[yt-users] How to efficiently control the memory in yt with large simulation?

[Junhwan Choi (최준환)]

Thank you Matt and Romain,

Now, I understand it is a more fundamental issue in the yt with
Ramses, and I need to wait yt to be updated.
Meanwhile, is there any way I can access small fraction of the
simulation data (such as thin slice of the data) and make some
visualization plots such as thin projection and slice?
If so, I can reduce the required memory.
Is it possible?

Thank you,
Junhwan


On Fri, Dec 6, 2013 at 8:56 AM, Matthew Turk <matthewturk at gmail.com> wrote:
> On Fri, Dec 6, 2013 at 9:54 AM, Romain Teyssier
> <romain.teyssier at gmail.com> wrote:
>> One possibility would be to define a bounding box that yt uses to upload
>> ramses data only when contained in this bounding box.
>> This can be done efficiently using the Hilbert key.
>>
>> Romain
>
> That would be good, and we could probably port the code Doug Rudd
> wrote to use SFC logic for detecting which zones to parse over to
> RAMSES, as well.
>
> Today is out for me on this, because of other things, but with this I
> think we might be able to make it work in the near future, like
> mid-next week, assuming all goes well otherwise.
>
> -Matt
>
>>
>> On 06 Dec 2013, at 15:49, Matthew Turk <matthewturk at gmail.com> wrote:
>>
>> Hi Junhwan,
>>
>> There's some more background on this issue here:
>>
>> http://lists.spacepope.org/pipermail/yt-users-spacepope.org/2013-November/004265.html
>>
>> Basically, what it amounts to is:
>>
>> * Right now RAMSES uses too much memory and duplicates the full mesh
>> on every processor
>> * This is not how it will always be
>> * Unfortunately changing this can't be prioritized in the next couple weeks
>>
>> Your mesh is *particularly* large for what we've dealt with before.
>> As it stands, I would be surprised if it will work.  The fix for this
>> would be to change the Octrees to only parse on demand rather than at
>> instantiation of the RAMSESHierarchy.  Sam Geen and I have talked a
>> bit about this, and he may be interested in working on it.  It's a
>> change that also will happen for N-body datasets (differently) and
>> it's definitely planned to come, but it isn't being prioritized at
>> this very moment because of other pressing concerns.
>>
>> On Thu, Dec 5, 2013 at 10:47 PM, Junhwan Choi (최준환)
>> <choi.junhwan at gmail.com> wrote:
>>
>> Hi all,
>>
>> I try to make some visualizations (density/temperature project) for
>> large simulation with yt.
>> The simulation is 4096^3 unigrid Ramses simulation.
>>
>> I try to implement the basics density and temperature projection plot
>> with following script and I got memory problem and yt run is crashed.
>> =====
>> from yt.mods import *
>>
>> ds = load("../output_00048/info_00048.txt", fields =
>> ["Density","x-velocity",
>> "y-velocity","z-velocity","Pressure","Metallicity","Rad"])
>> center = [0., 0., 0.]
>>
>> pw = ProjectionPlot(ds, "x", ("gas", "Density"),
>> weight_field="Density", center=center)
>> pw.zoom(1.01)
>> pw.save("allviewGas")
>> pw = ProjectionPlot(ds, "y", ("gas", "Density"),
>> weight_field="Density", center=center)
>> pw.zoom(1.01)
>> pw.save("allviewGas")
>> pw = ProjectionPlot(ds, "z", ("gas", "Density") ,
>> weight_field="Density", center=center)
>> pw.zoom(1.01)
>> pw.save("allviewGas")
>>
>> pw = ProjectionPlot(ds, "x", "Temperature", weight_field="Density",
>> center=center)
>> pw.zoom(1.01)
>> pw.save("allviewGas")
>> pw = ProjectionPlot(ds, "y", "Temperature", weight_field="Density",
>> center=center)
>> pw.zoom(1.01)
>> pw.save("allviewGas")
>> pw = ProjectionPlot(ds, "z", "Temperature" , weight_field="Density",
>> center=center)
>> pw.zoom(1.01)
>> pw.save("allviewGas")
>> ====
>>
>> Is there any way to reduce the memory usage when I make visualization?
>> If I use slice instead of projection, can I save the memory?
>> I saw there is parallelization for yt. Does parallelization distribute
>> the data at the beginning of the read?
>> (But, I prefer to do so w/o parallelization at this moment.)
>>
>>
>> In principle, yes, slices will considerably reduce the memory, modulo
>> the overhead from having all your octrees in memory at once -- which
>> will be considerable.
>>
>> I will spend some time thinking if there's a hotfix we can apply to
>> make this work for you right now, as is, but I suspect it may be a
>> little time until it can be properly implemented.
>>
>> -Matt
>>
>>
>> Thank you in advance,
>> Junhwan
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>>
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org