[yt-users] RAMSES multiple sims memory issue

Thor Andreas Seiff Ellewsen tellewsen at gmail.com
Mon Jan 2 03:04:57 PST 2017


Thank you for the great response.

Unfortunately I'm not able to share the data at this point.
I tried switching from using a timeseries to using the path to each output
folder as you suggested.

It seems that even if I strip my code down to the bare essentials where I
just load each sim in a for loop and load the field list, it keeps every
sim in memory as long as the for loop is still running. It does drop them
all at the end of the loop but I would like each to be dropped when the
next one loads.

However, if I try saving a projection plot, for each run through the loop,
it does not drop the simulations when the loop finishes.
By saving I mean both saving the plot to disk and keeping the figure object
in an array for later use.

Does this help at all?


The axis names I was referring to was the axes on the ProfilePlot object,
not the ProjectionPlot one. Specifically the 1D profile one with a simple x
and y axis. Could you please point me in the right direction?


Best,
Andreas Ellewsen

On Wed, 14 Dec 2016 at 16:18 Nathan Goldbaum <nathan12343 at gmail.com> wrote:

On Wed, Dec 14, 2016 at 5:30 AM, Thor Andreas Seiff Ellewsen <
tellewsen at gmail.com> wrote:

Hi all,

I'm working on a project where I'm comparing different RAMSES simulations.
However I'm running into a problem with memory.
It seems that yt loads the whole simulation into memory even though I'm
only interested in a number of dark matter halos which I have the
coordinates for.


Yes, this is a limitation of our current approach for octree data, we must
build a global octree for indexing purposes. For large simulations this can
be a prohibitive memory cost. We have long-term plans to fix this but as
far as I know no one is actively working on it.



It also seems like it doesn't drop the first simulation out of memory when
I'm done making the figures I want for that one.


This quickly fills up the memory.

You can find the relevant parts of my code at the pastebin at
http://paste.yt/p4946.html
I'm running yt version 3.3.2

What am I doing wrong?


In principle this shouldn't happen but it's entirely possible there is a
bug somewhere. In the python memory model, an object is retained in memory
until no other object have a reference to it, at which point it is garbage
collected. I don't see anything in your script that would cause references
to old datasets to be retained, so it might be the case that yt is
inadvertantly retaining a reference to datasets in a ramses time series
when it shouldn't be or if we need the reference that we should be using a
weakref. Unfortunately it looks like we don't have any Ramses time series
datasets available in our repository of public test datasets (
http://yt-project.org/data/) so I don't have a way of reproducing this
behavior. Is there any chance you can make a ramses time series available
for testing? You might find the yt curldrop to be useful for sharing a
large dataset:

https://docs.hub.yt/services.html#curldrop

A test that this is isolated to the time series object would be to instead
use a loop over filenames, loading a single filename in each loop
iteration, and then explicitly deleting the dataset object using the `del`
statement at the end of the loop. In principle this shouldn't be necessary
but it might help reveal where the issue is localized to.



Also, is there a way to change the axis names?
I'm making some dark matter density profiles using the
('deposit','io_density') field and this gives IO Density on the y axis, but
I just want it to say Dark matter density so my readers actually understand
what I'm talking about.


For the ProjectionPlot you can use the set_colorbar_label method:

http://yt-project.org/doc/reference/api/generated/yt.visualization.plot_window.ProjectionPlot.set_colorbar_label.html#yt.visualization.plot_window.ProjectionPlot.set_colorbar_label



Thanks for helping

Best,
Andreas Ellewsen



_______________________________________________
yt-users mailing list
yt-users at lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org


_______________________________________________
yt-users mailing list
yt-users at lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20170102/783e911e/attachment.htm>


More information about the yt-users mailing list