[yt-users] RAMSES multiple sims memory issue

[Nathan Goldbaum]

That does help. Like I said, I don't have access to any Ramses time series
datasets so I can't reproduce this behavior in practice, making it
difficult for me to debug.

If you can create a small Ramses time series dataset to debug with, that
will help here.

Nathan

On Mon, Jan 2, 2017 at 5:05 AM Thor Andreas Seiff Ellewsen <
tellewsen at gmail.com> wrote:

> Thank you for the great response.
>
> Unfortunately I'm not able to share the data at this point.
> I tried switching from using a timeseries to using the path to each output
> folder as you suggested.
>
> It seems that even if I strip my code down to the bare essentials where I
> just load each sim in a for loop and load the field list, it keeps every
> sim in memory as long as the for loop is still running. It does drop them
> all at the end of the loop but I would like each to be dropped when the
> next one loads.
>
> However, if I try saving a projection plot, for each run through the loop,
> it does not drop the simulations when the loop finishes.
> By saving I mean both saving the plot to disk and keeping the figure
> object in an array for later use.
>
> Does this help at all?
>
>
> The axis names I was referring to was the axes on the ProfilePlot object,
> not the ProjectionPlot one. Specifically the 1D profile one with a simple x
> and y axis. Could you please point me in the right direction?
>
>
> Best,
> Andreas Ellewsen
>
> On Wed, 14 Dec 2016 at 16:18 Nathan Goldbaum <nathan12343 at gmail.com>
> wrote:
>
> On Wed, Dec 14, 2016 at 5:30 AM, Thor Andreas Seiff Ellewsen <
> tellewsen at gmail.com> wrote:
>
> Hi all,
>
> I'm working on a project where I'm comparing different RAMSES simulations.
> However I'm running into a problem with memory.
> It seems that yt loads the whole simulation into memory even though I'm
> only interested in a number of dark matter halos which I have the
> coordinates for.
>
>
> Yes, this is a limitation of our current approach for octree data, we must
> build a global octree for indexing purposes. For large simulations this can
> be a prohibitive memory cost. We have long-term plans to fix this but as
> far as I know no one is actively working on it.
>
>
>
> It also seems like it doesn't drop the first simulation out of memory when
> I'm done making the figures I want for that one.
>
>
> This quickly fills up the memory.
>
> You can find the relevant parts of my code at the pastebin at
> http://paste.yt/p4946.html
> I'm running yt version 3.3.2
>
> What am I doing wrong?
>
>
> In principle this shouldn't happen but it's entirely possible there is a
> bug somewhere. In the python memory model, an object is retained in memory
> until no other object have a reference to it, at which point it is garbage
> collected. I don't see anything in your script that would cause references
> to old datasets to be retained, so it might be the case that yt is
> inadvertantly retaining a reference to datasets in a ramses time series
> when it shouldn't be or if we need the reference that we should be using a
> weakref. Unfortunately it looks like we don't have any Ramses time series
> datasets available in our repository of public test datasets (
> http://yt-project.org/data/) so I don't have a way of reproducing this
> behavior. Is there any chance you can make a ramses time series available
> for testing? You might find the yt curldrop to be useful for sharing a
> large dataset:
>
> https://docs.hub.yt/services.html#curldrop
>
> A test that this is isolated to the time series object would be to instead
> use a loop over filenames, loading a single filename in each loop
> iteration, and then explicitly deleting the dataset object using the `del`
> statement at the end of the loop. In principle this shouldn't be necessary
> but it might help reveal where the issue is localized to.
>
>
>
> Also, is there a way to change the axis names?
> I'm making some dark matter density profiles using the
> ('deposit','io_density') field and this gives IO Density on the y axis, but
> I just want it to say Dark matter density so my readers actually understand
> what I'm talking about.
>
>
> For the ProjectionPlot you can use the set_colorbar_label method:
>
>
> http://yt-project.org/doc/reference/api/generated/yt.visualization.plot_window.ProjectionPlot.set_colorbar_label.html#yt.visualization.plot_window.ProjectionPlot.set_colorbar_label
>
>
>
> Thanks for helping
>
> Best,
> Andreas Ellewsen
>
>
>
>
>
> _______________________________________________
>
>
> yt-users mailing list
>
>
> yt-users at lists.spacepope.org
>
>
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
>
>
>
>
> _______________________________________________
>
>
> yt-users mailing list
>
>
> yt-users at lists.spacepope.org
>
>
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
>
>
> _______________________________________________
>
> yt-users mailing list
>
> yt-users at lists.spacepope.org
>
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20170102/9d58ab80/attachment.html>