<p dir="ltr">That's fascinating. The superclass issues a bunch of deletes. Can you try either removing the del method (may leave dangling hdf5 files) or calling super() from it?</p>
<div class="gmail_quote">On Jun 11, 2014 5:54 PM, "Nathan Goldbaum" <<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">So one difference relative to Enzo is that FLASHDataset implements __del__ while EnzoDataset does not. This seems to be the reason that FLASHDataset objects are ending up in gc.garbage rather than being freed.<br>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jun 11, 2014 at 3:48 PM, Matthew Turk <span dir="ltr"><<a href="mailto:matthewturk@gmail.com" target="_blank">matthewturk@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">This shouldn't be preferentially affecting flash, though. I think it is a leftover from when we moved to unify index; when the hierarchy had the classes, pf was already a weakref.</p>
<div><div>
<div class="gmail_quote">On Jun 11, 2014 5:46 PM, "Nathan Goldbaum" <<a href="mailto:nathan12343@gmail.com" target="_blank">nathan12343@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div>After noticing that the FLASHDataset objects seem to have backreferences to the data object aliases that are attached to the dataset, I made the following modification:</div><div><br></div><div><div>diff -r d1de2160a4a8 yt/data_objects/static_output.py</div>
<div>--- a/yt/data_objects/static_output.py Wed Jun 11 13:25:17 2014 -0700</div><div>+++ b/yt/data_objects/static_output.py Wed Jun 11 15:42:56 2014 -0700</div><div>@@ -498,7 +498,7 @@</div><div> continue</div>
<div> cname = cls.__name__</div><div> if cname.endswith("Base"): cname = cname[:-4]</div><div>- self._add_object_class(name, cname, cls, {'pf':self})</div><div>+ self._add_object_class(name, cname, cls, {'pf':weakref.proxy(self)})</div>
<div> if self.refine_by != 2 and hasattr(self, 'proj') and \</div><div> hasattr(self, 'overlap_proj'):</div><div> mylog.warning("Refine by something other than two: reverting to"</div>
</div><div><br></div><div>After doing so, I find that the FLASHDataset objects are ending up in gc.garbage. The memory is still leaking, but now the garbage collector is able to at least find the objects that are supposed to be collected.</div>
<div><br></div><div>From my reading of the docs things will only end up in gc.garbage if they have a __del__ method that doesn't actually free the object. Any idea what might be happening here?</div><div><br></div><div>
-Nathan</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jun 11, 2014 at 3:37 PM, Matthew Turk <span dir="ltr"><<a href="mailto:matthewturk@gmail.com" target="_blank">matthewturk@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><p dir="ltr"></p>
<p dir="ltr">On Wed, Jun 11, 2014 at 5:28 PM, Nathan Goldbaum <<a href="mailto:nathan12343@gmail.com" target="_blank">nathan12343@gmail.com</a>> wrote:<br>
> On Wed, Jun 11, 2014 at 3:04 PM, Matthew Turk <<a href="mailto:matthewturk@gmail.com" target="_blank">matthewturk@gmail.com</a>> wrote:<br>
>><br>
>><br>
>> Please do, yeah. That should help us track down the memory increase.<br>
>> It's possible FLASH needs additional work; have you checked the<br>
>> refcounts for its grid objects?<br>
>><br>
><br>
> With this script: <a href="http://paste.yt-project.org/show/4779/" target="_blank">http://paste.yt-project.org/show/4779/</a><br>
><br>
> I also see steadily increasing memory usage, although it's not clear if<br>
> that's just because each successive Enzo dataset is larger than the one<br>
> before. The peak memory usage is 230 MB, so substantially better than the<br>
> FLASH dataset.</p>
</div><p dir="ltr">My guess is there is a leak in flash. You can evaluate whether it is the data size by looking at what the memory use is if you just load the final one or the final ten.</p>
<p dir="ltr">When I have a minute I will try to replicate here. I suspect it's not important that it's cylindrical.</p>
<p dir="ltr"></p><div>><br>
> I can privately share the FLASH dataset Philip was originally having trouble<br>
> with if that will help.<br>
><br>
> The root grid seems to have a refcount of 7. I'm not sure how many of those<br>
> references were generated by objgraph itself.<br>
><br></div><div>
> _______________________________________________<br>
> yt-dev mailing list<br>
> <a href="mailto:yt-dev@lists.spacepope.org" target="_blank">yt-dev@lists.spacepope.org</a><br>
> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
></div><p></p>
<br>_______________________________________________<br>
yt-dev mailing list<br>
<a href="mailto:yt-dev@lists.spacepope.org" target="_blank">yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
<br></blockquote></div><br></div>
<br>_______________________________________________<br>
yt-dev mailing list<br>
<a href="mailto:yt-dev@lists.spacepope.org" target="_blank">yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
<br></blockquote></div>
</div></div><br>_______________________________________________<br>
yt-dev mailing list<br>
<a href="mailto:yt-dev@lists.spacepope.org" target="_blank">yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
<br></blockquote></div><br></div></div>
<br>_______________________________________________<br>
yt-dev mailing list<br>
<a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
<br></blockquote></div>