<div dir="ltr">This works. I've implemented it for FLASH and hopefully will have the rest of the frontends done before bed tonight.<div><br></div><div>Thanks everyone for your input with this!</div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Wed, Jun 11, 2014 at 9:21 PM, Matthew Turk <span dir="ltr"><<a href="mailto:matthewturk@gmail.com" target="_blank">matthewturk@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="HOEnZb"><div class="h5">On Wed, Jun 11, 2014 at 6:35 PM, Nathan Goldbaum <<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a>> wrote:<br>
> I've hackily tried a few things - unfortunately there doesn't seem to be a<br>
> way to do this without deleting FLASH's __del__ method. That's unfortunate,<br>
> since I don't see a way to cleanly replace what it does without refactoring<br>
> the FLASH frontend to always close handles immediately after it uses them.<br>
><br>
> FWIW, __del__ is defined several other places in yt:<br>
><br>
> $ grin 'def __del__'<br>
> ./yt/analysis_modules/halo_finding/rockstar/rockstar.py:<br>
> 277 : def __del__(self):<br>
> ./yt/analysis_modules/halo_merger_tree/enzofof_merger_tree.py:<br>
> 110 : def __del__(self):<br>
> ./yt/frontends/chombo/data_structures.py:<br>
> 274 : def __del__(self):<br>
> ./yt/frontends/fits/data_structures.py:<br>
> 651 : def __del__(self):<br>
> ./yt/frontends/flash/data_structures.py:<br>
> 387 : def __del__(self):<br>
> ./yt/frontends/pluto/data_structures.py:<br>
> 183 : def __del__(self):<br>
> ./yt/geometry/geometry_handler.py:<br>
> 66 : def __del__(self):<br>
> ./yt/geometry/grid_geometry_handler.py:<br>
> 62 : def __del__(self):<br>
> ./yt/utilities/parallel_tools/parallel_analysis_interface.py:<br>
> 621 : def __del__(self):<br>
> 1214 : def __del__(self):<br>
><br>
> The other frontends that handle hdf5 i/o like FLASH does will also likely<br>
> need to be fixed to avoid this behavior.<br>
><br>
> Another option would be to figure out what is triggering the reference<br>
> cycles and add weakref.proxy's as needed. I've just tried to do that but<br>
> can't seem to find the where the culprit references are.<br>
<br>
</div></div>This is what we used to do. I'm almost certain they are now there<br>
because of the index and index proxy.<br>
<br>
I'm wondering if maybe we should have a filehandle object. That's<br>
where most of the issues with __del__ happen, as you note above. So<br>
if we had that, we'd end up in a situation where *it* (which would<br>
*only* be referenced by the Dataset object) would have a finalizer but<br>
the Dataset wouldn't.<br>
<div class="HOEnZb"><div class="h5"><br>
><br>
> I'm going to stop working on this for now. I hope my scratching at this<br>
> today has been helpful.<br>
><br>
><br>
><br>
> On Wed, Jun 11, 2014 at 4:53 PM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>> wrote:<br>
>><br>
>> This is tricky because of the way handles are held onto by h5py, but maybe<br>
>> we can figure it out somehow.<br>
>><br>
>> On Jun 11, 2014 5:41 PM, "Nathan Goldbaum" <<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a>> wrote:<br>
>>><br>
>>> Oh I see. Objects that implement __del__ cannot be garbage collected if<br>
>>> they participate in a cycle. Full stop. This is fixed in python 3.4.<br>
>>><br>
>>><br>
>>> <a href="http://objgraph.readthedocs.org/en/latest/uncollectable.html?highlight=del#uncollectable-garbage" target="_blank">http://objgraph.readthedocs.org/en/latest/uncollectable.html?highlight=del#uncollectable-garbage</a><br>
>>><br>
>>><br>
>>> On Wed, Jun 11, 2014 at 4:37 PM, Nathan Goldbaum <<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a>><br>
>>> wrote:<br>
>>>><br>
>>>> Deleting FLASHDataset.__del__ does seem to allow the dataset to be<br>
>>>> garbage collected. For some reason using super to call __del__ doesn't<br>
>>>> work.<br>
>>>><br>
>>>> That said, it seems that memory is still leaking, even though the<br>
>>>> datasets are being garbage collected. In particular, it looks like the<br>
>>>> FLASHGrid objects are *not* being garbage collected.<br>
>>>><br>
>>>> After spending a bit of time looking at backref graphs<br>
>>>> (<a href="http://i.imgur.com/zvAAYwA.png" target="_blank">http://i.imgur.com/zvAAYwA.png</a>, <a href="http://i.imgur.com/4JIT5Qz.png" target="_blank">http://i.imgur.com/4JIT5Qz.png</a>), I ended up<br>
>>>> trying the following patch:<br>
>>>><br>
>>>> <a href="http://paste.yt-project.org/show/4780/" target="_blank">http://paste.yt-project.org/show/4780/</a><br>
>>>><br>
>>>> With this patch I see substantially better memory performance. Where<br>
>>>> before peak memory usage was 1.3 GB after 20 outputs (and growing) now peak<br>
>>>> memory usage is only 300 MB, with visible deallocations in htop after each<br>
>>>> iteration of the loop.<br>
>>>><br>
>>>> Counterintuitively, *not* calling __del__ seems to substantially improve<br>
>>>> memory performance. In both cases I tried calling __del__ on the superclass<br>
>>>> using super() but unfortunately I still saw memory leaks when I did that. I<br>
>>>> have no idea why.<br>
>>>><br>
>>>><br>
>>>> On Wed, Jun 11, 2014 at 3:56 PM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
>>>> wrote:<br>
>>>>><br>
>>>>> That's fascinating. The superclass issues a bunch of deletes. Can you<br>
>>>>> try either removing the del method (may leave dangling hdf5 files) or<br>
>>>>> calling super() from it?<br>
>>>>><br>
>>>>> On Jun 11, 2014 5:54 PM, "Nathan Goldbaum" <<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a>><br>
>>>>> wrote:<br>
>>>>>><br>
>>>>>> So one difference relative to Enzo is that FLASHDataset implements<br>
>>>>>> __del__ while EnzoDataset does not. This seems to be the reason that<br>
>>>>>> FLASHDataset objects are ending up in gc.garbage rather than being freed.<br>
>>>>>><br>
>>>>>><br>
>>>>>> On Wed, Jun 11, 2014 at 3:48 PM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
>>>>>> wrote:<br>
>>>>>>><br>
>>>>>>> This shouldn't be preferentially affecting flash, though. I think it<br>
>>>>>>> is a leftover from when we moved to unify index; when the hierarchy had the<br>
>>>>>>> classes, pf was already a weakref.<br>
>>>>>>><br>
>>>>>>> On Jun 11, 2014 5:46 PM, "Nathan Goldbaum" <<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a>><br>
>>>>>>> wrote:<br>
>>>>>>>><br>
>>>>>>>> After noticing that the FLASHDataset objects seem to have<br>
>>>>>>>> backreferences to the data object aliases that are attached to the dataset,<br>
>>>>>>>> I made the following modification:<br>
>>>>>>>><br>
>>>>>>>> diff -r d1de2160a4a8 yt/data_objects/static_output.py<br>
>>>>>>>> --- a/yt/data_objects/static_output.py Wed Jun 11 13:25:17 2014<br>
>>>>>>>> -0700<br>
>>>>>>>> +++ b/yt/data_objects/static_output.py Wed Jun 11 15:42:56 2014<br>
>>>>>>>> -0700<br>
>>>>>>>> @@ -498,7 +498,7 @@<br>
>>>>>>>> continue<br>
>>>>>>>> cname = cls.__name__<br>
>>>>>>>> if cname.endswith("Base"): cname = cname[:-4]<br>
>>>>>>>> - self._add_object_class(name, cname, cls, {'pf':self})<br>
>>>>>>>> + self._add_object_class(name, cname, cls,<br>
>>>>>>>> {'pf':weakref.proxy(self)})<br>
>>>>>>>> if self.refine_by != 2 and hasattr(self, 'proj') and \<br>
>>>>>>>> hasattr(self, 'overlap_proj'):<br>
>>>>>>>> mylog.warning("Refine by something other than two:<br>
>>>>>>>> reverting to"<br>
>>>>>>>><br>
>>>>>>>> After doing so, I find that the FLASHDataset objects are ending up<br>
>>>>>>>> in gc.garbage. The memory is still leaking, but now the garbage collector is<br>
>>>>>>>> able to at least find the objects that are supposed to be collected.<br>
>>>>>>>><br>
>>>>>>>> From my reading of the docs things will only end up in gc.garbage if<br>
>>>>>>>> they have a __del__ method that doesn't actually free the object. Any idea<br>
>>>>>>>> what might be happening here?<br>
>>>>>>>><br>
>>>>>>>> -Nathan<br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>>> On Wed, Jun 11, 2014 at 3:37 PM, Matthew Turk<br>
>>>>>>>> <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>> wrote:<br>
>>>>>>>>><br>
>>>>>>>>> On Wed, Jun 11, 2014 at 5:28 PM, Nathan Goldbaum<br>
>>>>>>>>> <<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a>> wrote:<br>
>>>>>>>>> > On Wed, Jun 11, 2014 at 3:04 PM, Matthew Turk<br>
>>>>>>>>> > <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>> wrote:<br>
>>>>>>>>> >><br>
>>>>>>>>> >><br>
>>>>>>>>> >> Please do, yeah. That should help us track down the memory<br>
>>>>>>>>> >> increase.<br>
>>>>>>>>> >> It's possible FLASH needs additional work; have you checked the<br>
>>>>>>>>> >> refcounts for its grid objects?<br>
>>>>>>>>> >><br>
>>>>>>>>> ><br>
>>>>>>>>> > With this script: <a href="http://paste.yt-project.org/show/4779/" target="_blank">http://paste.yt-project.org/show/4779/</a><br>
>>>>>>>>> ><br>
>>>>>>>>> > I also see steadily increasing memory usage, although it's not<br>
>>>>>>>>> > clear if<br>
>>>>>>>>> > that's just because each successive Enzo dataset is larger than<br>
>>>>>>>>> > the one<br>
>>>>>>>>> > before. The peak memory usage is 230 MB, so substantially better<br>
>>>>>>>>> > than the<br>
>>>>>>>>> > FLASH dataset.<br>
>>>>>>>>><br>
>>>>>>>>> My guess is there is a leak in flash. You can evaluate whether it<br>
>>>>>>>>> is the data size by looking at what the memory use is if you just load the<br>
>>>>>>>>> final one or the final ten.<br>
>>>>>>>>><br>
>>>>>>>>> When I have a minute I will try to replicate here. I suspect it's<br>
>>>>>>>>> not important that it's cylindrical.<br>
>>>>>>>>><br>
>>>>>>>>> ><br>
>>>>>>>>> > I can privately share the FLASH dataset Philip was originally<br>
>>>>>>>>> > having trouble<br>
>>>>>>>>> > with if that will help.<br>
>>>>>>>>> ><br>
>>>>>>>>> > The root grid seems to have a refcount of 7. I'm not sure how<br>
>>>>>>>>> > many of those<br>
>>>>>>>>> > references were generated by objgraph itself.<br>
>>>>>>>>> ><br>
>>>>>>>>> > _______________________________________________<br>
>>>>>>>>> > yt-dev mailing list<br>
>>>>>>>>> > <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>>>>>>>> > <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>>>>>>>>> ><br>
>>>>>>>>><br>
>>>>>>>>><br>
>>>>>>>>> _______________________________________________<br>
>>>>>>>>> yt-dev mailing list<br>
>>>>>>>>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>>>>>>>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>>>>>>>>><br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>>> _______________________________________________<br>
>>>>>>>> yt-dev mailing list<br>
>>>>>>>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>>>>>>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>>>>>>>><br>
>>>>>>><br>
>>>>>>> _______________________________________________<br>
>>>>>>> yt-dev mailing list<br>
>>>>>>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>>>>>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>> _______________________________________________<br>
>>>>>> yt-dev mailing list<br>
>>>>>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>>>>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>>>>>><br>
>>>>><br>
>>>>> _______________________________________________<br>
>>>>> yt-dev mailing list<br>
>>>>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>>>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>>>>><br>
>>>><br>
>>><br>
>>><br>
>>> _______________________________________________<br>
>>> yt-dev mailing list<br>
>>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>>><br>
>><br>
>> _______________________________________________<br>
>> yt-dev mailing list<br>
>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>><br>
><br>
><br>
> _______________________________________________<br>
> yt-dev mailing list<br>
> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
><br>
_______________________________________________<br>
yt-dev mailing list<br>
<a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
</div></div></blockquote></div><br></div>