[yt-dev] Zombie jobs on eudora?

Nathan Goldbaum nathan12343 at gmail.com
Wed Jun 11 16:37:45 PDT 2014


Deleting FLASHDataset.__del__ does seem to allow the dataset to be garbage
collected.  For some reason using super to call __del__ doesn't work.

That said, it seems that memory is still leaking, even though the datasets
are being garbage collected.  In particular, it looks like the FLASHGrid
objects are *not* being garbage collected.

After spending a bit of time looking at backref graphs (
http://i.imgur.com/zvAAYwA.png, http://i.imgur.com/4JIT5Qz.png), I ended up
trying the following patch:

http://paste.yt-project.org/show/4780/

With this patch I see substantially better memory performance.  Where
before peak memory usage was 1.3 GB after 20 outputs (and growing) now peak
memory usage is only 300 MB, with visible deallocations in htop after each
iteration of the loop.

Counterintuitively, *not* calling __del__ seems to substantially improve
memory performance.  In both cases I tried calling __del__ on the
superclass using super() but unfortunately I still saw memory leaks when I
did that.  I have no idea why.


On Wed, Jun 11, 2014 at 3:56 PM, Matthew Turk <matthewturk at gmail.com> wrote:

> That's fascinating. The superclass issues a bunch of deletes. Can you try
> either removing the del method (may leave dangling hdf5 files) or calling
> super() from it?
>  On Jun 11, 2014 5:54 PM, "Nathan Goldbaum" <nathan12343 at gmail.com> wrote:
>
>> So one difference relative to Enzo is that FLASHDataset implements
>> __del__ while EnzoDataset does not.  This seems to be the reason that
>> FLASHDataset objects are ending up in gc.garbage rather than being freed.
>>
>>
>> On Wed, Jun 11, 2014 at 3:48 PM, Matthew Turk <matthewturk at gmail.com>
>> wrote:
>>
>>> This shouldn't be preferentially affecting flash, though. I think it is
>>> a leftover from when we moved to unify index; when the hierarchy had the
>>> classes, pf was already a weakref.
>>>  On Jun 11, 2014 5:46 PM, "Nathan Goldbaum" <nathan12343 at gmail.com>
>>> wrote:
>>>
>>>> After noticing that the FLASHDataset objects seem to have
>>>> backreferences to the data object aliases that are attached to the dataset,
>>>> I made the following modification:
>>>>
>>>> diff -r d1de2160a4a8 yt/data_objects/static_output.py
>>>> --- a/yt/data_objects/static_output.py  Wed Jun 11 13:25:17 2014 -0700
>>>> +++ b/yt/data_objects/static_output.py  Wed Jun 11 15:42:56 2014 -0700
>>>> @@ -498,7 +498,7 @@
>>>>                  continue
>>>>              cname = cls.__name__
>>>>              if cname.endswith("Base"): cname = cname[:-4]
>>>> -            self._add_object_class(name, cname, cls, {'pf':self})
>>>> +            self._add_object_class(name, cname, cls,
>>>> {'pf':weakref.proxy(self)})
>>>>          if self.refine_by != 2 and hasattr(self, 'proj') and \
>>>>              hasattr(self, 'overlap_proj'):
>>>>              mylog.warning("Refine by something other than two:
>>>> reverting to"
>>>>
>>>> After doing so, I find that the FLASHDataset objects are ending up in
>>>> gc.garbage. The memory is still leaking, but now the garbage collector is
>>>> able to at least find the objects that are supposed to be collected.
>>>>
>>>> From my reading of the docs things will only end up in gc.garbage if
>>>> they have a __del__ method that doesn't actually free the object.  Any idea
>>>> what might be happening here?
>>>>
>>>> -Nathan
>>>>
>>>>
>>>> On Wed, Jun 11, 2014 at 3:37 PM, Matthew Turk <matthewturk at gmail.com>
>>>> wrote:
>>>>
>>>>> On Wed, Jun 11, 2014 at 5:28 PM, Nathan Goldbaum <
>>>>> nathan12343 at gmail.com> wrote:
>>>>> > On Wed, Jun 11, 2014 at 3:04 PM, Matthew Turk <matthewturk at gmail.com>
>>>>> wrote:
>>>>> >>
>>>>> >>
>>>>> >> Please do, yeah. That should help us track down the memory increase.
>>>>> >> It's possible FLASH needs additional work; have you checked the
>>>>> >> refcounts for its grid objects?
>>>>> >>
>>>>> >
>>>>> > With this script: http://paste.yt-project.org/show/4779/
>>>>> >
>>>>> > I also see steadily increasing memory usage, although it's not clear
>>>>> if
>>>>> > that's just because each successive Enzo dataset is larger than the
>>>>> one
>>>>> > before. The peak memory usage is 230 MB, so substantially better
>>>>> than the
>>>>> > FLASH dataset.
>>>>>
>>>>> My guess is there is a leak in flash. You can evaluate whether it is
>>>>> the data size by looking at what the memory use is if you just load the
>>>>> final one or the final ten.
>>>>>
>>>>> When I have a minute I will try to replicate here. I suspect it's not
>>>>> important that it's cylindrical.
>>>>>
>>>>> >
>>>>> > I can privately share the FLASH dataset Philip was originally having
>>>>> trouble
>>>>> > with if that will help.
>>>>> >
>>>>> > The root grid seems to have a refcount of 7. I'm not sure how many
>>>>> of those
>>>>> > references were generated by objgraph itself.
>>>>> >
>>>>> > _______________________________________________
>>>>> > yt-dev mailing list
>>>>> > yt-dev at lists.spacepope.org
>>>>> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>> >
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> yt-dev mailing list
>>>>> yt-dev at lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> yt-dev mailing list
>>>> yt-dev at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>
>>>>
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>
>>>
>>
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20140611/8bf2666c/attachment.html>


More information about the yt-dev mailing list