[yt-dev] Zombie jobs on eudora?

Matthew Turk matthewturk at gmail.com
Wed Jun 11 15:56:39 PDT 2014


That's fascinating. The superclass issues a bunch of deletes. Can you try
either removing the del method (may leave dangling hdf5 files) or calling
super() from it?
On Jun 11, 2014 5:54 PM, "Nathan Goldbaum" <nathan12343 at gmail.com> wrote:

> So one difference relative to Enzo is that FLASHDataset implements __del__
> while EnzoDataset does not.  This seems to be the reason that FLASHDataset
> objects are ending up in gc.garbage rather than being freed.
>
>
> On Wed, Jun 11, 2014 at 3:48 PM, Matthew Turk <matthewturk at gmail.com>
> wrote:
>
>> This shouldn't be preferentially affecting flash, though. I think it is a
>> leftover from when we moved to unify index; when the hierarchy had the
>> classes, pf was already a weakref.
>>  On Jun 11, 2014 5:46 PM, "Nathan Goldbaum" <nathan12343 at gmail.com>
>> wrote:
>>
>>> After noticing that the FLASHDataset objects seem to have backreferences
>>> to the data object aliases that are attached to the dataset, I made the
>>> following modification:
>>>
>>> diff -r d1de2160a4a8 yt/data_objects/static_output.py
>>> --- a/yt/data_objects/static_output.py  Wed Jun 11 13:25:17 2014 -0700
>>> +++ b/yt/data_objects/static_output.py  Wed Jun 11 15:42:56 2014 -0700
>>> @@ -498,7 +498,7 @@
>>>                  continue
>>>              cname = cls.__name__
>>>              if cname.endswith("Base"): cname = cname[:-4]
>>> -            self._add_object_class(name, cname, cls, {'pf':self})
>>> +            self._add_object_class(name, cname, cls,
>>> {'pf':weakref.proxy(self)})
>>>          if self.refine_by != 2 and hasattr(self, 'proj') and \
>>>              hasattr(self, 'overlap_proj'):
>>>              mylog.warning("Refine by something other than two:
>>> reverting to"
>>>
>>> After doing so, I find that the FLASHDataset objects are ending up in
>>> gc.garbage. The memory is still leaking, but now the garbage collector is
>>> able to at least find the objects that are supposed to be collected.
>>>
>>> From my reading of the docs things will only end up in gc.garbage if
>>> they have a __del__ method that doesn't actually free the object.  Any idea
>>> what might be happening here?
>>>
>>> -Nathan
>>>
>>>
>>> On Wed, Jun 11, 2014 at 3:37 PM, Matthew Turk <matthewturk at gmail.com>
>>> wrote:
>>>
>>>> On Wed, Jun 11, 2014 at 5:28 PM, Nathan Goldbaum <nathan12343 at gmail.com>
>>>> wrote:
>>>> > On Wed, Jun 11, 2014 at 3:04 PM, Matthew Turk <matthewturk at gmail.com>
>>>> wrote:
>>>> >>
>>>> >>
>>>> >> Please do, yeah. That should help us track down the memory increase.
>>>> >> It's possible FLASH needs additional work; have you checked the
>>>> >> refcounts for its grid objects?
>>>> >>
>>>> >
>>>> > With this script: http://paste.yt-project.org/show/4779/
>>>> >
>>>> > I also see steadily increasing memory usage, although it's not clear
>>>> if
>>>> > that's just because each successive Enzo dataset is larger than the
>>>> one
>>>> > before. The peak memory usage is 230 MB, so substantially better than
>>>> the
>>>> > FLASH dataset.
>>>>
>>>> My guess is there is a leak in flash. You can evaluate whether it is
>>>> the data size by looking at what the memory use is if you just load the
>>>> final one or the final ten.
>>>>
>>>> When I have a minute I will try to replicate here. I suspect it's not
>>>> important that it's cylindrical.
>>>>
>>>> >
>>>> > I can privately share the FLASH dataset Philip was originally having
>>>> trouble
>>>> > with if that will help.
>>>> >
>>>> > The root grid seems to have a refcount of 7. I'm not sure how many of
>>>> those
>>>> > references were generated by objgraph itself.
>>>> >
>>>> > _______________________________________________
>>>> > yt-dev mailing list
>>>> > yt-dev at lists.spacepope.org
>>>> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>> >
>>>>
>>>>
>>>> _______________________________________________
>>>> yt-dev mailing list
>>>> yt-dev at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>
>>>>
>>>
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>
>>>
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>>
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20140611/8d5004b8/attachment.html>


More information about the yt-dev mailing list