[yt-dev] Zombie jobs on eudora?

Nathan Goldbaum nathan12343 at gmail.com
Wed Jun 11 15:54:19 PDT 2014


So one difference relative to Enzo is that FLASHDataset implements __del__
while EnzoDataset does not.  This seems to be the reason that FLASHDataset
objects are ending up in gc.garbage rather than being freed.


On Wed, Jun 11, 2014 at 3:48 PM, Matthew Turk <matthewturk at gmail.com> wrote:

> This shouldn't be preferentially affecting flash, though. I think it is a
> leftover from when we moved to unify index; when the hierarchy had the
> classes, pf was already a weakref.
>  On Jun 11, 2014 5:46 PM, "Nathan Goldbaum" <nathan12343 at gmail.com> wrote:
>
>> After noticing that the FLASHDataset objects seem to have backreferences
>> to the data object aliases that are attached to the dataset, I made the
>> following modification:
>>
>> diff -r d1de2160a4a8 yt/data_objects/static_output.py
>> --- a/yt/data_objects/static_output.py  Wed Jun 11 13:25:17 2014 -0700
>> +++ b/yt/data_objects/static_output.py  Wed Jun 11 15:42:56 2014 -0700
>> @@ -498,7 +498,7 @@
>>                  continue
>>              cname = cls.__name__
>>              if cname.endswith("Base"): cname = cname[:-4]
>> -            self._add_object_class(name, cname, cls, {'pf':self})
>> +            self._add_object_class(name, cname, cls,
>> {'pf':weakref.proxy(self)})
>>          if self.refine_by != 2 and hasattr(self, 'proj') and \
>>              hasattr(self, 'overlap_proj'):
>>              mylog.warning("Refine by something other than two: reverting
>> to"
>>
>> After doing so, I find that the FLASHDataset objects are ending up in
>> gc.garbage. The memory is still leaking, but now the garbage collector is
>> able to at least find the objects that are supposed to be collected.
>>
>> From my reading of the docs things will only end up in gc.garbage if they
>> have a __del__ method that doesn't actually free the object.  Any idea what
>> might be happening here?
>>
>> -Nathan
>>
>>
>> On Wed, Jun 11, 2014 at 3:37 PM, Matthew Turk <matthewturk at gmail.com>
>> wrote:
>>
>>> On Wed, Jun 11, 2014 at 5:28 PM, Nathan Goldbaum <nathan12343 at gmail.com>
>>> wrote:
>>> > On Wed, Jun 11, 2014 at 3:04 PM, Matthew Turk <matthewturk at gmail.com>
>>> wrote:
>>> >>
>>> >>
>>> >> Please do, yeah. That should help us track down the memory increase.
>>> >> It's possible FLASH needs additional work; have you checked the
>>> >> refcounts for its grid objects?
>>> >>
>>> >
>>> > With this script: http://paste.yt-project.org/show/4779/
>>> >
>>> > I also see steadily increasing memory usage, although it's not clear if
>>> > that's just because each successive Enzo dataset is larger than the one
>>> > before. The peak memory usage is 230 MB, so substantially better than
>>> the
>>> > FLASH dataset.
>>>
>>> My guess is there is a leak in flash. You can evaluate whether it is the
>>> data size by looking at what the memory use is if you just load the final
>>> one or the final ten.
>>>
>>> When I have a minute I will try to replicate here. I suspect it's not
>>> important that it's cylindrical.
>>>
>>> >
>>> > I can privately share the FLASH dataset Philip was originally having
>>> trouble
>>> > with if that will help.
>>> >
>>> > The root grid seems to have a refcount of 7. I'm not sure how many of
>>> those
>>> > references were generated by objgraph itself.
>>> >
>>> > _______________________________________________
>>> > yt-dev mailing list
>>> > yt-dev at lists.spacepope.org
>>> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> >
>>>
>>>
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>
>>>
>>
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20140611/39c5583a/attachment.htm>


More information about the yt-dev mailing list