[yt-dev] Zombie jobs on eudora?

Nathan Goldbaum nathan12343 at gmail.com
Wed Jun 11 15:45:43 PDT 2014


After noticing that the FLASHDataset objects seem to have backreferences to
the data object aliases that are attached to the dataset, I made the
following modification:

diff -r d1de2160a4a8 yt/data_objects/static_output.py
--- a/yt/data_objects/static_output.py  Wed Jun 11 13:25:17 2014 -0700
+++ b/yt/data_objects/static_output.py  Wed Jun 11 15:42:56 2014 -0700
@@ -498,7 +498,7 @@
                 continue
             cname = cls.__name__
             if cname.endswith("Base"): cname = cname[:-4]
-            self._add_object_class(name, cname, cls, {'pf':self})
+            self._add_object_class(name, cname, cls,
{'pf':weakref.proxy(self)})
         if self.refine_by != 2 and hasattr(self, 'proj') and \
             hasattr(self, 'overlap_proj'):
             mylog.warning("Refine by something other than two: reverting
to"

After doing so, I find that the FLASHDataset objects are ending up in
gc.garbage. The memory is still leaking, but now the garbage collector is
able to at least find the objects that are supposed to be collected.

>From my reading of the docs things will only end up in gc.garbage if they
have a __del__ method that doesn't actually free the object.  Any idea what
might be happening here?

-Nathan


On Wed, Jun 11, 2014 at 3:37 PM, Matthew Turk <matthewturk at gmail.com> wrote:

> On Wed, Jun 11, 2014 at 5:28 PM, Nathan Goldbaum <nathan12343 at gmail.com>
> wrote:
> > On Wed, Jun 11, 2014 at 3:04 PM, Matthew Turk <matthewturk at gmail.com>
> wrote:
> >>
> >>
> >> Please do, yeah. That should help us track down the memory increase.
> >> It's possible FLASH needs additional work; have you checked the
> >> refcounts for its grid objects?
> >>
> >
> > With this script: http://paste.yt-project.org/show/4779/
> >
> > I also see steadily increasing memory usage, although it's not clear if
> > that's just because each successive Enzo dataset is larger than the one
> > before. The peak memory usage is 230 MB, so substantially better than the
> > FLASH dataset.
>
> My guess is there is a leak in flash. You can evaluate whether it is the
> data size by looking at what the memory use is if you just load the final
> one or the final ten.
>
> When I have a minute I will try to replicate here. I suspect it's not
> important that it's cylindrical.
>
> >
> > I can privately share the FLASH dataset Philip was originally having
> trouble
> > with if that will help.
> >
> > The root grid seems to have a refcount of 7. I'm not sure how many of
> those
> > references were generated by objgraph itself.
> >
> > _______________________________________________
> > yt-dev mailing list
> > yt-dev at lists.spacepope.org
> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >
>
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20140611/dc46d21a/attachment.html>


More information about the yt-dev mailing list