<div dir="ltr">For the leaking YTArrays, Kacper suggested the following patch on IRC:<div><br></div><div><a href="http://bpaste.net/show/361120/">http://bpaste.net/show/361120/</a><br></div><div><br></div><div>This works for FLASH but seems to break field detection for enzo.</div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Jun 10, 2014 at 8:47 PM, Matthew Turk <span dir="ltr"><<a href="mailto:matthewturk@gmail.com" target="_blank">matthewturk@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Nathan,<br>
<br>
I believe there are two things at work here.<br>
<br>
1) (I do not have high confidence of this one.) YTArrays that are<br>
referenced with a .d and turned into numpy arrays which no longer *own<br>
the data* may be retaining a reference, but that reference doesn't get<br>
freed later. This happens often when we are doing things in the<br>
hierarchy instantiation phase. I haven't been able to figure out<br>
which references get lost; for me, over 40 outputs, I lost 1560. I<br>
think it's 39 YTArrays per hierarchy. This might also be related to<br>
field detection. I think this is not a substantial contributor.<br>
2) For some reason, when the .grids attribute (object array) is<br>
deleted on an index, the refcounts of those grids don't decrease. I<br>
am able to decrease their refcounts by manually setting<br>
pf.index.grids[:] = None. This eliminated all retained grid<br>
references.<br>
<br>
So, I think the root is that at some point, because of circular<br>
references or whatever, the finalizer isn't being called on the<br>
Gridndex (or on Index itself). This results in the reference to the<br>
grids array being kept, which then pumps up the lost object count. I<br>
don't know why it's not getting called (it's not guaranteed to be<br>
called, in any event).<br>
<br>
I have to take care of some other things (including Brendan's note<br>
about the memory problems with particle datasets) but I am pretty sure<br>
this is the root.<br>
<br>
-Matt<br>
<div class="HOEnZb"><div class="h5"><br>
On Tue, Jun 10, 2014 at 10:13 PM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>> wrote:<br>
> Hi Nathan,<br>
><br>
> All it requires is a call to .index; you don't need to do anything<br>
> else to get it to lose references.<br>
><br>
> I'm still looking into it.<br>
><br>
> -Matt<br>
><br>
> On Tue, Jun 10, 2014 at 9:26 PM, Nathan Goldbaum <<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a>> wrote:<br>
>><br>
>><br>
>><br>
>> On Tue, Jun 10, 2014 at 10:59 AM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
>> wrote:<br>
>>><br>
>>> Do you have a reproducible script?<br>
>><br>
>><br>
>> This should do the trick: <a href="http://paste.yt-project.org/show/4767/" target="_blank">http://paste.yt-project.org/show/4767/</a><br>
>><br>
>> (this is with an enzo dataset by the way)<br>
>><br>
>> That script prints (on my machine):<br>
>><br>
>> EnzoGrid 15065<br>
>> YTArray 1520<br>
>> list 704<br>
>> dict 2<br>
>> MaskedArray 1<br>
>><br>
>> Which indicates that 15000 EnzoGrid objects and 1520 YTArray objects have<br>
>> leaked.<br>
>><br>
>> The list I'm printing out at the end of the script should be the objects<br>
>> that leaked during the loop over the Enzo dataset. The<br>
>> objgraph.get_leaking_objects() function returns the list of all objects<br>
>> being tracked by the garbage collector that have no references but still<br>
>> have nonzero refcounts.<br>
>><br>
>> This means the "original_leaks" list isn't necessarily a list of leaky<br>
>> objects - most of the things in there are singletons that the interpreter<br>
>> keeps around. To create a list of leaky objects produced by iterating over<br>
>> the loop I take the set difference of the output of get_leaking_objects()<br>
>> before and after iterating over the dataset.<br>
>><br>
>>><br>
>>> If you make a bunch of symlinks to<br>
>>> one flash file and load them all in sequence, does that replicate the<br>
>>> behavior?<br>
>><br>
>><br>
>> Yes, it seems to. Compare the output of this script:<br>
>> <a href="http://paste.yt-project.org/show/4768/" target="_blank">http://paste.yt-project.org/show/4768/</a><br>
>><br>
>> Adjust the range of the for loop from 0 to 5 - creating the needed symlinks<br>
>> to WindTunnel/windtunnel_4lev_hdf5_plt_cnt_0040 as needed.<br>
>><br>
>>><br>
>>><br>
>>> On Tue, Jun 10, 2014 at 12:57 PM, Nathan Goldbaum <<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a>><br>
>>> wrote:<br>
>>> ><br>
>>> ><br>
>>> ><br>
>>> > On Tue, Jun 10, 2014 at 10:45 AM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
>>> > wrote:<br>
>>> >><br>
>>> >> Hi Nathan,<br>
>>> >><br>
>>> >> On Tue, Jun 10, 2014 at 12:43 PM, Nathan Goldbaum<br>
>>> >> <<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a>><br>
>>> >> wrote:<br>
>>> >> ><br>
>>> >> ><br>
>>> >> ><br>
>>> >> > On Tue, Jun 10, 2014 at 6:09 AM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
>>> >> > wrote:<br>
>>> >> >><br>
>>> >> >> Hi Nathan,<br>
>>> >> >><br>
>>> >> >> On Mon, Jun 9, 2014 at 11:02 PM, Nathan Goldbaum<br>
>>> >> >> <<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a>><br>
>>> >> >> wrote:<br>
>>> >> >> > Hey all,<br>
>>> >> >> ><br>
>>> >> >> > I'm looking at a memory leak that Philip (cc'd) is seeing when<br>
>>> >> >> > iterating<br>
>>> >> >> > over a long list of FLASH datasets. Just as an example of the<br>
>>> >> >> > type<br>
>>> >> >> > of<br>
>>> >> >> > behavior he is seeing - today he left his script running and ended<br>
>>> >> >> > up<br>
>>> >> >> > consuming 300 GB of RAM on a viz node.<br>
>>> >> >> ><br>
>>> >> >> > FWIW, the dataset is not particularly large - ~300 outputs and<br>
>>> >> >> > ~100<br>
>>> >> >> > MB<br>
>>> >> >> > per<br>
>>> >> >> > output. These are also FLASH cylindrical coordinate simulations -<br>
>>> >> >> > so<br>
>>> >> >> > perhaps<br>
>>> >> >> > this behavior will only occur in curvilinear geometries?<br>
>>> >> >><br>
>>> >> >> Hm, I don't know about that.<br>
>>> >> >><br>
>>> >> >> ><br>
>>> >> >> > I've been playing with objgraph to try to understand what's<br>
>>> >> >> > happening.<br>
>>> >> >> > Here's the script I've been using:<br>
>>> >> >> > <a href="http://paste.yt-project.org/show/4762/" target="_blank">http://paste.yt-project.org/show/4762/</a><br>
>>> >> >> ><br>
>>> >> >> > Here's the output after one iteration of the for loop:<br>
>>> >> >> > <a href="http://paste.yt-project.org/show/4761/" target="_blank">http://paste.yt-project.org/show/4761/</a><br>
>>> >> >> ><br>
>>> >> >> > It seems that for some reason a lot of data is not being garbage<br>
>>> >> >> > collected.<br>
>>> >> >> ><br>
>>> >> >> > Could there be a reference counting bug somewhere down in a cython<br>
>>> >> >> > routine?<br>
>>> >> >><br>
>>> >> >> Based on what you're running, the only Cython routines being called<br>
>>> >> >> are likely in the selection system.<br>
>>> >> >><br>
>>> >> >> > Objgraph is unable to find backreferences to root grid tiles in<br>
>>> >> >> > the<br>
>>> >> >> > flash<br>
>>> >> >> > dataset, and all the other yt objects that I've looked at seem to<br>
>>> >> >> > have<br>
>>> >> >> > backreference graphs that terminate at a FLASHGrid object that<br>
>>> >> >> > represents a<br>
>>> >> >> > root grid tile in one of the datasets. That's the best guess I<br>
>>> >> >> > have<br>
>>> >> >> > -<br>
>>> >> >> > but<br>
>>> >> >> > definitely nothing conclusive. I'd appreciate any other ideas<br>
>>> >> >> > anyone<br>
>>> >> >> > else<br>
>>> >> >> > has to help debug this.<br>
>>> >> >><br>
>>> >> >> I'm not entirely sure how to parse the output you've pasted, but I<br>
>>> >> >> do<br>
>>> >> >> have a thought. If you have a reproducible case, I can test it<br>
>>> >> >> myself. I am wondering if this could be related to the way that<br>
>>> >> >> grid<br>
>>> >> >> masks are cached. You should be able to test this by adding this<br>
>>> >> >> line<br>
>>> >> >> to _get_selector_mask in grid_patch.py, just before "return mask"<br>
>>> >> >><br>
>>> >> >> self._last_mask = self._last_selector_id = None<br>
>>> >> >><br>
>>> >> >> Something like this patch:<br>
>>> >> >><br>
>>> >> >> <a href="http://paste.yt-project.org/show/4316/" target="_blank">http://paste.yt-project.org/show/4316/</a><br>
>>> >> ><br>
>>> >> ><br>
>>> >> > Thanks for the code! I will look into this today.<br>
>>> >> ><br>
>>> >> > Sorry for not explaining the random terminal output I pasted from<br>
>>> >> > objgraph<br>
>>> >> > :/<br>
>>> >> ><br>
>>> >> > It's a list of objects created after yt operates on one dataset and<br>
>>> >> > after<br>
>>> >> > the garbage collector is explicitly called. Each iteration of the<br>
>>> >> > loop<br>
>>> >> > sees<br>
>>> >> > the creation of objects representing the FLASH grids, hierarchy, and<br>
>>> >> > associated metadata. With enough iterations this overhead from<br>
>>> >> > previous<br>
>>> >> > loop iterations begins to dominate the total memory budget.<br>
>>> >><br>
>>> >> The code snippet I sent might help reduce it, but I think it speaks to<br>
>>> >> a deeper problem in that somehow the FLASH stuff isn't being GC'd<br>
>>> >> anywhere. It really ought to be.<br>
>>> >><br>
>>> >> Can you try also doing:<br>
>>> >><br>
>>> >> yt.frontends.flash.FLASHDataset._skip_cache = True<br>
>>> ><br>
>>> ><br>
>>> > No effect, unfortunately.<br>
>>> ><br>
>>> >><br>
>>> >> and seeing if that helps?<br>
>>> >><br>
>>> >> ><br>
>>> >> >><br>
>>> >> >><br>
>>> >> >><br>
>>> >> >> -Matt<br>
>>> >> >><br>
>>> >> >> ><br>
>>> >> >> > Thanks for your help in debugging this!<br>
>>> >> >> ><br>
>>> >> >> > -Nathan<br>
>>> >> >> ><br>
>>> >> >> _______________________________________________<br>
>>> >> >> yt-dev mailing list<br>
>>> >> >> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>> >> >> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>>> >> ><br>
>>> >> ><br>
>>> >> ><br>
>>> >> > _______________________________________________<br>
>>> >> > yt-dev mailing list<br>
>>> >> > <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>> >> > <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>>> >> ><br>
>>> >> _______________________________________________<br>
>>> >> yt-dev mailing list<br>
>>> >> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>> >> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>>> ><br>
>>> ><br>
>>> ><br>
>>> > _______________________________________________<br>
>>> > yt-dev mailing list<br>
>>> > <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>> > <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>>> ><br>
>>> _______________________________________________<br>
>>> yt-dev mailing list<br>
>>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> yt-dev mailing list<br>
>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>><br>
_______________________________________________<br>
yt-dev mailing list<br>
<a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
</div></div></blockquote></div><br></div>