<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Jun 10, 2014 at 10:59 AM, Matthew Turk <span dir="ltr"><<a href="mailto:matthewturk@gmail.com" target="_blank">matthewturk@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Do you have a reproducible script? </blockquote><div><br>
</div><div>This should do the trick: <a href="http://paste.yt-project.org/show/4767/">http://paste.yt-project.org/show/4767/</a></div><div><br></div><div>(this is with an enzo dataset by the way)</div><div><br></div><div>
That script prints (on my machine):</div><div><br></div><div><div>EnzoGrid 15065</div><div>YTArray 1520</div><div>list 704</div><div>dict 2</div><div>MaskedArray 1</div></div><div><br></div><div>Which indicates that 15000 EnzoGrid objects and 1520 YTArray objects have leaked. </div>
<div><br></div><div>The list I'm printing out at the end of the script should be the objects that leaked during the loop over the Enzo dataset. The objgraph.get_leaking_objects() function returns the list of all objects being tracked by the garbage collector that have no references but still have nonzero refcounts.</div>
<div><br></div><div>This means the "original_leaks" list isn't necessarily a list of leaky objects - most of the things in there are singletons that the interpreter keeps around. To create a list of leaky objects produced by iterating over the loop I take the set difference of the output of get_leaking_objects() before and after iterating over the dataset.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">If you make a bunch of symlinks to<br>
one flash file and load them all in sequence, does that replicate the<br>
behavior?<br></blockquote><div><br></div><div>Yes, it seems to. Compare the output of this script: <a href="http://paste.yt-project.org/show/4768/">http://paste.yt-project.org/show/4768/</a></div><div><br></div><div>Adjust the range of the for loop from 0 to 5 - creating the needed symlinks to WindTunnel/windtunnel_4lev_hdf5_plt_cnt_0040 as needed.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class=""><div class="h5"><br>
On Tue, Jun 10, 2014 at 12:57 PM, Nathan Goldbaum <<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a>> wrote:<br>
><br>
><br>
><br>
> On Tue, Jun 10, 2014 at 10:45 AM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
> wrote:<br>
>><br>
>> Hi Nathan,<br>
>><br>
>> On Tue, Jun 10, 2014 at 12:43 PM, Nathan Goldbaum <<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a>><br>
>> wrote:<br>
>> ><br>
>> ><br>
>> ><br>
>> > On Tue, Jun 10, 2014 at 6:09 AM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
>> > wrote:<br>
>> >><br>
>> >> Hi Nathan,<br>
>> >><br>
>> >> On Mon, Jun 9, 2014 at 11:02 PM, Nathan Goldbaum<br>
>> >> <<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a>><br>
>> >> wrote:<br>
>> >> > Hey all,<br>
>> >> ><br>
>> >> > I'm looking at a memory leak that Philip (cc'd) is seeing when<br>
>> >> > iterating<br>
>> >> > over a long list of FLASH datasets. Just as an example of the type<br>
>> >> > of<br>
>> >> > behavior he is seeing - today he left his script running and ended up<br>
>> >> > consuming 300 GB of RAM on a viz node.<br>
>> >> ><br>
>> >> > FWIW, the dataset is not particularly large - ~300 outputs and ~100<br>
>> >> > MB<br>
>> >> > per<br>
>> >> > output. These are also FLASH cylindrical coordinate simulations - so<br>
>> >> > perhaps<br>
>> >> > this behavior will only occur in curvilinear geometries?<br>
>> >><br>
>> >> Hm, I don't know about that.<br>
>> >><br>
>> >> ><br>
>> >> > I've been playing with objgraph to try to understand what's<br>
>> >> > happening.<br>
>> >> > Here's the script I've been using:<br>
>> >> > <a href="http://paste.yt-project.org/show/4762/" target="_blank">http://paste.yt-project.org/show/4762/</a><br>
>> >> ><br>
>> >> > Here's the output after one iteration of the for loop:<br>
>> >> > <a href="http://paste.yt-project.org/show/4761/" target="_blank">http://paste.yt-project.org/show/4761/</a><br>
>> >> ><br>
>> >> > It seems that for some reason a lot of data is not being garbage<br>
>> >> > collected.<br>
>> >> ><br>
>> >> > Could there be a reference counting bug somewhere down in a cython<br>
>> >> > routine?<br>
>> >><br>
>> >> Based on what you're running, the only Cython routines being called<br>
>> >> are likely in the selection system.<br>
>> >><br>
>> >> > Objgraph is unable to find backreferences to root grid tiles in the<br>
>> >> > flash<br>
>> >> > dataset, and all the other yt objects that I've looked at seem to<br>
>> >> > have<br>
>> >> > backreference graphs that terminate at a FLASHGrid object that<br>
>> >> > represents a<br>
>> >> > root grid tile in one of the datasets. That's the best guess I have<br>
>> >> > -<br>
>> >> > but<br>
>> >> > definitely nothing conclusive. I'd appreciate any other ideas anyone<br>
>> >> > else<br>
>> >> > has to help debug this.<br>
>> >><br>
>> >> I'm not entirely sure how to parse the output you've pasted, but I do<br>
>> >> have a thought. If you have a reproducible case, I can test it<br>
>> >> myself. I am wondering if this could be related to the way that grid<br>
>> >> masks are cached. You should be able to test this by adding this line<br>
>> >> to _get_selector_mask in grid_patch.py, just before "return mask"<br>
>> >><br>
>> >> self._last_mask = self._last_selector_id = None<br>
>> >><br>
>> >> Something like this patch:<br>
>> >><br>
>> >> <a href="http://paste.yt-project.org/show/4316/" target="_blank">http://paste.yt-project.org/show/4316/</a><br>
>> ><br>
>> ><br>
>> > Thanks for the code! I will look into this today.<br>
>> ><br>
>> > Sorry for not explaining the random terminal output I pasted from<br>
>> > objgraph<br>
>> > :/<br>
>> ><br>
>> > It's a list of objects created after yt operates on one dataset and<br>
>> > after<br>
>> > the garbage collector is explicitly called. Each iteration of the loop<br>
>> > sees<br>
>> > the creation of objects representing the FLASH grids, hierarchy, and<br>
>> > associated metadata. With enough iterations this overhead from previous<br>
>> > loop iterations begins to dominate the total memory budget.<br>
>><br>
>> The code snippet I sent might help reduce it, but I think it speaks to<br>
>> a deeper problem in that somehow the FLASH stuff isn't being GC'd<br>
>> anywhere. It really ought to be.<br>
>><br>
>> Can you try also doing:<br>
>><br>
>> yt.frontends.flash.FLASHDataset._skip_cache = True<br>
><br>
><br>
> No effect, unfortunately.<br>
><br>
>><br>
>> and seeing if that helps?<br>
>><br>
>> ><br>
>> >><br>
>> >><br>
>> >><br>
>> >> -Matt<br>
>> >><br>
>> >> ><br>
>> >> > Thanks for your help in debugging this!<br>
>> >> ><br>
>> >> > -Nathan<br>
>> >> ><br>
>> >> _______________________________________________<br>
>> >> yt-dev mailing list<br>
>> >> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> >> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> ><br>
>> ><br>
>> ><br>
>> > _______________________________________________<br>
>> > yt-dev mailing list<br>
>> > <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> > <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> ><br>
>> _______________________________________________<br>
>> yt-dev mailing list<br>
>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
><br>
><br>
><br>
> _______________________________________________<br>
> yt-dev mailing list<br>
> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
><br>
_______________________________________________<br>
yt-dev mailing list<br>
<a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
</div></div></blockquote></div><br></div></div>