<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jun 11, 2014 at 2:53 PM, Matthew Turk <span dir="ltr"><<a href="mailto:matthewturk@gmail.com" target="_blank">matthewturk@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Nathan,<br>
<div class=""><br>
On Wed, Jun 11, 2014 at 3:30 PM, Nathan Goldbaum <<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a>> wrote:<br>
> For all of these tests I estimated the memory usage by watching the 'RES'<br>
> column under htop as the script ran.<br>
><br>
> On Wed, Jun 11, 2014 at 12:30 PM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
> wrote:<br>
>><br>
>> Can you try with:<br>
>><br>
>> * Just a slice<br>
><br>
><br>
> Roughly constant memory usage: <a href="http://paste.yt-project.org/show/4774/" target="_blank">http://paste.yt-project.org/show/4774/</a><br>
><br>
>><br>
>> * Just a slice you've accessed 'ones' in<br>
><br>
><br>
> Steadily increasing memory usage, peaking at about 1.3 gigabytes:<br>
> <a href="http://paste.yt-project.org/show/4776/" target="_blank">http://paste.yt-project.org/show/4776/</a><br>
<br>
</div>Okay, this seems like basically the same thing we were seeing before.<br>
I think the index is the problem. I was not seeing memory increase;<br>
is this with the fix for deallocating the grids?<br></blockquote><div><br></div><div>Yes, although it's with a FLASH cylindrical coordinate simulation.</div><div><br></div><div>I can check whether the same thing happens for an Enzo simulation time series.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="HOEnZb"><div class="h5"><br>
><br>
> Note that these datasets range from ~300 kilobytes in size up to about 30<br>
> megabytes.<br>
><br>
>><br>
>> * Slice => FRB<br>
><br>
><br>
> Roughly constant memory usage: <a href="http://paste.yt-project.org/show/4777/" target="_blank">http://paste.yt-project.org/show/4777/</a><br>
><br>
> (see the PR I just issued to get this script working properly).<br>
><br>
>><br>
>> * Just pf.index<br>
><br>
><br>
> Steadily increasing memory usage, peaking at about 1.2 GB:<br>
> <a href="http://paste.yt-project.org/show/4778/" target="_blank">http://paste.yt-project.org/show/4778/</a><br>
><br>
>><br>
>><br>
>> On Wed, Jun 11, 2014 at 2:26 PM, Nathan Goldbaum <<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a>><br>
>> wrote:<br>
>> > I can confirm that creating the list alias to .grids does eliminate the<br>
>> > Grid<br>
>> > objects from showing up in the objgraph output.<br>
>> ><br>
>> > That said, I'm still seeing steadily increasing memory usage when I<br>
>> > iterate<br>
>> > over a bunch of datasets (<a href="http://paste.yt-project.org/show/4773/" target="_blank">http://paste.yt-project.org/show/4773/</a>),<br>
>> > creating<br>
>> > SlicePlots for each one. I'm not sure yet where the memory is going,<br>
>> > just<br>
>> > that objgraph can't see it.<br>
>> ><br>
>> ><br>
>> > On Wed, Jun 11, 2014 at 12:02 PM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
>> > wrote:<br>
>> >><br>
>> >> On Wed, Jun 11, 2014 at 1:58 PM, Nathan Goldbaum<br>
>> >> <<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a>><br>
>> >> wrote:<br>
>> >> > Could this issue be related?<br>
>> >> ><br>
>> >> > <a href="https://github.com/numpy/numpy/issues/1601" target="_blank">https://github.com/numpy/numpy/issues/1601</a><br>
>> >><br>
>> >> Yeah, that's the one.<br>
>> >><br>
>> >> ><br>
>> >> > Can you elaborate a bit more about why we're using an object array in<br>
>> >> > the<br>
>> >> > first place? If switching to using a list solves these issues<br>
>> >> > perhaps<br>
>> >> > that<br>
>> >> > is the way to go.<br>
>> >><br>
>> >> Two reasons. One is that it's an OOM faster for some things that we<br>
>> >> do a lot, and the other is that it makes it much easier to index. We<br>
>> >> can do things like selection based on indices or booleans this way.<br>
>> >> But we don't do that very often anymore.<br>
>> >><br>
>> >> I don't want to switch it to a list; that's a nasty bandaid that<br>
>> >> breaks things. We can just add on an additional list, which is a<br>
>> >> nasty bandaid that doesn't break things. I think the memory overhead<br>
>> >> will be minimal for that. Really fixing it will require moving away<br>
>> >> from arrays completely, which we can slot in for 3.1.<br>
>> >><br>
>> >> ><br>
>> >> ><br>
>> >> > On Wed, Jun 11, 2014 at 7:25 AM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
>> >> > wrote:<br>
>> >> >><br>
>> >> >> I should also note, at some point in the future I want to get rid of<br>
>> >> >> the object arrays for grids, but that timescale is longer. Using<br>
>> >> >> John's grid tree is a much better approach.<br>
>> >> >><br>
>> >> >> On Wed, Jun 11, 2014 at 9:21 AM, Matthew Turk<br>
>> >> >> <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
>> >> >> wrote:<br>
>> >> >> > On Tue, Jun 10, 2014 at 10:53 PM, Matthew Turk<br>
>> >> >> > <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
>> >> >> > wrote:<br>
>> >> >> >> On Tue, Jun 10, 2014 at 10:50 PM, Nathan Goldbaum<br>
>> >> >> >> <<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a>> wrote:<br>
>> >> >> >>> For the leaking YTArrays, Kacper suggested the following patch<br>
>> >> >> >>> on<br>
>> >> >> >>> IRC:<br>
>> >> >> >>><br>
>> >> >> >>> <a href="http://bpaste.net/show/361120/" target="_blank">http://bpaste.net/show/361120/</a><br>
>> >> >> >>><br>
>> >> >> >>> This works for FLASH but seems to break field detection for<br>
>> >> >> >>> enzo.<br>
>> >> >> >><br>
>> >> >> >> I don't think this will ever be a big memory hog, but it is worth<br>
>> >> >> >> fixing.<br>
>> >> >> >><br>
>> >> >> ><br>
>> >> >> > I've spent a small bit of time at this again this morning, and<br>
>> >> >> > everything seems to come back down to the issue of having a numpy<br>
>> >> >> > array of grid objects. If I switch this to a list, the reference<br>
>> >> >> > counting is correct again and things get deallocated properly.<br>
>> >> >> > I've<br>
>> >> >> > tried a number of ways of changing how they're allocated, but none<br>
>> >> >> > seem to work for getting the refcount correct. Oddly enough, if I<br>
>> >> >> > track both a list *and* an array (i.e., set self._grids =<br>
>> >> >> > self.grids.tolist()) then the refcounting is correct.<br>
>> >> >> ><br>
>> >> >> > I'm sure there's an explanation for this, but I don't know it. It<br>
>> >> >> > looks to me like numpy thinks it owns the data and that it should<br>
>> >> >> > decrement the object refcount.<br>
>> >> >> ><br>
>> >> >> > By adding this line:<br>
>> >> >> ><br>
>> >> >> > self._grids = self.grids.tolist()<br>
>> >> >> ><br>
>> >> >> > after the call to _populate_grid_objects() in<br>
>> >> >> > grid_geometry_handler,<br>
>> >> >> > I<br>
>> >> >> > was able to get all references tracked and removed.<br>
>> >> >> ><br>
>> >> >> >>><br>
>> >> >> >>><br>
>> >> >> >>> On Tue, Jun 10, 2014 at 8:47 PM, Matthew Turk<br>
>> >> >> >>> <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
>> >> >> >>> wrote:<br>
>> >> >> >>>><br>
>> >> >> >>>> Hi Nathan,<br>
>> >> >> >>>><br>
>> >> >> >>>> I believe there are two things at work here.<br>
>> >> >> >>>><br>
>> >> >> >>>> 1) (I do not have high confidence of this one.) YTArrays that<br>
>> >> >> >>>> are<br>
>> >> >> >>>> referenced with a .d and turned into numpy arrays which no<br>
>> >> >> >>>> longer<br>
>> >> >> >>>> *own<br>
>> >> >> >>>> the data* may be retaining a reference, but that reference<br>
>> >> >> >>>> doesn't<br>
>> >> >> >>>> get<br>
>> >> >> >>>> freed later. This happens often when we are doing things in<br>
>> >> >> >>>> the<br>
>> >> >> >>>> hierarchy instantiation phase. I haven't been able to figure<br>
>> >> >> >>>> out<br>
>> >> >> >>>> which references get lost; for me, over 40 outputs, I lost<br>
>> >> >> >>>> 1560.<br>
>> >> >> >>>> I<br>
>> >> >> >>>> think it's 39 YTArrays per hierarchy. This might also be<br>
>> >> >> >>>> related<br>
>> >> >> >>>> to<br>
>> >> >> >>>> field detection. I think this is not a substantial<br>
>> >> >> >>>> contributor.<br>
>> >> >> >>>> 2) For some reason, when the .grids attribute (object array) is<br>
>> >> >> >>>> deleted on an index, the refcounts of those grids don't<br>
>> >> >> >>>> decrease.<br>
>> >> >> >>>> I<br>
>> >> >> >>>> am able to decrease their refcounts by manually setting<br>
>> >> >> >>>> pf.index.grids[:] = None. This eliminated all retained grid<br>
>> >> >> >>>> references.<br>
>> >> >> >>>><br>
>> >> >> >>>> So, I think the root is that at some point, because of circular<br>
>> >> >> >>>> references or whatever, the finalizer isn't being called on the<br>
>> >> >> >>>> Gridndex (or on Index itself). This results in the reference<br>
>> >> >> >>>> to<br>
>> >> >> >>>> the<br>
>> >> >> >>>> grids array being kept, which then pumps up the lost object<br>
>> >> >> >>>> count.<br>
>> >> >> >>>> I<br>
>> >> >> >>>> don't know why it's not getting called (it's not guaranteed to<br>
>> >> >> >>>> be<br>
>> >> >> >>>> called, in any event).<br>
>> >> >> >>>><br>
>> >> >> >>>> I have to take care of some other things (including Brendan's<br>
>> >> >> >>>> note<br>
>> >> >> >>>> about the memory problems with particle datasets) but I am<br>
>> >> >> >>>> pretty<br>
>> >> >> >>>> sure<br>
>> >> >> >>>> this is the root.<br>
>> >> >> >>>><br>
>> >> >> >>>> -Matt<br>
>> >> >> >>>><br>
>> >> >> >>>> On Tue, Jun 10, 2014 at 10:13 PM, Matthew Turk<br>
>> >> >> >>>> <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
>> >> >> >>>> wrote:<br>
>> >> >> >>>> > Hi Nathan,<br>
>> >> >> >>>> ><br>
>> >> >> >>>> > All it requires is a call to .index; you don't need to do<br>
>> >> >> >>>> > anything<br>
>> >> >> >>>> > else to get it to lose references.<br>
>> >> >> >>>> ><br>
>> >> >> >>>> > I'm still looking into it.<br>
>> >> >> >>>> ><br>
>> >> >> >>>> > -Matt<br>
>> >> >> >>>> ><br>
>> >> >> >>>> > On Tue, Jun 10, 2014 at 9:26 PM, Nathan Goldbaum<br>
>> >> >> >>>> > <<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a>><br>
>> >> >> >>>> > wrote:<br>
>> >> >> >>>> >><br>
>> >> >> >>>> >><br>
>> >> >> >>>> >><br>
>> >> >> >>>> >> On Tue, Jun 10, 2014 at 10:59 AM, Matthew Turk<br>
>> >> >> >>>> >> <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
>> >> >> >>>> >> wrote:<br>
>> >> >> >>>> >>><br>
>> >> >> >>>> >>> Do you have a reproducible script?<br>
>> >> >> >>>> >><br>
>> >> >> >>>> >><br>
>> >> >> >>>> >> This should do the trick:<br>
>> >> >> >>>> >> <a href="http://paste.yt-project.org/show/4767/" target="_blank">http://paste.yt-project.org/show/4767/</a><br>
>> >> >> >>>> >><br>
>> >> >> >>>> >> (this is with an enzo dataset by the way)<br>
>> >> >> >>>> >><br>
>> >> >> >>>> >> That script prints (on my machine):<br>
>> >> >> >>>> >><br>
>> >> >> >>>> >> EnzoGrid 15065<br>
>> >> >> >>>> >> YTArray 1520<br>
>> >> >> >>>> >> list 704<br>
>> >> >> >>>> >> dict 2<br>
>> >> >> >>>> >> MaskedArray 1<br>
>> >> >> >>>> >><br>
>> >> >> >>>> >> Which indicates that 15000 EnzoGrid objects and 1520 YTArray<br>
>> >> >> >>>> >> objects<br>
>> >> >> >>>> >> have<br>
>> >> >> >>>> >> leaked.<br>
>> >> >> >>>> >><br>
>> >> >> >>>> >> The list I'm printing out at the end of the script should be<br>
>> >> >> >>>> >> the<br>
>> >> >> >>>> >> objects<br>
>> >> >> >>>> >> that leaked during the loop over the Enzo dataset. The<br>
>> >> >> >>>> >> objgraph.get_leaking_objects() function returns the list of<br>
>> >> >> >>>> >> all<br>
>> >> >> >>>> >> objects<br>
>> >> >> >>>> >> being tracked by the garbage collector that have no<br>
>> >> >> >>>> >> references<br>
>> >> >> >>>> >> but<br>
>> >> >> >>>> >> still<br>
>> >> >> >>>> >> have nonzero refcounts.<br>
>> >> >> >>>> >><br>
>> >> >> >>>> >> This means the "original_leaks" list isn't necessarily a<br>
>> >> >> >>>> >> list<br>
>> >> >> >>>> >> of<br>
>> >> >> >>>> >> leaky<br>
>> >> >> >>>> >> objects - most of the things in there are singletons that<br>
>> >> >> >>>> >> the<br>
>> >> >> >>>> >> interpreter<br>
>> >> >> >>>> >> keeps around. To create a list of leaky objects produced by<br>
>> >> >> >>>> >> iterating<br>
>> >> >> >>>> >> over<br>
>> >> >> >>>> >> the loop I take the set difference of the output of<br>
>> >> >> >>>> >> get_leaking_objects()<br>
>> >> >> >>>> >> before and after iterating over the dataset.<br>
>> >> >> >>>> >><br>
>> >> >> >>>> >>><br>
>> >> >> >>>> >>> If you make a bunch of symlinks to<br>
>> >> >> >>>> >>> one flash file and load them all in sequence, does that<br>
>> >> >> >>>> >>> replicate<br>
>> >> >> >>>> >>> the<br>
>> >> >> >>>> >>> behavior?<br>
>> >> >> >>>> >><br>
>> >> >> >>>> >><br>
>> >> >> >>>> >> Yes, it seems to. Compare the output of this script:<br>
>> >> >> >>>> >> <a href="http://paste.yt-project.org/show/4768/" target="_blank">http://paste.yt-project.org/show/4768/</a><br>
>> >> >> >>>> >><br>
>> >> >> >>>> >> Adjust the range of the for loop from 0 to 5 - creating the<br>
>> >> >> >>>> >> needed<br>
>> >> >> >>>> >> symlinks<br>
>> >> >> >>>> >> to WindTunnel/windtunnel_4lev_hdf5_plt_cnt_0040 as needed.<br>
>> >> >> >>>> >><br>
>> >> >> >>>> >>><br>
>> >> >> >>>> >>><br>
>> >> >> >>>> >>> On Tue, Jun 10, 2014 at 12:57 PM, Nathan Goldbaum<br>
>> >> >> >>>> >>> <<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a>><br>
>> >> >> >>>> >>> wrote:<br>
>> >> >> >>>> >>> ><br>
>> >> >> >>>> >>> ><br>
>> >> >> >>>> >>> ><br>
>> >> >> >>>> >>> > On Tue, Jun 10, 2014 at 10:45 AM, Matthew Turk<br>
>> >> >> >>>> >>> > <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
>> >> >> >>>> >>> > wrote:<br>
>> >> >> >>>> >>> >><br>
>> >> >> >>>> >>> >> Hi Nathan,<br>
>> >> >> >>>> >>> >><br>
>> >> >> >>>> >>> >> On Tue, Jun 10, 2014 at 12:43 PM, Nathan Goldbaum<br>
>> >> >> >>>> >>> >> <<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a>><br>
>> >> >> >>>> >>> >> wrote:<br>
>> >> >> >>>> >>> >> ><br>
>> >> >> >>>> >>> >> ><br>
>> >> >> >>>> >>> >> ><br>
>> >> >> >>>> >>> >> > On Tue, Jun 10, 2014 at 6:09 AM, Matthew Turk<br>
>> >> >> >>>> >>> >> > <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
>> >> >> >>>> >>> >> > wrote:<br>
>> >> >> >>>> >>> >> >><br>
>> >> >> >>>> >>> >> >> Hi Nathan,<br>
>> >> >> >>>> >>> >> >><br>
>> >> >> >>>> >>> >> >> On Mon, Jun 9, 2014 at 11:02 PM, Nathan Goldbaum<br>
>> >> >> >>>> >>> >> >> <<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a>><br>
>> >> >> >>>> >>> >> >> wrote:<br>
>> >> >> >>>> >>> >> >> > Hey all,<br>
>> >> >> >>>> >>> >> >> ><br>
>> >> >> >>>> >>> >> >> > I'm looking at a memory leak that Philip (cc'd) is<br>
>> >> >> >>>> >>> >> >> > seeing<br>
>> >> >> >>>> >>> >> >> > when<br>
>> >> >> >>>> >>> >> >> > iterating<br>
>> >> >> >>>> >>> >> >> > over a long list of FLASH datasets. Just as an<br>
>> >> >> >>>> >>> >> >> > example<br>
>> >> >> >>>> >>> >> >> > of the<br>
>> >> >> >>>> >>> >> >> > type<br>
>> >> >> >>>> >>> >> >> > of<br>
>> >> >> >>>> >>> >> >> > behavior he is seeing - today he left his script<br>
>> >> >> >>>> >>> >> >> > running<br>
>> >> >> >>>> >>> >> >> > and<br>
>> >> >> >>>> >>> >> >> > ended<br>
>> >> >> >>>> >>> >> >> > up<br>
>> >> >> >>>> >>> >> >> > consuming 300 GB of RAM on a viz node.<br>
>> >> >> >>>> >>> >> >> ><br>
>> >> >> >>>> >>> >> >> > FWIW, the dataset is not particularly large - ~300<br>
>> >> >> >>>> >>> >> >> > outputs and<br>
>> >> >> >>>> >>> >> >> > ~100<br>
>> >> >> >>>> >>> >> >> > MB<br>
>> >> >> >>>> >>> >> >> > per<br>
>> >> >> >>>> >>> >> >> > output. These are also FLASH cylindrical coordinate<br>
>> >> >> >>>> >>> >> >> > simulations -<br>
>> >> >> >>>> >>> >> >> > so<br>
>> >> >> >>>> >>> >> >> > perhaps<br>
>> >> >> >>>> >>> >> >> > this behavior will only occur in curvilinear<br>
>> >> >> >>>> >>> >> >> > geometries?<br>
>> >> >> >>>> >>> >> >><br>
>> >> >> >>>> >>> >> >> Hm, I don't know about that.<br>
>> >> >> >>>> >>> >> >><br>
>> >> >> >>>> >>> >> >> ><br>
>> >> >> >>>> >>> >> >> > I've been playing with objgraph to try to<br>
>> >> >> >>>> >>> >> >> > understand<br>
>> >> >> >>>> >>> >> >> > what's<br>
>> >> >> >>>> >>> >> >> > happening.<br>
>> >> >> >>>> >>> >> >> > Here's the script I've been using:<br>
>> >> >> >>>> >>> >> >> > <a href="http://paste.yt-project.org/show/4762/" target="_blank">http://paste.yt-project.org/show/4762/</a><br>
>> >> >> >>>> >>> >> >> ><br>
>> >> >> >>>> >>> >> >> > Here's the output after one iteration of the for<br>
>> >> >> >>>> >>> >> >> > loop:<br>
>> >> >> >>>> >>> >> >> > <a href="http://paste.yt-project.org/show/4761/" target="_blank">http://paste.yt-project.org/show/4761/</a><br>
>> >> >> >>>> >>> >> >> ><br>
>> >> >> >>>> >>> >> >> > It seems that for some reason a lot of data is not<br>
>> >> >> >>>> >>> >> >> > being<br>
>> >> >> >>>> >>> >> >> > garbage<br>
>> >> >> >>>> >>> >> >> > collected.<br>
>> >> >> >>>> >>> >> >> ><br>
>> >> >> >>>> >>> >> >> > Could there be a reference counting bug somewhere<br>
>> >> >> >>>> >>> >> >> > down<br>
>> >> >> >>>> >>> >> >> > in<br>
>> >> >> >>>> >>> >> >> > a<br>
>> >> >> >>>> >>> >> >> > cython<br>
>> >> >> >>>> >>> >> >> > routine?<br>
>> >> >> >>>> >>> >> >><br>
>> >> >> >>>> >>> >> >> Based on what you're running, the only Cython<br>
>> >> >> >>>> >>> >> >> routines<br>
>> >> >> >>>> >>> >> >> being<br>
>> >> >> >>>> >>> >> >> called<br>
>> >> >> >>>> >>> >> >> are likely in the selection system.<br>
>> >> >> >>>> >>> >> >><br>
>> >> >> >>>> >>> >> >> > Objgraph is unable to find backreferences to root<br>
>> >> >> >>>> >>> >> >> > grid<br>
>> >> >> >>>> >>> >> >> > tiles<br>
>> >> >> >>>> >>> >> >> > in<br>
>> >> >> >>>> >>> >> >> > the<br>
>> >> >> >>>> >>> >> >> > flash<br>
>> >> >> >>>> >>> >> >> > dataset, and all the other yt objects that I've<br>
>> >> >> >>>> >>> >> >> > looked<br>
>> >> >> >>>> >>> >> >> > at<br>
>> >> >> >>>> >>> >> >> > seem<br>
>> >> >> >>>> >>> >> >> > to<br>
>> >> >> >>>> >>> >> >> > have<br>
>> >> >> >>>> >>> >> >> > backreference graphs that terminate at a FLASHGrid<br>
>> >> >> >>>> >>> >> >> > object<br>
>> >> >> >>>> >>> >> >> > that<br>
>> >> >> >>>> >>> >> >> > represents a<br>
>> >> >> >>>> >>> >> >> > root grid tile in one of the datasets. That's the<br>
>> >> >> >>>> >>> >> >> > best<br>
>> >> >> >>>> >>> >> >> > guess<br>
>> >> >> >>>> >>> >> >> > I<br>
>> >> >> >>>> >>> >> >> > have<br>
>> >> >> >>>> >>> >> >> > -<br>
>> >> >> >>>> >>> >> >> > but<br>
>> >> >> >>>> >>> >> >> > definitely nothing conclusive. I'd appreciate any<br>
>> >> >> >>>> >>> >> >> > other<br>
>> >> >> >>>> >>> >> >> > ideas<br>
>> >> >> >>>> >>> >> >> > anyone<br>
>> >> >> >>>> >>> >> >> > else<br>
>> >> >> >>>> >>> >> >> > has to help debug this.<br>
>> >> >> >>>> >>> >> >><br>
>> >> >> >>>> >>> >> >> I'm not entirely sure how to parse the output you've<br>
>> >> >> >>>> >>> >> >> pasted, but<br>
>> >> >> >>>> >>> >> >> I<br>
>> >> >> >>>> >>> >> >> do<br>
>> >> >> >>>> >>> >> >> have a thought. If you have a reproducible case, I<br>
>> >> >> >>>> >>> >> >> can<br>
>> >> >> >>>> >>> >> >> test it<br>
>> >> >> >>>> >>> >> >> myself. I am wondering if this could be related to<br>
>> >> >> >>>> >>> >> >> the<br>
>> >> >> >>>> >>> >> >> way<br>
>> >> >> >>>> >>> >> >> that<br>
>> >> >> >>>> >>> >> >> grid<br>
>> >> >> >>>> >>> >> >> masks are cached. You should be able to test this by<br>
>> >> >> >>>> >>> >> >> adding<br>
>> >> >> >>>> >>> >> >> this<br>
>> >> >> >>>> >>> >> >> line<br>
>> >> >> >>>> >>> >> >> to _get_selector_mask in grid_patch.py, just before<br>
>> >> >> >>>> >>> >> >> "return<br>
>> >> >> >>>> >>> >> >> mask"<br>
>> >> >> >>>> >>> >> >><br>
>> >> >> >>>> >>> >> >> self._last_mask = self._last_selector_id = None<br>
>> >> >> >>>> >>> >> >><br>
>> >> >> >>>> >>> >> >> Something like this patch:<br>
>> >> >> >>>> >>> >> >><br>
>> >> >> >>>> >>> >> >> <a href="http://paste.yt-project.org/show/4316/" target="_blank">http://paste.yt-project.org/show/4316/</a><br>
>> >> >> >>>> >>> >> ><br>
>> >> >> >>>> >>> >> ><br>
>> >> >> >>>> >>> >> > Thanks for the code! I will look into this today.<br>
>> >> >> >>>> >>> >> ><br>
>> >> >> >>>> >>> >> > Sorry for not explaining the random terminal output I<br>
>> >> >> >>>> >>> >> > pasted<br>
>> >> >> >>>> >>> >> > from<br>
>> >> >> >>>> >>> >> > objgraph<br>
>> >> >> >>>> >>> >> > :/<br>
>> >> >> >>>> >>> >> ><br>
>> >> >> >>>> >>> >> > It's a list of objects created after yt operates on<br>
>> >> >> >>>> >>> >> > one<br>
>> >> >> >>>> >>> >> > dataset<br>
>> >> >> >>>> >>> >> > and<br>
>> >> >> >>>> >>> >> > after<br>
>> >> >> >>>> >>> >> > the garbage collector is explicitly called. Each<br>
>> >> >> >>>> >>> >> > iteration<br>
>> >> >> >>>> >>> >> > of the<br>
>> >> >> >>>> >>> >> > loop<br>
>> >> >> >>>> >>> >> > sees<br>
>> >> >> >>>> >>> >> > the creation of objects representing the FLASH grids,<br>
>> >> >> >>>> >>> >> > hierarchy,<br>
>> >> >> >>>> >>> >> > and<br>
>> >> >> >>>> >>> >> > associated metadata. With enough iterations this<br>
>> >> >> >>>> >>> >> > overhead<br>
>> >> >> >>>> >>> >> > from<br>
>> >> >> >>>> >>> >> > previous<br>
>> >> >> >>>> >>> >> > loop iterations begins to dominate the total memory<br>
>> >> >> >>>> >>> >> > budget.<br>
>> >> >> >>>> >>> >><br>
>> >> >> >>>> >>> >> The code snippet I sent might help reduce it, but I<br>
>> >> >> >>>> >>> >> think<br>
>> >> >> >>>> >>> >> it<br>
>> >> >> >>>> >>> >> speaks<br>
>> >> >> >>>> >>> >> to<br>
>> >> >> >>>> >>> >> a deeper problem in that somehow the FLASH stuff isn't<br>
>> >> >> >>>> >>> >> being<br>
>> >> >> >>>> >>> >> GC'd<br>
>> >> >> >>>> >>> >> anywhere. It really ought to be.<br>
>> >> >> >>>> >>> >><br>
>> >> >> >>>> >>> >> Can you try also doing:<br>
>> >> >> >>>> >>> >><br>
>> >> >> >>>> >>> >> yt.frontends.flash.FLASHDataset._skip_cache = True<br>
>> >> >> >>>> >>> ><br>
>> >> >> >>>> >>> ><br>
>> >> >> >>>> >>> > No effect, unfortunately.<br>
>> >> >> >>>> >>> ><br>
>> >> >> >>>> >>> >><br>
>> >> >> >>>> >>> >> and seeing if that helps?<br>
>> >> >> >>>> >>> >><br>
>> >> >> >>>> >>> >> ><br>
>> >> >> >>>> >>> >> >><br>
>> >> >> >>>> >>> >> >><br>
>> >> >> >>>> >>> >> >><br>
>> >> >> >>>> >>> >> >> -Matt<br>
>> >> >> >>>> >>> >> >><br>
>> >> >> >>>> >>> >> >> ><br>
>> >> >> >>>> >>> >> >> > Thanks for your help in debugging this!<br>
>> >> >> >>>> >>> >> >> ><br>
>> >> >> >>>> >>> >> >> > -Nathan<br>
>> >> >> >>>> >>> >> >> ><br>
>> >> >> >>>> >>> >> >> _______________________________________________<br>
>> >> >> >>>> >>> >> >> yt-dev mailing list<br>
>> >> >> >>>> >>> >> >> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> >> >> >>>> >>> >> >><br>
>> >> >> >>>> >>> >> >><br>
>> >> >> >>>> >>> >> >><br>
>> >> >> >>>> >>> >> >> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> >> >> >>>> >>> >> ><br>
>> >> >> >>>> >>> >> ><br>
>> >> >> >>>> >>> >> ><br>
>> >> >> >>>> >>> >> > _______________________________________________<br>
>> >> >> >>>> >>> >> > yt-dev mailing list<br>
>> >> >> >>>> >>> >> > <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> >> >> >>>> >>> >> ><br>
>> >> >> >>>> >>> >> ><br>
>> >> >> >>>> >>> >> > <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> >> >> >>>> >>> >> ><br>
>> >> >> >>>> >>> >> _______________________________________________<br>
>> >> >> >>>> >>> >> yt-dev mailing list<br>
>> >> >> >>>> >>> >> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> >> >> >>>> >>> >><br>
>> >> >> >>>> >>> >><br>
>> >> >> >>>> >>> >> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> >> >> >>>> >>> ><br>
>> >> >> >>>> >>> ><br>
>> >> >> >>>> >>> ><br>
>> >> >> >>>> >>> > _______________________________________________<br>
>> >> >> >>>> >>> > yt-dev mailing list<br>
>> >> >> >>>> >>> > <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> >> >> >>>> >>> ><br>
>> >> >> >>>> >>> > <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> >> >> >>>> >>> ><br>
>> >> >> >>>> >>> _______________________________________________<br>
>> >> >> >>>> >>> yt-dev mailing list<br>
>> >> >> >>>> >>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> >> >> >>>> >>><br>
>> >> >> >>>> >>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> >> >> >>>> >><br>
>> >> >> >>>> >><br>
>> >> >> >>>> >><br>
>> >> >> >>>> >> _______________________________________________<br>
>> >> >> >>>> >> yt-dev mailing list<br>
>> >> >> >>>> >> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> >> >> >>>> >> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> >> >> >>>> >><br>
>> >> >> >>>> _______________________________________________<br>
>> >> >> >>>> yt-dev mailing list<br>
>> >> >> >>>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> >> >> >>>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> >> >> >>><br>
>> >> >> >>><br>
>> >> >> >>><br>
>> >> >> >>> _______________________________________________<br>
>> >> >> >>> yt-dev mailing list<br>
>> >> >> >>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> >> >> >>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> >> >> >>><br>
>> >> >> _______________________________________________<br>
>> >> >> yt-dev mailing list<br>
>> >> >> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> >> >> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> >> ><br>
>> >> ><br>
>> >> ><br>
>> >> > _______________________________________________<br>
>> >> > yt-dev mailing list<br>
>> >> > <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> >> > <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> >> ><br>
>> >> _______________________________________________<br>
>> >> yt-dev mailing list<br>
>> >> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> >> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> ><br>
>> ><br>
>> ><br>
>> > _______________________________________________<br>
>> > yt-dev mailing list<br>
>> > <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> > <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> ><br>
>> _______________________________________________<br>
>> yt-dev mailing list<br>
>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
><br>
><br>
><br>
> _______________________________________________<br>
> yt-dev mailing list<br>
> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
><br>
_______________________________________________<br>
yt-dev mailing list<br>
<a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
</div></div></blockquote></div><br></div></div>