<div dir="ltr">Hey all,<div><br></div><div>I'm looking at a memory leak that Philip (cc'd) is seeing when iterating over a long list of FLASH datasets. Just as an example of the type of behavior he is seeing - today he left his script running and ended up consuming 300 GB of RAM on a viz node.</div>
<div><br></div><div>FWIW, the dataset is not particularly large - ~300 outputs and ~100 MB per output. These are also FLASH cylindrical coordinate simulations - so perhaps this behavior will only occur in curvilinear geometries?</div>
<div><br></div><div>I've been playing with objgraph to try to understand what's happening. Here's the script I've been using: <a href="http://paste.yt-project.org/show/4762/">http://paste.yt-project.org/show/4762/</a></div>
<div><br></div><div>Here's the output after one iteration of the for loop: <a href="http://paste.yt-project.org/show/4761/">http://paste.yt-project.org/show/4761/</a></div><div><br></div><div>It seems that for some reason a lot of data is not being garbage collected. </div>
<div><br></div><div>Could there be a reference counting bug somewhere down in a cython routine? Objgraph is unable to find backreferences to root grid tiles in the flash dataset, and all the other yt objects that I've looked at seem to have backreference graphs that terminate at a FLASHGrid object that represents a root grid tile in one of the datasets. That's the best guess I have - but definitely nothing conclusive. I'd appreciate any other ideas anyone else has to help debug this.</div>
<div><br></div><div>Thanks for your help in debugging this!</div><div><br></div><div>-Nathan</div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jun 9, 2014 at 7:36 PM, Phillip Macias <span dir="ltr"><<a href="mailto:pmacias@ucsc.edu" target="_blank">pmacias@ucsc.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Done. Also this sim is just a test haha so don't judge the crappy quality :)</div><div class="HOEnZb">
<div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jun 9, 2014 at 7:28 PM, Nathan Goldbaum <span dir="ltr"><<a href="mailto:nathan12343@gmail.com" target="_blank">nathan12343@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Any chance you could make the folder that the dataset lives in visible to me?<div><br></div><div>chmod -R +r /pfs/pmacias/runs/hoseon/2pi/diff_angle/coriolis/<br>
</div><div><br></div><div>should do the trick.</div>
</div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jun 9, 2014 at 7:25 PM, Nathan Goldbaum <span dir="ltr"><<a href="mailto:nathan12343@gmail.com" target="_blank">nathan12343@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">This looks like a memory leak in yt. Let me see if can figure out what's going wrong.</div><div>
<div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jun 9, 2014 at 7:21 PM, Phillip Macias <span dir="ltr"><<a href="mailto:pmacias@ucsc.edu" target="_blank">pmacias@ucsc.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">No problem, sorry to be a pain! <div><br></div><div>I've attached the script. To run I'm just sourcing the yt installation on hyades and doing </div>
<div><br></div><div>python dens_plot.py</div><div>
<br></div><div>Maybe I need to be clearing some things as I go through the loop? Idk I'm sure it's something really silly :/</div><div><br></div><div>Thanks again!</div><div>-Phil</div></div><div><div>
<div class="gmail_extra">
<br><br><div class="gmail_quote">On Mon, Jun 9, 2014 at 6:56 PM, Nathan Goldbaum <span dir="ltr"><<a href="mailto:nathan12343@gmail.com" target="_blank">nathan12343@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Thanks for killing them. I'd be happy to take a look at your script.</div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jun 9, 2014 at 6:46 PM, Phillip Macias <span dir="ltr"><<a href="mailto:pmacias@ucsc.edu" target="_blank">pmacias@ucsc.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hey Nathan,<br>
<br>
Sorry about that!! I was actually running yt scripts, I'm not sure how they could use so much ram :(. When I get home can I send you my script and you can tell me how I'm being dumb?<br>
<br>
-Phil<br>
<div><div><br>
<br>
<br>
On Jun 9, 2014, at 6:43 PM, Nathan Goldbaum <<a href="mailto:nathan12343@gmail.com" target="_blank">nathan12343@gmail.com</a>> wrote:<br>
<br>
> Hey Philip,<br>
><br>
> It looks like you have two scripts that have been running for 7 hours - using ~300 GB of ram. Are these zombie processes?<br>
><br>
> -Nathan<br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>