Hello all, including Pragnesh, hope your request went through.<div><br></div><div>I have since finished the troubling 3200 cube dataset, so good news on my part, but now I'm trying to track down what made it so difficult.</div>
<div><br></div><div>Let me summarize the problem I had, Pragnesh was continuing a discussion we had off list:</div><div><br></div><div>I was analyzing the 3200 cube, and due to peak memory constraints of Natuilus, I tried to analyze subvolumes 1/64 of the whole volume one at a time. The behavior we found was that if we do not end the python script after each subvolume is analyzed with parallelHF() call, and start analyzing the subsequent 1/64 of the subvolume, the peak memory will go up, as per the print statements. If we were to restart the script after each subvolume is analyzed, we can get the peak memory back to a level consistent with the first subvolume.</div>
<div><br></div><div>Attached is the memory output plotted of parallelHF with 256 cores, so every 256 (in the x axis) it looks like a step function. Each step correspond to the peak memory used by parallelHF on a subvolume, and the numbers just go up, and if we were to end the calculation in the middle and restart at a particular subvolume, the numbers would drop back down. Since the numbers from each core varied, we were wondering if this is because the particle data remained on the core even after the calculation.<br>
<br></div><div>hg tip gave:</div><div><div>changeset: 4676:edd92d1eadf8</div><div>branch: yt</div><div>tag: tip</div><div>parent: 4671:85bc12462dff</div><div>parent: 4675:5efc2462a321</div><div>user: Stephen Skory <<a href="mailto:s@skory.us" target="_blank">s@skory.us</a>></div>
<div>date: Wed Oct 19 13:44:33 2011 -0600</div><div>summary: Merging from mainline.</div></div><div><br></div><div>This is the mod Stephen made to YT to help with this problem about a month ago. It definitely helped with the memory usage, but I just wanted to narrow this problem down to something more tangible besides "memory leak" inside YT. Hopefully others who ran into the same problem or are experts at parallelHF can help us out.</div>
<div><br></div><div>These issues were found before the new optional KDtree was put in by Stephen, so that is definitely something we can try. If the problem goes away, then we can probably say that the memory issue is inside the Fortran KDtree. I might be the only one encountering this problem, which only occurs when using YT in unconventional way (I believe the subvolume feature was made for analyzing a single halo, not the whole volume in pieces), so I was wondering if anyone else has come across this phenomenon, maybe running parallelHF on several datasets and seeing a peak memory increase?</div>
<div><br></div><div>From</div><div>G.S.</div><div><br></div><br><div class="gmail_quote">On Thu, Nov 17, 2011 at 9:49 AM, Patel, Pragneshkumar B <span dir="ltr"><<a href="mailto:pragnesh@utk.edu">pragnesh@utk.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div>
<div style="direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt"><br>
<div style="font-family:Times New Roman;color:rgb(0,0,0);font-size:16px">
<hr>
<div style="direction:ltr"><font color="#000000" face="Tahoma" size="2"><b>From:</b> Patel, Pragneshkumar B<br>
<b>Sent:</b> Thursday, November 17, 2011 12:18 PM<br>
<b>To:</b> Britton Smith<br>
<b>Cc:</b> Geoffrey So; <a href="mailto:s@skory.us" target="_blank">s@skory.us</a><br>
<b>Subject:</b> RE: [yt-dev] Memory leak issue with ParallelHOP code<br>
</font><br>
</div><div><div></div><div class="h5">
<div></div>
<div>
<div style="direction:ltr;font-family:Tahoma;color:rgb(0,0,0);font-size:10pt">
Hi Britton, <br>
<br>
I have subscribed to both yt-users and yt-dev. It may possible my requests are pending. As far as dataset is concern, Geoffrey can give you better explanation about it.
<br>
<br>
Dataset contains 0-64 regions. I tried to run ParallelHOP with all regions but because of memory issue, I could not able to complete all in once. Then I ran it with only 0-3 regions to debug code "parallel_hop_interface.py".
<br>
<br>
Please also find attached script. Let me know, if you have more questions. <br>
<br>
<br>
Thanks<br>
Pragnesh<br>
<br>
<br>
<br>
<div style="font-family:Times New Roman;color:rgb(0,0,0);font-size:16px">
<hr>
<div style="direction:ltr"><font color="#000000" face="Tahoma" size="2"><b>From:</b> Britton Smith [<a href="mailto:brittonsmith@gmail.com" target="_blank">brittonsmith@gmail.com</a>]<br>
<b>Sent:</b> Thursday, November 17, 2011 12:00 PM<br>
<b>To:</b> <a href="mailto:yt-dev@lists.spacepope.org" target="_blank">yt-dev@lists.spacepope.org</a><br>
<b>Cc:</b> Patel, Pragneshkumar B<br>
<b>Subject:</b> Re: [yt-dev] Memory leak issue with ParallelHOP code<br>
</font><br>
</div>
<div></div>
<div>Hi again Pragnesh,<br>
<br>
I mistakenly asked you to post this to the enzo-users mailing when I meant to say that you should post this to the yt-users mailing list. Sorry about that, not enough coffee this morning.<br>
<br>
Britton<br>
<br>
<div class="gmail_quote">On Thu, Nov 17, 2011 at 8:56 AM, Britton Smith <span dir="ltr">
<<a href="mailto:brittonsmith@gmail.com" target="_blank">brittonsmith@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Hi Pragnesh,<br>
<br>
You need to provide us with some more information. What do you mean when you say you ran for 0-3 regions? What kind of dataset are you trying to analyze? How big is it? Can you also please post the script you are using for this?<br>
<br>
Finally, could you please sign up for the enzo-users mailing list and post this there? You've posted this to the development list, which isn't the best forum for this question. Also, if you're not on the list you won't receive any of the responses to your
post.<br>
<br>
Britton<br>
<br>
<div class="gmail_quote">
<div>
<div>On Thu, Nov 17, 2011 at 7:29 AM, Patel, Pragneshkumar B <span dir="ltr">
<<a href="mailto:pragnesh@utk.edu" target="_blank">pragnesh@utk.edu</a>></span> wrote:<br>
</div>
</div>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>
<div>
<div style="direction:ltr;font-family:Tahoma;color:rgb(0,0,0);font-size:10pt">
Hi Stephen, <br>
<br>
I found memory leak issue, when I ran Parallel_HOP code. It may not be in python itself but may be in packages(e.g. kdtree etc..), which we call from"parallel_hop_interface.py". Please find attach file, in which you see memory keep increase in every region
(I ran program with 32 processes for 0-3 regions.)<br>
<br>
I like to get your comments/suggestions to solve this issue. <br>
<br>
I am trying to find more about it using Valgrind. I will let you know. <br>
<br>
Thanks<br>
Pragnesh<br>
<br>
<br>
</div>
</div>
<br>
</div>
</div>
_______________________________________________<br>
yt-dev mailing list<br>
<a href="mailto:yt-dev@lists.spacepope.org" target="_blank">yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
<br>
</blockquote>
</div>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div></div></div>
</div>
</div>
<br>_______________________________________________<br>
yt-dev mailing list<br>
<a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
<br></blockquote></div><br>