Sorry to butt in here, but could this be related to the same issue I had when trying to iterate many different subvolume for the halo profiler in a loop style, and see the peak max memory grow over each iteration of the loop? The behavior described seems awfully familiar to me.<div>
<br></div><div>The way I got around this problem is to star a new python instance for each of the subvolume in bash, and pass in the subvolume number via sys.argv[]. </div><div><br></div><div>in bash I have:</div><div><br>
</div><div><div>for j in {50..50}</div><div>do</div><div>mpirun -n 256 python <script name>.py $j --parallel 2>&1 | tee `printf %i $j`progress.txt</div><div>done</div></div><div><br></div><div>the above will analyze the piece number 50 of my data</div>
<div><br></div><div>in the yt script I have:</div><div><div><br></div><div>import sys</div><div>inputsv = int(sys.argv[1])</div></div><div><br></div><div>But of course this is a temporary fix of the symptoms not the cure, since I wasn't able to narrow the problem down further.</div>
<div><br></div><div>From</div><div>G.S.<br><br><div class="gmail_quote">On Sun, Jan 29, 2012 at 9:58 AM, John ZuHone <span dir="ltr"><<a href="mailto:jzuhone@milkyway.gsfc.nasa.gov">jzuhone@milkyway.gsfc.nasa.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Thanks to everyone for the help. The files are not very big, so I think this is an issue of memory not being freed up as it should, as per Matt's suggestion. I'm rewriting the script as he suggested and then trying that. <span class="HOEnZb"><font color="#888888"><div>
<br></div><div>John</div></font></span><div><br><div><div class="im"><div>On Jan 29, 2012, at 2:13 AM, Sam Skillman wrote:</div><br></div><div><div class="h5"><blockquote type="cite">Hi John,<div><br></div><div>Additionally, adding 'del cam, image' at the end of each loop should help considerably. The camera object contains the homogenized volume roughly equal to the size of the entire dataset for one field, so I'm guessing that is where most of it is coming from.</div>
<div><br></div><div>Let us know if that doesn't help.</div><div><br></div><div>Sam<br><br><div class="gmail_quote">On Sat, Jan 28, 2012 at 7:31 PM, Nathan Goldbaum <span dir="ltr"><<a href="mailto:goldbaum@ucolick.org" target="_blank">goldbaum@ucolick.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi John,<br>
<br>
I've found that calling the garbage collector via gc.collect() at the end of a loop over filenames fixes issues like this. Probably worth a try, anyway.<br>
<br>
-Nathan<br>
<div><div><br>
On Jan 28, 2012, at 8:52 AM, John ZuHone wrote:<br>
<br>
> Hi all,<br>
><br>
> I'm volume rendering a ~1 GB dataset on an 8 CPU machine with 32 GB of RAM. I keep using up the memory on the machine with the following script, which creates a time series object and runs through the pfs to create each image. Is there other memory I should be freeing up at each iteration in the loop?<br>
><br>
> <a href="http://paste.yt-project.org/show/2137/" target="_blank">http://paste.yt-project.org/show/2137/</a><br>
><br>
> Best,<br>
><br>
> John<br>
> _______________________________________________<br>
> yt-users mailing list<br>
> <a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a><br>
> <a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a><br>
><br>
</div></div>> !DSPAM:10175,4f24aeed3642254826273!<br>
<div><div>><br>
<br>
_______________________________________________<br>
yt-users mailing list<br>
<a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a><br>
</div></div></blockquote></div><br></div>
_______________________________________________<br>yt-users mailing list<br><a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a><br><a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a><br>
</blockquote></div></div></div><br></div></div><br>_______________________________________________<br>
yt-users mailing list<br>
<a href="mailto:yt-users@lists.spacepope.org">yt-users@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a><br>
<br></blockquote></div><br></div>