[yt-users] Memory leaks and volume rendering

[Geoffrey So]

Sorry to butt in here, but could this be related to the same issue I had
when trying to iterate many different subvolume for the halo profiler in a
loop style, and see the peak max memory grow over each iteration of the
loop?  The behavior described seems awfully familiar to me.

The way I got around this problem is to star a new python instance for each
of the subvolume in bash, and pass in the subvolume number via sys.argv[].

in bash I have:

for j in {50..50}
do
mpirun -n 256 python <script name>.py $j --parallel 2>&1 | tee `printf %i
$j`progress.txt
done

the above will analyze the piece number 50 of my data

in the yt script I have:

import sys
inputsv = int(sys.argv[1])

But of course this is a temporary fix of the symptoms not the cure, since I
wasn't able to narrow the problem down further.

From
G.S.

On Sun, Jan 29, 2012 at 9:58 AM, John ZuHone <jzuhone at milkyway.gsfc.nasa.gov
> wrote:

> Thanks to everyone for the help. The files are not very big, so I think
> this is an issue of memory not being freed up as it should, as per Matt's
> suggestion. I'm rewriting the script as he suggested and then trying that.
>
> John
>
> On Jan 29, 2012, at 2:13 AM, Sam Skillman wrote:
>
> Hi John,
>
> Additionally, adding 'del cam, image' at the end of each loop should help
> considerably.  The camera object contains the homogenized volume roughly
> equal to the size of the entire dataset for one field, so I'm guessing that
> is where most of it is coming from.
>
> Let us know if that doesn't help.
>
> Sam
>
> On Sat, Jan 28, 2012 at 7:31 PM, Nathan Goldbaum <goldbaum at ucolick.org>wrote:
>
>> Hi John,
>>
>> I've found that calling the garbage collector via gc.collect() at the end
>> of a loop over filenames fixes issues like this.  Probably worth a try,
>> anyway.
>>
>> -Nathan
>>
>> On Jan 28, 2012, at 8:52 AM, John ZuHone wrote:
>>
>> > Hi all,
>> >
>> > I'm volume rendering a ~1 GB dataset on an 8 CPU machine with 32 GB of
>> RAM. I keep using up the memory on the machine with the following script,
>> which creates a time series object and runs through the pfs to create each
>> image. Is there other memory I should be freeing up at each iteration in
>> the loop?
>> >
>> > http://paste.yt-project.org/show/2137/
>> >
>> > Best,
>> >
>> > John
>> > _______________________________________________
>> > yt-users mailing list
>> > yt-users at lists.spacepope.org
>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> >
>> > !DSPAM:10175,4f24aeed3642254826273!
>> >
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20120129/d5174353/attachment.htm>