[yt-users] Memory overloading on time series

[Sam Geen]

OK, so in the end I cheated a little and used the 
multiprocessing.Process object to spawn separate instances of python for 
each function, and isolated yt to those processes; I've attached the 
code if anyone's interested. I had a quick look at the gc debug output 
but it's quite hard to figure out what I'm looking at at first glance.

On 20/05/14 14:42, Sam Geen wrote:
> Thanks guys, will try those suggestions!
>
> On 20/05/14 14:40, tom fogal wrote:
>> Hi Sam,
>>
>> You might also try gleaning information about why it's not getting 
>> collected with gc.set_debug(...).  For instance, 
>> 'gc.set_debug(gc.DEBUG_LEAK)'.
>>
>> -tom
>>
>> On 05/20/2014 02:28 PM, Sam Geen wrote:
>>> Hmm, nope, memory usage still slowly creeping up (I also tried 
>>> reloading
>>> yt.mods). I have a couple more ideas to try - one is to run the 
>>> function
>>> in a separate process, the other is to check for existing images and
>>> skip that output if they exist, then just re-run every time the memory
>>> fills up. Is there any way of profiling what's in the memory with
>>> Python? I hope/assume it's not numpy or something leaking.
>>>
>>> On 20/05/14 13:48, Matthew Turk wrote:
>>>> Hi Sam,
>>>>
>>>> I've looked over your script and I'm not entirely sure where the
>>>> memory usage is not being freed.  My one thought is to try this:
>>>>
>>>> import yt
>>>> yt.frontends.ramses.RAMSESDataset._skip_cache=True
>>>>
>>>> at the top of your script and seeing if it helps.
>>>>
>>>> -Matt
>>>>
>>>> On Mon, May 19, 2014 at 9:10 AM, Sam Geen <samgeen at astro.ox.ac.uk> 
>>>> wrote:
>>>>> Hi,
>>>>>
>>>>> The memory usage in yt is filling up the machine for a time series
>>>>> with ~100
>>>>> outputs. It seems like the data from previous datasets isn't being
>>>>> cleared
>>>>> from memory, despite calling "del pf" and "gc.collect()". I've tried
>>>>> various
>>>>> suggestions made previously in the mailing list to no effect. I'm
>>>>> using the
>>>>> Ramses frontend with the script attached. Any ideas?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Sam
>>>>>
>>>>> _______________________________________________
>>>>> yt-users mailing list
>>>>> yt-users at lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>
>>>> _______________________________________________
>>>> yt-users mailing list
>>>> yt-users at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: quickplots.py
Type: text/x-python
Size: 2710 bytes
Desc: not available
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20140520/6e15993b/attachment.py>