[yt-dev] Zombie jobs on eudora?

[Nathan Goldbaum]

Hey all,

I'm looking at a memory leak that Philip (cc'd) is seeing when iterating
over a long list of FLASH datasets.  Just as an example of the type of
behavior he is seeing - today he left his script running and ended up
consuming 300 GB of RAM on a viz node.

FWIW, the dataset is not particularly large - ~300 outputs and ~100 MB per
output. These are also FLASH cylindrical coordinate simulations - so
perhaps this behavior will only occur in curvilinear geometries?

I've been playing with objgraph to try to understand what's happening.
 Here's the script I've been using: http://paste.yt-project.org/show/4762/

Here's the output after one iteration of the for loop:
http://paste.yt-project.org/show/4761/

It seems that for some reason a lot of data is not being garbage collected.


Could there be a reference counting bug somewhere down in a cython routine?
Objgraph is unable to find backreferences to root grid tiles in the flash
dataset, and all the other yt objects that I've looked at seem to have
backreference graphs that terminate at a FLASHGrid object that represents a
root grid tile in one of the datasets.  That's the best guess I have - but
definitely nothing conclusive.  I'd appreciate any other ideas anyone else
has to help debug this.

Thanks for your help in debugging this!

-Nathan



On Mon, Jun 9, 2014 at 7:36 PM, Phillip Macias <pmacias at ucsc.edu> wrote:

> Done. Also this sim is just a test haha so don't judge the crappy quality
> :)
>
>
> On Mon, Jun 9, 2014 at 7:28 PM, Nathan Goldbaum <nathan12343 at gmail.com>
> wrote:
>
>> Any chance you could make the folder that the dataset lives in visible to
>> me?
>>
>> chmod -R +r /pfs/pmacias/runs/hoseon/2pi/diff_angle/coriolis/
>>
>> should do the trick.
>>
>>
>> On Mon, Jun 9, 2014 at 7:25 PM, Nathan Goldbaum <nathan12343 at gmail.com>
>> wrote:
>>
>>> This looks like a memory leak in yt.  Let me see if can figure out
>>> what's going wrong.
>>>
>>>
>>> On Mon, Jun 9, 2014 at 7:21 PM, Phillip Macias <pmacias at ucsc.edu> wrote:
>>>
>>>> No problem, sorry to be a pain!
>>>>
>>>> I've attached the script. To run I'm just sourcing the yt installation
>>>> on hyades and doing
>>>>
>>>> python dens_plot.py
>>>>
>>>> Maybe I need to be clearing some things as I go through the loop? Idk
>>>> I'm sure it's something really silly :/
>>>>
>>>> Thanks again!
>>>> -Phil
>>>>
>>>>
>>>> On Mon, Jun 9, 2014 at 6:56 PM, Nathan Goldbaum <nathan12343 at gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks for killing them.  I'd be happy to take a look at your script.
>>>>>
>>>>>
>>>>> On Mon, Jun 9, 2014 at 6:46 PM, Phillip Macias <pmacias at ucsc.edu>
>>>>> wrote:
>>>>>
>>>>>> Hey  Nathan,
>>>>>>
>>>>>> Sorry about that!! I was actually running yt scripts, I'm not sure
>>>>>> how they could use so much ram :(. When I get home can I send you my script
>>>>>> and you can tell me how I'm being dumb?
>>>>>>
>>>>>> -Phil
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Jun 9, 2014, at 6:43 PM, Nathan Goldbaum <nathan12343 at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> > Hey Philip,
>>>>>> >
>>>>>> > It looks like you have two scripts that have been running for 7
>>>>>> hours - using ~300 GB of ram.  Are these zombie processes?
>>>>>> >
>>>>>> > -Nathan
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20140609/62353b2d/attachment.htm>