[yt-dev] profiles on non-prgio sets

[Matthew Turk]

On Wed, Mar 14, 2012 at 12:14 PM, david collins <antpuncher at gmail.com> wrote:
>> 512^3, bunch of subgrids, with 64 cores totalling 12 gb (i.e., just
>> shy of 200mb / core) sounds okay to me.  I would guess you are
>
> Ok, I've pulled together some more details.
>
> If I run this 8 cores running a [Density,Density] profile (so, should
> only have a single field) it fails claiming I'm using 31 Gb, which I
> have a hard time understanding-- that's nearly 4 Gb per core.

Yeah, puzzling.

>
> I did the same analysis with a more recent (PRGIO) 512^3 with
> thousands of grids (4 levels by 2) (many more than in the first set),
> same job script, and it runs fine.
>
> Anyhow, this isn't a huge deal, since non-prgio datasets are far from
> the norm these days, and it works fine in serial.
>
> Does cProfile work in parallel, and does it do memory work?

You have to manually call it to get it not to overwrite its profile
files; I've meant to add something to yt to make this easier, but have
not yet.  It doesn't do memory work.

-Matt

>
> d.
>
>> seriously disk limited for the analysis itself, but you can verify
>> this by running:
>>
>> python2.7 -m cProfile -o ytprof.cprof my_script.py
>>
>> and then using either pyprof2html (requires Jinja2, sold separately)
>> or the stats module to examine it.
>>
>> -Matt
>>
>> On Wed, Mar 14, 2012 at 11:15 AM, david collins <antpuncher at gmail.com> wrote:
>>>> 64 cores, using 12GB on each?  That's a problem.  MassFraction it
>>>
>>> 12Gb total, but still...  I'll grab some more failure info and get
>>> back to you.  In serial it ran fine, and took about 1 hour for both
>>> [density,cellmass] and [density,cellvolume] within the default
>>> head-node allowance on nautilus, which is 12Gb.  Timing wise longer
>>> than I want to deal with, but it does run.
>>>
>>> d.
>>>
>>>> sounds like has a handful of dependencies -- CellMassMsun, which
>>>> depends on Density and dx, but dx should be used as a single value not
>>>> as a field.  So that gives 3*512^3*64 bits = 3 gigabytes, but that's a
>>>> far cry from 12 gb/core.  Plus, the domain decomp for profiles is
>>>> per-grid -- so only one of your cores should get assigned the root
>>>> grid.  This is very bizarre.
>>>>
>>>>>
>>>>> Would lazy_reader off change things?
>>>>
>>>> Yes, but it would make them worse.  :)
>>>>
>>>> -Matt
>>>>
>>>>>
>>>>> Thanks,
>>>>> d.
>>>>>
>>>>>> Are you running in parallel? Which fields are you profiling? Do they have
>>>>>> lots of dependencies, or require ghost zones? Is yt using lazy_reader? Does
>>>>>> your data have many grids?
>>>>>
>>>>>
>>>>>>
>>>>>> Matt
>>>>>>
>>>>>> On Mar 14, 2012 12:37 AM, "david collins" <antpuncher at gmail.com> wrote:
>>>>>>>
>>>>>>> I should add that this was done on 64 cores-- in serial it works fine,
>>>>>>> just slow.
>>>>>>>
>>>>>>> On Tue, Mar 13, 2012 at 9:35 PM, david collins <antpuncher at gmail.com>
>>>>>>> wrote:
>>>>>>> > Hi, all--
>>>>>>> >
>>>>>>> > I have an old dataset that I'm trying to make profiles on.  It's a
>>>>>>> > 512^3 root grid, but was written with ParallelRootGridIO off.  I find
>>>>>>> > that it's using strange amounts of memory, more than 12 Gb.  Is this a
>>>>>>> > known problem with a straight forward work-around?
>>>>>>> >
>>>>>>> > d.
>>>>>>> >
>>>>>>> > --
>>>>>>> > Sent from my computer.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Sent from my computer.
>>>>>>> _______________________________________________
>>>>>>> yt-dev mailing list
>>>>>>> yt-dev at lists.spacepope.org
>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> yt-dev mailing list
>>>>>> yt-dev at lists.spacepope.org
>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Sent from my computer.
>>>>> _______________________________________________
>>>>> yt-dev mailing list
>>>>> yt-dev at lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>> _______________________________________________
>>>> yt-dev mailing list
>>>> yt-dev at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>
>>>
>>>
>>> --
>>> Sent from my computer.
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
>
> --
> Sent from my computer.
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org