[yt-dev] profiles on non-prgio sets

[Matthew Turk]

Hi Dave,

On Wed, Mar 14, 2012 at 2:18 PM, Matthew Turk <matthewturk at gmail.com> wrote:
> On Wed, Mar 14, 2012 at 12:14 PM, david collins <antpuncher at gmail.com> wrote:
>>> 512^3, bunch of subgrids, with 64 cores totalling 12 gb (i.e., just
>>> shy of 200mb / core) sounds okay to me.  I would guess you are
>>
>> Ok, I've pulled together some more details.
>>
>> If I run this 8 cores running a [Density,Density] profile (so, should
>> only have a single field) it fails claiming I'm using 31 Gb, which I
>> have a hard time understanding-- that's nearly 4 Gb per core.
>
> Yeah, puzzling.
>
>>
>> I did the same analysis with a more recent (PRGIO) 512^3 with
>> thousands of grids (4 levels by 2) (many more than in the first set),
>> same job script, and it runs fine.
>>
>> Anyhow, this isn't a huge deal, since non-prgio datasets are far from
>> the norm these days, and it works fine in serial.
>>
>> Does cProfile work in parallel, and does it do memory work?
>
> You have to manually call it to get it not to overwrite its profile
> files; I've meant to add something to yt to make this easier, but have
> not yet.  It doesn't do memory work.

For what it's worth, in the PR I just issued for including beta-status
Rockstar interfacing, I've added a parallel profiler.  You use it
like:

with parallel_profile("some_prefix"):
    do_something()
    something_else()

and it will write out .cprof files named as per processor in the top
communicator.

-Matt

>
> -Matt
>
>>
>> d.
>>
>>> seriously disk limited for the analysis itself, but you can verify
>>> this by running:
>>>
>>> python2.7 -m cProfile -o ytprof.cprof my_script.py
>>>
>>> and then using either pyprof2html (requires Jinja2, sold separately)
>>> or the stats module to examine it.
>>>
>>> -Matt
>>>
>>> On Wed, Mar 14, 2012 at 11:15 AM, david collins <antpuncher at gmail.com> wrote:
>>>>> 64 cores, using 12GB on each?  That's a problem.  MassFraction it
>>>>
>>>> 12Gb total, but still...  I'll grab some more failure info and get
>>>> back to you.  In serial it ran fine, and took about 1 hour for both
>>>> [density,cellmass] and [density,cellvolume] within the default
>>>> head-node allowance on nautilus, which is 12Gb.  Timing wise longer
>>>> than I want to deal with, but it does run.
>>>>
>>>> d.
>>>>
>>>>> sounds like has a handful of dependencies -- CellMassMsun, which
>>>>> depends on Density and dx, but dx should be used as a single value not
>>>>> as a field.  So that gives 3*512^3*64 bits = 3 gigabytes, but that's a
>>>>> far cry from 12 gb/core.  Plus, the domain decomp for profiles is
>>>>> per-grid -- so only one of your cores should get assigned the root
>>>>> grid.  This is very bizarre.
>>>>>
>>>>>>
>>>>>> Would lazy_reader off change things?
>>>>>
>>>>> Yes, but it would make them worse.  :)
>>>>>
>>>>> -Matt
>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> d.
>>>>>>
>>>>>>> Are you running in parallel? Which fields are you profiling? Do they have
>>>>>>> lots of dependencies, or require ghost zones? Is yt using lazy_reader? Does
>>>>>>> your data have many grids?
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Matt
>>>>>>>
>>>>>>> On Mar 14, 2012 12:37 AM, "david collins" <antpuncher at gmail.com> wrote:
>>>>>>>>
>>>>>>>> I should add that this was done on 64 cores-- in serial it works fine,
>>>>>>>> just slow.
>>>>>>>>
>>>>>>>> On Tue, Mar 13, 2012 at 9:35 PM, david collins <antpuncher at gmail.com>
>>>>>>>> wrote:
>>>>>>>> > Hi, all--
>>>>>>>> >
>>>>>>>> > I have an old dataset that I'm trying to make profiles on.  It's a
>>>>>>>> > 512^3 root grid, but was written with ParallelRootGridIO off.  I find
>>>>>>>> > that it's using strange amounts of memory, more than 12 Gb.  Is this a
>>>>>>>> > known problem with a straight forward work-around?
>>>>>>>> >
>>>>>>>> > d.
>>>>>>>> >
>>>>>>>> > --
>>>>>>>> > Sent from my computer.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Sent from my computer.
>>>>>>>> _______________________________________________
>>>>>>>> yt-dev mailing list
>>>>>>>> yt-dev at lists.spacepope.org
>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> yt-dev mailing list
>>>>>>> yt-dev at lists.spacepope.org
>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Sent from my computer.
>>>>>> _______________________________________________
>>>>>> yt-dev mailing list
>>>>>> yt-dev at lists.spacepope.org
>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>> _______________________________________________
>>>>> yt-dev mailing list
>>>>> yt-dev at lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>
>>>>
>>>>
>>>> --
>>>> Sent from my computer.
>>>> _______________________________________________
>>>> yt-dev mailing list
>>>> yt-dev at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>>
>>
>> --
>> Sent from my computer.
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org