[yt-dev] Parallelization & the ART frontend

Wed Feb 8 13:47:31 PST 2012

Hi Chris,

This does help.  My suspicion is that the load balancing is giving all
the lower-level grids to one processor, and all the upper level grids
to another.  If you have time tomorrow, let's work through this
together on IRC.  I'm free 1-2PM EST, 3-5PM EST.  I think it should be
a pretty straightforward fix.

It would also be cool to see your new method of accumulating octs into patches.

-Matt

On Wed, Feb 8, 2012 at 4:09 PM, Christopher Moody <cemoody at ucsc.edu> wrote:
> Hi Matt,
>
> pf.h.proj is of type <class 'yt.data_objects.hierarchy.AMRQuadTreeProj'> and
> refine_by is 2.
>
> Does this help? I'm not sure what you mean by overlaps -  doesn't the the
> RAMSES grid patching mechanism produce non-overlapping grids from the octs?
> Is quadtree proj checking for overlapping grids?
>
> chris
>
> On Wed, Feb 8, 2012 at 12:44 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>>
>> Hi Chris,
>>
>> Yeah, that's weird.  My guess is that load balancing is going haywire
>> for some reason, likely due to overlap versus quadtree proj.  Can you
>> tell me what type of object pf.h.proj is?  i.e., what's the output of
>> "print pf.h.proj"?  And then, what's pf.refine_by?
>>
>> -Matt
>>
>> On Wed, Feb 8, 2012 at 3:00 PM, Christopher Moody <cemoody at ucsc.edu>
>> wrote:
>> > Hi Matt,
>> >
>> > I've got the log output here:http://paste.yt-project.org/show/2153/ with
>> > the
>> > serial version here http://paste.yt-project.org/show/2154/ .
>> >
>> > The most interesting tidbit is below, where it looks like core 0
>> > projects
>> > Levels 0-5 and core 1 projects Level 6 (which takes up like 99% of the
>> > projection time.)
>> >
>> > chris
>> >
>> > P001 yt : [DEBUG    ] 2012-02-08 11:39:53,403 Going to obtain []
>> > P001 yt : [DEBUG    ] 2012-02-08 11:39:53,406 Preloading ['density']
>> > from 0
>> > grids
>> > P001 yt : [DEBUG    ] 2012-02-08 11:39:53,406 End of projecting level
>> > level
>> > 0, memory usage 3.545e-01
>> > P001 yt : [DEBUG    ] 2012-02-08 11:39:53,406 Preloading ['density']
>> > from 0
>> > grids
>> > P001 yt : [DEBUG    ] 2012-02-08 11:39:53,406 End of projecting level
>> > level
>> > 1, memory usage 3.545e-01
>> > P001 yt : [DEBUG    ] 2012-02-08 11:39:53,407 Preloading ['density']
>> > from 0
>> > grids
>> > P001 yt : [DEBUG    ] 2012-02-08 11:39:53,407 End of projecting level
>> > level
>> > 2, memory usage 3.545e-01
>> > P001 yt : [DEBUG    ] 2012-02-08 11:39:53,408 Preloading ['density']
>> > from 0
>> > grids
>> > P001 yt : [DEBUG    ] 2012-02-08 11:39:53,408 End of projecting level
>> > level
>> > 3, memory usage 3.545e-01
>> > P001 yt : [DEBUG    ] 2012-02-08 11:39:53,408 Preloading ['density']
>> > from 0
>> > grids
>> > P001 yt : [DEBUG    ] 2012-02-08 11:39:53,408 End of projecting level
>> > level
>> > 4, memory usage 3.545e-01
>> > P001 yt : [DEBUG    ] 2012-02-08 11:39:53,409 Preloading ['density']
>> > from 0
>> > grids
>> > P001 yt : [DEBUG    ] 2012-02-08 11:39:53,409 End of projecting level
>> > level
>> > 5, memory usage 3.545e-01
>> > P001 yt : [DEBUG    ] 2012-02-08 11:39:53,409 Preloading ['density']
>> > from 6
>> > grids
>> > P001 yt : [INFO     ] 2012-02-08 11:39:53,410 Starting 'Projecting
>> >  level  6
>> > /  6 '
>> > P000 yt : [INFO     ] 2012-02-08 11:39:54,057 Finishing 'Projecting
>> >  level
>> >  0 /  6 '
>> > P000 yt : [DEBUG    ] 2012-02-08 11:39:54,057 End of projecting level
>> > level
>> > 0, memory usage 4.482e-01
>> > P000 yt : [DEBUG    ] 2012-02-08 11:39:54,057 Preloading ['density']
>> > from 1
>> > grids
>> > P000 yt : [INFO     ] 2012-02-08 11:39:54,058 Starting 'Projecting
>> >  level  1
>> > /  6 'P000 yt : [INFO     ] 2012-02-08 11:39:54,070 Finishing
>> > 'Projecting
>> >  level  1 /  6 '
>> > P000 yt : [DEBUG    ] 2012-02-08 11:39:54,070 End of projecting level
>> > level
>> > 1, memory usage 4.482e-01
>> > P000 yt : [DEBUG    ] 2012-02-08 11:39:54,070 Preloading ['density']
>> > from 1
>> > gridsP000 yt : [INFO     ] 2012-02-08 11:39:54,071 Starting 'Projecting
>> >  level  2 /  6 '
>> > P000 yt : [INFO     ] 2012-02-08 11:39:54,130 Finishing 'Projecting
>> >  level
>> >  2 /  6 '
>> > P000 yt : [DEBUG    ] 2012-02-08 11:39:54,130 End of projecting level
>> > level
>> > 2, memory usage 4.482e-01
>> > P000 yt : [DEBUG    ] 2012-02-08 11:39:54,130 Preloading ['density']
>> > from 1
>> > grids
>> > P000 yt : [INFO     ] 2012-02-08 11:39:54,131 Starting 'Projecting
>> >  level  3
>> > /  6 '
>> > P000 yt : [INFO     ] 2012-02-08 11:39:54,783 Finishing 'Projecting
>> >  level
>> >  3 /  6 '
>> > P000 yt : [DEBUG    ] 2012-02-08 11:39:54,784 End of projecting level
>> > level
>> > 3, memory usage 4.482e-01
>> > P000 yt : [DEBUG    ] 2012-02-08 11:39:54,784 Preloading ['density']
>> > from 1
>> > grids
>> > P000 yt : [INFO     ] 2012-02-08 11:39:54,784 Starting 'Projecting
>> >  level  4
>> > /  6 '
>> > P000 yt : [INFO     ] 2012-02-08 11:39:59,389 Finishing 'Projecting
>> >  level
>> >  4 /  6 '
>> > P000 yt : [DEBUG    ] 2012-02-08 11:39:59,389 End of projecting level
>> > level
>> > 4, memory usage 5.918e-01
>> > P000 yt : [DEBUG    ] 2012-02-08 11:39:59,389 Preloading ['density']
>> > from 1
>> > grids
>> > P000 yt : [INFO     ] 2012-02-08 11:39:59,389 Starting 'Projecting
>> >  level  5
>> > /  6 '
>> > P000 yt : [INFO     ] 2012-02-08 11:40:17,735 Finishing 'Projecting
>> >  level
>> >  5 /  6 '
>> > P000 yt : [DEBUG    ] 2012-02-08 11:40:17,735 End of projecting level
>> > level
>> > 5, memory usage 1.569e+00
>> > P000 yt : [DEBUG    ] 2012-02-08 11:40:17,735 Preloading ['density']
>> > from 0
>> > grids
>> > P000 yt : [DEBUG    ] 2012-02-08 11:40:17,736 End of projecting level
>> > level
>> > 6, memory usage 1.569e+00
>> > P001 yt : [INFO     ] 2012-02-08 11:41:31,681 Finishing 'Projecting
>> >  level
>> >  6 /  6 '
>> > P001 yt : [DEBUG    ] 2012-02-08 11:41:31,681 End of projecting level
>> > level
>> > 6, memory usage 2.113e+00
>> > P000 yt : [DEBUG    ] 2012-02-08 11:41:33,807 Opening MPI Barrier on 0
>> > P000 yt : [INFO     ] 2012-02-08 11:41:34,502 Projection completed
>> > P001 yt : [DEBUG    ] 2012-02-08 11:41:34,502 Opening MPI Barrier on 1
>> > P001 yt : [INFO     ] 2012-02-08 11:41:34,502 Projection completed
>> > P001 yt : [DEBUG    ] 2012-02-08 11:41:34,579 Opening MPI Barrier on 1
>> >
>> >
>> >
>> >
>> > On Wed, Feb 8, 2012 at 6:46 AM, Matthew Turk <matthewturk at gmail.com>
>> > wrote:
>> >>
>> >> Hi Chris,
>> >>
>> >> On Tue, Feb 7, 2012 at 9:30 PM, Christopher Moody <cemoody at ucsc.edu>
>> >> wrote:
>> >> > Hi guys,
>> >> >
>> >> > I've been working hard on the ART frontend. Lately, I'm to the point
>> >> > where
>> >> > I'm playing around with more complex datasets that are taking much
>> >> > longer to
>> >> > project - so I'd really like to to start using the parallelization
>> >> > engines.
>> >> > I've tried Sam's workshop parallelization demos, and they all work.
>> >> > But
>> >> > launching with the ART frontend
>> >> > (http://paste.yt-project.org/show/2152/)
>> >> > spawns many independent processes which evidently are not actually
>> >> > splitting
>> >> > the projection job, but still taking up lots of processors.
>> >>
>> >> My guess is that parallelism is not enabled for the ART frontend
>> >> simply as a matter of how the IO is conducted.  To make it really work
>> >> in parallel, the IO needs to be split up so that when process 1 reads
>> >> a given grid patch, the rest of the processors don't also need to read
>> >> all the data for that grid patch.
>> >>
>> >> Can you lower your loglevel (by setting loglevel = 1 in ~/.yt/config
>> >> or by --config yt.loglevel=1 on the command line) and report back with
>> >> what it says during a projection job there?
>> >>
>> >> -MAtt
>> >>
>> >> >
>> >> > My mpi installation  works:
>> >> > yt : [INFO     ] 2012-02-07 18:12:28,207 Global parallel computation
>> >> > enabled: 0 / 8yt : [INFO     ] 2012-02-07 18:12:28,207 Global
>> >> > parallel
>> >> > computation enabled: 2 / 8
>> >> > yt : [INFO     ] 2012-02-07 18:12:28,208 Global parallel computation
>> >> > enabled: 1 / 8
>> >> > yt : [INFO     ] 2012-02-07 18:12:28,208 Global parallel computation
>> >> > enabled: 6 / 8
>> >> > yt : [INFO     ] 2012-02-07 18:12:28,208 Global parallel computation
>> >> > enabled: 3 / 8
>> >> > yt : [INFO     ] 2012-02-07 18:12:28,208 Global parallel computation
>> >> > enabled: 4 / 8
>> >> > yt : [INFO     ] 2012-02-07 18:12:28,208 Global parallel computation
>> >> > enabled: 5 / 8yt : [INFO     ] 2012-02-07 18:12:28,209 Global
>> >> > parallel
>> >> > computation enabled: 7 / 8
>> >> >
>> >> > But the script is just run 8 times, not any faster.
>> >> >
>> >> > What am I missing here?
>> >> >
>> >> > Many thanks!
>> >> > chris
>> >> >
>> >> > _______________________________________________
>> >> > yt-dev mailing list
>> >> > yt-dev at lists.spacepope.org
>> >> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >> >
>> >> _______________________________________________
>> >> yt-dev mailing list
>> >> yt-dev at lists.spacepope.org
>> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >
>> >
>> >
>> > _______________________________________________
>> > yt-dev mailing list
>> > yt-dev at lists.spacepope.org
>> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>