[yt-dev] Parallelization & the ART frontend
Christopher Moody
cemoody at ucsc.edu
Wed Feb 8 14:20:24 PST 2012
Hi Matt,
Sounds good! See ya at 1pm EST.
I've played around with lots of gridding mechanism now, with lots of
fiddling of parameters. I now don't think growing octs along a sparse
Hilbert curve (what I've been playing around with) is particularly more
efficient than splitting clumps on a coarse curve (default). It's also
tough to diagnose how to speed stuff up; an efficient hierarchy with many
small grids (+800 grids on a level) is easy on memory but takes x100 longer
to project. It's hard to guess how stuff scales with ncells, # of grids on
a level, and memory efficiency all being independent variables. In the
end, all I've done is made a few very, very, small changes to some of the
grid patch recursive splitting code, which has had a dramatic effect (~5x
speedup) for my simulations, but I don't think will be too helpful outside
of that.
chris
On Wed, Feb 8, 2012 at 1:47 PM, Matthew Turk <matthewturk at gmail.com> wrote:
> Hi Chris,
>
> This does help. My suspicion is that the load balancing is giving all
> the lower-level grids to one processor, and all the upper level grids
> to another. If you have time tomorrow, let's work through this
> together on IRC. I'm free 1-2PM EST, 3-5PM EST. I think it should be
> a pretty straightforward fix.
>
> It would also be cool to see your new method of accumulating octs into
> patches.
>
> -Matt
>
> On Wed, Feb 8, 2012 at 4:09 PM, Christopher Moody <cemoody at ucsc.edu>
> wrote:
> > Hi Matt,
> >
> > pf.h.proj is of type <class 'yt.data_objects.hierarchy.AMRQuadTreeProj'>
> and
> > refine_by is 2.
> >
> > Does this help? I'm not sure what you mean by overlaps - doesn't the the
> > RAMSES grid patching mechanism produce non-overlapping grids from the
> octs?
> > Is quadtree proj checking for overlapping grids?
> >
> > chris
> >
> > On Wed, Feb 8, 2012 at 12:44 PM, Matthew Turk <matthewturk at gmail.com>
> wrote:
> >>
> >> Hi Chris,
> >>
> >> Yeah, that's weird. My guess is that load balancing is going haywire
> >> for some reason, likely due to overlap versus quadtree proj. Can you
> >> tell me what type of object pf.h.proj is? i.e., what's the output of
> >> "print pf.h.proj"? And then, what's pf.refine_by?
> >>
> >> -Matt
> >>
> >> On Wed, Feb 8, 2012 at 3:00 PM, Christopher Moody <cemoody at ucsc.edu>
> >> wrote:
> >> > Hi Matt,
> >> >
> >> > I've got the log output here:http://paste.yt-project.org/show/2153/with
> >> > the
> >> > serial version here http://paste.yt-project.org/show/2154/ .
> >> >
> >> > The most interesting tidbit is below, where it looks like core 0
> >> > projects
> >> > Levels 0-5 and core 1 projects Level 6 (which takes up like 99% of the
> >> > projection time.)
> >> >
> >> > chris
> >> >
> >> > P001 yt : [DEBUG ] 2012-02-08 11:39:53,403 Going to obtain []
> >> > P001 yt : [DEBUG ] 2012-02-08 11:39:53,406 Preloading ['density']
> >> > from 0
> >> > grids
> >> > P001 yt : [DEBUG ] 2012-02-08 11:39:53,406 End of projecting level
> >> > level
> >> > 0, memory usage 3.545e-01
> >> > P001 yt : [DEBUG ] 2012-02-08 11:39:53,406 Preloading ['density']
> >> > from 0
> >> > grids
> >> > P001 yt : [DEBUG ] 2012-02-08 11:39:53,406 End of projecting level
> >> > level
> >> > 1, memory usage 3.545e-01
> >> > P001 yt : [DEBUG ] 2012-02-08 11:39:53,407 Preloading ['density']
> >> > from 0
> >> > grids
> >> > P001 yt : [DEBUG ] 2012-02-08 11:39:53,407 End of projecting level
> >> > level
> >> > 2, memory usage 3.545e-01
> >> > P001 yt : [DEBUG ] 2012-02-08 11:39:53,408 Preloading ['density']
> >> > from 0
> >> > grids
> >> > P001 yt : [DEBUG ] 2012-02-08 11:39:53,408 End of projecting level
> >> > level
> >> > 3, memory usage 3.545e-01
> >> > P001 yt : [DEBUG ] 2012-02-08 11:39:53,408 Preloading ['density']
> >> > from 0
> >> > grids
> >> > P001 yt : [DEBUG ] 2012-02-08 11:39:53,408 End of projecting level
> >> > level
> >> > 4, memory usage 3.545e-01
> >> > P001 yt : [DEBUG ] 2012-02-08 11:39:53,409 Preloading ['density']
> >> > from 0
> >> > grids
> >> > P001 yt : [DEBUG ] 2012-02-08 11:39:53,409 End of projecting level
> >> > level
> >> > 5, memory usage 3.545e-01
> >> > P001 yt : [DEBUG ] 2012-02-08 11:39:53,409 Preloading ['density']
> >> > from 6
> >> > grids
> >> > P001 yt : [INFO ] 2012-02-08 11:39:53,410 Starting 'Projecting
> >> > level 6
> >> > / 6 '
> >> > P000 yt : [INFO ] 2012-02-08 11:39:54,057 Finishing 'Projecting
> >> > level
> >> > 0 / 6 '
> >> > P000 yt : [DEBUG ] 2012-02-08 11:39:54,057 End of projecting level
> >> > level
> >> > 0, memory usage 4.482e-01
> >> > P000 yt : [DEBUG ] 2012-02-08 11:39:54,057 Preloading ['density']
> >> > from 1
> >> > grids
> >> > P000 yt : [INFO ] 2012-02-08 11:39:54,058 Starting 'Projecting
> >> > level 1
> >> > / 6 'P000 yt : [INFO ] 2012-02-08 11:39:54,070 Finishing
> >> > 'Projecting
> >> > level 1 / 6 '
> >> > P000 yt : [DEBUG ] 2012-02-08 11:39:54,070 End of projecting level
> >> > level
> >> > 1, memory usage 4.482e-01
> >> > P000 yt : [DEBUG ] 2012-02-08 11:39:54,070 Preloading ['density']
> >> > from 1
> >> > gridsP000 yt : [INFO ] 2012-02-08 11:39:54,071 Starting
> 'Projecting
> >> > level 2 / 6 '
> >> > P000 yt : [INFO ] 2012-02-08 11:39:54,130 Finishing 'Projecting
> >> > level
> >> > 2 / 6 '
> >> > P000 yt : [DEBUG ] 2012-02-08 11:39:54,130 End of projecting level
> >> > level
> >> > 2, memory usage 4.482e-01
> >> > P000 yt : [DEBUG ] 2012-02-08 11:39:54,130 Preloading ['density']
> >> > from 1
> >> > grids
> >> > P000 yt : [INFO ] 2012-02-08 11:39:54,131 Starting 'Projecting
> >> > level 3
> >> > / 6 '
> >> > P000 yt : [INFO ] 2012-02-08 11:39:54,783 Finishing 'Projecting
> >> > level
> >> > 3 / 6 '
> >> > P000 yt : [DEBUG ] 2012-02-08 11:39:54,784 End of projecting level
> >> > level
> >> > 3, memory usage 4.482e-01
> >> > P000 yt : [DEBUG ] 2012-02-08 11:39:54,784 Preloading ['density']
> >> > from 1
> >> > grids
> >> > P000 yt : [INFO ] 2012-02-08 11:39:54,784 Starting 'Projecting
> >> > level 4
> >> > / 6 '
> >> > P000 yt : [INFO ] 2012-02-08 11:39:59,389 Finishing 'Projecting
> >> > level
> >> > 4 / 6 '
> >> > P000 yt : [DEBUG ] 2012-02-08 11:39:59,389 End of projecting level
> >> > level
> >> > 4, memory usage 5.918e-01
> >> > P000 yt : [DEBUG ] 2012-02-08 11:39:59,389 Preloading ['density']
> >> > from 1
> >> > grids
> >> > P000 yt : [INFO ] 2012-02-08 11:39:59,389 Starting 'Projecting
> >> > level 5
> >> > / 6 '
> >> > P000 yt : [INFO ] 2012-02-08 11:40:17,735 Finishing 'Projecting
> >> > level
> >> > 5 / 6 '
> >> > P000 yt : [DEBUG ] 2012-02-08 11:40:17,735 End of projecting level
> >> > level
> >> > 5, memory usage 1.569e+00
> >> > P000 yt : [DEBUG ] 2012-02-08 11:40:17,735 Preloading ['density']
> >> > from 0
> >> > grids
> >> > P000 yt : [DEBUG ] 2012-02-08 11:40:17,736 End of projecting level
> >> > level
> >> > 6, memory usage 1.569e+00
> >> > P001 yt : [INFO ] 2012-02-08 11:41:31,681 Finishing 'Projecting
> >> > level
> >> > 6 / 6 '
> >> > P001 yt : [DEBUG ] 2012-02-08 11:41:31,681 End of projecting level
> >> > level
> >> > 6, memory usage 2.113e+00
> >> > P000 yt : [DEBUG ] 2012-02-08 11:41:33,807 Opening MPI Barrier on 0
> >> > P000 yt : [INFO ] 2012-02-08 11:41:34,502 Projection completed
> >> > P001 yt : [DEBUG ] 2012-02-08 11:41:34,502 Opening MPI Barrier on 1
> >> > P001 yt : [INFO ] 2012-02-08 11:41:34,502 Projection completed
> >> > P001 yt : [DEBUG ] 2012-02-08 11:41:34,579 Opening MPI Barrier on 1
> >> >
> >> >
> >> >
> >> >
> >> > On Wed, Feb 8, 2012 at 6:46 AM, Matthew Turk <matthewturk at gmail.com>
> >> > wrote:
> >> >>
> >> >> Hi Chris,
> >> >>
> >> >> On Tue, Feb 7, 2012 at 9:30 PM, Christopher Moody <cemoody at ucsc.edu>
> >> >> wrote:
> >> >> > Hi guys,
> >> >> >
> >> >> > I've been working hard on the ART frontend. Lately, I'm to the
> point
> >> >> > where
> >> >> > I'm playing around with more complex datasets that are taking much
> >> >> > longer to
> >> >> > project - so I'd really like to to start using the parallelization
> >> >> > engines.
> >> >> > I've tried Sam's workshop parallelization demos, and they all work.
> >> >> > But
> >> >> > launching with the ART frontend
> >> >> > (http://paste.yt-project.org/show/2152/)
> >> >> > spawns many independent processes which evidently are not actually
> >> >> > splitting
> >> >> > the projection job, but still taking up lots of processors.
> >> >>
> >> >> My guess is that parallelism is not enabled for the ART frontend
> >> >> simply as a matter of how the IO is conducted. To make it really
> work
> >> >> in parallel, the IO needs to be split up so that when process 1 reads
> >> >> a given grid patch, the rest of the processors don't also need to
> read
> >> >> all the data for that grid patch.
> >> >>
> >> >> Can you lower your loglevel (by setting loglevel = 1 in ~/.yt/config
> >> >> or by --config yt.loglevel=1 on the command line) and report back
> with
> >> >> what it says during a projection job there?
> >> >>
> >> >> -MAtt
> >> >>
> >> >> >
> >> >> > My mpi installation works:
> >> >> > yt : [INFO ] 2012-02-07 18:12:28,207 Global parallel
> computation
> >> >> > enabled: 0 / 8yt : [INFO ] 2012-02-07 18:12:28,207 Global
> >> >> > parallel
> >> >> > computation enabled: 2 / 8
> >> >> > yt : [INFO ] 2012-02-07 18:12:28,208 Global parallel
> computation
> >> >> > enabled: 1 / 8
> >> >> > yt : [INFO ] 2012-02-07 18:12:28,208 Global parallel
> computation
> >> >> > enabled: 6 / 8
> >> >> > yt : [INFO ] 2012-02-07 18:12:28,208 Global parallel
> computation
> >> >> > enabled: 3 / 8
> >> >> > yt : [INFO ] 2012-02-07 18:12:28,208 Global parallel
> computation
> >> >> > enabled: 4 / 8
> >> >> > yt : [INFO ] 2012-02-07 18:12:28,208 Global parallel
> computation
> >> >> > enabled: 5 / 8yt : [INFO ] 2012-02-07 18:12:28,209 Global
> >> >> > parallel
> >> >> > computation enabled: 7 / 8
> >> >> >
> >> >> > But the script is just run 8 times, not any faster.
> >> >> >
> >> >> > What am I missing here?
> >> >> >
> >> >> > Many thanks!
> >> >> > chris
> >> >> >
> >> >> > _______________________________________________
> >> >> > yt-dev mailing list
> >> >> > yt-dev at lists.spacepope.org
> >> >> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >> >> >
> >> >> _______________________________________________
> >> >> yt-dev mailing list
> >> >> yt-dev at lists.spacepope.org
> >> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >> >
> >> >
> >> >
> >> > _______________________________________________
> >> > yt-dev mailing list
> >> > yt-dev at lists.spacepope.org
> >> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >> >
> >> _______________________________________________
> >> yt-dev mailing list
> >> yt-dev at lists.spacepope.org
> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >
> >
> >
> > _______________________________________________
> > yt-dev mailing list
> > yt-dev at lists.spacepope.org
> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20120208/21db423d/attachment.htm>
More information about the yt-dev
mailing list