Hi Matt,<div>Sounds good! See ya at 1pm EST.</div><div><br></div><div>I've played around with lots of gridding mechanism now, with lots of fiddling of parameters. I now don't think growing octs along a sparse Hilbert curve (what I've been playing around with) is particularly more efficient than splitting clumps on a coarse curve (default). It's also tough to diagnose how to speed stuff up; an efficient hierarchy with many small grids (+800 grids on a level) is easy on memory but takes x100 longer to project. It's hard to guess how stuff scales with ncells, # of grids on a level, and memory efficiency all being independent variables. In the end, all I've done is made a few very, very, small changes to some of the grid patch recursive splitting code, which has had a dramatic effect (~5x speedup) for my simulations, but I don't think will be too helpful outside of that. </div>
<div><br></div><div>chris</div><div><br><div class="gmail_quote">On Wed, Feb 8, 2012 at 1:47 PM, Matthew Turk <span dir="ltr"><<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Chris,<br>
<br>
This does help. My suspicion is that the load balancing is giving all<br>
the lower-level grids to one processor, and all the upper level grids<br>
to another. If you have time tomorrow, let's work through this<br>
together on IRC. I'm free 1-2PM EST, 3-5PM EST. I think it should be<br>
a pretty straightforward fix.<br>
<br>
It would also be cool to see your new method of accumulating octs into patches.<br>
<br>
-Matt<br>
<div class="HOEnZb"><div class="h5"><br>
On Wed, Feb 8, 2012 at 4:09 PM, Christopher Moody <<a href="mailto:cemoody@ucsc.edu">cemoody@ucsc.edu</a>> wrote:<br>
> Hi Matt,<br>
><br>
> pf.h.proj is of type <class 'yt.data_objects.hierarchy.AMRQuadTreeProj'> and<br>
> refine_by is 2.<br>
><br>
> Does this help? I'm not sure what you mean by overlaps - doesn't the the<br>
> RAMSES grid patching mechanism produce non-overlapping grids from the octs?<br>
> Is quadtree proj checking for overlapping grids?<br>
><br>
> chris<br>
><br>
> On Wed, Feb 8, 2012 at 12:44 PM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>> wrote:<br>
>><br>
>> Hi Chris,<br>
>><br>
>> Yeah, that's weird. My guess is that load balancing is going haywire<br>
>> for some reason, likely due to overlap versus quadtree proj. Can you<br>
>> tell me what type of object pf.h.proj is? i.e., what's the output of<br>
>> "print pf.h.proj"? And then, what's pf.refine_by?<br>
>><br>
>> -Matt<br>
>><br>
>> On Wed, Feb 8, 2012 at 3:00 PM, Christopher Moody <<a href="mailto:cemoody@ucsc.edu">cemoody@ucsc.edu</a>><br>
>> wrote:<br>
>> > Hi Matt,<br>
>> ><br>
>> > I've got the log output here:<a href="http://paste.yt-project.org/show/2153/" target="_blank">http://paste.yt-project.org/show/2153/</a> with<br>
>> > the<br>
>> > serial version here <a href="http://paste.yt-project.org/show/2154/" target="_blank">http://paste.yt-project.org/show/2154/</a> .<br>
>> ><br>
>> > The most interesting tidbit is below, where it looks like core 0<br>
>> > projects<br>
>> > Levels 0-5 and core 1 projects Level 6 (which takes up like 99% of the<br>
>> > projection time.)<br>
>> ><br>
>> > chris<br>
>> ><br>
>> > P001 yt : [DEBUG ] 2012-02-08 11:39:53,403 Going to obtain []<br>
>> > P001 yt : [DEBUG ] 2012-02-08 11:39:53,406 Preloading ['density']<br>
>> > from 0<br>
>> > grids<br>
>> > P001 yt : [DEBUG ] 2012-02-08 11:39:53,406 End of projecting level<br>
>> > level<br>
>> > 0, memory usage 3.545e-01<br>
>> > P001 yt : [DEBUG ] 2012-02-08 11:39:53,406 Preloading ['density']<br>
>> > from 0<br>
>> > grids<br>
>> > P001 yt : [DEBUG ] 2012-02-08 11:39:53,406 End of projecting level<br>
>> > level<br>
>> > 1, memory usage 3.545e-01<br>
>> > P001 yt : [DEBUG ] 2012-02-08 11:39:53,407 Preloading ['density']<br>
>> > from 0<br>
>> > grids<br>
>> > P001 yt : [DEBUG ] 2012-02-08 11:39:53,407 End of projecting level<br>
>> > level<br>
>> > 2, memory usage 3.545e-01<br>
>> > P001 yt : [DEBUG ] 2012-02-08 11:39:53,408 Preloading ['density']<br>
>> > from 0<br>
>> > grids<br>
>> > P001 yt : [DEBUG ] 2012-02-08 11:39:53,408 End of projecting level<br>
>> > level<br>
>> > 3, memory usage 3.545e-01<br>
>> > P001 yt : [DEBUG ] 2012-02-08 11:39:53,408 Preloading ['density']<br>
>> > from 0<br>
>> > grids<br>
>> > P001 yt : [DEBUG ] 2012-02-08 11:39:53,408 End of projecting level<br>
>> > level<br>
>> > 4, memory usage 3.545e-01<br>
>> > P001 yt : [DEBUG ] 2012-02-08 11:39:53,409 Preloading ['density']<br>
>> > from 0<br>
>> > grids<br>
>> > P001 yt : [DEBUG ] 2012-02-08 11:39:53,409 End of projecting level<br>
>> > level<br>
>> > 5, memory usage 3.545e-01<br>
>> > P001 yt : [DEBUG ] 2012-02-08 11:39:53,409 Preloading ['density']<br>
>> > from 6<br>
>> > grids<br>
>> > P001 yt : [INFO ] 2012-02-08 11:39:53,410 Starting 'Projecting<br>
>> > level 6<br>
>> > / 6 '<br>
>> > P000 yt : [INFO ] 2012-02-08 11:39:54,057 Finishing 'Projecting<br>
>> > level<br>
>> > 0 / 6 '<br>
>> > P000 yt : [DEBUG ] 2012-02-08 11:39:54,057 End of projecting level<br>
>> > level<br>
>> > 0, memory usage 4.482e-01<br>
>> > P000 yt : [DEBUG ] 2012-02-08 11:39:54,057 Preloading ['density']<br>
>> > from 1<br>
>> > grids<br>
>> > P000 yt : [INFO ] 2012-02-08 11:39:54,058 Starting 'Projecting<br>
>> > level 1<br>
>> > / 6 'P000 yt : [INFO ] 2012-02-08 11:39:54,070 Finishing<br>
>> > 'Projecting<br>
>> > level 1 / 6 '<br>
>> > P000 yt : [DEBUG ] 2012-02-08 11:39:54,070 End of projecting level<br>
>> > level<br>
>> > 1, memory usage 4.482e-01<br>
>> > P000 yt : [DEBUG ] 2012-02-08 11:39:54,070 Preloading ['density']<br>
>> > from 1<br>
>> > gridsP000 yt : [INFO ] 2012-02-08 11:39:54,071 Starting 'Projecting<br>
>> > level 2 / 6 '<br>
>> > P000 yt : [INFO ] 2012-02-08 11:39:54,130 Finishing 'Projecting<br>
>> > level<br>
>> > 2 / 6 '<br>
>> > P000 yt : [DEBUG ] 2012-02-08 11:39:54,130 End of projecting level<br>
>> > level<br>
>> > 2, memory usage 4.482e-01<br>
>> > P000 yt : [DEBUG ] 2012-02-08 11:39:54,130 Preloading ['density']<br>
>> > from 1<br>
>> > grids<br>
>> > P000 yt : [INFO ] 2012-02-08 11:39:54,131 Starting 'Projecting<br>
>> > level 3<br>
>> > / 6 '<br>
>> > P000 yt : [INFO ] 2012-02-08 11:39:54,783 Finishing 'Projecting<br>
>> > level<br>
>> > 3 / 6 '<br>
>> > P000 yt : [DEBUG ] 2012-02-08 11:39:54,784 End of projecting level<br>
>> > level<br>
>> > 3, memory usage 4.482e-01<br>
>> > P000 yt : [DEBUG ] 2012-02-08 11:39:54,784 Preloading ['density']<br>
>> > from 1<br>
>> > grids<br>
>> > P000 yt : [INFO ] 2012-02-08 11:39:54,784 Starting 'Projecting<br>
>> > level 4<br>
>> > / 6 '<br>
>> > P000 yt : [INFO ] 2012-02-08 11:39:59,389 Finishing 'Projecting<br>
>> > level<br>
>> > 4 / 6 '<br>
>> > P000 yt : [DEBUG ] 2012-02-08 11:39:59,389 End of projecting level<br>
>> > level<br>
>> > 4, memory usage 5.918e-01<br>
>> > P000 yt : [DEBUG ] 2012-02-08 11:39:59,389 Preloading ['density']<br>
>> > from 1<br>
>> > grids<br>
>> > P000 yt : [INFO ] 2012-02-08 11:39:59,389 Starting 'Projecting<br>
>> > level 5<br>
>> > / 6 '<br>
>> > P000 yt : [INFO ] 2012-02-08 11:40:17,735 Finishing 'Projecting<br>
>> > level<br>
>> > 5 / 6 '<br>
>> > P000 yt : [DEBUG ] 2012-02-08 11:40:17,735 End of projecting level<br>
>> > level<br>
>> > 5, memory usage 1.569e+00<br>
>> > P000 yt : [DEBUG ] 2012-02-08 11:40:17,735 Preloading ['density']<br>
>> > from 0<br>
>> > grids<br>
>> > P000 yt : [DEBUG ] 2012-02-08 11:40:17,736 End of projecting level<br>
>> > level<br>
>> > 6, memory usage 1.569e+00<br>
>> > P001 yt : [INFO ] 2012-02-08 11:41:31,681 Finishing 'Projecting<br>
>> > level<br>
>> > 6 / 6 '<br>
>> > P001 yt : [DEBUG ] 2012-02-08 11:41:31,681 End of projecting level<br>
>> > level<br>
>> > 6, memory usage 2.113e+00<br>
>> > P000 yt : [DEBUG ] 2012-02-08 11:41:33,807 Opening MPI Barrier on 0<br>
>> > P000 yt : [INFO ] 2012-02-08 11:41:34,502 Projection completed<br>
>> > P001 yt : [DEBUG ] 2012-02-08 11:41:34,502 Opening MPI Barrier on 1<br>
>> > P001 yt : [INFO ] 2012-02-08 11:41:34,502 Projection completed<br>
>> > P001 yt : [DEBUG ] 2012-02-08 11:41:34,579 Opening MPI Barrier on 1<br>
>> ><br>
>> ><br>
>> ><br>
>> ><br>
>> > On Wed, Feb 8, 2012 at 6:46 AM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
>> > wrote:<br>
>> >><br>
>> >> Hi Chris,<br>
>> >><br>
>> >> On Tue, Feb 7, 2012 at 9:30 PM, Christopher Moody <<a href="mailto:cemoody@ucsc.edu">cemoody@ucsc.edu</a>><br>
>> >> wrote:<br>
>> >> > Hi guys,<br>
>> >> ><br>
>> >> > I've been working hard on the ART frontend. Lately, I'm to the point<br>
>> >> > where<br>
>> >> > I'm playing around with more complex datasets that are taking much<br>
>> >> > longer to<br>
>> >> > project - so I'd really like to to start using the parallelization<br>
>> >> > engines.<br>
>> >> > I've tried Sam's workshop parallelization demos, and they all work.<br>
>> >> > But<br>
>> >> > launching with the ART frontend<br>
>> >> > (<a href="http://paste.yt-project.org/show/2152/" target="_blank">http://paste.yt-project.org/show/2152/</a>)<br>
>> >> > spawns many independent processes which evidently are not actually<br>
>> >> > splitting<br>
>> >> > the projection job, but still taking up lots of processors.<br>
>> >><br>
>> >> My guess is that parallelism is not enabled for the ART frontend<br>
>> >> simply as a matter of how the IO is conducted. To make it really work<br>
>> >> in parallel, the IO needs to be split up so that when process 1 reads<br>
>> >> a given grid patch, the rest of the processors don't also need to read<br>
>> >> all the data for that grid patch.<br>
>> >><br>
>> >> Can you lower your loglevel (by setting loglevel = 1 in ~/.yt/config<br>
>> >> or by --config yt.loglevel=1 on the command line) and report back with<br>
>> >> what it says during a projection job there?<br>
>> >><br>
>> >> -MAtt<br>
>> >><br>
>> >> ><br>
>> >> > My mpi installation works:<br>
>> >> > yt : [INFO ] 2012-02-07 18:12:28,207 Global parallel computation<br>
>> >> > enabled: 0 / 8yt : [INFO ] 2012-02-07 18:12:28,207 Global<br>
>> >> > parallel<br>
>> >> > computation enabled: 2 / 8<br>
>> >> > yt : [INFO ] 2012-02-07 18:12:28,208 Global parallel computation<br>
>> >> > enabled: 1 / 8<br>
>> >> > yt : [INFO ] 2012-02-07 18:12:28,208 Global parallel computation<br>
>> >> > enabled: 6 / 8<br>
>> >> > yt : [INFO ] 2012-02-07 18:12:28,208 Global parallel computation<br>
>> >> > enabled: 3 / 8<br>
>> >> > yt : [INFO ] 2012-02-07 18:12:28,208 Global parallel computation<br>
>> >> > enabled: 4 / 8<br>
>> >> > yt : [INFO ] 2012-02-07 18:12:28,208 Global parallel computation<br>
>> >> > enabled: 5 / 8yt : [INFO ] 2012-02-07 18:12:28,209 Global<br>
>> >> > parallel<br>
>> >> > computation enabled: 7 / 8<br>
>> >> ><br>
>> >> > But the script is just run 8 times, not any faster.<br>
>> >> ><br>
>> >> > What am I missing here?<br>
>> >> ><br>
>> >> > Many thanks!<br>
>> >> > chris<br>
>> >> ><br>
>> >> > _______________________________________________<br>
>> >> > yt-dev mailing list<br>
>> >> > <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> >> > <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> >> ><br>
>> >> _______________________________________________<br>
>> >> yt-dev mailing list<br>
>> >> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> >> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> ><br>
>> ><br>
>> ><br>
>> > _______________________________________________<br>
>> > yt-dev mailing list<br>
>> > <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> > <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> ><br>
>> _______________________________________________<br>
>> yt-dev mailing list<br>
>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
><br>
><br>
><br>
> _______________________________________________<br>
> yt-dev mailing list<br>
> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
><br>
_______________________________________________<br>
yt-dev mailing list<br>
<a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
</div></div></blockquote></div><br></div>