Hi Matt,<div><br></div><div>pf.h.proj is of type <class 'yt.data_objects.hierarchy.AMRQuadTreeProj'> and refine_by is 2. </div><div><br></div><div>Does this help? I'm not sure what you mean by overlaps - doesn't the the RAMSES grid patching mechanism produce non-overlapping grids from the octs? Is quadtree proj checking for overlapping grids?</div>
<div><br></div><div>chris</div><div><br><div class="gmail_quote">On Wed, Feb 8, 2012 at 12:44 PM, Matthew Turk <span dir="ltr"><<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Chris,<br>
<br>
Yeah, that's weird. My guess is that load balancing is going haywire<br>
for some reason, likely due to overlap versus quadtree proj. Can you<br>
tell me what type of object pf.h.proj is? i.e., what's the output of<br>
"print pf.h.proj"? And then, what's pf.refine_by?<br>
<br>
-Matt<br>
<div class="HOEnZb"><div class="h5"><br>
On Wed, Feb 8, 2012 at 3:00 PM, Christopher Moody <<a href="mailto:cemoody@ucsc.edu">cemoody@ucsc.edu</a>> wrote:<br>
> Hi Matt,<br>
><br>
> I've got the log output here:<a href="http://paste.yt-project.org/show/2153/" target="_blank">http://paste.yt-project.org/show/2153/</a> with the<br>
> serial version here <a href="http://paste.yt-project.org/show/2154/" target="_blank">http://paste.yt-project.org/show/2154/</a> .<br>
><br>
> The most interesting tidbit is below, where it looks like core 0 projects<br>
> Levels 0-5 and core 1 projects Level 6 (which takes up like 99% of the<br>
> projection time.)<br>
><br>
> chris<br>
><br>
> P001 yt : [DEBUG ] 2012-02-08 11:39:53,403 Going to obtain []<br>
> P001 yt : [DEBUG ] 2012-02-08 11:39:53,406 Preloading ['density'] from 0<br>
> grids<br>
> P001 yt : [DEBUG ] 2012-02-08 11:39:53,406 End of projecting level level<br>
> 0, memory usage 3.545e-01<br>
> P001 yt : [DEBUG ] 2012-02-08 11:39:53,406 Preloading ['density'] from 0<br>
> grids<br>
> P001 yt : [DEBUG ] 2012-02-08 11:39:53,406 End of projecting level level<br>
> 1, memory usage 3.545e-01<br>
> P001 yt : [DEBUG ] 2012-02-08 11:39:53,407 Preloading ['density'] from 0<br>
> grids<br>
> P001 yt : [DEBUG ] 2012-02-08 11:39:53,407 End of projecting level level<br>
> 2, memory usage 3.545e-01<br>
> P001 yt : [DEBUG ] 2012-02-08 11:39:53,408 Preloading ['density'] from 0<br>
> grids<br>
> P001 yt : [DEBUG ] 2012-02-08 11:39:53,408 End of projecting level level<br>
> 3, memory usage 3.545e-01<br>
> P001 yt : [DEBUG ] 2012-02-08 11:39:53,408 Preloading ['density'] from 0<br>
> grids<br>
> P001 yt : [DEBUG ] 2012-02-08 11:39:53,408 End of projecting level level<br>
> 4, memory usage 3.545e-01<br>
> P001 yt : [DEBUG ] 2012-02-08 11:39:53,409 Preloading ['density'] from 0<br>
> grids<br>
> P001 yt : [DEBUG ] 2012-02-08 11:39:53,409 End of projecting level level<br>
> 5, memory usage 3.545e-01<br>
> P001 yt : [DEBUG ] 2012-02-08 11:39:53,409 Preloading ['density'] from 6<br>
> grids<br>
> P001 yt : [INFO ] 2012-02-08 11:39:53,410 Starting 'Projecting level 6<br>
> / 6 '<br>
> P000 yt : [INFO ] 2012-02-08 11:39:54,057 Finishing 'Projecting level<br>
> 0 / 6 '<br>
> P000 yt : [DEBUG ] 2012-02-08 11:39:54,057 End of projecting level level<br>
> 0, memory usage 4.482e-01<br>
> P000 yt : [DEBUG ] 2012-02-08 11:39:54,057 Preloading ['density'] from 1<br>
> grids<br>
> P000 yt : [INFO ] 2012-02-08 11:39:54,058 Starting 'Projecting level 1<br>
> / 6 'P000 yt : [INFO ] 2012-02-08 11:39:54,070 Finishing 'Projecting<br>
> level 1 / 6 '<br>
> P000 yt : [DEBUG ] 2012-02-08 11:39:54,070 End of projecting level level<br>
> 1, memory usage 4.482e-01<br>
> P000 yt : [DEBUG ] 2012-02-08 11:39:54,070 Preloading ['density'] from 1<br>
> gridsP000 yt : [INFO ] 2012-02-08 11:39:54,071 Starting 'Projecting<br>
> level 2 / 6 '<br>
> P000 yt : [INFO ] 2012-02-08 11:39:54,130 Finishing 'Projecting level<br>
> 2 / 6 '<br>
> P000 yt : [DEBUG ] 2012-02-08 11:39:54,130 End of projecting level level<br>
> 2, memory usage 4.482e-01<br>
> P000 yt : [DEBUG ] 2012-02-08 11:39:54,130 Preloading ['density'] from 1<br>
> grids<br>
> P000 yt : [INFO ] 2012-02-08 11:39:54,131 Starting 'Projecting level 3<br>
> / 6 '<br>
> P000 yt : [INFO ] 2012-02-08 11:39:54,783 Finishing 'Projecting level<br>
> 3 / 6 '<br>
> P000 yt : [DEBUG ] 2012-02-08 11:39:54,784 End of projecting level level<br>
> 3, memory usage 4.482e-01<br>
> P000 yt : [DEBUG ] 2012-02-08 11:39:54,784 Preloading ['density'] from 1<br>
> grids<br>
> P000 yt : [INFO ] 2012-02-08 11:39:54,784 Starting 'Projecting level 4<br>
> / 6 '<br>
> P000 yt : [INFO ] 2012-02-08 11:39:59,389 Finishing 'Projecting level<br>
> 4 / 6 '<br>
> P000 yt : [DEBUG ] 2012-02-08 11:39:59,389 End of projecting level level<br>
> 4, memory usage 5.918e-01<br>
> P000 yt : [DEBUG ] 2012-02-08 11:39:59,389 Preloading ['density'] from 1<br>
> grids<br>
> P000 yt : [INFO ] 2012-02-08 11:39:59,389 Starting 'Projecting level 5<br>
> / 6 '<br>
> P000 yt : [INFO ] 2012-02-08 11:40:17,735 Finishing 'Projecting level<br>
> 5 / 6 '<br>
> P000 yt : [DEBUG ] 2012-02-08 11:40:17,735 End of projecting level level<br>
> 5, memory usage 1.569e+00<br>
> P000 yt : [DEBUG ] 2012-02-08 11:40:17,735 Preloading ['density'] from 0<br>
> grids<br>
> P000 yt : [DEBUG ] 2012-02-08 11:40:17,736 End of projecting level level<br>
> 6, memory usage 1.569e+00<br>
> P001 yt : [INFO ] 2012-02-08 11:41:31,681 Finishing 'Projecting level<br>
> 6 / 6 '<br>
> P001 yt : [DEBUG ] 2012-02-08 11:41:31,681 End of projecting level level<br>
> 6, memory usage 2.113e+00<br>
> P000 yt : [DEBUG ] 2012-02-08 11:41:33,807 Opening MPI Barrier on 0<br>
> P000 yt : [INFO ] 2012-02-08 11:41:34,502 Projection completed<br>
> P001 yt : [DEBUG ] 2012-02-08 11:41:34,502 Opening MPI Barrier on 1<br>
> P001 yt : [INFO ] 2012-02-08 11:41:34,502 Projection completed<br>
> P001 yt : [DEBUG ] 2012-02-08 11:41:34,579 Opening MPI Barrier on 1<br>
><br>
><br>
><br>
><br>
> On Wed, Feb 8, 2012 at 6:46 AM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>> wrote:<br>
>><br>
>> Hi Chris,<br>
>><br>
>> On Tue, Feb 7, 2012 at 9:30 PM, Christopher Moody <<a href="mailto:cemoody@ucsc.edu">cemoody@ucsc.edu</a>><br>
>> wrote:<br>
>> > Hi guys,<br>
>> ><br>
>> > I've been working hard on the ART frontend. Lately, I'm to the point<br>
>> > where<br>
>> > I'm playing around with more complex datasets that are taking much<br>
>> > longer to<br>
>> > project - so I'd really like to to start using the parallelization<br>
>> > engines.<br>
>> > I've tried Sam's workshop parallelization demos, and they all work. But<br>
>> > launching with the ART frontend (<a href="http://paste.yt-project.org/show/2152/" target="_blank">http://paste.yt-project.org/show/2152/</a>)<br>
>> > spawns many independent processes which evidently are not actually<br>
>> > splitting<br>
>> > the projection job, but still taking up lots of processors.<br>
>><br>
>> My guess is that parallelism is not enabled for the ART frontend<br>
>> simply as a matter of how the IO is conducted. To make it really work<br>
>> in parallel, the IO needs to be split up so that when process 1 reads<br>
>> a given grid patch, the rest of the processors don't also need to read<br>
>> all the data for that grid patch.<br>
>><br>
>> Can you lower your loglevel (by setting loglevel = 1 in ~/.yt/config<br>
>> or by --config yt.loglevel=1 on the command line) and report back with<br>
>> what it says during a projection job there?<br>
>><br>
>> -MAtt<br>
>><br>
>> ><br>
>> > My mpi installation works:<br>
>> > yt : [INFO ] 2012-02-07 18:12:28,207 Global parallel computation<br>
>> > enabled: 0 / 8yt : [INFO ] 2012-02-07 18:12:28,207 Global parallel<br>
>> > computation enabled: 2 / 8<br>
>> > yt : [INFO ] 2012-02-07 18:12:28,208 Global parallel computation<br>
>> > enabled: 1 / 8<br>
>> > yt : [INFO ] 2012-02-07 18:12:28,208 Global parallel computation<br>
>> > enabled: 6 / 8<br>
>> > yt : [INFO ] 2012-02-07 18:12:28,208 Global parallel computation<br>
>> > enabled: 3 / 8<br>
>> > yt : [INFO ] 2012-02-07 18:12:28,208 Global parallel computation<br>
>> > enabled: 4 / 8<br>
>> > yt : [INFO ] 2012-02-07 18:12:28,208 Global parallel computation<br>
>> > enabled: 5 / 8yt : [INFO ] 2012-02-07 18:12:28,209 Global parallel<br>
>> > computation enabled: 7 / 8<br>
>> ><br>
>> > But the script is just run 8 times, not any faster.<br>
>> ><br>
>> > What am I missing here?<br>
>> ><br>
>> > Many thanks!<br>
>> > chris<br>
>> ><br>
>> > _______________________________________________<br>
>> > yt-dev mailing list<br>
>> > <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> > <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> ><br>
>> _______________________________________________<br>
>> yt-dev mailing list<br>
>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
><br>
><br>
><br>
> _______________________________________________<br>
> yt-dev mailing list<br>
> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
><br>
_______________________________________________<br>
yt-dev mailing list<br>
<a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
</div></div></blockquote></div><br></div>