[Yt-dev] QuadTree projection now works in parallel

[Matthew Turk]

Hi all,

In the development repo, I have swapped out quad tree for the old
style projection.  Old style is now the data object "overlap_proj" and
"quad_proj" is now "proj".

-Matt

On Thu, Jun 2, 2011 at 1:37 PM, Matthew Turk <matthewturk at gmail.com> wrote:
> John, Sam, Stephen,
>
> Thanks very much for testing.  I think maybe this is just about ready
> for production; sounds like we should push ahead with swapping it out
> for 2.2.  The only remaining test is the light cone stuff, but I was
> able to test the source-selection and that worked for me.  It would be
> a real feather in our caps I think to release 2.2 that included a
> completely new web GUI (which includes PyLab support as well as google
> maps-style widgets!), a projection speedup of 3.5x, and a new field
> system.  And, even a new logo!
>
> There are a few more optimizations I believe I can apply, which I will
> attempt to do over the next little while -- but they are less
> invasive.  Mainly these involve moving from pre-generated arrays of
> positions to generating positions inside the Cython code on an
> as-needed basis.
>
> John -- 350 s to 104 s is pretty good, I think.  We may now be IO
> dominated, but if you supply the argument preload="all" it might cut
> down even further on the run time.
>
> Thanks, everyone.
>
> -Matt
>
> On Thu, Jun 2, 2011 at 1:24 PM, John Wise <jwise at astro.princeton.edu> wrote:
>> Nice work, Matt!
>>
>>> pf.h.proj = pf.h.quad_proj
>>
>> Clever trick.  I didn't think about doing this before when running QuadProj in serial before.
>>
>> I just tested it on pleiades on 128 processes on a large nested grid simulation.  The run time decreased from 348s to 104s.  Very nice.  There were no errors, but there were some lingering RECV/SEND debugging messages.  Here are the stats on the dataset.
>>
>> level   # grids     # cells
>> ---------------------------
>>  0        512    134217728
>>  1        850     25412184
>>  2       1229    140608000
>>  3      13775    283353768
>>  4      41136    195020184
>>  5      29687     74512760
>>  6      14847     17540640
>>  7       4104      3998824
>>  8        527      1099896
>>  9         10         6984
>> ----------------------------
>>        106677    875770968
>>
>> John
>> _______________________________________________
>> Yt-dev mailing list
>> Yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>