[Yt-dev] [yt-users] parallel data retrieval

Matthew Turk matthewturk at gmail.com
Sat Feb 14 09:14:13 PST 2009


It blocks inside all calls to get information about halos that aren't
on the current processor unless _processing is set -- you can read the
code for this in yt/lagos/ParallelTools.py, specifically line 114.
All methods of HopGroup, unless they're named in dont_wrap or start
with a _, are wrapped in the function single_proc_results.  This
barriers, calculates, broadcasts, and barriers.  So everything inside
your if statement in your loop is fine; what's causing the problem is
get_size.

On Sat, Feb 14, 2009 at 9:08 AM, Stephen Skory <stephenskory at yahoo.com> wrote:
> Matt,
>
> I think I don't quite understand something fundamental about how parallel python works. In particular, how do I make the loop at line 29 run on all processors at the same time? As an experiment I'm trying to write separate HDF5 files per processor to speed things up, but from the timestamp (line 73) I can see that it's going serially in order of halo.id. For example, the first halo on proc 0 is the sixth, but it isn't written until the first five are written by proc 1, so it's going in strict order.
>
> http://paste.enzotools.org/show/49/
>
> wrote halo 0 from proc 1 at 2009-02-14 10:53:05.206974
> wrote halo 1 from proc 1 at 2009-02-14 10:53:05.296788
> wrote halo 2 from proc 1 at 2009-02-14 10:53:05.406035
> wrote halo 3 from proc 1 at 2009-02-14 10:53:05.506907
> wrote halo 4 from proc 1 at 2009-02-14 10:53:05.626588
> wrote halo 5 from proc 0 at 2009-02-14 10:53:06.474605
> wrote halo 6 from proc 0 at 2009-02-14 10:53:06.553487
> wrote halo 7 from proc 1 at 2009-02-14 10:53:06.688611
> wrote halo 8 from proc 1 at 2009-02-14 10:53:06.790755
> wrote halo 9 from proc 0 at 2009-02-14 10:53:06.879232
> wrote halo 10 from proc 1 at 2009-02-14 10:53:06.978932
>
>
> Thanks for answering my naive questions...
>
>  _______________________________________________________
> sskory at physics.ucsd.edu           o__  Stephen Skory
> http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student
> ________________________________(_)_\(_)_______________
> _______________________________________________
> Yt-dev mailing list
> Yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>



More information about the yt-dev mailing list