[yt-users] parallel data retrieval

[Matthew Turk]

Hi Stephen,

The parallel proxy class works so that it does all the barriers you
need.  In the second script, what is happening is that proc 0 is
inside the function parallel_simple_proxy:

http://yt.enzotools.org/browser/trunk/yt/lagos/ParallelTools.py#L114

because its metaclass is ParallelDummy:

http://yt.enzotools.org/browser/trunk/yt/lagos/ParallelTools.py#L130

These functions are not meant to be manually parallelized; this is a
flaw concealed inside a bonus.  If you want to try it again, you might
consider manually setting (and then unsetting) the _processing
attribute of your HopGroup to True; this should set it to turn off the
locking/barrierizing.  Keep in mind, though, that this means that all
of your lists of halo attributes will be processor specific and not
shared amongst the processors.  Locking of HDF5 files should also be
considered, so you'll need to handle that somehow as well, to ensure
you either have only one processor writing to your HDF5 files at a
time, or to use whatever (if any) parallel IO support PyTables has.

-Matt

On Fri, Feb 13, 2009 at 9:43 AM, Stephen Skory <stephenskory at yahoo.com> wrote:
> Matt,
>
> I have a script that runs parallel hop, and then writes out the results to an HDF5 file. I'd like to more efficiently parallelize a certain part of the process, where I retrieve and store the particle particulars, but it doesn't seem to work the way I'd like. This first example works, where I get the various values on all processors, and then store it only on the one that owns the halo:
>
> http://paste.enzotools.org/show/43/
>
>
> But this one doesn't work, where I only retrieve the values if the proc owns the halo. And by not working, I mean it hangs on halo 0 and doesn't finish after 10 minutes, while the script above takes less than a minute:
>
> http://paste.enzotools.org/show/44/
>
> Can you give me any ideas? In particular, the part of the script where it retrieves the particle_indexes is going serially in the functioning script (I can watch it in the output log) and it's taking far longer than it should. Thanks!
>
> _______________________________________________________
> sskory at physics.ucsd.edu           o__  Stephen Skory
> http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student
> ________________________________(_)_\(_)_______________
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>