[Yt-dev] parallel halo profiles

[Matthew Turk]

I've made my first pass at parallelizing the halo profiler.  The
writing out stage still probably won't work, as all the procs will
still try to write out at the same time.  You can see what I did in
these two changesets:

http://bitbucket.org/MatthewTurk/yt/changeset/556490d86872/
http://bitbucket.org/MatthewTurk/yt/changeset/91933c49f17f/

(note that I pushed a bunch more changesets where I fixed bugs I
missed...  oops!  the main stuff is in here)

I changed the GridIterator to be a general object iterator.  The halo
profiler then runs with this.  The halo profiler won't work just yet;
what needs to be done is to change hopHalos into a dict keyed by the
halo ids, and then it'll be a lot closer...  Maybe Britton could take
a look at where it is, suggest if it's almost ready?  I think it was
close to nearly there already.  Specifically, I suspect we'll need to
have multiple round-robins over the halos with the current workflow in
the HaloProfiler, so we'll have to use arguments to
initialize_parallel and finalize_parallel to govern what gets done.
(We probably only need a finalize_parallel.)

-Matt

On Fri, Mar 13, 2009 at 10:37 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>> Out of general curiosity, what aspect of the IO would have to change?
>
> Well, there are a couple problems.  What I'll present as the most
> problematic is the copying of arrays.  We constructs sphere to analyze
> halos, which then load-on-demand.  If you either ran these as
> field-cuts (inlineextractedregions) from the tiles or enabled
> preloading of data and disabled the 'pop' method in the DataQueue
> object, you could get around this.  It's something to think about for
> later, but I'm not yet it's an issue right now.  I think we should try
> implementing the round robin, then run cProfile on it to see if IO is
> the limiting factor.
>
>> Thank you very much, Matt! While I was certainly not requesting you do my work for me, I had the feeling that what I wanted to do would be related to (or impinge upon) overarching plans you already had for yt. Let me know what I can do for you.
>
> will do.  This is in support of a ticket I opened maybe a month or two
> ago, 192, which is really two-in-one.
>
> -Matt
>