[yt-dev] Clumps, parallel

[Matthew Turk]

Hi David,

On Mon, Mar 12, 2012 at 5:45 PM, david collins <antpuncher at gmail.com> wrote:
> Hi, All--
>
> What's the current state of the art with regard to parallel clump
> finding?

It's not implemented.

>  Out of curiosity, I did a clumpfind blocked into octant, and
> I found super-linear scaling (non-scientific, my 512^3 wouldn't finish
> in 24 hours, but each octant ran in 30 minutes).  Now I have these
> clumps, and I'd like to sew them together.  So I have a couple
> questions:
>
> 1.) What's the easiest way to join two extracted sets?  I figure I can
> take these 8 sets and sew them together pretty easy ex-post-facto.  If
> there's a tool that does
> new_joined_thing = yt.join( clump1.data, clump2.data)
> then writing the machinery to sort out which clumps hit which faces is
> pretty easy, as is sewing together the various joined things into a
> clump hierarchy.

Extracted sets have a built-in method for this, which is supposed to
intelligently join the indices to minimize overhead.  "join"  You can
also use boolean regions, but joining boolean regions won't coalesce
indices so they may be slightly higher overhead.

>
> 2.) Is there anything about the countour finder that would not block
> decompose like this?

Mmm, probably joins across boundaries and across corners, and any join
that extends over an octant.  Particularly, you will need to examine
every cell on the boundary between octants.  Almost invariably this
will involve the entire dataset residing in memory, or an OOC
structure that allows multiple r/w accesses.

>
> 3.) Has anyone made any progress on things like this?

There was a project going to insert an external level set
identification algorithm.  While I believe that project is
progressing, I would instead recommend you investigate a method or
algorithm yourself and work with a senior yt developer (Britton and I
have both expressed interest in taking this on) to contribute this
change back to the community.  A parallel contouring algorithm would
be very valuable, and while it would be challenging, it can certainly
be accomplished.

I would encourage you, if this is functionality you are interested in
utilizing, to take up the reins of a project like this, identify
collaborators, and push forward on developing this functionality and
returning it to the community.  As it stands, while there are
interested parties, I doubt that they will be willing to devote
substantial time to either private development of a feature for a
specific project (rather than something that advances the yt project
as a whole) or to work on it in isolation without a concrete use case.

The clump finder Britton and I developed initially targeted a very
specific project, but was developed not only in the open but designed
to be a general, multi-purpose module.  I would hope that efforts to
extend it to function in parallel would share those goals.

-Matt

>
> d.
>
> --
> Sent from my computer.
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org