[Yt-dev] Parallelism

[Matthew Turk]

Hi Stephen,

On Tue, Aug 2, 2011 at 10:52 AM, Stephen Skory <s at skory.us> wrote:
> Hi Matt,
>
>> 1) Would an opcode / function dispatch mechanism be an acceptable
>> replacement for a humongous proliferation of functions?
>
> I am not sure that there is a good reason to go all the way to a
> single function, if I understand you correctly. For example, there's a
> reason that the 'math' module doesn't work like this:
>
> x = math('sqrt', 4)
> y = math('pow', 2, 4)
>
> Because it's cumbersome. I think reducing the number of functions,
> especially those with type/precision in their names is fine, but I
> don't see the benefit in going farther than that. Or perhaps I don't
> understand what you're describing.

Perhaps you are correct, and I went too far.  But there is a very,
very important distinction between your analogy and what we're talking
about -- *lots* of things inherit from ParallelAnalysisInterface.  And
these functions aren't cordoned off into a sub-namespace, they're part
of the same namespace as "extract_connected_sets" and "print_stats"
and whatnot.  I should have been clearer, the problem is that right
now we inherit from a class that has TONS of methods, most of which
will never be relevant for a given object.

But, perhaps going to opcodes is too far.  Instead, perhaps just a few
conceptual routines, which perform introspection and handle getting
communicators correctly?  For instance,

parallel_concatenate
parallel_operation (max, min, etc)
parallel_broadcast

And then inside these, perform introspection and appropriate
delegation of tasks?  For the non-blocking send/recv, perhaps delegate
those to an object that tracks sends and receives.

Would this be a good compromise?

>
>> 2) Is anyone willing to write tests in the framework, so that we can
>> make these kind of changes to the code base?
>
> I'v already done some of this, and I'd be willing to help out some more.

Excellent!  I'd love it if these could show up in a .py file under tests/ .

Thanks for your thought,

Matt

>
>
> --
> Stephen Skory
> s at skory.us
> http://stephenskory.com/
> 510.621.3687 (google voice)
> _______________________________________________
> Yt-dev mailing list
> Yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>