[Yt-dev] Parallelism

[Matthew Turk]

Hi all,

This email turned out to be a bit long, but I would really appreciate
comments and suggestions.  I need some help with thinking through how
to restructure parallelism, and some help with writing tests that
cover parallelism.

The parallelism in yt has gotten a bit out of control.  It is largely
governed by inheriting from ParallelAnalysisInterface, which provides
a bunch of methods that assume MPI_COMM_WORLD and that do things
(mostly) to arrays.  I went in today to make some changes and found
myself a bit overwhelmed by the number of methods, the number of
leftover chunks of code, and the overall length of the
ParallelAnalysisInterface.  In fact, I'm a bit concerned simply
because we use PAI as a mixin in so many diverse situations -- this
leads to a pretty unweildy "dir()" call on those classes that use it
as a MixIn.

There are 61 methods defined, which I have appended to the end of this
email.  More worrisome, though, is that it seems when a method is
needed, rather than examining the existing methods, a new one is
simply added.  And, perhaps most challenging, it is not 100% clear how
to test all of these methods in isolation, or where the cascade-effect
will cause problems if a given method is changed.  I'm guilty of all
of these things, and the bulk of the problem to begin with.  But what
is clear is that it's becoming challenging if not impossible to make
changes, because it's not clear how fragile different methods are or
how they might behave.  And, as we move toward much bigger platforms
and co-scheduled and in-situ visualization, we will *need* to be more
flexible with parallelism; we'll need better and easier ways of
dispatching tasks, keeping processors busy, and sending/receiving
information.

To that end, I'd like to propose we change the behavior -- and mindset
-- somewhat.  I'm willing to accept that objects that are distributed
and in parallel should inherit from PAI (although I wonder if an
explicit 'communication' object might be more appropriate) but I would
like to propose that we make things a bit smarter.  Right now we
largely interface with MPI as thought we are writing a language with
no introspection and with no higher-level constructs.  I'd like to see
us develop something of a higher-level, more straightforward interface
with MPI.

As a first step, I'd like to see all of the functions named "_mpi_*"
be removed.  Rather than prefixing with _mpi, then the operation, then
the type of the array (for instance), we should instead provide a
single method (_parallel_operation, for instance) that accepts
operands, opcodes, arguments, etc, and then dispatches to the
appropriate sub-function.  One opportunity that this will bring up is
that the dispatch itself can determine the appropriate communicator to
use -- which would open up the ability to subdivide our parallelism.
The implicit next step after this is to avoid feature creep in the
parallel tools.

However, before we can make *any* set of changes, we need a set of
tests.  Right now there are a *large* number of operations that are in
parallel.  There are tests for these operations in *serial*, but not
parallel.  In yt/tests I have added a new setup for tests based on the
enzo_test mechanism, which can operate in parallel.  It covers a lot
of functionality, but notably no halo finding and no volume rendering.
 I did not think I was qualified to test these items.  The tests run
no more than once an hour, with every new changeset.  Additional tests
of parallel-analysis can be added as well.

The (tl;dr) two takeaways:

1) Would an opcode / function dispatch mechanism be an acceptable
replacement for a humongous proliferation of functions?
2) Is anyone willing to write tests in the framework, so that we can
make these kind of changes to the code base?  I would be willing to
lead a sprint on this.  I think it would be a lot easier than, say,
testing Enzo -- you just need to write up some functions that run
routines, and call it good.  But there are so many twiddles that I
don't know that I could do it myself.

Thoughts/responses?

-Matt

_barrier
_claim_object
_distributed
_do_not_claim_object
_finalize_parallel
_get_dependencies
_get_filename
_get_grid_objs
_get_grids
_get_objs
_grids
_initialize_parallel
_is_mine
_mpi_Allsum_double
_mpi_Allsum_long
_mpi_Irecv_double
_mpi_Irecv_long
_mpi_Isend_double
_mpi_Isend_long
_mpi_Request_Testall
_mpi_Request_Waitall
_mpi_Request_Waititer
_mpi_allmax
_mpi_allmin
_mpi_allsum
_mpi_bcast_long_dict_unpickled
_mpi_bcast_pickled
_mpi_cat_na_array
_mpi_catarray
_mpi_catdict
_mpi_catlist
_mpi_catrgb
_mpi_concatenate_array_double
_mpi_concatenate_array_long
_mpi_concatenate_array_on_root_double
_mpi_concatenate_array_on_root_int
_mpi_concatenate_array_on_root_long
_mpi_double_array_max
_mpi_exit_test
_mpi_find_neighbor_3d
_mpi_get_rank
_mpi_get_size
_mpi_info_dict
_mpi_joindict
_mpi_joindict_unpickled_double
_mpi_joindict_unpickled_long
_mpi_maxdict
_mpi_maxdict_dict
_mpi_minimum_array_long
_partition_hierarchy_2d
_partition_hierarchy_2d_inclined
_partition_hierarchy_3d
_partition_hierarchy_3d_bisection
_partition_hierarchy_3d_bisection_list
_partition_region_3d
_preload
_recv_quadtree
_send_quadtree
_should_i_write
_write_on_root
merge_quadtree_buffers