[yt-dev] RFI: Anybody making big changes?

[Matthew Turk]

Hi folks,

There are a couple big, backwards-incompatible changes that are coming
up in the future for yt.  None of these will come as a surprise, but
here they are:

 * The Demeshening: removing the global octree index for particle
datasets and replacing with a bitmap index.  This also includes
changing how particle fields are regarded, and makes the SPH
implementation much cleaner and faster.
 * Non-Spatial Coordinates: this one is a bit different than the title
suggests.  For example, ds.domain_left_edge and ds.domain_right_edge
will no longer be YTArray instances, instead they will be YTIndexArray
instances. YTIndexArray differs from YTArray in that instead of having
a single unit associated with it, it has a tuple of units, one for
each column of the array.  This will not, in its initial form, be a
big change to non-Cartesian coordinates, but that will certainly be
possible.

YTEPs are either done or being prepped for both of these.  Those are
kind of outside the main topic of this email, which is much more
social.

>From discussions, it seems pretty likely that these two would need to
be a yt 4.0.  So let's take that as a given.

Both of these are designed to be *as backwards compatible as
possible*.  The Demeshening will have many more breaking changes for
people that rely on yt for SPH data, but ideally the *surface area* of
those breakages will be small.  For the non-spatial changes, we
absolutely want that to be completely backwards compatible for
cartesian, spatial datasets.

So if we take as given that we want disruption to both users and
developers to be minimal, we're still kind of stuck with the
possibility that this will be a difficult transition.  We absolutely
want everything that worked before to work again.

What I'm getting at is this: how can we manage this change, when the
development from 3.x to 4.0 may be long?  We're at a different stage
than we were when we managed this for yt 3.0, which I did *not* do a
good job of managing.  We now encourage people to use binary
distributions a lot more, and we have a reliable bugfix release
system.

If we were to merge in both of those two big things, there is still
the possibility that unforeseen difficulties would crop up -- but, the
userbase will be much more isolated than they were when we did 3.0.
So the main problem is going to be if folks want to do big
development, but that big development conflicts with one of those two
things.

Here's the first thing I want to ask:

==
Is anybody planning or working toward some big development that they
would be nervous about doing in the codebase while we shake out any
lingering issues from those?
==

Ultimately, the question we need to figure out is, "how do we minimize
disruption to cartesian, grid-based codes while we fix this stuff up?"
 And I think until we know if folks have anything pretty big planned,
we can't *quite* answer that.

-Matt