[Yt-dev] The future of yt

[Matthew Turk]

Hi all,

I'm writing today about a change in the way yt is being developed,
which may have some visible changes.  I want to start out by
emphasizing that yt is, and will increasingly *be*, a *community*
project driven by pragmatic needs for scientific analysis.  Below I've
outlined a couple of my future projects as well as a brief discussion
of how to ensure that yt retains the community-developed approach that
has served it very well over the last few years.

As of January 1st, I've begun an NSF postdoctoral fellowship (the
Office of CyberInfrastructure's "Transformative Computational Science"
fellowship) designed to develop yt to the point that it can be used as
a full integrated science environment, starting with the generation of
initial conditions, moving through the conducting of a simulation,
continuing with analysis (both post-processing and during the course
of the calculation) and finishing with both scientific and outreach
quality visualization.  This will be driven forward most directly by
my own studies of the formation of the first stars and galaxies, but
it will also be conducted completely in the open and will be designed
to provide a new mechanism for conducting simulations for the
computational astrophysics community as a whole.  These changes will
be developed to be applicable to many different astrophysics codes,
not just Enzo.  My absence the last few weeks has been a result of
this; I packed up and moved from San Diego to New York City and
Columbia University.

However, I need to emphasize that this won't just be me working in
isolation for a few months and tossing things back at the source
control repository: this project will live or die based on how well it
is integrated into the workflows of others, and based on how engaged
the other users and developers of yt are in the process of making
these changes.  If conducted in isolation, this project surely would
fail; if conducted without the feedback from other working scientists,
this project won't ever gain traction.  This means that through the
process of making decisions about the design and implementation of
components up to and including initial deployment, I will be
soliciting feedback, encouraging interaction and testing changes.  As
users and developers, you are free to be as involved with or as
removed from this process as you desire.

This new fellowship will result in several nearly immediate changes:

1) The oft-delayed 2.0 release: I will be putting out a 2.0 release,
with the attendant documentation, by the 14th of January.  This will
mean the re-organized codebase will be the "stable" branch, and the
cookbook and documentation will be updated to reflect this.

2) Documentation will be prioritized from here on out; I am dedicated
to 100% coverage of yt.mods with docstrings, as well as newly written
narrative documentation for yt.

3) A new system we're tentatively calling "the Forge" will be deployed
to replace the ailing Barn, and I will focus on developing mechanisms
for sharing of scripts and analysis tools.

4) Regression and answer testing will be run on a daily or weekly
basis, to prevent errors and bitrot from creeping in.  This will be
initiated sometime in the couple weeks, hardware permitting.

There will also be several more medium-term changes.

1) Outreach: yt and the yt community will aim to provide a bridge
between scientists and outreach, by striving to enable both
connections between outreach facilitators and by providing intuitive,
narrative methods for visualization of astrophysical data.  I am going
to try to "bake in" narrative techniques into the visualization tools
provided by yt.

2) The de-enzo-fication of yt will continue.  Assumptions about the
nomenclature, the data types and the content of data will be
eliminated, and yt will continue to provide physically-motivated
astrophysical analysis that abstracts the underlying data objects.

3) Blogging: I will strive to blog once a week on my progress toward
these goals, on the Enzotools blog at blog.enzotools.org.

As work progresses on the formal goals of the fellowship, I will
update the mailing list, although I will attempt to keep the number of
updates to a minimum on yt-users and keep both yt-dev and the
Enzotools Blog as the main grounds for discussion.

Thanks to all the other developers -- even though I wrote yt
initially, it's a *community* project, and I do not see a future for
it except as a community project.

Best wishes,

Matt