[yt-dev] Announcing: yt 3.0 alpha 3

[Matthew Turk]

Hi everyone,

We're proud to announce the third ALPHA release of yt 3.0.  yt has
recently transitioned to a time-based release plan (
https://ytep.readthedocs.org/en/latest/YTEPs/YTEP-0008.html ) and this
is the third scheduled alpha of 3.0, although we've grossly missed the
estimated date of July 15.  No date for a "final" release has yet been
set, but there will likely be several more alphas before that time.

The yt 2.5 codebase, and further updates in the 2.x series, will be
supported for a considerable amount of time and you do not need to
upgrade.

= yt 3.0?! =

yt 3.0 represents a new direction forward for yt: getting rid of all
the underlying assumptions that data needs to be sectioned off into
nice little grid patches.  This includes supporting Octree codes
natively (NMSU-ART and RAMSES), eventual support for SPH codes, and
even opaque data structures where the data is extremely large (ARTIO).
 We're even planning support for natively handling cylindrical and
spherical coordinates.

More: http://blog.yt-project.org/post/WhatsUpWith30.html

However, this *is* an alpha release.  Not all of the existing codes
have been ported to 3.0.

Additionally, this release benefits from the technical and
non-technical contributions from many new people.  yt is developed in
the context of a community of contributors, and with the push toward a
new architecture, we aim to expand that community considerably.  In
particular, this release has considerably benefited from contributions
from many new individuals.

= Getting It! =

To try out yt 3.0, you can now pull from the main yt repository,
update to the yt-3.0 branch, and rebuild your extensions.  Or, if you
would like to create a new, safely sectioned off environment, simply
re-run the normal "development" install script after changing the
variable BRANCH to "yt-3.0".

If you would like to try out yt 3.0 and are having trouble, please
write to the yt-users mailing list for assistance.

The yt 3.0 install script may also work, which can be obtained by
executing these commands:

  wget http://hg.yt-project.org/yt/raw/yt-3.0/doc/install_script.sh
  bash install_script.sh

= What's New? =

A demo notebook demonstrating much new functionality, and including
the full release notes, can be found here:

http://hub.yt-project.org/nb/88u14f

The bullet-pointed release notes can be found at the end of this
email, as well.  The main improvements include *considerable* memory
usage reduction, ARTIO spatial data indexing support, better particle
support, an overhaul of the selection methods for data objects, and a
mechanism for on-the-fly definitions of particle types based on
boolean filters.

Additionally, this release was used in the production of the AGORA
project flagship paper.  The analysis script used there can be seen at
this short URL:

http://goo.gl/SwY9Uv

and the paper can be found here: http://arxiv.org/abs/1308.2669

= Reporting Problems =

If you test out yt 3.0 we want to hear if it DID or DID NOT work!
Feedback is crucial at this time. yt-users and yt-dev are both good
forums for discussion, asking questions, and reporting problems.  Lots
of things have changed on the backend, but we have attempted to
minimize the user-facing changes.

To report a bug please go here:

https://bitbucket.org/yt_analysis/yt/issues/new

Note that you will not receive updates if you are not logged in when
you create the bug.

= What's Next? =

The next alpha release (3.0a4) will be released sometime this fall,
but development can be monitored either at
http://bitbucket.org/yt_analysis/yt-3.0 or in the main yt repository
under the named branch "yt-3.0".  We hope to focus on generalizing
Octree support further, adding better non-Cartesian support, and
supporting generic smoothing kernel definitions.

If you'd like to participate in yt development, please stop by #yt on
irc.freenode.net ( http://yt-project.org/irc.html ) or yt-dev (
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org ), or
submit a pull request on BitBucket.

Thanks very much,

Matt, on behalf of the yt development team

Changelog:

(Including 434 changesets from 7 contributors since 3.0a2)

 * Fixes for volume rendering and porting of the Cython kD-tree.  VR
now works for octree codes, albeit slowly.
* Major reorganization and simplification of selector code, including
generic periodicity
* ARTIO now has spatial data support
* Major bug fix for FLASH IO.  FLASH works again, following a
regression in 3.0a2.
* Merged with mainline yt development, bringing developments from yt
2.5.3 and 2.5.4 into 3.0.
* OWLS (Gadget-HDF5) data now updated to match functionality of other
Gadget datasets.
* Creationg of "arbitrary_grid" data selector for flexible particle
deposition operations.  This enables creation of a grid of arbitrary
dimensions that can have particles smoothed or deposited within them.
* Conversion of all CIC operations to particle deposition operations.
This results in much more flexible particle type selection and speed
improvements.
* Refactoring of particle fields to enable simpler creation and
collection of particle fields.  Particle fields defined for Eulerian
codes and Lagrangian codes are now interchangeable.
* New, currently unused parallel ring iteration.  To be explored in
the future for smoothing kernels.
* Many improvements for NMSU-ART
* Fixed a crazy QuadTree deposition bug for projections that showed up
when projecting octree particle deposition results through a
restricted data object.
* Fixed a crazy off-by-one bug for binary Gadget IO.
* Enormous Octree diet.
    * Oct leaf nodes now cost 256 bits each, which may eventually be
reduced to 192 bits.  Previous versions were considerably larger.
    * Octree traversal is strictly recursive, enabling
distributed-memory octrees to be implemented.
    * RAMSES octrees are created by-domain instead of globally.  This
will enable better parallelism in the future and faster ghost zone
generation when that is implemented.
    * Particle octree are now constructed via Z-curve generation.
This is considerably faster (10-20x) as well as much more memory
conservative.  This was designed to utilize parallel octree
construction in the future.
    * Particle octrees are now indexed by coarse base-level indexing
of domain regions.  This enables unsorted particle files to be indexed
for IO as well as spatial selection without mandating that leaf nodes
are fully-contained within a single file.  Reduction in total oct leaf
count of ~8x for multi-file particle datasets.
* YTDataContainer no longer requires .shape and .size
* Only requested fields are plotted, not translated fields
* Reduce memory usage of spatially-chunked fields for patch datasets
* Added arbitrary particle loader, load_particles.
* Units fix for RAMSES boxlen!=1.0 datasets
* Particle filtering for dynamically-created particle types, similar
to derived fields
* Improve type consistency in TotalQuantity and other derived quantities.
* Fixed major error with ghost zone generation and uninitialized values
* Some resiliency for particle datasets that do not specify a domain
* Bugfix for 1- and 2-D Enzo datasets
* Improvements to the Tipsy frontend, including auto-detecting parameter files
* Enable specification of n_ref when creating particle dataset
* Enable particle deposition to back-act on particle fields
* Major fix for octree particle counting, resulting in smaller meshes
* Enable "source" specification for volume rendering
* Enable initial volume rendering of octree datasets
* Fixed error with RAMSES C/F ordering of data
* Many fixes related to field dependency calculation
* FLASH cylindrical & polar pixelization fix