[yt-dev] the future of analysis_modules

[Britton Smith]

Greetings,

I would like to open a discussion on the idea of moving most of yt’s
analysis modules into an external yt extensions package.  For ease of
reading, I will separate this email into what this would mean for the code,
what I see are the pros, cons, logistics, and open questions.  I would very
much appreciate comment on this.

What this means

If we did this, most of the contents of yt/analysis_modules would be moved
into a repository named something like yt_astro_analysis.  Installing this
would be an option in the install script and would likely also be pip
installable.  Imports would change from

from yt.analysis_modules.halo_analysis.api import HaloCatalog

to

from yt.extensions.astro_analysis.halo_analysis.api import HaloCatalog

After creating yt_astro_analysis, there would be a period where the old
analysis_modules would still exist but be deprecated before being removed
at some point down the road.

Pros

   -

   Almost all of the current analysis modules are specific to
   astrophysics.  As we continue to make the core functionality of yt less
   astro specific, it’s not clear how to make room for non-astro analysis
   modules.  Putting everything together under analysis_modules will make
   navigation and documentation messy and confusing.  This would also
   significantly slim down the yt codebase.
   -

   Many of the tools in analysis_modules are very old and are in need of
   API-breaking refactor.  Some of these, like two_point_functions, did not
   make the jump from yt-2 to yt-3 and are no longer usable.  Many tools no
   longer have a champion, someone interested in using, maintaining, and
   updating them as yt’s core functionality develops and changes.  Moving
   analysis_modules from yt decouples them from yt’s release cycle, allowing
   interested parties to make updates and releases on a separate, likely
   shorter timescale.  Some analysis_modules may even be better suited to be
   moved into other extensions that are actively developed, such as the case
   of the AbsorptionSpectrum with the Trident project.
   -

   Similar to the point above, yt releases would not be slowed by the need
   to update all of the championless modules.  Individual analysis modules can
   be tied to specific stable releases of yt and so assured to work there.


Cons

   -

   This will take a non-zero amount of work.  See below for a summary of
   the primary tasks.
   -

   There are some outstanding logistical questions.  See below.
   -

   Not having yt and analysis modules explicitly tied to the same
   codebase/releases could result in analysis tools falling behind and out of
   date.
   -

   The disruption and need to alter scripts could irritate and alienate
   users.


Proposed progression

This is roughly how this would happen.  Here is a table with all existing
analysis modules, their status, and potential future: https://goo.gl/HZykQA

   1.

   Create yt_astro_analysis repo with all analysis modules that are to be
   moved.  Add an entry to the extensions page on yt-project.org.  Make it
   installed by default in the install script, at least at first.
   2.

   Deprecate all moved modules in yt.
   3.

   After some time, remove deprecated modules from yt.


Open issues

Here are some logistics and questions that still need to be worked out.

   -

   Can we set up separate testing for yt_astro_analysis?  Would maintaining
   this be a pain?
   -

   How/where would the documentation be hosted?
   -

   How would we move the analysis modules source code and maintain its
   revision history?


Questions to yt-dev

   -

   Are you +/-1 on this?  Any other comments?
   -

   Changes to the analysis_modules spreadsheet (https://goo.gl/HZykQA)?
   -

   Interested in helping out with this?  If this happens, I propose anyone
   interested meets for a hangout to discuss how to proceed.


Thanks for reading!

Britton
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20161122/d190596e/attachment.htm>