[yt-dev] the future of analysis_modules

Tue Nov 29 10:10:03 PST 2016

Hi everyone,

We will have a hangout discussion about moving forward with this on
Thursday, Dec 1 at 2pm EST.  If you weren't on the poll and would like to
be invited to the hangout, send me an email.

Britton

On Wed, Nov 23, 2016 at 9:48 AM, Britton Smith <brittonsmith at gmail.com>
wrote:

> Hey all,
>
> Thanks to everyone for their feedback.  This has been a good discussion so
> far.  With Thanksgiving tomorrow, let's give people until the end of Monday
> to contribute comments.
>
> It looks like this idea is getting a good amount of support, so let's plan
> of having a live discussion about this in the near future.  After Monday, I
> will organize a doodle poll inviting everyone who has responded so far.  If
> you don't have anything to add to the discussion but would like to be
> involved, just drop me a note and I'll include you on the poll.
>
> Thanks again everyone and have a good weekend.
>
> Britton
>
> On Wed, Nov 23, 2016 at 8:19 AM, John ZuHone <jzuhone at gmail.com> wrote:
>
>> I’m +1 on this and I agree with Nathan’s comments. In particular, I think
>> that we shouldn’t bundle them all into one package but probably split them
>> off into separate packages. This requires more work on the behalf of the
>> maintainers of those modules, of course.
>>
>> In addition to keeping in clump_finder, particle_trajectories should also
>> probably be left in, and perhaps be turned into some kind of time series
>> object or something else.
>>
>> I’m happy to help out—in particular I can split off sunyaev_zeldovich and
>> ppv_cube and spin them off as separate packages. photon_simulator is
>> already deprecated in favor of pyXSIM.
>>
>> All that spectral_integrator does is generate fields, so perhaps its
>> functionality needs to be merged into the astro fields part somehow and
>> left inside yt itself.
>>
>> On Nov 22, 2016, at 8:54 PM, Nathan Goldbaum <nathan12343 at gmail.com>
>> wrote:
>>
>> Overall I think this is a good idea. I have a couple critiques about the
>> mechanics that I'll reply inline about. I think it makes sense to separate
>> out all of the astro-specific analysis modules.
>>
>> On Tue, Nov 22, 2016 at 8:37 PM, Britton Smith <brittonsmith at gmail.com>
>> wrote:
>>
>>> Greetings,
>>>
>>> I would like to open a discussion on the idea of moving most of yt’s
>>> analysis modules into an external yt extensions package.  For ease of
>>> reading, I will separate this email into what this would mean for the code,
>>> what I see are the pros, cons, logistics, and open questions.  I would very
>>> much appreciate comment on this.
>>>
>>> What this means
>>> If we did this, most of the contents of yt/analysis_modules would be
>>> moved into a repository named something like yt_astro_analysis.
>>> Installing this would be an option in the install script and would likely
>>> also be pip installable.  Imports would change from
>>> from yt.analysis_modules.halo_analysis.api import HaloCatalog
>>> to
>>> from yt.extensions.astro_analysis.halo_analysis.api import HaloCatalog
>>>
>>> After creating yt_astro_analysis, there would be a period where the old
>>> analysis_modules would still exist but be deprecated before being removed
>>> at some point down the road.
>>>
>>> Pros
>>>
>>>    - Almost all of the current analysis modules are specific to
>>>    astrophysics.  As we continue to make the core functionality of yt less
>>>    astro specific, it’s not clear how to make room for non-astro analysis
>>>    modules.  Putting everything together under analysis_modules will make
>>>    navigation and documentation messy and confusing.  This would also
>>>    significantly slim down the yt codebase.
>>>    - Many of the tools in analysis_modules are very old and are in need
>>>    of API-breaking refactor.  Some of these, like two_point_functions, did not
>>>    make the jump from yt-2 to yt-3 and are no longer usable.  Many tools no
>>>    longer have a champion, someone interested in using, maintaining, and
>>>    updating them as yt’s core functionality develops and changes.  Moving
>>>    analysis_modules from yt decouples them from yt’s release cycle, allowing
>>>    interested parties to make updates and releases on a separate, likely
>>>    shorter timescale.  Some analysis_modules may even be better suited to be
>>>    moved into other extensions that are actively developed, such as the case
>>>    of the AbsorptionSpectrum with the Trident project.
>>>    - Similar to the point above, yt releases would not be slowed by the
>>>    need to update all of the championless modules.  Individual analysis
>>>    modules can be tied to specific stable releases of yt and so assured to
>>>    work there.
>>>
>>>
>>> Cons
>>>
>>>    - This will take a non-zero amount of work.  See below for a summary
>>>    of the primary tasks.
>>>    - There are some outstanding logistical questions.  See below.
>>>    - Not having yt and analysis modules explicitly tied to the same
>>>    codebase/releases could result in analysis tools falling behind and out of
>>>    date.
>>>    - The disruption and need to alter scripts could irritate and
>>>    alienate users.
>>>
>>>
>>> Proposed progression
>>> This is roughly how this would happen.  Here is a table with all
>>> existing analysis modules, their status, and potential future:
>>> https://goo.gl/HZykQA
>>>
>>>    1. Create yt_astro_analysis repo with all analysis modules that are
>>>    to be moved.  Add an entry to the extensions page on yt-project.org.
>>>    Make it installed by default in the install script, at least at first.
>>>
>>>
>> Should there only be one repo? If we're going to do this, it might help
>> future maintainability to have one repo per analysis module. That way
>> maintainers that only care about one analysis module don't need to worry
>> about changes to other modules.
>>
>> I also don't think the clump finder in particular should be split out
>> into its own package. That one isn't astro-specific and it might make sense
>> to try to integrate it more deeply with the core of yt.
>>
>>
>>>
>>>    1. Deprecate all moved modules in yt.
>>>    2. After some time, remove deprecated modules from yt.
>>>
>>>
>>> Open issues
>>> Here are some logistics and questions that still need to be worked out.
>>>
>>>    - Can we set up separate testing for yt_astro_analysis?  Would
>>>    maintaining this be a pain?
>>>
>>>
>> If the tests don't require large datasets and are relatively quick, this
>> might be a good opportunity to explore bitbucket pipelines. There's also
>> drone.io or we can use the yt testing infrastructure. Since I'm not the
>> maintainer of the yt testing infra I can't speak to whether it's ok to
>> expand it to more packages.
>>
>>
>>>
>>>    - How/where would the documentation be hosted?
>>>
>>>
>> Readthedocs will likely be sufficient. If we care about inline code
>> examples or whatnot, then we can explore using the yt testing
>> infrastructure, with the same caveats as above.
>>
>>
>>>
>>>    - How would we move the analysis modules source code and maintain
>>>    its revision history?
>>>
>>>
>> This can be done relatively straightforwardly using the convert extension
>> for mercurial and a filemap.
>>
>>
>>>
>>> Questions to yt-dev
>>>
>>>    - Are you +/-1 on this?  Any other comments?
>>>
>>>
>> +1
>>
>>
>>>
>>>    - Changes to the analysis_modules spreadsheet (https://goo.gl/HZykQA
>>>    )?
>>>    - Interested in helping out with this?  If this happens, I propose
>>>    anyone interested meets for a hangout to discuss how to proceed.
>>>
>>>
>> I'm happy to help.
>>
>>
>>>
>>> Thanks for reading!
>>>
>>> Britton
>>>
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>
>>>
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>>
>>
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20161129/acf01fc0/attachment.html>