[yt-dev] the future of analysis_modules
Britton Smith
brittonsmith at gmail.com
Wed Nov 23 09:48:29 PST 2016
Hey all,
Thanks to everyone for their feedback. This has been a good discussion so
far. With Thanksgiving tomorrow, let's give people until the end of Monday
to contribute comments.
It looks like this idea is getting a good amount of support, so let's plan
of having a live discussion about this in the near future. After Monday, I
will organize a doodle poll inviting everyone who has responded so far. If
you don't have anything to add to the discussion but would like to be
involved, just drop me a note and I'll include you on the poll.
Thanks again everyone and have a good weekend.
Britton
On Wed, Nov 23, 2016 at 8:19 AM, John ZuHone <jzuhone at gmail.com> wrote:
> I’m +1 on this and I agree with Nathan’s comments. In particular, I think
> that we shouldn’t bundle them all into one package but probably split them
> off into separate packages. This requires more work on the behalf of the
> maintainers of those modules, of course.
>
> In addition to keeping in clump_finder, particle_trajectories should also
> probably be left in, and perhaps be turned into some kind of time series
> object or something else.
>
> I’m happy to help out—in particular I can split off sunyaev_zeldovich and
> ppv_cube and spin them off as separate packages. photon_simulator is
> already deprecated in favor of pyXSIM.
>
> All that spectral_integrator does is generate fields, so perhaps its
> functionality needs to be merged into the astro fields part somehow and
> left inside yt itself.
>
> On Nov 22, 2016, at 8:54 PM, Nathan Goldbaum <nathan12343 at gmail.com>
> wrote:
>
> Overall I think this is a good idea. I have a couple critiques about the
> mechanics that I'll reply inline about. I think it makes sense to separate
> out all of the astro-specific analysis modules.
>
> On Tue, Nov 22, 2016 at 8:37 PM, Britton Smith <brittonsmith at gmail.com>
> wrote:
>
>> Greetings,
>>
>> I would like to open a discussion on the idea of moving most of yt’s
>> analysis modules into an external yt extensions package. For ease of
>> reading, I will separate this email into what this would mean for the code,
>> what I see are the pros, cons, logistics, and open questions. I would very
>> much appreciate comment on this.
>>
>> What this means
>> If we did this, most of the contents of yt/analysis_modules would be
>> moved into a repository named something like yt_astro_analysis.
>> Installing this would be an option in the install script and would likely
>> also be pip installable. Imports would change from
>> from yt.analysis_modules.halo_analysis.api import HaloCatalog
>> to
>> from yt.extensions.astro_analysis.halo_analysis.api import HaloCatalog
>>
>> After creating yt_astro_analysis, there would be a period where the old
>> analysis_modules would still exist but be deprecated before being removed
>> at some point down the road.
>>
>> Pros
>>
>> - Almost all of the current analysis modules are specific to
>> astrophysics. As we continue to make the core functionality of yt less
>> astro specific, it’s not clear how to make room for non-astro analysis
>> modules. Putting everything together under analysis_modules will make
>> navigation and documentation messy and confusing. This would also
>> significantly slim down the yt codebase.
>> - Many of the tools in analysis_modules are very old and are in need
>> of API-breaking refactor. Some of these, like two_point_functions, did not
>> make the jump from yt-2 to yt-3 and are no longer usable. Many tools no
>> longer have a champion, someone interested in using, maintaining, and
>> updating them as yt’s core functionality develops and changes. Moving
>> analysis_modules from yt decouples them from yt’s release cycle, allowing
>> interested parties to make updates and releases on a separate, likely
>> shorter timescale. Some analysis_modules may even be better suited to be
>> moved into other extensions that are actively developed, such as the case
>> of the AbsorptionSpectrum with the Trident project.
>> - Similar to the point above, yt releases would not be slowed by the
>> need to update all of the championless modules. Individual analysis
>> modules can be tied to specific stable releases of yt and so assured to
>> work there.
>>
>>
>> Cons
>>
>> - This will take a non-zero amount of work. See below for a summary
>> of the primary tasks.
>> - There are some outstanding logistical questions. See below.
>> - Not having yt and analysis modules explicitly tied to the same
>> codebase/releases could result in analysis tools falling behind and out of
>> date.
>> - The disruption and need to alter scripts could irritate and
>> alienate users.
>>
>>
>> Proposed progression
>> This is roughly how this would happen. Here is a table with all existing
>> analysis modules, their status, and potential future:
>> https://goo.gl/HZykQA
>>
>> 1. Create yt_astro_analysis repo with all analysis modules that are
>> to be moved. Add an entry to the extensions page on yt-project.org.
>> Make it installed by default in the install script, at least at first.
>>
>>
> Should there only be one repo? If we're going to do this, it might help
> future maintainability to have one repo per analysis module. That way
> maintainers that only care about one analysis module don't need to worry
> about changes to other modules.
>
> I also don't think the clump finder in particular should be split out into
> its own package. That one isn't astro-specific and it might make sense to
> try to integrate it more deeply with the core of yt.
>
>
>>
>> 1. Deprecate all moved modules in yt.
>> 2. After some time, remove deprecated modules from yt.
>>
>>
>> Open issues
>> Here are some logistics and questions that still need to be worked out.
>>
>> - Can we set up separate testing for yt_astro_analysis? Would
>> maintaining this be a pain?
>>
>>
> If the tests don't require large datasets and are relatively quick, this
> might be a good opportunity to explore bitbucket pipelines. There's also
> drone.io or we can use the yt testing infrastructure. Since I'm not the
> maintainer of the yt testing infra I can't speak to whether it's ok to
> expand it to more packages.
>
>
>>
>> - How/where would the documentation be hosted?
>>
>>
> Readthedocs will likely be sufficient. If we care about inline code
> examples or whatnot, then we can explore using the yt testing
> infrastructure, with the same caveats as above.
>
>
>>
>> - How would we move the analysis modules source code and maintain its
>> revision history?
>>
>>
> This can be done relatively straightforwardly using the convert extension
> for mercurial and a filemap.
>
>
>>
>> Questions to yt-dev
>>
>> - Are you +/-1 on this? Any other comments?
>>
>>
> +1
>
>
>>
>> - Changes to the analysis_modules spreadsheet (https://goo.gl/HZykQA)?
>> - Interested in helping out with this? If this happens, I propose
>> anyone interested meets for a hangout to discuss how to proceed.
>>
>>
> I'm happy to help.
>
>
>>
>> Thanks for reading!
>>
>> Britton
>>
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20161123/5c475e2f/attachment-0001.htm>
More information about the yt-dev
mailing list