[yt-dev] the future of analysis_modules
Britton Smith
brittonsmith at gmail.com
Mon Dec 5 10:20:10 PST 2016
Hi again everyone,
Last week, a few of us had a hangout to discuss breaking most the
analysis modules off into external, pip-installable packages. Below
is a summary of this discussion. If anyone has any questions,
please feel free to post them here.
Britton
*What will happen:*
- Most of the contents of analysis_modules will be moved to individual
external packages under the yt_analysis user on bitbucket or on the
yt-project organization on github. Putting packages on github will
enable integration with testing services like travis, circleci and
appveyor. Some will be move to yt/data_objects.
- Modules without an interested maintainer will be collected into an
attic repository under yt_analysis.
- Status of each module can be seen here
<https://docs.google.com/spreadsheets/d/1uMGD5fjV7eHjNGWek1uGA-GP-xrfmWRMMN54Tkb2Gnc/edit#gid=0>,
with names of people
volunteering to relocate them.
*Logistics*
- Modules to be renamed yt_<module name> so as to be importable from
yt.extensions.
- Docs: individual docs for each module can be hosted on
readthedocs.io.
- Testing: continue to use in-house yt testing infrastructure or free
CI services like travis, circleci, and appveyor (on github) or
drone.io (on bitbucket). Kacper will help set this up. Exporting
version history: use the hg convert extension. Nathan to create
some instructions for this.
- Modules will be marked as deprecated with message to pip install
external package. Deprecated modules will be removed for yt-4.0
release.
- An extension page to be added to docs.
*Procedure*
1. Create new repo under yt_analysis and export source and docs.
2. Create readthedocs page.
3. Set up testing, work with Kacper.
4. Add entry to extensions page on yt-projects.org.
5. Issue PR to yt marking analysis module as deprecated.
On Tue, Nov 29, 2016 at 10:10 AM, Britton Smith <brittonsmith at gmail.com>
wrote:
> Hi everyone,
>
> We will have a hangout discussion about moving forward with this on
> Thursday, Dec 1 at 2pm EST. If you weren't on the poll and would like to
> be invited to the hangout, send me an email.
>
> Britton
>
> On Wed, Nov 23, 2016 at 9:48 AM, Britton Smith <brittonsmith at gmail.com>
> wrote:
>
>> Hey all,
>>
>> Thanks to everyone for their feedback. This has been a good discussion
>> so far. With Thanksgiving tomorrow, let's give people until the end of
>> Monday to contribute comments.
>>
>> It looks like this idea is getting a good amount of support, so let's
>> plan of having a live discussion about this in the near future. After
>> Monday, I will organize a doodle poll inviting everyone who has responded
>> so far. If you don't have anything to add to the discussion but would like
>> to be involved, just drop me a note and I'll include you on the poll.
>>
>> Thanks again everyone and have a good weekend.
>>
>> Britton
>>
>> On Wed, Nov 23, 2016 at 8:19 AM, John ZuHone <jzuhone at gmail.com> wrote:
>>
>>> I’m +1 on this and I agree with Nathan’s comments. In particular, I
>>> think that we shouldn’t bundle them all into one package but probably split
>>> them off into separate packages. This requires more work on the behalf of
>>> the maintainers of those modules, of course.
>>>
>>> In addition to keeping in clump_finder, particle_trajectories should
>>> also probably be left in, and perhaps be turned into some kind of time
>>> series object or something else.
>>>
>>> I’m happy to help out—in particular I can split off sunyaev_zeldovich
>>> and ppv_cube and spin them off as separate packages. photon_simulator is
>>> already deprecated in favor of pyXSIM.
>>>
>>> All that spectral_integrator does is generate fields, so perhaps its
>>> functionality needs to be merged into the astro fields part somehow and
>>> left inside yt itself.
>>>
>>> On Nov 22, 2016, at 8:54 PM, Nathan Goldbaum <nathan12343 at gmail.com>
>>> wrote:
>>>
>>> Overall I think this is a good idea. I have a couple critiques about the
>>> mechanics that I'll reply inline about. I think it makes sense to separate
>>> out all of the astro-specific analysis modules.
>>>
>>> On Tue, Nov 22, 2016 at 8:37 PM, Britton Smith <brittonsmith at gmail.com>
>>> wrote:
>>>
>>>> Greetings,
>>>>
>>>> I would like to open a discussion on the idea of moving most of yt’s
>>>> analysis modules into an external yt extensions package. For ease of
>>>> reading, I will separate this email into what this would mean for the code,
>>>> what I see are the pros, cons, logistics, and open questions. I would very
>>>> much appreciate comment on this.
>>>>
>>>> What this means
>>>> If we did this, most of the contents of yt/analysis_modules would be
>>>> moved into a repository named something like yt_astro_analysis.
>>>> Installing this would be an option in the install script and would likely
>>>> also be pip installable. Imports would change from
>>>> from yt.analysis_modules.halo_analysis.api import HaloCatalog
>>>> to
>>>> from yt.extensions.astro_analysis.halo_analysis.api import HaloCatalog
>>>>
>>>> After creating yt_astro_analysis, there would be a period where the old
>>>> analysis_modules would still exist but be deprecated before being removed
>>>> at some point down the road.
>>>>
>>>> Pros
>>>>
>>>> - Almost all of the current analysis modules are specific to
>>>> astrophysics. As we continue to make the core functionality of yt less
>>>> astro specific, it’s not clear how to make room for non-astro analysis
>>>> modules. Putting everything together under analysis_modules will make
>>>> navigation and documentation messy and confusing. This would also
>>>> significantly slim down the yt codebase.
>>>> - Many of the tools in analysis_modules are very old and are in
>>>> need of API-breaking refactor. Some of these, like two_point_functions,
>>>> did not make the jump from yt-2 to yt-3 and are no longer usable. Many
>>>> tools no longer have a champion, someone interested in using, maintaining,
>>>> and updating them as yt’s core functionality develops and changes. Moving
>>>> analysis_modules from yt decouples them from yt’s release cycle, allowing
>>>> interested parties to make updates and releases on a separate, likely
>>>> shorter timescale. Some analysis_modules may even be better suited to be
>>>> moved into other extensions that are actively developed, such as the case
>>>> of the AbsorptionSpectrum with the Trident project.
>>>> - Similar to the point above, yt releases would not be slowed by
>>>> the need to update all of the championless modules. Individual analysis
>>>> modules can be tied to specific stable releases of yt and so assured to
>>>> work there.
>>>>
>>>>
>>>> Cons
>>>>
>>>> - This will take a non-zero amount of work. See below for a
>>>> summary of the primary tasks.
>>>> - There are some outstanding logistical questions. See below.
>>>> - Not having yt and analysis modules explicitly tied to the same
>>>> codebase/releases could result in analysis tools falling behind and out of
>>>> date.
>>>> - The disruption and need to alter scripts could irritate and
>>>> alienate users.
>>>>
>>>>
>>>> Proposed progression
>>>> This is roughly how this would happen. Here is a table with all
>>>> existing analysis modules, their status, and potential future:
>>>> https://goo.gl/HZykQA
>>>>
>>>> 1. Create yt_astro_analysis repo with all analysis modules that are
>>>> to be moved. Add an entry to the extensions page on yt-project.org.
>>>> Make it installed by default in the install script, at least at first.
>>>>
>>>>
>>> Should there only be one repo? If we're going to do this, it might help
>>> future maintainability to have one repo per analysis module. That way
>>> maintainers that only care about one analysis module don't need to worry
>>> about changes to other modules.
>>>
>>> I also don't think the clump finder in particular should be split out
>>> into its own package. That one isn't astro-specific and it might make sense
>>> to try to integrate it more deeply with the core of yt.
>>>
>>>
>>>>
>>>> 1. Deprecate all moved modules in yt.
>>>> 2. After some time, remove deprecated modules from yt.
>>>>
>>>>
>>>> Open issues
>>>> Here are some logistics and questions that still need to be worked out.
>>>>
>>>> - Can we set up separate testing for yt_astro_analysis? Would
>>>> maintaining this be a pain?
>>>>
>>>>
>>> If the tests don't require large datasets and are relatively quick, this
>>> might be a good opportunity to explore bitbucket pipelines. There's also
>>> drone.io or we can use the yt testing infrastructure. Since I'm not
>>> the maintainer of the yt testing infra I can't speak to whether it's ok to
>>> expand it to more packages.
>>>
>>>
>>>>
>>>> - How/where would the documentation be hosted?
>>>>
>>>>
>>> Readthedocs will likely be sufficient. If we care about inline code
>>> examples or whatnot, then we can explore using the yt testing
>>> infrastructure, with the same caveats as above.
>>>
>>>
>>>>
>>>> - How would we move the analysis modules source code and maintain
>>>> its revision history?
>>>>
>>>>
>>> This can be done relatively straightforwardly using the convert
>>> extension for mercurial and a filemap.
>>>
>>>
>>>>
>>>> Questions to yt-dev
>>>>
>>>> - Are you +/-1 on this? Any other comments?
>>>>
>>>>
>>> +1
>>>
>>>
>>>>
>>>> - Changes to the analysis_modules spreadsheet (https://goo.gl/HZykQA
>>>> )?
>>>> - Interested in helping out with this? If this happens, I propose
>>>> anyone interested meets for a hangout to discuss how to proceed.
>>>>
>>>>
>>> I'm happy to help.
>>>
>>>
>>>>
>>>> Thanks for reading!
>>>>
>>>> Britton
>>>>
>>>> _______________________________________________
>>>> yt-dev mailing list
>>>> yt-dev at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>
>>>>
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>
>>>
>>>
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20161205/6846fe17/attachment-0001.htm>
More information about the yt-dev
mailing list