[yt-dev] the future of analysis_modules
Britton Smith
brittonsmith at gmail.com
Tue Nov 29 10:10:03 PST 2016
Hi everyone,
We will have a hangout discussion about moving forward with this on
Thursday, Dec 1 at 2pm EST. If you weren't on the poll and would like to
be invited to the hangout, send me an email.
Britton
On Wed, Nov 23, 2016 at 9:48 AM, Britton Smith <brittonsmith at gmail.com>
wrote:
> Hey all,
>
> Thanks to everyone for their feedback. This has been a good discussion so
> far. With Thanksgiving tomorrow, let's give people until the end of Monday
> to contribute comments.
>
> It looks like this idea is getting a good amount of support, so let's plan
> of having a live discussion about this in the near future. After Monday, I
> will organize a doodle poll inviting everyone who has responded so far. If
> you don't have anything to add to the discussion but would like to be
> involved, just drop me a note and I'll include you on the poll.
>
> Thanks again everyone and have a good weekend.
>
> Britton
>
> On Wed, Nov 23, 2016 at 8:19 AM, John ZuHone <jzuhone at gmail.com> wrote:
>
>> I’m +1 on this and I agree with Nathan’s comments. In particular, I think
>> that we shouldn’t bundle them all into one package but probably split them
>> off into separate packages. This requires more work on the behalf of the
>> maintainers of those modules, of course.
>>
>> In addition to keeping in clump_finder, particle_trajectories should also
>> probably be left in, and perhaps be turned into some kind of time series
>> object or something else.
>>
>> I’m happy to help out—in particular I can split off sunyaev_zeldovich and
>> ppv_cube and spin them off as separate packages. photon_simulator is
>> already deprecated in favor of pyXSIM.
>>
>> All that spectral_integrator does is generate fields, so perhaps its
>> functionality needs to be merged into the astro fields part somehow and
>> left inside yt itself.
>>
>> On Nov 22, 2016, at 8:54 PM, Nathan Goldbaum <nathan12343 at gmail.com>
>> wrote:
>>
>> Overall I think this is a good idea. I have a couple critiques about the
>> mechanics that I'll reply inline about. I think it makes sense to separate
>> out all of the astro-specific analysis modules.
>>
>> On Tue, Nov 22, 2016 at 8:37 PM, Britton Smith <brittonsmith at gmail.com>
>> wrote:
>>
>>> Greetings,
>>>
>>> I would like to open a discussion on the idea of moving most of yt’s
>>> analysis modules into an external yt extensions package. For ease of
>>> reading, I will separate this email into what this would mean for the code,
>>> what I see are the pros, cons, logistics, and open questions. I would very
>>> much appreciate comment on this.
>>>
>>> What this means
>>> If we did this, most of the contents of yt/analysis_modules would be
>>> moved into a repository named something like yt_astro_analysis.
>>> Installing this would be an option in the install script and would likely
>>> also be pip installable. Imports would change from
>>> from yt.analysis_modules.halo_analysis.api import HaloCatalog
>>> to
>>> from yt.extensions.astro_analysis.halo_analysis.api import HaloCatalog
>>>
>>> After creating yt_astro_analysis, there would be a period where the old
>>> analysis_modules would still exist but be deprecated before being removed
>>> at some point down the road.
>>>
>>> Pros
>>>
>>> - Almost all of the current analysis modules are specific to
>>> astrophysics. As we continue to make the core functionality of yt less
>>> astro specific, it’s not clear how to make room for non-astro analysis
>>> modules. Putting everything together under analysis_modules will make
>>> navigation and documentation messy and confusing. This would also
>>> significantly slim down the yt codebase.
>>> - Many of the tools in analysis_modules are very old and are in need
>>> of API-breaking refactor. Some of these, like two_point_functions, did not
>>> make the jump from yt-2 to yt-3 and are no longer usable. Many tools no
>>> longer have a champion, someone interested in using, maintaining, and
>>> updating them as yt’s core functionality develops and changes. Moving
>>> analysis_modules from yt decouples them from yt’s release cycle, allowing
>>> interested parties to make updates and releases on a separate, likely
>>> shorter timescale. Some analysis_modules may even be better suited to be
>>> moved into other extensions that are actively developed, such as the case
>>> of the AbsorptionSpectrum with the Trident project.
>>> - Similar to the point above, yt releases would not be slowed by the
>>> need to update all of the championless modules. Individual analysis
>>> modules can be tied to specific stable releases of yt and so assured to
>>> work there.
>>>
>>>
>>> Cons
>>>
>>> - This will take a non-zero amount of work. See below for a summary
>>> of the primary tasks.
>>> - There are some outstanding logistical questions. See below.
>>> - Not having yt and analysis modules explicitly tied to the same
>>> codebase/releases could result in analysis tools falling behind and out of
>>> date.
>>> - The disruption and need to alter scripts could irritate and
>>> alienate users.
>>>
>>>
>>> Proposed progression
>>> This is roughly how this would happen. Here is a table with all
>>> existing analysis modules, their status, and potential future:
>>> https://goo.gl/HZykQA
>>>
>>> 1. Create yt_astro_analysis repo with all analysis modules that are
>>> to be moved. Add an entry to the extensions page on yt-project.org.
>>> Make it installed by default in the install script, at least at first.
>>>
>>>
>> Should there only be one repo? If we're going to do this, it might help
>> future maintainability to have one repo per analysis module. That way
>> maintainers that only care about one analysis module don't need to worry
>> about changes to other modules.
>>
>> I also don't think the clump finder in particular should be split out
>> into its own package. That one isn't astro-specific and it might make sense
>> to try to integrate it more deeply with the core of yt.
>>
>>
>>>
>>> 1. Deprecate all moved modules in yt.
>>> 2. After some time, remove deprecated modules from yt.
>>>
>>>
>>> Open issues
>>> Here are some logistics and questions that still need to be worked out.
>>>
>>> - Can we set up separate testing for yt_astro_analysis? Would
>>> maintaining this be a pain?
>>>
>>>
>> If the tests don't require large datasets and are relatively quick, this
>> might be a good opportunity to explore bitbucket pipelines. There's also
>> drone.io or we can use the yt testing infrastructure. Since I'm not the
>> maintainer of the yt testing infra I can't speak to whether it's ok to
>> expand it to more packages.
>>
>>
>>>
>>> - How/where would the documentation be hosted?
>>>
>>>
>> Readthedocs will likely be sufficient. If we care about inline code
>> examples or whatnot, then we can explore using the yt testing
>> infrastructure, with the same caveats as above.
>>
>>
>>>
>>> - How would we move the analysis modules source code and maintain
>>> its revision history?
>>>
>>>
>> This can be done relatively straightforwardly using the convert extension
>> for mercurial and a filemap.
>>
>>
>>>
>>> Questions to yt-dev
>>>
>>> - Are you +/-1 on this? Any other comments?
>>>
>>>
>> +1
>>
>>
>>>
>>> - Changes to the analysis_modules spreadsheet (https://goo.gl/HZykQA
>>> )?
>>> - Interested in helping out with this? If this happens, I propose
>>> anyone interested meets for a hangout to discuss how to proceed.
>>>
>>>
>> I'm happy to help.
>>
>>
>>>
>>> Thanks for reading!
>>>
>>> Britton
>>>
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>
>>>
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>>
>>
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20161129/acf01fc0/attachment-0001.htm>
More information about the yt-dev
mailing list