<div dir="ltr">Hi Britton,<div><br></div><div>I like this idea a lot.  I think it would open us up to a much better development cycle, and even put more credibility in the notion of an extensions ecosystem.  It would certainly help us position yt as a lower-level library out of which analysis can be built, which I think is probably the most useful way to position it at this time.  I could go on with reasons I like this, but I don't think they'll be news to anyone.</div><div><br></div><div>I would be willing to help with this.  It's probably worth spinning up testing and docs for this on existing infrastructure, and I also think we might be able to get yt.amods to import from the extension, if that's helpful.  Getting the source code in is probably doable with something like hg convert.</div><div><br></div><div>Strong, strong +1.  (Sidenote: it's a good thing we don't do "price is right" scoring on votes, or else everyone would be doing 1.0+1e-16 all the time.)</div><div><br></div><div>-Matt</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Nov 22, 2016 at 7:37 PM, Britton Smith <span dir="ltr"><<a href="mailto:brittonsmith@gmail.com" target="_blank">brittonsmith@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span id="m_-8001026977587050164gmail-docs-internal-guid-14d1a6b0-8ed3-609f-b753-e59c9a72dec8"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Greetings,</span></p><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br></div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">I would like to open a discussion on the idea of moving most of yt’s analysis modules into an external yt extensions package.  For ease of reading, I will separate this email into what this would mean for the code, what I see are the pros, cons, logistics, and open questions.  I would very much appreciate comment on this.</span></p><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br></div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;font-weight:700;vertical-align:baseline;white-space:pre-wrap">What this means</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">If we did this, most of the contents of </span><span style="font-size:14.6667px;font-family:"courier new";color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">yt/analysis_modules</span><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> would be moved into a repository named something like </span><span style="font-size:14.6667px;font-family:"courier new";color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">yt_astro_analysis</span><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">.  Installing this would be an option in the install script and would likely also be pip installable.  Imports would change from</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:"courier new";color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">from yt.analysis_modules.halo_<wbr>analysis.api import HaloCatalog</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">to</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:"courier new";color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">from yt.extensions.astro_analysis.<wbr>halo_analysis.api import HaloCatalog</span></p><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br></div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">After creating yt_astro_analysis, there would be a period where the old analysis_modules would still exist but be deprecated before being removed at some point down the road.</span></p><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br></div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;font-weight:700;vertical-align:baseline;white-space:pre-wrap">Pros</span></p><ul style="margin-top:0pt;margin-bottom:0pt"><li dir="ltr" style="list-style-type:disc;font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Almost all of the current analysis modules are specific to astrophysics.  As we continue to make the core functionality of yt less astro specific, it’s not clear how to make room for non-astro analysis modules.  Putting everything together under analysis_modules will make navigation and documentation messy and confusing.  This would also significantly slim down the yt codebase.</span></p></li><li dir="ltr" style="list-style-type:disc;font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Many of the tools in analysis_modules are very old and are in need of API-breaking refactor.  Some of these, like two_point_functions, did not make the jump from yt-2 to yt-3 and are no longer usable.  Many tools no longer have a champion, someone interested in using, maintaining, and updating them as yt’s core functionality develops and changes.  Moving analysis_modules from yt decouples them from yt’s release cycle, allowing interested parties to make updates and releases on a separate, likely shorter timescale.  Some analysis_modules may even be better suited to be moved into other extensions that are actively developed, such as the case of the AbsorptionSpectrum with the Trident project.</span></p></li><li dir="ltr" style="list-style-type:disc;font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Similar to the point above, yt releases would not be slowed by the need to update all of the championless modules.  Individual analysis modules can be tied to specific stable releases of yt and so assured to work there.</span></p></li></ul><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br></div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;font-weight:700;vertical-align:baseline;white-space:pre-wrap">Cons</span></p><ul style="margin-top:0pt;margin-bottom:0pt"><li dir="ltr" style="list-style-type:disc;font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">This will take a non-zero amount of work.  See below for a summary of the primary tasks.</span></p></li><li dir="ltr" style="list-style-type:disc;font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">There are some outstanding logistical questions.  See below.</span></p></li><li dir="ltr" style="list-style-type:disc;font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Not having yt and analysis modules explicitly tied to the same codebase/releases could result in analysis tools falling behind and out of date.</span></p></li><li dir="ltr" style="list-style-type:disc;font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">The disruption and need to alter scripts could irritate and alienate users.</span></p></li></ul><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br></div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;font-weight:700;vertical-align:baseline;white-space:pre-wrap">Proposed progression</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">This is roughly how this would happen.  Here is a table with all existing analysis modules, their status, and potential future: <a href="https://goo.gl/HZykQA" target="_blank">https://goo.gl/HZykQA</a></span></p><ol style="margin-top:0pt;margin-bottom:0pt"><li dir="ltr" style="list-style-type:decimal;font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Create yt_astro_analysis repo with all analysis modules that are to be moved.  Add an entry to the extensions page on <a href="http://yt-project.org" target="_blank">yt-project.org</a>.  Make it installed by default in the install script, at least at first.</span></p></li><li dir="ltr" style="list-style-type:decimal;font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Deprecate all moved modules in yt.</span></p></li><li dir="ltr" style="list-style-type:decimal;font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">After some time, remove deprecated modules from yt.</span></p></li></ol><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br></div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;font-weight:700;vertical-align:baseline;white-space:pre-wrap">Open issues</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Here are some logistics and questions that still need to be worked out.</span></p><ul style="margin-top:0pt;margin-bottom:0pt"><li dir="ltr" style="list-style-type:disc;font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Can we set up separate testing for yt_astro_analysis?  Would maintaining this be a pain?</span></p></li><li dir="ltr" style="list-style-type:disc;font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">How/where would the documentation be hosted?</span></p></li><li dir="ltr" style="list-style-type:disc;font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">How would we move the analysis modules source code and maintain its revision history?</span></p></li></ul><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br></div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;font-weight:700;vertical-align:baseline;white-space:pre-wrap">Questions to yt-dev</span></p><ul style="margin-top:0pt;margin-bottom:0pt"><li dir="ltr" style="list-style-type:disc;font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Are you +/-1 on this?  Any other comments?</span></p></li><li dir="ltr" style="list-style-type:disc;font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Changes to the analysis_modules spreadsheet (</span><a href="https://goo.gl/HZykQA" style="text-decoration:none" target="_blank"><span style="font-size:14.6667px;background-color:transparent;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">https://goo.gl/HZykQA</span></a><span style="font-size:14.6667px;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">)?</span></p></li><li dir="ltr" style="list-style-type:disc;font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Interested in helping out with this?  If this happens, I propose anyone interested meets for a hangout to discuss how to proceed.</span></p></li></ul><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br></div><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Thanks for reading!</span></p><span class="HOEnZb"><font color="#888888"><div dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br></div><span style="font-size:14.6667px;font-family:arial;color:rgb(0,0,0);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Britton</span></font></span></span><br></div>
<br>______________________________<wbr>_________________<br>
yt-dev mailing list<br>
<a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" rel="noreferrer" target="_blank">http://lists.spacepope.org/<wbr>listinfo.cgi/yt-dev-spacepope.<wbr>org</a><br>
<br></blockquote></div><br></div>