<div dir="ltr">Hi everyone,<div><br></div><div>I have submitted a draft YTEP here:</div><div><br></div><div><a href="https://bitbucket.org/yt_analysis/ytep/pull-requests/60/ytep-0029-extension-packages/diff">https://bitbucket.org/yt_analysis/ytep/pull-requests/60/ytep-0029-extension-packages/diff</a><br></div><div><br></div><div>Please feel free to leave short comments there, but if they are larger comments, I think they should perhaps come here. I didn't discuss everything in it, but I think the biggest change is non-technical, and instead that of a mindset. Also, the website.</div><div><br></div><div>-Matt</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jan 19, 2016 at 9:20 AM, Matthew Turk <span dir="ltr"><<a href="mailto:matthewturk@gmail.com" target="_blank">matthewturk@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi all,<br>
<br>
Thanks for all your feedback. I think it seems like the proposal #1<br>
is a good way to go. I'm going to try to start writing up a YTEP that<br>
formalizes this, and then we can put that up for a vote -- let's not<br>
consider anything "settled" yet, but it seems pretty reasonable to<br>
move forward with a more detailed proposal. If anyone else has<br>
comments, please feel free to leave them here, or on the YTEP.<br>
<br>
-Matt<br>
<div class="HOEnZb"><div class="h5"><br>
On Tue, Jan 19, 2016 at 7:24 AM, Britton Smith <<a href="mailto:brittonsmith@gmail.com">brittonsmith@gmail.com</a>> wrote:<br>
> I guess this is clear from the original email, but I very much like the idea<br>
> of an ecosystem of a core yt package and separate extension packages.<br>
> Personally, I would stop short of making frontends into external packages.<br>
> I think one of the big values of yt is the library of frontends we maintain<br>
> and the ability of people to install the source and immediately be able to<br>
> load their data without further requirements. I think we run the risk of<br>
> losing new users who have to go through too many steps just to get their<br>
> data loaded.<br>
><br>
> Perhaps the first thing should be to start up the domain context efforts<br>
> again. By some later 3.x release, we could have an astro domain with domain<br>
> specific fields and extensions all still within the main yt codebase. Then,<br>
> moving to 4.0, we start to extract things into separate packages.<br>
><br>
> On Mon, Jan 18, 2016 at 6:16 PM, Michael Zingale<br>
> <<a href="mailto:michael.zingale@stonybrook.edu">michael.zingale@stonybrook.edu</a>> wrote:<br>
>><br>
>> Affiliated packages sound like a good idea to me, but I am a bit hesitant<br>
>> about splitting the volume renderer and frontends out of yt's core.<br>
>><br>
>> On Mon, Jan 18, 2016 at 1:11 PM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
>> wrote:<br>
>>><br>
>>> Hi Desika,<br>
>>><br>
>>> Yup, that's what I meant by it. And there are things that would<br>
>>> normally qualify as that type of extension, too, that are already in<br>
>>> the codebase.<br>
>>><br>
>>> Flask's extensions are another good example of how this type of thing is<br>
>>> done:<br>
>>><br>
>>> <a href="http://flask.pocoo.org/extensions/" rel="noreferrer" target="_blank">http://flask.pocoo.org/extensions/</a><br>
>>> <a href="http://flask.pocoo.org/docs/0.10/extensiondev/" rel="noreferrer" target="_blank">http://flask.pocoo.org/docs/0.10/extensiondev/</a><br>
>>><br>
>>> -Matt<br>
>>><br>
>>> On Mon, Jan 18, 2016 at 12:02 PM, Desika Narayanan<br>
>>> <<a href="mailto:desika.narayanan@gmail.com">desika.narayanan@gmail.com</a>> wrote:<br>
>>> > Hey All,<br>
>>> ><br>
>>> > Matt - just to maybe clarify by examples...are you thinking things like<br>
>>> > Trident or powderday are sorts of yt extensions that you mean? (If so,<br>
>>> > then<br>
>>> > I do like this idea a lot too). I think one might imagine having a<br>
>>> > clear<br>
>>> > set of guidelines for developers of extensions to aspire toward (even<br>
>>> > if<br>
>>> > it's for example, having some basic set of or style of documentation)<br>
>>> > so<br>
>>> > that from the user point, when looking at extensions on the yt docs,<br>
>>> > there's some modicum of uniformity. That might make it easier for<br>
>>> > users to<br>
>>> > browse and understand the extensions.<br>
>>> ><br>
>>> > -d<br>
>>> ><br>
>>> > On Mon, Jan 18, 2016 at 12:43 PM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
>>> > wrote:<br>
>>> >><br>
>>> >> On Mon, Jan 18, 2016 at 11:32 AM, John ZuHone <<a href="mailto:jzuhone@gmail.com">jzuhone@gmail.com</a>><br>
>>> >> wrote:<br>
>>> >> > Matt can correct me if I am wrong, but I was under the impression<br>
>>> >> > that<br>
>>> >> > the two components of the proposal were separate and not inevitably<br>
>>> >> > tied to<br>
>>> >> > one another.<br>
>>> >> ><br>
>>> >> > I am in strong support of the idea of extensions, but I strongly<br>
>>> >> > disagree with most of the second component of this proposal. I<br>
>>> >> > believe it is<br>
>>> >> > far too disruptive, and I honestly don’t see the argument for it<br>
>>> >> > (except<br>
>>> >> > maybe splitting some astro-specific stuff out into an extension).<br>
>>> >> > But<br>
>>> >> > spinning off stuff like volume rendering and other big components<br>
>>> >> > like that<br>
>>> >> > seems unnecessary.<br>
>>> >> ><br>
>>> >><br>
>>> >> No need to correct; the second one is indeed an extension of the<br>
>>> >> first, and a separate proposal.<br>
>>> >><br>
>>> >> >> On Jan 18, 2016, at 12:26 PM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
>>> >> >> wrote:<br>
>>> >> >><br>
>>> >> >> Hi Nathan,<br>
>>> >> >><br>
>>> >> >> Just to quickly address one component before anybody else replies:<br>
>>> >> >> I<br>
>>> >> >> would volunteer to do the majority of this work (which I think from<br>
>>> >> >> a<br>
>>> >> >> technical perspective is not necessarily too bad), and I don't see<br>
>>> >> >> it<br>
>>> >> >> being a 3.X release, but maybe a 4.0 release, and I don't have<br>
>>> >> >> answers<br>
>>> >> >> for when it would happen since I think that is a community decision<br>
>>> >> >> we<br>
>>> >> >> shouldn't rush into.<br>
>>> >> >><br>
>>> >> >> On Mon, Jan 18, 2016 at 11:21 AM, Nathan Goldbaum<br>
>>> >> >> <<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a>> wrote:<br>
>>> >> >>> Just a few thoughts about this.<br>
>>> >> >>><br>
>>> >> >>> I think encouraging more external packages to depend on yt will be<br>
>>> >> >>> great,<br>
>>> >> >>> and will keep us more honest about the backward compatibility<br>
>>> >> >>> guarantees<br>
>>> >> >>> we've committed ourselves to. Building social capital by<br>
>>> >> >>> encouraging<br>
>>> >> >>> an<br>
>>> >> >>> "affiliated" package ecosystem like astropy has is also good, both<br>
>>> >> >>> for<br>
>>> >> >>> social capital and also by enabling cool new features for our<br>
>>> >> >>> users.<br>
>>> >> >>><br>
>>> >> >>> That said, I'm skeptical about some of the technical components of<br>
>>> >> >>> Matt's<br>
>>> >> >>> proposal. In particular, I'm concerned with the amount of work<br>
>>> >> >>> required to<br>
>>> >> >>> make major changes to the core yt codebase, while still<br>
>>> >> >>> maintaining<br>
>>> >> >>> backward<br>
>>> >> >>> compatibility. Who will do this work? When will it happen? Would<br>
>>> >> >>> this<br>
>>> >> >>> constitute a yt 4.0 release?<br>
>>> >> >>><br>
>>> >> >>> While I do think making the core yt package less astronomy focused<br>
>>> >> >>> is<br>
>>> >> >>> good,<br>
>>> >> >>> I also think it's possible to do so while maintaining strong<br>
>>> >> >>> backward<br>
>>> >> >>> compatibility guarantees, or at least going through a deprecation<br>
>>> >> >>> cycle over<br>
>>> >> >>> the course of several minor versions. I don't think making another<br>
>>> >> >>> cycle of<br>
>>> >> >>> breaking changes so soon after yt-3.0 is a good idea, as that will<br>
>>> >> >>> impose a<br>
>>> >> >>> large amount of work both on the yt developers and on our users.<br>
>>> >> >>><br>
>>> >> >>> On Mon, Jan 18, 2016 at 9:43 AM, Matthew Turk<br>
>>> >> >>> <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
>>> >> >>> wrote:<br>
>>> >> >>>><br>
>>> >> >>>> Hi everyone,<br>
>>> >> >>>><br>
>>> >> >>>> The last couple weeks I’ve been thinking a lot about the future<br>
>>> >> >>>> of<br>
>>> >> >>>> yt.<br>
>>> >> >>>> What I’d like to propose is that we shift investment in yt as a<br>
>>> >> >>>> single, monolithic codebase into yt as a project, or an ecosystem<br>
>>> >> >>>> of<br>
>>> >> >>>> projects.<br>
>>> >> >>>><br>
>>> >> >>>> This came out of the discussion of the extension/affiliated<br>
>>> >> >>>> packages,<br>
>>> >> >>>> analysis modules, and so on. Britton has for a while been<br>
>>> >> >>>> pitching<br>
>>> >> >>>> the idea [which I will poorly paraphrase here] that yt can be the<br>
>>> >> >>>> framework on top of which killer apps can be built. I think this<br>
>>> >> >>>> is<br>
>>> >> >>>> great.<br>
>>> >> >>>><br>
>>> >> >>>> What’s holding us back in some ways from this is that yt is<br>
>>> >> >>>> currently<br>
>>> >> >>>> structured as a monolithic code base, with little to no discovery<br>
>>> >> >>>> of<br>
>>> >> >>>> other packages and apps and whatnot. We tried for a while to<br>
>>> >> >>>> change<br>
>>> >> >>>> this with The Barn, but it ended up not quite taking off. I<br>
>>> >> >>>> think<br>
>>> >> >>>> the<br>
>>> >> >>>> time is right to try to change the way we think about yt to be<br>
>>> >> >>>> more<br>
>>> >> >>>> about yt the Project, rather than yt the Codebase; the core<br>
>>> >> >>>> codebase<br>
>>> >> >>>> is an important component of this, but not the whole of it.<br>
>>> >> >>>><br>
>>> >> >>>> Encouraging an ecosystem of packages can have a few very<br>
>>> >> >>>> important<br>
>>> >> >>>> benefits:<br>
>>> >> >>>><br>
>>> >> >>>> * External packages will confer greater individual credit to the<br>
>>> >> >>>> folks who develop them.<br>
>>> >> >>>> * External packages can be versioned and developed independently;<br>
>>> >> >>>> the<br>
>>> >> >>>> review process can be different.<br>
>>> >> >>>> * yt’s core can be emphasized as a generic package, on top of<br>
>>> >> >>>> which<br>
>>> >> >>>> astronomy analysis can be built.<br>
>>> >> >>>> * Packages can be maintained wherever, including alternate<br>
>>> >> >>>> locations<br>
>>> >> >>>> such as github.<br>
>>> >> >>>><br>
>>> >> >>>> On the other hand, having packages inside the main distribution<br>
>>> >> >>>> makes<br>
>>> >> >>>> discoverability much, much easier. It also enables everything to<br>
>>> >> >>>> be<br>
>>> >> >>>> In The Box. And, the continuous integration and testing system<br>
>>> >> >>>> is<br>
>>> >> >>>> already set up for yt. But, these are all possible to overcome<br>
>>> >> >>>> -- we<br>
>>> >> >>>> can devise a strategy for adding packages to the CI system (and<br>
>>> >> >>>> if<br>
>>> >> >>>> they are externally managed, they can also rely on yt as a<br>
>>> >> >>>> dependency<br>
>>> >> >>>> and use whatever CI system they like!) and we can improve<br>
>>> >> >>>> discoverability by refocusing the website to enable this. I've<br>
>>> >> >>>> asked<br>
>>> >> >>>> Kacper about adding new packages, and it's not as easy as it<br>
>>> >> >>>> might<br>
>>> >> >>>> seem, so we may need to be careful about how that process occurs;<br>
>>> >> >>>> one<br>
>>> >> >>>> possibility would be to provide servers and ready-made setups,<br>
>>> >> >>>> but<br>
>>> >> >>>> have individuals do the heavy lifting. We could even have<br>
>>> >> >>>> something<br>
>>> >> >>>> in the codebase that describes some packages that are available.<br>
>>> >> >>>> External packages could have much looser dependency rules, which<br>
>>> >> >>>> means<br>
>>> >> >>>> they can be free to take advantage of things like OpenCL, numba,<br>
>>> >> >>>> etc,<br>
>>> >> >>>> without having to add them to the primary codebase.<br>
>>> >> >>>><br>
>>> >> >>>> Synchronizing APIs and versions across extension packages may be<br>
>>> >> >>>> difficult in some particular cases, but I suspect in practice<br>
>>> >> >>>> will<br>
>>> >> >>>> not<br>
>>> >> >>>> be an issue, as long as we continue to have a reasonably stable<br>
>>> >> >>>> *public* API, and graduate a few things (such as .blocks) into a<br>
>>> >> >>>> public API from semi-private.<br>
>>> >> >>>><br>
>>> >> >>>> To this end, of really encouraging an ecosystem of packages, I’d<br>
>>> >> >>>> like<br>
>>> >> >>>> to propose two things, in increasing order of disruptiveness.<br>
>>> >> >>>><br>
>>> >> >>>> First: Encourage extension packages. This would mean:<br>
>>> >> >>>> * Reorganize website to allow for extension packages to be<br>
>>> >> >>>> displayed<br>
>>> >> >>>> prominently<br>
>>> >> >>>> * Add support for name-space packages in yt<br>
>>> >> >>>> * (possible) split out some packages from analysis_modules,<br>
>>> >> >>>> including<br>
>>> >> >>>> halo finding<br>
>>> >> >>>> * Codify process of extension package creation, including how to<br>
>>> >> >>>> have<br>
>>> >> >>>> CI set up for them and build system.<br>
>>> >> >>>><br>
>>> >> >>>> The second, more disruptive proposal:<br>
>>> >> >>>> * Split yt into subprojects. This would include spinning out the<br>
>>> >> >>>> volume rendering and some or all of the frontends, and probably<br>
>>> >> >>>> the<br>
>>> >> >>>> testing infrastructure as well.<br>
>>> >> >>>> * Split further astro-specific routines into an astro extension,<br>
>>> >> >>>> and<br>
>>> >> >>>> begin the process of doing this with other domains as well. (As<br>
>>> >> >>>> in<br>
>>> >> >>>> the long-simmering domain context YTEP.)<br>
>>> >> >>>><br>
>>> >> >>>> I’ll invite comments from everyone, but particularly from folks<br>
>>> >> >>>> who<br>
>>> >> >>>> have either not contributed to an analysis module or extension<br>
>>> >> >>>> package<br>
>>> >> >>>> because of concerns that would be addressed by this, as well as<br>
>>> >> >>>> from<br>
>>> >> >>>> core developers this would impact. If the thread gets too<br>
>>> >> >>>> unweildy<br>
>>> >> >>>> we<br>
>>> >> >>>> may also want to table this for the next yt team meeting.<br>
>>> >> >>>><br>
>>> >> >>>> Thanks,<br>
>>> >> >>>><br>
>>> >> >>>> Matt<br>
>>> >> >>>> _______________________________________________<br>
>>> >> >>>> yt-dev mailing list<br>
>>> >> >>>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>> >> >>>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" rel="noreferrer" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>>> >> >>><br>
>>> >> >>><br>
>>> >> >>><br>
>>> >> >>> _______________________________________________<br>
>>> >> >>> yt-dev mailing list<br>
>>> >> >>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>> >> >>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" rel="noreferrer" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>>> >> >>><br>
>>> >> >> _______________________________________________<br>
>>> >> >> yt-dev mailing list<br>
>>> >> >> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>> >> >> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" rel="noreferrer" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>>> >> ><br>
>>> >> > _______________________________________________<br>
>>> >> > yt-dev mailing list<br>
>>> >> > <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>> >> > <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" rel="noreferrer" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>>> >> _______________________________________________<br>
>>> >> yt-dev mailing list<br>
>>> >> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>> >> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" rel="noreferrer" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>>> ><br>
>>> ><br>
>>> ><br>
>>> > _______________________________________________<br>
>>> > yt-dev mailing list<br>
>>> > <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>> > <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" rel="noreferrer" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>>> ><br>
>>> _______________________________________________<br>
>>> yt-dev mailing list<br>
>>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" rel="noreferrer" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>><br>
>><br>
>><br>
>><br>
>> --<br>
>> Michael Zingale<br>
>> Associate Professor<br>
>><br>
>> Dept. of Physics & Astronomy • Stony Brook University • Stony Brook, NY<br>
>> 11794-3800<br>
>> phone: <a href="tel:631-632-8225" value="+16316328225">631-632-8225</a><br>
>> e-mail: <a href="mailto:Michael.Zingale@stonybrook.edu">Michael.Zingale@stonybrook.edu</a><br>
>> web: <a href="http://www.astro.sunysb.edu/mzingale" rel="noreferrer" target="_blank">http://www.astro.sunysb.edu/mzingale</a><br>
>> github: <a href="http://github.com/zingale" rel="noreferrer" target="_blank">http://github.com/zingale</a><br>
>><br>
>><br>
>> _______________________________________________<br>
>> yt-dev mailing list<br>
>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" rel="noreferrer" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>><br>
><br>
><br>
> _______________________________________________<br>
> yt-dev mailing list<br>
> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" rel="noreferrer" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
><br>
</div></div></blockquote></div><br></div>