<meta http-equiv="Content-Type" content="text/html; charset=utf-8"><div dir="ltr">Affiliated packages sound like a good idea to me, but I am a bit hesitant about splitting the volume renderer and frontends out of yt's core.</div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jan 18, 2016 at 1:11 PM, Matthew Turk <span dir="ltr"><<a href="mailto:matthewturk@gmail.com" target="_blank">matthewturk@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Desika,<br>
<br>
Yup, that's what I meant by it. And there are things that would<br>
normally qualify as that type of extension, too, that are already in<br>
the codebase.<br>
<br>
Flask's extensions are another good example of how this type of thing is done:<br>
<br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__flask.pocoo.org_extensions_&d=BQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=hgcBC3x6dKFoTrmFmMYYbKNfiHZlGLKliIidd1LwmHI&m=1HwdG4qc9p2mvxARPwDgSeThRIrQZCb68UKjDS5-pmk&s=MhDetGrTXIBmv38YAw18VwPCRSuuhE5KD66MZGak4uQ&e=" rel="noreferrer" target="_blank">http://flask.pocoo.org/extensions/</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__flask.pocoo.org_docs_0.10_extensiondev_&d=BQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=hgcBC3x6dKFoTrmFmMYYbKNfiHZlGLKliIidd1LwmHI&m=1HwdG4qc9p2mvxARPwDgSeThRIrQZCb68UKjDS5-pmk&s=Pl7L5G5OtwKFSENPnCvg_XmFulGYXoKLyu-3B1E4j2c&e=" rel="noreferrer" target="_blank">http://flask.pocoo.org/docs/0.10/extensiondev/</a><br>
<br>
-Matt<br>
<div class="HOEnZb"><div class="h5"><br>
On Mon, Jan 18, 2016 at 12:02 PM, Desika Narayanan<br>
<<a href="mailto:desika.narayanan@gmail.com">desika.narayanan@gmail.com</a>> wrote:<br>
> Hey All,<br>
><br>
> Matt - just to maybe clarify by examples...are you thinking things like<br>
> Trident or powderday are sorts of yt extensions that you mean? (If so, then<br>
> I do like this idea a lot too). I think one might imagine having a clear<br>
> set of guidelines for developers of extensions to aspire toward (even if<br>
> it's for example, having some basic set of or style of documentation) so<br>
> that from the user point, when looking at extensions on the yt docs,<br>
> there's some modicum of uniformity. That might make it easier for users to<br>
> browse and understand the extensions.<br>
><br>
> -d<br>
><br>
> On Mon, Jan 18, 2016 at 12:43 PM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
> wrote:<br>
>><br>
>> On Mon, Jan 18, 2016 at 11:32 AM, John ZuHone <<a href="mailto:jzuhone@gmail.com">jzuhone@gmail.com</a>> wrote:<br>
>> > Matt can correct me if I am wrong, but I was under the impression that<br>
>> > the two components of the proposal were separate and not inevitably tied to<br>
>> > one another.<br>
>> ><br>
>> > I am in strong support of the idea of extensions, but I strongly<br>
>> > disagree with most of the second component of this proposal. I believe it is<br>
>> > far too disruptive, and I honestly don’t see the argument for it (except<br>
>> > maybe splitting some astro-specific stuff out into an extension). But<br>
>> > spinning off stuff like volume rendering and other big components like that<br>
>> > seems unnecessary.<br>
>> ><br>
>><br>
>> No need to correct; the second one is indeed an extension of the<br>
>> first, and a separate proposal.<br>
>><br>
>> >> On Jan 18, 2016, at 12:26 PM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
>> >> wrote:<br>
>> >><br>
>> >> Hi Nathan,<br>
>> >><br>
>> >> Just to quickly address one component before anybody else replies: I<br>
>> >> would volunteer to do the majority of this work (which I think from a<br>
>> >> technical perspective is not necessarily too bad), and I don't see it<br>
>> >> being a 3.X release, but maybe a 4.0 release, and I don't have answers<br>
>> >> for when it would happen since I think that is a community decision we<br>
>> >> shouldn't rush into.<br>
>> >><br>
>> >> On Mon, Jan 18, 2016 at 11:21 AM, Nathan Goldbaum<br>
>> >> <<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a>> wrote:<br>
>> >>> Just a few thoughts about this.<br>
>> >>><br>
>> >>> I think encouraging more external packages to depend on yt will be<br>
>> >>> great,<br>
>> >>> and will keep us more honest about the backward compatibility<br>
>> >>> guarantees<br>
>> >>> we've committed ourselves to. Building social capital by encouraging<br>
>> >>> an<br>
>> >>> "affiliated" package ecosystem like astropy has is also good, both for<br>
>> >>> social capital and also by enabling cool new features for our users.<br>
>> >>><br>
>> >>> That said, I'm skeptical about some of the technical components of<br>
>> >>> Matt's<br>
>> >>> proposal. In particular, I'm concerned with the amount of work<br>
>> >>> required to<br>
>> >>> make major changes to the core yt codebase, while still maintaining<br>
>> >>> backward<br>
>> >>> compatibility. Who will do this work? When will it happen? Would this<br>
>> >>> constitute a yt 4.0 release?<br>
>> >>><br>
>> >>> While I do think making the core yt package less astronomy focused is<br>
>> >>> good,<br>
>> >>> I also think it's possible to do so while maintaining strong backward<br>
>> >>> compatibility guarantees, or at least going through a deprecation<br>
>> >>> cycle over<br>
>> >>> the course of several minor versions. I don't think making another<br>
>> >>> cycle of<br>
>> >>> breaking changes so soon after yt-3.0 is a good idea, as that will<br>
>> >>> impose a<br>
>> >>> large amount of work both on the yt developers and on our users.<br>
>> >>><br>
>> >>> On Mon, Jan 18, 2016 at 9:43 AM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
>> >>> wrote:<br>
>> >>>><br>
>> >>>> Hi everyone,<br>
>> >>>><br>
>> >>>> The last couple weeks I’ve been thinking a lot about the future of<br>
>> >>>> yt.<br>
>> >>>> What I’d like to propose is that we shift investment in yt as a<br>
>> >>>> single, monolithic codebase into yt as a project, or an ecosystem of<br>
>> >>>> projects.<br>
>> >>>><br>
>> >>>> This came out of the discussion of the extension/affiliated packages,<br>
>> >>>> analysis modules, and so on. Britton has for a while been pitching<br>
>> >>>> the idea [which I will poorly paraphrase here] that yt can be the<br>
>> >>>> framework on top of which killer apps can be built. I think this is<br>
>> >>>> great.<br>
>> >>>><br>
>> >>>> What’s holding us back in some ways from this is that yt is currently<br>
>> >>>> structured as a monolithic code base, with little to no discovery of<br>
>> >>>> other packages and apps and whatnot. We tried for a while to change<br>
>> >>>> this with The Barn, but it ended up not quite taking off. I think<br>
>> >>>> the<br>
>> >>>> time is right to try to change the way we think about yt to be more<br>
>> >>>> about yt the Project, rather than yt the Codebase; the core codebase<br>
>> >>>> is an important component of this, but not the whole of it.<br>
>> >>>><br>
>> >>>> Encouraging an ecosystem of packages can have a few very important<br>
>> >>>> benefits:<br>
>> >>>><br>
>> >>>> * External packages will confer greater individual credit to the<br>
>> >>>> folks who develop them.<br>
>> >>>> * External packages can be versioned and developed independently; the<br>
>> >>>> review process can be different.<br>
>> >>>> * yt’s core can be emphasized as a generic package, on top of which<br>
>> >>>> astronomy analysis can be built.<br>
>> >>>> * Packages can be maintained wherever, including alternate locations<br>
>> >>>> such as github.<br>
>> >>>><br>
>> >>>> On the other hand, having packages inside the main distribution makes<br>
>> >>>> discoverability much, much easier. It also enables everything to be<br>
>> >>>> In The Box. And, the continuous integration and testing system is<br>
>> >>>> already set up for yt. But, these are all possible to overcome -- we<br>
>> >>>> can devise a strategy for adding packages to the CI system (and if<br>
>> >>>> they are externally managed, they can also rely on yt as a dependency<br>
>> >>>> and use whatever CI system they like!) and we can improve<br>
>> >>>> discoverability by refocusing the website to enable this. I've asked<br>
>> >>>> Kacper about adding new packages, and it's not as easy as it might<br>
>> >>>> seem, so we may need to be careful about how that process occurs; one<br>
>> >>>> possibility would be to provide servers and ready-made setups, but<br>
>> >>>> have individuals do the heavy lifting. We could even have something<br>
>> >>>> in the codebase that describes some packages that are available.<br>
>> >>>> External packages could have much looser dependency rules, which<br>
>> >>>> means<br>
>> >>>> they can be free to take advantage of things like OpenCL, numba, etc,<br>
>> >>>> without having to add them to the primary codebase.<br>
>> >>>><br>
>> >>>> Synchronizing APIs and versions across extension packages may be<br>
>> >>>> difficult in some particular cases, but I suspect in practice will<br>
>> >>>> not<br>
>> >>>> be an issue, as long as we continue to have a reasonably stable<br>
>> >>>> *public* API, and graduate a few things (such as .blocks) into a<br>
>> >>>> public API from semi-private.<br>
>> >>>><br>
>> >>>> To this end, of really encouraging an ecosystem of packages, I’d like<br>
>> >>>> to propose two things, in increasing order of disruptiveness.<br>
>> >>>><br>
>> >>>> First: Encourage extension packages. This would mean:<br>
>> >>>> * Reorganize website to allow for extension packages to be displayed<br>
>> >>>> prominently<br>
>> >>>> * Add support for name-space packages in yt<br>
>> >>>> * (possible) split out some packages from analysis_modules, including<br>
>> >>>> halo finding<br>
>> >>>> * Codify process of extension package creation, including how to have<br>
>> >>>> CI set up for them and build system.<br>
>> >>>><br>
>> >>>> The second, more disruptive proposal:<br>
>> >>>> * Split yt into subprojects. This would include spinning out the<br>
>> >>>> volume rendering and some or all of the frontends, and probably the<br>
>> >>>> testing infrastructure as well.<br>
>> >>>> * Split further astro-specific routines into an astro extension, and<br>
>> >>>> begin the process of doing this with other domains as well. (As in<br>
>> >>>> the long-simmering domain context YTEP.)<br>
>> >>>><br>
>> >>>> I’ll invite comments from everyone, but particularly from folks who<br>
>> >>>> have either not contributed to an analysis module or extension<br>
>> >>>> package<br>
>> >>>> because of concerns that would be addressed by this, as well as from<br>
>> >>>> core developers this would impact. If the thread gets too unweildy<br>
>> >>>> we<br>
>> >>>> may also want to table this for the next yt team meeting.<br>
>> >>>><br>
>> >>>> Thanks,<br>
>> >>>><br>
>> >>>> Matt<br>
>> >>>> _______________________________________________<br>
>> >>>> yt-dev mailing list<br>
>> >>>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> >>>> <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.spacepope.org_listinfo.cgi_yt-2Ddev-2Dspacepope.org&d=BQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=hgcBC3x6dKFoTrmFmMYYbKNfiHZlGLKliIidd1LwmHI&m=1HwdG4qc9p2mvxARPwDgSeThRIrQZCb68UKjDS5-pmk&s=UgYTeWCyf2INNcWK0wtNFtNJV4nhVAHWXwRIEPbooVk&e=" rel="noreferrer" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> >>><br>
>> >>><br>
>> >>><br>
>> >>> _______________________________________________<br>
>> >>> yt-dev mailing list<br>
>> >>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> >>> <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.spacepope.org_listinfo.cgi_yt-2Ddev-2Dspacepope.org&d=BQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=hgcBC3x6dKFoTrmFmMYYbKNfiHZlGLKliIidd1LwmHI&m=1HwdG4qc9p2mvxARPwDgSeThRIrQZCb68UKjDS5-pmk&s=UgYTeWCyf2INNcWK0wtNFtNJV4nhVAHWXwRIEPbooVk&e=" rel="noreferrer" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> >>><br>
>> >> _______________________________________________<br>
>> >> yt-dev mailing list<br>
>> >> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> >> <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.spacepope.org_listinfo.cgi_yt-2Ddev-2Dspacepope.org&d=BQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=hgcBC3x6dKFoTrmFmMYYbKNfiHZlGLKliIidd1LwmHI&m=1HwdG4qc9p2mvxARPwDgSeThRIrQZCb68UKjDS5-pmk&s=UgYTeWCyf2INNcWK0wtNFtNJV4nhVAHWXwRIEPbooVk&e=" rel="noreferrer" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> ><br>
>> > _______________________________________________<br>
>> > yt-dev mailing list<br>
>> > <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> > <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.spacepope.org_listinfo.cgi_yt-2Ddev-2Dspacepope.org&d=BQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=hgcBC3x6dKFoTrmFmMYYbKNfiHZlGLKliIidd1LwmHI&m=1HwdG4qc9p2mvxARPwDgSeThRIrQZCb68UKjDS5-pmk&s=UgYTeWCyf2INNcWK0wtNFtNJV4nhVAHWXwRIEPbooVk&e=" rel="noreferrer" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> _______________________________________________<br>
>> yt-dev mailing list<br>
>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.spacepope.org_listinfo.cgi_yt-2Ddev-2Dspacepope.org&d=BQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=hgcBC3x6dKFoTrmFmMYYbKNfiHZlGLKliIidd1LwmHI&m=1HwdG4qc9p2mvxARPwDgSeThRIrQZCb68UKjDS5-pmk&s=UgYTeWCyf2INNcWK0wtNFtNJV4nhVAHWXwRIEPbooVk&e=" rel="noreferrer" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
><br>
><br>
><br>
> _______________________________________________<br>
> yt-dev mailing list<br>
> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
> <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.spacepope.org_listinfo.cgi_yt-2Ddev-2Dspacepope.org&d=BQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=hgcBC3x6dKFoTrmFmMYYbKNfiHZlGLKliIidd1LwmHI&m=1HwdG4qc9p2mvxARPwDgSeThRIrQZCb68UKjDS5-pmk&s=UgYTeWCyf2INNcWK0wtNFtNJV4nhVAHWXwRIEPbooVk&e=" rel="noreferrer" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
><br>
_______________________________________________<br>
yt-dev mailing list<br>
<a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.spacepope.org_listinfo.cgi_yt-2Ddev-2Dspacepope.org&d=BQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=hgcBC3x6dKFoTrmFmMYYbKNfiHZlGLKliIidd1LwmHI&m=1HwdG4qc9p2mvxARPwDgSeThRIrQZCb68UKjDS5-pmk&s=UgYTeWCyf2INNcWK0wtNFtNJV4nhVAHWXwRIEPbooVk&e=" rel="noreferrer" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div>Michael Zingale</div><div>Associate Professor</div><div><br></div><div>Dept. of Physics & Astronomy • Stony Brook University • Stony Brook, NY 11794-3800</div><div><i>phone</i>: 631-632-8225</div><div><i>e-mail</i>: <a href="mailto:Michael.Zingale@stonybrook.edu" target="_blank">Michael.Zingale@stonybrook.edu</a></div><div><i>web</i>: <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.astro.sunysb.edu_mzingale&d=BQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=hgcBC3x6dKFoTrmFmMYYbKNfiHZlGLKliIidd1LwmHI&m=1HwdG4qc9p2mvxARPwDgSeThRIrQZCb68UKjDS5-pmk&s=cog0M67VU0tisLCreMqmtIM_o0tcaPYEz5UUr9_8WzE&e=" target="_blank">http://www.astro.sunysb.edu/mzingale</a></div><div>github: <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__github.com_zingale&d=BQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=hgcBC3x6dKFoTrmFmMYYbKNfiHZlGLKliIidd1LwmHI&m=1HwdG4qc9p2mvxARPwDgSeThRIrQZCb68UKjDS5-pmk&s=tSCYpkVhwJGc0dOVIfFrPUIBwB7lXZbcpTaO9ak5WNU&e=" target="_blank">http://github.com/zingale</a></div><div><br></div></div></div></div>
</div>