[yt-dev] The future of yt

[John ZuHone]

Matt can correct me if I am wrong, but I was under the impression that the two components of the proposal were separate and not inevitably tied to one another. 

I am in strong support of the idea of extensions, but I strongly disagree with most of the second component of this proposal. I believe it is far too disruptive, and I honestly don’t see the argument for it (except maybe splitting some astro-specific stuff out into an extension). But spinning off stuff like volume rendering and other big components like that seems unnecessary.

> On Jan 18, 2016, at 12:26 PM, Matthew Turk <matthewturk at gmail.com> wrote:
> 
> Hi Nathan,
> 
> Just to quickly address one component before anybody else replies: I
> would volunteer to do the majority of this work (which I think from a
> technical perspective is not necessarily too bad), and I don't see it
> being a 3.X release, but maybe a 4.0 release, and I don't have answers
> for when it would happen since I think that is a community decision we
> shouldn't rush into.
> 
> On Mon, Jan 18, 2016 at 11:21 AM, Nathan Goldbaum <nathan12343 at gmail.com> wrote:
>> Just a few thoughts about this.
>> 
>> I think encouraging more external packages to depend on yt will be great,
>> and will keep us more honest about the backward compatibility guarantees
>> we've committed ourselves to. Building social capital by encouraging an
>> "affiliated" package ecosystem like astropy has is also good, both for
>> social capital and also by enabling cool new features for our users.
>> 
>> That said, I'm skeptical about some of the technical components of Matt's
>> proposal. In particular, I'm concerned with the amount of work required to
>> make major changes to the core yt codebase, while still maintaining backward
>> compatibility. Who will do this work? When will it happen? Would this
>> constitute a yt 4.0 release?
>> 
>> While I do think making the core yt package less astronomy focused is good,
>> I also think it's possible to do so while maintaining strong backward
>> compatibility guarantees, or at least going through a deprecation cycle over
>> the course of several minor versions. I don't think making another cycle of
>> breaking changes so soon after yt-3.0 is a good idea, as that will impose a
>> large amount of work both on the yt developers and on our users.
>> 
>> On Mon, Jan 18, 2016 at 9:43 AM, Matthew Turk <matthewturk at gmail.com> wrote:
>>> 
>>> Hi everyone,
>>> 
>>> The last couple weeks I’ve been thinking a lot about the future of yt.
>>> What I’d like to propose is that we shift investment in yt as a
>>> single, monolithic codebase into yt as a project, or an ecosystem of
>>> projects.
>>> 
>>> This came out of the discussion of the extension/affiliated packages,
>>> analysis modules, and so on.  Britton has for a while been pitching
>>> the idea [which I will poorly paraphrase here] that yt can be the
>>> framework on top of which killer apps can be built.  I think this is
>>> great.
>>> 
>>> What’s holding us back in some ways from this is that yt is currently
>>> structured as a monolithic code base, with little to no discovery of
>>> other packages and apps and whatnot.  We tried for a while to change
>>> this with The Barn, but it ended up not quite taking off.  I think the
>>> time is right to try to change the way we think about yt to be more
>>> about yt the Project, rather than yt the Codebase; the core codebase
>>> is an important component of this, but not the whole of it.
>>> 
>>> Encouraging an ecosystem of packages can have a few very important
>>> benefits:
>>> 
>>> * External packages will confer greater individual credit to the
>>> folks who develop them.
>>> * External packages can be versioned and developed independently; the
>>> review process can be different.
>>> * yt’s core can be emphasized as a generic package, on top of which
>>> astronomy analysis can be built.
>>> * Packages can be maintained wherever, including alternate locations
>>> such as github.
>>> 
>>> On the other hand, having packages inside the main distribution makes
>>> discoverability much, much easier.  It also enables everything to be
>>> In The Box.  And, the continuous integration and testing system is
>>> already set up for yt.  But, these are all possible to overcome -- we
>>> can devise a strategy for adding packages to the CI system (and if
>>> they are externally managed, they can also rely on yt as a dependency
>>> and use whatever CI system they like!) and we can improve
>>> discoverability by refocusing the website to enable this.  I've asked
>>> Kacper about adding new packages, and it's not as easy as it might
>>> seem, so we may need to be careful about how that process occurs; one
>>> possibility would be to provide servers and ready-made setups, but
>>> have individuals do the heavy lifting.  We could even have something
>>> in the codebase that describes some packages that are available.
>>> External packages could have much looser dependency rules, which means
>>> they can be free to take advantage of things like OpenCL, numba, etc,
>>> without having to add them to the primary codebase.
>>> 
>>> Synchronizing APIs and versions across extension packages may be
>>> difficult in some particular cases, but I suspect in practice will not
>>> be an issue, as long as we continue to have a reasonably stable
>>> *public* API, and graduate a few things (such as .blocks) into a
>>> public API from semi-private.
>>> 
>>> To this end, of really encouraging an ecosystem of packages, I’d like
>>> to propose two things, in increasing order of disruptiveness.
>>> 
>>> First: Encourage extension packages.  This would mean:
>>> * Reorganize website to allow for extension packages to be displayed
>>> prominently
>>> * Add support for name-space packages in yt
>>> * (possible) split out some packages from analysis_modules, including
>>> halo finding
>>> * Codify process of extension package creation, including how to have
>>> CI set up for them and build system.
>>> 
>>> The second, more disruptive proposal:
>>> * Split yt into subprojects.  This would include spinning out the
>>> volume rendering and some or all of the frontends, and probably the
>>> testing infrastructure as well.
>>> * Split further astro-specific routines into an astro extension, and
>>> begin the process of doing this with other domains as well.  (As in
>>> the long-simmering domain context YTEP.)
>>> 
>>> I’ll invite comments from everyone, but particularly from folks who
>>> have either not contributed to an analysis module or extension package
>>> because of concerns that would be addressed by this, as well as from
>>> core developers this would impact.  If the thread gets too unweildy we
>>> may also want to table this for the next yt team meeting.
>>> 
>>> Thanks,
>>> 
>>> Matt
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> 
>> 
>> 
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> 
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

_______________________________________________
yt-dev mailing list
yt-dev at lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org