[yt-dev] The future of yt

[Matthew Turk]

Hi Nathan,

Just to quickly address one component before anybody else replies: I
would volunteer to do the majority of this work (which I think from a
technical perspective is not necessarily too bad), and I don't see it
being a 3.X release, but maybe a 4.0 release, and I don't have answers
for when it would happen since I think that is a community decision we
shouldn't rush into.

On Mon, Jan 18, 2016 at 11:21 AM, Nathan Goldbaum <nathan12343 at gmail.com> wrote:
> Just a few thoughts about this.
>
> I think encouraging more external packages to depend on yt will be great,
> and will keep us more honest about the backward compatibility guarantees
> we've committed ourselves to. Building social capital by encouraging an
> "affiliated" package ecosystem like astropy has is also good, both for
> social capital and also by enabling cool new features for our users.
>
> That said, I'm skeptical about some of the technical components of Matt's
> proposal. In particular, I'm concerned with the amount of work required to
> make major changes to the core yt codebase, while still maintaining backward
> compatibility. Who will do this work? When will it happen? Would this
> constitute a yt 4.0 release?
>
> While I do think making the core yt package less astronomy focused is good,
> I also think it's possible to do so while maintaining strong backward
> compatibility guarantees, or at least going through a deprecation cycle over
> the course of several minor versions. I don't think making another cycle of
> breaking changes so soon after yt-3.0 is a good idea, as that will impose a
> large amount of work both on the yt developers and on our users.
>
> On Mon, Jan 18, 2016 at 9:43 AM, Matthew Turk <matthewturk at gmail.com> wrote:
>>
>> Hi everyone,
>>
>> The last couple weeks I’ve been thinking a lot about the future of yt.
>> What I’d like to propose is that we shift investment in yt as a
>> single, monolithic codebase into yt as a project, or an ecosystem of
>> projects.
>>
>> This came out of the discussion of the extension/affiliated packages,
>> analysis modules, and so on.  Britton has for a while been pitching
>> the idea [which I will poorly paraphrase here] that yt can be the
>> framework on top of which killer apps can be built.  I think this is
>> great.
>>
>> What’s holding us back in some ways from this is that yt is currently
>> structured as a monolithic code base, with little to no discovery of
>> other packages and apps and whatnot.  We tried for a while to change
>> this with The Barn, but it ended up not quite taking off.  I think the
>> time is right to try to change the way we think about yt to be more
>> about yt the Project, rather than yt the Codebase; the core codebase
>> is an important component of this, but not the whole of it.
>>
>> Encouraging an ecosystem of packages can have a few very important
>> benefits:
>>
>>  * External packages will confer greater individual credit to the
>> folks who develop them.
>>  * External packages can be versioned and developed independently; the
>> review process can be different.
>>  * yt’s core can be emphasized as a generic package, on top of which
>> astronomy analysis can be built.
>>  * Packages can be maintained wherever, including alternate locations
>> such as github.
>>
>> On the other hand, having packages inside the main distribution makes
>> discoverability much, much easier.  It also enables everything to be
>> In The Box.  And, the continuous integration and testing system is
>> already set up for yt.  But, these are all possible to overcome -- we
>> can devise a strategy for adding packages to the CI system (and if
>> they are externally managed, they can also rely on yt as a dependency
>> and use whatever CI system they like!) and we can improve
>> discoverability by refocusing the website to enable this.  I've asked
>> Kacper about adding new packages, and it's not as easy as it might
>> seem, so we may need to be careful about how that process occurs; one
>> possibility would be to provide servers and ready-made setups, but
>> have individuals do the heavy lifting.  We could even have something
>> in the codebase that describes some packages that are available.
>> External packages could have much looser dependency rules, which means
>> they can be free to take advantage of things like OpenCL, numba, etc,
>> without having to add them to the primary codebase.
>>
>> Synchronizing APIs and versions across extension packages may be
>> difficult in some particular cases, but I suspect in practice will not
>> be an issue, as long as we continue to have a reasonably stable
>> *public* API, and graduate a few things (such as .blocks) into a
>> public API from semi-private.
>>
>> To this end, of really encouraging an ecosystem of packages, I’d like
>> to propose two things, in increasing order of disruptiveness.
>>
>> First: Encourage extension packages.  This would mean:
>>  * Reorganize website to allow for extension packages to be displayed
>> prominently
>>  * Add support for name-space packages in yt
>>  * (possible) split out some packages from analysis_modules, including
>> halo finding
>>  * Codify process of extension package creation, including how to have
>> CI set up for them and build system.
>>
>> The second, more disruptive proposal:
>>  * Split yt into subprojects.  This would include spinning out the
>> volume rendering and some or all of the frontends, and probably the
>> testing infrastructure as well.
>>  * Split further astro-specific routines into an astro extension, and
>> begin the process of doing this with other domains as well.  (As in
>> the long-simmering domain context YTEP.)
>>
>> I’ll invite comments from everyone, but particularly from folks who
>> have either not contributed to an analysis module or extension package
>> because of concerns that would be addressed by this, as well as from
>> core developers this would impact.  If the thread gets too unweildy we
>> may also want to table this for the next yt team meeting.
>>
>> Thanks,
>>
>> Matt
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
_______________________________________________
yt-dev mailing list
yt-dev at lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org