[yt-dev] The future of yt

Mon Jan 18 09:43:45 PST 2016

On Mon, Jan 18, 2016 at 11:32 AM, John ZuHone <jzuhone at gmail.com> wrote:
> Matt can correct me if I am wrong, but I was under the impression that the two components of the proposal were separate and not inevitably tied to one another.
>
> I am in strong support of the idea of extensions, but I strongly disagree with most of the second component of this proposal. I believe it is far too disruptive, and I honestly don’t see the argument for it (except maybe splitting some astro-specific stuff out into an extension). But spinning off stuff like volume rendering and other big components like that seems unnecessary.
>

No need to correct; the second one is indeed an extension of the
first, and a separate proposal.

>> On Jan 18, 2016, at 12:26 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>>
>> Hi Nathan,
>>
>> Just to quickly address one component before anybody else replies: I
>> would volunteer to do the majority of this work (which I think from a
>> technical perspective is not necessarily too bad), and I don't see it
>> being a 3.X release, but maybe a 4.0 release, and I don't have answers
>> for when it would happen since I think that is a community decision we
>> shouldn't rush into.
>>
>> On Mon, Jan 18, 2016 at 11:21 AM, Nathan Goldbaum <nathan12343 at gmail.com> wrote:
>>> Just a few thoughts about this.
>>>
>>> I think encouraging more external packages to depend on yt will be great,
>>> and will keep us more honest about the backward compatibility guarantees
>>> we've committed ourselves to. Building social capital by encouraging an
>>> "affiliated" package ecosystem like astropy has is also good, both for
>>> social capital and also by enabling cool new features for our users.
>>>
>>> That said, I'm skeptical about some of the technical components of Matt's
>>> proposal. In particular, I'm concerned with the amount of work required to
>>> make major changes to the core yt codebase, while still maintaining backward
>>> compatibility. Who will do this work? When will it happen? Would this
>>> constitute a yt 4.0 release?
>>>
>>> While I do think making the core yt package less astronomy focused is good,
>>> I also think it's possible to do so while maintaining strong backward
>>> compatibility guarantees, or at least going through a deprecation cycle over
>>> the course of several minor versions. I don't think making another cycle of
>>> breaking changes so soon after yt-3.0 is a good idea, as that will impose a
>>> large amount of work both on the yt developers and on our users.
>>>
>>> On Mon, Jan 18, 2016 at 9:43 AM, Matthew Turk <matthewturk at gmail.com> wrote:
>>>>
>>>> Hi everyone,
>>>>
>>>> The last couple weeks I’ve been thinking a lot about the future of yt.
>>>> What I’d like to propose is that we shift investment in yt as a
>>>> single, monolithic codebase into yt as a project, or an ecosystem of
>>>> projects.
>>>>
>>>> This came out of the discussion of the extension/affiliated packages,
>>>> analysis modules, and so on.  Britton has for a while been pitching
>>>> the idea [which I will poorly paraphrase here] that yt can be the
>>>> framework on top of which killer apps can be built.  I think this is
>>>> great.
>>>>
>>>> What’s holding us back in some ways from this is that yt is currently
>>>> structured as a monolithic code base, with little to no discovery of
>>>> other packages and apps and whatnot.  We tried for a while to change
>>>> this with The Barn, but it ended up not quite taking off.  I think the
>>>> time is right to try to change the way we think about yt to be more
>>>> about yt the Project, rather than yt the Codebase; the core codebase
>>>> is an important component of this, but not the whole of it.
>>>>
>>>> Encouraging an ecosystem of packages can have a few very important
>>>> benefits:
>>>>
>>>> * External packages will confer greater individual credit to the
>>>> folks who develop them.
>>>> * External packages can be versioned and developed independently; the
>>>> review process can be different.
>>>> * yt’s core can be emphasized as a generic package, on top of which
>>>> astronomy analysis can be built.
>>>> * Packages can be maintained wherever, including alternate locations
>>>> such as github.
>>>>
>>>> On the other hand, having packages inside the main distribution makes
>>>> discoverability much, much easier.  It also enables everything to be
>>>> In The Box.  And, the continuous integration and testing system is
>>>> already set up for yt.  But, these are all possible to overcome -- we
>>>> can devise a strategy for adding packages to the CI system (and if
>>>> they are externally managed, they can also rely on yt as a dependency
>>>> and use whatever CI system they like!) and we can improve
>>>> discoverability by refocusing the website to enable this.  I've asked
>>>> Kacper about adding new packages, and it's not as easy as it might
>>>> seem, so we may need to be careful about how that process occurs; one
>>>> possibility would be to provide servers and ready-made setups, but
>>>> have individuals do the heavy lifting.  We could even have something
>>>> in the codebase that describes some packages that are available.
>>>> External packages could have much looser dependency rules, which means
>>>> they can be free to take advantage of things like OpenCL, numba, etc,
>>>> without having to add them to the primary codebase.
>>>>
>>>> Synchronizing APIs and versions across extension packages may be
>>>> difficult in some particular cases, but I suspect in practice will not
>>>> be an issue, as long as we continue to have a reasonably stable
>>>> *public* API, and graduate a few things (such as .blocks) into a
>>>> public API from semi-private.
>>>>
>>>> To this end, of really encouraging an ecosystem of packages, I’d like
>>>> to propose two things, in increasing order of disruptiveness.
>>>>
>>>> First: Encourage extension packages.  This would mean:
>>>> * Reorganize website to allow for extension packages to be displayed
>>>> prominently
>>>> * Add support for name-space packages in yt
>>>> * (possible) split out some packages from analysis_modules, including
>>>> halo finding
>>>> * Codify process of extension package creation, including how to have
>>>> CI set up for them and build system.
>>>>
>>>> The second, more disruptive proposal:
>>>> * Split yt into subprojects.  This would include spinning out the
>>>> volume rendering and some or all of the frontends, and probably the
>>>> testing infrastructure as well.
>>>> * Split further astro-specific routines into an astro extension, and
>>>> begin the process of doing this with other domains as well.  (As in
>>>> the long-simmering domain context YTEP.)
>>>>
>>>> I’ll invite comments from everyone, but particularly from folks who
>>>> have either not contributed to an analysis module or extension package
>>>> because of concerns that would be addressed by this, as well as from
>>>> core developers this would impact.  If the thread gets too unweildy we
>>>> may also want to table this for the next yt team meeting.
>>>>
>>>> Thanks,
>>>>
>>>> Matt
>>>> _______________________________________________
>>>> yt-dev mailing list
>>>> yt-dev at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>
>>>
>>>
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
_______________________________________________
yt-dev mailing list
yt-dev at lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org