[yt-dev] The future of yt

Tue Jan 19 07:20:27 PST 2016

Hi all,

Thanks for all your feedback.  I think it seems like the proposal #1
is a good way to go.  I'm going to try to start writing up a YTEP that
formalizes this, and then we can put that up for a vote -- let's not
consider anything "settled" yet, but it seems pretty reasonable to
move forward with a more detailed proposal.  If anyone else has
comments, please feel free to leave them here, or on the YTEP.

-Matt

On Tue, Jan 19, 2016 at 7:24 AM, Britton Smith <brittonsmith at gmail.com> wrote:
> I guess this is clear from the original email, but I very much like the idea
> of an ecosystem of a core yt package and separate extension packages.
> Personally, I would stop short of making frontends into external packages.
> I think one of the big values of yt is the library of frontends we maintain
> and the ability of people to install the source and immediately be able to
> load their data without further requirements.  I think we run the risk of
> losing new users who have to go through too many steps just to get their
> data loaded.
>
> Perhaps the first thing should be to start up the domain context efforts
> again.  By some later 3.x release, we could have an astro domain with domain
> specific fields and extensions all still within the main yt codebase.  Then,
> moving to 4.0, we start to extract things into separate packages.
>
> On Mon, Jan 18, 2016 at 6:16 PM, Michael Zingale
> <michael.zingale at stonybrook.edu> wrote:
>>
>> Affiliated packages sound like a good idea to me, but I am a bit hesitant
>> about splitting the volume renderer and frontends out of yt's core.
>>
>> On Mon, Jan 18, 2016 at 1:11 PM, Matthew Turk <matthewturk at gmail.com>
>> wrote:
>>>
>>> Hi Desika,
>>>
>>> Yup, that's what I meant by it.  And there are things that would
>>> normally qualify as that type of extension, too, that are already in
>>> the codebase.
>>>
>>> Flask's extensions are another good example of how this type of thing is
>>> done:
>>>
>>> http://flask.pocoo.org/extensions/
>>> http://flask.pocoo.org/docs/0.10/extensiondev/
>>>
>>> -Matt
>>>
>>> On Mon, Jan 18, 2016 at 12:02 PM, Desika Narayanan
>>> <desika.narayanan at gmail.com> wrote:
>>> > Hey All,
>>> >
>>> > Matt - just to maybe clarify by examples...are you thinking things like
>>> > Trident or powderday are sorts of yt extensions that you mean?  (If so,
>>> > then
>>> > I do like this idea a lot too).  I think one might imagine having a
>>> > clear
>>> > set of guidelines for developers of extensions to aspire toward (even
>>> > if
>>> > it's for example, having some basic set of or style of documentation)
>>> > so
>>> > that from the user point, when looking at extensions  on the yt docs,
>>> > there's some modicum of uniformity.  That might make it easier for
>>> > users to
>>> > browse and understand the extensions.
>>> >
>>> > -d
>>> >
>>> > On Mon, Jan 18, 2016 at 12:43 PM, Matthew Turk <matthewturk at gmail.com>
>>> > wrote:
>>> >>
>>> >> On Mon, Jan 18, 2016 at 11:32 AM, John ZuHone <jzuhone at gmail.com>
>>> >> wrote:
>>> >> > Matt can correct me if I am wrong, but I was under the impression
>>> >> > that
>>> >> > the two components of the proposal were separate and not inevitably
>>> >> > tied to
>>> >> > one another.
>>> >> >
>>> >> > I am in strong support of the idea of extensions, but I strongly
>>> >> > disagree with most of the second component of this proposal. I
>>> >> > believe it is
>>> >> > far too disruptive, and I honestly don’t see the argument for it
>>> >> > (except
>>> >> > maybe splitting some astro-specific stuff out into an extension).
>>> >> > But
>>> >> > spinning off stuff like volume rendering and other big components
>>> >> > like that
>>> >> > seems unnecessary.
>>> >> >
>>> >>
>>> >> No need to correct; the second one is indeed an extension of the
>>> >> first, and a separate proposal.
>>> >>
>>> >> >> On Jan 18, 2016, at 12:26 PM, Matthew Turk <matthewturk at gmail.com>
>>> >> >> wrote:
>>> >> >>
>>> >> >> Hi Nathan,
>>> >> >>
>>> >> >> Just to quickly address one component before anybody else replies:
>>> >> >> I
>>> >> >> would volunteer to do the majority of this work (which I think from
>>> >> >> a
>>> >> >> technical perspective is not necessarily too bad), and I don't see
>>> >> >> it
>>> >> >> being a 3.X release, but maybe a 4.0 release, and I don't have
>>> >> >> answers
>>> >> >> for when it would happen since I think that is a community decision
>>> >> >> we
>>> >> >> shouldn't rush into.
>>> >> >>
>>> >> >> On Mon, Jan 18, 2016 at 11:21 AM, Nathan Goldbaum
>>> >> >> <nathan12343 at gmail.com> wrote:
>>> >> >>> Just a few thoughts about this.
>>> >> >>>
>>> >> >>> I think encouraging more external packages to depend on yt will be
>>> >> >>> great,
>>> >> >>> and will keep us more honest about the backward compatibility
>>> >> >>> guarantees
>>> >> >>> we've committed ourselves to. Building social capital by
>>> >> >>> encouraging
>>> >> >>> an
>>> >> >>> "affiliated" package ecosystem like astropy has is also good, both
>>> >> >>> for
>>> >> >>> social capital and also by enabling cool new features for our
>>> >> >>> users.
>>> >> >>>
>>> >> >>> That said, I'm skeptical about some of the technical components of
>>> >> >>> Matt's
>>> >> >>> proposal. In particular, I'm concerned with the amount of work
>>> >> >>> required to
>>> >> >>> make major changes to the core yt codebase, while still
>>> >> >>> maintaining
>>> >> >>> backward
>>> >> >>> compatibility. Who will do this work? When will it happen? Would
>>> >> >>> this
>>> >> >>> constitute a yt 4.0 release?
>>> >> >>>
>>> >> >>> While I do think making the core yt package less astronomy focused
>>> >> >>> is
>>> >> >>> good,
>>> >> >>> I also think it's possible to do so while maintaining strong
>>> >> >>> backward
>>> >> >>> compatibility guarantees, or at least going through a deprecation
>>> >> >>> cycle over
>>> >> >>> the course of several minor versions. I don't think making another
>>> >> >>> cycle of
>>> >> >>> breaking changes so soon after yt-3.0 is a good idea, as that will
>>> >> >>> impose a
>>> >> >>> large amount of work both on the yt developers and on our users.
>>> >> >>>
>>> >> >>> On Mon, Jan 18, 2016 at 9:43 AM, Matthew Turk
>>> >> >>> <matthewturk at gmail.com>
>>> >> >>> wrote:
>>> >> >>>>
>>> >> >>>> Hi everyone,
>>> >> >>>>
>>> >> >>>> The last couple weeks I’ve been thinking a lot about the future
>>> >> >>>> of
>>> >> >>>> yt.
>>> >> >>>> What I’d like to propose is that we shift investment in yt as a
>>> >> >>>> single, monolithic codebase into yt as a project, or an ecosystem
>>> >> >>>> of
>>> >> >>>> projects.
>>> >> >>>>
>>> >> >>>> This came out of the discussion of the extension/affiliated
>>> >> >>>> packages,
>>> >> >>>> analysis modules, and so on.  Britton has for a while been
>>> >> >>>> pitching
>>> >> >>>> the idea [which I will poorly paraphrase here] that yt can be the
>>> >> >>>> framework on top of which killer apps can be built.  I think this
>>> >> >>>> is
>>> >> >>>> great.
>>> >> >>>>
>>> >> >>>> What’s holding us back in some ways from this is that yt is
>>> >> >>>> currently
>>> >> >>>> structured as a monolithic code base, with little to no discovery
>>> >> >>>> of
>>> >> >>>> other packages and apps and whatnot.  We tried for a while to
>>> >> >>>> change
>>> >> >>>> this with The Barn, but it ended up not quite taking off.  I
>>> >> >>>> think
>>> >> >>>> the
>>> >> >>>> time is right to try to change the way we think about yt to be
>>> >> >>>> more
>>> >> >>>> about yt the Project, rather than yt the Codebase; the core
>>> >> >>>> codebase
>>> >> >>>> is an important component of this, but not the whole of it.
>>> >> >>>>
>>> >> >>>> Encouraging an ecosystem of packages can have a few very
>>> >> >>>> important
>>> >> >>>> benefits:
>>> >> >>>>
>>> >> >>>> * External packages will confer greater individual credit to the
>>> >> >>>> folks who develop them.
>>> >> >>>> * External packages can be versioned and developed independently;
>>> >> >>>> the
>>> >> >>>> review process can be different.
>>> >> >>>> * yt’s core can be emphasized as a generic package, on top of
>>> >> >>>> which
>>> >> >>>> astronomy analysis can be built.
>>> >> >>>> * Packages can be maintained wherever, including alternate
>>> >> >>>> locations
>>> >> >>>> such as github.
>>> >> >>>>
>>> >> >>>> On the other hand, having packages inside the main distribution
>>> >> >>>> makes
>>> >> >>>> discoverability much, much easier.  It also enables everything to
>>> >> >>>> be
>>> >> >>>> In The Box.  And, the continuous integration and testing system
>>> >> >>>> is
>>> >> >>>> already set up for yt.  But, these are all possible to overcome
>>> >> >>>> -- we
>>> >> >>>> can devise a strategy for adding packages to the CI system (and
>>> >> >>>> if
>>> >> >>>> they are externally managed, they can also rely on yt as a
>>> >> >>>> dependency
>>> >> >>>> and use whatever CI system they like!) and we can improve
>>> >> >>>> discoverability by refocusing the website to enable this.  I've
>>> >> >>>> asked
>>> >> >>>> Kacper about adding new packages, and it's not as easy as it
>>> >> >>>> might
>>> >> >>>> seem, so we may need to be careful about how that process occurs;
>>> >> >>>> one
>>> >> >>>> possibility would be to provide servers and ready-made setups,
>>> >> >>>> but
>>> >> >>>> have individuals do the heavy lifting.  We could even have
>>> >> >>>> something
>>> >> >>>> in the codebase that describes some packages that are available.
>>> >> >>>> External packages could have much looser dependency rules, which
>>> >> >>>> means
>>> >> >>>> they can be free to take advantage of things like OpenCL, numba,
>>> >> >>>> etc,
>>> >> >>>> without having to add them to the primary codebase.
>>> >> >>>>
>>> >> >>>> Synchronizing APIs and versions across extension packages may be
>>> >> >>>> difficult in some particular cases, but I suspect in practice
>>> >> >>>> will
>>> >> >>>> not
>>> >> >>>> be an issue, as long as we continue to have a reasonably stable
>>> >> >>>> *public* API, and graduate a few things (such as .blocks) into a
>>> >> >>>> public API from semi-private.
>>> >> >>>>
>>> >> >>>> To this end, of really encouraging an ecosystem of packages, I’d
>>> >> >>>> like
>>> >> >>>> to propose two things, in increasing order of disruptiveness.
>>> >> >>>>
>>> >> >>>> First: Encourage extension packages.  This would mean:
>>> >> >>>> * Reorganize website to allow for extension packages to be
>>> >> >>>> displayed
>>> >> >>>> prominently
>>> >> >>>> * Add support for name-space packages in yt
>>> >> >>>> * (possible) split out some packages from analysis_modules,
>>> >> >>>> including
>>> >> >>>> halo finding
>>> >> >>>> * Codify process of extension package creation, including how to
>>> >> >>>> have
>>> >> >>>> CI set up for them and build system.
>>> >> >>>>
>>> >> >>>> The second, more disruptive proposal:
>>> >> >>>> * Split yt into subprojects.  This would include spinning out the
>>> >> >>>> volume rendering and some or all of the frontends, and probably
>>> >> >>>> the
>>> >> >>>> testing infrastructure as well.
>>> >> >>>> * Split further astro-specific routines into an astro extension,
>>> >> >>>> and
>>> >> >>>> begin the process of doing this with other domains as well.  (As
>>> >> >>>> in
>>> >> >>>> the long-simmering domain context YTEP.)
>>> >> >>>>
>>> >> >>>> I’ll invite comments from everyone, but particularly from folks
>>> >> >>>> who
>>> >> >>>> have either not contributed to an analysis module or extension
>>> >> >>>> package
>>> >> >>>> because of concerns that would be addressed by this, as well as
>>> >> >>>> from
>>> >> >>>> core developers this would impact.  If the thread gets too
>>> >> >>>> unweildy
>>> >> >>>> we
>>> >> >>>> may also want to table this for the next yt team meeting.
>>> >> >>>>
>>> >> >>>> Thanks,
>>> >> >>>>
>>> >> >>>> Matt
>>> >> >>>> _______________________________________________
>>> >> >>>> yt-dev mailing list
>>> >> >>>> yt-dev at lists.spacepope.org
>>> >> >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> >> >>>
>>> >> >>>
>>> >> >>>
>>> >> >>> _______________________________________________
>>> >> >>> yt-dev mailing list
>>> >> >>> yt-dev at lists.spacepope.org
>>> >> >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> >> >>>
>>> >> >> _______________________________________________
>>> >> >> yt-dev mailing list
>>> >> >> yt-dev at lists.spacepope.org
>>> >> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> >> >
>>> >> > _______________________________________________
>>> >> > yt-dev mailing list
>>> >> > yt-dev at lists.spacepope.org
>>> >> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> >> _______________________________________________
>>> >> yt-dev mailing list
>>> >> yt-dev at lists.spacepope.org
>>> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > yt-dev mailing list
>>> > yt-dev at lists.spacepope.org
>>> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> >
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>>
>>
>>
>> --
>> Michael Zingale
>> Associate Professor
>>
>> Dept. of Physics & Astronomy • Stony Brook University • Stony Brook, NY
>> 11794-3800
>> phone:  631-632-8225
>> e-mail: Michael.Zingale at stonybrook.edu
>> web: http://www.astro.sunysb.edu/mzingale
>> github: http://github.com/zingale
>>
>>
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
_______________________________________________
yt-dev mailing list
yt-dev at lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org