[yt-dev] The future of yt

Matthew Turk matthewturk at gmail.com
Wed Jan 27 15:06:29 PST 2016 

Hi everyone,

I have submitted a draft YTEP here:

https://bitbucket.org/yt_analysis/ytep/pull-requests/60/ytep-0029-extension-packages/diff

Please feel free to leave short comments there, but if they are larger
comments, I think they should perhaps come here.  I didn't discuss
everything in it, but I think the biggest change is non-technical, and
instead that of a mindset.  Also, the website.

-Matt

On Tue, Jan 19, 2016 at 9:20 AM, Matthew Turk <matthewturk at gmail.com> wrote:

> Hi all,
>
> Thanks for all your feedback.  I think it seems like the proposal #1
> is a good way to go.  I'm going to try to start writing up a YTEP that
> formalizes this, and then we can put that up for a vote -- let's not
> consider anything "settled" yet, but it seems pretty reasonable to
> move forward with a more detailed proposal.  If anyone else has
> comments, please feel free to leave them here, or on the YTEP.
>
> -Matt
>
> On Tue, Jan 19, 2016 at 7:24 AM, Britton Smith <brittonsmith at gmail.com>
> wrote:
> > I guess this is clear from the original email, but I very much like the
> idea
> > of an ecosystem of a core yt package and separate extension packages.
> > Personally, I would stop short of making frontends into external
> packages.
> > I think one of the big values of yt is the library of frontends we
> maintain
> > and the ability of people to install the source and immediately be able
> to
> > load their data without further requirements.  I think we run the risk of
> > losing new users who have to go through too many steps just to get their
> > data loaded.
> >
> > Perhaps the first thing should be to start up the domain context efforts
> > again.  By some later 3.x release, we could have an astro domain with
> domain
> > specific fields and extensions all still within the main yt codebase.
> Then,
> > moving to 4.0, we start to extract things into separate packages.
> >
> > On Mon, Jan 18, 2016 at 6:16 PM, Michael Zingale
> > <michael.zingale at stonybrook.edu> wrote:
> >>
> >> Affiliated packages sound like a good idea to me, but I am a bit
> hesitant
> >> about splitting the volume renderer and frontends out of yt's core.
> >>
> >> On Mon, Jan 18, 2016 at 1:11 PM, Matthew Turk <matthewturk at gmail.com>
> >> wrote:
> >>>
> >>> Hi Desika,
> >>>
> >>> Yup, that's what I meant by it.  And there are things that would
> >>> normally qualify as that type of extension, too, that are already in
> >>> the codebase.
> >>>
> >>> Flask's extensions are another good example of how this type of thing
> is
> >>> done:
> >>>
> >>> http://flask.pocoo.org/extensions/
> >>> http://flask.pocoo.org/docs/0.10/extensiondev/
> >>>
> >>> -Matt
> >>>
> >>> On Mon, Jan 18, 2016 at 12:02 PM, Desika Narayanan
> >>> <desika.narayanan at gmail.com> wrote:
> >>> > Hey All,
> >>> >
> >>> > Matt - just to maybe clarify by examples...are you thinking things
> like
> >>> > Trident or powderday are sorts of yt extensions that you mean?  (If
> so,
> >>> > then
> >>> > I do like this idea a lot too).  I think one might imagine having a
> >>> > clear
> >>> > set of guidelines for developers of extensions to aspire toward (even
> >>> > if
> >>> > it's for example, having some basic set of or style of documentation)
> >>> > so
> >>> > that from the user point, when looking at extensions  on the yt docs,
> >>> > there's some modicum of uniformity.  That might make it easier for
> >>> > users to
> >>> > browse and understand the extensions.
> >>> >
> >>> > -d
> >>> >
> >>> > On Mon, Jan 18, 2016 at 12:43 PM, Matthew Turk <
> matthewturk at gmail.com>
> >>> > wrote:
> >>> >>
> >>> >> On Mon, Jan 18, 2016 at 11:32 AM, John ZuHone <jzuhone at gmail.com>
> >>> >> wrote:
> >>> >> > Matt can correct me if I am wrong, but I was under the impression
> >>> >> > that
> >>> >> > the two components of the proposal were separate and not
> inevitably
> >>> >> > tied to
> >>> >> > one another.
> >>> >> >
> >>> >> > I am in strong support of the idea of extensions, but I strongly
> >>> >> > disagree with most of the second component of this proposal. I
> >>> >> > believe it is
> >>> >> > far too disruptive, and I honestly don’t see the argument for it
> >>> >> > (except
> >>> >> > maybe splitting some astro-specific stuff out into an extension).
> >>> >> > But
> >>> >> > spinning off stuff like volume rendering and other big components
> >>> >> > like that
> >>> >> > seems unnecessary.
> >>> >> >
> >>> >>
> >>> >> No need to correct; the second one is indeed an extension of the
> >>> >> first, and a separate proposal.
> >>> >>
> >>> >> >> On Jan 18, 2016, at 12:26 PM, Matthew Turk <
> matthewturk at gmail.com>
> >>> >> >> wrote:
> >>> >> >>
> >>> >> >> Hi Nathan,
> >>> >> >>
> >>> >> >> Just to quickly address one component before anybody else
> replies:
> >>> >> >> I
> >>> >> >> would volunteer to do the majority of this work (which I think
> from
> >>> >> >> a
> >>> >> >> technical perspective is not necessarily too bad), and I don't
> see
> >>> >> >> it
> >>> >> >> being a 3.X release, but maybe a 4.0 release, and I don't have
> >>> >> >> answers
> >>> >> >> for when it would happen since I think that is a community
> decision
> >>> >> >> we
> >>> >> >> shouldn't rush into.
> >>> >> >>
> >>> >> >> On Mon, Jan 18, 2016 at 11:21 AM, Nathan Goldbaum
> >>> >> >> <nathan12343 at gmail.com> wrote:
> >>> >> >>> Just a few thoughts about this.
> >>> >> >>>
> >>> >> >>> I think encouraging more external packages to depend on yt will
> be
> >>> >> >>> great,
> >>> >> >>> and will keep us more honest about the backward compatibility
> >>> >> >>> guarantees
> >>> >> >>> we've committed ourselves to. Building social capital by
> >>> >> >>> encouraging
> >>> >> >>> an
> >>> >> >>> "affiliated" package ecosystem like astropy has is also good,
> both
> >>> >> >>> for
> >>> >> >>> social capital and also by enabling cool new features for our
> >>> >> >>> users.
> >>> >> >>>
> >>> >> >>> That said, I'm skeptical about some of the technical components
> of
> >>> >> >>> Matt's
> >>> >> >>> proposal. In particular, I'm concerned with the amount of work
> >>> >> >>> required to
> >>> >> >>> make major changes to the core yt codebase, while still
> >>> >> >>> maintaining
> >>> >> >>> backward
> >>> >> >>> compatibility. Who will do this work? When will it happen? Would
> >>> >> >>> this
> >>> >> >>> constitute a yt 4.0 release?
> >>> >> >>>
> >>> >> >>> While I do think making the core yt package less astronomy
> focused
> >>> >> >>> is
> >>> >> >>> good,
> >>> >> >>> I also think it's possible to do so while maintaining strong
> >>> >> >>> backward
> >>> >> >>> compatibility guarantees, or at least going through a
> deprecation
> >>> >> >>> cycle over
> >>> >> >>> the course of several minor versions. I don't think making
> another
> >>> >> >>> cycle of
> >>> >> >>> breaking changes so soon after yt-3.0 is a good idea, as that
> will
> >>> >> >>> impose a
> >>> >> >>> large amount of work both on the yt developers and on our users.
> >>> >> >>>
> >>> >> >>> On Mon, Jan 18, 2016 at 9:43 AM, Matthew Turk
> >>> >> >>> <matthewturk at gmail.com>
> >>> >> >>> wrote:
> >>> >> >>>>
> >>> >> >>>> Hi everyone,
> >>> >> >>>>
> >>> >> >>>> The last couple weeks I’ve been thinking a lot about the future
> >>> >> >>>> of
> >>> >> >>>> yt.
> >>> >> >>>> What I’d like to propose is that we shift investment in yt as a
> >>> >> >>>> single, monolithic codebase into yt as a project, or an
> ecosystem
> >>> >> >>>> of
> >>> >> >>>> projects.
> >>> >> >>>>
> >>> >> >>>> This came out of the discussion of the extension/affiliated
> >>> >> >>>> packages,
> >>> >> >>>> analysis modules, and so on.  Britton has for a while been
> >>> >> >>>> pitching
> >>> >> >>>> the idea [which I will poorly paraphrase here] that yt can be
> the
> >>> >> >>>> framework on top of which killer apps can be built.  I think
> this
> >>> >> >>>> is
> >>> >> >>>> great.
> >>> >> >>>>
> >>> >> >>>> What’s holding us back in some ways from this is that yt is
> >>> >> >>>> currently
> >>> >> >>>> structured as a monolithic code base, with little to no
> discovery
> >>> >> >>>> of
> >>> >> >>>> other packages and apps and whatnot.  We tried for a while to
> >>> >> >>>> change
> >>> >> >>>> this with The Barn, but it ended up not quite taking off.  I
> >>> >> >>>> think
> >>> >> >>>> the
> >>> >> >>>> time is right to try to change the way we think about yt to be
> >>> >> >>>> more
> >>> >> >>>> about yt the Project, rather than yt the Codebase; the core
> >>> >> >>>> codebase
> >>> >> >>>> is an important component of this, but not the whole of it.
> >>> >> >>>>
> >>> >> >>>> Encouraging an ecosystem of packages can have a few very
> >>> >> >>>> important
> >>> >> >>>> benefits:
> >>> >> >>>>
> >>> >> >>>> * External packages will confer greater individual credit to
> the
> >>> >> >>>> folks who develop them.
> >>> >> >>>> * External packages can be versioned and developed
> independently;
> >>> >> >>>> the
> >>> >> >>>> review process can be different.
> >>> >> >>>> * yt’s core can be emphasized as a generic package, on top of
> >>> >> >>>> which
> >>> >> >>>> astronomy analysis can be built.
> >>> >> >>>> * Packages can be maintained wherever, including alternate
> >>> >> >>>> locations
> >>> >> >>>> such as github.
> >>> >> >>>>
> >>> >> >>>> On the other hand, having packages inside the main distribution
> >>> >> >>>> makes
> >>> >> >>>> discoverability much, much easier.  It also enables everything
> to
> >>> >> >>>> be
> >>> >> >>>> In The Box.  And, the continuous integration and testing system
> >>> >> >>>> is
> >>> >> >>>> already set up for yt.  But, these are all possible to overcome
> >>> >> >>>> -- we
> >>> >> >>>> can devise a strategy for adding packages to the CI system (and
> >>> >> >>>> if
> >>> >> >>>> they are externally managed, they can also rely on yt as a
> >>> >> >>>> dependency
> >>> >> >>>> and use whatever CI system they like!) and we can improve
> >>> >> >>>> discoverability by refocusing the website to enable this.  I've
> >>> >> >>>> asked
> >>> >> >>>> Kacper about adding new packages, and it's not as easy as it
> >>> >> >>>> might
> >>> >> >>>> seem, so we may need to be careful about how that process
> occurs;
> >>> >> >>>> one
> >>> >> >>>> possibility would be to provide servers and ready-made setups,
> >>> >> >>>> but
> >>> >> >>>> have individuals do the heavy lifting.  We could even have
> >>> >> >>>> something
> >>> >> >>>> in the codebase that describes some packages that are
> available.
> >>> >> >>>> External packages could have much looser dependency rules,
> which
> >>> >> >>>> means
> >>> >> >>>> they can be free to take advantage of things like OpenCL,
> numba,
> >>> >> >>>> etc,
> >>> >> >>>> without having to add them to the primary codebase.
> >>> >> >>>>
> >>> >> >>>> Synchronizing APIs and versions across extension packages may
> be
> >>> >> >>>> difficult in some particular cases, but I suspect in practice
> >>> >> >>>> will
> >>> >> >>>> not
> >>> >> >>>> be an issue, as long as we continue to have a reasonably stable
> >>> >> >>>> *public* API, and graduate a few things (such as .blocks) into
> a
> >>> >> >>>> public API from semi-private.
> >>> >> >>>>
> >>> >> >>>> To this end, of really encouraging an ecosystem of packages,
> I’d
> >>> >> >>>> like
> >>> >> >>>> to propose two things, in increasing order of disruptiveness.
> >>> >> >>>>
> >>> >> >>>> First: Encourage extension packages.  This would mean:
> >>> >> >>>> * Reorganize website to allow for extension packages to be
> >>> >> >>>> displayed
> >>> >> >>>> prominently
> >>> >> >>>> * Add support for name-space packages in yt
> >>> >> >>>> * (possible) split out some packages from analysis_modules,
> >>> >> >>>> including
> >>> >> >>>> halo finding
> >>> >> >>>> * Codify process of extension package creation, including how
> to
> >>> >> >>>> have
> >>> >> >>>> CI set up for them and build system.
> >>> >> >>>>
> >>> >> >>>> The second, more disruptive proposal:
> >>> >> >>>> * Split yt into subprojects.  This would include spinning out
> the
> >>> >> >>>> volume rendering and some or all of the frontends, and probably
> >>> >> >>>> the
> >>> >> >>>> testing infrastructure as well.
> >>> >> >>>> * Split further astro-specific routines into an astro
> extension,
> >>> >> >>>> and
> >>> >> >>>> begin the process of doing this with other domains as well.
> (As
> >>> >> >>>> in
> >>> >> >>>> the long-simmering domain context YTEP.)
> >>> >> >>>>
> >>> >> >>>> I’ll invite comments from everyone, but particularly from folks
> >>> >> >>>> who
> >>> >> >>>> have either not contributed to an analysis module or extension
> >>> >> >>>> package
> >>> >> >>>> because of concerns that would be addressed by this, as well as
> >>> >> >>>> from
> >>> >> >>>> core developers this would impact.  If the thread gets too
> >>> >> >>>> unweildy
> >>> >> >>>> we
> >>> >> >>>> may also want to table this for the next yt team meeting.
> >>> >> >>>>
> >>> >> >>>> Thanks,
> >>> >> >>>>
> >>> >> >>>> Matt
> >>> >> >>>> _______________________________________________
> >>> >> >>>> yt-dev mailing list
> >>> >> >>>> yt-dev at lists.spacepope.org
> >>> >> >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >>> >> >>>
> >>> >> >>>
> >>> >> >>>
> >>> >> >>> _______________________________________________
> >>> >> >>> yt-dev mailing list
> >>> >> >>> yt-dev at lists.spacepope.org
> >>> >> >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >>> >> >>>
> >>> >> >> _______________________________________________
> >>> >> >> yt-dev mailing list
> >>> >> >> yt-dev at lists.spacepope.org
> >>> >> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >>> >> >
> >>> >> > _______________________________________________
> >>> >> > yt-dev mailing list
> >>> >> > yt-dev at lists.spacepope.org
> >>> >> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >>> >> _______________________________________________
> >>> >> yt-dev mailing list
> >>> >> yt-dev at lists.spacepope.org
> >>> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >>> >
> >>> >
> >>> >
> >>> > _______________________________________________
> >>> > yt-dev mailing list
> >>> > yt-dev at lists.spacepope.org
> >>> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >>> >
> >>> _______________________________________________
> >>> yt-dev mailing list
> >>> yt-dev at lists.spacepope.org
> >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >>
> >>
> >>
> >>
> >> --
> >> Michael Zingale
> >> Associate Professor
> >>
> >> Dept. of Physics & Astronomy • Stony Brook University • Stony Brook, NY
> >> 11794-3800
> >> phone:  631-632-8225
> >> e-mail: Michael.Zingale at stonybrook.edu
> >> web: http://www.astro.sunysb.edu/mzingale
> >> github: http://github.com/zingale
> >>
> >>
> >> _______________________________________________
> >> yt-dev mailing list
> >> yt-dev at lists.spacepope.org
> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >>
> >
> >
> > _______________________________________________
> > yt-dev mailing list
> > yt-dev at lists.spacepope.org
> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20160127/7fec7366/attachment.html>

More information about the yt-dev mailing list