[yt-dev] The future of yt

Michael Zingale michael.zingale at stonybrook.edu
Mon Jan 18 10:16:33 PST 2016


Affiliated packages sound like a good idea to me, but I am a bit hesitant
about splitting the volume renderer and frontends out of yt's core.

On Mon, Jan 18, 2016 at 1:11 PM, Matthew Turk <matthewturk at gmail.com> wrote:

> Hi Desika,
>
> Yup, that's what I meant by it.  And there are things that would
> normally qualify as that type of extension, too, that are already in
> the codebase.
>
> Flask's extensions are another good example of how this type of thing is
> done:
>
> http://flask.pocoo.org/extensions/
> http://flask.pocoo.org/docs/0.10/extensiondev/
>
> -Matt
>
> On Mon, Jan 18, 2016 at 12:02 PM, Desika Narayanan
> <desika.narayanan at gmail.com> wrote:
> > Hey All,
> >
> > Matt - just to maybe clarify by examples...are you thinking things like
> > Trident or powderday are sorts of yt extensions that you mean?  (If so,
> then
> > I do like this idea a lot too).  I think one might imagine having a clear
> > set of guidelines for developers of extensions to aspire toward (even if
> > it's for example, having some basic set of or style of documentation) so
> > that from the user point, when looking at extensions  on the yt docs,
> > there's some modicum of uniformity.  That might make it easier for users
> to
> > browse and understand the extensions.
> >
> > -d
> >
> > On Mon, Jan 18, 2016 at 12:43 PM, Matthew Turk <matthewturk at gmail.com>
> > wrote:
> >>
> >> On Mon, Jan 18, 2016 at 11:32 AM, John ZuHone <jzuhone at gmail.com>
> wrote:
> >> > Matt can correct me if I am wrong, but I was under the impression that
> >> > the two components of the proposal were separate and not inevitably
> tied to
> >> > one another.
> >> >
> >> > I am in strong support of the idea of extensions, but I strongly
> >> > disagree with most of the second component of this proposal. I
> believe it is
> >> > far too disruptive, and I honestly don’t see the argument for it
> (except
> >> > maybe splitting some astro-specific stuff out into an extension). But
> >> > spinning off stuff like volume rendering and other big components
> like that
> >> > seems unnecessary.
> >> >
> >>
> >> No need to correct; the second one is indeed an extension of the
> >> first, and a separate proposal.
> >>
> >> >> On Jan 18, 2016, at 12:26 PM, Matthew Turk <matthewturk at gmail.com>
> >> >> wrote:
> >> >>
> >> >> Hi Nathan,
> >> >>
> >> >> Just to quickly address one component before anybody else replies: I
> >> >> would volunteer to do the majority of this work (which I think from a
> >> >> technical perspective is not necessarily too bad), and I don't see it
> >> >> being a 3.X release, but maybe a 4.0 release, and I don't have
> answers
> >> >> for when it would happen since I think that is a community decision
> we
> >> >> shouldn't rush into.
> >> >>
> >> >> On Mon, Jan 18, 2016 at 11:21 AM, Nathan Goldbaum
> >> >> <nathan12343 at gmail.com> wrote:
> >> >>> Just a few thoughts about this.
> >> >>>
> >> >>> I think encouraging more external packages to depend on yt will be
> >> >>> great,
> >> >>> and will keep us more honest about the backward compatibility
> >> >>> guarantees
> >> >>> we've committed ourselves to. Building social capital by encouraging
> >> >>> an
> >> >>> "affiliated" package ecosystem like astropy has is also good, both
> for
> >> >>> social capital and also by enabling cool new features for our users.
> >> >>>
> >> >>> That said, I'm skeptical about some of the technical components of
> >> >>> Matt's
> >> >>> proposal. In particular, I'm concerned with the amount of work
> >> >>> required to
> >> >>> make major changes to the core yt codebase, while still maintaining
> >> >>> backward
> >> >>> compatibility. Who will do this work? When will it happen? Would
> this
> >> >>> constitute a yt 4.0 release?
> >> >>>
> >> >>> While I do think making the core yt package less astronomy focused
> is
> >> >>> good,
> >> >>> I also think it's possible to do so while maintaining strong
> backward
> >> >>> compatibility guarantees, or at least going through a deprecation
> >> >>> cycle over
> >> >>> the course of several minor versions. I don't think making another
> >> >>> cycle of
> >> >>> breaking changes so soon after yt-3.0 is a good idea, as that will
> >> >>> impose a
> >> >>> large amount of work both on the yt developers and on our users.
> >> >>>
> >> >>> On Mon, Jan 18, 2016 at 9:43 AM, Matthew Turk <
> matthewturk at gmail.com>
> >> >>> wrote:
> >> >>>>
> >> >>>> Hi everyone,
> >> >>>>
> >> >>>> The last couple weeks I’ve been thinking a lot about the future of
> >> >>>> yt.
> >> >>>> What I’d like to propose is that we shift investment in yt as a
> >> >>>> single, monolithic codebase into yt as a project, or an ecosystem
> of
> >> >>>> projects.
> >> >>>>
> >> >>>> This came out of the discussion of the extension/affiliated
> packages,
> >> >>>> analysis modules, and so on.  Britton has for a while been pitching
> >> >>>> the idea [which I will poorly paraphrase here] that yt can be the
> >> >>>> framework on top of which killer apps can be built.  I think this
> is
> >> >>>> great.
> >> >>>>
> >> >>>> What’s holding us back in some ways from this is that yt is
> currently
> >> >>>> structured as a monolithic code base, with little to no discovery
> of
> >> >>>> other packages and apps and whatnot.  We tried for a while to
> change
> >> >>>> this with The Barn, but it ended up not quite taking off.  I think
> >> >>>> the
> >> >>>> time is right to try to change the way we think about yt to be more
> >> >>>> about yt the Project, rather than yt the Codebase; the core
> codebase
> >> >>>> is an important component of this, but not the whole of it.
> >> >>>>
> >> >>>> Encouraging an ecosystem of packages can have a few very important
> >> >>>> benefits:
> >> >>>>
> >> >>>> * External packages will confer greater individual credit to the
> >> >>>> folks who develop them.
> >> >>>> * External packages can be versioned and developed independently;
> the
> >> >>>> review process can be different.
> >> >>>> * yt’s core can be emphasized as a generic package, on top of which
> >> >>>> astronomy analysis can be built.
> >> >>>> * Packages can be maintained wherever, including alternate
> locations
> >> >>>> such as github.
> >> >>>>
> >> >>>> On the other hand, having packages inside the main distribution
> makes
> >> >>>> discoverability much, much easier.  It also enables everything to
> be
> >> >>>> In The Box.  And, the continuous integration and testing system is
> >> >>>> already set up for yt.  But, these are all possible to overcome --
> we
> >> >>>> can devise a strategy for adding packages to the CI system (and if
> >> >>>> they are externally managed, they can also rely on yt as a
> dependency
> >> >>>> and use whatever CI system they like!) and we can improve
> >> >>>> discoverability by refocusing the website to enable this.  I've
> asked
> >> >>>> Kacper about adding new packages, and it's not as easy as it might
> >> >>>> seem, so we may need to be careful about how that process occurs;
> one
> >> >>>> possibility would be to provide servers and ready-made setups, but
> >> >>>> have individuals do the heavy lifting.  We could even have
> something
> >> >>>> in the codebase that describes some packages that are available.
> >> >>>> External packages could have much looser dependency rules, which
> >> >>>> means
> >> >>>> they can be free to take advantage of things like OpenCL, numba,
> etc,
> >> >>>> without having to add them to the primary codebase.
> >> >>>>
> >> >>>> Synchronizing APIs and versions across extension packages may be
> >> >>>> difficult in some particular cases, but I suspect in practice will
> >> >>>> not
> >> >>>> be an issue, as long as we continue to have a reasonably stable
> >> >>>> *public* API, and graduate a few things (such as .blocks) into a
> >> >>>> public API from semi-private.
> >> >>>>
> >> >>>> To this end, of really encouraging an ecosystem of packages, I’d
> like
> >> >>>> to propose two things, in increasing order of disruptiveness.
> >> >>>>
> >> >>>> First: Encourage extension packages.  This would mean:
> >> >>>> * Reorganize website to allow for extension packages to be
> displayed
> >> >>>> prominently
> >> >>>> * Add support for name-space packages in yt
> >> >>>> * (possible) split out some packages from analysis_modules,
> including
> >> >>>> halo finding
> >> >>>> * Codify process of extension package creation, including how to
> have
> >> >>>> CI set up for them and build system.
> >> >>>>
> >> >>>> The second, more disruptive proposal:
> >> >>>> * Split yt into subprojects.  This would include spinning out the
> >> >>>> volume rendering and some or all of the frontends, and probably the
> >> >>>> testing infrastructure as well.
> >> >>>> * Split further astro-specific routines into an astro extension,
> and
> >> >>>> begin the process of doing this with other domains as well.  (As in
> >> >>>> the long-simmering domain context YTEP.)
> >> >>>>
> >> >>>> I’ll invite comments from everyone, but particularly from folks who
> >> >>>> have either not contributed to an analysis module or extension
> >> >>>> package
> >> >>>> because of concerns that would be addressed by this, as well as
> from
> >> >>>> core developers this would impact.  If the thread gets too unweildy
> >> >>>> we
> >> >>>> may also want to table this for the next yt team meeting.
> >> >>>>
> >> >>>> Thanks,
> >> >>>>
> >> >>>> Matt
> >> >>>> _______________________________________________
> >> >>>> yt-dev mailing list
> >> >>>> yt-dev at lists.spacepope.org
> >> >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >> >>>
> >> >>>
> >> >>>
> >> >>> _______________________________________________
> >> >>> yt-dev mailing list
> >> >>> yt-dev at lists.spacepope.org
> >> >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >> >>>
> >> >> _______________________________________________
> >> >> yt-dev mailing list
> >> >> yt-dev at lists.spacepope.org
> >> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >> >
> >> > _______________________________________________
> >> > yt-dev mailing list
> >> > yt-dev at lists.spacepope.org
> >> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >> _______________________________________________
> >> yt-dev mailing list
> >> yt-dev at lists.spacepope.org
> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >
> >
> >
> > _______________________________________________
> > yt-dev mailing list
> > yt-dev at lists.spacepope.org
> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>



-- 
Michael Zingale
Associate Professor

Dept. of Physics & Astronomy • Stony Brook University • Stony Brook, NY
11794-3800
*phone*:  631-632-8225
*e-mail*: Michael.Zingale at stonybrook.edu
*web*: http://www.astro.sunysb.edu/mzingale
github: http://github.com/zingale
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20160118/71e18612/attachment.htm>
-------------- next part --------------
_______________________________________________
yt-dev mailing list
yt-dev at lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org


More information about the yt-dev mailing list