[yt-dev] The future of yt

[John ZuHone]

Hi Matt and everyone,

I support your first option of extension packages. I think the second option is far too disruptive. 

I’m working on two small packages that fit this description (no documentation or tests as of yet):

https://github.com/jzuhone/formulas <https://github.com/jzuhone/formulas>

https://github.com/jzuhone/equilibrate <https://github.com/jzuhone/equilibrate>

They both depend on yt, but in my view are too domain-specific and/or specialized to be incorporated into yt itself right now. 

Then of course there is YT.jl, which does have documentation and testing (but is also a Julia package):

https://github.com/jzuhone/YT.jl <https://github.com/jzuhone/YT.jl>

These are just some examples of what external packages might look like. 

Best,

John

> On Jan 18, 2016, at 10:43 AM, Matthew Turk <matthewturk at gmail.com> wrote:
> 
> Hi everyone,
> 
> The last couple weeks I’ve been thinking a lot about the future of yt.
> What I’d like to propose is that we shift investment in yt as a
> single, monolithic codebase into yt as a project, or an ecosystem of
> projects.
> 
> This came out of the discussion of the extension/affiliated packages,
> analysis modules, and so on.  Britton has for a while been pitching
> the idea [which I will poorly paraphrase here] that yt can be the
> framework on top of which killer apps can be built.  I think this is
> great.
> 
> What’s holding us back in some ways from this is that yt is currently
> structured as a monolithic code base, with little to no discovery of
> other packages and apps and whatnot.  We tried for a while to change
> this with The Barn, but it ended up not quite taking off.  I think the
> time is right to try to change the way we think about yt to be more
> about yt the Project, rather than yt the Codebase; the core codebase
> is an important component of this, but not the whole of it.
> 
> Encouraging an ecosystem of packages can have a few very important benefits:
> 
> * External packages will confer greater individual credit to the
> folks who develop them.
> * External packages can be versioned and developed independently; the
> review process can be different.
> * yt’s core can be emphasized as a generic package, on top of which
> astronomy analysis can be built.
> * Packages can be maintained wherever, including alternate locations
> such as github.
> 
> On the other hand, having packages inside the main distribution makes
> discoverability much, much easier.  It also enables everything to be
> In The Box.  And, the continuous integration and testing system is
> already set up for yt.  But, these are all possible to overcome -- we
> can devise a strategy for adding packages to the CI system (and if
> they are externally managed, they can also rely on yt as a dependency
> and use whatever CI system they like!) and we can improve
> discoverability by refocusing the website to enable this.  I've asked
> Kacper about adding new packages, and it's not as easy as it might
> seem, so we may need to be careful about how that process occurs; one
> possibility would be to provide servers and ready-made setups, but
> have individuals do the heavy lifting.  We could even have something
> in the codebase that describes some packages that are available.
> External packages could have much looser dependency rules, which means
> they can be free to take advantage of things like OpenCL, numba, etc,
> without having to add them to the primary codebase.
> 
> Synchronizing APIs and versions across extension packages may be
> difficult in some particular cases, but I suspect in practice will not
> be an issue, as long as we continue to have a reasonably stable
> *public* API, and graduate a few things (such as .blocks) into a
> public API from semi-private.
> 
> To this end, of really encouraging an ecosystem of packages, I’d like
> to propose two things, in increasing order of disruptiveness.
> 
> First: Encourage extension packages.  This would mean:
> * Reorganize website to allow for extension packages to be displayed
> prominently
> * Add support for name-space packages in yt
> * (possible) split out some packages from analysis_modules, including
> halo finding
> * Codify process of extension package creation, including how to have
> CI set up for them and build system.
> 
> The second, more disruptive proposal:
> * Split yt into subprojects.  This would include spinning out the
> volume rendering and some or all of the frontends, and probably the
> testing infrastructure as well.
> * Split further astro-specific routines into an astro extension, and
> begin the process of doing this with other domains as well.  (As in
> the long-simmering domain context YTEP.)
> 
> I’ll invite comments from everyone, but particularly from folks who
> have either not contributed to an analysis module or extension package
> because of concerns that would be addressed by this, as well as from
> core developers this would impact.  If the thread gets too unweildy we
> may also want to table this for the next yt team meeting.
> 
> Thanks,
> 
> Matt
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20160118/74370361/attachment.html>
-------------- next part --------------
_______________________________________________
yt-dev mailing list
yt-dev at lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org