<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Hi Matt and everyone,</div><div class=""><br class=""></div>I support your first option of extension packages. I think the second option is far too disruptive. <div class=""><br class=""></div><div class="">I’m working on two small packages that fit this description (no documentation or tests as of yet):</div><div class=""><br class=""></div><div class=""><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_jzuhone_formulas&d=BQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=hgcBC3x6dKFoTrmFmMYYbKNfiHZlGLKliIidd1LwmHI&m=ub-Ni1o3_R1yAcEElIMioZEXEubJJmCXlAYSYL0Ww3Q&s=FZpD_9YmLljD6CJlsZBjNqt9oT4rRGcld56R6uzep98&e=" class="">https://github.com/jzuhone/formulas</a></div><div class=""><br class=""></div><div class=""><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_jzuhone_equilibrate&d=BQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=hgcBC3x6dKFoTrmFmMYYbKNfiHZlGLKliIidd1LwmHI&m=ub-Ni1o3_R1yAcEElIMioZEXEubJJmCXlAYSYL0Ww3Q&s=dPTcnBTk_Ujon68o796WD8CcL8ibwXR-Nnd_jIuaKug&e=" class="">https://github.com/jzuhone/equilibrate</a></div><div class=""><br class=""></div><div class="">They both depend on yt, but in my view are too domain-specific and/or specialized to be incorporated into yt itself right now. </div><div class=""><br class=""></div><div class="">Then of course there is YT.jl, which does have documentation and testing (but is also a Julia package):</div><div class=""><br class=""></div><div class=""><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_jzuhone_YT.jl&d=BQMFaQ&c=8hUWFZcy2Z-Za5rBPlktOQ&r=hgcBC3x6dKFoTrmFmMYYbKNfiHZlGLKliIidd1LwmHI&m=ub-Ni1o3_R1yAcEElIMioZEXEubJJmCXlAYSYL0Ww3Q&s=p_vZF2xqXU6idKok8F_B6nS-ahue4ud0dxcJ74zIZkE&e=" class="">https://github.com/jzuhone/YT.jl</a></div><div class=""><br class=""></div><div class="">These are just some examples of what external packages might look like. </div><div class=""><br class=""></div><div class="">Best,</div><div class=""><br class=""></div><div class="">John</div><div class=""><br class=""></div><div class=""><div><blockquote type="cite" class=""><div class="">On Jan 18, 2016, at 10:43 AM, Matthew Turk <<a href="mailto:matthewturk@gmail.com" class="">matthewturk@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">Hi everyone,<br class=""><br class="">The last couple weeks I’ve been thinking a lot about the future of yt.<br class="">What I’d like to propose is that we shift investment in yt as a<br class="">single, monolithic codebase into yt as a project, or an ecosystem of<br class="">projects.<br class=""><br class="">This came out of the discussion of the extension/affiliated packages,<br class="">analysis modules, and so on. Britton has for a while been pitching<br class="">the idea [which I will poorly paraphrase here] that yt can be the<br class="">framework on top of which killer apps can be built. I think this is<br class="">great.<br class=""><br class="">What’s holding us back in some ways from this is that yt is currently<br class="">structured as a monolithic code base, with little to no discovery of<br class="">other packages and apps and whatnot. We tried for a while to change<br class="">this with The Barn, but it ended up not quite taking off. I think the<br class="">time is right to try to change the way we think about yt to be more<br class="">about yt the Project, rather than yt the Codebase; the core codebase<br class="">is an important component of this, but not the whole of it.<br class=""><br class="">Encouraging an ecosystem of packages can have a few very important benefits:<br class=""><br class=""> * External packages will confer greater individual credit to the<br class="">folks who develop them.<br class=""> * External packages can be versioned and developed independently; the<br class="">review process can be different.<br class=""> * yt’s core can be emphasized as a generic package, on top of which<br class="">astronomy analysis can be built.<br class=""> * Packages can be maintained wherever, including alternate locations<br class="">such as github.<br class=""><br class="">On the other hand, having packages inside the main distribution makes<br class="">discoverability much, much easier. It also enables everything to be<br class="">In The Box. And, the continuous integration and testing system is<br class="">already set up for yt. But, these are all possible to overcome -- we<br class="">can devise a strategy for adding packages to the CI system (and if<br class="">they are externally managed, they can also rely on yt as a dependency<br class="">and use whatever CI system they like!) and we can improve<br class="">discoverability by refocusing the website to enable this. I've asked<br class="">Kacper about adding new packages, and it's not as easy as it might<br class="">seem, so we may need to be careful about how that process occurs; one<br class="">possibility would be to provide servers and ready-made setups, but<br class="">have individuals do the heavy lifting. We could even have something<br class="">in the codebase that describes some packages that are available.<br class="">External packages could have much looser dependency rules, which means<br class="">they can be free to take advantage of things like OpenCL, numba, etc,<br class="">without having to add them to the primary codebase.<br class=""><br class="">Synchronizing APIs and versions across extension packages may be<br class="">difficult in some particular cases, but I suspect in practice will not<br class="">be an issue, as long as we continue to have a reasonably stable<br class="">*public* API, and graduate a few things (such as .blocks) into a<br class="">public API from semi-private.<br class=""><br class="">To this end, of really encouraging an ecosystem of packages, I’d like<br class="">to propose two things, in increasing order of disruptiveness.<br class=""><br class="">First: Encourage extension packages. This would mean:<br class=""> * Reorganize website to allow for extension packages to be displayed<br class="">prominently<br class=""> * Add support for name-space packages in yt<br class=""> * (possible) split out some packages from analysis_modules, including<br class="">halo finding<br class=""> * Codify process of extension package creation, including how to have<br class="">CI set up for them and build system.<br class=""><br class="">The second, more disruptive proposal:<br class=""> * Split yt into subprojects. This would include spinning out the<br class="">volume rendering and some or all of the frontends, and probably the<br class="">testing infrastructure as well.<br class=""> * Split further astro-specific routines into an astro extension, and<br class="">begin the process of doing this with other domains as well. (As in<br class="">the long-simmering domain context YTEP.)<br class=""><br class="">I’ll invite comments from everyone, but particularly from folks who<br class="">have either not contributed to an analysis module or extension package<br class="">because of concerns that would be addressed by this, as well as from<br class="">core developers this would impact. If the thread gets too unweildy we<br class="">may also want to table this for the next yt team meeting.<br class=""><br class="">Thanks,<br class=""><br class="">Matt<br class="">_______________________________________________<br class="">yt-dev mailing list<br class=""><a href="mailto:yt-dev@lists.spacepope.org" class="">yt-dev@lists.spacepope.org</a><br class="">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org<br class=""></div></div></blockquote></div><br class=""></div></body></html>