<div dir="ltr">I think both canopy and anaconda are good solutions that will create a useful python environment out of the box with a minimum of fuss on a wide variety of platforms.<div><br></div><div>I have more experience with anaconda so my preference for our main installation avenue is there. A concern with canopy is the somewhat arbitrary choice of packages in the free (as in beer) and non-free distributions.</div>
<div><br></div><div>Anaconda has a very liberal redistribution license (<a href="http://docs.continuum.io/anaconda/eula.html">http://docs.continuum.io/anaconda/eula.html</a>). We could conceivably distribute it ourselves on <a href="http://yt-project.org">yt-project.org</a>. I know Matt is also looking into a miniconda (<a href="http://repo.continuum.io/miniconda/">http://repo.continuum.io/miniconda/</a>) based deployment, which should work well. I think we stand a good chance of being included with the standard anaconda distribution or via conda.</div>
<div><br></div><div>Continuum has also set up binstar to allow projects to upload packaged versions of software which is then available via conda. One issue with binstar is that since we have C dependencies, we'll need buildbots to create packages for all supported OS/python combos (i.e. linux/2.7, OSX/2.7, linux/3.3 OSX/3.3).</div>
<div><br></div><div>Making the install script more lightweight is a good idea, although we'll still be susceptible to external changes in our hard dependencies. Numpy and matplotlib have historically been tough to keep up with.</div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Aug 29, 2013 at 12:26 PM, Matthew Turk <span dir="ltr"><<a href="mailto:matthewturk@gmail.com" target="_blank">matthewturk@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi all,<br>
<br>
We need to figure out yt packaging. This is becoming increasingly<br>
hard, particularly as the number of dependencies grows. (The upgrade<br>
to IPython 1.0 and Matplotlib 1.3.0 has caused several issues, which<br>
spurred this discussion.)<br>
<br>
As it stands, we mainly provide yt through the install script. Every<br>
time a new version comes out, we check compatibility, we update the<br>
install script, and we deploy that. Unfortunately, as packages evolve<br>
externally to yt, this results in occasional breakages, new (implicit)<br>
dependencies, and complexity that goes super-exponentially. I like<br>
the install script, and it is what I use, but I think we need to<br>
re-strategize. It was built many years ago when packaging was a<br>
different landscape, and when we needed a way to get a relatively<br>
small number of dependencies onto a relatively small set of system<br>
types.<br>
<br>
Every day, it seems, brings another problem with the install script.<br>
Not all of these are our fault. But more importantly, I don't think<br>
we should be spending our time on them, when we can only bandaid<br>
something for so long before it's not workable.<br>
<br>
That being said, installation is the single biggest impediment to<br>
people using yt, so we need to ensure it is still easy and simple.<br>
<br>
There are a few options for other installation procedures. I would<br>
like to retain a stripped down version of the install script for ease<br>
and simplicity, but removing many of the optional installs and<br>
focusing instead on the core packages.<br>
<br>
So here are the options. I'd prefer we choose *one* as the primary<br>
method, and then we (potentially) demonstrate how to use the others.<br>
As a note, part of this process will also be the relicensing as BSD<br>
and shoring up our source-based installations, ensuring that they are<br>
correctly packaged, following best-practices guidelines for Python<br>
source. I believe I may have dropped the ball somewhat on that front.<br>
<br>
* Conda / Anaconda: This package manager is gaining traction, and I<br>
think that once relicensing is done we stand a good chance of being<br>
included in the base install. This would mean that someone could<br>
download Conda and just use it. Even without that inclusion, however,<br>
I've heard good things. Conda is based on binary distributions, but<br>
we could also manage our own packaging (potentially in an automated<br>
way) and update with some frequency. Conda is also somewhat tied to<br>
the Wakari platform, and being part of Conda would mean being<br>
available on the IPython-in-the-cloud that is Wakari. I believe this<br>
works well on supers.<br>
* Canopy: This is the Enthought package manager, which Sam has had<br>
some good experience with it. I do not have a feeling for how it<br>
works on supers.<br>
* Source-only: This is the way some packages are managed, but it is<br>
essentially giving up, and while I think it is a good way to go<br>
forward, I'm not sure we'll ever be trivially pip-installable.<br>
* Keep trying to plug holes as they come up in the install script.<br>
<br>
What I think would be very productive is to hear people's experiences<br>
with these package managers. Sam, Nathan, anybody?<br>
<br>
Focusing on a platform-specific manager (brew, macports, apt, rpm) is<br>
a non-starter; they are good options, and we should develop a protocol<br>
for supporting platform-specific packaging systems, but they<br>
bottleneck quite seriously on person-time and we should think<br>
carefully before we tie ourselves to one.<br>
<br>
-Matt<br>
<br>
PS The period in the subject line was editorial. I'd very much like<br>
to settle on a path for all of this stuff; packaging remains one of<br>
the hardest issues in scientific python, as Software Carpentry has<br>
noted time and again. We're now pushing the install script, which is<br>
great for clusters, but it's a remnant of a time before packaging in<br>
Python was as mature as it is now, and before we had as many corner<br>
cases as we do now -- not because they didn't exist, but because we<br>
didn't have enough users to see them.<br>
_______________________________________________<br>
yt-dev mailing list<br>
<a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
</blockquote></div><br></div>