[yt-dev] Packaging.

Thu Aug 29 13:38:49 PDT 2013

I like this plan too.  Let's test out conda stuff and see how it all goes.
 I'm happy to assist in any way that I can.

I'm glad that we're addressing this.  It's pretty amazing how well the bash
script has worked in the past, and I think it is a testament to how hard
everyone has worked to keep it updated and patched (and Matt writing it in
the first place!).  So good work everyone, but I think moving to a more
generic means like conda will pay off in the long run now.

Cameron

On Thu, Aug 29, 2013 at 1:22 PM, Sam Skillman <samskillman at gmail.com> wrote:

> I think investigating these options is a great idea, especially if you
> think about it as allowing the install script to not have to be updated
> every single time numpy/matplotlib updates their version. We can let the
> install script stay more stable on the several month timescale and rely on
> other packaging to work out if people want up to the minute installs of
> some package.
>
> As for conda vs canopy, I think we should experiment with one first, and
> polish it off rather than attempting both at the same time.  Since Nathan
> has more experience with conda than I do with canopy, I'd be +1 on going
> down the conda route first.
>
> Sam
>
>
> On Thu, Aug 29, 2013 at 12:50 PM, Nathan Goldbaum <nathan12343 at gmail.com>wrote:
>
>> I think both canopy and anaconda are good solutions that will create a
>> useful python environment out of the box with a minimum of fuss on a wide
>> variety of platforms.
>>
>> I have more experience with anaconda so my preference for our main
>> installation avenue is there. A concern with canopy is the somewhat
>> arbitrary choice of packages in the free (as in beer) and non-free
>> distributions.
>>
>> Anaconda has a very liberal redistribution license (
>> http://docs.continuum.io/anaconda/eula.html).  We could conceivably
>> distribute it ourselves on yt-project.org.  I know Matt is also looking
>> into a miniconda (http://repo.continuum.io/miniconda/) based deployment,
>> which should work well.  I think we stand a good chance of being included
>> with the standard anaconda distribution or via conda.
>>
>> Continuum has also set up binstar to allow projects to upload packaged
>> versions of software which is then available via conda.  One issue with
>> binstar is that since we have C dependencies, we'll need buildbots to
>> create packages for all supported OS/python combos (i.e. linux/2.7,
>> OSX/2.7, linux/3.3 OSX/3.3).
>>
>> Making the install script more lightweight is a good idea, although we'll
>> still be susceptible to external changes in our hard dependencies. Numpy
>> and matplotlib have historically been tough to keep up with.
>>
>>
>> On Thu, Aug 29, 2013 at 12:26 PM, Matthew Turk <matthewturk at gmail.com>wrote:
>>
>>> Hi all,
>>>
>>> We need to figure out yt packaging.  This is becoming increasingly
>>> hard, particularly as the number of dependencies grows.  (The upgrade
>>> to IPython 1.0 and Matplotlib 1.3.0 has caused several issues, which
>>> spurred this discussion.)
>>>
>>> As it stands, we mainly provide yt through the install script.  Every
>>> time a new version comes out, we check compatibility, we update the
>>> install script, and we deploy that.  Unfortunately, as packages evolve
>>> externally to yt, this results in occasional breakages, new (implicit)
>>> dependencies, and complexity that goes super-exponentially.  I like
>>> the install script, and it is what I use, but I think we need to
>>> re-strategize.  It was built many years ago when packaging was a
>>> different landscape, and when we needed a way to get a relatively
>>> small number of dependencies onto a relatively small set of system
>>> types.
>>>
>>> Every day, it seems, brings another problem with the install script.
>>> Not all of these are our fault.  But more importantly, I don't think
>>> we should be spending our time on them, when we can only bandaid
>>> something for so long before it's not workable.
>>>
>>> That being said, installation is the single biggest impediment to
>>> people using yt, so we need to ensure it is still easy and simple.
>>>
>>> There are a few options for other installation procedures.  I would
>>> like to retain a stripped down version of the install script for ease
>>> and simplicity, but removing many of the optional installs and
>>> focusing instead on the core packages.
>>>
>>> So here are the options.  I'd prefer we choose *one* as the primary
>>> method, and then we (potentially) demonstrate how to use the others.
>>> As a note, part of this process will also be the relicensing as BSD
>>> and shoring up our source-based installations, ensuring that they are
>>> correctly packaged, following best-practices guidelines for Python
>>> source.  I believe I may have dropped the ball somewhat on that front.
>>>
>>>  * Conda / Anaconda: This package manager is gaining traction, and I
>>> think that once relicensing is done we stand a good chance of being
>>> included in the base install.  This would mean that someone could
>>> download Conda and just use it.  Even without that inclusion, however,
>>> I've heard good things.  Conda is based on binary distributions, but
>>> we could also manage our own packaging (potentially in an automated
>>> way) and update with some frequency.  Conda is also somewhat tied to
>>> the Wakari platform, and being part of Conda would mean being
>>> available on the IPython-in-the-cloud that is Wakari.  I believe this
>>> works well on supers.
>>>  * Canopy: This is the Enthought package manager, which Sam has had
>>> some good experience with it.  I do not have a feeling for how it
>>> works on supers.
>>>  * Source-only: This is the way some packages are managed, but it is
>>> essentially giving up, and while I think it is a good way to go
>>> forward, I'm not sure we'll ever be trivially pip-installable.
>>>  * Keep trying to plug holes as they come up in the install script.
>>>
>>> What I think would be very productive is to hear people's experiences
>>> with these package managers.  Sam, Nathan, anybody?
>>>
>>> Focusing on a platform-specific manager (brew, macports, apt, rpm) is
>>> a non-starter; they are good options, and we should develop a protocol
>>> for supporting platform-specific packaging systems, but they
>>> bottleneck quite seriously on person-time and we should think
>>> carefully before we tie ourselves to one.
>>>
>>> -Matt
>>>
>>> PS The period in the subject line was editorial.  I'd very much like
>>> to settle on a path for all of this stuff; packaging remains one of
>>> the hardest issues in scientific python, as Software Carpentry has
>>> noted time and again.  We're now pushing the install script, which is
>>> great for clusters, but it's a remnant of a time before packaging in
>>> Python was as mature as it is now, and before we had as many corner
>>> cases as we do now -- not because they didn't exist, but because we
>>> didn't have enough users to see them.
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>
>>
>>
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>>
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>

-- 
Cameron Hummels
Postdoctoral Researcher
Steward Observatory
University of Arizona
http://chummels.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20130829/906757ba/attachment.htm>