[yt-dev] Packaging.

Matthew Turk matthewturk at gmail.com
Tue Sep 10 10:59:22 PDT 2013


Hi Nathan,

On Tue, Sep 10, 2013 at 1:16 PM, Nathan Goldbaum <nathan12343 at gmail.com> wrote:
> Hey John,
>
> A few questions:
>
> 1. What did you choose for INST_YT_SOURCE in get_yt.sh? If INST_YT_SOURCE=1,
> what happens when you try INST_YT_SOURCE=0?
>
> 2. Again, if you had INST_YT_SOURCE=1, when you manually run `setup.py
> install` in the yt source repository, it should print out a few lines
> showing the root directory for the PNG, Freetype, and HDF5 libraries it
> tries to link against.  This should be the very first thing printed out by
> the setup script.  Is it linking against anaconda's libraries or system
> libraries?
>
> 3. What's the output of `ldd
> /u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/lib/png_writer.so`?
>
> I'm beginning to suspect that we cannot link against the libraries provided
> by anaconda unless we tailor our build environment to match the sort of
> gymnastics that `conda build` does to sanitize the build environment.  Just
> yesterday I was having no end of grief trying to link against anaconda's
> hdf5 library.  yt is able to link against this library inside a conda build
> environment, but when I try to link in my normal shell environment I run
> into issues - mostly because the OS X hdf5 library on anaconda was compiled
> on an OS X 10.5 machine.

If I read you correctly, you are saying that, practically speaking,
this would mainly mean not using the yt-conda HDF5 for Enzo/FLASH/etc
outside of yt.  Right?

>
>
> On Tue, Sep 10, 2013 at 9:54 AM, John ZuHone <jzuhone at gmail.com> wrote:
>>
>> Hi Nathan and all,
>>
>> Here's a little more info on that error you asked about regarding glibc.
>> It came up again for me on Pleiades, this time with yt itself.
>>
>> Traceback (most recent call last):
>>   File "/u/jzuhone/anaconda/bin/yt", line 4, in <module>
>>     from yt.utilities.command_line import run_main
>>   File
>> "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/command_line.py",
>> line 29, in <module>
>>     from yt.mods import *
>>   File "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/mods.py", line
>> 60, in <module>
>>     from yt.data_objects.api import \
>>   File
>> "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/api.py",
>> line 31, in <module>
>>     from grid_patch import \
>>   File
>> "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/grid_patch.py",
>> line 35, in <module>
>>     from yt.data_objects.data_containers import YTFieldData
>>   File
>> "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/data_containers.py",
>> line 45, in <module>
>>     from yt.data_objects.derived_quantities import GridChildMaskWrapper
>>   File
>> "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/data_objects/derived_quantities.py",
>> line 36, in <module>
>>     from yt.utilities.parallel_tools.parallel_analysis_interface import \
>>   File
>> "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/parallel_tools/parallel_analysis_interface.py",
>> line 39, in <module>
>>     from yt.utilities.lib import \
>>   File
>> "/u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/lib/__init__.py",
>> line 35, in <module>
>>     from .png_writer import *
>> ImportError: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by
>> /u/jzuhone/anaconda/lib/python2.7/site-packages/yt/utilities/lib/png_writer.so)
>>
>> Best,
>>
>> John
>>
>> On Sep 3, 2013, at 3:10 PM, Nathan Goldbaum <nathan12343 at gmail.com> wrote:
>>
>> John, can you give a little more info on that error?  I'm a little
>> concerned that there are binary incompatibilities that conda can't resolve.
>> Perhaps this is something we should report on the conda issue tracker.
>>
>> Matt, can't we get the recipes in the same way we get the latest dev
>> install script?  Something like:
>>
>> wget https://bitbucket.org/yt_analysis/yt_conda/raw/yt/meta.yaml
>>
>> This doesn't currently work, although it does work for the yt repo, I
>> think because we use named branches in that repository.
>>
>>
>> On Tue, Sep 3, 2013 at 12:00 PM, John ZuHone <jzuhone at gmail.com> wrote:
>>>
>>> Works fine for me on OS X x86_64.
>>>
>>> On my Goddard-controlled Linux x86_64 server, everything worked fine
>>> except Mercurial, which I "conda install"-ed using the yt link, but the
>>> binary had an incompatibility which the glibc that was installed on the
>>> machine. Using pip to install Mercurial (which did it from source) was the
>>> workaround.
>>>
>>> On Sep 3, 2013, at 2:54 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>>>
>>> > Hi all,
>>> >
>>> > Thanks to everybody who has reported back on testing.  After some
>>> > talking both offline and on IRC, as well as here, I think we would
>>> > need to do the following things:
>>> >
>>> > * Make a single script that grabs the appropriate distribution of
>>> > miniconda and installs it.  Right now I have a mechanism for doing
>>> > this, but it's currently tied to an architecture.
>>> > * Create a mechanism for installing all the packages we need.  Nearly
>>> > all are available inside the Continuum repos.  What we're hung up on
>>> > is that source installs require a "recipe", and transmitting the
>>> > recipe is where I don't have an idea of what to do.
>>> > * Test this out lots of places
>>> > * Clean up the edges in the (new) install script
>>> > * Move the old install script to maintenance mode
>>> > * Update all documentation to describe this and mothball other
>>> > methods of installation
>>> >
>>> > What would be nice:
>>> >
>>> > * Make available nightly builds of yt on several architectures using
>>> > binstar
>>> > * Utilize more of the packages included in conda elsewhere in yt, now
>>> > that we can!
>>> >
>>> > Here's my current recipe for get_yt.sh:
>>> >
>>> > http://paste.yt-project.org/show/3843/
>>> >
>>> > (The config thing may get switched to include the --system argument,
>>> > to modify the "yt-conda" condarc.)  The step that I'm most stuck on is
>>> > getting the yt recipe to people.  If we want to make it possible and
>>> > easy to build from source, we need to get the contents of a "conda
>>> > recipe" to people.  They can then run "conda build ." in the
>>> > directory.  Here are the recipes that we've been playing with:
>>> >
>>> > https://bitbucket.org/yt_analysis/yt_conda/src
>>> >
>>> > Basically, in get_yt.sh, to do from source instead of from binary we
>>> > need to insert a step at the end that downloads the recipes somehow
>>> > and then cd's into the right directory and builds them.  The reason
>>> > this is tricky is that we often need to bootstrap ourselves; we can't
>>> > assume anything exists.  We can download the .tar.bz2 of the current
>>> > tip of the repo, but it includes the hash in the directory name that
>>> > it extracts to.
>>> >
>>> > So I think what we need is a mechanism for getting the current state
>>> > of the repo, figuring out the name of the repo's directory, moves into
>>> > it, and then builds.  I believe that all/most of this becomes much,
>>> > much easier if hg gets included in Anaconda, which Nathan has
>>> > submitted a PR for.  So hopefully that will be taken care of, but
>>> > until that time we can possibly figure something out.  I'm not sure
>>> > that we have the resources to continually support binary nightly
>>> > builds in perpetuity for all the architectures that people run on, so
>>> > having source would be awesome.  Plus, one of the biggest appeals of
>>> > how we distribute yt is that the source is included; I would very much
>>> > not like to give this up.
>>> >
>>> > Thoughts?  Has anyone else tested any of this out?
>>> >
>>> > -Matt
>>> >
>>> > On Tue, Sep 3, 2013 at 1:04 PM, Britton Smith <brittonsmith at gmail.com>
>>> > wrote:
>>> >> Hi everyone,
>>> >>
>>> >> Sorry for chiming in late.  I just moved when this thread began and do
>>> >> not
>>> >> have regular internet access.
>>> >>
>>> >> I really like this idea of conda, especially as a package manager that
>>> >> only
>>> >> optionally makes its own edits to your .bashrc.  I have always really
>>> >> liked
>>> >> that the install script creates a clean python stack with basically
>>> >> everything a python user needs.  I have on occasion suggested it to
>>> >> people
>>> >> just looking to use numpy and matploblib.  It looks like conda will
>>> >> continue
>>> >> to provide this nice by-product, so I'm all for it.
>>> >>
>>> >> I won't be in a position to help with testing and such for another
>>> >> week or
>>> >> so when I get regular internet access, but I would be glad to do so
>>> >> then.
>>> >>
>>> >> Britton
>>> >>
>>> >>
>>> >> On Fri, Aug 30, 2013 at 9:29 PM, Nathan Goldbaum
>>> >> <nathan12343 at gmail.com>
>>> >> wrote:
>>> >>>
>>> >>> Everything should be available now for 64 bit linux and OS X.
>>> >>>
>>> >>>
>>> >>> On Fri, Aug 30, 2013 at 10:52 AM, Chris Malone
>>> >>> <chris.m.malone at gmail.com>
>>> >>> wrote:
>>> >>>>
>>> >>>> Hi Nathan,
>>> >>>>
>>> >>>> That appears to work as it built the environment and `conda install
>>> >>>> ...`
>>> >>>> added packages to my environment.
>>> >>>>
>>> >>>> One mistake I made was that I originally downloaded the "latest" OS
>>> >>>> X
>>> >>>> build of Miniconda, but that happened to be Miniconda3, which is
>>> >>>> python 3
>>> >>>> based.  Trying to build the environment with that yields an error
>>> >>>> regarding
>>> >>>> incompatibility of yt and python3.3, as it should.
>>> >>>>
>>> >>>> Chris
>>> >>>>
>>> >>>>
>>> >>>> On Fri, Aug 30, 2013 at 10:42 AM, Nathan Goldbaum
>>> >>>> <nathan12343 at gmail.com>
>>> >>>> wrote:
>>> >>>>>
>>> >>>>> Hey Chris,
>>> >>>>>
>>> >>>>> I don't think mercurial is strictly necessary, can you try again
>>> >>>>> without
>>> >>>>> it?  I think if Matt uploads a mercurial package for OS X this
>>> >>>>> won't be an
>>> >>>>> issue. I'll send him an updated tarball.
>>> >>>>>
>>> >>>>> I submitted a mercurial recipe to conda-recipes yesterday
>>> >>>>> (https://github.com/ContinuumIO/conda-recipes/pull/14) so hopefully
>>> >>>>> a
>>> >>>>> mercurial build will be included in future anaconda releases.
>>> >>>>>
>>> >>>>>
>>> >>>>> On Fri, Aug 30, 2013 at 10:39 AM, Chris Malone
>>> >>>>> <chris.m.malone at gmail.com> wrote:
>>> >>>>>>
>>> >>>>>> I just tried setting this up on OS X 10.7.5 and failed when
>>> >>>>>> attempting
>>> >>>>>> to create the conda environment due to a missing mercurial
>>> >>>>>> package:
>>> >>>>>>
>>> >>>>>> $ conda create -n ytenv -c http://conda.binstar.org/yt_project yt
>>> >>>>>> mercurial ipython tornado pyzmq pygments jinja2 sphinx
>>> >>>>>> Error: No packages found matching: mercurial
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> On Thu, Aug 29, 2013 at 9:01 PM, Nathan Goldbaum
>>> >>>>>> <nathan12343 at gmail.com> wrote:
>>> >>>>>>>
>>> >>>>>>> Yup, please try on OSX as well.  If you make sure Matt's binstar
>>> >>>>>>> is in
>>> >>>>>>> your .condarc, you should be able to get yt by doing 'conda
>>> >>>>>>> install yt'.
>>> >>>>>>>
>>> >>>>>>> I built the OSX binary on my laptop so I'd appreciate hearing
>>> >>>>>>> about
>>> >>>>>>> issues, particularly if there are issues on older OS X releases.
>>> >>>>>>>
>>> >>>>>>> On Thursday, August 29, 2013, Matthew Turk wrote:
>>> >>>>>>>>
>>> >>>>>>>> Hi all,
>>> >>>>>>>>
>>> >>>>>>>> Thank you for the feedback -- I am glad there is some agreement
>>> >>>>>>>> about
>>> >>>>>>>> possible ways forward, and so I'm happy to try to use this as an
>>> >>>>>>>> opportunity to explore simpler, more reliable methods than the
>>> >>>>>>>> install
>>> >>>>>>>> script.
>>> >>>>>>>>
>>> >>>>>>>> This afternoon, I spent a bit of time with Conda, and I think
>>> >>>>>>>> it's
>>> >>>>>>>> quite nice.  There are a few rough corners, particularly related
>>> >>>>>>>> to
>>> >>>>>>>> the binstar service, but it's so far pretty great.  With
>>> >>>>>>>> Nathan's
>>> >>>>>>>> help, I was able to upload a yt-2.5.5 package for linux x86_64
>>> >>>>>>>> and
>>> >>>>>>>> then install it.
>>> >>>>>>>>
>>> >>>>>>>> The workflow that seems to work:
>>> >>>>>>>>
>>> >>>>>>>> * Get miniconda: http://repo.continuum.io/miniconda/index.html
>>> >>>>>>>> * Run the installer for miniconda
>>> >>>>>>>> * Enter the conda environment and then install yt by doing
>>> >>>>>>>> "conda
>>> >>>>>>>> install yt -c http://conda.binstar.org/yt_project/ ".
>>> >>>>>>>>
>>> >>>>>>>> I think that this can likely all be stuck into a bash script.  A
>>> >>>>>>>> simple, first pass at this is here:
>>> >>>>>>>>
>>> >>>>>>>> http://paste.yt-project.org/show/3833/
>>> >>>>>>>>
>>> >>>>>>>> This right now only works on Linux x86_64, but getting it to
>>> >>>>>>>> work for
>>> >>>>>>>> other machines won't be too hard.  I suspect we will be able to
>>> >>>>>>>> do
>>> >>>>>>>> nightlies very easily as well.  If anyone out there has an
>>> >>>>>>>> x86_64
>>> >>>>>>>> machine they wouldn't mind trying it on, that would be very
>>> >>>>>>>> helpful!
>>> >>>>>>>> I did find that once I ran this script, I had to actually
>>> >>>>>>>> prepend the
>>> >>>>>>>> PATH afterwards as well.  This means doing:
>>> >>>>>>>>
>>> >>>>>>>> export LD_LIBRARY_PATH=$HOME/yt-conda:$LD_LIBRARY_PATH
>>> >>>>>>>> export PATH=$HOME/yt-conda:$PATH
>>> >>>>>>>> source activate ytenv
>>> >>>>>>>>
>>> >>>>>>>> At that point, everything was set up and working for me.  The
>>> >>>>>>>> miniconda install offers to add paths to .bashrc, but I don't
>>> >>>>>>>> think
>>> >>>>>>>> we
>>> >>>>>>>> should go down that route.  That being said, this is also a
>>> >>>>>>>> possible
>>> >>>>>>>> point of friction.
>>> >>>>>>>>
>>> >>>>>>>> One nice thing is that this also completely works with the full
>>> >>>>>>>> anaconda; if someone wants everything that is in the anaconda
>>> >>>>>>>> install,
>>> >>>>>>>> they can even simply do "conda install anaconda" from the
>>> >>>>>>>> command
>>> >>>>>>>> line
>>> >>>>>>>> to get it.  But the stripped down subset is the default.
>>> >>>>>>>>
>>> >>>>>>>> If anyone has a chance to try this out and has feedback, I'd
>>> >>>>>>>> greatly
>>> >>>>>>>> appreciate it!  I think Nathan has done something very similar
>>> >>>>>>>> for
>>> >>>>>>>> OSX.  I've also put a couple simple conda recipes here:
>>> >>>>>>>> https://bitbucket.org/yt_analysis/yt_conda which we can use as a
>>> >>>>>>>> basis
>>> >>>>>>>> for distributing builds and setting them up on buildbots and the
>>> >>>>>>>> like.
>>> >>>>>>>> I'm pretty optimistic about this.
>>> >>>>>>>>
>>> >>>>>>>> -Matt
>>> >>>>>>>>
>>> >>>>>>>> On Thu, Aug 29, 2013 at 5:50 PM, Nathan Goldbaum
>>> >>>>>>>> <nathan12343 at gmail.com> wrote:
>>> >>>>>>>>> I think to get everything working in a sustainable fashion, we
>>> >>>>>>>>> would need
>>> >>>>>>>>> buildbots for all platform combinations that we want to
>>> >>>>>>>>> support, so
>>> >>>>>>>>> all
>>> >>>>>>>>> permutations of the (32/64 bit,  linux / OS X / windows,
>>> >>>>>>>>> py27/py3.3) tuple.
>>> >>>>>>>>> At the moment anaconda seems to support 32 and 64 bit linux, 64
>>> >>>>>>>>> bit
>>> >>>>>>>>> OS X
>>> >>>>>>>>> (not totally clear if OS X version matters), and 32 and 64 bit
>>> >>>>>>>>> windows.
>>> >>>>>>>>>
>>> >>>>>>>>> Another option is to rely on conda build, which compiles
>>> >>>>>>>>> everything
>>> >>>>>>>>> from
>>> >>>>>>>>> source.
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>> On Thu, Aug 29, 2013 at 2:45 PM, Stephen Skory <s at skory.us>
>>> >>>>>>>>> wrote:
>>> >>>>>>>>>>
>>> >>>>>>>>>> Hi all,
>>> >>>>>>>>>>
>>> >>>>>>>>>> I have less of a skin in this than I used to, but I'd like to
>>> >>>>>>>>>> raise
>>> >>>>>>>>>> the issue of Windows & package managers. For example, Anaconda
>>> >>>>>>>>>> is
>>> >>>>>>>>>> available for Windows - would that mean that yt might "just
>>> >>>>>>>>>> work"
>>> >>>>>>>>>> on
>>> >>>>>>>>>> Windows? Or the opposite, and it would require a great deal of
>>> >>>>>>>>>> effort
>>> >>>>>>>>>> to get all the various things we expect to be .so's to work as
>>> >>>>>>>>>> .dll's
>>> >>>>>>>>>> (such as the Cython helpers or halo-finding stuff)?
>>> >>>>>>>>>>
>>> >>>>>>>>>> I don't know the answers to these questions, but I think it's
>>> >>>>>>>>>> worth
>>> >>>>>>>>>> thinking about.
>>> >>>>>>>>>>
>>> >>>>>>>>>> --
>>> >>>>>>>>>> Stephen Skory
>>> >>>>>>>>>> s at skory.us
>>> >>>>>>>>>> http://stephenskory.com/
>>> >>>>>>>>>> 510.621.3687 (google voice)
>>> >>>>>>>>>> _______________________________________________
>>> >>>>>>>>>> yt-dev mailing list
>>> >>>>>>>>>> yt-dev at lists.spacepope.org
>>> >>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>> _______________________________________________
>>> >>>>>>>>> yt-dev mailing list
>>> >>>>>>>>> yt-dev at lists.spacepope.org
>>> >>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> >>>>>>>>>
>>> >>>>>>>> _______________________________________________
>>> >>>>>>>> yt-dev mailing list
>>> >>>>>>>> yt-dev at lists.spacepope.org
>>> >>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> _______________________________________________
>>> >>>>>>> yt-dev mailing list
>>> >>>>>>> yt-dev at lists.spacepope.org
>>> >>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> _______________________________________________
>>> >>>>>> yt-dev mailing list
>>> >>>>>> yt-dev at lists.spacepope.org
>>> >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> >>>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> _______________________________________________
>>> >>>>> yt-dev mailing list
>>> >>>>> yt-dev at lists.spacepope.org
>>> >>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> >>>>>
>>> >>>>
>>> >>>>
>>> >>>> _______________________________________________
>>> >>>> yt-dev mailing list
>>> >>>> yt-dev at lists.spacepope.org
>>> >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> >>>>
>>> >>>
>>> >>>
>>> >>> _______________________________________________
>>> >>> yt-dev mailing list
>>> >>> yt-dev at lists.spacepope.org
>>> >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> >>>
>>> >>
>>> >>
>>> >> _______________________________________________
>>> >> yt-dev mailing list
>>> >> yt-dev at lists.spacepope.org
>>> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> >>
>>> > _______________________________________________
>>> > yt-dev mailing list
>>> > yt-dev at lists.spacepope.org
>>> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>>
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>>
>>
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>



More information about the yt-dev mailing list