[yt-users] yt install_script vs Blue Waters SSL

[Brian O'Shea]

Hi Stuart,

I ran into exactly this problem, and ultimately managed to get yt-dev
installed (and working in parallel) using Anaconda. Instructions are
attached; hopefully they'll work for you.

--Brian

-----------------------------------------------------------------------

Getting yt-3 to work on Blue Waters:

0) Get the GNU software environment set up:  “module swap PrgEnv-cray
PrgEnv-gnu” .  This will be necessary later.

1)  Install the Anaconda python distribution (
https://store.continuum.io/cshop/anaconda/) for 64-bit Linux systems.
Refresh your path so the Anaconda python’s in your path.

2)  Download mpi4py from https://bitbucket.org/mpi4py/mpi4py/, or grab it
from my home directory on Blue Waters (~bwoshea/mpi4py-1.3.1.tar.gz).

3)  Once you’ve unpacked it in your home directory, edit the file mpi.cfg,
which is in the top-level mpi4py directory, and add the following chunk of
text at the end of the file:

[cray]
mpicc = cc
mpicxx = CC
extra_link_args = -shared

then, in the top-level mpi4py directory, type:

python setup.py build --mpi=cray
python setp.py install

At this point, you should have a functional version of mpi4py that uses the
cray MPI libraries.

4)  You can then install yt via source in your home directory:

hg clone https://bitbucket.org/yt_analysis/yt
hg update yt
python setup.py develop

And it should work, though you’re going to need to modify your
LD_LIBRARY_PATH to point to your anaconda /lib directory.  You can verify
that yt and mpi4py are playing nicely together by running the scripts
hw_MPI.py and yt_parallel_test.py, both of which are sitting in my home
directory.  (Note that you can do this via batch script or through the
interactive queue, both with aprun - you can’t just do it from the login
node.)

5) and, optionally, if you want to install Rockstar using yt from source,
you do the following:

hg clone http://bitbucket.org/MatthewTurk/rockstar
cd rockstar
hg pull
hg up -C tip
make lib

and *then* make a file called rockstar.cfg that has a single line of text:
the path of the rockstar source code directory.  At that point, you need to
run “python setup.py develop” again, and yt will recognize that rockstar
has been installed and play nicely with it.  You will likely also have to
modify your LD_LIBRARY_PATH to point at your rockstar directory so it can
find the rockstar library (librockstar.so).



On Fri, Jan 8, 2016 at 9:30 PM, Levy, Stuart A <salevy at illinois.edu> wrote:

> Hello yt people,
>
> I'm trying to install current yt-3.3dev on Blue Waters at NCSA, using the
> install_script.sh route.
>
> After hacking around the fact that it has an obsolete & unsafe openssl
> library as the default (I needed to pass LDFLAGS/CPPFLAGS to the python
> build to set the -L and -rpath to the non-default usable openssl), I'm
> having lots of trouble getting other component installs to work.   The
> yt_install.log has things like
>
> Processing dependencies for matplotlib==1.4.3
> Searching for pytz
> Reading https://pypi.python.org/simple/pytz/
> Download error on https://pypi.python.org/simple/pytz/: [SSL:
> CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581) -- Some
> packages may not be found!
> Couldn't find index page for 'pytz' (maybe misspelled?)
> Scanning index of all packages (this may take a while)
> Reading https://pypi.python.org/simple/
> Download error on https://pypi.python.org/simple/: [SSL:
> CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581) -- Some
> packages may not be found!
> No local packages or download links found for pytz
> error: Could not find suitable distribution for Requirement.parse('pytz')
>
>
> Yet when I run what seems like the same install by hand  ($DIR/bin/pip -v
> install pytz) it works fine, without error.
>
> I can then re-run install_script.sh, until it reaches the next dependency
> and fails, etc.   Each cycle takes 5-10 minutes so it's pretty tedious.
>
> ***=> I don't understand why the internal-to-matplotlib-setup call to pip
> fails, while it succeeds when I run it by hand.
>
>
> A suspicion: could the CERTIFICATE_VERIFY_FAILED errors be hinting that
> Blue Waters has an https gateway to the outside world that rewrites SSL
> certificates?   I don't know whether that's true.   This page suggested a
> way to grab the needed root certificate and point env var PIP_CERT at it...
>
> http://stackoverflow.com/questions/25981703/pip-install-fails-with-connection-error-ssl-certificate-verify-failed-certi
> but trying that didn't help in this case.   The yt-installed "pip" works
> fine when run by hand, but whatever matplotlib's setup.py is running still
> fails.
>
>
>
>
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20160109/365ef305/attachment.html>
-------------- next part --------------
_______________________________________________
yt-users mailing list
yt-users at lists.spacepope.org
http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org