[yt-users] yt on Pleiades

Mateusz Ruszkowski mateuszr at umich.edu
Mon Oct 19 20:43:07 PDT 2015



Good call. It appears that cython was not included in miniconda3 (and was not present under miniconda3/bin/):

% conda update cython
Error: package 'cython' is not installed in /nobackupnfs2/mruszkow/miniconda3
% python test.py
Traceback (most recent call last):
  File "test.py", line 1, in <module>
    import cython
ImportError: No module named ‘python'


So I ran:

conda install cython     [ now miniconda3/bin/ does include cython ]
% python test.py
0.23.4
/nobackupnfs2/mruszkow/miniconda3/lib/python3.4/site-packages/cython.py

cd yt
python setup.py develop   

[ appears to have ran properly and, after a long list of warnings, I got: ]

[….]
Adding yt 3.3.dev0 to easy-install.pth file
Installing yt script to /nobackupnfs2/mruszkow/miniconda3/bin
Installing iyt script to /nobackupnfs2/mruszkow/miniconda3/bin

Installed /nobackupnfs2/mruszkow/yt
Processing dependencies for yt==3.3.dev0
Finished processing dependencies for yt==3.3.dev0
% 


So, how do I activate this installation? 









> On Oct 19, 2015, at 10:06 PM, Nathan Goldbaum <nathan12343 at gmail.com> wrote:
> 
> 
> 
> On Mon, Oct 19, 2015 at 6:52 PM, Mateusz Ruszkowski <mateuszr at umich.edu> wrote:
> 
> You were right that I didn’t have python from miniconda in my PATH and so I was picking up the (old) system version of python. I prepended PATH variable and “which python” now points to the right version of python from miniconda. Nevertheless, the problem remains as illustrated below. I am puzzled why this is happening ... 
> 
> % pwd
> /nobackupnfs2/mruszkow
> % ls
> Miniconda3-latest-Linux-x86_64.sh
> % bash Miniconda3-latest-Linux-x86_64.sh 
> % setenv PATH /nobackupnfs2/mruszkow/miniconda3/bin:${PATH}
> % echo $PATH
> % which python
> % conda install yt
> % module load pkgsrc/2015Q2
> % hg clone https://bitbucket.org/yt_analysis/yt
> % ls
> Miniconda3-latest-Linux-x86_64.sh  miniconda3  yt
> % pwd
> /nobackupnfs2/mruszkow
> % cd yt/
> % hg up yt
> % which python
> /nobackupnfs2/mruszkow/miniconda3/bin/python
> % python setup.py develop
> Cython is a build-time requirement for the source tree of yt.
> Please either install yt from a provided, release tarball,
> or install Cython (version 0.22 or higher).
> You may be able to accomplish this by typing:
>      pip install -U Cython
> 
> Does it persist if you do "conda update cython"?  There should be builds of Cython 0.23.4 on linux.
> 
> If it does persist, can you paste the output of
> 
> import cython
> print(cython.__version__)
> print(cython.__file__)
> 
> Using the miniconda python.
> 
> Sorry this is so complicated, unfortunately this is wrapped up with the general awfulness of packaging in python and one of several things could be going wrong.
>  
> 
> 
> 
>> On Oct 19, 2015, at 7:53 PM, Nathan Goldbaum <nathan12343 at gmail.com> wrote:
>> 
>> You don't have the python from miniconda in your PATH and are picking up a version of python 2.6 or older from the operating system on Pleiades.
>> 
>> What's the output of "which python"? If the path returned by that command doesn't have "miniconda" in it, it's the wrong python executable. You might need to manually add miniconda's bin directory to your PATH environment variable.
>> 
>> On Mon, Oct 19, 2015 at 4:51 PM, Mateusz Ruszkowski <mateuszr at umich.edu> wrote:
>> 
>> 
>> 
>> The reason I originally loaded python_2.7.2 was to avoid the error message below. Now that python_2.7.2 is unloaded this message is back ...
>> 
>> % module unload python_2.7.2
>> % python setup.py develop
>> yt currently requires Python version 2.7
>> certain features may fail unexpectedly and silently with older versions.
>> %
>> 
>> As the next step I will try to repeat the whole process from scratch w/o using module load pkgsrc/2015Q2. Instead, I will just use standalone mercurial installation to see if this helps.
>> 
>> 
>> 
>> 
>> 
>> 
>>> On Oct 19, 2015, at 7:31 PM, Nathan Goldbaum <nathan12343 at gmail.com> wrote:
>>> 
>>> 
>>> 
>>> On Mon, Oct 19, 2015 at 4:11 PM, Mateusz Ruszkowski <mateuszr at umich.edu> wrote:
>>> 
>>> 
>>> Hi Nathan/Desika,
>>> 
>>> Thanks! Following your suggestions, I replaced "conda build conda-recipes/yt"  with "conda install yt.” I am also able to do "conda install pip" w/o problem. But I nevertheless get the same error message when I execute the last command in this series:
>>> 
>>> bash Miniconda3-latest-Linux-x86_64.sh
>>> miniconda3/bin/conda install yt                      [instead of:  miniconda3/bin/conda build conda-recipes/yt]
>>> miniconda3/bin/conda install pip
>>> hg clone https://bitbucket.org/yt_analysis/yt
>>> cd yt/
>>> hg up yt
>>> python setup.py develop
>>> 
>>> Cython is a build-time requirement for the source tree of yt.
>>> Please either install yt from a provided, release tarball,
>>> or install Cython (version 0.22 or higher).
>>> You may be able to accomplish this by typing:
>>>      pip install -U Cython
>>> 
>>> Is there anything obvious in the above command sequence that is wrong? I am wondering if this problem has anything to do with how you setup various modules on Pleiades. For example, in order to use mercuria,l I do "module load pkgsrc/2015Q2”  I also do “module load python_2.7.2”  The pkgsrc/2015Q2 package loads a number of obsolete modules and perhaps they are messing things up?
>>> 
>>> Yes, that is possible.
>>> 
>>> You want to make sure that you are running the miniconda python and not the python loaded by the module. You can check by doing "which python". If you're using miniconda, you want to unload the python_2.7.2 module.
>>>  
>>> 
>>>    Thanks,
>>>      Mateusz
>>> 
>>> 
>>>    
>>>   
>>> 
>>> 
>>>> On Oct 19, 2015, at 5:05 PM, Nathan Goldbaum <nathan12343 at gmail.com> wrote:
>>>> 
>>>> 
>>>> 
>>>> On Mon, Oct 19, 2015 at 1:42 PM, Mateusz Ruszkowski <mateuszr at umich.edu> wrote:
>>>> 
>>>> 
>>>>    Hi Nathan,
>>>> 
>>>> Thanks for your help. I think also this almost works ...  I downloaded the latest miniconda and followed this procedure:
>>>> 
>>>> 
>>>> bash Miniconda3-latest-Linux-x86_64.sh
>>>> miniconda3/bin/conda build conda-recipes/yt
>>>> 
>>>> Here you want to do "conda install yt" to get all of yt's dependencies.  From there you can install the development version of yt. You can "conda install" must python packages.
>>>> 
>>>> As desika mentioned, you can also install pip with "conda install pip".
>>>> 
>>>> I'd caution against using the full anaconda distribution since anaconda's MPI will shadow the system MPI installation unless you explicitly remove it.
>>>>  
>>>> hg clone https://bitbucket.org/yt_analysis/yt
>>>> cd yt/
>>>> hy up yt
>>>> python setup.py develop
>>>> 
>>>> This resulted in:
>>>>  
>>>> Cython is a build-time requirement for the source tree of yt.
>>>> Please either install yt from a provided, release tarball,
>>>> or install Cython (version 0.22 or higher).
>>>> You may be able to accomplish this by typing:
>>>>      pip install -U Cython
>>>> 
>>>> which I am not able to install via "pip install -U Cython." So this is essentially the same error message I got when I attempted to install yt using the method described in my previous e-mail. Any idea what may still be wrong?
>>>> 
>>>> 
>>>>    Thanks,
>>>>       Mateusz
>>>> 
>>>> 
>>>> 
>>>> 
>>>>> On Oct 19, 2015, at 2:26 PM, Nathan Goldbaum <nathan12343 at gmail.com> wrote:
>>>>> 
>>>>> Hi Mateusz
>>>>> 
>>>>> Rather than using the python module on pleiades, I'd suggest setting up your python stack on your own in your home folder.
>>>>> 
>>>>> By far the quickest way to do this is to install miniconda and then install yt into the miniconda environment using "conda install yt". You could also use the yt install script, although you might have issues with that if Pleiades has an older (insecure) version of OpenSSL installed, which is sadly quite common on supercomputers these days...
>>>>> 
>>>>> Personally, I set up my python environment on Pleiades using miniconda.
>>>>> 
>>>>> -Nathan
>>>>> 
>>>>> On Mon, Oct 19, 2015 at 11:23 AM, Mateusz Ruszkowski <mateuszr at umich.edu> wrote:
>>>>> 
>>>>>   Hi,
>>>>> 
>>>>> I am trying to install yt on Pleiades. I was able to complete a number installation steps but I eventually got an error message related to Cython (Cython seems to be installed though). The error message is attached below. Does anybody have an idea what needs to be done to complete this installation? 
>>>>> 
>>>>>     Thanks,
>>>>>        Mateusz
>>>>> 
>>>>> ------------------------------------------------
>>>>> Dr. Mateusz Ruszkowski
>>>>> Associate Professor
>>>>> 
>>>>> University of Michigan 
>>>>> Department of Astronomy
>>>>> 301C West Hall
>>>>> 1085 South University Ave. 
>>>>> Ann Arbor, MI 48109-1107, USA
>>>>> 
>>>>> e-mail: mateuszr at umich.edu
>>>>> 
>>>>> 
>>>>> ———————————————————————— 
>>>>> 
>>>>> % module load pkgsrc/2014Q3
>>>>> % module load python/2.7.9
>>>>> % hg clone https://bitbucket.org/yt_analysis/yt
>>>>> warning: bitbucket.org certificate with fingerprint 46:de:34:e7:9b:18:cd:7f:ae:fd:8b:e3:bc:f4:1a:5e:38:d7:ac:24 not verified (check hostfingerprints or web.cacerts config setting)
>>>>> destination directory: yt
>>>>> requesting all changes
>>>>> adding changesets
>>>>> adding manifests
>>>>> adding file changes
>>>>> added 17568 changesets with 36510 changes to 3053 files (+3 heads)
>>>>> updating to bookmark @ on branch stable
>>>>> 1209 files updated, 0 files merged, 0 files removed, 0 files unresolved
>>>>> % cd yt
>>>>> % hg update yt
>>>>> 317 files updated, 0 files merged, 148 files removed, 0 files unresolved
>>>>> (leaving bookmark @)
>>>>> % python setup.py install --user --prefix=/nobackupnfs2/mruszkow
>>>>> Cython is a build-time requirement for the source tree of yt.
>>>>> Please either install yt from a provided, release tarball,
>>>>> or install Cython (version 0.22 or higher).
>>>>> You may be able to accomplish this by typing:
>>>>>      pip install -U Cython
>>>>> % pip install -U Cython
>>>>> Downloading/unpacking Cython from https://pypi.python.org/packages/source/C/Cython/Cython-0.23.4.tar.gz#md5=157df1f69bcec6b56fd97e0f2e057f6e
>>>>>   Downloading Cython-0.23.4.tar.gz (1.6MB): 1.6MB downloaded
>>>>>   Running setup.py (path:/tmp/pip_build_mruszkow/Cython/setup.py) egg_info for package Cython
>>>>>     Unable to find pgen, not compiling formal grammar.
>>>>>     
>>>>>     warning: no files found matching '*.pyx' under directory 'Cython/Debugger/Tests'
>>>>>     warning: no files found matching '*.pxd' under directory 'Cython/Debugger/Tests'
>>>>>     warning: no files found matching '*.h' under directory 'Cython/Debugger/Tests'
>>>>>     warning: no files found matching '*.pxd' under directory 'Cython/Utility'
>>>>> Installing collected packages: Cython
>>>>>   Found existing installation: Cython 0.21.1
>>>>>     Uninstalling Cython:
>>>>> Cleaning up...
>>>>> Exception:
>>>>> Traceback (most recent call last):
>>>>>   File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/basecommand.py", line 122, in main
>>>>>     status = self.run(options, args)
>>>>>   File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/commands/install.py", line 283, in run
>>>>>     requirement_set.install(install_options, global_options, root=options.root_path)
>>>>>   File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/req.py", line 1431, in install
>>>>>     requirement.uninstall(auto_confirm=True)
>>>>>   File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/req.py", line 598, in uninstall
>>>>>     paths_to_remove.remove(auto_confirm)
>>>>>   File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/req.py", line 1836, in remove
>>>>>     renames(path, new_path)
>>>>>   File "/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/pip/util.py", line 295, in renames
>>>>>     shutil.move(old, new)
>>>>>   File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 300, in move
>>>>>     rmtree(src)
>>>>>   File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 247, in rmtree
>>>>>     rmtree(fullname, ignore_errors, onerror)
>>>>>   File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 252, in rmtree
>>>>>     onerror(os.remove, fullname, sys.exc_info())
>>>>>   File "/nasa/pkgsrc/2014Q4/lib/python2.7/shutil.py", line 250, in rmtree
>>>>>     os.remove(fullname)
>>>>> OSError: [Errno 30] Read-only file system: '/nasa/pkgsrc/2014Q4/lib/python2.7/site-packages/Cython/Build/BuildExecutable.py'
>>>>> 
>>>>> Storing debug log for failure in /u/mruszkow/.pip/pip.log
>>>>> % 
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> yt-users mailing list
>>>>> yt-users at lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> yt-users mailing list
>>>>> yt-users at lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>> 
>>>> 
>>>> _______________________________________________
>>>> yt-users mailing list
>>>> yt-users at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>> 
>>>> 
>>>> _______________________________________________
>>>> yt-users mailing list
>>>> yt-users at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>> 
>>> 
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>> 
>>> 
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> 
>> 
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> 
>> 
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
> 
> 
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
> 
> 
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org




More information about the yt-users mailing list