[yt-users] Problem when run yt dev in parallel on Blue Waters

Brian O'Shea bwoshea at gmail.com
Fri Nov 20 03:34:33 PST 2015


Hi Pengfei,

Sorry for the delay in response.  Here are instructions for getting yt-3 to
work on Blue Waters using mpi4py.  I believe that Rob Sisneros has gotten
yt-3 working as a module; however, it is likely that it will not be updated
very frequently, at least compared to the pace of yt development.

--Brian

-----------------------------------------------------------------------

Getting yt-3 to work on Blue Waters:

0) Get the GNU software environment set up:  “module swap PrgEnv-cray
PrgEnv-gnu” .  This will be necessary later.

1)  Install the Anaconda python distribution (
https://store.continuum.io/cshop/anaconda/) for 64-bit Linux systems.
Refresh your path so the Anaconda python’s in your path.

2)  Download mpi4py from https://bitbucket.org/mpi4py/mpi4py/, or grab it
from my home directory on Blue Waters (~bwoshea/mpi4py-1.3.1.tar.gz).

3)  Once you’ve unpacked it in your home directory, edit the file mpi.cfg,
which is in the top-level mpi4py directory, and add the following chunk of
text at the end of the file:

[cray]
mpicc = cc
mpicxx = CC
extra_link_args = -shared

then, in the top-level mpi4py directory, type:

python setup.py build --mpi=cray
python setp.py install

At this point, you should have a functional version of mpi4py that uses the
cray MPI libraries.

4)  You can then install yt via source in your home directory:

hg clone https://bitbucket.org/yt_analysis/yt
hg update yt
python setup.py develop

And it should work, though you’re going to need to modify your
LD_LIBRARY_PATH to point to your anaconda /lib directory.  You can verify
that yt and mpi4py are playing nicely together by running the scripts
hw_MPI.py and yt_parallel_test.py, both of which are sitting in my home
directory.  (Note that you can do this via batch script or through the
interactive queue, both with aprun - you can’t just do it from the login
node.)

5) and, optionally, if you want to install Rockstar using yt from source,
you do the following:

hg clone http://bitbucket.org/MatthewTurk/rockstar
cd rockstar
hg pull
hg up -C tip
make lib

and *then* make a file called rockstar.cfg that has a single line of text:
the path of the rockstar source code directory.  At that point, you need to
run “python setup.py develop” again, and yt will recognize that rockstar
has been installed and play nicely with it.  You will likely also have to
modify your LD_LIBRARY_PATH to point at your rockstar directory so it can
find the rockstar library (librockstar.so).



On Fri, Nov 20, 2015 at 3:46 AM, Pengfei Chen <madcpf at gmail.com> wrote:

> Hi Brian,
>
> Thank you very much for your reply! No, I followed the instructions here
>  http://yt-project.org/docs/dev/installing.html
> <http://yt-project.org/docs/dev/installing.html>. I need the
> yt-development, but failed when using All-in-One Installation Script to
> install it on BW, so I used miniconda to install. Could you please send me
> your installation instructions for yt on BW? I will try it out.
>
> Thanks,
> Pengfei
>
> On Thu, Nov 19, 2015 at 8:45 AM, Brian O'Shea <bwoshea at gmail.com> wrote:
>
>> Hi Pengfei,
>>
>> Did you use my installation instructions for yt on Blue Waters?  If not,
>> I can send those along.
>>
>> B
>>
>>
>> On Thu, Nov 19, 2015 at 10:02 AM, Nathan Goldbaum <nathan12343 at gmail.com>
>> wrote:
>>
>>> So it looks like either mpi4py isn't able to properly link against the
>>> MPI installation on Blue waters or properly initialize itself at runtime.
>>>
>>> In either case, this points to the issue being a problem with mpi4py,
>>> not yt.
>>>
>>> Three things to try:
>>>
>>> * check which version version of mpi4py you have installed in your
>>> yt-2.x environment and install that version in your yt-3.x environment. You
>>> may have found a bug in mpi4py 2.0, which made major changes to the mpi4py
>>> library.
>>>
>>> * contact the Blue Waters sysadmins.
>>>
>>> * contact the mpi4py developers
>>>
>>> Hope that helps,
>>>
>>> Nathan
>>>
>>>
>>> On Thursday, November 19, 2015, Pengfei Chen <madcpf at gmail.com> wrote:
>>>
>>>> Hi Nathan,
>>>>
>>>> Thank you very much for your reply! When I run that test script in
>>>> parallel with 16 cores, I get 16 0s. Any suggestions to fix this?
>>>>
>>>> Thanks,
>>>> Pengfei
>>>>
>>>>
>>>>
>>>> On Wed, Nov 18, 2015 at 3:40 PM, Nathan Goldbaum <nathan12343 at gmail.com
>>>> > wrote:
>>>>
>>>>> Hi Pengfei,
>>>>>
>>>>> Can you try to run the following test script in parallel? This will
>>>>> determine if the issue is on the yt side of things or the mpi4py side of
>>>>> things:
>>>>>
>>>>>     from mpi4py import MPI
>>>>>
>>>>>     print(MPI.COMM_WORLD.rank)
>>>>>
>>>>> -Nathan
>>>>>
>>>>> On Wed, Nov 18, 2015 at 5:36 PM, Pengfei Chen <madcpf at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I have trouble running yt in parallel on Blue Waters. I installed yt
>>>>>> using miniconda, the version of yt is
>>>>>>
>>>>>> ~/miniconda/lib $yt version
>>>>>> yt module located at:
>>>>>>
>>>>>> /u/sciteam/madcpf/miniconda/lib/python2.7/site-packages/yt-3.3.dev0-py2.7-linux-x86_64.egg
>>>>>> The current version and changeset for the code is:
>>>>>> ---
>>>>>> Version = 3.3-dev
>>>>>> Changeset = 90f900be7a36+ yt
>>>>>>
>>>>>> Then with miniconda/bin in PATH I installed mpi4py-2.0.0. But when I
>>>>>> tried to make the following simple output in parallel, I get:
>>>>>>
>>>>>> import yt
>>>>>> yt.enable_parallelism()
>>>>>> from yt.utilities.parallel_tools.parallel_analysis_interface import\
>>>>>>  parallel_objects, communication_system
>>>>>>
>>>>>> comm = communication_system.communicators[-1]
>>>>>> print comm.rank, comm.size
>>>>>>
>>>>>> 0 1
>>>>>> 0 1
>>>>>> 0 1
>>>>>> 0 1
>>>>>> 0 1
>>>>>> 0 1
>>>>>> 0 1
>>>>>> 0 1
>>>>>> ...
>>>>>>
>>>>>> When I run a similar code but with yt-2.x also on Blue Waters, I get
>>>>>> what I expect:
>>>>>>
>>>>>> 7 16
>>>>>> 15 16
>>>>>> 6 16
>>>>>> 9 16
>>>>>> 11 16
>>>>>> 8 16
>>>>>> 0 16
>>>>>> 4 16
>>>>>> ...
>>>>>>
>>>>>> I'm confused about it. Could anyone give me some suggestions please?
>>>>>>
>>>>>> Thanks,
>>>>>> Pengfei
>>>>>>
>>>>>> _______________________________________________
>>>>>> yt-users mailing list
>>>>>> yt-users at lists.spacepope.org
>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> yt-users mailing list
>>>>> yt-users at lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>
>>>>>
>>>>
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>
>>>
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>>
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20151120/10de44a0/attachment.html>


More information about the yt-users mailing list