[yt-users] Problem when run yt dev in parallel on Blue Waters

Pengfei Chen madcpf at gmail.com
Mon Nov 23 12:13:35 PST 2015 

Hi Brian,

It works perfectly! Thank you so much!
I've saved the instructions for later use.

Thanks,
Pengfei


On Fri, Nov 20, 2015 at 3:34 AM, Brian O'Shea <bwoshea at gmail.com> wrote:

> Hi Pengfei,
>
> Sorry for the delay in response.  Here are instructions for getting yt-3
> to work on Blue Waters using mpi4py.  I believe that Rob Sisneros has
> gotten yt-3 working as a module; however, it is likely that it will not be
> updated very frequently, at least compared to the pace of yt development.
>
> --Brian
>
> -----------------------------------------------------------------------
>
> Getting yt-3 to work on Blue Waters:
>
> 0) Get the GNU software environment set up:  “module swap PrgEnv-cray
> PrgEnv-gnu” .  This will be necessary later.
>
> 1)  Install the Anaconda python distribution (
> https://store.continuum.io/cshop/anaconda/) for 64-bit Linux systems.
> Refresh your path so the Anaconda python’s in your path.
>
> 2)  Download mpi4py from https://bitbucket.org/mpi4py/mpi4py/, or grab it
> from my home directory on Blue Waters (~bwoshea/mpi4py-1.3.1.tar.gz).
>
> 3)  Once you’ve unpacked it in your home directory, edit the file mpi.cfg,
> which is in the top-level mpi4py directory, and add the following chunk of
> text at the end of the file:
>
> [cray]
> mpicc = cc
> mpicxx = CC
> extra_link_args = -shared
>
> then, in the top-level mpi4py directory, type:
>
> python setup.py build --mpi=cray
> python setp.py install
>
> At this point, you should have a functional version of mpi4py that uses
> the cray MPI libraries.
>
> 4)  You can then install yt via source in your home directory:
>
> hg clone https://bitbucket.org/yt_analysis/yt
> hg update yt
> python setup.py develop
>
> And it should work, though you’re going to need to modify your
> LD_LIBRARY_PATH to point to your anaconda /lib directory.  You can verify
> that yt and mpi4py are playing nicely together by running the scripts
> hw_MPI.py and yt_parallel_test.py, both of which are sitting in my home
> directory.  (Note that you can do this via batch script or through the
> interactive queue, both with aprun - you can’t just do it from the login
> node.)
>
> 5) and, optionally, if you want to install Rockstar using yt from source,
> you do the following:
>
> hg clone http://bitbucket.org/MatthewTurk/rockstar
> cd rockstar
> hg pull
> hg up -C tip
> make lib
>
> and *then* make a file called rockstar.cfg that has a single line of text:
> the path of the rockstar source code directory.  At that point, you need to
> run “python setup.py develop” again, and yt will recognize that rockstar
> has been installed and play nicely with it.  You will likely also have to
> modify your LD_LIBRARY_PATH to point at your rockstar directory so it can
> find the rockstar library (librockstar.so).
>
>
>
> On Fri, Nov 20, 2015 at 3:46 AM, Pengfei Chen <madcpf at gmail.com> wrote:
>
>> Hi Brian,
>>
>> Thank you very much for your reply! No, I followed the instructions here
>>  http://yt-project.org/docs/dev/installing.html
>> <http://yt-project.org/docs/dev/installing.html>. I need the
>> yt-development, but failed when using All-in-One Installation Script to
>> install it on BW, so I used miniconda to install. Could you please send me
>> your installation instructions for yt on BW? I will try it out.
>>
>> Thanks,
>> Pengfei
>>
>> On Thu, Nov 19, 2015 at 8:45 AM, Brian O'Shea <bwoshea at gmail.com> wrote:
>>
>>> Hi Pengfei,
>>>
>>> Did you use my installation instructions for yt on Blue Waters?  If not,
>>> I can send those along.
>>>
>>> B
>>>
>>>
>>> On Thu, Nov 19, 2015 at 10:02 AM, Nathan Goldbaum <nathan12343 at gmail.com
>>> > wrote:
>>>
>>>> So it looks like either mpi4py isn't able to properly link against the
>>>> MPI installation on Blue waters or properly initialize itself at runtime.
>>>>
>>>> In either case, this points to the issue being a problem with mpi4py,
>>>> not yt.
>>>>
>>>> Three things to try:
>>>>
>>>> * check which version version of mpi4py you have installed in your
>>>> yt-2.x environment and install that version in your yt-3.x environment. You
>>>> may have found a bug in mpi4py 2.0, which made major changes to the mpi4py
>>>> library.
>>>>
>>>> * contact the Blue Waters sysadmins.
>>>>
>>>> * contact the mpi4py developers
>>>>
>>>> Hope that helps,
>>>>
>>>> Nathan
>>>>
>>>>
>>>> On Thursday, November 19, 2015, Pengfei Chen <madcpf at gmail.com> wrote:
>>>>
>>>>> Hi Nathan,
>>>>>
>>>>> Thank you very much for your reply! When I run that test script in
>>>>> parallel with 16 cores, I get 16 0s. Any suggestions to fix this?
>>>>>
>>>>> Thanks,
>>>>> Pengfei
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Nov 18, 2015 at 3:40 PM, Nathan Goldbaum <
>>>>> nathan12343 at gmail.com> wrote:
>>>>>
>>>>>> Hi Pengfei,
>>>>>>
>>>>>> Can you try to run the following test script in parallel? This will
>>>>>> determine if the issue is on the yt side of things or the mpi4py side of
>>>>>> things:
>>>>>>
>>>>>>     from mpi4py import MPI
>>>>>>
>>>>>>     print(MPI.COMM_WORLD.rank)
>>>>>>
>>>>>> -Nathan
>>>>>>
>>>>>> On Wed, Nov 18, 2015 at 5:36 PM, Pengfei Chen <madcpf at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I have trouble running yt in parallel on Blue Waters. I installed yt
>>>>>>> using miniconda, the version of yt is
>>>>>>>
>>>>>>> ~/miniconda/lib $yt version
>>>>>>> yt module located at:
>>>>>>>
>>>>>>> /u/sciteam/madcpf/miniconda/lib/python2.7/site-packages/yt-3.3.dev0-py2.7-linux-x86_64.egg
>>>>>>> The current version and changeset for the code is:
>>>>>>> ---
>>>>>>> Version = 3.3-dev
>>>>>>> Changeset = 90f900be7a36+ yt
>>>>>>>
>>>>>>> Then with miniconda/bin in PATH I installed mpi4py-2.0.0. But when I
>>>>>>> tried to make the following simple output in parallel, I get:
>>>>>>>
>>>>>>> import yt
>>>>>>> yt.enable_parallelism()
>>>>>>> from yt.utilities.parallel_tools.parallel_analysis_interface import\
>>>>>>>  parallel_objects, communication_system
>>>>>>>
>>>>>>> comm = communication_system.communicators[-1]
>>>>>>> print comm.rank, comm.size
>>>>>>>
>>>>>>> 0 1
>>>>>>> 0 1
>>>>>>> 0 1
>>>>>>> 0 1
>>>>>>> 0 1
>>>>>>> 0 1
>>>>>>> 0 1
>>>>>>> 0 1
>>>>>>> ...
>>>>>>>
>>>>>>> When I run a similar code but with yt-2.x also on Blue Waters, I get
>>>>>>> what I expect:
>>>>>>>
>>>>>>> 7 16
>>>>>>> 15 16
>>>>>>> 6 16
>>>>>>> 9 16
>>>>>>> 11 16
>>>>>>> 8 16
>>>>>>> 0 16
>>>>>>> 4 16
>>>>>>> ...
>>>>>>>
>>>>>>> I'm confused about it. Could anyone give me some suggestions please?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Pengfei
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> yt-users mailing list
>>>>>>> yt-users at lists.spacepope.org
>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> yt-users mailing list
>>>>>> yt-users at lists.spacepope.org
>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>>
>>>>>>
>>>>>
>>>> _______________________________________________
>>>> yt-users mailing list
>>>> yt-users at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>
>>>>
>>>
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>
>>>
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>>
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20151123/a2b2919d/attachment.html>

More information about the yt-users mailing list