[yt-users] inline yt in parallel

Thu Apr 11 17:49:51 PDT 2013

Hi Christine,

On Thu, Apr 11, 2013 at 7:29 PM, Christine Simpson
<csimpson at astro.columbia.edu> wrote:
> Thanks so much for your help!
>
> So I did some tests.  I ran each test on 3 processors for 10 cycles and
> set PythonTopGridSkip = 1.  The hierarchy is static (so no amr).

Okay, that's good to hear -- it eliminates some common problems.

>
>> type 1:
>>
>> import numpy as np
>>
>> def main():
>>     print "Type 1"
>>
> This worked.  Type 1 printed a total of 60 times, which is
> ncycles*nproc*2 (for the 2 calls to python, one at the end of evolve
> level and the one at the end of evolve hierarchy).
>
>> type 2:
>>
>> from yt.mods import *
>>
>> def main:
>>     print "Type 2"
>>
>
> This failed.  The error I get is
>
> Successfully read in parameter file StarParticleTest.enzo.
> INITIALIZATION TIME =   4.38757896e-01
> yt : [INFO     ] 2013-04-11 13:14:08,924 Global parallel computation
> enabled: 1 / 3
> yt : [INFO     ] 2013-04-11 13:14:08,924 Global parallel computation
> enabled: 2 / 3
> yt : [INFO     ] 2013-04-11 13:14:08,924 Global parallel computation
> enabled: 0 / 3
> MPI process (rank: 0) terminated unexpectedly on trestles-1-10.local
> Exit code -5 signaled from trestles-1-10

Awesome.  This tells us a *lot* of information.  For starters,
importing numpy and having that fail is a very common use case that
can be difficult to fix.  Fortunately, that's not the case here!  :)
So here's a good start -- in the directory ~/.yt you may or may not
have a file named 'config'.  If you add to that file something like:

[yt]
loglevel: 1

then you can get a lot more info about how things proceed.  For me it
looks like this:

http://paste.yt-project.org/show/3357/

The next step is to put print statements in yt/mods.py around these areas:

http://paste.yt-project.org/show/3358/

Speciffically, I'm curious if it gets past startup tasks.  After that,
there is a series of import blocks that look like:

from yt.something \
    import flux_capacitor

or something.  Each of these blocks might be grabbing something.

Now, one very *last* thing is, can you run one last script identical
to the ones before, except test to see if h5py can be imported?  It
occurs to me that's another source of error that might be pernicious.

I'm *really* sorry this is so annoying -- getting this up and running
is a huge priority for me!  For what it's worth, part of the issue
here is that yt+sim code have to be in the same executable; the next
generation of in situ stuff will hopefully avoid this, and thus make
problems like this ... erased.  But that's a year or more off.

Anyhow, I'm really interested in getting this working for you, so
please let me know if any of that helps at all -- the unexplained
error (-5) doesn't help narrow it down too much, unfortunately.

Thanks again,

Matt

>
>
>> type 3:
>>
>> type 1:
>>
>> from mpi4py import MPI
>>
>> def main():
>>     print "Type 3"
>>
>>
>
> This works and prints out Type 3 a total of 60 times as in case 1.
>
>> One last thing to check is the output of ldd on the MPI.so file inside
>> your yt site-packages directory, under the mpi4py subdirectory, versus
>> the output of ldd for enzo.exe.  These should both link to the same
>> MPI library.
>>
> I'm not sure what the MPI library is called, but all the directory files
> listed for MPI.so from the yt site-packages directory and for the enzo
> executable look the same.  They are pasted below.
>
> So it looks like it is something with yt.mods/yt.pmods?  (I wasn't quite
> sure which one to use, by the way).  So I can try putting some print
> statements in that file to see how far it gets.  Are there likely
> problem places that I should focus on?
>
> Thanks
> Christine
>
> For MPI.so in the yt site-packages directory
> (yt-x86_64-shared)[csimpson at trestles-login1 mpi4py]$ ldd MPI.so
>         libpython2.7.so.1.0
> => /home/csimpson/yt-x86_64-shared/lib/libpython2.7.so.1.0
> (0x00002b9930541000)
>         libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b993091d000)
>         librdmacm.so.1 => /usr/lib64/librdmacm.so.1 (0x00002b9930b39000)
>         libibverbs.so.1 => /usr/lib64/libibverbs.so.1
> (0x00002b9930d3e000)
>         libibumad.so.3 => /usr/lib64/libibumad.so.3 (0x00002b9930f4b000)
>         libdl.so.2 => /lib64/libdl.so.2 (0x00002b9931152000)
>         librt.so.1 => /lib64/librt.so.1 (0x00002b9931356000)
>         libimf.so => /opt/intel/Compiler/11.1/072/lib/intel64/libimf.so
> (0x00002b993155f000)
>         libsvml.so
> => /opt/intel/Compiler/11.1/072/lib/intel64/libsvml.so
> (0x00002b99318f4000)
>         libm.so.6 => /lib64/libm.so.6 (0x00002b9931b0a000)
>         libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b9931d8d000)
>         libintlc.so.5
> => /opt/intel/Compiler/11.1/072/lib/intel64/libintlc.so.5
> (0x00002b9931f9c000)
>         libc.so.6 => /lib64/libc.so.6 (0x00002b99320da000)
>         libutil.so.1 => /lib64/libutil.so.1 (0x00002b9932431000)
>         /lib64/ld-linux-x86-64.so.2 (0x000000344ba00000)
>
> For enzo.exe:
> (yt-x86_64-shared)[csimpson at trestles-login1 mpi4py]$ ldd
> ~/temp_mom/enzo-dev-mom/src/enzo/enzo.exe
>         libhdf5.so.7 => /home/csimpson/yt-x86_64-shared/lib/libhdf5.so.7
> (0x00002b9496c96000)
>         libifcore.so.5
> => /opt/intel/Compiler/11.1/072/lib/intel64/libifcore.so.5
> (0x00002b9497139000)
>         libifport.so.5
> => /opt/intel/Compiler/11.1/072/lib/intel64/libifport.so.5
> (0x00002b94973b1000)
>         libpython2.7.so.1.0
> => /home/csimpson/yt-x86_64-shared/lib/libpython2.7.so.1.0
> (0x00002b94974ea000)
>         libreadline.so.5 => /usr/lib64/libreadline.so.5
> (0x0000003d96c00000)
>         libtermcap.so.2 => /lib64/libtermcap.so.2 (0x000000346bc00000)
>         libutil.so.1 => /lib64/libutil.so.1 (0x0000003452200000)
>         libpthread.so.0 => /lib64/libpthread.so.0 (0x000000344ca00000)
>         librdmacm.so.1 => /usr/lib64/librdmacm.so.1 (0x000000344ce00000)
>         libibverbs.so.1 => /usr/lib64/libibverbs.so.1
> (0x000000344c200000)
>         libibumad.so.3 => /usr/lib64/libibumad.so.3 (0x00002b94978c8000)
>         libdl.so.2 => /lib64/libdl.so.2 (0x000000344c600000)
>         librt.so.1 => /lib64/librt.so.1 (0x000000344da00000)
>         libm.so.6 => /lib64/libm.so.6 (0x00002b9497acf000)
>         libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00000035f8a00000)
>         libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000035f8e00000)
>         libc.so.6 => /lib64/libc.so.6 (0x000000344be00000)
>         libz.so.1 => /home/csimpson/yt-x86_64-shared/lib/libz.so.1
> (0x00002b9497d53000)
>         libimf.so => /opt/intel/Compiler/11.1/072/lib/intel64/libimf.so
> (0x00002b9497f6a000)
>         libintlc.so.5
> => /opt/intel/Compiler/11.1/072/lib/intel64/libintlc.so.5
> (0x00002b94982fe000)
>         /lib64/ld-linux-x86-64.so.2 (0x000000344ba00000)
>
>
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org