[yt-users] inline yt in parallel

[Christine Simpson]

Thanks so much for your help!

So I did some tests.  I ran each test on 3 processors for 10 cycles and
set PythonTopGridSkip = 1.  The hierarchy is static (so no amr).

> type 1:
> 
> import numpy as np
> 
> def main():
>     print "Type 1"
> 
This worked.  Type 1 printed a total of 60 times, which is
ncycles*nproc*2 (for the 2 calls to python, one at the end of evolve
level and the one at the end of evolve hierarchy).

> type 2:
> 
> from yt.mods import *
> 
> def main:
>     print "Type 2"
> 

This failed.  The error I get is 

Successfully read in parameter file StarParticleTest.enzo.
INITIALIZATION TIME =   4.38757896e-01
yt : [INFO     ] 2013-04-11 13:14:08,924 Global parallel computation
enabled: 1 / 3
yt : [INFO     ] 2013-04-11 13:14:08,924 Global parallel computation
enabled: 2 / 3
yt : [INFO     ] 2013-04-11 13:14:08,924 Global parallel computation
enabled: 0 / 3
MPI process (rank: 0) terminated unexpectedly on trestles-1-10.local
Exit code -5 signaled from trestles-1-10


> type 3:
> 
> type 1:
> 
> from mpi4py import MPI
> 
> def main():
>     print "Type 3"
> 
> 

This works and prints out Type 3 a total of 60 times as in case 1.

> One last thing to check is the output of ldd on the MPI.so file inside
> your yt site-packages directory, under the mpi4py subdirectory, versus
> the output of ldd for enzo.exe.  These should both link to the same
> MPI library.
> 
I'm not sure what the MPI library is called, but all the directory files
listed for MPI.so from the yt site-packages directory and for the enzo
executable look the same.  They are pasted below.

So it looks like it is something with yt.mods/yt.pmods?  (I wasn't quite
sure which one to use, by the way).  So I can try putting some print
statements in that file to see how far it gets.  Are there likely
problem places that I should focus on?

Thanks
Christine

For MPI.so in the yt site-packages directory
(yt-x86_64-shared)[csimpson at trestles-login1 mpi4py]$ ldd MPI.so
        libpython2.7.so.1.0
=> /home/csimpson/yt-x86_64-shared/lib/libpython2.7.so.1.0
(0x00002b9930541000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b993091d000)
        librdmacm.so.1 => /usr/lib64/librdmacm.so.1 (0x00002b9930b39000)
        libibverbs.so.1 => /usr/lib64/libibverbs.so.1
(0x00002b9930d3e000)
        libibumad.so.3 => /usr/lib64/libibumad.so.3 (0x00002b9930f4b000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00002b9931152000)
        librt.so.1 => /lib64/librt.so.1 (0x00002b9931356000)
        libimf.so => /opt/intel/Compiler/11.1/072/lib/intel64/libimf.so
(0x00002b993155f000)
        libsvml.so
=> /opt/intel/Compiler/11.1/072/lib/intel64/libsvml.so
(0x00002b99318f4000)
        libm.so.6 => /lib64/libm.so.6 (0x00002b9931b0a000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b9931d8d000)
        libintlc.so.5
=> /opt/intel/Compiler/11.1/072/lib/intel64/libintlc.so.5
(0x00002b9931f9c000)
        libc.so.6 => /lib64/libc.so.6 (0x00002b99320da000)
        libutil.so.1 => /lib64/libutil.so.1 (0x00002b9932431000)
        /lib64/ld-linux-x86-64.so.2 (0x000000344ba00000)

For enzo.exe:
(yt-x86_64-shared)[csimpson at trestles-login1 mpi4py]$ ldd
~/temp_mom/enzo-dev-mom/src/enzo/enzo.exe 
        libhdf5.so.7 => /home/csimpson/yt-x86_64-shared/lib/libhdf5.so.7
(0x00002b9496c96000)
        libifcore.so.5
=> /opt/intel/Compiler/11.1/072/lib/intel64/libifcore.so.5
(0x00002b9497139000)
        libifport.so.5
=> /opt/intel/Compiler/11.1/072/lib/intel64/libifport.so.5
(0x00002b94973b1000)
        libpython2.7.so.1.0
=> /home/csimpson/yt-x86_64-shared/lib/libpython2.7.so.1.0
(0x00002b94974ea000)
        libreadline.so.5 => /usr/lib64/libreadline.so.5
(0x0000003d96c00000)
        libtermcap.so.2 => /lib64/libtermcap.so.2 (0x000000346bc00000)
        libutil.so.1 => /lib64/libutil.so.1 (0x0000003452200000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x000000344ca00000)
        librdmacm.so.1 => /usr/lib64/librdmacm.so.1 (0x000000344ce00000)
        libibverbs.so.1 => /usr/lib64/libibverbs.so.1
(0x000000344c200000)
        libibumad.so.3 => /usr/lib64/libibumad.so.3 (0x00002b94978c8000)
        libdl.so.2 => /lib64/libdl.so.2 (0x000000344c600000)
        librt.so.1 => /lib64/librt.so.1 (0x000000344da00000)
        libm.so.6 => /lib64/libm.so.6 (0x00002b9497acf000)
        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00000035f8a00000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000035f8e00000)
        libc.so.6 => /lib64/libc.so.6 (0x000000344be00000)
        libz.so.1 => /home/csimpson/yt-x86_64-shared/lib/libz.so.1
(0x00002b9497d53000)
        libimf.so => /opt/intel/Compiler/11.1/072/lib/intel64/libimf.so
(0x00002b9497f6a000)
        libintlc.so.5
=> /opt/intel/Compiler/11.1/072/lib/intel64/libintlc.so.5
(0x00002b94982fe000)
        /lib64/ld-linux-x86-64.so.2 (0x000000344ba00000)