[yt-users] inline yt in parallel

Wed Apr 10 21:34:19 PDT 2013

Hi all,

Thanks for your observations and suggestions.  I had neglected to install mpi4py, which was the original problem.  I installed that and I can run parallel yt scripts, however, I'm still having trouble with using inline yt.  I've pasted the error I now get below.  It is not very informative (to me at least); the keyboard interrupt is the symptom, not the cause of the problem, I think.  I'm doing this on trestles and I tried to use their parallel debugger ddt to get some more information.  ddt seems to indicate that one of the processes is looking for a file called mpi4py.MPI.c in the /tmp directory, which I don't really understand, and maybe is a red herring.  I don't have any problems with single processor jobs.  I installed yt using shared libraries by adding the --enable-shared flag to the configure statement for python in the install script.  I've also pasted the enzo make file that I'm using below.  I'm thinking that I somehow have messed up the libraries or include files.  If anyone has successfully used inline yt on trestles and has any advice, I'd love to hear it.

Thanks for all your help
Christine

Error:

MPI_Init: NumberOfProcessors = 3
warning: the following parameter line was not interpreted:
TestStarParticleEnergy  =  0.00104392468495
warning: the following parameter line was not interpreted:
TestStarParticleDensity  =  1.0
warning: the following parameter line was not interpreted:
TestStarParticleStarMass  =  100.0
****** ReadUnits:  2.748961e+37 1.000000e-24 3.018025e+20 3.150000e+13 *******
Global Dir set to .
Initialdt in ReadParameterFile = 4.815337e-05
InitializeNew: Starting problem initialization.
Central Mass: 6813.382812 
Allocated 1 particles
Initialize Exterior
ExtBndry: BoundaryRank = 3
ExtBndry: GridDimension = 104 104 104 
ExtBndry: NumberOfBaryonFields = 6
InitializeExternalBoundaryFace
SimpleConstantBoundary FALSE
End of set exterior
InitializeNew: Initial grid hierarchy set
InitializeNew: Partition Initial Grid 0
Enter CommunicationPartitionGrid.
PartitionGrid (on all processors): Layout = 1 1 3
NumberOfNewGrids = 3
GridDims[0]:  98
GridDims[1]:  98
GridDims[2]:  33 32 33
StartIndex[0]:  0
StartIndex[1]:  0
StartIndex[2]:  0 33 65
Call ZeroSUS on TopGrid
ENZO_layout 1 x 1 x 3
Grid structure: 1576
SubGrids structure: 4728
Re-set Unigrid = 0
Grid distribution
Delete OldGrid
OldGrid deleted
Exit CommunicationPartitionGrid.
InitializeNew: Finished problem initialization.
Initializing Python interface
Successfully read in parameter file StarParticleTest.enzo.
INITIALIZATION TIME =   9.38615084e-01
Beginning parallel import block.
MPI process (rank: 1) terminated unexpectedly on trestles-12-20.local
Exit code -5 signaled from trestles-12-20
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "./user_script.py", line 1, in <module>
    from yt.pmods import *
  File "/home/csimpson/yt-x86_64-shared/src/yt-hg/yt/pmods.py", line 364, in <module>
    from yt.mods import *
  File "/home/csimpson/yt-x86_64-shared/src/yt-hg/yt/pmods.py", line 234, in __import_hook__
    q, tail = __find_head_package__(parent, name)
  File "/home/csimpson/yt-x86_64-shared/src/yt-hg/yt/pmods.py", line 323, in __find_head_package__
    q = __import_module__(head, qname, parent)
  File "/home/csimpson/yt-x86_64-shared/src/yt-hg/yt/pmods.py", line 268, in __import_module__
    pathname,stuff,ierror = mpi.bcast((pathname,stuff,ierror))
  File "/home/csimpson/yt-x86_64-shared/src/yt-hg/yt/pmods.py", line 201, in bcast
    return MPI.COMM_WORLD.bcast(obj,root)
KeyboardInterrupt
Caught fatal exception:

   'Importing user_script failed!'
at InitializePythonInterface.C:108

Backtrace:

BT symbol: ./enzo.exe [0x41ff8a]
BT symbol: ./enzo.exe [0x727e14]
BT symbol: ./enzo.exe [0x421147]
BT symbol: /lib64/libc.so.6(__libc_start_main+0xf4) [0x3c0121d994]
BT symbol: ./enzo.exe(__gxx_personality_v0+0x3d9) [0x41fea9]
terminate called after throwing an instance of 'EnzoFatalException'

Make file:

#=======================================================================
#
# FILE:        Make.mach.trestles
#
# DESCRIPTION: Makefile settings for the Trestles Resource at SDSC/UCSD
#
# AUTHOR:      John Wise (jwise at astro.princeton.edu)
#
# DATE:        07 Dec 2010
#
#
#=======================================================================

MACH_TEXT  = Trestles
MACH_VALID = 1
MACH_FILE  = Make.mach.trestles

MACHINE_NOTES = "MACHINE_NOTES for Trestles at SDSC/UCSD: \
        Load these modules, \
        'module add intel/11.1 mvapich2/1.5.1p1'"

#-----------------------------------------------------------------------
# Compiler settings
#-----------------------------------------------------------------------

LOCAL_MPI_INSTALL = /home/diag/opt/mvapich2/1.5.1p1/intel/
LOCAL_PYTHON_INSTALL = /home/csimpson/yt-x86_64-shared/
#LOCAL_COMPILER_DIR = /opt/pgi/linux86-64/10.5
LOCAL_COMPILER_DIR = /opt/intel/Compiler/11.1/072
LOCAL_HYPRE_INSTALL = 

# With MPI

MACH_CPP       = cpp
MACH_CC_MPI    = $(LOCAL_MPI_INSTALL)/bin/mpicc # C compiler when using MPI
MACH_CXX_MPI   = $(LOCAL_MPI_INSTALL)/bin/mpicxx # C++ compiler when using MPI
MACH_FC_MPI    = $(LOCAL_MPI_INSTALL)/bin/mpif90 # Fortran 77 compiler when using MPI
MACH_F90_MPI   = $(LOCAL_MPI_INSTALL)/bin/mpif90 # Fortran 90 compiler when using MPI
MACH_LD_MPI    = $(LOCAL_MPI_INSTALL)/bin/mpicxx # Linker when using MPI

# Without MPI

MACH_CC_NOMPI  = $(LOCAL_COMPILER_DIR)/bin/intel64/icc # C compiler when not using MPI
MACH_CXX_NOMPI = $(LOCAL_COMPILER_DIR)/bin/intel64/icpc # C++ compiler when not using MPI
MACH_FC_NOMPI  = $(LOCAL_COMPILER_DIR)/bin/intel64/ifort # Fortran 77 compiler when not using MPI
MACH_F90_NOMPI = $(LOCAL_COMPILER_DIR)/bin/intel64/ifort # Fortran 90 compiler when not using MPI
MACH_LD_NOMPI  = $(LOCAL_COMPILER_DIR)/bin/intel64/icpc # Linker when not using MPI

#-----------------------------------------------------------------------
# Machine-dependent defines
#-----------------------------------------------------------------------
# Defines for the architecture; e.g. -DSUN, -DLINUX, etc.
MACH_DEFINES   = -DLINUX -DH5_USE_16_API

#-----------------------------------------------------------------------
# Compiler flag settings
#-----------------------------------------------------------------------

MACH_CPPFLAGS = -P -traditional 
MACH_CFLAGS   = 
MACH_CXXFLAGS =
MACH_FFLAGS   = 
MACH_F90FLAGS = 
MACH_LDFLAGS  = 

#-----------------------------------------------------------------------
# Precision-related flags
#-----------------------------------------------------------------------

MACH_FFLAGS_INTEGER_32 = -i4
MACH_FFLAGS_INTEGER_64 = -i8
MACH_FFLAGS_REAL_32    = -r4
MACH_FFLAGS_REAL_64    = -r8

#-----------------------------------------------------------------------
# Optimization flags
#-----------------------------------------------------------------------

MACH_OPT_WARN        = -Wall # Flags for verbose compiler warnings
MACH_OPT_DEBUG       = -O0 -g # Flags for debugging
# Flags for high conservative optimization
#MACH_OPT_HIGH        = -O1 -ftz -mieee-fp -fp-speculation=off -prec-sqrt -prec-div
MACH_OPT_HIGH        = -O2
# Note that this breaks determinism, which is why it's commented out!
#
MACH_OPT_AGGRESSIVE  = -O3 # Flags for aggressive optimization
# This is the best we can do, from what I can tell.
#MACH_OPT_AGGRESSIVE  = -O1 -ftz -mieee-fp -fp-speculation=off -prec-sqrt -prec-div

#-----------------------------------------------------------------------
# Includes
#-----------------------------------------------------------------------

LOCAL_INCLUDES_MPI    = 
LOCAL_INCLUDES_HDF5   = -I/home/csimpson/yt-x86_64-shared/include # HDF5 includes
LOCAL_INCLUDES_HYPRE  = 
LOCAL_INCLUDES_PAPI   = # PAPI includes
LOCAL_INCLUDES_PYTHON = -I$(LOCAL_PYTHON_INSTALL)/include/python2.7 \
                        -I$(LOCAL_PYTHON_INSTALL)/lib/python2.7/site-packages/numpy/core/include

MACH_INCLUDES         = $(LOCAL_INCLUDES_HDF5)
MACH_INCLUDES_PYTHON  = $(LOCAL_INCLUDES_PYTHON)
MACH_INCLUDES_MPI     = $(LOCAL_INCLUDES_MPI)
MACH_INCLUDES_HYPRE   = $(LOCAL_INCLUDES_HYPRE)
MACH_INCLUDES_PAPI    = $(LOCAL_INCLUDES_PAPI)

#-----------------------------------------------------------------------
# Libraries
#-----------------------------------------------------------------------

LOCAL_LIBS_MPI    = 
LOCAL_LIBS_HDF5   = -L/home/csimpson/yt-x86_64-shared/lib -lhdf5 # HDF5 libraries
LOCAL_LIBS_HYPRE  = 
LOCAL_LIBS_PAPI   = # PAPI libraries
LOCAL_LIBS_PYTHON  = -L$(LOCAL_PYTHON_INSTALL)/lib -lpython2.7 \
                     -lreadline -ltermcap -lutil

#LOCAL_LIBS_MACH   = -L$(LOCAL_COMPILER_DIR)/lib \
#                       -lpgf90 -lpgf90_rpm1 -lpgf902 -lpgf90rtl -lpgftnrtl -lrt
LOCAL_LIBS_MACH  = -L$(LOCAL_COMPILER_DIR)/lib/intel64 -lifcore -lifport 

MACH_LIBS         = $(LOCAL_LIBS_HDF5) $(LOCAL_LIBS_MACH)
MACH_LIBS_MPI     = $(LOCAL_LIBS_MPI)
MACH_LIBS_HYPRE   = $(LOCAL_LIBS_HYPRE)
MACH_LIBS_PAPI    = $(LOCAL_LIBS_PAPI)
MACH_LIBS_PYTHON  = $(LOCAL_LIBS_PYTHON)