[yt-users] inline yt in parallel
Matthew Turk
matthewturk at gmail.com
Wed Apr 10 23:55:33 PDT 2013
Hi Christine,
On Thu, Apr 11, 2013 at 12:34 AM, Christine Simpson
<csimpson at astro.columbia.edu> wrote:
> Hi all,
>
> Thanks for your observations and suggestions. I had neglected to install mpi4py, which was the original problem. I installed that and I can run parallel yt scripts, however, I'm still having trouble with using inline yt. I've pasted the error I now get below. It is not very informative (to me at least); the keyboard interrupt is the symptom, not the cause of the problem, I think. I'm doing this on trestles and I tried to use their parallel debugger ddt to get some more information. ddt seems to indicate that one of the processes is looking for a file called mpi4py.MPI.c in the /tmp directory, which I don't really understand, and maybe is a red herring. I don't have any problems with single processor jobs. I installed yt using shared libraries by adding the --enable-shared flag to the configure statement for python in the install script. I've also pasted the enzo make file that I'm using below. I'm thinking that I somehow have messed up the libraries or include files
> . If anyone has successfully used inline yt on trestles and has any advice, I'd love to hear it.
>
So this could probably be better covered by the documentation, but the
inline yt process looks for a script called user_script.py in the yt
directory, within which it will call the main() function. This
function can get access to the in-memory output by doing something
like "pf = EnzoStaticOutputInMemory()" which will query the
appropriate items. Note that you can't access raw data like
"sphere['Density']" but you can do operations like
"sphere.quantities['Extrema']('Density')" and so on; anything that
uses an opaque object is fine, but arrays of concatenated data
generally aren't.
If you do have the script "user_script.py" in your directory, then
this generally means that there's a syntax error or something else
preventing it from being imported. I think if you have gotten this
far and you don't have user_script.py, you probably are fine for the
libraries and so on. If you do have it, are you able to run it with
"python2.7 user_script.py" ?
-Matt
> Thanks for all your help
> Christine
>
> Error:
>
> MPI_Init: NumberOfProcessors = 3
> warning: the following parameter line was not interpreted:
> TestStarParticleEnergy = 0.00104392468495
> warning: the following parameter line was not interpreted:
> TestStarParticleDensity = 1.0
> warning: the following parameter line was not interpreted:
> TestStarParticleStarMass = 100.0
> ****** ReadUnits: 2.748961e+37 1.000000e-24 3.018025e+20 3.150000e+13 *******
> Global Dir set to .
> Initialdt in ReadParameterFile = 4.815337e-05
> InitializeNew: Starting problem initialization.
> Central Mass: 6813.382812
> Allocated 1 particles
> Initialize Exterior
> ExtBndry: BoundaryRank = 3
> ExtBndry: GridDimension = 104 104 104
> ExtBndry: NumberOfBaryonFields = 6
> InitializeExternalBoundaryFace
> SimpleConstantBoundary FALSE
> End of set exterior
> InitializeNew: Initial grid hierarchy set
> InitializeNew: Partition Initial Grid 0
> Enter CommunicationPartitionGrid.
> PartitionGrid (on all processors): Layout = 1 1 3
> NumberOfNewGrids = 3
> GridDims[0]: 98
> GridDims[1]: 98
> GridDims[2]: 33 32 33
> StartIndex[0]: 0
> StartIndex[1]: 0
> StartIndex[2]: 0 33 65
> Call ZeroSUS on TopGrid
> ENZO_layout 1 x 1 x 3
> Grid structure: 1576
> SubGrids structure: 4728
> Re-set Unigrid = 0
> Grid distribution
> Delete OldGrid
> OldGrid deleted
> Exit CommunicationPartitionGrid.
> InitializeNew: Finished problem initialization.
> Initializing Python interface
> Successfully read in parameter file StarParticleTest.enzo.
> INITIALIZATION TIME = 9.38615084e-01
> Beginning parallel import block.
> MPI process (rank: 1) terminated unexpectedly on trestles-12-20.local
> Exit code -5 signaled from trestles-12-20
> Traceback (most recent call last):
> File "<string>", line 1, in <module>
> File "./user_script.py", line 1, in <module>
> from yt.pmods import *
> File "/home/csimpson/yt-x86_64-shared/src/yt-hg/yt/pmods.py", line 364, in <module>
> from yt.mods import *
> File "/home/csimpson/yt-x86_64-shared/src/yt-hg/yt/pmods.py", line 234, in __import_hook__
> q, tail = __find_head_package__(parent, name)
> File "/home/csimpson/yt-x86_64-shared/src/yt-hg/yt/pmods.py", line 323, in __find_head_package__
> q = __import_module__(head, qname, parent)
> File "/home/csimpson/yt-x86_64-shared/src/yt-hg/yt/pmods.py", line 268, in __import_module__
> pathname,stuff,ierror = mpi.bcast((pathname,stuff,ierror))
> File "/home/csimpson/yt-x86_64-shared/src/yt-hg/yt/pmods.py", line 201, in bcast
> return MPI.COMM_WORLD.bcast(obj,root)
> KeyboardInterrupt
> Caught fatal exception:
>
> 'Importing user_script failed!'
> at InitializePythonInterface.C:108
>
> Backtrace:
>
> BT symbol: ./enzo.exe [0x41ff8a]
> BT symbol: ./enzo.exe [0x727e14]
> BT symbol: ./enzo.exe [0x421147]
> BT symbol: /lib64/libc.so.6(__libc_start_main+0xf4) [0x3c0121d994]
> BT symbol: ./enzo.exe(__gxx_personality_v0+0x3d9) [0x41fea9]
> terminate called after throwing an instance of 'EnzoFatalException'
>
> Make file:
>
> #=======================================================================
> #
> # FILE: Make.mach.trestles
> #
> # DESCRIPTION: Makefile settings for the Trestles Resource at SDSC/UCSD
> #
> # AUTHOR: John Wise (jwise at astro.princeton.edu)
> #
> # DATE: 07 Dec 2010
> #
> #
> #=======================================================================
>
> MACH_TEXT = Trestles
> MACH_VALID = 1
> MACH_FILE = Make.mach.trestles
>
> MACHINE_NOTES = "MACHINE_NOTES for Trestles at SDSC/UCSD: \
> Load these modules, \
> 'module add intel/11.1 mvapich2/1.5.1p1'"
>
> #-----------------------------------------------------------------------
> # Compiler settings
> #-----------------------------------------------------------------------
>
> LOCAL_MPI_INSTALL = /home/diag/opt/mvapich2/1.5.1p1/intel/
> LOCAL_PYTHON_INSTALL = /home/csimpson/yt-x86_64-shared/
> #LOCAL_COMPILER_DIR = /opt/pgi/linux86-64/10.5
> LOCAL_COMPILER_DIR = /opt/intel/Compiler/11.1/072
> LOCAL_HYPRE_INSTALL =
>
> # With MPI
>
> MACH_CPP = cpp
> MACH_CC_MPI = $(LOCAL_MPI_INSTALL)/bin/mpicc # C compiler when using MPI
> MACH_CXX_MPI = $(LOCAL_MPI_INSTALL)/bin/mpicxx # C++ compiler when using MPI
> MACH_FC_MPI = $(LOCAL_MPI_INSTALL)/bin/mpif90 # Fortran 77 compiler when using MPI
> MACH_F90_MPI = $(LOCAL_MPI_INSTALL)/bin/mpif90 # Fortran 90 compiler when using MPI
> MACH_LD_MPI = $(LOCAL_MPI_INSTALL)/bin/mpicxx # Linker when using MPI
>
> # Without MPI
>
> MACH_CC_NOMPI = $(LOCAL_COMPILER_DIR)/bin/intel64/icc # C compiler when not using MPI
> MACH_CXX_NOMPI = $(LOCAL_COMPILER_DIR)/bin/intel64/icpc # C++ compiler when not using MPI
> MACH_FC_NOMPI = $(LOCAL_COMPILER_DIR)/bin/intel64/ifort # Fortran 77 compiler when not using MPI
> MACH_F90_NOMPI = $(LOCAL_COMPILER_DIR)/bin/intel64/ifort # Fortran 90 compiler when not using MPI
> MACH_LD_NOMPI = $(LOCAL_COMPILER_DIR)/bin/intel64/icpc # Linker when not using MPI
>
> #-----------------------------------------------------------------------
> # Machine-dependent defines
> #-----------------------------------------------------------------------
> # Defines for the architecture; e.g. -DSUN, -DLINUX, etc.
> MACH_DEFINES = -DLINUX -DH5_USE_16_API
>
> #-----------------------------------------------------------------------
> # Compiler flag settings
> #-----------------------------------------------------------------------
>
>
> MACH_CPPFLAGS = -P -traditional
> MACH_CFLAGS =
> MACH_CXXFLAGS =
> MACH_FFLAGS =
> MACH_F90FLAGS =
> MACH_LDFLAGS =
>
> #-----------------------------------------------------------------------
> # Precision-related flags
> #-----------------------------------------------------------------------
>
> MACH_FFLAGS_INTEGER_32 = -i4
> MACH_FFLAGS_INTEGER_64 = -i8
> MACH_FFLAGS_REAL_32 = -r4
> MACH_FFLAGS_REAL_64 = -r8
>
> #-----------------------------------------------------------------------
> # Optimization flags
> #-----------------------------------------------------------------------
>
> MACH_OPT_WARN = -Wall # Flags for verbose compiler warnings
> MACH_OPT_DEBUG = -O0 -g # Flags for debugging
> # Flags for high conservative optimization
> #MACH_OPT_HIGH = -O1 -ftz -mieee-fp -fp-speculation=off -prec-sqrt -prec-div
> MACH_OPT_HIGH = -O2
> # Note that this breaks determinism, which is why it's commented out!
> #
> MACH_OPT_AGGRESSIVE = -O3 # Flags for aggressive optimization
> # This is the best we can do, from what I can tell.
> #MACH_OPT_AGGRESSIVE = -O1 -ftz -mieee-fp -fp-speculation=off -prec-sqrt -prec-div
>
> #-----------------------------------------------------------------------
> # Includes
> #-----------------------------------------------------------------------
>
> LOCAL_INCLUDES_MPI =
> LOCAL_INCLUDES_HDF5 = -I/home/csimpson/yt-x86_64-shared/include # HDF5 includes
> LOCAL_INCLUDES_HYPRE =
> LOCAL_INCLUDES_PAPI = # PAPI includes
> LOCAL_INCLUDES_PYTHON = -I$(LOCAL_PYTHON_INSTALL)/include/python2.7 \
> -I$(LOCAL_PYTHON_INSTALL)/lib/python2.7/site-packages/numpy/core/include
>
> MACH_INCLUDES = $(LOCAL_INCLUDES_HDF5)
> MACH_INCLUDES_PYTHON = $(LOCAL_INCLUDES_PYTHON)
> MACH_INCLUDES_MPI = $(LOCAL_INCLUDES_MPI)
> MACH_INCLUDES_HYPRE = $(LOCAL_INCLUDES_HYPRE)
> MACH_INCLUDES_PAPI = $(LOCAL_INCLUDES_PAPI)
>
> #-----------------------------------------------------------------------
> # Libraries
> #-----------------------------------------------------------------------
>
> LOCAL_LIBS_MPI =
> LOCAL_LIBS_HDF5 = -L/home/csimpson/yt-x86_64-shared/lib -lhdf5 # HDF5 libraries
> LOCAL_LIBS_HYPRE =
> LOCAL_LIBS_PAPI = # PAPI libraries
> LOCAL_LIBS_PYTHON = -L$(LOCAL_PYTHON_INSTALL)/lib -lpython2.7 \
> -lreadline -ltermcap -lutil
>
> #LOCAL_LIBS_MACH = -L$(LOCAL_COMPILER_DIR)/lib \
> # -lpgf90 -lpgf90_rpm1 -lpgf902 -lpgf90rtl -lpgftnrtl -lrt
> LOCAL_LIBS_MACH = -L$(LOCAL_COMPILER_DIR)/lib/intel64 -lifcore -lifport
>
>
> MACH_LIBS = $(LOCAL_LIBS_HDF5) $(LOCAL_LIBS_MACH)
> MACH_LIBS_MPI = $(LOCAL_LIBS_MPI)
> MACH_LIBS_HYPRE = $(LOCAL_LIBS_HYPRE)
> MACH_LIBS_PAPI = $(LOCAL_LIBS_PAPI)
> MACH_LIBS_PYTHON = $(LOCAL_LIBS_PYTHON)
>
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
More information about the yt-users
mailing list