[yt-users] Problem with ParallellAnalysisInterface._mpi_allsum

[Matthew Turk]

Hi Anthony,

Using the grid cutting method, which I thought might be causing
problems, I was again unable to reproduce the issue.  If you could,
would you mind running with --detailed and sending me (off-list) the
log file, so that I can try to examine the problematic output?

-Matt

On Mon, Jun 6, 2011 at 7:17 PM, Anthony Harness
<anthony.harness at colorado.edu> wrote:
> The array shouldn't be too small. The data contain 1024^3 cells (20 million
> cells within the cut_region) and I am running it on 60 processors (120
> doesn't work either). This is my script:
> from yt.mods import *
> from yt.analysis_modules.api import EnzoSimulation
> import numpy as na
> from krakenPlugins import *
> from mpi4py import MPI
> ###########################################################
> simName = '50Mpc_1024unigrid.par'
> dataDir = '/lustre/scratch/britton/box_size_study/50Mpc_1024/run_17f_cl_5D'
> es = EnzoSimulation('%s/%s' %(dataDir,simName),get_redshift_outputs=False)
> dataCntr = 0
> numBins = 1000
> allHisty = na.array([na.zeros(numBins+1)])
> allHistx = na.array([na.zeros(numBins+1)])
> es = es.allOutputs[:85]
> for output in es:
> pf = load('%s%s' %(dataDir,output['filename'][1:]))
> dd = pf.h.all_data()
> pc = PlotCollection(pf)
> cut = dd.cut_region(["grid['Metallicity'] <= 1.e-6","grid['Temperature'] <=
> 10.**5.","grid['Temperature'] >= 300.","grid['Baryon_Overdensity'] >=
> 1.","grid['Baryon_Overdensity'] <= 100."])
> pc.add_profile_object(cut, ['Density','Ones'], weight=None,
> x_bins=numBins,x_log=True)
> ones = pc.plots[-1].data["Ones"]
> bod = pc.plots[-1].data["Density"]
> allHisty = na.concatenate((allHisty,[ones]))
> allHistx = na.concatenate((allHistx,[bod]))
> dataCntr += 1
> del pf,dd,pc,cut,ones,bod
> if MPI.COMM_WORLD.rank == 0:
> print '***Saving to .npy file. UTC Time: %s***'
> na.save('%s/histograms_y.npy'%saveDir,allHisty)
> na.save('%s/histograms_x.npy'%saveDir,allHistx)
>
>
> On Mon, Jun 6, 2011 at 6:24 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>>
>> Hi Anthony,
>>
>> I tried it on a small dataset and I was unable to reproduce it.  Do
>> you think that the array is small enough that some of the processors
>> aren't getting any data?  I was able to get the profile command to
>> work all the way down to arrays of size 19, run on 20 processors.
>>
>> Could you post the entirety of your script?
>>
>> -Matt
>>
>> On Mon, Jun 6, 2011 at 5:15 PM, Anthony Harness
>> <anthony.harness at colorado.edu> wrote:
>> > Hello,
>> >
>> > I am trying to add a profile object to a Plot Collection (via
>> > pc.add_profile_object(data,fields) ) while running in parallel on
>> > Kraken. I
>> > get the following error: "TypeError: message: expecting a list or tuple"
>> > which ultimately comes from mpi4py.MPI.Comm.Allreduce which is called by
>> > ParallelAnalysisInterface._mpi_allsum(). In ._mpi_allsum() there is the
>> > following comment: "# We use old-school pickling here on the assumption
>> > the
>> > arrays are relatively small ( < 1e7 elements )". The dataset I am
>> > working
>> > with is larger than 1e7 elements, so is _mpi_allsum not able to pass
>> > such a
>> > large array to Comm.Allreduce?
>> >
>> > Thanks,
>> > Anthony
>> >
>> > Here is the traceback:
>> >
>> > File "/yt-2.1stable-py2.7-linux-x86_64.egg/yt/data_objects/profiles.py",
>> > line 146, in add_fields
>> >     self._lazy_add_fields(fields, weight, accumulation)
>> > File "/yt-2.1stable-py2.7-linux-x86_64.egg/yt/data_objects/profiles.py",
>> > line 94, in _lazy_add_fields
>> >     for gi,grid in enumerate(self._get_grids(fields)):
>> > File
>> >
>> > "/yt-2.1stable-py2.7-linux-x86_64.egg/yt/utilities/parallel_tools/parallel_analysis_interface.py",
>> > line 134, in __iter__
>> >     if not self.just_list: self.pobj._finalize_parallel()
>> > File "/yt-2.1stable-py2.7-linux-x86_64.egg/yt/data_objects/profiles.py",
>> > line 122, in _finalize_parallel
>> >     self.__data[key] = self._mpi_allsum(self.__data[key])
>> > File
>> >
>> > "/yt-2.1stable-py2.7-linux-x86_64.egg/yt/utilities/parallel_tools/parallel_analysis_interface.py",
>> > line 185, in passage
>> >     return func(self, data)
>> > File
>> >
>> > "/yt-2.1stable-py2.7-linux-x86_64.egg/yt/utilities/parallel_tools/parallel_analysis_interface.py",
>> > line 1124, in _mpi_allsum
>> >     MPI.COMM_WORLD.Allreduce(data, tr, op=MPI.SUM)
>> > File "Comm.pyx", line 530, in mpi4py.MPI.Comm.Allreduce
>> > (src/mpi4py_MPI.c:43646)
>> > File "message.pxi", line 426, in mpi4py.MPI._p_msg_cco.for_allreduce
>> > (src/mpi4py_MPI.c:14446)
>> > File "message.pxi", line 33, in mpi4py.MPI.message_simple
>> > (src/mpi4py_MPI.c:11108)
>> > TypeError: message: expecting a list or tuple
>> >
>> >
>> > _______________________________________________
>> > yt-users mailing list
>> > yt-users at lists.spacepope.org
>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> >
>> >
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>