[yt-users] Problem with ParallellAnalysisInterface._mpi_allsum

[Matthew Turk]

Hi Anthony,

Stephen and I have chatted about this -- he brought up that some MPI
implementations are more tolerating than others.  What're the contents
of your module list?

It's also possible that we need to specify types in the Allreduce
call.  I am not sure why that would cause problems for you and not me,
however.

-Matt

On Mon, Jun 6, 2011 at 7:40 PM, Matthew Turk <matthewturk at gmail.com> wrote:
> Hi Anthony,
>
> Using the grid cutting method, which I thought might be causing
> problems, I was again unable to reproduce the issue.  If you could,
> would you mind running with --detailed and sending me (off-list) the
> log file, so that I can try to examine the problematic output?
>
> -Matt
>
> On Mon, Jun 6, 2011 at 7:17 PM, Anthony Harness
> <anthony.harness at colorado.edu> wrote:
>> The array shouldn't be too small. The data contain 1024^3 cells (20 million
>> cells within the cut_region) and I am running it on 60 processors (120
>> doesn't work either). This is my script:
>> from yt.mods import *
>> from yt.analysis_modules.api import EnzoSimulation
>> import numpy as na
>> from krakenPlugins import *
>> from mpi4py import MPI
>> ###########################################################
>> simName = '50Mpc_1024unigrid.par'
>> dataDir = '/lustre/scratch/britton/box_size_study/50Mpc_1024/run_17f_cl_5D'
>> es = EnzoSimulation('%s/%s' %(dataDir,simName),get_redshift_outputs=False)
>> dataCntr = 0
>> numBins = 1000
>> allHisty = na.array([na.zeros(numBins+1)])
>> allHistx = na.array([na.zeros(numBins+1)])
>> es = es.allOutputs[:85]
>> for output in es:
>> pf = load('%s%s' %(dataDir,output['filename'][1:]))
>> dd = pf.h.all_data()
>> pc = PlotCollection(pf)
>> cut = dd.cut_region(["grid['Metallicity'] <= 1.e-6","grid['Temperature'] <=
>> 10.**5.","grid['Temperature'] >= 300.","grid['Baryon_Overdensity'] >=
>> 1.","grid['Baryon_Overdensity'] <= 100."])
>> pc.add_profile_object(cut, ['Density','Ones'], weight=None,
>> x_bins=numBins,x_log=True)
>> ones = pc.plots[-1].data["Ones"]
>> bod = pc.plots[-1].data["Density"]
>> allHisty = na.concatenate((allHisty,[ones]))
>> allHistx = na.concatenate((allHistx,[bod]))
>> dataCntr += 1
>> del pf,dd,pc,cut,ones,bod
>> if MPI.COMM_WORLD.rank == 0:
>> print '***Saving to .npy file. UTC Time: %s***'
>> na.save('%s/histograms_y.npy'%saveDir,allHisty)
>> na.save('%s/histograms_x.npy'%saveDir,allHistx)
>>
>>
>> On Mon, Jun 6, 2011 at 6:24 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>>>
>>> Hi Anthony,
>>>
>>> I tried it on a small dataset and I was unable to reproduce it.  Do
>>> you think that the array is small enough that some of the processors
>>> aren't getting any data?  I was able to get the profile command to
>>> work all the way down to arrays of size 19, run on 20 processors.
>>>
>>> Could you post the entirety of your script?
>>>
>>> -Matt
>>>
>>> On Mon, Jun 6, 2011 at 5:15 PM, Anthony Harness
>>> <anthony.harness at colorado.edu> wrote:
>>> > Hello,
>>> >
>>> > I am trying to add a profile object to a Plot Collection (via
>>> > pc.add_profile_object(data,fields) ) while running in parallel on
>>> > Kraken. I
>>> > get the following error: "TypeError: message: expecting a list or tuple"
>>> > which ultimately comes from mpi4py.MPI.Comm.Allreduce which is called by
>>> > ParallelAnalysisInterface._mpi_allsum(). In ._mpi_allsum() there is the
>>> > following comment: "# We use old-school pickling here on the assumption
>>> > the
>>> > arrays are relatively small ( < 1e7 elements )". The dataset I am
>>> > working
>>> > with is larger than 1e7 elements, so is _mpi_allsum not able to pass
>>> > such a
>>> > large array to Comm.Allreduce?
>>> >
>>> > Thanks,
>>> > Anthony
>>> >
>>> > Here is the traceback:
>>> >
>>> > File "/yt-2.1stable-py2.7-linux-x86_64.egg/yt/data_objects/profiles.py",
>>> > line 146, in add_fields
>>> >     self._lazy_add_fields(fields, weight, accumulation)
>>> > File "/yt-2.1stable-py2.7-linux-x86_64.egg/yt/data_objects/profiles.py",
>>> > line 94, in _lazy_add_fields
>>> >     for gi,grid in enumerate(self._get_grids(fields)):
>>> > File
>>> >
>>> > "/yt-2.1stable-py2.7-linux-x86_64.egg/yt/utilities/parallel_tools/parallel_analysis_interface.py",
>>> > line 134, in __iter__
>>> >     if not self.just_list: self.pobj._finalize_parallel()
>>> > File "/yt-2.1stable-py2.7-linux-x86_64.egg/yt/data_objects/profiles.py",
>>> > line 122, in _finalize_parallel
>>> >     self.__data[key] = self._mpi_allsum(self.__data[key])
>>> > File
>>> >
>>> > "/yt-2.1stable-py2.7-linux-x86_64.egg/yt/utilities/parallel_tools/parallel_analysis_interface.py",
>>> > line 185, in passage
>>> >     return func(self, data)
>>> > File
>>> >
>>> > "/yt-2.1stable-py2.7-linux-x86_64.egg/yt/utilities/parallel_tools/parallel_analysis_interface.py",
>>> > line 1124, in _mpi_allsum
>>> >     MPI.COMM_WORLD.Allreduce(data, tr, op=MPI.SUM)
>>> > File "Comm.pyx", line 530, in mpi4py.MPI.Comm.Allreduce
>>> > (src/mpi4py_MPI.c:43646)
>>> > File "message.pxi", line 426, in mpi4py.MPI._p_msg_cco.for_allreduce
>>> > (src/mpi4py_MPI.c:14446)
>>> > File "message.pxi", line 33, in mpi4py.MPI.message_simple
>>> > (src/mpi4py_MPI.c:11108)
>>> > TypeError: message: expecting a list or tuple
>>> >
>>> >
>>> > _______________________________________________
>>> > yt-users mailing list
>>> > yt-users at lists.spacepope.org
>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>> >
>>> >
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>>
>