<div>The array shouldn't be too small. The data contain 1024^3 cells (20 million cells within the cut_region) and I am running it on 60 processors (120 doesn't work either). This is my script:</div><div><br></div><div>
<div>from yt.mods import *</div><div>from yt.analysis_modules.api import EnzoSimulation</div><div>import numpy as na</div><div>from krakenPlugins import *</div><div>from mpi4py import MPI</div><div>###########################################################</div>
<div>simName = '50Mpc_1024unigrid.par'</div><div>dataDir = '/lustre/scratch/britton/box_size_study/50Mpc_1024/run_17f_cl_5D'<span class="Apple-tab-span" style="white-space:pre"> </span></div><div><br></div>
<div>es = EnzoSimulation('%s/%s' %(dataDir,simName),get_redshift_outputs=False)</div><div><br></div><div>dataCntr = 0</div><div>numBins = 1000</div><div><br></div><div>allHisty = na.array([na.zeros(numBins+1)])</div>
<div>allHistx = na.array([na.zeros(numBins+1)])</div><div><br></div><div>es = es.allOutputs[:85]</div><div><br></div><div>for output in es:</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>pf = load('%s%s' %(dataDir,output['filename'][1:]))</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>dd = pf.h.all_data()</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>pc = PlotCollection(pf)</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>cut = dd.cut_region(["grid['Metallicity'] <= 1.e-6","grid['Temperature'] <= 10.**5.","grid['Temperature'] >= 300.","grid['Baryon_Overdensity'] >= 1.","grid['Baryon_Overdensity'] <= 100."])</div>
<div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>pc.add_profile_object(cut, ['Density','Ones'], weight=None, x_bins=numBins,x_log=True)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>ones = pc.plots[-1].data["Ones"]</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>bod = pc.plots[-1].data["Density"]</div><div><span class="Apple-tab-span" style="white-space:pre"> </span></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>allHisty = na.concatenate((allHisty,[ones]))</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>allHistx = na.concatenate((allHistx,[bod]))</div><div><span class="Apple-tab-span" style="white-space:pre"> </span></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>dataCntr += 1</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>del pf,dd,pc,cut,ones,bod</div></div><div><br></div><div><div>if MPI.COMM_WORLD.rank == 0:</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>print '***Saving to .npy file. UTC Time: %s***' </div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>na.save('%s/histograms_y.npy'%saveDir,allHisty)<span class="Apple-tab-span" style="white-space:pre"> </span></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>na.save('%s/histograms_x.npy'%saveDir,allHistx)</div>
</div><div><br></div><div> </div><div><br></div><br><div class="gmail_quote">On Mon, Jun 6, 2011 at 6:24 PM, Matthew Turk <span dir="ltr"><<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Hi Anthony,<br>
<br>
I tried it on a small dataset and I was unable to reproduce it. Do<br>
you think that the array is small enough that some of the processors<br>
aren't getting any data? I was able to get the profile command to<br>
work all the way down to arrays of size 19, run on 20 processors.<br>
<br>
Could you post the entirety of your script?<br>
<br>
-Matt<br>
<div><div></div><div class="h5"><br>
On Mon, Jun 6, 2011 at 5:15 PM, Anthony Harness<br>
<<a href="mailto:anthony.harness@colorado.edu">anthony.harness@colorado.edu</a>> wrote:<br>
> Hello,<br>
><br>
> I am trying to add a profile object to a Plot Collection (via<br>
> pc.add_profile_object(data,fields) ) while running in parallel on Kraken. I<br>
> get the following error: "TypeError: message: expecting a list or tuple"<br>
> which ultimately comes from mpi4py.MPI.Comm.Allreduce which is called by<br>
> ParallelAnalysisInterface._mpi_allsum(). In ._mpi_allsum() there is the<br>
> following comment: "# We use old-school pickling here on the assumption the<br>
> arrays are relatively small ( < 1e7 elements )". The dataset I am working<br>
> with is larger than 1e7 elements, so is _mpi_allsum not able to pass such a<br>
> large array to Comm.Allreduce?<br>
><br>
> Thanks,<br>
> Anthony<br>
><br>
> Here is the traceback:<br>
><br>
> File "/yt-2.1stable-py2.7-linux-x86_64.egg/yt/data_objects/profiles.py",<br>
> line 146, in add_fields<br>
> self._lazy_add_fields(fields, weight, accumulation)<br>
> File "/yt-2.1stable-py2.7-linux-x86_64.egg/yt/data_objects/profiles.py",<br>
> line 94, in _lazy_add_fields<br>
> for gi,grid in enumerate(self._get_grids(fields)):<br>
> File<br>
> "/yt-2.1stable-py2.7-linux-x86_64.egg/yt/utilities/parallel_tools/parallel_analysis_interface.py",<br>
> line 134, in __iter__<br>
> if not self.just_list: self.pobj._finalize_parallel()<br>
> File "/yt-2.1stable-py2.7-linux-x86_64.egg/yt/data_objects/profiles.py",<br>
> line 122, in _finalize_parallel<br>
> self.__data[key] = self._mpi_allsum(self.__data[key])<br>
> File<br>
> "/yt-2.1stable-py2.7-linux-x86_64.egg/yt/utilities/parallel_tools/parallel_analysis_interface.py",<br>
> line 185, in passage<br>
> return func(self, data)<br>
> File<br>
> "/yt-2.1stable-py2.7-linux-x86_64.egg/yt/utilities/parallel_tools/parallel_analysis_interface.py",<br>
> line 1124, in _mpi_allsum<br>
> MPI.COMM_WORLD.Allreduce(data, tr, op=MPI.SUM)<br>
> File "Comm.pyx", line 530, in mpi4py.MPI.Comm.Allreduce<br>
> (src/mpi4py_MPI.c:43646)<br>
> File "message.pxi", line 426, in mpi4py.MPI._p_msg_cco.for_allreduce<br>
> (src/mpi4py_MPI.c:14446)<br>
> File "message.pxi", line 33, in mpi4py.MPI.message_simple<br>
> (src/mpi4py_MPI.c:11108)<br>
> TypeError: message: expecting a list or tuple<br>
><br>
><br>
</div></div>> _______________________________________________<br>
> yt-users mailing list<br>
> <a href="mailto:yt-users@lists.spacepope.org">yt-users@lists.spacepope.org</a><br>
> <a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a><br>
><br>
><br>
_______________________________________________<br>
yt-users mailing list<br>
<a href="mailto:yt-users@lists.spacepope.org">yt-users@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a><br>
</blockquote></div><br>