Here's a list of my loaded modules:<br><br> 1) modules/<a href="http://3.1.6.5">3.1.6.5</a> 10) xt-os/2.2.74 19) xt-asyncpe/4.9<br>
2) torque/2.4.8 11) xt-boot/2.2.74 20) xt-pe/2.2.74<br> 3) moab/5.4.3.s16991 12) xt-lustre-ss/2.2.74_1.6.5 21) xt-mpt/5.2.3<br>
4) /opt/cray/xt-asyncpe/default/modulefiles/xtpe-istanbul 13) cray/job/1.5.5-0.1_2.0202.21413.56.7 22) pmi/2.1.2-1.0000.8396.13.1.ss<br> 5) tgusage/3.0-r2 14) cray/csa/3.0.0-1_2.0202.21426.77.7 23) xt-libsci/10.5.02<br>
6) altd/1.0 15) cray/account/1.0.0-2.0202.19482.49.18 24) gcc/4.5.3<br> 7) DefApps 16) cray/projdb/1.0.0-1.0202.19483.52.1 25) cray/MySQL/5.0.64-1.0202.2899.21.1<br>
8) xtpe-target-cnl 17) Base-opts/2.2.74 26) xt-mpt/5.0.0<br> 9) xt-service/2.2.74 18) PrgEnv-gnu/2.2.74 27) yt/2.1<br>
<br><br><br><br><br><br><div class="gmail_quote">On Tue, Jun 7, 2011 at 10:59 AM, Matthew Turk <span dir="ltr"><<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Hi Anthony,<br>
<br>
Stephen and I have chatted about this -- he brought up that some MPI<br>
implementations are more tolerating than others. What're the contents<br>
of your module list?<br>
<br>
It's also possible that we need to specify types in the Allreduce<br>
call. I am not sure why that would cause problems for you and not me,<br>
however.<br>
<br>
-Matt<br>
<div><div></div><div class="h5"><br>
On Mon, Jun 6, 2011 at 7:40 PM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>> wrote:<br>
> Hi Anthony,<br>
><br>
> Using the grid cutting method, which I thought might be causing<br>
> problems, I was again unable to reproduce the issue. If you could,<br>
> would you mind running with --detailed and sending me (off-list) the<br>
> log file, so that I can try to examine the problematic output?<br>
><br>
> -Matt<br>
><br>
> On Mon, Jun 6, 2011 at 7:17 PM, Anthony Harness<br>
> <<a href="mailto:anthony.harness@colorado.edu">anthony.harness@colorado.edu</a>> wrote:<br>
>> The array shouldn't be too small. The data contain 1024^3 cells (20 million<br>
>> cells within the cut_region) and I am running it on 60 processors (120<br>
>> doesn't work either). This is my script:<br>
>> from yt.mods import *<br>
>> from yt.analysis_modules.api import EnzoSimulation<br>
>> import numpy as na<br>
>> from krakenPlugins import *<br>
>> from mpi4py import MPI<br>
>> ###########################################################<br>
>> simName = '50Mpc_1024unigrid.par'<br>
>> dataDir = '/lustre/scratch/britton/box_size_study/50Mpc_1024/run_17f_cl_5D'<br>
>> es = EnzoSimulation('%s/%s' %(dataDir,simName),get_redshift_outputs=False)<br>
>> dataCntr = 0<br>
>> numBins = 1000<br>
>> allHisty = na.array([na.zeros(numBins+1)])<br>
>> allHistx = na.array([na.zeros(numBins+1)])<br>
>> es = es.allOutputs[:85]<br>
>> for output in es:<br>
>> pf = load('%s%s' %(dataDir,output['filename'][1:]))<br>
>> dd = pf.h.all_data()<br>
>> pc = PlotCollection(pf)<br>
>> cut = dd.cut_region(["grid['Metallicity'] <= 1.e-6","grid['Temperature'] <=<br>
>> 10.**5.","grid['Temperature'] >= 300.","grid['Baryon_Overdensity'] >=<br>
>> 1.","grid['Baryon_Overdensity'] <= 100."])<br>
>> pc.add_profile_object(cut, ['Density','Ones'], weight=None,<br>
>> x_bins=numBins,x_log=True)<br>
>> ones = pc.plots[-1].data["Ones"]<br>
>> bod = pc.plots[-1].data["Density"]<br>
>> allHisty = na.concatenate((allHisty,[ones]))<br>
>> allHistx = na.concatenate((allHistx,[bod]))<br>
>> dataCntr += 1<br>
>> del pf,dd,pc,cut,ones,bod<br>
>> if MPI.COMM_WORLD.rank == 0:<br>
>> print '***Saving to .npy file. UTC Time: %s***'<br>
>> na.save('%s/histograms_y.npy'%saveDir,allHisty)<br>
>> na.save('%s/histograms_x.npy'%saveDir,allHistx)<br>
>><br>
>><br>
>> On Mon, Jun 6, 2011 at 6:24 PM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>> wrote:<br>
>>><br>
>>> Hi Anthony,<br>
>>><br>
>>> I tried it on a small dataset and I was unable to reproduce it. Do<br>
>>> you think that the array is small enough that some of the processors<br>
>>> aren't getting any data? I was able to get the profile command to<br>
>>> work all the way down to arrays of size 19, run on 20 processors.<br>
>>><br>
>>> Could you post the entirety of your script?<br>
>>><br>
>>> -Matt<br>
>>><br>
>>> On Mon, Jun 6, 2011 at 5:15 PM, Anthony Harness<br>
>>> <<a href="mailto:anthony.harness@colorado.edu">anthony.harness@colorado.edu</a>> wrote:<br>
>>> > Hello,<br>
>>> ><br>
>>> > I am trying to add a profile object to a Plot Collection (via<br>
>>> > pc.add_profile_object(data,fields) ) while running in parallel on<br>
>>> > Kraken. I<br>
>>> > get the following error: "TypeError: message: expecting a list or tuple"<br>
>>> > which ultimately comes from mpi4py.MPI.Comm.Allreduce which is called by<br>
>>> > ParallelAnalysisInterface._mpi_allsum(). In ._mpi_allsum() there is the<br>
>>> > following comment: "# We use old-school pickling here on the assumption<br>
>>> > the<br>
>>> > arrays are relatively small ( < 1e7 elements )". The dataset I am<br>
>>> > working<br>
>>> > with is larger than 1e7 elements, so is _mpi_allsum not able to pass<br>
>>> > such a<br>
>>> > large array to Comm.Allreduce?<br>
>>> ><br>
>>> > Thanks,<br>
>>> > Anthony<br>
>>> ><br>
>>> > Here is the traceback:<br>
>>> ><br>
>>> > File "/yt-2.1stable-py2.7-linux-x86_64.egg/yt/data_objects/profiles.py",<br>
>>> > line 146, in add_fields<br>
>>> > self._lazy_add_fields(fields, weight, accumulation)<br>
>>> > File "/yt-2.1stable-py2.7-linux-x86_64.egg/yt/data_objects/profiles.py",<br>
>>> > line 94, in _lazy_add_fields<br>
>>> > for gi,grid in enumerate(self._get_grids(fields)):<br>
>>> > File<br>
>>> ><br>
>>> > "/yt-2.1stable-py2.7-linux-x86_64.egg/yt/utilities/parallel_tools/parallel_analysis_interface.py",<br>
>>> > line 134, in __iter__<br>
>>> > if not self.just_list: self.pobj._finalize_parallel()<br>
>>> > File "/yt-2.1stable-py2.7-linux-x86_64.egg/yt/data_objects/profiles.py",<br>
>>> > line 122, in _finalize_parallel<br>
>>> > self.__data[key] = self._mpi_allsum(self.__data[key])<br>
>>> > File<br>
>>> ><br>
>>> > "/yt-2.1stable-py2.7-linux-x86_64.egg/yt/utilities/parallel_tools/parallel_analysis_interface.py",<br>
>>> > line 185, in passage<br>
>>> > return func(self, data)<br>
>>> > File<br>
>>> ><br>
>>> > "/yt-2.1stable-py2.7-linux-x86_64.egg/yt/utilities/parallel_tools/parallel_analysis_interface.py",<br>
>>> > line 1124, in _mpi_allsum<br>
>>> > MPI.COMM_WORLD.Allreduce(data, tr, op=MPI.SUM)<br>
>>> > File "Comm.pyx", line 530, in mpi4py.MPI.Comm.Allreduce<br>
>>> > (src/mpi4py_MPI.c:43646)<br>
>>> > File "message.pxi", line 426, in mpi4py.MPI._p_msg_cco.for_allreduce<br>
>>> > (src/mpi4py_MPI.c:14446)<br>
>>> > File "message.pxi", line 33, in mpi4py.MPI.message_simple<br>
>>> > (src/mpi4py_MPI.c:11108)<br>
>>> > TypeError: message: expecting a list or tuple<br>
>>> ><br>
>>> ><br>
>>> > _______________________________________________<br>
>>> > yt-users mailing list<br>
>>> > <a href="mailto:yt-users@lists.spacepope.org">yt-users@lists.spacepope.org</a><br>
>>> > <a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a><br>
>>> ><br>
>>> ><br>
>>> _______________________________________________<br>
>>> yt-users mailing list<br>
>>> <a href="mailto:yt-users@lists.spacepope.org">yt-users@lists.spacepope.org</a><br>
>>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a><br>
>><br>
>><br>
>> _______________________________________________<br>
>> yt-users mailing list<br>
>> <a href="mailto:yt-users@lists.spacepope.org">yt-users@lists.spacepope.org</a><br>
>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a><br>
>><br>
>><br>
><br>
_______________________________________________<br>
yt-users mailing list<br>
<a href="mailto:yt-users@lists.spacepope.org">yt-users@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org</a><br>
</div></div></blockquote></div><br>