Sorry for the fragmented pieces of info, I was trying to determine what the problem is with one of the sys admin at Nautilus, so I'm not even sure yet if it is YT's problem.<div><br></div><div>Symptoms:</div><div>paralleHF fails for the 3200 cube dataset, but not always at the same place, which leads us to think this might be an memory issue.</div>
<div><br></div><div>1) What are you attempting to do, precisely?</div><div>Currently I'm trying to run parallelHF on pieces of the subvolume since I've found out the memory requirement of the whole dataset exceeds the machine's available memory (Nautilus with 4TB shared memory).</div>
<div><br>2) What type of data, and what size of data, are you applying this to?<br>I'm doing parallelHF with DM only on a piece of the subvolume that's 1/64th of the original volume.</div><div><br></div><div>3) What is the version of yt you are using (changeset hash)?</div>
<div>Was using the latest YT as of last week when I ran the unsuccessful runs, currently trying Stephen's modification which should help with memory:</div><div><div>(dev-yt)Geoffreys-MacBook-Air:yt-hg gso$ hg identify</div>
<div>2efcec06484e (yt) tip</div></div><div>I am going to modify my script and send it to the sys admin to run test on the 800 cube first</div><div>I've been asked not to submit jobs of the 3200 because the last time I did it, it brought half the machine to a standstill</div>
<div><br>4) How are you launching yt?</div><div>I was launching it with 512 cores and 2TB of total memory, but they said to try to decrease the mpi task count so I've also tried 256, 64, 32 but they all failed after a while, a couple was doing fine during the parallelHF phase but suddenly ended with:</div>
<div><div>MPI: MPI_COMM_WORLD rank 6 has terminated without calling MPI_Finalize()</div><div>MPI: aborting job</div><div>MPI: Received signal 9</div></div><div><br>5) What is the memory available to each individual process?</div>
<div>I've usually launched the 3200 with 2TB of memory with varying mpi task counts from 32 to 512.</div><div><br>6) Under what circumstances does yt crash?</div><div>I've also had</div><div><div>P100 yt : [INFO ] 2011-10-03 08:03:06,125 Getting field particle_position_x from 112</div>
<div>MPI: MPI_COMM_WORLD rank 153 has terminated without calling MPI_Finalize()</div><div>MPI: aborting job</div><div>MPI: Received signal 9</div><div><br></div><div><br></div><div><br></div><div>asallocash failed: system error trying to write a message header - Broken pipe</div>
</div><div><br></div><div>and with the same script</div><div><div>P180 yt : [INFO ] 2011-10-03 15:12:01,898 Finished with binary hierarchy reading</div><div>Traceback (most recent call last):</div><div> File "regionPHOP.py", line 23, in <module></div>
<div> sv = pf.h.region([i * delta[0] + delta[0] / 2.0,</div><div> File "/nics/b/home/gsiisg/NautilusYT/src/yt-hg/yt/data_objects/static_output.py", line 169, in hierarchy</div><div> self, data_style=self.data_style)</div>
<div> File "/nics/b/home/gsiisg/NautilusYT/src/yt-hg/yt/frontends/enzo/data_structures.py", line 162, in __init__</div><div> AMRHierarchy.__init__(self, pf, data_style)</div><div> File "/nics/b/home/gsiisg/NautilusYT/src/yt-hg/yt/data_objects/hierarchy.py", line 79, in __init__</div>
<div> self._detect_fields()</div><div> File "/nics/b/home/gsiisg/NautilusYT/src/yt-hg/yt/frontends/enzo/data_structures.py", line 405, in _detect_fields</div><div> self.save_data(list(field_list),"/","DataFields",passthrough=True)</div>
<div> File "/nics/b/home/gsiisg/NautilusYT/src/yt-hg/yt/utilities/parallel_tools/parallel_analysis_interface.py", line 216, in in_order</div><div> f1(*args, **kwargs)</div><div> File "/nics/b/home/gsiisg/NautilusYT/src/yt-hg/yt/data_objects/hierarchy.py", line 222, in _save_data</div>
<div> arr = myGroup.create_dataset(name,data=array)</div><div> File "/nics/b/home/gsiisg/NautilusYT/lib/python2.7/site-packages/h5py-1.3.1-py2.7-linux-x86_64.egg/h5py/highlevel.py", line 464, in create_dataset</div>
<div> return Dataset(self, name, *args, **kwds)</div><div> File "/nics/b/home/gsiisg/NautilusYT/lib/python2.7/site-packages/h5py-1.3.1-py2.7-linux-x86_64.egg/h5py/highlevel.py", line 1092, in __init__</div><div>
space_id = h5s.create_simple(shape, maxshape)</div><div> File "h5s.pyx", line 103, in h5py.h5s.create_simple (h5py/h5s.c:952)</div><div>h5py._stub.ValueError: Zero sized dimension for non-unlimited dimension (Invalid arguments to routine: Bad value)</div>
</div><div><br></div><div><br></div><div>7) How does yt report this crash to you, and is it deterministic?<br><br></div><div><div>And many times there isn't any associated error output in the logs, the process just hangs and become non-responsive, the admin has tried it a couple times and seeing the different errors on 2 different dataset, so right now it can also be the dataset that is corrupted, but so far not deterministic.</div>
</div><div><br></div><div>8) What have you attempted? How did it change #6 and #7?</div><div>I've tried:</div><div><br></div><div>- adding the environmental variables:</div><div><div>export MPI_BUFS_PER_PROC=64</div>
<div>export MPI_BUFS_PER_HOST=256</div></div><div>with no change in behavior, resulting in MPI_Finalize() error sometimes</div><div><br></div><div>- using my own installation of OpenMPI</div><div><div> from yt.mods import *</div>
<div> File "/nics/b/home/gsiisg/NautilusYT/src/yt-hg/yt/mods.py", line 44, in <module></div><div> from yt.data_objects.api import \</div><div> File "/nics/b/home/gsiisg/NautilusYT/src/yt-hg/yt/data_objects/api.py", line 34, in <module></div>
<div> from hierarchy import \</div><div> File "/nics/b/home/gsiisg/NautilusYT/src/yt-hg/yt/data_objects/hierarchy.py", line 40, in <module></div><div> from yt.utilities.parallel_tools.parallel_analysis_interface import \</div>
<div> File "/nics/b/home/gsiisg/NautilusYT/src/yt-hg/yt/utilities/parallel_tools/parallel_analysis_interface.py", line 49, in <module></div><div> from mpi4py import MPI</div><div>ImportError: /nics/b/home/gsiisg/NautilusYT/lib/python2.7/site-packages/mpi4py/MPI.so: undefined symbol: mpi_sgi_inplace</div>
</div><div><br></div><div>The system admin says there are bugs or incompatibilities with the network and I should use SGI's MPI by using the module mpt/2.04 which I was using before trying my own installation of openmpi.</div>
<div><br></div><div>- currently modifying my script with Stephen's proposed changes, once it runs on my laptop will let the sys admin try it on the small dataset of 800 cube before trying it on the 3200 dataset. At least when his job hangs the machine he can terminate it faster without waiting for someone to answer his emails. Hopefully these tests wouldn't be too much of a disruption to other Nautilus users.</div>
<div><br></div><div>- was speaking to Brian Crosby during the enzo meeting briefly about this, he said he's encountered MPI errors on Nautilus as well, but his issue might be a different one than mine. This may or may not be a YT issue after all, but since it seems like multiple people are interested in YT's performance on Nautilus, I'll keep everyone updated with the latest development.<br>
<br></div><div>From</div><div>G.S.</div><div><br><div class="gmail_quote">On Tue, Oct 18, 2011 at 7:59 PM, Matthew Turk <span dir="ltr"><<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Geoffrey,<br>
<br>
Parallel HOP definitely does not attempt to load all of the particles,<br>
simultaneously, on all processors. This is covered in the method<br>
papers for both p-hop and yt, the documentation for yt, the source<br>
code, and I believe on the yt-users mailing list a couple times when<br>
discussing estimates for resource usage in p-hop.<br>
<br>
The struggles you have been having with Nautilus may in fact be a yt<br>
problem, or an application-of-yt problem, a software problem on<br>
Nautilus, or even (if Nautilus is being exposed to an excessive number<br>
of cosmic rays, for instance) a hardware problem. It would probably<br>
be productive to properly debug exactly what is going on for you to<br>
provide to us:<br>
<br>
1) What are you attempting to do, precisely?<br>
2) What type of data, and what size of data, are you applying this to?<br>
3) What is the version of yt you are using (changeset hash)?<br>
4) How are you launching yt?<br>
5) What is the memory available to each individual process?<br>
6) Under what circumstances does yt crash?<br>
7) How does yt report this crash to you, and is it deterministic?<br>
8) What have you attempted? How did it change #6 and #7?<br>
<br>
We're interested in ensuring that yt functions well on Nautilus, and<br>
that it is able to successfully halo find, analyze, etc. However,<br>
right now it feels like we're being given about 10% of a bug report,<br>
and that is regrettably not enough to properly diagnose and repair the<br>
problem.<br>
<br>
Thanks,<br>
<br>
Matt<br>
<div><div></div><div class="h5"><br>
On Tue, Oct 18, 2011 at 7:51 PM, Geoffrey So <<a href="mailto:gsiisg@gmail.com">gsiisg@gmail.com</a>> wrote:<br>
> Ah yes, I think that answers our question.<br>
> We were worried that all the particles were read in by each processor (which<br>
> I told him I don't think it did, or it would have crashed my smaller 800<br>
> cube long ago), but I wanted to get the answer from pros.<br>
> Thanks!<br>
> From<br>
> G.S.<br>
><br>
> On Tue, Oct 18, 2011 at 4:21 PM, Stephen Skory <<a href="mailto:s@skory.us">s@skory.us</a>> wrote:<br>
>><br>
>> Geoffrey,<br>
>><br>
>> > "Is the particle IO in YT that calls h5py spawned by multiple processors<br>
>> > or is it doing it serially?"<br>
>><br>
>> For your purposes, h5py is only used to *write* particle data to disk<br>
>> after the halos have been found (if you are saving them to disk, which<br>
>> you must do explicitly, of course). And in this case, it will open up<br>
>> one file using h5py per MPI task.<br>
>><br>
>> I'm guessing that they're actually concerned about reading particle<br>
>> data, because that is more disk intensive. This is done with functions<br>
>> written in C that read the data, not h5py. Here each MPI task does its<br>
>> own reading of data, and may open up multiple files to retrieve the<br>
>> particle data it needs depending on the layouts of grids in the<br>
>> .cpuNNNN files.<br>
>><br>
>> Does that help?<br>
>><br>
>> --<br>
>> Stephen Skory<br>
>> <a href="mailto:s@skory.us">s@skory.us</a><br>
>> <a href="http://stephenskory.com/" target="_blank">http://stephenskory.com/</a><br>
>> <a href="tel:510.621.3687" value="+15106213687">510.621.3687</a> (google voice)<br>
>> _______________________________________________<br>
>> Yt-dev mailing list<br>
>> <a href="mailto:Yt-dev@lists.spacepope.org">Yt-dev@lists.spacepope.org</a><br>
>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
><br>
><br>
> _______________________________________________<br>
> Yt-dev mailing list<br>
> <a href="mailto:Yt-dev@lists.spacepope.org">Yt-dev@lists.spacepope.org</a><br>
> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
><br>
><br>
_______________________________________________<br>
Yt-dev mailing list<br>
<a href="mailto:Yt-dev@lists.spacepope.org">Yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
</div></div></blockquote></div><br></div>