<div dir="ltr">Hi <span style="font-size:12.8px">Suoqing, </span><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">I ran the code in the interactive mode and it returned me the same error as it was with submitting the job script. What happens is that It checks through all the files and instead of producing the output files or images, it returns me the error. </span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Best</span></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 18, 2016 at 4:48 PM, Suoqing Ji <span dir="ltr"><<a href="mailto:suoqing@physics.ucsb.edu" target="_blank">suoqing@physics.ucsb.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Hi Tazkera,<div><br></div><div>I have the access to Stampede but I can not reproduce your error either (using a slightly different script though).</div><span class=""><div><br></div><div><blockquote type="cite"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="m_-1158568485658934116HOEnZb"><div class="m_-1158568485658934116h5"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="m_-1158568485658934116m_-3081621645219472836HOEnZb"><div class="m_-1158568485658934116m_-3081621645219472836h5"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>apparently jobs can only be submitted through the $WORK folder on Stampede</div></div></blockquote></div></div></div></div></blockquote></div></div></div></div></blockquote></div></div></blockquote></div><div><br></div></span><div>Btw, I believe you could submit job from both work and scratch directory on Stampede.</div><div><br></div><div>Could you run your code in interactive mode, by</div><div><br></div><div>srun -p normal -t 0:30:00 -n 16 -A _your_project_code_ --pty /bin/bash -l</div><div><br></div><div>and then enter</div><div><br></div><div>python abundance.py</div><div><br></div><div>and see what’s going on?</div><div><br><div>
<div><div>Best wishes,</div><div>--</div><div>Suoqing JI</div><div>Ph.D Candidate</div><div>Department of Physics</div><div>University of California, Santa Barbara</div><div><a href="http://web.physics.ucsb.edu/~suoqing" target="_blank">http://web.physics.ucsb.edu/~<wbr>suoqing</a></div></div>
</div><div><div class="h5">
<br><div><blockquote type="cite"><div>On Oct 18, 2016, at 1:20 PM, tazkera haque <<a href="mailto:h.tazkera@gmail.com" target="_blank">h.tazkera@gmail.com</a>> wrote:</div><br class="m_-1158568485658934116Apple-interchange-newline"><div><div dir="ltr"><div>I typed </div><div><br></div><div>$ module avail HDF5</div><div><br></div><div>------------------------------<wbr>------------------- /opt/apps/intel15/mvapich2_2_<wbr>1/modulefiles ------------------------------<wbr>--------------------</div><div> phdf5/1.8.16</div><div><br></div><div>------------------------------<wbr>-------------------------- /opt/apps/intel15/modulefiles ------------------------------<wbr>--------------------------</div><div> hdf5/1.8.16 (m,L)</div><div><br></div><div> Where:</div><div> L: Module is loaded</div><div> m: built for host and native MIC</div><div><br></div><div>$ module avail h5py</div><div>returns blank, if that helps</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 18, 2016 at 4:08 PM, Nathan Goldbaum <span dir="ltr"><<a href="mailto:nathan12343@gmail.com" target="_blank">nathan12343@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Tazkera,<div><br></div><div>When I tried googling your error message last night, I found that it's associated with more than one MPI process trying to access an HDF5 file at the same time, presumably using a serial version of the HDF5 library.</div><div><br></div><div>I just did a quick test and I'm unable to reproduce here on my laptop. Unfortunately I don't have access to stampede so can't reproduce there.</div><div><br></div><div>Can you share exactly which h5py and HDF5 library versions you're using?</div><span class="m_-1158568485658934116HOEnZb"><font color="#888888"><div><br></div><div>-Nathan</div></font></span></div><div class="m_-1158568485658934116HOEnZb"><div class="m_-1158568485658934116h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 18, 2016 at 3:05 PM, tazkera haque <span dir="ltr"><<a href="mailto:h.tazkera@gmail.com" target="_blank">h.tazkera@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">HI Nathan, <div><br></div><div>Sorry to bother you again, but the problem seems to prevail even working from $WORK directory. I got the same error msg this morning again with a different script. Do you see anything wrong with the script I attached ? I have used it for a long time now without any sort of error.</div><div><br></div><div>Thanks</div></div><div class="m_-1158568485658934116m_-3081621645219472836HOEnZb"><div class="m_-1158568485658934116m_-3081621645219472836h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 18, 2016 at 2:34 AM, tazkera haque <span dir="ltr"><<a href="mailto:h.tazkera@gmail.com" target="_blank">h.tazkera@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">HI Nathan, <div><br></div><div>I figured out what was going wrong, I submitted my job script from the $SCRATCH folder. apparently jobs can only be submitted through the $WORK folder on Stampede. thanks for your prompt response though</div></div><div class="m_-1158568485658934116m_-3081621645219472836m_5274618939355493361HOEnZb"><div class="m_-1158568485658934116m_-3081621645219472836m_5274618939355493361h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 18, 2016 at 1:22 AM, tazkera haque <span dir="ltr"><<a href="mailto:h.tazkera@gmail.com" target="_blank">h.tazkera@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Nathan, <div><br></div><div>I tried with one file in my ipython notebook, it seems to work there</div></div><div class="m_-1158568485658934116m_-3081621645219472836m_5274618939355493361m_2549907504023929652HOEnZb"><div class="m_-1158568485658934116m_-3081621645219472836m_5274618939355493361m_2549907504023929652h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 18, 2016 at 12:54 AM, tazkera haque <span dir="ltr"><<a href="mailto:h.tazkera@gmail.com" target="_blank">h.tazkera@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">yes it's being run in parallel, I didn't check with one core yet, I will let you know what happens then</div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="m_-1158568485658934116m_-3081621645219472836m_5274618939355493361m_2549907504023929652m_3485652829179782912h5">On Tue, Oct 18, 2016 at 12:50 AM, Nathan Goldbaum <span dir="ltr"><<a href="mailto:nathan12343@gmail.com" target="_blank">nathan12343@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="m_-1158568485658934116m_-3081621645219472836m_5274618939355493361m_2549907504023929652m_3485652829179782912h5">Is the script being run in parallel? If so, does it crash if you run it on only one core?<span class="m_-1158568485658934116m_-3081621645219472836m_5274618939355493361m_2549907504023929652m_3485652829179782912m_-4374703221355933864HOEnZb"><font color="#888888"><span></span></font></span><div><span class="m_-1158568485658934116m_-3081621645219472836m_5274618939355493361m_2549907504023929652m_3485652829179782912m_-4374703221355933864HOEnZb"><font color="#888888"><div><br></div></font></span><div><span class="m_-1158568485658934116m_-3081621645219472836m_5274618939355493361m_2549907504023929652m_3485652829179782912m_-4374703221355933864HOEnZb"><font color="#888888">Nathan</font></span><div><div class="m_-1158568485658934116m_-3081621645219472836m_5274618939355493361m_2549907504023929652m_3485652829179782912m_-4374703221355933864h5"><br><br>On Monday, October 17, 2016, tazkera haque <<a href="mailto:h.tazkera@gmail.com" target="_blank">h.tazkera@gmail.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">HI people, <div><br></div><div>I am using yt 3.3.1 and submitting my SLURM script to stampede.</div><div>I was using this script to find abundances of C, O etc through checkpoint files in FLASH. while my script worked fine with old yt (3.1) , suddenly it crashed today and returned me the following error:</div><div><br></div><div><div><b>yt : [INFO ] 2016-10-17 23:22:24,295 Parameters: current_time = 28.1847530806</b></div><div><b>yt : [INFO ] 2016-10-17 23:22:24,295 Parameters: domain_dimensions = [128 128 128]</b></div><div><b>yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: domain_left_edge = [ -2.80000000e+10 -2.80000000e+10 -2.80000000e+10]</b></div><div><b>yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: domain_right_edge = [ 2.80000000e+10 2.80000000e+10 2.80000000e+10]</b></div><div><b>yt : [INFO ] 2016-10-17 23:22:24,296 Parameters: cosmological_simulation = 0.0</b></div><div><b>Executin lessg abundance.py</b></div><div><b>Traceback (most recent call last):</b></div><div><b> File "abundance2.py", line 304, in <module></b></div><div><b> main(chkFilenames_own)</b></div><div><b> File "abundance2.py", line 59, in main</b></div><div><b> pf = yt.load(filenames[n])</b></div><div><b> File "/work/03858/thaque56/sw/yt-ne<wbr>w-3.3/yt-conda/lib/python2.7/s<wbr>ite-packages/yt/convenience.py<wbr>", line 79, in load</b></div><div><b> if c._is_valid(*args, **kwargs): candidates.append(n)</b></div><div><b> File "/work/03858/thaque56/sw/yt-ne<wbr>w-3.3/yt-conda/lib/python2.7/s<wbr>ite-packages/yt/frontends/flas<wbr>h/data_structures.py", line 478, in _is_valid</b></div><div><b> if "bounding box" not in fileh["/"].keys() \</b></div><div><b> File "/work/03858/thaque56/sw/yt-ne<wbr>w-3.3/yt-conda/lib/python2.7/s<wbr>ite-packages/h5py/_hl/base.py"<wbr>, line 368, in keys</b></div><div><b> return list(self)</b></div><div><b> File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrappe<wbr>r (/home/ilan/minonda/conda-bld/<wbr>work/h5py/_objects.c:2696)</b></div><div><b> File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrappe<wbr>r (/home/ilan/minonda/conda-bld/<wbr>work/h5py/_objects.c:2654)</b></div><div><b> File "/work/03858/thaque56/sw/yt-ne<wbr>w-3.3/yt-conda/lib/python2.7/s<wbr>ite-packages/h5py/_hl/group.py<wbr>", line 303, in __len__</b></div><div><b> return self.id.get_num_objs()</b></div><div><b> File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrappe<wbr>r (/home/ilan/minonda/conda-bld/<wbr>work/h5py/_objects.c:2696)</b></div><div><b> File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrappe<wbr>r (/home/ilan/minonda/conda-bld/<wbr>work/h5py/_objects.c:2654)</b></div><div><b> File "h5py/h5g.pyx", line 321, in h5py.h5g.GroupID.get_num_objs (/home/ilan/minonda/conda-bld/<wbr>work/h5py/h5g.c:4194)</b></div><div><b>RuntimeError: Can't determine (Bad symbol table node signature)</b></div><div><b>[c560-102.stampede.tacc.utexas<wbr>.edu:mpispawn_0][child_handler<wbr>] MPI process (rank: 0, pid: 22137) exited with status 1</b></div><div><b>TACC: MPI job exited with code: 1</b></div><div><b><br></b></div><div><b>TACC: Shutdown complete. Exiting.</b></div></div><div><br></div><div>I was wondering if there is something wrong with my code or the new yt. I am also attaching my code here to look at. Thanks in advance</div><div><br></div><div>Best</div><div>Tazkera</div></div>
</blockquote></div></div></div></div>
<br></div></div>______________________________<wbr>_________________<br>
yt-users mailing list<br>
<a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" rel="noreferrer" target="_blank">http://lists.spacepope.org/lis<wbr>tinfo.cgi/yt-users-spacepope.o<wbr>rg</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div><br>______________________________<wbr>_________________<br>
yt-users mailing list<br>
<a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" rel="noreferrer" target="_blank">http://lists.spacepope.org/lis<wbr>tinfo.cgi/yt-users-spacepope.o<wbr>rg</a><br>
<br></blockquote></div><br></div>
</div></div><br>______________________________<wbr>_________________<br>
yt-users mailing list<br>
<a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" rel="noreferrer" target="_blank">http://lists.spacepope.org/lis<wbr>tinfo.cgi/yt-users-spacepope.<wbr>org</a><br>
<br></blockquote></div><br></div>
______________________________<wbr>_________________<br>yt-users mailing list<br><a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a><br><a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" target="_blank">http://lists.spacepope.org/<wbr>listinfo.cgi/yt-users-<wbr>spacepope.org</a><br></div></blockquote></div><br></div></div></div></div><br>______________________________<wbr>_________________<br>
yt-users mailing list<br>
<a href="mailto:yt-users@lists.spacepope.org">yt-users@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" rel="noreferrer" target="_blank">http://lists.spacepope.org/<wbr>listinfo.cgi/yt-users-<wbr>spacepope.org</a><br>
<br></blockquote></div><br></div>