[yt-users] strange errors

Tue Jul 3 11:09:11 PDT 2012

Hi Geoffrey,

Yeah, seems weird.  But without knowing more about how you are
launching your script or what it does, I think we might be out of
ideas.  If you hear back and it's definitely funny business with
yt/Python/etc, write back and let us know.

-Matt

On Tue, Jul 3, 2012 at 2:05 PM, Geoffrey So <gsiisg at gmail.com> wrote:
> I've verified that there's nothing wrong with the HDF5 files themselves,
> because I can restart the script at the DD of failure and it would continue
> without problem, until it dies again with similar error.  I've encountered
> corrupted files before, and it would be just the same error, except I cannot
> restart because it fails at the same spot again every time, this isn't one
> of those times though.
>
> From
> G.S.
>
>
> On Tue, Jul 3, 2012 at 10:32 AM, Britton Smith <brittonsmith at gmail.com>
> wrote:
>>
>> Hi Geoffrey,
>>
>> This could also be due simply to a single corrupt hdf5 file.  I have seen
>> these errors before where a file did not get fully written.  You might want
>> to verify that all of the files associated with that dataset are readable.
>>
>> Britton
>>
>>
>> On Tue, Jul 3, 2012 at 1:28 PM, Geoffrey So <gsiisg at gmail.com> wrote:
>>>
>>> Already submitted the ticket yesterday just haven't heard back.
>>> Yeah I didn't think it was OOM, just wanted to confirm it with the
>>> experts, thanks!
>>>
>>> From
>>> G.S.
>>>
>>> On Tue, Jul 3, 2012 at 3:22 AM, Matthew Turk <matthewturk at gmail.com>
>>> wrote:
>>>>
>>>> Hi Geoffrey,
>>>>
>>>> If you're using a script that iterates with an individual python
>>>> instance (i.e., calling "python my_script.py") for each data dump,
>>>> then I don't think it's an OOM unless somehow multiple instances are
>>>> running simultaneously.
>>>>
>>>> I'm not sure what to suggest other than talking to the help desk at
>>>> the particular supercomputer center.
>>>>
>>>> -Matt
>>>>
>>>> On Tue, Jul 3, 2012 at 5:32 AM, Geoffrey So <gsiisg at gmail.com> wrote:
>>>> > I am trying to make projections and slices in the same script and an
>>>> > iterating over several DD datasets, but after a while I get this error
>>>> > pasted below, it seems that there's some hdf5 error where h5py cannot
>>>> > open a
>>>> > file.  I've made sure the file exists and the script continues to run
>>>> > if I
>>>> > change the DD#### to continue where it failed.  I've already modified
>>>> > the
>>>> > script so that each DD is using a new python instace intead of
>>>> > iterating
>>>> > through the files with one python call.  I tried this in queue and
>>>> > interactive and get the same error, sometimes after many DD, but
>>>> > sometimes
>>>> > after only 1 or 2.  Is this indicative of memory issue?   If it is,
>>>> > I'm not
>>>> > getting the OOM messages even in interactive node.  Has anyone
>>>> > encountered
>>>> > this before?  I've stuck in import gc at the top and gc.collect() at
>>>> > the end
>>>> > of the script, too, but doesn't help.
>>>> >
>>>> > From
>>>> > G.S.
>>>> >
>>>> > Traceback (most recent call last):
>>>> >   File "moviesGrey.py", line 35, in <module>
>>>> >     proj = pf.h.proj(0, field, weight_field=field)
>>>> >   File
>>>> >
>>>> > "/home/ux455076/dev-yt/lib/python2.7/site-packages/yt-2.4dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py",
>>>> > line 2001, in __init__
>>>> >     if self._okay_to_serialize and self.serialize:
>>>> > self._serialize(node_name=self._node_name)
>>>> >   File
>>>> >
>>>> > "/home/ux455076/dev-yt/lib/python2.7/site-packages/yt-2.4dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py",
>>>> > line 970, in _serialize
>>>> >     self._store_fields(self._key_fields, node_name, force)
>>>> >   File
>>>> >
>>>> > "/home/ux455076/dev-yt/lib/python2.7/site-packages/yt-2.4dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py",
>>>> > line 947, in _store_fields
>>>> >     passthrough = True)
>>>> >   File
>>>> >
>>>> > "/home/ux455076/dev-yt/lib/python2.7/site-packages/yt-2.4dev-py2.7-linux-x86_64.egg/yt/utilities/parallel_tools/parallel_analysis_interface.py",
>>>> > line 241, in in_order
>>>> >     f2(*args, **kwargs)
>>>> >   File
>>>> >
>>>> > "/home/ux455076/dev-yt/lib/python2.7/site-packages/yt-2.4dev-py2.7-linux-x86_64.egg/yt/data_objects/hierarchy.py",
>>>> > line 278, in _reload_data_file
>>>> >     self._data_file = h5py.File(self.__data_filename, self._data_mode)
>>>> >   File
>>>> > "/home/ux455076/dev-yt/lib/python2.7/site-packages/h5py/_hl/files.py",
>>>> > line
>>>> > 150, in __init__
>>>> >     fid = make_fid(name, mode, fapl)
>>>> >   File
>>>> > "/home/ux455076/dev-yt/lib/python2.7/site-packages/h5py/_hl/files.py",
>>>> > line
>>>> > 45, in make_fid
>>>> >     fid = h5f.open(name, h5f.ACC_RDONLY, fapl=plist)
>>>> >   File "h5f.pyx", line 70, in h5py.h5f.open (h5py/h5f.c:1618)
>>>> > IOError: unable to open file (File accessability: Unable to open file)
>>>> > Exception AttributeError: "'EnzoHierarchy' object has no attribute
>>>> > '_data_file'" in <bound method EnzoHierarchy.__del__ of
>>>> > <yt.frontends.enzo.data_structures.EnzoHierarchy object at 0x32a4d10>>
>>>> > ignored
>>>> >
>>>> > _______________________________________________
>>>> > yt-users mailing list
>>>> > yt-users at lists.spacepope.org
>>>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>> >
>>>> _______________________________________________
>>>> yt-users mailing list
>>>> yt-users at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>
>>>
>>>
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>
>>
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>