[yt-users] h5py problem

[j s oishi]

Hi John,

I've been looking into your problem, and I cannot reproduce it using
your script on several enzo datasets I have available. It works fine
for me using yt revision 486d7131f1c2. Are you loading enzo data or
FLASH data? Which revision of yt were you having this problem on? I
was unable to reproduce the problem using either h5py 1.2 or 1.3.

If you are loading enzo data, loading the parameter file also opens an
hdf5 filehandle, and this could somehow be getting confused, though at
this point, I don't see how. Can you try not printing the grids, but
simply loading the parameter file, perhaps by replacing

print pf.h.grids

with

my_h = pf.h

and see if the crash persists.

thanks,

j


On Wed, Jul 6, 2011 at 1:25 PM, John ZuHone <jzuhone at cfa.harvard.edu> wrote:
> Hi all,
> Sam has come through and helped me solve this issue. He suggested deleting
> the file object ("del f") at the end of the loop over the file numbers
> (http://paste.enzotools.org/show/1730/), and it resolves the crash. Doesn't
> really explain it, but resolves it.
> Thanks to everyone who looked at this.
> Best,
> John ZuHone
> On Jul 6, 2011, at 3:28 PM, John ZuHone wrote:
>
> I should also point out that if you don't access the data (by not printing
> the list of grids as I do in my example), then you do not get a crash.
> John
> On Jul 5, 2011, at 8:06 PM, Sam Skillman wrote:
>
> Hi John,
> So I think the issue might be that the default mode of h5py.File is
> read/write if it exists, and create if it doesn't.  In this case, I'm
> guessing that it is trying to open it in read/write, then when it tries to
> access it again it freaks out.  If you want to try changing line 73 of
> yt/frontends/flash/data_structures.py
> from:
>         self._handle = h5py.File(self.hierarchy_filename)
> to:
>         self._handle = h5py.File(self.hierarchy_filename, 'r')
> it.....might work.  No promises. Let us know if that works.  If not, then
> seeing a pastebin of your script would be helpful.
> Best,
> Sam
> On Tue, Jul 5, 2011 at 6:03 PM, Cameron Hummels
> <chummels at astro.columbia.edu> wrote:
>>
>> Are you actively trying to create hdf5 files?  I know hdf5 won't
>> overwrite an existing file by default, so perhaps if you've already
>> created files in your loop, then try to run it again with those files
>> still present, it will bail like this.
>>
>> If that isn't the case, I'm not sure.  If you could send your script, or
>> pastebin it, that might provide more insight.
>>
>> Cameron
>>
>> On 07/05/2011 05:34 PM, John ZuHone wrote:
>> > Hi all,
>> >
>> > I  have a loop that goes over a list of filenames I've generated, loads
>> > parameter files for them, passes each pf object to a procedure which grabs
>> > data from the file, and then loops back over. After the first iteration, I
>> > get this crash:
>> >
>> > Traceback (most recent call last):
>> >   File
>> > "/local/data/duce2/zuhone/src/ctools/filter_particle_velocities.py", line
>> > 91, in <module>
>> >     main()
>> >   File
>> > "/local/data/duce2/zuhone/src/ctools/filter_particle_velocities.py", line
>> > 68, in main
>> >     compute_turb_vels(pf, numVelCells, xp, yp, zp, vxp, vyp, vzp)
>> >   File "/local/data/duce2/zuhone/src/ctools/compute_turb_vels.py", line
>> > 15, in compute_turb_vels
>> >     leaf_grids = [g for g in pf.h.grids if len(g.Children) == 0]
>> >   File
>> > "/local/data/duce2/zuhone/opt/yt-x86_64/src/yt-hg/yt/data_objects/static_output.py",
>> > line 163, in hierarchy
>> >     self, data_style=self.data_style)
>> >   File
>> > "/local/data/duce2/zuhone/opt/yt-x86_64/src/yt-hg/yt/frontends/flash/data_structures.py",
>> > line 73, in __init__
>> >     self._handle = h5py.File(self.hierarchy_filename)
>> >   File
>> > "/local/data/duce2/zuhone/opt/yt-x86_64/lib/python2.7/site-packages/h5py-1.3.1-py2.7-linux-x86_64.egg/h5py/highlevel.py",
>> > line 797, in __init__
>> >     self.fid = self._generate_fid(name, mode, plist)
>> >   File
>> > "/local/data/duce2/zuhone/opt/yt-x86_64/lib/python2.7/site-packages/h5py-1.3.1-py2.7-linux-x86_64.egg/h5py/highlevel.py",
>> > line 842, in _generate_fid
>> >     fid = h5f.create(name, h5f.ACC_EXCL, fapl=plist)
>> >   File "h5f.pyx", line 87, in h5py.h5f.create (h5py/h5f.c:1378)
>> > h5py._stub.IOError: File exists (File accessability: Unable to open
>> > file)
>> >
>> > It does not matter where in the list of files I start, so it's not due
>> > to a single bad file. It's like it doesn't detect the file it's trying to
>> > read somehow, so it tries to create it, but then it finds out it already
>> > exists, and then aborts. Very odd.
>> >
>> > Best,
>> >
>> > John ZuHone
>> > _______________________________________________
>> > yt-users mailing list
>> > yt-users at lists.spacepope.org
>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> >
>>
>> --
>> Cameron Hummels
>> PhD Candidate, Astronomy Department of Columbia University
>> Public Outreach Director, Astronomy Department of Columbia University
>> NASA IYA New York State Student Ambassador
>> http://outreach.astro.columbia.edu
>> PGP: 0x06F886E3
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>