[yt-users] h5py problem

John ZuHone jzuhone at cfa.harvard.edu
Wed Jul 6 21:47:17 PDT 2011 

Hi j s,

See my previous email... Sam solved this for me. But thank you for looking into this!

Best,

John

On Jul 6, 2011, at 6:24 PM, j s oishi wrote:

> Hi John,
> 
> I've been looking into your problem, and I cannot reproduce it using
> your script on several enzo datasets I have available. It works fine
> for me using yt revision 486d7131f1c2. Are you loading enzo data or
> FLASH data? Which revision of yt were you having this problem on? I
> was unable to reproduce the problem using either h5py 1.2 or 1.3.
> 
> If you are loading enzo data, loading the parameter file also opens an
> hdf5 filehandle, and this could somehow be getting confused, though at
> this point, I don't see how. Can you try not printing the grids, but
> simply loading the parameter file, perhaps by replacing
> 
> print pf.h.grids
> 
> with
> 
> my_h = pf.h
> 
> and see if the crash persists.
> 
> thanks,
> 
> j
> 
> 
> On Wed, Jul 6, 2011 at 1:25 PM, John ZuHone <jzuhone at cfa.harvard.edu> wrote:
>> Hi all,
>> Sam has come through and helped me solve this issue. He suggested deleting
>> the file object ("del f") at the end of the loop over the file numbers
>> (http://paste.enzotools.org/show/1730/), and it resolves the crash. Doesn't
>> really explain it, but resolves it.
>> Thanks to everyone who looked at this.
>> Best,
>> John ZuHone
>> On Jul 6, 2011, at 3:28 PM, John ZuHone wrote:
>> 
>> I should also point out that if you don't access the data (by not printing
>> the list of grids as I do in my example), then you do not get a crash.
>> John
>> On Jul 5, 2011, at 8:06 PM, Sam Skillman wrote:
>> 
>> Hi John,
>> So I think the issue might be that the default mode of h5py.File is
>> read/write if it exists, and create if it doesn't.  In this case, I'm
>> guessing that it is trying to open it in read/write, then when it tries to
>> access it again it freaks out.  If you want to try changing line 73 of
>> yt/frontends/flash/data_structures.py
>> from:
>>         self._handle = h5py.File(self.hierarchy_filename)
>> to:
>>         self._handle = h5py.File(self.hierarchy_filename, 'r')
>> it.....might work.  No promises. Let us know if that works.  If not, then
>> seeing a pastebin of your script would be helpful.
>> Best,
>> Sam
>> On Tue, Jul 5, 2011 at 6:03 PM, Cameron Hummels
>> <chummels at astro.columbia.edu> wrote:
>>> 
>>> Are you actively trying to create hdf5 files?  I know hdf5 won't
>>> overwrite an existing file by default, so perhaps if you've already
>>> created files in your loop, then try to run it again with those files
>>> still present, it will bail like this.
>>> 
>>> If that isn't the case, I'm not sure.  If you could send your script, or
>>> pastebin it, that might provide more insight.
>>> 
>>> Cameron
>>> 
>>> On 07/05/2011 05:34 PM, John ZuHone wrote:
>>>> Hi all,
>>>> 
>>>> I  have a loop that goes over a list of filenames I've generated, loads
>>>> parameter files for them, passes each pf object to a procedure which grabs
>>>> data from the file, and then loops back over. After the first iteration, I
>>>> get this crash:
>>>> 
>>>> Traceback (most recent call last):
>>>>   File
>>>> "/local/data/duce2/zuhone/src/ctools/filter_particle_velocities.py", line
>>>> 91, in <module>
>>>>     main()
>>>>   File
>>>> "/local/data/duce2/zuhone/src/ctools/filter_particle_velocities.py", line
>>>> 68, in main
>>>>     compute_turb_vels(pf, numVelCells, xp, yp, zp, vxp, vyp, vzp)
>>>>   File "/local/data/duce2/zuhone/src/ctools/compute_turb_vels.py", line
>>>> 15, in compute_turb_vels
>>>>     leaf_grids = [g for g in pf.h.grids if len(g.Children) == 0]
>>>>   File
>>>> "/local/data/duce2/zuhone/opt/yt-x86_64/src/yt-hg/yt/data_objects/static_output.py",
>>>> line 163, in hierarchy
>>>>     self, data_style=self.data_style)
>>>>   File
>>>> "/local/data/duce2/zuhone/opt/yt-x86_64/src/yt-hg/yt/frontends/flash/data_structures.py",
>>>> line 73, in __init__
>>>>     self._handle = h5py.File(self.hierarchy_filename)
>>>>   File
>>>> "/local/data/duce2/zuhone/opt/yt-x86_64/lib/python2.7/site-packages/h5py-1.3.1-py2.7-linux-x86_64.egg/h5py/highlevel.py",
>>>> line 797, in __init__
>>>>     self.fid = self._generate_fid(name, mode, plist)
>>>>   File
>>>> "/local/data/duce2/zuhone/opt/yt-x86_64/lib/python2.7/site-packages/h5py-1.3.1-py2.7-linux-x86_64.egg/h5py/highlevel.py",
>>>> line 842, in _generate_fid
>>>>     fid = h5f.create(name, h5f.ACC_EXCL, fapl=plist)
>>>>   File "h5f.pyx", line 87, in h5py.h5f.create (h5py/h5f.c:1378)
>>>> h5py._stub.IOError: File exists (File accessability: Unable to open
>>>> file)
>>>> 
>>>> It does not matter where in the list of files I start, so it's not due
>>>> to a single bad file. It's like it doesn't detect the file it's trying to
>>>> read somehow, so it tries to create it, but then it finds out it already
>>>> exists, and then aborts. Very odd.
>>>> 
>>>> Best,
>>>> 
>>>> John ZuHone
>>>> _______________________________________________
>>>> yt-users mailing list
>>>> yt-users at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>> 
>>> 
>>> --
>>> Cameron Hummels
>>> PhD Candidate, Astronomy Department of Columbia University
>>> Public Outreach Director, Astronomy Department of Columbia University
>>> NASA IYA New York State Student Ambassador
>>> http://outreach.astro.columbia.edu
>>> PGP: 0x06F886E3
>>> 
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> 
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> 
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> 
>> 
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> 
>> 
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org

More information about the yt-users mailing list