[yt-users] Crash when iterating over a long list of FLASH outputs

Nathan Goldbaum goldbaum at ucolick.org
Tue Jan 17 19:17:01 PST 2012


Hi Matt,

A test script that opens and closes the file using h5py runs with no problem.

Adding pf._handle.close() to the end of the loop in my original script seems to fix the issue I brought up in my previous e-mail.  Thanks for the suggestion.

Unfortunately, this leads to another failure somewhere else in the loop.  At some point, it tries to close a file and catches an exception:

Exception ValueError: 'invalid file identifier (Invalid arguments to routine: Inappropriate type)' in <bound method FLASHStaticOutput.__del__ of DiskDyn_hdf5_plt_cnt_1741> ignored

Some googling returns this discussion:

http://code.google.com/p/h5py/issues/detail?id=220

Apparently this error happens when h5py tries to close a file that has already been closed.  Can someone give me more details about how yt  closes files?  I appear to be triggering a race condition somehow.

-Nathan

On Jan 17, 2012, at 6:56 PM, Matthew Turk wrote:

> Hi Nathan,
> 
> On Tue, Jan 17, 2012 at 8:49 PM, Nathan Goldbaum <goldbaum at ucolick.org> wrote:
>> Hi all,
>> 
>> I'm having a curious issue.  I'm trying to open a large number of FLASH
>> parameter files, read in some vital statistics about each file, and then
>> write out the data to a ascii table.
>> 
>> However, for some reason I'm finding that this crashes after 1020 data files
>> have been opened. There's nothing special about the 1020th file since I can
>> open it and inspect it interactively.  This may have something to do with me
>> running this script on the lustre filesystem on Pleiades at NASA Ames.
> 
> This is puzzling.  My guess is that something about HDF5's internal
> reference counting, or h5py, or something like that is causing these
> issues.  My initial guess for a workaround would be to try os.fork,
> but having now tested it I'm not sure that will reset the file handles
> and so forth.
> 
>> 
>> My test script is pasted here:
>> 
>> http://paste.yt-project.org/show/2039/
>> 
>> And a sample traceback is pasted here:
>> 
>> http://paste.yt-project.org/show/2040/
>> 
>> It looks like the crash happens when yt tries to open the h5py handle for
>> the data file.  It crashes as if the data file doesn't exist, even though it
>> really does, I promise.  This seems to be related to an h5py issue that John
>> ZuHone noticed back in July:
>> 
>> http://lists.spacepope.org/htdig.cgi/yt-users-spacepope.org/2011-July/001703.html
>> 
>> Adding 'del pf._handle' to the end of my loop does not fix the issue.
> 
> Can you reproduce it inside a small loop, without yt, just using h5py
> and calling .close() on each file?
> 
> -Matt
> 
>> 
>> Thanks for your help with this!
>> 
>> Nathan Goldbaum
>> Graduate Student
>> Astronomy & Astrophysics, UCSC
>> goldbaum at ucolick.org
>> http://www.ucolick.org/~goldbaum
>> 
>> 
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> 
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
> 
> !DSPAM:10175,4f1634dc34752151017521!
> 




More information about the yt-users mailing list