[yt-users] Crash when iterating over a long list of FLASH outputs

[Matthew Turk]

Hi Nathan,

On Tue, Jan 17, 2012 at 8:49 PM, Nathan Goldbaum <goldbaum at ucolick.org> wrote:
> Hi all,
>
> I'm having a curious issue.  I'm trying to open a large number of FLASH
> parameter files, read in some vital statistics about each file, and then
> write out the data to a ascii table.
>
> However, for some reason I'm finding that this crashes after 1020 data files
> have been opened. There's nothing special about the 1020th file since I can
> open it and inspect it interactively.  This may have something to do with me
> running this script on the lustre filesystem on Pleiades at NASA Ames.

This is puzzling.  My guess is that something about HDF5's internal
reference counting, or h5py, or something like that is causing these
issues.  My initial guess for a workaround would be to try os.fork,
but having now tested it I'm not sure that will reset the file handles
and so forth.

>
> My test script is pasted here:
>
> http://paste.yt-project.org/show/2039/
>
> And a sample traceback is pasted here:
>
> http://paste.yt-project.org/show/2040/
>
> It looks like the crash happens when yt tries to open the h5py handle for
> the data file.  It crashes as if the data file doesn't exist, even though it
> really does, I promise.  This seems to be related to an h5py issue that John
> ZuHone noticed back in July:
>
> http://lists.spacepope.org/htdig.cgi/yt-users-spacepope.org/2011-July/001703.html
>
> Adding 'del pf._handle' to the end of my loop does not fix the issue.

Can you reproduce it inside a small loop, without yt, just using h5py
and calling .close() on each file?

-Matt

>
> Thanks for your help with this!
>
> Nathan Goldbaum
> Graduate Student
> Astronomy & Astrophysics, UCSC
> goldbaum at ucolick.org
> http://www.ucolick.org/~goldbaum
>
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>