[yt-users] Trouble with h5py + FLASH

[Matthew Turk]

Hi Nathan,

I'm not able to reproduce this; likely because I don't have any FLASH2
data.  I did fix a bug with particle counts in the most recent yt
branch tip, but that was for FLASH3 data.  Can you check a couple
things?

1) If you run with --detailed it should print out a couple helpful
variables, like the variables passed in: "fid = make_fid(name, mode,
fapl)".  In particular, what are those three variables' values?
2) Can you verify that there was no corruption in the variable names,
and that the HDF5 file still exists?  And, furthermore, can you verify
that h5ls -r works correctly on it?
3) Can you manually open the file with h5py, using h5py.File(..., "r")?

The change to h5py 2.0, which frankly I find unsettling, that files
remain open when the variables go out of scope could be problematic:

http://h5py.alfven.org/docs/intro/whatsnew.html#file-objects-must-be-manually-closed

Couple that with the fact that opening a file multiple times "leads to
undefined behavior" and I see this as deeply problematic.  (Plus, now
the entire h5e module has been removed ...)

My guess is that the behavior you are seeing is related to one of
these items.  We will have to sanitize all usages of h5py in yt.  I'm
going to go through and attempt to do this now (I take responsibility
for accepting the PR that updated h5py without double-checking the
behavior of h5py in the 2.0 series), but getting information about the
variables and the files will help address the issue you are seeing.
This I think serves as quite a good lesson for me, and the rest of the
project, that we should avoid upgrades without carefully considering
trickle-downs from minor (or major) API and behavioral changes.  This
is why yt has made an attempt to reduce dependencies across the board,
but as you can see even that effort is not always enough.

Thanks,

Matt


On Wed, Nov 2, 2011 at 8:06 PM, Nathan Goldbaum <goldbaum at ucolick.org> wrote:
> In fact, it looks like upgrading to h5py 2.0 actually broke my ability to
> read FLASH files - yt crashes whenever I try to directly access the data
> through a FLASHStaticOutput object. I've pasted a script that exhibits the
> problem as well as the corresponding crash traceback.
> If I go back to h5py1.3.1 the same script does not crash.
> -Nathan
> Nathan Goldbaum
> Graduate Student
> Astronomy & Astrophysics, UCSC
> goldbaum at ucolick.org
> http://www.ucolick.org/~goldbaum
> On Nov 2, 2011, at 10:41 AM, Nathan Goldbaum wrote:
>
> Hi Matt,
> Thanks for the suggestion.  Unfortunately, h5py still crashes:
>  http://paste.yt-project.org/show/1907/
> Nathan Goldbaum
> Graduate Student
> Astronomy & Astrophysics, UCSC
> goldbaum at ucolick.org
> http://www.ucolick.org/~goldbaum
> On Nov 2, 2011, at 5:13 AM, Matthew Turk wrote:
>
> Hi Nathan, John,
>
> This bug is annoying.  Any chance you could try with the latest
> install script, which uses h5py 2.0?  If you re-run the install script
> in place it should simply update the out-of-date dependencies.
>
> Thanks,
>
> Matt
>
> On Wed, Nov 2, 2011 at 2:12 AM, John ZuHone
> <jzuhone at milkyway.gsfc.nasa.gov> wrote:
>
> Hi Nathan,
>
> I would suspect that your grid data is probably fine, yes. I'm pretty sure
> the relevant data structures are essentially identical.
>
> Happy FLASH-ing!
>
> Best,
>
> John
>
> On Nov 2, 2011, at 2:10 AM, Nathan Goldbaum wrote:
>
> Thanks for looking into this, John.  Adding those two lines fixes my
> problem.
>
> No worries about yt not supporting FLASH2 particle data - as you can see
> it's not a problem to read it in directly.
>
> I haven't had any problems reading in my FLASH2 AMR data using yt.  I get
> results that are consistent with my old IDL scripts so I don't think there
> are any conversion issues.
>
> On Nov 1, 2011, at 11:00 PM, John ZuHone wrote:
>
> Hi Nathan,
>
> You'd have to double-check with Matt, but as I recall we didn't design the
> yt interface to support FLASH2 data. Now, I wouldn't be surprised if it does
> for the most part, since the grid data is written in almost exactly the same
> format by design to ease people switching over to FLASH3.
>
> The particles, however, are a different story. In FLASH2, the particles are
> a composite HDF5 data structure of ints and doubles, whereas in FLASH3 they
> are a 2D array of doubles. In the latter case, simplicity was chosen over
> design sophistication. I know for a fact that yt only supports particle data
> in FLASH3.
>
> I'm looking at this file of yours now... I was not aware that the way
> compound data types work in HDF5 that you get Python dictionaries when you
> read them in. That is a lot simpler than what I have to do to read FLASH3
> particles.
>
> In any case, I reproduce the error you get. If you add the lines
>
> del ParticleData
>
> del f
>
> after your "f.close()" call, I no longer get the error. This is what we had
> to do with the previous issue you mentioned. For some reason the FLASH files
> are sometimes a bit picky about having all objects allocated from the
> previous read deleted before accessing the file again.
>
> Best,
>
> John
>
>
>
> _______________________________________________
>
> yt-users mailing list
>
> yt-users at lists.spacepope.org
>
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
> _______________________________________________
>
> yt-users mailing list
>
> yt-users at lists.spacepope.org
>
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
> !DSPAM:10175,4eb1340175682950323881!
>
>
>
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>