[Yt-dev] Packed AMR hierarchy problem

[Stephen Skory]

Matt & Britton,


> > Commenting out __obtain_filenames will work as long as the paths in the

Commenting out __obtain_filenames in trunk doesn't seem to do it for me. My dataset is located where the full pathnames say they are. It's dying inside of _save_data in HierarchyType where it's trying to save the .yt file, due to h5py being unwilling to save an empty array (h5py.h5.ArgsError: Zero sized dimension for non-unlimited dimension). From pdb:

-> arr = myGroup.create_dataset(name,data=array)
(Pdb) name
'DataFields'
(Pdb) array
[]

> LAst night I discovered another set of data that was broken with the
> current trunk parser but works with the new hg parser -- there seems

This is a WoC-created dataset, so that may introduce more wrinkles into this.

...

I've spent the last while trying to get this to work in the hg yt tip, but I think I need some help. I've discovered a couple things in HierarchyType.py that seems wrong. I haven't committed them because I'm not sure I understand what's going on well enough to do that.

h5py doesn't like directly accessing the file pointer. Adding an intermediate step fixes this hangup:

diff -r d2386cb60c6d yt/lagos/HierarchyType.py
--- a/yt/lagos/HierarchyType.py    Wed Nov 18 10:07:25 2009 -0800
+++ b/yt/lagos/HierarchyType.py    Thu Nov 19 10:06:07 2009 -0800
@@ -345,7 +345,8 @@
             "%s.hierarchy" % (pf.parameter_filename))
         harray_fn = self.hierarchy_filename[:-9] + "harrays"
         if os.path.exists(harray_fn):
-            self.num_grids = h5py.File(harray_fn)["/Level"].len()
+            harray_fp = h5py.File(harray_fn)
+            self.num_grids = harray_fp["/Level"].len()
         elif os.path.getsize(self.hierarchy_filename) == 0:
             raise IOError(-1,"File empty", self.hierarchy_filename)
         self.directory = os.path.dirname(self.hierarchy_filename)

These three datasets in the .harrays file are not 1D, they're 3D, so trying to put them in a flatiter array won't work. I might be doing something wrong here, but as it is doesn't seem quite right. Do we want to keep the self...arrays as flatiters?

@@ -483,9 +484,9 @@
     def _parse_binary_hierarchy(self):
         mylog.info("Getting the binary hierarchy")
         f = h5py.File(self.hierarchy_filename[:-9] + "harrays")
-        self.grid_dimensions.flat[:] = f["/ActiveDimensions"][:]
-        self.grid_left_edge.flat[:] = f["/LeftEdges"][:]
-        self.grid_right_edge.flat[:] = f["/RightEdges"][:]
+        self.grid_dimensions[:] = f["/ActiveDimensions"][:]
+        self.grid_left_edge[:] = f["/LeftEdges"][:]
+        self.grid_right_edge[:] = f["/RightEdges"][:]
         levels = f["/Level"][:]
         parents = f["/ParentIDs"][:]
         procs = f["/Processor"][:]

With these fixes I'm hanging in the "for level in xrange(self.max_level+1):" loop in "_initialize_level_stats()" because self.max_level+1 is much much greater than the actual max:

(Pdb) self.max_level
2867
(Pdb) self.level_stats
rec.array([(8, 16777216, 0), (239, 256240, 1), (2, 944, 2), (1, 384, 3),
       (3, 9536, 4), (2, 4992, 5), (2, 1488, 6), (1, 384, 7), (1, 384, 8),
       (1, 216, 9), (1, 480, 10), (1, 384, 11), (2, 928, 12), (1, 512, 13),
       (1, 288, 14), (3, 3200, 15), (1, 1200, 16), (1, 800, 17),
       (1, 288, 18), (1, 512, 19), (1, 640, 20), (1, 384, 21),
       (1, 216, 22), (1, 512, 23), (1, 288, 24), (1, 192, 25),
       (1, 216, 26), (2, 8608, 27), (3, 4224, 28), (1, 384, 29),
       (1, 1440, 30), (1, 640, 31), (1, 144, 32), (3, 1120, 33),
       (1, 216, 34), (2, 728, 35), (1, 512, 36), (2, 432, 37),
       (1, 384, 38), (1, 288, 39), (1, 288, 40), (1, 216, 41),
       (2, 504, 42), (1, 5040, 43), (1, 3528, 44), (1, 1680, 45),
       (1, 640, 46), (2, 1216, 47)], 
      dtype=[('numgrids', '<i4'), ('numcells', '<i4'), ('level', '<i4')])
(Pdb) self.grid_levels
array([[   0],
       [   0],
       [   0],
       ..., 
       [2865],
       [2866],
       [2867]], dtype=int32)
(Pdb) self.grid_levels.size  
5403
(Pdb) self.grid_levels.max()
2867
(Pdb) MAXLEVEL
48

Why is MAXLEVEL different than self.max_level?

Thanks!

 _______________________________________________________
sskory at physics.ucsd.edu           o__  Stephen Skory
http://physics.ucsd.edu/~sskory/ _.>/ _Graduate Student
________________________________(_)_\(_)_______________