[yt-users] Accessing data

[Matthew Turk]

Hi Stuart,

On Wed, Nov 6, 2013 at 6:37 AM, Stuart Mumford <stuart at mumford.me.uk> wrote:
> Hello,
>
> I am starting upon an adventure of actually using yt and I am already
> getting the impression I may be using a chainsaw to cut kindling.

Well, first off, glad to hear you're giving yt a shot!  :)  I hope
it's not chainsaw/kindling territory.  :)

>
> My data is uniform grid, not that large (128^3 for now) and I need to access
> the fields, both on disk and derived in 3D array format. I have tried using
> a covering_grid thus:
>
> ```
> ts =
> yt.load("/archive/io_testing/3D_tube128_Slog_p90_A20r2-3_B005/3D_tube128_Slog_p90_A20r2-3_B005_*.gdf")
>
> ds = ts[10]
>
> cg = ds.h.covering_grid(0, ds.domain_left_edge, ds.domain_dimensions)
>
> cg['density_pert']
> ```
>
> which soemtimes seems to give me an array full of -999 and sometimes gives
> me this lovely traceback:
> ```
>
> ---------------------------------------------------------------------------
> DataCubeError                             Traceback (most recent call last)
> <ipython-input-55-62e42995fe53> in <module>()
> ----> 1 cg['density_pert']
>
> /home/stuart/BitBucket/yt/yt/data_objects/data_containers.pyc in
> __getitem__(self, key)
>     316             if key not in self.fields:
>     317                 self.fields.append(key)
> --> 318             self.get_data(key)
>     319         return self.field_data[key]
>     320
>
> /home/stuart/BitBucket/yt/yt/data_objects/data_containers.pyc in
> get_data(self, fields)
>    3757         for i, grid in enumerate(self._grids):
>    3758             if self._use_pbar: pbar.update(i)
> -> 3759             count -= self._get_data_from_grid(grid, obtain_fields)
>    3760             if count <= 0: break
>    3761         if self._use_pbar: pbar.finish()
>
> /home/stuart/BitBucket/yt/yt/data_objects/data_containers.pyc in
> save_state(self, grid, field, *args, **kwargs)
>      79         old_keys = grid.field_data.keys()
>      80         grid.field_parameters = self.field_parameters
> ---> 81         tr = func(self, grid, field, *args, **kwargs)
>      82         grid.field_parameters = old_params
>      83         grid.field_data = YTFieldData( [(k, grid.field_data[k]) for
> k in old_keys] )
>
> /home/stuart/BitBucket/yt/yt/data_objects/data_containers.pyc in
> _get_data_from_grid(self, grid, fields)
>    3801             c_fields, g_fields,
>    3802             self.ActiveDimensions, grid.ActiveDimensions,
> -> 3803             grid.child_mask, self.domain_width, ll, 0)
>    3804         return count
>    3805
>
> DataCubeError: DataCubeGeneric: Invalid parameters.
> ```
>
> I only want my arrays :'(

Yes, we definitely want you to get your arrays.  -999 means an unfilled cell.

Which version of yt are you using?  Can you get the changeset hash?
And, maybe we can run some diagnostics to check that the data is in
fact correctly read in.  What's the output of ts[10].h.print_stats() ?

>
> On another, related, note:
> I have a bit of code that executes in serial at the top of a otherwise
> embarrassingly parallel piece of vtk code, I need the most optimal io
> possible for this, as it is all io limited, how well is yt suited to getting
> one slice out of my gdf files as fast as humanly possible?

It depends.  We actually found for *many* types of data, where the
grids are below a threshold, that it's faster to read in and discard
slices than to read hyperslabs.  This may not be the case now that we
are using low-level h5py code.  But I'd actually say, just give it a
shot?

>
> Stuart
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>