[yt-dev] Dealing with overlapping grids in Athena data

[John Zuhone]

Hi all,

I wanted to call attention to an oddity I discovered yesterday with
regard to some Athena data I'm working with.

For background, Athena data consists of "domains" which may or may not
exist on different levels of refinement. They adhere to some fairly
strict rules, which may be found here:

https://trac.princeton.edu/Athena/wiki/AthenaDocsSMRGeom

Domains are further subdivided into "grids". If the run was done on a
single processor, then each domain just has one grid, and the two are
identical. If it was a parallel job, then a domain will usually have
multiple grids. The important point here is that there is no
requirement that a grid on a given refinement level needs to be fully
self-contained within a parent grid on the level just below it (though
Athena does adhere to refine_by = 2).

To further complicate things, for parallel runs the grids are split
into different vtk files in different directories for each processor
and also different directories for each refinement level. We've
constructed the Athena frontend so that this data may be read as well
as data that was produced by a serial run. There is a tool in the
Athena source, join_vtk, which allows one to take the data from a
parallel run and merge it into single vtk files, which has the
consequence of reducing the number of grids to one per domain.

This brings me to the dataset I mentioned in the beginning. It
consists of a simple set of 5 3D cubical nested domains, all centered
on the domain center, which all of them except the highest-level
domain exactly half the size on a side of the previous one. The last
one is 3/4 the size on a side of the previous one. The run was done on
512 processors, and each domain is split up into 512 grids.

The result looks like this:

https://hub.yt-project.org/nb/bfd6uv

As you can see, the issue is that the grids on the highest refinement
level really don't have a well-defined notion of parent grids, because
these grids are overlapping multiple grids on the parent level.
However, at the cell level there are no overlaps of this type--every
cell on a given level is fully contained within a cell on the level
below it.

Here is the same data, except that it has been merged with join_vtk
and now there are only five grids:

https://hub.yt-project.org/nb/7qxvas

In this case, every grid has a well-defined parent, so everything looks correct.

In fact, I realized today that there is no provision in the Athena
frontend for establishing parent-child relationships amongst grids--so
apparently this is being determined somewhere in the code as a
fallback when the frontend doesn't provide this information?

I'd just like to open this up for discussion as to how we might make
it possible to at least work with this data in a limited way, since it
seems like it's a situation that might come up often enough. I've
created an issue:

https://bitbucket.org/yt_analysis/yt/issue/636/overlapping-grids-in-athena-smr-data

Best,

John Z

-- 
John ZuHone

Postdoctoral Researcher
NASA/Goddard Space Flight Center

jzuhone at gmail.com
john.zuhone at nasa.g