[yt-dev] RFC: (yt-3.0) Geometry handling redesign

[Matthew Turk]

Hi everybody,

Just a quick reminder -- at 1PM today I'll be holding a G+ hangout
about geometry refactoring.  I've prepared a bit of a presentation,
and I'll also go over some code, but I mainly want to get some
feedback and talk about ideas!  This will be a great time to
brainstorm as well as provide feedback on what is already in place.
The hangout will be on my profile, which is here:

https://plus.google.com/106495956319144885505/

That's in just about three hours.  No further reminders will be forthcoming.

I also wanted to thank everyone for replying, and I'll respond briefly
to comments here.  One thing I really want to convey, though, is that
this refactor is mostly about stopping thinking of everything in terms
of *grids*.  This is the myopia that would prevent yt from really
moving forward; to that end, I do not want *any* data objects in yt to
directly access grid objects unless they are specifically tasked with
handling grid geometry or handling a

Casey -- returning a chunk object is probably cleaner, but I didn't
want to add on a bunch of new overloads and potentially duplicated
logic.  I don't see reading in a chunked fashion as something that is
commonly (or ever) exposed to analysis scripts, and if a script author
does choose to use chunking, they should know what they are doing.
Each data *object* (i.e., sphere, region, etc) does *not* define its
own chunking routine, but rather calls the chunking routine from the
geometry handler, which is something like a grid-based handler,
particle-based, oct-based.  These can be overridden by specific codes
(for instance, if IO chunking proceeds differently in Nyx versus in
Enzo, even though they are both grid based codes.)

Nathan -- I like the syntax you and Casey have come up with.  We can
also add [:] or "all" for particle typing.

Chris -- yup, data_objects needs rewriting.  That's what I have been
doing, and I got first light on plotting slices, cutting planes and
profiles this weekend.  The harder one will be doing data objects that
require some degree of construction, like profiles and covering grids.
 I hope to start work on those soon.  As for if this would help with
SPH, the answer is emphatically yes!  While actually calculating fluid
quantities in particles that require neighbor searches will still be
challenging, the process of loading and looking at SPH data in the raw
should be greatly improved.  As an example of what we will be able to
do with Octs, we should be able to load ART data, store the index in
memory, chunk by whatever is best for IO (sorted by resolution,
perhaps) and add to the exported, resorted octree without any
regridding.  I'd like to work on this with you very soon.

Stephen -- I am not yet sure about implementation of particle IO, but
I would see selection of all particles in a given region being
available via something like obj.particles[:]["whatever_field"].  I
think we can mostly avoid concatenation in this way.  As for accessing
things via .grid[field], I am -1 on this since I don't really want to
be thinking about grids.  We could consider .fluid, but in general I
think I am okay with continuing to keep fluids first class and
particles separate from them.  I am willing to be convinced otherwise.

Thanks everybody,

Matt

On Sat, Feb 25, 2012 at 7:26 PM, Stephen Skory <s at skory.us> wrote:
> Hi all,
>
> One thought comes to mind with regards to selecting particle types as
> part of the new method: There needs to be a simple and memory
> efficient way to select multiple types of particles (or all types).
> For example, for the halo finders, sometimes only dark matter is
> wanted, and other times stars (or perhaps even only certain types of
> stars) are wanted as well. In the past we've done things to avoid
> na.concatenate with particles because it fragments memory. I think
> Matt has thought of this already, but I'd like to extend this
> discussion to how to allow users to select particle types without
> having to do na.concatenate manually (or have it happen behind the
> scenes).
>
> With regards to obj['particle_position_x'] or
> obj.particles['position_x'], I think the argument for changing it is
> not a bad one, but that if we do this, I think it would make the most
> sense to extend it to other things, like obj.grid['Density'], for
> continuity's sake.
>
> I'm for a user survey if we think we'll get replies. Have users
> generally been forthcoming with opinions on technical stuff like this
> before?
>
> I'll see you in the chat on Monday!
>
> --
> Stephen Skory
> s at skory.us
> http://stephenskory.com/
> 510.621.3687 (google voice)
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org