<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>I agree with Casey about the particle UI. And am +1 on changing the particle UI.</div><div><br></div><div>It should be easy to programmatically get the list of all particle types in a simulation and from there get the list of all particle fields associated with that particle type. The dict of dicts organization - obj.particles["<span>PopIICluster"]["position_x"]) - makes the most sense to me since you can use particles.keys() and particles["PopIICluster"].keys() functions.</span></div><br><div>
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Nathan Goldbaum<br>Graduate Student<br>Astronomy & Astrophysics, UCSC<br><a href="mailto:goldbaum@ucolick.org">goldbaum@ucolick.org</a><br>http://www.ucolick.org/~goldbaum</div></span></div></span></div></span></span>
</div>
<br><div><div>On Feb 24, 2012, at 10:46 AM, Casey W. Stark wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">Great stuff Matt.<div><br></div><div>+1 on the presented GeometryHandler design. It doesn't sound like it changes performance much and it's a nicer organization.<div><br></div><div>I like the chunking work, but I'm confused about syntax and what goes where. Does each data object define its own chunks method? In the chunks method, would it be clearer to return something like a "chunk" data object instead of self? This probably doesn't matter though, since a user would not care about type(ds).</div>
<div><br></div><div>I haven't worked with particle types in yt (it's all DM to me!), but I would prefer the `obj.particles["position_x"]` syntax. With the current design, I don't like that I have to inspect fields to see if they are particle fields or on the grid. This is fine now because I am familiar with the fields in my data, but I remember this being confusing as a new user.</div>
<div><br></div><div>For particle type selection, I think syntax like `obj.particles["<span style="">PopIICluster"]["position_x"]` or even `obj.particles.type("PopIICluster")["position_x"]` is clearer. With a tuple it almost looks like you are selecting two fields. If we go with the tuple though, can we make the particle type the first element? </span><span style="">I have no idea about mandating a particle type, but we can hash that out in the hangout.</span></div>
<div><br></div><div>Best,</div><div>Casey</div><div><br><div><br><div class="gmail_quote">On Fri, Feb 24, 2012 at 8:12 AM, Matthew Turk <span dir="ltr"><<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi all,<br>
<br>
This email is going to be long, but the changes I am proposing in it<br>
will require feedback. Next Monday at 1PM Eastern I'll be holding a<br>
hangout on Google+, if you would like to join in and discuss it with<br>
lower latency. (Of course, if nobody shows up, that's okay too.) We<br>
don't really have a detailed review process for design decisions like<br>
this, and I hope that this discussion will guide developing that kind<br>
of review process in the future.<br>
<br>
Everything described here is in the branch geometry_handling at<br>
<a href="https://bitbucket.org/MatthewTurk/yt-refactor" target="_blank">https://bitbucket.org/MatthewTurk/yt-refactor</a><br>
<br>
= Background =<br>
<br>
Geometry in yt has always been designed around two things:<br>
<br>
1) Grid patches<br>
2) Spatial location based on bounding boxes<br>
<br>
This has presented a number of excellent opportunities for optimizing<br>
patch-based AMR data. However, it has posed challenges for supporting<br>
particle-based datasets. (Particles inside patch-based data are<br>
somewhat easier, although if you want to see one of the worst pieces<br>
of yt, look no further than FakeGridForParticles, which mocks up<br>
particles as grids with varying x,y,z, so that existing<br>
fluid-selection routines can be used.) Furthermore, the state of<br>
Octree codes is not good. For relatively small (up to about 64)<br>
domains in RAMSES, we can do analysis with relative ease -- volume<br>
rendering, projections, fluid analysis, etc, all work very nicely.<br>
However, what is required is a costly step of regridding octs at the<br>
outset, to form grid patches with some efficiency. This regridding is<br>
not stored between runs, and it's not *efficient*.<br>
<br>
This has worked so far, but it is showing its strain. As it stands,<br>
yt cannot directly support SPH codes without somehow making SPH look<br>
like grids. Octree codes can't be supported without making them look<br>
like grid codes. Both of these are very slow operations. And on top<br>
of that, the current state of how data is read in, even for grid<br>
codes, is a bit gross. There are a number of conditionals, mechanisms<br>
for striding over data (i.e., "_get_grids()" which is called in<br>
parallel, and iterates over grids, and sometimes you have to call<br>
_get_point_indices and sometimes you don't), and probably worst of all<br>
a lot of unnecessary memory duplication because arrays in<br>
regions/spheres/etc are constructed by iterating over grids, filling<br>
in a list of arrays, and then concatenating that list into a second<br>
array.<br>
<br>
Furthermore, several times recently the question has come up of<br>
handling and supporting more generic geometries: spherical, polar,<br>
cylindrical. Unfortunately, because we're so heavily based around the<br>
idea of grids as is, these are difficult to support. All of our<br>
collision detection has been designed with rectilinear surfaces in<br>
mind. That being said, I'm really quite happy with how yt *functions*<br>
for patch-based grids. I'm just not sure that as it stands it is<br>
maintainable, and I think that addressing the geometry issues could<br>
ultimately also help out a great deal with scalability. (More on that<br>
below.)<br>
<br>
= Geometry Overhaul =<br>
<br>
For the last couple weeks, I have been working on an overhaul of the<br>
geometry system in yt, attempting to handle parallel striding in a<br>
format-agnostic way, deal with generating fields in as simple a way as<br>
possible, and also attempting to speed things up. I'm going to<br>
summarize below the changes that I've made, the new design, and the<br>
current status.<br>
<br>
== Design: Selectors ==<br>
<br>
As it stands, yt data objects (such as Region, Sphere, etc) are mostly<br>
thought of as conduits for data. They apply a criteria to the domain<br>
and select all of the fluid points or particles within a region or<br>
that fit a specific non-spatial criteria. I have extended this<br>
concept to construct very lightweight objects called selectors,<br>
similar to how ParticleIO is now conducted for Enzo. These are<br>
objects in Cython that accept either a set of bounding-box arrays or a<br>
set of spatial points and decide if those points are included in a<br>
given region. This is essentially a combination of the<br>
get_point_indices and get_cut_mask methods that currently exist.<br>
These have all been designed to be GIL-free, so that in the future<br>
they can be threaded using OpenMP (although it turns out not very much<br>
time is spent on them.) To date I have implemented spheres, regions,<br>
disks, cutting planes, slices, ortho rays, and grid collections. They<br>
all give bitwise identical results to the old method and are almost<br>
all substantially faster. (Cutting planes are particularly better,<br>
regions are slightly slower, and the others are all about 25-40%<br>
faster.)<br>
<br>
One primary advantage is that these can (and do) trivially provide<br>
counting methods, masking methods, and point-evaluation methods. So<br>
we can use the same routines to select particles and individual points<br>
(for collision detection) that we use for selecting grid cells. This<br>
will allow reuse of data object code with particle outputs, and is a<br>
primary advantage of the refactor.<br>
<br>
This also led to separating selection-based data containers from<br>
constructed data containers.<br>
<br>
== Design: Ditching the Hierarchy ==<br>
<br>
(Hopefully that heading caught some eyes.) In line with attempting to<br>
reduce Enzo-isms in the code, I have been moving away from<br>
specifically calling something a "hierarchy," and instead moving<br>
toward referring to geometry handlers. grids, grid_left_edges,<br>
grid_right_edges, and so on will no longer be guaranteed to appear<br>
hanging off these objects. The new class structure:<br>
<br>
GeometryHandler<br>
GridGeometryHandler(GeometryHandler)<br>
OctreeGeometryHandler(GeometryHandler)<br>
<br>
and eventually, other methods. The concept behind this is to isolate<br>
those items that *must* require domain knowledge, mesh knowledge, or<br>
particle/fluid distribution knowledge, and place those in the<br>
GeometryHandler. All items specific to patch based or block based AMR<br>
will go in GridGeometryHandler and so on. This will allow for<br>
fundamentally different method of IO for octrees from grids, for<br>
instance. The geometry handlers are all required to implement the<br>
following methods:<br>
<br>
_read_selection(self, fields, data_object, chunk)<br>
_chunk(self, data_object, chunking_style, ngz = 0)<br>
<br>
I will describe chunks later. Note that the first routine,<br>
_read_selection, accepts the names of the fields, and the data object<br>
that describes how to select which portions of a dataset participate<br>
in a given selection. For the specific case of Enzo (as I am still in<br>
design phase, I have only been working with Enzo data, although I will<br>
move on to FLASH and Boxlib next) read_selection calls a routine that<br>
counts and masks grids, batches operations to files, and then returns<br>
only the necessary data for a selection. The data object does no<br>
discarding of out-of-bounds or child-masked data.<br>
<br>
== Design: Chunking ==<br>
<br>
Currently, yt uses de facto chunking of data in the form of grids.<br>
When Derived Quantities want to handle individual grids, one at a<br>
time, they "preload" the data from whatever grids the<br>
ParallelAnalysisInterface thinks they deserve. These grids are<br>
iterated over, and handled individually, then the result is combined<br>
at the end. Profiles do something similar. However, both of these<br>
are de facto, and not really designed. They rely on calls to<br>
semi-private functions on data objects, manually masking data, on and<br>
on.<br>
<br>
To get around this ugly spot, I have started working on redesigning IO<br>
and data handling to function in "chunks." In this method, data<br>
objects will request that the geometry handler supply a set of chunks<br>
(the function _chunk, described above, conducts this task.) Chunks<br>
are of the form (IO_unit, size), where IO_unit is only ever managed or<br>
handled by _read_selection, above.<br>
<br>
To start the Chunks shuffling over the output, the code calls<br>
data_source.chunks(fields, chunking_style). Right now only "spatial",<br>
"io" and "all" are supported for chunking styles. This corresponds to<br>
spatially-oriented division, IO-conserving, and all-at-once (not<br>
usually relevant.) What happens next is kind of fun. The chunks<br>
function looks like this:<br>
<br>
def chunks(self, fields, chunking_style, **kwargs):<br>
for chunk in self.hierarchy._chunk(self, chunking_style, **kwargs):<br>
with self._chunked_read(chunk):<br>
self.get_data(fields)<br>
yield self<br>
<br>
Note what it does here -- it actually yields *itself*. However,<br>
inside the chunked_read function, what happens is that the attributes<br>
corresponding to the size, the current data source, and so on, are set<br>
by the geometry handler (still called a hierarchy here.) So, for<br>
instance, execution might look like this:<br>
<br>
for ds in my_obj.chunks(["Density"], "spatial"):<br>
print ds is my_obj<br>
print ds["Density"].size<br>
<br>
The first line will actually print True, but the results from the<br>
second one will be the size of (for instance) the grid it's currently<br>
iterating over. In this way, it becomes much easier to stride over<br>
subsets of data. Derived quantities now look like this:<br>
<br>
chunks = self._data_source.chunks([], chunking_style="io")<br>
for ds in parallel_objects(chunks, -1):<br>
rv = self.func(ds, *args, **kwargs)<br>
<br>
It chunks things, evaluates and then stores intermediate results.<br>
<br>
This is not meant to replace spatial decomposition in parallel jobs,<br>
but it *is* designed to enable much easier and *mesh-neutral* division<br>
of labor for parallelism and for IO. If we were to call chunk on an<br>
octree, it no longer has to make things look like grids; it just makes<br>
them look like flattened arrays (unless you chunk over spatial, which<br>
I haven't gotten into yet.)<br>
<br>
Essentially, by making the method of subsetting and striding over<br>
subsetted data more compartmentalized, the code becomes more clear and<br>
more maintainable.<br>
<br>
= Status =<br>
<br>
As it stands, I am able to select data very nicely -- this means<br>
slices, profiles, derived quantities all work. However (see below)<br>
particles still need some work and handling, and constructed data<br>
objects (smoothed covering grids, projections) have yet to be ported.<br>
<br>
= Open Questions =<br>
<br>
However, the main reason I am writing -- when the code is not yet<br>
fully functional! -- is to ask for advice and consideration on a few<br>
points.<br>
<br>
== Particle Types ==<br>
<br>
In the past, we've talked about handling particle types differently<br>
depending on the particle. Because this is destined for yt-3.0, I<br>
would like to propose that we break compatibility with old scripts in<br>
favor of clarity in our particle types. Specifically, rather than<br>
doing:<br>
<br>
obj["particle_position_x"][obj["particle_type"] == 1]<br>
<br>
We do,<br>
<br>
obj[("particle_positions_x", 1)]<br>
<br>
(or obj[("particle_positions_x", "PopIICluster")] if you are using a<br>
code that supports such expression.)<br>
<br>
And if a field is found to be a particle type field, but without a<br>
tuple, we simply raise a KeyError. We can emulate old behavior with:<br>
<br>
obj[("particle_position_x",None)]<br>
<br>
But I am inclined to say that I would rather *mandate* particle type<br>
selection, rather than merely allow it. This will lead to much<br>
greater clarity, I think.<br>
<br>
[+-][01] on<br>
<br>
== Particle Access ==<br>
<br>
Currently, we access particles off a data_object:<br>
<br>
sphere["particle_position_x"]<br>
<br>
Do we want to keep this, or move to being much more explicit, that we<br>
require sphere.particles["particle_position_x"] ? I can see both<br>
sides to this. Explicit is good, and would allow us to be both more<br>
clear with users and about fields (and with our IO methods) but<br>
allowing old behavior encourages not having to necessarily worry about<br>
if something is a fluid or a particle, which is a good thing I think.<br>
<br>
== User Survey ==<br>
<br>
Should we conduct a brief (only a handful of questions) polling users<br>
for their feelings on the changes or potential changes to<br>
nomenclature, particles, and so on? We could use it also as a time to<br>
identify other outstanding issues with yt.<br>
<br>
= Conclusion =<br>
<br>
Does this all make a sort of sense?<br>
<br>
[+-][01] on the design presented so far<br>
<br>
If you're interested in helping out, please get in touch. I'm still<br>
laying groundwork in some areas, but in other areas I would appreciate<br>
help and advice moving forward.<br>
<br>
Thanks to everyone who made it this far,<br>
<br>
Matt<br>
_______________________________________________<br>
yt-dev mailing list<br>
<a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
</blockquote></div><br></div></div></div>
!DSPAM:10175,4f47db1f213559955021!
_______________________________________________<br>yt-dev mailing list<br><a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org<br><br><br>!DSPAM:10175,4f47db1f213559955021!<br></blockquote></div><br></body></html>