[yt-dev] A "taxonomy" for fields

[Matthew Turk]

Hi Jeff,

On Wed, Oct 16, 2013 at 11:55 PM, j s oishi <jsoishi at gmail.com> wrote:
> Hi Matt,
>
> If I'm understanding you properly, I am strongly in support of this. My
> understanding is that you proposing a hierarchical taxonomy, such that all
> data access will now be two layered. Thus one coudl do something like this
>
> dd = pf.h.all_data()
>
> dd['gas','density']
> dd['deposited','density']
>
> then below this, 'gas' will resolve to 'enzo' or 'flash' or 'pencil',
> performing unit conversions, and something similar for "deposited"? Thus, it
> would also be possible to get completely raw, unconverted code units just by
> doing
>
> dd["enzo", "Bx"]?
>
> If so, I think this is a great idea. We could also then have a set of
> defaults for single-key dict style access, e.g. dd['Density'] would do
> something by default. I think this will also make all derived fields work in
> any units in a much easier way.
>

Yup, this is basically what I'm proposing, and what I ended up testing
out an implementation of yesterday.  The defaults are actually already
in the 3.0 code base, so it will search for a field if you don't
specify the "field type" first.  Additionally, I want to note that
these fields are all being discussed in the context of the *default*
units, but since they are being implemented in the unitful arrays, it
only governs the defaults, not the limits -- in fact, they will all be
able to be freely converted back and forth using the YTArray
functionality Casey and Nathan implemented.

There is a "field type" and "field name" hierarchy in the 3.0
codebase.  Currently this mostly includes "deposit", "smooth", "gas",
"[some particle type]", "index", and new particle types can be created
by filtering or unioning other particle types.

So here's a more concrete laying out of the system:

 * Enumerate all the fields that "yt knows about" for various code
frontends.  This is not a comprehensive list, but we would explicitly
name the fields that yt is free to use in its derived fields.  As an
example, we would have "density" (lowercase d), "thermal_energy" and
the like.
 * Code frontends would create "on-disk" fields, called, say, "enzo"
which are defaulted to code units.  They'd create "alias" fields which
then say, this field corresponds to ("gas", "density").
 * We'd default units for on-disk fields to code, and yt-fields to cgs
(or something else; this is flexible by frontend and instantiation.)

I'm not entirely sure how to handle particles since each particle type
is currently its own namespace.  Maybe just prefixing with the
frontend in the namespace of the fields is appropriate to get a
code-unit default.  Regardless, here is a primitive set of examples
from the Enzo frontend.  Note that the "gas" fields are all defaulted
into CGS, and the "enzo" fields are in code, and that you can convert
back and forth by calling methods on the *arrays themselves* since
they're YTArray objects.

Script:

http://paste.yt-project.org/show/3951/

Results:

http://paste.yt-project.org/show/3950/

(I haven't applied the same logic to the particle fields yet.)

As you noted, there are defaults -- we can set up a search order for
field name resolution.  This is already in place, but in a primtiive
way as there's just a "default fluid type" and "default particle type"
and so on.  Additionally, one thing I'm really happy about is that
with these changes we're getting to the point of setting it up such
that you can load field plugins pretty easily -- and field detection
is considerably simpler.

The code for this (which currently only works with the enzo frontend)
is in http://bitbucket.org/MatthewTurk/yt-3.0 in the bookmark
"unitrefactor".  Here are a few field plugins:

https://bitbucket.org/MatthewTurk/yt-3.0/src/2d34221450e78446235bd322e6ea42233209d223/yt/fields?at=yt-3.0
https://bitbucket.org/MatthewTurk/yt-3.0/src/2d34221450e78446235bd322e6ea42233209d223/yt/fields/magnetic_field.py?at=yt-3.0
https://bitbucket.org/MatthewTurk/yt-3.0/src/2d34221450e78446235bd322e6ea42233209d223/yt/fields/fluid_fields.py?at=yt-3.0

(Note that all of these have 'ftype' arguments -- which means they can
be applied to *any* field type, for instances of, say, multiple fluids
or 'ghost' fluids, or even possibly loading in two different outputs
simultaneously and comparing them.)

and using this I simplified how the enzo thermal energy field, and the
finite stencils, are created:

https://bitbucket.org/MatthewTurk/yt-3.0/src/2d34221450e78446235bd322e6ea42233209d223/yt/frontends/enzo/fields.py

The system this sets up is eventually going to be refactored, once I
have the kinks worked out for Enzo and stream.  I am rather liking the
mechanism of having a list of aliases, units and so on, and then
converting freely between them.

What do you think about particles?  How should the namespacing of
particles work, in case we have the situation of multiple particle
types?  What I had been doing was dumping them into ftype == "io", but
I think that will not work for multiple inhomogeneous particle types.
Would something like "[frontend]_[fluidtype]"
"[frontend]_[particletype]" be better?  I am kind of against a
three-level hierarchy in objects, but having the string keys be
meaningful (as long as we did no parsing of them) would be okay, I
think.

-Matt

> j
>
>
> On Tue, Oct 15, 2013 at 9:54 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>>
>> Hi all,
>>
>> Nathan and I have been doing some work on units recently, and how to
>> handle on-disk fields, derived fields, and so on came up a little bit.
>>
>> I speculated a little bit about this and was wondering if this is a
>> good enough idea to propose a modification to YTEP-0003:
>>
>> https://ytep.readthedocs.org/en/latest/YTEPs/YTEP-0003.html
>>
>> Basically, what about having a taxonomy of fields?  What this would be:
>>
>>  * An enumeration of the fields that yt may call upon that it may
>> expect to find in a dataset.
>>  * Explicit translations, rather than de facto/implicit translations.
>>  * Calling on-disk fields by a fluid name other than "gas"
>>
>> So what I mean here is, right now we have support for multiple
>> "fluids", but the only ones we have are "gas" and "deposit".  I'm
>> proposing that code frontends:
>>
>>  * Define their own fluid types that mean, fields on disk.  These will
>> by default be in code units.  So you could access
>> dataobj["enzo","Density"] and it will return a YTArray by default in
>> code units that is the "Density" field.
>>  * Frontends provide translations of these to yt-specific fields.
>> This would mean, for the example above, ["gas","density"].  (Note that
>> by default the search process would be "gas", then "enzo" in this
>> case, so you could still ask for "Density" but it would default to a
>> unitful code-unit array, and yt knows how to convert all of that.)
>>  * Somewhere (YTEP?), enumerate the fields that yt knows how to manage
>> and operate on.  This way we can also more easily detect derived
>> fields, and indicate to frontend developers how to hook their code up
>> to yt.
>>
>> In the unit_refactor I've started working toward this by adding:
>>
>> field_info.add_output_field( ... )
>> field_info.alias( ... )
>>
>> to set these things up a bit more clearly.  It also devises a
>> different way of doing field plugins, to make it more deliberate and
>> reduce a LOT of the nastiness of the process.
>>
>> So what I'm looking for feedback on:
>>
>>  * What about new fluid type for on-disk datasets?
>>  * What about enumerating field names that yt knows about?
>>
>> -Matt
>>
>> PS I know 3.0alpha4 was scheduled for yesterday.  I was a bit
>> jet-lagged, but I'll put out what I can as soon as I can for a release
>> announcement.  But, the recent changes are in yt_analysis/yt now!
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>