[Yt-dev] Field definitions, derived fields, whats-in-a-file and the "deliberate_fields" branch

david collins antpuncher at gmail.com
Wed Nov 9 14:02:25 PST 2011


Hi, Matt--

This is exciting.  I have dabbled a little with units and derived
fields in my time.  I should note that I did all these tests before I
got a chance to read your examples, so I didn't incorporate any of the
modifications to add_field/add_enzo_field that you mentioned.

I've checked this out and banged on it.  I ran into one bug, one odd
warning, have one feature question, and several successes.  I think I
fixed the bug.  Everything I've tried seems to work so far, even my
old wonky scripts.

The  bug:  It fails with a syntax error (full trace is below.)  on
"import yt.mods" From other bits of code, it looks like, in the
definition of AMRBooleanRegionBase, (line 3407 in data_containers.py)

_con_args = {"regions"}

should be

_con_args = ("regions")

Fixing that lets the import yt.mods work.

Then reading my dataset, I tried some tests.  For the record this
dataset was from and Enzo who's closest parent is enzo1.0.1, with all
of my horrible additions: old unpacked hdf5 plus MHDCT garbage.

>>> pf = load(filename)
>>> print pf.h.grids[0]['Density'].min(), 'Denisty'
>>> print pf.h.grids[0]['MagneticField_C_1'].min(), "MagField"
0.018 'Density'
-13.2, "MagField"

which works fine.  If I start a new python session,
>>> pf2 = load(filename, conversion_override={'MagneticField_C_1':100}
>>> print pf2.h.grids[0]['MagneticField_C_1'].min()
-1328

which is also successful.  Also successful were similar tests using my
overloaded EnzoStaticOutput classes.  I also tried other derived
fields, without changing any of my definitions, and the worked
correctly, as far as I can tell.

The one minor I've run into involves a hack-tastic I did to deal with
uncertainty in field names.  I think this might be what your
"translate" thing is for? This field  throws the exception when I load
the parameter file, but otherwise works--  Does your new stuff have a
better fix for this kind of code?

def _Bxu(field,data):
    try:
        return data['Bx']
    except:
        try:
            return data['MagneticField_C_1']
        except:
            print "no magnetic field x"

add_field('Bxu',function=_Bxu,validators=[ValidateGridType()], take_log=False)

I call this a roaring success.

Thanks a ton!
d.

Trace for the first bug:
 File "go", line 20, in <module>
    from yt.mods import *
  File "/Users/dccollins/local/src/yt_turk/yt/mods.py", line 44, in <module>
    from yt.data_objects.api import \
  File "/Users/dccollins/local/src/yt_turk/yt/data_objects/api.py",
line 31, in <module>
    from grid_patch import \
  File "/Users/dccollins/local/src/yt_turk/yt/data_objects/grid_patch.py",
line 35, in <module>
    from yt.data_objects.data_containers import YTFieldData
  File "/Users/dccollins/local/src/yt_turk/yt/data_objects/data_containers.py",
line 3407
    _con_args = {"regions"}
                          ^
SyntaxError: invalid syntax



On Wed, Nov 9, 2011 at 1:25 PM, j s oishi <jsoishi at gmail.com> wrote:
> Hi Matt,
>
> This sounds like a much needed overhaul. However, I'm not quite clear
> on exactly what this will entail, or how it will work once
> implemented. Could you or Casey provide an example of a new field or
> two, demonstrating how these dictionaries, fallbacks, and Null
> functions work? I think this is likely a very simple thing, but I'm
> having trouble visualizing it.
>
> thanks,
>
> j
>
> On Wed, Nov 9, 2011 at 9:05 AM, Matthew Turk <matthewturk at gmail.com> wrote:
>> Hi all,
>>
>> Over the last couple months, Casey and I have been working -- on and
>> off! -- on a new branch of the code called "deliberate_fields."  This
>> branch will change, in a substantial but easy-to-update way, how
>> fields are handled in yt.
>>
>> I recognize this email is long.  But if you use non-standard fields, a
>> bunch of derived fields, unit modifications, any of that, it may
>> affect you.  So I *please* ask that you read it and, if you like,
>> contribute back to the discussion.
>>
>> This is one of the items I really want to have done for a hypothetical
>> 2.3 release.
>>
>> = Background =
>>
>> The way fields work currently was designed a bit haphazardly.  They
>> use FieldInfoContainers, objects which share state and which contain
>> unions of the known derived fields and the known IO-based fields.  One
>> of the problems with this is that the only thing that separates a
>> derived field from a known field is that function that generates the
>> field: the IO-based fields all use a lambda which returns None, and
>> the non-IO based fields return actual fields.  This is pretty
>> sub-optimal, and it actually lands us in trouble when (for instance)
>> we have fields wandering around named things like "Thermal_Energy" and
>> "ThermalEnergy"; the mechanism by which one is selected and the other
>> not is problematic, and to get around infinite recursion, hacks have
>> had to be applied.
>>
>> As it stands, to find a field, the shared-state "field info" on a
>> parameter file is queried; this then will try to check universal
>> fields.  But because of how the fields are stored, the field info
>> cascade can also operate in reverse.  The big problem is that the
>> field selection mechanism doesn't seem to have a bus factor >= 1.0.
>> And, it has a number of hacks to make it work with conflicting field
>> definitiosn and the like.
>>
>> Unfortunately, layering these hacks on top of each other makes it much
>> harder for other codes to be supported; translations are not reliable,
>> and sometimes cause too many levels of recursion to be added.
>> Something simpler is necessary.
>>
>> = What this does =
>>
>> Essentially, this creates multi-level, explicit fallbacks.  The field
>> info container, which was a bloated, weird shared state object, is now
>> simply a dictionary subclass with a "fallback" option.  When you
>> create them, you can either create it in isolation (with no fallback)
>> or with a fallback.  When you query it, if it does not have a field,
>> it checks its fallback.  There are, additionally, two new functions
>> for IO: the translation function and the null function.  The first is
>> to translate, for instance, "density" to "Density" and the second is
>> to indicate that a field is expected to be found in an output from the
>> simulation code.
>>
>> There are now affiliated with each simulation code two field info
>> objects: the "known" fields, which may appear in files, and the
>> non-known (i.e., code-specific derived) fields.  These live as the
>> attributes _fieldinfo_fallback and _fieldinfo_known on the
>> StaticOutput sublcass corresponding to a simulation code.  When the
>> Hierarchy (not static output) is instantiated, the first step is to
>> create a new field_info object.  This has, as a fallback, the
>> _fieldinfo_fallback, which itself has as a fallback the
>> universally-known derived fields.  The hierarchy then queries the
>> output file for which fields are available.  This process then looks
>> for a corresponding field in fieldinfo_known, and if it finds it, it
>> adds it to the field_info object, *overriding* any possible derived
>> fields.  (In this manner, for instance, yt will not recalculate a
>> "CoolingTime" field if one exists in the output.)
>>
>> = What it aims to do in the future =
>>
>> This will be utilized in three main ways:
>>
>> 1) Making it more clear which fields belong to which code, and which
>> come from disk and which are derived
>> 2) Help move IO into fields, to optimize for geometries and data containers
>> 3) Make units more clear and specific
>> 4) This is all designed around better supporting the GDF.
>>
>> = Where from here? =
>>
>> It would be hugely beneficial if you could test this and report back.
>> I have created a pull request:
>>
>> https://bitbucket.org/yt_analysis/yt/pull-request/27/field-overhaul-to-utilize-explicit
>>
>> This is by no means a settled matter; I think we need to have testing
>> on this, buy-in from developers and users, and to make sure that old
>> code doesn't beak.  The test cases all pass for me for Enzo.
>>
>> Before this can be merged, I would hope we can get some testing from:
>>
>>  * Enzo
>>  * Nyx
>>  * FLASH
>>  * Orion
>>
>> and any other codes that can hear me.
>>
>> Thanks very much for your time; please let me know if you have any
>> questions, concerns, jokes, comments, improvements, CDs of your band,
>> suggestions, and so on.  For this major of a change I'd like to keep
>> discussion on list, so the record of this is a bit more prominent.
>>
>> Best,
>>
>> Matt
>> _______________________________________________
>> Yt-dev mailing list
>> Yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
> _______________________________________________
> Yt-dev mailing list
> Yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>



-- 
Sent from my computer.



More information about the yt-dev mailing list