[Yt-dev] Field definitions, derived fields, whats-in-a-file and the "deliberate_fields" branch
david collins
antpuncher at gmail.com
Wed Nov 9 14:02:25 PST 2011
Hi, Matt--
This is exciting. I have dabbled a little with units and derived
fields in my time. I should note that I did all these tests before I
got a chance to read your examples, so I didn't incorporate any of the
modifications to add_field/add_enzo_field that you mentioned.
I've checked this out and banged on it. I ran into one bug, one odd
warning, have one feature question, and several successes. I think I
fixed the bug. Everything I've tried seems to work so far, even my
old wonky scripts.
The bug: It fails with a syntax error (full trace is below.) on
"import yt.mods" From other bits of code, it looks like, in the
definition of AMRBooleanRegionBase, (line 3407 in data_containers.py)
_con_args = {"regions"}
should be
_con_args = ("regions")
Fixing that lets the import yt.mods work.
Then reading my dataset, I tried some tests. For the record this
dataset was from and Enzo who's closest parent is enzo1.0.1, with all
of my horrible additions: old unpacked hdf5 plus MHDCT garbage.
>>> pf = load(filename)
>>> print pf.h.grids[0]['Density'].min(), 'Denisty'
>>> print pf.h.grids[0]['MagneticField_C_1'].min(), "MagField"
0.018 'Density'
-13.2, "MagField"
which works fine. If I start a new python session,
>>> pf2 = load(filename, conversion_override={'MagneticField_C_1':100}
>>> print pf2.h.grids[0]['MagneticField_C_1'].min()
-1328
which is also successful. Also successful were similar tests using my
overloaded EnzoStaticOutput classes. I also tried other derived
fields, without changing any of my definitions, and the worked
correctly, as far as I can tell.
The one minor I've run into involves a hack-tastic I did to deal with
uncertainty in field names. I think this might be what your
"translate" thing is for? This field throws the exception when I load
the parameter file, but otherwise works-- Does your new stuff have a
better fix for this kind of code?
def _Bxu(field,data):
try:
return data['Bx']
except:
try:
return data['MagneticField_C_1']
except:
print "no magnetic field x"
add_field('Bxu',function=_Bxu,validators=[ValidateGridType()], take_log=False)
I call this a roaring success.
Thanks a ton!
d.
Trace for the first bug:
File "go", line 20, in <module>
from yt.mods import *
File "/Users/dccollins/local/src/yt_turk/yt/mods.py", line 44, in <module>
from yt.data_objects.api import \
File "/Users/dccollins/local/src/yt_turk/yt/data_objects/api.py",
line 31, in <module>
from grid_patch import \
File "/Users/dccollins/local/src/yt_turk/yt/data_objects/grid_patch.py",
line 35, in <module>
from yt.data_objects.data_containers import YTFieldData
File "/Users/dccollins/local/src/yt_turk/yt/data_objects/data_containers.py",
line 3407
_con_args = {"regions"}
^
SyntaxError: invalid syntax
On Wed, Nov 9, 2011 at 1:25 PM, j s oishi <jsoishi at gmail.com> wrote:
> Hi Matt,
>
> This sounds like a much needed overhaul. However, I'm not quite clear
> on exactly what this will entail, or how it will work once
> implemented. Could you or Casey provide an example of a new field or
> two, demonstrating how these dictionaries, fallbacks, and Null
> functions work? I think this is likely a very simple thing, but I'm
> having trouble visualizing it.
>
> thanks,
>
> j
>
> On Wed, Nov 9, 2011 at 9:05 AM, Matthew Turk <matthewturk at gmail.com> wrote:
>> Hi all,
>>
>> Over the last couple months, Casey and I have been working -- on and
>> off! -- on a new branch of the code called "deliberate_fields." This
>> branch will change, in a substantial but easy-to-update way, how
>> fields are handled in yt.
>>
>> I recognize this email is long. But if you use non-standard fields, a
>> bunch of derived fields, unit modifications, any of that, it may
>> affect you. So I *please* ask that you read it and, if you like,
>> contribute back to the discussion.
>>
>> This is one of the items I really want to have done for a hypothetical
>> 2.3 release.
>>
>> = Background =
>>
>> The way fields work currently was designed a bit haphazardly. They
>> use FieldInfoContainers, objects which share state and which contain
>> unions of the known derived fields and the known IO-based fields. One
>> of the problems with this is that the only thing that separates a
>> derived field from a known field is that function that generates the
>> field: the IO-based fields all use a lambda which returns None, and
>> the non-IO based fields return actual fields. This is pretty
>> sub-optimal, and it actually lands us in trouble when (for instance)
>> we have fields wandering around named things like "Thermal_Energy" and
>> "ThermalEnergy"; the mechanism by which one is selected and the other
>> not is problematic, and to get around infinite recursion, hacks have
>> had to be applied.
>>
>> As it stands, to find a field, the shared-state "field info" on a
>> parameter file is queried; this then will try to check universal
>> fields. But because of how the fields are stored, the field info
>> cascade can also operate in reverse. The big problem is that the
>> field selection mechanism doesn't seem to have a bus factor >= 1.0.
>> And, it has a number of hacks to make it work with conflicting field
>> definitiosn and the like.
>>
>> Unfortunately, layering these hacks on top of each other makes it much
>> harder for other codes to be supported; translations are not reliable,
>> and sometimes cause too many levels of recursion to be added.
>> Something simpler is necessary.
>>
>> = What this does =
>>
>> Essentially, this creates multi-level, explicit fallbacks. The field
>> info container, which was a bloated, weird shared state object, is now
>> simply a dictionary subclass with a "fallback" option. When you
>> create them, you can either create it in isolation (with no fallback)
>> or with a fallback. When you query it, if it does not have a field,
>> it checks its fallback. There are, additionally, two new functions
>> for IO: the translation function and the null function. The first is
>> to translate, for instance, "density" to "Density" and the second is
>> to indicate that a field is expected to be found in an output from the
>> simulation code.
>>
>> There are now affiliated with each simulation code two field info
>> objects: the "known" fields, which may appear in files, and the
>> non-known (i.e., code-specific derived) fields. These live as the
>> attributes _fieldinfo_fallback and _fieldinfo_known on the
>> StaticOutput sublcass corresponding to a simulation code. When the
>> Hierarchy (not static output) is instantiated, the first step is to
>> create a new field_info object. This has, as a fallback, the
>> _fieldinfo_fallback, which itself has as a fallback the
>> universally-known derived fields. The hierarchy then queries the
>> output file for which fields are available. This process then looks
>> for a corresponding field in fieldinfo_known, and if it finds it, it
>> adds it to the field_info object, *overriding* any possible derived
>> fields. (In this manner, for instance, yt will not recalculate a
>> "CoolingTime" field if one exists in the output.)
>>
>> = What it aims to do in the future =
>>
>> This will be utilized in three main ways:
>>
>> 1) Making it more clear which fields belong to which code, and which
>> come from disk and which are derived
>> 2) Help move IO into fields, to optimize for geometries and data containers
>> 3) Make units more clear and specific
>> 4) This is all designed around better supporting the GDF.
>>
>> = Where from here? =
>>
>> It would be hugely beneficial if you could test this and report back.
>> I have created a pull request:
>>
>> https://bitbucket.org/yt_analysis/yt/pull-request/27/field-overhaul-to-utilize-explicit
>>
>> This is by no means a settled matter; I think we need to have testing
>> on this, buy-in from developers and users, and to make sure that old
>> code doesn't beak. The test cases all pass for me for Enzo.
>>
>> Before this can be merged, I would hope we can get some testing from:
>>
>> * Enzo
>> * Nyx
>> * FLASH
>> * Orion
>>
>> and any other codes that can hear me.
>>
>> Thanks very much for your time; please let me know if you have any
>> questions, concerns, jokes, comments, improvements, CDs of your band,
>> suggestions, and so on. For this major of a change I'd like to keep
>> discussion on list, so the record of this is a bit more prominent.
>>
>> Best,
>>
>> Matt
>> _______________________________________________
>> Yt-dev mailing list
>> Yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
> _______________________________________________
> Yt-dev mailing list
> Yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
--
Sent from my computer.
More information about the yt-dev
mailing list