[Yt-dev] Field definitions, derived fields, whats-in-a-file and the "deliberate_fields" branch
Matthew Turk
matthewturk at gmail.com
Wed Nov 9 14:22:14 PST 2011
Hi David,
On Wed, Nov 9, 2011 at 5:02 PM, david collins <antpuncher at gmail.com> wrote:
> Hi, Matt--
>
> This is exciting. I have dabbled a little with units and derived
> fields in my time. I should note that I did all these tests before I
> got a chance to read your examples, so I didn't incorporate any of the
> modifications to add_field/add_enzo_field that you mentioned.
>
> I've checked this out and banged on it. I ran into one bug, one odd
> warning, have one feature question, and several successes. I think I
> fixed the bug. Everything I've tried seems to work so far, even my
> old wonky scripts.
>
Wow, thank you for such a thorough testing! To be perfectly honest, you
were in my list of top one people I thought would either love or hate this
proposed change. ;-) I really appreciate that it has improved things for
you.
>
> The bug: It fails with a syntax error (full trace is below.) on
> "import yt.mods" From other bits of code, it looks like, in the
> definition of AMRBooleanRegionBase, (line 3407 in data_containers.py)
>
> _con_args = {"regions"}
>
> should be
>
> _con_args = ("regions")
>
> Fixing that lets the import yt.mods work.
>
Excellent, I've updated.
>
> Then reading my dataset, I tried some tests. For the record this
> dataset was from and Enzo who's closest parent is enzo1.0.1, with all
> of my horrible additions: old unpacked hdf5 plus MHDCT garbage.
>
> >>> pf = load(filename)
> >>> print pf.h.grids[0]['Density'].min(), 'Denisty'
> >>> print pf.h.grids[0]['MagneticField_C_1'].min(), "MagField"
> 0.018 'Density'
> -13.2, "MagField"
>
> which works fine. If I start a new python session,
> >>> pf2 = load(filename, conversion_override={'MagneticField_C_1':100}
> >>> print pf2.h.grids[0]['MagneticField_C_1'].min()
> -1328
>
> which is also successful. Also successful were similar tests using my
> overloaded EnzoStaticOutput classes. I also tried other derived
> fields, without changing any of my definitions, and the worked
> correctly, as far as I can tell.
>
> The one minor I've run into involves a hack-tastic I did to deal with
> uncertainty in field names. I think this might be what your
> "translate" thing is for? This field throws the exception when I load
> the parameter file, but otherwise works-- Does your new stuff have a
> better fix for this kind of code?
>
> def _Bxu(field,data):
> try:
> return data['Bx']
> except:
> try:
> return data['MagneticField_C_1']
> except:
> print "no magnetic field x"
>
> add_field('Bxu',function=_Bxu,validators=[ValidateGridType()],
> take_log=False)
>
Ah! I think this is what TranslateFunc is designed for. I think the way I
envision it working is to choose a canonical name, maybe Bx, and then do:
add_enzo_field("Bx", function=NullFunc)
add_field("Bx", function=TranslateFunction("MagneticField_C_1"))
I think ti mgiht be a failing in the system that right now this mapping, of
one-or-the-other doesn't work ... I will have to think about that. But it
should be straightforward to fix this up. :)
>
> I call this a roaring success.
>
Thank you. :) I have updated the pull request with the new changeset
where con_args is fixed.
Best,
Matt
>
> Thanks a ton!
> d.
>
> Trace for the first bug:
> File "go", line 20, in <module>
> from yt.mods import *
> File "/Users/dccollins/local/src/yt_turk/yt/mods.py", line 44, in <module>
> from yt.data_objects.api import \
> File "/Users/dccollins/local/src/yt_turk/yt/data_objects/api.py",
> line 31, in <module>
> from grid_patch import \
> File "/Users/dccollins/local/src/yt_turk/yt/data_objects/grid_patch.py",
> line 35, in <module>
> from yt.data_objects.data_containers import YTFieldData
> File
> "/Users/dccollins/local/src/yt_turk/yt/data_objects/data_containers.py",
> line 3407
> _con_args = {"regions"}
> ^
> SyntaxError: invalid syntax
>
>
>
> On Wed, Nov 9, 2011 at 1:25 PM, j s oishi <jsoishi at gmail.com> wrote:
> > Hi Matt,
> >
> > This sounds like a much needed overhaul. However, I'm not quite clear
> > on exactly what this will entail, or how it will work once
> > implemented. Could you or Casey provide an example of a new field or
> > two, demonstrating how these dictionaries, fallbacks, and Null
> > functions work? I think this is likely a very simple thing, but I'm
> > having trouble visualizing it.
> >
> > thanks,
> >
> > j
> >
> > On Wed, Nov 9, 2011 at 9:05 AM, Matthew Turk <matthewturk at gmail.com>
> wrote:
> >> Hi all,
> >>
> >> Over the last couple months, Casey and I have been working -- on and
> >> off! -- on a new branch of the code called "deliberate_fields." This
> >> branch will change, in a substantial but easy-to-update way, how
> >> fields are handled in yt.
> >>
> >> I recognize this email is long. But if you use non-standard fields, a
> >> bunch of derived fields, unit modifications, any of that, it may
> >> affect you. So I *please* ask that you read it and, if you like,
> >> contribute back to the discussion.
> >>
> >> This is one of the items I really want to have done for a hypothetical
> >> 2.3 release.
> >>
> >> = Background =
> >>
> >> The way fields work currently was designed a bit haphazardly. They
> >> use FieldInfoContainers, objects which share state and which contain
> >> unions of the known derived fields and the known IO-based fields. One
> >> of the problems with this is that the only thing that separates a
> >> derived field from a known field is that function that generates the
> >> field: the IO-based fields all use a lambda which returns None, and
> >> the non-IO based fields return actual fields. This is pretty
> >> sub-optimal, and it actually lands us in trouble when (for instance)
> >> we have fields wandering around named things like "Thermal_Energy" and
> >> "ThermalEnergy"; the mechanism by which one is selected and the other
> >> not is problematic, and to get around infinite recursion, hacks have
> >> had to be applied.
> >>
> >> As it stands, to find a field, the shared-state "field info" on a
> >> parameter file is queried; this then will try to check universal
> >> fields. But because of how the fields are stored, the field info
> >> cascade can also operate in reverse. The big problem is that the
> >> field selection mechanism doesn't seem to have a bus factor >= 1.0.
> >> And, it has a number of hacks to make it work with conflicting field
> >> definitiosn and the like.
> >>
> >> Unfortunately, layering these hacks on top of each other makes it much
> >> harder for other codes to be supported; translations are not reliable,
> >> and sometimes cause too many levels of recursion to be added.
> >> Something simpler is necessary.
> >>
> >> = What this does =
> >>
> >> Essentially, this creates multi-level, explicit fallbacks. The field
> >> info container, which was a bloated, weird shared state object, is now
> >> simply a dictionary subclass with a "fallback" option. When you
> >> create them, you can either create it in isolation (with no fallback)
> >> or with a fallback. When you query it, if it does not have a field,
> >> it checks its fallback. There are, additionally, two new functions
> >> for IO: the translation function and the null function. The first is
> >> to translate, for instance, "density" to "Density" and the second is
> >> to indicate that a field is expected to be found in an output from the
> >> simulation code.
> >>
> >> There are now affiliated with each simulation code two field info
> >> objects: the "known" fields, which may appear in files, and the
> >> non-known (i.e., code-specific derived) fields. These live as the
> >> attributes _fieldinfo_fallback and _fieldinfo_known on the
> >> StaticOutput sublcass corresponding to a simulation code. When the
> >> Hierarchy (not static output) is instantiated, the first step is to
> >> create a new field_info object. This has, as a fallback, the
> >> _fieldinfo_fallback, which itself has as a fallback the
> >> universally-known derived fields. The hierarchy then queries the
> >> output file for which fields are available. This process then looks
> >> for a corresponding field in fieldinfo_known, and if it finds it, it
> >> adds it to the field_info object, *overriding* any possible derived
> >> fields. (In this manner, for instance, yt will not recalculate a
> >> "CoolingTime" field if one exists in the output.)
> >>
> >> = What it aims to do in the future =
> >>
> >> This will be utilized in three main ways:
> >>
> >> 1) Making it more clear which fields belong to which code, and which
> >> come from disk and which are derived
> >> 2) Help move IO into fields, to optimize for geometries and data
> containers
> >> 3) Make units more clear and specific
> >> 4) This is all designed around better supporting the GDF.
> >>
> >> = Where from here? =
> >>
> >> It would be hugely beneficial if you could test this and report back.
> >> I have created a pull request:
> >>
> >>
> https://bitbucket.org/yt_analysis/yt/pull-request/27/field-overhaul-to-utilize-explicit
> >>
> >> This is by no means a settled matter; I think we need to have testing
> >> on this, buy-in from developers and users, and to make sure that old
> >> code doesn't beak. The test cases all pass for me for Enzo.
> >>
> >> Before this can be merged, I would hope we can get some testing from:
> >>
> >> * Enzo
> >> * Nyx
> >> * FLASH
> >> * Orion
> >>
> >> and any other codes that can hear me.
> >>
> >> Thanks very much for your time; please let me know if you have any
> >> questions, concerns, jokes, comments, improvements, CDs of your band,
> >> suggestions, and so on. For this major of a change I'd like to keep
> >> discussion on list, so the record of this is a bit more prominent.
> >>
> >> Best,
> >>
> >> Matt
> >> _______________________________________________
> >> Yt-dev mailing list
> >> Yt-dev at lists.spacepope.org
> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >>
> > _______________________________________________
> > Yt-dev mailing list
> > Yt-dev at lists.spacepope.org
> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >
>
>
>
> --
> Sent from my computer.
> _______________________________________________
> Yt-dev mailing list
> Yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20111109/34545bde/attachment.htm>
More information about the yt-dev
mailing list