[yt-dev] [Yt-dev] Field definitions, derived fields, whats-in-a-file and the "deliberate_fields" branch

Britton Smith brittonsmith at gmail.com
Fri Nov 11 11:34:04 PST 2011 

I've been pretty swamped this week, but I can try it out now.

Britton

On Fri, Nov 11, 2011 at 2:15 PM, Matthew Turk <matthewturk at gmail.com> wrote:

> Hi all,
>
> It's been a couple days.  Would somebody mind revisiting the pull
> request?  I have been running with it here and I haven't seen any
> problems.
>
> Thanks,
>
> Matt
>
> On Wed, Nov 9, 2011 at 5:37 PM, Matthew Turk <matthewturk at gmail.com>
> wrote:
> > Hi David,
> >
> > On Wed, Nov 9, 2011 at 5:34 PM, david collins <antpuncher at gmail.com>
> wrote:
> >>
> >> > To be perfectly honest, you were in my list of top one people I
> thought
> >> > would
> >> > either love or hate this proposed change.
> >>
> >> It wouldn't be so much "hate" as much as "gripe because my
> >> non-standard code that worked with the 1987 vintage of yt doesn't work
> >> with the changes you made in 1995." :D  I'm actually really impressed
> >> that it all went so smoothly, that's a pretty major change with many
> >> opportunities for bear traps to step in.
> >
> > Well, in fairness, 1995 was a good year for yt.
> >
> >>
> >> > Ah!  I think this is what TranslateFunc is designed for.  I think the
> >> > way I
> >> > envision it working is to choose a canonical name, maybe Bx, and then
> >> > do:
> >> > add_enzo_field("Bx", function=NullFunc)
> >> > add_field("Bx", function=TranslateFunction("MagneticField_C_1"))
> >>
> >> Is this now functioning?   I don't quite get the logic behind the
> >> function calls here, why is Bx itself an "add_enzo_field" but the
> >> translate is an add_field?
> >
> > That *should* function.  What it does is say, the canonical name is Bx,
> and
> > that's the one you'll find in a file (even if you find another) and that
> > we'll address in yt.  But, just in case, we want it also to return
> > MagneticField_C_1, if it can't find Bx itself.  So the first goes to the
> > "known" fields, and the second goes to the "derived" fields.  Does that
> make
> > any more sense?
> > -Matt
> >
> >>
> >> Thanks a ton,
> >> d.
> >>
> >> > Thank you.  :)  I have updated the pull request with the new changeset
> >> > where
> >> > con_args is fixed.
> >> > Best,
> >> > Matt
> >> >
> >> >>
> >> >> Thanks a ton!
> >> >> d.
> >> >>
> >> >> Trace for the first bug:
> >> >>  File "go", line 20, in <module>
> >> >>    from yt.mods import *
> >> >>  File "/Users/dccollins/local/src/yt_turk/yt/mods.py", line 44, in
> >> >> <module>
> >> >>    from yt.data_objects.api import \
> >> >>  File "/Users/dccollins/local/src/yt_turk/yt/data_objects/api.py",
> >> >> line 31, in <module>
> >> >>    from grid_patch import \
> >> >>  File
> >> >> "/Users/dccollins/local/src/yt_turk/yt/data_objects/grid_patch.py",
> >> >> line 35, in <module>
> >> >>    from yt.data_objects.data_containers import YTFieldData
> >> >>  File
> >> >>
> >> >>
> "/Users/dccollins/local/src/yt_turk/yt/data_objects/data_containers.py",
> >> >> line 3407
> >> >>    _con_args = {"regions"}
> >> >>                          ^
> >> >> SyntaxError: invalid syntax
> >> >>
> >> >>
> >> >>
> >> >> On Wed, Nov 9, 2011 at 1:25 PM, j s oishi <jsoishi at gmail.com> wrote:
> >> >> > Hi Matt,
> >> >> >
> >> >> > This sounds like a much needed overhaul. However, I'm not quite
> clear
> >> >> > on exactly what this will entail, or how it will work once
> >> >> > implemented. Could you or Casey provide an example of a new field
> or
> >> >> > two, demonstrating how these dictionaries, fallbacks, and Null
> >> >> > functions work? I think this is likely a very simple thing, but I'm
> >> >> > having trouble visualizing it.
> >> >> >
> >> >> > thanks,
> >> >> >
> >> >> > j
> >> >> >
> >> >> > On Wed, Nov 9, 2011 at 9:05 AM, Matthew Turk <
> matthewturk at gmail.com>
> >> >> > wrote:
> >> >> >> Hi all,
> >> >> >>
> >> >> >> Over the last couple months, Casey and I have been working -- on
> and
> >> >> >> off! -- on a new branch of the code called "deliberate_fields."
> >> >> >>  This
> >> >> >> branch will change, in a substantial but easy-to-update way, how
> >> >> >> fields are handled in yt.
> >> >> >>
> >> >> >> I recognize this email is long.  But if you use non-standard
> fields,
> >> >> >> a
> >> >> >> bunch of derived fields, unit modifications, any of that, it may
> >> >> >> affect you.  So I *please* ask that you read it and, if you like,
> >> >> >> contribute back to the discussion.
> >> >> >>
> >> >> >> This is one of the items I really want to have done for a
> >> >> >> hypothetical
> >> >> >> 2.3 release.
> >> >> >>
> >> >> >> = Background =
> >> >> >>
> >> >> >> The way fields work currently was designed a bit haphazardly.
>  They
> >> >> >> use FieldInfoContainers, objects which share state and which
> contain
> >> >> >> unions of the known derived fields and the known IO-based fields.
> >> >> >>  One
> >> >> >> of the problems with this is that the only thing that separates a
> >> >> >> derived field from a known field is that function that generates
> the
> >> >> >> field: the IO-based fields all use a lambda which returns None,
> and
> >> >> >> the non-IO based fields return actual fields.  This is pretty
> >> >> >> sub-optimal, and it actually lands us in trouble when (for
> instance)
> >> >> >> we have fields wandering around named things like "Thermal_Energy"
> >> >> >> and
> >> >> >> "ThermalEnergy"; the mechanism by which one is selected and the
> >> >> >> other
> >> >> >> not is problematic, and to get around infinite recursion, hacks
> have
> >> >> >> had to be applied.
> >> >> >>
> >> >> >> As it stands, to find a field, the shared-state "field info" on a
> >> >> >> parameter file is queried; this then will try to check universal
> >> >> >> fields.  But because of how the fields are stored, the field info
> >> >> >> cascade can also operate in reverse.  The big problem is that the
> >> >> >> field selection mechanism doesn't seem to have a bus factor >=
> 1.0.
> >> >> >> And, it has a number of hacks to make it work with conflicting
> field
> >> >> >> definitiosn and the like.
> >> >> >>
> >> >> >> Unfortunately, layering these hacks on top of each other makes it
> >> >> >> much
> >> >> >> harder for other codes to be supported; translations are not
> >> >> >> reliable,
> >> >> >> and sometimes cause too many levels of recursion to be added.
> >> >> >> Something simpler is necessary.
> >> >> >>
> >> >> >> = What this does =
> >> >> >>
> >> >> >> Essentially, this creates multi-level, explicit fallbacks.  The
> >> >> >> field
> >> >> >> info container, which was a bloated, weird shared state object, is
> >> >> >> now
> >> >> >> simply a dictionary subclass with a "fallback" option.  When you
> >> >> >> create them, you can either create it in isolation (with no
> >> >> >> fallback)
> >> >> >> or with a fallback.  When you query it, if it does not have a
> field,
> >> >> >> it checks its fallback.  There are, additionally, two new
> functions
> >> >> >> for IO: the translation function and the null function.  The first
> >> >> >> is
> >> >> >> to translate, for instance, "density" to "Density" and the second
> is
> >> >> >> to indicate that a field is expected to be found in an output from
> >> >> >> the
> >> >> >> simulation code.
> >> >> >>
> >> >> >> There are now affiliated with each simulation code two field info
> >> >> >> objects: the "known" fields, which may appear in files, and the
> >> >> >> non-known (i.e., code-specific derived) fields.  These live as the
> >> >> >> attributes _fieldinfo_fallback and _fieldinfo_known on the
> >> >> >> StaticOutput sublcass corresponding to a simulation code.  When
> the
> >> >> >> Hierarchy (not static output) is instantiated, the first step is
> to
> >> >> >> create a new field_info object.  This has, as a fallback, the
> >> >> >> _fieldinfo_fallback, which itself has as a fallback the
> >> >> >> universally-known derived fields.  The hierarchy then queries the
> >> >> >> output file for which fields are available.  This process then
> looks
> >> >> >> for a corresponding field in fieldinfo_known, and if it finds it,
> it
> >> >> >> adds it to the field_info object, *overriding* any possible
> derived
> >> >> >> fields.  (In this manner, for instance, yt will not recalculate a
> >> >> >> "CoolingTime" field if one exists in the output.)
> >> >> >>
> >> >> >> = What it aims to do in the future =
> >> >> >>
> >> >> >> This will be utilized in three main ways:
> >> >> >>
> >> >> >> 1) Making it more clear which fields belong to which code, and
> which
> >> >> >> come from disk and which are derived
> >> >> >> 2) Help move IO into fields, to optimize for geometries and data
> >> >> >> containers
> >> >> >> 3) Make units more clear and specific
> >> >> >> 4) This is all designed around better supporting the GDF.
> >> >> >>
> >> >> >> = Where from here? =
> >> >> >>
> >> >> >> It would be hugely beneficial if you could test this and report
> >> >> >> back.
> >> >> >> I have created a pull request:
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> https://bitbucket.org/yt_analysis/yt/pull-request/27/field-overhaul-to-utilize-explicit
> >> >> >>
> >> >> >> This is by no means a settled matter; I think we need to have
> >> >> >> testing
> >> >> >> on this, buy-in from developers and users, and to make sure that
> old
> >> >> >> code doesn't beak.  The test cases all pass for me for Enzo.
> >> >> >>
> >> >> >> Before this can be merged, I would hope we can get some testing
> >> >> >> from:
> >> >> >>
> >> >> >>  * Enzo
> >> >> >>  * Nyx
> >> >> >>  * FLASH
> >> >> >>  * Orion
> >> >> >>
> >> >> >> and any other codes that can hear me.
> >> >> >>
> >> >> >> Thanks very much for your time; please let me know if you have any
> >> >> >> questions, concerns, jokes, comments, improvements, CDs of your
> >> >> >> band,
> >> >> >> suggestions, and so on.  For this major of a change I'd like to
> keep
> >> >> >> discussion on list, so the record of this is a bit more prominent.
> >> >> >>
> >> >> >> Best,
> >> >> >>
> >> >> >> Matt
> >> >> >> _______________________________________________
> >> >> >> Yt-dev mailing list
> >> >> >> Yt-dev at lists.spacepope.org
> >> >> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >> >> >>
> >> >> > _______________________________________________
> >> >> > Yt-dev mailing list
> >> >> > Yt-dev at lists.spacepope.org
> >> >> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Sent from my computer.
> >> >> _______________________________________________
> >> >> Yt-dev mailing list
> >> >> Yt-dev at lists.spacepope.org
> >> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >> >
> >> >
> >> > _______________________________________________
> >> > Yt-dev mailing list
> >> > Yt-dev at lists.spacepope.org
> >> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Sent from my computer.
> >> _______________________________________________
> >> Yt-dev mailing list
> >> Yt-dev at lists.spacepope.org
> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >
> >
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20111111/0a1d904a/attachment.html>

More information about the yt-dev mailing list