I've been pretty swamped this week, but I can try it out now.<br><br>Britton<br><br><div class="gmail_quote">On Fri, Nov 11, 2011 at 2:15 PM, Matthew Turk <span dir="ltr"><<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Hi all,<br>
<br>
It's been a couple days. Would somebody mind revisiting the pull<br>
request? I have been running with it here and I haven't seen any<br>
problems.<br>
<br>
Thanks,<br>
<br>
Matt<br>
<div><div class="h5"><br>
On Wed, Nov 9, 2011 at 5:37 PM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>> wrote:<br>
> Hi David,<br>
><br>
> On Wed, Nov 9, 2011 at 5:34 PM, david collins <<a href="mailto:antpuncher@gmail.com">antpuncher@gmail.com</a>> wrote:<br>
>><br>
>> > To be perfectly honest, you were in my list of top one people I thought<br>
>> > would<br>
>> > either love or hate this proposed change.<br>
>><br>
>> It wouldn't be so much "hate" as much as "gripe because my<br>
>> non-standard code that worked with the 1987 vintage of yt doesn't work<br>
>> with the changes you made in 1995." :D I'm actually really impressed<br>
>> that it all went so smoothly, that's a pretty major change with many<br>
>> opportunities for bear traps to step in.<br>
><br>
> Well, in fairness, 1995 was a good year for yt.<br>
><br>
>><br>
>> > Ah! I think this is what TranslateFunc is designed for. I think the<br>
>> > way I<br>
>> > envision it working is to choose a canonical name, maybe Bx, and then<br>
>> > do:<br>
>> > add_enzo_field("Bx", function=NullFunc)<br>
>> > add_field("Bx", function=TranslateFunction("MagneticField_C_1"))<br>
>><br>
>> Is this now functioning? I don't quite get the logic behind the<br>
>> function calls here, why is Bx itself an "add_enzo_field" but the<br>
>> translate is an add_field?<br>
><br>
> That *should* function. What it does is say, the canonical name is Bx, and<br>
> that's the one you'll find in a file (even if you find another) and that<br>
> we'll address in yt. But, just in case, we want it also to return<br>
> MagneticField_C_1, if it can't find Bx itself. So the first goes to the<br>
> "known" fields, and the second goes to the "derived" fields. Does that make<br>
> any more sense?<br>
> -Matt<br>
><br>
>><br>
>> Thanks a ton,<br>
>> d.<br>
>><br>
>> > Thank you. :) I have updated the pull request with the new changeset<br>
>> > where<br>
>> > con_args is fixed.<br>
>> > Best,<br>
>> > Matt<br>
>> ><br>
>> >><br>
>> >> Thanks a ton!<br>
>> >> d.<br>
>> >><br>
>> >> Trace for the first bug:<br>
>> >> File "go", line 20, in <module><br>
>> >> from yt.mods import *<br>
>> >> File "/Users/dccollins/local/src/yt_turk/yt/mods.py", line 44, in<br>
>> >> <module><br>
>> >> from yt.data_objects.api import \<br>
>> >> File "/Users/dccollins/local/src/yt_turk/yt/data_objects/api.py",<br>
>> >> line 31, in <module><br>
>> >> from grid_patch import \<br>
>> >> File<br>
>> >> "/Users/dccollins/local/src/yt_turk/yt/data_objects/grid_patch.py",<br>
>> >> line 35, in <module><br>
>> >> from yt.data_objects.data_containers import YTFieldData<br>
>> >> File<br>
>> >><br>
>> >> "/Users/dccollins/local/src/yt_turk/yt/data_objects/data_containers.py",<br>
>> >> line 3407<br>
>> >> _con_args = {"regions"}<br>
>> >> ^<br>
>> >> SyntaxError: invalid syntax<br>
>> >><br>
>> >><br>
>> >><br>
>> >> On Wed, Nov 9, 2011 at 1:25 PM, j s oishi <<a href="mailto:jsoishi@gmail.com">jsoishi@gmail.com</a>> wrote:<br>
>> >> > Hi Matt,<br>
>> >> ><br>
>> >> > This sounds like a much needed overhaul. However, I'm not quite clear<br>
>> >> > on exactly what this will entail, or how it will work once<br>
>> >> > implemented. Could you or Casey provide an example of a new field or<br>
>> >> > two, demonstrating how these dictionaries, fallbacks, and Null<br>
>> >> > functions work? I think this is likely a very simple thing, but I'm<br>
>> >> > having trouble visualizing it.<br>
>> >> ><br>
>> >> > thanks,<br>
>> >> ><br>
>> >> > j<br>
>> >> ><br>
>> >> > On Wed, Nov 9, 2011 at 9:05 AM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
>> >> > wrote:<br>
>> >> >> Hi all,<br>
>> >> >><br>
>> >> >> Over the last couple months, Casey and I have been working -- on and<br>
>> >> >> off! -- on a new branch of the code called "deliberate_fields."<br>
>> >> >> This<br>
>> >> >> branch will change, in a substantial but easy-to-update way, how<br>
>> >> >> fields are handled in yt.<br>
>> >> >><br>
>> >> >> I recognize this email is long. But if you use non-standard fields,<br>
>> >> >> a<br>
>> >> >> bunch of derived fields, unit modifications, any of that, it may<br>
>> >> >> affect you. So I *please* ask that you read it and, if you like,<br>
>> >> >> contribute back to the discussion.<br>
>> >> >><br>
>> >> >> This is one of the items I really want to have done for a<br>
>> >> >> hypothetical<br>
>> >> >> 2.3 release.<br>
>> >> >><br>
>> >> >> = Background =<br>
>> >> >><br>
>> >> >> The way fields work currently was designed a bit haphazardly. They<br>
>> >> >> use FieldInfoContainers, objects which share state and which contain<br>
>> >> >> unions of the known derived fields and the known IO-based fields.<br>
>> >> >> One<br>
>> >> >> of the problems with this is that the only thing that separates a<br>
>> >> >> derived field from a known field is that function that generates the<br>
>> >> >> field: the IO-based fields all use a lambda which returns None, and<br>
>> >> >> the non-IO based fields return actual fields. This is pretty<br>
>> >> >> sub-optimal, and it actually lands us in trouble when (for instance)<br>
>> >> >> we have fields wandering around named things like "Thermal_Energy"<br>
>> >> >> and<br>
>> >> >> "ThermalEnergy"; the mechanism by which one is selected and the<br>
>> >> >> other<br>
>> >> >> not is problematic, and to get around infinite recursion, hacks have<br>
>> >> >> had to be applied.<br>
>> >> >><br>
>> >> >> As it stands, to find a field, the shared-state "field info" on a<br>
>> >> >> parameter file is queried; this then will try to check universal<br>
>> >> >> fields. But because of how the fields are stored, the field info<br>
>> >> >> cascade can also operate in reverse. The big problem is that the<br>
>> >> >> field selection mechanism doesn't seem to have a bus factor >= 1.0.<br>
>> >> >> And, it has a number of hacks to make it work with conflicting field<br>
>> >> >> definitiosn and the like.<br>
>> >> >><br>
>> >> >> Unfortunately, layering these hacks on top of each other makes it<br>
>> >> >> much<br>
>> >> >> harder for other codes to be supported; translations are not<br>
>> >> >> reliable,<br>
>> >> >> and sometimes cause too many levels of recursion to be added.<br>
>> >> >> Something simpler is necessary.<br>
>> >> >><br>
>> >> >> = What this does =<br>
>> >> >><br>
>> >> >> Essentially, this creates multi-level, explicit fallbacks. The<br>
>> >> >> field<br>
>> >> >> info container, which was a bloated, weird shared state object, is<br>
>> >> >> now<br>
>> >> >> simply a dictionary subclass with a "fallback" option. When you<br>
>> >> >> create them, you can either create it in isolation (with no<br>
>> >> >> fallback)<br>
>> >> >> or with a fallback. When you query it, if it does not have a field,<br>
>> >> >> it checks its fallback. There are, additionally, two new functions<br>
>> >> >> for IO: the translation function and the null function. The first<br>
>> >> >> is<br>
>> >> >> to translate, for instance, "density" to "Density" and the second is<br>
>> >> >> to indicate that a field is expected to be found in an output from<br>
>> >> >> the<br>
>> >> >> simulation code.<br>
>> >> >><br>
>> >> >> There are now affiliated with each simulation code two field info<br>
>> >> >> objects: the "known" fields, which may appear in files, and the<br>
>> >> >> non-known (i.e., code-specific derived) fields. These live as the<br>
>> >> >> attributes _fieldinfo_fallback and _fieldinfo_known on the<br>
>> >> >> StaticOutput sublcass corresponding to a simulation code. When the<br>
>> >> >> Hierarchy (not static output) is instantiated, the first step is to<br>
>> >> >> create a new field_info object. This has, as a fallback, the<br>
>> >> >> _fieldinfo_fallback, which itself has as a fallback the<br>
>> >> >> universally-known derived fields. The hierarchy then queries the<br>
>> >> >> output file for which fields are available. This process then looks<br>
>> >> >> for a corresponding field in fieldinfo_known, and if it finds it, it<br>
>> >> >> adds it to the field_info object, *overriding* any possible derived<br>
>> >> >> fields. (In this manner, for instance, yt will not recalculate a<br>
>> >> >> "CoolingTime" field if one exists in the output.)<br>
>> >> >><br>
>> >> >> = What it aims to do in the future =<br>
>> >> >><br>
>> >> >> This will be utilized in three main ways:<br>
>> >> >><br>
>> >> >> 1) Making it more clear which fields belong to which code, and which<br>
>> >> >> come from disk and which are derived<br>
>> >> >> 2) Help move IO into fields, to optimize for geometries and data<br>
>> >> >> containers<br>
>> >> >> 3) Make units more clear and specific<br>
>> >> >> 4) This is all designed around better supporting the GDF.<br>
>> >> >><br>
>> >> >> = Where from here? =<br>
>> >> >><br>
>> >> >> It would be hugely beneficial if you could test this and report<br>
>> >> >> back.<br>
>> >> >> I have created a pull request:<br>
>> >> >><br>
>> >> >><br>
>> >> >><br>
>> >> >> <a href="https://bitbucket.org/yt_analysis/yt/pull-request/27/field-overhaul-to-utilize-explicit" target="_blank">https://bitbucket.org/yt_analysis/yt/pull-request/27/field-overhaul-to-utilize-explicit</a><br>
>> >> >><br>
>> >> >> This is by no means a settled matter; I think we need to have<br>
>> >> >> testing<br>
>> >> >> on this, buy-in from developers and users, and to make sure that old<br>
>> >> >> code doesn't beak. The test cases all pass for me for Enzo.<br>
>> >> >><br>
>> >> >> Before this can be merged, I would hope we can get some testing<br>
>> >> >> from:<br>
>> >> >><br>
>> >> >> * Enzo<br>
>> >> >> * Nyx<br>
>> >> >> * FLASH<br>
>> >> >> * Orion<br>
>> >> >><br>
>> >> >> and any other codes that can hear me.<br>
>> >> >><br>
>> >> >> Thanks very much for your time; please let me know if you have any<br>
>> >> >> questions, concerns, jokes, comments, improvements, CDs of your<br>
>> >> >> band,<br>
>> >> >> suggestions, and so on. For this major of a change I'd like to keep<br>
>> >> >> discussion on list, so the record of this is a bit more prominent.<br>
>> >> >><br>
>> >> >> Best,<br>
>> >> >><br>
>> >> >> Matt<br>
>> >> >> _______________________________________________<br>
>> >> >> Yt-dev mailing list<br>
>> >> >> <a href="mailto:Yt-dev@lists.spacepope.org">Yt-dev@lists.spacepope.org</a><br>
>> >> >> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> >> >><br>
>> >> > _______________________________________________<br>
>> >> > Yt-dev mailing list<br>
>> >> > <a href="mailto:Yt-dev@lists.spacepope.org">Yt-dev@lists.spacepope.org</a><br>
>> >> > <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> >> ><br>
>> >><br>
>> >><br>
>> >><br>
>> >> --<br>
>> >> Sent from my computer.<br>
>> >> _______________________________________________<br>
>> >> Yt-dev mailing list<br>
>> >> <a href="mailto:Yt-dev@lists.spacepope.org">Yt-dev@lists.spacepope.org</a><br>
>> >> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> ><br>
>> ><br>
>> > _______________________________________________<br>
>> > Yt-dev mailing list<br>
>> > <a href="mailto:Yt-dev@lists.spacepope.org">Yt-dev@lists.spacepope.org</a><br>
>> > <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> ><br>
>> ><br>
>><br>
>><br>
>><br>
>> --<br>
>> Sent from my computer.<br>
>> _______________________________________________<br>
>> Yt-dev mailing list<br>
>> <a href="mailto:Yt-dev@lists.spacepope.org">Yt-dev@lists.spacepope.org</a><br>
>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
><br>
><br>
</div></div>_______________________________________________<br>
yt-dev mailing list<br>
<a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
</blockquote></div><br>