Hi David,<br><br><div class="gmail_quote">On Wed, Nov 9, 2011 at 5:34 PM, david collins <span dir="ltr"><<a href="mailto:antpuncher@gmail.com">antpuncher@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im">> To be perfectly honest, you were in my list of top one people I thought would<br>
> either love or hate this proposed change.<br>
<br>
</div>It wouldn't be so much "hate" as much as "gripe because my<br>
non-standard code that worked with the 1987 vintage of yt doesn't work<br>
with the changes you made in 1995." :D I'm actually really impressed<br>
that it all went so smoothly, that's a pretty major change with many<br>
opportunities for bear traps to step in.<br></blockquote><div><br></div><div>Well, in fairness, 1995 was a good year for yt.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im"><br>
> Ah! I think this is what TranslateFunc is designed for. I think the way I<br>
> envision it working is to choose a canonical name, maybe Bx, and then do:<br>
> add_enzo_field("Bx", function=NullFunc)<br>
> add_field("Bx", function=TranslateFunction("MagneticField_C_1"))<br>
<br>
</div>Is this now functioning? I don't quite get the logic behind the<br>
function calls here, why is Bx itself an "add_enzo_field" but the<br>
translate is an add_field?<br></blockquote><div><br></div><div>That *should* function. What it does is say, the canonical name is Bx, and that's the one you'll find in a file (even if you find another) and that we'll address in yt. But, just in case, we want it also to return MagneticField_C_1, if it can't find Bx itself. So the first goes to the "known" fields, and the second goes to the "derived" fields. Does that make any more sense?</div>
<div><br></div><div>-Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<br>
Thanks a ton,<br>
<div class="HOEnZb"><div class="h5">d.<br>
<br>
> Thank you. :) I have updated the pull request with the new changeset where<br>
> con_args is fixed.<br>
> Best,<br>
> Matt<br>
><br>
>><br>
>> Thanks a ton!<br>
>> d.<br>
>><br>
>> Trace for the first bug:<br>
>> File "go", line 20, in <module><br>
>> from yt.mods import *<br>
>> File "/Users/dccollins/local/src/yt_turk/yt/mods.py", line 44, in<br>
>> <module><br>
>> from yt.data_objects.api import \<br>
>> File "/Users/dccollins/local/src/yt_turk/yt/data_objects/api.py",<br>
>> line 31, in <module><br>
>> from grid_patch import \<br>
>> File "/Users/dccollins/local/src/yt_turk/yt/data_objects/grid_patch.py",<br>
>> line 35, in <module><br>
>> from yt.data_objects.data_containers import YTFieldData<br>
>> File<br>
>> "/Users/dccollins/local/src/yt_turk/yt/data_objects/data_containers.py",<br>
>> line 3407<br>
>> _con_args = {"regions"}<br>
>> ^<br>
>> SyntaxError: invalid syntax<br>
>><br>
>><br>
>><br>
>> On Wed, Nov 9, 2011 at 1:25 PM, j s oishi <<a href="mailto:jsoishi@gmail.com">jsoishi@gmail.com</a>> wrote:<br>
>> > Hi Matt,<br>
>> ><br>
>> > This sounds like a much needed overhaul. However, I'm not quite clear<br>
>> > on exactly what this will entail, or how it will work once<br>
>> > implemented. Could you or Casey provide an example of a new field or<br>
>> > two, demonstrating how these dictionaries, fallbacks, and Null<br>
>> > functions work? I think this is likely a very simple thing, but I'm<br>
>> > having trouble visualizing it.<br>
>> ><br>
>> > thanks,<br>
>> ><br>
>> > j<br>
>> ><br>
>> > On Wed, Nov 9, 2011 at 9:05 AM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
>> > wrote:<br>
>> >> Hi all,<br>
>> >><br>
>> >> Over the last couple months, Casey and I have been working -- on and<br>
>> >> off! -- on a new branch of the code called "deliberate_fields." This<br>
>> >> branch will change, in a substantial but easy-to-update way, how<br>
>> >> fields are handled in yt.<br>
>> >><br>
>> >> I recognize this email is long. But if you use non-standard fields, a<br>
>> >> bunch of derived fields, unit modifications, any of that, it may<br>
>> >> affect you. So I *please* ask that you read it and, if you like,<br>
>> >> contribute back to the discussion.<br>
>> >><br>
>> >> This is one of the items I really want to have done for a hypothetical<br>
>> >> 2.3 release.<br>
>> >><br>
>> >> = Background =<br>
>> >><br>
>> >> The way fields work currently was designed a bit haphazardly. They<br>
>> >> use FieldInfoContainers, objects which share state and which contain<br>
>> >> unions of the known derived fields and the known IO-based fields. One<br>
>> >> of the problems with this is that the only thing that separates a<br>
>> >> derived field from a known field is that function that generates the<br>
>> >> field: the IO-based fields all use a lambda which returns None, and<br>
>> >> the non-IO based fields return actual fields. This is pretty<br>
>> >> sub-optimal, and it actually lands us in trouble when (for instance)<br>
>> >> we have fields wandering around named things like "Thermal_Energy" and<br>
>> >> "ThermalEnergy"; the mechanism by which one is selected and the other<br>
>> >> not is problematic, and to get around infinite recursion, hacks have<br>
>> >> had to be applied.<br>
>> >><br>
>> >> As it stands, to find a field, the shared-state "field info" on a<br>
>> >> parameter file is queried; this then will try to check universal<br>
>> >> fields. But because of how the fields are stored, the field info<br>
>> >> cascade can also operate in reverse. The big problem is that the<br>
>> >> field selection mechanism doesn't seem to have a bus factor >= 1.0.<br>
>> >> And, it has a number of hacks to make it work with conflicting field<br>
>> >> definitiosn and the like.<br>
>> >><br>
>> >> Unfortunately, layering these hacks on top of each other makes it much<br>
>> >> harder for other codes to be supported; translations are not reliable,<br>
>> >> and sometimes cause too many levels of recursion to be added.<br>
>> >> Something simpler is necessary.<br>
>> >><br>
>> >> = What this does =<br>
>> >><br>
>> >> Essentially, this creates multi-level, explicit fallbacks. The field<br>
>> >> info container, which was a bloated, weird shared state object, is now<br>
>> >> simply a dictionary subclass with a "fallback" option. When you<br>
>> >> create them, you can either create it in isolation (with no fallback)<br>
>> >> or with a fallback. When you query it, if it does not have a field,<br>
>> >> it checks its fallback. There are, additionally, two new functions<br>
>> >> for IO: the translation function and the null function. The first is<br>
>> >> to translate, for instance, "density" to "Density" and the second is<br>
>> >> to indicate that a field is expected to be found in an output from the<br>
>> >> simulation code.<br>
>> >><br>
>> >> There are now affiliated with each simulation code two field info<br>
>> >> objects: the "known" fields, which may appear in files, and the<br>
>> >> non-known (i.e., code-specific derived) fields. These live as the<br>
>> >> attributes _fieldinfo_fallback and _fieldinfo_known on the<br>
>> >> StaticOutput sublcass corresponding to a simulation code. When the<br>
>> >> Hierarchy (not static output) is instantiated, the first step is to<br>
>> >> create a new field_info object. This has, as a fallback, the<br>
>> >> _fieldinfo_fallback, which itself has as a fallback the<br>
>> >> universally-known derived fields. The hierarchy then queries the<br>
>> >> output file for which fields are available. This process then looks<br>
>> >> for a corresponding field in fieldinfo_known, and if it finds it, it<br>
>> >> adds it to the field_info object, *overriding* any possible derived<br>
>> >> fields. (In this manner, for instance, yt will not recalculate a<br>
>> >> "CoolingTime" field if one exists in the output.)<br>
>> >><br>
>> >> = What it aims to do in the future =<br>
>> >><br>
>> >> This will be utilized in three main ways:<br>
>> >><br>
>> >> 1) Making it more clear which fields belong to which code, and which<br>
>> >> come from disk and which are derived<br>
>> >> 2) Help move IO into fields, to optimize for geometries and data<br>
>> >> containers<br>
>> >> 3) Make units more clear and specific<br>
>> >> 4) This is all designed around better supporting the GDF.<br>
>> >><br>
>> >> = Where from here? =<br>
>> >><br>
>> >> It would be hugely beneficial if you could test this and report back.<br>
>> >> I have created a pull request:<br>
>> >><br>
>> >><br>
>> >> <a href="https://bitbucket.org/yt_analysis/yt/pull-request/27/field-overhaul-to-utilize-explicit" target="_blank">https://bitbucket.org/yt_analysis/yt/pull-request/27/field-overhaul-to-utilize-explicit</a><br>
>> >><br>
>> >> This is by no means a settled matter; I think we need to have testing<br>
>> >> on this, buy-in from developers and users, and to make sure that old<br>
>> >> code doesn't beak. The test cases all pass for me for Enzo.<br>
>> >><br>
>> >> Before this can be merged, I would hope we can get some testing from:<br>
>> >><br>
>> >> * Enzo<br>
>> >> * Nyx<br>
>> >> * FLASH<br>
>> >> * Orion<br>
>> >><br>
>> >> and any other codes that can hear me.<br>
>> >><br>
>> >> Thanks very much for your time; please let me know if you have any<br>
>> >> questions, concerns, jokes, comments, improvements, CDs of your band,<br>
>> >> suggestions, and so on. For this major of a change I'd like to keep<br>
>> >> discussion on list, so the record of this is a bit more prominent.<br>
>> >><br>
>> >> Best,<br>
>> >><br>
>> >> Matt<br>
>> >> _______________________________________________<br>
>> >> Yt-dev mailing list<br>
>> >> <a href="mailto:Yt-dev@lists.spacepope.org">Yt-dev@lists.spacepope.org</a><br>
>> >> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> >><br>
>> > _______________________________________________<br>
>> > Yt-dev mailing list<br>
>> > <a href="mailto:Yt-dev@lists.spacepope.org">Yt-dev@lists.spacepope.org</a><br>
>> > <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>> ><br>
>><br>
>><br>
>><br>
>> --<br>
>> Sent from my computer.<br>
>> _______________________________________________<br>
>> Yt-dev mailing list<br>
>> <a href="mailto:Yt-dev@lists.spacepope.org">Yt-dev@lists.spacepope.org</a><br>
>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
><br>
><br>
> _______________________________________________<br>
> Yt-dev mailing list<br>
> <a href="mailto:Yt-dev@lists.spacepope.org">Yt-dev@lists.spacepope.org</a><br>
> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
><br>
><br>
<br>
<br>
<br>
--<br>
Sent from my computer.<br>
_______________________________________________<br>
Yt-dev mailing list<br>
<a href="mailto:Yt-dev@lists.spacepope.org">Yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
</div></div></blockquote></div><br>