Hi David,<div><br><div class="gmail_quote">On Wed, Nov 9, 2011 at 5:02 PM, david collins <span dir="ltr"><<a href="mailto:antpuncher@gmail.com">antpuncher@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Hi, Matt--<br>
<br>
This is exciting. I have dabbled a little with units and derived<br>
fields in my time. I should note that I did all these tests before I<br>
got a chance to read your examples, so I didn't incorporate any of the<br>
modifications to add_field/add_enzo_field that you mentioned.<br>
<br>
I've checked this out and banged on it. I ran into one bug, one odd<br>
warning, have one feature question, and several successes. I think I<br>
fixed the bug. Everything I've tried seems to work so far, even my<br>
old wonky scripts.<br></blockquote><div><br></div><div>Wow, thank you for such a thorough testing! To be perfectly honest, you were in my list of top one people I thought would either love or hate this proposed change. ;-) I really appreciate that it has improved things for you.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<br>
The bug: It fails with a syntax error (full trace is below.) on<br>
"import yt.mods" From other bits of code, it looks like, in the<br>
definition of AMRBooleanRegionBase, (line 3407 in data_containers.py)<br>
<br>
_con_args = {"regions"}<br>
<br>
should be<br>
<br>
_con_args = ("regions")<br>
<br>
Fixing that lets the import yt.mods work.<br></blockquote><div><br></div><div>Excellent, I've updated.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<br>
Then reading my dataset, I tried some tests. For the record this<br>
dataset was from and Enzo who's closest parent is enzo1.0.1, with all<br>
of my horrible additions: old unpacked hdf5 plus MHDCT garbage.<br>
<br>
>>> pf = load(filename)<br>
>>> print pf.h.grids[0]['Density'].min(), 'Denisty'<br>
>>> print pf.h.grids[0]['MagneticField_C_1'].min(), "MagField"<br>
0.018 'Density'<br>
-13.2, "MagField"<br>
<br>
which works fine. If I start a new python session,<br>
>>> pf2 = load(filename, conversion_override={'MagneticField_C_1':100}<br>
>>> print pf2.h.grids[0]['MagneticField_C_1'].min()<br>
-1328<br>
<br>
which is also successful. Also successful were similar tests using my<br>
overloaded EnzoStaticOutput classes. I also tried other derived<br>
fields, without changing any of my definitions, and the worked<br>
correctly, as far as I can tell.<br>
<br>
The one minor I've run into involves a hack-tastic I did to deal with<br>
uncertainty in field names. I think this might be what your<br>
"translate" thing is for? This field throws the exception when I load<br>
the parameter file, but otherwise works-- Does your new stuff have a<br>
better fix for this kind of code?<br>
<br>
def _Bxu(field,data):<br>
try:<br>
return data['Bx']<br>
except:<br>
try:<br>
return data['MagneticField_C_1']<br>
except:<br>
print "no magnetic field x"<br>
<br>
add_field('Bxu',function=_Bxu,validators=[ValidateGridType()], take_log=False)<br></blockquote><div><br></div><div>Ah! I think this is what TranslateFunc is designed for. I think the way I envision it working is to choose a canonical name, maybe Bx, and then do:</div>
<div><br></div><div>add_enzo_field("Bx", function=NullFunc)</div><div>add_field("Bx", function=TranslateFunction("MagneticField_C_1"))</div><div><br></div><div>I think ti mgiht be a failing in the system that right now this mapping, of one-or-the-other doesn't work ... I will have to think about that. But it should be straightforward to fix this up. :)</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<br>
I call this a roaring success.<br></blockquote><div><br></div><div>Thank you. :) I have updated the pull request with the new changeset where con_args is fixed.</div><div><br></div><div>Best,</div><div><br></div><div>Matt</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<br>
Thanks a ton!<br>
d.<br>
<br>
Trace for the first bug:<br>
File "go", line 20, in <module><br>
from yt.mods import *<br>
File "/Users/dccollins/local/src/yt_turk/yt/mods.py", line 44, in <module><br>
from yt.data_objects.api import \<br>
File "/Users/dccollins/local/src/yt_turk/yt/data_objects/api.py",<br>
line 31, in <module><br>
from grid_patch import \<br>
File "/Users/dccollins/local/src/yt_turk/yt/data_objects/grid_patch.py",<br>
line 35, in <module><br>
from yt.data_objects.data_containers import YTFieldData<br>
File "/Users/dccollins/local/src/yt_turk/yt/data_objects/data_containers.py",<br>
line 3407<br>
_con_args = {"regions"}<br>
^<br>
SyntaxError: invalid syntax<br>
<div class="im HOEnZb"><br>
<br>
<br>
On Wed, Nov 9, 2011 at 1:25 PM, j s oishi <<a href="mailto:jsoishi@gmail.com">jsoishi@gmail.com</a>> wrote:<br>
</div><div class="HOEnZb"><div class="h5">> Hi Matt,<br>
><br>
> This sounds like a much needed overhaul. However, I'm not quite clear<br>
> on exactly what this will entail, or how it will work once<br>
> implemented. Could you or Casey provide an example of a new field or<br>
> two, demonstrating how these dictionaries, fallbacks, and Null<br>
> functions work? I think this is likely a very simple thing, but I'm<br>
> having trouble visualizing it.<br>
><br>
> thanks,<br>
><br>
> j<br>
><br>
> On Wed, Nov 9, 2011 at 9:05 AM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>> wrote:<br>
>> Hi all,<br>
>><br>
>> Over the last couple months, Casey and I have been working -- on and<br>
>> off! -- on a new branch of the code called "deliberate_fields." This<br>
>> branch will change, in a substantial but easy-to-update way, how<br>
>> fields are handled in yt.<br>
>><br>
>> I recognize this email is long. But if you use non-standard fields, a<br>
>> bunch of derived fields, unit modifications, any of that, it may<br>
>> affect you. So I *please* ask that you read it and, if you like,<br>
>> contribute back to the discussion.<br>
>><br>
>> This is one of the items I really want to have done for a hypothetical<br>
>> 2.3 release.<br>
>><br>
>> = Background =<br>
>><br>
>> The way fields work currently was designed a bit haphazardly. They<br>
>> use FieldInfoContainers, objects which share state and which contain<br>
>> unions of the known derived fields and the known IO-based fields. One<br>
>> of the problems with this is that the only thing that separates a<br>
>> derived field from a known field is that function that generates the<br>
>> field: the IO-based fields all use a lambda which returns None, and<br>
>> the non-IO based fields return actual fields. This is pretty<br>
>> sub-optimal, and it actually lands us in trouble when (for instance)<br>
>> we have fields wandering around named things like "Thermal_Energy" and<br>
>> "ThermalEnergy"; the mechanism by which one is selected and the other<br>
>> not is problematic, and to get around infinite recursion, hacks have<br>
>> had to be applied.<br>
>><br>
>> As it stands, to find a field, the shared-state "field info" on a<br>
>> parameter file is queried; this then will try to check universal<br>
>> fields. But because of how the fields are stored, the field info<br>
>> cascade can also operate in reverse. The big problem is that the<br>
>> field selection mechanism doesn't seem to have a bus factor >= 1.0.<br>
>> And, it has a number of hacks to make it work with conflicting field<br>
>> definitiosn and the like.<br>
>><br>
>> Unfortunately, layering these hacks on top of each other makes it much<br>
>> harder for other codes to be supported; translations are not reliable,<br>
>> and sometimes cause too many levels of recursion to be added.<br>
>> Something simpler is necessary.<br>
>><br>
>> = What this does =<br>
>><br>
>> Essentially, this creates multi-level, explicit fallbacks. The field<br>
>> info container, which was a bloated, weird shared state object, is now<br>
>> simply a dictionary subclass with a "fallback" option. When you<br>
>> create them, you can either create it in isolation (with no fallback)<br>
>> or with a fallback. When you query it, if it does not have a field,<br>
>> it checks its fallback. There are, additionally, two new functions<br>
>> for IO: the translation function and the null function. The first is<br>
>> to translate, for instance, "density" to "Density" and the second is<br>
>> to indicate that a field is expected to be found in an output from the<br>
>> simulation code.<br>
>><br>
>> There are now affiliated with each simulation code two field info<br>
>> objects: the "known" fields, which may appear in files, and the<br>
>> non-known (i.e., code-specific derived) fields. These live as the<br>
>> attributes _fieldinfo_fallback and _fieldinfo_known on the<br>
>> StaticOutput sublcass corresponding to a simulation code. When the<br>
>> Hierarchy (not static output) is instantiated, the first step is to<br>
>> create a new field_info object. This has, as a fallback, the<br>
>> _fieldinfo_fallback, which itself has as a fallback the<br>
>> universally-known derived fields. The hierarchy then queries the<br>
>> output file for which fields are available. This process then looks<br>
>> for a corresponding field in fieldinfo_known, and if it finds it, it<br>
>> adds it to the field_info object, *overriding* any possible derived<br>
>> fields. (In this manner, for instance, yt will not recalculate a<br>
>> "CoolingTime" field if one exists in the output.)<br>
>><br>
>> = What it aims to do in the future =<br>
>><br>
>> This will be utilized in three main ways:<br>
>><br>
>> 1) Making it more clear which fields belong to which code, and which<br>
>> come from disk and which are derived<br>
>> 2) Help move IO into fields, to optimize for geometries and data containers<br>
>> 3) Make units more clear and specific<br>
>> 4) This is all designed around better supporting the GDF.<br>
>><br>
>> = Where from here? =<br>
>><br>
>> It would be hugely beneficial if you could test this and report back.<br>
>> I have created a pull request:<br>
>><br>
>> <a href="https://bitbucket.org/yt_analysis/yt/pull-request/27/field-overhaul-to-utilize-explicit" target="_blank">https://bitbucket.org/yt_analysis/yt/pull-request/27/field-overhaul-to-utilize-explicit</a><br>
>><br>
>> This is by no means a settled matter; I think we need to have testing<br>
>> on this, buy-in from developers and users, and to make sure that old<br>
>> code doesn't beak. The test cases all pass for me for Enzo.<br>
>><br>
>> Before this can be merged, I would hope we can get some testing from:<br>
>><br>
>> * Enzo<br>
>> * Nyx<br>
>> * FLASH<br>
>> * Orion<br>
>><br>
>> and any other codes that can hear me.<br>
>><br>
>> Thanks very much for your time; please let me know if you have any<br>
>> questions, concerns, jokes, comments, improvements, CDs of your band,<br>
>> suggestions, and so on. For this major of a change I'd like to keep<br>
>> discussion on list, so the record of this is a bit more prominent.<br>
>><br>
>> Best,<br>
>><br>
>> Matt<br>
>> _______________________________________________<br>
>> Yt-dev mailing list<br>
>> <a href="mailto:Yt-dev@lists.spacepope.org">Yt-dev@lists.spacepope.org</a><br>
>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>><br>
> _______________________________________________<br>
> Yt-dev mailing list<br>
> <a href="mailto:Yt-dev@lists.spacepope.org">Yt-dev@lists.spacepope.org</a><br>
> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
><br>
<br>
<br>
<br>
</div></div><span class="HOEnZb"><font color="#888888">--<br>
Sent from my computer.<br>
</font></span><div class="HOEnZb"><div class="h5">_______________________________________________<br>
Yt-dev mailing list<br>
<a href="mailto:Yt-dev@lists.spacepope.org">Yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
</div></div></blockquote></div><br></div>