[yt-dev] Native field discussion

Tom Abel tabel at slac.stanford.edu
Tue May 7 14:37:57 PDT 2013 

Let me add 2 cents as well. 
When I worried about this for jacques I tried to separate units from data fields as well. There I decided to only ever affect labels and never the data itself. So e.g. if I look at a slice of the density field and I wanted to switch between physical and code units I only adjust the labels and axis not the data itself. This had the very nice advantage that it was essentially instantaneous and did not involve ever having to multiply the entire data with a scaling factor or worse have to reread data because I only implemented a method to translate to cvs but not back. 

This has the consequence that I had to specify how units are defined for derived fields as well. That however, had the nice opportunity also that I had extra control over the units of these derived fields. 

For an interactive GUI this is relevant. For the scripting you may not care. 

Anyhow. This discussion is very interesting! 
Best,
Tom



On May 7, 2013, at 2:10 PM, Matthew Turk <matthewturk at gmail.com> wrote:

> On Tue, May 7, 2013 at 5:06 PM, j s oishi <jsoishi at gmail.com> wrote:
>> 
>> 
>> 
>> On Tue, May 7, 2013 at 1:25 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>>> 
>>> On Tue, May 7, 2013 at 1:24 PM, Casey W. Stark <caseywstark at gmail.com>
>>> wrote:
>>>> Ok, I just tried this and it doesn't work. Even though the Unit objects
>>>> are
>>>> separate instances, the sympy Symbol objects in the expressions are
>>>> treated
>>>> as the same. We will have to think about the design some more.
>>> 
>>> Although, on reflection, maybe *this is okay*.  We should perhaps
>>> assume that anyone who is actually dividing or multiplying or
>>> something the native code units knows what they're doing.  If they
>>> want to look at the cgs units, they should be doing so.  I'm not
>>> entirely sure.
>> 
>> 
>> I'm not so sure about this. What about simulations that are performed in
>> dimensionless units? This is something that yt handles poorly right now, and
>> I think this system should fix that. For example, in a dimensionless system,
>> the code units are fully functional and probably what you want to work in.
>> Or am I missing something here?
> 
> What you're describing will be much, much easier to do in the system
> Casey and Nathan have outlined.  What I was describing was comparing
> two code unitful arrays with the same name where that name means
> something different for each of the two arrays.  For instance,
> comparing the same Enzo cosmology run in *code units* at z=10 and z=1.
> I at first thought that this would be an undefined operation that we
> don't know how to manage, but then I realized that anyone who is doing
> this probably knows what they are doing.
> 
> For dimensionless simulations, you will be able to access the native
> fields with less overhead in this new system, simply by querying -- as
> Casey showed above -- the native fields.
> 
> This will also simplify (as Casey noted) the field generation and
> field info container logic.
> 
>> 
>>> 
>>> 
>>>> 
>>>> 
>>>> On Tue, May 7, 2013 at 10:17 AM, Casey W. Stark <caseywstark at gmail.com>
>>>> wrote:
>>>>> 
>>>>>> think of having two different parameter
>>>>> files from the same simulation open at the same time.
>>>>> 
>>>>> Ah, got it. You're right that this will be a problem. The two datasets
>>>>> will have separate unit registries, so something like "pccm" units can
>>>>> be
>>>>> different from each. However, if you try something like
>>>>> ds1_dd["enzo_comoving_density"] / ds2_dd["enzo_comoving_density"],
>>>>> sympy
>>>>> might just cancel out the symbols in the unit expressions and the
>>>>> conversion
>>>>> will not work. I'll try something...
>>>>> 
>>>>> 
>>>>> On Tue, May 7, 2013 at 10:05 AM, Matthew Turk <matthewturk at gmail.com>
>>>>> wrote:
>>>>>> 
>>>>>> On Tue, May 7, 2013 at 12:13 PM, Casey W. Stark
>>>>>> <caseywstark at gmail.com>
>>>>>> wrote:
>>>>>>> Hey Matt.
>>>>>>> 
>>>>>>> Excellent! It's a considerable change, so we should dread it a bit,
>>>>>>> but
>>>>>>> really glad you like the idea too.
>>>>>>> 
>>>>>>> About where the conversion should happen, I like the idea to return
>>>>>>> any
>>>>>>> universal fields in cgs by default. Operations between "code" fields
>>>>>>> ("mycode_density") would return in code units and ops between
>>>>>>> universal
>>>>>>> fields ("density") would return in cgs. That means most users won't
>>>>>>> see
>>>>>>> a
>>>>>>> change too. Mixing units is mostly straightforward. I think the only
>>>>>>> gotcha
>>>>>>> is for add and sub, where we return in the left object's units.
>>>>>> 
>>>>>> That sounds good to me.
>>>>>> 
>>>>>>> 
>>>>>>>> How do we handle the fact
>>>>>>> that converting "code_density" to cgs is different for different
>>>>>>> arrays, depending on the code?  i.e., if I have two "code_density"
>>>>>>> arrays from a code that stores them in comoving code units, how do
>>>>>>> we
>>>>>>> distinguish between those two things during division, say?
>>>>>>> 
>>>>>>> I'm not sure I'm following here, but is this a case where density
>>>>>>> field
>>>>>>> a
>>>>>>> has different units from density field b? If so, I think we have to
>>>>>>> define
>>>>>>> separate units for the code densities. Here's a quick Nyx example...
>>>>>> 
>>>>>> Not exactly, but close -- think of having two different parameter
>>>>>> files from the same simulation open at the same time.  If the sim
>>>>>> stores internally in comoving units, the native->propercgs will vary
>>>>>> between those two, but the registered units might be the same name,
>>>>>> for instance "enzo_density" or something.
>>>>>> 
>>>>>>> 
>>>>>>> unit_registry.add("nyx_cm_density", "Msun/Mpccm**3")
>>>>>>> unit_registry.add("nyx_prop_density", "g/cm**3")
>>>>>>> 
>>>>>>> add_field("nyx_density", units="nyx_cm_density")
>>>>>>> add_field("nyx_hi_density", units="nyx_prop_density")
>>>>>>> 
>>>>>>> dd["nyx_density"] / dd["nyx_hi_density"]    (should return
>>>>>>> dimensionless
>>>>>>> YTArray)
>>>>>>> 
>>>>>>> - Casey
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Tue, May 7, 2013 at 8:18 AM, Matthew Turk <matthewturk at gmail.com>
>>>>>>> wrote:
>>>>>>>> 
>>>>>>>> Hi Casey and Nathan,
>>>>>>>> 
>>>>>>>> Last night when I read this I felt a bit of dread in my stomach,
>>>>>>>> worrying that this would make things a lot more complex and
>>>>>>>> difficult.
>>>>>>>> But I have to say, now having thought it through and read your
>>>>>>>> pastes, reflected on the contents of the YTEP ... I think this is
>>>>>>>> beautiful and elegant.  Very nicely thought out.
>>>>>>>> 
>>>>>>>> I completely agree that -- particularly in 3.0 -- we no longer need
>>>>>>>> to
>>>>>>>> perform any conversions at IO time, and instead we should be
>>>>>>>> pushing
>>>>>>>> this to the places where the data will be presented to individual
>>>>>>>> users.  This will require some thought about where we want this
>>>>>>>> conversion to take place (i.e., at the display layer?) but it
>>>>>>>> should
>>>>>>>> be completely workable to do so.  The ensure_cgs decorator is a
>>>>>>>> great
>>>>>>>> way to manage this.
>>>>>>>> 
>>>>>>>> I'm +1 on this, and can help with the technical details.  As it
>>>>>>>> stands, all unit conversions happen in a single place in the code,
>>>>>>>> so
>>>>>>>> we can simply remove the items there and instead just wrap them in
>>>>>>>> the
>>>>>>>> appropriate "code" units.  We'll need to figure out the specific
>>>>>>>> semantics of that but it should be okay.  How do we handle the fact
>>>>>>>> that converting "code_density" to cgs is different for different
>>>>>>>> arrays, depending on the code?  i.e., if I have two "code_density"
>>>>>>>> arrays from a code that stores them in comoving code units, how do
>>>>>>>> we
>>>>>>>> distinguish between those two things during division, say?
>>>>>>>> 
>>>>>>>> -Matt
>>>>>>>> 
>>>>>>>> On Mon, May 6, 2013 at 6:50 PM, Casey W. Stark
>>>>>>>> <caseywstark at gmail.com>
>>>>>>>> wrote:
>>>>>>>>> Hey Nathan.
>>>>>>>>> 
>>>>>>>>> Thanks for the reminder about `ensure_cgs`. Definitely helps to
>>>>>>>>> see
>>>>>>>>> the
>>>>>>>>> pseudocode too. Users could also write out the
>>>>>>>>> array.convert_to(...)
>>>>>>>>> in
>>>>>>>>> the
>>>>>>>>> field function for whatever obscure units. I guess this is
>>>>>>>>> similar
>>>>>>>>> to
>>>>>>>>> the
>>>>>>>>> fields with the units tacked on to the name, but much cleaner.
>>>>>>>>> 
>>>>>>>>> I think making the code to universal field mapping more explicit
>>>>>>>>> will be
>>>>>>>>> very helpful. I've been confused about where some derived fields
>>>>>>>>> come
>>>>>>>>> from
>>>>>>>>> many times!
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Mon, May 6, 2013 at 3:39 PM, Nathan Goldbaum
>>>>>>>>> <goldbaum at ucolick.org>
>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> Hey Casey,
>>>>>>>>>> 
>>>>>>>>>> That's pretty much what I was thinking, although I wanted to add
>>>>>>>>>> a
>>>>>>>>>> little
>>>>>>>>>> bit of syntactic sugar so the CGS conversion happens
>>>>>>>>>> automatically:
>>>>>>>>>> 
>>>>>>>>>> http://paste.yt-project.org/show/3443/
>>>>>>>>>> 
>>>>>>>>>> The ensure_cgs decorator would simply call convert_to_cgs on the
>>>>>>>>>> returned
>>>>>>>>>> YTArray.  The nice thing about this is the flow of data is
>>>>>>>>>> clear,
>>>>>>>>>> things
>>>>>>>>>> come in to the field definition code units and leave in CGS.
>>>>>>>>>> This
>>>>>>>>>> also
>>>>>>>>>> eliminates the need for the unit conversion functions that are
>>>>>>>>>> scattered
>>>>>>>>>> throughout the frontends and universal_fields.py.
>>>>>>>>>> 
>>>>>>>>>> This has the nice feature that each of the code frontends will
>>>>>>>>>> need
>>>>>>>>>> to
>>>>>>>>>> explicitly define mappings to the CGS fields (like Density)
>>>>>>>>>> needed
>>>>>>>>>> by
>>>>>>>>>> universal fields.
>>>>>>>>>> 
>>>>>>>>>> -Nathan
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Mon, May 6, 2013 at 3:33 PM, Casey W. Stark
>>>>>>>>>> <caseywstark at gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Hey yt.
>>>>>>>>>>> 
>>>>>>>>>>> Nathan and I had a cool idea as an extension of the units work,
>>>>>>>>>>> but I
>>>>>>>>>>> am
>>>>>>>>>>> hitting a yt internals knowledge wall writing a YTEP. I would
>>>>>>>>>>> like
>>>>>>>>>>> some
>>>>>>>>>>> input on technical details or general feelings before I move
>>>>>>>>>>> ahead
>>>>>>>>>>> with the
>>>>>>>>>>> write-up.
>>>>>>>>>>> 
>>>>>>>>>>> One of the things mentioned in the units YTEP is handling code
>>>>>>>>>>> units.
>>>>>>>>>>> Basically each frontend static output should register the code
>>>>>>>>>>> units.
>>>>>>>>>>> In Nyx
>>>>>>>>>>> for instance, we would do something like registering
>>>>>>>>>>> "code_length"
>>>>>>>>>>> as
>>>>>>>>>>> Mpccm,
>>>>>>>>>>> "code_mass" as Msun, etc. Then any field from the dataset can
>>>>>>>>>>> be
>>>>>>>>>>> loaded in
>>>>>>>>>>> code units with something like
>>>>>>>>>>> dd["density"].in_units("code_density").
>>>>>>>>>>> 
>>>>>>>>>>> This got us thinking -- why load native fields in CGS now?
>>>>>>>>>>> Instead, we
>>>>>>>>>>> can define native fields exactly as they are on disk and users
>>>>>>>>>>> who
>>>>>>>>>>> want code
>>>>>>>>>>> units will never have to convert by hand again. If frontend
>>>>>>>>>>> developers
>>>>>>>>>>> define the fields and units on disk, and some mapping to the
>>>>>>>>>>> universal
>>>>>>>>>>> fields, we shouldn't have to define or convert units anywhere
>>>>>>>>>>> else.
>>>>>>>>>>> 
>>>>>>>>>>> I made a quick and dirty example of what we're thinking here:
>>>>>>>>>>> http://paste.yt-project.org/show/3441/
>>>>>>>>>>> 
>>>>>>>>>>> So, is something like this a good idea? Is it realistic? Would
>>>>>>>>>>> this
>>>>>>>>>>> make
>>>>>>>>>>> for a simpler field info container? Nathan, is there anything I
>>>>>>>>>>> missed?
>>>>>>>>>>> 
>>>>>>>>>>> - Casey
>>>>>>>>>>> 
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> yt-dev mailing list
>>>>>>>>>>> yt-dev at lists.spacepope.org
>>>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> _______________________________________________
>>>>>>>>>> yt-dev mailing list
>>>>>>>>>> yt-dev at lists.spacepope.org
>>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> _______________________________________________
>>>>>>>>> yt-dev mailing list
>>>>>>>>> yt-dev at lists.spacepope.org
>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>>>> 
>>>>>>>> _______________________________________________
>>>>>>>> yt-dev mailing list
>>>>>>>> yt-dev at lists.spacepope.org
>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> yt-dev mailing list
>>>>>>> yt-dev at lists.spacepope.org
>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>> 
>>>>>> _______________________________________________
>>>>>> yt-dev mailing list
>>>>>> yt-dev at lists.spacepope.org
>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> yt-dev mailing list
>>>> yt-dev at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>> 
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> 
>> 
>> 
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> 
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

More information about the yt-dev mailing list