[yt-dev] YTArray YTEP

Nathan Goldbaum nathan12343 at gmail.com
Tue Mar 12 11:09:34 PDT 2013 

> It's probably also worth noting that the *vast* majority of times this
> will be used in yt will be from within a call to __getitem__, where I
> think we wrap with units anyway, right?

Right, we already wrap the units to create the YTArray in __getitem__.  While most of the YTArrays will be created there, we need to make sure that math operations work as well, leading to the complications Casey is bringing up.

On Mar 12, 2013, at 11:03 AM, Matthew Turk <matthewturk at gmail.com> wrote:

> On Tue, Mar 12, 2013 at 2:01 PM, Nathan Goldbaum <nathan12343 at gmail.com> wrote:
>> Hi Casey,
>> 
>> Hmm, I'm not sure and like Matt suggests it might be worth enquiring on the
>> numpy list or doing some digging for other subclasses of NDArray.
>> 
>> I worry about doing to much in __array_wrap__ and __array_finalize__ because
>> of the nonuniform way the context keyword is populated by different numpy
>> operations.  If there is an exhaustive list of operations we can look up or
>> generate, that might be enough to engineer a solution around that.
> 
> I like that solution -- is there a spec?
> 
> Also, it seems to me that my first reply was perhaps misguided -- I
> like Britton's point about addressing this in the array_wrap.
> 
> It's probably also worth noting that the *vast* majority of times this
> will be used in yt will be from within a call to __getitem__, where I
> think we wrap with units anyway, right?
> 
>> 
>> -Nathan
>> 
>> On Mar 12, 2013, at 10:20 AM, "Casey W. Stark" <caseywstark at gmail.com>
>> wrote:
>> 
>> Hi all.
>> 
>> I'm writing the YTEP and I came across a technical problem.
>> 
>> For the operation overloading, I realized that we might be able/have to
>> remove the __op__ methods and do everything inside the __array_wrap__
>> method. Currently if you call `a + b` (where they're YTArray objects), it
>> will check the units and make sure that's valid. But if you call np.add(a,
>> b), it goes straight to __array_wrap__ without checking units!
>> 
>> Maybe we can have two ufunc registries -- one for unit conversion functions
>> and one for unit check functions?
>> 
>> - Casey
>> 
>> 
>> On Mon, Mar 11, 2013 at 11:28 AM, Matthew Turk <matthewturk at gmail.com>
>> wrote:
>>> 
>>> Hi Doug,
>>> 
>>> On Mon, Mar 11, 2013 at 1:40 PM, Douglas Harvey Rudd <drudd at uchicago.edu>
>>> wrote:
>>>> Hi Matt,
>>>> 
>>>> Let us know when you'd like us to push ARTIO changes to the mainline.
>>>> It's still
>>>> being actively worked on, but the main features should work, and we
>>>> really would like
>>>> to get the new code into the first 3.0 alpha so we can push our users to
>>>> start testing.
>>> 
>>> Awesome!  I think I will have time to read and test them in detail
>>> probably starting Wednesday, but I suspect that they're probably in a
>>> good state already to go in.  I'll ping you once I've put in a handful
>>> of other changes I'm looking at.
>>> 
>>> Incidentally, I'm planning to put all this on a blog post about the
>>> workshop, but I wanted to take this opportunity to say how cool it was
>>> to see over the course of the few days the transition for ARTIO to a
>>> frontend based on opaque data objects.
>>> 
>>> -Matt
>>> 
>>>> 
>>>> Douglas Rudd
>>>> Scientific Computing Consultant
>>>> Research Computing Center
>>>> drudd at uchicago.edu
>>>> 
>>>> 
>>>> 
>>>> On Mar 11, 2013, at 12:21 PM, Matthew Turk <matthewturk at gmail.com>
>>>> wrote:
>>>> 
>>>>> On Mon, Mar 11, 2013 at 1:15 PM, Nathan Goldbaum
>>>>> <nathan12343 at gmail.com> wrote:
>>>>>> Hi Matt,
>>>>>> 
>>>>>> I think the steps you outline are a good start.  My plan to continue
>>>>>> the
>>>>>> units work was to try to get the unit tests passing and see what needs
>>>>>> to
>>>>>> get done as that process proceeds.  I'm glad to hear you're excited
>>>>>> enough
>>>>>> about this to get started on that work.
>>>>> 
>>>>> Absolutely.  On the plane I was able to get it down to 193 fails, out
>>>>> of ~2600 or so.  The biggest issues seem to be a problem with how the
>>>>> broadcasting is working (as evidenced in the radial velocity fields),
>>>>> things like Baroclinic vorticity disagreeing with how we're changing
>>>>> units, and some misc other problems.  I've pushed my changes up to
>>>>> Casey's fork.
>>>>> 
>>>>>> 
>>>>>> At the moment the work of subclassing NDArray is probably ~75%
>>>>>> complete.  To
>>>>>> finish we'll need to make sure we cover all of the possible ufuncs and
>>>>>> do
>>>>>> appropriate operations in all cases.  Having a formal spec will help
>>>>>> in that
>>>>>> effort.  There are also a couple of odd issues that still need to be
>>>>>> worked
>>>>>> out to make the code cleaner, i.e. the issue referenced in fce1e6c.
>>>>> 
>>>>> I definitely think we need to address the ufuncs through design.
>>>>> There are several issues I'm concerned about, specifically those cases
>>>>> where we do *not* want to return a subclass of YTArray.  As an
>>>>> example, any of the ufuncs that return logical operators.  These could
>>>>> in principle return dimensionless (which is what I have done) but I
>>>>> think I'm more comfortable returning just standard arrays . I've also
>>>>> had to put in what I think is something of a hack in returning scalars
>>>>> when a ufunc drops to a single value.
>>>>> 
>>>>>> 
>>>>>> Right now all of this work is going on in Casey's yt-3.0 fork:
>>>>>> https://bitbucket.org/caseywstark/yt-3.0.  Last week we were just
>>>>>> pushing
>>>>>> directly to the fork but I think now that we're all in different time
>>>>>> zones
>>>>>> it's probably a good idea to process further changes via pull requests
>>>>>> to
>>>>>> that fork.  Once everything is ready, Casey will be able to issue a
>>>>>> pull
>>>>>> request to the yt_analysis/yt-3.0 repository.  Does anyone disagree
>>>>>> with
>>>>>> that idea?
>>>>> 
>>>>> Yes, I think it should live separately for a while -- it's
>>>>> *incredibly* invasive, and I'd like to roll out 3.0a1 on the 15th.  I
>>>>> think we should wait until the tagging of 3.0a1 to do this.  I'm also
>>>>> a bit concerned that there's heavy development on the ART and ARTIO
>>>>> fronts that has not yet been pulled back in to mainline, and I want to
>>>>> try to pull that in before this change.  I'm not trying to put the
>>>>> brakes on the development, just that I think we need to take this a
>>>>> bit carefully since it's quite invasive and we now have a set of
>>>>> people using 3.0 for their work.  Maybe we could aim to have the YTEP
>>>>> done, and then see how ready the code base is?
>>>>> 
>>>>>> 
>>>>>> Cheers,
>>>>>> 
>>>>>> Nathan
>>>>>> 
>>>>>> On Mar 11, 2013, at 9:09 AM, Matthew Turk <matthewturk at gmail.com>
>>>>>> wrote:
>>>>>> 
>>>>>> Hi all,
>>>>>> 
>>>>>> Last week the units stuff really took off -- great work on that front!
>>>>>> I was wondering if I might solicit a YTEP?  As I have been going
>>>>>> through the YTArray changes to get the unit tests passing, I'm
>>>>>> realizing just how invasive it is in some ways.  I think having an
>>>>>> enumeration of the behavior of the YTArray under ufuncs as well as
>>>>>> common operations will be pretty essential.  That is something we
>>>>>> should also probably discuss on this list.
>>>>>> 
>>>>>> I'll be submitting a few changes for review by Casey and Nathan as
>>>>>> well.  Do either of you have a feeling for where things stand now and
>>>>>> what is needed next?
>>>>>> 
>>>>>> -Matt
>>>>>> _______________________________________________
>>>>>> yt-dev mailing list
>>>>>> yt-dev at lists.spacepope.org
>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> yt-dev mailing list
>>>>>> yt-dev at lists.spacepope.org
>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>> 
>>>>> _______________________________________________
>>>>> yt-dev mailing list
>>>>> yt-dev at lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>> 
>>>> _______________________________________________
>>>> yt-dev mailing list
>>>> yt-dev at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> 
>> 
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> 
>> 
>> 
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> 
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

More information about the yt-dev mailing list