[yt-dev] YTArray YTEP

Matthew Turk matthewturk at gmail.com
Wed Mar 13 11:20:10 PDT 2013 

Hi Casey,

On Wed, Mar 13, 2013 at 12:24 PM, Casey W. Stark <caseywstark at gmail.com> wrote:
> Hey Matt.
>
> I think Britton suggested something like this too. Something like
> units="unknown" that works like nan, but for units. So whenever you combine
> a YTArray with valid units and another YTArray with "unknown" units, the
> valid units get clobbered into this "unknown" state. Is this something we
> need to implement?

Probably not for now -- we can do this anywhere an error is thrown, so
it should be an easy change to make if we deem it necessary.

-Matt

>
> - Casey
>
>
> On Tue, Mar 12, 2013 at 11:13 AM, Matthew Turk <matthewturk at gmail.com>
> wrote:
>>
>> On Tue, Mar 12, 2013 at 2:09 PM, Nathan Goldbaum <nathan12343 at gmail.com>
>> wrote:
>> >> It's probably also worth noting that the *vast* majority of times this
>> >> will be used in yt will be from within a call to __getitem__, where I
>> >> think we wrap with units anyway, right?
>> >
>> > Right, we already wrap the units to create the YTArray in __getitem__.
>> > While most of the YTArrays will be created there, we need to make sure that
>> > math operations work as well, leading to the complications Casey is bringing
>> > up.
>>
>> Yes, agreed.  One point I think I didn't make at the workshop that I
>> wanted to is that there should be a mechanism of noting that a YTArray
>> is in an "unknown state", or an easy way of stripping the units.  This
>> would enable people to continue to do relatively nasty things, if they
>> really want to.  (And some people, myself included, will want to and
>> will get frustrated if we are hindered from doing so.)
>>
>> >
>> > On Mar 12, 2013, at 11:03 AM, Matthew Turk <matthewturk at gmail.com>
>> > wrote:
>> >
>> >> On Tue, Mar 12, 2013 at 2:01 PM, Nathan Goldbaum
>> >> <nathan12343 at gmail.com> wrote:
>> >>> Hi Casey,
>> >>>
>> >>> Hmm, I'm not sure and like Matt suggests it might be worth enquiring
>> >>> on the
>> >>> numpy list or doing some digging for other subclasses of NDArray.
>> >>>
>> >>> I worry about doing to much in __array_wrap__ and __array_finalize__
>> >>> because
>> >>> of the nonuniform way the context keyword is populated by different
>> >>> numpy
>> >>> operations.  If there is an exhaustive list of operations we can look
>> >>> up or
>> >>> generate, that might be enough to engineer a solution around that.
>> >>
>> >> I like that solution -- is there a spec?
>> >>
>> >> Also, it seems to me that my first reply was perhaps misguided -- I
>> >> like Britton's point about addressing this in the array_wrap.
>> >>
>> >> It's probably also worth noting that the *vast* majority of times this
>> >> will be used in yt will be from within a call to __getitem__, where I
>> >> think we wrap with units anyway, right?
>> >>
>> >>>
>> >>> -Nathan
>> >>>
>> >>> On Mar 12, 2013, at 10:20 AM, "Casey W. Stark" <caseywstark at gmail.com>
>> >>> wrote:
>> >>>
>> >>> Hi all.
>> >>>
>> >>> I'm writing the YTEP and I came across a technical problem.
>> >>>
>> >>> For the operation overloading, I realized that we might be able/have
>> >>> to
>> >>> remove the __op__ methods and do everything inside the __array_wrap__
>> >>> method. Currently if you call `a + b` (where they're YTArray objects),
>> >>> it
>> >>> will check the units and make sure that's valid. But if you call
>> >>> np.add(a,
>> >>> b), it goes straight to __array_wrap__ without checking units!
>> >>>
>> >>> Maybe we can have two ufunc registries -- one for unit conversion
>> >>> functions
>> >>> and one for unit check functions?
>> >>>
>> >>> - Casey
>> >>>
>> >>>
>> >>> On Mon, Mar 11, 2013 at 11:28 AM, Matthew Turk <matthewturk at gmail.com>
>> >>> wrote:
>> >>>>
>> >>>> Hi Doug,
>> >>>>
>> >>>> On Mon, Mar 11, 2013 at 1:40 PM, Douglas Harvey Rudd
>> >>>> <drudd at uchicago.edu>
>> >>>> wrote:
>> >>>>> Hi Matt,
>> >>>>>
>> >>>>> Let us know when you'd like us to push ARTIO changes to the
>> >>>>> mainline.
>> >>>>> It's still
>> >>>>> being actively worked on, but the main features should work, and we
>> >>>>> really would like
>> >>>>> to get the new code into the first 3.0 alpha so we can push our
>> >>>>> users to
>> >>>>> start testing.
>> >>>>
>> >>>> Awesome!  I think I will have time to read and test them in detail
>> >>>> probably starting Wednesday, but I suspect that they're probably in a
>> >>>> good state already to go in.  I'll ping you once I've put in a
>> >>>> handful
>> >>>> of other changes I'm looking at.
>> >>>>
>> >>>> Incidentally, I'm planning to put all this on a blog post about the
>> >>>> workshop, but I wanted to take this opportunity to say how cool it
>> >>>> was
>> >>>> to see over the course of the few days the transition for ARTIO to a
>> >>>> frontend based on opaque data objects.
>> >>>>
>> >>>> -Matt
>> >>>>
>> >>>>>
>> >>>>> Douglas Rudd
>> >>>>> Scientific Computing Consultant
>> >>>>> Research Computing Center
>> >>>>> drudd at uchicago.edu
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On Mar 11, 2013, at 12:21 PM, Matthew Turk <matthewturk at gmail.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>>> On Mon, Mar 11, 2013 at 1:15 PM, Nathan Goldbaum
>> >>>>>> <nathan12343 at gmail.com> wrote:
>> >>>>>>> Hi Matt,
>> >>>>>>>
>> >>>>>>> I think the steps you outline are a good start.  My plan to
>> >>>>>>> continue
>> >>>>>>> the
>> >>>>>>> units work was to try to get the unit tests passing and see what
>> >>>>>>> needs
>> >>>>>>> to
>> >>>>>>> get done as that process proceeds.  I'm glad to hear you're
>> >>>>>>> excited
>> >>>>>>> enough
>> >>>>>>> about this to get started on that work.
>> >>>>>>
>> >>>>>> Absolutely.  On the plane I was able to get it down to 193 fails,
>> >>>>>> out
>> >>>>>> of ~2600 or so.  The biggest issues seem to be a problem with how
>> >>>>>> the
>> >>>>>> broadcasting is working (as evidenced in the radial velocity
>> >>>>>> fields),
>> >>>>>> things like Baroclinic vorticity disagreeing with how we're
>> >>>>>> changing
>> >>>>>> units, and some misc other problems.  I've pushed my changes up to
>> >>>>>> Casey's fork.
>> >>>>>>
>> >>>>>>>
>> >>>>>>> At the moment the work of subclassing NDArray is probably ~75%
>> >>>>>>> complete.  To
>> >>>>>>> finish we'll need to make sure we cover all of the possible ufuncs
>> >>>>>>> and
>> >>>>>>> do
>> >>>>>>> appropriate operations in all cases.  Having a formal spec will
>> >>>>>>> help
>> >>>>>>> in that
>> >>>>>>> effort.  There are also a couple of odd issues that still need to
>> >>>>>>> be
>> >>>>>>> worked
>> >>>>>>> out to make the code cleaner, i.e. the issue referenced in
>> >>>>>>> fce1e6c.
>> >>>>>>
>> >>>>>> I definitely think we need to address the ufuncs through design.
>> >>>>>> There are several issues I'm concerned about, specifically those
>> >>>>>> cases
>> >>>>>> where we do *not* want to return a subclass of YTArray.  As an
>> >>>>>> example, any of the ufuncs that return logical operators.  These
>> >>>>>> could
>> >>>>>> in principle return dimensionless (which is what I have done) but I
>> >>>>>> think I'm more comfortable returning just standard arrays . I've
>> >>>>>> also
>> >>>>>> had to put in what I think is something of a hack in returning
>> >>>>>> scalars
>> >>>>>> when a ufunc drops to a single value.
>> >>>>>>
>> >>>>>>>
>> >>>>>>> Right now all of this work is going on in Casey's yt-3.0 fork:
>> >>>>>>> https://bitbucket.org/caseywstark/yt-3.0.  Last week we were just
>> >>>>>>> pushing
>> >>>>>>> directly to the fork but I think now that we're all in different
>> >>>>>>> time
>> >>>>>>> zones
>> >>>>>>> it's probably a good idea to process further changes via pull
>> >>>>>>> requests
>> >>>>>>> to
>> >>>>>>> that fork.  Once everything is ready, Casey will be able to issue
>> >>>>>>> a
>> >>>>>>> pull
>> >>>>>>> request to the yt_analysis/yt-3.0 repository.  Does anyone
>> >>>>>>> disagree
>> >>>>>>> with
>> >>>>>>> that idea?
>> >>>>>>
>> >>>>>> Yes, I think it should live separately for a while -- it's
>> >>>>>> *incredibly* invasive, and I'd like to roll out 3.0a1 on the 15th.
>> >>>>>> I
>> >>>>>> think we should wait until the tagging of 3.0a1 to do this.  I'm
>> >>>>>> also
>> >>>>>> a bit concerned that there's heavy development on the ART and ARTIO
>> >>>>>> fronts that has not yet been pulled back in to mainline, and I want
>> >>>>>> to
>> >>>>>> try to pull that in before this change.  I'm not trying to put the
>> >>>>>> brakes on the development, just that I think we need to take this a
>> >>>>>> bit carefully since it's quite invasive and we now have a set of
>> >>>>>> people using 3.0 for their work.  Maybe we could aim to have the
>> >>>>>> YTEP
>> >>>>>> done, and then see how ready the code base is?
>> >>>>>>
>> >>>>>>>
>> >>>>>>> Cheers,
>> >>>>>>>
>> >>>>>>> Nathan
>> >>>>>>>
>> >>>>>>> On Mar 11, 2013, at 9:09 AM, Matthew Turk <matthewturk at gmail.com>
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>> Hi all,
>> >>>>>>>
>> >>>>>>> Last week the units stuff really took off -- great work on that
>> >>>>>>> front!
>> >>>>>>> I was wondering if I might solicit a YTEP?  As I have been going
>> >>>>>>> through the YTArray changes to get the unit tests passing, I'm
>> >>>>>>> realizing just how invasive it is in some ways.  I think having an
>> >>>>>>> enumeration of the behavior of the YTArray under ufuncs as well as
>> >>>>>>> common operations will be pretty essential.  That is something we
>> >>>>>>> should also probably discuss on this list.
>> >>>>>>>
>> >>>>>>> I'll be submitting a few changes for review by Casey and Nathan as
>> >>>>>>> well.  Do either of you have a feeling for where things stand now
>> >>>>>>> and
>> >>>>>>> what is needed next?
>> >>>>>>>
>> >>>>>>> -Matt
>> >>>>>>> _______________________________________________
>> >>>>>>> yt-dev mailing list
>> >>>>>>> yt-dev at lists.spacepope.org
>> >>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> _______________________________________________
>> >>>>>>> yt-dev mailing list
>> >>>>>>> yt-dev at lists.spacepope.org
>> >>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >>>>>>>
>> >>>>>> _______________________________________________
>> >>>>>> yt-dev mailing list
>> >>>>>> yt-dev at lists.spacepope.org
>> >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >>>>>
>> >>>>> _______________________________________________
>> >>>>> yt-dev mailing list
>> >>>>> yt-dev at lists.spacepope.org
>> >>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >>>> _______________________________________________
>> >>>> yt-dev mailing list
>> >>>> yt-dev at lists.spacepope.org
>> >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> yt-dev mailing list
>> >>> yt-dev at lists.spacepope.org
>> >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >>>
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> yt-dev mailing list
>> >>> yt-dev at lists.spacepope.org
>> >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >>>
>> >> _______________________________________________
>> >> yt-dev mailing list
>> >> yt-dev at lists.spacepope.org
>> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> >
>> > _______________________________________________
>> > yt-dev mailing list
>> > yt-dev at lists.spacepope.org
>> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>

More information about the yt-dev mailing list