[yt-dev] YTArray YTEP

Casey W. Stark caseywstark at gmail.com
Wed Mar 13 09:24:28 PDT 2013 

Hey Matt.

I think Britton suggested something like this too. Something like
units="unknown" that works like nan, but for units. So whenever you combine
a YTArray with valid units and another YTArray with "unknown" units, the
valid units get clobbered into this "unknown" state. Is this something we
need to implement?

- Casey


On Tue, Mar 12, 2013 at 11:13 AM, Matthew Turk <matthewturk at gmail.com>wrote:

> On Tue, Mar 12, 2013 at 2:09 PM, Nathan Goldbaum <nathan12343 at gmail.com>
> wrote:
> >> It's probably also worth noting that the *vast* majority of times this
> >> will be used in yt will be from within a call to __getitem__, where I
> >> think we wrap with units anyway, right?
> >
> > Right, we already wrap the units to create the YTArray in __getitem__.
>  While most of the YTArrays will be created there, we need to make sure
> that math operations work as well, leading to the complications Casey is
> bringing up.
>
> Yes, agreed.  One point I think I didn't make at the workshop that I
> wanted to is that there should be a mechanism of noting that a YTArray
> is in an "unknown state", or an easy way of stripping the units.  This
> would enable people to continue to do relatively nasty things, if they
> really want to.  (And some people, myself included, will want to and
> will get frustrated if we are hindered from doing so.)
>
> >
> > On Mar 12, 2013, at 11:03 AM, Matthew Turk <matthewturk at gmail.com>
> wrote:
> >
> >> On Tue, Mar 12, 2013 at 2:01 PM, Nathan Goldbaum <nathan12343 at gmail.com>
> wrote:
> >>> Hi Casey,
> >>>
> >>> Hmm, I'm not sure and like Matt suggests it might be worth enquiring
> on the
> >>> numpy list or doing some digging for other subclasses of NDArray.
> >>>
> >>> I worry about doing to much in __array_wrap__ and __array_finalize__
> because
> >>> of the nonuniform way the context keyword is populated by different
> numpy
> >>> operations.  If there is an exhaustive list of operations we can look
> up or
> >>> generate, that might be enough to engineer a solution around that.
> >>
> >> I like that solution -- is there a spec?
> >>
> >> Also, it seems to me that my first reply was perhaps misguided -- I
> >> like Britton's point about addressing this in the array_wrap.
> >>
> >> It's probably also worth noting that the *vast* majority of times this
> >> will be used in yt will be from within a call to __getitem__, where I
> >> think we wrap with units anyway, right?
> >>
> >>>
> >>> -Nathan
> >>>
> >>> On Mar 12, 2013, at 10:20 AM, "Casey W. Stark" <caseywstark at gmail.com>
> >>> wrote:
> >>>
> >>> Hi all.
> >>>
> >>> I'm writing the YTEP and I came across a technical problem.
> >>>
> >>> For the operation overloading, I realized that we might be able/have to
> >>> remove the __op__ methods and do everything inside the __array_wrap__
> >>> method. Currently if you call `a + b` (where they're YTArray objects),
> it
> >>> will check the units and make sure that's valid. But if you call
> np.add(a,
> >>> b), it goes straight to __array_wrap__ without checking units!
> >>>
> >>> Maybe we can have two ufunc registries -- one for unit conversion
> functions
> >>> and one for unit check functions?
> >>>
> >>> - Casey
> >>>
> >>>
> >>> On Mon, Mar 11, 2013 at 11:28 AM, Matthew Turk <matthewturk at gmail.com>
> >>> wrote:
> >>>>
> >>>> Hi Doug,
> >>>>
> >>>> On Mon, Mar 11, 2013 at 1:40 PM, Douglas Harvey Rudd <
> drudd at uchicago.edu>
> >>>> wrote:
> >>>>> Hi Matt,
> >>>>>
> >>>>> Let us know when you'd like us to push ARTIO changes to the mainline.
> >>>>> It's still
> >>>>> being actively worked on, but the main features should work, and we
> >>>>> really would like
> >>>>> to get the new code into the first 3.0 alpha so we can push our
> users to
> >>>>> start testing.
> >>>>
> >>>> Awesome!  I think I will have time to read and test them in detail
> >>>> probably starting Wednesday, but I suspect that they're probably in a
> >>>> good state already to go in.  I'll ping you once I've put in a handful
> >>>> of other changes I'm looking at.
> >>>>
> >>>> Incidentally, I'm planning to put all this on a blog post about the
> >>>> workshop, but I wanted to take this opportunity to say how cool it was
> >>>> to see over the course of the few days the transition for ARTIO to a
> >>>> frontend based on opaque data objects.
> >>>>
> >>>> -Matt
> >>>>
> >>>>>
> >>>>> Douglas Rudd
> >>>>> Scientific Computing Consultant
> >>>>> Research Computing Center
> >>>>> drudd at uchicago.edu
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Mar 11, 2013, at 12:21 PM, Matthew Turk <matthewturk at gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> On Mon, Mar 11, 2013 at 1:15 PM, Nathan Goldbaum
> >>>>>> <nathan12343 at gmail.com> wrote:
> >>>>>>> Hi Matt,
> >>>>>>>
> >>>>>>> I think the steps you outline are a good start.  My plan to
> continue
> >>>>>>> the
> >>>>>>> units work was to try to get the unit tests passing and see what
> needs
> >>>>>>> to
> >>>>>>> get done as that process proceeds.  I'm glad to hear you're excited
> >>>>>>> enough
> >>>>>>> about this to get started on that work.
> >>>>>>
> >>>>>> Absolutely.  On the plane I was able to get it down to 193 fails,
> out
> >>>>>> of ~2600 or so.  The biggest issues seem to be a problem with how
> the
> >>>>>> broadcasting is working (as evidenced in the radial velocity
> fields),
> >>>>>> things like Baroclinic vorticity disagreeing with how we're changing
> >>>>>> units, and some misc other problems.  I've pushed my changes up to
> >>>>>> Casey's fork.
> >>>>>>
> >>>>>>>
> >>>>>>> At the moment the work of subclassing NDArray is probably ~75%
> >>>>>>> complete.  To
> >>>>>>> finish we'll need to make sure we cover all of the possible ufuncs
> and
> >>>>>>> do
> >>>>>>> appropriate operations in all cases.  Having a formal spec will
> help
> >>>>>>> in that
> >>>>>>> effort.  There are also a couple of odd issues that still need to
> be
> >>>>>>> worked
> >>>>>>> out to make the code cleaner, i.e. the issue referenced in fce1e6c.
> >>>>>>
> >>>>>> I definitely think we need to address the ufuncs through design.
> >>>>>> There are several issues I'm concerned about, specifically those
> cases
> >>>>>> where we do *not* want to return a subclass of YTArray.  As an
> >>>>>> example, any of the ufuncs that return logical operators.  These
> could
> >>>>>> in principle return dimensionless (which is what I have done) but I
> >>>>>> think I'm more comfortable returning just standard arrays . I've
> also
> >>>>>> had to put in what I think is something of a hack in returning
> scalars
> >>>>>> when a ufunc drops to a single value.
> >>>>>>
> >>>>>>>
> >>>>>>> Right now all of this work is going on in Casey's yt-3.0 fork:
> >>>>>>> https://bitbucket.org/caseywstark/yt-3.0.  Last week we were just
> >>>>>>> pushing
> >>>>>>> directly to the fork but I think now that we're all in different
> time
> >>>>>>> zones
> >>>>>>> it's probably a good idea to process further changes via pull
> requests
> >>>>>>> to
> >>>>>>> that fork.  Once everything is ready, Casey will be able to issue a
> >>>>>>> pull
> >>>>>>> request to the yt_analysis/yt-3.0 repository.  Does anyone disagree
> >>>>>>> with
> >>>>>>> that idea?
> >>>>>>
> >>>>>> Yes, I think it should live separately for a while -- it's
> >>>>>> *incredibly* invasive, and I'd like to roll out 3.0a1 on the 15th.
>  I
> >>>>>> think we should wait until the tagging of 3.0a1 to do this.  I'm
> also
> >>>>>> a bit concerned that there's heavy development on the ART and ARTIO
> >>>>>> fronts that has not yet been pulled back in to mainline, and I want
> to
> >>>>>> try to pull that in before this change.  I'm not trying to put the
> >>>>>> brakes on the development, just that I think we need to take this a
> >>>>>> bit carefully since it's quite invasive and we now have a set of
> >>>>>> people using 3.0 for their work.  Maybe we could aim to have the
> YTEP
> >>>>>> done, and then see how ready the code base is?
> >>>>>>
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>>
> >>>>>>> Nathan
> >>>>>>>
> >>>>>>> On Mar 11, 2013, at 9:09 AM, Matthew Turk <matthewturk at gmail.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>> Hi all,
> >>>>>>>
> >>>>>>> Last week the units stuff really took off -- great work on that
> front!
> >>>>>>> I was wondering if I might solicit a YTEP?  As I have been going
> >>>>>>> through the YTArray changes to get the unit tests passing, I'm
> >>>>>>> realizing just how invasive it is in some ways.  I think having an
> >>>>>>> enumeration of the behavior of the YTArray under ufuncs as well as
> >>>>>>> common operations will be pretty essential.  That is something we
> >>>>>>> should also probably discuss on this list.
> >>>>>>>
> >>>>>>> I'll be submitting a few changes for review by Casey and Nathan as
> >>>>>>> well.  Do either of you have a feeling for where things stand now
> and
> >>>>>>> what is needed next?
> >>>>>>>
> >>>>>>> -Matt
> >>>>>>> _______________________________________________
> >>>>>>> yt-dev mailing list
> >>>>>>> yt-dev at lists.spacepope.org
> >>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> _______________________________________________
> >>>>>>> yt-dev mailing list
> >>>>>>> yt-dev at lists.spacepope.org
> >>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >>>>>>>
> >>>>>> _______________________________________________
> >>>>>> yt-dev mailing list
> >>>>>> yt-dev at lists.spacepope.org
> >>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >>>>>
> >>>>> _______________________________________________
> >>>>> yt-dev mailing list
> >>>>> yt-dev at lists.spacepope.org
> >>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >>>> _______________________________________________
> >>>> yt-dev mailing list
> >>>> yt-dev at lists.spacepope.org
> >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >>>
> >>>
> >>> _______________________________________________
> >>> yt-dev mailing list
> >>> yt-dev at lists.spacepope.org
> >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> yt-dev mailing list
> >>> yt-dev at lists.spacepope.org
> >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >>>
> >> _______________________________________________
> >> yt-dev mailing list
> >> yt-dev at lists.spacepope.org
> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> >
> > _______________________________________________
> > yt-dev mailing list
> > yt-dev at lists.spacepope.org
> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20130313/dd4bcb81/attachment.html>

More information about the yt-dev mailing list