[yt-dev] Merging the open unitrefactor PR

Thu Feb 6 12:37:53 PST 2014

Hi everyone,

I'm worried this is going to get out of hand.  I'd like to ask
everyone to take a bit of time to think and reflect and so on.  I know
that I will be taking time to think about this from a couple different
perspectives before I formulate my opinion.  I'm concerned we are
going to very quickly start burning social capital, which concerns me
greatly.

One thing that is certainly going to guide this is something Anthony
Scopatz said on another mailing list yesterday, which I will quote
liberally from:

"I think the short answer is from the style guide, namely "have
extreme empathy for your users."  This is lesson hard learned by not
having such empathy.  [...]  Now, what constitutes extreme empathy &
caution? [...]"

The reason I'm reluctant to say, well, we should merge it, or, well,
we should just keep on going with documenting and keeping the fork
separate, is that we are all users *and* developers, and we should try
to have extreme empathy for people using the code, developing it, and
understanding it.  And I think there are arguments to be made on both
the side of "Merge right away" and "Keep developing and documenting"
for empathy.  And I don't think either is the right one.

But I do know that what I want to think about is in the context of
people that are using yt and in the context of people developing yt.
And there's a large overlap there.  So I would ask that before you
reply to any additional messages in this thread, you give it some time
to settle, think about it from all the perspectives -- the original
users from Enzo, Orion, FLASH, the people who want to use SPH, the
people who want to use Octrees (without changing field names), the
people who have mission critical results that depend on yt, the people
who want to contribute features, and on and on.

So please, I'd like if we could have some contemplation time -- until
tomorrow at least -- before we continue getting upset.

Also, this would be a lot easier if everyone wasn't ... right.  But
the problem is, both sides *are*.

-Matt

On Thu, Feb 6, 2014 at 3:23 PM, Sam Skillman <samskillman at gmail.com> wrote:
> I would agree with Cameron if there was any current non-SPH/units
> development going on in 3.0.
>
> https://bitbucket.org/yt_analysis/yt/pull-requests
> vs
> https://bitbucket.org/MatthewTurk/yt/pull-requests
>
> However, given that right now 3.0 is defacto located in matthewturk/yt, the
> only difference to me between leaving things as they are now and merging is
> that it *easier* for people other than the unit/sph devs to contribute
> fixes/docs/features.
>
> Sam
>
>
>
> On Thu, Feb 6, 2014 at 12:18 PM, Cameron Hummels <chummels at gmail.com> wrote:
>>
>> I agree that having this PR merged into the main code branch will ease
>> some of the difficulties that have arisen in PRs to PRs in individual user's
>> repositories.  That's why I'm +1 on merging it, but I guess I am -1 on
>> merging it immediately since there are outstanding issues.
>>
>> But I'm confused by what's being said here.  Yesterday, Matt said
>> documentation already existed for all of the units stuff in the unit
>> refactor, but Nathan is saying he'll write it this weekend.  I thought the
>> only thing left to be documented was the SPH smoothing stuff, which Matt
>> said he wouldn't do until March.   Is there more to document about the unit
>> refactor and how it works?
>>
>> My main concern is to avoid that which has happened in the past--that code
>> gets introduced and the authors say they'll introduce documentation on it
>> later, which may get postponed for a long period or even indefinitely.  As
>> John rightly points out, this PR changes a great deal of the underlying
>> structure in yt, and it's going to be confusing for both users and
>> developers accustomed to the old way.  I want to make sure that those of us
>> who were not actively involved in the development of the unit refactor can
>> figure out what is going on, so that we can continue to contribute to the
>> code.  Documentation enables this.
>>
>> Lastly, if the docs cannot even be built in the PR, then I would strongly
>> recommend waiting to accept until they can.  Now that the yt-docs repository
>> is deprecated, the yt-3.0 branch is the only location where the current
>> state of the documentation exists.  If we cannot even build it, then we lack
>> up to date docs, and no one will contribute docs on other new featuers
>> because they cannot compile their documentation.
>>
>>
>> On Thu, Feb 6, 2014 at 12:54 PM, Nathan Goldbaum <nathan12343 at gmail.com>
>> wrote:
>>>
>>> On Thu, Feb 6, 2014 at 11:47 AM, John ZuHone <jzuhone at gmail.com> wrote:
>>> > Thinking about this some more...
>>> >
>>> > The unit and field refactoring are very big changes that effect nearly
>>> > every part of the codebase. Since a lot of our documentation relies on
>>> > notebooks, it would seem to me that writing these would be a lot easier if
>>> > we integrated the major parts of this release together first. That way
>>> > issues that crop up and unexpected bugs can be addressed immediately, as
>>> > opposed to writing docs and then finding out later that they need to be
>>> > changed after the big merge is made.
>>> >
>>>
>>> We've been trying to merge with mainline dev as much as possible.  It
>>> doesn't happen immediately because it adds cognitive overhead to
>>> dealing with development, so it ends up happening as a PR every couple
>>> of weeks.
>>>
>>> Merging into the main repo will help ease the burden of periodic
>>> merging and fixing bugs exposed by the merge.
>>>
>>> I think Matt was actually trying to get a docs build to run last
>>> night.  I'm not sure how much work it would be to get the docs into a
>>> state where they can be built in yt-3.0, even if the narrative docs
>>> have not been updated to be consistent with the codebase yet.
>>>
>>> > I suppose the response to that is that we ought to merge from yt-dev
>>> > into the units fork, but this seems to the six of one vs half-dozen of the
>>> > other.
>>> >
>>> > John ZuHone
>>> > Laboratory for High-Energy Astrophysics
>>> > NASA/Goddard Space Flight Center
>>> > 8800 Greenbelt Rd., Mail Code 662
>>> > Greenbelt, MD 20771
>>> > (w) 301-286-2531
>>> > (m) 781-708-5004
>>> > john.zuhone at nasa.gov
>>> > jzuhone at gmail.com
>>> >
>>> >> On Feb 6, 2014, at 2:30 PM, Nathan Goldbaum <nathan12343 at gmail.com>
>>> >> wrote:
>>> >>
>>> >> I can prepare docs for units this weekend.
>>> >>
>>> >> Documentation for SPH smoothing and the new field system will depend
>>> >> on Matt, who in a previous e-mail said he wouldn't have time to work
>>> >> seriously on docs until March.  I don't think "wait until March" makes
>>> >> sense here...
>>> >>
>>> >> -Nathan
>>> >>
>>> >>> On Thu, Feb 6, 2014 at 11:27 AM, Cameron Hummels <chummels at gmail.com>
>>> >>> wrote:
>>> >>> I'm +1 on merging into the 3.0 dev branch, but only after
>>> >>> documentation
>>> >>> exists for the features that are being introduced by this PR.  I know
>>> >>> this
>>> >>> isn't what people want to hear, but I think it will be much easier to
>>> >>> support if people other than the main authors know how it works and
>>> >>> how to
>>> >>> use it.
>>> >>>
>>> >>>
>>> >>>> On Thu, Feb 6, 2014 at 11:52 AM, John ZuHone <jzuhone at gmail.com>
>>> >>>> wrote:
>>> >>>>
>>> >>>> +1 on merging into the 3.0 dev branch.
>>> >>>>
>>> >>>> -1 on separate branch.
>>> >>>>
>>> >>>> On Feb 6, 2014, at 1:45 PM, Sam Skillman <samskillman at gmail.com>
>>> >>>> wrote:
>>> >>>>
>>> >>>> I am a +1 on this for all of the reasons you've stated.  I would be
>>> >>>> much
>>> >>>> more likely to contribute bugfixes/enhancements/docs as well as
>>> >>>> test/report
>>> >>>> issues if it was on the main branch, even if it means the 3.0 branch
>>> >>>> is less
>>> >>>> "stable" than it is currently.  I'd suggest we tag the last
>>> >>>> non-unitrefactor
>>> >>>> commit so that 3.0 users can have a easy way to revert to
>>> >>>> non-unitful yt if
>>> >>>> they way, but besides that I think we should go for it.
>>> >>>>
>>> >>>> I'm -1 on a separate stable/dev branch for 3.0.
>>> >>>>
>>> >>>> Sam
>>> >>>>
>>> >>>>
>>> >>>> On Thu, Feb 6, 2014 at 10:34 AM, Nathan Goldbaum
>>> >>>> <nathan12343 at gmail.com>
>>> >>>> wrote:
>>> >>>>>
>>> >>>>> Hi all,
>>> >>>>>
>>> >>>>> Over the past month or so development has been proceeding apace on
>>> >>>>> the
>>> >>>>> unit refactor in Matt's fork of yt. We did this because Matt
>>> >>>>> initially
>>> >>>>> opened the unitrefactor PR and at that time it was not ready for
>>> >>>>> merging.
>>> >>>>>
>>> >>>>> However, thanks to everyone's hard work, unitrefactor is getting
>>> >>>>> more
>>> >>>>> and more stable and I think it's time to think about merging it
>>> >>>>> into
>>> >>>>> the main repo, even if there are open issues or some remaining bits
>>> >>>>> of
>>> >>>>> functionality that haven't been updated yet.
>>> >>>>>
>>> >>>>> With tons of development going on in Matt's fork, I think we're
>>> >>>>> possibly leaving out people who aren't watching his repo.
>>> >>>>> Additionally, since Matt is the only one who can merge pull
>>> >>>>> requests
>>> >>>>> into his repo, we need to use a lot more of his attention to keep
>>> >>>>> work
>>> >>>>> moving forward.
>>> >>>>>
>>> >>>>> It's true that there is some functionality that still needs to be
>>> >>>>> ported and bugs that need to be fixed. Matt's trello board
>>> >>>>> summarizes
>>> >>>>> most of the issues (BTW, I see a couple missing issues, would you
>>> >>>>> be
>>> >>>>> open to adding more users to it?):
>>> >>>>>
>>> >>>>> https://trello.com/b/yv7o0dTp/unit-refactor
>>> >>>>>
>>> >>>>> One option would be to open a new named branch in the main repo,
>>> >>>>> while
>>> >>>>> still keeping the current 3.0 tip in a 'stable' state. I'm less
>>> >>>>> inclined to go this route because I think unitrefactor is such a
>>> >>>>> big
>>> >>>>> improvement over the current codebase, since many new features have
>>> >>>>> snuck in besides just adding units.
>>> >>>>>
>>> >>>>> Another concern is that there aren't any docs yet.  That's
>>> >>>>> definitely
>>> >>>>> true, but there aren't any docs for the current 3.0 tip either.  In
>>> >>>>> fact, now that there are a set of docs bundled in the repo in the
>>> >>>>> unitrefactor bookmark, merging should improve the documentation
>>> >>>>> effort
>>> >>>>> for 3.0 going forward by making it more straightforward to enforce
>>> >>>>> a
>>> >>>>> rule that things need to be documented before they get merged.
>>> >>>>>
>>> >>>>> What do you all think?
>>> >>>>>
>>> >>>>> -Nathan
>>> >>>>> _______________________________________________
>>> >>>>> yt-dev mailing list
>>> >>>>> yt-dev at lists.spacepope.org
>>> >>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> >>>>
>>> >>>>
>>> >>>> _______________________________________________
>>> >>>> yt-dev mailing list
>>> >>>> yt-dev at lists.spacepope.org
>>> >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> _______________________________________________
>>> >>>> yt-dev mailing list
>>> >>>> yt-dev at lists.spacepope.org
>>> >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Cameron Hummels
>>> >>> Postdoctoral Researcher
>>> >>> Steward Observatory
>>> >>> University of Arizona
>>> >>> http://chummels.org
>>> >>>
>>> >>> _______________________________________________
>>> >>> yt-dev mailing list
>>> >>> yt-dev at lists.spacepope.org
>>> >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> >> _______________________________________________
>>> >> yt-dev mailing list
>>> >> yt-dev at lists.spacepope.org
>>> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> > _______________________________________________
>>> > yt-dev mailing list
>>> > yt-dev at lists.spacepope.org
>>> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>> _______________________________________________
>>> yt-dev mailing list
>>> yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>>
>>
>>
>> --
>> Cameron Hummels
>> Postdoctoral Researcher
>> Steward Observatory
>> University of Arizona
>> http://chummels.org
>>
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>