[yt-dev] proposed change to development process

[Nathan Goldbaum]

On Thu, Sep 10, 2015 at 2:45 PM, Britton Smith <brittonsmith at gmail.com>
wrote:

> Hi Nathan,
>
> That is a good point about Bitbucket's bookmark support.  As I see it, a
> feature-named bookmark would only exist until the point at which the
> feature is merged.  We would immediately remove it after that.
>

I'm not sure if my main concern came across. Someone would need to manually
update the bookmarks after every single pull request is merged. I don't
think that's sustainable in the long run and we will quickly get into
states where the tip of a head doesn't have a bookmark on it.

How long the bookmark stays in the repository is a side issue.


>
> This idea came mainly after thinking about the volume render refactor and
> what we might do if we had two such features being worked on concurrently.
> My feeling was that the name "experimental" wasn't very descriptive and
> also allowed for only one such effort to exist.  As it stands, I would not
> consider my reloadable datasets PR to be needing of this type of action as
> it's really only me that's working on it.  I think what I proposed should
> only be used when we have a clear situation of something that multiple
> people need to work on together.  So far, I can only think of two instances
> where this should have been used: 1) the current vr refactor and 2) yt-3.0
> and the units work.  This should really be just a last resort.
>
> I would disagree with your last point about easing restrictions on pulling
> in experimental work.  If anything, I think our history has shown that
> simply making a commitment to eventually getting docs done is not a good
> idea.  What's more, by merging multiple new features into a single branch,
> we end up with a situation where a single undocumented feature can hold up
> a release because it is tangled up with everything else that's ready to
> go.  I don't mean to say people are deceptive or break promises, only that
> life just gets in the way of plans far too often.
>
> I do think your concern of having too many heads in yt is very reasonable
> and we should use something like this only when absolutely necessary.
>
> Britton
>
> On Thu, Sep 10, 2015 at 3:01 PM, Nathan Goldbaum <nathan12343 at gmail.com>
> wrote:
>
>> Just a first initial reaction: I don't think introducing even more heads
>> to the yet branch is a good idea, even if they are marked by bookmarks. In
>> fact I'd prefer to only have one head on the yt branch right now.
>>
>> In addition, bitbucket's support for bookmark's in the pull request
>> interface amounts only to labeling entries in the source and destination
>> dropdown menu, meaning that once a pull request is merged, someone would
>> need to manually update the corresponding bookmark to the correct head.
>> This is both error prone and easy to skip, so I predict that if we *did*
>> start doing this, bookmarks would wuickly go out of sync with heads,
>> leaving a bunch of un-bookmarked heads.
>>
>> Was this prompted by your work on the reloadable datasets PR? Without
>> something like this do you expect it to sit in limbo for a long time?
>>
>> I think the solution we need is to jot be afraid to pull in more
>> "experimental" work-in-progress stuff to the yt branch. Of course that
>> would need to come with assurances from the author or community
>> members that they're committed to "finishing" the code before the next
>> major release. That way people will be able to test new code more easily
>> (they'll get it without updating explicitly to it) and we'll catch
>> breakages to unrelated code and wordflows more quickly.
>>
>> Nathan
>>
>>
>> On Thursday, September 10, 2015, Britton Smith <brittonsmith at gmail.com>
>> wrote:
>>
>>> Hi everyone,
>>>
>>> I had some ideas for improving the yt development process that I
>>> wanted to run by everyone.  This can be discussed further at our
>>> upcoming team meeting and if people are in favor, I will issue a pull
>>> request to the relevant YTEP.
>>>
>>> STATEMENT OF PROBLEM
>>> Currently, development proceeds roughly as follows.  The two main
>>> active branches within the central yt repository are *yt* and *stable*.
>>> The tip of *stable* is the latest release and the *yt* branch is the de
>>> facto "development version" of the code.  Until recently, we have not
>>> been very good at regularly scheduled minor releases and so the *stable*
>>> branch sits for quite some time with many bugs that are fixed within
>>> the development branch.  This effectively makes *stable* unusable and
>>> pushes most users to the *yt* branch.
>>>
>>> When new features are developed, pull requests are issued to the
>>> single head of the *yt* branch.  Because this is the version most people
>>> are actually using, the current policy is to not allow PR with new
>>> functionality to be accepted until they are 100% ready (full
>>> functionality, tests, docs, etc).  As we have already seen, this makes
>>> collaborative development very cumbersome, as it requires people to
>>> create forks of the fork from which the PR originates.  They then must
>>> issue PRs to that fork after which time the original PR is updated.
>>> The current volume render refactor is the perfect example of this.
>>>
>>> PROPOSED SOLUTION
>>> Before I lay out the proposed solution, I want to list a number of
>>> recent developments that I think will make this possible:
>>> 1. Nathan's new script for backporting changes now keeps *stable* and
>>> *yt*
>>>    synced on bugfixes.
>>> 2. We have returned to doing minor releases containing only bugfixes,
>>>    thanks again to Nathan's hard work.  This and point 1 means that
>>>    users are once again safe to be on *stable*, and now *should* be
>>> there
>>>    most of the time.
>>> 3. Bitbucket now supports bookmarks, meaning that PRs can be issued to
>>>    specific bookmarks instead of to branches or heads named only by the
>>>    changeset hash.
>>> 4. The weekly PR triage hangouts are making it easier to process PRs
>>>    and also providing a place to strategize getting larger PRs
>>>    accepted.  Thanks to Hilary for keeping this going.
>>>
>>> With the above in mind, I propose the following:
>>> 1. Create a "development" bookmark to sit at the tip of the *yt*
>>>    branch.  All PRs containing relatively small new features are
>>>    issued to this.  The requirements for acceptance remain the same:
>>>    PRs accepted to "development" must contain all intended
>>>    functionality and be fully documented.  This allows the
>>>    "development" bookmark to be defined explicitly as everything that
>>>    will be included in the next major release.
>>> 2. Large new features should have a corresponding YTEP that has been
>>>    accepted.  After the YTEP has been accepted, a PR should be issued
>>>    to the *yt* branch.  After some initial discussion, this PR is pulled
>>>    into the main yt repo with a bookmark named after the feature.
>>>    Once this has happened, developers can now issue new PRs
>>>    specifically to this bookmark.  This is effectively what we have
>>>    now with the volume render work in the "experimental" bookmark,
>>>    only we would rename the bookmark to something like "vr-refactor".
>>>    As with PRs issued directly to "development", only after the new
>>>    feature is 100% ready shall it be merged into the "development"
>>>    bookmark.
>>> 3. We continue to make use of the PR triage hangouts to establish when
>>>    large features are ready to be merged.
>>>
>>> I believe this will have the following benefits:
>>> 1. Large, new features can be developed collaboratively without the
>>>    need for forks of forks of forks.
>>> 2. New, underdevelopment features are more accessible to the larger
>>>    community by simply updating to named bookmarks from the main repo
>>>    (no need for "just pull these changes from my fork").
>>> 3. The "development" branch is preserved as a place only for
>>>    ready-to-be-released features (i.e., polished and documented).
>>>
>>>
>>> All told, this is really just a small tweak on our current process.
>>> Please comment with any thoughts, or even a +/-1 if your feelings can
>>> be summed up thusly.
>>>
>>> Britton
>>>
>>
>> _______________________________________________
>> yt-dev mailing list
>> yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>>
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20150910/347555d0/attachment.htm>