[yt-dev] proposed change to development process

Thu Sep 10 12:18:23 PDT 2015

In general, I like Britton's idea as it fosters more group development as
opposed to the problems of forks of forks that many of us experience in
trying to develop unmerged features like VR-refactor and ytdata frontend.

That said, there are some issues to be aware of.  By default, I think when
you pull from the repository, hg defaults to updating to the most recent
commit (ie the tip), which may not be in your active bookmark.  This means
that if someone just committed to the "ytdata frontend" head/bookmark, and
I pull from the main repository expecting to get the main "development"
bookmark, I'm going to be put on the wrong version of the code.  This can
be confusing unless we educate everyone to always make sure after they pull
to update to the "development" bookmark or whatever.

I'm dubious of the idea of merging more work-in-progress code into the main
head of the dev branch as Nathan suggests.  As he points out, the success
of this model is entirely dependent on the author/community "finishing"
code that goes into the dev branch before it gets tagged as a stable
release.  All this does is simply kick the can down the road as to when
code gets "finished" (i.e. documented, tested, polished, the parts no one
wants to do).  By delaying this process until there is a deadline (the
stable release) it separates it from when the main development went on, and
makes it less likely for authors to remember what was going on in the
code.  In the past, this results in code kinda getting pushed through, just
to not hold up the release any longer, and it ends up being unnecessarily
acrimonious.  I'm in favor or the separate heads/bookmarks model that
Britton suggests, but not merging those heads into the main dev bookmark
until it is "finished".

Cameron

On Thu, Sep 10, 2015 at 12:01 PM, Nathan Goldbaum <nathan12343 at gmail.com>
wrote:

> Just a first initial reaction: I don't think introducing even more heads
> to the yet branch is a good idea, even if they are marked by bookmarks. In
> fact I'd prefer to only have one head on the yt branch right now.
>
> In addition, bitbucket's support for bookmark's in the pull request
> interface amounts only to labeling entries in the source and destination
> dropdown menu, meaning that once a pull request is merged, someone would
> need to manually update the corresponding bookmark to the correct head.
> This is both error prone and easy to skip, so I predict that if we *did*
> start doing this, bookmarks would wuickly go out of sync with heads,
> leaving a bunch of un-bookmarked heads.
>
> Was this prompted by your work on the reloadable datasets PR? Without
> something like this do you expect it to sit in limbo for a long time?
>
> I think the solution we need is to jot be afraid to pull in more
> "experimental" work-in-progress stuff to the yt branch. Of course that
> would need to come with assurances from the author or community
> members that they're committed to "finishing" the code before the next
> major release. That way people will be able to test new code more easily
> (they'll get it without updating explicitly to it) and we'll catch
> breakages to unrelated code and wordflows more quickly.
>
> Nathan
>
>
> On Thursday, September 10, 2015, Britton Smith <brittonsmith at gmail.com>
> wrote:
>
>> Hi everyone,
>>
>> I had some ideas for improving the yt development process that I
>> wanted to run by everyone.  This can be discussed further at our
>> upcoming team meeting and if people are in favor, I will issue a pull
>> request to the relevant YTEP.
>>
>> STATEMENT OF PROBLEM
>> Currently, development proceeds roughly as follows.  The two main
>> active branches within the central yt repository are *yt* and *stable*.
>> The tip of *stable* is the latest release and the *yt* branch is the de
>> facto "development version" of the code.  Until recently, we have not
>> been very good at regularly scheduled minor releases and so the *stable*
>> branch sits for quite some time with many bugs that are fixed within
>> the development branch.  This effectively makes *stable* unusable and
>> pushes most users to the *yt* branch.
>>
>> When new features are developed, pull requests are issued to the
>> single head of the *yt* branch.  Because this is the version most people
>> are actually using, the current policy is to not allow PR with new
>> functionality to be accepted until they are 100% ready (full
>> functionality, tests, docs, etc).  As we have already seen, this makes
>> collaborative development very cumbersome, as it requires people to
>> create forks of the fork from which the PR originates.  They then must
>> issue PRs to that fork after which time the original PR is updated.
>> The current volume render refactor is the perfect example of this.
>>
>> PROPOSED SOLUTION
>> Before I lay out the proposed solution, I want to list a number of
>> recent developments that I think will make this possible:
>> 1. Nathan's new script for backporting changes now keeps *stable* and
>> *yt*
>>    synced on bugfixes.
>> 2. We have returned to doing minor releases containing only bugfixes,
>>    thanks again to Nathan's hard work.  This and point 1 means that
>>    users are once again safe to be on *stable*, and now *should* be there
>>    most of the time.
>> 3. Bitbucket now supports bookmarks, meaning that PRs can be issued to
>>    specific bookmarks instead of to branches or heads named only by the
>>    changeset hash.
>> 4. The weekly PR triage hangouts are making it easier to process PRs
>>    and also providing a place to strategize getting larger PRs
>>    accepted.  Thanks to Hilary for keeping this going.
>>
>> With the above in mind, I propose the following:
>> 1. Create a "development" bookmark to sit at the tip of the *yt*
>>    branch.  All PRs containing relatively small new features are
>>    issued to this.  The requirements for acceptance remain the same:
>>    PRs accepted to "development" must contain all intended
>>    functionality and be fully documented.  This allows the
>>    "development" bookmark to be defined explicitly as everything that
>>    will be included in the next major release.
>> 2. Large new features should have a corresponding YTEP that has been
>>    accepted.  After the YTEP has been accepted, a PR should be issued
>>    to the *yt* branch.  After some initial discussion, this PR is pulled
>>    into the main yt repo with a bookmark named after the feature.
>>    Once this has happened, developers can now issue new PRs
>>    specifically to this bookmark.  This is effectively what we have
>>    now with the volume render work in the "experimental" bookmark,
>>    only we would rename the bookmark to something like "vr-refactor".
>>    As with PRs issued directly to "development", only after the new
>>    feature is 100% ready shall it be merged into the "development"
>>    bookmark.
>> 3. We continue to make use of the PR triage hangouts to establish when
>>    large features are ready to be merged.
>>
>> I believe this will have the following benefits:
>> 1. Large, new features can be developed collaboratively without the
>>    need for forks of forks of forks.
>> 2. New, underdevelopment features are more accessible to the larger
>>    community by simply updating to named bookmarks from the main repo
>>    (no need for "just pull these changes from my fork").
>> 3. The "development" branch is preserved as a place only for
>>    ready-to-be-released features (i.e., polished and documented).
>>
>>
>> All told, this is really just a small tweak on our current process.
>> Please comment with any thoughts, or even a +/-1 if your feelings can
>> be summed up thusly.
>>
>> Britton
>>
>
> _______________________________________________
> yt-dev mailing list
> yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>
>

-- 
Cameron Hummels
NSF Postdoctoral Fellow
Department of Astronomy
California Institute of Technology
http://chummels.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-dev-spacepope.org/attachments/20150910/1cf41cfc/attachment.html>