Establish a Beta Release

[jginsburgn]

Having a beta release (or better said: prerelease) will let us incrementally produce a major (with breaking changes), better tested, more significant stable release. PRs tagged with breaking should be filed against the beta branch.

[jginsburgn]

RFC
@XhmikosR
@mtrea
@devoto13
@nicojs
@sgravrock

[devoto13]

How do you envision it to work?

My thinking is that with the limited maintainer time and given the small amount of breaking changes continuously maintaining multiple branches and backporting all changes is not worth the effort. So I would suggest the below workflow.

master branch contains the latest version of the library as it is today. Most of the time it is the latest stable version of the library, so PRs sent to it should be non-breaking and will be immediately released as stable. Periodically, we would switch the master branch into a prerelease mode (by changing the Semantic Release config). In this mode, we can merge both breaking and non-breaking PRs and they will be released immediately as next/pre-release versions on NPM so that users can install and test them. Once we've merged all breaking changes, we switch the master branch back into stable mode and Semantic Release will cut a single major release with all breaking changes.

Whenever we switch the master branch into the pre-release mode, we'll also create release/N.x (where N is the current major number) and protect it. We can send PRs against this branch to backport critical fixes to the stable version or contributors can open PRs against it in the future. The branch is also automatically released by the Semantic Release.

[jginsburgn]

I was thinking about following the pre-releases recipe from semantic release.

• The master branch would always track the @latest "stable release".
• We create a protected beta branch to track @beta "prereleases".
• All breaking PRs should be filed against beta and each merge triggers a new beta prerelease (e.g. v5.0.0-beta.0).
• When deemed necessary we promote beta to stable:
  • We merge beta to master, releasing a new major stable version.
  • The old major stable version goes out of support.
  • When necessary beta can diverge from master again.

[jginsburgn]

Sibling bug: karma-runner/karma#3774

[jginsburgn]

One question that is unclear to me still:

Will each breaking commit (indicated by the "BREAKING CHANGE:" footer) trigger a new major pre-release in the beta branch?

For example, let's say that the latest stable version currently is v4.0.2. We diverge from it via the beta branch with a PR that contains a breaking commit. This should immediately release version v5.0.0-beta.0. Let's say there is another PR with a breaking commit merged onto the beta branch. Will that trigger the release of version v5.0.0-beta.1 or v6.0.0-beta.0? It is not immediately clear from the pre-releases recipe. I would need to find out via experimentation. I think the right behavior should be that it triggers the release of v5.0.0-beta.1 as beta is considered unstable and it is permissible to introduce new breaking changes upon every PR merge.

WDYT @devoto13 ?

[devoto13]

Will each breaking commit (indicated by the "BREAKING CHANGE:" footer) trigger a new major pre-release in the beta branch?

I think it will make v5.0.0-beta.1, but better make sure through experimentation as you suggested.

Otherwise, the proposed approach sounds good to me!

[jginsburgn]

This should be the same across all the other repos that need a beta release. Do you agree @devoto13 ?

[jginsburgn]

@karma-runner/google-web-test-team

[mtrea]

It sounds like @jginsburgn's clarifications addressed @devoto13's concerns re:limited maintainer time, continuously maintaining multiple branches and backporting all changes.

I would like to nail down the positive/negative impacts of this change a bit more. Positive is that we mitigate breaking changes reaching stable. Negative is some increased complexity in the project; will there also be a tax on contributors and maintainers once this is set up? Was there some incident that inspired filing this issue, or are you just being proactive?

When you talk about "breaking changes", do you mean our CI still passes but downstream users may be broken e.g. due to API migrations/etc? It seems you're referring to intentionally breaking changes, which contrasts with unintentionally breaking changes which should be caught by CI.

[devoto13]

@mtrea

will there also be a tax on contributors and maintainers once this is set up?

I don't think contributors are affected in any way. Slight extra effort required from the maintainers to backport occasional fixes to another branch, but IMO that shouldn't be significant.

Was there some incident that inspired filing this issue, or are you just being proactive?

We use semantic-release to automatically publish each merged PR as a new version. When merging breaking changes, it will automatically bump a major version. The primary problem we're solving here is that we don't want every breaking change to result in a new major release (which quickly gets out of control and results in noise and update friction), but instead have a way to batch several breaking changes into a single major release.

We would still use semantic-release to automatically publish each merged PR, but from the beta branch it would result in 6.0.0-beta.0, 6.0.0-beta-1, 6.0.0-beta.2 (and eventually 6.0.0 containing all previous changes) instead of 6.0.0, 7.0.0, 8.0.0, etc.

When you talk about "breaking changes", do you mean our CI still passes but downstream users may be broken e.g. due to API migrations/etc? It seems you're referring to intentionally breaking changes, which contrasts with unintentionally breaking changes which should be caught by CI.

Yes, it's intential breaking changes. E.g. drop support for the obsolete Node version, remove deprecated or obsolete APIs, etc. As I understand these should not be breaking changes for Google as all usages should be cleaned up prior to landing the change, but for external user it goes other way around and we should communicate such changes with major version bump.

[jginsburgn]

Plus one to what @devoto13 responded. Nothing to add.

Do you have any other concerns here @mtrea ?

[mtrea]

Nope, go for it!

[nicojs]

I'd be happy to test beta releases out in our StrykerJS e2e test suite. Especially if it saves us having to fix breaking issues for users later on 😅

Q: How will these beta releases be communicated?

[jginsburgn]

I'd be happy to test beta releases out in our StrykerJS e2e test suite. Especially if it saves us having to fix breaking issues for users later on 😅

Q: How will these beta releases be communicated?

@nicojs that is a great idea! I think it would make our release process safer.

I haven't thought about how to decide when to merge the beta branch discussed above. A suggestion would be to stick to the Node.js lifecycle. WDYT?

In parallel, however, I think we could add the Stryker e2e test suite as an extension of our integration tests repo and run it as a step in every PR.

Tell me your thoughts.