Automating point releases

[thomaseizinger]

I am transferring some ideas from #2902 over to this discussion because I am hoping that we can get some more structure into it as a result. Plus, many things changed since we started that thread so it is time to consolidate that knowledge into a new place.

Overall, I think it is really important that we minimize breaking changes as much as possible. I think we've been doing a good job on that front and we should continue that way. By postponing breaking changes, we've effectively cut many more point releases, delivering features to our users earlier that previously.

I think this is a good direction and I'd like to propose that - as a next step - we automate those releases, with the end goal that they happen in a fully automated way every week or so.

There are a number of problems to solve and I am going to create a separate thread for each of them below here.

[thomaseizinger]

Releasing from master automatically

It is important that releases happen on the master branch to ensure that the generated git-tag is in the mainline history. As such, any release automation can only run on the master branch.

What we have to avoid is a divergence of a "release PR" after it has been branched off from master and when it gets merged. Thus, I think it would be easiest if we try and get rid of "release PR"s entirely and instead try to create a workflow where master is always in a state that can be released.

That way, we can simply run a release workflow on a cron trigger or invoke it manually using a workflow_dispatch event. This release job would then simplify invoke a predefined cargo release command that releases all crates in the workspace.

[thomaseizinger]

That way, we can simply run a release workflow on a cron trigger or invoke it manually using a workflow_dispatch event. This release job would then simplify invoke a predefined cargo release command that releases all crates in the workspace.

A good simplification would actually be to not use a cron trigger to start with. That frees us from building all sorts of validation checks to not run this at the wrong point in time. Having to only click a single button is already very convenient and can in fact then also be used to make minor releases.

[thomaseizinger]

The next thing we need is a script that runs the release when we trigger the workflow. Thanks to cargo release, that should be pretty simple!

[thomaseizinger]

Changelog entries

Assuming that we want master to be in an always release-able state, we have two options regarding changelog entries:

1. Drop the - unreleased suffix that we are currently adding to the not yet released version:

This would mean the final git tag is "clean" in that when checking out the released tag, there is no notion of anything being "unreleased" in the changelog.

2. Remove the - unreleased suffix in a PR after the release has happened:

This leaves the git tag in a kind of dirty state. Plus, there might be a divergence between when this PR gets opened and merged. We'd have to be careful that no PR in-between adds an entry to the unreleased section.

3. Drop the - unreleased suffix in a PR before the release:

This is essentially what we are doing now but has the same problem as (2). There can be a divergence between when that PR gets opened and when it gets merged (which would then trigger the release?). Worse than (2), this might allow features to slip into a release that were not intended to be released yet.

---

(2) and (3) have some kind of coordination cost which I'd like to avoid as much as possible. Thus, I think it makes the most sense if we push for (1).

(1) has the downside that it is not clear whether the top-most version in the changelog is an unreleased version or not. The same applies to the crate's version. There are two problems / possible errors we need to solve:

a. Detecting that the crate needs a version bump: This should be fairly easy as a CI check by checking if a PR contains and code change within a certain directory and diffing the crate version against what is on crates.io.
b. Detecting that we are modifying an old changelog entry.

(b) is not super important but would be good to deal with somehow. Perhaps, to get a solution off the ground, we could ignore that for now and spot-check it manually in PRs? If we have the above CI check regarding a crate-version bump, then we could probably tie it to that? If that CI check fails, it means we need to add a new entry. If that CI check passes, the existing top-entry can be modified.

[thomaseizinger]

a. Detecting that the crate needs a version bump: This should be fairly easy as a CI check by checking if a PR contains and code change within a certain directory and diffing the crate version against what is on crates.io.

We currently actually have the same issue and it is already captured :)

#4126

[thomaseizinger]

#4620 implements several lints that ensure Cargo.toml and CHANGELOG.md are updated when changes are made to a certain crate.

[thomaseizinger]

Dealing with breaking changes

Once our master is "dirty", we should definitely prohibit a scheduled release. This should be rather easy by adding a check to the triggered release job that all about to be done releases are only point releases and it abort otherwise.

[thomaseizinger]

Previously, we've bumped the minor versions in the PRs that made the necessary changes. I think we should move away from that in favor of a dedicated PR that "opens" a merge window for breaking changes. That PR by itself wouldn't make any changes to code other than bump the minor versions of all publicly exposed crates (i.e. all libp2p-* crates).

Once this PR is merged, the scheduled release process would "sleep" (we could also turn it off as part of this PR if "auto-detecting" is too hard). Merging this PR will open the merge window for the milestone collecting all the breaking changes. Once the release is done, we'd activate the automated releases again and a new cycle begins.