Release procedure notes

[rvagg]

This is for discussion with @nodejs/release and intended as a first step in updating https://github.com/nodejs/node/blob/master/doc/releases.md to be better reflective of current practice.

Given that myself and @Fishrock123 have been cutting the majority of recent releases, certainly the only ones for io.js v3 and Node.js v4 and that we both would like to see additional members of @nodejs/release be empowered to cut releases and actually start doing so, I wanted to document some of the recent procedures I've been following to make a release happen so that we can update our docs and land on a shared agreement of how it should be done. So here it is!

1. Cherry-picking from master (or other)

This is something we should be doing often but it probably should be limited to a subset of maintainers, I would suggest that the @nodejs/release team be responsible for this with @nodejs/lts guiding how the LTS branches are maintained. In practice I've been doing most of this since we switched to the master-as-primary model, Jeremiah has also done some along the way. We don't have nightlies re-enabled yet so it's not as vital but it will become important that we do it very regularly so that our nightlies on stable branches have changes to build and users can start testing them in between releases.

I have a tool to help me with this, branch-diff that is like a git log b1..b2 but uses our metadata to make proper comparisons. When I run branch-diff v4 master I get a similar output to changelog-maker (because they share some code) showing me the commits in master that are not in v4, calculated by comparing the commit summary (first line) and the PR-URL. So far this has worked really well and it'll also show the semver tags we attach so the semver-minor and semver-major commits are obvious. The only drawback is that when PR-URL is left off accidentally, which it often is, the commit will show up because it's unsure if it's a duplicate or not.

For example, running this right now shows 47 commits, some of which are meant to be cherry-picked, some of which are semver-major and some of which are missing a PR-URL, here's a snippet:

* [[`342c3a1bbb`](https://github.com/nodejs/node/commit/342c3a1bbb)] - **test**: remove arguments.callee usage (Roman Reiss) [#3167](https://github.com/nodejs/node/pull/3167)
* [[`6fff47ffac`](https://github.com/nodejs/node/commit/6fff47ffac)] - **deps**: backport 03ef3cd from V8 upstream (Ali Ijaz Sheikh) [#3165](https://github.com/nodejs/node/pull/3165)
* [[`2e6ece44e1`](https://github.com/nodejs/node/commit/2e6ece44e1)] - **(SEMVER-MINOR)** **node**: add -c|--check CLI arg to syntax check script (Dave Eddy) [#2411](https://github.com/nodejs/node/pull/2411)
* [[`5bbc6df7de`](https://github.com/nodejs/node/commit/5bbc6df7de)] - **test**: change call to deprecated util.isError() (Rich Trott) [#3084](https://github.com/nodejs/node/pull/3084)
* [[`87e820ead5`](https://github.com/nodejs/node/commit/87e820ead5)] - **(SEMVER-MINOR)** **fs**: include filename in watch errors (charlierudolph) [#2748](https://github.com/nodejs/node/pull/2748)
* [[`30b8bb0aab`](https://github.com/nodejs/node/commit/30b8bb0aab)] - **(SEMVER-MINOR)** **src**: add process.versions.icu (Evan Lucas) [#3102](https://github.com/nodejs/node/pull/3102)
* [[`680dda8023`](https://github.com/nodejs/node/commit/680dda8023)] - **dns**: remove nonexistant exports.ADNAME (Roman Reiss) [#3051](https://github.com/nodejs/node/pull/3051)
* [[`6192c9892f`](https://github.com/nodejs/node/commit/6192c9892f)] - **(SEMVER-MAJOR)** **http**: add checkIsHttpToken check for header fields (James M Snell) [#2526](https://github.com/nodejs/node/pull/2526)
* [[`4a6f1feecb`](https://github.com/nodejs/node/commit/4a6f1feecb)] - 2015-09-22, Version 4.1.1 (Stable) Release (Rod Vagg) [#2995](https://github.com/nodejs/node/pull/2995)
* [[`5f6579d366`](https://github.com/nodejs/node/commit/5f6579d366)] - **(SEMVER-MAJOR)** **buffer**: remove raw & raws encoding (Sakthipriyan Vairamani) [#2859](https://github.com/nodejs/node/pull/2859)
* [[`b9813641dc`](https://github.com/nodejs/node/commit/b9813641dc)] - 2015-09-15 io.js v3.3.1 Release (Rod Vagg) [#2698](https://github.com/nodejs/node/pull/2698)
* [[`380a3d89c3`](https://github.com/nodejs/node/commit/380a3d89c3)] - 2015-09-08, Version 4.0.0 (Stable) Release (Rod Vagg) [#2742](https://github.com/nodejs/node/pull/2742)
* [[`cdfa271164`](https://github.com/nodejs/node/commit/cdfa271164)] - **doc**: update AUTHORS list (Rod Vagg)
* [[`42a8a0a53e`](https://github.com/nodejs/node/commit/42a8a0a53e)] - Working on v5.0.0 (Rod Vagg)

(you can also use --simple to make it more console friendly but I like having the full URLs that I can click on in my console).

In this case, the release commits and version bump (5.0.0) don't get cherry-picked, there's a few that don't have PR-URL like **doc**: update AUTHORS list which was my fault, and there's also dns: remove nonexistant exports.ADNAME which has the wrong PR-URL and should be showing as semver-major (I had to force push this out of the v4.x branch when someone pointed out that I had accidentally brought it over prior to 4.1.2—fwiw I consider force pushing less of a crime on these release branches when you're doing this kind of work because there's so few of us touching those branches).

2. Propose a release

Release proposal timing is largely based on how long it's been since we've had a release and the kind of commit-debt we have built up. We have a few objectives to meet wrt release cadence IMO:

1. Release with regularity to keep up the feedback loop, signal to the ecosystem that there is a lot of energy in core and a desire to keep it active
2. Minimise the number of commits that remain unreleased, I see this as a special form of technical-debt (which is why I'm calling it commit-debt) where the longer we hold on to commits the greater the delta for users and the longer we have before we get feedback on how those commits meet the needs of our user-base. Our test suite is large but it's so far from covering all of the novel ways that Node is used that having code in the hands of users is unfortunately one of the aspects of our testing procedures, so best give it to them in increments rather than dumping it in very large chunks.
3. Don't release too often lest it be a signal of instability in core, we absolutely need to retain and build trust in our users and making them have to upgrade too often has the risk of decreasing that trust because it feels like a moving target or perhaps we have enough flaws in our releases that it becomes objectively unstable for subsets of users. This is a tough one to balance, mainly because it's a soft measure and the fact that we also have an LTS line now that should be where people go if they want deep stability.

We seem to have carried over the approximate once-per-week cadence that we had for much of io.js (or aimed for anyway) and for v4 we are averaging just under once every 7 days. My personal comfort level would be somewhere around once every 1.5 weeks so it's not a relentless "patch Tuesday" type deal.

Also, we don't release at the end of the week. Based on strong feedback (a.k.a. criticism), users don't want releases on a Friday, so if it gets too late in the week then punt till Monday.

Release proposals should live for at least 24 hours, in practice they live for 48+ hours before resulting in tagging. Get a proposal up even when you have a vague idea of timing, best to start the process on GitHub early than rushing it through to meet a deadline.

Proposing a release is a matter of making a pull request now, using the following steps:

1. Make a new branch, with a name such as v4.1.2-proposal
2. Edit src/node_version.h to set the version number right and set NODE_VERSION_IS_RELEASE to 0
3. Run changelog-maker --group to get the commits list of the changelog, use --simple if you want to view it on your console without markdown and use --start-ref=vX.Y.Z to begin the commits from the last release if there has been a tag created since the last release on that branch (i.e. the start point is uses is calculated by grabbing the last tag in the repo and assuming it's the last release but this often not the case, particularly when we have multiple branches cutting releases). If you use --start-ref note that you'll also get the release commit and the Working on X.Y.Z+1 commit which you should not include in the changelog—they are removed automatically when you let changelog-maker decide the start-ref for you.
4. Edit CHANGELOG.md to put in the commits list, copy the Known Issues section and edit accordingly, put in a skeleton Notable Changes section that you can work on now or later and put a release header copying the format used by other recent releases.
5. Commit and push to _this repository_, not your own fork—this is so that the release could be taken over by another releaser for whatever reason (this has had to happen a couple of times in the past with @Fishrock123 and I handing them off to each other).
6. Create a PR from that branch and put some or all of the CHANGELOG modifications in the OP so that collaborators can see easily what's going on. Over the life of one of these PRs I tend to upgrade the OP with the latest information as I re-run changelog-maker after new commits and edit the Notable Items.
7. Hunt down contributors for more details on the changes if needed—I tend to do this for libuv changes and other areas that appear to be notable but where I don't have enough background or information at hand to make a good judgement on it. I also like to @-mention people who's changes I'm including in Notable Items if there's any uncertainty or I'd just like a sanity-check.

3. Test and maintain PR

Now is the time for testing. Use CI on the release branch, as many times as necessary to have confidence that all's well with that code. Continue to do this as more commits are cherry-picked into this branch.

Perform some smoke-testing. We have citgm for this, it's not been quite ready when I've tried to use it extensively for this and often fall-back to manually testing a bunch of ecosystem modules. We also have a job in Jenkins called smoker that uses citgm to run across a bunch of these modules on different platforms but it's not quite ready yet either. The trickiest bit here is doing cross-platform checks which is where Jenkins + citgm is supposed to make life so much better, we'll get there but as a releaser you should probably be contributing to making citgm even better! Remember that node-gyp and npm both take a --nodedir flag to point to your local repository so that you can test unreleased versions without needing node-gyp to download headers for you.

Continually update the PR, edit the OP with new commits, amend the release commit as new changes make it in, etc.

4. Build

When the time is right and the code is stable, make sure your release commit is ready. Check out previous release commits to make sure it looks right (e.g. git show v4.1.2 to see an example). The commit itself should include a summary of predictable format, such as 2015-10-05, Version 4.1.2 (Stable) Release and the Notable Items in the commit description. I usually take the rendered HTML version of the Notable Items and paste it in then reformat to have the right line-length and indenting for the commit message.

**Do not tag the commit until you have successful builds*.

Build using the iojs+release job (yes it needs renaming still ..) by pointing at your HEAD commit (the release commit). This can involve some pain and yak shaving:

1. If there are errors on any of them you should start again, however also note that you could collect binaries from multiple runs, you don't strictly need an all-green build to cut a release, just one green for each slave for the same commit you're building against. Note that you can cancel Jenkins jobs, the "X" button does that.
2. If you have an error on Windows and need to start again, be aware that you'll get immediate failure unless you wait up to 2 minutes for the linker to stop from previous jobs. i.e. if a build fails after having started compiling, that slave will still have a linker process that's running for another couple of minutes which will prevent Jenkins from clearing the workspace to start a new one. This isn't a big deal, it's just a hassle because it'll result in another failed build if you start again!
3. ARMv7 takes the longest to compile. Unfortunately ccache isn't as effective on release builds, I think it's because of the additional macro settings that go in to a release build that nullify previous builds. Also most of the release build machines are separate to the test build machines so they don't get any benefit from ongoing compiles between releases. You can expect 1.5 hours for the ARMv7 builder to complete and you should normally wait for this to finish. It is possible to rush a release out if you want and add additional builds later but we normally provide ARMv7 from initial promotion.
4. Don't wait for ARMv6 / Raspberry Pi, it can take 6h+, just don't forget that it's compiling and needs promotion later on (tho someone will likely remind you if you do forget).

Each of the slaves doing the builds will push their tarballs/packages to the webserver in that staging directory when they are done, they will wait there until you manually promote. Multiple runs of the Jenkins build job with the same parameters will result in the binaries going into the same place, so it's not a one-shot deal.

5. Test your builds

Jenkins collects the artifacts from the builds so you can go into the slaves for your build and grab tarballs and packages to test out the new build to make sure it's what you wanted (make sure version number is right, do some basic checks to confirm that it's all good before moving forward).

6. Tag

Only one you've got successful builds that you're happy with should you tag. This is so that you can bail on a release if something goes wrong at the last minute and more commits/changes are needed to the code. One tagged and the tag is pushed to the repo you _should not_ delete and re-tag. If you make a mistake after tagging then you'll have to version-bump and start again and count that tag/version as lost.

Tag summaries have a predictable format, look at a recent tag to see, git tag -v v4.1.2. The message should look something like 2015-10-05 Node.js v4.1.2 Release.

Tag using something like this: git tag v4.1.2 -sm '2015-10-05 Node.js v4.1.2 Release'. The tag must be signed using the GPG key that's listed for you on the project README, that's where the -s comes in.

Push the tag to the repo before you promote the builds. Part of the promotion process involves some code scraping code from the repo by using the tag for the version you are promoting in order to populate index.json and index.tab. If you haven't pushed your tag before promoting then this won't work properly.

7. Update the repo

This means two things:

1. Add a new commit after your release commit where you change src/node_verisions.h to set NODE_VERSION_IS_RELEASE back to 0 and increment the NODE_PATCH_VERSION value. Then commit with a predictable summary, the format we've been using is something like this: Working on v4.1.3). This new commit goes to the branch you released on.
2. Merge your release branch back into the stable branch you are releasing from.
3. Cherry-pick the release commit to master so it has your CHANGELOG information (the website points to this but that's just one reason to do this. After cherry-picking, edit src/node_version.h and make sure it's what it was on master, i.e. don't bring any changes to that file from the release branch, leave it as the master version number.

8. Promote

Use tools/release.sh to promote your builds. This logs in to the server and runs the promotion process for you, you'll be prompted for a y or n when it finds something to promote, then when it's done it'll make a SHASUMS.txt and ask you to sign it with your GPG key so it can upload that to the release directory.

Note that promotion is only final for builds that are in staging. Any additional builds, including replacement builds, that land in staging can be promoted with tools/release.sh. You shouldn't be promoting new tarballs or packages to overwrite existing ones for any release (the shasum should be final for each file) but you can use it to add new tarballs or packages to the release. In practice this is normally about adding ARMv6 binaries when they are done. Just run tools/release.sh again to make this happen and you'll be prompted again and will have to sign the new SHASUMS.txt that now includes your new files.

9. Announce release on nodejs.org

There is an automatic build that is kicked off when you promote new builds, so within a few minutes nodejs.org will be listing your new version as the latest without you having to do anything extra. But the blog post is not yet fully automatic.

Clone new.nodejs.org, npm install, and run ./scripts/release-post.js. This has been contributed by @phillipj who has been very responsive in adjusting it to work better with each release, be sure to provide feedback. The program will inspect the latest promoted version and create a new post for you using all of the information it can collect.

• You can add a short blurb just under the main heading if you want to say something important, otherwise the text should be good to go, straight from your CHANGELOG.
• The links to the download files won't be complete unless you waited for ARMv6 builds for some reason, any downloads that are missing will have *Coming soon* next to them. It's your responsibility to update these later when you have builds and it's a manual process of editing the post to add in the links.
• The SHASUMS256.txt.asc content is at the bottom and when you update the list of tarballs you'll need to copy/paste the new contents of this file to reflect those changes.
• Always use pull-requests on the new.nodejs.org repo, be respectful of that working group but you shouldn't have to wait for sign-off, opening a PR and merging it straight away should be fine.
• Changes to master on the new.nodejs.org repo will trigger a new build of nodejs.org so your changes should appear in a few minutes after pushing.

10. Announce on twitter

Link to the download directory for your builds, link to the changelog, mention the version and @-mention @nodejs so it gets retweeted when one of those individuals notice it.

11. Cleanup

Close your PR, remove your branch, etc.

Ping myself, @Fishrock123 or @nodejs/build if any assistance is needed during this whole process.

[rvagg]

https://github.com/rvagg/gfm2html is another tool of mine I find useful in this process, making local html versions of markdown like they would be rendered on github

[jasnell]

The instructions were quite helpful. Thank you. Couple of nits:

• Section 2, second set of Bullet points, item test: don't remove empty.txt on win32 #2, needs to read set NODE_VERSION_IS_RELEASE to 1 instead of 0
• Section 9 needs to mention possibly needing to update the informative banner on the home page (it's updated in the build.js file)
• It's still not entirely clear where all of the artifacts are located after the iojs+release build. That could be made clearer.

[jasnell]

@rvagg ... couple more bits on this: the process for doing a RC release should be included and this would likely be best as a file in the docs folder. I can open a PR for that a bit later.

[jasnell]

Note: when running ./scripts/release-post.js to generate a post for anything other than the current stable, you will need to include the release version number, e.g. ./scripts/release-post.js 4.2.4 to generate the announce post for v4.2.4

[cjihrig]

I landed a big update to releases.md in f1653cc. That commit used the existing information, as well as the material in this issue. I'm going to close this.