Skip to content
This repository was archived by the owner on Feb 12, 2024. It is now read-only.

refactor: improvements to the release process #2408

Merged
merged 13 commits into from
Sep 4, 2019
Merged

refactor: improvements to the release process #2408

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
4229a14
feat: improvements to the release process
Aug 29, 2019
d929e46
feat: add cartoon
Aug 30, 2019
fef01c1
fix: copy tweaks
Aug 30, 2019
19a7a5f
fix: ToC
Aug 30, 2019
f0ab4d3
fix: more tweak
Aug 30, 2019
ed3b138
fix: stage 3 link
Aug 30, 2019
7a296a7
fix: tweaks and changes to release issue template
Aug 30, 2019
4805832
feat: estimated shipping date?
Aug 30, 2019
d2965dc
docs: fix blog url
Sep 3, 2019
8680975
docs: fix discuss url
Sep 3, 2019
1754255
docs: fix discuss url again
Sep 3, 2019
c5b86b2
refactor: use MINOR for breaks and features, PATCH for bug fixes
Sep 4, 2019
8af5edc
refactor: suggest fuzzy shipping date
Sep 4, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 0 additions & 66 deletions RELEASE.md
View file
Open in desktop

This file was deleted.

29 changes: 29 additions & 0 deletions doc/EARLY_TESTERS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# Early Testers Programme

## What is it?

The early testers programme allows groups using js-ipfs in production to self-volunteer to help test js-ipfs release candidates to ensure that no regressions that might affect production systems make it into the final release. While we invite the _entire_ community to help test releases, members of the early testers program are expected to participate directly and actively in every release.

## What are the expectations?

Members of the early tester program are expected to work closely with us to:

* Provide high quality, actionable feedback.
* Work directly with us to debug regressions in the release.
* Help ensure a rock-solid, timely release.

We will ask early testers to participate at two points in the process:

* When js-ipfs enters the second release stage, early testers will be asked to test js-ipfs on non-production infrastructure. This may involve things like:
- Running integration tests against the release candidate.
- Running simulations/benchmarks on the release candidate.
- Manually testing the release candidate to check for regressions.
* When js-ipfs enters the third release stage (soft release), early testers will be asked to partially deploy the release candidate to production infrastructure. Release candidates at this stage are expected to be identical to the final release. However, this stage allows the js-ipfs team to fix any last-minute regressions without cutting an entirely new release.

## Who has signed up?

No early testers volunteers yet!

## How to sign up?

Simply submit a PR to this document by adding your project name and contact.
126 changes: 126 additions & 0 deletions doc/RELEASE_ISSUE_TEMPLATE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
# Release Template

> short tl;dr; of the release

# 🗺 What's left for release

<List of items with PRs and/or Issues to be considered for this release>

# 🚢 Estimated shipping date

<Date this release will ship on if everything goes to plan>

# 🔦 Highlights

<Top highlights for this release>

# 🏗 API Changes

<Any API changes breaking or otherwise that people should know of>

# ✅ Release Checklist

- [ ] **Stage 0 - Automated Testing**
- [ ] Feature freeze. If any "non-trivial" changes (see the footnotes of [doc/releases.md](https://github.com/ipfs/js-ipfs/tree/master/doc/releases.md) for a definition) get added to the release, uncheck all the checkboxes and return to this stage.
- [ ] Automated Testing (already tested in CI) - Ensure that all tests are passing, this includes:
- [ ] unit/functional/integration/e2e
- [ ] interop
- [ ] sharness
- [ ] **Stage 1 - Internal Testing**
- [ ] Publish a release candidate to npm
```sh
# Clean out node_modules and re-install dependencies
# (Ensures you have the latest versions for the browser build)
rm -rf node_modules package-lock.json
npm install

# Minor prerelease (e.g. 0.33.1 -> 0.34.0-rc.0)
npx aegir release --type preminor --preid rc --dist-tag next

# Increment prerelease (e.g. 0.34.0-rc.0 -> 0.34.0-rc.1)
npx aegir release --type prerelease --preid rc --dist-tag next
```
- Network Testing:
- test lab things - TBD
- Infrastructure Testing:
- TBD
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

js-ipfs currently has no deployments to internal infrastructure so there's nothing to partially roll out to here yet.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not itself, but it does rely on *-star signalling servers, preload nodes, relay nodes and bootstrappers

- [ ] IPFS Application Testing - Run the tests of the following applications:
- [ ] [webui](https://github.com/ipfs-shipyard/ipfs-webui)
- [ ] [ipfs-desktop](https://github.com/ipfs-shipyard/ipfs-desktop)
- [ ] [ipfs-companion](https://github.com/ipfs-shipyard/ipfs-companion)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If success then I'd like for @lidel to publish a new version of companion to a beta channel 🙏

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we should include ipfs-postmsg-proxy as a separate thing. Thoughts?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I think we should but I think it needs a big update before we can include it here....

- [ ] [npm-on-ipfs](https://github.com/ipfs-shipyard/npm-on-ipfs)
- [ ] [ipfs-pubsub-room](https://github.com/ipfs-shipyard/ipfs-pubsub-room)
- [ ] [peer-base](https://github.com/peer-base/peer-base)
- [ ] [service-worker-gateway](https://github.com/ipfs-shipyard/service-worker-gateway)
- [ ] **Stage 2 - Community Dev Testing**
- [ ] Reach out to the IPFS _early testers_ listed in [doc/EARLY_TESTERS.md](https://github.com/ipfs/js-ipfs/tree/master/doc/EARLY_TESTERS.md) for testing this release (check when no more problems have been reported). If you'd like to be added to this list, please file a PR.
- [ ] Reach out on IRC for additional early testers.
- [ ] Run tests available in the following repos with the latest RC:
- [ ] [ipfs-log](https://github.com/orbitdb/ipfs-log)
- [ ] [orbit-db](https://github.com/orbitdb/orbit-db)
- [ ] **Stage 3 - Community Prod Testing**
- [ ] Documentation
- [ ] Ensure that [README.md](https://github.com/ipfs/js-ipfs/tree/master/README.md) is up to date
- [ ] Ensure that all the examples run without problems:
- [ ] https://github.com/ipfs/js-ipfs/tree/master/examples/browser-add-readable-stream
- [ ] https://github.com/ipfs/js-ipfs/tree/master/examples/browser-browserify
- [ ] https://github.com/ipfs/js-ipfs/tree/master/examples/browser-mfs
- [ ] https://github.com/ipfs/js-ipfs/tree/master/examples/browser-parceljs
- [ ] https://github.com/ipfs/js-ipfs/tree/master/examples/browser-readablestream
- [ ] https://github.com/ipfs/js-ipfs/tree/master/examples/browser-script-tag
- [ ] https://github.com/ipfs/js-ipfs/tree/master/examples/browser-video-streaming
- [ ] https://github.com/ipfs/js-ipfs/tree/master/examples/browser-vue
- [ ] https://github.com/ipfs/js-ipfs/tree/master/examples/browser-webpack
- [ ] https://github.com/ipfs/js-ipfs/tree/master/examples/circuit-relaying
- [ ] https://github.com/ipfs/js-ipfs/tree/master/examples/custom-ipfs-repo
- [ ] https://github.com/ipfs/js-ipfs/tree/master/examples/custom-libp2p
- [ ] https://github.com/ipfs/js-ipfs/tree/master/examples/exchange-files-in-browser
- [ ] https://github.com/ipfs/js-ipfs/tree/master/examples/explore-ethereum-blockchain
- [ ] https://github.com/ipfs/js-ipfs/tree/master/examples/ipfs-101
- [ ] https://github.com/ipfs/js-ipfs/tree/master/examples/run-in-electron
- [ ] https://github.com/ipfs/js-ipfs/tree/master/examples/running-multiple-nodes
- [ ] https://github.com/ipfs/js-ipfs/tree/master/examples/traverse-ipld-graphs
Copy link
Member Author

@alanshaw alanshaw Aug 30, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A bit onerous to keep up to date, but I imagine we can easily just add a new one to this list if we notice one we forgot to add while it was being created...

- [ ] Update [js.ipfs.io](https://js.ipfs.io) examples to use the latest js-ipfs
- [ ] Invite the IPFS [_early testers_](https://github.com/ipfs/js-ipfs/tree/master/doc/EARLY_TESTERS.md) to deploy the release to part of their production infrastructure.
- [ ] Invite the wider community (link to the release issue):
- [ ] [discuss.ipfs.io](https://discuss.ipfs.io/c/announcements)
- [ ] Twitter
- [ ] IRC
- [ ] **Stage 4 - Release**
- [ ] Take a snapshot of everyone that has contributed to this release (including its direct dependencies in IPFS, libp2p, IPLD and multiformats) using [this script](https://gist.github.com/alanshaw/5a2d9465c5a05b201d949551bdb1fcc3).
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eventually I'll find time to turn this into an easy to use CLI tool like I did here: https://github.com/filecoin-project/filecoin-contributors

- [ ] Publish to npm:
```sh
# Clean out node_modules and re-install dependencies
# (Ensures you have the latest versions for the browser build)
rm -rf node_modules package-lock.json
npm install

# lint, build, test, tag, publish
npm run release-minor
```
- [ ] Publish a blog post to [github.com/ipfs/blog](https://github.com/ipfs/blog) (at minimum, a c&p of this release issue with all the highlights, API changes and thank yous)
- [ ] Broadcasting (link to blog post)
- [ ] Twitter
- [ ] IRC
- [ ] [Reddit](https://reddit.com/r/ipfs)
- [ ] [discuss.ipfs.io](https://discuss.ipfs.io/c/announcements)
- [ ] Announce it on the [IPFS Users Mailing List](https://groups.google.com/forum/#!forum/ipfs-users)
- [ ] Copy release notes to the [GitHub Release description](https://github.com/ipfs/js-ipfs/releases)

# ❤️ Huge thank you to everyone that made this release possible

<Generated contributor list>

# 🙌🏽 Want to contribute?

Would you like to contribute to the IPFS project and don't know how? Well, there are a few places you can get started:

- Check the issues with the `help wanted` label in the [js-ipfs repo](https://github.com/ipfs/js-ipfs/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22)
- Join an IPFS All Hands, introduce yourself and let us know where you would like to contribute - https://github.com/ipfs/team-mgmt/#weekly-ipfs-all-hands
- Hack with IPFS and show us what you made! The All Hands call is also the perfect venue for demos, join in and show us what you built
- Join the discussion at http://discuss.ipfs.io/ and help users finding their answers.
- Join the [🚀 IPFS Core Implementations Weekly Sync 🛰](https://github.com/ipfs/team-mgmt/issues/992) and be part of the action!

# ⁉️ Do you have questions?

The best place to ask your questions about IPFS, how it works and what you can do with it is at [discuss.ipfs.io](http://discuss.ipfs.io). We are also available at the `#ipfs` channel on Freenode.
120 changes: 120 additions & 0 deletions doc/releases.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
# Releases

## Table of Contents

- [Release Philosophy](#release-philosophy)
- [Release Flow](#release-flow)
- [Stage 0 - Automated Testing](#stage-0---automated-testing)
- [Stage 1 - Internal Testing](#stage-1---internal-testing)
- [Stage 2 - Community Dev Testing](#stage-2---community-dev-testing)
- [Stage 3 - Community Prod Testing](#stage-3---community-prod-testing)
- [Stage 4 - Release](#stage-4---release)
- [Release Cycle](#release-cycle)
- [Patch Releases](#patch-releases)
- [Performing a Release](#performing-a-release)
- [Release Version Numbers](#release-version-numbers)
- [Pre-Releases and Release Candidates](#pre-releases-and-release-candidates)

## Release Philosophy

js-ipfs aims to have release every six weeks, two releases per quarter. During these 6 week releases, we go through 4 different stages that gives us the opportunity to test the new version against our test environments (unit, interop, integration), QA in our current production environment, IPFS apps (e.g. Desktop and WebUI) and with our community and _early testers_<sup>[1]</sup> that have IPFS running in production.

We might expand the six week release schedule in case of:

- No new updates to be added
- In case of a large community event that takes the core team availability away (e.g. IPFS Conf, Dev Meetings, IPFS Camp, etc.)

## Release Flow

js-ipfs releases come in 5 stages designed to gradually roll out changes and reduce the impact of any regressions that may have been introduced. If we need to merge non-trivial<sup>[2]</sup> changes during the process, we start over at stage 0.

![js-ipfs release flow cartoon](https://ipfs.io/ipfs/QmU5pwcGh38DqzLy3rK8GAHuWm2kK87oGqDAtqZYWhxjab)

### Stage 0 - Automated Testing

At this stage, we expect _all_ automated tests (unit, functional, integration, interop, testlab, performance, etc.) to pass.

### Stage 1 - Internal Testing

At this stage, we'll:

1. Start a partial-rollout to our own infrastructure.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, this is a noop for js-ipfs right now but with the addition of /refs and now GC we're one step closer to being able to be our own preload node.

2. Test against applications in the [ipfs](https://github.com/ipfs/) and [ipfs-shipyard](https://github.com/ipfs-shipyard/) organisations and a selection of other hand picked projects.

**Goals:**

1. Make sure we haven't introduced any obvious regressions.
2. Test the release in an environment we can monitor and easily roll back (i.e. our own infra).

### Stage 2 - Community Dev Testing

At this stage, we'll announce the impending release to the community and ask for pre-release testers.

**Goal:**

Test the release in as many non-production environments as possible. This is relatively low-risk but gives us a _breadth_ of testing internal testing can't.

### Stage 3 - Community Prod Testing

At this stage, we consider the release to be "production ready" and will ask the community and our early testers to (partially) deploy the release to their production infrastructure.

**Goals:**

1. Test the release in some production environments with heavy workloads.
2. Partially roll-out an upgrade to see how it affects the network.
3. Retain the ability to ship last-minute fixes before the final release.

### Stage 4 - Release

At this stage, the release is "battle hardened" and ready for wide deployment. A new version is published to npm, announcements are made and a blog post is published to [ipfs.io/blog](https://ipfs.io/blog).

## Release Cycle

A full release process should take about 3 weeks, a week per stage 1-3. We will start a new process every 6 weeks, regardless of when the previous release landed unless it's still ongoing.

### Patch Releases

If we encounter a serious bug in the stable latest release, we will create a patch release based on this release. For now, bug fixes will _not_ be backported to previous releases.

Patch releases will usually follow a compressed release cycle and should take 2-3 days. In a patch release:

1. Automated and internal testing (stage 0 and 1) will be compressed into a few hours - ideally less than a day.
2. Stage 2 will be skipped.
3. Community production testing will be shortened to 1-2 days of opt-in testing in production (early testers can choose to pass).

Some patch releases, especially ones fixing one or more complex bugs, may undergo the full release process.

## Performing a Release

The release is managed by the "Lead Maintainer" for js-ipfs. It starts with the opening of an issue containing the content available on the [RELEASE_ISSUE_TEMPLATE](./RELEASE_ISSUE_TEMPLATE.md) not more than **48 hours** after the previous release.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the 48 hours which the go-ipfs release process doc currently doesn't have. This communicates expectations for when the new release is due and will help with planning of included features. The community could even suggest inclusions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Stebalien not sure if you want to adopt this idea too (might help with writing release notes as we go?)


This issue is pinned and labeled ["release"](https://github.com/ipfs/js-ipfs/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+label%3Arelease). When the cycle is due to begin the 5 stages will be followed until the release is done.

## Release Version Numbers

js-ipfs is currently pre-1.0. In semver terms this means [anything may change at any time](https://semver.org/#spec-item-4).

However, js-ipfs follows semver "shifted by one". This means js-ipfs reserves MINOR version increments for BREAKING CHANGES and/or major feature additions and PATCH version increments for bug fixes and _backwards compatible_ minor feature additions.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is what we do now. Is this working for everyone? Do we think that with the more strict release schedule we could actually use MINOR for all features (and breaks) and PATCH exclusively for bug fixes?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is reasonable with a 6 week release cycle.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is reasonable with a 6 week release cycle.

Just to clarify, you are voting for a change from what we do now to this?:

MINOR for all features (and breaks) and PATCH exclusively for bug fixes?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to clarify, you are voting for a change from what we do now to this?

Yes, I'm in favor of keeping the versioning simple and predictable. It also allows us to minimize the time investment on releases, and ensure that all new features have gone through the full release testing cycle, which should help reduce the need for patch releases.

Copy link
Contributor

@mikeal mikeal Sep 3, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This still isn’t going to conform to any of the tooling people have written around semver. Regardless of what we say, developers and the automation written by those developers will tend to assume that there aren’t intentional breaking changes between these releases — or they will assume that any version will break and may not adopt it because there isn’t any kind of guarantee.

I don’t see why we can’t just bump the major version number. Version numbers are essentially free, we have 2,147,483,647 of them available.

We like to talk about “1.0” as this big milestone, but we could just call that milestone something else and untangle it from version numbers. Version numbers, especially in JS, have a lot of expectations around them that we are not currently conforming to. It’s easier to adjust to the community than to try and document/educate every JS developer who finds js-ipfs.

We’ve already done far more work to stabilize js-ipfs and ensure there aren’t breaking changes between these releases than the average package on npm, it seems silly that we wouldn’t just adjust the versioning along with it in order to gain more adoption given that we’re putting in all of this effort to avoid breaking changes anyway.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with @mikeal, but we got here as there was a desire to make 1.0 mean something about defining an API that we want to commit to. It feels like we did that a long time ago, but still #1563 (comment)

Otherwise I am fine with patches for bug fixes, minor for everything else. Seems like it 'd be 0.40.0 then 0.40.1 when we spot the terrible regression, then 0.41.0 and 0.41.1 ~6 weeks later which seems nice and predictable

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mikeal - I don't think this is the right place to have that discussion. Feel free to start a separate issue, but I don't want to block merging this release process description on changing a very embedded IPFS standard - aka that 1.0 means interface reliability / forwards compatibility. Changing that current practice is a separate discussion.

I'm supportive of consistency between the js and go - aka patches should be for bug fixes or very scoped improvements - and minor should be for all new features (regardless of compatibility) such that they have testing.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with @momack2, we should have that discussion elsewhere as it's not the question on the table right now and I don't want to further expand the scope of this PR.

It sounds as if there's at least agreement that we should switch to MINOR for all features (and breaks) and PATCH exclusively for bug fixes which I'm happy to do right now and would be consistent with go-ipfs so lets do that! 🚢🚢🚢

FWIW I have a strong desire to use the major version number and I dislike that 1.0 is this big milestone for stable, production ready, "finished" to a lot of people. We lose a lot of power of semver by not using the major version number and as @mikeal pointed out many people (and machines) are still going to view anything pre-1.0 as unstable and likely to change and break, whatever convention we use for pre-1.0 versioning.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, wasn’t intending to block the merge, that’s why it’s just a comment and a not a full review. the change being discussed is a positive move in the right direction, i don’t see any need to block it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No worries - it is a necessary point to make, I'm glad someone bought it up 🙂

Lets open an issue and pick up where we left off.


Post `1.x.x` (future), MAJOR version number increments will contain BREAKING CHANGES, MINOR version increments will be reserved for backwards compatible new features and PATCH version increments for bug fixes.

We do not yet retroactively apply fixes to older releases (no Long Term Support releases for now), which means that we always recommend users to update to the latest, whenever possible.

### Pre-Releases and Release Candidates

Prior to or during the early stages of a release cycle, js-ipfs may release a pre-release version for users who want to try out the "bleeding edge". This typically happens when a new major feature or breaking change lands in master. When this happens is entirely at the discretion of the Lead Maintainer.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this make sense? I've been doing this already for non-feature complete pre-releases but I realised I've not openly communicated this.


Pre-release version numbers take the form `x.y.z-pre.n`.

Where `x`, `y`, and `z` are the usual MAJOR, MINOR and PATCH version numbers and `n` (starting at 0) increments by 1 for every pre-release version that is released.

Following any pre-releases, at least one release candidate will be published during stage 1/2.

Release candidate version numbers take the form `x.y.z-rc.n`.

Where `x`, `y`, and `z` are the usual MAJOR, MINOR and PATCH version numbers and `n` (starting at 0) increments by 1 for every release candidate that is released.

---

- <sup>**[1]**</sup> - _early testers_ is an IPFS programme in which members of the community can self-volunteer to help test `js-ipfs` Release Candidates. You find more info about it at [EARLY_TESTERS.md](./EARLY_TESTERS.md)
- <sup>**[2]**</sup> - A non-trivial change is any change that could potentially introduce an issue that could not categorically be caught by automated testing. This is up to the discretion of the Lead Maintainer but the assumption is that every change is non-trivial unless proven otherwise.