diff --git a/README.md b/README.md index 2328b6340..692550356 100644 --- a/README.md +++ b/README.md @@ -13,14 +13,14 @@ The [contributing documentation](CONTRIBUTING.md) covers licencing and the usual [Cincinnati][] is configured to track the master branch, so it will automatically react to updates made to this repository. -### Schema version +### Schema Version The layout of this repository is versioned via [a `version` file](version), which contains the [Semantic Version][semver] of the schema. As a schema version, the patch level is likely to remain 0, but the minor version will be incremented if backwards-compatible features are added, and the major version will be incremented if backwards-incompatible changes are made. Consumers, such as [Cincinnati][], who support *x.y.0* may safely consume this repository when the stated major version matches the understood *x* and the stated minor version is less than or equal to the understood *y*. For example, a consumer that supports 1.3.0 and 2.1.0 could safely consume 1.2.0, 1.3.0, 2.0.0, 2.1.0, etc., but could not safely consume 1.4.0, 2.2.0, 3.0.0, etc. -### Release names +### Release Names Release names are used for [adding releases to channels](#add-releases-to-channels) and [blocking edges](#block-edges). Architecture-agnostic names will apply to all images with that exact name in the `version` property of the `release-metadata` file included in the release image. @@ -31,13 +31,60 @@ And 4.2.14+amd64 would only apply to the amd64 release image. ### Add Releases To Channels Edit the appropriate file in `channels/`. -For example, to add a release to stable-4.2 you would edit `channels/stable-4.2.yaml`. -Channel semantics are documented [here][channel-semantics]. +For example, to add a release to candidate-4.2 you would edit [`channels/candidate-4.2.yaml`](channels/candidate-4.2.yaml). The file contains a list of versions. Please keep the versions in order. And LEAVE COMMENTS if you skip a version. +#### Feeder Channels + +Channel semantics, as documented [here][channel-semantics], show nodes and edges being promoted to successive channels as they prove their stability. +For example, a 4.2.z release will appear in `candidate-4.2` first. +Upon proving itself sufficiently stable in the candidate channel, it will be promoted into `fast-4.2`. +Some time after landing in `fast-4.2`, it will appear in `stable-4.2`. + +_Note:_ Once we have phased release rollouts, we will drop the fast/stable distinction from this repository and promote to a unified fast/stable channel with a start time and rollout duration. +Until then, we are using fast channels to feed stable channels with a delay, just like candidate channels feed fast channels. + +In this repository, the intended promotion flow is reflected by a `feeder` property in the channel declaration. +For example, for [`channels/fast-4.2.yaml`](channels/fast-4.2.yaml): + +```yaml +feeder: + name: candidate-4.2 + delay: P1W + filter: 4\.[0-9]+\.[0-9]+(.*hotfix.*|\+amd64|-s390x)? +``` + +which declares the intention that nodes and edges will be considered for promotion into `fast-4.2` after cooking for one week in `candidate-4.2`. +The `delay` value is an [ISO 8601][rfc-3339-p13] [duration][iso-8601-durations]. +The `filter` value excludes `4.2.0-rc.5` and other releases, while allowing for `4.2.0-0.hotfix-2020-09-19-234758` and `4.2.10-s390x` and `4.2.14+amd64`. + +This is the expected delay, but it does not mean that promotion will happen at that moment. +For example, it is possible that release architects decide that there is insufficient data for a `fast-4.2` promotion, in which case the promotion can be delated until sufficient data accumulates. + +To see recommended feeder promotions, run: + +```console +$ hack/stabilization-change.py +``` + +##### Tombstones + +Removing a node from a channel can strand existing clusters with a `VersionNotFound` error. +To avoid that, unstable nodes are left in their existing channels, but should not be promoted to additional channels. +This is reflected through entries in the optional `tombstones` property. +For example, [`channels/candidate-4.2.yaml`](channels/candidate-4.2.yaml) has: + +```yaml +tombstones: +- 4.1.18 +- 4.1.20 +``` + +declaring that, while 4.1.18 and 4.1.20 are in `candidate-4.2`, they should not be promoted to subsequent channels (in this case, `fast-4.2`). + ### Block Edges Create/edit an appropriate file in `blocked_edges/`. @@ -59,5 +106,7 @@ from: 4\.1\.(18|20) [channel-semantics]: https://docs.openshift.com/container-platform/4.3/updating/updating-cluster-between-minor.html#understanding-upgrade-channels_updating-cluster-between-minor [Cincinnati]: https://github.com/openshift/cincinnati/ [image-arch]: https://github.com/opencontainers/image-spec/blame/v1.0.1/config.md#L103 +[iso-8601-durations]: https://en.wikipedia.org/wiki/ISO_8601#Durations +[rfc-3339-p13]: https://tools.ietf.org/html/rfc3339#page-13 [semver]: https://semver.org/spec/v2.0.0.html [semver-build]: https://semver.org/spec/v2.0.0.html#spec-item-10 diff --git a/channels/candidate-4.2.yaml b/channels/candidate-4.2.yaml index 2d218a5d0..aebb75098 100644 --- a/channels/candidate-4.2.yaml +++ b/channels/candidate-4.2.yaml @@ -1,4 +1,13 @@ name: candidate-4.2 +tombstones: +- 4.1.18 +# no 4.1.37 because of https://bugzilla.redhat.com/show_bug.cgi?id=1810393 (Jenkins webconsole 403 Forbidden) +- 4.1.37 +- 4.2.14 +# no 4.2.30 because broken build causing machine-api-operator CrashLoopBackoff, https://bugzilla.redhat.com/show_bug.cgi?id=1829028 +- 4.2.30 +# no 4.2.32 because of RHEL 8.2 ICMP, https://bugzilla.redhat.com/show_bug.cgi?id=1781575 +- 4.2.32 versions: - 4.1.18 - 4.1.20 diff --git a/channels/candidate-4.3.yaml b/channels/candidate-4.3.yaml index 99d75a1b0..cd3b6b8e0 100644 --- a/channels/candidate-4.3.yaml +++ b/channels/candidate-4.3.yaml @@ -1,4 +1,21 @@ name: candidate-4.3 +tombstones: +# no 4.2.30 because broken build causing machine-api-operator CrashLoopBackoff, https://bugzilla.redhat.com/show_bug.cgi?id=1829028 +- 4.2.30 +# no 4.2.32 because of RHEL 8.2 ICMP, https://bugzilla.redhat.com/show_bug.cgi?id=1781575 +- 4.2.32 +# No 4.3.11 because of build issues: https://bugzilla.redhat.com/show_bug.cgi?id=1823378 +- 4.3.11 +# No 4.3.14 (did not include 4.3.0 as an update source: https://github.com/openshift/cincinnati-graph-data/pull/194#issuecomment-618080792 ) +- 4.3.14 +# No 4.3.15 (did not include DefaultSecurityContextConstraints_Mutated clear: https://bugzilla.redhat.com/show_bug.cgi?id=1827337 ) +- 4.3.15 +# No 4.3.16 (OVN container has unsigned RPMs: https://github.com/openshift/cincinnati-graph-data/pull/205#issuecomment-619049150 ) +- 4.3.16 +# No 4.3.17 because of CRI-O OCINI context, https://bugzilla.redhat.com/show_bug.cgi?id=1828104 +- 4.3.17 +# No 4.3.24, because we were minimizing Quay load +- 4.3.24 versions: - 4.2.36 - 4.2.34 diff --git a/channels/candidate-4.4.yaml b/channels/candidate-4.4.yaml index c0ba0a9ae..717bd0ec8 100644 --- a/channels/candidate-4.4.yaml +++ b/channels/candidate-4.4.yaml @@ -1,4 +1,46 @@ name: candidate-4.4 +tombstones: +# No early 4.3.z will be promoted to fast, these are just for testing -> RC updates +- 4.3.5 +- 4.3.8 +- 4.3.9 +- 4.3.10 +# No 4.3.11 because of build issues: https://bugzilla.redhat.com/show_bug.cgi?id=1823378 +- 4.3.11 +# No 4.3.14 (did not include 4.3.0 as an update source: https://github.com/openshift/cincinnati-graph-data/pull/194#issuecomment-618080792 ) +- 4.3.14+amd64 +# No 4.3.15 (did not include DefaultSecurityContextConstraints_Mutated clear: https://bugzilla.redhat.com/show_bug.cgi?id=1827337 ) +- 4.3.15+amd64 +# No 4.3.16 (OVN container has unsigned RPMs: https://github.com/openshift/cincinnati-graph-data/pull/205#issuecomment-619049150 ) +- 4.3.16 +# No 4.3.17 because of CRI-O OCINI context, https://bugzilla.redhat.com/show_bug.cgi?id=1828104 +- 4.3.17+amd64 +# No 4.3.24, because we were minimizing Quay load +- 4.3.24 +# RC releases are never promoted to fast channels +- 4.4.0-rc.0 +- 4.4.0-rc.1 +- 4.4.0-rc.2 +- 4.4.0-rc.4 +- 4.4.0-rc.6 +- 4.4.0-rc.7 +- 4.4.0-rc.8 +- 4.4.0-rc.9 +- 4.4.0-rc.10 +- 4.4.0-rc.11 +- 4.4.0-rc.12 +- 4.4.0-rc.13 +# No 4.4.0 because MachineConfig CRD does not define all fields which users might have set in 4.3, https://bugzilla.redhat.com/show_bug.cgi?id=1829651 +- 4.4.0 +# No 4.4.2 because 4.4 MachineSet with 4.2 or earlier bootimages fails to scale up because old CRI-O chokes on new CRI-O config, https://bugzilla.redhat.com/show_bug.cgi?id=1830102 +- 4.4.2 +# No 4.4.7, because we were minimizing Quay load +- 4.4.7 +# Shares an errata with a later release +- 4.4.22 +- 4.4.24 +- 4.4.25 +- 4.4.28 versions: - 4.3.40 - 4.3.38 diff --git a/channels/candidate-4.5.yaml b/channels/candidate-4.5.yaml index ada3c2fa3..27a2f6283 100644 --- a/channels/candidate-4.5.yaml +++ b/channels/candidate-4.5.yaml @@ -1,4 +1,22 @@ name: candidate-4.5 +tombstones: +# No 4.4.0 because MachineConfig CRD does not define all fields which users might have set in 4.3, https://bugzilla.redhat.com/show_bug.cgi?id=1829651 +- 4.4.0 +# No 4.4.2 because 4.4 MachineSet with 4.2 or earlier bootimages fails to scale up because old CRI-O chokes on new CRI-O config, https://bugzilla.redhat.com/show_bug.cgi?id=1830102 +- 4.4.2 +# No 4.4.7, because we were minimizing Quay load +- 4.4.7 +# Never got an errata +- 4.5.0-0.hotfix-2020-11-28-021842 +# Shares an errata with a later release +- 4.4.22 +- 4.4.24 +- 4.4.25 +- 4.4.28 +- 4.5.0 +- 4.5.25 +- 4.5.29 +- 4.5.32 versions: - 4.4.33 diff --git a/channels/candidate-4.6.yaml b/channels/candidate-4.6.yaml index 2c4feb28a..773703481 100644 --- a/channels/candidate-4.6.yaml +++ b/channels/candidate-4.6.yaml @@ -1,4 +1,22 @@ name: candidate-4.6 +tombstones: +# Never got an errata +- 4.5.0-0.hotfix-2020-11-28-021842 +# Shares an errata with a later release +- 4.5.0 +- 4.5.25 +- 4.5.29 +- 4.5.32 +- 4.6.0 +- 4.6.2 +# Rerolling as 4.6.6 with a rebuilt CRI-O RPM +- 4.6.5 +# Blocked on born-in-4.1 ingress regression: https://bugzilla.redhat.com/show_bug.cgi?id=1904582#c4 +- 4.6.7 +# Did not fix 4.6.9 OVS regression https://bugzilla.redhat.com/show_bug.cgi?id=1914284 +- 4.6.10 +# 4.6.24 introduced a regression in registry ca-trust handling: https://bugzilla.redhat.com/show_bug.cgi?id=1949040#c4 +- 4.6.24 versions: - 4.5.37 diff --git a/channels/candidate-4.7.yaml b/channels/candidate-4.7.yaml index 07297d428..f5301c9f1 100644 --- a/channels/candidate-4.7.yaml +++ b/channels/candidate-4.7.yaml @@ -1,4 +1,9 @@ name: candidate-4.7 +tombstones: +# Did not fix 4.6.9 OVS regression https://bugzilla.redhat.com/show_bug.cgi?id=1914284 +- 4.6.10 +# 4.6.24 introduced a regression in registry ca-trust handling: https://bugzilla.redhat.com/show_bug.cgi?id=1949040#c4 +- 4.6.24 versions: - 4.6.25 diff --git a/channels/eus-4.6.yaml b/channels/eus-4.6.yaml index 699149dc9..24d39b3f6 100644 --- a/channels/eus-4.6.yaml +++ b/channels/eus-4.6.yaml @@ -1,4 +1,8 @@ name: eus-4.6 +feeder: + name: stable-4.6 + delay: PT0H + filter: 4\.6\.[0-9]+(.*hotfix.*)? versions: - 4.6.1 - 4.6.3 diff --git a/channels/fast-4.2.yaml b/channels/fast-4.2.yaml index 5f9f6f64f..a626dd569 100644 --- a/channels/fast-4.2.yaml +++ b/channels/fast-4.2.yaml @@ -1,4 +1,10 @@ name: fast-4.2 +feeder: + name: candidate-4.2 + delay: P1W + filter: 4\.[0-9]+\.[0-9]+(.*hotfix.*|\+amd64|-s390x)? +tombstones: +- 4.2.10-s390x versions: - 4.1.20 - 4.1.21 diff --git a/channels/fast-4.3.yaml b/channels/fast-4.3.yaml index cfdfef419..370a1306d 100644 --- a/channels/fast-4.3.yaml +++ b/channels/fast-4.3.yaml @@ -1,4 +1,8 @@ name: fast-4.3 +feeder: + name: candidate-4.3 + delay: P1W + filter: 4\.[0-9]+\.[0-9]+(.*hotfix.*)? versions: - 4.2.36 - 4.2.34 diff --git a/channels/fast-4.4.yaml b/channels/fast-4.4.yaml index ae68f6897..8260e3ee3 100644 --- a/channels/fast-4.4.yaml +++ b/channels/fast-4.4.yaml @@ -1,4 +1,8 @@ name: fast-4.4 +feeder: + name: candidate-4.4 + delay: P1W + filter: 4\.[0-9]+\.[0-9]+(.*hotfix.*)? versions: - 4.3.40 - 4.3.38 diff --git a/channels/fast-4.5.yaml b/channels/fast-4.5.yaml index 3bdb2c190..72bbdbd54 100644 --- a/channels/fast-4.5.yaml +++ b/channels/fast-4.5.yaml @@ -1,4 +1,8 @@ name: fast-4.5 +feeder: + name: candidate-4.5 + delay: P1W + filter: 4\.[0-9]+\.[0-9]+(.*hotfix.*)? versions: - 4.4.33 diff --git a/channels/fast-4.6.yaml b/channels/fast-4.6.yaml index 4ca31355f..e6c7925c2 100644 --- a/channels/fast-4.6.yaml +++ b/channels/fast-4.6.yaml @@ -1,4 +1,8 @@ name: fast-4.6 +feeder: + name: candidate-4.6 + delay: P1W + filter: 4\.[0-9]+\.[0-9]+(.*hotfix.*)? versions: - 4.5.37 diff --git a/channels/fast-4.7.yaml b/channels/fast-4.7.yaml index 08a83c49f..4fba731e5 100644 --- a/channels/fast-4.7.yaml +++ b/channels/fast-4.7.yaml @@ -1,4 +1,8 @@ name: fast-4.7 +feeder: + name: candidate-4.7 + delay: P1W + filter: 4\.[0-9]+\.[0-9]+(.*hotfix.*)? versions: - 4.6.23 - 4.6.22 diff --git a/channels/fast-4.8.yaml b/channels/fast-4.8.yaml new file mode 100644 index 000000000..aba3e3892 --- /dev/null +++ b/channels/fast-4.8.yaml @@ -0,0 +1,6 @@ +name: fast-4.8 +feeder: + name: candidate-4.8 + delay: P1W + filter: 4\.[0-9]+\.[0-9]+(.*hotfix.*)? +versions: [] diff --git a/channels/prerelease-4.1.yaml b/channels/prerelease-4.1.yaml index a58801d7e..a4d2941d9 100644 --- a/channels/prerelease-4.1.yaml +++ b/channels/prerelease-4.1.yaml @@ -1,4 +1,7 @@ name: prerelease-4.1 +tombstones: +# no 4.1.37 because of https://bugzilla.redhat.com/show_bug.cgi?id=1810393 (Jenkins webconsole 403 Forbidden) +- 4.1.37 versions: - 4.1.0 - 4.1.0-rc.0 diff --git a/channels/stable-4.1.yaml b/channels/stable-4.1.yaml index e1a162fc6..6442cff7e 100644 --- a/channels/stable-4.1.yaml +++ b/channels/stable-4.1.yaml @@ -1,4 +1,8 @@ name: stable-4.1 +feeder: + name: prerelease-4.1 + delay: P1W + filter: 4\.[0-9]+\.[0-9]+(.*hotfix.*)? versions: - 4.1.0 - 4.1.1 diff --git a/channels/stable-4.2.yaml b/channels/stable-4.2.yaml index 775898fcd..92e2963f9 100644 --- a/channels/stable-4.2.yaml +++ b/channels/stable-4.2.yaml @@ -1,4 +1,7 @@ name: stable-4.2 +feeder: + name: fast-4.2 + delay: PT48H versions: - 4.1.20 - 4.1.21 diff --git a/channels/stable-4.3.yaml b/channels/stable-4.3.yaml index e697e4a6e..7420f1d05 100644 --- a/channels/stable-4.3.yaml +++ b/channels/stable-4.3.yaml @@ -1,4 +1,7 @@ name: stable-4.3 +feeder: + name: fast-4.3 + delay: PT48H versions: - 4.2.36 - 4.2.34 diff --git a/channels/stable-4.4.yaml b/channels/stable-4.4.yaml index c8af4b890..2e823c77a 100644 --- a/channels/stable-4.4.yaml +++ b/channels/stable-4.4.yaml @@ -1,4 +1,7 @@ name: stable-4.4 +feeder: + name: fast-4.4 + delay: PT48H versions: - 4.3.40 - 4.3.38 diff --git a/channels/stable-4.5.yaml b/channels/stable-4.5.yaml index 3333ca774..2ff90392c 100644 --- a/channels/stable-4.5.yaml +++ b/channels/stable-4.5.yaml @@ -1,4 +1,7 @@ name: stable-4.5 +feeder: + name: fast-4.5 + delay: PT48H versions: - 4.4.33 @@ -37,7 +40,9 @@ versions: - 4.5.7 - 4.5.8 - 4.5.9 +- 4.5.10 - 4.5.11 +- 4.5.12 - 4.5.13 - 4.5.14 - 4.5.15 diff --git a/channels/stable-4.6.yaml b/channels/stable-4.6.yaml index b5855255b..e63505a87 100644 --- a/channels/stable-4.6.yaml +++ b/channels/stable-4.6.yaml @@ -1,4 +1,7 @@ name: stable-4.6 +feeder: + name: fast-4.6 + delay: PT48H versions: - 4.5.36 - 4.5.35 diff --git a/channels/stable-4.7.yaml b/channels/stable-4.7.yaml index 048823a5d..332ed6746 100644 --- a/channels/stable-4.7.yaml +++ b/channels/stable-4.7.yaml @@ -1,4 +1,7 @@ name: stable-4.7 +feeder: + name: fast-4.7 + delay: PT48H versions: - 4.6.23 - 4.6.22 diff --git a/channels/stable-4.8.yaml b/channels/stable-4.8.yaml new file mode 100644 index 000000000..627b848b9 --- /dev/null +++ b/channels/stable-4.8.yaml @@ -0,0 +1,5 @@ +name: stable-4.8 +feeder: + name: fast-4.8 + delay: PT48H +versions: [] diff --git a/hack/stabilization-changes.py b/hack/stabilization-changes.py new file mode 100755 index 000000000..dfc6247a9 --- /dev/null +++ b/hack/stabilization-changes.py @@ -0,0 +1,109 @@ +#!/usr/bin/env python + +import datetime +import logging +import os +import re +import subprocess + +import yaml + + +logging.basicConfig(format='%(levelname)s: %(message)s') +_LOGGER = logging.getLogger(__name__) +_LOGGER.setLevel(logging.DEBUG) +_ISO_8601_DELAY_REGEXP = re.compile('^P((?P\d+)W)?(T(?P\d+)H)?$') +_GIT_BLAME_COMMIT_REGEXP = re.compile('^(?P[0-9a-f]{40}) .*') +_GIT_BLAME_HEADER_REGEXP = re.compile('^(?P[^ \t]+) (?P.*)$') +_GIT_BLAME_LINE_REGEXP = re.compile('^\t(?P.*)$') + + +def parse_iso8601_delay(delay): + # https://tools.ietf.org/html/rfc3339#page-13 + match = _ISO_8601_DELAY_REGEXP.match(delay) + if not match: + raise ValueError('invalid or unsupported ISO 8601 duration {!r}. Tooling currently only supports PWTH for week and/or hour offsets') + weeks = int(match.group('weeks') or 0) + hours = int(match.group('hours') or 0) + return datetime.timedelta(days=7*weeks, hours=hours) + + +def stabilization_changes(directory): + channels = {} + paths = {} + for root, _, files in os.walk(directory): + for filename in files: + if not filename.endswith('.yaml'): + continue + path = os.path.join(root, filename) + with open(path) as f: + try: + data = yaml.load(f, Loader=yaml.SafeLoader) + except ValueError as error: + raise ValueError('failed to load YAML from {}: {}'.format(path, error)) + channel = data['name'] + if channel in channels: + raise ValueError('multiple definitions for {}: {} and {}'.format(channel, paths[channel], path)) + paths[channel] = path + channels[channel] = data + for name, channel in sorted(channels.items()): + if not channel.get('feeder'): + continue + feeder = channel['feeder']['name'] + delay_string = channel['feeder']['delay'] + delay = parse_iso8601_delay(delay=delay_string) + version_filter = re.compile('^{}$'.format(channel['feeder'].get('filter', '.*'))) + feeder_data = channels[feeder] + unpromoted = set(feeder_data['versions']) - set(channel['versions']) - set(feeder_data.get('tombstones', {})) + candidates = set(v for v in unpromoted if version_filter.match(v)) + if not candidates: + continue + promotions = get_promotions(paths[feeder]) + now = datetime.datetime.now() + _LOGGER.info('considering promotions from {} to {} after {}'.format(feeder, name, delay_string)) + for version in sorted(candidates): + promotion = promotions[version] + version_delay = now - promotion['committer-time'] + if version_delay > delay: + _LOGGER.info(' recommended: {} ({})'.format(version, version_delay)) + _LOGGER.debug(' {}: Promote {} to {}'.format(paths[name].rsplit('.', 1)[0], version, name)) + _LOGGER.debug(' It was promoted the feeder {} by {} ({}, {}).'.format(feeder, promotion['hash'][:10], promotion['summary'], promotion['committer-time'].date().isoformat())) + else: + _LOGGER.info(' waiting: {} ({})'.format(version, version_delay)) + + +def get_promotions(path): + # https://git-scm.com/docs/git-blame#_the_porcelain_format + output = subprocess.check_output(['git', 'blame', '--first-parent', '--porcelain', path]).decode('utf-8') + commits = {} + lines = {} + for line in output.strip().split('\n'): + match = _GIT_BLAME_COMMIT_REGEXP.match(line) + if match: + commit = match.group('hash') + if commit not in commits: + commits[commit] = {'hash': commit} + continue + match = _GIT_BLAME_HEADER_REGEXP.match(line) + if match: + key = match.group('key') + value = match.group('value') + if key == 'committer-time': + commits[commit]['committer-time'] = datetime.datetime.fromtimestamp(int(value)) + else: + commits[commit][key] = value + continue + match = _GIT_BLAME_LINE_REGEXP.match(line) + if not match: + raise ValueError('unrecognized blame output for {} (blame line {}): {}'.format(path, i, line)) + lines[match.group('value')] = commit + promotions = {} + for line, commit in lines.items(): + if line.startswith('- '): + version = line[2:] + promotions[version] = commits[commit] + return promotions + + +if __name__ == '__main__': + stabilization_changes(directory='channels')