Bug 1781708: templates/container-storage: Add a "this is generated" note #1320

cgwalters · 2019-12-10T19:27:58Z

It's identical between master/worker, so move it into the
common pool.

Just a cleanup while looking at
#1319

templates/container-storage: Add a "this is generated" note

This is a best practice, and particularly in this case
when we're intending to modify a config file, it can't be identical
to the shipped version.

This avoids having libostree replace it with a newer version.

Closes: #1319

It's identical between master/worker, so move it into the common pool. Just a cleanup while looking at openshift#1319

cgwalters · 2019-12-10T19:36:44Z

If my theory is correct, then in this PR what I'll do is add a second commit which adds a dummy comment like # Generated from the Machine Config Operator's containerruntimeconfig - that will mark the file as modified and libostree won't replace it with a new default.

cgwalters · 2019-12-10T19:57:39Z

/test verify
/test unit
/test images

This is a best practice, and particularly in this case when we're intending to modify a config file, it can't be identical to the shipped version. This avoids having libostree replace it with a newer version. Closes: openshift#1319

cgwalters · 2019-12-10T20:48:39Z

Now with a new 🆕 commit which should fix the bug.
And hopefully CI is fixed now (some errors were unrelated).

cgwalters · 2019-12-10T21:26:55Z

Just logged into the cluster spun up by that e2e-aws and things are looking good there so far!

walters@toolbox ~/s/g/o/ostree> oc get machineconfigpool
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-5497b82f2b60d6a6e17bd16345e22dc1   True      False      False      3              3                   3                     0                      9m53s
worker   rendered-worker-4fe1fc25bc80ac51a692798946497e92   True      False      False      3              3                   3                     0                      9m53s

cgwalters · 2019-12-10T21:36:37Z

I just want to say it's pretty awesome that we're fixing a critical bug by adding a comment.

Funny enough, this bug is effectively the inverse of https://src.fedoraproject.org/rpms/selinux-policy/pull-request/2

vrutkovs · 2019-12-10T22:16:49Z

Clean install works, but gcp-upgrade still fails - not sure why though, I expected this to resolve upgrades as well

cgwalters · 2019-12-10T22:25:17Z

I think if the current release payload is broken, then upgrades can't pass. Let's retry once, but if it still fails I think we should pull out the /override 🔨 .
/test e2e-gcp-upgrade

cgwalters · 2019-12-10T22:26:14Z

One mystery here is how we got into a situation where the promoted payload cause this issue.

kikisdeliveryservice · 2019-12-10T22:35:41Z

Looking here it seems as tho it's failing bc of the storage.conf thing:
https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_machine-config-operator/1320/pull-ci-openshift-machine-config-operator-master-e2e-gcp-upgrade/605/artifacts/e2e-gcp-upgrade/pods/openshift-machine-config-operator_machine-config-daemon-4wfbw_machine-config-daemon.log

If that's that case we can either hold this until that is resolved (since this isn't emergency) or merge it bc it helps solve that (which doesn't seemlike that case?)

vrutkovs · 2019-12-10T22:36:12Z

IIUC the broken machine-os-content never got promoted to .ci (https://openshift-release.svc.ci.openshift.org/ 4.3.0.0.ci broken for ~5 days), but CI tests compose latest release from latest tags, so it attempts to update from a broken payload to a "fixed" payload. I assumed this would fix it too, not quite sure why it doesn't.

Perhaps its MCD on the host not being updated (this requires a new MCD RPM with this PR)?

cgwalters · 2019-12-11T00:38:18Z

If that's that case we can either hold this until that is resolved (since this isn't emergency)

The storage.conf conflict is exactly what this PR is addressing. Note this one passed e2e-aws, but other PRs on master aren't.

but CI tests compose latest release from latest tags, so it attempts to update from a broken payload to a "fixed" payload.

Hmm. Oh, wait, so CI runs are also pulling in the latest machine-os-content that's tagged, but before promotion? That seems like a bug if so.

Anyways, I want to more fully outline how this bug occurred. First, libostree has a semantic for config files shared with dpkg/rpm and other OS update systems in general, which is "keep modified config files, replace with latest version otherwise".

We previously tried to replace the file with its default version (i.e. a no-op), and that worked OK because the new version was the same thing.

But the MCD will validate that it's in its expected state, and as soon as we do an OS upgrade to a file with a different default version (which libostree will use because it didn't see it as modified locally), the MCD will fail validation.

Here we're just adding a comment to flag the file as modified and ensure that our version always wins, regardless of what changes in the package.

(Now we should also stop overwriting the package version and merge our changes, but that's a separate bug)

We need to backport this probably as far back as 4.2 or maybe even 4.1 - otherwise it's a time bomb waiting to bite whenever podman is updated in those streams.

(And we go degraded rather than just replacing the file with what we expect to avoid reboot loops which wouldn't occur in this case, but did in the past with multiple components writing to a config file)

cgwalters · 2019-12-11T00:51:04Z

I think the upgrade is failing because we need this PR backported to older branches too.

So, someone mind validating what I'm saying here and then dropping the /override e2e-gcp-upgrade (and a /lgtm) ?

kikisdeliveryservice · 2019-12-11T01:28:52Z

So I'm pretty sure this is failing in bootstrap (as always) bc the bootstrap version wouldn't have the changes the PR has it would be upgrading to this version. (ie a cluster comes up without images from this PR and then upgrades to this PR image). It will always fail until the fix is merged and future prs would be upgrading from the version with the fix. Right?

kikisdeliveryservice · 2019-12-11T02:02:18Z

Ok double checked my e2e understanding and yes: cluster comes up -> upgrades to image with PR's changes -> runs e2e. This PR is still getting stuck in the initial bootstrap (exactly how/where all the other e2e runs were failing before this PR) bc it doesn't have the fix. So talked to Colin and will override.

/lgtm

openshift-ci-robot · 2019-12-11T02:02:32Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cgwalters, kikisdeliveryservice

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [cgwalters,kikisdeliveryservice]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2019-12-11T02:03:32Z

/retest

Please review the full test history for this PR and help us cut down flakes.

kikisdeliveryservice · 2019-12-11T02:05:32Z

/override ci/prow/e2e-gcp-upgrade

openshift-ci-robot · 2019-12-11T02:05:46Z

@kikisdeliveryservice: Overrode contexts on behalf of kikisdeliveryservice: ci/prow/e2e-gcp-upgrade

Details

In response to this:

/override ci/prow/e2e-gcp-upgrade

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2019-12-11T02:19:00Z

@cgwalters: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
ci/prow/e2e-gcp-upgrade	`46c4e27`	link	`/test e2e-gcp-upgrade`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

vrutkovs · 2019-12-11T09:33:53Z

/cherry-pick release-4.3

openshift-cherrypick-robot · 2019-12-11T09:34:19Z

@vrutkovs: new pull request created: #1322

Details

In response to this:

/cherry-pick release-4.3

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

vrutkovs · 2019-12-11T09:46:44Z

/cherry-pick release-4.2

openshift-cherrypick-robot · 2019-12-11T09:46:55Z

@vrutkovs: new pull request created: #1323

Details

In response to this:

/cherry-pick release-4.2

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

vrutkovs · 2019-12-11T09:52:48Z

/cherry-pick release-4.1

openshift-cherrypick-robot · 2019-12-11T09:52:56Z

@vrutkovs: new pull request created: #1324

Details

In response to this:

/cherry-pick release-4.1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

cgwalters · 2019-12-11T12:01:28Z

/cherrypick release-4.3

openshift-cherrypick-robot · 2019-12-11T12:01:34Z

@cgwalters: failed to push cherry-picked changes in GitHub: pushing failed, output: "To https://github.com/openshift-cherrypick-robot/machine-config-operator\n ! [rejected] cherry-pick-1320-to-release-4.3 -> cherry-pick-1320-to-release-4.3 (non-fast-forward)\nerror: failed to push some refs to 'https://openshift-cherrypick-robot:[email protected]/openshift-cherrypick-robot/machine-config-operator'\nhint: Updates were rejected because the tip of your current branch is behind\nhint: its remote counterpart. Integrate the remote changes (e.g.\nhint: 'git pull ...') before pushing again.\nhint: See the 'Note about fast-forwards' in 'git push --help' for details.\n", error: exit status 1

Details

In response to this:

/cherrypick release-4.3

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

cgwalters · 2019-12-11T12:01:49Z

Oh nice thanks vrutkovs!

vrutkovs · 2019-12-11T17:13:32Z

/retitle Bug 1781708: templates/container-storage: Add a "this is generated" note

openshift-ci-robot · 2019-12-11T17:13:36Z

@cgwalters: Bugzilla bug 1781708 is in an unrecognized state (MODIFIED) and will not be moved to the MODIFIED state.

Details

In response to this:

Bug 1781708: templates/container-storage: Add a "this is generated" note

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

vrutkovs · 2019-12-11T17:15:43Z

/bugzilla refresh

openshift-ci-robot · 2019-12-11T17:15:48Z

@vrutkovs: All pull requests linked via external trackers have merged. Bugzilla bug 1781708 has been moved to the MODIFIED state.

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

vrutkovs · 2020-01-15T18:12:02Z

/cherry-pick fcos

openshift-cherrypick-robot · 2020-01-15T18:12:13Z

@vrutkovs: new pull request created: #1372

Details

In response to this:

/cherry-pick fcos

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

The machine-config operator had a bug where MachineConfig entries lead the machine-config daemon (MCD) to lay down a storage.conf that exactly matched the content installed by the containers-common RPM. On update, the RHCOS machine pivots to a new OSTree image (defined in the machine-os-content image referenced from the release image). Seeing storage.conf content that matched the old OSTree image, libostree replaced storage.conf with the version defined in the new OSTree image [1]. Then, when the MCD comes back up post-pivot, it sees the divergent storage.conf content and freaks out with logs like [2]: E1210 16:15:51.105286 11181 daemon.go:1350] content mismatch for file /etc/containers/storage.conf: and the machine-config operator goes Degraded=True with RequiredPoolsFailed "nodes are reporting degraded status on sync" [3]. The narrow machine-config fix was to annotate storage.conf that it writes, libostree doesn't touch the files on pivot [4]. This addresses the storage.conf case, but leaves the MCD vulnerable to other instances of "MCD writes exactly the OSTree contents to $FILE and expects it to remain untouched during an OSTree pivot that bumps the file". I'm not aware of a generic fix at the moment, although [5] might be related. You can guard a cluster against the narrow bug by setting a MachineConfig [6] or higher level object such as a ContainerRuntimeConfig [7] that will cause the MCD to write a storage.conf that diverges (even just by a comment or whitespace) from the OSTree original. Tracking the narrow fix through the various z streams: The 4.1 machine-config bug was introduced in d2c44d7 [8], which landed before 4.1.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.1.0-rc.0 | grep machine-config machine-config-controller https://github.com/openshift/machine-config-operator de9998eb37e90b3ee2fcfdbb3eda7ba26870ab6e machine-config-daemon https://github.com/openshift/machine-config-operator de9998eb37e90b3ee2fcfdbb3eda7ba26870ab6e machine-config-operator https://github.com/openshift/machine-config-operator de9998eb37e90b3ee2fcfdbb3eda7ba26870ab6e machine-config-server https://github.com/openshift/machine-config-operator de9998eb37e90b3ee2fcfdbb3eda7ba26870ab6e setup-etcd-environment https://github.com/openshift/machine-config-operator de9998eb37e90b3ee2fcfdbb3eda7ba26870ab6e $ git --no-pager log --oneline --first-parent de9998eb37 | grep d2c44d7 d2c44d7c Merge pull request openshift#330 from umohnani8/runtime The 4.1 machine-config fix was [9], landed in 1301934 [10], which is new in 4.1.34: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.1.34-x86_64 | grep machine-config machine-config-controller https://github.com/openshift/machine-config-operator f56d736e74af8fb0dc85c4b1ee3cc8d1d1f6600b machine-config-daemon https://github.com/openshift/machine-config-operator f56d736e74af8fb0dc85c4b1ee3cc8d1d1f6600b machine-config-operator https://github.com/openshift/machine-config-operator f56d736e74af8fb0dc85c4b1ee3cc8d1d1f6600b machine-config-server https://github.com/openshift/machine-config-operator f56d736e74af8fb0dc85c4b1ee3cc8d1d1f6600b setup-etcd-environment https://github.com/openshift/machine-config-operator f56d736e74af8fb0dc85c4b1ee3cc8d1d1f6600b $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.1.31-x86_64 | grep machine-config machine-config-controller https://github.com/openshift/machine-config-operator b38afe6e5b79a3e11e881429dc4c7c70e8784e84 machine-config-daemon https://github.com/openshift/machine-config-operator b38afe6e5b79a3e11e881429dc4c7c70e8784e84 machine-config-operator https://github.com/openshift/machine-config-operator b38afe6e5b79a3e11e881429dc4c7c70e8784e84 machine-config-server https://github.com/openshift/machine-config-operator b38afe6e5b79a3e11e881429dc4c7c70e8784e84 setup-etcd-environment https://github.com/openshift/machine-config-operator b38afe6e5b79a3e11e881429dc4c7c70e8784e84 $ git --no-pager log --oneline --first-parent -2 f56d736e74a f56d736e (origin/release-4.1) Merge pull request openshift#1147 from openshift-cherrypick-robot/cherry-pick-1114-to-release-4.1 1301934a Merge pull request openshift#1382 from vrutkovs/4.1-containers-conf-generated The 4.2 machine-config fix was [2], landed in bd358bb [11], which is new in 4.2.18: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.16-x86_64 | grep machine-config machine-config-operator https://github.com/openshift/machine-config-operator 31fed93186c9f84708f5cdfd0227ffe4f79b31cd $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.18-x86_64 | grep machine-config machine-config-operator https://github.com/openshift/machine-config-operator 9366460085b2a24d825380759f554769ec5ab4f9 $ git --no-pager log --oneline --first-parent -2 9366460085 93664600 Merge pull request openshift#1362 from rphillips/fixes/1787581_4.2 bd358bb7 Merge pull request openshift#1323 from openshift-cherrypick-robot/cherry-pick-1320-to-release-4.2 The 4.3 machine-config fix was [12], landed in 9fd53bd [13], which landed early enough for 4.3.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.0-rc.0-x86_64 | grep machine-config machine-config-operator https://github.com/openshift/machine-config-operator 23a6e6fb37e73501bc3216183ef5e6ebb15efc7a $ git --no-pager log --oneline --first-parent -8 23a6e6fb37 23a6e6fb Merge pull request openshift#1348 from openshift-cherrypick-robot/cherry-pick-1285-to-release-4.3 80c8aed7 Merge pull request openshift#1343 from retroflexer/cherry-pick-backup-restore-kube-static-resources 269990a3 Merge pull request openshift#1344 from openshift-cherrypick-robot/cherry-pick-1296-to-release-4.3 fd3ca395 Merge pull request openshift#1338 from runcom/fix-go-mod ba304dbb Merge pull request openshift#1333 from openshift-cherrypick-robot/cherry-pick-1278-to-release-4.3 787f3fa9 Merge pull request openshift#1332 from runcom/reserved-cpus-4.3 2b85d6ba Merge pull request openshift#1329 from openshift-cherrypick-robot/cherry-pick-1314-to-release-4.3 9fd53bd5 Merge pull request openshift#1322 from openshift-cherrypick-robot/cherry-pick-1320-to-release-4.3 The 4.4 machine-config fix was [3] which has landed before any 4.4 RCs have been cut. Even in 4.4, the generated note was the first content touch to this template: $ git --no-pager log --oneline --follow origin/release-4.4 -- templates/common/_base/files/container-storage.yaml 46c4e27a (origin/pr/1320) templates/container-storage: Add a "this is generated" note 47a6321c templates: Move container-storage.yaml into common/ 74ae3b31 (origin/pr/330) Add ContainerRuntime CRD and Controller (47a6321c was a pure rename). So the MCD has been annotating storage.conf since 4.1.34, 4.2.18, and all 4.3 and later releases. When has the RPM-installed storage.conf changed? Figuring this part out is a bit awkward, because we need to drill down machine-os-content -> RHCOS -> RPM -> file. For example, from 4.2.16 -> 4.2.18 [14]: $ oc image info --output json $(oc adm release info --image-for=machine-os-content quay.io/openshift-release-dev/ocp-release:4.2.16-x86_64) | jq -r .config.config.Labels.version 42.81.20200114.0 $ oc image info --output json $(oc adm release info --image-for=machine-os-content quay.io/openshift-release-dev/ocp-release:4.2.18-x86_64) | jq -r .config.config.Labels.version 42.81.20200203.1 $ ./differ.py --first-endpoint art --first-version 42.81.20200114.0 --second-endpoint art --second-version 42.81.20200203.1 | jq -r '.diff | keys | sort[]' cri-o ignition libarchive machine-config-daemon openshift-clients openshift-hyperkube sqlite-libs storage.conf is managed by the containers-common RPM, so no change from 4.2.16 to 4.2.18, and that update will safely pull in the fixed MCD without a surprising pivot change. Here are our changes to the RPM across the various z streams: $ for OCP in 4.1.1 4.1.23 4.1.24 4.1.31-x86_64 4.1.34-x86_64; do RHCOS="$(oc image info --output json $(oc adm release info --image-for=machine-os-content "quay.io/openshift-release-dev/ocp-release:${OCP}") | jq -r .config.config.Labels.version)"; COMMON="$(curl -s "https://releases-rhcos-art.cloud.privileged.psi.redhat.com/storage/releases/rhcos-4.1/${RHCOS}/commitmeta.json" | jq -r '.["rpmostree.rpmdb.pkglist"][] | select(.[0] == "containers-common") | .[2]')"; echo "${RHCOS} ${COMMON} ${OCP}"; done 410.8.20190606.0 0.1.32 4.1.1 410.8.20191030.0 0.1.32 4.1.23 410.81.20191112.2 0.1.37 4.1.24 410.81.20200114.0 0.1.37 4.1.31-x86_64 410.81.20200204.1 0.1.40 4.1.34-x86_64 $ for OCP in 4.2.0-rc.0 4.2.2 4.2.4 4.2.16-x86_64 4.2.18-x86_64 4.2.19-x86_64; do RHCOS="$(oc image info --output json $(oc adm release info --image-for=machine-os-content "quay.io/openshift-release-dev/ocp-release:${OCP}") | jq -r .config.config.Labels.version)"; COMMON="$(curl -s "https://releases-rhcos-art.cloud.privileged.psi.redhat.com/storage/releases/rhcos-4.2/${RHCOS}/commitmeta.json" | jq -r '.["rpmostree.rpmdb.pkglist"][] | select(.[0] == "containers-common") | .[2]')"; echo "${RHCOS} ${COMMON} ${OCP}"; done 42.80.20190930.1 0.1.32 4.2.0-rc.0 42.80.20191022.0 0.1.32 4.2.2 42.81.20191107.0 0.1.37 4.2.4 42.81.20200114.0 0.1.37 4.2.16-x86_64 42.81.20200203.1 0.1.37 4.2.18-x86_64 42.81.20200210.0 0.1.40 4.2.19-x86_64 $ for OCP in 4.3.0-rc.0-x86_64 4.3.3-x86_64; do RHCOS="$(oc image info --output json $(oc adm release info --image-for=machine-os-content "quay.io/openshift-release-dev/ocp-release:${OCP}") | jq -r .config.config.Labels.version)"; COMMON="$(curl -s "https://releases-rhcos-art.cloud.privileged.psi.redhat.com/storage/releases/rhcos-4.3/${RHCOS}/x86_64/commitmeta.json" | jq -r '.["rpmostree.rpmdb.pkglist"][] | select(.[0] == "containers-common") | .[2]')"; echo "${RHCOS} ${COMMON} ${OCP}"; done 43.81.202001072253.0 0.1.40 4.3.0-rc.0-x86_64 43.81.202002170853.0 0.1.40 4.3.3-x86_64 Fetching a source RPM for containers-common, e.g. from [15,16] shows the source packages coming from skopeo. Checking [17]: $ git --no-pager log --follow --oneline --stat=200 -M50% -- vendor/github.com/containers/storage/storage.conf afaa9e7f Bump github.com/containers/storage from 1.15.1 to 1.15.2 vendor/github.com/containers/storage/storage.conf | 3 --- 1 file changed, 3 deletions(-) 39ff039b Image encryption/decryption support in skopeo vendor/github.com/containers/storage/storage.conf | 44 +++++++++++++++++++++++++------------------- 1 file changed, 25 insertions(+), 19 deletions(-) 05ae513b Bump github.com/containers/buildah from 1.8.4 to 1.11.4 vendor/github.com/containers/storage/storage.conf | 7 ------- 1 file changed, 7 deletions(-) 700b3102 update github.com/containers/{image,storage} vendor/github.com/containers/storage/storage.conf | 8 ++++++++ 1 file changed, 8 insertions(+) 033b2902 migrate to go modules vendor/github.com/containers/storage/storage.conf | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+) $ git --no-pager log --follow --oneline --stat=200 -M50% 033b2902^ -- contrib/storage.conf fe259105 add storage.conf and manpage in contrib/ contrib/storage.conf | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) $ for HASH in fe259105 033b2902 700b3102 05ae513b 39ff039b afaa9e7f; do git describe --contains "${HASH}"; done v0.1.29~3^2 v0.1.38~14^2~2 v0.1.39~1 v0.1.41~25^2 v0.1.41~21^2 v0.1.41~12^2 So changes may have been made in 0.1.29 (when the file landed for the first time, likely from wherever we store post-Git patches), and were likely made in 0.1.38, 0.1.39, and 0.1.41. Comparing with our machine-os-content, that means vulnerable transitions are: * 4.1.* -> 4.1.34, since 4.1.31 -> 4.1.34 takes containers-common from 0.1.37 to 0.1.40, picking up the v0.1.38~14^2~2 and v0.1.39~1 bumps. There may be no safe way to get to 4.1.34. * 4.1.* -> 4.2... FIXME * 4.2.16 and earler -> 4.2.19, since 4.2.18 -> 4.2.19 takes containers-common from 0.1.37 to 0.1.40, picking up the v0.1.38~14^2~2 and v0.1.39~1 bumps. 4.2.16 and earlier -> 4.2.18 is fine, because there were no RPM-induced storage.conf bumps. 4.2.18 -> 4.2.* is fine, because 4.2.18 has the patched machine-config source. * 4.2.16 and earlier -> 4.3, since 4.2.18 -> 4.3 takes containers-common from 0.1.37 to 0.1.40, picking up the v0.1.38~14^2~2 and v0.1.39~1 bumps. 4.2.18 -> 4.3 is fine, because 4.2.18 has the patched machine-config source. * 4.3 -> 4.3 are fine, since they all have the patched machine-config source. So ideally this pull would block edges from 4.2.16 and earlier into 4.3. But because blocked-edges requires explicit to, I've just added the 4.3.0 blocker (other 4.3.z releases either already blocked 4.2.* or only give 4.2.18+ as update sources). I've also dropped 4.2.16 from the *-4.3 channels with a comment about this bug. There shouldn't be much pushback on pulling the edge, because users can still move from 4.2 to 4.3 via 4.2.19 -> 4.3.2. Also simplify the wording on the GCP bug 1793635, which remains unfixed. [1]: openshift/machine-config-operator#1320 (comment) [2]: https://bugzilla.redhat.com/show_bug.cgi?id=1782152#c5 [3]: https://bugzilla.redhat.com/show_bug.cgi?id=1781708#c0 [4]: https://github.com/openshift/machine-config-operator/pull/1320/files [5]: openshift/machine-config-operator#1190 [6]: https://github.com/openshift/machine-config-operator/blob/13f0dda734262c3edbd23c007e42b7704125e88f/docs/MachineConfiguration.md [7]: https://github.com/openshift/machine-config-operator/blob/13f0dda734262c3edbd23c007e42b7704125e88f/docs/ContainerRuntimeConfigDesign.md [8]: openshift/machine-config-operator#330 (comment) [9]: https://bugzilla.redhat.com/show_bug.cgi?id=1782153 [10]: openshift/machine-config-operator#1382 (comment) [11]: openshift/machine-config-operator#1323 (comment) [12]: https://bugzilla.redhat.com/show_bug.cgi?id=1782149 [13]: openshift/machine-config-operator#1322 (comment) [14]: https://gitlab.cee.redhat.com/coretools/differ Internal link, sorry :/ But you can also browse the history at: https://releases-rhcos-art.cloud.privileged.psi.redhat.com/?stream=releases/rhcos-4.2&release=42.81.20200114.0 etc. [15]: https://access.redhat.com/downloads/content/290/ver=4.2/rhel---8/4.2.0/x86_64/packages [16]: https://access.redhat.com/downloads/content/rhel---8/x86_64/8841/containers-common/0.1.32-5.git1715c90.el8/x86_64/fd431d51/package [17]: https://github.com/containers/skopeo/

The machine-config operator had a bug where MachineConfig entries lead the machine-config daemon (MCD) to lay down a storage.conf that exactly matched the content installed by the containers-common RPM. On update, the RHCOS machine pivots to a new OSTree image (defined in the machine-os-content image referenced from the release image). Seeing storage.conf content that matched the old OSTree image, libostree replaced storage.conf with the version defined in the new OSTree image [1]. Then, when the MCD comes back up post-pivot, it sees the divergent storage.conf content and freaks out with logs like [2]: E1210 16:15:51.105286 11181 daemon.go:1350] content mismatch for file /etc/containers/storage.conf: and the machine-config operator goes Degraded=True with RequiredPoolsFailed "nodes are reporting degraded status on sync" [3]. The narrow machine-config fix was to annotate storage.conf that it writes, libostree doesn't touch the files on pivot [4]. This addresses the storage.conf case, but leaves the MCD vulnerable to other instances of "MCD writes exactly the OSTree contents to $FILE and expects it to remain untouched during an OSTree pivot that bumps the file". I'm not aware of a generic fix at the moment, although [5] might be related. You can guard a cluster against the narrow bug by setting a MachineConfig [6] or higher level object such as a ContainerRuntimeConfig [7] that will cause the MCD to write a storage.conf that diverges (even just by a comment or whitespace) from the OSTree original. Tracking the narrow fix through the various z streams: The 4.1 machine-config bug was introduced in d2c44d7 [8], which landed before 4.1.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.1.0-rc.0 | grep machine-config machine-config-controller https://github.com/openshift/machine-config-operator de9998eb37e90b3ee2fcfdbb3eda7ba26870ab6e machine-config-daemon https://github.com/openshift/machine-config-operator de9998eb37e90b3ee2fcfdbb3eda7ba26870ab6e machine-config-operator https://github.com/openshift/machine-config-operator de9998eb37e90b3ee2fcfdbb3eda7ba26870ab6e machine-config-server https://github.com/openshift/machine-config-operator de9998eb37e90b3ee2fcfdbb3eda7ba26870ab6e setup-etcd-environment https://github.com/openshift/machine-config-operator de9998eb37e90b3ee2fcfdbb3eda7ba26870ab6e $ git --no-pager log --oneline --first-parent de9998eb37 | grep d2c44d7 d2c44d7c Merge pull request openshift#330 from umohnani8/runtime The 4.1 machine-config fix was [9], landed in 1301934 [10], which is new in 4.1.34: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.1.34-x86_64 | grep machine-config machine-config-controller https://github.com/openshift/machine-config-operator f56d736e74af8fb0dc85c4b1ee3cc8d1d1f6600b machine-config-daemon https://github.com/openshift/machine-config-operator f56d736e74af8fb0dc85c4b1ee3cc8d1d1f6600b machine-config-operator https://github.com/openshift/machine-config-operator f56d736e74af8fb0dc85c4b1ee3cc8d1d1f6600b machine-config-server https://github.com/openshift/machine-config-operator f56d736e74af8fb0dc85c4b1ee3cc8d1d1f6600b setup-etcd-environment https://github.com/openshift/machine-config-operator f56d736e74af8fb0dc85c4b1ee3cc8d1d1f6600b $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.1.31-x86_64 | grep machine-config machine-config-controller https://github.com/openshift/machine-config-operator b38afe6e5b79a3e11e881429dc4c7c70e8784e84 machine-config-daemon https://github.com/openshift/machine-config-operator b38afe6e5b79a3e11e881429dc4c7c70e8784e84 machine-config-operator https://github.com/openshift/machine-config-operator b38afe6e5b79a3e11e881429dc4c7c70e8784e84 machine-config-server https://github.com/openshift/machine-config-operator b38afe6e5b79a3e11e881429dc4c7c70e8784e84 setup-etcd-environment https://github.com/openshift/machine-config-operator b38afe6e5b79a3e11e881429dc4c7c70e8784e84 $ git --no-pager log --oneline --first-parent -2 f56d736e74a f56d736e (origin/release-4.1) Merge pull request openshift#1147 from openshift-cherrypick-robot/cherry-pick-1114-to-release-4.1 1301934a Merge pull request openshift#1382 from vrutkovs/4.1-containers-conf-generated The 4.2 machine-config fix was [2], landed in bd358bb [11], which is new in 4.2.18: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.16-x86_64 | grep machine-config machine-config-operator https://github.com/openshift/machine-config-operator 31fed93186c9f84708f5cdfd0227ffe4f79b31cd $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.18-x86_64 | grep machine-config machine-config-operator https://github.com/openshift/machine-config-operator 9366460085b2a24d825380759f554769ec5ab4f9 $ git --no-pager log --oneline --first-parent -2 9366460085 93664600 Merge pull request openshift#1362 from rphillips/fixes/1787581_4.2 bd358bb7 Merge pull request openshift#1323 from openshift-cherrypick-robot/cherry-pick-1320-to-release-4.2 The 4.3 machine-config fix was [12], landed in 9fd53bd [13], which landed early enough for 4.3.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.0-rc.0-x86_64 | grep machine-config machine-config-operator https://github.com/openshift/machine-config-operator 23a6e6fb37e73501bc3216183ef5e6ebb15efc7a $ git --no-pager log --oneline --first-parent -8 23a6e6fb37 23a6e6fb Merge pull request openshift#1348 from openshift-cherrypick-robot/cherry-pick-1285-to-release-4.3 80c8aed7 Merge pull request openshift#1343 from retroflexer/cherry-pick-backup-restore-kube-static-resources 269990a3 Merge pull request openshift#1344 from openshift-cherrypick-robot/cherry-pick-1296-to-release-4.3 fd3ca395 Merge pull request openshift#1338 from runcom/fix-go-mod ba304dbb Merge pull request openshift#1333 from openshift-cherrypick-robot/cherry-pick-1278-to-release-4.3 787f3fa9 Merge pull request openshift#1332 from runcom/reserved-cpus-4.3 2b85d6ba Merge pull request openshift#1329 from openshift-cherrypick-robot/cherry-pick-1314-to-release-4.3 9fd53bd5 Merge pull request openshift#1322 from openshift-cherrypick-robot/cherry-pick-1320-to-release-4.3 The 4.4 machine-config fix was [3] which has landed before any 4.4 RCs have been cut. Even in 4.4, the generated note was the first content touch to this template: $ git --no-pager log --oneline --follow origin/release-4.4 -- templates/common/_base/files/container-storage.yaml 46c4e27a (origin/pr/1320) templates/container-storage: Add a "this is generated" note 47a6321c templates: Move container-storage.yaml into common/ 74ae3b31 (origin/pr/330) Add ContainerRuntime CRD and Controller (47a6321c was a pure rename). So the MCD has been annotating storage.conf since 4.1.34, 4.2.18, and all 4.3 and later releases. When has the RPM-installed storage.conf changed? Figuring this part out is a bit awkward, because we need to drill down machine-os-content -> RHCOS -> RPM -> file. For example, from 4.2.16 -> 4.2.18 [14]: $ oc image info --output json $(oc adm release info --image-for=machine-os-content quay.io/openshift-release-dev/ocp-release:4.2.16-x86_64) | jq -r .config.config.Labels.version 42.81.20200114.0 $ oc image info --output json $(oc adm release info --image-for=machine-os-content quay.io/openshift-release-dev/ocp-release:4.2.18-x86_64) | jq -r .config.config.Labels.version 42.81.20200203.1 $ ./differ.py --first-endpoint art --first-version 42.81.20200114.0 --second-endpoint art --second-version 42.81.20200203.1 | jq -r '.diff | keys | sort[]' cri-o ignition libarchive machine-config-daemon openshift-clients openshift-hyperkube sqlite-libs storage.conf is managed by the containers-common RPM, so no change from 4.2.16 to 4.2.18, and that update will safely pull in the fixed MCD without a surprising pivot change. Here are our changes to the RPM across the various z streams: $ for OCP in 4.1.1 4.1.23 4.1.24 4.1.31-x86_64 4.1.34-x86_64; do RHCOS="$(oc image info --output json $(oc adm release info --image-for=machine-os-content "quay.io/openshift-release-dev/ocp-release:${OCP}") | jq -r .config.config.Labels.version)"; COMMON="$(curl -s "https://releases-rhcos-art.cloud.privileged.psi.redhat.com/storage/releases/rhcos-4.1/${RHCOS}/commitmeta.json" | jq -r '.["rpmostree.rpmdb.pkglist"][] | select(.[0] == "containers-common") | .[2]')"; echo "${RHCOS} ${COMMON} ${OCP}"; done 410.8.20190606.0 0.1.32 4.1.1 410.8.20191030.0 0.1.32 4.1.23 410.81.20191112.2 0.1.37 4.1.24 410.81.20200114.0 0.1.37 4.1.31-x86_64 410.81.20200204.1 0.1.40 4.1.34-x86_64 $ for OCP in 4.2.0-rc.0 4.2.2 4.2.4 4.2.16-x86_64 4.2.18-x86_64 4.2.19-x86_64; do RHCOS="$(oc image info --output json $(oc adm release info --image-for=machine-os-content "quay.io/openshift-release-dev/ocp-release:${OCP}") | jq -r .config.config.Labels.version)"; COMMON="$(curl -s "https://releases-rhcos-art.cloud.privileged.psi.redhat.com/storage/releases/rhcos-4.2/${RHCOS}/commitmeta.json" | jq -r '.["rpmostree.rpmdb.pkglist"][] | select(.[0] == "containers-common") | .[2]')"; echo "${RHCOS} ${COMMON} ${OCP}"; done 42.80.20190930.1 0.1.32 4.2.0-rc.0 42.80.20191022.0 0.1.32 4.2.2 42.81.20191107.0 0.1.37 4.2.4 42.81.20200114.0 0.1.37 4.2.16-x86_64 42.81.20200203.1 0.1.37 4.2.18-x86_64 42.81.20200210.0 0.1.40 4.2.19-x86_64 $ for OCP in 4.3.0-rc.0-x86_64 4.3.3-x86_64; do RHCOS="$(oc image info --output json $(oc adm release info --image-for=machine-os-content "quay.io/openshift-release-dev/ocp-release:${OCP}") | jq -r .config.config.Labels.version)"; COMMON="$(curl -s "https://releases-rhcos-art.cloud.privileged.psi.redhat.com/storage/releases/rhcos-4.3/${RHCOS}/x86_64/commitmeta.json" | jq -r '.["rpmostree.rpmdb.pkglist"][] | select(.[0] == "containers-common") | .[2]')"; echo "${RHCOS} ${COMMON} ${OCP}"; done 43.81.202001072253.0 0.1.40 4.3.0-rc.0-x86_64 43.81.202002170853.0 0.1.40 4.3.3-x86_64 Fetching a source RPM for containers-common, e.g. from [15,16] shows the source packages coming from skopeo. Checking [17]: $ git --no-pager log --follow --oneline --stat=200 -M50% -- vendor/github.com/containers/storage/storage.conf afaa9e7f Bump github.com/containers/storage from 1.15.1 to 1.15.2 vendor/github.com/containers/storage/storage.conf | 3 --- 1 file changed, 3 deletions(-) 39ff039b Image encryption/decryption support in skopeo vendor/github.com/containers/storage/storage.conf | 44 +++++++++++++++++++++++++------------------- 1 file changed, 25 insertions(+), 19 deletions(-) 05ae513b Bump github.com/containers/buildah from 1.8.4 to 1.11.4 vendor/github.com/containers/storage/storage.conf | 7 ------- 1 file changed, 7 deletions(-) 700b3102 update github.com/containers/{image,storage} vendor/github.com/containers/storage/storage.conf | 8 ++++++++ 1 file changed, 8 insertions(+) 033b2902 migrate to go modules vendor/github.com/containers/storage/storage.conf | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+) $ git --no-pager log --follow --oneline --stat=200 -M50% 033b2902^ -- contrib/storage.conf fe259105 add storage.conf and manpage in contrib/ contrib/storage.conf | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) $ for HASH in fe259105 033b2902 700b3102 05ae513b 39ff039b afaa9e7f; do git describe --contains "${HASH}"; done v0.1.29~3^2 v0.1.38~14^2~2 v0.1.39~1 v0.1.41~25^2 v0.1.41~21^2 v0.1.41~12^2 So changes may have been made in 0.1.29 (when the file landed for the first time, likely from wherever we store post-Git patches), and were likely made in 0.1.38, 0.1.39, and 0.1.41. However, the skopeo and derivative containers-common RPMs may have had patched versions of the file tracked in dist-git [18]. Comparing the dist-git 4.1 tip with the machine-config template: $ git -C containers/skopeo remote -v | grep 'dist-git.*fetch' dist-git git://pkgs.devel.redhat.com/rpms/skopeo.git (fetch) $ git --no-pager -C containers/skopeo log --date=short --format='%ad %h %s' -2 dist-git/rhaos-4.1-rhel-8 -- storage.conf 2018-07-18 3757b210 add statx to seccomp.json to containers-config add seccomp.json to containers-config 2017-11-08 284f9024 Force storage.conf to default to overlay $ git --no-pager -C containers/skopeo grep '^Version:' 3757b210 3757b210:skopeo.spec:Version: 0.1.31 $ diff -U3 <(git -C containers/skopeo cat-file -p 3757b210:storage.conf) <(sed 's/^ //' openshift/machine-config-operator/templates/common/_base/files/container-storage.yaml)--- /dev/fd/63 2020-02-20 01:13:48.073704685 -0800 +++ /dev/fd/62 2020-02-20 01:13:48.073704685 -0800 @@ -1,3 +1,10 @@ +filesystem: "root" +mode: 0644 +path: "/etc/containers/storage.conf" +contents: + inline: | +# This file is generated by the Machine Config Operator's containerruntimeconfig controller. +# # storage.conf is the configuration file for all tools # that share the containers/storage libraries # See man 5 containers-storage.conf for more information So the machine-config master (5ed0aee72c) only differs from the old 0.1.31 RPM storage.conf by the "file is generated" marker. There does not seem to be any 4.2-specific content. Presumably they're using the same rhaos-4.1-rhel-8 RPMs. 4.3 has some changes: $ git --no-pager log --date=short --format='%ad %h %s' -2 --stat=80 dist-git/rhaos-4.3-rhel-8 -- storage.conf 2019-12-09 4a131916 skopeo-0.1.40-2.el8 storage.conf | 39 +++++++++++++++++++++++++++++---------- 1 file changed, 29 insertions(+), 10 deletions(-) 2019-10-08 13a4ce10 skopeo-1:0.1.40-0.1.gitf72e39f storage.conf | 114 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 114 insertions(+) So it looks like we can ignore the dev skopeo repository, focus on the dist-git skopeo repository, and say that before 0.1.40-2.el8 we had a version of storage.conf in the RPMs that matched the unpatched machine-config templates, and with 0.1.40-2.el8 and later the RPMs had different content. Sanity checking via [19,20]: $ diff -U3 <(rpm2cpio containers-common-0.1.32-5.git1715c90.el8.x86_64.rpm | cpio -i --to-stdout ./etc/containers/storage.conf 2>/dev/null) <(sed 's/^ //' templates/common/_base/files/container-storage.yaml) --- /dev/fd/63 2020-02-20 01:36:23.031918968 -0800 +++ /dev/fd/62 2020-02-20 01:36:23.031918968 -0800 @@ -1,3 +1,10 @@ +filesystem: "root" +mode: 0644 +path: "/etc/containers/storage.conf" +contents: + inline: | +# This file is generated by the Machine Config Operator's containerruntimeconfig controller. +# # storage.conf is the configuration file for all tools # that share the containers/storage libraries # See man 5 containers-storage.conf for more information but I'm not clear on why the product pages are claiming containers-common-0.1.32 for 4.1.34 [19,20]. FIXME Comparing with our machine-os-content, that means vulnerable transitions are: * 4.1.* -> 4.1.34, since 4.1.31 -> 4.1.34 takes containers-common from 0.1.37 to 0.1.40, picking up the v0.1.38~14^2~2 and v0.1.39~1 bumps. There may be no safe way to get to 4.1.34. * 4.1.* -> 4.2... FIXME * 4.2.16 and earler -> 4.2.19, since 4.2.18 -> 4.2.19 takes containers-common from 0.1.37 to 0.1.40, picking up the v0.1.38~14^2~2 and v0.1.39~1 bumps. 4.2.16 and earlier -> 4.2.18 is fine, because there were no RPM-induced storage.conf bumps. 4.2.18 -> 4.2.* is fine, because 4.2.18 has the patched machine-config source. * 4.2.16 and earlier -> 4.3, since 4.2.18 -> 4.3 takes containers-common from 0.1.37 to 0.1.40, picking up the v0.1.38~14^2~2 and v0.1.39~1 bumps. 4.2.18 -> 4.3 is fine, because 4.2.18 has the patched machine-config source. * 4.3 -> 4.3 are fine, since they all have the patched machine-config source. So ideally this pull would block edges from 4.2.16 and earlier into 4.3. But because blocked-edges requires explicit to, I've just added the 4.3.0 blocker (other 4.3.z releases either already blocked 4.2.* or only give 4.2.18+ as update sources). I've also dropped 4.2.16 from the *-4.3 channels with a comment about this bug. There shouldn't be much pushback on pulling the edge, because users can still move from 4.2 to 4.3 via 4.2.19 -> 4.3.2. Also simplify the wording on the GCP bug 1793635, which remains unfixed. [1]: openshift/machine-config-operator#1320 (comment) [2]: https://bugzilla.redhat.com/show_bug.cgi?id=1782152#c5 [3]: https://bugzilla.redhat.com/show_bug.cgi?id=1781708#c0 [4]: https://github.com/openshift/machine-config-operator/pull/1320/files [5]: openshift/machine-config-operator#1190 [6]: https://github.com/openshift/machine-config-operator/blob/13f0dda734262c3edbd23c007e42b7704125e88f/docs/MachineConfiguration.md [7]: https://github.com/openshift/machine-config-operator/blob/13f0dda734262c3edbd23c007e42b7704125e88f/docs/ContainerRuntimeConfigDesign.md [8]: openshift/machine-config-operator#330 (comment) [9]: https://bugzilla.redhat.com/show_bug.cgi?id=1782153 [10]: openshift/machine-config-operator#1382 (comment) [11]: openshift/machine-config-operator#1323 (comment) [12]: https://bugzilla.redhat.com/show_bug.cgi?id=1782149 [13]: openshift/machine-config-operator#1322 (comment) [14]: https://gitlab.cee.redhat.com/coretools/differ Internal link, sorry :/ But you can also browse the history at: https://releases-rhcos-art.cloud.privileged.psi.redhat.com/?stream=releases/rhcos-4.2&release=42.81.20200114.0 etc. [15]: https://access.redhat.com/downloads/content/290/ver=4.2/rhel---8/4.2.0/x86_64/packages [16]: https://access.redhat.com/downloads/content/rhel---8/x86_64/8841/containers-common/0.1.32-5.git1715c90.el8/x86_64/fd431d51/package [17]: https://github.com/containers/skopeo/ [18]: http://pkgs.devel.redhat.com/cgit/rpms/skopeo/ [19]: https://access.redhat.com/downloads/content/290/ver=4.1/rhel---8/4.1.34/x86_64/packages [20]: https://access.redhat.com/downloads/content/rhel---8/x86_64/8384/containers-common/0.1.32-5.git1715c90.el8/x86_64/fd431d51/package

The machine-config operator had a bug where MachineConfig entries lead the machine-config daemon (MCD) to lay down a storage.conf that exactly matched the content installed by the containers-common RPM. On update, the RHCOS machine pivots to a new OSTree image (defined in the machine-os-content image referenced from the release image). Seeing storage.conf content that matched the old OSTree image, libostree replaced storage.conf with the version defined in the new OSTree image [1]. Then, when the MCD comes back up post-pivot, it sees the divergent storage.conf content and freaks out with logs like [2]: E1210 16:15:51.105286 11181 daemon.go:1350] content mismatch for file /etc/containers/storage.conf: and the machine-config operator goes Degraded=True with RequiredPoolsFailed "nodes are reporting degraded status on sync" [3]. The narrow machine-config fix was to annotate storage.conf that it writes, libostree doesn't touch the files on pivot [4]. This addresses the storage.conf case, but leaves the MCD vulnerable to other instances of "MCD writes exactly the OSTree contents to $FILE and expects it to remain untouched during an OSTree pivot that bumps the file". I'm not aware of a generic fix at the moment, although [5] might be related. You can guard a cluster against the narrow bug by setting a MachineConfig [6] or higher level object such as a ContainerRuntimeConfig [7] that will cause the MCD to write a storage.conf that diverges (even just by a comment or whitespace) from the OSTree original. Tracking the narrow fix through the various z streams: The 4.1 machine-config bug was introduced in d2c44d7 [8], which landed before 4.1.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.1.0-rc.0 | grep machine-config machine-config-controller https://github.com/openshift/machine-config-operator de9998eb37e90b3ee2fcfdbb3eda7ba26870ab6e machine-config-daemon https://github.com/openshift/machine-config-operator de9998eb37e90b3ee2fcfdbb3eda7ba26870ab6e machine-config-operator https://github.com/openshift/machine-config-operator de9998eb37e90b3ee2fcfdbb3eda7ba26870ab6e machine-config-server https://github.com/openshift/machine-config-operator de9998eb37e90b3ee2fcfdbb3eda7ba26870ab6e setup-etcd-environment https://github.com/openshift/machine-config-operator de9998eb37e90b3ee2fcfdbb3eda7ba26870ab6e $ git --no-pager log --oneline --first-parent de9998eb37 | grep d2c44d7 d2c44d7c Merge pull request openshift#330 from umohnani8/runtime The 4.1 machine-config fix was [9], landed in 1301934 [10], which is new in 4.1.34: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.1.34-x86_64 | grep machine-config machine-config-controller https://github.com/openshift/machine-config-operator f56d736e74af8fb0dc85c4b1ee3cc8d1d1f6600b machine-config-daemon https://github.com/openshift/machine-config-operator f56d736e74af8fb0dc85c4b1ee3cc8d1d1f6600b machine-config-operator https://github.com/openshift/machine-config-operator f56d736e74af8fb0dc85c4b1ee3cc8d1d1f6600b machine-config-server https://github.com/openshift/machine-config-operator f56d736e74af8fb0dc85c4b1ee3cc8d1d1f6600b setup-etcd-environment https://github.com/openshift/machine-config-operator f56d736e74af8fb0dc85c4b1ee3cc8d1d1f6600b $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.1.31-x86_64 | grep machine-config machine-config-controller https://github.com/openshift/machine-config-operator b38afe6e5b79a3e11e881429dc4c7c70e8784e84 machine-config-daemon https://github.com/openshift/machine-config-operator b38afe6e5b79a3e11e881429dc4c7c70e8784e84 machine-config-operator https://github.com/openshift/machine-config-operator b38afe6e5b79a3e11e881429dc4c7c70e8784e84 machine-config-server https://github.com/openshift/machine-config-operator b38afe6e5b79a3e11e881429dc4c7c70e8784e84 setup-etcd-environment https://github.com/openshift/machine-config-operator b38afe6e5b79a3e11e881429dc4c7c70e8784e84 $ git --no-pager log --oneline --first-parent -2 f56d736e74a f56d736e (origin/release-4.1) Merge pull request openshift#1147 from openshift-cherrypick-robot/cherry-pick-1114-to-release-4.1 1301934a Merge pull request openshift#1382 from vrutkovs/4.1-containers-conf-generated The 4.2 machine-config fix was [2], landed in bd358bb [11], which is new in 4.2.18: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.16-x86_64 | grep machine-config machine-config-operator https://github.com/openshift/machine-config-operator 31fed93186c9f84708f5cdfd0227ffe4f79b31cd $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.18-x86_64 | grep machine-config machine-config-operator https://github.com/openshift/machine-config-operator 9366460085b2a24d825380759f554769ec5ab4f9 $ git --no-pager log --oneline --first-parent -2 9366460085 93664600 Merge pull request openshift#1362 from rphillips/fixes/1787581_4.2 bd358bb7 Merge pull request openshift#1323 from openshift-cherrypick-robot/cherry-pick-1320-to-release-4.2 The 4.3 machine-config fix was [12], landed in 9fd53bd [13], which landed early enough for 4.3.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.0-rc.0-x86_64 | grep machine-config machine-config-operator https://github.com/openshift/machine-config-operator 23a6e6fb37e73501bc3216183ef5e6ebb15efc7a $ git --no-pager log --oneline --first-parent -8 23a6e6fb37 23a6e6fb Merge pull request openshift#1348 from openshift-cherrypick-robot/cherry-pick-1285-to-release-4.3 80c8aed7 Merge pull request openshift#1343 from retroflexer/cherry-pick-backup-restore-kube-static-resources 269990a3 Merge pull request openshift#1344 from openshift-cherrypick-robot/cherry-pick-1296-to-release-4.3 fd3ca395 Merge pull request openshift#1338 from runcom/fix-go-mod ba304dbb Merge pull request openshift#1333 from openshift-cherrypick-robot/cherry-pick-1278-to-release-4.3 787f3fa9 Merge pull request openshift#1332 from runcom/reserved-cpus-4.3 2b85d6ba Merge pull request openshift#1329 from openshift-cherrypick-robot/cherry-pick-1314-to-release-4.3 9fd53bd5 Merge pull request openshift#1322 from openshift-cherrypick-robot/cherry-pick-1320-to-release-4.3 The 4.4 machine-config fix was [3] which has landed before any 4.4 RCs have been cut. Even in 4.4, the generated note was the first content touch to this template: $ git --no-pager log --oneline --follow origin/release-4.4 -- templates/common/_base/files/container-storage.yaml 46c4e27a (origin/pr/1320) templates/container-storage: Add a "this is generated" note 47a6321c templates: Move container-storage.yaml into common/ 74ae3b31 (origin/pr/330) Add ContainerRuntime CRD and Controller (47a6321c was a pure rename). So the MCD has been annotating storage.conf since 4.1.34, 4.2.18, and all 4.3 and later releases. When has the RPM-installed storage.conf changed? Figuring this part out is a bit awkward, because we need to drill down machine-os-content -> RHCOS -> RPM -> file. For example, from 4.2.16 -> 4.2.18 [14]: $ oc image info --output json $(oc adm release info --image-for=machine-os-content quay.io/openshift-release-dev/ocp-release:4.2.16-x86_64) | jq -r .config.config.Labels.version 42.81.20200114.0 $ oc image info --output json $(oc adm release info --image-for=machine-os-content quay.io/openshift-release-dev/ocp-release:4.2.18-x86_64) | jq -r .config.config.Labels.version 42.81.20200203.1 $ ./differ.py --first-endpoint art --first-version 42.81.20200114.0 --second-endpoint art --second-version 42.81.20200203.1 | jq -r '.diff | keys | sort[]' cri-o ignition libarchive machine-config-daemon openshift-clients openshift-hyperkube sqlite-libs storage.conf is managed by the containers-common RPM, so no change from 4.2.16 to 4.2.18, and that update will safely pull in the fixed MCD without a surprising pivot change. Here are our changes to the RPM across the various z streams: $ for OCP in 4.1.1 4.1.16 4.1.17 4.1.23 4.1.24 4.1.28 4.1.29 4.1.31-x86_64 4.1.34-x86_64; do RHCOS="$(oc image info --output json $(oc adm release info --image-for=machine-os-content "quay.io/openshift-release-dev/ocp-release:${OCP}") | jq -r .config.config.Labels.version)"; COMMON="$(curl -s "https://releases-rhcos-art.cloud.privileged.psi.redhat.com/storage/releases/rhcos-4.1/${RHCOS}/commitmeta.json" | jq -c '.["rpmostree.rpmdb.pkglist"][] | select(.[0] == "containers-common") | .')"; echo "${COMMON} ${RHCOS} ${OCP}"; done ["containers-common","1","0.1.32","4.git1715c90.el8","x86_64"] 410.8.20190606.0 4.1.1 ["containers-common","1","0.1.32","4.git1715c90.el8","x86_64"] 410.8.20190910.1 4.1.16 ["containers-common","1","0.1.32","5.git1715c90.el8","x86_64"] 410.8.20190918.0 4.1.17 ["containers-common","1","0.1.32","5.git1715c90.el8","x86_64"] 410.8.20191030.0 4.1.23 ["containers-common","1","0.1.37","5.module+el8.1.0+4240+893c1ab8","x86_64"] 410.81.20191112.2 4.1.24 ["containers-common","1","0.1.37","5.module+el8.1.0+4240+893c1ab8","x86_64"] 410.81.20191210.0 4.1.28 ["containers-common","1","0.1.37","6.module+el8.1.0+4876+e678a192","x86_64"] 410.81.20191223.0 4.1.29 ["containers-common","1","0.1.37","6.module+el8.1.0+4876+e678a192","x86_64"] 410.81.20200114.0 4.1.31-x86_64 ["containers-common","1","0.1.40","8.module+el8.1.1+5351+506397b0","x86_64"] 410.81.20200204.1 4.1.34-x86_64 $ for OCP in 4.2.0-rc.0 4.2.2 4.2.4 4.2.12 4.2.13 4.2.18-x86_64 4.2.19-x86_64 4.2.20-x86_64; do RHCOS="$(oc image info --output json $(oc adm release info --image-for=machine-os-content "quay.io/openshift-release-dev/ocp-release:${OCP}") | jq -r .config.config.Labels.version)"; COMMON="$(curl -s "https://releases-rhcos-art.cloud.privileged.psi.redhat.com/storage/releases/rhcos-4.2/${RHCOS}/commitmeta.json" | jq -c '.["rpmostree.rpmdb.pkglist"][] | select(.[0] == "containers-common") | .')"; echo "${COMMON} ${RHCOS} ${OCP}"; done ["containers-common","1","0.1.32","5.git1715c90.el8","x86_64"] 42.80.20190930.1 4.2.0-rc.0 ["containers-common","1","0.1.32","5.git1715c90.el8","x86_64"] 42.80.20191022.0 4.2.2 ["containers-common","1","0.1.37","5.module+el8.1.0+4240+893c1ab8","x86_64"] 42.81.20191107.0 4.2.4 ["containers-common","1","0.1.37","5.module+el8.1.0+4240+893c1ab8","x86_64"] 42.81.20191210.1 4.2.12 ["containers-common","1","0.1.37","6.module+el8.1.0+4876+e678a192","x86_64"] 42.81.20191223.0 4.2.13 ["containers-common","1","0.1.37","6.module+el8.1.0+4876+e678a192","x86_64"] 42.81.20200203.1 4.2.18-x86_64 ["containers-common","1","0.1.40","8.module+el8.1.1+5351+506397b0","x86_64"] 42.81.20200210.0 4.2.19-x86_64 ["containers-common","1","0.1.40","8.module+el8.1.1+5351+506397b0","x86_64"] 42.81.20200217.0 4.2.20-x86_64 $ for OCP in 4.3.0-rc.0-x86_64 4.3.0-x86_64 4.3.1-x86_64 4.3.2-x86_64 4.3.3-x86_64; do RHCOS="$(oc image info --output json $(oc adm release info --image-for=machine-os-content "quay.io/openshift-release-dev/ocp-release:${OCP}") | jq -r .config.config.Labels.version)"; COMMON="$(curl -s "https://releases-rhcos-art.cloud.privileged.psi.redhat.com/storage/releases/rhcos-4.3/${RHCOS}/x86_64/commitmeta.json" | jq -c '.["rpmostree.rpmdb.pkglist"][] | select(.[0] == "containers-common") | .')"; echo "${COMMON} ${RHCOS} ${OCP}"; done ["containers-common","1","0.1.40","2.el8","x86_64"] 43.81.202001072253.0 4.3.0-rc.0-x86_64 ["containers-common","1","0.1.40","2.el8","x86_64"] 43.81.202001142154.0 4.3.0-x86_64 ["containers-common","1","0.1.40","3.rhaos.el8","x86_64"] 43.81.202002032142.0 4.3.1-x86_64 ["containers-common","1","0.1.40","8.module+el8.1.1+5351+506397b0","x86_64"] 43.81.202002110953.0 4.3.2-x86_64 ["containers-common","1","0.1.40","8.module+el8.1.1+5351+506397b0","x86_64"] 43.81.202002170853.0 4.3.3-x86_64 Fetching a source RPM for containers-common, e.g. from [15,16] shows the source packages coming from skopeo. Checking [17]: $ git --no-pager log --follow --oneline --stat=200 -M50% -- vendor/github.com/containers/storage/storage.conf afaa9e7f Bump github.com/containers/storage from 1.15.1 to 1.15.2 vendor/github.com/containers/storage/storage.conf | 3 --- 1 file changed, 3 deletions(-) 39ff039b Image encryption/decryption support in skopeo vendor/github.com/containers/storage/storage.conf | 44 +++++++++++++++++++++++++------------------- 1 file changed, 25 insertions(+), 19 deletions(-) 05ae513b Bump github.com/containers/buildah from 1.8.4 to 1.11.4 vendor/github.com/containers/storage/storage.conf | 7 ------- 1 file changed, 7 deletions(-) 700b3102 update github.com/containers/{image,storage} vendor/github.com/containers/storage/storage.conf | 8 ++++++++ 1 file changed, 8 insertions(+) 033b2902 migrate to go modules vendor/github.com/containers/storage/storage.conf | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+) $ git --no-pager log --follow --oneline --stat=200 -M50% 033b2902^ -- contrib/storage.conf fe259105 add storage.conf and manpage in contrib/ contrib/storage.conf | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) $ for HASH in fe259105 033b2902 700b3102 05ae513b 39ff039b afaa9e7f; do git describe --contains "${HASH}"; done v0.1.29~3^2 v0.1.38~14^2~2 v0.1.39~1 v0.1.41~25^2 v0.1.41~21^2 v0.1.41~12^2 So changes may have been made in 0.1.29 (when the file landed for the first time, likely from wherever we store post-Git patches), and were likely made in 0.1.38, 0.1.39, and 0.1.41. However, the skopeo and derivative containers-common RPMs may have had patched versions of the file tracked in dist-git [18]. Comparing the dist-git 4.1 tip with the machine-config template: $ git -C containers/skopeo remote -v | grep 'dist-git.*fetch' dist-git git://pkgs.devel.redhat.com/rpms/skopeo.git (fetch) $ git --no-pager -C containers/skopeo log --date=short --format='%ad %h %s' -2 dist-git/rhaos-4.1-rhel-8 -- storage.conf 2018-07-18 3757b210 add statx to seccomp.json to containers-config add seccomp.json to containers-config 2017-11-08 284f9024 Force storage.conf to default to overlay $ git --no-pager -C containers/skopeo grep '^Version:' 3757b210 3757b210:skopeo.spec:Version: 0.1.31 $ diff -U3 <(git -C containers/skopeo cat-file -p 3757b210:storage.conf) <(sed 's/^ //' openshift/machine-config-operator/templates/common/_base/files/container-storage.yaml)--- /dev/fd/63 2020-02-20 01:13:48.073704685 -0800 +++ /dev/fd/62 2020-02-20 01:13:48.073704685 -0800 @@ -1,3 +1,10 @@ +filesystem: "root" +mode: 0644 +path: "/etc/containers/storage.conf" +contents: + inline: | +# This file is generated by the Machine Config Operator's containerruntimeconfig controller. +# # storage.conf is the configuration file for all tools # that share the containers/storage libraries # See man 5 containers-storage.conf for more information So the machine-config master (5ed0aee72c) only differs from the old 0.1.31 RPM storage.conf by the "file is generated" marker. There does not seem to be any 4.2-specific content. Presumably they're using the same rhaos-4.1-rhel-8 RPMs. 4.3 has some changes: $ git --no-pager log --date=short --format='%ad %h %s' -2 --stat=80 dist-git/rhaos-4.3-rhel-8 -- storage.conf 2019-12-09 4a131916 skopeo-0.1.40-2.el8 storage.conf | 39 +++++++++++++++++++++++++++++---------- 1 file changed, 29 insertions(+), 10 deletions(-) 2019-10-08 13a4ce10 skopeo-1:0.1.40-0.1.gitf72e39f storage.conf | 114 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 114 insertions(+) So it looks like we can ignore the dev skopeo repository, focus on the dist-git skopeo repository, and say that before 0.1.40-2.el8 we had a version of storage.conf in the RPMs that matched the unpatched machine-config templates, and with 0.1.40-2.el8 and later the RPMs had different content. Can we check the RPMs to confirm? The product pages are claiming containers-common-0.1.32 for 4.1.34 [19,20]. Those product pages are fed from RPM Errata reports, and ART builds those Errata by sweeping RPM repositories in the viscinity of the RHCOS builds. So there's a potential for races like: 1. RPM Errata sweep fires and grabs RPM A v1. 2. New RPM A v2 pushed to the repository. 3. RHCOS build hits repositories and grabs RPM A v2. The RPMs referenced by releases-rhcos-art.cloud are reliable, but actually tracking down the referenced RPMs to download them is complicated (especially for module builds like containers-common). But here are two RPM-lookup procedures that seem more reliable: A. From [21]: 1. On [21], find the matching skopeo package, e.g. skopeo-0.1.40-2.el8. Click through to the Advisory, e.g. [22]. 2. On [22], find the matching skopeo package, expand the CDN RPMs section to see the containers-common RPM link, e.g. [23]. 3. Click through to /etc/containers/storage.conf, e.g. [24]. 4. See the sha256, e.g. a6423cca39d0cde0d6ee82163630d288e8876ab7d39d2678f6d86d804bf61044. B. From [25]. This works better for module builds. 1. Search for the skopeo package from [25], e.g. [26], takes me to [27]. 2. Find the matching package, e.g. skopeo-0.1.37-5.module+el8.1.0+4240+893c1ab8, and click through to [28]. 3. Find the x86_64 containers-common RPM, and click through to info [29]. Continue from step A.3. Summarizing storage.conf digests for the various RPMs: * containers-common-1:0.1.32-4.git1715c90.el8.x86_64 Used for 4.1.1 through 4.1.16. ee7daca89532d5a80da391fc358776ec11eff256c497652c49505acc70b96822 [30] * containers-common-1:0.1.32-5.git1715c90.el8.x86_64 Used for 4.1.7 through 4.1.23, 4.2.0-rc.0 through 4.2.2. ee7daca89532d5a80da391fc358776ec11eff256c497652c49505acc70b96822 [31] * containers-common-1:0.1.37-5.module+el8.1.0+4240+893c1ab8.x86_64 Used for 4.1.24 through 4.1.28, 4.2.4 through 4.2.12. ee7daca89532d5a80da391fc358776ec11eff256c497652c49505acc70b96822 [32] * containers-common-1:0.1.37-6.module+el8.1.0+4876+e678a192.x86_64 Used for 4.1.29 through 4.1.31, 4.2.13 through 4.2.18. ee7daca89532d5a80da391fc358776ec11eff256c497652c49505acc70b96822 [33] * containers-common-1:0.1.40-2.el8.x86_64.rpm Used for 4.3.0-rc.0 through 4.3.0. a6423cca39d0cde0d6ee82163630d288e8876ab7d39d2678f6d86d804bf61044 [24] * containers-common-1:0.1.40-3.rhaos.el8.x86_64 Used for 4.3.1. a6423cca39d0cde0d6ee82163630d288e8876ab7d39d2678f6d86d804bf61044 [34] * containers-common-1:0.1.40-8.module+el8.1.1+5351+506397b0.x86_64 Used for 4.2.19, 4.2.20, and 4.1.34. a6423cca39d0cde0d6ee82163630d288e8876ab7d39d2678f6d86d804bf61044 [35] So there are only two versions in the RPMs, ee7daca895 used for all 4.1 and 4.2, and a6423cca39 used for all 4.3. That means that the vulnerable transitions are 4.2.16 and earlier going into 4.3. It also means that there's a potential for future trouble in transitions from 4.1.31 and earlier to a future 4.1 or 4.2 where the RPM-installed content is different, and from 4.2.16 and earlier to a future 4.2 where the RPM-installed content is different, but that we have no such 4.1 or 4.2 changes at the moment. So ideally this pull would block edges from 4.2.16 and earlier into 4.3. This commit drops 4.2.16 from the *-4.3 channels with a comment about this bug. This also explicitly blocks edges from 4.2 into 4.3.0, because 4.3.0 is the only 4.3 release which recommends 4.2.16 or earlier as an update edge. $ for i in $(seq 0 3); do echo -n "$i "; oc adm release info "quay.io/openshift-release-dev/ocp-release:4.3.$i-x86_64" | grep Upgrades; done 0 Upgrades: 4.2.16, 4.3.0-rc.0, 4.3.0-rc.1, 4.3.0-rc.2, 4.3.0-rc.3 1 Upgrades: 4.2.18, 4.3.0-rc.0, 4.3.0-rc.3, 4.3.0 2 Upgrades: 4.2.19, 4.3.0, 4.3.1 3 Upgrades: 4.2.20, 4.3.0, 4.3.1, 4.3.2 There shouldn't be much pushback on pulling the edge, because users can still move from 4.2 to 4.3 via 4.2.18 -> 4.3.1, both of which are already in fast-4.3. Also simplify the wording on the GCP bug 1793635, which remains unfixed. [1]: openshift/machine-config-operator#1320 (comment) [2]: https://bugzilla.redhat.com/show_bug.cgi?id=1782152#c5 [3]: https://bugzilla.redhat.com/show_bug.cgi?id=1781708#c0 [4]: https://github.com/openshift/machine-config-operator/pull/1320/files [5]: openshift/machine-config-operator#1190 [6]: https://github.com/openshift/machine-config-operator/blob/13f0dda734262c3edbd23c007e42b7704125e88f/docs/MachineConfiguration.md [7]: https://github.com/openshift/machine-config-operator/blob/13f0dda734262c3edbd23c007e42b7704125e88f/docs/ContainerRuntimeConfigDesign.md [8]: openshift/machine-config-operator#330 (comment) [9]: https://bugzilla.redhat.com/show_bug.cgi?id=1782153 [10]: openshift/machine-config-operator#1382 (comment) [11]: openshift/machine-config-operator#1323 (comment) [12]: https://bugzilla.redhat.com/show_bug.cgi?id=1782149 [13]: openshift/machine-config-operator#1322 (comment) [14]: https://gitlab.cee.redhat.com/coretools/differ Internal link, sorry :/ But you can also browse the history at: https://releases-rhcos-art.cloud.privileged.psi.redhat.com/?stream=releases/rhcos-4.2&release=42.81.20200114.0 etc. [15]: https://access.redhat.com/downloads/content/290/ver=4.2/rhel---8/4.2.0/x86_64/packages [16]: https://access.redhat.com/downloads/content/rhel---8/x86_64/8841/containers-common/0.1.32-5.git1715c90.el8/x86_64/fd431d51/package [17]: https://github.com/containers/skopeo/ [18]: http://pkgs.devel.redhat.com/cgit/rpms/skopeo/ [19]: https://access.redhat.com/downloads/content/290/ver=4.1/rhel---8/4.1.34/x86_64/packages [20]: https://access.redhat.com/downloads/content/rhel---8/x86_64/8384/containers-common/0.1.32-5.git1715c90.el8/x86_64/fd431d51/package [21]: https://errata.devel.redhat.com/package/show/skopeo [22]: https://errata.devel.redhat.com/errata/content/46255 [23]: https://brewweb.engineering.redhat.com/brew/rpminfo?rpmID=7604818 [24]: https://brewweb.engineering.redhat.com/brew/fileinfo?rpmID=7604818&filename=/etc/containers/storage.conf [25]: https://brewweb.engineering.redhat.com/brew/search [26]: https://brewweb.engineering.redhat.com/brew/search?match=glob&type=package&terms=skopeo [27]: https://brewweb.engineering.redhat.com/brew/packageinfo?packageID=58395 [28]: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=971200 [29]: https://brewweb.engineering.redhat.com/brew/rpminfo?rpmID=7349205 [30]: https://brewweb.engineering.redhat.com/brew/fileinfo?rpmID=6958325&filename=/etc/containers/storage.conf [31]: https://brewweb.engineering.redhat.com/brew/fileinfo?rpmID=7334504&filename=/etc/containers/storage.conf [32]: https://brewweb.engineering.redhat.com/brew/fileinfo?rpmID=7349205&filename=/etc/containers/storage.conf [33]: https://brewweb.engineering.redhat.com/brew/fileinfo?rpmID=7550403&filename=/etc/containers/storage.conf [34]: https://brewweb.engineering.redhat.com/brew/fileinfo?rpmID=7727297&filename=/etc/containers/storage.conf [35]: https://brewweb.engineering.redhat.com/brew/fileinfo?rpmID=7656074&filename=/etc/containers/storage.conf

templates: Move container-storage.yaml into common/

47a6321

It's identical between master/worker, so move it into the common pool. Just a cleanup while looking at openshift#1319

openshift-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Dec 10, 2019

cgwalters mentioned this pull request Dec 10, 2019

Recent containers storage.conf breaking RHCOS promotion #1319

Closed

openshift-ci-robot requested review from mtrmac and umohnani8 December 10, 2019 19:28

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 10, 2019

cgwalters changed the title ~~templates: Move container-storage.yaml into common/~~ templates/container-storage: Add a "this is generated" note Dec 10, 2019

openshift-ci-robot assigned kikisdeliveryservice Dec 11, 2019

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Dec 11, 2019

openshift-merge-robot merged commit 41e16d8 into openshift:master Dec 11, 2019

openshift-cherrypick-robot mentioned this pull request Dec 11, 2019

[release-4.3] Bug 1782149: templates/container-storage: Add a "this is generated" note #1322

Merged

openshift-cherrypick-robot mentioned this pull request Dec 11, 2019

Bug 1782152: templates/container-storage: Add a "this is generated" note #1323

Merged

openshift-cherrypick-robot mentioned this pull request Dec 11, 2019

[release-4.1] Bug 1782153: templates/container-storage: Add a "this is generated" note #1324

Closed

openshift-ci-robot changed the title ~~templates/container-storage: Add a "this is generated" note~~ Bug 1781708: templates/container-storage: Add a "this is generated" note Dec 11, 2019

vrutkovs mentioned this pull request Jan 15, 2020

MCO /etc/crio/crio.conf file conflict upon upgrading releases okd-project/okd#36

Closed

openshift-cherrypick-robot mentioned this pull request Jan 15, 2020

[fcos] templates/container-storage: Add a "this is generated" note #1372

Merged

vrutkovs mentioned this pull request Jan 17, 2020

Bug 1782153: templates: Add a "this is generated" note #1382

Merged

Bug 1781708: templates/container-storage: Add a "this is generated" note #1320

Bug 1781708: templates/container-storage: Add a "this is generated" note #1320

Uh oh!

Conversation

cgwalters commented Dec 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cgwalters commented Dec 10, 2019

Uh oh!

cgwalters commented Dec 10, 2019

Uh oh!

cgwalters commented Dec 10, 2019

Uh oh!

cgwalters commented Dec 10, 2019

Uh oh!

cgwalters commented Dec 10, 2019

Uh oh!

vrutkovs commented Dec 10, 2019

Uh oh!

cgwalters commented Dec 10, 2019

Uh oh!

cgwalters commented Dec 10, 2019

Uh oh!

kikisdeliveryservice commented Dec 10, 2019

Uh oh!

vrutkovs commented Dec 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cgwalters commented Dec 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cgwalters commented Dec 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kikisdeliveryservice commented Dec 11, 2019

Uh oh!

kikisdeliveryservice commented Dec 11, 2019

Uh oh!

openshift-ci-robot commented Dec 11, 2019

Uh oh!

openshift-bot commented Dec 11, 2019

Uh oh!

kikisdeliveryservice commented Dec 11, 2019

Uh oh!

openshift-ci-robot commented Dec 11, 2019

Uh oh!

openshift-ci-robot commented Dec 11, 2019

Uh oh!

vrutkovs commented Dec 11, 2019

Uh oh!

openshift-cherrypick-robot commented Dec 11, 2019

Uh oh!

vrutkovs commented Dec 11, 2019

Uh oh!

openshift-cherrypick-robot commented Dec 11, 2019

Uh oh!

vrutkovs commented Dec 11, 2019

Uh oh!

openshift-cherrypick-robot commented Dec 11, 2019

Uh oh!

cgwalters commented Dec 11, 2019

Uh oh!

openshift-cherrypick-robot commented Dec 11, 2019

Uh oh!

cgwalters commented Dec 11, 2019

Uh oh!

vrutkovs commented Dec 11, 2019

Uh oh!

openshift-ci-robot commented Dec 11, 2019

Uh oh!

vrutkovs commented Dec 11, 2019

Uh oh!

openshift-ci-robot commented Dec 11, 2019

Uh oh!

vrutkovs commented Jan 15, 2020

Uh oh!

openshift-cherrypick-robot commented Jan 15, 2020

Uh oh!

Reviewers

Assignees

cgwalters commented Dec 10, 2019 •

edited

Loading

vrutkovs commented Dec 10, 2019 •

edited

Loading

cgwalters commented Dec 11, 2019 •

edited

Loading

cgwalters commented Dec 11, 2019 •

edited

Loading