Bug 1782546: rhcos: Bump to 43.81.201912091858.0 #2777

cgwalters · 2019-12-10T00:23:43Z

This has encryption/FIPS fixes. Among other things, TPM2 binding
by default on metal is disabled by default. See also
openshift/enhancements#140

Note this is not a direct cherry-pick because RHCOS has branched for 4.4.

https://bugzilla.redhat.com/show_bug.cgi?id=1775388

openshift-ci-robot · 2019-12-10T00:23:44Z

@cgwalters: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

Details

In response to this:

BZ 1775388: rhcos: Bump to 43.81.201912091858.0

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jcpowermac · 2019-12-10T00:38:47Z

/test e2e-vsphere

abhinavdahiya · 2019-12-10T06:43:25Z

/test e2e-metal
/test e2e-gcp
/test e2e-azure

hardys · 2019-12-10T11:40:01Z

/label platform/baremetal

hardys · 2019-12-10T11:40:44Z

I added the platform/baremetal to trigger the metal3 CI (we don't yet have a fully integrated IPI baremetal e2e test)

metal3ci · 2019-12-10T11:46:52Z

Build FAILURE, see build http://10.8.144.11:8080/job/dev-tools/1362/

hardys · 2019-12-10T12:07:22Z

/remove-label platform/baremetal

hardys · 2019-12-10T12:11:25Z

Sorry for the noise, I missed this is a backport to 4.3 - we need to fix metal3 CI to support testing from a branch or revive openshift/release#5016 to add e2e IPI baremetal coverage

cgwalters · 2019-12-10T14:21:54Z

/hold
podman broke in this one

sdodson · 2019-12-11T01:36:22Z

43.81.201912101506.0 at least gets e2e-metal through the installation. ref https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_installer/2695/pull-ci-openshift-installer-master-e2e-metal/124

sdodson · 2019-12-11T14:47:17Z

/test e2e-metal
/test e2e-gcp
/test e2e-azure

sdodson · 2019-12-11T14:47:35Z

/test e2e-vsphere

cgwalters · 2019-12-11T15:10:28Z

/hold cancel
podman isn't reverted anymore

sdodson · 2019-12-11T15:19:44Z

No e2e-metal presubmit for 4.3 branch, bummer.
openshift/release#6344

cgwalters · 2019-12-11T16:37:27Z

/test e2e-aws
Storage flakes

/test e2e-aws-scaleup-rhel7
One single network flaek

/test e2e-gcp-upi
Looks like rate limiting in GCP APIs

sdodson · 2019-12-11T16:54:13Z

/lgtm

sdodson · 2019-12-11T17:14:21Z

/bugzilla refresh

Note this is not a direct cherry-pick because RHCOS has branched for 4.4.

Actually, it looks like master branch has not diverged with respect to RHCOS images and this PR cites a 4.4 BZ. Do you mind if we re-target this to master branch, cherry-pick, and target a cloned BZ (or one like https://bugzilla.redhat.com/show_bug.cgi?id=1776011) ? While it doesn't look like there have been any 4.4 RHCOS builds yet I'm not sure we'd want release-4.3 to have a newer image than master branch.

openshift-ci-robot · 2019-12-11T17:14:23Z

@sdodson: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

Details

In response to this:

/bugzilla refresh

Note this is not a direct cherry-pick because RHCOS has branched for 4.4.

Actually, it looks like master branch has not diverged with respect to RHCOS images and this PR cites a 4.4 BZ. Do you mind if we re-target this to master branch, cherry-pick, and target a cloned BZ (or one like https://bugzilla.redhat.com/show_bug.cgi?id=1776011) ? While it doesn't look like there have been any 4.4 RHCOS builds yet I'm not sure we'd want release-4.3 to have a newer image than master branch.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

If we can't reach the bootstrap machine via SSH. Before this commit, we would occasionally see connection issues like [1]: level=info msg="Waiting up to 30m0s for the Kubernetes API at https://api.ci-op-6266tp8r-77109.origin-ci-int-aws.dev.rhcloud.com:6443..." level=error msg="Attempted to gather ClusterOperator status after installation failure: listing ClusterOperator objects: Get https://api.ci-op-6266tp8r-77109.origin-ci-int-aws.dev.rhcloud.com:6443/apis/config.openshift.io/v1/clusteroperators: dial tcp 3.221.214.197:6443: connect: connection refused" level=info msg="Pulling debug logs from the bootstrap machine" level=error msg="Attempted to gather debug logs after installation failure: failed to create SSH client, ensure the proper ssh key is in your keyring or specify with --key: dial tcp 3.84.188.207:22: connect: connection refused" level=fatal msg="Bootstrap failed to complete: waiting for Kubernetes API: context deadline exceeded" With this commit, when we see those connection-refused errors, we attempt to retrieve console logs for the bootstrap instance. This will make it easier for users and users to see why the machine failed to come up. It should be especially useful in continuous integration when bumping RHCOS boot images [2], when such boot-time failures are more likely. I've only implemented it on AWS for the moment, but I've set it up so we can extend it to other platforms going forward. [1]: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/12076 [2]: openshift#2777 (comment)

sdodson · 2019-12-13T02:54:36Z

/test e2e-metal

sdodson · 2019-12-13T03:32:34Z

rhcos-fips.service fails on bootstrap host on e2e-metal at least

:/# journalctl --no-pager -u rhcos-fips.service                                                                                                                                                                                               
-- Logs begin at Fri 2019-12-13 03:29:48 UTC, end at Fri 2019-12-13 03:30:23 UTC. --
Dec 13 03:30:03 localhost systemd[1]: Starting Check for FIPS mode...
Dec 13 03:30:03 localhost rhcos-fips[1678]: INFO     : Ignition 0.34.0
Dec 13 03:30:03 localhost rhcos-fips[1678]: INFO     : Stage: files
Dec 13 03:30:03 localhost rhcos-fips[1678]: INFO     : reading system config file "/usr/lib/ignition/base.ign"
Dec 13 03:30:03 localhost rhcos-fips[1678]: INFO     : no config at "/usr/lib/ignition/base.ign"
Dec 13 03:30:03 localhost rhcos-fips[1678]: DEBUG    : files: targeting root without /usr/sbin/restorecon, skipping relabel
Dec 13 03:30:03 localhost rhcos-fips[1678]: INFO     : files: createFilesystemsFiles: createFiles: op(1): [started]  writing file "/etc/ignition-machine-config-encapsulated.json"
Dec 13 03:30:03 localhost rhcos-fips[1678]: CRITICAL : files: createFilesystemsFiles: createFiles: op(1): [failed]   writing file "/etc/ignition-machine-config-encapsulated.json": No such user "root": lookup failed: no such file or directory
Dec 13 03:30:03 localhost rhcos-fips[1678]: files failedFull config:
Dec 13 03:30:03 localhost rhcos-fips[1678]: {
Dec 13 03:30:03 localhost rhcos-fips[1678]:   "ignition": {
Dec 13 03:30:03 localhost rhcos-fips[1678]:     "config": {},
Dec 13 03:30:03 localhost rhcos-fips[1678]:     "proxy": {},
Dec 13 03:30:03 localhost rhcos-fips[1678]:     "security": {
Dec 13 03:30:03 localhost rhcos-fips[1678]:       "tls": {}
Dec 13 03:30:03 localhost rhcos-fips[1678]:     },
Dec 13 03:30:03 localhost rhcos-fips[1678]:     "timeouts": {},
Dec 13 03:30:03 localhost rhcos-fips[1678]:     "version": "2.4.0-experimental"
Dec 13 03:30:03 localhost rhcos-fips[1678]:   },
Dec 13 03:30:03 localhost rhcos-fips[1678]:   "networkd": {},
Dec 13 03:30:03 localhost rhcos-fips[1678]:   "passwd": {},
Dec 13 03:30:03 localhost rhcos-fips[1678]:   "storage": {
Dec 13 03:30:03 localhost rhcos-fips[1678]:     "files": [
Dec 13 03:30:03 localhost rhcos-fips[1678]:       {
Dec 13 03:30:03 localhost rhcos-fips[1678]:         "filesystem": "root",
Dec 13 03:30:03 localhost rhcos-fips[1678]:         "path": "/etc/ignition-machine-config-encapsulated.json",
Dec 13 03:30:03 localhost rhcos-fips[1678]:         "user": {
Dec 13 03:30:03 localhost rhcos-fips[1678]:           "name": "root"
Dec 13 03:30:03 localhost rhcos-fips[1678]:         },
Dec 13 03:30:03 localhost rhcos-fips[1678]:         "contents": {
Dec 13 03:30:03 localhost rhcos-fips[1678]:           "source": "data:text/plain;charset=utf-8;base64,ewogICJtZXRhZGF0YSI6IHsKICAgICJuYW1lIjogImJvb3RzdHJhcC1maXBzIgogIH0sCiAgInNwZWMiOiB7CiAgICAiY29uZmlnIjogewogICAgICAiaWduaXRpb24iOiB7CiAgICAgICAgInZlcnNpb24iOiAiMi4yLjAiCiAgICAgIH0KICAgIH0sCiAgICAia2VybmVsQXJndW1lbnRzIjogW10sCiAgICAiZmlwcyI6IGZhbHNlCiAgfQp9Cg==",
Dec 13 03:30:03 localhost rhcos-fips[1678]:           "verification": {}
Dec 13 03:30:03 localhost rhcos-fips[1678]:         },
Dec 13 03:30:03 localhost rhcos-fips[1678]:         "mode": 384
Dec 13 03:30:03 localhost rhcos-fips[1678]:       }
Dec 13 03:30:03 localhost rhcos-fips[1678]:     ],
Dec 13 03:30:03 localhost rhcos-fips[1678]:     "filesystems": [
Dec 13 03:30:03 localhost rhcos-fips[1678]:       {
Dec 13 03:30:03 localhost rhcos-fips[1678]:         "name": "root",
Dec 13 03:30:03 localhost rhcos-fips[1678]:         "path": "/run/rhcos-fips/sysroot"
Dec 13 03:30:03 localhost rhcos-fips[1678]:       }
Dec 13 03:30:03 localhost rhcos-fips[1678]:     ]
Dec 13 03:30:03 localhost rhcos-fips[1678]:   },
Dec 13 03:30:03 localhost rhcos-fips[1678]:   "systemd": {}
Dec 13 03:30:03 localhost rhcos-fips[1678]: }CRITICAL : Ignition failed: failed to create files: failed to create files: failed to create file "/etc/ignition-machine-config-encapsulated.json": No such user "root": lookup failed: no such file or directory
Dec 13 03:30:03 localhost systemd[1]: rhcos-fips.service: Main process exited, code=exited, status=1/FAILURE
Dec 13 03:30:03 localhost systemd[1]: rhcos-fips.service: Failed with result 'exit-code'.
Dec 13 03:30:03 localhost systemd[1]: Failed to start Check for FIPS mode.
Dec 13 03:30:03 localhost systemd[1]: rhcos-fips.service: Triggering OnFailure= dependencies.

cgwalters · 2019-12-13T12:33:26Z

/hold

ashcrow · 2019-12-13T13:45:02Z

coreos/mantle#1137 merged

ashcrow · 2019-12-13T15:10:42Z

This PR ends up being referenced in multiple BZs ... however, I believe a new bootimage will likely be available based on some other fixes.

This has encryption/FIPS fixes. Among other things, TPM2 binding by default on metal is disabled by default. See also openshift/enhancements#140 Update: Now a regression causing partitioning to fail is fixed. https://bugzilla.redhat.com/show_bug.cgi?id=1775388

cgwalters · 2019-12-13T19:31:53Z

/hold cancel

cgwalters · 2019-12-13T20:12:12Z

Infra limits
/test e2e-gcp-upi

sdodson · 2019-12-13T20:19:24Z

/test e2e-metal
/test e2e-vsphere

sdodson · 2019-12-13T21:01:51Z

e2e-metal installation complete in https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_installer/2777/pull-ci-openshift-installer-release-4.3-e2e-metal/10
/lgtm

cgwalters · 2019-12-13T21:06:45Z

azure and gcp-upi look like usual infra provisioning failures.

sdodson · 2019-12-13T21:26:37Z

/approve

openshift-ci-robot · 2019-12-13T21:26:51Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sdodson

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [sdodson]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2019-12-13T21:32:31Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci-robot · 2019-12-13T21:43:52Z

@cgwalters: All pull requests linked via external trackers have merged. Bugzilla bug 1782546 has been moved to the MODIFIED state.

Details

In response to this:

Bug 1782546: rhcos: Bump to 43.81.201912091858.0

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2019-12-13T23:12:07Z

@cgwalters: The following tests failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
ci/prow/e2e-vsphere	`3544350`	link	`/test e2e-vsphere`
ci/prow/e2e-metal	`3544350`	link	`/test e2e-metal`
ci/prow/e2e-aws-scaleup-rhel7	`3544350`	link	`/test e2e-aws-scaleup-rhel7`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

…1912131630.0 Catching up with openshift/installer@3544350cc (BZ 1775388: rhcos: Bump to 43.81.201912131630.0, 2019-12-10, openshift/installer#2777). Generated with: $ date --iso=m --utc 2020-01-09T00:57+0000 $ git fetch origin $ git --no-pager log -1 --oneline origin/release-4.3 -- data/data/rhcos.json 3544350cc (origin/pr/2777) BZ 1775388: rhcos: Bump to 43.81.201912131630.0 $ git cat-file -p 3544350cc:data/data/rhcos.json | jq -r '.amis | to_entries | sort_by(.key)[] | "\n|`" + .key + "`\n|`" + .value.hvm + "`"' and pasting the output into the module doc.

openshift-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Dec 10, 2019

openshift-ci-robot requested review from jstuever and patrickdillon December 10, 2019 00:24

cgwalters mentioned this pull request Dec 10, 2019

Bug 1775388: rhcos: Bump to latest RHCOS #2776

Merged

openshift-ci-robot added the platform/baremetal IPI bare metal hosts platform label Dec 10, 2019

openshift-ci-robot removed the platform/baremetal IPI bare metal hosts platform label Dec 10, 2019

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 10, 2019

cgwalters force-pushed the rhcos-bump-4.3-20191209 branch from 8a8f7f4 to c4ab50a Compare December 11, 2019 14:45

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 11, 2019

openshift-ci-robot assigned sdodson Dec 11, 2019

openshift-ci-robot added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Dec 11, 2019

wking mentioned this pull request Dec 13, 2019

WIP: cmd/openshift-install/gather: Gather bootstrap console logs #2811

Closed

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 13, 2019

lucab mentioned this pull request Dec 13, 2019

kola/fips: Add user/group to file coreos/mantle#1137

Merged

cgwalters force-pushed the rhcos-bump-4.3-20191209 branch from 52413ba to 3544350 Compare December 13, 2019 19:31

openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Dec 13, 2019

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 13, 2019

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Dec 13, 2019

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 13, 2019

openshift-merge-robot merged commit db8e88d into openshift:release-4.3 Dec 13, 2019

wking mentioned this pull request Jan 9, 2020

modules/installation-aws-user-infra-rhcos-ami: Bump to RHCOS 43.81.201912131630.0 openshift/openshift-docs#18917

Merged

Bug 1782546: rhcos: Bump to 43.81.201912091858.0 #2777

Bug 1782546: rhcos: Bump to 43.81.201912091858.0 #2777

Uh oh!

Conversation

cgwalters commented Dec 10, 2019

Uh oh!

openshift-ci-robot commented Dec 10, 2019

Uh oh!

jcpowermac commented Dec 10, 2019

Uh oh!

abhinavdahiya commented Dec 10, 2019

Uh oh!

hardys commented Dec 10, 2019

Uh oh!

hardys commented Dec 10, 2019

Uh oh!

metal3ci commented Dec 10, 2019

Uh oh!

hardys commented Dec 10, 2019

Uh oh!

hardys commented Dec 10, 2019

Uh oh!

cgwalters commented Dec 10, 2019

Uh oh!

sdodson commented Dec 11, 2019

Uh oh!

sdodson commented Dec 11, 2019

Uh oh!

sdodson commented Dec 11, 2019

Uh oh!

cgwalters commented Dec 11, 2019

Uh oh!

sdodson commented Dec 11, 2019

Uh oh!

cgwalters commented Dec 11, 2019

Uh oh!

sdodson commented Dec 11, 2019

Uh oh!

sdodson commented Dec 11, 2019

Uh oh!

openshift-ci-robot commented Dec 11, 2019

Uh oh!

sdodson commented Dec 13, 2019

Uh oh!

sdodson commented Dec 13, 2019

Uh oh!

cgwalters commented Dec 13, 2019

Uh oh!

ashcrow commented Dec 13, 2019

Uh oh!

ashcrow commented Dec 13, 2019

Uh oh!

cgwalters commented Dec 13, 2019

Uh oh!

cgwalters commented Dec 13, 2019

Uh oh!

sdodson commented Dec 13, 2019

Uh oh!

sdodson commented Dec 13, 2019

Uh oh!

cgwalters commented Dec 13, 2019

Uh oh!

sdodson commented Dec 13, 2019

Uh oh!

openshift-ci-robot commented Dec 13, 2019

Uh oh!

openshift-bot commented Dec 13, 2019

Uh oh!

openshift-ci-robot commented Dec 13, 2019

Uh oh!

openshift-ci-robot commented Dec 13, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants