daemon: Accept osImageURL with matching digests #463

cgwalters · 2019-02-20T02:00:53Z

The way CI works, a release payload is generated in the CI namespace
but reusing the digested pull of existing components. This means
the pull specs for everything change format - but the actual
content stays the same.

Let's handle this for the OS image at least.

In practice...I feel like we're going to want higher level special
handling for this - maybe the CVO does pull-through to the registry
and rewrites to an internal canonical form, but this should help
us for now.

Closes: #462

cgwalters · 2019-02-20T02:01:34Z

Not tested locally though beyond new unit tests; looks like the UBI work broke make image-daemon for me with:

Loaded plugins: ovl, product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
http://base-4-0.ocp.svc/rhel-fast-datapath/repodata/repomd.xml: [Errno 14] curl#6 - "Could not resolve host: base-4-0.ocp.svc; Unknown error"
Trying other mirror.

pkg/daemon/daemon.go

runcom · 2019-02-20T08:36:56Z

/approve

jlebon

Some minor comments, otherwise LGTM!

pkg/daemon/daemon.go

jlebon · 2019-02-20T14:15:17Z

/approve

jlebon · 2019-02-20T14:51:19Z

/lgtm

openshift-bot · 2019-02-20T15:54:07Z

/retest

Please review the full test history for this PR and help us cut down flakes.

vrutkovs · 2019-02-20T16:36:39Z

/retest

openshift-bot · 2019-02-20T17:55:03Z

/retest

Please review the full test history for this PR and help us cut down flakes.

cgwalters · 2019-02-20T18:19:25Z

Just for general interest, here's a workflow I have for testing PRs like this locally. As part of submitting a PR, a release image will be generated. You can log into the CI namespace (see HACKING.md) and from there run: oc logs -c release release-latest

In those logs you'll see a message like:
Pushed image sha256:3b3ab2bfd9301f11ae0130c47010828e25a5f26f9caf7322f8d9b14a2571a881 to registry.svc.ci.openshift.org/ci-op-6br3phj4/release:latest

Now, because the CI namespaces only last ~1h, if you want to do testing you'll need to mirror it, e.g.:

oc adm release mirror --from registry.svc.ci.openshift.org/ci-op-6br3phj4/release:latest --to quay.io/cgwalters/ostest

Now, you can pass it to the installer to create a fresh cluster with it:

env OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=quay.io/cgwalters/ostest:release openshift-install ...

Or you can use it as a target for oc adm upgrade --to-image.

ashcrow · 2019-02-20T19:41:38Z

level=fatal msg="waiting for Kubernetes API: context deadline exceeded"
2019/02/20 18:37:00 Container setup in pod e2e-aws failed, exit code 1, reason Error

openshift-bot · 2019-02-20T23:57:54Z

/retest

Please review the full test history for this PR and help us cut down flakes.

cgwalters · 2019-02-21T03:22:10Z

Hmm, that job failed with:

0221 02:06:34.537828    5053 update.go:647] Updating OS to registry.svc.ci.openshift.org/ci-op-39zm8d3x/stable@sha256:660061d6eae3ee6d93ca836cd52e6033f1d611c629c1ce47cf272c9e9bda2488
E0221 02:06:34.557315    5053 daemon.go:436] Fatal error checking initial state of node: Checking initial state: Failed to run pivot: error starting job: Transaction is destructive.
E0221 02:06:34.557411    5053 writer.go:90] Marking degraded due to: Checking initial state: Failed to run pivot: error starting job: Transaction is destructive.

Hmm. In non-ancient systemd looks like that error has a more useful message which would have told us what was conflicting.

I feel like this is another case where we should have retried instead of going degraded maybe?

But clearly need to figure out why this happened.

openshift-bot · 2019-02-21T04:01:46Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-02-21T06:00:53Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-02-21T08:01:53Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-02-21T10:02:59Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2019-02-21T12:04:00Z

/retest

Please review the full test history for this PR and help us cut down flakes.

cgwalters · 2019-02-21T13:45:41Z

Not 100% certain but I think what happened here is since we don't have this pivot commit we ended up in a race where the system was shutting down for reboot, but the MCD had come up and tried to start pivot again.

/retest

pkg/daemon/daemon.go

runcom · 2019-02-21T15:07:15Z

/retest

runcom · 2019-02-21T17:08:20Z

/retest

cgwalters · 2019-02-21T20:17:24Z

(Any reason not to drop the lgtm?)

runcom · 2019-02-21T20:18:35Z

/lgtm

@jlebon ptal

ashcrow · 2019-02-21T21:09:31Z

/retest

jlebon · 2019-02-21T22:29:53Z

@jlebon ptal

Yup, this is still
/lgtm
to me. :)

Not 100% certain but I think what happened here is since we don't have this pivot commit we ended up in a race where the system was shutting down for reboot, but the MCD had come up and tried to start pivot again.

Yeah, that sounds plausible to me.

openshift-ci-robot · 2019-02-21T22:30:02Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cgwalters, jlebon, runcom

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [cgwalters,jlebon,runcom]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

cgwalters · 2019-02-22T13:13:04Z

/retest

runcom · 2019-02-22T15:33:58Z

/retest

kikisdeliveryservice · 2019-02-22T17:08:47Z

level=error msg="\t* aws_route.to_nat_gw.5: Error creating route: timeout while waiting for state to become 'success' (timeout: 2m0s)"

/retest

runcom · 2019-02-22T19:30:42Z

aws limit hit

/retest

kikisdeliveryservice · 2019-02-22T20:38:16Z

I'll retest in a bit, we keep hitting aws rate limits errors ☹️

runcom · 2019-02-22T21:41:08Z

/retest

ashcrow · 2019-02-22T23:18:34Z

/retest

cgwalters · 2019-02-23T01:00:46Z

/retest

smarterclayton · 2019-02-23T02:42:16Z

There is a reason we do this - we will almost certainly migrate our content to another registry location in the future. It’s good to be smart, but don’t assume that if a new url comes in that it will always be at that location in the future.

This is a lowering of openshift/machine-config-operator#463 to pivot. We need it for the case of doing an early pivot before the MCD comes up.

openshift-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Feb 20, 2019

cgwalters mentioned this pull request Feb 20, 2019

upgrade fails in CI due to different generated release image names with same content #462

Closed

openshift-ci-robot requested review from jlebon and yuqi-zhang February 20, 2019 02:04

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 20, 2019

hexfusion mentioned this pull request Feb 20, 2019

image: MCD depends on util-linux (mount) which is being removed from base #445

Merged

kikisdeliveryservice reviewed Feb 20, 2019

View reviewed changes

pkg/daemon/daemon.go Outdated Show resolved Hide resolved

jlebon reviewed Feb 20, 2019

View reviewed changes

pkg/daemon/daemon.go Outdated Show resolved Hide resolved

pkg/daemon/daemon.go Outdated Show resolved Hide resolved

cgwalters force-pushed the pivot-identical branch from b8ac65b to a02917b Compare February 20, 2019 14:45

openshift-ci-robot assigned jlebon Feb 20, 2019

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 20, 2019

runcom reviewed Feb 21, 2019

View reviewed changes

pkg/daemon/daemon.go Outdated Show resolved Hide resolved

openshift-ci-robot assigned runcom Feb 21, 2019

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 21, 2019

openshift-merge-robot merged commit b25616a into openshift:master Feb 23, 2019

runcom mentioned this pull request Feb 23, 2019

Fix oncefrom with MachineConfig #483

Merged

cgwalters mentioned this pull request Feb 24, 2019

daemon: Don't reboot for noops #485

Closed

cgwalters added a commit to cgwalters/pivot that referenced this pull request Mar 8, 2019

WIP: Don't pivot if identical sha256

f705b41

This is a lowering of openshift/machine-config-operator#463 to pivot. We need it for the case of doing an early pivot before the MCD comes up.

cgwalters mentioned this pull request Mar 8, 2019

Don't pivot if identical sha256 openshift/pivot#43

Merged

cgwalters added a commit to cgwalters/pivot that referenced this pull request Mar 8, 2019

WIP: Don't pivot if identical sha256

ca0909a

This is a lowering of openshift/machine-config-operator#463 to pivot. We need it for the case of doing an early pivot before the MCD comes up.

cgwalters added a commit to cgwalters/pivot that referenced this pull request Mar 8, 2019

WIP: Don't pivot if identical sha256

cc36276

This is a lowering of openshift/machine-config-operator#463 to pivot. We need it for the case of doing an early pivot before the MCD comes up.

cgwalters added a commit to cgwalters/pivot that referenced this pull request Mar 8, 2019

WIP: Don't pivot if identical sha256

f91621d

This is a lowering of openshift/machine-config-operator#463 to pivot. We need it for the case of doing an early pivot before the MCD comes up.

cgwalters added a commit to cgwalters/pivot that referenced this pull request Mar 8, 2019

Don't pivot if identical sha256

b26b9e1

This is a lowering of openshift/machine-config-operator#463 to pivot. We need it for the case of doing an early pivot before the MCD comes up.

cgwalters added a commit to cgwalters/pivot that referenced this pull request Mar 8, 2019

Don't pivot if identical sha256

4a065e8

This is a lowering of openshift/machine-config-operator#463 to pivot. We need it for the case of doing an early pivot before the MCD comes up.

cgwalters added a commit to cgwalters/pivot that referenced this pull request Mar 11, 2019

Don't pivot if identical sha256

e0e0317

This is a lowering of openshift/machine-config-operator#463 to pivot. We need it for the case of doing an early pivot before the MCD comes up.

cgwalters added a commit to cgwalters/pivot that referenced this pull request Mar 11, 2019

Don't pivot if identical sha256

71d6d77

This is a lowering of openshift/machine-config-operator#463 to pivot. We need it for the case of doing an early pivot before the MCD comes up.

ashcrow pushed a commit to openshift/pivot that referenced this pull request Mar 11, 2019

Don't pivot if identical sha256

297a0ec

This is a lowering of openshift/machine-config-operator#463 to pivot. We need it for the case of doing an early pivot before the MCD comes up.

daemon: Accept osImageURL with matching digests #463

daemon: Accept osImageURL with matching digests #463

Uh oh!

Conversation

cgwalters commented Feb 20, 2019

Uh oh!

cgwalters commented Feb 20, 2019

Uh oh!

Uh oh!

runcom commented Feb 20, 2019

Uh oh!

jlebon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jlebon commented Feb 20, 2019

Uh oh!

jlebon commented Feb 20, 2019

Uh oh!

openshift-bot commented Feb 20, 2019

Uh oh!

vrutkovs commented Feb 20, 2019

Uh oh!

openshift-bot commented Feb 20, 2019

Uh oh!

cgwalters commented Feb 20, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ashcrow commented Feb 20, 2019

Uh oh!

openshift-bot commented Feb 20, 2019

Uh oh!

cgwalters commented Feb 21, 2019

Uh oh!

openshift-bot commented Feb 21, 2019

Uh oh!

openshift-bot commented Feb 21, 2019

Uh oh!

openshift-bot commented Feb 21, 2019

Uh oh!

openshift-bot commented Feb 21, 2019

Uh oh!

openshift-bot commented Feb 21, 2019

Uh oh!

cgwalters commented Feb 21, 2019

Uh oh!

Uh oh!

runcom commented Feb 21, 2019

Uh oh!

runcom commented Feb 21, 2019

Uh oh!

cgwalters commented Feb 21, 2019

Uh oh!

runcom commented Feb 21, 2019

Uh oh!

ashcrow commented Feb 21, 2019

Uh oh!

jlebon commented Feb 21, 2019

Uh oh!

openshift-ci-robot commented Feb 21, 2019

Uh oh!

cgwalters commented Feb 22, 2019

Uh oh!

runcom commented Feb 22, 2019

Uh oh!

kikisdeliveryservice commented Feb 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

runcom commented Feb 22, 2019

Uh oh!

kikisdeliveryservice commented Feb 22, 2019

Uh oh!

runcom commented Feb 22, 2019

Uh oh!

ashcrow commented Feb 22, 2019

Uh oh!

cgwalters commented Feb 23, 2019

Uh oh!

smarterclayton commented Feb 23, 2019

cgwalters commented Feb 20, 2019 •

edited

Loading

kikisdeliveryservice commented Feb 22, 2019 •

edited

Loading