Design for osImageURL updates - integration with CVO/release payload

[cgwalters]

I wanted to elaborate here on the current status of this. We have a PR in #363 which will finally close the loop and inject machine-os-content from the release payload all the way into the MachineConfig objects, which will result in the MCD updating.

The final architecture will be:

New kernel errata, turns into RPM, converted into ostree then oscontainer. A bit more information on the build system side here. The oscontainer makes it into a new release payload published on quay.io.

At some point the release payload pulled down by CVO, which includes a osimageurl ConfigMap that references that container (same thing as the machine-os-content ImageStream). The CVO updates the ConfigMap, which you can see via

oc -n openshift-machine-config-operator get configmap/machine-config-osimageurl

The operator notices the change to the configmap and updates the "controllerconfig" which is an internal CRD that is used as the primary input to the MCC. See oc get -o yaml controllerconfig.

The "template" sub-controller of the MCC then updates machineconfigs/00-master and machineconfigs/00-worker.

The "render" sub-controller of the MCC generates new "rendered" MCs that look like machineconfigs/master-<hash> and machineconfigs/worker-<hash> and updates the MachineConfigPools to target them. For more information on this, see the MCC docs.

On each node the MCD will get the new osimageurl, and if it's different than what's booted, it will pull down the container and rebase to it and reboot. This is also the same as any other config change.

[jlebon]

For completeness, one alternative is to make it a MachineConfig object instead, which is then used as the base config containing the osImageURL in which all other (Ignition) configs are merged into. I think this maps slightly more closely to what's currently done today.

That said, using a separate ConfigMap allows us to have richer metadata. E.g. pkg lists & diffs, OSTree commits & versions, etc... The idea being that an "Update available" button can show all that stuff without having to hunt down the osImage.

[ashcrow]

Remember, this data will need to flow one way or another down into a MachineConfig. If it's easier we could add some more information to MCs themselves.

[jlebon]

Strawman:

apiVersion: v1
kind: ConfigMap
metadata:
  name: rhcos-release-info
data:
  metadata.json: '{"osImageUrl": "registry.svc.ci.openshift.org/rhcos/os-maipo@sha256:4efb36f476405f8e30256733ac900e11b70833dc6a6b54179db60a501fa47124", "ostree-version": "47.115", "ostree-checksum": "b0edcc594e65ccebcc917b0ead76ea6894c0d4d672a63ceee5bdc59976c55bf9", "pkgdiff": [["origin-clients", 2, {"NewPackage": ["origin-clients", "4.0.0-0.alpha.0.610.98ebf23", "x86_64"], "PreviousPackage": ["origin-clients", "4.0.0-0.alpha.0.607.49e9f08", "x86_64"]}], ...], "pkglist": ["GeoIP-1.5.0-13.el7.x86_64", "NetworkManager-1:1.12.0-7.el7_6.x86_64", ...]}'

Basically, we can include a subset of what coreos-assembler already outputs. Or maybe just a link to that meta.json instead?

[jlebon]

Remember, this data will need to flow one way or another down into a MachineConfig. If it's easier we could add some more information to MCs themselves.

I think it shouldn't be too hard to adapt the MCC either way so we shouldn't constrain ourselves too much on what's easier.

[ashcrow]

I think it shouldn't be too hard to adapt the MCC either way so we shouldn't constrain ourselves too much on what's easier.

Fair. Rephrased: what's doable and consumable within specific constraints 😃

[cgwalters]

Basically, we can include a subset of what coreos-assembler already outputs.

Do we need anything other than the "registry.svc.ci.openshift.org/rhcos/os-maipo@sha256:4efb36f476405f8e30256733ac900e11b70833dc6a6b54179db60a501fa47124", string?

EDIT: Ah sorry I missed this:

The idea being that an "Update available" button can show all that stuff without having to hunt down the osImage.

Yeah, though...a tricky part about this is that the pkgdiff is against the previous version, which may not be the one they're updating to...

Mmm. I'm OK stuffing the whole meta.json in there for now, but I suspect we're going to need to add something smarter later.

[jlebon]

Yeah, though...a tricky part about this is that the pkgdiff is against the previous version, which may not be the one they're updating to...

Yeah, I think to do this correctly, the pkg diff would have to be done from the pkg lists instead rather than precomputed. Anyway, those are things that could easily come later. We just have to make sure we leave the door open for it.

Mmm. I'm OK stuffing the whole meta.json in there for now, but I suspect we're going to need to add something smarter later.

Yeah, that sounds fine to me.

[smarterclayton]

Be aware you’re going to eventually need to have a distinction between “the config says this is the latest” and “I’m ready to roll that out to the nodes”. Design with that in mind because new kubeelts are going to happen ~ 1/week

[smarterclayton]

Pkgdiff is useless. There is no guarantee it has any relevance. Think package manifest instead of diffs. Our errata and higher level tools will calculate the diff

[jlebon]

Be aware you’re going to eventually need to have a distinction between “the config says this is the latest” and “I’m ready to roll that out to the nodes”. Design with that in mind because new kubeelts are going to happen ~ 1/week

Can you elaborate on that? Do you mean making sure we actually upgrade to the selected release instead of whatever happens to now be the latest at the time we're actually ready to upgrade? I think that should be covered yeah. The metadata includes the full sha256 of the oscontainer.

Pkgdiff is useless. There is no guarantee it has any relevance. Think package manifest instead of diffs. Our errata and higher level tools will calculate the diff

Yup, see coreos/coreos-assembler#226.

[smarterclayton]

We don't decide on version 4.0.6 until the last moment. So the goal of automation is to continuously have a set of 4.0.6 candidates that we then pick one and ship it. The "train" mindset, not the "artisanal release payload".

[smarterclayton]

Expectation is that you will build an image and push it to the openshift origin integration image stream, then reference the component “os” from this operator and have the dummy value substituted.

In OCP, we do something similar where the OS content gets built via whatever process and shows up in the prerelease list, and is processed the same way.

You can build anywhere - we just need you to push/be imported to the right place

[smarterclayton]

Note the dummy value can be real - but since we need to mirror the content you have to be referenced in the operator list which means you need to be sucked into the right place.

Push vs scheduled import is also possible, but if we do scheduled import the source location has to be appropriately gated like a push would (only changes if you test against latest in an install)