OCPBUGS-69674: Skip boot image updates if arch annotation is not found #5508

djoshy · 2025-12-18T16:17:42Z

- What I did
This change prevents the boot image controller from defaulting to the control plane architecture while performing boot image updates to a machineset whose autoscalar annotation(capacity.cluster-autoscaler.kubernetes.io/labels) is not yet present. This was causing incorrect updates in multi arch clusters, where the control plane and worker nodes are of different architectures and the arch annotation was not yet applied by the MAPI controller. Once this annotation is applied, another sync loop will be triggered, which should result in a boot image update if needed. I've also updated the unit test to account for this behavior.

It is important to note that:

This annotation is a collection of infra specific labels. If the arch label is missing within this annotation, it will still cause an error; the change is only relevant to the scenario where the whole annotation is missing.
This behavior change is only targeted at multi-arch clusters, for single arch, the controller behavior should remain the same.

- How to verify it
The existing e2es and units should be sufficient to test this behavior.

openshift-ci-robot · 2025-12-18T16:17:49Z

@djoshy: This pull request references Jira Issue OCPBUGS-69674, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.22.0) matches configured target version for branch (4.22.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

- What I did
This change prevents the boot image controller from updating the machineset when the autoscalar annotation(capacity.cluster-autoscaler.kubernetes.io/labels) is not present, instead of defaulting to the control plane architecture. This was causing incorrect updates in multi arch clusters, where the control plane and worker nodes are of different architectures and the arch annotation was not yet applied by the MAPI controller. Once this annotation is applied, another sync loop will be triggered, which should result in a boot image update if needed. I've also updated the unit test to account for this behavior.

It is important to note that this annotation is a collection of infra specific labels. If the arch label is missing within this annotation, it will still cause an error; the change is only relevant to the scenario where the whole annotation is missing.

- How to verify it
The existing e2es and units should be sufficient to test this behavior.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

yuqi-zhang · 2025-12-18T16:20:33Z

/payload periodic-ci-openshift-multiarch-master-nightly-4.21-ocp-e2e-aws-ovn-multi-day-0-x-a

openshift-ci · 2025-12-18T16:20:35Z

@yuqi-zhang: it appears that you have attempted to use some version of the payload command, but your comment was incorrectly formatted and cannot be acted upon. See the docs for usage info.

yuqi-zhang · 2025-12-18T16:23:15Z

/payload-aggregate periodic-ci-openshift-multiarch-master-nightly-4.21-ocp-e2e-aws-ovn-multi-day-0-x-a 10

openshift-ci · 2025-12-18T16:23:18Z

@yuqi-zhang: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-multiarch-master-nightly-4.21-ocp-e2e-aws-ovn-multi-day-0-x-a

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/db9bb860-dc2d-11f0-9867-e48d28b83c35-0

djoshy · 2025-12-18T16:49:18Z

/hold

Looks like vsphere machinesets won't have the arch label populated, so we'll have to alter this approach. Perhaps auto detect a multi arch cluster and just avoid defaulting in those cases.

yuqi-zhang · 2025-12-18T16:53:22Z

/payload-abort

openshift-ci · 2025-12-18T16:53:49Z

@yuqi-zhang: aborted active payload jobs for pull request #5508

openshift-ci-robot · 2025-12-18T17:37:12Z

@djoshy: This pull request references Jira Issue OCPBUGS-69674, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.22.0) matches configured target version for branch (4.22.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

- What I did
This change prevents the boot image controller from defaulting to the control plane architecture while performing boot image updates to a machineset whose autoscalar annotation(capacity.cluster-autoscaler.kubernetes.io/labels) is not yet present. This was causing incorrect updates in multi arch clusters, where the control plane and worker nodes are of different architectures and the arch annotation was not yet applied by the MAPI controller. Once this annotation is applied, another sync loop will be triggered, which should result in a boot image update if needed. I've also updated the unit test to account for this behavior.

It is important to note that:

This annotation is a collection of infra specific labels. If the arch label is missing within this annotation, it will still cause an error; the change is only relevant to the scenario where the whole annotation is missing.

This behavior change is only targeted at multi-arch clusters, for single arch, the controller behavior should remain the same.

- How to verify it
The existing e2es and units should be sufficient to test this behavior.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

djoshy · 2025-12-18T17:48:41Z

/payload-job periodic-ci-openshift-machine-config-operator-release-4.21-periodics-e2e-vsphere-mco-disruptive
periodic-ci-openshift-machine-config-operator-release-4.21-periodics-e2e-azure-mco-disruptive

openshift-ci · 2025-12-18T17:48:46Z

@djoshy: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-machine-config-operator-release-4.21-periodics-e2e-vsphere-mco-disruptive

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/caefb8c0-dc39-11f0-8cc1-cad44f029f3b-0

djoshy · 2025-12-18T17:49:15Z

/payload-aggregate periodic-ci-openshift-multiarch-master-nightly-4.21-ocp-e2e-aws-ovn-multi-day-0-x-a 7

/payload-job periodic-ci-openshift-machine-config-operator-release-4.21-periodics-e2e-azure-mco-disruptive

openshift-ci · 2025-12-18T17:49:30Z

@djoshy: trigger 2 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-machine-config-operator-release-4.21-periodics-e2e-azure-mco-disruptive
periodic-ci-openshift-multiarch-master-nightly-4.21-ocp-e2e-aws-ovn-multi-day-0-x-a

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/df2ed3c0-dc39-11f0-9b5f-87be6a5507dd-0

djoshy · 2025-12-18T17:56:41Z

/payload abort

urghh need to rebase

openshift-ci · 2025-12-18T17:56:43Z

@djoshy: it appears that you have attempted to use some version of the payload command, but your comment was incorrectly formatted and cannot be acted upon. See the docs for usage info.

djoshy · 2025-12-18T17:56:55Z

/payload-abort

openshift-ci · 2025-12-18T17:56:57Z

@djoshy: aborted active payload jobs for pull request #5508

djoshy · 2025-12-18T18:00:39Z

/payload-aggregate periodic-ci-openshift-multiarch-master-nightly-4.21-ocp-e2e-aws-ovn-multi-day-0-x-a 7

/payload-job periodic-ci-openshift-machine-config-operator-release-4.21-periodics-e2e-azure-mco-disruptive

/payload-job periodic-ci-openshift-machine-config-operator-release-4.21-periodics-e2e-vsphere-mco-disruptive

openshift-ci · 2025-12-18T18:00:46Z

@djoshy: trigger 3 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-machine-config-operator-release-4.21-periodics-e2e-azure-mco-disruptive
periodic-ci-openshift-machine-config-operator-release-4.21-periodics-e2e-vsphere-mco-disruptive
periodic-ci-openshift-multiarch-master-nightly-4.21-ocp-e2e-aws-ovn-multi-day-0-x-a

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/77520310-dc3b-11f0-9b79-757f652daf04-0

djoshy · 2025-12-18T18:04:07Z

/unhold

yuqi-zhang

/lgtm

Logically seems fine to me. I'd wanted to validate via the failing job, but looks like a) that's flaky and b) it's not running on the PR for some reason. Maybe we can verify later via payload runs in CI?

openshift-ci · 2025-12-18T22:23:54Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: djoshy, yuqi-zhang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [djoshy,yuqi-zhang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

djoshy · 2025-12-19T02:07:20Z

/payload-job periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-vsphere-mco-disruptive

openshift-ci · 2025-12-19T02:07:23Z

@djoshy: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-vsphere-mco-disruptive

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/73f41a20-dc7f-11f0-945d-e92d5e09a67d-0

djoshy · 2025-12-19T02:15:48Z

/payload-job periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-vsphere-mco-disruptive-techpreview-1of2 periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-vsphere-mco-disruptive-techpreview-2of2

openshift-ci · 2025-12-19T02:15:51Z

@djoshy: trigger 2 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-vsphere-mco-disruptive-techpreview-1of2
periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-vsphere-mco-disruptive-techpreview-2of2

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/a2fc3e50-dc80-11f0-9643-298efe9898c5-0

djoshy · 2025-12-19T10:39:08Z

/payload-job periodic-ci-openshift-multiarch-master-nightly-4.21-ocp-e2e-aws-ovn-multi-day-0-x-a

openshift-ci · 2025-12-19T10:39:12Z

@djoshy: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-multiarch-master-nightly-4.21-ocp-e2e-aws-ovn-multi-day-0-x-a

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/f33111c0-dcc6-11f0-83b6-0b923cfd29d1-0

djoshy · 2025-12-19T14:40:23Z

Boot image tests are passing, so I think we should just pull the trigger on this and monitor how the multi job behaves

/payload-job periodic-ci-openshift-multiarch-master-nightly-4.22-ocp-e2e-aws-ovn-multi-day-0-x-a

openshift-ci · 2025-12-19T14:40:26Z

@djoshy: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-multiarch-master-nightly-4.22-ocp-e2e-aws-ovn-multi-day-0-x-a

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/a71e2760-dce8-11f0-8e94-f4ad260f7f34-0

djoshy · 2025-12-19T14:46:46Z

/verified by e2es

openshift-ci-robot · 2025-12-19T14:46:57Z

@djoshy: This PR has been marked as verified by e2es.

Details

In response to this:

/verified by e2es

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

djoshy · 2025-12-19T14:48:11Z

/hold

djoshy · 2025-12-19T16:43:33Z

keeps failing to bootstrap, not sure if something we broke or org wide

/payload-job periodic-ci-openshift-multiarch-master-nightly-4.22-ocp-e2e-aws-ovn-multi-day-0-x-a

one more just to be sure

openshift-ci · 2025-12-19T16:43:37Z

@djoshy: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-multiarch-master-nightly-4.22-ocp-e2e-aws-ovn-multi-day-0-x-a

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/dc5f83e0-dcf9-11f0-9791-72a52f189e72-0

yuqi-zhang · 2025-12-19T18:47:48Z

/hold cancel

I don't think this will break any payloads, so let's try to get the fix in and see if we can verify if this helps tests

openshift-ci-robot · 2025-12-19T23:38:49Z

/retest-required

Remaining retests: 0 against base HEAD bf7ba38 and 2 for PR HEAD 89d1c1c in total

openshift-ci · 2025-12-20T02:53:25Z

@djoshy: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/bootstrap-unit	`551fe4c`	link	false	`/test bootstrap-unit`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

yuqi-zhang · 2025-12-24T02:10:43Z

/retest-required

openshift-ci-robot · 2025-12-24T05:13:10Z

@djoshy: Jira Issue Verification Checks: Jira Issue OCPBUGS-69674
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-69674 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

- What I did
This change prevents the boot image controller from defaulting to the control plane architecture while performing boot image updates to a machineset whose autoscalar annotation(capacity.cluster-autoscaler.kubernetes.io/labels) is not yet present. This was causing incorrect updates in multi arch clusters, where the control plane and worker nodes are of different architectures and the arch annotation was not yet applied by the MAPI controller. Once this annotation is applied, another sync loop will be triggered, which should result in a boot image update if needed. I've also updated the unit test to account for this behavior.

It is important to note that:

This annotation is a collection of infra specific labels. If the arch label is missing within this annotation, it will still cause an error; the change is only relevant to the scenario where the whole annotation is missing.

This behavior change is only targeted at multi-arch clusters, for single arch, the controller behavior should remain the same.

- How to verify it
The existing e2es and units should be sufficient to test this behavior.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-merge-robot · 2025-12-24T20:48:11Z

Fix included in accepted release 4.22.0-0.nightly-2025-12-24-152746

openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. labels Dec 18, 2025

openshift-ci bot requested review from RishabhSaini and isabella-janssen December 18, 2025 16:18

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 18, 2025

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 18, 2025

djoshy force-pushed the skip-unknown-arch branch from 551fe4c to 10e9a51 Compare December 18, 2025 17:31

msbic: skip update if arch annotation not found

89d1c1c

djoshy force-pushed the skip-unknown-arch branch from 10e9a51 to 89d1c1c Compare December 18, 2025 17:58

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 18, 2025

yuqi-zhang approved these changes Dec 18, 2025

View reviewed changes

openshift-ci bot assigned yuqi-zhang Dec 18, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Dec 18, 2025

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Dec 19, 2025

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 19, 2025

This was referenced Dec 19, 2025

[release-4.21] OCPBUGS-69965: Skip boot image updates if arch annotation is not found #5513

Merged

OCPBUGS-72394: Move operator metrics into the controller #5498

Merged

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 19, 2025

openshift-merge-bot bot merged commit 710f50d into openshift:main Dec 24, 2025
13 of 14 checks passed

djoshy deleted the skip-unknown-arch branch January 5, 2026 15:20

OCPBUGS-69674: Skip boot image updates if arch annotation is not found #5508

OCPBUGS-69674: Skip boot image updates if arch annotation is not found #5508

Uh oh!

Conversation

djoshy commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Dec 18, 2025

Uh oh!

yuqi-zhang commented Dec 18, 2025

Uh oh!

openshift-ci bot commented Dec 18, 2025

Uh oh!

yuqi-zhang commented Dec 18, 2025

Uh oh!

openshift-ci bot commented Dec 18, 2025

Uh oh!

djoshy commented Dec 18, 2025

Uh oh!

yuqi-zhang commented Dec 18, 2025

Uh oh!

openshift-ci bot commented Dec 18, 2025

Uh oh!

openshift-ci-robot commented Dec 18, 2025

Uh oh!

djoshy commented Dec 18, 2025

Uh oh!

openshift-ci bot commented Dec 18, 2025

Uh oh!

djoshy commented Dec 18, 2025

Uh oh!

openshift-ci bot commented Dec 18, 2025

Uh oh!

djoshy commented Dec 18, 2025

Uh oh!

openshift-ci bot commented Dec 18, 2025

Uh oh!

djoshy commented Dec 18, 2025

Uh oh!

openshift-ci bot commented Dec 18, 2025

Uh oh!

djoshy commented Dec 18, 2025

Uh oh!

openshift-ci bot commented Dec 18, 2025

Uh oh!

djoshy commented Dec 18, 2025

Uh oh!

yuqi-zhang left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Dec 18, 2025

Uh oh!

djoshy commented Dec 19, 2025

Uh oh!

openshift-ci bot commented Dec 19, 2025

Uh oh!

djoshy commented Dec 19, 2025

Uh oh!

openshift-ci bot commented Dec 19, 2025

Uh oh!

djoshy commented Dec 19, 2025

Uh oh!

openshift-ci bot commented Dec 19, 2025

Uh oh!

djoshy commented Dec 19, 2025

Uh oh!

openshift-ci bot commented Dec 19, 2025

Uh oh!

djoshy commented Dec 19, 2025

Uh oh!

openshift-ci-robot commented Dec 19, 2025

Uh oh!

djoshy commented Dec 19, 2025

Uh oh!

djoshy commented Dec 19, 2025

Uh oh!

openshift-ci bot commented Dec 19, 2025

Uh oh!

yuqi-zhang commented Dec 19, 2025

Uh oh!

djoshy commented Dec 18, 2025 •

edited

Loading

openshift-ci bot commented Dec 20, 2025 •

edited

Loading