OCPBUGS-57423: pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition #5119

wking · 2025-06-12T17:41:44Z

Manually picking #4760 / #5111 back to 4.16, which is using coStatus instead of coStatusCondition, because it lacks #4321. Otherwise identical:

$ diff -U1 <(git show --oneline 377a78bb9fd2) <(git show --oneline 6bf0d7135d09)
--- /dev/fd/63  2025-06-12 10:41:31.352490108 -0700
+++ /dev/fd/62  2025-06-12 10:41:31.352490108 -0700
@@ -1,7 +1,7 @@
-377a78bb9 pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition
+6bf0d7135 pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition
 diff --git a/pkg/operator/status.go b/pkg/operator/status.go
-index 63699ec5c..eb015e005 100644
+index 69d37ad29..21a5ccc16 100644
 --- a/pkg/operator/status.go
 +++ b/pkg/operator/status.go
-@@ -285,13 +285,6 @@ func (optr *Operator) syncUpgradeableStatus(co *configv1.ClusterOperator) error
+@@ -316,13 +316,6 @@ func (optr *Operator) syncUpgradeableStatus() error {
                        break
@@ -12,5 +12,5 @@
 -      if updating && !degraded && !interrupted {
--              coStatusCondition.Status = configv1.ConditionFalse
--              coStatusCondition.Reason = "PoolUpdating"
--              coStatusCondition.Message = "One or more machine config pools are updating, please see `oc get mcp` for further details"
+-              coStatus.Status = configv1.ConditionFalse
+-              coStatus.Reason = "PoolUpdating"
+-              coStatus.Message = "One or more machine config pools are updating, please see `oc get mcp` for further details"
 -      }

956e787 (Implement Upgrade-Monitor, FeatureGate, and MachineConfigNode types, 2023-11-28, openshift#4012) had added the "this should no longer trigger when adding a node to a pool" comment, but unfortunately, it's still triggering. For example, in [1]: $ curl -s https://storage.googleapis.com/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.19-e2e-gcp-ovn-serial-crun/1868424902256627712/build-log.txt | grep 'PoolUpdating' | sort | uniq time="2024-12-16T01:43:52Z" level=info msg="operator status: processing event" event="Dec 16 00:55:35.662 W clusteroperator/machine-config condition/Upgradeable reason/PoolUpdating status/False One or more machine config pools are updating, please see `oc get mcp` for further details" operator=machine-config Checking PromeCIeus, the Upgradeable=False window seems to have been 00:56 through 00:59, which correlates with the scale-up/scale-down of the serial suite: $ curl -s https://storage.googleapis.com/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.19-e2e-gcp-ovn-serial-crun/1868424902256627712/build-log.txt | grep 'Managed cluster should grow and decrease when scaling different machineSets simultaneously' started: 0/20/74 "[sig-cluster-lifecycle][Feature:Machines][Serial] Managed cluster should grow and decrease when scaling different machineSets simultaneously [Timeout:30m][apigroup:machine.openshift.io] [Suite:openshift/conformance/serial]" passed: (5m42s) 2024-12-16T00:57:49 "[sig-cluster-lifecycle][Feature:Machines][Serial] Managed cluster should grow and decrease when scaling different machineSets simultaneously [Timeout:30m][apigroup:machine.openshift.io] [Suite:openshift/conformance/serial]" confirmed via MCC logs: $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.19-e2e-gcp-ovn-serial-crun/1868424902256627712/artifacts/e2e-gcp-ovn-serial-crun/gather-extra/artifacts/pods/openshift-machine-config-operator_machine-config-controller-6f4f46457c-v8b2l_machine-config-controller.log | grep rendered- I1216 00:55:35.430231 1 node_controller.go:584] Pool worker[zone=us-central1-f]: node ci-op-k8c03v6z-9149a-r27w7-worker-f-t7rmb: changed annotation machineconfiguration.openshift.io/currentConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2 I1216 00:55:35.430252 1 node_controller.go:584] Pool worker[zone=us-central1-f]: node ci-op-k8c03v6z-9149a-r27w7-worker-f-t7rmb: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2 I1216 00:55:36.174629 1 node_controller.go:584] Pool worker[zone=us-central1-a]: node ci-op-k8c03v6z-9149a-r27w7-worker-a-f7hkj: changed annotation machineconfiguration.openshift.io/currentConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2 I1216 00:55:36.174738 1 node_controller.go:584] Pool worker[zone=us-central1-a]: node ci-op-k8c03v6z-9149a-r27w7-worker-a-f7hkj: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2 I1216 00:55:41.296273 1 node_controller.go:584] Pool worker[zone=us-central1-b]: node ci-op-k8c03v6z-9149a-r27w7-worker-b-554bt: changed annotation machineconfiguration.openshift.io/currentConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2 I1216 00:55:41.296306 1 node_controller.go:584] Pool worker[zone=us-central1-b]: node ci-op-k8c03v6z-9149a-r27w7-worker-b-554bt: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2 I1216 00:55:47.106173 1 node_controller.go:584] Pool worker[zone=us-central1-c]: node ci-op-k8c03v6z-9149a-r27w7-worker-c-hshj2: changed annotation machineconfiguration.openshift.io/currentConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2 I1216 00:55:47.106201 1 node_controller.go:584] Pool worker[zone=us-central1-c]: node ci-op-k8c03v6z-9149a-r27w7-worker-c-hshj2: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2 In this commit, I'm dropping the code that had been moving the ClusterOperator to Upgradeable=False on PoolUpdating entirely, instead of hoping that it doesn't trip. I haven't dug into why the code had still been tripping. But we want to stay Upgradeable=True while new nodes scale in, because clusters where nodes are joining should still be able to update to 4.(y+1). There are node-vs.-control-plane skew issues that should block updates to 4.(y+1), but they're enforced by the Kube API server operator [2], and don't need the MCO chipping in. [1]: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.19-e2e-gcp-ovn-serial-crun/1868424902256627712 [2]: openshift/cluster-kube-apiserver-operator@9ce4f74

wking · 2025-06-12T17:42:23Z

/jira cherry-pick OCPBUGS-57135

openshift-ci-robot · 2025-06-12T17:42:35Z

@wking: Jira Issue OCPBUGS-57135 has been cloned as Jira Issue OCPBUGS-57423. Will retitle bug to link to clone.
/retitle OCPBUGS-57423: pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition

Details

In response to this:

/jira cherry-pick OCPBUGS-57135

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2025-06-12T17:42:47Z

@wking: This pull request references Jira Issue OCPBUGS-57423, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.16.z) matches configured target version for branch (4.16.z)
bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
release note text is set and does not match the template
dependent bug Jira Issue OCPBUGS-57135 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
dependent Jira Issue OCPBUGS-57135 targets the "4.17.z" version, which is one of the valid target versions: 4.17.0, 4.17.z
bug has dependents

Requesting review from QA contact:
/cc @sergiordlr

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Manually picking #4760 / #5111 back to 4.16, which is using coStatus instead of coStatusCondition, because it lacks #4321. Otherwise identical:

$ diff -U1 <(git show --oneline 377a78bb9fd2) <(git show --oneline 6bf0d7135d09)
--- /dev/fd/63  2025-06-12 10:41:31.352490108 -0700
+++ /dev/fd/62  2025-06-12 10:41:31.352490108 -0700
@@ -1,7 +1,7 @@
-377a78bb9 pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition
+6bf0d7135 pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition
diff --git a/pkg/operator/status.go b/pkg/operator/status.go
-index 63699ec5c..eb015e005 100644
+index 69d37ad29..21a5ccc16 100644
--- a/pkg/operator/status.go
+++ b/pkg/operator/status.go
-@@ -285,13 +285,6 @@ func (optr *Operator) syncUpgradeableStatus(co *configv1.ClusterOperator) error
+@@ -316,13 +316,6 @@ func (optr *Operator) syncUpgradeableStatus() error {
                       break
@@ -12,5 +12,5 @@
-      if updating && !degraded && !interrupted {
--              coStatusCondition.Status = configv1.ConditionFalse
--              coStatusCondition.Reason = "PoolUpdating"
--              coStatusCondition.Message = "One or more machine config pools are updating, please see `oc get mcp` for further details"
+-              coStatus.Status = configv1.ConditionFalse
+-              coStatus.Reason = "PoolUpdating"
+-              coStatus.Message = "One or more machine config pools are updating, please see `oc get mcp` for further details"
-      }

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

wking · 2025-06-13T04:55:42Z

unit TestBuildControllerMultipleOptedInPools/Happy_Path_Multiple_Configs/master/CustomBuildPod panic doesn't look like it's touching any of the code I'm changing.

/retest-required

sergiordlr · 2025-06-19T10:32:24Z

/label cherry-pick-approved

sergiordlr · 2025-07-14T15:28:45Z

pre-merge verified using IPI on AWS

Install 4.15.0-0.nightly-2025-07-12-030542
Upgrade to a 4.16 release with our fix
Scale machinesets up and down, while watching the Upgradeable condition status value, making sure that is always True

oc get -w -o 'jsonpath={.status.conditions[?(@.type == "Upgradeable")]}{"\n"}' clusteroperator machine-config

Upgrade to registry.ci.openshift.org/ocp/release:4.17.0-0.nightly-2025-07-07-070757

No issues found

/label qe-approved
/label cherry-pick-approved

openshift-ci-robot · 2025-07-14T15:28:52Z

@wking: This pull request references Jira Issue OCPBUGS-57423, which is valid.

7 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.16.z) matches configured target version for branch (4.16.z)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
release note text is set and does not match the template
dependent bug Jira Issue OCPBUGS-57135 is in the state Closed (Done-Errata), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
dependent Jira Issue OCPBUGS-57135 targets the "4.17.z" version, which is one of the valid target versions: 4.17.0, 4.17.z
bug has dependents

Requesting review from QA contact:
/cc @sergiordlr

Details

In response to this:

Manually picking #4760 / #5111 back to 4.16, which is using coStatus instead of coStatusCondition, because it lacks #4321. Otherwise identical:

$ diff -U1 <(git show --oneline 377a78bb9fd2) <(git show --oneline 6bf0d7135d09)
--- /dev/fd/63  2025-06-12 10:41:31.352490108 -0700
+++ /dev/fd/62  2025-06-12 10:41:31.352490108 -0700
@@ -1,7 +1,7 @@
-377a78bb9 pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition
+6bf0d7135 pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition
diff --git a/pkg/operator/status.go b/pkg/operator/status.go
-index 63699ec5c..eb015e005 100644
+index 69d37ad29..21a5ccc16 100644
--- a/pkg/operator/status.go
+++ b/pkg/operator/status.go
-@@ -285,13 +285,6 @@ func (optr *Operator) syncUpgradeableStatus(co *configv1.ClusterOperator) error
+@@ -316,13 +316,6 @@ func (optr *Operator) syncUpgradeableStatus() error {
                       break
@@ -12,5 +12,5 @@
-      if updating && !degraded && !interrupted {
--              coStatusCondition.Status = configv1.ConditionFalse
--              coStatusCondition.Reason = "PoolUpdating"
--              coStatusCondition.Message = "One or more machine config pools are updating, please see `oc get mcp` for further details"
+-              coStatus.Status = configv1.ConditionFalse
+-              coStatus.Reason = "PoolUpdating"
+-              coStatus.Message = "One or more machine config pools are updating, please see `oc get mcp` for further details"
-      }

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

yuqi-zhang

/lgtm
/backport-risk-assessed

openshift-ci · 2025-08-20T14:23:34Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wking, yuqi-zhang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [yuqi-zhang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

dkhater-redhat · 2025-09-24T17:18:47Z

/retest-required

openshift-ci · 2025-09-24T18:09:16Z

@wking: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-azure-ovn-upgrade-out-of-change	`6bf0d71`	link	false	`/test e2e-azure-ovn-upgrade-out-of-change`
ci/prow/e2e-gcp-op-techpreview	`6bf0d71`	link	false	`/test e2e-gcp-op-techpreview`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

isabella-janssen · 2025-10-24T14:21:30Z

/verified by @sergiordlr

Applying the verified label per Sergio's verification in #5119 (comment).

openshift-ci-robot · 2025-10-24T14:21:43Z

@isabella-janssen: This PR has been marked as verified by @sergiordlr.

Details

In response to this:

/verified by @sergiordlr

Applying the verified label per Sergio's verification in #5119 (comment).

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

isabella-janssen · 2025-10-24T14:22:55Z

/label backport-risk-assessed

Looks like a clean backport that has been verified and approved by the team.

isabella-janssen · 2025-10-24T14:26:01Z

/retest-required

openshift-ci-robot · 2025-10-24T21:00:19Z

@wking: Jira Issue Verification Checks: Jira Issue OCPBUGS-57423
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-57423 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

Manually picking #4760 / #5111 back to 4.16, which is using coStatus instead of coStatusCondition, because it lacks #4321. Otherwise identical:

$ diff -U1 <(git show --oneline 377a78bb9fd2) <(git show --oneline 6bf0d7135d09)
--- /dev/fd/63  2025-06-12 10:41:31.352490108 -0700
+++ /dev/fd/62  2025-06-12 10:41:31.352490108 -0700
@@ -1,7 +1,7 @@
-377a78bb9 pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition
+6bf0d7135 pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition
diff --git a/pkg/operator/status.go b/pkg/operator/status.go
-index 63699ec5c..eb015e005 100644
+index 69d37ad29..21a5ccc16 100644
--- a/pkg/operator/status.go
+++ b/pkg/operator/status.go
-@@ -285,13 +285,6 @@ func (optr *Operator) syncUpgradeableStatus(co *configv1.ClusterOperator) error
+@@ -316,13 +316,6 @@ func (optr *Operator) syncUpgradeableStatus() error {
                       break
@@ -12,5 +12,5 @@
-      if updating && !degraded && !interrupted {
--              coStatusCondition.Status = configv1.ConditionFalse
--              coStatusCondition.Reason = "PoolUpdating"
--              coStatusCondition.Message = "One or more machine config pools are updating, please see `oc get mcp` for further details"
+-              coStatus.Status = configv1.ConditionFalse
+-              coStatus.Reason = "PoolUpdating"
+-              coStatus.Message = "One or more machine config pools are updating, please see `oc get mcp` for further details"
-      }

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-merge-robot · 2025-10-25T17:38:45Z

Fix included in accepted release 4.16.0-0.nightly-2025-10-25-135048

openshift-ci bot requested review from jkyros and sinnykumari June 12, 2025 17:42

openshift-ci bot changed the title ~~pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition~~ OCPBUGS-57423: pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition Jun 12, 2025

openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. labels Jun 12, 2025

openshift-ci bot requested a review from sergiordlr June 12, 2025 17:42

openshift-ci bot added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Jun 19, 2025

openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label Jul 14, 2025

yuqi-zhang approved these changes Aug 20, 2025

View reviewed changes

openshift-ci bot assigned yuqi-zhang Aug 20, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Aug 20, 2025

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 20, 2025

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Oct 24, 2025

openshift-ci bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Oct 24, 2025

openshift-ci bot assigned anuragthehatter and eurijon Oct 24, 2025

openshift-ci bot assigned imatza-rh, lyman9966, mhanss, ptalgulk01, rbbratta, rioliu-rh and sergiordlr Oct 24, 2025

openshift-merge-bot bot merged commit f508597 into openshift:release-4.16 Oct 24, 2025
17 of 19 checks passed

OCPBUGS-57423: pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition #5119

OCPBUGS-57423: pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition #5119

Uh oh!

Conversation

wking commented Jun 12, 2025

Uh oh!

wking commented Jun 12, 2025

Uh oh!

openshift-ci-robot commented Jun 12, 2025

Uh oh!

openshift-ci-robot commented Jun 12, 2025

Uh oh!

wking commented Jun 13, 2025

Uh oh!

sergiordlr commented Jun 19, 2025

Uh oh!

sergiordlr commented Jul 14, 2025

Uh oh!

openshift-ci-robot commented Jul 14, 2025

Uh oh!

yuqi-zhang left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Aug 20, 2025

Uh oh!

dkhater-redhat commented Sep 24, 2025

Uh oh!

openshift-ci bot commented Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

isabella-janssen commented Oct 24, 2025

Uh oh!

openshift-ci-robot commented Oct 24, 2025

Uh oh!

isabella-janssen commented Oct 24, 2025

Uh oh!

isabella-janssen commented Oct 24, 2025

Uh oh!

Uh oh!

openshift-ci-robot commented Oct 24, 2025

Uh oh!

openshift-merge-robot commented Oct 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

15 participants

openshift-ci bot commented Sep 24, 2025 •

edited

Loading