OCPBUGS-56517: pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition #5065

openshift-cherrypick-robot · 2025-05-20T15:51:22Z

This is an automated cherry-pick of #4760

/assign wking

956e787 (Implement Upgrade-Monitor, FeatureGate, and MachineConfigNode types, 2023-11-28, openshift#4012) had added the "this should no longer trigger when adding a node to a pool" comment, but unfortunately, it's still triggering. For example, in [1]: $ curl -s https://storage.googleapis.com/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.19-e2e-gcp-ovn-serial-crun/1868424902256627712/build-log.txt | grep 'PoolUpdating' | sort | uniq time="2024-12-16T01:43:52Z" level=info msg="operator status: processing event" event="Dec 16 00:55:35.662 W clusteroperator/machine-config condition/Upgradeable reason/PoolUpdating status/False One or more machine config pools are updating, please see `oc get mcp` for further details" operator=machine-config Checking PromeCIeus, the Upgradeable=False window seems to have been 00:56 through 00:59, which correlates with the scale-up/scale-down of the serial suite: $ curl -s https://storage.googleapis.com/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.19-e2e-gcp-ovn-serial-crun/1868424902256627712/build-log.txt | grep 'Managed cluster should grow and decrease when scaling different machineSets simultaneously' started: 0/20/74 "[sig-cluster-lifecycle][Feature:Machines][Serial] Managed cluster should grow and decrease when scaling different machineSets simultaneously [Timeout:30m][apigroup:machine.openshift.io] [Suite:openshift/conformance/serial]" passed: (5m42s) 2024-12-16T00:57:49 "[sig-cluster-lifecycle][Feature:Machines][Serial] Managed cluster should grow and decrease when scaling different machineSets simultaneously [Timeout:30m][apigroup:machine.openshift.io] [Suite:openshift/conformance/serial]" confirmed via MCC logs: $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.19-e2e-gcp-ovn-serial-crun/1868424902256627712/artifacts/e2e-gcp-ovn-serial-crun/gather-extra/artifacts/pods/openshift-machine-config-operator_machine-config-controller-6f4f46457c-v8b2l_machine-config-controller.log | grep rendered- I1216 00:55:35.430231 1 node_controller.go:584] Pool worker[zone=us-central1-f]: node ci-op-k8c03v6z-9149a-r27w7-worker-f-t7rmb: changed annotation machineconfiguration.openshift.io/currentConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2 I1216 00:55:35.430252 1 node_controller.go:584] Pool worker[zone=us-central1-f]: node ci-op-k8c03v6z-9149a-r27w7-worker-f-t7rmb: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2 I1216 00:55:36.174629 1 node_controller.go:584] Pool worker[zone=us-central1-a]: node ci-op-k8c03v6z-9149a-r27w7-worker-a-f7hkj: changed annotation machineconfiguration.openshift.io/currentConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2 I1216 00:55:36.174738 1 node_controller.go:584] Pool worker[zone=us-central1-a]: node ci-op-k8c03v6z-9149a-r27w7-worker-a-f7hkj: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2 I1216 00:55:41.296273 1 node_controller.go:584] Pool worker[zone=us-central1-b]: node ci-op-k8c03v6z-9149a-r27w7-worker-b-554bt: changed annotation machineconfiguration.openshift.io/currentConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2 I1216 00:55:41.296306 1 node_controller.go:584] Pool worker[zone=us-central1-b]: node ci-op-k8c03v6z-9149a-r27w7-worker-b-554bt: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2 I1216 00:55:47.106173 1 node_controller.go:584] Pool worker[zone=us-central1-c]: node ci-op-k8c03v6z-9149a-r27w7-worker-c-hshj2: changed annotation machineconfiguration.openshift.io/currentConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2 I1216 00:55:47.106201 1 node_controller.go:584] Pool worker[zone=us-central1-c]: node ci-op-k8c03v6z-9149a-r27w7-worker-c-hshj2: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2 In this commit, I'm dropping the code that had been moving the ClusterOperator to Upgradeable=False on PoolUpdating entirely, instead of hoping that it doesn't trip. I haven't dug into why the code had still been tripping. But we want to stay Upgradeable=True while new nodes scale in, because clusters where nodes are joining should still be able to update to 4.(y+1). There are node-vs.-control-plane skew issues that should block updates to 4.(y+1), but they're enforced by the Kube API server operator [2], and don't need the MCO chipping in. [1]: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.19-e2e-gcp-ovn-serial-crun/1868424902256627712 [2]: openshift/cluster-kube-apiserver-operator@9ce4f74

openshift-ci-robot · 2025-05-20T15:51:25Z

@openshift-cherrypick-robot: Ignoring requests to cherry-pick non-bug issues: MCO-1482

Details

In response to this:

This is an automated cherry-pick of #4760

/assign wking

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

wking · 2025-05-20T16:53:11Z

/jira cherrypick OCPBUGS-56516

wking · 2025-05-20T17:23:27Z

Maybe the message got lost. Trying again:

/jira cherrypick OCPBUGS-56516

openshift-ci-robot · 2025-05-20T17:23:35Z

@wking: Jira Issue OCPBUGS-56516 has been cloned as Jira Issue OCPBUGS-56517. Will retitle bug to link to clone.
/retitle OCPBUGS-56517: [release-4.18] MCO-1482: pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition

Details

In response to this:

Maybe the message got lost. Trying again:

/jira cherrypick OCPBUGS-56516

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2025-05-20T17:24:27Z

@openshift-cherrypick-robot: An error was encountered updating to the POST state for bug OCPBUGS-56517 on the Jira server at https://issues.redhat.com/. No known errors were detected, please see the full error message for details.

Full error message.


No response returned: Post "https://issues.redhat.com/rest/api/2/issue/16995460/transitions": POST https://issues.redhat.com/rest/api/2/issue/16995460/transitions giving up after 5 attempt(s)

Please contact an administrator to resolve this issue, then request a bug refresh with /jira refresh.

Details

In response to this:

This is an automated cherry-pick of #4760

/assign wking

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

In 4.19: * 377a78b (pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition, 2024-12-16, openshift#4760). * 0c21907 (pkg/operator/status: Drop kubelet skew guard, 2025-04-03, openshift#4970). But in 4.18, we're using the other order: * 13cceb0 (pkg/operator/status: Drop kubelet skew guard, add RHEL guard, 2025-03-26, openshift#4956). * 20fe075 (pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition, 2024-12-16, openshift#5065). So I'm adding this follow-up commit within openshift#5065 to remove the 'updating' variable that both the kubelet-skew-guard and the PoolUpdating guard had used, but which we no longer need now that both are gone in 4.18.

wking · 2025-05-20T23:12:42Z

/jira refresh
/retest-required

openshift-ci-robot · 2025-05-20T23:12:49Z

@wking: This pull request references Jira Issue OCPBUGS-56517, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.18.z) matches configured target version for branch (4.18.z)
bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
release note text is set and does not match the template
dependent bug Jira Issue OCPBUGS-56516 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
dependent Jira Issue OCPBUGS-56516 targets the "4.19.0" version, which is one of the valid target versions: 4.19.0
bug has dependents

Requesting review from QA contact:
/cc @sergiordlr

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

/jira refresh
/retest-required

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

yuqi-zhang

/lgtm
/label backport-risk-assessed

Technically a slight change in behaviour but generally should only help unblock upgrades

openshift-ci · 2025-05-21T00:48:55Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: openshift-cherrypick-robot, yuqi-zhang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [yuqi-zhang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci · 2025-05-21T05:13:23Z

@openshift-cherrypick-robot: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-gcp-op-ocl	`69be318`	link	false	`/test e2e-gcp-op-ocl`
ci/prow/e2e-azure-ovn-upgrade-out-of-change	`69be318`	link	false	`/test e2e-azure-ovn-upgrade-out-of-change`
ci/prow/e2e-gcp-op-techpreview	`69be318`	link	false	`/test e2e-gcp-op-techpreview`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

sergiordlr · 2025-06-04T17:23:23Z

/label cherry-pick-approved

openshift-ci-robot · 2025-06-04T17:30:33Z

/retest-required

Remaining retests: 0 against base HEAD 00143af and 2 for PR HEAD 69be318 in total

openshift-ci-robot · 2025-06-05T02:04:19Z

@openshift-cherrypick-robot: Jira Issue OCPBUGS-56517: All pull requests linked via external trackers have merged:

openshift/machine-config-operator#5065

Jira Issue OCPBUGS-56517 has been moved to the MODIFIED state.

Details

In response to this:

This is an automated cherry-pick of #4760

/assign wking

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-bot · 2025-06-05T05:16:29Z

[ART PR BUILD NOTIFIER]

Distgit: ose-machine-config-operator
This PR has been included in build ose-machine-config-operator-container-v4.18.0-202506050433.p0.g255436e.assembly.stream.el9.
All builds following this will include this PR.

openshift-merge-robot · 2025-06-05T14:30:35Z

Fix included in accepted release 4.18.0-0.nightly-2025-06-05-073427

wking · 2025-06-05T21:02:10Z

/cherrypick release-4.17

openshift-cherrypick-robot · 2025-06-05T21:02:55Z

@wking: #5065 failed to apply on top of branch "release-4.17":

Applying: pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition
Using index info to reconstruct a base tree...
M	pkg/operator/status.go
Falling back to patching base and 3-way merge...
Auto-merging pkg/operator/status.go
Applying: pkg/operator/status: Drop unused 'updating' variable
Using index info to reconstruct a base tree...
M	pkg/operator/status.go
Falling back to patching base and 3-way merge...
Auto-merging pkg/operator/status.go
CONFLICT (content): Merge conflict in pkg/operator/status.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Patch failed at 0002 pkg/operator/status: Drop unused 'updating' variable

Details

In response to this:

/cherrypick release-4.17

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

wking · 2025-06-05T21:10:27Z

Picked directly from dev/4.19's #4760 back to 4.17 with #5111, because 4.17 doesn't need this pull's 69be318 backport-ordering workaround.

openshift-cherrypick-robot assigned wking May 20, 2025

openshift-cherrypick-robot mentioned this pull request May 20, 2025

MCO-1482: pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition #4760

Merged

openshift-ci bot requested review from RishabhSaini and cheesesashimi May 20, 2025 15:52

openshift-ci bot changed the title ~~[release-4.18] MCO-1482: pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition~~ OCPBUGS-56517: [release-4.18] MCO-1482: pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition May 20, 2025

wking changed the title ~~OCPBUGS-56517: [release-4.18] MCO-1482: pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition~~ OCPBUGS-56517: pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition May 20, 2025

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label May 20, 2025

openshift-ci-robot added the jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. label May 20, 2025

openshift-ci bot requested a review from sergiordlr May 20, 2025 23:12

yuqi-zhang approved these changes May 21, 2025

View reviewed changes

openshift-ci bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label May 21, 2025

openshift-ci bot assigned anuragthehatter, eurijon, mhanss, ptalgulk01, rbbratta, rioliu-rh, rlobillo, sergiordlr and zhaozhanqi May 21, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label May 21, 2025

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 21, 2025

openshift-ci bot added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Jun 4, 2025

openshift-merge-bot bot merged commit 255436e into openshift:release-4.18 Jun 5, 2025
16 of 19 checks passed

OCPBUGS-56517: pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition #5065

OCPBUGS-56517: pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition #5065

Uh oh!

Conversation

openshift-cherrypick-robot commented May 20, 2025

Uh oh!

openshift-ci-robot commented May 20, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wking commented May 20, 2025

Uh oh!

wking commented May 20, 2025

Uh oh!

openshift-ci-robot commented May 20, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented May 20, 2025

Uh oh!

wking commented May 20, 2025

Uh oh!

openshift-ci-robot commented May 20, 2025

Uh oh!

yuqi-zhang left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented May 21, 2025

Uh oh!

openshift-ci bot commented May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sergiordlr commented Jun 4, 2025

Uh oh!

openshift-ci-robot commented Jun 4, 2025

Uh oh!

Uh oh!

openshift-ci-robot commented Jun 5, 2025

Uh oh!

openshift-bot commented Jun 5, 2025

Uh oh!

openshift-merge-robot commented Jun 5, 2025

Uh oh!

wking commented Jun 5, 2025

Uh oh!

openshift-cherrypick-robot commented Jun 5, 2025

Uh oh!

wking commented Jun 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

15 participants

openshift-ci-robot commented May 20, 2025 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented May 20, 2025 •

edited by openshift-ci bot

Loading

openshift-ci bot commented May 21, 2025 •

edited

Loading