Skip to content

OCPBUGS-57135: pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition #5111

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
openshift-merge-bot merged 1 commit into from
Jun 11, 2025
Merged

OCPBUGS-57135: pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition #5111

openshift-merge-bot merged 1 commit into from
Jun 11, 2025

Conversation

@openshift-cherrypick-robot
Copy link

@openshift-cherrypick-robot openshift-cherrypick-robot commented Jun 5, 2025

This is an automated cherry-pick of #4760

/assign wking

@wking
pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition
79d012a 
956e787 (Implement Upgrade-Monitor, FeatureGate, and
MachineConfigNode types, 2023-11-28, openshift#4012) had added the "this should
no longer trigger when adding a node to a pool" comment, but
unfortunately, it's still triggering.  For example, in [1]:

  $ curl -s https://storage.googleapis.com/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.19-e2e-gcp-ovn-serial-crun/1868424902256627712/build-log.txt | grep 'PoolUpdating' | sort | uniq
  time="2024-12-16T01:43:52Z" level=info msg="operator status: processing event" event="Dec 16 00:55:35.662 W clusteroperator/machine-config condition/Upgradeable reason/PoolUpdating status/False One or more machine config pools are updating, please see `oc get mcp` for further details" operator=machine-config

Checking PromeCIeus, the Upgradeable=False window seems to have been
00:56 through 00:59, which correlates with the scale-up/scale-down of
the serial suite:

  $ curl -s https://storage.googleapis.com/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.19-e2e-gcp-ovn-serial-crun/1868424902256627712/build-log.txt | grep 'Managed cluster should grow and decrease when scaling different machineSets simultaneously'
  started: 0/20/74 "[sig-cluster-lifecycle][Feature:Machines][Serial] Managed cluster should grow and decrease when scaling different machineSets simultaneously [Timeout:30m][apigroup:machine.openshift.io] [Suite:openshift/conformance/serial]"
  passed: (5m42s) 2024-12-16T00:57:49 "[sig-cluster-lifecycle][Feature:Machines][Serial] Managed cluster should grow and decrease when scaling different machineSets simultaneously [Timeout:30m][apigroup:machine.openshift.io] [Suite:openshift/conformance/serial]"

confirmed via MCC logs:

  $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.19-e2e-gcp-ovn-serial-crun/1868424902256627712/artifacts/e2e-gcp-ovn-serial-crun/gather-extra/artifacts/pods/openshift-machine-config-operator_machine-config-controller-6f4f46457c-v8b2l_machine-config-controller.log | grep rendered-
  I1216 00:55:35.430231       1 node_controller.go:584] Pool worker[zone=us-central1-f]: node ci-op-k8c03v6z-9149a-r27w7-worker-f-t7rmb: changed annotation machineconfiguration.openshift.io/currentConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2
  I1216 00:55:35.430252       1 node_controller.go:584] Pool worker[zone=us-central1-f]: node ci-op-k8c03v6z-9149a-r27w7-worker-f-t7rmb: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2
  I1216 00:55:36.174629       1 node_controller.go:584] Pool worker[zone=us-central1-a]: node ci-op-k8c03v6z-9149a-r27w7-worker-a-f7hkj: changed annotation machineconfiguration.openshift.io/currentConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2
  I1216 00:55:36.174738       1 node_controller.go:584] Pool worker[zone=us-central1-a]: node ci-op-k8c03v6z-9149a-r27w7-worker-a-f7hkj: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2
  I1216 00:55:41.296273       1 node_controller.go:584] Pool worker[zone=us-central1-b]: node ci-op-k8c03v6z-9149a-r27w7-worker-b-554bt: changed annotation machineconfiguration.openshift.io/currentConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2
  I1216 00:55:41.296306       1 node_controller.go:584] Pool worker[zone=us-central1-b]: node ci-op-k8c03v6z-9149a-r27w7-worker-b-554bt: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2
  I1216 00:55:47.106173       1 node_controller.go:584] Pool worker[zone=us-central1-c]: node ci-op-k8c03v6z-9149a-r27w7-worker-c-hshj2: changed annotation machineconfiguration.openshift.io/currentConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2
  I1216 00:55:47.106201       1 node_controller.go:584] Pool worker[zone=us-central1-c]: node ci-op-k8c03v6z-9149a-r27w7-worker-c-hshj2: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2

In this commit, I'm dropping the code that had been moving the
ClusterOperator to Upgradeable=False on PoolUpdating entirely, instead
of hoping that it doesn't trip.  I haven't dug into why the code had
still been tripping.  But we want to stay Upgradeable=True while new
nodes scale in, because clusters where nodes are joining should still
be able to update to 4.(y+1).  There are node-vs.-control-plane skew
issues that should block updates to 4.(y+1), but they're enforced by
the Kube API server operator [2], and don't need the MCO chipping in.

[1]: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.19-e2e-gcp-ovn-serial-crun/1868424902256627712
[2]: openshift/cluster-kube-apiserver-operator@9ce4f74
@openshift-cherrypick-robot openshift-cherrypick-robot mentioned this pull request Jun 5, 2025
Merged
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Jun 5, 2025 •
edited by openshift-ci bot
Loading

@openshift-cherrypick-robot: Ignoring requests to cherry-pick non-bug issues: MCO-1482

Details

In response to this:

This is an automated cherry-pick of #4760

/assign wking

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@wking wking mentioned this pull request Jun 5, 2025
Merged
@wking
Copy link
Member

wking commented Jun 5, 2025

/jira cherry-pick OCPBUGS-56517

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Jun 5, 2025 •
edited by openshift-ci bot
Loading

@wking: Jira Issue OCPBUGS-56517 has been cloned as Jira Issue OCPBUGS-57135. Will retitle bug to link to clone.
/retitle OCPBUGS-57135: [release-4.17] MCO-1482: pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition

Details

In response to this:

/jira cherry-pick OCPBUGS-56517

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot changed the title [release-4.17] MCO-1482: pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition OCPBUGS-57135: [release-4.17] MCO-1482: pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition Jun 5, 2025
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jun 5, 2025
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Jun 5, 2025

@openshift-cherrypick-robot: This pull request references Jira Issue OCPBUGS-57135, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.17.z) matches configured target version for branch (4.17.z)
  • bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-56517 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-56517 targets the "4.18.z" version, which is one of the valid target versions: 4.18.0, 4.18.z
  • bug has dependents

Requesting review from QA contact:
/cc @sergiordlr

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

This is an automated cherry-pick of #4760

/assign wking

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot added the jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. label Jun 5, 2025
@openshift-ci openshift-ci bot requested a review from sergiordlr June 5, 2025 21:11
@wking
Copy link
Member

wking commented Jun 5, 2025

Trim out the original non-bug Jira from dev/4.19, and also drop [release-4.17], because that's structured GitHub information, and we don't need to spend pull-request title-text space on it.

/retitle OCPBUGS-57135: pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition

@openshift-ci openshift-ci bot changed the title OCPBUGS-57135: [release-4.17] MCO-1482: pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition OCPBUGS-57135: pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition Jun 5, 2025
yuqi-zhang
yuqi-zhang approved these changes Jun 5, 2025
View reviewed changes
Copy link
Contributor

@yuqi-zhang yuqi-zhang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/label backport-risk-assessed

@openshift-ci openshift-ci bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Jun 5, 2025
@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 5, 2025
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jun 5, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: openshift-cherrypick-robot, yuqi-zhang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 5, 2025
@sergiordlr
Copy link
Contributor

sergiordlr commented Jun 6, 2025

/label cherry-pick-approved

@openshift-ci openshift-ci bot added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Jun 6, 2025
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Jun 6, 2025

/retest-required

Remaining retests: 0 against base HEAD f54ce41 and 2 for PR HEAD 79d012a in total

3 similar comments
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Jun 6, 2025

/retest-required

Remaining retests: 0 against base HEAD f54ce41 and 2 for PR HEAD 79d012a in total

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Jun 9, 2025

/retest-required

Remaining retests: 0 against base HEAD f54ce41 and 2 for PR HEAD 79d012a in total

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Jun 9, 2025

/retest-required

Remaining retests: 0 against base HEAD f54ce41 and 2 for PR HEAD 79d012a in total

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Jun 9, 2025

/retest-required

Remaining retests: 0 against base HEAD ca15369 and 1 for PR HEAD 79d012a in total

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Jun 10, 2025

/retest-required

Remaining retests: 0 against base HEAD ca15369 and 2 for PR HEAD 79d012a in total

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jun 10, 2025 •
edited
Loading

@openshift-cherrypick-robot: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-azure-ovn-upgrade-out-of-change 79d012a link false /test e2e-azure-ovn-upgrade-out-of-change
ci/prow/e2e-gcp-op-techpreview 79d012a link false /test e2e-gcp-op-techpreview

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Jun 10, 2025

/retest-required

Remaining retests: 0 against base HEAD 3027d07 and 2 for PR HEAD 79d012a in total

@openshift-merge-bot openshift-merge-bot bot merged commit 5f8e2a1 into openshift:release-4.17 Jun 11, 2025
15 of 17 checks passed
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Jun 11, 2025

@openshift-cherrypick-robot: Jira Issue OCPBUGS-57135: All pull requests linked via external trackers have merged:

Jira Issue OCPBUGS-57135 has been moved to the MODIFIED state.

Details

In response to this:

This is an automated cherry-pick of #4760

/assign wking

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-bot
Copy link
Contributor

openshift-bot commented Jun 11, 2025

[ART PR BUILD NOTIFIER]

Distgit: ose-machine-config-operator
This PR has been included in build ose-machine-config-operator-container-v4.17.0-202506110310.p0.g5f8e2a1.assembly.stream.el9.
All builds following this will include this PR.

@openshift-merge-robot
Copy link
Contributor

openshift-merge-robot commented Jun 11, 2025

Fix included in accepted release 4.17.0-0.nightly-2025-06-11-184010

@wking
Copy link
Member

wking commented Jun 12, 2025

/jira cherrypick release-4.16

@wking
Copy link
Member

wking commented Jun 12, 2025

oops, not Jira, I meant to ask the Prow plugin...

/cherry-pick release-4.16

@openshift-cherrypick-robot
Copy link
Author

openshift-cherrypick-robot commented Jun 12, 2025

@wking: #5111 failed to apply on top of branch "release-4.16":

Applying: pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition
Using index info to reconstruct a base tree...
M	pkg/operator/status.go
Falling back to patching base and 3-way merge...
Auto-merging pkg/operator/status.go
CONFLICT (content): Merge conflict in pkg/operator/status.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Patch failed at 0001 pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition
Details

In response to this:

oops, not Jira, I meant to ask the Prow plugin...

/cherry-pick release-4.16

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@wking wking mentioned this pull request Jun 12, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yuqi-zhang yuqi-zhang yuqi-zhang approved these changes

@cheesesashimi cheesesashimi Awaiting requested review from cheesesashimi

@sergiordlr sergiordlr Awaiting requested review from sergiordlr

Assignees

@wking wking

@rbbratta rbbratta

@eurijon eurijon

@zhaozhanqi zhaozhanqi

@anuragthehatter anuragthehatter

@sergiordlr sergiordlr

@rlobillo rlobillo

@rioliu-rh rioliu-rh

@mhanss mhanss

@ptalgulk01 ptalgulk01

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.