OCPBUGS-84169: Handle zero-worker HyperShift clusters in daemonset rollout [4.21 backport] by weliang1 · Pull Request #2972 · openshift/cluster-network-operator

weliang1 · 2026-04-24T15:48:36Z

Summary

Backport of #2897 to release-4.21

This PR fixes control-plane upgrade blocking in zero-worker HyperShift clusters by:

Adding early return in daemonSetProgressing() when DesiredNumberScheduled == 0
Treating rollout as complete when there are no nodes to schedule pods on

Problem

In zero-worker HyperShift clusters, the ovnkube-node DaemonSet has no pods to roll out. The existing logic in daemonSetProgressing() would return true indefinitely because NumberAvailable == 0, blocking control-plane upgrades waiting for a node rollout that will never complete.

Solution

Add an early check for DesiredNumberScheduled == 0 and return false (not progressing) since there is nothing to roll out.

Backport Details

Original PR: CORENET-6871: Handle zero-worker HyperShift clusters in daemonset rollout #2897
Jira Bug: https://redhat.atlassian.net/browse/OCPBUGS-84169
Original Bug: https://redhat.atlassian.net/browse/OCPBUGS-32382
Target Version: 4.21

Cherry-picked commits

4fe8ca6 - Handle zero-worker HyperShift clusters in daemonset rollout
04718f8 - Address CodeRabbit review: add docstring and assertion messages

🤖 Generated with Claude Code

Fix daemonSetProgressing() to treat a DaemonSet with DesiredNumberScheduled == 0 as not progressing, by changing the condition from: status.NumberAvailable == 0 to: status.NumberAvailable == 0 && status.DesiredNumberScheduled > 0 This aligns with the existing logic in status_manager.go and fixes control-plane upgrade blocking in zero-worker HyperShift clusters, where the ovnkube-node DaemonSet has no pods to roll out and would otherwise block indefinitely. Add unit tests for daemonSetProgressing() covering the zero-worker case and existing progressing conditions.

- Add doc comment to TestDaemonSetProgressing describing its purpose - Add descriptive failure message to Gomega assertion so test failures report the case name and expected vs actual values Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

openshift-ci-robot · 2026-04-24T15:48:42Z

@weliang1: This pull request references Jira Issue OCPBUGS-84169, which is invalid:

release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
expected Jira Issue OCPBUGS-84169 to depend on a bug targeting a version in 4.22.0 and in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

Backport of #2897 to release-4.21

This PR fixes control-plane upgrade blocking in zero-worker HyperShift clusters by:

Adding early return in daemonSetProgressing() when DesiredNumberScheduled == 0

Treating rollout as complete when there are no nodes to schedule pods on

Problem

In zero-worker HyperShift clusters, the ovnkube-node DaemonSet has no pods to roll out. The existing logic in daemonSetProgressing() would return true indefinitely because NumberAvailable == 0, blocking control-plane upgrades waiting for a node rollout that will never complete.

Solution

Add an early check for DesiredNumberScheduled == 0 and return false (not progressing) since there is nothing to roll out.

Backport Details

Original PR: CORENET-6871: Handle zero-worker HyperShift clusters in daemonset rollout #2897

Jira Bug: https://redhat.atlassian.net/browse/OCPBUGS-84169

Original Bug: https://redhat.atlassian.net/browse/OCPBUGS-32382

Target Version: 4.21

Cherry-picked commits

4fe8ca6 - Handle zero-worker HyperShift clusters in daemonset rollout

04718f8 - Address CodeRabbit review: add docstring and assertion messages

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

coderabbitai · 2026-04-24T15:48:49Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 59a5f2e5-fca0-42ad-85de-af6ce306d801

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

weliang1 · 2026-04-24T19:35:43Z

/retest

weliang1 · 2026-04-27T13:17:38Z

/retest

weliang1 · 2026-04-27T19:29:28Z

/retest

weliang1 · 2026-04-29T19:26:00Z

/retest

weliang1 · 2026-04-30T17:31:25Z

/verified by @weliang1

openshift-ci-robot · 2026-04-30T17:31:38Z

@weliang1: This PR has been marked as verified by @weliang1.

Details

In response to this:

/verified by @weliang1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

weliang1 · 2026-04-30T17:32:38Z

/retest

weliang1 · 2026-05-01T13:34:35Z

/test e2e-azure-ovn-upgrade

weliang1 · 2026-05-04T18:52:55Z

/test e2e-azure-ovn-dualstack

weliang1 · 2026-05-05T16:19:49Z

/retest

weliang1 · 2026-05-05T22:32:13Z

/test e2e-aws-hypershift-ovn-kubevirt

weliang1 · 2026-05-06T16:03:15Z

/test e2e-azure-ovn-dualstack

weliang1 · 2026-05-06T16:03:17Z

/test e2e-vsphere-ovn-dualstack-primaryv6

weliang1 · 2026-05-06T16:03:25Z

/test e2e-openstack-ovn

weliang1 · 2026-05-06T16:03:27Z

/test e2e-aws-hypershift-ovn-kubevirt

weliang1 · 2026-05-06T19:46:39Z

/test e2e-aws-hypershift-ovn-kubevirt

weliang1 · 2026-05-06T19:46:41Z

/test e2e-azure-ovn-dualstack

weliang1 · 2026-05-06T19:46:45Z

/test e2e-vsphere-ovn-dualstack-primaryv6

weliang1 · 2026-05-07T14:36:05Z

Re test the failed many times, below analysis approved all failures are not related the new code itself:

Test Failure Analysis Complete                                                                                                                                             
                                                                                                                                                                             
Prow Job: pull-ci-openshift-cluster-network-operator-release-4.21-e2e-vsphere-ovn-dualstack-primaryv6                                                                      
Build ID: 2052112886200799232                 
Error: Cluster installation failed - failed to provision control-plane machines within 15m0s                                                                                                                                                                                                                                                                                                                                                                                                                             
Root Cause                                                                                                                                                                                                                                                                                                                                            
This is a vSphere CI infrastructure issue, NOT a code issue with PR #2972.                                                                                                 
The installation failed because IPv6 DHCP is disabled in the vSphere test environment, but it's required for a dualstack primaryv6 deployment.     

Prow Job: pull-ci-openshift-cluster-network-operator-release-4.21-e2e-openstack-ovn                                                                                        
Build ID: 2052056976401108992                                                                                                                                              
Error: 1 blocking test failure out of 1832 passed tests                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
Root Cause                                                                                                                                                                                                                                                                                                                                           
This is an OpenStack infrastructure issue, NOT a code issue with PR #2972.                                                                                                                                                                                                                                                                           
The test suite failed due to insufficient CPU resources and a severe metric overflow bug in the OpenStack worker node.     

Prow Job: pull-ci-openshift-cluster-network-operator-release-4.21-e2e-azure-ovn-dualstack                                                                                  
Build ID: 2052112873068433408                             
Error: Cluster installation failed during bootstrap - timed out waiting for network operator                                                                                                                                                                                                                                                                                                                                                                                                                            
Root Cause                                                                                                                                                                                                                                                                                                                                             
This is an Azure infrastructure issue, NOT a code issue with PR #2972.                                                                                                                                                                                                                                                                                 
The installation failed because Azure Load Balancer NAT rules conflicted during master node provisioning.         

Prow Job: pull-ci-openshift-cluster-network-operator-release-4.21-e2e-aws-hypershift-ovn-kubevirt                                                                          
Build ID: 2052112860506492928                                                                                                                                              
Error: Massive test suite failure - 110 blocking failures out of 1764 tests                                                                                                                                                                                                                                                                                                                                                                                                                                      
Root Cause                                                                                                                                                                                                                                                                                                                                            
This is a HyperShift KubeVirt infrastructure issue, NOT a code issue with PR #2972.                                                                                                                                                                                                                                                                  
The test suite failed due to severe API server performance degradation caused by client rate limiting and widespread timeouts.

@kyrtapz Please override failed job and approve this PR.

kyrtapz · 2026-05-08T07:18:51Z

@weliang1 why didn't you use cherry-pick automation?
/lgtm
/label backport-risk-assessed
/retest-required

retesting required jobs, infra issues likely mean that CNO didn't get a chance to prove it works fine.
We don't usually just override all failing required jobs without a bug reported.

openshift-ci · 2026-05-08T07:19:11Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kyrtapz, weliang1

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [kyrtapz]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kyrtapz · 2026-05-08T09:15:26Z

/test e2e-aws-hypershift-ovn-kubevirt

weliang1 · 2026-05-11T14:48:24Z

@weliang1 why didn't you use cherry-pick automation? /lgtm /label backport-risk-assessed /retest-required

retesting required jobs, infra issues likely mean that CNO didn't get a chance to prove it works fine. We don't usually just override all failing required jobs without a bug reported.

@kyrtapz Should I close this PR and using cherry-pick automation to create a new one?

kyrtapz · 2026-05-11T16:48:30Z

@weliang1 why didn't you use cherry-pick automation? /lgtm /label backport-risk-assessed /retest-required
retesting required jobs, infra issues likely mean that CNO didn't get a chance to prove it works fine. We don't usually just override all failing required jobs without a bug reported.

@kyrtapz Should I close this PR and using cherry-pick automation to create a new one?

It's fine to keep it, I was just interested.

weliang1 · 2026-05-11T18:12:47Z

@weliang1 why didn't you use cherry-pick automation? /lgtm /label backport-risk-assessed /retest-required
retesting required jobs, infra issues likely mean that CNO didn't get a chance to prove it works fine. We don't usually just override all failing required jobs without a bug reported.

@kyrtapz Should I close this PR and using cherry-pick automation to create a new one?

It's fine to keep it, I was just interested.

@kyrtapz What should I do next to merge this PR? Same backport need for 4.20 and 4.19

kyrtapz · 2026-05-12T07:29:39Z

@weliang1 why didn't you use cherry-pick automation? /lgtm /label backport-risk-assessed /retest-required
retesting required jobs, infra issues likely mean that CNO didn't get a chance to prove it works fine. We don't usually just override all failing required jobs without a bug reported.

@kyrtapz Should I close this PR and using cherry-pick automation to create a new one?

It's fine to keep it, I was just interested.

@kyrtapz What should I do next to merge this PR? Same backport need for 4.20 and 4.19

You can see what is missing in the tide job:

[tide](https://prow.ci.openshift.org/pr?query=is%3Apr+repo%3Aopenshift%2Fcluster-network-operator+author%3Aweliang1+head%3AOCPBUGS-84169-backport-4.21)
tideWaiting for status to be reported — Not mergeable. Needs jira/valid-bug label.

You will need to create a 4.22.0 bug and close it as done:

This pull request references [Jira Issue OCPBUGS-84169](https://redhat.atlassian.net/browse/OCPBUGS-84169), which is invalid:

    release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the [OpenShift Bug Process](https://source.redhat.com/groups/public/openshift/openshift_wiki/openshift_bugzilla_process#doc-text-for-bugs).
    expected [Jira Issue OCPBUGS-84169](https://redhat.atlassian.net/browse/OCPBUGS-84169) to depend on a bug targeting a version in 4.22.0 and in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but no dependents were found

kyrtapz · 2026-05-12T07:30:23Z

/jira refresh

openshift-ci-robot · 2026-05-12T07:30:28Z

@kyrtapz: This pull request references Jira Issue OCPBUGS-84169, which is invalid:

release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
expected Jira Issue OCPBUGS-84169 to depend on a bug targeting a version in 4.22.0 and in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

weliang1 · 2026-05-14T14:58:18Z

/jira refresh

openshift-ci-robot · 2026-05-14T14:58:23Z

@weliang1: This pull request references Jira Issue OCPBUGS-84169, which is invalid:

expected dependent Jira Issue OCPBUGS-85657 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is ON_QA instead
expected dependent Jira Issue OCPBUGS-85657 to target a version in 4.22.0, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

weliang1 · 2026-05-14T14:59:10Z

/jira refresh

openshift-ci-robot · 2026-05-14T14:59:15Z

@weliang1: This pull request references Jira Issue OCPBUGS-84169, which is invalid:

expected dependent Jira Issue OCPBUGS-85657 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is ON_QA instead
expected dependent Jira Issue OCPBUGS-85657 to target a version in 4.22.0, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

weliang1 · 2026-05-14T15:00:04Z

Jira Bug Resolution Status

I've completed most of the setup to resolve the Jira validation issue:

✅ Completed Actions

Created OCPBUGS-85657 - 4.22.0 parent bug
- Status: Currently ON_QA (automation prevents VERIFIED transition without Story Points)
- Target Version: 4.22.0 ✓
- Release Notes: Added ✓
- Linked to PR CORENET-6871: Handle zero-worker HyperShift clusters in daemonset rollout #2897 (already merged to master) ✓
Created dependency link: OCPBUGS-84169 (4.21) → depends on → OCPBUGS-85657 (4.22) ✓
Added release notes to OCPBUGS-84169 ✓

⚠️ Remaining Issue

OCPBUGS-85657 keeps reverting from VERIFIED to ON_QA due to Jira automation rules requiring:

Story Points
Original Story Points
Sprint label

These fields are typically managed by the development team/PM, not QE.

📋 Next Steps

@kyrtapz Since you've already approved this PR with /lgtm, could you please either:

Set Story Points and Sprint on OCPBUGS-85657 and transition it to VERIFIED, or
Override the Jira validation using /jira valid-bug or similar command to allow merge

The code changes are approved and verified - this is purely a Jira workflow/automation issue.

weliang1 · 2026-05-14T15:08:01Z

/jira refresh

openshift-ci-robot · 2026-05-14T15:08:05Z

@weliang1: This pull request references Jira Issue OCPBUGS-84169, which is invalid:

expected dependent Jira Issue OCPBUGS-85657 to target a version in 4.22.0, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

weliang1 · 2026-05-14T15:08:48Z

/jira refresh

openshift-ci-robot · 2026-05-14T15:08:54Z

@weliang1: This pull request references Jira Issue OCPBUGS-84169, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.21.z) matches configured target version for branch (4.21.z)
bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
release note text is set and does not match the template
dependent bug Jira Issue OCPBUGS-85657 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
dependent Jira Issue OCPBUGS-85657 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
bug has dependents

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

weliang1 · 2026-05-14T15:12:16Z

/retest

openshift-ci-robot · 2026-05-14T15:13:09Z

@weliang1: Jira Issue Verification Checks: Jira Issue OCPBUGS-84169
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-84169 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

Summary

Backport of #2897 to release-4.21

This PR fixes control-plane upgrade blocking in zero-worker HyperShift clusters by:

Adding early return in daemonSetProgressing() when DesiredNumberScheduled == 0

Treating rollout as complete when there are no nodes to schedule pods on

Problem

In zero-worker HyperShift clusters, the ovnkube-node DaemonSet has no pods to roll out. The existing logic in daemonSetProgressing() would return true indefinitely because NumberAvailable == 0, blocking control-plane upgrades waiting for a node rollout that will never complete.

Solution

Add an early check for DesiredNumberScheduled == 0 and return false (not progressing) since there is nothing to roll out.

Backport Details

Original PR: CORENET-6871: Handle zero-worker HyperShift clusters in daemonset rollout #2897

Jira Bug: https://redhat.atlassian.net/browse/OCPBUGS-84169

Original Bug: https://redhat.atlassian.net/browse/OCPBUGS-32382

Target Version: 4.21

Cherry-picked commits

4fe8ca6 - Handle zero-worker HyperShift clusters in daemonset rollout

04718f8 - Address CodeRabbit review: add docstring and assertion messages

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

weliang1 and others added 2 commits April 24, 2026 11:48

openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Apr 24, 2026

openshift-ci Bot requested review from arkadeepsen and jcaamano April 24, 2026 15:49

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Apr 30, 2026

openshift-ci Bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label May 8, 2026

openshift-ci Bot assigned imatza-rh, jechen0648, mffiedler, rbbratta, weliang1 and kyrtapz May 8, 2026

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label May 8, 2026

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 8, 2026

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels May 14, 2026

openshift-merge-bot Bot merged commit b5955e9 into openshift:release-4.21 May 14, 2026
48 of 53 checks passed

Uh oh!

Conversation

weliang1 commented Apr 24, 2026

Summary

Problem

Solution

Backport Details

Cherry-picked commits

Uh oh!

openshift-ci-robot commented Apr 24, 2026

Summary

Problem

Solution

Backport Details

Cherry-picked commits

Uh oh!

coderabbitai Bot commented Apr 24, 2026

Review skipped

Uh oh!

weliang1 commented Apr 24, 2026

Uh oh!

weliang1 commented Apr 27, 2026

Uh oh!

weliang1 commented Apr 27, 2026

Uh oh!

weliang1 commented Apr 29, 2026

Uh oh!

weliang1 commented Apr 30, 2026

Uh oh!

openshift-ci-robot commented Apr 30, 2026

Uh oh!

weliang1 commented Apr 30, 2026

Uh oh!

weliang1 commented May 1, 2026

Uh oh!

weliang1 commented May 4, 2026

Uh oh!

weliang1 commented May 5, 2026

Uh oh!

weliang1 commented May 5, 2026

Uh oh!

weliang1 commented May 6, 2026

Uh oh!

weliang1 commented May 6, 2026

Uh oh!

weliang1 commented May 6, 2026

Uh oh!

weliang1 commented May 6, 2026

Uh oh!

weliang1 commented May 6, 2026

Uh oh!

weliang1 commented May 6, 2026

Uh oh!

weliang1 commented May 6, 2026

Uh oh!

weliang1 commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kyrtapz commented May 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci Bot commented May 8, 2026

Uh oh!

kyrtapz commented May 8, 2026

Uh oh!

weliang1 commented May 11, 2026

Uh oh!

kyrtapz commented May 11, 2026

Uh oh!

weliang1 commented May 11, 2026

Uh oh!

kyrtapz commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kyrtapz commented May 12, 2026

Uh oh!

openshift-ci-robot commented May 12, 2026

Uh oh!

weliang1 commented May 14, 2026

Uh oh!

weliang1 commented May 7, 2026 •

edited

Loading

kyrtapz commented May 8, 2026 •

edited

Loading

kyrtapz commented May 12, 2026 •

edited

Loading