OCPBUGS-88531: Remove CPO-side restart logic for CNO operands

[bryan-cox]

Summary

• Remove redundant CPO-side restart-date annotation propagation for multus-admission-controller, network-node-identity, and ovnkube-control-plane
• CNO handles restart-date propagation to all its operands directly, making the CPO-side logic redundant

The cleanupClusterNetworkOperatorResources function previously contained logic to read hypershift.openshift.io/restart-date from the HCP and patch it onto CNO-managed deployments. CNO reads the annotation from HCP itself and sets it as a pod template annotation on all rendered workloads, so the CPO-side logic is unnecessary duplication.

Test plan

Tested and verified. Test verification report here - https://bryan-cox.github.io/architectural-artifact-sharing/test-verification-report-ocpbugs-84239/index.html.

Summary by CodeRabbit

• Refactor
  • Simplified cluster network operator resource cleanup by removing restart-annotation handling for CNO-managed components.
  • Cleanup now focuses directly on removing the ovnkube-sbdb Route when applicable, and deleting the ovnkube-master-external and ovnkube-master-internal Services.

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[openshift-ci-robot]

@bryan-cox: This pull request references Jira Issue OCPBUGS-88531, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

• Remove redundant CPO-side restart-date annotation propagation for multus-admission-controller, network-node-identity, and ovnkube-control-plane
• CNO now handles restart-date propagation to all its operands directly via OCPBUGS-88531: Propagate restart-date annotation to CNO operand pod templates cluster-network-operator#3030

The cleanupClusterNetworkOperatorResources function previously contained logic to read hypershift.openshift.io/restart-date from the HCP and patch it onto CNO-managed deployments. With the CNO fix, CNO reads the annotation from HCP itself and sets it as a pod template annotation on all rendered workloads, making the CPO-side logic redundant.

Test plan

• Verify make build passes (confirmed locally)
• Deploy with CNO PR HOSTEDCP-1205: Move network policies reconciliation to the CPO #3030 and confirm all CNO operands restart when restart-date annotation is set on HostedCluster
• CI: e2e-aws-ovn-hypershift-conformance, hypershift-e2e-aks

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

📝 Walkthrough

Walkthrough

In cleanupClusterNetworkOperatorResources on HostedControlPlaneReconciler, the 24-line block that read hyperv1.RestartDateAnnotation from the HostedControlPlane annotations and patched it onto the multus-admission-controller, network-node-identity, and ovnkube-control-plane deployments has been deleted. The function now proceeds directly to its remaining tasks: conditionally deleting the ovnkube-sbdb Route when hasRouteCap is true, and unconditionally deleting the ovnkube-master-external and ovnkube-master-internal Services.

🚥 Pre-merge checks | ✅ 11

✅ Passed checks (11 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically describes the main change: removing CPO-side restart logic for CNO operands, which aligns directly with the changeset that removes restart-date annotation propagation logic.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	PR modifies only controller code (hostedcontrolplane_controller.go), not test files. No Ginkgo tests to verify, so check is not applicable.
Test Structure And Quality	✅ Passed	The PR does not contain Ginkgo test code. The test file added uses standard Go unit testing with Gomega assertions, not Ginkgo BDD patterns (Describe, It, BeforeEach, Eventually, etc.), so the Gink...
Topology-Aware Scheduling Compatibility	✅ Passed	PR removes annotation propagation code from cleanupClusterNetworkOperatorResources; no new deployment manifests, scheduling constraints, affinity rules, topology spread constraints, nodeSelectors,...
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	No new Ginkgo e2e tests are added in this PR. The changes only remove 24 lines of annotation propagation logic from hostedcontrolplane_controller.go, making this check not applicable.
No-Weak-Crypto	✅ Passed	No weak cryptography detected. The PR removes annotation propagation logic, not crypto code. The file uses only standard Go crypto/rand for secure randomness.
Container-Privileges	✅ Passed	PR modifies Go controller code only (hostedcontrolplane_controller.go), removing restart annotation logic. No Kubernetes manifests, container specs, or security configurations present.
No-Sensitive-Data-In-Logs	✅ Passed	The modified cleanupClusterNetworkOperatorResources function contains no logging statements that expose passwords, tokens, API keys, PII, session IDs, internal hostnames, or customer data. Logging...
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bryan-cox

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [bryan-cox]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 41.86%. Comparing base (392fd5a) to head (407530a).
⚠️ Report is 322 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8751      +/-   ##
==========================================
+ Coverage   41.75%   41.86%   +0.11%     
==========================================
  Files         758      759       +1     
  Lines       93981    94019      +38     
==========================================
+ Hits        39240    39362     +122     
+ Misses      51988    51902      -86     
- Partials     2753     2755       +2     

Files with missing lines	Coverage Δ
...ostedcontrolplane/hostedcontrolplane_controller.go	46.04% <ø> (+0.33%)	⬆️
...or/controllers/hostedcontrolplane/manifests/cno.go	0.00% <ø> (ø)
...controllers/hostedcontrolplane/v2/cno/component.go	5.94% <ø> (+0.85%)	⬆️

... and 13 files with indirect coverage changes

Flag	Coverage Δ
cmd-support	35.13% <ø> (+0.10%)	⬆️
cpo-hostedcontrolplane	44.25% <ø> (+0.15%)	⬆️
cpo-other	43.45% <ø> (ø)
hypershift-operator	52.02% <ø> (+0.19%)	⬆️
other	31.56% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[jparrill]

/hold

Dropped some comments. Thanks!

Holding until companion PR from cluster-network-operator#3030 got merged. Feel free to remove it when that happens.

[jparrill]

This function has zero callers after your PR — the three call sites you removed were its only consumers. Worth deleting it here to avoid leaving dead exported code in the package. Same for MultusAdmissionControllerDeployment, NetworkNodeIdentityDeployment, OVNKubeControlPlaneDeployment and their constants in manifests/cno.go — all orphaned now.

Happy to help with a follow-up if you prefer to keep this PR minimal, but since it's all in the same ownership boundary it fits naturally here.

[bryan-cox]

Isn't it still called in

hypershift/control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go

Line 1234 in 4755e9c

if err := r.cleanupClusterNetworkOperatorResources(ctx, hcp, r.ManagementClusterCapabilities.Has(capabilities.CapabilityRoute)); err != nil {

?

[mgencur]

I was going to comment the same - let's remove the function now because it's unused.

Isn't it still called in

The function SetRestartAnnotationAndPatch was only called in this cleanupClusterNetworkOperatorResources.
See:

ᐅ grep -Ri SetRestartAnnotationAndPatch                            
control-plane-operator/controllers/hostedcontrolplane/v2/cno/component.go:func SetRestartAnnotationAndPatch(ctx context.Context, crclient client.Client, dep *appsv1.Deployment, restartAnnotation string) error {
control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go:			if err := cnov2.SetRestartAnnotationAndPatch(ctx, r.Client, multusDeployment, restartAnnotation); err != nil {
control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go:		if err := cnov2.SetRestartAnnotationAndPatch(ctx, r.Client, networkNodeIdentityDeployment, restartAnnotation); err != nil {
control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go:		// CNO manages overall ovnkube-control-plane deployment. CPO manages restarts.  Note that cnov2.SetRestartAnnotationAndPatch just returns err == nil if the deployment isn't found (so if OVN isn't being used)
control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go:		if err := cnov2.SetRestartAnnotationAndPatch(ctx, r.Client, ovnKubeControlPlaneDeployment, restartAnnotation); err != nil {

[bryan-cox]

Done — removed SetRestartAnnotationAndPatch and the orphaned manifest helpers (MultusAdmissionControllerDeployment, NetworkNodeIdentityDeployment, OVNKubeControlPlaneDeployment) along with their unused constants and imports.

[mgencur]

/lgtm

[openshift-merge-bot]

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-v2-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke

[hypershift-jira-solve-ci]

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-aks | Build: 2067559611954630656 | Cost: $3.23479775 | Failed step: hypershift-azure-run-e2e

View full analysis report

---

_{Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6}

[cwbotbot]

Test Results

e2e-aws

• Status: ❌ FAIL
• Started: 2026-06-18T10:46:50Z
• View Job
• View Job History

e2e-aks

• Status: ❌ FAIL
• Started: 2026-06-18T10:46:50Z
• View Job
• View Job History

Failed Tests

Total failed tests: 3

• TestNodePool
• TestNodePool/HostedCluster0
• TestNodePool/HostedCluster0/EnsureHostedCluster

[bryan-cox]

/hold cancel

CNO PR merged

[bryan-cox]

/retest

[hypershift-jira-solve-ci]

Now I have a clear picture. Let me produce the final report:

Test Failure Analysis Complete

Job Information

• Prow Job: pull-ci-openshift-hypershift-main-e2e-aws
• Build ID: 2076678412021272576
• Target: e2e-aws
• PR: OCPBUGS-88531: Remove CPO-side restart logic for CNO operands #8751 (OCPBUGS-88531: Remove CPO-side restart logic for CNO operands)
• Result: 556 passed, 30 skipped, 10 failures out of 566 tests

Test Failure Analysis

Error

DONE 566 tests, 30 skipped, 10 failures in 0.019s

Leaf failures:
1. TestKarpenterUpgradeControlPlane/EnsureNoCrashingPods: aws-ebs-csi-driver-operator restartCount > 0 (1)
2. TestCreateClusterRequestServingIsolation/ValidateHostedCluster: expected 3 nodes, got 2 (45m timeout)
3. TestNodePool/HostedCluster0/Main/TestKMSRootVolumeEncryption: ReachedIgnitionEndpoint=False after 20m
4. TestNodePool/HostedCluster0/Main/TestNodePoolDay2Tags: expected 1 nodes, got 0 (45m timeout)
+ 6 cascaded parent failures from the above

Summary

All 10 test failures are pre-existing flaky infrastructure issues unrelated to the PR changes. PR #8751 only removes CPO-side restart logic for CNO operands (multus-admission-controller, network-node-identity, ovnkube-control-plane) from three files in the control-plane-operator. None of the failures involve these components. The 4 root leaf failures are: (1) an aws-ebs-csi-driver-operator container restart on the KarpenterUpgradeControlPlane cluster, (2) an AWS node provisioning timeout where only 2 of 3 nodes joined for RequestServingIsolation, (3) a NodePool ignition endpoint never being reached despite the node being ready for KMS root volume test, and (4) a node never joining for Day2Tags. Meanwhile, 16+ other NodePool tests on the same HostedCluster0 provisioned nodes successfully, and 556 out of 566 tests passed, confirming these are sporadic failures.

Root Cause

The 10 reported failures decompose into 4 independent, unrelated leaf failures plus 6 cascaded parent test failures:

Failure 1: TestKarpenterUpgradeControlPlane/EnsureNoCrashingPods

• The aws-ebs-csi-driver-operator container in pod aws-ebs-csi-driver-operator-869dcbc7f4-l8hqk had restartCount=1. This is a sporadic container restart completely unrelated to the PR — the PR removes restart annotation logic for CNO operands (multus, network-node-identity, ovnkube-control-plane), not for the EBS CSI driver operator. Leader election failures were detected across all 16+ hosted clusters in this run but only caused EnsureNoCrashingPods to fail for this one cluster due to the additional container restart.

Failure 2: TestCreateClusterRequestServingIsolation/ValidateHostedCluster

• Only 2 of 3 expected nodes joined after 45 minutes. This is an AWS infrastructure timeout — one worker node failed to provision in the request-serving-isolation-l6jw5 hosted cluster. The test also experienced DNS resolution failures and TCP i/o timeouts connecting to the guest API server initially, suggesting transient networking issues.

Failure 3: TestNodePool/HostedCluster0/Main/TestKMSRootVolumeEncryption

• The node actually became ready (Successfully waited for 1 nodes to become ready... in 9m30s), but the NodePool's ReachedIgnitionEndpoint condition never transitioned to True within 20 minutes. A client rate limiter error (Wait(n=1) would exceed context deadline) also appeared, indicating API pressure. This is a sporadic ignition endpoint delivery issue.

Failure 4: TestNodePool/HostedCluster0/Main/TestNodePoolDay2Tags

• The provisioned machine never reached Ready state (0 of 1 expected nodes after 45 minutes). CAPI conditions showed the machine was stuck at InstanceProvisionStarted. This is an AWS infrastructure issue where the EC2 instance provisioning timed out. 16+ other NodePool tests on the same HostedCluster0 successfully provisioned nodes.

Cascaded failures (6): TestKarpenterUpgradeControlPlane, TestKarpenterUpgradeControlPlane/ValidateHostedCluster, TestCreateClusterRequestServingIsolation, TestNodePool/HostedCluster0/Main, TestNodePool/HostedCluster0, and TestNodePool all fail because their children failed. The HostedCluster0 teardown condition check (wanted ClusterVersionSucceeding=False, got True) is a post-failure diagnostic artifact — because HostedCluster0 was created with NodePoolReplicas=0, the teardown validation expects "not ready" conditions, but the cluster is actually fully ready since sub-tests created worker nodes.

None of these failures are related to PR #8751, which only removes the cleanupClusterNetworkOperatorResources restart annotation propagation and associated manifest/helper functions.

Recommendations

1. Re-trigger the job — These are all sporadic infrastructure failures unrelated to the code changes. A rerun should pass.

2. Consider merging — The PR's code change (removing CPO-side restart annotation logic for CNO operands) is clean and the failures are well-understood flakes affecting multiple unrelated components (EBS CSI driver, AWS node provisioning, ignition endpoint delivery).

3. For test infrastructure improvements (unrelated to this PR):

   • The EnsureNoCrashingPods test could be improved to tolerate single container restarts for non-critical pods, as the EBS CSI driver operator restart appears intermittent.
   • Leader election failures appear across ALL 16+ hosted clusters — this may warrant investigation as a systemic CI environment issue (74 leader election failure log entries).

Evidence

Evidence	Detail
PR scope	Removes restart annotation logic for 3 CNO operands (multus-admission-controller, network-node-identity, ovnkube-control-plane) in 3 files
Failed component	aws-ebs-csi-driver-operator — not touched by the PR
Test pass rate	556/566 tests passed (98.2%)
NodePool tests on same cluster	16+ other NodePool tests on HostedCluster0 provisioned nodes successfully
Leader election failures	74 leader election failure log entries across ALL hosted clusters — systemic CI issue
KMS test	Node became ready (9m30s) but ignition endpoint condition never satisfied (20min timeout)
Day2Tags test	Machine stuck at InstanceProvisionStarted, 0/1 nodes ready after 45min — AWS infra timeout
RequestServingIsolation	2/3 nodes joined, 1 node failed to provision after 45min — AWS infra timeout
Condition mismatch	ClusterVersionSucceeding=True (wanted False) is a post-failure diagnostic, not the actual failure
PR files changed	hostedcontrolplane_controller.go, manifests/cno.go, v2/cno/component.go

---

[bryan-cox]

/verified by e2e

[bryan-cox]

/test e2e-aws

[openshift-ci-robot]

@bryan-cox: This PR has been marked as verified by e2e.

Details

In response to this:

/verified by e2e

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[bryan-cox]

/label acknowledge-critical-fixes-only

[openshift-ci]

@bryan-cox: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci-robot]

@bryan-cox: Jira Issue OCPBUGS-88531 is in an unrecognized state (Verified) and will not be moved to the MODIFIED state.

Details

In response to this:

Summary

• Remove redundant CPO-side restart-date annotation propagation for multus-admission-controller, network-node-identity, and ovnkube-control-plane
• CNO handles restart-date propagation to all its operands directly, making the CPO-side logic redundant

The cleanupClusterNetworkOperatorResources function previously contained logic to read hypershift.openshift.io/restart-date from the HCP and patch it onto CNO-managed deployments. CNO reads the annotation from HCP itself and sets it as a pod template annotation on all rendered workloads, so the CPO-side logic is unnecessary duplication.

Test plan

Tested and verified. Test verification report here - https://bryan-cox.github.io/architectural-artifact-sharing/test-verification-report-ocpbugs-84239/index.html.

Summary by CodeRabbit

• Refactor
• Simplified cluster network operator resource cleanup by removing restart-annotation handling for CNO-managed components.
• Cleanup now focuses directly on removing the ovnkube-sbdb Route when applicable, and deleting the ovnkube-master-external and ovnkube-master-internal Services.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.