OCPBUGS-94178: Remove --skip-crd-migration-phases flags from CAPI deployment by csrwng · Pull Request #8881 · openshift/hypershift

csrwng · 2026-06-30T16:40:12Z

Summary

Removes --skip-crd-migration-phases=StorageVersionMigration and --skip-crd-migration-phases=CleanupManagedFields flags from the core CAPI deployment template
These flags were introduced in PR OSASINFRA-4368, OCPBUGS-84114: Update CAPO to latest stable release #8687 but are not supported by older CAPI images shipped with earlier OCP releases, causing the CAPI controller to fail to start
The OpenStack-specific CAPO deployment retains the flags since CAPO v0.14 supports them
The flags should be re-added later conditionally based on CAPI version support

Test plan

make test passes
make verify passes
Verify CAPI controller starts successfully on older OCP clusters

🤖 Generated with Claude Code

Summary by CodeRabbit

Bug Fixes
- Updated the cluster API controller startup settings so it no longer skips CRD migration-related phases. This helps ensure migration and cleanup steps run as expected.

These flags were introduced in PR openshift#8687 but are not supported by older CAPI images shipped with earlier OCP releases, causing the CAPI controller to fail to start. Remove them unconditionally for now; they should be re-added later gated on CAPI version support. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

openshift-merge-bot · 2026-06-30T16:40:15Z

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

openshift-ci-robot · 2026-06-30T16:40:19Z

@csrwng: This pull request references Jira Issue OCPBUGS-94178, which is invalid:

expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

Removes --skip-crd-migration-phases=StorageVersionMigration and --skip-crd-migration-phases=CleanupManagedFields flags from the core CAPI deployment template

These flags were introduced in PR OSASINFRA-4368, OCPBUGS-84114: Update CAPO to latest stable release #8687 but are not supported by older CAPI images shipped with earlier OCP releases, causing the CAPI controller to fail to start

The OpenStack-specific CAPO deployment retains the flags since CAPO v0.14 supports them

The flags should be re-added later conditionally based on CAPI version support

Test plan

make test passes

make verify passes

Verify CAPI controller starts successfully on older OCP clusters

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

coderabbitai · 2026-06-30T16:40:32Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: a7e1b9b8-29a6-40e9-8d6e-c74ef08cda6e

📥 Commits

Reviewing files that changed from the base of the PR and between 4d087b3 and 50f29cf.

⛔ Files ignored due to path filters (1)

hypershift-operator/controllers/hostedcluster/testdata/cluster-api/zz_fixture_TestReconcileComponents.yaml is excluded by !**/testdata/**

📒 Files selected for processing (1)

control-plane-operator/controllers/hostedcontrolplane/v2/assets/cluster-api/deployment.yaml

💤 Files with no reviewable changes (1)

control-plane-operator/controllers/hostedcontrolplane/v2/assets/cluster-api/deployment.yaml

📝 Walkthrough

Walkthrough

Two --skip-crd-migration-phases flags (StorageVersionMigration and CleanupManagedFields) are removed from the manager container's argument list in the cluster-api Deployment manifest, re-enabling those CRD migration phases during controller startup.

Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error)

Check name	Status	Explanation	Resolution
Stable And Deterministic Test Names	❌ Error	test/envtest/generator.go adds It(fmt.Sprintf(..., featureSet)), so the title varies by input and is not deterministic.	Use a static title like 'should install all CRDs for a feature set' and keep featureSet only in the test body or logs.

✅ Passed checks (10 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: removing CRD migration skip flags from the CAPI deployment.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Test Structure And Quality	✅ Passed	PASS: The PR only changes a deployment YAML manifest; no Ginkgo test code, It blocks, or test files were modified, so the test-quality check is not applicable.
Topology-Aware Scheduling Compatibility	✅ Passed	The only manifest change removes CAPI flags; no nodeSelector, affinity, topology spread, or other topology-sensitive scheduling constraints were added.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	No new Ginkgo e2e tests were added; the PR only removes args from deployment YAML and updates a test fixture, with no IPv4-only or external-connectivity assumptions introduced.
No-Weak-Crypto	✅ Passed	Only a deployment YAML arg change was made; no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret comparisons were introduced.
Container-Privileges	✅ Passed	Only startup args were removed; the modified Deployment has no privileged, hostPID/Network/IPC, SYS_ADMIN, or allowPrivilegeEscalation settings.
No-Sensitive-Data-In-Logs	✅ Passed	The only affected file is a deployment YAML removing two args; no log statements or sensitive data exposure found.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

openshift-ci · 2026-06-30T16:41:12Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: csrwng

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [csrwng]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

csrwng · 2026-06-30T16:41:36Z

/jira refresh

openshift-ci-robot · 2026-06-30T16:41:43Z

@csrwng: This pull request references Jira Issue OCPBUGS-94178, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (5.0.0) matches configured target version for branch (5.0.0)
bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

csrwng · 2026-06-30T16:46:17Z

@coderabbitai re: Pre-merge checks failed - this is not introduced by this PR

coderabbitai · 2026-06-30T16:46:42Z

Tip

For best results, initiate chat on the files or code changes.

@csrwng Thanks for the clarification — understood. If the pre-merge failures are unrelated to the changes in this PR, I won’t treat them as actionable feedback for this review.

codecov · 2026-06-30T16:48:40Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 43.19%. Comparing base (4d087b3) to head (50f29cf).
⚠️ Report is 12 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #8881   +/-   ##
=======================================
  Coverage   43.19%   43.19%           
=======================================
  Files         767      767           
  Lines       94910    94910           
=======================================
  Hits        40997    40997           
  Misses      51050    51050           
  Partials     2863     2863

Flag	Coverage Δ
cmd-support	`36.43% <ø> (ø)`
cpo-hostedcontrolplane	`45.31% <ø> (ø)`
cpo-other	`45.10% <ø> (ø)`
hypershift-operator	`53.53% <ø> (ø)`
other	`31.69% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

stephenfin · 2026-06-30T16:57:11Z

/lgtm

Sorry about that 🙈 I wonder why I didn't see this in a failure on another platform? Missing test coverage?

openshift-merge-bot · 2026-06-30T16:58:00Z

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks-4-22
/test e2e-aws-4-22
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-v2-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke

cwbotbot · 2026-06-30T19:15:03Z

Test Results

e2e-aws

Status: ✅ PASS
Started: 2026-06-30T23:33:53Z
View Job
View Job History

e2e-aks

Status: ✅ PASS
Started: 2026-06-30T16:59:17Z
View Job
View Job History

csrwng · 2026-06-30T23:33:39Z

/retest-required

hypershift-jira-solve-ci · 2026-07-01T01:28:43Z

No interval files for this job. Now I have all the evidence I need. Let me compile the final report:

Test Failure Analysis Complete

Job Information

Prow Job: pull-ci-openshift-hypershift-main-e2e-aks-4-22
Build ID: 2072101306343886848
Target: e2e-aks-4-22
Failed Step: hypershift-azure-run-e2e
Result: 352 tests, 33 skipped, 4 failures (all in TestNodePool/HostedCluster0)

Test Failure Analysis

Error

Container openshift-oauth-apiserver in pod openshift-oauth-apiserver-75fc568d67-fztbk has a restartCount > 0 (1)

Summary

The TestNodePool/HostedCluster0/ValidateHostedCluster/EnsureNoCrashingPods test failed because the openshift-oauth-apiserver container in pod openshift-oauth-apiserver-75fc568d67-fztbk had 1 restart. The container crashed 21 seconds after starting (00:12:20 → 00:12:41) due to transient etcd connectivity issues during hosted cluster control plane startup — the oauth-apiserver could not connect to etcd-client:2379 (gRPC connection errors: dial tcp 10.224.3.92:2379: i/o timeout, operation was canceled), causing it to fail with "error building REST storage: context deadline exceeded". The container successfully restarted at 00:12:43 and was healthy thereafter. This is a startup timing flake, not related to the PR's changes (removing --skip-crd-migration-phases flags from CAPI deployment). All CAPI-related tests passed, and the identical EnsureNoCrashingPods check passed on HostedCluster2 in the same job run.

Root Cause

The root cause is a transient etcd connectivity timeout during hosted cluster control plane startup, which is a known class of infrastructure flake unrelated to the PR under test.

Detailed chain of events:

The etcd pod (etcd-0) started at 00:08:48 and its containers were running by 00:09:29–00:09:37.
The openshift-oauth-apiserver pod started at 00:12:20 — approximately 3 minutes after etcd was running.
Despite etcd being healthy, the oauth-apiserver was unable to establish gRPC connections to etcd-client:2379 within its startup deadline. The termination message shows multiple connection failures: dial tcp 10.224.3.92:2379: i/o timeout and dial tcp: lookup etcd-client: operation was canceled.
After 21 seconds, the oauth-apiserver exited with code 1 and the error "error building REST storage: context deadline exceeded".
The kubelet restarted the container at 00:12:43, and it came up healthy with ready: True.

The EnsureNoCrashingPods test asserts that no pod in the hosted control plane namespace has restartCount > 0 (for platforms without explicit crash tolerations). Since openshift-oauth-apiserver is not in the podCrashTolerations map, even a single transient restart during startup causes the test to fail.

Why this is unrelated to PR #8881:

The PR removes --skip-crd-migration-phases flags from CAPI deployment — this affects Cluster API CRD migration behavior.
The failing component (openshift-oauth-apiserver) is in the hosted cluster control plane, not the CAPI provider.
All CAPI-specific tests (EnsureCAPIFinalizers, CAPI propagation checks) passed.
The same EnsureNoCrashingPods check passed on HostedCluster2 in the same run, confirming it's a non-deterministic flake.
348 of 352 tests passed; the 4 failures are all parent/child propagation from the single EnsureNoCrashingPods assertion failure.

Recommendations

Retry the job — This is a transient infrastructure flake. The oauth-apiserver recovered immediately after restart. A re-run is expected to pass.
Consider adding a crash toleration for openshift-oauth-apiserver — Similar to what's done for ingress-operator, dns-operator, and other components that can transiently restart during startup. A toleration of 1 in the podCrashTolerations map in test/e2e/util/util.go would prevent this known startup timing issue from failing the test, with a comment referencing the etcd connectivity race during HCP bootstrap.
No code changes needed for PR OCPBUGS-94178: Remove --skip-crd-migration-phases flags from CAPI deployment #8881 — The PR's changes (removing --skip-crd-migration-phases) are functionally correct and all related tests pass.

Evidence

Evidence	Detail
Failed test	`TestNodePool/HostedCluster0/ValidateHostedCluster/EnsureNoCrashingPods`
Failed pod	`openshift-oauth-apiserver-75fc568d67-fztbk`
Container	`openshift-oauth-apiserver` (restartCount: 1, exitCode: 1)
Crash reason	`error building REST storage: context deadline exceeded`
Crash detail	gRPC connections to `etcd-client:2379` failed with `i/o timeout` and `operation was canceled`
Container started	`2026-07-01T00:12:20Z`
Container crashed	`2026-07-01T00:12:41Z` (21s after start)
Container restarted	`2026-07-01T00:12:43Z` (healthy thereafter)
etcd pod start time	`2026-07-01T00:08:48Z` (running since `00:09:29`)
HostedCluster2 same check	PASSED (confirming flake, not systematic)
CAPI tests	All PASSED (`EnsureCAPIFinalizers`, CAPI propagation)
Overall results	352 tests, 33 skipped, 4 failures (all from single EnsureNoCrashingPods)
Crash toleration	`openshift-oauth-apiserver` not in `podCrashTolerations` map (default: 0)
PR relevance	None — PR changes CAPI `--skip-crd-migration-phases` flags, unrelated to oauth-apiserver/etcd

csrwng · 2026-07-01T02:01:29Z

The e2e-aks-4-22 test failed because openshift-oauth-apiserver had 1 restart
/override ci/prow/e2e-aks-4-22

openshift-ci · 2026-07-01T02:01:35Z

@csrwng: Overrode contexts on behalf of csrwng: ci/prow/e2e-aks-4-22

Details

In response to this:

The e2e-aks-4-22 test failed because openshift-oauth-apiserver had 1 restart
/override ci/prow/e2e-aks-4-22

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

csrwng · 2026-07-01T02:57:55Z

/verified later @jkaurredhat

openshift-ci-robot · 2026-07-01T02:58:08Z

@csrwng: This PR has been marked to be verified later by @jkaurredhat.

Details

In response to this:

/verified later @jkaurredhat

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2026-07-01T03:21:32Z

@csrwng: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci-robot · 2026-07-01T03:24:20Z

@csrwng: Jira Issue OCPBUGS-94178: All pull requests linked via external trackers have merged:

openshift/hypershift#8881

This pull request has the verified-later tag and will need to be manually moved to VERIFIED after testing. Jira Issue OCPBUGS-94178 has been moved to the MODIFIED state.

Details

In response to this:

Summary

Removes --skip-crd-migration-phases=StorageVersionMigration and --skip-crd-migration-phases=CleanupManagedFields flags from the core CAPI deployment template

These flags were introduced in PR OSASINFRA-4368, OCPBUGS-84114: Update CAPO to latest stable release #8687 but are not supported by older CAPI images shipped with earlier OCP releases, causing the CAPI controller to fail to start

The OpenStack-specific CAPO deployment retains the flags since CAPO v0.14 supports them

The flags should be re-added later conditionally based on CAPI version support

Test plan

make test passes

make verify passes

Verify CAPI controller starts successfully on older OCP clusters

🤖 Generated with Claude Code

Summary by CodeRabbit

Bug Fixes

Updated the cluster API controller startup settings so it no longer skips CRD migration-related phases. This helps ensure migration and cleanup steps run as expected.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-merge-robot · 2026-07-01T19:04:37Z

Fix included in release 5.0.0-0.nightly-2026-07-01-125918

openshift-ci Bot added the do-not-merge/needs-area label Jun 30, 2026

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 30, 2026

openshift-ci Bot requested review from enxebre and muraee June 30, 2026 16:41

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jun 30, 2026

openshift-ci Bot assigned stephenfin Jun 30, 2026

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jun 30, 2026

openshift-ci-robot added verified-later verified Signifies that the PR passed pre-merge verification criteria labels Jul 1, 2026

openshift-merge-bot Bot merged commit d6ec188 into openshift:main Jul 1, 2026
43 checks passed

hypershift-jira-solve-ci Bot mentioned this pull request Jul 1, 2026

STOR-2954: inject centralized TLS configuration for storage operators #8887

Merged

4 tasks

This was referenced Jul 2, 2026

OCPBUGS-86661: Konnectivity retry proxy connection on timeout #8579

Open

NO-JIRA: fix(ci): prevent Claude from breaking push credentials in fork PRs #8831

Closed

Uh oh!

Conversation

csrwng commented Jun 30, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Summary by CodeRabbit

Uh oh!

openshift-merge-bot Bot commented Jun 30, 2026

Uh oh!

openshift-ci-robot commented Jun 30, 2026

Summary

Test plan

Uh oh!

coderabbitai Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Pre-merge checks failed

❌ Failed checks (1 error)

Uh oh!

openshift-ci Bot commented Jun 30, 2026

Uh oh!

csrwng commented Jun 30, 2026

Uh oh!

openshift-ci-robot commented Jun 30, 2026

Uh oh!

csrwng commented Jun 30, 2026

Uh oh!

coderabbitai Bot commented Jun 30, 2026

Uh oh!

codecov Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

stephenfin commented Jun 30, 2026

Uh oh!

openshift-merge-bot Bot commented Jun 30, 2026

Uh oh!

cwbotbot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

e2e-aws

e2e-aks

Uh oh!

csrwng commented Jun 30, 2026

Uh oh!

hypershift-jira-solve-ci Bot commented Jul 1, 2026

Test Failure Analysis Complete

Job Information

Test Failure Analysis

Error

Summary

Uh oh!

csrwng commented Jul 1, 2026

Uh oh!

openshift-ci Bot commented Jul 1, 2026

Uh oh!

csrwng commented Jul 1, 2026

Uh oh!

openshift-ci-robot commented Jul 1, 2026

Uh oh!

openshift-ci Bot commented Jul 1, 2026

Uh oh!

Uh oh!

openshift-ci-robot commented Jul 1, 2026

Summary

Test plan

Summary by CodeRabbit

Uh oh!

openshift-merge-robot commented Jul 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

csrwng commented Jun 30, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jun 30, 2026 •

edited

Loading

codecov Bot commented Jun 30, 2026 •

edited

Loading

cwbotbot commented Jun 30, 2026 •

edited

Loading