Skip to content

CNTRLPLANE-502: Add CRD breaking changes validation to HyperShift CI - #8535

Merged
openshift-merge-bot[bot] merged 2 commits into
openshift:mainfrom
hypershift-community:fix-CNTRLPLANE-502
Jun 23, 2026
Merged

CNTRLPLANE-502: Add CRD breaking changes validation to HyperShift CI#8535
openshift-merge-bot[bot] merged 2 commits into
openshift:mainfrom
hypershift-community:fix-CNTRLPLANE-502

Conversation

@hypershift-jira-solve-ci

@hypershift-jira-solve-ci hypershift-jira-solve-ci Bot commented May 18, 2026

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

Adds automated CRD schema breaking-change detection to HyperShift CI.

A new standalone Go tool (hack/tools/crd-schema-check) uses the kubernetes-sigs/crdify library to compare CRD YAML files between a base git commit and the current working tree, catching unintentional breaking changes before merge.

What the checker validates (via crdify):

  • No incompatible schema changes (field removals, type changes, enum changes)
  • No new required fields that break existing CRs
  • CRD structural integrity (status subresource, SSA merge tags)
  • CEL cost budget compliance

Featuregate variant handling:

  • Only the CustomNoUpgrade superset CRD variant is checked
  • -Default and -TechPreviewNoUpgrade variants are skipped to avoid false positives when fields move between featuregate levels

A new verify-crd-schema Makefile target is wired into verify-parallel, so it runs automatically as part of make verify and CI presubmit checks. The target uses PULL_BASE_SHA (set by Prow in CI, defaults to upstream/main locally) as the comparison baseline, following the same pattern used by api-lint. Both the HyperShift operator CRD directory and the Karpenter CRD directory are checked.

Which issue(s) this PR fixes:

Fixes https://redhat.atlassian.net/browse/CNTRLPLANE-502

Special notes for your reviewer:

  • The tool lives under hack/tools/crd-schema-check/ as a standalone binary, consistent with the project's pattern for CI tooling.
  • Uses kubernetes-sigs/crdify — the upstream successor to openshift/crd-schema-checker — for CRD validation.
  • Uses crdify's git loader for reading CRD files from base commits, and crdify's default configuration for validation rules.
  • Comprehensive unit tests cover: non-breaking changes, breaking change detection (field removal, enum changes, new required fields, type changes), new CRD handling, featuregate variant skipping, and version/schema filtering.

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label May 18, 2026
@openshift-ci-robot

openshift-ci-robot commented May 18, 2026

Copy link
Copy Markdown

@hypershift-jira-solve-ci[bot]: This pull request references CNTRLPLANE-502 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

What this PR does / why we need it:

Adds automated CRD schema breaking-change detection to HyperShift CI.

A new standalone Go tool (hack/tools/crd-schema-check) uses the openshift/crd-schema-checker library to compare CRD YAML files between a base git commit and the current working tree, catching unintentional breaking changes before merge.

What the checker validates:

  • No field removals from existing CRDs
  • No enum value removals
  • No new required fields that break existing CRs
  • No data type changes on existing fields
  • CRDs must have status subresource
  • Lists must have SSA merge tags
  • CEL cost budget compliance

Comparators already enforced by the KAL linter (NoBools, NoFloats, NoUints, NoMaps, ConditionsMustHaveProperSSATags) are disabled to avoid duplicate enforcement.

A new verify-crd-schema Makefile target is wired into verify-parallel, so it runs automatically as part of make verify and CI presubmit checks. The target uses PULL_BASE_SHA (set by Prow in CI, defaults to upstream/main locally) as the comparison baseline, following the same pattern used by api-lint.

Which issue(s) this PR fixes:

Fixes https://redhat.atlassian.net/browse/CNTRLPLANE-502

Special notes for your reviewer:

  • The tool lives under hack/tools/crd-schema-check/ as a standalone binary, consistent with the project's pattern for CI tooling.
  • Comprehensive unit tests cover: non-breaking changes, breaking change detection (field removal, enum removal, new required fields, type changes), new CRD handling, and comparator configuration.

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Always review AI generated responses prior to use.
Generated with Claude Code via /jira:solve [CNTRLPLANE-502](https://redhat.atlassian.net/browse/CNTRLPLANE-502)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented May 18, 2026

Copy link
Copy Markdown
Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

This pull request adds a new crd-schema-check CLI (hack/tools/crd-schema-check) that scans specified directories for CRD YAML, filters versions with OpenAPI v3 schemas, loads the corresponding CRDs from a base git commit, and runs schema comparisons via the comparator registry. The tool prints warnings, aggregates errors, and exits non‑zero on breaking changes. The Makefile is updated to build the tool and add a verify-crd-schema target included in verify-parallel. Tests and YAML helper builders were added, and hack/tools/go.mod promotes required dependencies.

Sequence Diagram(s)

sequenceDiagram
  participant User as CLI (crd-schema-check)
  participant Repo as git.Repository
  participant FS as Filesystem
  participant Comparator as comparator.Registry
  User->>Repo: findRepoRoot / resolve base commit
  User->>FS: scan --crd-dir for .yaml files
  FS->>User: CRD YAML bytes
  User->>User: isCRDYAML / filterVersionsWithSchema
  User->>Repo: loadCRDFromCommit(path, baseCommit)
  User->>Comparator: CompareCRDs(oldCRD, newCRD, config)
  Comparator->>User: ComparisonResults, Errors/Warnings
  User->>User: aggregate and exit with status
Loading

Possibly related PRs

  • openshift/hypershift#8547: Overlaps on promoting/bumping the github.com/go-git/go-git/v5 dependency used by the new tooling.

Suggested reviewers

  • muraee
  • clebs
  • cblecker
🚥 Pre-merge checks | ✅ 11 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 36.36% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (11 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately describes the main change: adding CRD breaking changes validation to HyperShift CI. It directly reflects the primary purpose of the changeset.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The PR adds only standard Go tests using package "testing", not Ginkgo tests, so the Ginkgo test name stability check is not applicable.
Test Structure And Quality ✅ Passed This PR contains standard Go unit tests, not Ginkgo tests. The custom check is Ginkgo-specific and does not apply to standard Go testing patterns.
Microshift Test Compatibility ✅ Passed This PR adds a utility tool (crd-schema-check) and standard Go unit tests, not Ginkgo e2e tests. The custom check only applies to new Ginkgo e2e tests, which are not present in this PR.
Single Node Openshift (Sno) Test Compatibility ✅ Passed PR adds no Ginkgo e2e tests—only standard Go unit tests using testing.T for the crd-schema-check CLI tool. Check is not applicable.
Topology-Aware Scheduling Compatibility ✅ Passed PR adds only CI tooling (crd-schema-check) for CRD schema validation. No deployment manifests, operators, controllers, or pod scheduling constraints introduced.
Ote Binary Stdout Contract ✅ Passed The crd-schema-check tool is a CI verification utility, not an OTE test binary, so the check is not applicable. Additionally, all output properly goes to stderr, not stdout.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed This PR adds no Ginkgo e2e tests; it only adds unit tests for a Go CLI tool using standard testing package. Custom check only applies to Ginkgo e2e tests.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci openshift-ci Bot added the area/ci-tooling Indicates the PR includes changes for CI or tooling label May 18, 2026
@openshift-ci
openshift-ci Bot requested review from jparrill and muraee May 18, 2026 09:27

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
hack/tools/crd-schema-check/main_test.go (1)

67-68: ⚡ Quick win

Strengthen filterVersionsWithSchema assertions to validate retained version names, not only count.

Current checks can pass even if the wrong version is retained. Add expected version names per case and assert exact match (order or set).

Suggested test hardening
 	tests := []struct {
 		name             string
 		yaml             string
 		expectedVersions int
+		expectedNames    []string
 	}{
 		{
 			name: "When all versions have schemas it should keep all versions",
 			yaml: baseCRDWithVersions([]versionSpec{
 				{name: "v1", hasSchema: true},
 				{name: "v1beta1", hasSchema: true},
 			}),
 			expectedVersions: 2,
+			expectedNames:    []string{"v1", "v1beta1"},
 		},
 		{
 			name: "When some versions lack schemas it should filter them out",
 			yaml: baseCRDWithVersions([]versionSpec{
 				{name: "v1", hasSchema: true},
 				{name: "v1beta1", hasSchema: false},
 			}),
 			expectedVersions: 1,
+			expectedNames:    []string{"v1"},
 		},
 		{
 			name: "When no versions have schemas it should return zero versions",
 			yaml: baseCRDWithVersions([]versionSpec{
 				{name: "v1", hasSchema: false},
 			}),
 			expectedVersions: 0,
+			expectedNames:    nil,
 		},
 	}
@@
 			result := filterVersionsWithSchema(crd)
 			if len(result.Spec.Versions) != tt.expectedVersions {
 				t.Errorf("filterVersionsWithSchema() returned %d versions, want %d", len(result.Spec.Versions), tt.expectedVersions)
 			}
+			got := make([]string, 0, len(result.Spec.Versions))
+			for _, v := range result.Spec.Versions {
+				got = append(got, v.Name)
+			}
+			if len(got) != len(tt.expectedNames) {
+				t.Fatalf("filterVersionsWithSchema() names count = %d, want %d", len(got), len(tt.expectedNames))
+			}
+			for i := range got {
+				if got[i] != tt.expectedNames[i] {
+					t.Fatalf("filterVersionsWithSchema() names = %v, want %v", got, tt.expectedNames)
+				}
+			}

Also applies to: 100-103

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/tools/crd-schema-check/main_test.go` around lines 67 - 68, The tests for
filterVersionsWithSchema only assert the count; change the test cases in
main_test.go to include expected version names (e.g., replace expectedVersions
int with expectedVersions []string) and update the assertion after calling
filterVersionsWithSchema to compare the actual retained version names against
the expected slice (either by exact order or by set equality). Locate the test
table entries and the assertion block that currently checks len(filtered) and
replace it with logic that extracts the version names from filtered and asserts
equality with the expectedVersions for each test case (also update the
additional cases around the other occurrence referenced in the comment).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@hack/tools/crd-schema-check/main.go`:
- Around line 167-170: The call-site currently treats any error from
loadCRDFromCommit as "new CRD" and continues with oldCRD == nil; instead, change
loadCRDFromCommit so it returns (nil, nil) when the file truly does not exist by
checking errors.Is(err, object.ErrFileNotFound) inside loadCRDFromCommit, and
for any other error propagate it upward; then update the caller (where oldCRD,
err := loadCRDFromCommit(baseCommit, relPath) is invoked) to return or fail on
non-nil err rather than logging "new", and only log "new CRD" when
loadCRDFromCommit returned (nil, nil) meaning file-not-found.
- Line 104: The printed success message in main.main (the fmt.Printf call that
uses totalChecked and allWarnings) writes non-JSON output to stdout; change it
to write to stderr instead so stdout remains JSON-only—replace the
fmt.Printf(...) with fmt.Fprintf(os.Stderr, ...) or fmt.Fprintln(os.Stderr, ...)
(importing os if needed) or use the logger that writes to stderr; ensure no
other non-JSON prints remain in main().

---

Nitpick comments:
In `@hack/tools/crd-schema-check/main_test.go`:
- Around line 67-68: The tests for filterVersionsWithSchema only assert the
count; change the test cases in main_test.go to include expected version names
(e.g., replace expectedVersions int with expectedVersions []string) and update
the assertion after calling filterVersionsWithSchema to compare the actual
retained version names against the expected slice (either by exact order or by
set equality). Locate the test table entries and the assertion block that
currently checks len(filtered) and replace it with logic that extracts the
version names from filtered and asserts equality with the expectedVersions for
each test case (also update the additional cases around the other occurrence
referenced in the comment).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: f14c7b00-9679-4518-af7d-51f8e7eaa728

📥 Commits

Reviewing files that changed from the base of the PR and between f76be88 and 1f8bfd0.

📒 Files selected for processing (3)
  • Makefile
  • hack/tools/crd-schema-check/main.go
  • hack/tools/crd-schema-check/main_test.go

Comment thread hack/tools/crd-schema-check/main.go Outdated
Comment thread hack/tools/crd-schema-check/main.go Outdated
@codecov

codecov Bot commented May 18, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 41.86%. Comparing base (d86f3d4) to head (9155e6b).
⚠️ Report is 377 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8535      +/-   ##
==========================================
+ Coverage   40.34%   41.86%   +1.51%     
==========================================
  Files         755      759       +4     
  Lines       93167    94101     +934     
==========================================
+ Hits        37587    39392    +1805     
+ Misses      52877    51949     -928     
- Partials     2703     2760      +57     

see 95 files with indirect coverage changes

Flag Coverage Δ
cmd-support 35.13% <ø> (+0.82%) ⬆️
cpo-hostedcontrolplane 44.15% <ø> (+2.38%) ⬆️
cpo-other 43.45% <ø> (+3.30%) ⬆️
hypershift-operator 52.05% <ø> (+1.33%) ⬆️
other 31.56% <ø> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@hypershift-jira-solve-ci
hypershift-jira-solve-ci Bot force-pushed the fix-CNTRLPLANE-502 branch 2 times, most recently from 1898e5a to 70586fe Compare May 20, 2026 17:03

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
hack/tools/crd-schema-check/main_test.go (1)

19-297: ⚡ Quick win

Consider using gomega assertions and enabling parallel execution for all test functions.

The test functions use standard Go testing assertions (t.Errorf, t.Fatalf) and do not call t.Parallel(). These tests are independent with no shared state and would benefit from:

  1. gomega matchers for more expressive assertions and clearer failure messages
  2. t.Parallel() calls to improve test execution performance in CI

Example refactor for one test function:

Suggested pattern with gomega and parallel execution
+import (
+	. "github.com/onsi/gomega"
+)

 func TestIsCRDYAML(t *testing.T) {
+	RegisterTestingT(t)
+	
 	tests := []struct {
 		name     string
 		yaml     string
 		expected bool
 	}{
 		// ... test cases ...
 	}

 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+			g := NewWithT(t)
 			result := isCRDYAML([]byte(tt.yaml))
-			if result != tt.expected {
-				t.Errorf("isCRDYAML() = %v, want %v", result, tt.expected)
-			}
+			g.Expect(result).To(Equal(tt.expected))
 		})
 	}
 }

Apply similar changes to all test functions in this file. As per coding guidelines, prefer gomega for unit test assertions and use race detection and parallel execution for unit tests.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/tools/crd-schema-check/main_test.go` around lines 19 - 297, All tests in
this file should use Gomega matchers and run in parallel: add t.Parallel() at
the start of each top-level test (TestIsCRDYAML, TestFilterVersionsWithSchema,
TestCompareCRDs_NonBreakingChanges, TestCompareCRDs_BreakingChanges,
TestCompareCRDs_NewCRDWithNilOld, TestBuildComparatorConfig) and inside each
t.Run subtest where appropriate, replace t.Fatalf/t.Errorf assertions with
Gomega expectations (use NewGomegaWithT(t) or GomegaWithT(t) to create a matcher
bound to the test and assert equality, empty error lists, presence/absence of
comparator names, lengths, etc.), and add the gomega import; keep existing test
logic and helper functions (isCRDYAML, filterVersionsWithSchema, CompareCRDs,
buildComparatorConfig, resourceread usage) but convert all assertion checks to
gomega.Expect(...) calls for clearer failures and concurrency safety.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@hack/tools/crd-schema-check/main.go`:
- Around line 171-176: The code logs that a CRD is new and should skip
comparison but then still calls config.ComparatorRegistry.Compare(oldCRD,
newCRD, ...) and increments results.checked; fix by adding an immediate continue
after the fmt.Fprintf(...) when oldCRD == nil so Compare is not invoked and
results.checked is not incremented for new CRDs, and if you want to track those
files add a separate counter (e.g., results.new) instead of counting them in
results.checked; reference oldCRD, newCRD, relPath,
config.ComparatorRegistry.Compare, compResults/compErrors and results.checked
when applying the change.

---

Nitpick comments:
In `@hack/tools/crd-schema-check/main_test.go`:
- Around line 19-297: All tests in this file should use Gomega matchers and run
in parallel: add t.Parallel() at the start of each top-level test
(TestIsCRDYAML, TestFilterVersionsWithSchema,
TestCompareCRDs_NonBreakingChanges, TestCompareCRDs_BreakingChanges,
TestCompareCRDs_NewCRDWithNilOld, TestBuildComparatorConfig) and inside each
t.Run subtest where appropriate, replace t.Fatalf/t.Errorf assertions with
Gomega expectations (use NewGomegaWithT(t) or GomegaWithT(t) to create a matcher
bound to the test and assert equality, empty error lists, presence/absence of
comparator names, lengths, etc.), and add the gomega import; keep existing test
logic and helper functions (isCRDYAML, filterVersionsWithSchema, CompareCRDs,
buildComparatorConfig, resourceread usage) but convert all assertion checks to
gomega.Expect(...) calls for clearer failures and concurrency safety.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 4d17fc14-c76c-429c-befe-e740b884a2d9

📥 Commits

Reviewing files that changed from the base of the PR and between 1898e5a and 70586fe.

📒 Files selected for processing (4)
  • Makefile
  • hack/tools/crd-schema-check/main.go
  • hack/tools/crd-schema-check/main_test.go
  • hack/tools/go.mod

Comment thread hack/tools/crd-schema-check/main.go

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
hack/tools/go.mod (1)

18-18: ⚡ Quick win

Align declared and replaced Kubernetes API extension versions.

Line 18 declares k8s.io/apiextensions-apiserver v0.34.3, but Line 335 forces v0.34.2 via replace. Align these to the same version to avoid version-skew confusion in tooling and maintenance.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/tools/go.mod` at line 18, The go.mod currently declares
k8s.io/apiextensions-apiserver at v0.34.3 while a replace directive forces
v0.34.2, causing a version mismatch; update one so both match—either change the
declared module version k8s.io/apiextensions-apiserver to v0.34.2 or
(preferably) update the replace directive to v0.34.3—ensure the declared module
version and the replace entry for k8s.io/apiextensions-apiserver use the
identical semver string.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@hack/tools/crd-schema-check/main.go`:
- Around line 171-173: The current logic treats oldCRD == nil the same whether
the base file is missing or was filtered to no versions with OpenAPIV3Schema by
loadCRDFromCommit; update loadCRDFromCommit to return a distinct error/value
(e.g., ErrNoSchema or a typed sentinel) when the file exists but all versions
lack OpenAPIV3Schema, and then change the caller handling the oldCRD check in
main.go to distinguish the two cases: if the error is ErrNoSchema, print a
specific message like "info: %s present in base but no versions with
OpenAPIV3Schema, skipping schema comparison", while only using "not found in
base" for a true missing-file case; ensure references to oldCRD and
loadCRDFromCommit are updated accordingly so both the early return and the log
message reflect the distinct states.

---

Nitpick comments:
In `@hack/tools/go.mod`:
- Line 18: The go.mod currently declares k8s.io/apiextensions-apiserver at
v0.34.3 while a replace directive forces v0.34.2, causing a version mismatch;
update one so both match—either change the declared module version
k8s.io/apiextensions-apiserver to v0.34.2 or (preferably) update the replace
directive to v0.34.3—ensure the declared module version and the replace entry
for k8s.io/apiextensions-apiserver use the identical semver string.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 42394e2d-6833-4293-997a-b4d7e469f7f0

📥 Commits

Reviewing files that changed from the base of the PR and between 70586fe and 0b4378f.

📒 Files selected for processing (2)
  • hack/tools/crd-schema-check/main.go
  • hack/tools/go.mod

Comment thread hack/tools/crd-schema-check/main.go

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
hack/tools/crd-schema-check/main.go (2)

138-140: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Include .yml files in the CRD scan.

This filter silently skips CRDs committed as .yml, which leaves a gap in the breaking-change check.

Proposed fix
-		if filepath.Ext(entry.Name()) != ".yaml" {
+		ext := strings.ToLower(filepath.Ext(entry.Name()))
+		if ext != ".yaml" && ext != ".yml" {
 			continue
 		}

As per coding guidelines, "**/*.{go,yaml,yml}: Provide API definitions that align with OpenShift and Kubernetes best practices".

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/tools/crd-schema-check/main.go` around lines 138 - 140, The current
file-extension filter in crd-schema-check only accepts ".yaml" and skips ".yml"
files; update the check that uses filepath.Ext(entry.Name()) so it allows both
".yaml" and ".yml" (e.g., compute ext := filepath.Ext(entry.Name()) and continue
only if ext is neither ".yaml" nor ".yml"), ensuring CRDs saved with .yml are
included in the scan.

41-42: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Use upstream/main as the local fallback.

Falling back to main diverges from the documented api-lint behavior and can compare against a stale local branch—or fail on clones that only have upstream/main—so local verification may report the wrong schema diff.

Proposed fix
-	comparisonBase := flag.String("comparison-base", envOrDefault("PULL_BASE_SHA", "main"),
+	comparisonBase := flag.String("comparison-base", envOrDefault("PULL_BASE_SHA", "upstream/main"),
 		"git ref to compare CRD schemas against (branch, tag, or SHA)")
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/tools/crd-schema-check/main.go` around lines 41 - 42, The flag default
for comparisonBase uses envOrDefault(..., "main") which can point to a stale
local branch; change the fallback default string to "upstream/main" in the
flag.String call that defines comparisonBase so the CLI uses upstream/main as
the local fallback (keep the envOrDefault and flag.String usage intact; update
only the default value passed to comparisonBase).
🧹 Nitpick comments (1)
hack/tools/go.mod (1)

18-18: ⚡ Quick win

Align direct require version with the pinned replace version.

k8s.io/apiextensions-apiserver is required at v0.34.3 but force-replaced to v0.34.2 (line 335). Keeping these aligned avoids misleading dependency declarations and tooling confusion.

Suggested diff
-	k8s.io/apiextensions-apiserver v0.34.3
+	k8s.io/apiextensions-apiserver v0.34.2
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/tools/go.mod` at line 18, The go.mod requires
k8s.io/apiextensions-apiserver at v0.34.3 but the file contains a replace pin to
v0.34.2; update the direct require entry for k8s.io/apiextensions-apiserver in
hack/tools/go.mod to v0.34.2 so it matches the replace directive (ensure you
edit the require line that mentions k8s.io/apiextensions-apiserver).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@hack/tools/crd-schema-check/main.go`:
- Around line 138-140: The current file-extension filter in crd-schema-check
only accepts ".yaml" and skips ".yml" files; update the check that uses
filepath.Ext(entry.Name()) so it allows both ".yaml" and ".yml" (e.g., compute
ext := filepath.Ext(entry.Name()) and continue only if ext is neither ".yaml"
nor ".yml"), ensuring CRDs saved with .yml are included in the scan.
- Around line 41-42: The flag default for comparisonBase uses envOrDefault(...,
"main") which can point to a stale local branch; change the fallback default
string to "upstream/main" in the flag.String call that defines comparisonBase so
the CLI uses upstream/main as the local fallback (keep the envOrDefault and
flag.String usage intact; update only the default value passed to
comparisonBase).

---

Nitpick comments:
In `@hack/tools/go.mod`:
- Line 18: The go.mod requires k8s.io/apiextensions-apiserver at v0.34.3 but the
file contains a replace pin to v0.34.2; update the direct require entry for
k8s.io/apiextensions-apiserver in hack/tools/go.mod to v0.34.2 so it matches the
replace directive (ensure you edit the require line that mentions
k8s.io/apiextensions-apiserver).

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 055a5765-7bdf-4da1-98da-f89b538788f7

📥 Commits

Reviewing files that changed from the base of the PR and between 0b4378f and c423f10.

📒 Files selected for processing (2)
  • hack/tools/crd-schema-check/main.go
  • hack/tools/go.mod

@hypershift-jira-solve-ci

Copy link
Copy Markdown
Contributor Author

Re: #8535 (review)

Thanks for the review suggestions — here's my assessment of each:

1. Include .yml files in the CRD scan (lines 138-140)
Declined. All CRD files in the scanned directories (cmd/install/assets/crds/hypershift-operator/) use .yaml exclusively — HyperShift's CRD generation tooling (make api) only produces .yaml files, consistent with OpenShift/Kubernetes convention. Adding .yml support would be dead code with no practical benefit.

2. Use upstream/main as the local fallback (lines 41-42)
Declined. In CI (Prow), PULL_BASE_SHA is always set, so the fallback is never used there. For local development, main is the standard remote-tracking branch name — using upstream/main would break for developers who use origin as their primary remote. The flag is configurable via --comparison-base for other setups.

3. Align require version with replace version (go.mod line 18)
The replace directive (v0.34.2) overrides the require (v0.34.3), so this is functionally correct. The version in require is managed by go mod tidy and reflects the minimum version in the dependency graph — manually aligning it would be overridden on the next go mod tidy run.


AI-assisted response via Claude Code

@hypershift-jira-solve-ci

Copy link
Copy Markdown
Contributor Author

Re: #8535 (review)

Thanks for the review, @coderabbitai. Addressing the three suggestions:

1. Include .yml files in the CRD scan (lines 138-140) ✅ Done.
Updated the extension filter to accept both .yaml and .yml. While all CRDs in this repo are generated as .yaml, this is a good defensive measure.

2. Use upstream/main as the local fallback (lines 41-42) — Intentionally kept as main.
In CI (Prow), PULL_BASE_SHA is always set, so the fallback is only used for local development. Using main is more universal — it works for developers who clone the main repo directly (where origin points to openshift/hypershift) and for fork-based workflows (where main typically tracks upstream). Using upstream/main would fail for anyone without an upstream remote configured.

3. Align go.mod require version with replace (go.mod line 18) ✅ Done.
Aligned k8s.io/apiextensions-apiserver require to v0.34.2 to match the replace directive. Ran go mod vendor to sync.


AI-assisted response via Claude Code

@bryan-cox

Copy link
Copy Markdown
Member

If there is already a library at openshift/crd-schema-checker, why do we need the go files? Is this how that repo says to set it up?

@bryan-cox

Copy link
Copy Markdown
Member

/address-review-comments

@hypershift-jira-solve-ci

Copy link
Copy Markdown
Contributor Author

Re: #8535 (comment)

Good question! openshift/crd-schema-checker is a library (Go packages), not a standalone CLI tool — it doesn't ship a main.go or binary. It provides the comparison engine (pkg/manifestcomparators), CRD parsing (pkg/resourceread), and comparator configuration (pkg/cmd/options), but leaves it to each consumer to wire up the git integration, file discovery, and comparator selection.

The custom Go tool in hack/tools/crd-schema-check/ is needed to provide HyperShift-specific integration on top of the library:

  • Git comparison: Uses go-git to read CRD YAML from a base commit (PULL_BASE_SHA) and compare against the current working tree — the library itself has no git awareness.
  • Directory walking: Recursively finds CRD .yaml/.yml files in the specified --crd-dir paths.
  • Comparator filtering: Disables comparators already enforced by KAL (NoBools, NoFloats, NoUints, NoMaps, ConditionsMustHaveProperSSATags) to avoid duplicate enforcement.
  • CI integration: Reads PULL_BASE_SHA from the Prow environment for the comparison baseline, following the same pattern used by api-lint.

This is how openshift/api uses it too — they have their own custom tool that imports the library and adds repo-specific logic. The pattern is consistent with other tools in hack/tools/ (controller-gen, yq, codegen, staticcheck, etc.) that are built from Go source in the tools module.


AI-assisted response via Claude Code

@hypershift-jira-solve-ci

Copy link
Copy Markdown
Contributor Author

Re: #8535 (comment)

Good question. The openshift/crd-schema-checker is a Go library (packages under pkg/), not a standalone CLI tool — it provides the comparison engine, comparator registry, and CRD parsing utilities, but expects each consuming project to write its own thin wrapper. Looking at the vendored code, there's no main package or pre-built binary in that repo.

The Go files in this PR use the library's packages to add HyperShift-specific integration:

  1. Git-based comparison — uses go-git to read old CRD YAML from the base commit (PULL_BASE_SHA in Prow CI, main locally), so it diffs the before/after CRDs automatically without needing two files on disk
  2. Custom comparator configuration — disables comparators already enforced by the KAL linter (NoBools, NoFloats, NoUints, NoMaps, ConditionsMustHaveProperSSATags) to avoid duplicate enforcement
  3. Multi-directory CRD scanning — walks CRD directories recursively, filters for actual CRD YAMLs, and skips new CRDs that don't exist in the base

This is the same pattern openshift/api follows — they also have their own wrapper tool in their repo that calls into the library's pkg/ packages.


AI-assisted response via Claude Code

@bryan-cox bryan-cox left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Staff Engineer Review

Overall the approach is sound — integrating openshift/crd-schema-checker aligns with openshift/api patterns and this is a valuable addition to CI. The tool is well-structured and follows existing hack/tools/ conventions. See inline comments for specific issues to address before merge.

Good:

  • Correct KAL comparator exclusions matching openshift/api
  • Proper nil CRD handling (upstream library comparators handle it correctly)
  • Clean separation of concerns and testable design
  • CodeRabbit feedback has been incorporated well

Blocking issues (3): featuregated CRD variants, PULL_BASE_SHA double-defaulting, exported unused function
Suggestions (4): gomega assertions, dead code, karpenter CRDs, recursive scan fragility

Comment thread Makefile
Comment thread Makefile Outdated
.PHONY: verify-crd-schema
verify-crd-schema: $(CRD_SCHEMA_CHECK) ## Verify CRD schemas for breaking changes against base branch.
$(CRD_SCHEMA_CHECK) --comparison-base=$(PULL_BASE_SHA) \
--crd-dir=cmd/install/assets/crds/hypershift-operator

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] Karpenter CRDs are not checked.

There is also a karpenter CRD at karpenter-operator/controllers/karpenter/assets/zz_generated.crd-manifests/openshiftec2nodeclasses.crd.yaml. If this is an intentional exclusion (karpenter CRDs may have a different lifecycle), add a comment explaining why. Otherwise, add another --crd-dir.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Added --crd-dir=karpenter-operator/controllers/karpenter/assets/zz_generated.crd-manifests to the Makefile target.


AI-assisted response via Claude Code

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Added --crd-dir=karpenter-operator/controllers/karpenter/assets/zz_generated.crd-manifests to the verify-crd-schema Makefile target.


AI-assisted response via Claude Code

Comment thread hack/tools/crd-schema-check/main.go
Comment thread hack/tools/crd-schema-check/main.go Outdated
return fallback
}

// CompareCRDs is the testable core of the comparison logic.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocking] CompareCRDs is exported but only used in tests.

The production code in checkCRDsInDir calls config.ComparatorRegistry.Compare() directly. This exported wrapper exists only for the test file. Per project conventions, unexported symbols should be preferred unless there is an explicit external consumer.

Fix: Either unexport it (compareCRDs) or remove it entirely and have tests call config.ComparatorRegistry.Compare() directly.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Unexported to compareCRDs — no external consumers, only used in tests.


AI-assisted response via Claude Code

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Unexported to compareCRDs — no external consumers, only used in tests.


AI-assisted response via Claude Code

Comment thread hack/tools/crd-schema-check/main.go Outdated
for _, e := range allErrors {
fmt.Fprintf(os.Stderr, " ERROR: %s\n", e)
}
fmt.Fprintf(os.Stderr, "\nIf this is a pre-existing violation from a moved/renamed file, use Prow /override to bypass.\n")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] The override guidance is vague:

If this is a pre-existing violation from a moved/renamed file, use Prow /override to bypass.

Consider adding the specific job name that would need to be overridden, or referencing a doc/process for intentional breaking changes. In openshift/api, the crd-schema-checker annotations provide a more granular exception mechanism.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Updated to reference crd-schema-checker annotations for intentional changes and added the specific verify-crd-schema job name for /override.


AI-assisted response via Claude Code

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Updated to reference crd-schema-checker annotations for intentional changes and added the specific verify-crd-schema job name for /override.


AI-assisted response via Claude Code

Comment thread hack/tools/crd-schema-check/main_test.go
Comment thread hack/tools/crd-schema-check/main_test.go
Comment thread hack/tools/crd-schema-check/main.go
@bryan-cox

Copy link
Copy Markdown
Member

/address-review-comments

@github-actions

github-actions Bot commented Jun 2, 2026

Copy link
Copy Markdown

🤖 Addressing review comments: workflow run

@hypershift-jira-solve-ci

Copy link
Copy Markdown
Contributor Author

Re: #8535 (review)

Thanks for the thorough review, @bryan-cox. All inline comments have been addressed:

Blocking (3):

  • Featuregate variants: Added skipFeatureGateVariant() to filter out -Default.crd.yaml and -TechPreviewNoUpgrade.crd.yaml, only checking the CustomNoUpgrade superset variant. Added test coverage.
  • PULL_BASE_SHA double-defaulting: Removed envOrDefault function entirely; default is now plain "main". The Makefile always passes the correct value via --comparison-base.
  • CompareCRDs exported: Unexported to compareCRDs — no external consumers.

Nit (1):

  • Override guidance: Updated error message to reference crd-schema-checker annotations for intentional changes and the specific verify-crd-schema job name for /override.

Suggestions (4):

  • Karpenter CRDs: Added --crd-dir=karpenter-operator/controllers/karpenter/assets/zz_generated.crd-manifests to the Makefile target.
  • Dead code: Removed unused mustParseCRD function.
  • Gomega assertions: Converted all test assertions to gomega (g.Expect(...).To(...), .NotTo(HaveOccurred()), .To(BeEmpty()), .To(ContainElement(...)), etc.). Added gomega as a test dependency in hack/tools/go.mod.
  • Recursive scan docs: Added comprehensive doc comment on checkCRDsInDir documenting the content-based filtering assumption and featuregate variant skipping.

All tests pass locally.


AI-assisted response via Claude Code

@hypershift-jira-solve-ci

Copy link
Copy Markdown
Contributor Author

Re: #8535 (review)

@bryan-cox Thank you for the thorough review. All 3 blocking issues and 4 suggestions have been addressed:

Blocking (3):

  • Featuregated CRD variants: Added skipFeatureGateVariant() to only check CustomNoUpgrade superset variant
  • PULL_BASE_SHA double-defaulting: Removed envOrDefault, default is now plain "main"
  • Exported CompareCRDs: Unexported to compareCRDs

Suggestions (4):

  • Gomega assertions: Converted all tests to use gomega
  • Dead code: Removed unused mustParseCRD
  • Karpenter CRDs: Added --crd-dir for karpenter CRD manifests
  • Recursive scan: Added doc comment on checkCRDsInDir explaining content-based filtering

AI-assisted response via Claude Code

@github-actions

Copy link
Copy Markdown

🤖 Addressing review comments: workflow run

@hypershift-jira-solve-ci

hypershift-jira-solve-ci Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor Author

Re: #8535 (comment)

Review Comments Status

All review threads have been addressed. Here's the current state:

✅ Resolved in code (bryan-cox, jparrill reviews)

  • Featuregate CRD variants: skipFeatureGateVariant() skips Default/TechPreview, only checks CustomNoUpgrade superset
  • PULL_BASE_SHA double-defaulting: Tool uses --comparison-base CLI flag only; no env var fallback
  • CompareCRDs usage: Now used in both production code (line 197) and tests
  • Override guidance: Specifies the exact job name (/override verify-crd-schema)
  • Dead code removal: mustParseCRD removed
  • Gomega + NewWithT: All tests use NewWithT(t) (not deprecated NewGomegaWithT)
  • Karpenter CRDs: Included via second --crd-dir in Makefile
  • Test parallelization: All test functions and subtests call t.Parallel()
  • crdify migration: Switched from crd-schema-checker to sigs.k8s.io/crdify
  • Enum additions allowed: additionPolicy: Allow in crdify-config.yaml

✅ Answered today (JoelSpeed)

  • Which manifests are checked? — Replied inline: cmd/install/assets/crds/hypershift-operator (CustomNoUpgrade only) + karpenter-operator/.../zz_generated.crd-manifests
  • Follow-up tracking for integration tests — Replied inline: not in a dedicated Jira yet; parent ticket is CNTRLPLANE-502

⏳ Open question (needs human decision)

  • Enum removal policy (thread): @JoelSpeed noted o/api allows enum value removals. Currently, this PR still detects enum removals as breaking. Should we add removalPolicy: Allow to the crdify config to match o/api? Waiting for confirmation.

No code changes needed this round

No new code changes were required — all feedback is either addressed in existing commits or awaiting a policy decision.


AI-assisted response via Claude Code

@bryan-cox

Copy link
Copy Markdown
Member

/restructure-commits

@github-actions

Copy link
Copy Markdown

🤖 Restructuring commits: workflow run

@bryan-cox

Copy link
Copy Markdown
Member

/restructure-commits

@github-actions

Copy link
Copy Markdown

🤖 Restructuring commits: workflow run

OpenShift CI Bot added 2 commits June 19, 2026 00:05
- Add crd-schema-check tool using crdify library to detect breaking
  CRD schema changes between git revisions
- Wire verify-crd-schema target into Makefile verify pipeline
- Support configurable allowed changes via crdify-config.yaml
- Include unit tests for enum addition validation

Signed-off-by: OpenShift CI Bot <ci-bot@redhat.com>
Commit-Message-Assisted-by: Claude (via Claude Code)
- Add sigs.k8s.io/crdify library for CRD schema validation
- Add github.com/onsi/gomega for test assertions

Signed-off-by: OpenShift CI Bot <ci-bot@redhat.com>
Commit-Message-Assisted-by: Claude (via Claude Code)
@JoelSpeed

Copy link
Copy Markdown
Contributor

/approve

1 similar comment
@bryan-cox

Copy link
Copy Markdown
Member

/approve

@openshift-ci

openshift-ci Bot commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bryan-cox, hypershift-jira-solve-ci[bot], JoelSpeed

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 23, 2026
@enxebre

enxebre commented Jun 23, 2026

Copy link
Copy Markdown
Member

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jun 23, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-v2-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke

@hypershift-jira-solve-ci

Copy link
Copy Markdown
Contributor Author

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-aks | Build: 2069369535986667520 | Cost: $3.3625057500000004 | Failed step: hypershift-azure-run-e2e

View full analysis report


Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6

@hypershift-jira-solve-ci

Copy link
Copy Markdown
Contributor Author

I have all the evidence I need. The analysis is clear:

  1. 5 test failures — all in TestNodePool (HostedCluster0 and HostedCluster2 subtests)
  2. Root cause: cluster-policy-controller image failed to pull from quay-proxy.ci.openshift.org — a transient CI image registry issue
  3. Proof it's transient: The same image (sha256:ab6e20bd...) was successfully pulled ~30 min later by TestAutoscaling
  4. PR CNTRLPLANE-502: Add CRD breaking changes validation to HyperShift CI #8535 is unrelated: Only adds CI tooling under hack/tools/crd-schema-check/ — no production code changes
  5. 333 of 338 tests passed (35 skipped, 5 failed)

Test Failure Analysis Complete

Job Information

Test Failure Analysis

Error

TestNodePool/HostedCluster0/ValidateHostedCluster: Failed to wait for HostedCluster
e2e-clusters-96w8d/node-pool-d6rq6 to have valid conditions in 20m0s: context deadline exceeded
  - Degraded=True: UnavailableReplicas(cluster-policy-controller deployment has 1 unavailable replicas)
  - Available=False: ComponentsNotAvailable(Waiting for components to be available: cluster-policy-controller)
  - controlPlaneVersion state is Partial, expected Completed

TestNodePool/HostedCluster2/ValidateHostedCluster: Failed to wait for HostedCluster
e2e-clusters-gswfv/node-pool-vb7s5 to have valid conditions in 20m0s: context deadline exceeded
  - Degraded=True: UnavailableReplicas(cluster-policy-controller deployment has 2 unavailable replicas)
  - Available=False: ComponentsNotAvailable(Waiting for components to be available: cluster-policy-controller)
  - controlPlaneVersion state is Partial, expected Completed

Summary

All 5 test failures are in the TestNodePool test suite, caused by a transient CI image registry unavailability at quay-proxy.ci.openshift.org. The cluster-policy-controller container image could not be pulled for ~30 minutes, preventing two HostedClusters (HostedCluster0 and HostedCluster2) from reaching Available=True. The same image was successfully pulled by TestAutoscaling approximately 30 minutes later, confirming the issue was transient. This failure is unrelated to PR #8535, which only adds a CRD schema breaking-change validation tool under hack/tools/crd-schema-check/ with no production code changes.

Root Cause

Transient CI image registry unavailability at quay-proxy.ci.openshift.org

The cluster-policy-controller deployment requires an image from the CI image mirror (quay-proxy.ci.openshift.org/openshift/ci@sha256:ab6e20bd2581fc043a40bd157f2dd21a6125944d46c826b0e855828d791cae12). During the TestNodePool test execution window (~12:04 UTC), this image could not be pulled:

  • HostedCluster0 (node-pool-d6rq6): Image pull initiated at 12:04:17Z but hung — no Pulled, Created, or Started events were recorded. Pod remained in PodInitializing with imageID: "". Deployment hit ProgressDeadlineExceeded at 12:14:19Z.
  • HostedCluster2 (node-pool-vb7s5): Two pods affected — one with ImagePullBackOff (not found error), another with ErrImagePull (TLS handshake timeout).

Both HostedClusters remained Degraded=True and Available=False because the cluster-policy-controller deployment could not reach its desired replica count. The ValidateHostedCluster test timed out after 20 minutes waiting for valid conditions.

Proof of transient nature: The TestAutoscaling cluster successfully pulled the exact same image digest (sha256:ab6e20bd...) at ~12:34:25Z — confirming the CI registry recovered. 9 other top-level test suites (TestCreateCluster, TestHAEtcdChaos, TestAutoscaling, TestAzureScheduler, etc.) all passed.

PR #8535 changes are entirely unrelated: Only 6 non-vendor files changed, all under hack/tools/crd-schema-check/ and Makefile. No controller, operator, or HyperShift production code was modified.

Recommendations
  1. Retry the CI job — The transient image registry issue has resolved, as proven by the TestAutoscaling test pulling the same image ~30 minutes later in the same job run.
  2. No code changes needed — PR CNTRLPLANE-502: Add CRD breaking changes validation to HyperShift CI #8535 only adds CI tooling (hack/tools/crd-schema-check/) with no production code changes. The failure is purely infrastructure-related.
  3. This is a known CI infrastructure pattern — Transient unavailability of quay-proxy.ci.openshift.org periodically causes image pull failures in HyperShift e2e tests, particularly when multiple HostedClusters attempt to pull images during the same outage window.
Evidence
Evidence Detail
Failed tests TestNodePool/HostedCluster0/ValidateHostedCluster (1331.92s), TestNodePool/HostedCluster0 (2525.47s), TestNodePool/HostedCluster2/ValidateHostedCluster (1341.11s), TestNodePool/HostedCluster2 (2526.44s), TestNodePool (parent)
Failing component cluster-policy-controller deployment — image pull failure
Affected image quay-proxy.ci.openshift.org/openshift/ci@sha256:ab6e20bd2581fc043a40bd157f2dd21a6125944d46c826b0e855828d791cae12
HostedCluster0 error Image pull hung at 12:04:17Z, ProgressDeadlineExceeded at 12:14:19Z, pod stuck in PodInitializing with empty imageID
HostedCluster2 errors ImagePullBackOff: not found and ErrImagePull: TLS handshake timeout
Transient proof Same image successfully pulled by TestAutoscaling at ~12:34:25Z (confirmed matching digest)
Passing tests 333/338 passed including TestCreateCluster, TestHAEtcdChaos, TestAutoscaling, TestAzureScheduler, TestUpgradeControlPlane, TestCreateClusterCustomConfig, etc.
PR #8535 scope 6 non-vendor files: Makefile, hack/tools/crd-schema-check/{main.go, main_test.go, crdify-config.yaml}, hack/tools/{go.mod, go.sum} — no production code
Failed step e2e-aks-hypershift-azure-run-e2e (test phase, 1h39m41s)
CI analysis artifact hypershift-analyze-e2e-failure/artifacts/failure-analysis.md confirms same root cause

@bryan-cox

Copy link
Copy Markdown
Member

/test e2e-aks

@bryan-cox

Copy link
Copy Markdown
Member

/verified by GHA verify test

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jun 23, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@bryan-cox: This PR has been marked as verified by GHA verify test.

Details

In response to this:

/verified by GHA verify test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-merge-bot
openshift-merge-bot Bot merged commit 7eea4bd into openshift:main Jun 23, 2026
32 checks passed
@bryan-cox
bryan-cox deleted the fix-CNTRLPLANE-502 branch June 23, 2026 17:22
@openshift-ci

openshift-ci Bot commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

@hypershift-jira-solve-ci: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/ci-tooling Indicates the PR includes changes for CI or tooling jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants