NO-JIRA: Skip TestKarpenter for releases < 4.22

[enxebre]

Summary

• Skip TestKarpenter e2e suite for releases < 4.22. Karpenter is only supported from 4.22 onwards.
• Unblock periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-aws-ovn and periodic-ci-openshift-hypershift-release-4.19-periodics-e2e-aws-ovn failing since April 30

Test plan

• Verify e2e tests compile with go build -tags e2e ./test/e2e/
• Verify 4.20/4.19 periodic jobs start passing (Karpenter tests will be skipped)
• Verify 4.22+ periodic jobs still run Karpenter tests

🤖 Generated with Claude Code

Summary by CodeRabbit

• Tests
  • Raised the minimum end-to-end test environment version requirement from 4.19 to 4.22. This affects execution of AWS Karpenter end-to-end tests and ensures they run only on environments meeting the new version threshold.

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[openshift-ci-robot]

@enxebre: This pull request explicitly references no jira issue.

Details

In response to this:

Summary

• Skip TestKarpenter e2e suite when running against clusters < 4.21
• Fixes 100% failure rate on periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-aws-ovn and periodic-ci-openshift-hypershift-release-4.19-periodics-e2e-aws-ovn since April 30

Root Cause

The periodic e2e jobs for 4.19/4.20 use the hypershift-tests binary built from main (not from the release branch). After df33c3502b removed the TECH_PREVIEW_NO_UPGRADE skip guard on main, TestKarpenter started running against 4.19/4.20 clusters for the first time.

The test code computes n-2 versions and constructs .0 patch versions (e.g., 4.18.0 for 4.20, 4.15.1 for 4.19) that don't exist in Cincinnati fast channels, causing deterministic ResolutionFailed errors.

Additionally, the capacity reservation test fails because the upstream karpenter ReservedCapacity feature gate is disabled by default.

These tests were never intended to run against pre-GA Karpenter releases (they were behind the tech preview gate).

Test plan

• Verify e2e tests compile with go build -tags e2e ./test/e2e/
• Verify 4.20/4.19 periodic jobs start passing (Karpenter tests will be skipped)
• Verify 4.21+ periodic jobs still run Karpenter tests

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: f9e77127-0370-489a-b92d-a55749ba9c5c

📥 Commits

Reviewing files that changed from the base of the PR and between c146d81 and 604f52a.

📒 Files selected for processing (1)

• test/e2e/karpenter_test.go

🚧 Files skipped from review as they are similar to previous changes (1)

• test/e2e/karpenter_test.go

---

📝 Walkthrough

Walkthrough

The TestKarpenter function in the e2e test suite now requires a minimum environment version of e2eutil.Version422 (previously e2eutil.Version419) via the e2eutil.AtLeast(...) check before executing AWS platform-specific Karpenter test logic.

🚥 Pre-merge checks | ✅ 11 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (11 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the main change: skipping TestKarpenter for releases below 4.22, which matches the version constraint added in the code.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	PR modifies only the minimum environment version check from e2eutil.Version419 to e2eutil.Version422 in TestKarpenter function precondition; all test names remain stable with descriptive static strings containing no dynamic information.
Test Structure And Quality	✅ Passed	Test code demonstrates adequate structure and quality with focused responsibilities, proper cleanup using Go's t.Cleanup() function, and consistent timeout implementations.
Microshift Test Compatibility	✅ Passed	This PR modifies the minimum version requirement of an existing test, not adding new Ginkgo e2e tests, so the custom check does not apply.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	The PR does not add new Ginkgo e2e tests, only modifies version preconditions on an existing test, making the SNO compatibility check not applicable.
Topology-Aware Scheduling Compatibility	✅ Passed	The PR only modifies a test file to add a version gate; no deployment manifests, operator code, or controllers are modified.
Ote Binary Stdout Contract	✅ Passed	The PR changes version requirement in TestKarpenter test function from Version419 to Version422. The version.go utility uses fmt.Fprintf(ginkgo.GinkgoWriter,...) which writes to GinkgoWriter (not stdout), explicitly allowed. No process-level stdout violations detected.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	The custom check applies only to newly added Ginkgo e2e tests. This PR modifies an existing test's version precondition, not adding new tests.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Review rate limit: 9/10 reviews remaining, refill in 6 minutes.}

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: enxebre

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [enxebre]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 37.22%. Comparing base (68106f0) to head (604f52a).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #8409   +/-   ##
=======================================
  Coverage   37.22%   37.22%           
=======================================
  Files         750      750           
  Lines       91789    91789           
=======================================
  Hits        34172    34172           
  Misses      54978    54978           
  Partials     2639     2639           

Flag	Coverage Δ
cmd-support	32.06% <ø> (ø)
cpo-hostedcontrolplane	36.45% <ø> (ø)
cpo-other	37.73% <ø> (ø)
hypershift-operator	47.85% <ø> (ø)
other	27.77% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[joshbranham]

/lgtm

[enxebre]

/verified by e2e @enxebre

[openshift-ci-robot]

@enxebre: This PR has been marked as verified by e2e @enxebre.

Details

In response to this:

/verified by e2e @enxebre

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-merge-bot]

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks-4-22
/test e2e-aws-4-22
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws

[cwbotbot]

Test Results

e2e-aws

• Status: ❌ FAIL
• Started: 2026-05-05T01:16:18Z
• View Job
• View Job History

e2e-aks

• Status: ✅ PASS
• Started: 2026-05-04T15:13:53Z
• View Job
• View Job History

[openshift-merge-bot]

/retest-required

Remaining retests: 0 against base HEAD ad88854 and 2 for PR HEAD 604f52a in total

[hypershift-jira-solve-ci]

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-aws | Build: 2051366291926683648 | Cost: $4.27864245 | Failed step: hypershift-aws-run-e2e-nested

View full analysis report

---

_{Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6}

[hypershift-jira-solve-ci]

Now I have all the evidence. Here is the final report:

Test Failure Analysis Complete

Job Information

• Prow Job: pull-ci-openshift-hypershift-main-e2e-aws
• Build ID: 2051366291926683648
• Target: e2e-aws
• PR: NO-JIRA: Skip TestKarpenter for releases < 4.22 #8409 (NO-JIRA: Skip TestKarpenter for releases < 4.22)
• Failed Test: TestKarpenter/Main/Billing_vCPUs,_consolidation,_and_cluster_deletion_with_blocking_PDB

Test Failure Analysis

Error

karpenter_test.go:1245: Failed to wait for HostedCluster e2e-clusters-zw4g6/karpenter-nxlst AutoNode.VCPUs=0 in 1m0s: context deadline exceeded
eventually.go:225: observed *v1beta1.HostedCluster e2e-clusters-zw4g6/karpenter-nxlst invalid at RV 130771 after 1m0s: AutoNode.VCPUs=4, want 0
eventually.go:105: Failed to get *v1beta1.HostedCluster: client rate limiter Wait returned an error: context deadline exceeded

Summary

This failure is unrelated to PR #8409 (which only changes e2eutil.AtLeast(t, e2eutil.Version419) to e2eutil.AtLeast(t, e2eutil.Version422)). Since the e2e-aws job runs on the main branch at version 5.0.0, the version gate change has zero impact on test execution. The failure is a pre-existing race condition in the Karpenter e2e test suite: the Billing_vCPUs subtest expects AutoNode.VCPUs=0 at entry, but the preceding OpenshiftEC2NodeClass_version_field_and_MetadataOptions parallel subtest provisions a Karpenter-managed node (ip-10-0-135-132.ec2.internal, 4 vCPUs) and does not wait for it to terminate before returning. When the AutoNode enable/disable lifecycle test then disables and re-enables AutoNode, the karpenter-operator reconciles and reports the still-existing node's vCPUs, leaving AutoNode.VCPUs=4 when the billing test starts and expects 0. The 1-minute timeout is insufficient for the node to fully terminate and the status to propagate through the HCP → HostedCluster chain.

Root Cause

The root cause is a race condition between parallel Karpenter subtests and the sequential billing vCPUs test:

1. Parallel provisioning subtests create Karpenter nodes: Five subtests run in parallel (ARM64_instance_provisioning [skipped], Capacity_reservation_selector_propagation, Arbitrary_subnet_propagation, Instance_profile_annotation_propagation, OpenshiftEC2NodeClass_version_field_and_MetadataOptions). Each creates a Karpenter NodePool and provisions nodes.

2. Incomplete node cleanup in OpenshiftEC2NodeClass_version_field_and_MetadataOptions: This subtest provisions node ip-10-0-135-132.ec2.internal (instance i-0cff0c6c2370242b1, 4 vCPUs) but only cleans up the OpenshiftEC2NodeClass resource ("version-skew-test"). It does not explicitly wait for the provisioned node to terminate (unlike the Instance_profile_annotation_propagation test which calls "Waiting for Karpenter nodes to be deleted" and waits for 0 nodes).

3. AutoNode disable/re-enable exposes the stale node: The AutoNode_enable/disable_lifecycle test disables AutoNode (clearing HCP.Status.AutoNode) then re-enables it. On re-enable, the karpenter-operator reconciles and discovers the still-existing Karpenter-managed node, writing VCPUs=4 to HCP status.

4. Billing test precondition fails: The Billing_vCPUs test starts immediately after and asserts AutoNode.VCPUs=0 with a 1-minute timeout. The residual node's 4 vCPUs never converge to 0 within the timeout because the node hasn't finished terminating. Additionally, a client rate limiter error (client rate limiter Wait returned an error: context deadline exceeded) compounds the issue by reducing the number of successful status polls within the timeout window.

This is a test infrastructure flake — the product is behaving correctly (reporting real Karpenter-managed nodes), but the test's precondition assumes all prior Karpenter nodes have been fully cleaned up, which is not guaranteed by the test teardown logic.

Recommendations

1. Re-trigger the job — This failure is unrelated to PR NO-JIRA: Skip TestKarpenter for releases < 4.22 #8409's changes and is a pre-existing flake in the Karpenter test suite. A re-run is likely to pass.

2. Fix the OpenshiftEC2NodeClass_version_field_and_MetadataOptions subtest — Add an explicit wait for Karpenter node termination (similar to the Instance_profile_annotation_propagation test's karpenter_test.go:413: Waiting for Karpenter nodes to be deleted + Successfully waited for 0 nodes) after the MetadataOptions validation completes.

3. Add a synchronization barrier after all parallel provisioning tests — Before the sequential AutoNode_enable/disable_lifecycle test runs, insert a wait ensuring all Karpenter-managed nodes have been terminated (i.e., AutoNode.VCPUs=0 or node count with karpenter.sh/nodepool label is 0).

4. Increase the billing test precondition timeout — The current 1-minute timeout at karpenter_test.go:1245 may be insufficient when Karpenter nodes are still draining from prior subtests. Consider increasing to 3–5 minutes as a safety margin, though the proper fix is option 2/3.

Evidence

Evidence	Detail
PR #8409 diff	Single-line change: e2eutil.AtLeast(t, e2eutil.Version419) → e2eutil.AtLeast(t, e2eutil.Version422) — no impact on main branch (version 5.0.0)
Failed test	TestKarpenter/Main/Billing_vCPUs,_consolidation,_and_cluster_deletion_with_blocking_PDB (60.00s timeout)
Error message	AutoNode.VCPUs=4, want 0 — 4 vCPUs from an uncleared Karpenter node
Residual node	ip-10-0-135-132.ec2.internal (instance i-0cff0c6c2370242b1) provisioned by OpenshiftEC2NodeClass_version_field_and_MetadataOptions subtest
Missing cleanup	OpenshiftEC2NodeClass_version_field_and_MetadataOptions cleans up NodeClass but does NOT wait for node termination (unlike Instance_profile_annotation_propagation which calls Waiting for Karpenter nodes to be deleted)
Rate limiter	client rate limiter Wait returned an error: context deadline exceeded — API throttling reduces effective polling during the 1-minute timeout
Initial VCPUs=0 pass	karpenter_test.go:136: Successfully waited for HostedCluster ... AutoNode.VCPUs=0 in 0s — VCPUs were 0 before parallel tests provisioned nodes
Test results	521 tests total, 25 skipped, 3 failures (all from TestKarpenter cascade: parent test + Main + billing subtest)
Failed step	e2e-aws-hypershift-aws-run-e2e-nested (test phase), all other steps passed
Cluster health	All HostedCluster conditions healthy — Available=True, Degraded=False, AutoNodeEnabled=True, no operator issues

---

[openshift-merge-bot]

/retest-required

Remaining retests: 0 against base HEAD 0ca85b1 and 1 for PR HEAD 604f52a in total

[hypershift-jira-solve-ci]

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-aws | Build: 2051470920450576384 | Cost: $5.266338399999999 | Failed step: hypershift-aws-run-e2e-nested

View full analysis report

---

_{Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6}

[openshift-ci]

@enxebre: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws	604f52a	link	true	/test e2e-aws

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.