OCPBUGS-84969: fix(e2e): wait for Karpenter node cleanup in parallel tests to prevent vCPU flake

[bryan-cox]

What this PR does / why we need it:

Fixes a 50% flake rate in TestKarpenter/Main/Billing_vCPUs,_consolidation,_and_cluster_deletion_with_blocking_PDB across all e2e-aws runs.

testCapacityReservation and testNodeClassVersionField provision Karpenter nodes but only rely on t.Cleanup for deletion without waiting for nodes to fully terminate. When the sequential testBillingConsolidationAndPDB runs next and asserts AutoNode.VCPUs=0, stale NodeClaims from these parallel tests cause the assertion to fail with AutoNode.VCPUs=4, want 0 (one lingering t3.xlarge = 4 vCPUs).

This adds explicit WaitForReadyNodesByLabels(..., 0, ...) calls after deleting workloads and NodePools, matching the pattern already used by testARM64Provisioning and testInstanceProfileAnnotation.

Which issue(s) this PR fixes:

6 out of 12 e2e-aws runs in the last 24h failed due to this flake, blocking unrelated PRs (#8385, #8396, #8404, #8409).

https://redhat.atlassian.net/browse/OCPBUGS-84969

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Summary by CodeRabbit

• Tests
  • Enhanced test flow to ensure explicit cleanup of managed resources and full node termination before running subsequent checks, preventing resource leakage and improving overall test reliability across multiple test scenarios.

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[openshift-ci-robot]

@bryan-cox: This pull request explicitly references no jira issue.

Details

In response to this:

What this PR does / why we need it:

Fixes a 50% flake rate in TestKarpenter/Main/Billing_vCPUs,_consolidation,_and_cluster_deletion_with_blocking_PDB across all e2e-aws runs.

testCapacityReservation and testNodeClassVersionField provision Karpenter nodes but only rely on t.Cleanup for deletion without waiting for nodes to fully terminate. When the sequential testBillingConsolidationAndPDB runs next and asserts AutoNode.VCPUs=0, stale NodeClaims from these parallel tests cause the assertion to fail with AutoNode.VCPUs=4, want 0 (one lingering t3.xlarge = 4 vCPUs).

This adds explicit WaitForReadyNodesByLabels(..., 0, ...) calls after deleting workloads and NodePools, matching the pattern already used by testARM64Provisioning and testInstanceProfileAnnotation.

Which issue(s) this PR fixes:

6 out of 12 e2e-aws runs in the last 24h failed due to this flake, blocking unrelated PRs (#8385, #8396, #8404, #8409).

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[coderabbitai]

📝 Walkthrough

Walkthrough

The test file test/e2e/karpenter_test.go was modified to include explicit cleanup steps in two test paths. In the version field test scenario, after validating EC2 instance metadata options, the test now actively deletes the test workloads and node pool, then waits for node termination. Similarly, in the capacity reservation test scenario, after verifying instance placement, the test deletes the workload and node pool and waits for provisioned nodes to fully disappear. These changes ensure deterministic cleanup completion before proceeding to subsequent test operations.

🚥 Pre-merge checks | ✅ 11 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (11 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	All test names are static strings with no dynamic identifiers. No pod names, timestamps, UUIDs, node names, namespaces, IPs, or changing values found. PR changes only add cleanup logic.
Test Structure And Quality	✅ Passed	Tests meet all quality requirements: proper single responsibility, cleanup with explicit waits, timeouts configured, meaningful assertions, and consistency with codebase patterns.
Microshift Test Compatibility	✅ Passed	PR modifies existing e2e tests (not adding new ones). Only test entry point has explicit AWS platform skip guard, preventing MicroShift execution.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	This PR modifies tests using Go's standard testing package (t.Run), not Ginkgo (It/Describe/Context). The custom check applies only to Ginkgo e2e tests, so this check does not apply.
Topology-Aware Scheduling Compatibility	✅ Passed	This PR modifies only e2e test code. It does not add deployment manifests, operator code, controllers, or scheduling constraints affecting topology compatibility (SNO, TNF, TNA, HyperShift).
Ote Binary Stdout Contract	✅ Passed	Changes are inside test function closures only. No fmt/log/klog stdout writes. No process-level code modified. No OTE Binary Stdout Contract violations.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	No new Ginkgo tests added. Changes modify only existing test functions with cleanup improvements. Custom check applies to new tests only.
Title check	✅ Passed	The title clearly and specifically describes the main change: fixing Karpenter node cleanup in parallel e2e tests to prevent a vCPU flake, which directly matches the file changes that add explicit node termination waits.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bryan-cox

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [bryan-cox]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

@bryan-cox: This pull request references Jira Issue OCPBUGS-84969, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

What this PR does / why we need it:

Fixes a 50% flake rate in TestKarpenter/Main/Billing_vCPUs,_consolidation,_and_cluster_deletion_with_blocking_PDB across all e2e-aws runs.

testCapacityReservation and testNodeClassVersionField provision Karpenter nodes but only rely on t.Cleanup for deletion without waiting for nodes to fully terminate. When the sequential testBillingConsolidationAndPDB runs next and asserts AutoNode.VCPUs=0, stale NodeClaims from these parallel tests cause the assertion to fail with AutoNode.VCPUs=4, want 0 (one lingering t3.xlarge = 4 vCPUs).

This adds explicit WaitForReadyNodesByLabels(..., 0, ...) calls after deleting workloads and NodePools, matching the pattern already used by testARM64Provisioning and testInstanceProfileAnnotation.

Which issue(s) this PR fixes:

6 out of 12 e2e-aws runs in the last 24h failed due to this flake, blocking unrelated PRs (#8385, #8396, #8404, #8409).

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Summary by CodeRabbit

• Tests
• Enhanced test flow to ensure explicit cleanup of managed resources and full node termination before running subsequent checks, preventing resource leakage and improving overall test reliability across multiple test scenarios.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@bryan-cox: This pull request references Jira Issue OCPBUGS-84969, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

What this PR does / why we need it:

Fixes a 50% flake rate in TestKarpenter/Main/Billing_vCPUs,_consolidation,_and_cluster_deletion_with_blocking_PDB across all e2e-aws runs.

testCapacityReservation and testNodeClassVersionField provision Karpenter nodes but only rely on t.Cleanup for deletion without waiting for nodes to fully terminate. When the sequential testBillingConsolidationAndPDB runs next and asserts AutoNode.VCPUs=0, stale NodeClaims from these parallel tests cause the assertion to fail with AutoNode.VCPUs=4, want 0 (one lingering t3.xlarge = 4 vCPUs).

This adds explicit WaitForReadyNodesByLabels(..., 0, ...) calls after deleting workloads and NodePools, matching the pattern already used by testARM64Provisioning and testInstanceProfileAnnotation.

Which issue(s) this PR fixes:

6 out of 12 e2e-aws runs in the last 24h failed due to this flake, blocking unrelated PRs (#8385, #8396, #8404, #8409).

https://redhat.atlassian.net/browse/OCPBUGS-84969

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Summary by CodeRabbit

• Tests
• Enhanced test flow to ensure explicit cleanup of managed resources and full node termination before running subsequent checks, preventing resource leakage and improving overall test reliability across multiple test scenarios.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 37.23%. Comparing base (ad88854) to head (8e84889).
⚠️ Report is 17 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #8414   +/-   ##
=======================================
  Coverage   37.23%   37.23%           
=======================================
  Files         752      752           
  Lines       91829    91829           
=======================================
  Hits        34195    34195           
  Misses      54993    54993           
  Partials     2641     2641           

Flag	Coverage Δ
cmd-support	32.06% <ø> (ø)
cpo-hostedcontrolplane	36.50% <ø> (ø)
cpo-other	37.73% <ø> (ø)
hypershift-operator	47.85% <ø> (ø)
other	27.77% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bryan-cox]

/test e2e-aws

[bryan-cox]

/verified by e2e

[openshift-ci-robot]

@bryan-cox: This PR has been marked as verified by e2e.

Details

In response to this:

/verified by e2e

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[joshbranham]

/lgtm

[openshift-merge-bot]

Tests from second stage were triggered manually. Pipeline can be controlled only manually, until HEAD changes. Use command to trigger second stage.

[bryan-cox]

/pipeline auto

[openshift-merge-bot]

Pipeline controller notification

The pipeline-auto label has been added to this PR. Second-stage tests will be triggered automatically when all first-stage tests pass.

[openshift-merge-bot]

Tests from second stage were triggered manually. Pipeline can be controlled only manually, until HEAD changes. Use command to trigger second stage.

[bryan-cox]

/pipeline required

[openshift-merge-bot]

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks-4-22
/test e2e-aws-4-22
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws

[openshift-merge-bot]

/retest-required

Remaining retests: 0 against base HEAD 0ca85b1 and 2 for PR HEAD 8e84889 in total

[cwbotbot]

Test Results

e2e-aws

• Status: ✅ PASS
• Started: 2026-05-05T00:42:40Z
• View Job
• View Job History

e2e-aks

• Status: ✅ PASS
• Started: 2026-05-05T02:48:44Z
• View Job
• View Job History

[bryan-cox]

/retest

[bryan-cox]

Confirmed from the e2e-aws build log that both tests waited for nodes to fully terminate before the billing test ran:

testCapacityReservation — waited 1m39s for nodes to drain:

=== NAME  TestKarpenter/Main/Parallel_provisioning_tests/Capacity_reservation_selector_propagation
    util.go:571: Successfully waited for 0 nodes to become ready in 1m39s

testNodeClassVersionField — waited 1m36s for nodes to drain:

=== NAME  TestKarpenter/Main/Parallel_provisioning_tests/OpenshiftEC2NodeClass_version_field_and_MetadataOptions
    util.go:571: Successfully waited for 0 nodes to become ready in 1m36s

testBillingConsolidationAndPDB then ran with a clean slate — AutoNode.VCPUs=0 converged immediately (0s), confirming no stale NodeClaims leaked:

=== RUN   TestKarpenter/Main/Billing_vCPUs,_consolidation,_and_cluster_deletion_with_blocking_PDB
    karpenter_test.go:1253: Validating AutoNode.VCPUs converges to 0
    karpenter_test.go:1253: Successfully waited for HostedCluster e2e-clusters-kzpsb/karpenter-tj4k9 AutoNode.VCPUs=0 in 0s

All Karpenter subtests passed, including the previously flaky billing test:

--- PASS: TestKarpenter/Main/Parallel_provisioning_tests/Capacity_reservation_selector_propagation (455.93s)
--- PASS: TestKarpenter/Main/Parallel_provisioning_tests/OpenshiftEC2NodeClass_version_field_and_MetadataOptions (420.88s)
--- PASS: TestKarpenter/Main/Billing_vCPUs,_consolidation,_and_cluster_deletion_with_blocking_PDB (777.35s)

[openshift-ci]

@bryan-cox: you cannot LGTM your own PR.

Details

In response to this:

Confirmed that both test functions now properly wait for node termination before proceeding:

1. testNodeClassVersionField — after deleting testWorkLoads and testNodePool, calls WaitForReadyNodesByLabels(..., 0, testNodeLabels) to block until all nodes with those labels are fully terminated.

2. testCapacityReservation — after deleting crWorkload and crNodePool, calls WaitForReadyNodesByLabels(..., 0, crNodeLabels) to do the same.

Both use a target count of 0, ensuring no stale NodeClaims can leak vCPUs into the subsequent sequential testBillingConsolidationAndPDB. This matches the existing cleanup pattern already used by testARM64Provisioning and testInstanceProfileAnnotation.

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-ci]

@bryan-cox: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci-robot]

@bryan-cox: Jira Issue Verification Checks: Jira Issue OCPBUGS-84969
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-84969 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

What this PR does / why we need it:

Fixes a 50% flake rate in TestKarpenter/Main/Billing_vCPUs,_consolidation,_and_cluster_deletion_with_blocking_PDB across all e2e-aws runs.

testCapacityReservation and testNodeClassVersionField provision Karpenter nodes but only rely on t.Cleanup for deletion without waiting for nodes to fully terminate. When the sequential testBillingConsolidationAndPDB runs next and asserts AutoNode.VCPUs=0, stale NodeClaims from these parallel tests cause the assertion to fail with AutoNode.VCPUs=4, want 0 (one lingering t3.xlarge = 4 vCPUs).

This adds explicit WaitForReadyNodesByLabels(..., 0, ...) calls after deleting workloads and NodePools, matching the pattern already used by testARM64Provisioning and testInstanceProfileAnnotation.

Which issue(s) this PR fixes:

6 out of 12 e2e-aws runs in the last 24h failed due to this flake, blocking unrelated PRs (#8385, #8396, #8404, #8409).

https://redhat.atlassian.net/browse/OCPBUGS-84969

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Summary by CodeRabbit

• Tests
• Enhanced test flow to ensure explicit cleanup of managed resources and full node termination before running subsequent checks, preventing resource leakage and improving overall test reliability across multiple test scenarios.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.