CNTRLPLANE-3093,CNTRLPLANE-3096,CNTRLPLANE-3095,CNTRLPLANE-3098: lower Azure WI webhook test version gate to 4.20

[celebdor]

What this PR does / why we need it:

Lowers the Azure workload identity webhook e2e test version gate from Version422 to Version420. The feature was originally implemented for 4.22 (#7867) but was backported to 4.20 (#7998) and 4.21 (#7997). The e2e test should run on all versions where the feature is available.

Which issue(s) this PR fixes:

Fixes CNTRLPLANE-3093, fixes CNTRLPLANE-3096

Special notes for your reviewer:

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Summary by CodeRabbit

• Tests
  • Adjusted version compatibility testing for Azure Workload Identity Webhook support to cover earlier versions.

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[coderabbitai]

📝 Walkthrough

Walkthrough

The end-to-end test EnsureAzureWorkloadIdentityWebhookMutation in the Azure test utilities had its version gate requirement adjusted. The minimum version constraint was lowered from Version422 to Version420 using the AtLeast(t, ...) function. No modifications were made to test logic, object creation, or predicate assertions. This represents a single-line change affecting only the version compatibility requirement for the test.

🚥 Pre-merge checks | ✅ 9 | ❌ 3

❌ Failed checks (3 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Test Structure And Quality	⚠️ Warning	Test creates cluster-scoped resources (Namespace, ServiceAccount, Pod) without cleanup using t.Cleanup() or similar mechanism, violating established repository patterns.	Add cleanup logic using t.Cleanup() to delete test resources after test completes, ensuring resources don't leak into the cluster.
Ipv6 And Disconnected Network Test Compatibility	⚠️ Warning	Test pulls container image from public registry registry.k8s.io without using cluster-internal mirror, causing failure in disconnected environments with no internet access.	Use cluster-internal registry/image stream instead of public registry, implement image mirroring for disconnected environments, or add [Skipped:Disconnected] to test name.

✅ Passed checks (9 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	The test name is stable and deterministic with no dynamic information. The PR only changes the version gate parameter, not the test title. Dynamic values are correctly placed in the test body, not the test name.
Microshift Test Compatibility	✅ Passed	The test only uses core Kubernetes APIs and does not reference OpenShift-specific features unavailable on MicroShift.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	The EnsureAzureWorkloadIdentityWebhookMutation test creates and validates webhook mutations on a single pod in isolation, with no assumptions about multiple nodes, HA setup, or cluster topology.
Topology-Aware Scheduling Compatibility	✅ Passed	PR only modifies e2e test code (test/e2e/util/azure.go), not deployment manifests, operator code, or controllers where topology-aware scheduling would be a concern.
Ote Binary Stdout Contract	✅ Passed	The pull request does not violate the OTE Binary Stdout Contract as the modified test utility file contains no process-level code that could write to stdout.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and specifically describes the main change: lowering the Azure workload identity webhook test version gate to 4.20, with relevant ticket references.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[openshift-ci-robot]

@celebdor: This pull request references CNTRLPLANE-3093 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

What this PR does / why we need it:

Lowers the Azure workload identity webhook e2e test version gate from Version422 to Version420. The feature was originally implemented for 4.22 (#7867) but was backported to 4.20 (#7998) and 4.21 (#7997). The e2e test should run on all versions where the feature is available.

Which issue(s) this PR fixes:

Fixes CNTRLPLANE-3093, fixes CNTRLPLANE-3096

Special notes for your reviewer:

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@celebdor: This pull request references CNTRLPLANE-3093 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

This pull request references CNTRLPLANE-3096 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

This pull request references CNTRLPLANE-3095 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

This pull request references CNTRLPLANE-3098 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

What this PR does / why we need it:

Lowers the Azure workload identity webhook e2e test version gate from Version422 to Version420. The feature was originally implemented for 4.22 (#7867) but was backported to 4.20 (#7998) and 4.21 (#7997). The e2e test should run on all versions where the feature is available.

Which issue(s) this PR fixes:

Fixes CNTRLPLANE-3093, fixes CNTRLPLANE-3096

Special notes for your reviewer:

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Summary by CodeRabbit

• Tests
• Adjusted version compatibility testing for Azure Workload Identity Webhook support to cover earlier versions.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[celebdor]

/jira refresh

[openshift-ci-robot]

@celebdor: This pull request references CNTRLPLANE-3093 which is a valid jira issue.

This pull request references CNTRLPLANE-3096 which is a valid jira issue.

This pull request references CNTRLPLANE-3095 which is a valid jira issue.

This pull request references CNTRLPLANE-3098 which is a valid jira issue.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 41.66%. Comparing base (4755e9c) to head (7416bf2).
⚠️ Report is 21 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8323      +/-   ##
==========================================
+ Coverage   41.54%   41.66%   +0.11%     
==========================================
  Files         758      758              
  Lines       93838    93929      +91     
==========================================
+ Hits        38986    39135     +149     
+ Misses      52107    52046      -61     
- Partials     2745     2748       +3     

see 12 files with indirect coverage changes

Flag	Coverage Δ
cmd-support	34.96% <ø> (-0.01%)	⬇️
cpo-hostedcontrolplane	44.00% <ø> (+0.40%)	⬆️
cpo-other	43.45% <ø> (+0.27%)	⬆️
hypershift-operator	51.65% <ø> (+0.03%)	⬆️
other	31.56% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[enxebre]

/lgtm

[openshift-merge-bot]

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks-4-22
/test e2e-aws-4-22
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws

[cwbotbot]

Test Results

e2e-aws

• Status: ❌ FAIL
• Started: 2026-04-28T19:08:51Z
• View Job
• View Job History

e2e-aks

• Status: ✅ PASS
• Started: 2026-04-28T17:20:10Z
• View Job
• View Job History

[hypershift-jira-solve-ci]

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-aks | Build: 2047572605442985984 | Cost: $2.8566305 | Failed step: hypershift-azure-run-e2e

View full analysis report

---

_{Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6}

[hypershift-jira-solve-ci]

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-azure-self-managed | Build: 2047572605686255616 | Cost: $2.0333102499999995 | Failed step: hypershift-azure-run-e2e-self-managed

View full analysis report

---

_{Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6}

[celebdor]

/retest-required

[hypershift-jira-solve-ci]

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-aws | Build: 2049204112108556288 | Cost: $3.50828595 | Failed step: hypershift-aws-run-e2e-nested

View full analysis report

---

_{Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6}

[openshift-ci]

Stale PRs are closed after 21d of inactivity.

If this PR is still relevant, comment to refresh it or remove the stale label.
Mark the PR as fresh by commenting /remove-lifecycle stale.

If this PR is safe to close now please do so with /close.

/lifecycle stale

[bryan-cox]

/lgtm

[openshift-merge-bot]

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks-4-22
/test e2e-aws-4-22
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-v2-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bryan-cox, celebdor

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [bryan-cox]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[hypershift-jira-solve-ci]

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-aws | Build: 2066529501940027392 | Cost: $1.8139094999999996 | Failed step: hypershift-aws-run-e2e-nested

View full analysis report

---

_{Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6}

[hypershift-jira-solve-ci]

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-aks | Build: 2066529501847752704 | Cost: $3.875517749999999 | Failed step: hypershift-azure-run-e2e

View full analysis report

---

_{Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6}

[celebdor]

/retest-required

Since the issues were infrastructure related

[openshift-ci]

@celebdor: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-azure-self-managed	883c21e	link	true	/test e2e-azure-self-managed
ci/prow/e2e-aws	7416bf2	link	true	/test e2e-aws
ci/prow/e2e-aws-4-22	7416bf2	link	true	/test e2e-aws-4-22

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[hypershift-jira-solve-ci]

There it is. nodePool.Status.Replicas is set from machineDeployment.Status.AvailableReplicas (line 641). This means it reflects the number of available nodes (nodes that are ready and running). If one of the 2 native nodes wasn't yet available when the baseline was read, Status.Replicas = 1, giving baseline = 2 vCPUs. Later when both nodes become available, Status.Replicas = 2, adding 4 vCPUs to the metric total.

This is the root cause: the test reads the baseline hypershift_cluster_vcpus metric before all native NodePool replicas have become available, creating a stale baseline. When the metric is later queried, the native replicas have fully converged, inflating the total by the additional vCPUs.

This is completely unrelated to PR #8323 (Azure WI version gate change). Now let me produce the final report.

Test Failure Analysis Complete

Job Information

• Prow Job: pull-ci-openshift-hypershift-main-e2e-aws
• Build ID: 2067611357305376768
• Target: e2e-aws
• Test: TestKarpenter/Main/Billing_vCPUs,_consolidation,_and_cluster_deletion_with_blocking_PDB
• PR: CNTRLPLANE-3093,CNTRLPLANE-3096,CNTRLPLANE-3095,CNTRLPLANE-3098: lower Azure WI webhook test version gate to 4.20 #8323 — CNTRLPLANE-3093,CNTRLPLANE-3096,CNTRLPLANE-3095,CNTRLPLANE-3098: lower Azure WI webhook test version gate to 4.20

Test Failure Analysis

Error

karpenter_test.go:1464:
    failed to validate hypershift_cluster_vcpus metric
    Unexpected error:
        <context.deadlineExceededError>:
        context deadline exceeded
        {}
    occurred

hypershift_cluster_vcpus=12, want 10 for cluster e2e-clusters-8hw25/karpenter-qtntl

Summary

The TestKarpenter/Main/Billing_vCPUs,_consolidation,_and_cluster_deletion_with_blocking_PDB test failed because the hypershift_cluster_vcpus billing metric reported 12 vCPUs instead of the expected 10. This is a pre-existing flaky test caused by a race condition in baseline measurement — completely unrelated to PR #8323, which only changes Azure WI webhook test version gates from 4.22 to 4.20.

Root Cause

The test reads a baseline vCPU count from the hypershift_cluster_vcpus Prometheus metric before all native NodePool replicas have become available, then later asserts that baseline + karpenter_vcpus equals the metric total. The metric computation (metrics.go:466-473) uses nodePool.Status.Replicas, which is sourced from machineDeployment.Status.AvailableReplicas (capi.go:641). This reflects the count of nodes that are fully available — not the desired count.

The race:

1. The cluster is created with NodePoolReplicas: 2 and a default m5.large instance type (2 vCPUs each).
2. The test reads the baseline at karpenter_test.go:1441. At this moment, only 1 of 2 native nodes is available (Status.Replicas = 1), so baseline = 1 × 2 = 2.
3. Karpenter provisions 2 × t3.xlarge nodes (4 vCPUs each), and AutoNode.VCPUs = 8.
4. The test expects the metric to be baseline + 8 = 10.
5. By the time the metric is checked, the second native node has become available (Status.Replicas = 2), so the actual metric is 8 (Karpenter) + 2 × 2 (native) = 12.
6. The metric never converges to 10 because the true total is 12. The test times out after 1 minute of polling.

This is a test-level race condition in testBillingConsolidationAndPDB(). The baseline is captured at a point where NodePool.Status.Replicas has not fully converged to the desired replica count. The metric later reflects the fully-converged state, causing a permanent 2-vCPU discrepancy.

PR #8323 only modifies test/e2e/util/azure.go and test/e2e/v2/tests/hosted_cluster_azure_test.go to lower a version gate from Version422 to Version420 — it has no overlap with Karpenter, billing metrics, or NodePool logic.

Recommendations

1. Fix the test baseline measurement: The test at karpenter_test.go:1441-1443 should wait for NodePool.Status.Replicas to equal Spec.Replicas before reading the baseline metric, ensuring native nodes are fully available. Alternatively, wait for the metric to stabilize (e.g., read it twice with a delay and assert convergence) before using it as a baseline.

2. Retrigger the job: Since this failure is unrelated to PR CNTRLPLANE-3093,CNTRLPLANE-3096,CNTRLPLANE-3095,CNTRLPLANE-3098: lower Azure WI webhook test version gate to 4.20 #8323, rerunning with /test e2e-aws should pass. The race is timing-dependent and does not reproduce consistently.

3. File a bug for the flaky test: Track this as a known flake in TestKarpenter/Main/Billing_vCPUs,_consolidation,_and_cluster_deletion_with_blocking_PDB to prevent it from blocking unrelated PRs.

Evidence

Evidence	Detail
Failing test	TestKarpenter/Main/Billing_vCPUs,_consolidation,_and_cluster_deletion_with_blocking_PDB at karpenter_test.go:1464
Error	hypershift_cluster_vcpus=12, want 10 — metric never converges, context deadline exceeded after 1 min
Baseline reading	karpenter_test.go:1443: "Baseline billing metric vCPUs from native NodePools: 2"
Expected calculation	baseline(2) + Karpenter(8) = 10
Actual metric	12 = Karpenter(8) + 2 native nodes × 2 vCPUs each(4)
Root cause location	metrics.go:467-473: uses nodePool.Status.Replicas (available count, not desired)
Status.Replicas source	capi.go:641: nodePool.Status.Replicas = ptr.Deref(machineDeployment.Status.AvailableReplicas, 0)
PR #8323 changes	Only test/e2e/util/azure.go and test/e2e/v2/tests/hosted_cluster_azure_test.go — Azure WI version gate, unrelated to Karpenter
Cluster config	NodePoolReplicas: 2, m5.large (2 vCPUs), AutoNode: true
Test duration	765.20s (12m 45s) — spent most of it polling the metric that never converges

---