NO-JIRA: fix(e2e-v2): gate ConfigOperatorReconciliationSucceeded on 4.23+ by muraee · Pull Request #8875 · openshift/hypershift

muraee · 2026-06-30T15:40:04Z

Summary

The v2 health test unconditionally expects ConfigOperatorReconciliationSucceeded, which was introduced in PR CNTRLPLANE-597: Surface HCCO reconciliation failures in HostedCluster status #8340 for 4.23+
The v1 test already gates this with IsLessThan(Version423), but the v2 test was missing the same check
This causes e2e-v2-aws failures on all release-4.22 PRs since the condition doesn't exist on 4.22 clusters

Test plan

Verify e2e-v2-aws passes on release-4.22 PRs after this merges
Verify e2e-v2-aws still passes on main (4.23+ clusters should have the condition)

🤖 Generated with Claude Code

Summary by CodeRabbit

Tests
- Updated hosted cluster health checks to align expected conditions with older platform versions, reducing false failures when validating version-specific hosted cluster behavior.

openshift-merge-bot · 2026-06-30T15:40:06Z

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

openshift-ci-robot · 2026-06-30T15:40:08Z

@muraee: This pull request explicitly references no jira issue.

Details

In response to this:

Summary

The v2 health test unconditionally expects ConfigOperatorReconciliationSucceeded, which was introduced in PR CNTRLPLANE-597: Surface HCCO reconciliation failures in HostedCluster status #8340 for 4.23+

The v1 test already gates this with IsLessThan(Version423), but the v2 test was missing the same check

This causes e2e-v2-aws failures on all release-4.22 PRs since the condition doesn't exist on 4.22 clusters

Test plan

Verify e2e-v2-aws passes on release-4.22 PRs after this merges

Verify e2e-v2-aws still passes on main (4.23+ clusters should have the condition)

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

coderabbitai · 2026-06-30T15:41:50Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 863e8314-6113-43df-be74-fa98998fd252

📥 Commits

Reviewing files that changed from the base of the PR and between 4d087b3 and e7e1898.

📒 Files selected for processing (1)

test/e2e/v2/tests/hosted_cluster_health_test.go

📝 Walkthrough

Walkthrough

ValidateHostedClusterConditionsTest now removes hyperv1.ConfigOperatorReconciliationSucceeded from the expected conditions map when the e2e version is below Version423.

🚥 Pre-merge checks | ✅ 11

✅ Passed checks (11 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: gating the ConfigOperatorReconciliationSucceeded expectation for e2e-v2 on 4.23+.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	The PR only adds version-gated condition logic; the Ginkgo titles in the touched test remain static and deterministic.
Test Structure And Quality	✅ Passed	The change only gates one expected condition by version; the test already uses BeforeEach, bounded Eventually timeouts, and helpful assertion messages.
Topology-Aware Scheduling Compatibility	✅ Passed	Only an e2e health test changed; it gates a condition by version and adds no scheduling constraints or workload placement logic.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	The modified health test only gates an expected condition by version; it contains no IPv4 literals, IP parsing, host URL building, or public internet dependencies.
No-Weak-Crypto	✅ Passed	Changed test only adds a version gate for ConfigOperatorReconciliationSucceeded; no weak crypto, custom crypto, or secret/token comparisons appear in the modified file.
Container-Privileges	✅ Passed	The PR only changes a Go e2e test; no container/K8s manifest privilege flags (privileged, hostPID, hostNetwork, hostIPC, SYS_ADMIN, allowPrivilegeEscalation) were added.
No-Sensitive-Data-In-Logs	✅ Passed	The touched test code only adds a version gate and introduces no logging, prints, or sensitive-data strings.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

openshift-ci · 2026-06-30T15:42:12Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: muraee

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [muraee]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

codecov · 2026-06-30T15:49:31Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 43.19%. Comparing base (4d087b3) to head (e7e1898).
⚠️ Report is 17 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #8875   +/-   ##
=======================================
  Coverage   43.19%   43.19%           
=======================================
  Files         767      767           
  Lines       94910    94910           
=======================================
  Hits        40997    40997           
  Misses      51050    51050           
  Partials     2863     2863

Flag	Coverage Δ
cmd-support	`36.43% <ø> (ø)`
cpo-hostedcontrolplane	`45.31% <ø> (ø)`
cpo-other	`45.10% <ø> (ø)`
hypershift-operator	`53.53% <ø> (ø)`
other	`31.69% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The v2 health test unconditionally expects the ConfigOperatorReconciliationSucceeded condition, which was introduced in PR openshift#8340 for 4.23+. The v1 test already gates this with IsLessThan(Version423), but the v2 test was missing the same check, causing e2e-v2-aws failures on release-4.22 branches. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

csrwng · 2026-06-30T16:07:51Z

/lgtm

openshift-merge-bot · 2026-06-30T16:10:34Z

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks-4-22
/test e2e-aws-4-22
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-v2-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke

hypershift-jira-solve-ci · 2026-06-30T18:01:00Z

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-aks | Build: 2071990283515465728 | Cost: $3.2868255000000004 | Failed step: hypershift-azure-run-e2e

View full analysis report

_{Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6}

cwbotbot · 2026-06-30T18:20:03Z

Test Results

e2e-aks

Status: ❌ FAIL
Started: 2026-07-01T12:36:22Z
View Job
View Job History

e2e-aws

Status: ❌ FAIL
Started: 2026-07-01T12:36:21Z
View Job
View Job History

hypershift-jira-solve-ci · 2026-06-30T19:08:04Z

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-aws | Build: 2071990283549020160 | Cost: $2.74329625 | Failed step: hypershift-aws-run-e2e-nested

View full analysis report

_{Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6}

cblecker · 2026-06-30T19:09:11Z

/test e2e-v2-gke

muraee · 2026-07-01T10:01:45Z

/retest-required

muraee · 2026-07-01T10:46:41Z

/verified by e2e

openshift-ci-robot · 2026-07-01T10:46:53Z

@muraee: This PR has been marked as verified by e2e.

Details

In response to this:

/verified by e2e

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

hypershift-jira-solve-ci · 2026-07-01T12:02:52Z

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-aks | Build: 2072259377338781696 | Cost: $3.8785522500000003 | Failed step: hypershift-azure-run-e2e

View full analysis report

_{Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6}

openshift-merge-bot · 2026-07-01T12:23:13Z

/retest-required

Remaining retests: 0 against base HEAD ca3d347 and 2 for PR HEAD e7e1898 in total

muraee · 2026-07-01T13:39:40Z

/override e2e-aks
/override e2e-aws
/override e2e-aws-4-22

PR only changes v2 test, those tests are failing because of infra issues.

hypershift-jira-solve-ci · 2026-07-01T13:40:24Z

I have all the evidence needed. Both failures are identical CI infrastructure issues — not related to the PR code changes at all. Here's the final report:

Test Failure Analysis Complete

Job Information

Prow Job 1: pull-ci-openshift-hypershift-main-e2e-aks — Build ID: 2072298227402543104
Prow Job 2: pull-ci-openshift-hypershift-main-e2e-aws — Build ID: 2072298227335434240
Targets: e2e-aks, e2e-aws
CI Cluster: build01 — Namespace: ci-op-lbf03gvq (shared by both jobs)
PR: NO-JIRA: fix(e2e-v2): gate ConfigOperatorReconciliationSucceeded on 4.23+ #8875

Test Failure Analysis

Error

step [release:initial] failed: failed to get CLI image: pod "release-images-initial-cli"
pending for more than 1h0m0s — pod has not been scheduled.

Reason: executing_graph:step_failed:importing_release:pod_pending

Scheduling failure (82 events): 0/74-85 nodes are available: ~12 node(s) didn't match
Pod's node affinity/selector, ~65-70 node(s) had untolerated taint(s), 1 node(s) didn't
match pod anti-affinity rules, 1-6 node(s) were unschedulable. Preemption: not helpful.

Summary

Both e2e-aks and e2e-aws jobs failed due to a CI infrastructure scheduling issue on the build01 cluster — not due to any code change in PR #8875. The jobs shared the same CI namespace (ci-op-lbf03gvq) and both attempted to schedule a pod release-images-initial-cli to extract the CLI image from the initial release. This pod remained in Pending state for over 1 hour due to cluster-wide resource exhaustion (node taints, affinity mismatches, and unschedulable nodes), after which ci-operator timed out and reported failure. The test code itself was never executed.

Root Cause

The root cause is CI infrastructure resource pressure on the build01 cluster, not a product or test code bug.

Timeline:

Both jobs started at 2026-07-01T12:36:24-25Z and shared namespace ci-op-lbf03gvq
Source builds (src, hypershift-operator, hypershift, hypershift-tests, hypershift-cli) all completed successfully by 12:57:40Z
The release image was snapshotted at 12:59:12Z
ci-operator then attempted to import the initial release by scheduling pod release-images-initial-cli
The pod was gated briefly by the multiarch-tuning-operator (gate applied and removed at 12:36:26-27Z) but then entered a permanent scheduling failure loop
For the next 60 minutes, the Kubernetes scheduler repeatedly failed to place the pod across 74-85 available nodes due to:
- ~65-70 nodes with untolerated taints (reserved for other workloads)
- ~12 nodes that didn't match pod node affinity/selector
- 1 node blocked by pod anti-affinity rules
- 1-6 nodes marked unschedulable (cordoned/draining)
- Occasional Insufficient memory events
At 13:36:26Z, ci-operator's 1-hour timeout fired and both jobs were reported as failed with reason executing_graph:step_failed:importing_release:pod_pending

This is a transient CI infrastructure issue. The PR's code changes were never tested — the jobs failed during the release image import phase, before any test step could begin.

Recommendations

Retest the PR — Simply trigger a retest (/retest or /test e2e-aks and /test e2e-aws). This is a transient infrastructure issue that typically resolves when cluster load decreases.
No code changes needed — The PR code (gating ConfigOperatorReconciliationSucceeded on 4.23+) was never executed. The failure is entirely in the CI release-import infrastructure step.
If retests continue to fail with the same pod_pending reason, escalate to the CI infrastructure team (TRT / DPTP) as it indicates sustained resource pressure on the build01 cluster.

Evidence

Evidence	Detail
Failure reason	`executing_graph:step_failed:importing_release:pod_pending`
Pending pod	`release-images-initial-cli` — stuck in Pending for 1h0m0s
CI cluster	`build01` (namespace `ci-op-lbf03gvq`, shared by both jobs)
Scheduling events	82 events showing no schedulable node (taints, affinity mismatch, unschedulable nodes)
Dominant constraint	~65-70 of 74-85 nodes had untolerated taints
Memory pressure	Occasional `Insufficient memory` events observed
All builds succeeded	`src` (2m4s), `hypershift-operator` (5m56s), `hypershift` (3m41s), `hypershift-cli` (1m45s), `hypershift-tests` (12m45s) — all completed OK
Test execution	Never reached — failure occurred during release image import, before any test step
PR code relevance	None — the gating change in the PR was never evaluated

openshift-ci · 2026-07-01T13:43:38Z

@muraee: /override requires failed status contexts, check run or a prowjob name to operate on.
The following unknown contexts/checkruns were given:

e2e-aks
e2e-aws
e2e-aws-4-22

Only the following failed contexts/checkruns were expected:

CodeRabbit
ci/prow/e2e-aks
ci/prow/e2e-aks-4-22
ci/prow/e2e-aws
ci/prow/e2e-aws-4-22
ci/prow/e2e-aws-upgrade-hypershift-operator
ci/prow/e2e-azure-v2-self-managed
ci/prow/e2e-kubevirt-aws-ovn-reduced
ci/prow/e2e-v2-aws
ci/prow/e2e-v2-gke
ci/prow/images
ci/prow/okd-scos-images
ci/prow/security
ci/prow/verify-deps
envtest-kube / Envtest Vanilla Kube ${{ matrix.version }}
envtest-ocp / Envtest OCP (K8s ${{ matrix.version }})
pull-ci-openshift-hypershift-main-e2e-aks
pull-ci-openshift-hypershift-main-e2e-aks-4-22
pull-ci-openshift-hypershift-main-e2e-aws
pull-ci-openshift-hypershift-main-e2e-aws-4-22
pull-ci-openshift-hypershift-main-e2e-aws-upgrade-hypershift-operator
pull-ci-openshift-hypershift-main-e2e-azure-v2-self-managed
pull-ci-openshift-hypershift-main-e2e-kubevirt-aws-ovn-reduced
pull-ci-openshift-hypershift-main-e2e-v2-aws
pull-ci-openshift-hypershift-main-e2e-v2-gke
pull-ci-openshift-hypershift-main-images
pull-ci-openshift-hypershift-main-okd-scos-images
pull-ci-openshift-hypershift-main-security
pull-ci-openshift-hypershift-main-verify-deps
tide

If you are trying to override a checkrun that has a space in it, you must put a double quote on the context.

Details

In response to this:

/override e2e-aks
/override e2e-aws
/override e2e-aws-4-22

PR only changes v2 test, those tests are failing because of infra issues.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

muraee · 2026-07-01T13:54:00Z

/override "ci/prow/e2e-aks"
/override "ci/prow/e2e-aws"
/override "ci/prow/e2e-aws-4-22"

PR only changes v2 test, those tests are failing because of infra issues.

openshift-ci · 2026-07-01T13:56:18Z

@muraee: Overrode contexts on behalf of muraee: ci/prow/e2e-aks, ci/prow/e2e-aws, ci/prow/e2e-aws-4-22

Details

In response to this:

/override "ci/prow/e2e-aks"
/override "ci/prow/e2e-aws"
/override "ci/prow/e2e-aws-4-22"

PR only changes v2 test, those tests are failing because of infra issues.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci · 2026-07-01T14:35:51Z

@muraee: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jun 30, 2026

openshift-ci Bot added the do-not-merge/needs-area label Jun 30, 2026

openshift-ci Bot requested review from cblecker and csrwng June 30, 2026 15:42

openshift-ci Bot added the area/testing Indicates the PR includes changes for e2e testing label Jun 30, 2026

openshift-ci Bot added approved Indicates a PR has been approved by an approver from all required OWNERS files. and removed do-not-merge/needs-area labels Jun 30, 2026

muraee force-pushed the fix-v2-health-test-version-gate branch from 8165647 to e7e1898 Compare June 30, 2026 16:01

openshift-ci Bot assigned csrwng Jun 30, 2026

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jun 30, 2026

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jul 1, 2026

openshift-merge-bot Bot merged commit ce9dd2c into openshift:main Jul 1, 2026
41 checks passed

Uh oh!

Conversation

muraee commented Jun 30, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Summary by CodeRabbit

Uh oh!

openshift-merge-bot Bot commented Jun 30, 2026

Uh oh!

openshift-ci-robot commented Jun 30, 2026

Summary

Test plan

Uh oh!

coderabbitai Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Uh oh!

openshift-ci Bot commented Jun 30, 2026

Uh oh!

codecov Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

csrwng commented Jun 30, 2026

Uh oh!

openshift-merge-bot Bot commented Jun 30, 2026

Uh oh!

hypershift-jira-solve-ci Bot commented Jun 30, 2026

AI Test Failure Analysis

Uh oh!

cwbotbot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

e2e-aks

e2e-aws

Uh oh!

hypershift-jira-solve-ci Bot commented Jun 30, 2026

AI Test Failure Analysis

Uh oh!

cblecker commented Jun 30, 2026

Uh oh!

muraee commented Jul 1, 2026

Uh oh!

muraee commented Jul 1, 2026

Uh oh!

openshift-ci-robot commented Jul 1, 2026

Uh oh!

hypershift-jira-solve-ci Bot commented Jul 1, 2026

AI Test Failure Analysis

Uh oh!

openshift-merge-bot Bot commented Jul 1, 2026

Uh oh!

muraee commented Jul 1, 2026

Uh oh!

hypershift-jira-solve-ci Bot commented Jul 1, 2026

Test Failure Analysis Complete

Job Information

Test Failure Analysis

Error

Summary

Uh oh!

openshift-ci Bot commented Jul 1, 2026

Uh oh!

muraee commented Jul 1, 2026

Uh oh!

openshift-ci Bot commented Jul 1, 2026

Uh oh!

Uh oh!

openshift-ci Bot commented Jul 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

muraee commented Jun 30, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jun 30, 2026 •

edited

Loading

codecov Bot commented Jun 30, 2026 •

edited

Loading

cwbotbot commented Jun 30, 2026 •

edited

Loading