OCPBUGS-81544: requeue when AutoNodeEnabled is progressing

[maxcao13]

What this PR does / why we need it:

The HostedCluster reconciler reads ControlPlaneComponent status to set the
AutoNodeEnabled condition, but it does not watch CPC resources. When AutoNode
is enabled or disabled, the condition gets stuck at AutoNodeProgressing until
an unrelated resource change happens to trigger reconciliation. In CI this
causes the TestKarpenter/Main/AutoNode_enable/disable_lifecycle e2e test to
time out — the karpenter CPC reports RolloutComplete=True within ~1 minute
but the HC condition is never updated.

This adds a 15-second requeue when reconcileAutoNodeEnabledCondition reports
a progressing state, so the HC reconciler polls until the transition completes.
This is a targeted fix that avoids adding a broad CPC watch (which would fire
for all ~30-40 CPCs on every status change across every HostedCluster).

Which issue(s) this PR fixes:

Fixes periodic TestKarpenter/Main/AutoNode_enable/disable_lifecycle failures
in periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn.

Special notes for your reviewer:

The alternative considered was adding ControlPlaneComponent to the HC
reconciler's managedResources() watch list. That would provide immediate
reactivity but at the cost of watching all CPCs across all HCP namespaces
(~30-40 per HostedCluster), when only 2 karpenter CPCs matter during
enable/disable transitions. A bounded 15s poll during transitions is the
better tradeoff.

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Assisted-by: Cursor Agent

Made with Cursor

Summary by CodeRabbit

• Improvements

  • Enhanced AutoNode reconciliation with optimized requeue timing, ensuring the system polls for automatic node management progress more frequently (every 15 seconds maximum).

• Tests

  • Updated unit and end-to-end tests to validate AutoNode progress tracking and reconciliation behavior.

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[openshift-ci-robot]

@maxcao13: This pull request explicitly references no jira issue.

Details

In response to this:

What this PR does / why we need it:

The HostedCluster reconciler reads ControlPlaneComponent status to set the
AutoNodeEnabled condition, but it does not watch CPC resources. When AutoNode
is enabled or disabled, the condition gets stuck at AutoNodeProgressing until
an unrelated resource change happens to trigger reconciliation. In CI this
causes the TestKarpenter/Main/AutoNode_enable/disable_lifecycle e2e test to
time out — the karpenter CPC reports RolloutComplete=True within ~1 minute
but the HC condition is never updated.

This adds a 15-second requeue when reconcileAutoNodeEnabledCondition reports
a progressing state, so the HC reconciler polls until the transition completes.
This is a targeted fix that avoids adding a broad CPC watch (which would fire
for all ~30-40 CPCs on every status change across every HostedCluster).

Which issue(s) this PR fixes:

Fixes periodic TestKarpenter/Main/AutoNode_enable/disable_lifecycle failures
in periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn.

Special notes for your reviewer:

The alternative considered was adding ControlPlaneComponent to the HC
reconciler's managedResources() watch list. That would provide immediate
reactivity but at the cost of watching all CPCs across all HCP namespaces
(~30-40 per HostedCluster), when only 2 karpenter CPCs matter during
enable/disable transitions. A bounded 15s poll during transitions is the
better tradeoff.

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Assisted-by: Cursor Agent

Made with Cursor

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci-robot]

@maxcao13: This pull request references Jira Issue OCPBUGS-81544, which is invalid:

• expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

What this PR does / why we need it:

The HostedCluster reconciler reads ControlPlaneComponent status to set the
AutoNodeEnabled condition, but it does not watch CPC resources. When AutoNode
is enabled or disabled, the condition gets stuck at AutoNodeProgressing until
an unrelated resource change happens to trigger reconciliation. In CI this
causes the TestKarpenter/Main/AutoNode_enable/disable_lifecycle e2e test to
time out — the karpenter CPC reports RolloutComplete=True within ~1 minute
but the HC condition is never updated.

This adds a 15-second requeue when reconcileAutoNodeEnabledCondition reports
a progressing state, so the HC reconciler polls until the transition completes.
This is a targeted fix that avoids adding a broad CPC watch (which would fire
for all ~30-40 CPCs on every status change across every HostedCluster).

Which issue(s) this PR fixes:

Fixes periodic TestKarpenter/Main/AutoNode_enable/disable_lifecycle failures
in periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn.

Special notes for your reviewer:

The alternative considered was adding ControlPlaneComponent to the HC
reconciler's managedResources() watch list. That would provide immediate
reactivity but at the cost of watching all CPCs across all HCP namespaces
(~30-40 per HostedCluster), when only 2 karpenter CPCs matter during
enable/disable transitions. A bounded 15s poll during transitions is the
better tradeoff.

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Assisted-by: Cursor Agent

Made with Cursor

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 10823b46-9eba-488f-a4b3-2bb05cf0e107

📥 Commits

Reviewing files that changed from the base of the PR and between 6f44bab and 20be0ba.

📒 Files selected for processing (4)

• hypershift-operator/controllers/hostedcluster/hostedcluster_controller.go
• hypershift-operator/controllers/hostedcluster/karpenter.go
• hypershift-operator/controllers/hostedcluster/karpenter_test.go
• test/e2e/karpenter_test.go

🚧 Files skipped from review as they are similar to previous changes (4)

• hypershift-operator/controllers/hostedcluster/karpenter.go
• hypershift-operator/controllers/hostedcluster/hostedcluster_controller.go
• hypershift-operator/controllers/hostedcluster/karpenter_test.go
• test/e2e/karpenter_test.go

---

📝 Walkthrough

Walkthrough

This PR extends the AutoNode reconciliation to signal when Karpenter component provisioning is in progress. The reconcileAutoNodeEnabledCondition() function now returns a boolean alongside the condition, indicating whether the controller should poll for status updates. The controller captures this flag and adjusts requeue timing to 15 seconds when polling is active, ensuring timely status checks. Unit tests validate the new return value across enable/disable paths. E2E tests were refactored to use fresh hosted cluster state snapshots in multiple test scenarios and the pod-readiness helper was extended to filter by pod labels.

Possibly related PRs

• openshift/hypershift#8510: Concurrent update to the Karpenter E2E pod-readiness flow filtering by workload labels via waitForReadyKarpenterPods.
• openshift/hypershift#8498: Related E2E test changes that also refactor hosted cluster state handling to use deep-copied snapshots.

Suggested reviewers

• muraee
• clebs

🚥 Pre-merge checks | ✅ 10 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Single Node Openshift (Sno) Test Compatibility	⚠️ Warning	testBillingConsolidationAndPDB creates 2-replica workloads with pod anti-affinity requiring separate nodes. No SNO protection present (no IsSingleNode check or [Skipped:SingleReplicaTopology] label).	Add SNO protection to TestKarpenter: either [Skipped:SingleReplicaTopology] label, or guard with isSingleNode check and g.Skip(). Pod anti-affinity cannot be satisfied on single-node clusters.

✅ Passed checks (10 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title directly addresses the main change: it specifies that the PR requeues reconciliation when AutoNodeEnabled is progressing, which is precisely what the changes implement across the HostedCluster reconciler and Karpenter reconciliation logic.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	All test names are stable and deterministic. No dynamic information appears in test titles. All tests use static, descriptive strings.
Test Structure And Quality	✅ Passed	All requirements met: single responsibility (focused tests), proper setup/cleanup (t.Cleanup blocks), timeouts on all Eventually calls, meaningful assertion messages, follows repository patterns.
Microshift Test Compatibility	✅ Passed	No new Ginkgo e2e tests added. The PR modifies existing test functions in test/e2e/karpenter_test.go using standard Go t.Run() pattern, not Ginkgo framework. Custom check is not applicable.
Topology-Aware Scheduling Compatibility	✅ Passed	PR makes no changes to deployment manifests or scheduling constraints. Changes are limited to controller reconciliation logic and test infrastructure. No topology-aware scheduling issues introduced.
Ote Binary Stdout Contract	✅ Passed	No stdout contract violations detected. Code changes are within test functions; no module-level stdout writes, klog redirections, or process-level issues found.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	No new Ginkgo e2e tests (It(), Describe(), Context(), When()) are added. The PR modifies existing test behavior and controller logic, not new tests. Check requirement not applicable.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[openshift-ci-robot]

@maxcao13: This pull request references Jira Issue OCPBUGS-81544, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

What this PR does / why we need it:

The HostedCluster reconciler reads ControlPlaneComponent status to set the
AutoNodeEnabled condition, but it does not watch CPC resources. When AutoNode
is enabled or disabled, the condition gets stuck at AutoNodeProgressing until
an unrelated resource change happens to trigger reconciliation. In CI this
causes the TestKarpenter/Main/AutoNode_enable/disable_lifecycle e2e test to
time out — the karpenter CPC reports RolloutComplete=True within ~1 minute
but the HC condition is never updated.

This adds a 15-second requeue when reconcileAutoNodeEnabledCondition reports
a progressing state, so the HC reconciler polls until the transition completes.
This is a targeted fix that avoids adding a broad CPC watch (which would fire
for all ~30-40 CPCs on every status change across every HostedCluster).

Which issue(s) this PR fixes:

Fixes periodic TestKarpenter/Main/AutoNode_enable/disable_lifecycle failures
in periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn.

Special notes for your reviewer:

The alternative considered was adding ControlPlaneComponent to the HC
reconciler's managedResources() watch list. That would provide immediate
reactivity but at the cost of watching all CPCs across all HCP namespaces
(~30-40 per HostedCluster), when only 2 karpenter CPCs matter during
enable/disable transitions. A bounded 15s poll during transitions is the
better tradeoff.

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Assisted-by: Cursor Agent

Made with Cursor

Summary by CodeRabbit

• Bug Fixes
• Improved AutoNode component reconciliation polling to more intelligently detect when the system should requeue to monitor component rollout and teardown progress, ensuring better responsiveness during Karpenter component transitions.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

hypershift-operator/controllers/hostedcluster/karpenter.go (1)

183-187: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Requeue should stay enabled when AutoNode evaluation fails

At Line 187 and Line 245, transient client/list/get errors return progressing=false, which can leave AutoNodeEnabled stuck in Unknown until an unrelated event. Returning true here preserves polling and avoids stale status.

Suggested patch

-       return condition, false
+       return condition, true
...
-           return condition, false
+           return condition, true

Also applies to: 241-245

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hypershift-operator/controllers/hostedcluster/karpenter.go` around lines 183
- 187, The client.List and client.Get error paths in the AutoNode evaluation
currently return (condition, false) which disables requeue; update the error
returns in the r.Client.List(...) and r.Client.Get(...) error branches (the
blocks that set condition.Status = metav1.ConditionUnknown, condition.Reason =
hyperv1.AutoNodeEvaluationFailedReason and set condition.Message with the
fmt.Sprintf of the error) to return (condition, true) so retry/polling stays
enabled and AutoNodeEnabled doesn't remain stale.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@hypershift-operator/controllers/hostedcluster/hostedcluster_controller.go`:
- Around line 2106-2108: When autoNodeProgressing is true, don't unconditionally
overwrite result.RequeueAfter; instead set it to the minimum non-zero interval
between the existing result.RequeueAfter and 15s. Update the autoNodeProgressing
branch to: if result.RequeueAfter is zero, assign 15s; otherwise assign 15s only
if 15s is less than the current result.RequeueAfter (i.e., result.RequeueAfter =
minNonZero(result.RequeueAfter, 15*time.Second)). This preserves shorter
previously-computed intervals while ensuring a 15s fallback when none was set.

---

Outside diff comments:
In `@hypershift-operator/controllers/hostedcluster/karpenter.go`:
- Around line 183-187: The client.List and client.Get error paths in the
AutoNode evaluation currently return (condition, false) which disables requeue;
update the error returns in the r.Client.List(...) and r.Client.Get(...) error
branches (the blocks that set condition.Status = metav1.ConditionUnknown,
condition.Reason = hyperv1.AutoNodeEvaluationFailedReason and set
condition.Message with the fmt.Sprintf of the error) to return (condition, true)
so retry/polling stays enabled and AutoNodeEnabled doesn't remain stale.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 6d9b0b24-a636-484f-8709-0f8ab232c9e2

📥 Commits

Reviewing files that changed from the base of the PR and between 4341d0c and 83f477e.

📒 Files selected for processing (3)

• hypershift-operator/controllers/hostedcluster/hostedcluster_controller.go
• hypershift-operator/controllers/hostedcluster/karpenter.go
• hypershift-operator/controllers/hostedcluster/karpenter_test.go

[codecov]

Codecov Report

❌ Patch coverage is 53.33333% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 40.07%. Comparing base (10fd799) to head (20be0ba).
⚠️ Report is 147 commits behind head on main.

Files with missing lines	Patch %	Lines
...trollers/hostedcluster/hostedcluster_controller.go	28.57%	4 Missing and 1 partial ⚠️
...ft-operator/controllers/hostedcluster/karpenter.go	75.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8497      +/-   ##
==========================================
- Coverage   40.07%   40.07%   -0.01%     
==========================================
  Files         751      751              
  Lines       92863    92868       +5     
==========================================
  Hits        37215    37215              
- Misses      52956    52960       +4     
- Partials     2692     2693       +1     

Files with missing lines	Coverage Δ
...ft-operator/controllers/hostedcluster/karpenter.go	75.16% <75.00%> (ø)
...trollers/hostedcluster/hostedcluster_controller.go	43.59% <28.57%> (-0.06%)	⬇️

Flag	Coverage Δ
cmd-support	34.31% <ø> (ø)
cpo-hostedcontrolplane	40.56% <ø> (ø)
cpo-other	40.14% <ø> (ø)
hypershift-operator	50.51% <53.33%> (-0.02%)	⬇️
other	31.54% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[openshift-ci-robot]

@maxcao13: This pull request references Jira Issue OCPBUGS-81544, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

What this PR does / why we need it:

The HostedCluster reconciler reads ControlPlaneComponent status to set the
AutoNodeEnabled condition, but it does not watch CPC resources. When AutoNode
is enabled or disabled, the condition gets stuck at AutoNodeProgressing until
an unrelated resource change happens to trigger reconciliation. In CI this
causes the TestKarpenter/Main/AutoNode_enable/disable_lifecycle e2e test to
time out — the karpenter CPC reports RolloutComplete=True within ~1 minute
but the HC condition is never updated.

This adds a 15-second requeue when reconcileAutoNodeEnabledCondition reports
a progressing state, so the HC reconciler polls until the transition completes.
This is a targeted fix that avoids adding a broad CPC watch (which would fire
for all ~30-40 CPCs on every status change across every HostedCluster).

Which issue(s) this PR fixes:

Fixes periodic TestKarpenter/Main/AutoNode_enable/disable_lifecycle failures
in periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn.

Special notes for your reviewer:

The alternative considered was adding ControlPlaneComponent to the HC
reconciler's managedResources() watch list. That would provide immediate
reactivity but at the cost of watching all CPCs across all HCP namespaces
(~30-40 per HostedCluster), when only 2 karpenter CPCs matter during
enable/disable transitions. A bounded 15s poll during transitions is the
better tradeoff.

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Assisted-by: Cursor Agent

Made with Cursor

Summary by CodeRabbit

• Bug Fixes
• Improved AutoNode reconciliation: the system now more reliably detects when Karpenter-related components are still rolling out or tearing down and will poll more appropriately, including ensuring progress is rechecked at least every 15 seconds when rollout/teardown is in flight, improving responsiveness during component transitions.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[maxcao13]

/test e2e-aws-autonode
/test e2e-aws

[maxcao13]

Added a commit to fix some data racing because we ran into concurrent map writes related to the shared hostedcluster pointer object.

/test e2e-aws-autonode
/test e2e-aws

[hypershift-jira-solve-ci]

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-aws | Build: 2054345191254396928 | Cost: $3.2408330000000003 | Failed step: hypershift-aws-run-e2e-nested

View full analysis report

---

_{Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6}

[cwbotbot]

Test Results

e2e-aws

• Status: ✅ PASS
• Started: 2026-05-13T05:06:25Z
• View Job
• View Job History

[maxcao13]

/test e2e-aws

[hypershift-jira-solve-ci]

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-aws | Build: 2054387343774715904 | Cost: $2.9509135 | Failed step: hypershift-aws-run-e2e-nested

View full analysis report

---

_{Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6}

[maxcao13]

cc @enxebre @jkyros

[enxebre]

/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: enxebre, maxcao13

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [enxebre]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[joshbranham]

/lgtm

[openshift-merge-bot]

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks-4-22
/test e2e-aws-4-22
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke

[maxcao13]

/retest-required

[hypershift-jira-solve-ci]

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-aws | Build: 2060041550489980928 | Cost: $2.457118 | Failed step: hypershift-aws-run-e2e-nested

View full analysis report

---

_{Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6}

[maxcao13]

/retest-required

[maxcao13]

/test e2e-aws

[maxcao13]

/test e2e-aws-4-22

[maxcao13]

/test e2e-aws-4-22

[maxcao13]

/verified by unit-tests,e2e-aws

[openshift-ci-robot]

@maxcao13: This PR has been marked as verified by unit-tests,e2e-aws.

Details

In response to this:

/verified by unit-tests,e2e-aws

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-merge-bot]

/retest-required

Remaining retests: 0 against base HEAD 9b67f7b and 2 for PR HEAD 20be0ba in total

[hypershift-jira-solve-ci]

I have enough evidence. The hypershift-tests Docker image build failed with DockerBuildFailed during the Go compilation of the e2e test suite. The build log was never captured (completely empty — suggesting the build process was killed abruptly, consistent with an OOM kill). The build had a 4Gi memory limit but was compiling 8+ large Go binaries. No build log file was uploaded to GCS (unlike the 3 other successful image builds). The code compiles cleanly (verified), the PR's code changes are correct, and the other related image builds with simpler compilation targets all succeeded.

Test Failure Analysis Complete

Job Information

• Prow Job: pull-ci-openshift-hypershift-main-images
• Build ID: 2060269457309175808
• Target: images
• PR: #8497 — OCPBUGS-81544: requeue when AutoNodeEnabled is progressing
• State: failure

Test Failure Analysis

Error

the build hypershift-tests-amd64 failed after 4m52s with reason DockerBuildFailed:
Dockerfile build strategy has failed.

Summary

The hypershift-tests Docker image build failed during Go compilation of the e2e test suite (make e2e hypershift e2ev2-create-guests e2ev2-run-tests e2ev2-destroy-guests e2ev2-dump-guests). The build ran for 4 minutes 52 seconds before dying with DockerBuildFailed, and critically, no build log was captured — the log file was never uploaded to GCS artifacts (unlike the 3 other successful image builds). This empty-log pattern is characteristic of an OOM kill during Docker-in-Docker builds: the Go compiler is killed mid-process by the kernel OOM killer, the build container exits non-zero, and the build log is truncated/lost. The PR's code changes are verified correct — the code compiles cleanly, and the three other images (hypershift, hypershift-operator, hypershift-cli) that compile smaller subsets of the same codebase all built successfully in the same CI run.

Root Cause

The hypershift-tests-amd64 build pod was allocated a 4Gi memory limit with only 200Mi memory request to compile the largest and most memory-intensive image in the hypershift CI pipeline. The Dockerfile.e2e runs make e2e hypershift e2ev2-create-guests e2ev2-run-tests e2ev2-destroy-guests e2ev2-dump-guests, which compiles 8+ separate Go binaries sequentially:

1. test-reqserving (reqserving-e2e, CGO_ENABLED=1)
2. test-e2e-v2 (e2ev2)
3. test-backuprestore (backuprestore-e2e)
4. test-e2e (main e2e test suite — the largest binary)
5. test-setup
6. gotestsum (tools build)
7. hypershift (the CLI binary)
8. create-guests, run-tests, destroy-guests, dump-guests (e2ev2 helpers)

The Go compiler is known to use significant memory (often 2-4GB for large packages), especially when compiling test binaries that pull in extensive dependency trees. With a 4Gi limit, the build pod likely exceeded this limit during compilation of one of the larger binaries (most likely test-e2e which imports the full karpenter test suite including all AWS SDK dependencies).

Key evidence that this is NOT a code error:

• The PR's code compiles cleanly — all callers of the modified reconcileAutoNodeEnabledCondition function correctly handle the new (metav1.Condition, bool) return signature
• The e2e test changes (DeepCopy() refactor) are type-correct and compile without errors
• The hypershift-operator-amd64 build (which compiles the controller code including the modified karpenter.go) succeeded in the same CI run
• The hypershift-amd64 and hypershift-cli-amd64 builds also succeeded
• The only build that failed is the one with the most intensive compilation workload
• The build log file for hypershift-tests-amd64 was never uploaded to GCS, while all 3 successful builds have their logs — empty log capture is the hallmark of an abrupt process kill (OOM)
• The ci-operator logged "Build hypershift-tests-amd64 failed, printing logs:" followed immediately by no output and "classified as legitimate failure" — confirming no log data was available

Infrastructure context:

• The build pod spent ~4 minutes in FailedScheduling due to cluster resource contention ("Insufficient cpu" on eligible nodes) before being scheduled
• The node that eventually ran the build (ip-10-0-153-34.ec2.internal) was scaled up on-demand by the cluster autoscaler specifically for this pod
• The build pod requested 6 CPU cores but only 200Mi memory, creating a CPU-rich but memory-constrained environment for parallel Go compilation

Recommendations

1. Retry the job — Run /retest or /retest-required. This is a transient infrastructure failure unrelated to the PR's code changes. The build should succeed on retry if memory pressure is lower.
2. No code changes needed — The PR's modifications to reconcileAutoNodeEnabledCondition (adding the bool return value) and the DeepCopy() refactor in e2e tests are correct and compile cleanly.
3. Consider filing a CI infra issue — The hypershift-tests image build has a 4Gi memory limit which may be insufficient for the growing e2e test suite. The memory request (200Mi) vs limit (4Gi) ratio is extremely wide, making the build susceptible to OOM kills on memory-constrained nodes. A possible fix would be increasing the memory limit in the CI configuration or the Dockerfile.e2e build resources.

Evidence

Evidence	Detail
Failed step	hypershift-tests image build — DockerBuildFailed after 4m52s
Build log captured	No — hypershift-tests-amd64.log missing from GCS build-logs/ (other 3 builds have logs)
ci-operator output	"printing logs:" followed by zero log lines then "classified as legitimate failure"
Memory limit	4Gi (pod spec: resources.limits.memory: 4Gi)
Memory request	200Mi (pod spec: resources.requests.memory: 200Mi)
CPU request	6 cores
Compilation target	make e2e hypershift e2ev2-create-guests e2ev2-run-tests e2ev2-destroy-guests e2ev2-dump-guests (8+ Go binaries)
Successful builds (same run)	hypershift-amd64 (306s), hypershift-operator-amd64 (320s), hypershift-cli-amd64 (77s)
Node	ip-10-0-153-34.ec2.internal (autoscaled on-demand)
Scheduling delay	~4 minutes of FailedScheduling due to Insufficient cpu
Pod final status	pod_phase: "Failed" (from metrics event)
PR files changed	karpenter.go, hostedcluster_controller.go, karpenter_test.go, test/e2e/karpenter_test.go
Code compiles	✅ Verified — make e2e completes successfully on this commit

---

[openshift-merge-bot]

/retest-required

Remaining retests: 0 against base HEAD 8b13140 and 1 for PR HEAD 20be0ba in total

[openshift-ci]

@maxcao13: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci-robot]

@maxcao13: Jira Issue Verification Checks: Jira Issue OCPBUGS-81544
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-81544 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

What this PR does / why we need it:

The HostedCluster reconciler reads ControlPlaneComponent status to set the
AutoNodeEnabled condition, but it does not watch CPC resources. When AutoNode
is enabled or disabled, the condition gets stuck at AutoNodeProgressing until
an unrelated resource change happens to trigger reconciliation. In CI this
causes the TestKarpenter/Main/AutoNode_enable/disable_lifecycle e2e test to
time out — the karpenter CPC reports RolloutComplete=True within ~1 minute
but the HC condition is never updated.

This adds a 15-second requeue when reconcileAutoNodeEnabledCondition reports
a progressing state, so the HC reconciler polls until the transition completes.
This is a targeted fix that avoids adding a broad CPC watch (which would fire
for all ~30-40 CPCs on every status change across every HostedCluster).

Which issue(s) this PR fixes:

Fixes periodic TestKarpenter/Main/AutoNode_enable/disable_lifecycle failures
in periodic-ci-openshift-hypershift-release-5.0-periodics-e2e-aws-ovn.

Special notes for your reviewer:

The alternative considered was adding ControlPlaneComponent to the HC
reconciler's managedResources() watch list. That would provide immediate
reactivity but at the cost of watching all CPCs across all HCP namespaces
(~30-40 per HostedCluster), when only 2 karpenter CPCs matter during
enable/disable transitions. A bounded 15s poll during transitions is the
better tradeoff.

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Assisted-by: Cursor Agent

Made with Cursor

Summary by CodeRabbit

• Improvements

• Enhanced AutoNode reconciliation with optimized requeue timing, ensuring the system polls for automatic node management progress more frequently (every 15 seconds maximum).

• Tests

• Updated unit and end-to-end tests to validate AutoNode progress tracking and reconciliation behavior.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-merge-robot]

Fix included in release 5.0.0-0.nightly-2026-05-30-015000