OCPBUGS-98213: fix router component ordering to prevent missing HAProxy backends

[hypershift-jira-solve-ci]

OCPBUGS-98213: fix router component ordering to prevent missing HAProxy backends

What this PR does / why we need it:

The router's adaptConfig function lists existing Route objects to generate the HAProxy ConfigMap. When the router was registered early in the component list (position #8), components that create Routes — ignition-server (#35) and metrics-proxy (#38) — had not yet run during the first reconcile pass. This caused the initial HAProxy config to miss ignition and metrics-proxy backends, routing ignition requests to the KAS default backend and failing NodePool ignition.

This PR:

• Moves the router component to after metricsproxyv2 (the last route-creating component) so that all Routes exist when the router config is generated.
• Declares ignition-server and metrics-proxy as explicit WithDependencies on the router component, blocking its reconciliation until both dependencies are Available and RolloutComplete. This provides a stronger guarantee than positional ordering alone, though the informer cache update from watch events is still async.
• Adds TestRouterComponentComesAfterRouteCreatingComponents to verify the router is always registered after ignition-server and metrics-proxy, preventing future regressions from component reordering.

Which issue(s) this PR fixes:

Fixes https://redhat.atlassian.net/browse/OCPBUGS-98213

Special notes for your reviewer:

The fix uses two complementary mechanisms: positional ordering in registerComponents (belt) and WithDependencies via the component framework (suspenders). WithDependencies ensures the router's reconciliation is skipped until ignition-server and metrics-proxy report Available + RolloutComplete, giving the informer cache time to reflect their routes.

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

---

Always review AI generated responses prior to use.
Generated with Claude Code via openshift-developer plugin

---

Note: This PR was auto-generated by the jira-agent periodic CI job in response to OCPBUGS-98213. See the full report for token usage, cost breakdown, and detailed phase output.

Summary by CodeRabbit

• Bug Fixes
  • Ensured the router component initializes after components that create prerequisite Routes.
  • Made router reconciliation explicitly honor required dependencies; on ARO hosted control planes, it now waits for required Routes to exist and be ready before proceeding.
• Tests
  • Added and expanded unit tests to verify router component registration order, ARO Route requirements, readiness checks, and router predicate behavior.

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[openshift-ci-robot]

@hypershift-jira-solve-ci[bot]: This pull request references Jira Issue OCPBUGS-98213, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

What this PR does / why we need it:

The router's adaptConfig function lists existing Route objects to generate the HAProxy ConfigMap. When the router was registered early in the component list (position #8), components that create Routes — ignition-server (#35) and metrics-proxy (#38) — had not yet run during the first reconcile pass. This caused the initial HAProxy config to miss ignition and metrics-proxy backends, routing ignition requests to the KAS default backend and failing NodePool ignition.

This PR:

• Moves the router component to after metricsproxyv2 (the last route-creating component) so that all Routes exist when the router config is generated, eliminating the race window.
• Adds TestRouterComponentComesAfterRouteCreatingComponents to verify the router is always registered after ignition-server and metrics-proxy, preventing future regressions from component reordering.

Which issue(s) this PR fixes:

Fixes https://redhat.atlassian.net/browse/OCPBUGS-98213

Special notes for your reviewer:

The fix is a component ordering change only — no behavioral changes to the router or route-creating components themselves. The test locks down the ordering constraint so it cannot silently regress.

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

---

Always review AI generated responses prior to use.
Generated with Claude Code via openshift-developer plugin

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@hypershift-jira-solve-ci[bot]: This pull request references Jira Issue OCPBUGS-98213, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

What this PR does / why we need it:

The router's adaptConfig function lists existing Route objects to generate the HAProxy ConfigMap. When the router was registered early in the component list (position #8), components that create Routes — ignition-server (#35) and metrics-proxy (#38) — had not yet run during the first reconcile pass. This caused the initial HAProxy config to miss ignition and metrics-proxy backends, routing ignition requests to the KAS default backend and failing NodePool ignition.

This PR:

• Moves the router component to after metricsproxyv2 (the last route-creating component) so that all Routes exist when the router config is generated, eliminating the race window.
• Adds TestRouterComponentComesAfterRouteCreatingComponents to verify the router is always registered after ignition-server and metrics-proxy, preventing future regressions from component reordering.

Which issue(s) this PR fixes:

Fixes https://redhat.atlassian.net/browse/OCPBUGS-98213

Special notes for your reviewer:

The fix is a component ordering change only — no behavioral changes to the router or route-creating components themselves. The test locks down the ordering constraint so it cannot silently regress.

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

---

Always review AI generated responses prior to use.
Generated with Claude Code via openshift-developer plugin

---

Note: This PR was auto-generated by the jira-agent periodic CI job in response to OCPBUGS-98213. See the full report for token usage, cost breakdown, and detailed phase output.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

routerv2 now declares dependencies on ignitionserverv2 and metricsproxyv2, and its predicate checks that the required Routes exist and are ready in ARO HCP mode. HostedControlPlaneReconciler.registerComponents now adds routerv2 later in the component list. A new test asserts that router registration occurs after the ignition server and metrics proxy components.

Sequence Diagram(s)

sequenceDiagram
  participant Reconciler as HostedControlPlaneReconciler
  participant Router as routerv2 component
  participant API as Kubernetes Route API
  participant HCP as HostedControlPlane
  Reconciler->>Router: register component with dependencies
  Router->>HCP: inspect HCP router mode and platform
  Router->>API: list Routes in HCP namespace
  API-->>Router: return Route objects
  Router->>Router: check expected names and spec.host readiness
  Router-->>Reconciler: permit or block reconciliation

Loading

🚥 Pre-merge checks | ✅ 10 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Test Structure And Quality	⚠️ Warning	Two fake-client router tests discard routev1.Install errors, so setup can proceed with an invalid scheme and yield misleading results.	Handle scheme registration failures in both tests with t.Fatalf if routev1.Install(scheme) returns an error.

✅ Passed checks (10 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: reordering the router component to prevent missing HAProxy backends.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	Added test and subtest names are static, descriptive strings; no timestamps, random IDs, node/namespace names, or other run-to-run variable data appear in the new titles.
Topology-Aware Scheduling Compatibility	✅ Passed	No topology-sensitive scheduling changes were introduced; the patch only reorders router registration and adds route/dependency tests, with no affinity, nodeSelector, spread, or replica logic.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	No new Ginkgo e2e tests were added; the new tests are plain Go unit tests using fake clients and no IPv4 hardcodes or external connectivity.
No-Weak-Crypto	✅ Passed	PR only reorders router components and adds route-readiness tests; no weak crypto, custom crypto, or non-constant-time secret comparisons were added.
Container-Privileges	✅ Passed	The PR only reorders router registration and updates tests/fixture statuses; no touched manifest adds privileged, hostPID/Network/IPC, SYS_ADMIN, or allowPrivilegeEscalation.
No-Sensitive-Data-In-Logs	✅ Passed	No new runtime logging was added; the PR only reorders components and adds tests, and the new errors/messages expose only route names, not secrets or PII.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go (1)

279-285: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Consider WithDependencies instead of positional ordering.

Comment-guarded slice ordering is fragile: any future insertion/reordering in registerComponents, or a move away from strictly sequential reconciliation of r.components, silently reintroduces this bug. This framework already has an explicit ordering primitive for exactly this case — WithDependencies(...), which blocks a component's reconciliation until its declared dependencies are ready — used elsewhere (e.g. clusterpolicy, dnsoperator, konnectivity_agent all call WithDependencies(oapiv2.ComponentName)/kasv2.ComponentName). Declaring routerv2.NewComponent().WithDependencies(ignitionserverv2.ComponentName, metricsproxyv2.ComponentName) (if the framework's builder API supports appending dependencies) would make the ordering contract explicit and self-enforcing rather than implicit in registration order.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go`
around lines 279 - 285, The router registration in registerComponents is relying
on fragile slice order to wait for ignition-server and metrics-proxy, which can
be broken by future reordering. Update routerv2.NewComponent to declare an
explicit dependency on the components that create Route objects using the
framework’s WithDependencies API, so router reconciliation is blocked until
those dependencies are ready. Use the existing component names in this
controller (for example ignitionserverv2.ComponentName and
metricsproxyv2.ComponentName) to make the ordering contract explicit and
self-enforcing.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In
`@control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go`:
- Around line 279-285: The router registration in registerComponents is relying
on fragile slice order to wait for ignition-server and metrics-proxy, which can
be broken by future reordering. Update routerv2.NewComponent to declare an
explicit dependency on the components that create Route objects using the
framework’s WithDependencies API, so router reconciliation is blocked until
those dependencies are ready. Use the existing component names in this
controller (for example ignitionserverv2.ComponentName and
metricsproxyv2.ComponentName) to make the ordering contract explicit and
self-enforcing.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 8b89eabd-9f22-4e18-a641-6d02aef5df25

📥 Commits

Reviewing files that changed from the base of the PR and between 433a404 and b092cf4.

📒 Files selected for processing (2)

• control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go
• control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller_test.go

[codecov]

Codecov Report

❌ Patch coverage is 85.45455% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 43.81%. Comparing base (1f2811f) to head (d565a6e).
⚠️ Report is 63 commits behind head on main.

Files with missing lines	Patch %	Lines
...trollers/hostedcontrolplane/v2/router/component.go	85.18%	6 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8971      +/-   ##
==========================================
+ Coverage   43.53%   43.81%   +0.28%     
==========================================
  Files         771      772       +1     
  Lines       95798    96106     +308     
==========================================
+ Hits        41707    42113     +406     
+ Misses      51192    51076     -116     
- Partials     2899     2917      +18     

Files with missing lines	Coverage Δ
...ostedcontrolplane/hostedcontrolplane_controller.go	45.73% <100.00%> (ø)
support/controlplane-component/status.go	72.28% <ø> (ø)
...trollers/hostedcontrolplane/v2/router/component.go	63.88% <85.18%> (+63.88%)	⬆️

... and 24 files with indirect coverage changes

Flag	Coverage Δ
cmd-support	37.43% <ø> (+0.20%)	⬆️
cpo-hostedcontrolplane	46.02% <85.45%> (+0.50%)	⬆️
cpo-other	45.11% <ø> (+0.01%)	⬆️
hypershift-operator	54.07% <ø> (+0.41%)	⬆️
other	32.12% <ø> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bryan-cox]

Review

How Reconciliation Actually Works

1. reconcileInfrastructure() runs first (line 1084) — creates KAS routes, konnectivity routes, and OAuth routes via infra/infra.go
2. AdmitHCPManagedRoutes() runs next (line 1201) — sets host/status on HCP-managed routes
3. for _, c := range r.components (line 1263) — iterates sequentially, calling each component's Reconcile(), which calls update(), which applies manifests via server-side apply

Does component order matter?

Partially. The router's adaptConfig (v2/router/config.go:43) does:

cpContext.Client.List(cpContext, routeList, client.InNamespace(cpContext.HCP.Namespace))

cpContext.Client is r.Client (line 1247) — the controller-runtime cached client backed by informer caches. When ignition-server's Reconcile() creates a Route via server-side apply, the Route hits etcd, but the informer cache update from the watch event is async. So ordering the router after route-creating components gives the informer more time to process the watch event, making it more likely the routes are visible — but it's not a hard guarantee.

What routes does the router care about?

The router's adaptConfig handles: KAS (3 routes), ignition, konnectivity, OAuth (3 routes), metrics_forwarder, and metrics_proxy.

• KAS, konnectivity, OAuth routes are created in reconcileInfrastructure(), which runs before the component loop. These are not affected by component ordering at all.
• Only ignition-server and metrics-proxy create routes via the component framework (v2/assets/ignition-server/route.yaml and v2/assets/metrics-proxy/route.yaml). These are the ones affected by this PR.

The route-creating component list is complete

The PR correctly identifies ignition-server and metrics-proxy as the only two route-creating components in the framework. Confirmed by:

• Only two route.yaml assets exist under v2/assets/
• No other components reference routev1.Route outside of router itself

Test quality

The test is well-structured:

• Uses exported ComponentName constants
• Properly sets up the reconciler with required mocks
• Verifies position ordering in the actual component slice
• Only tests AWS platform, but the route-creating components aren't platform-conditional so this is fine

Issues

1. The PR description overstates the fix. It says this "eliminates the race window" — it doesn't. It reduces the window by giving the informer cache more time to sync, but the cache update is still async. A fully robust fix would use WithDependencies() on the router component (the framework supports this — see metricsproxyv2 which already uses WithDependencies(endpointresolverv2.ComponentName)), or use an uncached read in adaptConfig.

2. The 6-line inline comment is verbose. 2 lines would suffice:

   // Router must come after route-creating components (ignition-server, metrics-proxy)
   // so their routes exist in the cache when adaptConfig generates the HAProxy config.

3. Not a blocker, but worth noting: the router already handles missing routes gracefully — if a route doesn't exist, it just doesn't get a backend entry. The next reconcile pass will pick it up. So this is a first-reconcile latency optimization, not a hard correctness bug.

Verdict

Conditional ACK — the ordering change is a reasonable improvement and the test is good. But the PR description should be corrected to not claim it "eliminates" the race, and the comment should be shortened.

[bryan-cox]

/test address-review-comments

[hypershift-jira-solve-ci]

Review agent triggered. View job

[hypershift-jira-solve-ci]

Done. Addressed both items:

1. Comment shortened to 2 lines as suggested.
2. Commit message corrected — "eliminates the race window" → "reduces the race window" with a note that the informer cache update is still async.

Good point on #3 — the router does handle missing routes gracefully via subsequent reconcile passes, so this is indeed a first-reconcile latency optimization rather than a hard correctness fix.

---

AI-assisted response via Claude Code

[openshift-ci-robot]

@hypershift-jira-solve-ci[bot]: This pull request references Jira Issue OCPBUGS-98213, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

What this PR does / why we need it:

The router's adaptConfig function lists existing Route objects to generate the HAProxy ConfigMap. When the router was registered early in the component list (position #8), components that create Routes — ignition-server (#35) and metrics-proxy (#38) — had not yet run during the first reconcile pass. This caused the initial HAProxy config to miss ignition and metrics-proxy backends, routing ignition requests to the KAS default backend and failing NodePool ignition.

This PR:

• Moves the router component to after metricsproxyv2 (the last route-creating component) so that all Routes exist when the router config is generated, eliminating the race window.
• Adds TestRouterComponentComesAfterRouteCreatingComponents to verify the router is always registered after ignition-server and metrics-proxy, preventing future regressions from component reordering.

Which issue(s) this PR fixes:

Fixes https://redhat.atlassian.net/browse/OCPBUGS-98213

Special notes for your reviewer:

The fix is a component ordering change only — no behavioral changes to the router or route-creating components themselves. The test locks down the ordering constraint so it cannot silently regress.

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

---

Always review AI generated responses prior to use.
Generated with Claude Code via openshift-developer plugin

---

Note: This PR was auto-generated by the jira-agent periodic CI job in response to OCPBUGS-98213. See the full report for token usage, cost breakdown, and detailed phase output.

Summary by CodeRabbit

• Bug Fixes
• Improved startup/registration sequencing so routing configuration happens after related route components are initialized, reducing the risk of initial requests being routed to the wrong backend.
• Tests
• Added a unit test to confirm the routing component is registered after key components that create routes, preventing regressions in routing order.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[bryan-cox]

Was https://redhat.atlassian.net/browse/OCPBUGS-98213?focusedCommentId=17546910 taken into account when generating this PR?

[bryan-cox]

/test address-review-comments

[hypershift-jira-solve-ci]

Review agent triggered. View job

[openshift-ci-robot]

@hypershift-jira-solve-ci[bot]: This pull request references Jira Issue OCPBUGS-98213, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

OCPBUGS-98213: fix router component ordering to prevent missing HAProxy backends

What this PR does / why we need it:

The router's adaptConfig function lists existing Route objects to generate the HAProxy ConfigMap. When the router was registered early in the component list (position #8), components that create Routes — ignition-server (#35) and metrics-proxy (#38) — had not yet run during the first reconcile pass. This caused the initial HAProxy config to miss ignition and metrics-proxy backends, routing ignition requests to the KAS default backend and failing NodePool ignition.

This PR:

• Moves the router component to after metricsproxyv2 (the last route-creating component) so that all Routes exist when the router config is generated.
• Declares ignition-server and metrics-proxy as explicit WithDependencies on the router component, blocking its reconciliation until both dependencies are Available and RolloutComplete. This provides a stronger guarantee than positional ordering alone, though the informer cache update from watch events is still async.
• Adds TestRouterComponentComesAfterRouteCreatingComponents to verify the router is always registered after ignition-server and metrics-proxy, preventing future regressions from component reordering.

Which issue(s) this PR fixes:

Fixes https://redhat.atlassian.net/browse/OCPBUGS-98213

Special notes for your reviewer:

The fix uses two complementary mechanisms: positional ordering in registerComponents (belt) and WithDependencies via the component framework (suspenders). WithDependencies ensures the router's reconciliation is skipped until ignition-server and metrics-proxy report Available + RolloutComplete, giving the informer cache time to reflect their routes.

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

---

Always review AI generated responses prior to use.
Generated with Claude Code via openshift-developer plugin

---

Note: This PR was auto-generated by the jira-agent periodic CI job in response to OCPBUGS-98213. See the full report for token usage, cost breakdown, and detailed phase output.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[hypershift-jira-solve-ci]

Addressed review feedback:

1. WithDependencies instead of positional ordering — Added WithDependencies(ignitionserverv2.ComponentName, metricsproxyv2.ComponentName) to the router component in router/component.go. This uses the framework's dependency mechanism to block the router's reconciliation until both route-creating components are Available and RolloutComplete, providing a stronger guarantee than positional ordering alone. The comment-guarded ordering has been removed.

2. PR description updated — Removed the "eliminating the race window" claim. The description now accurately states that WithDependencies provides a stronger guarantee but the informer cache update is still async.

3. Test fixtures updated — The router's ControlPlaneComponent status now shows WaitingForDependencies: ignition-server, metrics-proxy instead of WaitingForRolloutComplete since the dependency check runs before reconciliation.

The positional ordering test is preserved as a secondary safety net — WithDependencies is the primary mechanism.

---

AI-assisted response via Claude Code

[bryan-cox]

/restructure-commits

[github-actions]

🤖 Restructuring commits: workflow run

[bryan-cox]

/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bryan-cox, hypershift-jira-solve-ci[bot]

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [bryan-cox]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[hypershift-jira-solve-ci]

I now have all the evidence needed. Here is the final report:

Test Failure Analysis Complete

Job Information

• Prow Job: pull-ci-openshift-hypershift-main-e2e-aws
• Build ID: 2076929237461241856
• Target: e2e-aws
• PR: OCPBUGS-98213: fix router component ordering to prevent missing HAProxy backends #8971 (OCPBUGS-98213: fix router component ordering to prevent missing HAProxy backends)
• Failed Step: hypershift-aws-run-e2e-nested
• Total Tests: 645 (643 passed, 30 skipped, 2 failures)

Test Failure Analysis

Error

TestAutoscaling/Teardown (1419.58s):
  fixture.go:338: Failed to wait for infra resources in guest cluster to be deleted: context deadline exceeded
  fixture.go:345: Failed to clean up 11 remaining resources for guest cluster
  
Leaked resources: 10 EC2 EBS volumes + 1 NLB (all owned by autoscaling-4llb9 hosted cluster)

Summary

The TestAutoscaling/Teardown test failed because AWS infrastructure resources (10 EBS volumes from worker nodes and 1 Network Load Balancer for router-default) could not be deleted within the ~24-minute teardown deadline. All actual autoscaling functional tests passed (TestAutoscaling, TestAutoscalerRespectsNodePoolPause, TestAutoscalingBalancing). This is a known flaky infrastructure cleanup timing issue completely unrelated to PR #8971's changes, which modify router component ordering in the control-plane-operator — a different subsystem from the e2e test framework's AWS resource cleanup.

Root Cause

The failure is a flaky AWS infrastructure teardown timeout, unrelated to the PR.

The TestAutoscaling test creates a hosted cluster (autoscaling-4llb9) and scales it up/down across multiple nodepools (thzr4 initial pool with 7 nodes, additional-t82j8 pool with 3 nodes). After the functional tests pass, the Teardown phase must wait for all AWS infrastructure (EBS volumes attached to worker nodes + the ingress NLB) to be deleted.

The teardown timed out after ~24 minutes (context deadline exceeded) with 11 resources still pending deletion:

• 10 EBS volumes: Root volumes of CAPA-managed worker Machines from two node pools
• 1 NLB: The openshift-ingress/router-default service load balancer

This is a race condition between the hosted cluster destruction and AWS's asynchronous resource cleanup. Notably, TestNodePool/HostedCluster0/Teardown took even longer (1458.24s) but barely passed, showing that multiple tests were close to the timeout boundary in this run. The autoscaling test has more nodes to clean up (due to scale-up tests), making it more susceptible to this timeout.

PR #8971 unrelatedness confirmed: The PR modifies control-plane-operator/controllers/hostedcontrolplane/ files to reorder the router component registration and add WithDependencies. These changes affect reconciliation ordering within the hosted control plane operator — they have zero interaction with the e2e test framework's fixture.go AWS resource cleanup logic.

Recommendations

1. Retry the job — This is a transient AWS infrastructure cleanup timing issue. A re-run should pass.
2. This failure should not block PR merge — The failure is in test teardown infrastructure cleanup, not in any functional test, and is completely unrelated to the PR's router component ordering changes.
3. Known flaky pattern — The TestAutoscaling/Teardown timeout is a known flaky issue in the HyperShift e2e suite when AWS resource deletion takes longer than expected. Consider filing a test infrastructure issue to extend the teardown deadline for tests that scale to many nodes.

Evidence

Evidence	Detail
Failed test	TestAutoscaling/Teardown (1419.58s)
Error message	Failed to wait for infra resources in guest cluster to be deleted: context deadline exceeded
Leaked resources	10 EBS volumes + 1 NLB for cluster autoscaling-4llb9
All functional tests	PASSED — TestAutoscaling/Main/TestAutoscaling (1020s), TestAutoscalerRespectsNodePoolPause (956s), TestAutoscalingBalancing (808s)
Near-miss	TestNodePool/HostedCluster0/Teardown took 1458.24s (longer) but passed — showing system-wide slow cleanup
PR changes scope	control-plane-operator/controllers/hostedcontrolplane/ — router component ordering, not e2e test infra
Cluster destroyed	hypershift_framework.go:591: Destroyed cluster. Namespace: e2e-clusters-kj6xf, name: autoscaling-4llb9 — cluster deletion itself succeeded
Test result summary	645 tests, 30 skipped, 2 failures (both TestAutoscaling/Teardown and parent TestAutoscaling)

---

[enxebre]

/lgtm
/pipeline requried
/verified by @bennerv

[openshift-ci-robot]

@enxebre: This PR has been marked as verified by @bennerv.

Details

In response to this:

/lgtm
/pipeline requried
/verified by @bennerv

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[enxebre]

/pipeline required

[openshift-merge-bot]

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-v2-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke

[openshift-merge-bot]

Tests from second stage were triggered manually. Pipeline can be controlled only manually, until HEAD changes. Use command to trigger second stage.

[enxebre]

/retest

[bennerv]

/hold cancel

three scenarios look good with the new CPO image built from this PR. See comment for details.

[openshift-ci]

@hypershift-jira-solve-ci[bot]: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci-robot]

@hypershift-jira-solve-ci[bot]: Jira Issue Verification Checks: Jira Issue OCPBUGS-98213
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-98213 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

OCPBUGS-98213: fix router component ordering to prevent missing HAProxy backends

What this PR does / why we need it:

The router's adaptConfig function lists existing Route objects to generate the HAProxy ConfigMap. When the router was registered early in the component list (position #8), components that create Routes — ignition-server (#35) and metrics-proxy (#38) — had not yet run during the first reconcile pass. This caused the initial HAProxy config to miss ignition and metrics-proxy backends, routing ignition requests to the KAS default backend and failing NodePool ignition.

This PR:

• Moves the router component to after metricsproxyv2 (the last route-creating component) so that all Routes exist when the router config is generated.
• Declares ignition-server and metrics-proxy as explicit WithDependencies on the router component, blocking its reconciliation until both dependencies are Available and RolloutComplete. This provides a stronger guarantee than positional ordering alone, though the informer cache update from watch events is still async.
• Adds TestRouterComponentComesAfterRouteCreatingComponents to verify the router is always registered after ignition-server and metrics-proxy, preventing future regressions from component reordering.

Which issue(s) this PR fixes:

Fixes https://redhat.atlassian.net/browse/OCPBUGS-98213

Special notes for your reviewer:

The fix uses two complementary mechanisms: positional ordering in registerComponents (belt) and WithDependencies via the component framework (suspenders). WithDependencies ensures the router's reconciliation is skipped until ignition-server and metrics-proxy report Available + RolloutComplete, giving the informer cache time to reflect their routes.

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

---

Always review AI generated responses prior to use.
Generated with Claude Code via openshift-developer plugin

---

Note: This PR was auto-generated by the jira-agent periodic CI job in response to OCPBUGS-98213. See the full report for token usage, cost breakdown, and detailed phase output.

Summary by CodeRabbit

• Bug Fixes
• Ensured the router component initializes after components that create prerequisite Routes.
• Made router reconciliation explicitly honor required dependencies; on ARO hosted control planes, it now waits for required Routes to exist and be ready before proceeding.
• Tests
• Added and expanded unit tests to verify router component registration order, ARO Route requirements, readiness checks, and router predicate behavior.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-cherrypick-robot]

@enxebre: #8971 failed to apply on top of branch "release-4.22":

Applying: fix(control-plane-operator): ensure router component runs after route-creating components
Using index info to reconstruct a base tree...
M	control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go
M	control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller_test.go
Falling back to patching base and 3-way merge...
Auto-merging control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go
Auto-merging control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller_test.go
CONFLICT (content): Merge conflict in control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller_test.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Patch failed at 0001 fix(control-plane-operator): ensure router component runs after route-creating components

Details

In response to this:

/cherry-pick release-4.22

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[enxebre]

/cherry-pick release-4.22

[openshift-cherrypick-robot]

@enxebre: #8971 failed to apply on top of branch "release-4.22":

Applying: fix(control-plane-operator): ensure router component runs after route-creating components
Using index info to reconstruct a base tree...
M	control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go
M	control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller_test.go
Falling back to patching base and 3-way merge...
Auto-merging control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go
Auto-merging control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller_test.go
CONFLICT (content): Merge conflict in control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller_test.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Patch failed at 0001 fix(control-plane-operator): ensure router component runs after route-creating components

Details

In response to this:

/cherry-pick release-4.22

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.