CNTRLPLANE-2775: Expose KAS availability and latency metrics from the control-plane-operator

[enxebre]

Description

Instruments the existing healthCheckKASEndpoint() function in the control-plane-operator to expose two new Prometheus metrics for KAS health monitoring:

Metric	Type	Description
hypershift_kube_apiserver_available	Gauge	1 if /healthz returns HTTP 200, 0 otherwise
hypershift_kube_apiserver_request_duration_seconds	Histogram	Latency of the /healthz probe (buckets: 0.01–10s)

Why

HCP offerings (ROSA HCP, ARO HCP) need to monitor customer API endpoint availability and latency for SLA purposes. ROSA HCP currently relies on an external tool (route-monitor-operator) solely for this. These native metrics eliminate that dependency.

How

• Metrics are registered with the controller-runtime metrics registry and automatically scraped by the existing PodMonitor for the CPO — no new monitoring infrastructure required
• Each CPO pod runs in its own HCP namespace, so metrics are naturally scoped per hosted cluster
• The existing HostedControlPlaneAvailable condition logic is unchanged — metrics are a side-effect, not a replacement
• Works across all endpoint topologies: private clusters, public with Route, public with LoadBalancer, shared ingress (ARO HCP / ROSA HCP)

Key files

• control-plane-operator/controllers/hostedcontrolplane/kas/metrics.go — metric definitions and registration
• control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go — instrumented health check
• control-plane-operator/main.go — metrics initialization

Testing

• Unit tests verify gauge and histogram are correctly set for success (200), failure (503), and unreachable scenarios
• E2e test validates metric presence on the CPO pod using GetMetricsFromPod/ValidateMetricPresence (Karpenter pattern)
• All existing tests pass (make test exits 0)

Jira

CNTRLPLANE-2775

🤖 Generated with Claude Code via /jira:solve [CNTRLPLANE-2775](https://issues.redhat.com/browse/CNTRLPLANE-2775)

Summary by CodeRabbit

• New Features

  • Added KAS health metrics (availability gauge and request-duration histogram) and registered them for collection.

• Tests

  • Added unit tests for KAS health metrics and end-to-end validation to check metrics are emitted during runs.

• Chores

  • Metrics instrumentation initialized at startup so controller exposes KAS health metrics for monitoring.

[openshift-ci-robot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[openshift-ci-robot]

@enxebre: This pull request references CNTRLPLANE-2775 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the epic to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Description

Instruments the existing healthCheckKASEndpoint() function in the control-plane-operator to expose two new Prometheus metrics for KAS health monitoring:

Metric	Type	Description
hypershift_kube_apiserver_available	Gauge	1 if /healthz returns HTTP 200, 0 otherwise
hypershift_kube_apiserver_request_duration_seconds	Histogram	Latency of the /healthz probe (buckets: 0.01–10s)

Why

HCP offerings (ROSA HCP, ARO HCP) need to monitor customer API endpoint availability and latency for SLA purposes. ROSA HCP currently relies on an external tool (route-monitor-operator) solely for this. These native metrics eliminate that dependency.

How

• Metrics are registered with the controller-runtime metrics registry and automatically scraped by the existing PodMonitor for the CPO — no new monitoring infrastructure required
• Each CPO pod runs in its own HCP namespace, so metrics are naturally scoped per hosted cluster
• The existing HostedControlPlaneAvailable condition logic is unchanged — metrics are a side-effect, not a replacement
• Works across all endpoint topologies: private clusters, public with Route, public with LoadBalancer, shared ingress (ARO HCP / ROSA HCP)

Key files

• control-plane-operator/controllers/hostedcontrolplane/kas/metrics.go — metric definitions and registration
• control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go — instrumented health check
• control-plane-operator/main.go — metrics initialization

Testing

• Unit tests verify gauge and histogram are correctly set for success (200), failure (503), and unreachable scenarios
• E2e test validates metric presence on the CPO pod using GetMetricsFromPod/ValidateMetricPresence (Karpenter pattern)
• All existing tests pass (make test exits 0)

Jira

CNTRLPLANE-2775

🤖 Generated with Claude Code via /jira:solve [CNTRLPLANE-2775](https://issues.redhat.com/browse/CNTRLPLANE-2775)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

---

Walkthrough

Adds Prometheus metrics for KAS health (availability and request duration), threads an optional KASHealthMetrics through KAS health checks, initializes metrics at startup, and adds unit and E2E tests and helpers to validate metrics exposure. Some test and helper additions are duplicated in the diff.

Changes

Cohort / File(s)	Summary
Controller instrumentation control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go	Added HostedControlPlaneReconciler.KASHealthMetrics *kas.KASHealthMetrics; changed healthCheckKASEndpoint to accept m *kas.KASHealthMetrics; record request duration and set availability (guarded by nil check).
KAS metrics implementation control-plane-operator/controllers/hostedcontrolplane/kas/metrics.go	New metrics: KASAvailableMetricName, KASRequestDurationMetricName, KASRequestDurationBuckets; KASHealthMetrics struct with Available gauge and RequestDuration histogram; NewKASHealthMetrics() registering metrics.
Controller unit tests control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller_test.go	Added TestHealthCheckKASEndpointMetrics with subtests (200, 503, unreachable, nil-metrics), helpers newTestKASHealthMetrics and parseHostPort. Note: test and helpers appear duplicated in the diff — inspect for unintended repeats.
KAS metrics unit tests control-plane-operator/controllers/hostedcontrolplane/kas/metrics_test.go	New test validating metric registration, gauge initial state, gauge update, and histogram observation via a Prometheus registry.
Startup wiring control-plane-operator/main.go	Instantiates kas.NewKASHealthMetrics() and injects it into HostedControlPlaneReconciler during startup/setup.
E2E helpers & integration test/e2e/util/hypershift_framework.go, test/e2e/util/util.go	Adds ValidateCPOMetrics E2E helper and invokes it in after-phase; helper polls control-plane-operator metrics for kas.KASAvailableMetricName and kas.KASRequestDurationMetricName. Note: ValidateCPOMetrics appears duplicated in the diff — verify and dedupe.

Sequence Diagram(s)

sequenceDiagram
  participant Tests as Tests/E2E
  participant Reconciler as HostedControlPlaneReconciler
  participant KAS as KAS/API
  participant Metrics as Prometheus/Registry

  Tests->>Reconciler: trigger health check
  Reconciler->>KAS: HTTP request to ingress (healthCheckKASEndpoint with m)
  activate KAS
  KAS-->>Reconciler: HTTP response (200/503/timeout)
  deactivate KAS
  alt m != nil
    Reconciler->>Metrics: observe RequestDuration
    Reconciler->>Metrics: set Available = 1 or 0
  end
  Tests->>Metrics: query metrics endpoint to validate metrics present/values

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 27.27% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Test Structure And Quality	⚠️ Warning	PR contains duplicated test implementations in hostedcontrolplane_controller_test.go and util.go, lacks meaningful assertion messages, and has insufficient timeout safeguards in E2E polling operations.	Remove duplicate test functions, add descriptive failure messages to all assertions, and ensure explicit timeout configuration for all polling operations.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and specifically describes the main objective: exposing KAS availability and latency metrics from the control-plane-operator, which aligns with the core implementation across all modified files.
Stable And Deterministic Test Names	✅ Passed	All test names in the PR use stable, deterministic naming with no dynamic content, formatted strings, or variable substitution.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[openshift-ci-robot]

@enxebre: This pull request references CNTRLPLANE-2775 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the epic to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Description

Instruments the existing healthCheckKASEndpoint() function in the control-plane-operator to expose two new Prometheus metrics for KAS health monitoring:

Metric	Type	Description
hypershift_kube_apiserver_available	Gauge	1 if /healthz returns HTTP 200, 0 otherwise
hypershift_kube_apiserver_request_duration_seconds	Histogram	Latency of the /healthz probe (buckets: 0.01–10s)

Why

HCP offerings (ROSA HCP, ARO HCP) need to monitor customer API endpoint availability and latency for SLA purposes. ROSA HCP currently relies on an external tool (route-monitor-operator) solely for this. These native metrics eliminate that dependency.

How

• Metrics are registered with the controller-runtime metrics registry and automatically scraped by the existing PodMonitor for the CPO — no new monitoring infrastructure required
• Each CPO pod runs in its own HCP namespace, so metrics are naturally scoped per hosted cluster
• The existing HostedControlPlaneAvailable condition logic is unchanged — metrics are a side-effect, not a replacement
• Works across all endpoint topologies: private clusters, public with Route, public with LoadBalancer, shared ingress (ARO HCP / ROSA HCP)

Key files

• control-plane-operator/controllers/hostedcontrolplane/kas/metrics.go — metric definitions and registration
• control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go — instrumented health check
• control-plane-operator/main.go — metrics initialization

Testing

• Unit tests verify gauge and histogram are correctly set for success (200), failure (503), and unreachable scenarios
• E2e test validates metric presence on the CPO pod using GetMetricsFromPod/ValidateMetricPresence (Karpenter pattern)
• All existing tests pass (make test exits 0)

Jira

CNTRLPLANE-2775

🤖 Generated with Claude Code via /jira:solve [CNTRLPLANE-2775](https://issues.redhat.com/browse/CNTRLPLANE-2775)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[enxebre]

/cc @muraee @csrwng

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: enxebre

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [enxebre]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[enxebre]

/auto-cc

[typeid]

Just to clarify a bit further, while these metrics are great, in ROSA HCP we intend to probe KAS availability externally via RHOBS synthetic monitoring (SREP-333), where Blackbox Exporter runs on RHOBS cells outside the management cluster. This gives us the advantage of testing the actual customer-facing network path (only partially for private API), including DNS resolution, load balancer health, and regional routing, rather than probing from within the MC's own network.

I understand ARO HCP wants to avoid the RMO dependency, and these metrics help with that. However, since the CPO probe originates from within the MC, it's not a full replacement for external synthetic monitoring for SLA purposes IMO.

That said, these CPO-local metrics are definitely useful even for the ROSA side for faster internal detection of control plane issues, for example catching KAS pod crashes or pinpointing network issues to in-cluster networking failures.

LGTM & thanks for the addition!

[typeid]

Also cc @dustman9000 as a FYI that this exists and is now being extended with latency as well :)

[muraee]

lgtm

[enxebre]

/test e2e-aws

[openshift-ci-robot]

@enxebre: This pull request references CNTRLPLANE-2775 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the epic to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Description

Instruments the existing healthCheckKASEndpoint() function in the control-plane-operator to expose two new Prometheus metrics for KAS health monitoring:

Metric	Type	Description
hypershift_kube_apiserver_available	Gauge	1 if /healthz returns HTTP 200, 0 otherwise
hypershift_kube_apiserver_request_duration_seconds	Histogram	Latency of the /healthz probe (buckets: 0.01–10s)

Why

HCP offerings (ROSA HCP, ARO HCP) need to monitor customer API endpoint availability and latency for SLA purposes. ROSA HCP currently relies on an external tool (route-monitor-operator) solely for this. These native metrics eliminate that dependency.

How

• Metrics are registered with the controller-runtime metrics registry and automatically scraped by the existing PodMonitor for the CPO — no new monitoring infrastructure required
• Each CPO pod runs in its own HCP namespace, so metrics are naturally scoped per hosted cluster
• The existing HostedControlPlaneAvailable condition logic is unchanged — metrics are a side-effect, not a replacement
• Works across all endpoint topologies: private clusters, public with Route, public with LoadBalancer, shared ingress (ARO HCP / ROSA HCP)

Key files

• control-plane-operator/controllers/hostedcontrolplane/kas/metrics.go — metric definitions and registration
• control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go — instrumented health check
• control-plane-operator/main.go — metrics initialization

Testing

• Unit tests verify gauge and histogram are correctly set for success (200), failure (503), and unreachable scenarios
• E2e test validates metric presence on the CPO pod using GetMetricsFromPod/ValidateMetricPresence (Karpenter pattern)
• All existing tests pass (make test exits 0)

Jira

CNTRLPLANE-2775

🤖 Generated with Claude Code via /jira:solve [CNTRLPLANE-2775](https://issues.redhat.com/browse/CNTRLPLANE-2775)

Summary by CodeRabbit

Release Notes

• New Features

• Added Kubernetes API Server (KAS) health metrics monitoring with Prometheus instrumentation, tracking request duration and availability status.

• Tests

• Added comprehensive validation tests for KAS health metrics functionality.

• Integrated Control Plane Operator metrics validation into end-to-end test workflows.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@test/e2e/util/util.go`:
- Around line 2712-2743: The loop in ValidateCPOMetrics uses
ValidateMetricPresence which expects labeled metrics and therefore never matches
label-less KAS metrics; modify the inner check after GetMetricsFromPod to look
for the MetricFamily by name (from the returned mf MetricFamily map or slice)
for kas.KASAvailableMetricName and kas.KASRequestDurationMetricName instead of
calling ValidateMetricPresence, i.e., verify the MetricFamily exists and has at
least one metric (no label checks) before returning true; keep the surrounding
wait.PollUntilContextTimeout, GetMetricsFromPod, and error handling unchanged.

[openshift-ci-robot]

@enxebre: This pull request references CNTRLPLANE-2775 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the epic to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Description

Instruments the existing healthCheckKASEndpoint() function in the control-plane-operator to expose two new Prometheus metrics for KAS health monitoring:

Metric	Type	Description
hypershift_kube_apiserver_available	Gauge	1 if /healthz returns HTTP 200, 0 otherwise
hypershift_kube_apiserver_request_duration_seconds	Histogram	Latency of the /healthz probe (buckets: 0.01–10s)

Why

HCP offerings (ROSA HCP, ARO HCP) need to monitor customer API endpoint availability and latency for SLA purposes. ROSA HCP currently relies on an external tool (route-monitor-operator) solely for this. These native metrics eliminate that dependency.

How

• Metrics are registered with the controller-runtime metrics registry and automatically scraped by the existing PodMonitor for the CPO — no new monitoring infrastructure required
• Each CPO pod runs in its own HCP namespace, so metrics are naturally scoped per hosted cluster
• The existing HostedControlPlaneAvailable condition logic is unchanged — metrics are a side-effect, not a replacement
• Works across all endpoint topologies: private clusters, public with Route, public with LoadBalancer, shared ingress (ARO HCP / ROSA HCP)

Key files

• control-plane-operator/controllers/hostedcontrolplane/kas/metrics.go — metric definitions and registration
• control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go — instrumented health check
• control-plane-operator/main.go — metrics initialization

Testing

• Unit tests verify gauge and histogram are correctly set for success (200), failure (503), and unreachable scenarios
• E2e test validates metric presence on the CPO pod using GetMetricsFromPod/ValidateMetricPresence (Karpenter pattern)
• All existing tests pass (make test exits 0)

Jira

CNTRLPLANE-2775

🤖 Generated with Claude Code via /jira:solve [CNTRLPLANE-2775](https://issues.redhat.com/browse/CNTRLPLANE-2775)

Summary by CodeRabbit

• New Features

• Added KAS health metrics: availability and request-duration metrics exposed for monitoring.

• Tests

• Added unit tests for KAS health metrics and integrated control-plane metrics validation into end-to-end test flows.

• Chores

• Instrumentation initialized at startup so metrics are available from the controller runtime.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[enxebre]

/test e2e-aws
/verified by e2e

[typeid]

/lgtm

[openshift-merge-bot]

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks-4-22
/test e2e-aws-4-22
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke

[devguyio]

/lgtm yolo

[enxebre]

/test e2e-aws

[enxebre]

/test e2e-aws-4-22

[muraee]

/lgtm

[openshift-merge-bot]

Tests from second stage were triggered manually. Pipeline can be controlled only manually, until HEAD changes. Use command to trigger second stage.

[hypershift-jira-solve-ci]

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-aws | Build: 2061747161749524480 | Cost: $1.8019642499999997 | Failed step: hypershift-aws-run-e2e-nested

View full analysis report

---

_{Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6}

[openshift-ci]

@enxebre: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/unit	e94ec9e	link	true	/test unit
ci/prow/verify-workflows	e94ec9e	link	true	/test verify-workflows
ci/prow/e2e-aks-4-22	4946853	link	true	/test e2e-aks-4-22
ci/prow/e2e-aws-4-22	db0c022	link	true	/test e2e-aws-4-22
ci/prow/e2e-aws	db0c022	link	true	/test e2e-aws

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci]

Stale PRs are closed after 21d of inactivity.

If this PR is still relevant, comment to refresh it or remove the stale label.
Mark the PR as fresh by commenting /remove-lifecycle stale.

If this PR is safe to close now please do so with /close.

/lifecycle stale

[openshift-ci]

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[hypershift-jira-solve-ci]

Now I have all the evidence I need. Both jobs show the exact same root cause — AWS EC2 API rate limiting. This is a well-known CI infrastructure flake that is unrelated to the PR's code changes (which expose KAS availability/latency metrics from the control-plane-operator).

Test Failure Analysis Complete

Job Information

• Prow Job 1: pull-ci-openshift-hypershift-main-e2e-aws
• Build ID 1: 2061747161749524480
• Prow Job 2: pull-ci-openshift-hypershift-main-e2e-aws-4-22
• Build ID 2: 2061747230338977792
• PR: CNTRLPLANE-2775: Expose KAS availability and latency metrics from the control-plane-operator #7749 — CNTRLPLANE-2775: Expose KAS availability and latency metrics from the control-plane-operator
• Targets: e2e-aws, e2e-aws-4-22

Test Failure Analysis

Error

failed to create cluster, tearing down: failed to create infra: cannot create VPC S3 endpoint:
operation error EC2: CreateVpcEndpoint, exceeded maximum number of attempts, 11,
https response error StatusCode: 503, api error RequestLimitExceeded:
Request limit exceeded. Account 820196288204 has been throttled on ec2:CreateVpcEndpoint
because it exceeded its request rate limit.

Summary

Both CI jobs (e2e-aws and e2e-aws-4-22) failed due to AWS EC2 API rate limiting on the CreateVpcEndpoint operation for AWS account 820196288204. The two jobs ran concurrently (both started at ~09:50 UTC on 2026-06-02) and collectively overwhelmed the account's API rate limits. In e2e-aws, 2 tests failed (TestKarpenterUpgradeControlPlane); in e2e-aws-4-22, 7 tests failed (TestCreateClusterProxy, TestNodePoolAutoscalingScaleFromZero, TestUpgradeControlPlane, TestAutoscaling, TestCreateCluster, TestKarpenterUpgradeControlPlane). Every failure has the identical RequestLimitExceeded error on ec2:CreateVpcEndpoint. The remaining 567+ tests in e2e-aws and 347+ tests in e2e-aws-4-22 passed. This is a CI infrastructure flake — the PR changes (exposing KAS availability/latency metrics) do not touch AWS infrastructure, VPC, or cluster creation code.

Root Cause

Root cause: AWS EC2 API rate limiting (CI infrastructure flake)

The shared AWS account 820196288204 was throttled on the ec2:CreateVpcEndpoint API call. Both PR jobs ran concurrently (started within 16 seconds of each other at ~09:50 UTC), and each HyperShift E2E test creates its own hosted cluster — which involves creating VPC infrastructure including S3 VPC endpoints. When many tests attempt cluster creation in parallel across two concurrent job runs sharing the same AWS account, the aggregate CreateVpcEndpoint call rate exceeded AWS's per-account API rate limit.

The e2e-aws-4-22 job was more severely impacted (7 failures vs 2) because it runs a different test suite where more tests happened to be in the cluster-creation phase simultaneously when the throttling kicked in.

Key indicators this is infrastructure-only:

1. Identical error across all failures — every single failed test hit the exact same RequestLimitExceeded error on CreateVpcEndpoint, with different AWS RequestIDs confirming independent API calls being throttled.
2. HTTP 503 from AWS — the error originates from AWS's rate limiter, not from any code in the PR.
3. PR changes are unrelated — PR CNTRLPLANE-2775: Expose KAS availability and latency metrics from the control-plane-operator #7749 exposes KAS availability/latency metrics from the control-plane-operator. It does not modify VPC creation, AWS API calls, or cluster infrastructure code.
4. Majority of tests passed — 567/594 tests passed in e2e-aws and 347/379 passed in e2e-aws-4-22, demonstrating the test framework and PR code work correctly when AWS API calls succeed.

Recommendations

1. Retest the PR — /retest the failing jobs. This is a transient AWS rate-limiting issue that will resolve when the account is under less load.
2. No code changes needed — the PR's metric-exposure changes are not related to these failures.
3. The tide error state is a consequence of the two required jobs failing; it will clear once both jobs pass on retry.

Evidence

Evidence	Detail
Error type	AWS EC2 API RequestLimitExceeded on ec2:CreateVpcEndpoint (HTTP 503)
AWS Account	820196288204
e2e-aws failures	2/594 tests: TestKarpenterUpgradeControlPlane
e2e-aws-4-22 failures	7/379 tests: TestCreateClusterProxy, TestNodePoolAutoscalingScaleFromZero, TestUpgradeControlPlane, TestAutoscaling, TestCreateCluster, TestKarpenterUpgradeControlPlane
Job concurrency	Both jobs started within 16s of each other (~09:50 UTC, 2026-06-02) on the same AWS account
Error consistency	All 9 unique test failures across both jobs show identical CreateVpcEndpoint rate-limit errors with distinct AWS RequestIDs
PR relevance	PR #7749 exposes KAS metrics — no changes to VPC/AWS/infra code
Failure source	hypershift_framework.go:518 — cluster creation infra setup, not test logic

---

[openshift-ci]

Stale PRs rot after 14d of inactivity.

Mark the PR as fresh by commenting /remove-lifecycle rotten.
Rotten PRs close after an additional 7d of inactivity.

If this PR is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-ci]

Rotten PRs close after 7d of inactivity.

Reopen the PR by commenting /reopen.
Mark the PR as fresh by commenting /remove-lifecycle rotten.

/close

[openshift-ci]

@openshift-ci[bot]: Closed this PR.

Details

In response to this:

Rotten PRs close after 7d of inactivity.

Reopen the PR by commenting /reopen.
Mark the PR as fresh by commenting /remove-lifecycle rotten.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.