Skip to content

OCPBUGS-84508: Fix NLB name parsing for EKS Auto Mode hostnames - #8343

Merged
openshift-merge-bot[bot] merged 1 commit into
openshift:mainfrom
typeid:fix/nlb-name-parsing-eks-auto-mode
May 1, 2026
Merged

OCPBUGS-84508: Fix NLB name parsing for EKS Auto Mode hostnames#8343
openshift-merge-bot[bot] merged 1 commit into
openshift:mainfrom
typeid:fix/nlb-name-parsing-eks-auto-mode

Conversation

@typeid

@typeid typeid commented Apr 27, 2026

Copy link
Copy Markdown
Member

Summary

  • The PrivateServiceObserver extracts the NLB name from the Kubernetes Service's load balancer hostname to populate AWSEndpointService.Spec.NetworkLoadBalancerName. The current parsing splits on the first -, which fails for NLBs provisioned by the AWS Load Balancer Controller (EKS Auto Mode), where names contain hyphens (e.g. k8s-clusters-kubeapis-db6fee3a62).
  • Extracts the NLB name by stripping only the last dash-delimited segment (the AWS-assigned ID suffix), which works for both standard and EKS Auto Mode NLB hostnames.

Background

AWS NLB DNS hostnames follow the format {name}-{id}.elb.{region}.amazonaws.com, where {name} is the exact value passed to CreateLoadBalancer and {id} is an AWS-assigned hex suffix (AWS docs).

Backwards compatibility: The {id} suffix is always hex (no hyphens), as shown in every AWS API example. This is also structurally required — since {name} may contain hyphens, a hyphenated {id} would make the hostname format ambiguous. LastIndex("-") therefore always lands on the boundary between {name} and {id}, producing the same result as Split("-")[0] when the name has no hyphens.

The two NLB name generation paths:

  • In-tree cloud provider (OpenShift): generates "a" + serviceUID with all hyphens stripped — purely alphanumeric, so both old and new parsing are identical.
  • AWS LB Controller (EKS Auto Mode): generates k8s-{ns}-{svc}-{hash} — always contains hyphens, where the old Split("-")[0] extracted only k8s.

Jira

https://issues.redhat.com/browse/OCPBUGS-84508

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes

    • Corrected extraction of network load balancer names from AWS hostnames so names with multiple hyphens are preserved and resolved correctly, preventing misidentification of services.
  • Tests

    • Added unit tests validating name-extraction behavior across several hostname formats to prevent regressions.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Apr 27, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@typeid: This pull request references Jira Issue OCPBUGS-84508, which is invalid:

  • expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

  • The PrivateServiceObserver extracts the NLB name from the Kubernetes Service's load balancer hostname to populate AWSEndpointService.Spec.NetworkLoadBalancerName. The current parsing splits on the first -, which fails for NLBs provisioned by the AWS Load Balancer Controller (EKS Auto Mode), where names contain hyphens (e.g. k8s-clusters-kubeapis-db6fee3a62).
  • Extracts the NLB name by stripping only the last dash-delimited segment (the AWS-assigned ID suffix), which works for both standard and EKS Auto Mode NLB hostnames.

Background

AWS NLB DNS hostnames follow the format {name}-{id}.elb.{region}.amazonaws.com (AWS docs). The AWS Load Balancer Controller generates NLB names containing hyphens using the convention k8s-{namespace}-{service}-{hash} (EKS docs). The previous parsing assumed no hyphens in the name portion, which only holds for the in-tree AWS cloud provider used on OpenShift.

Jira

https://issues.redhat.com/browse/OCPBUGS-84508

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Apr 27, 2026

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Walkthrough

A new package-level function extractNLBName(hostname string) string was added and PrivateServiceObserver.Reconcile now uses it to derive awsEndpointService.Spec.NetworkLoadBalancerName. extractNLBName obtains the service LoadBalancer ingress hostname's first DNS label and removes only the final dash-delimited segment (the AWS-assigned ID), preserving other hyphens in the NLB name. A unit test TestExtractNLBName was added to validate behavior across standard NLB hostnames, EKS Auto Mode hostnames, and hostnames without hyphens.

🚥 Pre-merge checks | ✅ 10 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Test Structure And Quality ⚠️ Warning The TestExtractNLBName test assertion lacks a meaningful failure message as required by the custom check for Ginkgo test quality. Add a descriptive message to the assertion: g.Expect(extractNLBName(tc.hostname)).To(Equal(tc.expected), "extracting NLB name from hostname %q", tc.hostname).
✅ Passed checks (10 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and specifically describes the main change: fixing NLB name parsing to handle EKS Auto Mode hostnames, which is the core issue addressed in the PR.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The new test function TestExtractNLBName follows stable and deterministic naming requirements with descriptive, static strings in test titles and dynamic values in test body.
Microshift Test Compatibility ✅ Passed No new Ginkgo e2e tests were added. The only new test is TestExtractNLBName, a standard Go unit test using the testing package that tests a pure utility function and does not interact with Kubernetes or OpenShift APIs.
Single Node Openshift (Sno) Test Compatibility ✅ Passed The test file uses standard Go testing patterns with t.Run, not Ginkgo e2e test framework patterns, and tests a pure utility function with no cluster topology assumptions.
Topology-Aware Scheduling Compatibility ✅ Passed PR introduces no topology-aware scheduling constraints; changes are limited to hostname parsing utility function for AWS load balancer DNS format handling.
Ote Binary Stdout Contract ✅ Passed PR adds helper function and modifies controller without stdout operations or initialization-level code that would violate OTE Binary Stdout Contract.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed The PR adds only a standard Go unit test using testing.T, not a Ginkgo e2e test. Since the check targets Ginkgo e2e tests, it is not applicable.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci
openshift-ci Bot requested review from Nirshal and bryan-cox April 27, 2026 14:49
@openshift-ci openshift-ci Bot added area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release area/platform/aws PR/issue for AWS (AWSPlatform) platform and removed do-not-merge/needs-area labels Apr 27, 2026
@typeid

typeid commented Apr 27, 2026

Copy link
Copy Markdown
Member Author

/jira refresh

@openshift-ci-robot

Copy link
Copy Markdown

@typeid: This pull request references Jira Issue OCPBUGS-84508, which is invalid:

  • expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@typeid

typeid commented Apr 27, 2026

Copy link
Copy Markdown
Member Author

/jira refresh

@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Apr 27, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@typeid: This pull request references Jira Issue OCPBUGS-84508, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
control-plane-operator/controllers/awsprivatelink/awsprivatelink_controller_test.go (1)

914-942: Add an empty-hostname edge case to lock parser behavior.

Given parser output is persisted into spec, include a case like hostname: ""expected: "" (or whatever behavior you want to guarantee) to prevent silent regressions around uninitialized ingress data.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@control-plane-operator/controllers/awsprivatelink/awsprivatelink_controller_test.go`
around lines 914 - 942, Add an explicit empty-hostname test to
TestExtractNLBName to lock down parser behavior: in the testCases slice inside
TestExtractNLBName, add a case with hostname: "" and expected: "" so
extractNLBName("") is asserted to return the empty string; this prevents
regressions when persisted spec contains uninitialized/empty ingress hostnames
and clearly documents the intended behavior of extractNLBName.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In
`@control-plane-operator/controllers/awsprivatelink/awsprivatelink_controller.go`:
- Line 173: Check and validate the LoadBalancer hostname before
deriving/persisting an NLB name: ensure svc.Status.LoadBalancer != nil,
len(svc.Status.LoadBalancer.Ingress) > 0 and
svc.Status.LoadBalancer.Ingress[0].Hostname is non-empty before calling
extractNLBName; if the hostname is empty, avoid calling extractNLBName and do
not set/persist NetworkLoadBalancerName (or return/error out of the reconcile
path) and log a clear warning. Update the code around the extractNLBName call
and any subsequent assignment to NetworkLoadBalancerName to use this guard so
DescribeLoadBalancers is never invoked with an empty name.

---

Nitpick comments:
In
`@control-plane-operator/controllers/awsprivatelink/awsprivatelink_controller_test.go`:
- Around line 914-942: Add an explicit empty-hostname test to TestExtractNLBName
to lock down parser behavior: in the testCases slice inside TestExtractNLBName,
add a case with hostname: "" and expected: "" so extractNLBName("") is asserted
to return the empty string; this prevents regressions when persisted spec
contains uninitialized/empty ingress hostnames and clearly documents the
intended behavior of extractNLBName.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: c599f7ef-b59b-46ac-9fa4-6ca87ac9dc5e

📥 Commits

Reviewing files that changed from the base of the PR and between a6e1d11 and 79cfb63.

📒 Files selected for processing (2)
  • control-plane-operator/controllers/awsprivatelink/awsprivatelink_controller.go
  • control-plane-operator/controllers/awsprivatelink/awsprivatelink_controller_test.go

@openshift-ci-robot

Copy link
Copy Markdown

@typeid: This pull request references Jira Issue OCPBUGS-84508, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
Details

In response to this:

Summary

  • The PrivateServiceObserver extracts the NLB name from the Kubernetes Service's load balancer hostname to populate AWSEndpointService.Spec.NetworkLoadBalancerName. The current parsing splits on the first -, which fails for NLBs provisioned by the AWS Load Balancer Controller (EKS Auto Mode), where names contain hyphens (e.g. k8s-clusters-kubeapis-db6fee3a62).
  • Extracts the NLB name by stripping only the last dash-delimited segment (the AWS-assigned ID suffix), which works for both standard and EKS Auto Mode NLB hostnames.

Background

AWS NLB DNS hostnames follow the format {name}-{id}.elb.{region}.amazonaws.com (AWS docs). The AWS-assigned ID suffix is a 16-char hex string that never contains hyphens.

The in-tree AWS cloud provider generates NLB names as "a" + serviceUID with all hyphens stripped, producing purely alphanumeric names. For these, Split("-")[0] and LastIndex("-") produce identical results — this change is backwards compatible for OpenShift management clusters.

The AWS Load Balancer Controller generates NLB names in the format k8s-{namespace}-{service}-{hash} (source), which always contains hyphens. The old parsing extracted only k8s, causing DescribeLoadBalancers to fail with LoadBalancerNotFound.

Jira

https://issues.redhat.com/browse/OCPBUGS-84508

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@codecov

codecov Bot commented Apr 27, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.50000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 36.09%. Comparing base (a6e1d11) to head (4e22e73).
⚠️ Report is 68 commits behind head on main.

Files with missing lines Patch % Lines
...ollers/awsprivatelink/awsprivatelink_controller.go 87.50% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #8343   +/-   ##
=======================================
  Coverage   36.08%   36.09%           
=======================================
  Files         767      767           
  Lines       93486    93493    +7     
=======================================
+ Hits        33737    33744    +7     
  Misses      57042    57042           
  Partials     2707     2707           
Files with missing lines Coverage Δ
...ollers/awsprivatelink/awsprivatelink_controller.go 18.95% <87.50%> (+0.64%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@typeid
typeid force-pushed the fix/nlb-name-parsing-eks-auto-mode branch 2 times, most recently from 19b27cc to 8421390 Compare April 27, 2026 15:24
EKS Auto Mode provisions NLBs via the AWS LB Controller, which
generates names containing hyphens (e.g. k8s-ns-svc-hash). The
previous parsing split on the first hyphen, extracting only the
first segment. Use strings.LastIndex to strip only the AWS-assigned
ID suffix, which works correctly for both standard and EKS Auto Mode
NLB hostnames.

Signed-off-by: Claudio Busse <cbusse@redhat.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@typeid
typeid force-pushed the fix/nlb-name-parsing-eks-auto-mode branch from 8421390 to 4e22e73 Compare April 27, 2026 15:25

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (1)
control-plane-operator/controllers/awsprivatelink/awsprivatelink_controller.go (1)

173-173: ⚠️ Potential issue | 🟠 Major

Validate Ingress[0].Hostname before deriving NLB name.

At Line 173, an empty hostname still produces an empty NetworkLoadBalancerName, which can later fail AWS load balancer lookup by name.

💡 Proposed fix
-	lbName := extractNLBName(svc.Status.LoadBalancer.Ingress[0].Hostname)
+	hostname := strings.TrimSpace(svc.Status.LoadBalancer.Ingress[0].Hostname)
+	if hostname == "" {
+		r.log.Info("load balancer hostname not provisioned yet")
+		return ctrl.Result{}, nil
+	}
+	lbName := extractNLBName(hostname)
+	if lbName == "" {
+		return ctrl.Result{}, fmt.Errorf("failed to extract NLB name from hostname %q", hostname)
+	}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@control-plane-operator/controllers/awsprivatelink/awsprivatelink_controller.go`
at line 173, The code calls extractNLBName(...) on
svc.Status.LoadBalancer.Ingress[0].Hostname without validating the Ingress slice
or Hostname, which can produce an empty lbName and break subsequent AWS lookups;
update the reconcile logic to first check that svc.Status.LoadBalancer != nil,
len(svc.Status.LoadBalancer.Ingress) > 0 and that
svc.Status.LoadBalancer.Ingress[0].Hostname is non-empty before calling
extractNLBName, and if any check fails, log a clear error via the controller
logger and return/requeue (don’t proceed with using lbName).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In
`@control-plane-operator/controllers/awsprivatelink/awsprivatelink_controller.go`:
- Line 173: The code calls extractNLBName(...) on
svc.Status.LoadBalancer.Ingress[0].Hostname without validating the Ingress slice
or Hostname, which can produce an empty lbName and break subsequent AWS lookups;
update the reconcile logic to first check that svc.Status.LoadBalancer != nil,
len(svc.Status.LoadBalancer.Ingress) > 0 and that
svc.Status.LoadBalancer.Ingress[0].Hostname is non-empty before calling
extractNLBName, and if any check fails, log a clear error via the controller
logger and return/requeue (don’t proceed with using lbName).

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 18e9634c-912c-484b-91db-4c343f3ff999

📥 Commits

Reviewing files that changed from the base of the PR and between 19b27cc and 4e22e73.

📒 Files selected for processing (2)
  • control-plane-operator/controllers/awsprivatelink/awsprivatelink_controller.go
  • control-plane-operator/controllers/awsprivatelink/awsprivatelink_controller_test.go

@bryan-cox bryan-cox left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve

@openshift-ci

openshift-ci Bot commented Apr 27, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bryan-cox, typeid

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 27, 2026
@typeid

typeid commented Apr 28, 2026

Copy link
Copy Markdown
Member Author

/retest

@typeid

typeid commented Apr 28, 2026

Copy link
Copy Markdown
Member Author

/test e2e-aks e2e-aws e2e-aws-upgrade-hypershift-operator e2e-azure-self-managed e2e-kubevirt-aws-ovn-reduced e2e-v2-aws

@typeid

typeid commented Apr 28, 2026

Copy link
Copy Markdown
Member Author

/verified by @typeid

Tested on EKS - names are recognized and cluster installation proceeds with this build instead of hanging on failing endpoint service reconciles. E2E tests here cover that there's no regression: we're able to still recognize the OpenShift NLB names as not being able to would mean clusters in E2Es don't come up.

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Apr 28, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@typeid: This PR has been marked as verified by @typeid.

Details

In response to this:

/verified by @typeid

Tested on EKS - names are recognized. E2E tests here cover that there's no regression: we're able to still recognize the OpenShift NLB names as not being able to would mean clusters in E2Es don't come up.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@hypershift-jira-solve-ci

Copy link
Copy Markdown
Contributor

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-aws | Build: 2049030266814468096 | Cost: $1.8857702499999995 | Failed step: hypershift-aws-run-e2e-nested

View full analysis report


Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6

@hypershift-jira-solve-ci

Copy link
Copy Markdown
Contributor

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-azure-self-managed | Build: 2049030266906742784 | Cost: $4.327970249999998 | Failed step: hypershift-azure-run-e2e-self-managed

View full analysis report


Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6

@cwbotbot

cwbotbot commented Apr 28, 2026

Copy link
Copy Markdown

Test Results

e2e-aks

e2e-aws

@typeid

typeid commented Apr 28, 2026

Copy link
Copy Markdown
Member Author

/test e2e-aws e2e-azure-self-managed

@typeid

typeid commented Apr 28, 2026

Copy link
Copy Markdown
Member Author

/test e2e-aws

@typeid

typeid commented Apr 28, 2026

Copy link
Copy Markdown
Member Author

/test e2e-azure-self-managed

@typeid

typeid commented Apr 28, 2026

Copy link
Copy Markdown
Member Author

/retest

@hypershift-jira-solve-ci

Copy link
Copy Markdown
Contributor

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-azure-self-managed | Build: 2049113046529347584 | Cost: $2.5947809999999993 | Failed step: hypershift-azure-run-e2e-self-managed

View full analysis report


Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6

@typeid

typeid commented Apr 29, 2026

Copy link
Copy Markdown
Member Author

/test e2e-azure-self-managed

@csrwng

csrwng commented Apr 30, 2026

Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Apr 30, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws

@typeid

typeid commented Apr 30, 2026

Copy link
Copy Markdown
Member Author

/retest

@hypershift-jira-solve-ci

Copy link
Copy Markdown
Contributor

Test Failure Analysis Complete

Job Information

  • Prow Job: pull-ci-openshift-hypershift-main-e2e-kubevirt-aws-ovn-reduced
  • Build ID: 2049864372095815680
  • Target: e2e-kubevirt-aws-ovn-reduced
  • Failed Test: TestCreateCluster/ValidateHostedCluster
  • PR: #8343 — OCPBUGS-84508: Fix NLB name parsing for EKS Auto Mode hostnames

Test Failure Analysis

Error

Failed to wait for HostedCluster e2e-clusters-f4q5s/create-cluster-94dwl to rollout in 30m0s: context deadline exceeded
wanted most recent version history to have state Completed, has state Partial
ClusterVersionSucceeding=False: ClusterOperatorsNotAvailable(Cluster operators console, monitoring are not available)
ClusterVersionProgressing=True: ClusterOperatorsNotAvailable(Unable to apply 5.0.0-0.ci-2026-04-30-150750-test-ci-op-kj2j7k2k-latest: some cluster operators are not available)

Summary

The TestCreateCluster/ValidateHostedCluster test timed out after 30 minutes waiting for the KubeVirt-hosted HostedCluster create-cluster-94dwl to reach a Completed rollout state. The rollout was stuck in Partial because the console and monitoring cluster operators never became available. Investigation of the cluster dump reveals this was caused by a widespread issue where pods across many namespaces on the guest cluster worker nodes were stuck in ContainerCreating or PodInitializing — affecting not just console/monitoring but also pods in openshift-insights, openshift-multus, openshift-network-diagnostics, openshift-cluster-csi-drivers, and more. Meanwhile, TestAutoscaling (running in parallel on a different HostedCluster) passed completely, confirming this is an environment-specific issue with one particular hosted cluster instance, not a code regression. The PR only modifies awsprivatelink_controller.go (NLB name parsing for EKS Auto Mode), which is completely unrelated to KubeVirt, guest cluster pod scheduling, or cluster operators.

Root Cause

The root cause is a KubeVirt-hosted guest cluster infrastructure flake — not a code regression from PR #8343.

What happened:

  1. Two HostedClusters were created in parallel: create-cluster-94dwl (TestCreateCluster) and autoscaling-v7js7 (TestAutoscaling)
  2. Both clusters had their nodes become Ready (2 nodes for TestCreateCluster in ~10m, 1 node for TestAutoscaling in ~9m)
  3. On the create-cluster-94dwl cluster, pods across nearly all guest cluster namespaces became stuck in ContainerCreating or PodInitializing state on the worker nodes:
    • openshift-monitoring: alertmanager-main-0, kube-state-metrics, metrics-server, monitoring-plugin, node-exporter, openshift-state-metrics, prometheus-k8s-0, telemeter-client, thanos-querier — all stuck
    • openshift-console: console pods and downloads pod — all stuck in ContainerCreating
    • openshift-insights, openshift-multus, openshift-network-diagnostics, openshift-cluster-csi-drivers — similarly stuck
  4. Because so many pods were stuck, the console and monitoring operators never became Available, causing ClusterVersion to remain in Partial state
  5. After 30 minutes, the ValidateHostedCluster wait timed out
  6. The autoscaling-v7js7 cluster had no such problem — it completed rollout in ~5m36s and all its tests passed

Why it's not related to PR #8343:

  • The PR only modifies control-plane-operator/controllers/awsprivatelink/awsprivatelink_controller.go and its unit test
  • These changes affect NLB hostname parsing for EKS Auto Mode — a completely different platform (AWS) and code path from KubeVirt
  • The KubeVirt guest cluster pod scheduling issue has no relationship to AWS Private Link NLB name parsing

Why pods were stuck:
The widespread ContainerCreating state across many namespaces on the guest cluster indicates a node-level issue on the KubeVirt worker VMs — likely related to container runtime readiness, CNI plugin initialization, or volume mount delays on the KubeVirt virtualized nodes. The CreateContainerError seen on init-textfile (node-exporter) and download-server (console downloads) suggests the container runtime on one or more guest nodes was having transient issues.

Recommendations
  1. Retest the PR — This failure is unrelated to the PR changes. Run /test e2e-kubevirt-aws-ovn-reduced to retry.
  2. No code changes needed — The PR modifies only AWS Private Link NLB name parsing (awsprivatelink_controller.go), which has no interaction with KubeVirt guest cluster pod scheduling.
  3. Known flake pattern — KubeVirt-hosted guest clusters occasionally experience widespread pod startup delays on their virtualized worker nodes. This is an infrastructure-level flake in the CI environment.
Evidence
Evidence Detail
Failed test TestCreateCluster/ValidateHostedCluster — timed out after 30m0s
Passed parallel test TestAutoscaling — completed successfully (3816s) on separate HostedCluster
Rollout state Partial — wanted Completed
Unavailable operators console, monitoring
Guest cluster pods stuck alertmanager-main-0, kube-state-metrics, prometheus-k8s-0, thanos-querier, console pods, metrics-server, monitoring-plugin, node-exporter, telemeter-client, openshift-state-metrics, network-metrics-daemon, kubevirt-csi-node — all in ContainerCreating or PodInitializing
CreateContainerError init-textfile (node-exporter), download-server (console downloads)
Affected namespaces openshift-monitoring, openshift-console, openshift-insights, openshift-multus, openshift-network-diagnostics, openshift-cluster-csi-drivers
PR files changed control-plane-operator/controllers/awsprivatelink/awsprivatelink_controller.go, awsprivatelink_controller_test.go only
PR scope NLB name parsing for EKS Auto Mode hostnames — unrelated to KubeVirt
Management cluster nodes All 4 nodes healthy (Ready=True, no pressure conditions)

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

/retest-required

Remaining retests: 0 against base HEAD afd63bd and 2 for PR HEAD 4e22e73 in total

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

/retest-required

Remaining retests: 0 against base HEAD 0ee6567 and 1 for PR HEAD 4e22e73 in total

@openshift-ci

openshift-ci Bot commented May 1, 2026

Copy link
Copy Markdown
Contributor

@typeid: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/verify-workflows 4e22e73 link true /test verify-workflows

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-merge-bot
openshift-merge-bot Bot merged commit 6e734a9 into openshift:main May 1, 2026
36 checks passed
@openshift-ci-robot

Copy link
Copy Markdown

@typeid: Jira Issue Verification Checks: Jira Issue OCPBUGS-84508
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-84508 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

Summary

  • The PrivateServiceObserver extracts the NLB name from the Kubernetes Service's load balancer hostname to populate AWSEndpointService.Spec.NetworkLoadBalancerName. The current parsing splits on the first -, which fails for NLBs provisioned by the AWS Load Balancer Controller (EKS Auto Mode), where names contain hyphens (e.g. k8s-clusters-kubeapis-db6fee3a62).
  • Extracts the NLB name by stripping only the last dash-delimited segment (the AWS-assigned ID suffix), which works for both standard and EKS Auto Mode NLB hostnames.

Background

AWS NLB DNS hostnames follow the format {name}-{id}.elb.{region}.amazonaws.com, where {name} is the exact value passed to CreateLoadBalancer and {id} is an AWS-assigned hex suffix (AWS docs).

Backwards compatibility: The {id} suffix is always hex (no hyphens), as shown in every AWS API example. This is also structurally required — since {name} may contain hyphens, a hyphenated {id} would make the hostname format ambiguous. LastIndex("-") therefore always lands on the boundary between {name} and {id}, producing the same result as Split("-")[0] when the name has no hyphens.

The two NLB name generation paths:

  • In-tree cloud provider (OpenShift): generates "a" + serviceUID with all hyphens stripped — purely alphanumeric, so both old and new parsing are identical.
  • AWS LB Controller (EKS Auto Mode): generates k8s-{ns}-{svc}-{hash} — always contains hyphens, where the old Split("-")[0] extracted only k8s.

Jira

https://issues.redhat.com/browse/OCPBUGS-84508

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes

  • Corrected extraction of network load balancer names from AWS hostnames so names with multiple hyphens are preserved and resolved correctly, preventing misidentification of services.

  • Tests

  • Added unit tests validating name-extraction behavior across several hostname formats to prevent regressions.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-merge-robot

Copy link
Copy Markdown
Contributor

Fix included in release 5.0.0-0.nightly-2026-05-01-102026

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release area/platform/aws PR/issue for AWS (AWSPlatform) platform jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants