Skip to content

CNTRLPLANE-3277: Add Azure OAuth LoadBalancer private topology e2e test - #8584

Open
Nirshal wants to merge 2 commits into
openshift:mainfrom
Nirshal:CNTRLPLANE-3277
Open

CNTRLPLANE-3277: Add Azure OAuth LoadBalancer private topology e2e test#8584
Nirshal wants to merge 2 commits into
openshift:mainfrom
Nirshal:CNTRLPLANE-3277

Conversation

@Nirshal

@Nirshal Nirshal commented May 25, 2026

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

Adds the AzureOAuthLoadBalancerPrivateTest to the v2 Ginkgo e2e suite, validating
that the OAuth server works correctly when published via LoadBalancer on an Azure
Private topology cluster.

The test verifies:

  • The oauth-openshift Service is created as type LoadBalancer with an allocated endpoint
  • The Service carries the Azure internal LoadBalancer annotation (confirming it's an ILB, not public)
  • The OAuth token flow (kubeadmin login + htpasswd IDP setup + token request) works through the private endpoint

Changes to lifecycle/azure.go:

  • Adds a dedicated oauth-lb-private ClusterSpec combining --endpoint-access=Private
    with --oauth-publishing-strategy=LoadBalancer
  • Restores the private ClusterSpec to match main (no OAuth LoadBalancer override),
    resolving the conflict with AzureEndpointAccessTransitionTest (PR CNTRLPLANE-3276: Add Azure ExternalPrivateService and endpoint access transition test #8718) which
    expects Route-based OAuth on the private cluster
  • Removes the private-and-oauth SequentialGroup — with separate clusters, there is
    no shared-state ordering concern (the sequential group was the original compromise
    per CB-2 review feedback to avoid an extra cluster; Bryan's endpoint access transition
    test makes the dedicated cluster necessary)
  • Moves private and oauth-lb-private TestGroups to Parallel

Note: The OAuth token flow test requires the test runner to have network connectivity
to the Azure VNet (e.g., running in the management cluster or via VPN/peering).

Which issue(s) this PR fixes:

Fixes https://redhat.atlassian.net/browse/CNTRLPLANE-3277

Special notes for your reviewer:

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests. No unit tests needed — changes are e2e test code and CI cluster configuration only, with no new business logic.

Summary by CodeRabbit

  • Tests
    • Expanded Azure end-to-end coverage for private clusters using OAuth with internal LoadBalancers.
    • Added validation for OAuth and guest API access through secure port-forward connections.
    • Added coverage for the combined private endpoint and OAuth LoadBalancer configuration.
    • Added checks for expected initial node counts across Azure cluster variants.
    • Improved test reliability by separating LoadBalancer endpoint allocation from readiness checks and restoring authentication configuration after validation.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label May 25, 2026
@openshift-ci-robot

openshift-ci-robot commented May 25, 2026

Copy link
Copy Markdown

@Nirshal: This pull request references CNTRLPLANE-3277 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "5.0.0" version, but no target version was set.

Details

In response to this:

What this PR does / why we need it:

Adds the AzureOAuthLoadBalancerPrivateTest to the v2 Ginkgo e2e suite, validating
that the OAuth server works correctly when published via LoadBalancer on an Azure
Private topology cluster.

The test verifies:

  • The oauth-openshift Service is created as type LoadBalancer with an allocated endpoint
  • The Service carries the Azure internal LoadBalancer annotation (confirming it's an ILB, not public)
  • The OAuth token flow (kubeadmin login + htpasswd IDP setup + token request) works through the private endpoint

Also adds:

  • oauth-lb-private cluster spec in lifecycle/azure.go combining --endpoint-access=Private with --oauth-publishing-strategy=LoadBalancer
  • Corresponding test matrix entry with label self-managed-azure-oauth-lb-private

Note: The OAuth token flow test requires the test runner to have network connectivity
to the Azure VNet (e.g., running in the management cluster or via VPN/peering).

Companion PR needed: openshift/release changes to create/destroy/dump the
oauth-lb-private cluster and wire the label filter in the CI job.

Which issue(s) this PR fixes:

Fixes https://redhat.atlassian.net/browse/CNTRLPLANE-3277

Special notes for your reviewer:

  • This builds on top of CNTRLPLANE-3222: Port v1 lifecycle tests to v2 Ginkgo framework #8527 (CNTRLPLANE-3222) which ported the v1 lifecycle tests to v2 and established the Azure self-managed test infrastructure.
  • The naming oauth-lb-private follows the existing oauth-lb convention rather than private-oauth-lb to stay consistent with Bryan's naming. Open to renaming both to public-oauth-lb / private-oauth-lb for clarity if preferred.
  • The ValidateOAuthWithIdentityProviderViaLoadBalancer helper in test/e2e/util/oauth.go does not clean up the htpasswd Secret it creates — this is pre-existing and safe since each test variant uses a separate cluster, but worth a follow-up.

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests. No unit tests needed — changes are e2e test code and CI cluster configuration only, with no new business logic.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 25, 2026
@openshift-ci

openshift-ci Bot commented May 25, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@coderabbitai

coderabbitai Bot commented May 25, 2026

Copy link
Copy Markdown
Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: f22db274-8160-435f-9069-0ea21630c1d6

📥 Commits

Reviewing files that changed from the base of the PR and between 5043db2 and 93d6344.

📒 Files selected for processing (1)
  • hack/tools/hypershiftlinter/analyzers/e2eutilallowlist/e2eutilallowlist.go

Included review availability: Your plan provides up to 12 included reviews per hour; 9 remain after this review.


📝 Walkthrough

Walkthrough

Adds the private Azure oauth-lb-private cluster variant and its end-to-end test. The test validates private topology, internal LoadBalancer annotations, endpoint allocation, and OAuth access through OAuth and guest KAS port-forward transports. OAuth utilities now support configurable transports, endpoint-only readiness checks, consolidated identity-provider validation, configuration restoration, and reusable SPDY forwarding.

Sequence Diagram(s)

sequenceDiagram
  participant AzureOAuthLoadBalancerPrivateTest
  participant AzureLoadBalancerService
  participant OAuthPortForward
  participant GuestKASPortForward
  AzureOAuthLoadBalancerPrivateTest->>AzureLoadBalancerService: validate private LoadBalancer and ingress
  AzureOAuthLoadBalancerPrivateTest->>OAuthPortForward: configure OAuth transport
  AzureOAuthLoadBalancerPrivateTest->>GuestKASPortForward: configure guest KAS transport
  AzureOAuthLoadBalancerPrivateTest->>OAuthPortForward: validate OAuth identity-provider flow
  AzureOAuthLoadBalancerPrivateTest->>GuestKASPortForward: validate guest API access
Loading

Suggested reviewers: ironcladlou

Merge Risk: ⚪ Minimal · up to 93d63

The PR adds Azure private-topology OAuth load balancer coverage without any supplied actionable merge-blocking risk; it is merge-ready after normal checks and review.


Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error, 1 warning)

Check name Status Explanation Resolution
No-Sensitive-Data-In-Logs ❌ Error The pull request adds logs that can expose private cluster endpoints. SetupGuestKASPortForwardConfig logs serverName, which it extracts from the guest kubeconfig host, and the new Azure private te… Remove dynamic OAuth hostnames, KAS serverName values, and LoadBalancer ingress addresses from t.Logf, failure messages, and predicate reason strings. Use fixed status messages or sanitized identifiers instead. Confirm that the private …
Test Structure And Quality ⚠️ Warning The new private Azure test introduces unbounded cluster waits. test/e2e/v2/tests/suite_test.go initializes the shared test context with context.Background(). The new establishPodPortForward help… Bound port-forward readiness with an explicit timeout derived from ctx. Ensure the KAS connectivity request itself has a bounded timeout and honors the poll context, for example by using a context-aware REST request or setting a suitable …
✅ Passed checks (9 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: adding an Azure OAuth LoadBalancer end-to-end test for a private topology.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed No unstable test title was introduced. The Azure PR additions use static Ginkgo titles for the private topology context and its three specs, and the updated public title is also static. The lifecycle …
Topology-Aware Scheduling Compatibility ✅ Passed PASS: The PR changes only e2e tests, OAuth test utilities, and lifecycle cluster-creation configuration. It does not add or modify operator controllers, deployment manifests, or production workload sc…
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PASS — the new Azure Ginkgo suite does not introduce an IPv4-only assumption or public-internet dependency. The OAuth flow uses management-cluster API port-forwards to the OAuth and guest KAS pods, an…
No-Weak-Crypto ✅ Passed No custom-check failure was introduced. The PR diff adds no MD5, SHA-1, DES, RC4, 3DES, Blowfish, or ECB usage, and no cryptographic primitive calls. The new code uses standard TLS and SPDY transport …
Container-Privileges ✅ Passed PASS — The PR diff contains 10 changed Go files and no Kubernetes/container manifest files. No added lines set privileged: true, hostPID, hostNetwork, hostIPC, SYS_ADMIN, `allowPrivilegeEsca…
Full details: Stable And Deterministic Test Names

Explanation

No unstable test title was introduced. The Azure PR additions use static Ginkgo titles for the private topology context and its three specs, and the updated public title is also static. The lifecycle table uses fixed literal t.Run names. Dynamic endpoint data appears only in assertion/diagnostic code, not in It, Context, Describe, When, or subtest titles. The titles describe stable resources and behavior without generated suffixes, timestamps, UUIDs, node names, random namespaces, or IP addresses.

Full details: Test Structure And Quality

Explanation

The new private Azure test introduces unbounded cluster waits. test/e2e/v2/tests/suite_test.go initializes the shared test context with context.Background(). The new establishPodPortForward helper waits on readyCh with only ctx.Done() (test/e2e/util/oauth.go:645-655), so port-forward startup can block indefinitely. The KAS readiness poll has a five-minute poll timeout, but its testClient.Discovery().ServerVersion() call (oauth.go:800-804) does not use the poll context, and guestRestConfig does not set a REST timeout. The new OAuth port-forward helper also contains an assertion without a diagnostic message at the health-request construction (oauth.go:713).

Resolution

Bound port-forward readiness with an explicit timeout derived from ctx. Ensure the KAS connectivity request itself has a bounded timeout and honors the poll context, for example by using a context-aware REST request or setting a suitable timeout on the guest REST config. Use a timeout context for the htpasswd cleanup delete instead of context.Background(). Add a meaningful message to the new health-request assertion, such as failed to create OAuth health check request.

Full details: Topology-Aware Scheduling Compatibility

Explanation

PASS: The PR changes only e2e tests, OAuth test utilities, and lifecycle cluster-creation configuration. It does not add or modify operator controllers, deployment manifests, or production workload scheduling. The added initial replica values configure Azure test NodePools through --node-pool-replicas; they do not derive operator workload replicas from topology. No added diff lines introduce affinity, topology spread, node selectors, tolerations, PDBs, or control-plane/arbiter scheduling constraints. The existing testWorkload Deployment helper is unchanged from origin/main.

Full details: Ipv6 And Disconnected Network Test Compatibility

Explanation

PASS — the new Azure Ginkgo suite does not introduce an IPv4-only assumption or public-internet dependency. The OAuth flow uses management-cluster API port-forwards to the OAuth and guest KAS pods, and the only local endpoint is localhost, which is not a hardcoded IPv4 address. The OAuth URL uses oauthHost, but this value is the LoadBalancer Hostname, whose API validation requires a DNS domain name, not an IPv6 literal. No hardcoded IPv4 addresses, IPv4-only parsing, public hosts, downloads, or external API calls were added.

Full details: No-Weak-Crypto

Explanation

No custom-check failure was introduced. The PR diff adds no MD5, SHA-1, DES, RC4, 3DES, Blowfish, or ECB usage, and no cryptographic primitive calls. The new code uses standard TLS and SPDY transport setup. Credential handling remains HTTPS Basic authentication and bearer-token use. The only credential comparison is an identity-name assertion, not a secret or token comparison. The $2y$ htpasswd value is bcrypt test data and was already present in the base utility.

Full details: Container-Privileges

Explanation

PASS — The PR diff contains 10 changed Go files and no Kubernetes/container manifest files. No added lines set privileged: true, hostPID, hostNetwork, hostIPC, SYS_ADMIN, allowPrivilegeEscalation: true, or a root user. The added Karpenter test reuses a workload helper that sets RunAsUser: 1000 and AllowPrivilegeEscalation: false.

Full details: No-Sensitive-Data-In-Logs

Explanation

The pull request adds logs that can expose private cluster endpoints. SetupGuestKASPortForwardConfig logs serverName, which it extracts from the guest kubeconfig host, and the new Azure private test invokes this helper for a KAS behind an Azure Internal Load Balancer. WaitForOAuthLoadBalancerEndpoint logs the OAuth hostname and the LoadBalancer IP/hostname. The new oauthServiceLBPredicates also returns the ingress address in its predicate reason, and EventuallyObject writes predicate reasons to test logs. No password or token value is logged, but the internal-hostname condition is met.

Resolution

Remove dynamic OAuth hostnames, KAS serverName values, and LoadBalancer ingress addresses from t.Logf, failure messages, and predicate reason strings. Use fixed status messages or sanitized identifiers instead. Confirm that the private OAuth test output contains no internal endpoint values.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@openshift-ci openshift-ci Bot added area/platform/azure PR/issue for Azure (AzurePlatform) platform area/testing Indicates the PR includes changes for e2e testing and removed do-not-merge/needs-area labels May 25, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
test/e2e/v2/tests/hosted_cluster_azure_test.go (1)

304-320: ⚡ Quick win

Move platform/topology guard skips to BeforeEach with standardized skip text.

In Line 304-320, the new suite uses BeforeAll and custom skip messages for platform checks. Please switch to BeforeEach and the repository’s required message format for platform-specific skips.

♻️ Suggested adjustment
-		BeforeAll(func() {
+		BeforeEach(func() {
 			testCtx = getTestCtx()
 			hc = testCtx.GetHostedCluster()
 			if hc == nil || hc.Spec.Platform.Type != hyperv1.AzurePlatform {
-				Skip("Azure OAuth LB Private tests are only for Azure platform")
+				Skip("[Azure OAuth LB Private] test is only for Azure platform")
 			}
 			if hc.Spec.Platform.Azure == nil || hc.Spec.Platform.Azure.Topology != hyperv1.AzureTopologyPrivate {
 				Skip("Azure OAuth LB Private tests require Private topology")
 			}

As per coding guidelines, "test/e2e/v2/tests/**/*_test.go: Use BeforeEach with Skip() for platform-specific tests with message format: [Feature] test is only for [Platform] platform".

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/v2/tests/hosted_cluster_azure_test.go` around lines 304 - 320,
Replace the platform/topology guard block currently inside BeforeAll with a
BeforeEach that performs the same checks (use
getTestCtx()/testCtx.GetHostedCluster(), validate hc.Spec.Platform.Type,
hc.Spec.Platform.Azure.Topology, and
netutil.ServicePublishingStrategyByTypeByHC(hc, hyperv1.OAuthServer)), move
controlPlaneNamespace assignment into this BeforeEach, and call Skip() with the
repository-standard messages: for platform use "Azure OAuth LB Private tests is
only for Azure platform" and for topology use "Azure OAuth LB Private tests
requires Private topology" (also keep the existing Skip for publishing strategy
if present); update references to BeforeAll -> BeforeEach and keep the same
symbol names (getTestCtx, testCtx.GetHostedCluster, controlPlaneNamespace,
netutil.ServicePublishingStrategyByTypeByHC, hyperv1.OAuthServer).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@test/e2e/v2/tests/hosted_cluster_azure_test.go`:
- Around line 304-320: Replace the platform/topology guard block currently
inside BeforeAll with a BeforeEach that performs the same checks (use
getTestCtx()/testCtx.GetHostedCluster(), validate hc.Spec.Platform.Type,
hc.Spec.Platform.Azure.Topology, and
netutil.ServicePublishingStrategyByTypeByHC(hc, hyperv1.OAuthServer)), move
controlPlaneNamespace assignment into this BeforeEach, and call Skip() with the
repository-standard messages: for platform use "Azure OAuth LB Private tests is
only for Azure platform" and for topology use "Azure OAuth LB Private tests
requires Private topology" (also keep the existing Skip for publishing strategy
if present); update references to BeforeAll -> BeforeEach and keep the same
symbol names (getTestCtx, testCtx.GetHostedCluster, controlPlaneNamespace,
netutil.ServicePublishingStrategyByTypeByHC, hyperv1.OAuthServer).

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 4942b228-db2d-42ed-83bf-2dd3738e5e94

📥 Commits

Reviewing files that changed from the base of the PR and between 7625684 and b8a5e30.

📒 Files selected for processing (2)
  • test/e2e/v2/lifecycle/azure.go
  • test/e2e/v2/tests/hosted_cluster_azure_test.go

@codecov

codecov Bot commented May 25, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 47.45%. Comparing base (2f8ff64) to head (59e06b9).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #8584   +/-   ##
=======================================
  Coverage   47.45%   47.45%           
=======================================
  Files         795      795           
  Lines       99839    99839           
=======================================
  Hits        47383    47383           
  Misses      49292    49292           
  Partials     3164     3164           
Flag Coverage Δ
cmd-support 41.08% <ø> (ø)
cpo-hostedcontrolplane 50.50% <ø> (ø)
cpo-other 48.38% <ø> (ø)
hypershift-operator 57.68% <ø> (ø)
other 34.70% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Nirshal

Nirshal commented May 26, 2026

Copy link
Copy Markdown
Contributor Author

Re: CodeRabbit suggestion to change BeforeAll to BeforeEach — intentionally keeping BeforeAll here.

This Context uses the Ordered decorator with shared state (testCtx, hc, controlPlaneNamespace) across three sequential It blocks. BeforeAll runs once and initializes the shared variables; BeforeEach would redundantly re-fetch the test context and re-run the guards before every It.

This matches the existing pattern in AzurePrivateTopologyTest (same file, line 91) which also uses Ordered + BeforeAll for the same reason.

@Nirshal
Nirshal marked this pull request as ready for review May 26, 2026 13:56
@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 26, 2026
@openshift-ci
openshift-ci Bot requested review from cblecker and sjenning May 26, 2026 13:58

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e/v2/tests/hosted_cluster_azure_test.go`:
- Around line 379-382: The test calls
ValidateOAuthWithIdentityProviderViaLoadBalancer which mutates the HostedCluster
OAuth config and creates an htpasswd Secret without restoring them; capture the
original HostedCluster.Spec.OAuth (and any other mutated fields) and the
existence/data of the htpasswd Secret before calling
ValidateOAuthWithIdentityProviderViaLoadBalancer, then defer a restoration
function that re-applies the original OAuth config and deletes or restores the
htpasswd Secret, and in that deferred cleanup handle apierrors.IsNotFound(error)
when attempting to restore/delete resources so cleanup tolerates resources
already removed; use the HostedCluster object and the Secret name returned/used
by ValidateOAuthWithIdentityProviderViaLoadBalancer to identify what to restore.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: bcc13421-364c-476a-94f1-73cf7c97a172

📥 Commits

Reviewing files that changed from the base of the PR and between b8a5e30 and 5d52eb0.

📒 Files selected for processing (2)
  • test/e2e/v2/lifecycle/azure.go
  • test/e2e/v2/tests/hosted_cluster_azure_test.go

Comment on lines +379 to +382
It("should complete OAuth token flow through LoadBalancer endpoint", func() {
ctx := testCtx.Context
e2eutil.ValidateOAuthWithIdentityProviderViaLoadBalancer(GinkgoTB(), ctx, testCtx.MgmtClient, hc)
})

@coderabbitai coderabbitai Bot May 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Restore cluster auth state after OAuth flow mutation.

Line 381 calls ValidateOAuthWithIdentityProviderViaLoadBalancer, which mutates HostedCluster OAuth config and creates an htpasswd Secret without restoring/deleting them. This makes outcomes order-dependent on shared clusters. Please capture original state and defer restoration, with apierrors.IsNotFound() handling during cleanup.

As per coding guidelines: "When a test mutates cluster state, capture original state before mutation and defer restoration; check apierrors.IsNotFound() in cleanup to handle already-deleted resources".

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/v2/tests/hosted_cluster_azure_test.go` around lines 379 - 382, The
test calls ValidateOAuthWithIdentityProviderViaLoadBalancer which mutates the
HostedCluster OAuth config and creates an htpasswd Secret without restoring
them; capture the original HostedCluster.Spec.OAuth (and any other mutated
fields) and the existence/data of the htpasswd Secret before calling
ValidateOAuthWithIdentityProviderViaLoadBalancer, then defer a restoration
function that re-applies the original OAuth config and deletes or restores the
htpasswd Secret, and in that deferred cleanup handle apierrors.IsNotFound(error)
when attempting to restore/delete resources so cleanup tolerates resources
already removed; use the HostedCluster object and the Secret name returned/used
by ValidateOAuthWithIdentityProviderViaLoadBalancer to identify what to restore.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The mutation happens inside the shared helper ValidateOAuthWithIdentityProviderViaLoadBalancer (in test/e2e/util/oauth.go), which is also used by the existing AzureOAuthLoadBalancerTest (line 284, same file) without cleanup. Adding cleanup in the caller would mean duplicating restore logic that belongs in the helper itself.

This is a pre-existing concern, not introduced by this PR. Noted in the PR description as a known tradeoff. If we address it, it should be done in the shared helper as a follow-up.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Nirshal, that's a fair point — the concern is pre-existing and your reasoning is correct. Duplicating restore logic in each caller would be the wrong approach; the cleanup belongs inside ValidateOAuthWithIdentityProviderViaLoadBalancer itself so all callers benefit automatically.

Would you like me to open a follow-up GitHub issue to track adding htpasswd Secret cleanup (and OAuth config restoration) to the shared helper?


🧠 Learnings used
Learnt from: cblecker
Repo: openshift/hypershift PR: 8511
File: test/e2e/v2/tests/hosted_cluster_metrics_test.go:203-245
Timestamp: 2026-05-15T03:34:00.446Z
Learning: When reviewing HyperShift v2 e2e tests under test/e2e/v2/, do not require MicroShift “guard” logic or markers. HyperShift v2 e2e tests in this directory run only against hosted clusters (no MicroShift targets), so missing exutil.IsMicroShiftCluster() checks and/or [Skipped:MicroShift] labels should not be flagged.

Learnt from: bryan-cox
Repo: openshift/hypershift PR: 8527
File: test/e2e/v2/tests/nodepool_autoscaling_test.go:236-236
Timestamp: 2026-05-15T14:34:25.062Z
Learning: In openshift/hypershift v2 e2e lifecycle CI tests under test/e2e/v2/tests/, it is OK to reference external image registries because the CI environment is not air-gapped. Image references like registry.access.redhat.com/ubi9/ubi-minimal:latest in lifecycle DaemonSet definitions should not be flagged for requiring internal registries. For the autoscaling workload created by newAutoscalingWorkload (nodepool_autoscaling_test.go), ensure it uses registry.k8s.io/pause:3.9 (not ubi-minimal).

Learnt from: bryan-cox
Repo: openshift/hypershift PR: 8469
File: test/e2e/util/util_test.go:17-131
Timestamp: 2026-05-17T22:37:02.057Z
Learning: In Go e2e tests under test/e2e, do not call t.Parallel() (neither at the top-level test nor in any subtest) when the test (or any helper it calls) uses t.Setenv—for example, when calling azureutil.SetAsAroHCPTest(t). The Go test runner panics with "testing: test using t.Setenv can not use t.Parallel" because environment variables are process-global. Keep such tests serial to ensure the environment setup is safe and deterministic.

@Nirshal

Nirshal commented May 26, 2026

Copy link
Copy Markdown
Contributor Author

/test ?

1 similar comment
@Nirshal

Nirshal commented May 26, 2026

Copy link
Copy Markdown
Contributor Author

/test ?

@Nirshal

Nirshal commented May 26, 2026

Copy link
Copy Markdown
Contributor Author

/test e2e-azure-v2-self-managed

@cblecker cblecker left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test structure itself is solid — good use of Ordered + BeforeAll, thorough skip guards, and the internal LB annotation check is a meaningful addition for private topology coverage.

Two things to address before this can merge:

1. Healthcheck unreachable from CI (blocking): WaitForOAuthLoadBalancerReady dials the OAuth hostname directly via TLS, which resolves to the internal LB IP for private topology — unreachable from the build cluster. The CI run confirmed this (job 2059278516129632256, oauth.go:375, 300s timeout). The token flow step worked; it's only the raw healthcheck that fails. Fix needs to happen in the shared helper.

2. Test isolation on the shared cluster: The public AzureOAuthLoadBalancerTest works because it has a dedicated cluster. This private variant shares a cluster with read-only tests and relies on declaration order for isolation. Either a dedicated cluster, or a sequential matrix step, would give the same guarantee the public variant has.

Minor nit: the Service type + endpoint predicates in the first It block are nearly identical to the public variant (~30 lines). Not blocking, but worth a shared helper if more LB variants get added.

It("should complete OAuth token flow through LoadBalancer endpoint", func() {
ctx := testCtx.Context
e2eutil.ValidateOAuthWithIdentityProviderViaLoadBalancer(GinkgoTB(), ctx, testCtx.MgmtClient, hc)
})

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CI run showed this test can't pass as-is — WaitForOAuthLoadBalancerReady (in test/e2e/util/oauth.go:350-375) does a direct TLS dial to https://<oauth-hostname>/healthz. For private topology that hostname resolves to the Azure internal LB IP (e.g., 10.0.0.22), which is unreachable from the CI build cluster.

Interestingly the OAuth token flow itself succeeds because it routes through the guest kubeconfig transport — it's specifically the raw healthcheck that fails. So the underlying validation logic works; it's just the connectivity check that's incompatible with private topology.

The fix needs to happen in the shared helper. A few approaches:

  1. Skip WaitForOAuthLoadBalancerReady entirely for private topology and rely on the token flow validation alone — it already exercises the full OAuth stack end-to-end.
  2. Use the guest kubeconfig transport for the healthcheck too, the same way the token flow step does.
  3. Route it through a pod on the management cluster that has VNet connectivity.

This needs to be addressed in the helper before this can land.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in c274044. Extracted WaitForOAuthLoadBalancerEndpoint from WaitForOAuthLoadBalancerReady — it waits for the LB endpoint without the direct /healthz TLS dial. The private test now calls WaitForOAuthLoadBalancerEndpoint + ValidateOAuthIdentityProviderFlow (also extracted), skipping the health check entirely while still validating the full OAuth IDP token flow end-to-end.

WaitForOAuthLoadBalancerReady is unchanged in behavior — it now composes WaitForOAuthLoadBalancerEndpoint + the health check, so the public test path is unaffected.

AzureOAuthLoadBalancerPrivateTest(getTestCtx)
}

var _ = Describe("Hosted Cluster Azure", Label("hosted-cluster-azure"), func() {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The public variant (AzureOAuthLoadBalancerTest) works cleanly because it runs on a dedicated cluster (cluster-name-oauth-lb) created specifically for it — mutation is safe there since nothing else runs on that cluster. This private variant is trying to replicate that pattern but without the dedicated cluster, sharing cluster-name-private with the read-only AzurePrivateTopologyTest instead.

Registering this last is a reasonable mitigation, but declaration order between Context blocks isn't enforced by Ginkgo's Ordered decorator — that only applies within a single Context. If the registration order changes, or a new test gets inserted between the two, the read-only private topology tests could run after the HC has been mutated (htpasswd IDP created, kubeadmin secret deleted).

The cleanest fix that matches how the public variant is structured would be a dedicated cluster-name-oauth-lb-private cluster. I understand the reviewer feedback to reduce cluster count, so if a dedicated cluster is off the table, the other option is to make this a sequential step in the test matrix (after the read-only private tests), the way etcd-chaos runs sequentially after upgrade on the upgrade cluster. That would give the same isolation guarantee without an extra cluster.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in c274044. Moved the private test group from Parallel to Sequential in the test matrix, following the same pattern as upgrade-and-chaos. The new private-and-oauth sequential group runs read-only private topology tests first, then the mutating OAuth LB test second — guaranteeing isolation regardless of registration order.

@Nirshal

Nirshal commented Jun 17, 2026

Copy link
Copy Markdown
Contributor Author

/test e2e-azure-v2-self-managed

@Nirshal

Nirshal commented Jun 17, 2026

Copy link
Copy Markdown
Contributor Author

CI failure on e2e-azure-v2-self-managed was an infrastructure issue — the hypershift-install step timed out after ~5m due to API server rate limiting (client rate limiter Wait returned an error: context deadline exceeded). The HyperShift operator never deployed, so no test clusters were created and none of our test code ran. Retriggering.

/test e2e-azure-v2-self-managed

@cblecker cblecker left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both previous blocking items are addressed — thank you for the updates! One minor annotation fix needed for CI tooling.

@openshift-ci openshift-ci Bot added the area/ci-tooling Indicates the PR includes changes for CI or tooling label Sep 2, 2026
@openshift-ci

openshift-ci Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bryan-cox, Nirshal

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@Nirshal

Nirshal commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

/test e2e-v2-azure-self-managed

@clebs

clebs commented Sep 8, 2026

Copy link
Copy Markdown
Member

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Sep 8, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks-5-0
/test e2e-aws-5-0
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-azure-self-managed
/test e2e-v2-gke

@openshift-ci openshift-ci Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 8, 2026
@openshift-ci openshift-ci Bot removed the lgtm Indicates that a PR is ready to be merged. label Sep 9, 2026
@openshift-ci openshift-ci Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 9, 2026
@clebs

clebs commented Sep 9, 2026

Copy link
Copy Markdown
Member

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Sep 9, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks-5-0
/test e2e-aws-5-0
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-azure-self-managed
/test e2e-v2-gke

@Nirshal

Nirshal commented Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

/test e2e-v2-aws

@bryan-cox

Copy link
Copy Markdown
Member

/verified by e2e

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Sep 10, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@bryan-cox: This PR has been marked as verified by e2e.

Details

In response to this:

/verified by e2e

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

/retest-required

Remaining retests: 0 against base HEAD 2f8ff64 and 2 for PR HEAD cc37897 in total

Add an e2e test for the oauth-lb-private cluster variant, which
validates the OAuth identity provider flow on Azure private clusters
with oauthPublishingStrategy=LoadBalancer.

The test creates a dedicated oauth-lb-private cluster variant,
configures an htpasswd identity provider, and validates token
acquisition through port-forwarded connections to both the OAuth
server and guest KAS, since direct network access is not available
on private topology clusters.

Key changes:
- Add oauth-lb-private cluster variant in azure.go TestMatrix
- Add port-forward helpers for OAuth server and guest KAS with
  DeletionTimestamp filtering and polling for pod readiness
- Register cleanup before readiness assertions to prevent
  goroutine leaks on test abort
- Use per-cluster htpasswd secret names to avoid collision
- Re-establish port-forward after OAuth server rollout
- Adopt ClusterManifest pattern from Dan Mace's refactor

Signed-off-by: Alessandro Rossi <alesross@redhat.com>
Commit-Message-Assisted-by: Claude (via Claude Code)
…utine leaks

Move t.Cleanup registration into establishPodPortForward using a
sync.Once-wrapped stop function so early assertion failures always
close the forwarder. Use t.Fatalf for the errCh readiness case to
fail unconditionally (including nil error from a dead tunnel).
Callers use the idempotent pf.stop() for re-establishment.

Signed-off-by: Alessandro Rossi <alesross@redhat.com>
Commit-Message-Assisted-by: Claude (via Claude Code)
@openshift-ci-robot openshift-ci-robot removed the verified Signifies that the PR passed pre-merge verification criteria label Sep 10, 2026
@openshift-ci openshift-ci Bot removed the lgtm Indicates that a PR is ready to be merged. label Sep 10, 2026
@clebs

clebs commented Sep 10, 2026

Copy link
Copy Markdown
Member

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Sep 10, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks-5-0
/test e2e-aws-5-0
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-azure-self-managed
/test e2e-v2-gke

@openshift-ci

openshift-ci Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

@Nirshal: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-azure-v2-self-managed 9d20619 link true /test e2e-azure-v2-self-managed
ci/prow/e2e-kubevirt-aws-ovn-reduced 59e06b9 link true /test e2e-kubevirt-aws-ovn-reduced
ci/prow/e2e-v2-aws 59e06b9 link true /test e2e-v2-aws
ci/prow/e2e-aks 59e06b9 link true /test e2e-aks
ci/prow/images cc37897 link unknown /test images
ci/prow/e2e-aks-5-0 59e06b9 link true /test e2e-aks-5-0
ci/prow/e2e-v2-azure-self-managed 59e06b9 link true /test e2e-v2-azure-self-managed
ci/prow/e2e-aws-5-0 59e06b9 link true /test e2e-aws-5-0
ci/prow/e2e-aws-upgrade-hypershift-operator 59e06b9 link true /test e2e-aws-upgrade-hypershift-operator
ci/prow/e2e-aws 59e06b9 link true /test e2e-aws
ci/prow/e2e-v2-gke 59e06b9 link true /test e2e-v2-gke

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/ci-tooling Indicates the PR includes changes for CI or tooling area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release area/platform/azure PR/issue for Azure (AzurePlatform) platform area/testing Indicates the PR includes changes for e2e testing jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants