Skip to content

GCP-429: Add e2e tests for CNCC GCP credential authentication methods - #30832

Open
apahim wants to merge 3 commits into
openshift:mainfrom
apahim:cncc-tests
Open

GCP-429: Add e2e tests for CNCC GCP credential authentication methods#30832
apahim wants to merge 3 commits into
openshift:mainfrom
apahim:cncc-tests

Conversation

@apahim

@apahim apahim commented Mar 4, 2026

Copy link
Copy Markdown

Validates the three-tier GCP credential fallback chain (WIF → SA → GOOGLE_APPLICATION_CREDENTIALS) with six test cases covering smoke verification, credential rotation, WIF priority, invalid credentials, secret deletion/recovery, and IP management persistence across rotation.

Summary by CodeRabbit

  • Tests
    • Added comprehensive end-to-end CNCC GCP credential tests covering credential rotation, WIF vs service-account JSON priority, invalid-credential behavior, secret deletion/recovery, and recovery after restarts.
    • Validates CNCC readiness, pod restart and log checks, and verifies cloud IP management via an egress-IP workflow — ensuring CloudPrivateIPConfig creation, assignment, and persistence through credential rotations.

@openshift-ci-robot

Copy link
Copy Markdown

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Mar 4, 2026
@openshift-ci-robot

openshift-ci-robot commented Mar 4, 2026

Copy link
Copy Markdown

@apahim: This pull request references GCP-429 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Validates the three-tier GCP credential fallback chain (WIF → SA → GOOGLE_APPLICATION_CREDENTIALS) with six test cases covering smoke verification, credential rotation, WIF priority, invalid credentials, secret deletion/recovery, and IP management persistence across rotation.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci
openshift-ci Bot requested review from martinkennelly and tssurya March 4, 2026 14:52
@coderabbitai

coderabbitai Bot commented Mar 4, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are limited based on label configuration.

🚫 Review skipped — only excluded labels are configured. (1)
  • do-not-merge/work-in-progress

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 767257b6-e825-4b9d-b402-2388a4fc2e1b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

Introduces a new end-to-end Go test for CNCC on GCP that snapshots/restores the cloud-credentials secret, detects credential type (WIF/SA), verifies CNCC readiness and restart behavior, exercises credential rotation and invalid-secret handling, tests secret deletion/recovery, and validates EgressIP-driven CloudPrivateIPConfig workflows.

Changes

Cohort / File(s) Summary
CNCC GCP Credentials Test Suite
test/extended/networking/cncc_gcp_credentials.go
Adds a ~500-line end-to-end test implementing CNCC GCP credential scenarios: platform gating (GCP-only), client initialization, cloud-credentials secret snapshot/restore, credential type detection (WIF vs SA), CNCC readiness and pod restart/crash checks, credential rotation tests, invalid-credential handling, secret deletion/recovery, EgressIP -> CloudPrivateIPConfig allocation/assignment workflow, and helper utilities (logs, restart counts, polling, createEgressIPFromJSON). No exported API changes.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Test Structure And Quality ⚠️ Warning Test file has critical nil pointer dereference risk in AfterEach due to uninitialized variables when BeforeEach skips, plus 20+ assertions lack descriptive messages, and multiple tests violate single responsibility principle. Add guard condition in AfterEach to check if clientset and originalSecret are nil before use. Add descriptive messages to all Expect() assertions. Split multi-behavior tests into focused single-responsibility tests.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'GCP-429: Add e2e tests for CNCC GCP credential authentication methods' directly and clearly describes the main change: adding end-to-end tests for CNCC GCP credential authentication.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Stable And Deterministic Test Names ✅ Passed All Ginkgo test titles (It, Describe declarations) in the test file are stable and deterministic with no dynamic content.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci-robot

openshift-ci-robot commented Mar 4, 2026

Copy link
Copy Markdown

@apahim: This pull request references GCP-429 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Validates the three-tier GCP credential fallback chain (WIF → SA → GOOGLE_APPLICATION_CREDENTIALS) with six test cases covering smoke verification, credential rotation, WIF priority, invalid credentials, secret deletion/recovery, and IP management persistence across rotation.

Summary by CodeRabbit

  • Tests
  • Added comprehensive end-to-end test suite for CNCC GCP credentials, validating credential rotation, WIF and service account JSON support, failure recovery, and IP configuration resilience.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@test/extended/networking/cncc_gcp_credentials.go`:
- Line 329: The test fixture embeds a PEM-like private key in the dummySA string
which triggers secret scanners; update the dummySA fixture (and the similar
occurrence at the other spot) to remove any real-looking PEM content and replace
the "private_key" value with a non-secret placeholder (e.g. a short
"<REDACTED_PRIVATE_KEY>" or clearly fake token) or build the JSON
programmatically without PEM text so the variable dummySA no longer contains PEM
markers like "-----BEGIN" / "-----END"; keep the same JSON keys and structure so
tests still parse but do not include PEM-looking secrets.
- Around line 70-87: The restore flow currently treats any error from
clientset.CoreV1().Secrets(cnccNamespace).Get(...) as a deleted secret; change
it to first check if the error is a NotFound (use apierrors.IsNotFound(err)) and
only then create the new secret with cnccSecretName/cnccNamespace; for all other
errors return/fail the test (o.Expect(err).NotTo(o.HaveOccurred(...)) or
propagate the error) instead of attempting Create — keep the existing Update
path for the else case where Get succeeds, and reference the same
clientset.CoreV1().Secrets(...).Get, Create and Update calls and the o.Expect
assertions to place the checks.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 23cc057b-129f-4c25-b816-2e98201af7cc

📥 Commits

Reviewing files that changed from the base of the PR and between aa56844 and d992119.

📒 Files selected for processing (1)
  • test/extended/networking/cncc_gcp_credentials.go

Comment thread test/extended/networking/cncc_gcp_credentials.go
Comment thread test/extended/networking/cncc_gcp_credentials.go Outdated
@openshift-ci-robot

openshift-ci-robot commented Mar 4, 2026

Copy link
Copy Markdown

@apahim: This pull request references GCP-429 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Validates the three-tier GCP credential fallback chain (WIF → SA → GOOGLE_APPLICATION_CREDENTIALS) with six test cases covering smoke verification, credential rotation, WIF priority, invalid credentials, secret deletion/recovery, and IP management persistence across rotation.

Summary by CodeRabbit

  • Tests
  • Added comprehensive end-to-end tests for CNCC GCP credentials covering credential rotation, WIF vs service-account JSON priority, invalid-credential behaviors, secret deletion/recovery, and system resilience.
  • Includes validation of CNCC readiness, IP management via an egress IP workflow, and verification that Cloud IP assignments persist through credential rotations and restarts.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (2)
test/extended/networking/cncc_gcp_credentials.go (2)

50-57: ⚠️ Potential issue | 🟠 Major

Replace PEM-like private key literals with non-secret placeholders.

The dummySAJSON and invalidSAJSON constants contain PEM-formatted private key strings that trigger secret scanners (Gitleaks flagged both). While these are test fixtures, they will cause CI noise and security triage overhead.

Replace the private_key values with clearly fake placeholders:

Suggested fix
-	dummySAJSON = `{"type":"service_account","project_id":"dummy-project","private_key_id":"dummy","private_key":"-----BEGIN RSA PRIVATE KEY-----\nMIIBogIBAAJBALRo/7FBMHpMx2ls9k5A5feDteQa4ARWV5jNz9/v3mPpM4CPYB/m\nmivnN4g3a3+0kqe5W9USNWQ/lC3qLNjdXnECAwEAAQJAC/wG4sB/YxYILqLywb11\nNpviLqQgeKi7ZXSXT1OBF0gU8bB0OP8OLjkKGfhSW3niqxoM1RRoLvNsL7A0mBp\nwQIhAO0/GNXSdA98JxmNxVdIjPTCSAP6XEoH3R8WFiJ0JBSvAiEAxjn0EDBFKOI7\nhBk1RXlTfMHbDQHrPKFmL7y2GIPBYU8CIGl9a3PJ2QeB2Nz3IlJjSPjQ5U0ASNS\nSN1GMH2eJwYnAiEAoRxE6QUCRzBL7F3l/Dq+IYbwxZkN95Nf8CPm8GjYHD8CIQC8\nuqbJMxbRbhsPNi88e1ZYayVjPhFKEXZfCihqT/YE1Q==\n-----END RSA PRIVATE KEY-----\n","client_email":"dummy@dummy-project.iam.gserviceaccount.com","client_id":"123456789","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token"}`
+	dummySAJSON = `{"type":"service_account","project_id":"dummy-project","private_key_id":"dummy","private_key":"not-a-real-key","client_email":"dummy@dummy-project.iam.gserviceaccount.com","client_id":"123456789","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token"}`

-	invalidSAJSON = `{"type":"service_account","project_id":"invalid-project","private_key_id":"invalid","private_key":"-----BEGIN RSA PRIVATE KEY-----\nMIIBogIBAAJBALRo/7FBMHpMx2ls9k5A5feDteQa4ARWV5jNz9/v3mPpM4CPYB/m\nmivnN4g3a3+0kqe5W9USNWQ/lC3qLNjdXnECAwEAAQJAC/wG4sB/YxYILqLywb11\nNpviLqQgeKi7ZXSXT1OBF0gU8bB0OP8OLjkKGfhSW3niqxoM1RRoLvNsL7A0mBp\nwQIhAO0/GNXSdA98JxmNxVdIjPTCSAP6XEoH3R8WFiJ0JBSvAiEAxjn0EDBFKOI7\nhBk1RXlTfMHbDQHrPKFmL7y2GIPBYU8CIGl9a3PJ2QeB2Nz3IlJjSPjQ5U0ASNS\nSN1GMH2eJwYnAiEAoRxE6QUCRzBL7F3l/Dq+IYbwxZkN95Nf8CPm8GjYHD8CIQC8\nuqbJMxbRbhsPNi88e1ZYayVjPhFKEXZfCihqT/YE1Q==\n-----END RSA PRIVATE KEY-----\n","client_email":"invalid@invalid-project.iam.gserviceaccount.com","client_id":"000000000","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token"}`
+	invalidSAJSON = `{"type":"service_account","project_id":"invalid-project","private_key_id":"invalid","private_key":"not-a-real-key","client_email":"invalid@invalid-project.iam.gserviceaccount.com","client_id":"000000000","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token"}`
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/extended/networking/cncc_gcp_credentials.go` around lines 50 - 57, The
dummySAJSON and invalidSAJSON test fixtures include real-looking PEM private_key
literals that trigger secret scanners; update the JSON string values for the
"private_key" fields in both dummySAJSON and invalidSAJSON to use clearly fake
placeholders (e.g., a short non-sensitive token or a clearly marked placeholder
like "<PRIVATE_KEY_PLACEHOLDER>" or "-----BEGIN PRIVATE
KEY-----\nFAKE-KEY\n-----END PRIVATE KEY-----") so the JSON remains
syntactically valid but contains no PEM-like secret material; ensure you only
change the private_key content within the dummySAJSON and invalidSAJSON
constants.

78-98: ⚠️ Potential issue | 🟠 Major

Differentiate NotFound from other secret-read errors in restore flow.

The code treats any Get() error as if the secret was deleted (line 79), without distinguishing NotFound from actual failures like permission denied or API server issues. This hides real failures and triggers incorrect remediation.

Suggested fix
+import (
+    apierrors "k8s.io/apimachinery/pkg/api/errors"
+)
...
 existing, err := clientset.CoreV1().Secrets(cnccNamespace).Get(context.Background(), cnccSecretName, metav1.GetOptions{})
 if err != nil {
+    if !apierrors.IsNotFound(err) {
+        o.Expect(err).NotTo(o.HaveOccurred(), "failed to read CNCC secret during restore")
+        return
+    }
     // Secret was deleted; recreate it.
     newSecret := &corev1.Secret{
         ...
     }
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/extended/networking/cncc_gcp_credentials.go` around lines 78 - 98, The
restore flow currently treats any error from
clientset.CoreV1().Secrets(cnccNamespace).Get(...) as a missing secret; change
it to distinguish a NotFound error using apierrors.IsNotFound(err) (from
k8s.io/apimachinery/pkg/api/errors) and only recreate the secret (newSecret with
Name cnccSecretName, Namespace cnccNamespace, Annotations/Labels/Data/Type from
originalSecret) in that case; for any other Get() error, fail the test or
surface the error via o.Expect(err).NotTo(o.HaveOccurred(), ...) (or return the
error) instead of attempting to recreate, ensuring existing is only used when
Get() succeeds and originalSecret/existing are preserved as before.
🧹 Nitpick comments (2)
test/extended/networking/cncc_gcp_credentials.go (2)

447-449: Clarify node label value intention.

Line 448 sets k8s.ovn.org/egress-assignable= with an empty value. While this may work (OVN-Kubernetes typically just checks for label presence), explicitly setting =true is more conventional and self-documenting.

Suggested fix
-		_, err = oc.AsAdmin().Run("label").Args("node", workerNodes[0].Name, "k8s.ovn.org/egress-assignable=", "--overwrite").Output()
+		_, err = oc.AsAdmin().Run("label").Args("node", workerNodes[0].Name, "k8s.ovn.org/egress-assignable=true", "--overwrite").Output()
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/extended/networking/cncc_gcp_credentials.go` around lines 447 - 449, The
label call that presently sets "k8s.ovn.org/egress-assignable=" with an empty
value should explicitly set a boolean value; update the
oc.AsAdmin().Run("label").Args invocation that targets workerNodes[0].Name to
use "k8s.ovn.org/egress-assignable=true" (and keep --overwrite) so the label
value is explicit and self-documenting while preserving the existing error check
after Output().

200-219: Use defer for logStream.Close() to prevent resource leaks.

If logStream.Read() panics, the stream won't be closed. Using defer ensures cleanup regardless of how the function exits.

Suggested fix
 		logStream, err := req.Stream(context.Background())
 		if err != nil {
 			framework.Logf("Failed to get logs for pod %s: %v", pod.Name, err)
 			continue
 		}
+		defer logStream.Close()
 		buf := new(strings.Builder)
 		_, _ = fmt.Fprintf(buf, "--- Logs from pod %s ---\n", pod.Name)
 		b := make([]byte, 4096)
 		for {
 			n, readErr := logStream.Read(b)
 			if n > 0 {
 				buf.Write(b[:n])
 			}
 			if readErr != nil {
 				break
 			}
 		}
-		logStream.Close()
 		allLogs.WriteString(buf.String())
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/extended/networking/cncc_gcp_credentials.go` around lines 200 - 219, The
log stream opened by req.Stream(...) is closed only at the end and can leak if
Read panics or returns early; immediately after acquiring logStream in the block
using clientset.CoreV1().Pods(cnccNamespace).GetLogs(...).Stream, check err as
you already do and then call defer logStream.Close() (instead of the later
explicit logStream.Close()) so the stream is always closed; remove or avoid the
later explicit Close call to prevent double-close and ensure buf and allLogs
logic remains unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In `@test/extended/networking/cncc_gcp_credentials.go`:
- Around line 50-57: The dummySAJSON and invalidSAJSON test fixtures include
real-looking PEM private_key literals that trigger secret scanners; update the
JSON string values for the "private_key" fields in both dummySAJSON and
invalidSAJSON to use clearly fake placeholders (e.g., a short non-sensitive
token or a clearly marked placeholder like "<PRIVATE_KEY_PLACEHOLDER>" or
"-----BEGIN PRIVATE KEY-----\nFAKE-KEY\n-----END PRIVATE KEY-----") so the JSON
remains syntactically valid but contains no PEM-like secret material; ensure you
only change the private_key content within the dummySAJSON and invalidSAJSON
constants.
- Around line 78-98: The restore flow currently treats any error from
clientset.CoreV1().Secrets(cnccNamespace).Get(...) as a missing secret; change
it to distinguish a NotFound error using apierrors.IsNotFound(err) (from
k8s.io/apimachinery/pkg/api/errors) and only recreate the secret (newSecret with
Name cnccSecretName, Namespace cnccNamespace, Annotations/Labels/Data/Type from
originalSecret) in that case; for any other Get() error, fail the test or
surface the error via o.Expect(err).NotTo(o.HaveOccurred(), ...) (or return the
error) instead of attempting to recreate, ensuring existing is only used when
Get() succeeds and originalSecret/existing are preserved as before.

---

Nitpick comments:
In `@test/extended/networking/cncc_gcp_credentials.go`:
- Around line 447-449: The label call that presently sets
"k8s.ovn.org/egress-assignable=" with an empty value should explicitly set a
boolean value; update the oc.AsAdmin().Run("label").Args invocation that targets
workerNodes[0].Name to use "k8s.ovn.org/egress-assignable=true" (and keep
--overwrite) so the label value is explicit and self-documenting while
preserving the existing error check after Output().
- Around line 200-219: The log stream opened by req.Stream(...) is closed only
at the end and can leak if Read panics or returns early; immediately after
acquiring logStream in the block using
clientset.CoreV1().Pods(cnccNamespace).GetLogs(...).Stream, check err as you
already do and then call defer logStream.Close() (instead of the later explicit
logStream.Close()) so the stream is always closed; remove or avoid the later
explicit Close call to prevent double-close and ensure buf and allLogs logic
remains unchanged.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 85ed3122-0f3b-4687-9c74-7eb01c306201

📥 Commits

Reviewing files that changed from the base of the PR and between d992119 and d90ffc5.

📒 Files selected for processing (1)
  • test/extended/networking/cncc_gcp_credentials.go

@openshift-ci-robot

openshift-ci-robot commented Mar 4, 2026

Copy link
Copy Markdown

@apahim: This pull request references GCP-429 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Validates the three-tier GCP credential fallback chain (WIF → SA → GOOGLE_APPLICATION_CREDENTIALS) with six test cases covering smoke verification, credential rotation, WIF priority, invalid credentials, secret deletion/recovery, and IP management persistence across rotation.

Summary by CodeRabbit

  • Tests
  • Added comprehensive end-to-end CNCC GCP credential tests covering credential rotation, WIF vs service-account JSON priority, invalid-credential behavior, secret deletion/recovery, and system resilience.
  • Validates CNCC readiness, restart/log checks, and verifies Cloud IP management via an egress-IP workflow — ensuring Cloud IP assignments persist through credential rotations and restarts.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
test/extended/networking/cncc_gcp_credentials.go (2)

205-223: Consider using defer for stream cleanup.

The logStream.Close() call could be missed if a panic occurs between opening and closing. Using defer would ensure proper cleanup.

♻️ Suggested improvement
 		logStream, err := req.Stream(context.Background())
 		if err != nil {
 			framework.Logf("Failed to get logs for pod %s: %v", pod.Name, err)
 			continue
 		}
+		defer logStream.Close()
 		buf := new(strings.Builder)
 		_, _ = fmt.Fprintf(buf, "--- Logs from pod %s ---\n", pod.Name)
 		b := make([]byte, 4096)
 		for {
 			n, readErr := logStream.Read(b)
 			if n > 0 {
 				buf.Write(b[:n])
 			}
 			if readErr != nil {
 				break
 			}
 		}
-		logStream.Close()
 		allLogs.WriteString(buf.String())
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/extended/networking/cncc_gcp_credentials.go` around lines 205 - 223, The
log stream opened via req.Stream(...) is closed explicitly later which can be
skipped on panic—after successfully acquiring logStream (the variable returned
from req.Stream), immediately schedule cleanup with defer logStream.Close() so
the stream is always closed; remove the later explicit logStream.Close() call
and keep the read loop as-is (references: req.Stream, logStream, and the read
loop writing into buf/allLogs).

451-453: Clarify the egress-assignable label value.

The label is set with an empty value (k8s.ovn.org/egress-assignable=). While this works for presence-based checks, consider using an explicit value like "true" for clarity and consistency with common label conventions.

♻️ Optional clarification
-		_, err = oc.AsAdmin().Run("label").Args("node", workerNodes[0].Name, "k8s.ovn.org/egress-assignable=", "--overwrite").Output()
+		_, err = oc.AsAdmin().Run("label").Args("node", workerNodes[0].Name, "k8s.ovn.org/egress-assignable=true", "--overwrite").Output()
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/extended/networking/cncc_gcp_credentials.go` around lines 451 - 453, The
label on the first worker node is being set with an empty value; change the
label call that currently sets "k8s.ovn.org/egress-assignable=" to use an
explicit value (e.g., "k8s.ovn.org/egress-assignable=true") in the
oc.AsAdmin().Run("label") invocation that references workerNodes[0].Name, and
update any related presence-based assertions elsewhere in the test to check for
the explicit value if needed.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@test/extended/networking/cncc_gcp_credentials.go`:
- Around line 205-223: The log stream opened via req.Stream(...) is closed
explicitly later which can be skipped on panic—after successfully acquiring
logStream (the variable returned from req.Stream), immediately schedule cleanup
with defer logStream.Close() so the stream is always closed; remove the later
explicit logStream.Close() call and keep the read loop as-is (references:
req.Stream, logStream, and the read loop writing into buf/allLogs).
- Around line 451-453: The label on the first worker node is being set with an
empty value; change the label call that currently sets
"k8s.ovn.org/egress-assignable=" to use an explicit value (e.g.,
"k8s.ovn.org/egress-assignable=true") in the oc.AsAdmin().Run("label")
invocation that references workerNodes[0].Name, and update any related
presence-based assertions elsewhere in the test to check for the explicit value
if needed.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: ff3f78ea-3aff-4ca8-9452-79cc6b816d38

📥 Commits

Reviewing files that changed from the base of the PR and between d90ffc5 and ed8cc01.

📒 Files selected for processing (1)
  • test/extended/networking/cncc_gcp_credentials.go

@openshift-ci-robot

openshift-ci-robot commented Mar 4, 2026

Copy link
Copy Markdown

@apahim: This pull request references GCP-429 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Validates the three-tier GCP credential fallback chain (WIF → SA → GOOGLE_APPLICATION_CREDENTIALS) with six test cases covering smoke verification, credential rotation, WIF priority, invalid credentials, secret deletion/recovery, and IP management persistence across rotation.

Summary by CodeRabbit

  • Tests
  • Added comprehensive end-to-end CNCC GCP credential tests covering credential rotation, WIF vs service-account JSON priority, invalid-credential behavior, secret deletion/recovery, and system resilience.
  • Validates CNCC readiness, pod restart and log checks, and verifies Cloud IP management via an egress-IP workflow — ensuring CloudPrivateIPConfig creation, assignment, and persistence through credential rotations and restarts.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
test/extended/networking/cncc_gcp_credentials.go (1)

97-101: Restore secret metadata on update path too.

When the secret exists, only Data and Type are restored. Restoring labels/annotations as well keeps “original state” behavior consistent with the recreate path.

Proposed fix
 		} else {
 			// Secret exists; update data back to original.
 			existing.Data = originalSecret.Data
 			existing.Type = originalSecret.Type
+			existing.Labels = originalSecret.Labels
+			existing.Annotations = originalSecret.Annotations
 			_, err = clientset.CoreV1().Secrets(cnccNamespace).Update(context.Background(), existing, metav1.UpdateOptions{})
 			o.Expect(err).NotTo(o.HaveOccurred(), "failed to restore CNCC secret data")
 		}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/extended/networking/cncc_gcp_credentials.go` around lines 97 - 101, When
restoring an existing secret in the update path, also copy metadata labels and
annotations from originalSecret so the secret's original state matches the
recreate path; update the block that currently sets existing.Data =
originalSecret.Data and existing.Type = originalSecret.Type to also set
existing.Labels = originalSecret.Labels and existing.Annotations =
originalSecret.Annotations before calling
clientset.CoreV1().Secrets(cnccNamespace).Update(...), and keep the same error
check using o.Expect(err).NotTo(o.HaveOccurred()).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@test/extended/networking/cncc_gcp_credentials.go`:
- Around line 432-433: The EgressIP setup uses a fixed name (egressIPName) and
then applies an EgressIP object (type EgressIP), which can silently reuse a
stale object from prior runs; change the test to generate a unique name per run
(e.g., append a timestamp/UUID) or explicitly ensure idempotence by deleting any
existing EgressIP with that name before creating/applying it; update all
occurrences around egressIPName and the EgressIP creation blocks (including the
other locations noted) so the test always creates a fresh resource or cleans up
the old one prior to apply.
- Around line 281-284: When reading the cloud-credentials secret (originalSecret
via clientset.CoreV1().Secrets(cnccNamespace).Get(...)) only treat a NotFound
error as a reason to skipper.Skipf; for all other errors (API/auth/transient
failures) fail the test instead of skipping by returning or calling the test
failure helper and include the error details. Use apierrors.IsNotFound(err)
(import "k8s.io/apimachinery/pkg/api/errors" as apierrors) to distinguish
NotFound from other errors and keep the existing skipper.Skipf(cnccNamespace,
cnccSecretName...) branch for the NotFound case.

---

Nitpick comments:
In `@test/extended/networking/cncc_gcp_credentials.go`:
- Around line 97-101: When restoring an existing secret in the update path, also
copy metadata labels and annotations from originalSecret so the secret's
original state matches the recreate path; update the block that currently sets
existing.Data = originalSecret.Data and existing.Type = originalSecret.Type to
also set existing.Labels = originalSecret.Labels and existing.Annotations =
originalSecret.Annotations before calling
clientset.CoreV1().Secrets(cnccNamespace).Update(...), and keep the same error
check using o.Expect(err).NotTo(o.HaveOccurred()).

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 5878b4a7-11d4-4310-98b0-bf338906f908

📥 Commits

Reviewing files that changed from the base of the PR and between ed8cc01 and 6a54273.

📒 Files selected for processing (1)
  • test/extended/networking/cncc_gcp_credentials.go

Comment thread test/extended/networking/cncc_gcp_credentials.go
Comment thread test/extended/networking/cncc_gcp_credentials.go Outdated
@apahim

apahim commented Mar 4, 2026

Copy link
Copy Markdown
Author

/payload-job-with-prs periodic-ci-openshift-release-main-ci-4.22-e2e-gcp-ovn openshift/cloud-network-config-controller#206

@openshift-ci

openshift-ci Bot commented Mar 4, 2026

Copy link
Copy Markdown
Contributor

@apahim: it appears that you have attempted to use some version of the payload command, but your comment was incorrectly formatted and cannot be acted upon. See the docs for usage info.

@openshift-ci

openshift-ci Bot commented Mar 4, 2026

Copy link
Copy Markdown
Contributor

@apahim: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-ci-4.22-e2e-gcp-ovn

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/67ccb430-1813-11f1-9f44-422dfd494bcd-0

@openshift-ci-robot

Copy link
Copy Markdown

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

@ricky-rav

Copy link
Copy Markdown

The new should have a working CNCC deployment with valid GCP credentials" is exactly what's failing :) https://prow.ci.openshift.org/view/gs/test-platform-results/logs/openshift-cloud-network-config-controller-206-openshift-origin-30832-ci-4.22-e2e-gcp-ovn/2029312221011513344

@apahim

apahim commented Mar 5, 2026

Copy link
Copy Markdown
Author

/retest

@openshift-trt

openshift-trt Bot commented Mar 5, 2026

Copy link
Copy Markdown

Risk analysis has seen new tests most likely introduced by this PR.
Please ensure that new tests meet guidelines for naming and stability.

New Test Risks for sha: 6a54273

Job Name New Test Risk
pull-ci-openshift-origin-main-e2e-gcp-ovn High - "[sig-network][Feature:CNCC][apigroup:operator.openshift.io] should have a working CNCC deployment with valid GCP credentials [Suite:openshift/conformance/parallel]" is a new test that was not present in all runs against the current commit, and also failed 1 time(s).

New tests seen in this PR at sha: 6a54273

  • "[sig-network][Feature:CNCC][apigroup:operator.openshift.io] should have a working CNCC deployment with valid GCP credentials [Suite:openshift/conformance/parallel]" [Total: 2, Pass: 1, Fail: 1, Flake: 0]

@apahim

apahim commented Mar 6, 2026

Copy link
Copy Markdown
Author

/retest-required

@apahim

apahim commented Mar 6, 2026

Copy link
Copy Markdown
Author

The new should have a working CNCC deployment with valid GCP credentials" is exactly what's failing :) https://prow.ci.openshift.org/view/gs/test-platform-results/logs/openshift-cloud-network-config-controller-206-openshift-origin-30832-ci-4.22-e2e-gcp-ovn/2029312221011513344

the issue is that getCnccPodLogs(100) at line 309 doesn't capture the startup log message because the pod has been running for over an hour by the time the smoke test executes. fixing.

Validates the three-tier GCP credential fallback chain (WIF → SA →
GOOGLE_APPLICATION_CREDENTIALS) with six test cases covering smoke
verification, credential rotation, WIF priority, invalid credentials,
secret deletion/recovery, and IP management persistence across rotation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Amador Pahim <apahim@redhat.com>
@openshift-ci-robot

openshift-ci-robot commented Mar 6, 2026

Copy link
Copy Markdown

@apahim: This pull request references GCP-429 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Validates the three-tier GCP credential fallback chain (WIF → SA → GOOGLE_APPLICATION_CREDENTIALS) with six test cases covering smoke verification, credential rotation, WIF priority, invalid credentials, secret deletion/recovery, and IP management persistence across rotation.

Summary by CodeRabbit

  • Tests
  • Added comprehensive end-to-end CNCC GCP credential tests covering credential rotation, WIF vs service-account JSON priority, invalid-credential behavior, secret deletion/recovery, and recovery after restarts.
  • Validates CNCC readiness, pod restart and log checks, and verifies cloud IP management via an egress-IP workflow — ensuring CloudPrivateIPConfig creation, assignment, and persistence through credential rotations.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@apahim

apahim commented Mar 9, 2026

Copy link
Copy Markdown
Author

/retest-required

@apahim

apahim commented Mar 9, 2026

Copy link
Copy Markdown
Author

The e2e-gcp-ovn test is failing because it needs the CNCC PR merged:

It passes here, with a CNCC build including the PR:

@apahim

apahim commented Mar 11, 2026

Copy link
Copy Markdown
Author

/retest-required

@apahim

apahim commented Mar 11, 2026

Copy link
Copy Markdown
Author

@openshift-trt

openshift-trt Bot commented Mar 11, 2026

Copy link
Copy Markdown

Risk analysis has seen new tests most likely introduced by this PR.
Please ensure that new tests meet guidelines for naming and stability.

New Test Risks for sha: f3423dc

Job Name New Test Risk
pull-ci-openshift-origin-main-e2e-gcp-ovn High - "[sig-network][Feature:CNCC][apigroup:operator.openshift.io] should have a working CNCC deployment with valid GCP credentials [Suite:openshift/conformance/parallel]" is a new test, was only seen in one job, and failed 4 time(s) against the current commit.

New tests seen in this PR at sha: f3423dc

  • "[sig-network][Feature:CNCC][apigroup:operator.openshift.io] should have a working CNCC deployment with valid GCP credentials [Suite:openshift/conformance/parallel]" [Total: 4, Pass: 0, Fail: 4, Flake: 0]

@ricky-rav ricky-rav left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks very good, I've left two minor comments

Comment thread test/extended/networking/cncc_gcp_credentials.go Outdated
Comment thread test/extended/networking/cncc_gcp_credentials.go
- Consolidate dummySAJSON and invalidSAJSON into single fakeSAJSON constant
- Differentiate NotFound from other errors when reading cloud-credentials secret
- Guard AfterEach teardown against nil clientset/originalSecret on skipped tests
- Use unique EgressIP names per run to prevent stale-object reuse
- Switch from oc apply to oc create for EgressIP creation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@apahim

apahim commented Mar 12, 2026

Copy link
Copy Markdown
Author

/payload-job-with-prs periodic-ci-openshift-release-main-ci-4.22-e2e-gcp-ovn openshift/cloud-network-config-controller#206

@openshift-ci

openshift-ci Bot commented Mar 12, 2026

Copy link
Copy Markdown
Contributor

@apahim: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-ci-4.22-e2e-gcp-ovn

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/b4819320-1e00-11f1-9f43-e9ee7a909854-0

@openshift-ci-robot

Copy link
Copy Markdown

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

@ricky-rav ricky-rav left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/LGTM

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Mar 12, 2026
@openshift-trt

openshift-trt Bot commented Mar 12, 2026

Copy link
Copy Markdown

Risk analysis has seen new tests most likely introduced by this PR.
Please ensure that new tests meet guidelines for naming and stability.

New Test Risks for sha: d0f8466

Job Name New Test Risk
pull-ci-openshift-origin-main-e2e-gcp-ovn Medium - "[sig-network][Feature:CNCC][apigroup:operator.openshift.io] should have a working CNCC deployment with valid GCP credentials [Suite:openshift/conformance/parallel]" is a new test, and was only seen in one job.

New tests seen in this PR at sha: d0f8466

  • "[sig-network][Feature:CNCC][apigroup:operator.openshift.io] should have a working CNCC deployment with valid GCP credentials [Suite:openshift/conformance/parallel]" [Total: 1, Pass: 1, Fail: 0, Flake: 0]

1 similar comment
@openshift-trt

openshift-trt Bot commented Mar 12, 2026

Copy link
Copy Markdown

Risk analysis has seen new tests most likely introduced by this PR.
Please ensure that new tests meet guidelines for naming and stability.

New Test Risks for sha: d0f8466

Job Name New Test Risk
pull-ci-openshift-origin-main-e2e-gcp-ovn Medium - "[sig-network][Feature:CNCC][apigroup:operator.openshift.io] should have a working CNCC deployment with valid GCP credentials [Suite:openshift/conformance/parallel]" is a new test, and was only seen in one job.

New tests seen in this PR at sha: d0f8466

  • "[sig-network][Feature:CNCC][apigroup:operator.openshift.io] should have a working CNCC deployment with valid GCP credentials [Suite:openshift/conformance/parallel]" [Total: 1, Pass: 1, Fail: 0, Flake: 0]

for k, v := range dep.Spec.Selector.MatchLabels {
parts = append(parts, fmt.Sprintf("%s=%s", k, v))
}
return strings.Join(parts, ",")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return metav1.FormatLabelSelector(dep.Spec.Selector) ?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure!

Comment thread test/extended/networking/cncc_gcp_credentials.go
case credentialTypeWIF:
o.Expect(logs).To(o.ContainSubstring("Using GCP Workload Identity Federation credentials from secret"))
case credentialTypeSA:
o.Expect(logs).To(o.ContainSubstring("Using GCP service account JSON credentials from secret"))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We shouldn't be asserting on exact strings logged by another component.

(In general, it seems like this test ought to be using that new openshift testing thing where we build the tests in the repo of the component that they're testing, and then this would be fine. But I don't know the details of how that all works...)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CNCC doen't use addopt the OTE framework yet.

Ideally we would rely on CNCC conditions here, but we don't have any status field that exposes which credential type was selected.

If you're ok with it, we can remove the log assertions. The test already proves WIF priority indirectly: it adds a fake SA alongside the real WIF config. If CNCC picked the fake SA, it would crash.

Comment thread test/extended/networking/cncc_gcp_credentials.go Outdated
// Test 1: Smoke — current credential type works
g.It("should have a working CNCC deployment with valid GCP credentials", func() {
g.By("Verifying CNCC deployment is ready")
waitForCnccReady()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If CNCC isn't ready then CNO should report not ready, in which case the installer should report not ready, in which case the tests should not be running yet. So is this really needed?

If it is needed, isn't it needed in the other tests too? (The tests aren't guaranteed to be run in the order they appear in. Note that getCnccPodRestartCount() in the other tests will just return 0 if CNCC isn't running, rather than waiting for it to be running.)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, removing then waitForCnccReady().

Comment thread test/extended/networking/cncc_gcp_credentials.go
- Rename Cncc to CNCC in function names per Go acronym convention
- Use metav1.FormatLabelSelector instead of manual label construction
- Remove waitForCNCCReady from smoke test (CNO guarantees readiness)
- Remove log string assertions (avoid coupling to another component's
  log output); rely on functional verification instead
- Remove unused getCNCCPodLogs function and strings import

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-ci openshift-ci Bot removed the lgtm Indicates that a PR is ready to be merged. label Mar 23, 2026
@openshift-ci

openshift-ci Bot commented Mar 23, 2026

Copy link
Copy Markdown
Contributor

@apahim: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-ovn-microshift d0f8466 link true /test e2e-aws-ovn-microshift

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@danwinship

Copy link
Copy Markdown
Contributor

/lgtm

it would be cool if there was some TRT dashboard that could notice if some e2e test got skipped everywhere but for now, 🤷‍♂️

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Mar 31, 2026
@openshift-ci

openshift-ci Bot commented Mar 31, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: apahim, danwinship, ricky-rav

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 31, 2026
@apahim

apahim commented Mar 31, 2026

Copy link
Copy Markdown
Author

/retest-required

1 similar comment
@apahim

apahim commented Apr 6, 2026

Copy link
Copy Markdown
Author

/retest-required

@danwinship

Copy link
Copy Markdown
Contributor

@apahim this is apparently waiting for a /verified tag

@apahim

apahim commented Jul 29, 2026

Copy link
Copy Markdown
Author

/verified by @apahim

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jul 29, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@apahim: This PR has been marked as verified by @apahim.

Details

In response to this:

/verified by @apahim

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants