OCPBUGS-60620: e2e: Deflake tests by using ReplicaSet for test workload by alebedev87 · Pull Request #1262 · openshift/cluster-ingress-operator

alebedev87 · 2025-08-10T21:15:18Z

This PR aims at trying to de-flake the e2e tests which were flaking during the CI testing of #1257:

Gateway API e2e test: switch the test HTTPRoute's backend workload to a ReplicaSet to better tolerate pod evictions or deletions during runs.
IdleConnection test: switch the test route's backend workload to a ReplicaSet to better tolerate pod evictions or deletions during runs.

alebedev87 · 2025-08-12T13:03:24Z

/hold

Holding this PR to better split between different flakes (e.g. #1265).

alebedev87 · 2025-08-12T21:18:04Z

All operator tests finished green. Let's retry them by pushing an amended commit with no real changed.

alebedev87 · 2025-08-13T08:03:30Z

No operator test failures, second time in a row. Let's try again...

openshift-ci-robot · 2025-08-18T22:49:15Z

@alebedev87: This pull request references Jira Issue OCPBUGS-60620, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.20.0) matches configured target version for branch (4.20.0)
bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira (iamin@redhat.com), skipping review request.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

This PR aims at trying to de-flake the e2e tests which we were flaking during the CI testing of #1257:

Gateway API e2e test: switch the test HTTPRoute's backend workload to a ReplicaSet to better tolerate pod evictions or deletions during runs.

IdleConnection test: switch the test route's backend workload to a ReplicaSet to better tolerate pod evictions or deletions during runs.

Gateway API e2e test: Increase timeouts for CVO scale-ups/downs and VAP assertions.

Gateway API e2e test: Retry Gateway CRD VAP creation on network errors.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

alebedev87 · 2025-08-18T22:52:17Z

/unhold

openshift-ci-robot · 2025-08-18T22:52:41Z

@alebedev87: This pull request references Jira Issue OCPBUGS-60620, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.20.0) matches configured target version for branch (4.20.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira (iamin@redhat.com), skipping review request.

Details

In response to this:

This PR aims at trying to de-flake the e2e tests which we were flaking during the CI testing of #1257:

Gateway API e2e test: switch the test HTTPRoute's backend workload to a ReplicaSet to better tolerate pod evictions or deletions during runs.

IdleConnection test: switch the test route's backend workload to a ReplicaSet to better tolerate pod evictions or deletions during runs.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

rikatz · 2025-08-19T15:17:26Z

/assign

test/e2e/idle_connection_test.go

rikatz · 2025-08-19T16:49:50Z

test/e2e/idle_connection_test.go

 	}

-	return pod, nil
+	if err := kclient.Create(ctx, rs); err != nil {


do you wanna retry here for the sake of network problems? (as we've been doing on other PRs)

Added createWithRetryOnConflict helper function (similar to existing update*WithRetryOnConflict helpers) and introduced it here.

rikatz · 2025-08-19T16:51:42Z

test/e2e/util_gatewayapi_test.go

-		return nil, fmt.Errorf("failed to create pod %s/%s: %v", namespace, echoPod.Name, err)
+	// buildEchoReplicaSet builds a replicaset which creates a pod that listens on port 8080.
+	echoRs := buildEchoReplicaSet(backendRefname, namespace)
+	if err := kclient.Create(context.TODO(), echoRs); err != nil {


same comment as above on the create and retry. I am wondering if we care on actually creating some helper function for these "create" operations, with something like "create or retry" to avoid network issues.

Added createWithRetryOnConflict helper function and introduced it here.

rikatz · 2025-08-20T12:21:14Z

test/e2e/util_test.go


+// createWithRetryOnConflict creates the given object. If there is a conflict error on create
+// then the create is retried until the timeout is reached.
+func createWithRetryOnConflict[T client.Object](ctx context.Context, obj T, timeout time.Duration) error {


I don't think you need generics/type parameters here, you can just pass client.Object as the argument of object. kclient already accepts a client.Object

Converted to non generic one.

rikatz · 2025-08-20T12:22:54Z

test/e2e/util_test.go

+func createWithRetryOnConflict[T client.Object](ctx context.Context, obj T, timeout time.Duration) error {
+	return wait.PollUntilContextTimeout(ctx, 2*time.Second, timeout, true, func(ctx context.Context) (bool, error) {
+		if err := kclient.Create(ctx, obj); err != nil && !errors.IsAlreadyExists(err) {
+			if errors.IsConflict(err) {


I don't know where this can happen on a kclient.Create operation, at least for me the API will answer with Conflict only when you are doing patch/update operation. In case Create, I can think of:

Already exists - you cover on the if branch

Does not exist - So you create it

Another error (network, permission, etc) - you retry

Maybe I am missing something on the Conflict situation here

Maybe I am missing something on the Conflict situation here

No, you are right. I copied the function from an existing update*OnConflict function. Updated it to retry on any error but AlreadyExists.

Switch the test HTTPRoute's backend workload to a ReplicaSet to better tolerate pod evictions or deletions during runs.

Switch the test route's backend workload to a ReplicaSet to better tolerate pod evictions or deletions during runs. Update test logic to expect a response prefix, since the echo pod names returned in the response are now suffixed with a random hash by the ReplicaSet controller.

rikatz · 2025-08-20T12:48:29Z

/lgtm
/approve
Thanks!

openshift-ci · 2025-08-20T12:50:44Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rikatz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [rikatz]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2025-08-20T15:05:04Z

/retest-required

Remaining retests: 0 against base HEAD 52a0048 and 2 for PR HEAD edfab23 in total

rikatz · 2025-08-20T19:22:07Z

/retest

rikatz · 2025-08-20T19:22:38Z

The tests failing are not required.
The only required test failing is GCP one, which is a well known failure.

@Miciah this test may need an override

alebedev87 · 2025-08-20T22:48:17Z

@rikatz : just for info: we can use /retest-required if we see optional job failing on things which we cannot relate to the PR.

openshift-ci-robot · 2025-08-20T23:24:59Z

/retest-required

Remaining retests: 0 against base HEAD 52a0048 and 2 for PR HEAD edfab23 in total

openshift-ci-robot · 2025-08-21T11:18:13Z

/retest-required

Remaining retests: 0 against base HEAD 52a0048 and 2 for PR HEAD edfab23 in total

candita · 2025-08-21T14:16:17Z

This test is failing and has been reported to the Cloud Platform team. Resolution depends at least on merge of openshift/cloud-provider-gcp#86.

/override ci/prow/e2e-gcp-operator

openshift-ci · 2025-08-21T14:18:34Z

@candita: Overrode contexts on behalf of candita: ci/prow/e2e-gcp-operator

Details

In response to this:

This test is failing and has been reported to the Cloud Platform team. Resolution depends at least on merge of openshift/cloud-provider-gcp#86.

/override ci/prow/e2e-gcp-operator

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci · 2025-08-21T14:18:40Z

@alebedev87: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/okd-scos-e2e-aws-ovn	`edfab23`	link	false	`/test okd-scos-e2e-aws-ovn`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci-robot · 2025-08-21T14:21:58Z

@alebedev87: Jira Issue OCPBUGS-60620: All pull requests linked via external trackers have merged:

openshift/cluster-ingress-operator#1262

Jira Issue OCPBUGS-60620 has been moved to the MODIFIED state.

Details

In response to this:

This PR aims at trying to de-flake the e2e tests which were flaking during the CI testing of #1257:

Gateway API e2e test: switch the test HTTPRoute's backend workload to a ReplicaSet to better tolerate pod evictions or deletions during runs.

IdleConnection test: switch the test route's backend workload to a ReplicaSet to better tolerate pod evictions or deletions during runs.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci bot requested review from candita and grzpiotrowski August 10, 2025 21:15

alebedev87 mentioned this pull request Aug 10, 2025

NE-2096: Bump to OSSM 3.1.0 and Istio 1.26.2 #1257

Merged

alebedev87 force-pushed the e2e-echo-rs-2 branch from bbc76c4 to b574b3f Compare August 11, 2025 09:17

alebedev87 changed the title ~~e2e: GWAPI - Use ReplicaSet for backend pod~~ e2e: Deflake some tests by switching to a ReplicaSet for test workload Aug 11, 2025

alebedev87 changed the title ~~e2e: Deflake some tests by switching to a ReplicaSet for test workload~~ e2e: Deflake tests by using ReplicaSet for test workload Aug 11, 2025

alebedev87 force-pushed the e2e-echo-rs-2 branch 3 times, most recently from 7bb7c34 to 5647a21 Compare August 11, 2025 14:29

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 12, 2025

alebedev87 mentioned this pull request Aug 12, 2025

OCPBUGS-60302: e2e - Reduce flakiness in testGatewayAPIResourcesProtection #1265

Merged

alebedev87 force-pushed the e2e-echo-rs-2 branch from 62a2d3b to 71e2a03 Compare August 12, 2025 21:20

alebedev87 force-pushed the e2e-echo-rs-2 branch from 71e2a03 to e9e5d82 Compare August 13, 2025 08:03

alebedev87 mentioned this pull request Aug 18, 2025

NE-2066: Set degraded=true when OSSM 3 can't be installed #1268

Merged

alebedev87 changed the title ~~e2e: Deflake tests by using ReplicaSet for test workload~~ OCPBUGS-60620: e2e: Deflake tests by using ReplicaSet for test workload Aug 18, 2025

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Aug 18, 2025

openshift-ci-robot added the jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. label Aug 18, 2025

alebedev87 force-pushed the e2e-echo-rs-2 branch from e9e5d82 to d4fe23d Compare August 18, 2025 22:50

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 18, 2025

openshift-ci bot assigned rikatz Aug 19, 2025

rikatz reviewed Aug 19, 2025

View reviewed changes

test/e2e/idle_connection_test.go Outdated Show resolved Hide resolved

rikatz reviewed Aug 19, 2025

View reviewed changes

alebedev87 force-pushed the e2e-echo-rs-2 branch from d4fe23d to 75a0e3c Compare August 20, 2025 10:21

rikatz reviewed Aug 20, 2025

View reviewed changes

alebedev87 added 3 commits August 20, 2025 14:46

e2e: Add helper to create any object with a retry on conflict

bdbac7e

e2e: Use ReplicaSet for GatewayAPI test workload

19897f5

Switch the test HTTPRoute's backend workload to a ReplicaSet to better tolerate pod evictions or deletions during runs.

alebedev87 force-pushed the e2e-echo-rs-2 branch from 75a0e3c to edfab23 Compare August 20, 2025 12:47

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Aug 20, 2025

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 20, 2025

openshift-merge-bot bot merged commit 2f21242 into openshift:master Aug 21, 2025
20 of 21 checks passed

Conversation

alebedev87 commented Aug 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alebedev87 commented Aug 12, 2025

Uh oh!

alebedev87 commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alebedev87 commented Aug 13, 2025

Uh oh!

openshift-ci-robot commented Aug 18, 2025

Uh oh!

alebedev87 commented Aug 18, 2025

Uh oh!

openshift-ci-robot commented Aug 18, 2025

Uh oh!

rikatz commented Aug 19, 2025

Uh oh!

Uh oh!

rikatz Aug 19, 2025

Choose a reason for hiding this comment

Uh oh!

alebedev87 Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rikatz Aug 19, 2025

Choose a reason for hiding this comment

Uh oh!

alebedev87 Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

rikatz Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

alebedev87 Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

rikatz Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

alebedev87 Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

rikatz commented Aug 20, 2025

Uh oh!

openshift-ci bot commented Aug 20, 2025

Uh oh!

openshift-ci-robot commented Aug 20, 2025

Uh oh!

rikatz commented Aug 20, 2025

Uh oh!

rikatz commented Aug 20, 2025

Uh oh!

alebedev87 commented Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Aug 20, 2025

Uh oh!

openshift-ci-robot commented Aug 21, 2025

Uh oh!

candita commented Aug 21, 2025

Uh oh!

openshift-ci bot commented Aug 21, 2025

Uh oh!

openshift-ci bot commented Aug 21, 2025

Uh oh!

Uh oh!

openshift-ci-robot commented Aug 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Comments

alebedev87 commented Aug 10, 2025 •

edited

Loading

alebedev87 commented Aug 12, 2025 •

edited

Loading

alebedev87 Aug 20, 2025 •

edited

Loading

alebedev87 commented Aug 20, 2025 •

edited

Loading