E2E: Fixes to stabilize the multicluster tests further by mkolesnik · Pull Request #1472 · istio-ecosystem/sail-operator

mkolesnik · 2025-12-18T19:09:31Z

This collection of fixes adds further verifications to the multicluster tests to make them more robust and isolate problems stemming from the underlying cloud infrastructure when deploying the east-west gateway LB service.

codecov · 2025-12-18T19:13:54Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.94%. Comparing base (4868852) to head (5129c7a).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1472      +/-   ##
==========================================
+ Coverage   80.55%   80.94%   +0.39%     
==========================================
  Files          44       44              
  Lines        2299     2299              
==========================================
+ Hits         1852     1861       +9     
+ Misses        328      322       -6     
+ Partials      119      116       -3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

tests/e2e/multicluster/common.go

This might take a while, expecially on a public cloud. Make sure the tests wait for the IP to be assigned before proceeding, otherwise they'll get stuck later waiting for the communication to proceed (and sometimes fail). Signed-off-by: Mike Kolesnik <mkolesni@redhat.com>

To rule out infrastructure problems, a sanity check is added to try and reach the gateway on the other side. If this check fails, this could indicate an infrastructure problem. Signed-off-by: Mike Kolesnik <mkolesni@redhat.com>

Having curl fetch the URL several times saves time waiting on responses since it's much cheaper than each `exec` call. This makes successful tests finish much faster instead of waiting for the response to randomly land on the expected version. Signed-off-by: Mike Kolesnik <mkolesni@redhat.com>

* upstream/main: (34 commits) E2E: Fixes to stabilize the multicluster tests further (istio-ecosystem#1472) fix: Minimize wildcard use in operator ClusterRoles (istio-ecosystem#1477) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1475) fix unbound shell variable (istio-ecosystem#1491) [claude] Add "/submit-pr" slash command (istio-ecosystem#1487) claude: generate-release-notes slash command (istio-ecosystem#1471) Adding docs for integrating Istio with ZTWIM (SPIFFE/SPIRE) (istio-ecosystem#1319) fix dual-stack kind cluster sample and add beta version (istio-ecosystem#1488) fix: dual-stack doc list (istio-ecosystem#1479) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1473) fix: allow automator commits without message (istio-ecosystem#1470) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1466) set preserve-unknown-fields on gatewayClasses (istio-ecosystem#1465) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1462) Add Operator image build trigger on "versions.yaml" change (istio-ecosystem#1461) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1460) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1457) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1453) Bump operator version to 1.29.0 (istio-ecosystem#1451) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1452) ...

* upstream/main: (35 commits) Use the setup buildx action in nightly image build (istio-ecosystem#1501) E2E: Fixes to stabilize the multicluster tests further (istio-ecosystem#1472) fix: Minimize wildcard use in operator ClusterRoles (istio-ecosystem#1477) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1475) fix unbound shell variable (istio-ecosystem#1491) [claude] Add "/submit-pr" slash command (istio-ecosystem#1487) claude: generate-release-notes slash command (istio-ecosystem#1471) Adding docs for integrating Istio with ZTWIM (SPIFFE/SPIRE) (istio-ecosystem#1319) fix dual-stack kind cluster sample and add beta version (istio-ecosystem#1488) fix: dual-stack doc list (istio-ecosystem#1479) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1473) fix: allow automator commits without message (istio-ecosystem#1470) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1466) set preserve-unknown-fields on gatewayClasses (istio-ecosystem#1465) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1462) Add Operator image build trigger on "versions.yaml" change (istio-ecosystem#1461) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1460) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1457) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1453) Bump operator version to 1.29.0 (istio-ecosystem#1451) ...

* upstream/main: (36 commits) Use the setup buildx action in release workflow (istio-ecosystem#1503) Use the setup buildx action in nightly image build (istio-ecosystem#1501) E2E: Fixes to stabilize the multicluster tests further (istio-ecosystem#1472) fix: Minimize wildcard use in operator ClusterRoles (istio-ecosystem#1477) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1475) fix unbound shell variable (istio-ecosystem#1491) [claude] Add "/submit-pr" slash command (istio-ecosystem#1487) claude: generate-release-notes slash command (istio-ecosystem#1471) Adding docs for integrating Istio with ZTWIM (SPIFFE/SPIRE) (istio-ecosystem#1319) fix dual-stack kind cluster sample and add beta version (istio-ecosystem#1488) fix: dual-stack doc list (istio-ecosystem#1479) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1473) fix: allow automator commits without message (istio-ecosystem#1470) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1466) set preserve-unknown-fields on gatewayClasses (istio-ecosystem#1465) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1462) Add Operator image build trigger on "versions.yaml" change (istio-ecosystem#1461) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1460) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1457) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1453) ...

* upstream/main: (38 commits) Transform structpb types back into their k8s types (istio-ecosystem#1493) E2E: Improve Cleaner waiting logic (istio-ecosystem#1494) Use the setup buildx action in release workflow (istio-ecosystem#1503) Use the setup buildx action in nightly image build (istio-ecosystem#1501) E2E: Fixes to stabilize the multicluster tests further (istio-ecosystem#1472) fix: Minimize wildcard use in operator ClusterRoles (istio-ecosystem#1477) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1475) fix unbound shell variable (istio-ecosystem#1491) [claude] Add "/submit-pr" slash command (istio-ecosystem#1487) claude: generate-release-notes slash command (istio-ecosystem#1471) Adding docs for integrating Istio with ZTWIM (SPIFFE/SPIRE) (istio-ecosystem#1319) fix dual-stack kind cluster sample and add beta version (istio-ecosystem#1488) fix: dual-stack doc list (istio-ecosystem#1479) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1473) fix: allow automator commits without message (istio-ecosystem#1470) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1466) set preserve-unknown-fields on gatewayClasses (istio-ecosystem#1465) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1462) Add Operator image build trigger on "versions.yaml" change (istio-ecosystem#1461) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1460) ...

* upstream/main: (39 commits) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1506) Transform structpb types back into their k8s types (istio-ecosystem#1493) E2E: Improve Cleaner waiting logic (istio-ecosystem#1494) Use the setup buildx action in release workflow (istio-ecosystem#1503) Use the setup buildx action in nightly image build (istio-ecosystem#1501) E2E: Fixes to stabilize the multicluster tests further (istio-ecosystem#1472) fix: Minimize wildcard use in operator ClusterRoles (istio-ecosystem#1477) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1475) fix unbound shell variable (istio-ecosystem#1491) [claude] Add "/submit-pr" slash command (istio-ecosystem#1487) claude: generate-release-notes slash command (istio-ecosystem#1471) Adding docs for integrating Istio with ZTWIM (SPIFFE/SPIRE) (istio-ecosystem#1319) fix dual-stack kind cluster sample and add beta version (istio-ecosystem#1488) fix: dual-stack doc list (istio-ecosystem#1479) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1473) fix: allow automator commits without message (istio-ecosystem#1470) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1466) set preserve-unknown-fields on gatewayClasses (istio-ecosystem#1465) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1462) Add Operator image build trigger on "versions.yaml" change (istio-ecosystem#1461) ...

* upstream/main: (40 commits) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1507) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1506) Transform structpb types back into their k8s types (istio-ecosystem#1493) E2E: Improve Cleaner waiting logic (istio-ecosystem#1494) Use the setup buildx action in release workflow (istio-ecosystem#1503) Use the setup buildx action in nightly image build (istio-ecosystem#1501) E2E: Fixes to stabilize the multicluster tests further (istio-ecosystem#1472) fix: Minimize wildcard use in operator ClusterRoles (istio-ecosystem#1477) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1475) fix unbound shell variable (istio-ecosystem#1491) [claude] Add "/submit-pr" slash command (istio-ecosystem#1487) claude: generate-release-notes slash command (istio-ecosystem#1471) Adding docs for integrating Istio with ZTWIM (SPIFFE/SPIRE) (istio-ecosystem#1319) fix dual-stack kind cluster sample and add beta version (istio-ecosystem#1488) fix: dual-stack doc list (istio-ecosystem#1479) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1473) fix: allow automator commits without message (istio-ecosystem#1470) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1466) set preserve-unknown-fields on gatewayClasses (istio-ecosystem#1465) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1462) ...

* upstream/main: (41 commits) Use the setup buildx action in versions-triggered-build flow (istio-ecosystem#1511) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1507) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1506) Transform structpb types back into their k8s types (istio-ecosystem#1493) E2E: Improve Cleaner waiting logic (istio-ecosystem#1494) Use the setup buildx action in release workflow (istio-ecosystem#1503) Use the setup buildx action in nightly image build (istio-ecosystem#1501) E2E: Fixes to stabilize the multicluster tests further (istio-ecosystem#1472) fix: Minimize wildcard use in operator ClusterRoles (istio-ecosystem#1477) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1475) fix unbound shell variable (istio-ecosystem#1491) [claude] Add "/submit-pr" slash command (istio-ecosystem#1487) claude: generate-release-notes slash command (istio-ecosystem#1471) Adding docs for integrating Istio with ZTWIM (SPIFFE/SPIRE) (istio-ecosystem#1319) fix dual-stack kind cluster sample and add beta version (istio-ecosystem#1488) fix: dual-stack doc list (istio-ecosystem#1479) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1473) fix: allow automator commits without message (istio-ecosystem#1470) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1466) set preserve-unknown-fields on gatewayClasses (istio-ecosystem#1465) ...

* upstream/main: (42 commits) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1512) Use the setup buildx action in versions-triggered-build flow (istio-ecosystem#1511) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1507) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1506) Transform structpb types back into their k8s types (istio-ecosystem#1493) E2E: Improve Cleaner waiting logic (istio-ecosystem#1494) Use the setup buildx action in release workflow (istio-ecosystem#1503) Use the setup buildx action in nightly image build (istio-ecosystem#1501) E2E: Fixes to stabilize the multicluster tests further (istio-ecosystem#1472) fix: Minimize wildcard use in operator ClusterRoles (istio-ecosystem#1477) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1475) fix unbound shell variable (istio-ecosystem#1491) [claude] Add "/submit-pr" slash command (istio-ecosystem#1487) claude: generate-release-notes slash command (istio-ecosystem#1471) Adding docs for integrating Istio with ZTWIM (SPIFFE/SPIRE) (istio-ecosystem#1319) fix dual-stack kind cluster sample and add beta version (istio-ecosystem#1488) fix: dual-stack doc list (istio-ecosystem#1479) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1473) fix: allow automator commits without message (istio-ecosystem#1470) Automator: Update dependencies in istio-ecosystem/sail-operator@main (istio-ecosystem#1466) ...

…em#1472) * E2E: Add a validation for E/W GW to get LB IP This might take a while, expecially on a public cloud. Make sure the tests wait for the IP to be assigned before proceeding, otherwise they'll get stuck later waiting for the communication to proceed (and sometimes fail). Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> * E2E: Add check that multicluster gateway rechable To rule out infrastructure problems, a sanity check is added to try and reach the gateway on the other side. If this check fails, this could indicate an infrastructure problem. Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> * E2E: Improve connectivity checking in multicluster Having curl fetch the URL several times saves time waiting on responses since it's much cheaper than each `exec` call. This makes successful tests finish much faster instead of waiting for the response to randomly land on the expected version. Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> --------- Signed-off-by: Mike Kolesnik <mkolesni@redhat.com>

* E2E code consolidations (istio-ecosystem#1397) * E2E: Extract common function to await conditions This code is very repetitive and verbose, extracting it to a common function allows the tests to be much more concise and readable, without affecting functionality. The main benefit of this is making the tests easier to read and maintain, and avoid repeating code (and making mistakes). Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> * E2E: Add specific AwaitDeployment function Most of the tests that await for the DeploymentAvailable condition do it on the control plane namespace for a Deployment. Extracting this specific case makes the tests even easier to read and understand. Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> * E2E: Add CheckSamplePodsReady function Reuse CheckPodsReady and add a specific function CheckSamplePodsReady since most of the calls are on the sample namespace. Both checks are now in the `checks.go` file for logical consistency. Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> * E2E: Extract waiting for CNI DaemonSet to be ready This is repetitive and verbose, extracting this makes the tests easier to read and maintain. Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> --------- Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> * Add Ambient multi cluster E2E tests for Multi Primary deployments (istio-ecosystem#1353) * Expose common namespaces in E2E tests To be able to use directly without redefining or passing them all over. The namespaces are reused in the test suite files (instead of redefined) in order to minimize the amount of changes, but we can further refactor this in the future. Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> * Extract multicluster resource creation This males tests easier to maintain while avoiding code duplication Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> * Add Ambient tests to E2E Multi Primary Istio 1.27+ supports running Multi-Primary deployments in Abmient mode, so tests were adapted to run both on Sidecar and Ambient mode. Both modes run the same tests, sequentialy & independently of each other. Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> --------- Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> * E2E: Fixes to stabilize the multicluster tests further (istio-ecosystem#1472) * E2E: Add a validation for E/W GW to get LB IP This might take a while, expecially on a public cloud. Make sure the tests wait for the IP to be assigned before proceeding, otherwise they'll get stuck later waiting for the communication to proceed (and sometimes fail). Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> * E2E: Add check that multicluster gateway rechable To rule out infrastructure problems, a sanity check is added to try and reach the gateway on the other side. If this check fails, this could indicate an infrastructure problem. Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> * E2E: Improve connectivity checking in multicluster Having curl fetch the URL several times saves time waiting on responses since it's much cheaper than each `exec` call. This makes successful tests finish much faster instead of waiting for the response to randomly land on the expected version. Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> --------- Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> --------- Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> Co-authored-by: Mike Kolesnik <mkolesni@redhat.com>

…em#1472) * E2E: Add a validation for E/W GW to get LB IP This might take a while, expecially on a public cloud. Make sure the tests wait for the IP to be assigned before proceeding, otherwise they'll get stuck later waiting for the communication to proceed (and sometimes fail). Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> * E2E: Add check that multicluster gateway rechable To rule out infrastructure problems, a sanity check is added to try and reach the gateway on the other side. If this check fails, this could indicate an infrastructure problem. Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> * E2E: Improve connectivity checking in multicluster Having curl fetch the URL several times saves time waiting on responses since it's much cheaper than each `exec` call. This makes successful tests finish much faster instead of waiting for the response to randomly land on the expected version. Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> --------- Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> Signed-off-by: Daniel Grimm <dgrimm@redhat.com>

mkolesnik requested a review from a team as a code owner December 18, 2025 19:09

istio-testing added the size/M label Dec 18, 2025

sridhargaddam reviewed Dec 26, 2025

View reviewed changes

tests/e2e/multicluster/common.go Outdated Show resolved Hide resolved

mkolesnik force-pushed the e2e-timeouts branch from d4c54bd to 73aebf8 Compare January 7, 2026 14:00

mkolesnik added 2 commits January 7, 2026 16:03

E2E: Add check that multicluster gateway rechable

8e2cecf

To rule out infrastructure problems, a sanity check is added to try and reach the gateway on the other side. If this check fails, this could indicate an infrastructure problem. Signed-off-by: Mike Kolesnik <mkolesni@redhat.com>

mkolesnik force-pushed the e2e-timeouts branch from 73aebf8 to 5129c7a Compare January 7, 2026 14:04

mkolesnik requested a review from sridhargaddam January 8, 2026 06:55

sridhargaddam approved these changes Jan 8, 2026

View reviewed changes

istio-testing merged commit e5b85f4 into istio-ecosystem:main Jan 8, 2026
17 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

E2E: Fixes to stabilize the multicluster tests further#1472

E2E: Fixes to stabilize the multicluster tests further#1472
istio-testing merged 3 commits intoistio-ecosystem:mainfrom
mkolesnik:e2e-timeouts

mkolesnik commented Dec 18, 2025

Uh oh!

codecov bot commented Dec 18, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

mkolesnik commented Dec 18, 2025

Uh oh!

codecov bot commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov bot commented Dec 18, 2025 •

edited

Loading