Simplify logic when using `KEEP_ON_FAILURE` by mkolesnik · Pull Request #858 · istio-ecosystem/sail-operator

mkolesnik · 2025-05-29T11:28:21Z

When using KEEP_ON_FAILURE=true also set SKIP_CLEANUP=true. This keeps the cluster running after the tests, without a need to defined our own trap logic.

The downside is that the cluster will keep running even if the tests passed, but seeing as using the KEEP_ON_FAILURE flag is excpected mainly when there are failures, it seems that it's resonable to do.

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

Related Issue/PR #832

Additional information:

When using `KEEP_ON_FAILURE=true` also set `SKIP_CLEANUP=true`. This keeps the cluster running after the tests, without a need to defined our own trap logic. The downside is that the cluster will keep running even if the tests passed, but seeing as using the `KEEP_ON_FAILURE` flag is excpected mainly when there are failures, it seems that it's resonable to do. Signed-off-by: Mike Kolesnik <mkolesni@redhat.com>

codecov · 2025-05-29T11:32:25Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 76.32%. Comparing base (dee0d23) to head (86d2d49).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #858      +/-   ##
==========================================
- Coverage   76.47%   76.32%   -0.16%     
==========================================
  Files          44       44              
  Lines        2640     2640              
==========================================
- Hits         2019     2015       -4     
- Misses        533      536       +3     
- Partials       88       89       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

* upstream/main: Simplify logic when using `KEEP_ON_FAILURE` (istio-ecosystem#858) Add KEEP_ON_FAILURE flag to e2e kind tests (istio-ecosystem#832) switch to ubi-micro (istio-ecosystem#856) Speed up docs tests by running on a single cluster (istio-ecosystem#855) Add misspell to do basic spell validation in our docs (istio-ecosystem#853)

* upstream/main: Updates release and nightly-images GH workflows to work with new FBC used in community-operators-prod (istio-ecosystem#854) Simplify logic when using `KEEP_ON_FAILURE` (istio-ecosystem#858) Add KEEP_ON_FAILURE flag to e2e kind tests (istio-ecosystem#832) switch to ubi-micro (istio-ecosystem#856) Speed up docs tests by running on a single cluster (istio-ecosystem#855) Add misspell to do basic spell validation in our docs (istio-ecosystem#853)

* upstream/main: Add to vendors_default the possibility to add values for IstioCNI (istio-ecosystem#845) Updates release and nightly-images GH workflows to work with new FBC used in community-operators-prod (istio-ecosystem#854) Simplify logic when using `KEEP_ON_FAILURE` (istio-ecosystem#858) Add KEEP_ON_FAILURE flag to e2e kind tests (istio-ecosystem#832) switch to ubi-micro (istio-ecosystem#856) Speed up docs tests by running on a single cluster (istio-ecosystem#855) Add misspell to do basic spell validation in our docs (istio-ecosystem#853)

* upstream/main: Split Documentation: splitting docs, adding table of content and fixing links (istio-ecosystem#881) Do not block the workflow on errors when publishing to operatorhub.io (istio-ecosystem#877) chore(docs): cleans up IstioCNI section (istio-ecosystem#867) Force push to operator hub fork (istio-ecosystem#873) Make a yaml linter in the community-operators-prod happy (istio-ecosystem#869) Adding description for release workflow run (istio-ecosystem#868) Docs: adding istiod HA documentations (istio-ecosystem#865) Add to vendors_default the possibility to add values for IstioCNI (istio-ecosystem#845) Updates release and nightly-images GH workflows to work with new FBC used in community-operators-prod (istio-ecosystem#854) Simplify logic when using `KEEP_ON_FAILURE` (istio-ecosystem#858) Add KEEP_ON_FAILURE flag to e2e kind tests (istio-ecosystem#832) switch to ubi-micro (istio-ecosystem#856) Speed up docs tests by running on a single cluster (istio-ecosystem#855) Add misspell to do basic spell validation in our docs (istio-ecosystem#853)

* upstream/main: Fix "Check broken link" workflow (istio-ecosystem#887) Split Documentation: splitting docs, adding table of content and fixing links (istio-ecosystem#881) Do not block the workflow on errors when publishing to operatorhub.io (istio-ecosystem#877) chore(docs): cleans up IstioCNI section (istio-ecosystem#867) Force push to operator hub fork (istio-ecosystem#873) Make a yaml linter in the community-operators-prod happy (istio-ecosystem#869) Adding description for release workflow run (istio-ecosystem#868) Docs: adding istiod HA documentations (istio-ecosystem#865) Add to vendors_default the possibility to add values for IstioCNI (istio-ecosystem#845) Updates release and nightly-images GH workflows to work with new FBC used in community-operators-prod (istio-ecosystem#854) Simplify logic when using `KEEP_ON_FAILURE` (istio-ecosystem#858) Add KEEP_ON_FAILURE flag to e2e kind tests (istio-ecosystem#832) switch to ubi-micro (istio-ecosystem#856) Speed up docs tests by running on a single cluster (istio-ecosystem#855) Add misspell to do basic spell validation in our docs (istio-ecosystem#853)

* upstream/main: Update github action for broken links (istio-ecosystem#888) Fix "Check broken link" workflow (istio-ecosystem#887) Split Documentation: splitting docs, adding table of content and fixing links (istio-ecosystem#881) Do not block the workflow on errors when publishing to operatorhub.io (istio-ecosystem#877) chore(docs): cleans up IstioCNI section (istio-ecosystem#867) Force push to operator hub fork (istio-ecosystem#873) Make a yaml linter in the community-operators-prod happy (istio-ecosystem#869) Adding description for release workflow run (istio-ecosystem#868) Docs: adding istiod HA documentations (istio-ecosystem#865) Add to vendors_default the possibility to add values for IstioCNI (istio-ecosystem#845) Updates release and nightly-images GH workflows to work with new FBC used in community-operators-prod (istio-ecosystem#854) Simplify logic when using `KEEP_ON_FAILURE` (istio-ecosystem#858) Add KEEP_ON_FAILURE flag to e2e kind tests (istio-ecosystem#832) switch to ubi-micro (istio-ecosystem#856) Speed up docs tests by running on a single cluster (istio-ecosystem#855) Add misspell to do basic spell validation in our docs (istio-ecosystem#853)

* upstream/main: Enchance debug information when e2e fails: add istioctl proxy status (istio-ecosystem#891) Update github action for broken links (istio-ecosystem#888) Fix "Check broken link" workflow (istio-ecosystem#887) Split Documentation: splitting docs, adding table of content and fixing links (istio-ecosystem#881) Do not block the workflow on errors when publishing to operatorhub.io (istio-ecosystem#877) chore(docs): cleans up IstioCNI section (istio-ecosystem#867) Force push to operator hub fork (istio-ecosystem#873) Make a yaml linter in the community-operators-prod happy (istio-ecosystem#869) Adding description for release workflow run (istio-ecosystem#868) Docs: adding istiod HA documentations (istio-ecosystem#865) Add to vendors_default the possibility to add values for IstioCNI (istio-ecosystem#845) Updates release and nightly-images GH workflows to work with new FBC used in community-operators-prod (istio-ecosystem#854) Simplify logic when using `KEEP_ON_FAILURE` (istio-ecosystem#858) Add KEEP_ON_FAILURE flag to e2e kind tests (istio-ecosystem#832) switch to ubi-micro (istio-ecosystem#856) Speed up docs tests by running on a single cluster (istio-ecosystem#855) Add misspell to do basic spell validation in our docs (istio-ecosystem#853)

* upstream/main: Improve e2e test cleanup by recording & cleaning up (istio-ecosystem#889) Enchance debug information when e2e fails: add istioctl proxy status (istio-ecosystem#891) Update github action for broken links (istio-ecosystem#888) Fix "Check broken link" workflow (istio-ecosystem#887) Split Documentation: splitting docs, adding table of content and fixing links (istio-ecosystem#881) Do not block the workflow on errors when publishing to operatorhub.io (istio-ecosystem#877) chore(docs): cleans up IstioCNI section (istio-ecosystem#867) Force push to operator hub fork (istio-ecosystem#873) Make a yaml linter in the community-operators-prod happy (istio-ecosystem#869) Adding description for release workflow run (istio-ecosystem#868) Docs: adding istiod HA documentations (istio-ecosystem#865) Add to vendors_default the possibility to add values for IstioCNI (istio-ecosystem#845) Updates release and nightly-images GH workflows to work with new FBC used in community-operators-prod (istio-ecosystem#854) Simplify logic when using `KEEP_ON_FAILURE` (istio-ecosystem#858) Add KEEP_ON_FAILURE flag to e2e kind tests (istio-ecosystem#832) switch to ubi-micro (istio-ecosystem#856) Speed up docs tests by running on a single cluster (istio-ecosystem#855) Add misspell to do basic spell validation in our docs (istio-ecosystem#853)

When using `KEEP_ON_FAILURE=true` also set `SKIP_CLEANUP=true`. This keeps the cluster running after the tests, without a need to defined our own trap logic. The downside is that the cluster will keep running even if the tests passed, but seeing as using the `KEEP_ON_FAILURE` flag is excpected mainly when there are failures, it seems that it's resonable to do. Signed-off-by: Mike Kolesnik <mkolesni@redhat.com>

* upstream/main: Improve e2e test cleanup by recording & cleaning up (istio-ecosystem#889) Enchance debug information when e2e fails: add istioctl proxy status (istio-ecosystem#891) Update github action for broken links (istio-ecosystem#888) Fix "Check broken link" workflow (istio-ecosystem#887) Split Documentation: splitting docs, adding table of content and fixing links (istio-ecosystem#881) Do not block the workflow on errors when publishing to operatorhub.io (istio-ecosystem#877) chore(docs): cleans up IstioCNI section (istio-ecosystem#867) Force push to operator hub fork (istio-ecosystem#873) Make a yaml linter in the community-operators-prod happy (istio-ecosystem#869) Adding description for release workflow run (istio-ecosystem#868) Docs: adding istiod HA documentations (istio-ecosystem#865) Add to vendors_default the possibility to add values for IstioCNI (istio-ecosystem#845) Updates release and nightly-images GH workflows to work with new FBC used in community-operators-prod (istio-ecosystem#854) Simplify logic when using `KEEP_ON_FAILURE` (istio-ecosystem#858) Add KEEP_ON_FAILURE flag to e2e kind tests (istio-ecosystem#832) switch to ubi-micro (istio-ecosystem#856) Speed up docs tests by running on a single cluster (istio-ecosystem#855) Add misspell to do basic spell validation in our docs (istio-ecosystem#853)

* Add KEEP_ON_FAILURE flag to e2e kind tests (#832) Using this flag allows debugging the failed test on the environment, instead of destroying it. The default behavior is still to clean up the cluster when failing. Fixes: #675 Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> * Simplify logic when using `KEEP_ON_FAILURE` (#858) When using `KEEP_ON_FAILURE=true` also set `SKIP_CLEANUP=true`. This keeps the cluster running after the tests, without a need to defined our own trap logic. The downside is that the cluster will keep running even if the tests passed, but seeing as using the `KEEP_ON_FAILURE` flag is excpected mainly when there are failures, it seems that it's resonable to do. Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> * Speed up docs tests by running on a single cluster (#855) Refactor the tests to run all tags on a single cluster without redeploying it. After each tag the cluster is cleaned back to its original state. Dual stack tests are still being run in their own cluster, as it needs to be deployed in dual stack mode to support it. Fixes: #827 Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> * Improve e2e test cleanup by recording & cleaning up (#889) * Consolidate namespace cleanup in a new package The new package tracks namespaces and cleans up any namespaces it doesn't know about. The new `Cleaner` type can record and clean up the resources, given it's called with `Record()` before the tests and `Cleanup()` after. This is an initial change and more resources will be added. Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> * Add CRD/CR cleanup logic for tests Now the `Cleaner` can also record and clean up any rogue CRDs/CRs that were created during the tests. Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> * Replace operator uninstall with Cleanup Now that Cleanup takes care of all resources that we care about, replace the operator uninstall with it. Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> * Add docs how to use `Cleaner` Add docs to E2E tests about the `Cleaner` and how it should be used. Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> --------- Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> * Add `make cluster` command (#705) Add the ability to create a cluster for testing by using `make cluster`. Multiple clusters can be created by utilizing the `MULTICLUSTER=true` option. This allows developers to easily create a cluster to run manual testing on. Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> --------- Signed-off-by: Mike Kolesnik <mkolesni@redhat.com> Co-authored-by: Mike Kolesnik <mkolesni@redhat.com>

* upstream/main: Improve e2e test cleanup by recording & cleaning up (istio-ecosystem#889) Enchance debug information when e2e fails: add istioctl proxy status (istio-ecosystem#891) Update github action for broken links (istio-ecosystem#888) Fix "Check broken link" workflow (istio-ecosystem#887) Split Documentation: splitting docs, adding table of content and fixing links (istio-ecosystem#881) Do not block the workflow on errors when publishing to operatorhub.io (istio-ecosystem#877) chore(docs): cleans up IstioCNI section (istio-ecosystem#867) Force push to operator hub fork (istio-ecosystem#873) Make a yaml linter in the community-operators-prod happy (istio-ecosystem#869) Adding description for release workflow run (istio-ecosystem#868) Docs: adding istiod HA documentations (istio-ecosystem#865) Add to vendors_default the possibility to add values for IstioCNI (istio-ecosystem#845) Updates release and nightly-images GH workflows to work with new FBC used in community-operators-prod (istio-ecosystem#854) Simplify logic when using `KEEP_ON_FAILURE` (istio-ecosystem#858) Add KEEP_ON_FAILURE flag to e2e kind tests (istio-ecosystem#832) switch to ubi-micro (istio-ecosystem#856) Speed up docs tests by running on a single cluster (istio-ecosystem#855) Add misspell to do basic spell validation in our docs (istio-ecosystem#853)

* upstream/main: Consolidate multicluster operator deploy/cleanup (istio-ecosystem#902) Improve e2e test cleanup by recording & cleaning up (istio-ecosystem#889) Enchance debug information when e2e fails: add istioctl proxy status (istio-ecosystem#891) Update github action for broken links (istio-ecosystem#888) Fix "Check broken link" workflow (istio-ecosystem#887) Split Documentation: splitting docs, adding table of content and fixing links (istio-ecosystem#881) Do not block the workflow on errors when publishing to operatorhub.io (istio-ecosystem#877) chore(docs): cleans up IstioCNI section (istio-ecosystem#867) Force push to operator hub fork (istio-ecosystem#873) Make a yaml linter in the community-operators-prod happy (istio-ecosystem#869) Adding description for release workflow run (istio-ecosystem#868) Docs: adding istiod HA documentations (istio-ecosystem#865) Add to vendors_default the possibility to add values for IstioCNI (istio-ecosystem#845) Updates release and nightly-images GH workflows to work with new FBC used in community-operators-prod (istio-ecosystem#854) Simplify logic when using `KEEP_ON_FAILURE` (istio-ecosystem#858) Add KEEP_ON_FAILURE flag to e2e kind tests (istio-ecosystem#832) switch to ubi-micro (istio-ecosystem#856) Speed up docs tests by running on a single cluster (istio-ecosystem#855) Add misspell to do basic spell validation in our docs (istio-ecosystem#853)

* upstream/main: Fix log message for cleaner when waiting (istio-ecosystem#903) Consolidate multicluster operator deploy/cleanup (istio-ecosystem#902) Improve e2e test cleanup by recording & cleaning up (istio-ecosystem#889) Enchance debug information when e2e fails: add istioctl proxy status (istio-ecosystem#891) Update github action for broken links (istio-ecosystem#888) Fix "Check broken link" workflow (istio-ecosystem#887) Split Documentation: splitting docs, adding table of content and fixing links (istio-ecosystem#881) Do not block the workflow on errors when publishing to operatorhub.io (istio-ecosystem#877) chore(docs): cleans up IstioCNI section (istio-ecosystem#867) Force push to operator hub fork (istio-ecosystem#873) Make a yaml linter in the community-operators-prod happy (istio-ecosystem#869) Adding description for release workflow run (istio-ecosystem#868) Docs: adding istiod HA documentations (istio-ecosystem#865) Add to vendors_default the possibility to add values for IstioCNI (istio-ecosystem#845) Updates release and nightly-images GH workflows to work with new FBC used in community-operators-prod (istio-ecosystem#854) Simplify logic when using `KEEP_ON_FAILURE` (istio-ecosystem#858) Add KEEP_ON_FAILURE flag to e2e kind tests (istio-ecosystem#832) switch to ubi-micro (istio-ecosystem#856) Speed up docs tests by running on a single cluster (istio-ecosystem#855) Add misspell to do basic spell validation in our docs (istio-ecosystem#853)

* upstream/main: (21 commits) Annotate only our clusters when using kind (istio-ecosystem#904) Fix log message for cleaner when waiting (istio-ecosystem#903) Consolidate multicluster operator deploy/cleanup (istio-ecosystem#902) Improve e2e test cleanup by recording & cleaning up (istio-ecosystem#889) Enchance debug information when e2e fails: add istioctl proxy status (istio-ecosystem#891) Update github action for broken links (istio-ecosystem#888) Fix "Check broken link" workflow (istio-ecosystem#887) Split Documentation: splitting docs, adding table of content and fixing links (istio-ecosystem#881) Do not block the workflow on errors when publishing to operatorhub.io (istio-ecosystem#877) chore(docs): cleans up IstioCNI section (istio-ecosystem#867) Force push to operator hub fork (istio-ecosystem#873) Make a yaml linter in the community-operators-prod happy (istio-ecosystem#869) Adding description for release workflow run (istio-ecosystem#868) Docs: adding istiod HA documentations (istio-ecosystem#865) Add to vendors_default the possibility to add values for IstioCNI (istio-ecosystem#845) Updates release and nightly-images GH workflows to work with new FBC used in community-operators-prod (istio-ecosystem#854) Simplify logic when using `KEEP_ON_FAILURE` (istio-ecosystem#858) Add KEEP_ON_FAILURE flag to e2e kind tests (istio-ecosystem#832) switch to ubi-micro (istio-ecosystem#856) Speed up docs tests by running on a single cluster (istio-ecosystem#855) ...

dgn · 2025-07-03T14:13:31Z

/cherry-pick release-1.26

istio-testing · 2025-07-03T14:14:08Z

@dgn: new pull request could not be created: failed to create pull request against istio-ecosystem/sail-operator#release-1.26 from head istio-testing:cherry-pick-858-to-release-1.26: status code 422 not one of [201], body: {"message":"Validation Failed","errors":[{"resource":"PullRequest","code":"custom","message":"No commits between istio-ecosystem:release-1.26 and istio-testing:cherry-pick-858-to-release-1.26"}],"documentation_url":"https://docs.github.com/rest/pulls/pulls#create-a-pull-request","status":"422"}

Details

In response to this:

/cherry-pick release-1.26

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

mkolesnik requested a review from a team as a code owner May 29, 2025 11:28

istio-testing added the size/S label May 29, 2025

mkolesnik mentioned this pull request May 29, 2025

Add KEEP_ON_FAILURE flag to e2e kind tests #832

Merged

6 tasks

fjglira approved these changes May 29, 2025

View reviewed changes

istio-testing merged commit 751850a into istio-ecosystem:main May 29, 2025
16 of 17 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Simplify logic when using `KEEP_ON_FAILURE`#858

Simplify logic when using `KEEP_ON_FAILURE`#858
istio-testing merged 1 commit intoistio-ecosystem:mainfrom
mkolesnik:skip-clean-keep-fail

mkolesnik commented May 29, 2025

Uh oh!

codecov bot commented May 29, 2025 •

edited

Loading

Uh oh!

Uh oh!

dgn commented Jul 3, 2025

Uh oh!

istio-testing commented Jul 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

mkolesnik commented May 29, 2025

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Additional information:

Uh oh!

codecov bot commented May 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

dgn commented Jul 3, 2025

Uh oh!

istio-testing commented Jul 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented May 29, 2025 •

edited

Loading