Improve e2e test cleanup by recording & cleaning up

[mkolesnik]

What type of PR is this?

• Enhancement / New Feature
• Bug Fix
• Refactor
• Optimization
• Test
• Documentation Update

What this PR does / why we need it:

This set of commits adds a Cleaner type which can record the cluster state and revert to that state by removing resources added via tests.

Currently only pertinent resources are tracked for the sake of simplicity, but this can be further expanded and generalized if necessary.

Cleaning up the operator is handled automatically, and when the operator is deployed externally if would be skipped (as it's recorded at the start of the test).

Which issue(s) this PR fixes:

Fixes #833

Related Issue/PR #

Additional information:

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 76.31%. Comparing base (b01d498) to head (d180ccd).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #889   +/-   ##
=======================================
  Coverage   76.31%   76.31%           
=======================================
  Files          44       44           
  Lines        2685     2685           
=======================================
  Hits         2049     2049           
  Misses        543      543           
  Partials       93       93           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[fjglira]

Hey @mkolesnik, can we add a small explanation of the use of the cleaner into the docs here? I think writing a test will be great (I'll also take a look this week to those docs to update them because I think they are not completely up to date)

[mkolesnik]

Hey @mkolesnik, can we add a small explanation of the use of the cleaner into the docs here? I think writing a test will be great (I'll also take a look this week to those docs to update them because I think they are not completely up to date)

Sure thing!
I think I'll add it to "Best practices" unless you have a suggestion?

[fjglira]

Hey @mkolesnik, can we add a small explanation of the use of the cleaner into the docs here? I think writing a test will be great (I'll also take a look this week to those docs to update them because I think they are not completely up to date)

Sure thing! I think I'll add it to "Best practices" unless you have a suggestion?

Sounds good to me

[fjglira]

lgtm, I really like this approarch of saving the current state of the cluster to avoid missing cleanup during teardown

[fjglira]

STEP: Cleaning up the workloadgroups.networking.istio.io-v1alpha3-admin ClusterRole @ 06/10/25 13:14:27.013
  STEP: Cleaning up the workloadgroups.networking.istio.io-v1alpha3-crdview ClusterRole @ 06/10/25 13:14:27.214
  STEP: Cleaning up the workloadgroups.networking.istio.io-v1alpha3-edit ClusterRole @ 06/10/25 13:14:27.414
  STEP: Cleaning up the workloadgroups.networking.istio.io-v1alpha3-view ClusterRole @ 06/10/25 13:14:27.614
  STEP: Cleaning up the workloadgroups.networking.istio.io-v1beta1-admin ClusterRole @ 06/10/25 13:14:27.814
  STEP: Waiting for resources to be deleted @ 06/10/25 13:14:28.015
  * Finished cleaning up resources
  << Timeline
  [FAILED] Expected success, but got an error:
      <*fmt.wrapError | 0xc0019efc00>: 
      client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
      {
          msg: "client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
          err: <*errors.errorString | 0xc001bbdd30>{
              s: "rate: Wait(n=1) would exceed context deadline",
          },
      }
  In [AfterAll] at: /home/prow/go/src/github.com/istio-ecosystem/sail-operator/tests/e2e/util/cleaner/cleaner.go:270 @ 06/10/25 13:14:09.213

The client is rate-limiting us? Too many deletions?

[mkolesnik]

The client is rate-limiting us? Too many deletions?

No, the problem is that OLM recreates these resources so we wait until it times out.
I addressed it (hopefully).

[fjglira]

LGTM

[fjglira]

@pbajjuri20, when this PR is merged, please take a look at the errors on P and Z execution. They should be solved