OCPBUGS-104543: fix test isolation bug in EnsureDefaultSecurityGroupTagsTest

[ironcladlou]

Before this commit, EnsureDefaultSecurityGroupTagsTest() mutated the cluster but used a fire-and-forget deferred cleanup, causing downstream tests which observe the same state to sometimes see the polluted state, leading to failed assertions (e.g. EnsureInfrastructureResourceTagsTest()).

The test also read cached hostedcluster state from the test context which is mutated by other tests. That issue is systemic.

This commit updates the deferred cleanup method to await convergence of the mutation rollback to make the test more fully isolated. It also uses a targeted fix for the cached hostedcluster lookup by fetching a fresh instance for assertions.

Summary by CodeRabbit

• Bug Fixes
  • Improved verification that removed AWS infrastructure tags are successfully cleared after restoring the HostedCluster’s original tags.
  • Added retry handling to account for eventual updates, with verification continuing for up to five minutes.
  • Refined tag validation to use the latest HostedCluster information, reducing the risk of inaccurate results from stale data.

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[openshift-ci-robot]

@ironcladlou: This pull request references Jira Issue OCPBUGS-104543, which is invalid:

• expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Before this commit, EnsureDefaultSecurityGroupTagsTest() mutated the cluster but used a fire-and-forget deferred cleanup, causing downstream tests which observe the same state to sometimes see the polluted state, leading to failed assertions (e.g. EnsureInfrastructureResourceTagsTest()).

This commit updates the deferred cleanup method to await convergence of the mutation rollback to make the test more fully isolated.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

📝 Walkthrough

Walkthrough

The AWS hosted cluster end-to-end test now validates tag cleanup. After restoring the original AWS resource tags, it retrieves the HostedCluster Infrastructure resource and waits up to five minutes for the removed day-2 tag to disappear. The infrastructure tag test also re-fetches the HostedCluster from the management API before reading AWS resource tags.

Possibly related PRs

• openshift/hypershift#9229: Updates the same test to re-fetch the HostedCluster and avoid stale cached state.

Suggested reviewers: devguyio, muraee

🚥 Pre-merge checks | ✅ 10 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Test Structure And Quality	⚠️ Warning	The cleanup uses a bounded 5-minute Eventually and restores the HostedCluster, but new Get and status assertions at lines 122-124 have no diagnostic messages.	Add meaningful messages to each new cleanup assertion, such as identifying the infrastructure object and the missing platform or AWS status.

✅ Passed checks (10 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	The patch adds no Ginkgo title declarations. All existing It, When, Context, and Describe titles use static string literals without run-dependent values.
Topology-Aware Scheduling Compatibility	✅ Passed	The only changed file is an AWS end-to-end test; the diff adds cleanup polling and a fresh API read, with no deployment, controller, replica, affinity, node, taint, or topology scheduling changes.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	The PR adds no new Ginkgo test declarations. Added code uses management and hosted-cluster clients only, with no IPv4 literals, IP parsing, public URLs, or external service connections.
No-Weak-Crypto	✅ Passed	The PR adds only HostedCluster/AWS tag cleanup and re-fetch logic; no MD5, SHA1, DES, RC4, Blowfish, ECB, custom crypto, or secret comparisons appear in the diff.
Container-Privileges	✅ Passed	The PR changes only a Go e2e test; no container/Kubernetes manifest or privilege-related setting was added.
No-Sensitive-Data-In-Logs	✅ Passed	The PR adds no logging calls or sensitive interpolations. New output is limited to fixed day-2 tag text and assertion results.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the test isolation fix and names the affected test, matching the pull request objectives.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 44.96%. Comparing base (4009f0f) to head (ca12889).
⚠️ Report is 39 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9228      +/-   ##
==========================================
+ Coverage   44.95%   44.96%   +0.01%     
==========================================
  Files         778      778              
  Lines       97434    97452      +18     
==========================================
+ Hits        43797    43820      +23     
+ Misses      50615    50607       -8     
- Partials     3022     3025       +3     

see 2 files with indirect coverage changes

Flag	Coverage Δ
cmd-support	38.62% <ø> (ø)
cpo-hostedcontrolplane	47.28% <ø> (ø)
cpo-other	45.67% <ø> (ø)
hypershift-operator	55.00% <ø> (+0.05%)	⬆️
other	34.30% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ironcladlou]

/jira refresh

[openshift-ci-robot]

@ironcladlou: This pull request references Jira Issue OCPBUGS-104543, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[ironcladlou]

/pipeline required

[openshift-merge-bot]

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks-4-22
/test e2e-aws-4-22
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-azure-self-managed
/test e2e-v2-gke

[ironcladlou]

/test e2e-v2-azure-self-managed

[ironcladlou]

Pretty sure the issue is not limited to the cleanup, it's also an issue with the test reading stale hostedcluster state, which I'm addressing systemically in #9229

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[openshift-ci-robot]

@ironcladlou: This pull request references Jira Issue OCPBUGS-104543, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

Before this commit, EnsureDefaultSecurityGroupTagsTest() mutated the cluster but used a fire-and-forget deferred cleanup, causing downstream tests which observe the same state to sometimes see the polluted state, leading to failed assertions (e.g. EnsureInfrastructureResourceTagsTest()).

The test also read cached hostedcluster state from the test context which is mutated by other tests. That issue is systemic.

This commit updates the deferred cleanup method to await convergence of the mutation rollback to make the test more fully isolated. It also uses a targeted fix for the cached hostedcluster lookup by fetching a fresh instance for assertions.

Summary by CodeRabbit

• Bug Fixes
• Improved verification that removed AWS infrastructure tags are successfully cleared after restoring the HostedCluster’s original tags.
• Added retry handling to account for eventual updates, with verification continuing for up to five minutes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[ironcladlou]

/test e2e-v2-aws

[ironcladlou]

With the latest commit I was unable to reproduce the issue with 10 consecutive e2e v2 aws runs locally

/pipeline required

[openshift-merge-bot]

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks-4-22
/test e2e-aws-4-22
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-azure-self-managed
/test e2e-v2-gke

[openshift-merge-bot]

Tests from second stage were triggered manually. Pipeline can be controlled only manually, until HEAD changes. Use command to trigger second stage.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bryan-cox, ironcladlou

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• test/e2e/OWNERS [bryan-cox,ironcladlou]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[cwbotbot]

Test Results

e2e-aws

• Status: ✅ PASS
• Started: 2026-08-06T11:46:12Z
• View Job
• View Job History

e2e-aks

• Status: ✅ PASS
• Started: 2026-08-05T20:20:30Z
• View Job
• View Job History

[ironcladlou]

/retest

[ironcladlou]

/verified by e2e regression analysis and manual testing

executed full v2 e2e aws workflow (create-guests / run-tests) 10x against with no errors

[openshift-ci-robot]

@ironcladlou: This PR has been marked as verified by e2e regression analysis and manual testing.

Details

In response to this:

/verified by e2e regression analysis and manual testing

executed full v2 e2e aws workflow (create-guests / run-tests) 10x against with no errors

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[ironcladlou]

/test e2e-kubevirt-aws-ovn-reduced

[openshift-ci]

@ironcladlou: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci-robot]

@ironcladlou: Jira Issue Verification Checks: Jira Issue OCPBUGS-104543
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-104543 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

Before this commit, EnsureDefaultSecurityGroupTagsTest() mutated the cluster but used a fire-and-forget deferred cleanup, causing downstream tests which observe the same state to sometimes see the polluted state, leading to failed assertions (e.g. EnsureInfrastructureResourceTagsTest()).

The test also read cached hostedcluster state from the test context which is mutated by other tests. That issue is systemic.

This commit updates the deferred cleanup method to await convergence of the mutation rollback to make the test more fully isolated. It also uses a targeted fix for the cached hostedcluster lookup by fetching a fresh instance for assertions.

Summary by CodeRabbit

• Bug Fixes
• Improved verification that removed AWS infrastructure tags are successfully cleared after restoring the HostedCluster’s original tags.
• Added retry handling to account for eventual updates, with verification continuing for up to five minutes.
• Refined tag validation to use the latest HostedCluster information, reducing the risk of inaccurate results from stale data.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-merge-robot]

Fix included in release 5.0.0-0.nightly-2026-08-07-105601