OCPBUGS-99783: fix(e2e): check pre-upgrade HO version for shared role support - #8891
Conversation
|
Pipeline controller notification For optional jobs, comment This repository is configured in: LGTM mode |
📝 WalkthroughWalkthroughThis change modifies the HyperShift operator upgrade e2e test to add a version-gated workaround. When the test environment runs a HyperShift version earlier than 421, the test disables Changes
Related Issues: None specified. Related PRs: None specified. Suggested Labels: test, e2e Suggested Reviewers: None specified. 🐰 A hop, a check, a version's tale, 🚥 Pre-merge checks | ✅ 11✅ Passed checks (11 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: devguyio The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/autocc |
|
/auto-cc |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8891 +/- ##
==========================================
+ Coverage 44.51% 44.56% +0.04%
==========================================
Files 774 774
Lines 96997 97003 +6
==========================================
+ Hits 43179 43228 +49
+ Misses 50830 50783 -47
- Partials 2988 2992 +4 see 6 files with indirect coverage changes
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
|
/lgtm |
|
Scheduling tests matching the |
|
I have all the evidence I need. The failure is crystal clear — it's an AWS IAM OIDC provider quota exhaustion issue in the shared CI AWS account, completely unrelated to the PR's code changes. Let me produce the final report. Test Failure Analysis CompleteJob Information
Test Failure AnalysisErrorSummaryThe e2e test suite failed immediately during Root CauseThe root cause is AWS IAM OIDC provider quota exhaustion in the shared CI AWS account ( How it happens:
Why this is unrelated to PR #8891:
Recommendations
Evidence
|
Test Resultse2e-aws
e2e-aks
|
|
/retest |
|
/retest-required
|
|
/retest-required same capacity error |
|
/retest-required |
|
/retest-required all steps passed including the 4.22 e2e, but the post cluster dump failed, timing out pulling an image. |
274450c to
4d75137
Compare
4d75137 to
d269545
Compare
|
@clebs the logic of shared role is only there for our CI infra, that logic isn't used at all for any production envs. Seth introduced it as a way to reduce the number of needed roles on our CI clusters. |
|
/retest-required |
d269545 to
2c5ac5f
Compare
|
/pipeline required |
|
Scheduling tests matching the |
|
/test e2e-aws |
|
/retest-required |
| // The supported-versions ConfigMap is reconciled asynchronously by the HO after | ||
| // its deployment becomes Available, so poll briefly to allow for controller startup. | ||
| var preUpgradeHOVersion semver.Version | ||
| err = wait.PollUntilContextTimeout(ctx, 2*time.Second, 30*time.Second, true, func(ctx context.Context) (bool, error) { |
There was a problem hiding this comment.
failure to read the version is fatal, so an arbitrary inner timeouts like this only add another flake vector... generally I like to rely on the highest level context possible and let the CI timeout take care of it (i.e. let it try until the overall test times out). If it ends up getting stuck here in practice that would be indicative of another issue to solve. So basically I'm suggesting just use PollUntilContextCancel
There was a problem hiding this comment.
Good point, definitely makes sense. This should be considered as an unrecoverable failure and bubble up as a timeout CI test run, not an inner failure. Pushing an update.
Shared role credential reconciliation only landed in 4.21+. When upgrading from an older HO, the test must disable SharedRole to avoid failures on unsupported infrastructure. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Ahmed Abdalla <aabdelre@redhat.com>
2c5ac5f to
6555035
Compare
|
thanks! /lgtm |
|
Scheduling tests matching the |
|
@devguyio: This pull request references Jira Issue OCPBUGS-99783, which is valid. The bug has been moved to the POST state. 3 validation(s) were run on this bug
The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/retest-required |
|
/retest-required |
|
retested since I thought #9167 was merged, now that I see it's not, I aborted the e2e-aws job |
|
/retest-required |
|
/verified by e2e |
|
@devguyio: This PR has been marked as verified by DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@devguyio: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
@devguyio: Jira Issue Verification Checks: Jira Issue OCPBUGS-99783 Jira Issue OCPBUGS-99783 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Fix included in release 5.0.0-0.nightly-2026-07-30-103726 |
Summary
Context
#7331 introduced shared role support to be used in our CI infra as a measure for reducing our CI infra footprint. The PR was not backported, that's why hypershift cli prior to 4.21 does not include this feature.
Since we run all our e2e tests from main, the TestUpgradeHypershiftOperator needs to skip this flag when creating a hypershift cluster using the hypershift cli for hypershift operator < 4.21.
Changes
supported-versionsConfigMap it reconciles, usingsupportedversion.GetLatestSupportedOCPVersion()Related
Test plan
e2e-aws-upgrade-hypershift-operatoron release-4.20 should pass (shared role disabled)e2e-aws-upgrade-hypershift-operatoron release-4.21+ should still use shared role🤖 Generated with Claude Code