OCPBUGS-98763: patch AWS master IAM role in 4.21/4.22→5.0 upgrade jobs by mfbonfigli · Pull Request #81974 · openshift/release

mfbonfigli · 2026-07-15T20:20:42Z

The OpenShift installer did not include elasticloadbalancing:SetSecurityGroups in the master IAM role policy until 5.0.

Clusters installed with 4.21 or 4.22 and upgraded to 5.0 hit AccessDenied when the 5.0 CCM calls SetSecurityGroups on NLBs, since the master role is never updated during cluster upgrades.

This PR adds a new step openshift-upgrade-aws-patch-master-iam that attaches a dedicated inline policy with the missing permission to the master role before the upgrade test runs, and wires it into the pre phase of the three jobs that upgrade from 4.x to 5.0:

periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-aws-ovn-upgrade
periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-from-stable-4.21-e2e-aws-ovn-upgrade
periodic-ci-openshift-release-main-nightly-5.0-upgrade-from-stable-4.21-e2e-aws-ovn-upgrade-paused

Summary by CodeRabbit

Adds a new CI Operator step-registry step (openshift-upgrade-aws-patch-master-iam) in the openshift/release config to pre-patch AWS master IAM roles during OpenShift upgrades from 4.21/4.22 to 5.0. The step runs before the affected AWS upgrade workflows and, for 4.x → 5.x upgrades, uses aws iam put-role-policy to attach an inline policy granting elasticloadbalancing:SetSecurityGroups (skipping when the upgrade path doesn’t match). This wiring is added to the AWS OVN and paused-workers upgrade workflows to prevent CCM NLB-related AccessDenied failures.

openshift-ci · 2026-07-15T20:20:46Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

openshift-ci-robot · 2026-07-15T20:20:49Z

@mfbonfigli: This pull request references Jira Issue OCPBUGS-98763, which is invalid:

expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

The OpenShift installer did not include elasticloadbalancing:SetSecurityGroups in the master IAM role policy until 5.0.

Clusters installed with 4.21 or 4.22 and upgraded to 5.0 hit AccessDenied when the 5.0 CCM calls SetSecurityGroups on NLBs, since the master role is never updated during cluster upgrades.

This PR adds a new step openshift-upgrade-aws-patch-master-iam that attaches a dedicated inline policy with the missing permission to the master role before the upgrade test runs, and wires it into the pre phase of the three jobs that upgrade from 4.x to 5.0:

periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-aws-ovn-upgrade

periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-from-stable-4.21-e2e-aws-ovn-upgrade

periodic-ci-openshift-release-main-nightly-5.0-upgrade-from-stable-4.21-e2e-aws-ovn-upgrade-paused

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

mfbonfigli · 2026-07-15T20:21:41Z

/jira refresh

openshift-ci-robot · 2026-07-15T20:21:48Z

@mfbonfigli: This pull request references Jira Issue OCPBUGS-98763, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (5.0.0) matches configured target version for branch (5.0.0)
bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

coderabbitai · 2026-07-15T20:23:03Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

Walkthrough

Adds a CI Operator step that conditionally grants elasticloadbalancing:SetSecurityGroups to the OpenShift master AWS IAM role during 4.x-to-5.x upgrades. The step includes ownership and metadata definitions and is added to the OVN and paused-workers AWS upgrade workflows.

Changes

AWS master IAM patch

Layer / File(s)	Summary
Define and implement IAM patch step `ci-operator/step-registry/openshift/upgrade/aws/patch-master-iam/*`	Adds ownership, metadata, step configuration, and a strict-mode Bash script that identifies eligible upgrades, derives AWS IAM identifiers, and applies the inline policy.
Wire IAM patch into upgrade workflows `ci-operator/step-registry/openshift/upgrade/aws/ovn/...`, `ci-operator/step-registry/openshift/upgrade/aws/pausedworkers/...`	Adds the IAM patch step to both workflows’ pre-step chains.

Estimated code review effort: 2 (Simple) | ~10 minutes

Sequence Diagram(s)

sequenceDiagram
  participant UpgradeWorkflow
  participant IAMPatchStep
  participant OpenShiftAPI
  participant AWSIAM
  UpgradeWorkflow->>IAMPatchStep: Run pre-step
  IAMPatchStep->>OpenShiftAPI: Read source and target release versions
  OpenShiftAPI-->>IAMPatchStep: Return release versions
  IAMPatchStep->>AWSIAM: Put inline policy on computed master role
  AWSIAM-->>IAMPatchStep: Return operation result

Suggested labels: lgtm, rehearsals-ack

Suggested reviewers: neisw, xueqzhan

🚥 Pre-merge checks | ✅ 15

✅ Passed checks (15 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: adding an AWS master IAM patch for 4.21/4.22 to 5.0 upgrade jobs.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	No Ginkgo test titles or test code were added; the diff only changes step-registry YAML, a shell helper, and OWNERS metadata.
Test Structure And Quality	✅ Passed	No Ginkgo test code was changed; the PR only touches step-registry YAML, metadata, OWNERS, and a shell script, so the test-structure check is not applicable.
Microshift Test Compatibility	✅ Passed	No new Ginkgo test definitions were added; the diff only changes step-registry YAML, a shell step, and OWNERS files.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	PR only adds step-registry YAML/bash for upgrade jobs; no new Ginkgo tests or SNO-relevant topology assumptions were introduced.
Topology-Aware Scheduling Compatibility	✅ Passed	Only step-registry/workflow wiring and a Bash IAM patch step changed; no pod spec, controller, affinity, nodeSelector, PDB, or spread constraints were added.
Ote Binary Stdout Contract	✅ Passed	No OTE binary/main process was changed; only step-registry YAML and a shell step script with normal logging, so the stdout contract isn’t implicated.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	No new Ginkgo e2e tests were added; the PR only changes step-registry workflows and a Bash IAM patch script.
No-Weak-Crypto	✅ Passed	Changed files only add IAM/workflow YAML and a Bash AWS policy patch; no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret comparisons found.
Container-Privileges	✅ Passed	Touched manifests only add a ref and workflow step; no privileged, hostPID/Network/IPC, SYS_ADMIN, root, or allowPrivilegeEscalation settings were added.
No-Sensitive-Data-In-Logs	✅ Passed	Changed logs are generic status messages; no passwords, tokens, PII, session IDs, hostnames, or customer data appeared in the new files.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

openshift-ci-robot · 2026-07-15T20:24:33Z

@mfbonfigli: This pull request references Jira Issue OCPBUGS-98763, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (5.0.0) matches configured target version for branch (5.0.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

The OpenShift installer did not include elasticloadbalancing:SetSecurityGroups in the master IAM role policy until 5.0.

Clusters installed with 4.21 or 4.22 and upgraded to 5.0 hit AccessDenied when the 5.0 CCM calls SetSecurityGroups on NLBs, since the master role is never updated during cluster upgrades.

This PR adds a new step openshift-upgrade-aws-patch-master-iam that attaches a dedicated inline policy with the missing permission to the master role before the upgrade test runs, and wires it into the pre phase of the three jobs that upgrade from 4.x to 5.0:

periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-aws-ovn-upgrade

periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-from-stable-4.21-e2e-aws-ovn-upgrade

periodic-ci-openshift-release-main-nightly-5.0-upgrade-from-stable-4.21-e2e-aws-ovn-upgrade-paused

Summary by CodeRabbit

Adds a reusable CI step that grants OpenShift AWS master IAM roles the elasticloadbalancing:SetSecurityGroups permission required by the OpenShift 5.0 CCM.

Runs the IAM patch before AWS OVN upgrade tests from OpenShift 4.21 and 4.22 to prevent NLB-related AccessDenied failures.

Registers the step with its script, metadata, ownership, and installer resource configuration.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

mfbonfigli · 2026-07-15T20:26:49Z

/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-aws-ovn-upgrade

openshift-merge-bot · 2026-07-15T20:28:28Z

@mfbonfigli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

mfbonfigli · 2026-07-15T23:46:06Z

/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-aws-ovn-upgrade

openshift-merge-bot · 2026-07-15T23:46:09Z

@mfbonfigli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

mfbonfigli · 2026-07-16T07:25:36Z

The test seems to be a flake unrelated to the change. The specific step added ran correctly:

[...]
[2026-07-16T00:39:08Z] Step phase pre succeeded after 47m39s.       
[2026-07-16T00:39:08Z] Running multi-stage phase test               
[2026-07-16T00:39:08Z] Running step e2e-aws-ovn-upgrade-openshift-upgrade-aws-patch-master-iam. 
[2026-07-16T00:42:53Z] Step e2e-aws-ovn-upgrade-openshift-upgrade-aws-patch-master-iam succeeded after 3m45s. 
[2026-07-16T00:42:53Z] Running step e2e-aws-ovn-upgrade-openshift-e2e-test. 
[...]

mfbonfigli · 2026-07-16T07:43:01Z

/pj-rehearse

openshift-merge-bot · 2026-07-16T07:43:03Z

@mfbonfigli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

mfbonfigli · 2026-07-16T12:08:39Z

/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-from-stable-4.21-e2e-aws-ovn-upgrade periodic-ci-openshift-release-main-nightly-5.0-upgrade-from-stable-4.21-e2e-aws-ovn-upgrade-paused

openshift-merge-bot · 2026-07-16T12:10:27Z

@mfbonfigli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

mfbonfigli · 2026-07-16T12:21:39Z

ci/rehearse/periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-aws-ovn-upgrade passed, the other seemed again unrelated failures so triggered a retry run

vr4manta · 2026-07-16T12:34:22Z

/lgtm

openshift-merge-bot · 2026-07-20T10:19:08Z

@mfbonfigli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

mfbonfigli · 2026-07-20T11:37:17Z

This job seems to be stuck on the new step:
https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_release/81974/rehearse-81974-periodic-ci-openshift-release-main-ci-4.22-upgrade-from-stable-4.21-from-stable-4.20-e2e-aws-ovn-upgrade/2079149880101572608#1:build-log.txt%3A1

this doesn't have upi-installer in the base images and the new step which normally takes 10 seconds has been running for over 20 minutes, I will keep monitoring. In case it is proven a regression, I will revert the CR to not use from: upi-installer given it touches 1633 jobs

mfbonfigli · 2026-07-20T12:37:32Z

Indeed the above e2e failed with error:

 * Container test is not ready with reason ImagePullBackOff and message Back-off pulling image "image-registry.openshift-image-registry.svc:5000/ci-op-kizx94z0/stable:upi-installer": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://image-registry.openshift-image-registry.svc:5000/ci-op-kizx94z0/stable:upi-installer: reading manifest upi-installer in image-registry.openshift-image-registry.svc:5000/ci-op-kizx94z0/stable: manifest unknown; artifact err: get manifest: build image source: reading manifest upi-installer in image-registry.openshift-image-registry.svc:5000/ci-op-kizx94z0/stable: manifest unknown

This proves that not all jobs carry a direct or indirect reference to upi-installer.

In particular at risk could be the 344 periodic-ci-* affected jobs where stable is populated from the assembled release payload and hence upi-installer is not in the payload. Out of these one should exclude those that explicitly define upi-installer in the base images. From an initial investigation, that might not be fully accurate, it seems that out of all the configs for these jobs:

87 configs have upi-installer in base_images → not affected
50 configs are missing it → from: upi-installer falls through to stable:upi-installer → not in the candidate payload → will fail

Given that scope of this PR is to enable e2e tests for the AWS BYO NLB feature in CCM, and not identify and retrofit tens of jobs to deprecate a pattern that is widely in use across the codebase, I believe it is appropriate to not use from: upi-installer but rely on an explicit version definition, and postpone any eventual deprecation of this pattern to a dedicated task.

For this reason am going to revert this PR to the original approach of using a specific version pin.

The OpenShift installer did not include elasticloadbalancing:SetSecurityGroups in the master IAM role policy until 4.23. Clusters installed with 4.21 or 4.22 and upgraded to 5.0 hit AccessDenied when the 5.0 CCM calls SetSecurityGroups on NLBs, since the master role is never updated during upgrade. Add a new step openshift-upgrade-aws-patch-master-iam that attaches a dedicated inline policy with the missing permission to the master role before the upgrade test runs. Wire it into the test phase of the three affected jobs: - periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-aws-ovn-upgrade - periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-from-stable-4.21-e2e-aws-ovn-upgrade - periodic-ci-openshift-release-main-nightly-5.0-upgrade-from-stable-4.21-e2e-aws-ovn-upgrade-paused Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

openshift-merge-bot · 2026-07-20T12:52:43Z

[REHEARSALNOTIFIER]
@mfbonfigli: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name	Repo	Type	Reason
pull-ci-openshift-kubernetes-master-e2e-aws-ovn-upgrade	openshift/kubernetes	presubmit	Registry content changed
pull-ci-openshift-kubernetes-master-e2e-aws-ovn-downgrade	openshift/kubernetes	presubmit	Registry content changed
pull-ci-openshift-kubernetes-release-5.1-e2e-aws-ovn-upgrade	openshift/kubernetes	presubmit	Registry content changed
pull-ci-openshift-kubernetes-release-5.1-e2e-aws-ovn-downgrade	openshift/kubernetes	presubmit	Registry content changed
pull-ci-openshift-kubernetes-release-5.0-e2e-aws-ovn-upgrade	openshift/kubernetes	presubmit	Registry content changed
pull-ci-openshift-kubernetes-release-5.0-e2e-aws-ovn-downgrade	openshift/kubernetes	presubmit	Registry content changed
pull-ci-openshift-kubernetes-release-4.23-e2e-aws-ovn-upgrade	openshift/kubernetes	presubmit	Registry content changed
pull-ci-openshift-kubernetes-release-4.23-e2e-aws-ovn-downgrade	openshift/kubernetes	presubmit	Registry content changed
pull-ci-openshift-kubernetes-release-4.22-e2e-aws-ovn-upgrade	openshift/kubernetes	presubmit	Registry content changed
pull-ci-openshift-kubernetes-release-4.22-e2e-aws-ovn-downgrade	openshift/kubernetes	presubmit	Registry content changed
pull-ci-openshift-kubernetes-release-4.21-e2e-aws-ovn-upgrade	openshift/kubernetes	presubmit	Registry content changed
pull-ci-openshift-kubernetes-release-4.21-e2e-aws-ovn-downgrade	openshift/kubernetes	presubmit	Registry content changed
pull-ci-openshift-kubernetes-release-4.20-e2e-aws-ovn-upgrade	openshift/kubernetes	presubmit	Registry content changed
pull-ci-openshift-kubernetes-release-4.20-e2e-aws-ovn-downgrade	openshift/kubernetes	presubmit	Registry content changed
pull-ci-openshift-kubernetes-release-4.19-e2e-aws-ovn-upgrade	openshift/kubernetes	presubmit	Registry content changed
pull-ci-openshift-kubernetes-release-4.19-e2e-aws-ovn-downgrade	openshift/kubernetes	presubmit	Registry content changed
pull-ci-openshift-kubernetes-release-4.18-e2e-aws-ovn-upgrade	openshift/kubernetes	presubmit	Registry content changed
pull-ci-openshift-kubernetes-release-4.18-e2e-aws-ovn-downgrade	openshift/kubernetes	presubmit	Registry content changed
pull-ci-openshift-kubernetes-release-4.17-e2e-aws-ovn-upgrade	openshift/kubernetes	presubmit	Registry content changed
pull-ci-openshift-kubernetes-release-4.17-e2e-aws-ovn-downgrade	openshift/kubernetes	presubmit	Registry content changed
pull-ci-openshift-kubernetes-release-4.16-e2e-aws-ovn-upgrade	openshift/kubernetes	presubmit	Registry content changed
pull-ci-openshift-kubernetes-release-4.16-e2e-aws-ovn-downgrade	openshift/kubernetes	presubmit	Registry content changed
pull-ci-openshift-kubernetes-release-4.15-e2e-aws-ovn-upgrade	openshift/kubernetes	presubmit	Registry content changed
pull-ci-openshift-kubernetes-release-4.15-e2e-aws-ovn-downgrade	openshift/kubernetes	presubmit	Registry content changed
pull-ci-openshift-kubernetes-release-4.14-e2e-aws-ovn-upgrade	openshift/kubernetes	presubmit	Registry content changed

A total of 1633 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

mfbonfigli · 2026-07-20T13:00:41Z

/pj-rehearse periodic-ci-openshift-release-main-ci-4.22-upgrade-from-stable-4.21-from-stable-4.20-e2e-aws-ovn-upgrade periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-aws-ovn-upgrade periodic-ci-openshift-release-main-nightly-5.0-upgrade-from-stable-4.21-e2e-aws-ovn-upgrade-paused

openshift-merge-bot · 2026-07-20T13:00:44Z

@mfbonfigli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

mfbonfigli · 2026-07-20T18:57:45Z

Rehearsal results

Three rehearsals were run against this PR. All three successfully executed the new openshift-upgrade-aws-patch-master-iam step. Here is the breakdown:

Job	Result	New step
`periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-aws-ovn-upgrade`	✅ PASS	✅ SUCCESS (step installs IAM permission for 4.22 to 5.0 upgrade)
`periodic-ci-openshift-release-main-ci-4.22-upgrade-from-stable-4.21-from-stable-4.20-e2e-aws-ovn-upgrade`	❌ FAIL (known flake)	✅ SUCCESS (step is no-op correctly for 4.x to 4.x upgrade)
`periodic-ci-openshift-release-main-nightly-5.0-upgrade-from-stable-4.21-e2e-aws-ovn-upgrade-paused`	❌ FAIL (known flake)	✅ SUCCESS (step installs IAM permisison for 4.21 to 5.0 upgrade)

Why the failures are unrelated to this PR

Both failures occur in the openshift-e2e-test step, after the cluster was successfully installed and upgraded. All pre-upgrade and upgrade steps passed, including the new step introduced by this PR. The failures appear to be unrelated and the fialing tests are consistently reporting as failing on Sippy.

What this PR does, confirmed working

The new step uses from_image pinned to ocp/5.0:upi-installer, so it resolves cleanly in all job types. The failure that was previously reported affecting some jobs that cannot resolve the upi-installer image tag is no longer present.

mfbonfigli · 2026-07-20T18:59:45Z

/pj-rehearse periodic-ci-openshift-release-main-nightly-4.22-upgrade-from-stable-4.20-e2e-aws-ovn-upgrade-paused

openshift-merge-bot · 2026-07-20T18:59:49Z

@mfbonfigli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

tthvo · 2026-07-20T19:09:44Z

/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-aws-ovn-upgrade

openshift-merge-bot · 2026-07-20T19:09:47Z

@tthvo: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

tthvo

/lgtm

Based on latest findings 👍

openshift-ci · 2026-07-20T19:35:57Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mfbonfigli, tthvo, vr4manta

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~ci-operator/step-registry/openshift/upgrade/OWNERS~~ [vr4manta]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tthvo · 2026-07-20T19:36:06Z

/hold cancel

We can wait for rehearsals and ack it as the final step.

openshift-ci · 2026-07-20T23:50:50Z

@mfbonfigli: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/rehearse/periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-from-stable-4.21-e2e-aws-ovn-upgrade	`b97ede2`	link	unknown	`/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-from-stable-4.21-e2e-aws-ovn-upgrade`
ci/rehearse/periodic-ci-openshift-release-main-ci-4.22-upgrade-from-stable-4.21-from-stable-4.20-e2e-aws-ovn-upgrade	`aa2c49d`	link	unknown	`/pj-rehearse periodic-ci-openshift-release-main-ci-4.22-upgrade-from-stable-4.21-from-stable-4.20-e2e-aws-ovn-upgrade`
ci/rehearse/periodic-ci-openshift-release-main-nightly-4.22-upgrade-from-stable-4.20-e2e-aws-ovn-upgrade-paused	`aa2c49d`	link	unknown	`/pj-rehearse periodic-ci-openshift-release-main-nightly-4.22-upgrade-from-stable-4.20-e2e-aws-ovn-upgrade-paused`
ci/rehearse/openshift/api/release-4.22/e2e-upgrade	`3fe979a`	link	unknown	`/pj-rehearse pull-ci-openshift-api-release-4.22-e2e-upgrade`
ci/rehearse/periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-aws-ovn-upgrade	`aa2c49d`	link	unknown	`/pj-rehearse periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-aws-ovn-upgrade`
ci/rehearse/periodic-ci-openshift-release-main-nightly-5.0-upgrade-from-stable-4.21-e2e-aws-ovn-upgrade-paused	`aa2c49d`	link	unknown	`/pj-rehearse periodic-ci-openshift-release-main-nightly-5.0-upgrade-from-stable-4.21-e2e-aws-ovn-upgrade-paused`
ci/rehearse/periodic-ci-openshift-release-main-ci-4.21-upgrade-from-stable-4.20-from-stable-4.19-e2e-aws-ovn-upgrade	`3fe979a`	link	unknown	`/pj-rehearse periodic-ci-openshift-release-main-ci-4.21-upgrade-from-stable-4.20-from-stable-4.19-e2e-aws-ovn-upgrade`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

mfbonfigli · 2026-07-21T08:13:51Z

The failing rehearsals appear all unrelated. Unfortunately most of the affected tests of this PR seem to be, according to Sippy, have a high fail rate, as explained above.

The last two test run where:

periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-aws-ovn-upgrade

This test previously passed, and in the re-run failed. The test has a pass percentage on sippy of 49%, so it is known to flake. The failing test cases are fully unrelated to this PR, and this PR step successfully completed installed the IAM permission. From the logs:

Detected 4.x to 5.x upgrade (4.22.0-0.ci-2026-07-19-051950 to 5.0.0-0.ci-2026-07-20-122541), patching master IAM role.
Adding inline policy ci-op-ff29p098-b66d0-zjvxl-master-upgrade-policy to role: ci-op-ff29p098-b66d0-zjvxl-master-role
Successfully added inline policy ci-op-ff29p098-b66d0-zjvxl-master-upgrade-policy to ci-op-ff29p098-b66d0-zjvxl-master-role

periodic-ci-openshift-release-main-nightly-4.22-upgrade-from-stable-4.20-e2e-aws-ovn-upgrade-paused

This one too shows up on Sippy as frequently affected by "internal infrastructure failures" manifesting with exact the same failed test cases as the run that failed here.
This PR step successfully completed skipping correctly to add the IAM permission since it's not a 4.x to 5.x cluster upgrade. From the logs:

Not a 4.x to 5.x upgrade (4.20.0-0.nightly-2026-07-19-034625 to 4.22.0-0.nightly-2026-07-20-035241), skipping.

mfbonfigli · 2026-07-22T11:49:13Z

/pj-rehearse ack

openshift-merge-bot · 2026-07-22T11:49:16Z

@mfbonfigli: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

openshift-ci-robot · 2026-07-22T12:07:27Z

@mfbonfigli: Jira Issue OCPBUGS-98763: All pull requests linked via external trackers have merged:

openshift/release#81974

Jira Issue OCPBUGS-98763 has been moved to the MODIFIED state.

Details

In response to this:

The OpenShift installer did not include elasticloadbalancing:SetSecurityGroups in the master IAM role policy until 5.0.

Clusters installed with 4.21 or 4.22 and upgraded to 5.0 hit AccessDenied when the 5.0 CCM calls SetSecurityGroups on NLBs, since the master role is never updated during cluster upgrades.

This PR adds a new step openshift-upgrade-aws-patch-master-iam that attaches a dedicated inline policy with the missing permission to the master role before the upgrade test runs, and wires it into the pre phase of the three jobs that upgrade from 4.x to 5.0:

periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-aws-ovn-upgrade

periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-from-stable-4.21-e2e-aws-ovn-upgrade

periodic-ci-openshift-release-main-nightly-5.0-upgrade-from-stable-4.21-e2e-aws-ovn-upgrade-paused

Summary by CodeRabbit

Adds a new CI Operator step-registry step (openshift-upgrade-aws-patch-master-iam) in the openshift/release config to pre-patch AWS master IAM roles during OpenShift upgrades from 4.21/4.22 to 5.0. The step runs before the affected AWS upgrade workflows and, for 4.x → 5.x upgrades, uses aws iam put-role-policy to attach an inline policy granting elasticloadbalancing:SetSecurityGroups (skipping when the upgrade path doesn’t match). This wiring is added to the AWS OVN and paused-workers upgrade workflows to prevent CCM NLB-related AccessDenied failures.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift#81974) The OpenShift installer did not include elasticloadbalancing:SetSecurityGroups in the master IAM role policy until 4.23. Clusters installed with 4.21 or 4.22 and upgraded to 5.0 hit AccessDenied when the 5.0 CCM calls SetSecurityGroups on NLBs, since the master role is never updated during upgrade. Add a new step openshift-upgrade-aws-patch-master-iam that attaches a dedicated inline policy with the missing permission to the master role before the upgrade test runs. Wire it into the test phase of the three affected jobs: - periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-e2e-aws-ovn-upgrade - periodic-ci-openshift-release-main-ci-5.0-upgrade-from-stable-4.22-from-stable-4.21-e2e-aws-ovn-upgrade - periodic-ci-openshift-release-main-nightly-5.0-upgrade-from-stable-4.21-e2e-aws-ovn-upgrade-paused Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 15, 2026

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jul 15, 2026

mfbonfigli force-pushed the OCPBUGS-98763/patch-master-iam-role-on-upgrade branch from cc627ad to b4e8831 Compare July 15, 2026 23:45

mfbonfigli marked this pull request as ready for review July 16, 2026 07:42

openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 16, 2026

openshift-ci Bot requested review from neisw and xueqzhan July 16, 2026 07:44

mfbonfigli mentioned this pull request Jul 16, 2026

SPLAT-2713: Reapply "Merge pull request #476" to bring in e2e for BYO SG for AWS NLB openshift/cluster-cloud-controller-manager-operator#492

Merged

openshift-ci Bot assigned vr4manta Jul 16, 2026

mfbonfigli force-pushed the OCPBUGS-98763/patch-master-iam-role-on-upgrade branch from 3fe979a to aa2c49d Compare July 20, 2026 12:45

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 20, 2026

tthvo reviewed Jul 20, 2026

View reviewed changes

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 20, 2026

openshift-ci Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 20, 2026

openshift-merge-bot Bot added the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Jul 22, 2026

openshift-merge-bot Bot merged commit c63238e into openshift:main Jul 22, 2026
11 of 15 checks passed

mfbonfigli mentioned this pull request Aug 4, 2026

Inject AWS IAM SetSecurityGroups perm on upgrades to 4.23 #82862

Merged

Uh oh!

Conversation

mfbonfigli commented Jul 15, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

openshift-ci Bot commented Jul 15, 2026

Uh oh!

openshift-ci-robot commented Jul 15, 2026

Uh oh!

mfbonfigli commented Jul 15, 2026

Uh oh!

openshift-ci-robot commented Jul 15, 2026

Uh oh!

coderabbitai Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

openshift-ci-robot commented Jul 15, 2026

Summary by CodeRabbit

Uh oh!

mfbonfigli commented Jul 15, 2026

Uh oh!

openshift-merge-bot Bot commented Jul 15, 2026

Uh oh!

mfbonfigli commented Jul 15, 2026

Uh oh!

openshift-merge-bot Bot commented Jul 15, 2026

Uh oh!

mfbonfigli commented Jul 16, 2026

Uh oh!

mfbonfigli commented Jul 16, 2026

Uh oh!

openshift-merge-bot Bot commented Jul 16, 2026

Uh oh!

mfbonfigli commented Jul 16, 2026

Uh oh!

openshift-merge-bot Bot commented Jul 16, 2026

Uh oh!

mfbonfigli commented Jul 16, 2026

Uh oh!

vr4manta commented Jul 16, 2026

Uh oh!

openshift-merge-bot Bot commented Jul 20, 2026

Uh oh!

mfbonfigli commented Jul 20, 2026

Uh oh!

mfbonfigli commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-merge-bot Bot commented Jul 20, 2026

Uh oh!

mfbonfigli commented Jul 20, 2026

Uh oh!

openshift-merge-bot Bot commented Jul 20, 2026

Uh oh!

mfbonfigli commented Jul 20, 2026

Rehearsal results

Why the failures are unrelated to this PR

What this PR does, confirmed working

Uh oh!

mfbonfigli commented Jul 20, 2026

Uh oh!

openshift-merge-bot Bot commented Jul 20, 2026

Uh oh!

tthvo commented Jul 20, 2026

Uh oh!

openshift-merge-bot Bot commented Jul 20, 2026

Uh oh!

tthvo left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci Bot commented Jul 20, 2026

Uh oh!

tthvo commented Jul 20, 2026

Uh oh!

openshift-ci Bot commented Jul 20, 2026

mfbonfigli commented Jul 15, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 15, 2026 •

edited

Loading

mfbonfigli commented Jul 20, 2026 •

edited

Loading

mfbonfigli commented Jul 21, 2026 •

edited

Loading