[release-4.21] OCPBUGS-93927: fix(cpo): deduplicate VPC endpoint subnets by AZ to prevent DuplicateSubnetsInSameZone by reedcort · Pull Request #8862 · openshift/hypershift

reedcort · 2026-06-29T15:46:25Z

What this PR does / why we need it:

Backport of #8651 to release-4.21.

When multiple NodePools reference subnets in the same AWS availability zone, the CPO's VPC endpoint reconciler now deduplicates them before calling CreateVpcEndpoint or ModifyVpcEndpoint, preventing the AWS DuplicateSubnetsInSameZone error.

Adds an in-memory AZ cache (sync.RWMutex + map) to the AWSEndpointServiceReconciler to avoid repeated DescribeSubnets calls
Lexicographic subnet selection ensures deterministic deduplication across concurrent reconciles
Graceful degradation: if DescribeSubnets fails (e.g. missing IAM permission), the original subnet list passes through unchanged
Adds ec2:DescribeSubnets to the control-plane-operator and shared-vpc-endpoint IAM policies

Which issue(s) this PR fixes:

Fixes https://issues.redhat.com/browse/OCPBUGS-93927

Special notes for your reviewer:

Adapted from the 4.22 backport (#8724) for AWS SDK v1 (aws-sdk-go) used on release-4.21. The logic is identical; only the AWS API call patterns differ (DescribeSubnetsWithContext, aws.StringValue, ec2iface.EC2API).

Checklist:

Subject and description added to both, commit and PR.
Relevant issues have been referenced.
This change includes docs.
This change includes unit tests.

🤖 Generated with Claude Code

…SubnetsInSameZone When multiple NodePools reference subnets in the same AWS availability zone, the VPC endpoint reconciler now deduplicates them before calling CreateVpcEndpoint or ModifyVpcEndpoint. This prevents the AWS DuplicateSubnetsInSameZone error that blocks endpoint reconciliation. The fix uses an in-memory cache (sync.RWMutex + map) to avoid repeated DescribeSubnets calls. If DescribeSubnets fails (e.g., missing IAM permission), the original subnet list is used unchanged, preserving existing behavior. Also adds ec2:DescribeSubnets to the control-plane-operator and shared-vpc-endpoint IAM policies. Closes: OCPBUGS-93927 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

openshift-merge-bot · 2026-06-29T15:46:28Z

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

openshift-ci-robot · 2026-06-29T15:46:32Z

@reedcort: This pull request references Jira Issue OCPBUGS-93927, which is invalid:

expected Jira Issue OCPBUGS-93927 to depend on a bug targeting a version in 4.22.0 and in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

What this PR does / why we need it:

Backport of #8651 to release-4.21.

When multiple NodePools reference subnets in the same AWS availability zone, the CPO's VPC endpoint reconciler now deduplicates them before calling CreateVpcEndpoint or ModifyVpcEndpoint, preventing the AWS DuplicateSubnetsInSameZone error.

Adds an in-memory AZ cache (sync.RWMutex + map) to the AWSEndpointServiceReconciler to avoid repeated DescribeSubnets calls

Lexicographic subnet selection ensures deterministic deduplication across concurrent reconciles

Graceful degradation: if DescribeSubnets fails (e.g. missing IAM permission), the original subnet list passes through unchanged

Adds ec2:DescribeSubnets to the control-plane-operator and shared-vpc-endpoint IAM policies

Which issue(s) this PR fixes:

Fixes https://issues.redhat.com/browse/OCPBUGS-93927

Special notes for your reviewer:

Adapted from the 4.22 backport (#8724) for AWS SDK v1 (aws-sdk-go) used on release-4.21. The logic is identical; only the AWS API call patterns differ (DescribeSubnetsWithContext, aws.StringValue, ec2iface.EC2API).

Checklist:

Subject and description added to both, commit and PR.

Relevant issues have been referenced.

This change includes docs.

This change includes unit tests.

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

coderabbitai · 2026-06-29T15:46:34Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 7cc07d00-2f42-4732-8825-d882d8061d69

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

reedcort · 2026-06-29T20:29:53Z

/jira refresh

openshift-ci-robot · 2026-06-29T20:29:57Z

@reedcort: This pull request references Jira Issue OCPBUGS-93927, which is invalid:

expected dependent Jira Issue OCPBUGS-88356 to target a version in 4.22.0, but it targets "4.22.z" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

reedcort · 2026-06-29T20:32:15Z

/jira refresh

openshift-ci-robot · 2026-06-29T20:32:22Z

@reedcort: This pull request references Jira Issue OCPBUGS-93927, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.21.z) matches configured target version for branch (4.21.z)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)
release note text is set and does not match the template
dependent bug Jira Issue OCPBUGS-88356 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
dependent Jira Issue OCPBUGS-88356 targets the "4.22.0" version, which is one of the valid target versions: 4.22.0
bug has dependents

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

reedcort · 2026-06-29T20:32:45Z

/area control-plane-operator
/area cli
/area platform/aws

reedcort · 2026-07-01T11:16:40Z

/test e2e-aws-upgrade-hypershift-operator

reedcort · 2026-07-01T13:21:00Z

/test e2e-aws-upgrade-hypershift-operator

csrwng · 2026-07-01T19:02:33Z

/approve

openshift-ci · 2026-07-01T19:03:02Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: csrwng, reedcort

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [csrwng]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

reedcort · 2026-07-02T17:00:07Z

Test Evidence

Tested on a ROSA HCP 4.21 staging cluster with two subnets in the same AZ (us-east-2a).

CPO image overridden with CI build from this PR (4.21.0-0-2026-07-01-192751, release-4.21 + #8862).

Before — graceful degradation without `ec2:DescribeSubnets` in managed policy

The CPO attempts to deduplicate but DescribeSubnets is denied. It falls back to the original subnet list, and ModifyVpcEndpoint fails with DuplicateSubnetsInSameZone. This is the existing fallback behavior — no regression when the ROSA managed policy hasn't been updated yet.

{"level":"error","ts":"2026-07-02T16:55:31Z","msg":"failed to deduplicate subnets by AZ, proceeding with original list","controller":"awsendpointservice","controllerKind":"AWSEndpointService","name":"private-router","error":"failed to describe subnets for AZ deduplication: UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:sts::374459400198:assumed-role/creed-hcp-ugst-kube-system-control-plane-operator/1783011330969480238 is not authorized to perform: ec2:DescribeSubnets because no identity-based policy allows the ec2:DescribeSubnets action\n\tstatus code: 403"}

{"level":"error","ts":"2026-07-02T16:55:31Z","msg":"failed to modify vpc endpoint","controller":"awsendpointservice","controllerKind":"AWSEndpointService","name":"private-router","addSubnets":["subnet-0ab84217ba33a3613"],"removeSubnets":null,"addSG":null,"error":"DuplicateSubnetsInSameZone: Found another VPC endpoint subnet in the availability zone of subnet-0ab84217ba33a3613. VPC endpoint subnets should be in different availability zones supported by the VPC endpoint service.\n\tstatus code: 400"}

After — with `ec2:DescribeSubnets` added to the CPO IAM role

After adding ec2:DescribeSubnets to the CPO role, the dedup resolves both subnets to us-east-2a, picks one, and the endpoint reconciles successfully.

{"level":"info","ts":"2026-07-02T16:57:10Z","msg":"endpoint exists","controller":"awsendpointservice","controllerKind":"AWSEndpointService","name":"private-router","endpointID":"vpce-080c51d629e97ecd0"}

{"level":"info","ts":"2026-07-02T16:57:10Z","msg":"endpoint subnets are unchanged","controller":"awsendpointservice","controllerKind":"AWSEndpointService","name":"private-router"}

Condition:

AWSEndpointAvailable=True

/verified by @reedcort — tested on ROSA HCP 4.21 staging cluster with two subnets in same AZ

openshift-ci-robot · 2026-07-02T17:00:21Z

@reedcort: This PR has been marked as verified by @reedcort — tested on ROSA HCP 4.21 staging cluster with two subnets in same AZ.

Details

In response to this:

Test Evidence

Tested on a ROSA HCP 4.21 staging cluster with two subnets in the same AZ (us-east-2a).

CPO image overridden with CI build from this PR (4.21.0-0-2026-07-01-192751, release-4.21 + #8862).

Before — graceful degradation without ec2:DescribeSubnets in managed policy

The CPO attempts to deduplicate but DescribeSubnets is denied. It falls back to the original subnet list, and ModifyVpcEndpoint fails with DuplicateSubnetsInSameZone. This is the existing fallback behavior — no regression when the ROSA managed policy hasn't been updated yet.
{"level":"error","ts":"2026-07-02T16:55:31Z","msg":"failed to deduplicate subnets by AZ, proceeding with original list","controller":"awsendpointservice","controllerKind":"AWSEndpointService","name":"private-router","error":"failed to describe subnets for AZ deduplication: UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:sts::374459400198:assumed-role/creed-hcp-ugst-kube-system-control-plane-operator/1783011330969480238 is not authorized to perform: ec2:DescribeSubnets because no identity-based policy allows the ec2:DescribeSubnets action\n\tstatus code: 403"}
{"level":"error","ts":"2026-07-02T16:55:31Z","msg":"failed to modify vpc endpoint","controller":"awsendpointservice","controllerKind":"AWSEndpointService","name":"private-router","addSubnets":["subnet-0ab84217ba33a3613"],"removeSubnets":null,"addSG":null,"error":"DuplicateSubnetsInSameZone: Found another VPC endpoint subnet in the availability zone of subnet-0ab84217ba33a3613. VPC endpoint subnets should be in different availability zones supported by the VPC endpoint service.\n\tstatus code: 400"}
After — with ec2:DescribeSubnets added to the CPO IAM role

After adding ec2:DescribeSubnets to the CPO role, the dedup resolves both subnets to us-east-2a, picks one, and the endpoint reconciles successfully.
{"level":"info","ts":"2026-07-02T16:57:10Z","msg":"endpoint exists","controller":"awsendpointservice","controllerKind":"AWSEndpointService","name":"private-router","endpointID":"vpce-080c51d629e97ecd0"}
{"level":"info","ts":"2026-07-02T16:57:10Z","msg":"endpoint subnets are unchanged","controller":"awsendpointservice","controllerKind":"AWSEndpointService","name":"private-router"}
Condition:
AWSEndpointAvailable=True
/verified by @reedcort — tested on ROSA HCP 4.21 staging cluster with two subnets in same AZ

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

joshbranham · 2026-07-02T19:33:07Z

/lgtm

openshift-merge-bot · 2026-07-02T19:33:22Z

Scheduling required tests:
/test verify-deps

openshift-merge-bot · 2026-07-02T20:22:40Z

/retest-required

Remaining retests: 0 against base HEAD c84f807 and 2 for PR HEAD 7973b82 in total

reedcort · 2026-07-03T13:25:30Z

/test e2e-aks
/test e2e-aws

reedcort · 2026-07-07T20:56:02Z

/test e2e-aks
/test e2e-aws

reedcort · 2026-07-09T11:37:59Z

/retest

deepsm007 · 2026-07-09T13:33:45Z

/test e2e-aks
/test e2e-aws

deepsm007 · 2026-07-09T14:10:39Z

The job should be importing last accepted release but instead it is looking at 4.20.0-0.ci-2026-06-30-224937 to tag release:n1minor:
The job should be picking from below

deepsm007 · 2026-07-09T14:13:51Z

/test e2e-aks
/test e2e-aws

reedcort · 2026-07-09T17:27:19Z

/test e2e-aws

reedcort · 2026-07-09T17:54:30Z

/override e2e-aks

openshift-ci · 2026-07-09T17:54:44Z

@reedcort: reedcort unauthorized: /override is restricted to Repo administrators, approvers in top level OWNERS file, and the following github teams:openshift: openshift-release-oversight openshift-staff-engineers openshift-sustaining-engineers.

Details

In response to this:

/override e2e-aks

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

reedcort · 2026-07-09T20:29:45Z

/test e2e-aks
/test e2e-aws

reedcort · 2026-07-10T00:25:09Z

/test e2e-aws

openshift-merge-bot · 2026-07-10T01:19:56Z

/retest-required

Remaining retests: 0 against base HEAD b5439e4 and 1 for PR HEAD 7973b82 in total

reedcort · 2026-07-10T11:04:31Z

/retest-required

openshift-merge-bot · 2026-07-10T18:07:10Z

/retest-required

Remaining retests: 0 against base HEAD c975abb and 0 for PR HEAD 7973b82 in total

openshift-merge-bot · 2026-07-10T22:08:45Z

/hold

Revision 7973b82 was retested 3 times: holding

reedcort · 2026-07-10T23:33:12Z

/retest-required

reedcort · 2026-07-11T11:43:35Z

/retest-required

reedcort · 2026-07-13T12:23:50Z

/retest-required

reedcort · 2026-07-13T13:00:35Z

/test e2e-aks

deepsm007 · 2026-07-13T14:04:43Z

/test e2e-aks

openshift-ci · 2026-07-13T16:18:27Z

@reedcort: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

reedcort · 2026-07-13T16:20:51Z

/hold cancel

openshift-ci-robot · 2026-07-13T16:25:15Z

@reedcort: Jira Issue Verification Checks: Jira Issue OCPBUGS-93927
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-93927 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

What this PR does / why we need it:

Backport of #8651 to release-4.21.

When multiple NodePools reference subnets in the same AWS availability zone, the CPO's VPC endpoint reconciler now deduplicates them before calling CreateVpcEndpoint or ModifyVpcEndpoint, preventing the AWS DuplicateSubnetsInSameZone error.

Adds an in-memory AZ cache (sync.RWMutex + map) to the AWSEndpointServiceReconciler to avoid repeated DescribeSubnets calls

Lexicographic subnet selection ensures deterministic deduplication across concurrent reconciles

Graceful degradation: if DescribeSubnets fails (e.g. missing IAM permission), the original subnet list passes through unchanged

Adds ec2:DescribeSubnets to the control-plane-operator and shared-vpc-endpoint IAM policies

Which issue(s) this PR fixes:

Fixes https://issues.redhat.com/browse/OCPBUGS-93927

Special notes for your reviewer:

Adapted from the 4.22 backport (#8724) for AWS SDK v1 (aws-sdk-go) used on release-4.21. The logic is identical; only the AWS API call patterns differ (DescribeSubnetsWithContext, aws.StringValue, ec2iface.EC2API).

Checklist:

Subject and description added to both, commit and PR.

Relevant issues have been referenced.

This change includes docs.

This change includes unit tests.

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-merge-robot · 2026-07-14T05:32:06Z

Fix included in release 4.21.0-0.nightly-2026-07-13-235512

openshift-ci Bot added the do-not-merge/needs-area label Jun 29, 2026

reedcort changed the title ~~OCPBUGS-93927: fix(cpo): deduplicate VPC endpoint subnets by AZ to prevent DuplicateSubnetsInSameZone~~ [release-4.21] OCPBUGS-93927: fix(cpo): deduplicate VPC endpoint subnets by AZ to prevent DuplicateSubnetsInSameZone Jun 29, 2026

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jun 29, 2026

openshift-ci Bot added area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release area/cli Indicates the PR includes changes for CLI area/platform/aws PR/issue for AWS (AWSPlatform) platform and removed do-not-merge/needs-area labels Jun 29, 2026

csrwng added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Jul 1, 2026

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 1, 2026

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jul 2, 2026

openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 10, 2026

openshift-ci Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 13, 2026

openshift-merge-bot Bot merged commit 85ac58c into openshift:release-4.21 Jul 13, 2026
19 checks passed

reedcort mentioned this pull request Jul 13, 2026

OCPBUGS-98574: [release-4.20] fix(cpo): deduplicate VPC endpoint subnets by AZ to prevent DuplicateSubnetsInSameZone #8994

Merged

4 tasks

Uh oh!

Conversation

reedcort commented Jun 29, 2026

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Checklist:

Uh oh!

openshift-merge-bot Bot commented Jun 29, 2026

Uh oh!

openshift-ci-robot commented Jun 29, 2026

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Checklist:

Uh oh!

coderabbitai Bot commented Jun 29, 2026

Review skipped

Uh oh!

reedcort commented Jun 29, 2026

Uh oh!

openshift-ci-robot commented Jun 29, 2026

Uh oh!

reedcort commented Jun 29, 2026

Uh oh!

openshift-ci-robot commented Jun 29, 2026

Uh oh!

reedcort commented Jun 29, 2026

Uh oh!

reedcort commented Jul 1, 2026

Uh oh!

reedcort commented Jul 1, 2026

Uh oh!

csrwng commented Jul 1, 2026

Uh oh!

openshift-ci Bot commented Jul 1, 2026

Uh oh!

reedcort commented Jul 2, 2026

Test Evidence

Before — graceful degradation without ec2:DescribeSubnets in managed policy

After — with ec2:DescribeSubnets added to the CPO IAM role

Uh oh!

openshift-ci-robot commented Jul 2, 2026

Test Evidence

Before — graceful degradation without ec2:DescribeSubnets in managed policy

After — with ec2:DescribeSubnets added to the CPO IAM role

Uh oh!

joshbranham commented Jul 2, 2026

Uh oh!

openshift-merge-bot Bot commented Jul 2, 2026

Uh oh!

openshift-merge-bot Bot commented Jul 2, 2026

Uh oh!

reedcort commented Jul 3, 2026

Uh oh!

reedcort commented Jul 7, 2026

Uh oh!

reedcort commented Jul 9, 2026

Uh oh!

deepsm007 commented Jul 9, 2026

Uh oh!

deepsm007 commented Jul 9, 2026

Uh oh!

deepsm007 commented Jul 9, 2026

Uh oh!

reedcort commented Jul 9, 2026

Uh oh!

reedcort commented Jul 9, 2026

Uh oh!

openshift-ci Bot commented Jul 9, 2026

Uh oh!

reedcort commented Jul 9, 2026

Uh oh!

reedcort commented Jul 10, 2026

Uh oh!

openshift-merge-bot Bot commented Jul 10, 2026

Uh oh!

reedcort commented Jul 10, 2026

Uh oh!

openshift-merge-bot Bot commented Jul 10, 2026

Before — graceful degradation without `ec2:DescribeSubnets` in managed policy

After — with `ec2:DescribeSubnets` added to the CPO IAM role

Before — graceful degradation without `ec2:DescribeSubnets` in managed policy

After — with `ec2:DescribeSubnets` added to the CPO IAM role