Skip to content

OCPBUGS-81309: Fix vSphere with static IPs on TPNU#10442

Merged
openshift-merge-bot[bot] merged 1 commit intoopenshift:mainfrom
openshift-cloud-team:OCPBUGS-81309
Apr 2, 2026
Merged

OCPBUGS-81309: Fix vSphere with static IPs on TPNU#10442
openshift-merge-bot[bot] merged 1 commit intoopenshift:mainfrom
openshift-cloud-team:OCPBUGS-81309

Conversation

@mdbooth
Copy link
Copy Markdown
Contributor

@mdbooth mdbooth commented Mar 30, 2026

The bootstrap KAS cannot currently reach conversion webhooks, so installer-generated manifests must use the storage version of any CRDs they rely on. If they use a non-storage version then KAS will try to reach a conversion webhook, which will fail, and the object will not be admitted.

In this case the IPAM CRD storage versions were reverted to v1beta1 from v1beta2, so the installer manifests must be updated to follow.

@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Mar 30, 2026
@openshift-ci-robot
Copy link
Copy Markdown
Contributor

@mdbooth: This pull request references Jira Issue OCPBUGS-81309, which is invalid:

  • expected the bug to target the "4.22.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Mar 30, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: cbbd1f4b-9c6d-4e05-b6ea-0eb9b2da485f

📥 Commits

Reviewing files that changed from the base of the PR and between be916e8 and 7a59da3.

📒 Files selected for processing (1)
  • images/openstack/Dockerfile.ci
🚧 Files skipped from review as they are similar to previous changes (1)
  • images/openstack/Dockerfile.ci

Walkthrough

Removed feature-gate conditional that rewrote IPAM object APIVersion; code now always appends generated IPClaim and IPAddress objects unchanged. Updated OpenStack CI Dockerfile base image tags from OpenShift 4.17 to 4.22 and removed several Ansible/OpenStack yum packages from test deps.

Changes

Cohort / File(s) Summary
VSphere IPAM logic
pkg/asset/machines/vsphere/machines.go
Deleted conditional on ClusterAPIMachineManagement that mutated APIVersion for IPClaims/IPAddresses; now unconditionally appends generated claim and address without APIVersion rewriting and includes clarifying comments about relying on installed CRD storage/API versions.
OpenStack CI Dockerfile
images/openstack/Dockerfile.ci
Bumped OpenShift CI image tags from 4.17 to 4.22 for multiple build stages; removed several yum-installed Ansible/OpenStack packages from test dependency installation and added commented notes about their unavailability in default repos.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@mdbooth
Copy link
Copy Markdown
Contributor Author

mdbooth commented Mar 30, 2026

/jira refresh

@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Mar 30, 2026
@openshift-ci-robot
Copy link
Copy Markdown
Contributor

@mdbooth: This pull request references Jira Issue OCPBUGS-81309, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.22.0) matches configured target version for branch (4.22.0)
  • bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@mdbooth
Copy link
Copy Markdown
Contributor Author

mdbooth commented Mar 30, 2026

/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-vsphere-static-ovn-techpreview

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci bot commented Mar 30, 2026

@mdbooth: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-nightly-4.22-e2e-vsphere-static-ovn-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/9d7528f0-2c46-11f1-81eb-188b8f5fdaa1-0

@patrickdillon
Copy link
Copy Markdown
Contributor

The code being backed out here was added as a fix for https://redhat.atlassian.net/browse/OCPBUGS-69434--it seemed v1beta2 IPAM was expected. Are we not going to hit the same issue? Perhaps something changed and now v1beta is ok?

@mdbooth
Copy link
Copy Markdown
Contributor Author

mdbooth commented Mar 30, 2026

The code being backed out here was added as a fix for https://redhat.atlassian.net/browse/OCPBUGS-69434--it seemed v1beta2 IPAM was expected. Are we not going to hit the same issue? Perhaps something changed and now v1beta is ok?

We flipped the requirement again in openshift/cluster-api#263. The technical constraint here is that installer-generated manifests which are applied during bootstrap cannot require a conversion webhook.

When we initially bumped CAPI to whichever version contains v1beta2 manifests, it also bumped the CRD storage version to v1beta2. This meant that applying v1beta1 required calling a conversion webhook. Consequently we added this change which ensured that we used v1beta2, which didn't require conversion.

We later realised that the CAPI bump broke upgrades because it is not permitted to both introduce a new version and make it the storage version at the same time. Consequently we added openshift/cluster-api#263 which bumped the storage version temporarily back to v1beta1 but retained v1beta2. This second change broke vSphere again because now it's v1beta2 that requires conversion, not v1beta1.

This is very fragile and I suspect we will break it several more times in the future before we have a proper handle on it, but for now v1beta1 should be the 'correct' version here. Again.

@mdbooth
Copy link
Copy Markdown
Contributor Author

mdbooth commented Mar 30, 2026

/test yaml-lint

@mdbooth
Copy link
Copy Markdown
Contributor Author

mdbooth commented Mar 30, 2026

Looks like prow was unhappy for a while there.

/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-vsphere-static-ovn-techpreview

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci bot commented Mar 30, 2026

@mdbooth: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-nightly-4.22-e2e-vsphere-static-ovn-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/6d7def30-2c58-11f1-875e-82471d9c3a14-0

@mdbooth
Copy link
Copy Markdown
Contributor Author

mdbooth commented Mar 30, 2026

/test artifacts-images images okd-scos-images verify-deps verify-vendor images gover golint gofmt

@mdbooth
Copy link
Copy Markdown
Contributor Author

mdbooth commented Mar 30, 2026

/test govet shellcheck

@mdbooth
Copy link
Copy Markdown
Contributor Author

mdbooth commented Mar 30, 2026

/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-vsphere-static-ovn-techpreview

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci bot commented Mar 30, 2026

@mdbooth: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-nightly-4.22-e2e-vsphere-static-ovn-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/f1cd39b0-2c5f-11f1-8dfe-851cd79328a7-0

@openshift-ci-robot
Copy link
Copy Markdown
Contributor

@mdbooth: This pull request references Jira Issue OCPBUGS-81309, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.22.0) matches configured target version for branch (4.22.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
Details

In response to this:

Includes a fix to a broken OpenStack image, required to allow the payload job to run.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@mdbooth
Copy link
Copy Markdown
Contributor Author

mdbooth commented Mar 30, 2026

/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-vsphere-static-ovn-techpreview

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci bot commented Mar 30, 2026

@mdbooth: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-nightly-4.22-e2e-vsphere-static-ovn-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/fc33b710-2c66-11f1-8c1c-0d3632815be7-0

Copy link
Copy Markdown

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@images/openstack/Dockerfile.ci`:
- Line 26: The Dockerfile.ci copies the OpenStack UPI playbooks (COPY
--from=builder ... upi/openstack) which include network.yaml,
security-groups.yaml, control-plane.yaml, compute-nodes.yaml, bootstrap.yaml and
cleanup scripts that import common.yaml and require Ansible + openstacksdk;
either ensure no CI path will ever execute those playbooks or install the
required runtime before copying/using them: locate the COPY line that copies
/upi/openstack in Dockerfile.ci and either (A) remove or move those playbooks
out of the CI image so they are not present for any CI job that might invoke
them, or (B) add installation of Ansible (ansible-core or ansible package) and
openstacksdk (python3-openstacksdk or pip install openstacksdk) and any other
Ansible plugin deps (e.g., ansible-collections or python3-ansible-*) to the
Dockerfile.ci so any accidental execution will succeed.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: a28ec0a9-65ed-4cae-826b-dfaf1f0a3f55

📥 Commits

Reviewing files that changed from the base of the PR and between fccf426 and be916e8.

📒 Files selected for processing (1)
  • images/openstack/Dockerfile.ci

@mdbooth
Copy link
Copy Markdown
Contributor Author

mdbooth commented Mar 30, 2026

/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-vsphere-static-ovn-techpreview

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci bot commented Mar 30, 2026

@mdbooth: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-nightly-4.22-e2e-vsphere-static-ovn-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/054c9510-2c8a-11f1-9a85-602c80bd0161-0

@mdbooth
Copy link
Copy Markdown
Contributor Author

mdbooth commented Mar 31, 2026

The payload job passed: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/openshift-installer-10442-nightly-4.22-e2e-vsphere-static-ovn-techpreview/2038749310417375232

I'm going to take out the OpenStack image changes as they were only required to let me run the payload job and they're not related to this.

@mdbooth
Copy link
Copy Markdown
Contributor Author

mdbooth commented Mar 31, 2026

/pipeline required

@mdbooth
Copy link
Copy Markdown
Contributor Author

mdbooth commented Mar 31, 2026

All failures should be resolved by openshift/release#77140

/retest-required

@mdbooth
Copy link
Copy Markdown
Contributor Author

mdbooth commented Mar 31, 2026

For completeness I'll run the payload again. It should pass now that the OpenStack image is fixed.

/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-vsphere-static-ovn-techpreview

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci bot commented Mar 31, 2026

@mdbooth: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-nightly-4.22-e2e-vsphere-static-ovn-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/17a823c0-2d1e-11f1-9428-6fb3738e17af-0

@tthvo
Copy link
Copy Markdown
Member

tthvo commented Apr 1, 2026

/test e2e-vsphere-static-ovn

@mandre
Copy link
Copy Markdown
Member

mandre commented Apr 1, 2026

/retest-required

Copy link
Copy Markdown
Member

@damdo damdo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Apr 1, 2026
@mdbooth
Copy link
Copy Markdown
Contributor Author

mdbooth commented Apr 1, 2026

The payload job above ran successfully, which verifies this is fixed.

/verified by CI

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Apr 1, 2026
@openshift-ci-robot
Copy link
Copy Markdown
Contributor

@mdbooth: This PR has been marked as verified by CI.

Details

In response to this:

The payload job above ran successfully, which verifies this is fixed.

/verified by CI

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@patrickdillon
Copy link
Copy Markdown
Contributor

/approve

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci bot commented Apr 1, 2026

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: patrickdillon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 1, 2026
@damdo
Copy link
Copy Markdown
Member

damdo commented Apr 1, 2026

/retest-required

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci bot commented Apr 1, 2026

@mdbooth: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-openstack-ovn 7a59da3 link true /test e2e-openstack-ovn
ci/prow/e2e-openstack-nfv-intel 7a59da3 link false /test e2e-openstack-nfv-intel
ci/prow/openstack-manifests 7a59da3 link true /test openstack-manifests
ci/prow/e2e-openstack-dualstack-upi 7a59da3 link false /test e2e-openstack-dualstack-upi
ci/prow/e2e-openstack-proxy 7a59da3 link false /test e2e-openstack-proxy
ci/prow/e2e-vsphere-ovn-hybrid-env d37d0a7 link false /test e2e-vsphere-ovn-hybrid-env
ci/prow/e2e-vsphere-multi-vcenter-ovn d37d0a7 link false /test e2e-vsphere-multi-vcenter-ovn
ci/prow/e2e-vsphere-ovn-zones d37d0a7 link false /test e2e-vsphere-ovn-zones
ci/prow/e2e-vsphere-ovn-disk-setup-techpreview d37d0a7 link false /test e2e-vsphere-ovn-disk-setup-techpreview
ci/prow/e2e-vsphere-ovn-techpreview d37d0a7 link false /test e2e-vsphere-ovn-techpreview
ci/prow/e2e-vsphere-ovn-devpreview d37d0a7 link false /test e2e-vsphere-ovn-devpreview

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

/retest-required

Remaining retests: 0 against base HEAD c93ae9f and 2 for PR HEAD d37d0a7 in total

@openshift-merge-bot openshift-merge-bot bot merged commit 30edf44 into openshift:main Apr 2, 2026
17 of 23 checks passed
@openshift-ci-robot
Copy link
Copy Markdown
Contributor

@mdbooth: Jira Issue Verification Checks: Jira Issue OCPBUGS-81309
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-81309 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

The bootstrap KAS cannot currently reach conversion webhooks, so installer-generated manifests must use the storage version of any CRDs they rely on. If they use a non-storage version then KAS will try to reach a conversion webhook, which will fail, and the object will not be admitted.

In this case the IPAM CRD storage versions were reverted to v1beta1 from v1beta2, so the installer manifests must be updated to follow.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@mdbooth mdbooth deleted the OCPBUGS-81309 branch April 2, 2026 07:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants