OCPBUGS-85778: Fix NodePool reconciliation failure when updating mirrored immutable ConfigMaps

[vsolanki12]

What this PR does / why we need it:

When multiple NodePools reference the same kubelet ConfigMap and the source ConfigMap is modified after mirroring, the nodepool-controller's NTO reconciliation fails because mirrored ConfigMaps are created with immutable: true. Kubernetes rejects in-place updates to immutable ConfigMaps, causing NodePools to get stuck in UpdatingConfig=True with:

failed to reconcile mirrored ConfigMap: data: Forbidden: field is immutable when `immutable` is set

This PR:

• Removes the immutable: true flag from mutateMirroredConfig so newly created mirrored ConfigMaps are mutable
• Adds deleteImmutableConfigMapIfNeeded to reconcileMirroredConfigs to handle migration — existing immutable mirrored ConfigMaps are deleted before CreateOrUpdate recreates them as mutable
• This mirrors the parallel fix already applied on the CPO/HCCO side (commit bbd8f63, PR OCPBUGS-60936:fix(hccp): remove immutable field from kubeletconfigmap #6726) which fixed Stage 2 (HCP namespace → guest cluster), while this PR fixes Stage 1 (user namespace → HCP namespace)

Which issue(s) this PR fixes:

Fixes https://issues.redhat.com/browse/OCPBUGS-85778

Special notes for your reviewer:

• Tested on a live KubeVirt HCP cluster (OCP 4.21.9 management, 4.18.30 guest) with two NodePools sharing the same custom-kubelet ConfigMap
• Bug reproduced: both NodePools stuck in UpdatingConfig=True with immutable field error
• After deploying patched operator image: mirrored ConfigMaps recreated as mutable, reconciliation succeeded, no new errors

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Summary by CodeRabbit

• Bug Fixes

  • Prevented reconciliation failures by removing immutable ConfigMaps and recreating them as mutable so updates apply reliably.
  • Ensured mirrored ConfigMaps are set mutable during reconciliation to allow safe updates.

• Tests

  • Updated tests to expect mutable mirrored ConfigMaps and to validate recreation behavior when immutable configs are encountered.

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci-robot]

@vsolanki12: This pull request references Jira Issue OCPBUGS-85778, which is invalid:

• expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

What this PR does / why we need it:

When multiple NodePools reference the same kubelet ConfigMap and the source ConfigMap is modified after mirroring, the nodepool-controller's NTO reconciliation fails because mirrored ConfigMaps are created with immutable: true. Kubernetes rejects in-place updates to immutable ConfigMaps, causing NodePools to get stuck in UpdatingConfig=True with:

failed to reconcile mirrored ConfigMap: data: Forbidden: field is immutable when `immutable` is set

This PR:

• Removes the immutable: true flag from mutateMirroredConfig so newly created mirrored ConfigMaps are mutable
• Adds deleteImmutableConfigMapIfNeeded to reconcileMirroredConfigs to handle migration — existing immutable mirrored ConfigMaps are deleted before CreateOrUpdate recreates them as mutable
• This mirrors the parallel fix already applied on the CPO/HCCO side (commit bbd8f63, PR OCPBUGS-60936:fix(hccp): remove immutable field from kubeletconfigmap #6726) which fixed Stage 2 (HCP namespace → guest cluster), while this PR fixes Stage 1 (user namespace → HCP namespace)

Which issue(s) this PR fixes:

Fixes https://issues.redhat.com/browse/OCPBUGS-85778

Special notes for your reviewer:

• Tested on a live KubeVirt HCP cluster (OCP 4.21.9 management, 4.18.30 guest) with two NodePools sharing the same custom-kubelet ConfigMap
• Bug reproduced: both NodePools stuck in UpdatingConfig=True with immutable field error
• After deploying patched operator image: mirrored ConfigMaps recreated as mutable, reconciliation succeeded, no new errors

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: e191d3ad-5359-4ce6-a18b-0091fcd3ac82

📥 Commits

Reviewing files that changed from the base of the PR and between 5324fc1 and c04d38b.

📒 Files selected for processing (2)

• hypershift-operator/controllers/nodepool/nto.go
• hypershift-operator/controllers/nodepool/nto_test.go

🚧 Files skipped from review as they are similar to previous changes (1)

• hypershift-operator/controllers/nodepool/nto.go

---

📝 Walkthrough

Walkthrough

This PR makes the reconciler delete an existing mirrored ConfigMap if it is immutable before recreating it, changes the mirrored ConfigMap mutation to explicitly set Immutable = false, and updates unit tests to expect mutable mirrored ConfigMaps across containerruntime, kubeletconfig, deletion, and immutable-existing scenarios.

Sequence Diagram(s)

sequenceDiagram
  participant NodePoolReconciler
  participant KubeAPIServer
  participant ConfigMapResource
  NodePoolReconciler->>KubeAPIServer: GET mirrored ConfigMap
  KubeAPIServer-->>NodePoolReconciler: existing ConfigMap / NotFound
  alt existing.Immutable == true and labels match nodePool
    NodePoolReconciler->>KubeAPIServer: DELETE existing immutable ConfigMap
    KubeAPIServer-->>NodePoolReconciler: DeleteResult
  end
  NodePoolReconciler->>KubeAPIServer: CreateOrUpdate mirrored ConfigMap (Immutable=false)
  KubeAPIServer-->>NodePoolReconciler: CreateOrUpdateResult

Loading

🚥 Pre-merge checks | ✅ 10 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Test Structure And Quality	⚠️ Warning	Key assertions lack meaningful failure messages at lines 206, 209, 213, 222. Bare Expect() calls without diagnostic context make test failures harder to diagnose.	Add failure messages to assertions: replace g.Expect(err).ToNot(HaveOccurred()) with g.Expect(err).ToNot(HaveOccurred(), "failed to reconcile mirrored configs") throughout.

✅ Passed checks (10 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and specifically describes the main change: fixing NodePool reconciliation failures caused by updating immutable mirrored ConfigMaps, which directly addresses the problem described in the PR objectives.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	Test file uses standard Go testing.T, not Ginkgo. All test case names are static strings with no dynamic content, timestamps, UUIDs, pod/namespace names, or other variable information.
Topology-Aware Scheduling Compatibility	✅ Passed	PR only modifies ConfigMap reconciliation logic (Immutable field), no scheduling constraints, node affinities, topology assumptions, or HA-specific logic introduced.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	The check targets Ginkgo e2e tests, but this PR only modifies unit tests (func TestXXX(t *testing.T)) in hypershift-operator/controllers/nodepool/nto_test.go. No e2e tests added.
No-Weak-Crypto	✅ Passed	No weak crypto detected. PR contains no crypto imports (MD5, SHA1, DES, RC4, 3DES, Blowfish, ECB) or custom crypto implementations. Changes involve only Kubernetes ConfigMap reconciliation.
Container-Privileges	✅ Passed	PR contains no container privilege escalation flags or configurations. Changes only modify ConfigMap.Immutable field for ConfigMap reconciliation.
No-Sensitive-Data-In-Logs	✅ Passed	The new log statement in deleteImmutableConfigMapIfNeeded logs only a descriptive message and ConfigMap namespace/name. No sensitive data like passwords, tokens, or PII are exposed in logs.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@hypershift-operator/controllers/nodepool/nto.go`:
- Around line 213-225: The deletion currently removes any immutable ConfigMap at
the target key; restrict it to only remove ConfigMaps that this controller
manages by checking ownership/management metadata before deleting. Update
deleteImmutableConfigMapIfNeeded to verify existingCM is owned/managed by this
controller (e.g., check for an OwnerReference with Controller==true and
Kind=="NodePool" and/or a controller management label you use for mirrored
ConfigMaps such as a "mirrored" or "managed-by" label) and only call
k8sutil.DeleteIfNeeded when that check passes; if ownership/label is absent, log
and return without deleting. Use existingCM.OwnerReferences and
existingCM.Labels in the NodePoolReconciler.deleteImmutableConfigMapIfNeeded
function to implement the guard.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 40fe0a06-ff2c-48c8-8989-7c6b6eef2da0

📥 Commits

Reviewing files that changed from the base of the PR and between cf2b91f and 53511be.

📒 Files selected for processing (2)

• hypershift-operator/controllers/nodepool/nto.go
• hypershift-operator/controllers/nodepool/nto_test.go

[vsolanki12]

/jira refresh

[openshift-ci-robot]

@vsolanki12: This pull request references Jira Issue OCPBUGS-85778, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[codecov]

Codecov Report

❌ Patch coverage is 82.35294% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 41.28%. Comparing base (d433acd) to head (c04d38b).
⚠️ Report is 198 commits behind head on main.

Files with missing lines	Patch %	Lines
hypershift-operator/controllers/nodepool/nto.go	82.35%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8543      +/-   ##
==========================================
+ Coverage   40.51%   41.28%   +0.76%     
==========================================
  Files         755      755              
  Lines       93213    93462     +249     
==========================================
+ Hits        37768    38586     +818     
+ Misses      52742    52145     -597     
- Partials     2703     2731      +28     

Files with missing lines	Coverage Δ
hypershift-operator/controllers/nodepool/nto.go	51.54% <82.35%> (+2.68%)	⬆️

... and 44 files with indirect coverage changes

Flag	Coverage Δ
cmd-support	34.86% <ø> (+0.45%)	⬆️
cpo-hostedcontrolplane	43.50% <ø> (+1.73%)	⬆️
cpo-other	42.79% <ø> (+1.72%)	⬆️
hypershift-operator	51.05% <82.35%> (+0.29%)	⬆️
other	31.64% <ø> (+0.05%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[openshift-ci-robot]

@vsolanki12: This pull request references Jira Issue OCPBUGS-85778, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

What this PR does / why we need it:

When multiple NodePools reference the same kubelet ConfigMap and the source ConfigMap is modified after mirroring, the nodepool-controller's NTO reconciliation fails because mirrored ConfigMaps are created with immutable: true. Kubernetes rejects in-place updates to immutable ConfigMaps, causing NodePools to get stuck in UpdatingConfig=True with:

failed to reconcile mirrored ConfigMap: data: Forbidden: field is immutable when `immutable` is set

This PR:

• Removes the immutable: true flag from mutateMirroredConfig so newly created mirrored ConfigMaps are mutable
• Adds deleteImmutableConfigMapIfNeeded to reconcileMirroredConfigs to handle migration — existing immutable mirrored ConfigMaps are deleted before CreateOrUpdate recreates them as mutable
• This mirrors the parallel fix already applied on the CPO/HCCO side (commit bbd8f63, PR OCPBUGS-60936:fix(hccp): remove immutable field from kubeletconfigmap #6726) which fixed Stage 2 (HCP namespace → guest cluster), while this PR fixes Stage 1 (user namespace → HCP namespace)

Which issue(s) this PR fixes:

Fixes https://issues.redhat.com/browse/OCPBUGS-85778

Special notes for your reviewer:

• Tested on a live KubeVirt HCP cluster (OCP 4.21.9 management, 4.18.30 guest) with two NodePools sharing the same custom-kubelet ConfigMap
• Bug reproduced: both NodePools stuck in UpdatingConfig=True with immutable field error
• After deploying patched operator image: mirrored ConfigMaps recreated as mutable, reconciliation succeeded, no new errors

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Summary by CodeRabbit

• Bug Fixes

• Prevented reconciliation failures caused by immutable ConfigMaps by removing and recreating them as mutable so updates succeed.

• Mirrored ConfigMaps are now ensured to be mutable during reconciliation to allow safe updates.

• Tests

• Updated tests to expect mutable mirrored ConfigMaps and to verify correct recreation when immutable configs existed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[jparrill]

/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jparrill, vsolanki12

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [jparrill]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[csrwng]

/lgtm

[openshift-merge-bot]

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks-4-22
/test e2e-aws-4-22
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-self-managed
/test e2e-azure-v2-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke

[hypershift-jira-solve-ci]

All 5 jobs have the identical failure pattern. Here is the consolidated report:

Test Failure Analysis Complete

Job Information

• Prow Jobs: pull-ci-openshift-hypershift-main-e2e-aws, pull-ci-openshift-hypershift-main-e2e-aws-4-22, pull-ci-openshift-hypershift-main-e2e-azure-self-managed, pull-ci-openshift-hypershift-main-e2e-aks-4-22, pull-ci-openshift-hypershift-main-e2e-aks
• Build IDs: 2064724084729581568, 2064724084763136000, 2064724084838633472, 2064724084712804352, 2064724084645695488
• PR: OCPBUGS-85778: Fix NodePool reconciliation failure when updating mirrored immutable ConfigMaps #8543 (OCPBUGS-85778: Fix NodePool reconciliation failure when updating mirrored immutable ConfigMaps)
• Failed Steps: Release image imports for OCP 4.19, 4.20, and 4.21 payloads

Test Failure Analysis

Error

error importing release "n2minor" (4.21.0): import of component "agent-installer-ui" failed:
Internal error occurred: [dockerimage.image.openshift.io "quay.io/openshift/ci@sha256:abea17a3a199d668f82fba64afa060764126d780008ed9641ddcd5e8fd7c0a67" not found,
dockerimage.image.openshift.io "quay-proxy.ci.openshift.org/openshift/ci@sha256:abea17a3a199d668f82fba64afa060764126d780008ed9641ddcd5e8fd7c0a67" not found]
on the image stream even after (6) imports: timed out waiting for the condition

error importing release "n3minor" (4.20.0): import of component "hypershift" failed:
Internal error occurred: [dockerimage.image.openshift.io "quay.io/openshift/ci@sha256:ef1b3047fb8915cf4bfd3e7a08ed90a1312b8399a152d8acf69767055a2a446d" not found,
dockerimage.image.openshift.io "quay-proxy.ci.openshift.org/openshift/ci@sha256:ef1b3047fb8915cf4bfd3e7a08ed90a1312b8399a152d8acf69767055a2a446d" not found]
on the image stream even after (6) imports: timed out waiting for the condition

error importing release "n4minor" (4.19.0): import of component "machine-config-operator" failed:
Internal error occurred: [dockerimage.image.openshift.io "quay.io/openshift/ci@sha256:90428a823afd00acb6191e416a4aebf86363688ccd47d5f50150c38a8d1d214e" not found,
dockerimage.image.openshift.io "quay-proxy.ci.openshift.org/openshift/ci@sha256:90428a823afd00acb6191e416a4aebf86363688ccd47d5f50150c38a8d1d214e" not found]
on the image stream even after (6) imports: timed out waiting for the condition

Summary

All 5 Prow jobs failed with the identical root cause: a CI infrastructure issue where specific container images for OCP releases 4.19, 4.20, and 4.21 were missing from both the primary registry (quay.io) and the proxy registry (quay-proxy.ci.openshift.org). The failures occurred during the release image import phase — before any test code was executed — meaning the PR's code changes were never actually tested. Releases 4.18 and 4.22 imported successfully, indicating this was not a total registry outage but rather specific images that were unavailable or had been garbage-collected.

Root Cause

CI Infrastructure / Registry Issue — Not a code problem.

The ci-operator multi-version test workflow requires importing multiple OCP release payloads (4.18 through 5.0) to validate upgrade and cross-version compatibility. Three specific component images were missing from both quay.io/openshift/ci and quay-proxy.ci.openshift.org/openshift/ci:

1. OCP 4.21 — agent-installer-ui component image (sha256:abea17a3a199...) not found
2. OCP 4.20 — hypershift component image (sha256:ef1b3047fb89...) not found
3. OCP 4.19 — machine-config-operator component image (sha256:90428a823afd...) not found

Each image import was retried 6 times before timing out. Since these release imports are a prerequisite for the test workflow, the entire job was aborted before any test step could run.

This is a transient infrastructure issue — the images were likely garbage-collected or temporarily unavailable from the CI image mirrors. The PR code changes (ConfigMap reconciliation fix) are completely unrelated to image registry operations.

Recommendations

1. Retest the PR — Run /retest or trigger individual jobs. The registry issue is transient and may already be resolved.
2. If retests fail identically — Report the missing image digests to the Test Platform team (OpenShift CI), as the images may need to be re-mirrored to the CI registry.
3. No code changes needed — The PR's changes to NodePool reconciliation for mirrored immutable ConfigMaps are unrelated to these failures. The code was never exercised.

Evidence

Evidence	Detail
Failure phase	Release image import (pre-test setup) — no test code was executed
Failed releases	4.19, 4.20, 4.21 (3 of 6 releases)
Successful releases	4.18, 4.22, 5.0 — proves partial registry availability
Missing image: 4.21	agent-installer-ui @ sha256:abea17a3a199d668f82fba64afa060764126d780008ed9641ddcd5e8fd7c0a67
Missing image: 4.20	hypershift @ sha256:ef1b3047fb8915cf4bfd3e7a08ed90a1312b8399a152d8acf69767055a2a446d
Missing image: 4.19	machine-config-operator @ sha256:90428a823afd00acb6191e416a4aebf86363688ccd47d5f50150c38a8d1d214e
Registries tried	quay.io/openshift/ci and quay-proxy.ci.openshift.org/openshift/ci — both returned "not found"
Retry attempts	6 per image, all timed out
Affected jobs	All 5 jobs (e2e-aws, e2e-aws-4-22, e2e-azure-self-managed, e2e-aks, e2e-aks-4-22) — identical pattern
PR code built successfully	All PR images (hypershift, hypershift-operator, hypershift-tests, hypershift-cli) built without errors

---

[openshift-ci]

@vsolanki12: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-azure-self-managed	c04d38b	link	true	/test e2e-azure-self-managed

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[vsolanki12]

/retest

[vsolanki12]

/verified by @vsolanki12

Before fix:

$ oc describe nodepool ocpbugs-85778 -n clusters | tail -10
Events:
  Type     Reason          Age   From                 Message
  ----     ------          ----  ----                 -------
  Warning  ReconcileError  15m   nodepool-controller  failed to reconcile NTO:
    failed to mirror configs: failed to reconcile mirrored
    clusters-ocpbugs-85778/custom-kubelet-ocpbugs-85778 ConfigMap:
    ConfigMap "custom-kubelet-ocpbugs-85778" is invalid:
    data: Forbidden: field is immutable when `immutable` is set

After fix:

$ oc get cm -n clusters-ocpbugs-85778 custom-kubelet-ocpbugs-85778 -o jsonpath='{.immutable}'
(empty — no longer immutable)

$ oc get cm -n clusters-ocpbugs-85778 custom-kubelet-ocpbugs-85778-b -o jsonpath='{.immutable}'
(empty — no longer immutable)

2. No more reconciliation errors:

$ oc describe nodepool ocpbugs-85778-b -n clusters | tail -10
Events:
  Type     Reason          Age                   From                 Message
  ----     ------          ----                  ----                 -------
  Warning  ReconcileError  100m (x2 over 100m)   nodepool-controller  ...

3. Operator logs clean:

$ oc logs -n hypershift deployment/operator --tail=50 | grep -i "immutable\|mirrored"

[openshift-ci-robot]

@vsolanki12: This PR has been marked as verified by @vsolanki12.

Details

In response to this:

/verified by @vsolanki12

Before fix:

$ oc describe nodepool ocpbugs-85778 -n clusters | tail -10
Events:
 Type     Reason          Age   From                 Message
 ----     ------          ----  ----                 -------
 Warning  ReconcileError  15m   nodepool-controller  failed to reconcile NTO:
   failed to mirror configs: failed to reconcile mirrored
   clusters-ocpbugs-85778/custom-kubelet-ocpbugs-85778 ConfigMap:
   ConfigMap "custom-kubelet-ocpbugs-85778" is invalid:
   data: Forbidden: field is immutable when `immutable` is set

After fix:

$ oc get cm -n clusters-ocpbugs-85778 custom-kubelet-ocpbugs-85778 -o jsonpath='{.immutable}'
(empty — no longer immutable)

$ oc get cm -n clusters-ocpbugs-85778 custom-kubelet-ocpbugs-85778-b -o jsonpath='{.immutable}'
(empty — no longer immutable)

2. No more reconciliation errors:

$ oc describe nodepool ocpbugs-85778-b -n clusters | tail -10
Events:
 Type     Reason          Age                   From                 Message
 ----     ------          ----                  ----                 -------
 Warning  ReconcileError  100m (x2 over 100m)   nodepool-controller  ...

3. Operator logs clean:

$ oc logs -n hypershift deployment/operator --tail=50 | grep -i "immutable\|mirrored"

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@vsolanki12: Jira Issue Verification Checks: Jira Issue OCPBUGS-85778
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-85778 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

What this PR does / why we need it:

When multiple NodePools reference the same kubelet ConfigMap and the source ConfigMap is modified after mirroring, the nodepool-controller's NTO reconciliation fails because mirrored ConfigMaps are created with immutable: true. Kubernetes rejects in-place updates to immutable ConfigMaps, causing NodePools to get stuck in UpdatingConfig=True with:

failed to reconcile mirrored ConfigMap: data: Forbidden: field is immutable when `immutable` is set

This PR:

• Removes the immutable: true flag from mutateMirroredConfig so newly created mirrored ConfigMaps are mutable
• Adds deleteImmutableConfigMapIfNeeded to reconcileMirroredConfigs to handle migration — existing immutable mirrored ConfigMaps are deleted before CreateOrUpdate recreates them as mutable
• This mirrors the parallel fix already applied on the CPO/HCCO side (commit bbd8f63, PR OCPBUGS-60936:fix(hccp): remove immutable field from kubeletconfigmap #6726) which fixed Stage 2 (HCP namespace → guest cluster), while this PR fixes Stage 1 (user namespace → HCP namespace)

Which issue(s) this PR fixes:

Fixes https://issues.redhat.com/browse/OCPBUGS-85778

Special notes for your reviewer:

• Tested on a live KubeVirt HCP cluster (OCP 4.21.9 management, 4.18.30 guest) with two NodePools sharing the same custom-kubelet ConfigMap
• Bug reproduced: both NodePools stuck in UpdatingConfig=True with immutable field error
• After deploying patched operator image: mirrored ConfigMaps recreated as mutable, reconciliation succeeded, no new errors

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Summary by CodeRabbit

• Bug Fixes

• Prevented reconciliation failures by removing immutable ConfigMaps and recreating them as mutable so updates apply reliably.

• Ensured mirrored ConfigMaps are set mutable during reconciliation to allow safe updates.

• Tests

• Updated tests to expect mutable mirrored ConfigMaps and to validate recreation behavior when immutable configs are encountered.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-merge-robot]

Fix included in release 5.0.0-0.nightly-2026-06-18-000016

[vsolanki12]

/jira backport release-4.22,release-4.21,release-4.20

[openshift-ci-robot]

@vsolanki12: The following backport issues have been created:

• OCPBUGS-90083 for branch release-4.22
• OCPBUGS-90084 for branch release-4.21
• OCPBUGS-90085 for branch release-4.20

Queuing cherrypicks to the requested branches to be created after this PR merges:
/cherrypick release-4.22
/cherrypick release-4.21
/cherrypick release-4.20

Details

In response to this:

/jira backport release-4.22,release-4.21,release-4.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-cherrypick-robot]

@openshift-ci-robot: new pull request created: #8765

Details

In response to this:

@vsolanki12: The following backport issues have been created:

• OCPBUGS-90083 for branch release-4.22
• OCPBUGS-90084 for branch release-4.21
• OCPBUGS-90085 for branch release-4.20

Queuing cherrypicks to the requested branches to be created after this PR merges:
/cherrypick release-4.22
/cherrypick release-4.21
/cherrypick release-4.20

In response to this:

/jira backport release-4.22,release-4.21,release-4.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-cherrypick-robot]

@openshift-ci-robot: new pull request created: #8766

Details

In response to this:

@vsolanki12: The following backport issues have been created:

• OCPBUGS-90083 for branch release-4.22
• OCPBUGS-90084 for branch release-4.21
• OCPBUGS-90085 for branch release-4.20

Queuing cherrypicks to the requested branches to be created after this PR merges:
/cherrypick release-4.22
/cherrypick release-4.21
/cherrypick release-4.20

In response to this:

/jira backport release-4.22,release-4.21,release-4.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-cherrypick-robot]

@openshift-ci-robot: new pull request created: #8767

Details

In response to this:

@vsolanki12: The following backport issues have been created:

• OCPBUGS-90083 for branch release-4.22
• OCPBUGS-90084 for branch release-4.21
• OCPBUGS-90085 for branch release-4.20

Queuing cherrypicks to the requested branches to be created after this PR merges:
/cherrypick release-4.22
/cherrypick release-4.21
/cherrypick release-4.20

In response to this:

/jira backport release-4.22,release-4.21,release-4.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.