Skip to content

OCPBUGS-92062: reduce memory usage in MCC and MCD - #6259

Merged
openshift-merge-bot[bot] merged 4 commits into
openshift:mainfrom
cheesesashimi:zzlotnik/fix-memory-regression
Aug 6, 2026
Merged

OCPBUGS-92062: reduce memory usage in MCC and MCD#6259
openshift-merge-bot[bot] merged 4 commits into
openshift:mainfrom
cheesesashimi:zzlotnik/fix-memory-regression

Conversation

@cheesesashimi

@cheesesashimi cheesesashimi commented Jul 1, 2026

Copy link
Copy Markdown
Member

- What I did

In OCPBUGS-92062, numerous memory usage regressions were identified. The proximate cause was that numerous features were no longer kept behind a FeatureGate. However, the root cause was mostly due to widely-scoped informer configurations. To remedy this, I've done the following:

  1. Tightened up the podLister for the NodeController to only look for the MCO pod. Prior to this, it was looking at all pods in all namespaces. This alone should yield a significant reduction in overall memory usage for very large clusters.
  2. Added copy-on-write behavior to the LayeredNodeState objects. This will only perform a single deep-copy and only when the object is actually mutated. Otherwise, it will return the original object reference to the caller. I've also added a NewMutableLayeredNodeState() constructor which disables this behavior.
  3. Removed the API server calls in NodeController for getting MachineConfigNode objects. Since these are read-only and the NodeController already has a copy of every MachineConfigNode object in memory, we should use them instead of retrieving one from the API server.
  4. Although not directly related to the Machine Config Controller, I tightened up the MachineConfigNodeLister in the MCD to scope it only to the node which the MCD is running on. This will ensure that it will only have the actual MachineConfigNode for the current node.

- How to verify it

Verifying this outside of the test scenario described in OCPBUGS-92062 is difficult because we do not (yet) have a good way to enable pprof analysis at runtime and capture it. Therefore, verification should ensure that the objects are being updated as expected. The current E2E test suite should be able to verify this.

- Description for the changelog
Reduce memory usage in both the MCC and MCD

Summary by CodeRabbit

  • New Features
    • Enhanced controller startup by wiring additional machine-configuration pod informers for more consistent node/pod state tracking.
    • Added support for scoped machine-configuration node informers when the relevant feature gate is enabled, plus internal release image informer updates.
  • Bug Fixes
    • Reduced direct API lookups during MachineConfigNode status synchronization by relying on cached data.
    • Improved machine OS/image updating condition handling, including paused-pool behavior.
    • Made node state annotation mutation safer via deferred copy-on-write, and ensured arbiter/master taints are fully cleared.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jul 1, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@cheesesashimi: This pull request references Jira Issue OCPBUGS-92062, which is invalid:

  • expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

- What I did

In OCPBUGS-92062, numerous memory usage regressions were identified. The proximate cause was that numerous features were no longer kept behind a FeatureGate. However, the root cause was mostly due to widely-scoped informer configurations. To remedy this, I've done the following:

  1. Tightened up the podLister for the NodeController to only look for the MCO pod. Prior to this, it was looking at all pods in all namespaces. This alone should yield a significant reduction in overall memory usage for very large clusters.
  2. Added copy-on-write behavior to the LayeredNodeState objects. This will only perform a single deep-copy and only when the object is actually mutated. Otherwise, it will return the original object reference to the caller. I've also added a NewMutableLayeredNodeState() constructor which disables this behavior.
  3. Removed the API server calls in NodeController for getting MachineConfigNode objects. Since these are read-only and the NodeController already has a copy of every MachineConfigNode object in memory, we should use them instead of retrieving one from the API server.
  4. Although not directly related to the Machine Config Controller, I tightened up the MachineConfigNodeLister in the MCD to scope it only to the node which the MCD is running on. This will ensure that it will only have the actual MachineConfigNode for the current node.

- How to verify it

Verifying this outside of the test scenario described in OCPBUGS-92062 is difficult because we do not (yet) have a good way to enable pprof analysis at runtime and capture it. Therefore, verification should ensure that the objects are being updated as expected. The current E2E test suite should be able to verify this.

- Description for the changelog
Reduce memory usage in both the MCC and MCD

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Jul 1, 2026

Copy link
Copy Markdown

Walkthrough

This PR adds scoped MachineConfig informers to controller and daemon startup, updates node reconciliation and status handling, and changes LayeredNodeState to deep-copy only on its first mutation.

Changes

Node controller runtime and status flow

Layer / File(s) Summary
Informer synchronization and status reconciliation
pkg/controller/node/node_controller.go, pkg/controller/node/status.go
Waits for the MachineConfigNode lister cache, reads nodes from the lister, and expands build-state handling for pool updating conditions, including paused pools.
Pool and master-node reconciliation
pkg/controller/node/node_controller.go
Adds arbiter-specific scheduling and availability handling, coordinates pool updates, and passes control-plane topology explicitly when setting cluster annotations.

Scoped informer integration

Layer / File(s) Summary
Controller context scoped informers
pkg/controller/common/controller_context.go
Adds an MCO pod informer factory filtered by namespace and label, plus node-scoped MachineConfigNode informer helpers.
Machine-config-controller informer startup
cmd/machine-config-controller/start.go
Starts the MCO pod informer factory and supplies its pod informer to the node controller.
Machine-config-daemon informer startup
cmd/machine-config-daemon/start.go
Under the feature gate, wires a node-scoped informer and v1 internal release image informer, then starts the scoped informer.

Layered node state

Layer / File(s) Summary
Copy-on-write node mutation
pkg/controller/common/layered_node_state.go
Tracks defensive copying, deep-copies on the first mutation, and returns the underlying node pointer from Node().

Estimated code review effort: 4 (Complex) | ~45 minutes

Suggested labels: verified

Suggested reviewers: djoshy, umohnani8

🚥 Pre-merge checks | ✅ 14 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (14 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main goal of reducing memory usage in MCC and MCD.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed No *_test.go files were changed, so there are no Ginkgo titles to audit for unstable dynamic content.
Test Structure And Quality ✅ Passed No Ginkgo test files or test blocks were changed in this PR; only production code was modified, so this check is not applicable.
Microshift Test Compatibility ✅ Passed PASS: The diff only touches non-test Go files and contains no It/Describe/Context/When blocks or MicroShift-sensitive e2e specs.
Single Node Openshift (Sno) Test Compatibility ✅ Passed No new Ginkgo e2e tests were added in the diff; only controller/daemon code changed, so SNO-specific test review is not applicable.
Topology-Aware Scheduling Compatibility ✅ Passed No new topology-unaware scheduling constraints were introduced; the node controller already handles HighlyAvailableArbiter topology explicitly.
Ote Binary Stdout Contract ✅ Passed Changed startup code only sets logtostderr=true; no new fmt.Print/os.Stdout/log.SetOutput in init/main/setup of touched files.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed No new Ginkgo e2e tests were added; the changed files are controller/daemon code only, so IPv4/disconnected-network compatibility is not applicable.
No-Weak-Crypto ✅ Passed Diff only adds scoped informer wiring; no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret/token comparisons were introduced.
Container-Privileges ✅ Passed Code-only changes touched no K8s manifests, so no new privileged/hostPID/hostNetwork/hostIPC/allowPrivilegeEscalation settings were introduced.
No-Sensitive-Data-In-Logs ✅ Passed The patch only scopes informers and changes wiring; it adds no new logs or sensitive fields in the touched code.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci
openshift-ci Bot requested review from djoshy and umohnani8 July 1, 2026 18:41
@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 1, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/controller/common/layered_node_state.go`:
- Around line 183-186: The LayeredNodeState.Node method is currently documented
and used as write-safe, but it returns the underlying cached node directly and
bypasses ensureSafeForMutation, so update the API contract to make Node()
read-only and introduce a separate mutation-safe accessor or guard writes before
exposing a writable pointer. Adjust the Node() documentation and any callers
relying on lns.Node() for mutations so they use the safe mutation path in
LayeredNodeState instead of mutating informer-backed state directly.

In `@pkg/controller/node/status.go`:
- Line 34: syncStatusOnly now reads from ctrl.mcnLister.Get, but Run() only
waits on the other cache syncs, so the MachineConfigNode lister can still be
empty at startup and cause false “not found” results. Update the controller’s
WaitForCacheSync in Run() to include ctrl.mcnListerSynced alongside the existing
cache sync checks, keeping the syncStatusOnly path and mcnLister lookup
consistent during startup.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: fed91d30-eac8-47e3-ac95-6d12303c4510

📥 Commits

Reviewing files that changed from the base of the PR and between 9149412 and dd69753.

📒 Files selected for processing (5)
  • cmd/machine-config-controller/start.go
  • cmd/machine-config-daemon/start.go
  • pkg/controller/common/controller_context.go
  • pkg/controller/common/layered_node_state.go
  • pkg/controller/node/status.go

Comment thread pkg/controller/common/layered_node_state.go Outdated
Comment thread pkg/controller/node/status.go
@cheesesashimi
cheesesashimi force-pushed the zzlotnik/fix-memory-regression branch 2 times, most recently from a3901bc to 081cd3d Compare July 1, 2026 19:17
@cheesesashimi

Copy link
Copy Markdown
Member Author

/test e2e-aws-ovn e2e-gcp-op-part1 e2e-gcp-op-part2

@isabella-janssen

Copy link
Copy Markdown
Member

/payload-job periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disruptive-techpreview-1of3 periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disruptive-techpreview-2of3 periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disruptive-techpreview-3of3 periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-serial-1of2 periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-serial-2of2

This should cover any MCN tests

@openshift-ci

openshift-ci Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

@isabella-janssen: trigger 5 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disruptive-techpreview-1of3
  • periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disruptive-techpreview-2of3
  • periodic-ci-openshift-machine-config-operator-release-5.0-periodics-e2e-aws-mco-disruptive-techpreview-3of3
  • periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-serial-1of2
  • periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-serial-2of2

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/12185cd0-7586-11f1-87c5-ce5461a95323-0

@isabella-janssen isabella-janssen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

Change looks reasonable to me & MCN tests are still passing as expected.

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 2, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Tests from second stage were triggered manually. Pipeline can be controlled only manually, until HEAD changes. Use command to trigger second stage.

@cheesesashimi

Copy link
Copy Markdown
Member Author

/jira refresh

@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jul 8, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@cheesesashimi: This pull request references Jira Issue OCPBUGS-92062, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@cheesesashimi

Copy link
Copy Markdown
Member Author

/verified by payload jobs

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jul 8, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@cheesesashimi: This PR has been marked as verified by payload jobs.

Details

In response to this:

/verified by payload jobs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Assisted-By: Claude Sonnet 4.5

Previously, the podLister was looking at all pods in all namespaces.
This consumes a lot of unnecessary CPU and memory since the podLister
must not only synchronize its internal state against the API server, it
must also keep a copy of those objects in memory. The reason why this is
unnecessary is because node controller is only looking for the node that
the MCO pod is running on, which is a single pod.
Assisted-By: Claude Sonnet 4.5

Previously, the LayeredNodeState object would perform a DeepCopy() for
every single mutation of the node object. This could cause significant
memory consumption whenever multiple mutations are performed across many
nodes. Instead, this new implementation will only perform a DeepCopy()
whenever a mutation is actually made.
Assisted-By: Claude Sonnet 4.5

The backing cache for the MachineConfigNodeLister has all of the
MachineConfigNode objects already in memory. Instead of constructing a
potentially large list of MachineConfigNodes incrementally from the API
server, they should be retrieved from the lister instead. This is more
memory efficient because the lister will return references to the
objects in its cache instead of a whole new copy of the object.
Assisted-By: Claude Sonnet 4.5

The MachineConfigNodeInformer was syncing all of the MachineConfigNode
objects when it should only be syncing the MachineConfigNode for the
node in which the Daemon is executing. By scoping the informer only to
the current node, only the current MachineConfigNode will be cached,
yielding memory savings as well as reducing API server load.
@cheesesashimi
cheesesashimi force-pushed the zzlotnik/fix-memory-regression branch from 081cd3d to 5691c56 Compare July 30, 2026 17:41
@openshift-ci-robot openshift-ci-robot removed the verified Signifies that the PR passed pre-merge verification criteria label Jul 30, 2026
@openshift-ci openshift-ci Bot removed the lgtm Indicates that a PR is ready to be merged. label Jul 30, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@cheesesashimi: This pull request references Jira Issue OCPBUGS-92062, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
Details

In response to this:

- What I did

In OCPBUGS-92062, numerous memory usage regressions were identified. The proximate cause was that numerous features were no longer kept behind a FeatureGate. However, the root cause was mostly due to widely-scoped informer configurations. To remedy this, I've done the following:

  1. Tightened up the podLister for the NodeController to only look for the MCO pod. Prior to this, it was looking at all pods in all namespaces. This alone should yield a significant reduction in overall memory usage for very large clusters.
  2. Added copy-on-write behavior to the LayeredNodeState objects. This will only perform a single deep-copy and only when the object is actually mutated. Otherwise, it will return the original object reference to the caller. I've also added a NewMutableLayeredNodeState() constructor which disables this behavior.
  3. Removed the API server calls in NodeController for getting MachineConfigNode objects. Since these are read-only and the NodeController already has a copy of every MachineConfigNode object in memory, we should use them instead of retrieving one from the API server.
  4. Although not directly related to the Machine Config Controller, I tightened up the MachineConfigNodeLister in the MCD to scope it only to the node which the MCD is running on. This will ensure that it will only have the actual MachineConfigNode for the current node.

- How to verify it

Verifying this outside of the test scenario described in OCPBUGS-92062 is difficult because we do not (yet) have a good way to enable pprof analysis at runtime and capture it. Therefore, verification should ensure that the objects are being updated as expected. The current E2E test suite should be able to verify this.

- Description for the changelog
Reduce memory usage in both the MCC and MCD

Summary by CodeRabbit

  • New Features
  • Enhanced controller startup by wiring additional machine-configuration pod informers for more consistent node/pod state tracking.
  • Added support for scoped machine-configuration node informers when the relevant feature gate is enabled, plus internal release image informer updates.
  • Bug Fixes
  • Reduced direct API lookups during MachineConfigNode status synchronization by relying on cached data.
  • Improved machine OS/image updating condition handling, including paused-pool behavior.
  • Made node state annotation mutation safer via deferred copy-on-write, and ensured arbiter/master taints are fully cleared.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
pkg/controller/node/status.go (1)

268-343: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Clear Updating for paused pools when non-build degradation is present.

The paused-pool MOSB dispatch sets Updating=True for IsBuildPrepared, IsBuilding, and IsInInitialState without checking MachineConfigPoolRenderDegraded, MachineConfigPoolNodeDegraded, or MachineConfigPoolPinnedImageSetsDegraded, while the unpaused branch gates those on !isNonBuildDegraded. Add the same guard here, and extract the shared MOSB state switch to keep the two branches in sync.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/controller/node/status.go` around lines 268 - 343, Update the paused-pool
handling in the status reconciliation logic to compute and honor the existing
non-build degradation conditions before setting Updating=True. Extract the
duplicated mosbState dispatch used by both paused and unpaused paths into a
shared helper or flow, ensuring RenderDegraded, NodeDegraded, and
PinnedImageSetsDegraded clear Updating consistently while preserving
paused-specific messages.
🧹 Nitpick comments (2)
pkg/controller/node/node_controller.go (2)

1477-1477: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

context.TODO() used despite Run() now threading a real context.Context.

Run() was migrated to accept and propagate ctx for cancellation (Lines 321-354), but updatePools still creates a detached context.TODO() for the taint-update calls, so those calls won't be cancelled on shutdown. As per path instructions, Go code should use context.Context for cancellation and timeouts.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/controller/node/node_controller.go` at line 1477, Replace the detached
context.TODO() in updatePools with the context.Context received from Run and
threaded through the call chain. Ensure the taint-update calls use this
propagated ctx so cancellation and shutdown signals reach them, without changing
unrelated behavior.

Source: Path instructions


1382-1552: 📐 Maintainability & Code Quality | 🔵 Trivial

Master/arbiter sequencing depends on undocumented ordering of pools.

masterUnavailableCount is only populated while processing a pool named master (Lines 1517-1525) and only consumed while processing a pool named arbiter (Lines 1505-1512). This is correct today only because the sole caller (syncMachineConfigPool) always builds poolsToUpdate as [master, arbiter]. If updatePools is ever called with a different order (or a future caller changes the construction), the arbiter would silently skip the master-unavailability guard since masterUnavailableCount would still be zero. Consider asserting/documenting the required ordering directly in updatePools, or deriving the master state by name lookup instead of loop-order dependency.

Separately, getNodesForPool is called for the arbiter pool twice per master-processing pass — once for the pre-check at Line 1455 and again for its own iteration later in the same call — a minor redundant listing.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/controller/node/node_controller.go` around lines 1382 - 1552, Update
updatePools so master/arbiter sequencing does not rely on the caller’s pools
order: explicitly process the master pool before the arbiter, or otherwise
derive the master unavailable state by name lookup before applying the arbiter
guard. Preserve the existing masterUnavailableCount behavior, and reuse cached
arbiter nodes within the same updatePools invocation instead of calling
getNodesForPool twice.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@pkg/controller/node/status.go`:
- Around line 268-343: Update the paused-pool handling in the status
reconciliation logic to compute and honor the existing non-build degradation
conditions before setting Updating=True. Extract the duplicated mosbState
dispatch used by both paused and unpaused paths into a shared helper or flow,
ensuring RenderDegraded, NodeDegraded, and PinnedImageSetsDegraded clear
Updating consistently while preserving paused-specific messages.

---

Nitpick comments:
In `@pkg/controller/node/node_controller.go`:
- Line 1477: Replace the detached context.TODO() in updatePools with the
context.Context received from Run and threaded through the call chain. Ensure
the taint-update calls use this propagated ctx so cancellation and shutdown
signals reach them, without changing unrelated behavior.
- Around line 1382-1552: Update updatePools so master/arbiter sequencing does
not rely on the caller’s pools order: explicitly process the master pool before
the arbiter, or otherwise derive the master unavailable state by name lookup
before applying the arbiter guard. Preserve the existing masterUnavailableCount
behavior, and reuse cached arbiter nodes within the same updatePools invocation
instead of calling getNodesForPool twice.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 7ab490f7-10ff-416e-93c4-d79a0f85ebb0

📥 Commits

Reviewing files that changed from the base of the PR and between dd69753 and 5691c56.

📒 Files selected for processing (6)
  • cmd/machine-config-controller/start.go
  • cmd/machine-config-daemon/start.go
  • pkg/controller/common/controller_context.go
  • pkg/controller/common/layered_node_state.go
  • pkg/controller/node/node_controller.go
  • pkg/controller/node/status.go

@umohnani8

Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 30, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-ovn
/test e2e-aws-ovn-upgrade
/test e2e-gcp-op-ocl-part1
/test e2e-gcp-op-ocl-part2
/test e2e-gcp-op-part1
/test e2e-gcp-op-part2
/test e2e-gcp-op-single-node
/test e2e-hypershift
/test tls-pqc-readiness

@openshift-ci

openshift-ci Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cheesesashimi, isabella-janssen, umohnani8

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [cheesesashimi,isabella-janssen,umohnani8]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@cheesesashimi

Copy link
Copy Markdown
Member Author

/test e2e-gcp-op-ocl-part2 e2e-gcp-op-part1

@cheesesashimi

Copy link
Copy Markdown
Member Author

/test e2e-gcp-op-part1

@cheesesashimi

Copy link
Copy Markdown
Member Author

/test tls-pqc-readiness

@isabella-janssen

Copy link
Copy Markdown
Member

/override ci/prow/e2e-gcp-op-part1

@openshift-ci

openshift-ci Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

@isabella-janssen: Overrode contexts on behalf of isabella-janssen: ci/prow/e2e-gcp-op-part1

Details

In response to this:

/override ci/prow/e2e-gcp-op-part1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@cheesesashimi

Copy link
Copy Markdown
Member Author

/verified by CI test suites

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Aug 6, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@cheesesashimi: This PR has been marked as verified by CI test suites.

Details

In response to this:

/verified by CI test suites

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci

openshift-ci Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

@cheesesashimi: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-merge-bot
openshift-merge-bot Bot merged commit d690173 into openshift:main Aug 6, 2026
18 checks passed
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@cheesesashimi: Jira Issue Verification Checks: Jira Issue OCPBUGS-92062
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-92062 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

- What I did

In OCPBUGS-92062, numerous memory usage regressions were identified. The proximate cause was that numerous features were no longer kept behind a FeatureGate. However, the root cause was mostly due to widely-scoped informer configurations. To remedy this, I've done the following:

  1. Tightened up the podLister for the NodeController to only look for the MCO pod. Prior to this, it was looking at all pods in all namespaces. This alone should yield a significant reduction in overall memory usage for very large clusters.
  2. Added copy-on-write behavior to the LayeredNodeState objects. This will only perform a single deep-copy and only when the object is actually mutated. Otherwise, it will return the original object reference to the caller. I've also added a NewMutableLayeredNodeState() constructor which disables this behavior.
  3. Removed the API server calls in NodeController for getting MachineConfigNode objects. Since these are read-only and the NodeController already has a copy of every MachineConfigNode object in memory, we should use them instead of retrieving one from the API server.
  4. Although not directly related to the Machine Config Controller, I tightened up the MachineConfigNodeLister in the MCD to scope it only to the node which the MCD is running on. This will ensure that it will only have the actual MachineConfigNode for the current node.

- How to verify it

Verifying this outside of the test scenario described in OCPBUGS-92062 is difficult because we do not (yet) have a good way to enable pprof analysis at runtime and capture it. Therefore, verification should ensure that the objects are being updated as expected. The current E2E test suite should be able to verify this.

- Description for the changelog
Reduce memory usage in both the MCC and MCD

Summary by CodeRabbit

  • New Features
  • Enhanced controller startup by wiring additional machine-configuration pod informers for more consistent node/pod state tracking.
  • Added support for scoped machine-configuration node informers when the relevant feature gate is enabled, plus internal release image informer updates.
  • Bug Fixes
  • Reduced direct API lookups during MachineConfigNode status synchronization by relying on cached data.
  • Improved machine OS/image updating condition handling, including paused-pool behavior.
  • Made node state annotation mutation safer via deferred copy-on-write, and ensured arbiter/master taints are fully cleared.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-merge-robot

Copy link
Copy Markdown
Contributor

Fix included in release 5.0.0-0.nightly-2026-08-07-030024

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants