Skip to content

CNTRLPLANE-3532: Add shared status patching helpers with optimistic locking - #8782

Merged
openshift-merge-bot[bot] merged 1 commit into
openshift:mainfrom
vsolanki12:vsolanki/cntrlplane-3532-statuspatching-helper
Jul 1, 2026
Merged

CNTRLPLANE-3532: Add shared status patching helpers with optimistic locking#8782
openshift-merge-bot[bot] merged 1 commit into
openshift:mainfrom
vsolanki12:vsolanki/cntrlplane-3532-statuspatching-helper

Conversation

@vsolanki12

@vsolanki12 vsolanki12 commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Introduces a support/statuspatching package with two helpers that give controllers a single, consistent way to patch status subresources safely.

PatchStatus takes a mutate callback, re-fetches the object, deep-copies it, runs the callback, and uses MergeFromWithOptimisticLock so a stale write returns a conflict instead of silently winning. It skips the API call entirely when nothing changed. On conflict, the entire cycle (re-fetch → mutate → patch) is retried automatically.

PatchStatusCondition does the same thing for a single metav1.Condition. It accepts a *[]metav1.Condition pointer (e.g. &hcp.Status.Conditions) because HCP exposes conditions as a bare field rather than through getter/setter methods. It uses SetStatusCondition's own change detection to skip no-ops reliably, avoiding false positives from LastTransitionTime being stamped with time.Now().

Why is this needed?

Several controllers (CPO, HCCO, HO, karpenter-operator) write to the same HostedControlPlane status. Without optimistic locking the last writer silently wins. This package gives everyone a single correct implementation to call, with built-in retry-on-conflict handling so callers never need to handle 409s themselves.

How to test

  • Unit tests cover no-op skipping, happy path patching, optimistic lock verification, mutate error propagation, retry-on-conflict with persisted state verification, and Get failure (NotFound) propagation
  • Run with go test ./support/statuspatching/...

Jira

CNTRLPLANE-3532

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci

openshift-ci Bot commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 19, 2026
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jun 19, 2026
@openshift-ci-robot

openshift-ci-robot commented Jun 19, 2026

Copy link
Copy Markdown

@vsolanki12: This pull request references CNTRLPLANE-3532 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

What does this PR do?

Introduces a support/statuspatching package with three helpers that give controllers a single, consistent way to patch status subresources safely.

PatchStatus takes a mutate callback, deep-copies the object beforehand, and uses MergeFromWithOptimisticLock so a stale write returns a conflict instead of silently winning. It skips the API call entirely when nothing changed.

PatchStatusCondition does the same thing for a single metav1.Condition. It accepts a *[]metav1.Condition pointer (e.g. &hcp.Status.Conditions) because HCP exposes conditions as a bare field rather than through getter/setter methods.

PatchStatusJSON is for status fields that can legitimately be null. JSON Merge Patch (RFC 7386) treats null as "delete this field", which corrupts fields like availableUpdates or completionTime. This helper generates RFC 6902 ops instead, filters them to /status/*, and prepends a test op on resourceVersion for optimistic locking.

Why is this needed?

Several controllers (CPO, HCCO, HO, karpenter-operator) write to the same HostedControlPlane status. Without optimistic locking the last writer silently wins. PR #8562 already migrated most call sites, but the pattern was duplicated in each controller. This package gives everyone a single correct implementation to call.

How to test

  • Unit tests cover no-op skipping, happy path patching, optimistic lock verification, status-only filtering, and null field preservation
  • Run with go test ./support/statuspatching/...

Jira

CNTRLPLANE-3532

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Jun 19, 2026

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Walkthrough

The support/statuspatching package adds helpers for optimistic-lock status updates and condition updates. The helpers re-fetch the latest object, skip no-op changes, and patch the status subresource only when the mutated status differs. The test file adds fake-client interception for status patch calls and covers no-op cases, mutate errors, missing objects, and conflict retries.


Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error)

Check name Status Explanation Resolution
Container-Privileges ❌ Error PR adds new manifests with privileged:true, hostPID/hostNetwork, and allowPrivilegeEscalation:true in kubelet-config and kubevirt CSI daemonsets. Remove or justify the privileged settings in those manifests; container-privileges policy forbids privileged:true, hostPID/hostNetwork, and allowPrivilegeEscalation:true.
✅ Passed checks (10 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately summarizes the main change: new shared status patching helpers with optimistic locking.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed No Ginkgo-style titles were added, and all t.Run names in the new tests are static, descriptive strings with no dynamic data.
Test Structure And Quality ✅ Passed These are plain table-driven unit tests, not Ginkgo; they use repo-standard t.Run/Gomega style and have no cluster waits or cleanup issues.
Topology-Aware Scheduling Compatibility ✅ Passed Only status-patching helpers and tests were added; no deployment, replica, affinity, nodeSelector, or topology-aware scheduling logic appears in the changed code.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed The PR only adds standard Go unit tests in support/statuspatching; no Ginkgo e2e tests, IPv4-only assumptions, or external connectivity were found.
No-Weak-Crypto ✅ Passed No MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret comparisons appear in the touched statuspatching files.
No-Sensitive-Data-In-Logs ✅ Passed No logging calls or sensitive strings were added in the changed statuspatching files.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci openshift-ci Bot added area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release and removed do-not-merge/needs-area labels Jun 19, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@support/statuspatching/statuspatching.go`:
- Line 108: The comment at line 108 that says "8. Apply the patch" has duplicate
step numbering since step 8 already appears at line 102. Update the step number
in the comment from 8 to 9 to maintain sequential numbering in the patch
application logic.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 4447923e-c270-401e-b7ee-5cf73e96f58a

📥 Commits

Reviewing files that changed from the base of the PR and between dca6f75 and fb03e25.

📒 Files selected for processing (2)
  • support/statuspatching/statuspatching.go
  • support/statuspatching/statuspatching_test.go

Comment thread support/statuspatching/statuspatching.go Outdated
@codecov

codecov Bot commented Jun 19, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 42.30%. Comparing base (25817d4) to head (3114a9c).
⚠️ Report is 288 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8782      +/-   ##
==========================================
+ Coverage   41.43%   42.30%   +0.87%     
==========================================
  Files         756      773      +17     
  Lines       93658    97413    +3755     
==========================================
+ Hits        38807    41215    +2408     
- Misses      52128    53315    +1187     
- Partials     2723     2883     +160     
Files with missing lines Coverage Δ
support/statuspatching/statuspatching.go 100.00% <100.00%> (ø)

... and 100 files with indirect coverage changes

Flag Coverage Δ
cmd-support 35.88% <100.00%> (+1.00%) ⬆️
cpo-hostedcontrolplane 45.31% <ø> (+1.81%) ⬆️
cpo-other 45.10% <ø> (+2.35%) ⬆️
hypershift-operator 50.63% <ø> (-0.94%) ⬇️
other 31.69% <ø> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
support/statuspatching/statuspatching_test.go (1)

69-76: ⚡ Quick win

Narrow the conflict interceptor to the status subresource.

Line 70 currently returns a conflict for every subresource patch. That can let this test pass even if callers patch the wrong subresource. Restricting this to "status" makes the test precise.

Proposed diff
 return interceptor.NewClient(underlying, interceptor.Funcs{
 	SubResourcePatch: func(ctx context.Context, c client.Client, subResourceName string, obj client.Object, patch client.Patch, opts ...client.SubResourcePatchOption) error {
+		if subResourceName != "status" {
+			return fmt.Errorf("unexpected subresource patch: %s", subResourceName)
+		}
 		return apierrors.NewConflict(
 			schema.GroupResource{Group: "", Resource: "nodes"},
 			obj.GetName(),
 			fmt.Errorf("the object has been modified"),
 		)
 	},
 })
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@support/statuspatching/statuspatching_test.go` around lines 69 - 76, The
SubResourcePatch interceptor function currently returns a conflict error for any
subresource patch operation, but it should only return this conflict when
patching the status subresource. Add a condition at the beginning of the
SubResourcePatch function to check if the subResourceName parameter equals
"status", and only return the conflict error in that case. For other
subresources, delegate to the underlying client by calling its SubResourcePatch
method or return nil to allow the patch to proceed.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@support/statuspatching/statuspatching_test.go`:
- Around line 69-76: The SubResourcePatch interceptor function currently returns
a conflict error for any subresource patch operation, but it should only return
this conflict when patching the status subresource. Add a condition at the
beginning of the SubResourcePatch function to check if the subResourceName
parameter equals "status", and only return the conflict error in that case. For
other subresources, delegate to the underlying client by calling its
SubResourcePatch method or return nil to allow the patch to proceed.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 3c69c41d-fb19-40ac-9d8f-35c5ab55677b

📥 Commits

Reviewing files that changed from the base of the PR and between fb03e25 and d91c7b9.

📒 Files selected for processing (2)
  • support/statuspatching/statuspatching.go
  • support/statuspatching/statuspatching_test.go
🚧 Files skipped from review as they are similar to previous changes (1)
  • support/statuspatching/statuspatching.go

@vsolanki12
vsolanki12 force-pushed the vsolanki/cntrlplane-3532-statuspatching-helper branch 2 times, most recently from b929e39 to 1c7030d Compare June 19, 2026 12:13
Comment on lines +33 to +38
original := obj.DeepCopyObject().(client.Object)
mutate()
if equality.Semantic.DeepEqual(original, obj) {
return nil
}
return c.Status().Patch(ctx, obj, client.MergeFromWithOptions(original, client.MergeFromWithOptimisticLock{}))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to minimize requeues, I think we should get the original first and wrap in retryOnConflict
see:

func UpdateObject[T client.Object](ctx context.Context, c client.Client, obj T, mutate func() error) error {
return retry.RetryOnConflict(retry.DefaultBackoff, func() error {
if err := c.Get(ctx, client.ObjectKeyFromObject(obj), obj); err != nil {
return err
}
original := obj.DeepCopyObject().(T)
if err := mutate(); err != nil {
return err
}
return c.Patch(ctx, obj, client.MergeFrom(original))
})
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. All three functions now use retry.RetryOnConflict(retry.DefaultBackoff, ...).

// last TransitionTime in certain states).
// mutate() must only modify status fields on obj.
// The patch is filtered to only /status/* operations before applying.
func PatchStatusJSON(ctx context.Context, c client.Client, obj client.Object, mutate func()) error {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need this? this was meant as a workaround, we should not expose it as reusable function.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed. Dropped from this PR since it was a workaround specific to hcpstatus.go.

// It captures a deep copy before calling mutate, then patches with
// MergeFromWithOptimisticLock so concurrent writers get a conflict error instead of silently overwriting.
// Mutate must only modify status fields.
func PatchStatus(ctx context.Context, c client.Client, obj client.Object, mutate func()) error {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mutate() should return an error

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Changed to mutate func() error.

return nil
}
return c.Status().Patch(ctx, obj, client.MergeFromWithOptions(original, client.MergeFromWithOptimisticLock{}))
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the main CPO reconciler finction does multiple changes in sequence to the HCP status, then a bulk patch at the end, which doesn't fit the mutate() pattern. We should have a separate function to simply patch an object. e.g.

func PatchObjectStatus(ctx context.Context, c client.Client, obj client.Object) error {
	return retry.RetryOnConflict(retry.DefaultBackoff, func() error {
		existing := obj.DeepCopyObject().(client.Object)
		if err := c.Get(ctx, client.ObjectKeyFromObject(obj), existing); err != nil {
			return err
		}

		return c.Status().Patch(ctx, obj, client.MergeFromWithOptions(existing, client.MergeFromWithOptimisticLock{}))
	})
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Added PatchObjectStatusthat patches an already mutated object against server state, for the CPO bulk-mutation-then-patch pattern.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how is this better/different from just passing a no op mutate func here and we keep a single authoritative func?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If really need to keep the separation I couldn't tell the intent different by the current func naming

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason why we can't have one single exported function and let the others be part of the implementation?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

vibe coded example fwiw

 // Condition binds a condition to the slice it should be set on.
  type Condition struct {
        Target    *[]metav1.Condition
        Condition metav1.Condition
  }

  // PatchStatus re-fetches obj, applies mutate (if non-nil), sets any
  // conditions, and patches the status subresource with optimistic locking.
  // Conflicts are retried automatically. mutate must only modify status fields.
  //
  // Usage patterns:
  //
  //    // Mutate callback only:
  //    statuspatching.PatchStatus(ctx, c, hcp, func() error {
  //        hcp.Status.Version = newVersion
  //        return nil
  //    }, nil)
  //
  //    // Single condition only:
  //    statuspatching.PatchStatus(ctx, c, hcp, nil, &statuspatching.Condition{
  //        Target:    &hcp.Status.Conditions,
  //        Condition: metav1.Condition{Type: "Ready", Status: metav1.ConditionTrue, Reason: "AllGood"},
  //    })
  //
  //    // Both:
  //    statuspatching.PatchStatus(ctx, c, hcp, func() error {
  //        hcp.Status.Version = newVersion
  //        return nil
  //    }, &statuspatching.Condition{
  //        Target:    &hcp.Status.Conditions,
  //        Condition: metav1.Condition{Type: "Ready", Status: metav1.ConditionTrue, Reason: "AllGood"},
  //    })
  func PatchStatus(ctx context.Context, c client.Client, obj client.Object, mutate func() error, cond *Condition) error {
        return retry.RetryOnConflict(retry.DefaultBackoff, func() error {
                if err := c.Get(ctx, client.ObjectKeyFromObject(obj), obj); err != nil {
                        return err
                }
                original := obj.DeepCopyObject().(client.Object)
                if mutate != nil {
                        if err := mutate(); err != nil {
                                return err
                        }
                }
                if cond != nil {
                        meta.SetStatusCondition(cond.Target, cond.Condition)
                }
                if equality.Semantic.DeepEqual(original, obj) {
                        return nil
                }
                return c.Status().Patch(ctx, obj, client.MergeFromWithOptions(original, client.MergeFromWithOptimisticLock{}))
        })
  }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how is this better/different from just passing a no op mutate func here and we keep a single authoritative func?

the original function fetches the obj and save it in the passed obj and then apply mutate. This fetches the existing obj in a separate variable to be used a original for patching the passed in obj

@vsolanki12 vsolanki12 Jun 30, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped PatchObjectStatus after cblecker identified the stale-state retry bug, same class of issue as OCPBUGS-93462. On consolidating into a single function, PatchStatusCondition relies on SetStatusCondition changed return value for no-op detection, which DeepEqual can't do reliably due to LastTransitionTime being stamped on every call. Keeping them as two separate functions for now, open to revisiting if you feel otherwise.

@vsolanki12
vsolanki12 force-pushed the vsolanki/cntrlplane-3532-statuspatching-helper branch from 1c7030d to 2525891 Compare June 24, 2026 04:34

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
support/statuspatching/statuspatching_test.go (1)

350-354: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add optimistic-lock payload assertions to the remaining helper test suites.

TestPatchStatusCondition and TestPatchObjectStatus verify invocation (and type for one path), but they don’t assert that the generated patch payload carries resourceVersion. Since optimistic locking is a key contract, asserting this in both suites would better protect against regressions.

Suggested hardening
@@
            err := PatchStatusCondition(context.Background(), c, svc, &svc.Status.Conditions, tt.newCondition)
            g.Expect(err).ToNot(HaveOccurred())
            g.Expect(recorder.called).To(Equal(tt.expectPatchCalled))
+           if tt.expectPatchCalled {
+               g.Expect(recorder.patchType).To(Equal(types.MergePatchType))
+               g.Expect(string(recorder.patchData)).To(ContainSubstring("resourceVersion"))
+           }
@@
            if tt.expectPatchCalled {
                g.Expect(recorder.patchType).To(Equal(types.MergePatchType))
+               g.Expect(string(recorder.patchData)).To(ContainSubstring("resourceVersion"))
            }

Also applies to: 420-422

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@support/statuspatching/statuspatching_test.go` around lines 350 - 354, The
test suites TestPatchStatusCondition and TestPatchObjectStatus currently verify
that the patch functions are invoked but do not assert that the generated patch
payload includes the resourceVersion field for optimistic locking. Add
assertions in both test suites to inspect the patch payload captured by the
recorder (likely in the recorder.patch field or similar) and verify that it
contains the resourceVersion field. This should be done after verifying the
patch was called to ensure the optimistic locking contract is maintained across
both test cases.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@support/statuspatching/statuspatching_test.go`:
- Around line 50-53: The SubResourcePatch function at line 52 ignores the error
returned by patch.Data(obj) using a blank identifier, which violates Go security
guidelines about handling error returns. Capture the error return value from
patch.Data(obj) instead of discarding it with _, check if the error is not nil,
and handle it appropriately by either returning the error or failing the test
with an error message. This same fix must also be applied to the identical
pattern at line 85 in the file where patch.Data is called.

---

Nitpick comments:
In `@support/statuspatching/statuspatching_test.go`:
- Around line 350-354: The test suites TestPatchStatusCondition and
TestPatchObjectStatus currently verify that the patch functions are invoked but
do not assert that the generated patch payload includes the resourceVersion
field for optimistic locking. Add assertions in both test suites to inspect the
patch payload captured by the recorder (likely in the recorder.patch field or
similar) and verify that it contains the resourceVersion field. This should be
done after verifying the patch was called to ensure the optimistic locking
contract is maintained across both test cases.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 0efa4a0d-5110-4408-b62a-c45a714d112c

📥 Commits

Reviewing files that changed from the base of the PR and between 1c7030d and 2525891.

📒 Files selected for processing (2)
  • support/statuspatching/statuspatching.go
  • support/statuspatching/statuspatching_test.go
🚧 Files skipped from review as they are similar to previous changes (1)
  • support/statuspatching/statuspatching.go

Comment thread support/statuspatching/statuspatching_test.go
@vsolanki12
vsolanki12 marked this pull request as ready for review June 24, 2026 04:38
@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 24, 2026
@openshift-ci
openshift-ci Bot requested review from devguyio and muraee June 24, 2026 04:38
@vsolanki12
vsolanki12 force-pushed the vsolanki/cntrlplane-3532-statuspatching-helper branch from 2525891 to dfafe87 Compare June 24, 2026 04:45
@muraee

muraee commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

/approve

@openshift-ci

openshift-ci Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: muraee, vsolanki12

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 24, 2026
// TEST: PatchStatusCondition
//==================================================================

func TestPatchStatusCondition(t *testing.T) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PatchStatus and PatchObjectStatus both have dedicated RetryOnConflict tests, but PatchStatusCondition doesn't. The retry path is especially worth testing here because of the conditions pointer — after c.Get refreshes obj on retry, the *[]metav1.Condition pointer must still reference the correct slice. A TestPatchStatusCondition_RetryOnConflict using newFakeClientWithConflictThenSuccess would lock that in.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added TestPatchStatusCondition_RetryOnConflict to verify that the conditions pointer remains valid after re-fetching the object during a retry.

if err := c.Get(ctx, client.ObjectKeyFromObject(obj), existing); err != nil {
return err
}
return c.Status().Patch(ctx, obj, client.MergeFromWithOptions(existing, client.MergeFromWithOptimisticLock{}))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PatchStatus skips no-ops via DeepEqual (line 36) and PatchStatusCondition via SetStatusCondition's return value (line 55), but PatchObjectStatus always calls Status().Patch() even when the local object matches server state. The package doc says these helpers "skip no-op updates" — this function doesn't.

Unnecessary patches bump resourceVersion, triggering watch events and reconcile loops across every controller watching the object. Adding if equality.Semantic.DeepEqual(existing, obj) { return nil } before the patch would make this consistent. (Or if this function gets folded into PatchStatus per the other thread, the skip comes for free.)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed PatchObjectStatus entirely — no longer applies.

// current server state. Use this when the caller has already made multiple
// status mutations in sequence (e.g. the main CPO reconciler) and wants a
// single bulk patch at the end rather than the mutate-callback pattern.
func PatchObjectStatus(ctx context.Context, c client.Client, obj client.Object) error {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Building on the API consolidation discussion — there's a concrete correctness issue with the retry semantics here. When PatchObjectStatus retries after a 409, line 68 deep-copies the caller's stale, pre-mutated object on every iteration. Line 69 fetches fresh server state into that copy. The merge patch is then diff(fresh server, stale caller) — which includes ALL fields that differ, not just what the caller intended to change. This silently reverts concurrent status changes from other controllers.

Contrast with PatchStatus which re-fetches into obj and re-runs the mutate callback on every retry, always operating on fresh state. That's another argument for consolidating into PatchStatus — the retry-replay-mutation pattern is inherently correct for concurrent writers, while retry-with-stale-diff is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, this has the same stale-state problem as OCPBUGS-93462. Dropped PatchObjectStatus and its tests. If we need a bulk-patch pattern later, it can be designed with correct retry semantics during the CPO migration.

// avoiding false positives from LastTransitionTime being stamped with time.Now().
// Pass a pointer to the object's conditions slice (e.g. &hcp.Status.Conditions)
// since HCP types expose conditions as a bare field, not via getter/setter methods.
func PatchStatusCondition(ctx context.Context, c client.Client, obj client.Object, conditions *[]metav1.Condition, condition metav1.Condition) error {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: the conditions pointer must alias a field within obj for correctness (so Get refreshes the same memory the pointer references). The doc comment explains this well, and Go's type system can't enforce it, so this is inherently documentation-enforced. Just flagging it — enxebre's Condition struct proposal in the other thread would make the relationship more visible at call sites.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This follows the same pattern as &hcp.Status.Conditions used throughout the codebase. Wrapping it in a struct would only relocate the aliasing, not eliminate it.

}
}

func TestPatchObjectStatus_RetryOnConflict(t *testing.T) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TestPatchObjectStatus_RetryOnConflict verifies the retry happened (recorder.called) but doesn't assert the correct status values were written afterward. Fetching the node from the client after the call and checking Phase == NodeTerminated would strengthen the contract.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed along with PatchObjectStatus.

// mutate must only modify status fields on obj.
func PatchStatus(ctx context.Context, c client.Client, obj client.Object, mutate func() error) error {
return retry.RetryOnConflict(retry.DefaultBackoff, func() error {
if err := c.Get(ctx, client.ObjectKeyFromObject(obj), obj); err != nil {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No test covers the path where the initial Get fails (e.g., object was deleted). A single test calling PatchStatus against a non-seeded fake client and asserting NotFound is returned would document the error-propagation contract.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added TestPatchStatus_GetFailure to cover the unseeded object scenario, verifying a NotFound error is returned and no patch is applied.

@cblecker

Copy link
Copy Markdown
Member

@coderabbitai resume

@coderabbitai

coderabbitai Bot commented Jun 26, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Reviews resumed.

@vsolanki12
vsolanki12 force-pushed the vsolanki/cntrlplane-3532-statuspatching-helper branch from c958425 to d5ab2b5 Compare June 30, 2026 04:59

@cblecker cblecker left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: PatchStatus now has TestPatchStatus_GetFailure (thanks for adding that), but there's no equivalent for PatchStatusCondition. The code path is identical so the risk is low, but a symmetric TestPatchStatusCondition_GetFailure would round out the error contract coverage.

node.Status.Phase = corev1.NodeTerminated
return nil
})
g.Expect(err).ToNot(HaveOccurred())

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same note I left on the removed PatchObjectStatus retry test — this verifies the retry mechanism fired but doesn't assert the final persisted state. TestPatchStatusCondition_RetryOnConflict (below) does this correctly by checking the conditions after the call. Would be good to add a c.Get + assert on result.Status.Phase == corev1.NodeTerminated here too, to catch subtle retry-loop bugs like the stale-state issue we saw before.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — added c.Get + result.Status.Phase == NodeTerminated assertion to TestPatchStatus_RetryOnConflict in the latest push, so it now verifies the mutation actually persisted.

…king

Signed-off-by: Vimal Solanki <vsolanki@redhat.com>
@vsolanki12
vsolanki12 force-pushed the vsolanki/cntrlplane-3532-statuspatching-helper branch from d5ab2b5 to 3114a9c Compare July 1, 2026 04:20

@vsolanki12 vsolanki12 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added TestPatchStatusCondition_GetFailure — creates a Service not seeded into the fake client, calls PatchStatusCondition, and asserts IsNotFound + no patch call. Symmetric with TestPatchStatus_GetFailure. Both changes are in the latest push (3114a9c).

@cblecker

cblecker commented Jul 1, 2026

Copy link
Copy Markdown
Member

/lgtm

nit: your PR description may be slightly out of date with the changes you made, but not blocking

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 1, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-v2-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke

@vsolanki12

Copy link
Copy Markdown
Contributor Author

thank you @cblecker updated as per suggestion.

@vsolanki12

Copy link
Copy Markdown
Contributor Author

/test e2e-aws-upgrade-hypershift-operator

earlier it got failed due to resource issues on CI side.

0001-01-01T00:00:00Z 0x : 0/60 nodes are available: 1 Insufficient memory, 13 node(s) didn't match Pod's node affinity/selector, 46 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/60 nodes are available: 1 No preemption victims found for incoming pod, 59 Preemption is not helpful for scheduling.

@hypershift-jira-solve-ci

hypershift-jira-solve-ci Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Confirmed: 37 tests, 0 failures, 0 errors. The test itself passed completely. Now I have all the evidence I need. Let me compile the final report.

Test Failure Analysis Complete

Job Information

Test Failure Analysis

Error

"e2e-aws-upgrade-hypershift-operator" post steps failed: pod "e2e-aws-upgrade-hypershift-operator-dump-management-cluster" failed: pod pending for more than 1h0m0s: pod has not been scheduled in 1h0m0.001045092s:
0/56 nodes are available: 1 node(s) didn't match pod anti-affinity rules, 12 node(s) didn't match Pod's node affinity/selector, 43 node(s) had untolerated taint(s).

Summary

All 37 e2e tests (TestUpgradeHyperShiftOperator) passed successfully with 0 failures. The pre phase (cluster creation, HyperShift install) and the test phase (upgrade e2e tests) both completed without errors. The job failed solely because the dump-management-cluster post phase step — a diagnostic artifact-collection step that runs after all tests finish — could not be scheduled onto any node in the CI build cluster for over 1 hour due to node affinity/selector mismatches, untolerated taints, and pod anti-affinity rules. This is a CI infrastructure scheduling issue completely unrelated to the PR's code changes.

Root Cause

The root cause is a CI infrastructure pod scheduling failure on the shared OpenShift CI build cluster, not a product or test code bug.

When the post phase began at 2026-07-01T09:18:06Z, the dump-management-cluster pod was created to collect diagnostic artifacts from the management cluster. However, the pod could never be scheduled because no node in the CI cluster satisfied all scheduling constraints simultaneously:

  1. 12 nodes were excluded by the pod's nodeAffinity/nodeSelector — the pod requires specific node labels that these nodes don't have.
  2. 37–55 nodes (varying over the 1-hour window) had untolerated taints — likely reserved for other workloads or in maintenance.
  3. 1–12 nodes were unschedulable (cordoned) — likely undergoing rolling updates or maintenance during the observation window.
  4. 1 node was excluded by pod anti-affinity rules — another pod with conflicting anti-affinity was already running there.
  5. At one point, 1 node reported insufficient memory, indicating resource exhaustion.

The combination of all constraints left zero eligible nodes for the entire 1-hour timeout period. After the timeout, ci-operator deleted the pending pod and marked the post phase as failed, which caused the overall job to be reported as failed despite all actual tests passing.

Critically:

  • The hypershift-dump-extra post step (which ran after dump-management-cluster failed) succeeded in 24 seconds.
  • The destroy-management-cluster post step also succeeded in 7m16s.
  • This confirms the CI cluster was capable of scheduling pods — the scheduling failure was specific to the dump-management-cluster pod's constraints intersecting with a congested/maintained cluster state.

This is a flaky CI infrastructure issue — not caused by PR #8782's code changes.

Recommendations
  1. Retry the job — This is a transient CI infrastructure issue. A re-run when the build cluster has available nodes matching the scheduling constraints should succeed.
  2. This failure is unrelated to PR CNTRLPLANE-3532: Add shared status patching helpers with optimistic locking #8782 — The PR adds shared status patching helpers with optimistic locking. All 37 HyperShift operator upgrade tests passed, including the core upgrade flow (Calculate invariants → Upgrade → Verify invariants), hosted cluster validation, and post-teardown checks.
  3. No code changes needed — The failure is in a CI artifact-collection step (dump-management-cluster), not in any product or test code.
  4. If the issue recurs frequently, the CI infra team should investigate the node affinity/selector requirements and taint tolerations for the dump-management-cluster step definition (see step registry reference) to ensure they are not overly restrictive for the shared build cluster.
Evidence
Evidence Detail
E2E Test Result 37 tests passed, 0 failures, 0 errors (junit.xml: <testsuites tests="37" failures="0" errors="0">)
Pre Phase Succeeded in 18m36s (ipi-install-rbac → create-management-cluster → hypershift-install → resolve-nodepool-releases)
Test Phase Succeeded in 32m24s (hypershift-aws-run-e2e-nested, all PASS including TestUpgradeHyperShiftOperator)
Failed Step dump-management-cluster (post phase) — pod pending 1h0m0s, never scheduled
Scheduling Failure 91 scheduler events, all showing no eligible nodes: node affinity mismatches (12), untolerated taints (37-55), unschedulable/cordoned (1-12), anti-affinity conflicts (1)
Post Steps After Failure hypershift-dump-extra succeeded (24s), destroy-management-cluster succeeded (7m16s)
Failure Start Time 2026-07-01T09:18:06Z (post phase start)
Failure End Time 2026-07-01T10:18:06Z (1h timeout reached)
Overall Job Duration 2h0m8s
PR Relation None — failure is in CI infrastructure (pod scheduling), not in product/test code

@cblecker

cblecker commented Jul 1, 2026

Copy link
Copy Markdown
Member

/override ci/prow/e2e-aws-upgrade-hypershift-operator
/verified later @vsolanki12

This job succeeded but failed in the dump. Considering this PR doesn't wire in these changes yet, this is okay.

Setting verified to later for the same reason -- will be wired in, in a follow up.

@openshift-ci-robot openshift-ci-robot added verified-later verified Signifies that the PR passed pre-merge verification criteria labels Jul 1, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@cblecker: This PR has been marked to be verified later by @vsolanki12.

Details

In response to this:

/override ci/prow/e2e-aws-upgrade-hypershift-operator
/verified later @vsolanki12

This job succeeded but failed in the dump. Considering this PR doesn't wire in these changes yet, this is okay.

Setting verified to later for the same reason -- will be wired in, in a follow up.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci

openshift-ci Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

@cblecker: Overrode contexts on behalf of cblecker: ci/prow/e2e-aws-upgrade-hypershift-operator

Details

In response to this:

/override ci/prow/e2e-aws-upgrade-hypershift-operator
/verified later @vsolanki12

This job succeeded but failed in the dump. Considering this PR doesn't wire in these changes yet, this is okay.

Setting verified to later for the same reason -- will be wired in, in a follow up.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci

openshift-ci Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

@cblecker: Overrode contexts on behalf of cblecker: ci/prow/e2e-aws-upgrade-hypershift-operator

Details

In response to this:

/override ci/prow/e2e-aws-upgrade-hypershift-operator

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci

openshift-ci Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

@vsolanki12: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria verified-later

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants