fix(device): hand out copies from PodManager instead of stored pointers - #2472
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthrough
ChangesPod snapshot isolation
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related issues
Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests.
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 1 file with indirect coverage changes 🚀 New features to boost your workflow:
|
4b4b849 to
85da57a
Compare
GetScheduledPods copied the map but returned the stored *PodInfo pointers, and AddPod writes to those structs in place at pods.go:72. The metrics collector ranges over Devices after the read lock is released while the informer keeps calling AddPod, so go test --race reports it on master. Copy the entries too, reusing the PodInfo.DeepCopy that ListPodsInfo already uses. GetPod gets the same treatment. Mirrors Project-HAMi#2333. Fixes Project-HAMi#2471 Signed-off-by: Lakshya77089 <lakshyasharma7708@gmail.com>
85da57a to
29b5821
Compare
|
/lgtm |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: FouoF, Lakshya77089 The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
What type of PR is this?
/kind bug
What this PR does / why we need it:
PodManager.GetScheduledPodscopies the map but returns the same*PodInfopointers it stores:
That does keep callers off the manager's own map, but
AddPodwrites to thestored structs in place:
and
UpdatePoddoes the same withpi.Pod.The only caller is the Prometheus collector in
cmd/scheduler/metrics.go, whichruns on the scrape goroutine and ranges over
val.Devicesand readsval.NamespaceafterGetScheduledPodshas returned and the read lock isgone. The pod informer calls
AddPodon every pod add and update. The pointerhandoff is synchronised; the field reads are not.
go test ./pkg/device/ --racereports it on master:with a second report on the devices map itself,
mapIterStartagainstmapassign_faststr.The fix
Copy the entries as well as the map, reusing the
PodInfo.DeepCopythatListPodsInfoin the same file already relies on.GetPodgets the sametreatment: it has no non-test callers today, but it returns the stored pointer
in exactly the same way and is the same trap for the next caller.
This mirrors #2333, which made
nodeManager.GetNodereturn a deep copy for thenode cache.
Tests
TestGetScheduledPodsCopiesEntriesdrivesAddPodon one goroutine while asecond does what the collector does.
TestGetScheduledPodsReturnsDetachedEntriesand
TestGetPodReturnsDetachedCopymutate what was handed back and assert themanager's own state is untouched. All three fail on master, the first with five
DATA RACEreports, and pass here.Which issue(s) this PR fixes:
Fixes #2471
Special notes for your reviewer:
The cost is a deep copy per scrape rather than a map copy.
ListPodsInfoalready pays exactly that, so this is not a new class of cost, but it is worth
naming: on a large cluster the collector now copies every tracked pod's device
list once per scrape interval.
The cheaper alternative is to hold the read lock while the collector builds its
metric values, which avoids the copy but holds the lock across metric
construction. I went with the copy because it matches
ListPodsInfoand #2333,and keeps the locking discipline inside the manager rather than spreading it to
callers. Happy to switch if you would rather have the other trade.
Not validated on real hardware. The change is confined to the scheduler's
in-memory pod cache and is covered by unit tests, which CONTRIBUTING allows for
scheduler-scoped changes.
Does this PR introduce a user-facing change?:
AI assistance disclosure: this change was developed with Claude Code — spotting
the race, the fix, and the tests. Flagging the extent up front per CONTRIBUTING.
Summary by CodeRabbit