fix(monitor): prevent fatal data race during feedback observation - #2574
fix(monitor): prevent fatal data race during feedback observation#2574Rickydama3 wants to merge 1 commit into
Conversation
Signed-off-by: Ricky Dama <rickydama2006@gmail.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: leodon33 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthrough
ChangesvGPU monitor concurrency
Estimated code review effort: 1 (Trivial) | ~5 minutes Possibly related issues
Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report❌ Patch coverage is
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 6 files with indirect coverage changes 🚀 New features to boost your workflow:
|
|
This is being closed because it does not comply with the contribution guidelines. |
What type of PR is this?
/kind bug
What this PR does / why we need it:
This PR fixes a critical data race in the
HAMi-vGPUmonitorthat causes the monitor to crash with a fatalconcurrent map iteration and map writeruntime panic.In
cmd/vGPUmonitor/feedback.go, theObserve()function fetches thecontainersmap vialister.ListContainers()and iterates over it in a background ticker loop. However, it was missing a lock acquisition before doing so. At the same time, the Kubernetes Pod informer events (handled bypodInformer.AddEventHandlerinpkg/monitor/nvidia/cudevshr.go) concurrently triggerlister.Update(), which explicitly locks the mutex and modifies the exact same map.Iterating over a map without a lock while another goroutine modifies it triggers a fatal runtime panic in Go. This PR adds the missing
lister.Lock()anddefer lister.UnLock()toObserve(), safely synchronizing it with the informer events. This strictly mirrors the correct locking behavior already implemented for the exact same map incmd/vGPUmonitor/metrics.go.Which issue(s) this PR fixes:
Fixes #2573
Special notes for your reviewer:
This is a standard Go concurrency bug fix. The lock is only held during the fast in-memory map iteration inside
Observe(), so there is no risk of deadlock or blocking the informer goroutines for a significant amount of time.Does this PR introduce a user-facing change?: