fix(vGPUmonitor): bound v0 spec aggregation loops by procnum - #2328
Conversation
fix(vGPUmonitor): bound v0 spec aggregation loops by procnum The six metric-aggregation methods in pkg/monitor/nvidia/v0/spec.go iterated over the full 1024-slot procs array instead of stopping at procnum. Any slot beyond the active process count may retain stale, non-zeroed data from a previous CUDA process, causing DeviceMemoryTotal and the five related methods to inflate reported usage. Added an activeProcCount() helper that clamps procnum to [0, len(procs)] before slicing, since procnum is read from raw mmap'd shared memory with no upstream validation and could otherwise be negative or >1024, causing a panic rather than just wrong data. The v1 implementation already bounds these loops by procnum but has the same unguarded-value issue; that's tracked separately under #2310 and intentionally left out of scope here. Fixes #2327
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: Brijesh-Thakkar The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe v0 NVIDIA monitor now clamps Changesv0 process bounds
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related issues
Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
This PR fixes vGPUmonitor’s NVIDIA v0 shared-memory spec metric aggregation to only consider active process slots (bounded by procnum) rather than iterating all 1024 fixed slots, preventing stale-slot data from inflating memory/utilization metrics and triggering false throttling.
Changes:
- Add
Spec.activeProcCount()to clampprocnumto[0, len(procs)]before slicing. - Update all six v0 aggregation methods to iterate over
procs[:activeProcCount()]. - Strengthen unit tests with explicit
procnumvalues, boundary clamp coverage, and “stale slots excluded” regressions.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| pkg/monitor/nvidia/v0/spec.go | Clamp procnum and bound all aggregation loops to active proc slots to avoid stale-data inflation and slice panics. |
| pkg/monitor/nvidia/v0/spec_test.go | Add boundary tests for clamping and regression coverage ensuring stale slots beyond procnum are excluded. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
Done in #2282 |
What type of PR is this?
/kind bug
What this PR does / why we need it:
The six metric-aggregation methods in
pkg/monitor/nvidia/v0/spec.go(DeviceMemoryContextSize,DeviceMemoryModuleSize,DeviceMemoryBufferSize,DeviceMemoryOffset,DeviceMemoryTotal,DeviceSmUtil) iterated over the full 1024-slotprocsarray instead of stopping atprocnum.Any slot beyond the active process count may retain stale, non-zeroed data from a previous CUDA
process, causing
DeviceMemoryTotal()and the five related methods to inflate reported usage.Inflated usage can trigger false throttling of live GPU workloads via the feedback loop in
feedback.go.This PR adds an
activeProcCount()helper that clampsprocnumto[0, len(procs)]beforeslicing. This is needed because
procnumis read directly from raw mmap'd shared memory(written by
libvgpu.soinside the container) with no field-level validation upstream — theexisting file-size check only guarantees the buffer is the right total size, not that individual
field values are sane. A naive fix using
procs[:int(procnum)]would trade the stale-data bugfor a new panic (slice bounds out of range) if
procnumwere ever corrupted to a negative valueor a value greater than 1024. The clamp makes out-of-range values degrade safely instead.
Which issue(s) this PR fixes:
Fixes #2327
Note:
v1implementation (pkg/monitor/nvidia/v1/spec.go) already bounds these loops byprocnum, but has the same unguarded-value issue — a corruptedprocnumthere would panictoo. That's tracked separately under bug(vGPUmonitor): v1 spec aggregations ignore
statusfield, inflating memory metrics for dead process slots #2310 (which covers a different, pre-existing v1 defect:a missing
p.statusguard). This PR intentionally does not touchv1/spec.goto keep scopeminimal; flagging it here for whoever picks up bug(vGPUmonitor): v1 spec aggregations ignore
statusfield, inflating memory metrics for dead process slots #2310.spec_test.gohadprocnumimplicitly left at its zero value, meaning theyweren't actually exercising the bounded-iteration path before this fix — they'd have passed
regardless of correctness. I added explicit
procnumvalues to all affected existing cases,plus:
TestSpec_ActiveProcCounttable covering 7 boundary values (zero, normal, exactcapacity, negative, large negative, over-capacity, large over-capacity)
TestSpec_DeviceMemoryTotalproving negative/overflowprocnumvalues don't panic
go vetandgolangci-lintboth pass clean on the changed package; fullv0test suite(26 tests) passes with
-race.Does this PR introduce a user-facing change?:
Summary by CodeRabbit
Bug Fixes
Tests