fix(monitor): clamp DeviceNum to maxDevices to prevent scrape-path panic - #2400
Conversation
📝 WalkthroughWalkthrough
ChangesNVIDIA device count bounds
Estimated code review effort: 2 (Simple) | ~5 minutes Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pkg/monitor/nvidia/v0/spec.go`:
- Line 81: Clamp the uint64 device count before converting it to int in both
DeviceNum implementations: update pkg/monitor/nvidia/v0/spec.go lines 81-81 and
pkg/monitor/nvidia/v1/spec.go lines 95-95 to apply min against
uint64(maxDevices), then narrow the result to int.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 39b388cc-b553-4322-863c-f301c0ae1ade
📒 Files selected for processing (4)
pkg/monitor/nvidia/v0/spec.gopkg/monitor/nvidia/v0/spec_test.gopkg/monitor/nvidia/v1/spec.gopkg/monitor/nvidia/v1/spec_test.go
DeviceNum() returns int(s.sr.num) — a raw read from the shared-memory region. collectContainerMetrics loops over DeviceNum() and calls DeviceUUID(i) and DeviceMemoryLimit(i), which index into [16]uuid and [16]uint64 arrays. A corrupt sr.num field (version mismatch or partial write) larger than 16 causes an out-of-bounds panic on every Prometheus scrape, silencing all GPU metrics on that node. Apply the same min(_, maxDevices) clamp already used in the setter loops and activeProcs() to DeviceNum() in both v0 and v1. Signed-off-by: Nakshatra Sharma <nakshatra.sharma3012@gmail.com>
da80db8 to
b2cb1a2
Compare
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
Codecov Report✅ All modified and coverable lines are covered by tests.
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 4 files with indirect coverage changes 🚀 New features to boost your workflow:
|
|
@Nakshatra480 just add ai disclosure. you can take ref from contributing.md |
Done |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: archlitchi, Nakshatra480 The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Summary
DeviceNum()inv0/spec.goandv1/spec.goreturnsint(s.sr.num), anunclamped raw read from the shared-memory region.
collectContainerMetricsincmd/vGPUmonitor/metrics.goloopsfor i := range c.Info.DeviceNum()andcalls
DeviceUUID(i)andDeviceMemoryLimit(i), which index into[16]uuidand
[16]uint64arrays respectively. Whensr.numexceeds 16 due to aversion mismatch or a partial libvgpu write, every Prometheus scrape causes an
index out of range panic that crashes vGPUmonitor and silences all GPU metrics
on that node.
pkg/monitor/nvidia/v0/spec.go:DeviceNum()now returnsmin(int(s.sr.num), maxDevices)pkg/monitor/nvidia/v1/spec.go: samepkg/monitor/nvidia/v0/spec_test.go: added"num larger than maxDevices is clamped"toTestSpec_DeviceNumpkg/monitor/nvidia/v1/spec_test.go: added the same toTest_DeviceNum; also tightened existing table entries to inline formgo test ./pkg/monitor/nvidia/... -count=1This applies the same
min(_, maxDevices)guard already used inactiveProcs()(via
#2282) and the setter loops (via#2362) to the getter that drives thescrape loop.
Which issue(s) this PR fixes:
Part of #2126 (LFX observability hardening - vGPU monitor stability)
Special notes for your reviewer:
DeviceUUID(i)andDeviceMemoryLimit(i)are called immediately after theDeviceNum()loop bound, both access fixed-size[16]arrays soi >= 16is an out-of-bounds panic, not a silent wrong value.
Does this PR introduce a user-facing change?
Yes vGPUmonitor no longer panics when
sr.numexceeds 16; per-device GPUmetrics continue to be reported correctly.
Summary by CodeRabbit
Bug Fixes
Tests
AI Disclosure:
AI assistance was used for code inspection and draft formatting.