fix(vGPUmonitor): skip GPU utilization gracefully when not supported - #2619
fix(vGPUmonitor): skip GPU utilization gracefully when not supported#2619Nakshatra480 wants to merge 1 commit into
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: Nakshatra480 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughThe GPU monitor treats ChangesGPU utilization handling
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@cmd/vGPUmonitor/metrics_gpu_test.go`:
- Around line 83-84: Update the test around sendLegacyMetric to isolate the
package-global legacyHostGPUUtilizationdesc state before asserting the metric
count, or ensure legacy emission is gated by LegacyMetrics. Preserve the
expectation of exactly one metric when legacy metrics are disabled and prevent
other tests’ descriptor initialization from affecting this collector.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: f309f563-26cd-4b0c-865b-7719c7744e87
📒 Files selected for processing (2)
cmd/vGPUmonitor/metrics.gocmd/vGPUmonitor/metrics_gpu_test.go
collectGPUMemoryMetrics already returns nil on nvml.ERROR_NOT_SUPPORTED (unified memory devices, Jetson, MIG configurations). collectGPUUtilizationMetrics treated the same return code as a hard error, causing collectGPUDeviceMetrics to surface a misleading failure even though memory metrics had already been emitted to the channel. Mirror the existing memory guard so a device that does not support GetUtilizationRates still produces its memory metrics without a spurious error in the logs. Signed-off-by: Nakshatra Sharma <nakshatra.sharma3012@gmail.com>
fd85797 to
9f01fff
Compare
Codecov Report✅ All modified and coverable lines are covered by tests.
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 2 files with indirect coverage changes 🚀 New features to boost your workflow:
|
|
This is being closed because it does not comply with the contribution guidelines. |
|
Thanks for your feedback. I will review the contribution guidelines and ensure that my future submissions follow them. If you can, please let me know what was missing or not compliant in this contribution so I can improve. |
Summary
collectGPUMemoryMetricsalready returnsnilonnvml.ERROR_NOT_SUPPORTED(unified memory architectures, Jetson, certain MIG configurations).collectGPUUtilizationMetricstreated the same NVML return code as a hard error.collectGPUDeviceMetricsreturned an error after memory metrics had already been emitted to the channel, producing a misleading "Failed to collect metrics for GPU device N" log even though memory data was fine.ERROR_NOT_SUPPORTEDguard thatcollectGPUMemoryMetricsalready uses.Files changed:
cmd/vGPUmonitor/metrics.go: four-line guard matching the existing memory patternTested:
go test ./cmd/vGPUmonitor/... -count=1passes. The NVML path requires hardware; the guard mirrors an already-tested pattern fromcollectGPUMemoryMetrics.Special notes for your reviewer:
The only change is mirroring the
if nvret == nvml.ERROR_NOT_SUPPORTED { return nil }guard from the memory function. No logic change for the success path.Does this PR introduce a user-facing change?
Yes devices that previously triggered a spurious error log now silently skip utilization (with a
klog.V(3)message) and continue emitting memory metrics.Summary by CodeRabbit