fix(nvidia): treat gpumem-percentage of 0 as unset - #2156
Conversation
📝 WalkthroughWalkthroughNVIDIA resource request generation now preserves the unset sentinel for zero and negative ChangesNVIDIA memory percentage fallback
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests.
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 1 file with indirect coverage changes 🚀 New features to boost your workflow:
|
|
Could we handle |
A gpumem-percentage of 0 made the scheduler book 0 memory, so the pod fit any card and the device plugin injected CUDA_DEVICE_MEMORY_LIMIT=0m, which hami-core reads as no limit. Handle it like nvidia.com/gpumem: 0: fall back to defaultMemory, or to the whole card when no default is configured. Signed-off-by: wangmin <wangmin@riseunion.io>
7a5be78 to
ab79651
Compare
done, handled entirely in GenerateResourceRequests now. thank you |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: archlitchi, Wangmin362 The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
What type of PR is this?
/kind bug
What this PR does / why we need it:
nvidia.com/gpumem-percentage: 0is accepted today, so the scheduler computesmemreq = Totalmem * 0 / 100 = 0: the pod fits any card however full it is, is booked with 0 memory, and the device plugin injectsCUDA_DEVICE_MEMORY_LIMIT_0=0m, which hami-core reads as "no limit". The container can then use the whole card while the scheduler still thinks it is free.nvidia.com/gpumem: 0is fine because it leaves the percentage unset and falls through to the defaultMemory / whole-card branch below. This makes a non-positive percentage take the same path.Which issue(s) this PR fixes:
NONE
Special notes for your reviewer:
Clamping to 100 instead would also stop the 0-memory booking, but it skips that defaultMemory branch, and unlike the 101 sentinel, 100 is not excluded in
ComputeScore, so scoring would count a whole card.On a V100-32GB node, a pod with
gpumem-percentage: 0got0mand allocated 8 GiB while still booked as 0; a second pod asking forgpumem: 30000on the same card was then admitted and died withcuMemoryAllocate failed res=2. I also ran a before/after A/B on an A100-80GB node with two shadow schedulers differing only in this change: an empty card is now booked as 81920 instead of 0, with 80000/81920 already booked the same request goes Pending withCardInsufficientMemory, and withdefaultMemory: 512it is booked as 512 and the container gets512minstead of0m.The tests added for a zero percentage fail on master and pass here. This PR was written primarily by Claude Code, and I reviewed and tested it.
Does this PR introduce a user-facing change?:
A
nvidia.com/gpumem-percentageof 0 is now treated as unset: the container gets the whole card, ordefaultMemorywhen one is configured, instead of running with no GPU memory limit.Summary by CodeRabbit