fix(nvidia): treat gpumem-percentage of 0 as unset - #11
Closed
Wangmin362 wants to merge 1 commit into
Closed
Conversation
A gpumem-percentage of 0 made the scheduler book 0 memory, so the pod fit any card and the device plugin injected CUDA_DEVICE_MEMORY_LIMIT=0m, which hami-core reads as no limit. Handle it like nvidia.com/gpumem: 0: fall back to defaultMemory, or to the whole card when no default is configured. Signed-off-by: wangmin <wangmin@riseunion.io>
Wangmin362
force-pushed
the
fix/gpumem-percentage-zero-v2
branch
from
August 4, 2026 06:54
b212c15 to
837e4b1
Compare
Wangmin362
changed the title
Owner
Author
|
内容已合入社区 PR Project-HAMi#2156 （head |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
/kind bug
What this PR does / why we need it:
nvidia.com/gpumem-percentage: 0is accepted today, so the scheduler computesmemreq = Totalmem * 0 / 100 = 0: the pod fits any card however full it is, is booked with 0 memory, and the device plugin injectsCUDA_DEVICE_MEMORY_LIMIT_0=0m, which hami-core reads as "no limit". The container can then use the whole card while the scheduler still thinks it is free.nvidia.com/gpumem: 0is fine because it leaves the percentage unset and falls through to the defaultMemory / whole-card branch below. This makes a non-positive percentage take the same path.Which issue(s) this PR fixes:
NONE
Special notes for your reviewer:
Clamping to 100 instead would also stop the 0-memory booking, but it skips that defaultMemory branch, and unlike the 101 sentinel, 100 is not excluded in
ComputeScore, so scoring would count a whole card.On a V100-32GB node, a pod with
gpumem-percentage: 0got0mand allocated 8 GiB while still booked as 0; a second pod asking forgpumem: 30000on the same card was then admitted and died withcuMemoryAllocate failed res=2. I also ran a before/after A/B on an A100-80GB node with two shadow schedulers differing only in this change: an empty card is now booked as 81920 instead of 0, with 80000/81920 already booked the same request goes Pending withCardInsufficientMemory, and withdefaultMemory: 512it is booked as 512 and the container gets512minstead of0m.The tests added for a zero percentage fail on master and pass here. This PR was written primarily by Claude Code, and I reviewed and tested it.
Does this PR introduce a user-facing change?:
A
nvidia.com/gpumem-percentageof 0 is now treated as unset: the container gets the whole card, ordefaultMemorywhen one is configured, instead of running with no GPU memory limit.