fix(nvidia): reject gpumem-percentage of 0 - #10
Closed
Wangmin362 wants to merge 3 commits into
Closed
Conversation
Wangmin362
force-pushed
the
fix/gpumem-percentage-zero
branch
2 times, most recently
from
July 27, 2026 13:59
efaadd9 to
5db8987
Compare
…AMi#2105) * fix(scheduler): scope per-type device list in fitInDevices When a single container requested more than one device type, the devs accumulator was declared once outside the per-type loop and never reset, so the second type's allocation entry inherited the first type's devices. Append each requested type's own devices (tmpDevs[k.Type]) directly to its devinput entry so a type records only its own devices. Signed-off-by: imantaba <itn.taba@gmail.com> * test(scheduler): cover multi-device-type partition in fitInDevices Signed-off-by: imantaba <itn.taba@gmail.com> --------- Signed-off-by: imantaba <itn.taba@gmail.com>
* fix webhook to deny privileged containers Signed-off-by: Jay2006sawant <jay242902@gmail.com> * check privileged init containers before scheduler bypass Signed-off-by: Jay2006sawant <jay242902@gmail.com> * deny privileged containers only when gpu resources are requested Signed-off-by: Jay2006sawant <jay242902@gmail.com> --------- Signed-off-by: Jay2006sawant <jay242902@gmail.com>
A container asking for nvidia.com/gpumem-percentage: 0 passes validation, so the scheduler computes memreq = Totalmem * 0 / 100 = 0. The pod fits any card regardless of how full it is, is booked with 0 memory, and the device plugin injects CUDA_DEVICE_MEMORY_LIMIT_0=0m, which HAMi-core reads as no limit. The container can then use the whole card while the scheduler still counts the card as free and keeps placing other pods on it, which OOM for real. Reject 0 at admission and clamp it to 100 in GenerateResourceRequests for pods that skip the webhook, same as the existing out-of-range handling. Signed-off-by: wangmin <wangmin@riseunion.io>
Wangmin362
force-pushed
the
fix/gpumem-percentage-zero
branch
from
July 28, 2026 06:12
5db8987 to
7a5be78
Compare
Owner
Author
|
Superseded by the upstream PR: Project-HAMi#2156 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
/kind bug
What this PR does / why we need it:
nvidia.com/gpumem-percentage: 0is accepted today. The scheduler then computesmemreq = Totalmem * 0 / 100 = 0, so the pod fits any card no matter how full it is, gets booked with 0 memory, and the device plugin injectsCUDA_DEVICE_MEMORY_LIMIT_0=0m— which HAMi-core treats as "no limit" (do_init_device_memory_limitsleaves the limit at 0 andoom_checkreturns early). The container can use the whole card while the scheduler still thinks it is free and keeps placing other pods on it; those pods then hit a real driver OOM inside their own quota.This rejects 0 at admission and clamps it to 100 in
GenerateResourceRequests, the same way out-of-range values are already handled. The clamp is not just a fallback for pods that skip the webhook: the webhook only walkspod.Spec.Containers, so an init container is never validated, whileGenerateResourceRequestsstill runs for it — there the clamp is the only guard.Which issue(s) this PR fixes:
NONE
Special notes for your reviewer:
Verified on two clusters, one pod per case in a throwaway namespace.
On a Tesla V100-PCIE-32GB node (HAMi 2.9.0 + hami-core):
gpumem-percentage: 0→ allocation annotationGPU-00552014-...,NVIDIA,0,0, container envCUDA_DEVICE_MEMORY_LIMIT_0=0m, and an 8 GiB torch allocation succeeded (hostnvidia-smishowed 8501 MiB in use, booked as 0). hami-core itself logsinvalid device memory limit CUDA_DEVICE_MEMORY_LIMIT_0=0mand then runs with no limit.gpumem: 30000on the same UUID was still admitted, and its allocation died withcuMemoryAllocate failed res=2/torch.OutOfMemoryError— its own view said 29 GiB were free.gpumem: 2000on a 1080 Ti → env2000m, i.e. a 2000 MiB cap: an oversized 4 GiB allocation is rejected as expected, so the limiter itself works.I also ran a before/after A/B with two shadow schedulers built from this branch — identical except for these two lines — each with its own namespace, scheduler name and no webhook, filtering on a single real GPU node. Same pod spec every time:
gpumem-percentage: 0...,NVIDIA,0,0...,NVIDIA,32768,001 nodes CardInsufficientMemory/webhookAdmissionReview carrying percentage 0allowed: trueallowed: false,must be an integer between 1 and 100The new tests fail on master and pass here.
This PR was written primarily by Claude Code, and I reviewed and tested it.
Does this PR introduce a user-facing change?:
Pods requesting
nvidia.com/gpumem-percentage: 0are now rejected at admission instead of silently running with no GPU memory limit.