fix(scheduler): skip devices with malformed MIG UUID in getNodesUsage - #1
Closed
Wangmin362 wants to merge 1 commit into
Closed
fix(scheduler): skip devices with malformed MIG UUID in getNodesUsage#1Wangmin362 wants to merge 1 commit into
Wangmin362 wants to merge 1 commit into
Conversation
getNodesUsage discarded the error from ExtractMigTemplatesFromUUID and used the resulting index without checking it, so a pod carrying a malformed or corrupt MIG UUID annotation panicked with "index out of range". getNodesUsage runs in the background register loop (a goroutine with no recover), so the panic killed the whole scheduler process and put it into CrashLoopBackOff. Handle it in three places: - skip the device when ExtractMigTemplatesFromUUID returns an error; - bound-check the instance index before indexing UsageList; - bound-check the template index inside PlatternMIG so an out-of-range index returns instead of indexing templates[idx]. Signed-off-by: wangmin <wangmin@riseunion.io>
Wangmin362
force-pushed
the
fix/mig-uuid-panic-getnodesusage
branch
from
July 14, 2026 03:52
84df1be to
0d7fa93
Compare
Owner
Author
|
Fixed upstream by Project-HAMi#2088 (bound-check MIG template/instance index parsed from UUID annotation). Closing. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
/kind bug
What this PR does / why we need it:
getNodesUsagediscards the error returned byExtractMigTemplatesFromUUIDand uses the returned indices without checking them. A pod carrying a malformed or corrupt MIG UUID annotation therefore panics withindex out of range:GPU-MIG33[999](no-inside the brackets):ExtractMigTemplatesFromUUIDreturns an error and-1indices; the discarded error lets-1reachPlatternMIG(templates[-1]) andMigUsage.UsageList[Instance].GPU-MIG33[5-0](well-formed but the template index does not exist on the node):ExtractMigTemplatesFromUUIDreturns(5, 0, nil)— no error — andPlatternMIGthen indexestemplates[5]out of range.getNodesUsageruns in the background register loop started bymain.start— a goroutine with no HTTP recover — so any of these panics kills the whole scheduler process and puts it into CrashLoopBackOff. Any pod with such an annotation can repeatedly crash the scheduler.This PR handles it in three places:
ExtractMigTemplatesFromUUIDreturns an error (mirrors the existing handling inpkg/device-plugin/nvidiadevice/nvinternal/plugin/util.go);MigUsage.UsageList;PlatternMIGso an out-of-range index returns instead of indexingtemplates[idx](this also protects the otherPlatternMIGcallers).Which issue(s) this PR fixes:
NONE
Special notes for your reviewer:
Real-machine before/after verification in an isolated single-node kind cluster (its own apiserver, zero interaction with any shared/production cluster), using scheduler images built from this branch:
hami.io/vgpu-devices-allocated: "GPU-MIG33[999],NVIDIA,4096,0:;"assigned to a MIG node crashes the scheduler within ~8s (RESTARTS 0->1, CrashLoopBackOff), with the panic stackindex out of range [-1]atPlatternMIG <- getNodesUsage <- register <- RegisterFromNodeAnnotations, created bymain.start.GPU-MIG33[999](malformed) andGPU-MIG33[5-0](out-of-range template index) pods are processed every register tick and skipped with a logged error; the scheduler stays up (RESTARTS 0) across multiple ticks.Unit regression test added (
Test_getNodesUsage_MalformedMIGUUID, three sub-cases: malformed UUID, out-of-range instance index, out-of-range template index): each panics before the corresponding guard and passes after; fullpkg/schedulerandpkg/devicesuites pass.AI assistance: the analysis is based on my own testing; the code was drafted with AI (Claude Code) assistance and reviewed and verified by me.
Does this PR introduce a user-facing change?:
Yes - the scheduler no longer crashes when a pod carries a malformed or out-of-range MIG UUID annotation; the offending device is skipped instead.