Skip to content

fix(scheduler): skip devices with malformed MIG UUID in getNodesUsage - #1

Closed
Wangmin362 wants to merge 1 commit into
r3m33-oss-basefrom
fix/mig-uuid-panic-getnodesusage
Closed

fix(scheduler): skip devices with malformed MIG UUID in getNodesUsage#1
Wangmin362 wants to merge 1 commit into
r3m33-oss-basefrom
fix/mig-uuid-panic-getnodesusage

Conversation

@Wangmin362

@Wangmin362 Wangmin362 commented Jul 14, 2026

Copy link
Copy Markdown
Owner

What type of PR is this?

/kind bug

What this PR does / why we need it:

getNodesUsage discards the error returned by ExtractMigTemplatesFromUUID and uses the returned indices without checking them. A pod carrying a malformed or corrupt MIG UUID annotation therefore panics with index out of range:

  • GPU-MIG33[999] (no - inside the brackets): ExtractMigTemplatesFromUUID returns an error and -1 indices; the discarded error lets -1 reach PlatternMIG (templates[-1]) and MigUsage.UsageList[Instance].
  • GPU-MIG33[5-0] (well-formed but the template index does not exist on the node): ExtractMigTemplatesFromUUID returns (5, 0, nil) — no error — and PlatternMIG then indexes templates[5] out of range.

getNodesUsage runs in the background register loop started by main.start — a goroutine with no HTTP recover — so any of these panics kills the whole scheduler process and puts it into CrashLoopBackOff. Any pod with such an annotation can repeatedly crash the scheduler.

This PR handles it in three places:

  • skip the device when ExtractMigTemplatesFromUUID returns an error (mirrors the existing handling in pkg/device-plugin/nvidiadevice/nvinternal/plugin/util.go);
  • bound-check the instance index before indexing MigUsage.UsageList;
  • bound-check the template index inside PlatternMIG so an out-of-range index returns instead of indexing templates[idx] (this also protects the other PlatternMIG callers).

Which issue(s) this PR fixes:
NONE

Special notes for your reviewer:

Real-machine before/after verification in an isolated single-node kind cluster (its own apiserver, zero interaction with any shared/production cluster), using scheduler images built from this branch:

  • Before (unfixed): applying a pod with hami.io/vgpu-devices-allocated: "GPU-MIG33[999],NVIDIA,4096,0:;" assigned to a MIG node crashes the scheduler within ~8s (RESTARTS 0->1, CrashLoopBackOff), with the panic stack index out of range [-1] at PlatternMIG <- getNodesUsage <- register <- RegisterFromNodeAnnotations, created by main.start.
  • After (fixed): both the GPU-MIG33[999] (malformed) and GPU-MIG33[5-0] (out-of-range template index) pods are processed every register tick and skipped with a logged error; the scheduler stays up (RESTARTS 0) across multiple ticks.

Unit regression test added (Test_getNodesUsage_MalformedMIGUUID, three sub-cases: malformed UUID, out-of-range instance index, out-of-range template index): each panics before the corresponding guard and passes after; full pkg/scheduler and pkg/device suites pass.

AI assistance: the analysis is based on my own testing; the code was drafted with AI (Claude Code) assistance and reviewed and verified by me.

Does this PR introduce a user-facing change?:

Yes - the scheduler no longer crashes when a pod carries a malformed or out-of-range MIG UUID annotation; the offending device is skipped instead.

getNodesUsage discarded the error from ExtractMigTemplatesFromUUID and
used the resulting index without checking it, so a pod carrying a
malformed or corrupt MIG UUID annotation panicked with "index out of
range". getNodesUsage runs in the background register loop (a goroutine
with no recover), so the panic killed the whole scheduler process and
put it into CrashLoopBackOff.

Handle it in three places:
- skip the device when ExtractMigTemplatesFromUUID returns an error;
- bound-check the instance index before indexing UsageList;
- bound-check the template index inside PlatternMIG so an out-of-range
  index returns instead of indexing templates[idx].

Signed-off-by: wangmin <wangmin@riseunion.io>
@Wangmin362
Wangmin362 force-pushed the fix/mig-uuid-panic-getnodesusage branch from 84df1be to 0d7fa93 Compare July 14, 2026 03:52
@Wangmin362

Copy link
Copy Markdown
Owner Author

Fixed upstream by Project-HAMi#2088 (bound-check MIG template/instance index parsed from UUID annotation). Closing.

@Wangmin362 Wangmin362 closed this Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant