fix(nvidia): stamp flattened UsageList index in MIG UUID suffix by Eshiv-Pandey · Pull Request #2513 · Project-HAMi/HAMi

Eshiv-Pandey · 2026-08-09T14:13:41Z

What this PR does

AddResourceUsage records which MIG slot a container got by appending a [templateIdx-position] suffix to the device UUID. The two allocation branches disagreed on what position means:

The fresh-template branch (when the geometry is reset) reserved the slot at the flattened UsageList offset (usageListIdx) but stamped the geometry item index (idx) into the UUID.
The reuse branch stamps the flattened offset.

ExtractMigTemplatesFromUUID, and therefore getNodesUsage, reads thesuffix as a flattened offset into UsageList. When an earlier item in the geometry has Count > 1, the geometry index and the flattened offset diverge, so getNodesUsage marks a different slot in use than the one that was actually reserved. The allocated slot appears free while an unallocated one appears taken, which can lead to double-booking a MIG instance.

The fix

Stamp usageListIdx in the fresh-template branch so both branches and the reader agree on the meaning of the suffix.

Why it was not caught earlier

The shipped A100-40GB geometry is 1g.5gb x1, 2g.10gb x3, where the geometry index and flattened offset of the first fitting slot coincide, so the bug is masked. It surfaces on geometries where an earlier item has Count > 1, such as the 2g.10gb x3, 1g.5gb x1 example in docs/develop/dynamic-mig.md.

Testing

Added TestAddResourceUsage_MigFreshTemplateStampsFlattenedIndex, which builds a 1g.5gb x2, 2g.10gb x1 geometry (flattened UsageList index 2 = geometry index 1) and asserts the stamped suffix round-trips through ExtractMigTemplatesFromUUID to the slot actually reserved. The test fails without the fix (stamps 1, expects 2) and passes with it.

Also corrected the existing TestDevices_AddResourceUsage: its Count > 1 case pinned the buggy suffix (dev-3[0-1]) while its assertion helper silently re-derived the flattened offset, masking the mismatch. It now expects the flattened suffix (dev-3[0-2]) and reads the UsageList index directly. Full unit suite run with -race.

AI assistance disclosure

This change was prepared with AI assistance (Claude Opus 4.8). All code and tests were reviewed by me.

Summary by CodeRabbit

Bug Fixes

Corrected NVIDIA MIG allocation identifiers to consistently match the reserved MIG slot.
Improved allocation reliability for templates containing repeated geometry configurations.
Updated MIG usage tracking so allocated slots are represented accurately.

Tests

Added coverage for fresh-template allocations.
Added validation for repeated geometry configurations and correct slot identification.

hami-robot · 2026-08-09T14:13:45Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Eshiv-Pandey
Once this PR has been reviewed and has the lgtm label, please assign shouren for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

coderabbitai · 2026-08-09T14:14:02Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 47f1025f-5600-4c1a-adc1-9d06ce8990aa

📥 Commits

Reviewing files that changed from the base of the PR and between 757c440 and 3579148.

📒 Files selected for processing (2)

pkg/device/nvidia/device.go
pkg/device/nvidia/device_test.go

🚧 Files skipped from review as they are similar to previous changes (2)

pkg/device/nvidia/device.go
pkg/device/nvidia/device_test.go

📝 Walkthrough

Walkthrough

NVIDIA MIG allocation now records the flattened UsageList slot in generated UUID suffixes. Tests validate flattened indices and cover repeated geometry counts, usage tracking, memory recording, and UUID parsing.

Changes

MIG UUID indexing

Layer / File(s)	Summary
Flattened index allocation and validation `pkg/device/nvidia/device.go`, `pkg/device/nvidia/device_test.go`	MIG allocation uses the reserved flattened `UsageList` index in the UUID suffix. Tests validate expected indices and repeated geometry counts.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

Project-HAMi/HAMi#2088: Both changes address NVIDIA MIG UUID index validation and usage mapping.
Project-HAMi/HAMi#2245: Both changes modify NVIDIA MIG allocation logic and its tests.

Suggested reviewers: archlitchi

Poem

A rabbit checks the MIG slot’s place,
The flattened index sets the pace.
UUIDs now identify the right slot,
Tests confirm each reserved spot.
Hop, hop—allocation is correct!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main NVIDIA MIG fix: using the flattened UsageList index in the UUID suffix.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

codecov · 2026-08-09T14:20:40Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

Flag	Coverage Δ
unittests	`64.24% <100.00%> (+0.03%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
pkg/device/nvidia/device.go	`97.71% <100.00%> (ø)`

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

AddResourceUsage stamps the MIG slot position into the container UUID suffix, but the two allocation branches disagreed on what that position means. The fresh-template branch stamped the geometry item index while reserving the slot at the flattened UsageList offset, whereas the reuse branch stamped the flattened offset. ExtractMigTemplatesFromUUID, and therefore getNodesUsage, reads the suffix as a flattened offset into UsageList. When an earlier item in the geometry has Count > 1 the two diverge, so getNodesUsage marks a different slot in use than the one actually reserved: the allocated slot appears free while an unallocated one appears taken. Stamp usageListIdx in the fresh-template branch so both branches and the reader agree. Also correct TestDevices_AddResourceUsage, whose Count > 1 case pinned the buggy suffix (dev-3[0-1]) while its assertion helper silently re-derived the flattened offset, masking the mismatch. It now expects the flattened suffix (dev-3[0-2]) and reads the UsageList index directly. Adds TestAddResourceUsage_MigFreshTemplateStampsFlattenedIndex as a focused regression. Signed-off-by: Eshiv Pandey <eshivpandey18@gmail.com>

FouoF · 2026-08-10T04:57:59Z

#2378 have refactored dynamic MIG, please check is this bug still exists on it.

mesutoezdil · 2026-08-10T08:12:33Z

+						// and getNodesUsage read back), not the geometry item index.
 						if !strings.Contains(ctr.UUID, "[") {
-							ctr.UUID = ctr.UUID + "[" + fmt.Sprint(tidx) + "-" + fmt.Sprint(idx) + "]"
+							ctr.UUID = ctr.UUID + "[" + fmt.Sprint(tidx) + "-" + fmt.Sprint(usageListIdx) + "]"


usagelistidx and tidx are gone in master now. #2378 rewrote this function. pls check vs new code.

i just checked the bug is gone now after the refactor. closing this. thanks for the review!

hami-robot Bot added the dco-signoff: yes label Aug 9, 2026

hami-robot Bot requested review from FouoF and lengrongfu August 9, 2026 14:13

github-actions Bot added the kind/bug Something isn't working label Aug 9, 2026

hami-robot Bot added the size/M label Aug 9, 2026

coderabbitai Bot requested a review from archlitchi August 9, 2026 14:14

Eshiv-Pandey temporarily deployed to nvidia August 9, 2026 14:26 — with GitHub Actions Inactive

Eshiv-Pandey force-pushed the fix/mig-uuid-flattened-index branch from 757c440 to 3579148 Compare August 9, 2026 14:43

Eshiv-Pandey temporarily deployed to nvidia August 9, 2026 14:55 — with GitHub Actions Inactive

mesutoezdil reviewed Aug 10, 2026

View reviewed changes

Eshiv-Pandey closed this Aug 10, 2026

Navigation Menu

Uh oh!

fix(nvidia): stamp flattened UsageList index in MIG UUID suffix - #2513

fix(nvidia): stamp flattened UsageList index in MIG UUID suffix#2513
Eshiv-Pandey wants to merge 1 commit into
Project-HAMi:masterfrom
Eshiv-Pandey:fix/mig-uuid-flattened-index

Eshiv-Pandey commented Aug 9, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

hami-robot Bot commented Aug 9, 2026

Uh oh!

coderabbitai Bot commented Aug 9, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

codecov Bot commented Aug 9, 2026 •

edited

Loading

Uh oh!

FouoF commented Aug 10, 2026

Uh oh!

mesutoezdil Aug 10, 2026 •

edited

Loading

Uh oh!

Eshiv-Pandey Aug 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Eshiv-Pandey commented Aug 9, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does

The fix

Why it was not caught earlier

Testing

AI assistance disclosure

Summary by CodeRabbit

Bug Fixes

Tests

Uh oh!

hami-robot Bot commented Aug 9, 2026

Uh oh!

coderabbitai Bot commented Aug 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

codecov Bot commented Aug 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

FouoF commented Aug 10, 2026

Uh oh!

mesutoezdil Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Eshiv-Pandey Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Eshiv-Pandey commented Aug 9, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Aug 9, 2026 •

edited

Loading

codecov Bot commented Aug 9, 2026 •

edited

Loading

mesutoezdil Aug 10, 2026 •

edited

Loading