Skip to content

docs(mig): document reservation-first Dynamic MIG #794

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
saiyam1814 wants to merge 3 commits into
base: master
Choose a base branch
from
Open

docs(mig): document reservation-first Dynamic MIG #794

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
901d277
docs(mig): document reservation-first Dynamic MIG
saiyam1814 Aug 19, 2026
e80b40d
docs(mig): replace stale README anchor with the installation guide
saiyam1814 Aug 21, 2026
1d70630
docs(mig): pin chart links to the v2.10.0 release tag
saiyam1814 Aug 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
320 changes: 192 additions & 128 deletions docs/developers/dynamic-mig.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,157 +1,221 @@
---
title: NVIDIA GPU MPS and MIG dynamic slice plugin
title: NVIDIA dynamic MIG implementation
sidebar_label: Dynamic MIG Implementation
---

## Special Thanks

This feature will not be implemented without the help of @sailorvii.

## Introduction

The NVIDIA GPU built-in sharing method includes: time-slice, MPS and MIG. The context switch for time slice sharing would waste some time, so MPS and MIG are preferred. The GPU MIG profile is variable, the user could acquire the MIG device in the profile definition, but current implementation only defines the dedicated profile before the user requirement. That limits the usage of MIG. The goal is to develop an automatic slice plugin and create the slice when the user requires it. For the scheduling method, node-level binpack and spread will be supported. Referring to the binpack plugin, the scheduler considers CPU, Mem, GPU memory and other user-defined resources. HAMi is done by using [hami-core](https://github.com/Project-HAMi/HAMi-core), which is a cuda-hacking library. But MIG is also widely used across the world. A unified API for dynamic-mig and hami-core is needed.
HAMi v2.10 replaces predefined `knownMigGeometries` and whole-GPU template switching with a reservation-first Dynamic MIG architecture. Hardware capability originates on the node, the scheduler reserves an exact MIG placement before binding a Pod, and the NVIDIA device plugin realizes and reclaims the corresponding GPU instance (GI) and compute instance (CI).

The design keeps four responsibilities separate:

| Authority | Responsibility |
| --- | --- |
| NVML on the node | Reports profile memory, compute capacity, slice count, and legal placements |
| HAMi scheduler | Selects the physical GPU, profile, and placement and accounts for the reservation |
| NVIDIA device plugin | Serializes hardware mutation and creates, verifies, adopts, and destroys GI/CI instances |
| Pod annotation | Persists the allocation identity shared by the scheduler and device plugin |

This model creates hardware-isolated MIG devices per Pod without making the scheduler mutate GPU hardware or making the device plugin choose a different placement at runtime.

## Design goals

- Treat the node and NVML as the source of truth for hardware capability.
- Reserve topology, not only aggregate MIG slice count.
- Realize exactly the profile and placement accepted by the scheduler.
- Use stable workload metadata for reconciliation and restart recovery.
- Reclaim one Pod's instance without reconfiguring instances owned by other Pods.
- Keep the shared scheduler/device-plugin contract compact as GPU density grows.

## Architecture

```text
Kubernetes control plane

+----------------+ Node capability +----------------+
| Device Plugin | --------------------------> | HAMi Scheduler |
| | | |
| NVML discovery | Pod reservation | Placement |
| GI/CI manager | <-------------------------- | policy |
| Reconciler | | Capacity model |
+-------+--------+ +--------+-------+
| |
| exact GI/CI realization | bind
v v
+----------------+ +----------------+
| NVIDIA GPU | | Workload Pod |
| MIG topology | | Allocation |
| and instances | | annotation |
+----------------+ +----------------+
```

## Targets
The device plugin publishes capability in the NVIDIA node registration annotation. The scheduler reconstructs topology occupancy from active Pod reservations. The Pod annotation is the durable handoff between the scheduling and runtime phases.

- CPU, Mem, and GPU combined schedule
- GPU dynamic slice: HAMi-core and MIG
- Support node-level binpack and spread by GPU memory, CPU and Mem
- A unified vGPU Pool different virtualization techniques
- Tasks can choose to use MIG, use HAMi-core, or use both.
## Capability discovery contract

### Config maps
### Policy comes from `migProfileAllowlist`

- hami-scheduler-device-configMap This configmap defines the plugin configurations including resourceName, and MIG geometries, and node-level configurations.
The scheduler device configuration names the profiles that cluster policy allows for each model:

```yaml
apiVersion: v1
data:
device-config.yaml: |
nvidia:
resourceCountName: nvidia.com/gpu
resourceMemoryName: nvidia.com/gpumem
resourceCoreName: nvidia.com/gpucores
knownMigGeometries:
- models: [ "A30" ]
allowedGeometries:
-
- name: 1g.6gb
memory: 6144
count: 4
-
- name: 2g.12gb
memory: 12288
count: 2
-
- name: 4g.24gb
memory: 24576
count: 1
- models: [ "A100-SXM4-40GB", "A100-40GB-PCIe", "A100-PCIE-40GB" ]
allowedGeometries:
-
- name: 1g.5gb
memory: 5120
count: 7
-
- name: 2g.10gb
memory: 10240
count: 3
- name: 1g.5gb
memory: 5120
count: 1
-
- name: 3g.20gb
memory: 20480
count: 2
-
- name: 7g.40gb
memory: 40960
count: 1
- models: [ "A100-SXM4-80GB", "A100-80GB-PCIe", "A100-PCIE-80GB"]
allowedGeometries:
-
- name: 1g.10gb
memory: 10240
count: 7
-
- name: 2g.20gb
memory: 20480
count: 3
- name: 1g.10gb
memory: 10240
count: 1
-
- name: 3g.40gb
memory: 40960
count: 2
-
- name: 7g.79gb
memory: 80896
count: 1
nodeconfig:
- name: nodeA
operatingmode: hami-core
- name: nodeB
operatingmode: mig
nvidia:
migProfileAllowlist:
- models: ["A100-SXM4-40GB"]
profiles: ["1g.5gb", "2g.10gb", "3g.20gb", "7g.40gb"]
- models: ["RTX PRO 6000 Blackwell Server Edition"]
profiles: ["1g.24gb", "2g.48gb", "4g.96gb"]
```

## Structure
The [Chart's default configuration](https://github.com/Project-HAMi/HAMi/blob/v2.10.0/charts/hami/templates/scheduler/device-configmap.yaml) also includes A30, A100 80 GB, H100, H20, H200, and B200 mappings. The allowlist determines what the cluster permits; it does not define capacity or a complete geometry.

<img src="/img/docs/en/dynamic-mig/hami-dynamic-mig-structure.png" width="600" alt="HAMi dynamic MIG structure diagram showing vGPU Pool and Scheduler components" />
A custom `device-config.content` value or external ConfigMap replaces the Chart default. Such configurations must use `migProfileAllowlist` explicitly. Legacy `knownMigGeometries` fields are not converted automatically.

## Examples
### Capability comes from NVML

Dynamic MIG is compatible with HAMi tasks, as shown in the example below: Set `nvidia.com/gpu` and `nvidia.com/gpumem`.
For every allowlisted profile on a matching GPU, the device plugin queries NVML for profile information and possible placements. It publishes scheduler-facing fields in the per-GPU `migProfiles` array of `hami.io/node-nvidia-register`:

```yaml
apiVersion: v1
kind: Pod
metadata:
name: gpu-pod1
spec:
containers:
- name: ubuntu-container1
image: ubuntu:22.04
command: ["bash", "-c", "sleep 86400"]
resources:
limits:
nvidia.com/gpu: 2 # requesting 2 vGPUs
nvidia.com/gpumem: 8000 # Each vGPU contains 8000m device memory (Optional,Integer)
```json
{
"name": "2g.10gb",
"memoryMB": 9984,
"core": 29,
"sliceCount": 2,
"placements": [
{ "start": 0, "size": 2 },
{ "start": 2, "size": 2 },
{ "start": 4, "size": 2 }
]
}
```

A task can decide only to use `mig` or `hami-core` by setting `annotations.nvidia.com/vgpu-mode` to corresponding value, as the example below shows:

```yaml
apiVersion: v1
kind: Pod
metadata:
name: gpu-pod1
annotations:
nvidia.com/vgpu-mode: "mig"
spec:
containers:
- name: ubuntu-container1
image: ubuntu:22.04
command: ["bash", "-c", "sleep 86400"]
resources:
limits:
nvidia.com/gpu: 2 # requesting 2 vGPUs
nvidia.com/gpumem: 8000 # Each vGPU contains 8000m device memory (Optional,Integer)
| Field | Scheduler use |
| ------------ | ----------------------------------------------------------------- |
| `name` | Stable profile identity across components |
| `memoryMB` | Match `nvidia.com/gpumem` demand against actual reported capacity |
| `core` | Account for the profile's compute share |
| `sliceCount` | Order profile candidates deterministically |
| `placements` | Select a legal, non-overlapping physical slice interval |

Profile names are policy labels, but their memory, compute, and placement data are node-local facts. A profile that is not allowlisted or that NVML cannot resolve is omitted. Device-plugin-only discovery data, such as the maximum instance count, is not added to the shared wire format when placements already express schedulable capacity.

## Scheduling and reservation

For a Pod selecting `nvidia.com/vgpu-mode: "mig"`, scheduling follows this sequence:

1. Reconstruct each GPU's occupied intervals from active Pod reservations.
2. Sort discovered profile candidates by NVML-reported memory and slice count.
3. Choose the smallest profile that satisfies the container's memory request.
4. Select a deterministic legal placement that does not overlap an existing or newly accepted reservation.
5. Add the reservation to scheduler occupancy immediately, before the Pod is bound.
6. Persist the complete logical reservation in the Pod.

A placement occupies the half-open interval `[placement.start, placement.start + placement.size)`. Aggregate free slices are not sufficient if no placement reported for the requested profile fits without overlap. In that case the Pod remains Pending. The scheduler does not request a whole-GPU template change and does not move a running GI.

### Pod reservation contract

The scheduler writes a JSON array to `hami.io/vgpu-mig-allocations`, with one entry per requested MIG device:

```json
[
{
"containerIndex": 0,
"deviceIndex": 0,
"gpuUUID": "GPU-xxxxxxxx",
"profile": "2g.10gb",
"placement": { "start": 2, "size": 2 },
"migUUID": "MIG-xxxxxxxx",
"gpuInstanceID": 4,
"computeInstanceID": 0
}
]
```

## Procedures
The scheduler writes `containerIndex`, `deviceIndex`, `gpuUUID`, `profile`, and `placement`. The runtime identity fields are initially absent. After realization, the device plugin patches the same record with `migUUID`, `gpuInstanceID`, and `computeInstanceID`.

The Procedure of a vGPU task which uses dynamic-mig is shown below:
Container and device indexes disambiguate repeated allocations on the same physical GPU. Runtime identity is valid only when all three runtime fields are present; partial runtime identity fails validation. This annotation is an internal protocol and must not be generated or modified by users.

<img src="/img/docs/en/dynamic-mig/hami-dynamic-mig-procedure.png" width="800" alt="HAMi dynamic MIG procedure flowchart showing task scheduling process" />
## Runtime realization

After submitting a task, the deviceshare plugin iterates over templates defined in configMap `hami-scheduler-device` and finds the first available template to fit. You can always change the content of that configMap, and restart vc-scheduler to customize.
During kubelet `Allocate`, the device plugin resolves the entries for the container being started and performs the following operations:

If you submit the example on an empty A100-PCIE-40GB node, then it will select a GPU and choose MIG template below:
1. Reconcile tracked instances against the current active-Pod snapshot.
2. Verify that the reserved profile and placement still exist in NVML capability.
3. Acquire the per-physical-GPU mutation lock.
4. Create the GI at exactly the scheduler-selected placement and create its CI.
5. Resolve the MIG UUID, GI ID, and CI ID and expose the MIG device to the container.
6. Patch the Pod's allocation annotation with that runtime identity.

```yaml
2g.10gb: 3
1g.5gb: 1
The manager key is the physical GPU index, profile, placement start, and placement size. Repeated `Allocate` calls for the same reservation return the same managed MIG UUID. If a multi-device allocation fails partway through, instances created by that attempt are rolled back; already adopted instances are not destroyed as part of that rollback.

The device plugin never retries a different physical placement. Doing so would violate the scheduler's topology accounting and could overlap another accepted reservation.

## Reconciliation and reclaim

The device plugin periodically lists Pods assigned to its node. Pods with valid reservations form the desired allocation set if they are not deleting and have not reached Succeeded or Failed. Managed instances absent from that set are released by destroying their exact CI and GI.

Reconciliation is intentionally conservative. A complete Kubernetes list and valid allocation annotations authorize a cleanup pass. API failures, malformed records, or partial runtime identity cause destructive reconciliation to be skipped rather than allowing the plugin to infer ownership.

This convergent loop makes a placement reusable after its Pod terminates while leaving unrelated placements on the same GPU intact.

## Device plugin restart adoption

Startup recovery uses both Kubernetes allocation state and NVML activity:

1. Active Pod reservations and GPU process state identify GPUs carrying work.
2. Idle GPUs are prepared in a clean MIG-ready state; old GI/CI instances on those idle GPUs can be removed.
3. For each active v2.10 record, the plugin verifies the annotated profile, placement, MIG UUID, GI ID, and CI ID against NVML.
4. A matching live instance is adopted into the new manager's allocation maps.
5. Normal reconciliation resumes from the reconstructed ownership state.

If allocation state cannot be read reliably during startup, the plugin preserves GPUs instead of applying destructive idle-GPU cleanup. Legacy `GPU-UUID[template-slot]` identifiers cannot be adopted because they do not prove a physical placement and complete runtime identity.

## Metrics and observability

The scheduler exports realized instances through the current metric:

```text
# HELP hami_node_gpu_mig_instance_info Realized MIG instance identity and scheduler placement
# TYPE hami_node_gpu_mig_instance_info gauge
hami_node_gpu_mig_instance_info{compute_instance_id="0",device_index="0",device_uuid="GPU-xxxxxxxx",gpu_instance_id="4",mig_uuid="MIG-xxxxxxxx",node="MIG-NODE-A",placement_size="2",placement_start="2",profile="2g.10gb"} 1
```

Then start the container with 2g.10gb instances \* 2
Only reservations enriched with complete runtime identity produce this series. Its labels join the scheduler reservation to the physical GPU, placement, MIG UUID, GI ID, and CI ID. `device_uuid` plus `gpu_instance_id` can also correlate with DCGM series carrying `UUID` and `GPU_I_ID`.

`nodeGPUMigInstance` is a compatibility metric emitted only when `legacyMetrics: true`; the current Chart default is `false`. The standard scheduler endpoint is `<scheduler-ip>:31993/metrics` with the default NodePort service.

Useful operational signals include:

- device plugin discovery logs and non-empty per-GPU `migProfiles`;
- scheduler placement decisions and Pods that remain Pending under capacity or fragmentation pressure;
- the `hami.io/vgpu-mig-allocations` transition from logical reservation to runtime identity;
- GI/CI visibility through `nvidia-smi` or NVML;
- reclaim and startup-adoption logs; and
- sustained workload progress before and after a device plugin restart.

## Migration and ownership boundaries

### Legacy HAMi geometry implementation

The v2.9 and v2.10 protocols cannot safely serve MIG workloads as a mixed scheduler/device-plugin pair. A legacy scheduler publishes and consumes `migtemplate` and template/slot identifiers; v2.10 consumes `migProfiles` and requires an exact profile/placement reservation.

Initial migration therefore requires a controlled handover:

1. stop new MIG scheduling and drain legacy MIG Pods one node at a time;
2. convert the union of required legacy profiles from `knownMigGeometries` into `migProfileAllowlist`;
3. upgrade the scheduler before the node device plugins;
4. validate capability publication, realization, reclaim, and restart adoption; and
5. uncordon each node only after its lifecycle test passes.

Routine mixed-profile creation after migration does not require a whole-GPU geometry switch when a legal free placement exists. Hardware fragmentation, MIG mode changes, driver maintenance, rollback, or a layout that requires moving an active GI can still require draining or rebooting.

### NVIDIA MIG Manager

MIG Manager applies node- or GPU-level geometries, while HAMi Dynamic MIG creates and destroys GI/CI instances from per-Pod reservations. Both mutate the same hardware state and must not reconcile the same physical GPU concurrently.

GPU Operator may continue to provide the NVIDIA driver, Container Toolkit, DCGM, and other infrastructure. Before HAMi assumes mutation ownership, stop MIG Manager reconciliation and ensure a controller cannot recreate it or reapply `nvidia.com/mig.config`. Deleting one MIG Manager Pod without changing its controller policy does not establish that boundary.

See the [Dynamic MIG user guide](../userguide/nvidia-device/dynamic-mig-support) for the current Chart allowlist, migration checklist, workload example, and validation commands.

## Special thanks

Thanks to @sailorvii for helping make the original Dynamic MIG feature possible.
Loading