diff --git a/docs/developers/dynamic-mig.md b/docs/developers/dynamic-mig.md index ce4cc579d..1200e0289 100644 --- a/docs/developers/dynamic-mig.md +++ b/docs/developers/dynamic-mig.md @@ -1,157 +1,221 @@ --- -title: NVIDIA GPU MPS and MIG dynamic slice plugin +title: NVIDIA dynamic MIG implementation sidebar_label: Dynamic MIG Implementation --- -## Special Thanks - -This feature will not be implemented without the help of @sailorvii. - ## Introduction -The NVIDIA GPU built-in sharing method includes: time-slice, MPS and MIG. The context switch for time slice sharing would waste some time, so MPS and MIG are preferred. The GPU MIG profile is variable, the user could acquire the MIG device in the profile definition, but current implementation only defines the dedicated profile before the user requirement. That limits the usage of MIG. The goal is to develop an automatic slice plugin and create the slice when the user requires it. For the scheduling method, node-level binpack and spread will be supported. Referring to the binpack plugin, the scheduler considers CPU, Mem, GPU memory and other user-defined resources. HAMi is done by using [hami-core](https://github.com/Project-HAMi/HAMi-core), which is a cuda-hacking library. But MIG is also widely used across the world. A unified API for dynamic-mig and hami-core is needed. +HAMi v2.10 replaces predefined `knownMigGeometries` and whole-GPU template switching with a reservation-first Dynamic MIG architecture. Hardware capability originates on the node, the scheduler reserves an exact MIG placement before binding a Pod, and the NVIDIA device plugin realizes and reclaims the corresponding GPU instance (GI) and compute instance (CI). + +The design keeps four responsibilities separate: + +| Authority | Responsibility | +| --- | --- | +| NVML on the node | Reports profile memory, compute capacity, slice count, and legal placements | +| HAMi scheduler | Selects the physical GPU, profile, and placement and accounts for the reservation | +| NVIDIA device plugin | Serializes hardware mutation and creates, verifies, adopts, and destroys GI/CI instances | +| Pod annotation | Persists the allocation identity shared by the scheduler and device plugin | + +This model creates hardware-isolated MIG devices per Pod without making the scheduler mutate GPU hardware or making the device plugin choose a different placement at runtime. + +## Design goals + +- Treat the node and NVML as the source of truth for hardware capability. +- Reserve topology, not only aggregate MIG slice count. +- Realize exactly the profile and placement accepted by the scheduler. +- Use stable workload metadata for reconciliation and restart recovery. +- Reclaim one Pod's instance without reconfiguring instances owned by other Pods. +- Keep the shared scheduler/device-plugin contract compact as GPU density grows. + +## Architecture + +```text + Kubernetes control plane + + +----------------+ Node capability +----------------+ + | Device Plugin | --------------------------> | HAMi Scheduler | + | | | | + | NVML discovery | Pod reservation | Placement | + | GI/CI manager | <-------------------------- | policy | + | Reconciler | | Capacity model | + +-------+--------+ +--------+-------+ + | | + | exact GI/CI realization | bind + v v + +----------------+ +----------------+ + | NVIDIA GPU | | Workload Pod | + | MIG topology | | Allocation | + | and instances | | annotation | + +----------------+ +----------------+ +``` -## Targets +The device plugin publishes capability in the NVIDIA node registration annotation. The scheduler reconstructs topology occupancy from active Pod reservations. The Pod annotation is the durable handoff between the scheduling and runtime phases. -- CPU, Mem, and GPU combined schedule -- GPU dynamic slice: HAMi-core and MIG -- Support node-level binpack and spread by GPU memory, CPU and Mem -- A unified vGPU Pool different virtualization techniques -- Tasks can choose to use MIG, use HAMi-core, or use both. +## Capability discovery contract -### Config maps +### Policy comes from `migProfileAllowlist` -- hami-scheduler-device-configMap This configmap defines the plugin configurations including resourceName, and MIG geometries, and node-level configurations. +The scheduler device configuration names the profiles that cluster policy allows for each model: ```yaml -apiVersion: v1 -data: - device-config.yaml: | - nvidia: - resourceCountName: nvidia.com/gpu - resourceMemoryName: nvidia.com/gpumem - resourceCoreName: nvidia.com/gpucores - knownMigGeometries: - - models: [ "A30" ] - allowedGeometries: - - - - name: 1g.6gb - memory: 6144 - count: 4 - - - - name: 2g.12gb - memory: 12288 - count: 2 - - - - name: 4g.24gb - memory: 24576 - count: 1 - - models: [ "A100-SXM4-40GB", "A100-40GB-PCIe", "A100-PCIE-40GB" ] - allowedGeometries: - - - - name: 1g.5gb - memory: 5120 - count: 7 - - - - name: 2g.10gb - memory: 10240 - count: 3 - - name: 1g.5gb - memory: 5120 - count: 1 - - - - name: 3g.20gb - memory: 20480 - count: 2 - - - - name: 7g.40gb - memory: 40960 - count: 1 - - models: [ "A100-SXM4-80GB", "A100-80GB-PCIe", "A100-PCIE-80GB"] - allowedGeometries: - - - - name: 1g.10gb - memory: 10240 - count: 7 - - - - name: 2g.20gb - memory: 20480 - count: 3 - - name: 1g.10gb - memory: 10240 - count: 1 - - - - name: 3g.40gb - memory: 40960 - count: 2 - - - - name: 7g.79gb - memory: 80896 - count: 1 - nodeconfig: - - name: nodeA - operatingmode: hami-core - - name: nodeB - operatingmode: mig +nvidia: + migProfileAllowlist: + - models: ["A100-SXM4-40GB"] + profiles: ["1g.5gb", "2g.10gb", "3g.20gb", "7g.40gb"] + - models: ["RTX PRO 6000 Blackwell Server Edition"] + profiles: ["1g.24gb", "2g.48gb", "4g.96gb"] ``` -## Structure +The [Chart's default configuration](https://github.com/Project-HAMi/HAMi/blob/v2.10.0/charts/hami/templates/scheduler/device-configmap.yaml) also includes A30, A100 80 GB, H100, H20, H200, and B200 mappings. The allowlist determines what the cluster permits; it does not define capacity or a complete geometry. -HAMi dynamic MIG structure diagram showing vGPU Pool and Scheduler components +A custom `device-config.content` value or external ConfigMap replaces the Chart default. Such configurations must use `migProfileAllowlist` explicitly. Legacy `knownMigGeometries` fields are not converted automatically. -## Examples +### Capability comes from NVML -Dynamic MIG is compatible with HAMi tasks, as shown in the example below: Set `nvidia.com/gpu` and `nvidia.com/gpumem`. +For every allowlisted profile on a matching GPU, the device plugin queries NVML for profile information and possible placements. It publishes scheduler-facing fields in the per-GPU `migProfiles` array of `hami.io/node-nvidia-register`: -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: gpu-pod1 -spec: - containers: - - name: ubuntu-container1 - image: ubuntu:22.04 - command: ["bash", "-c", "sleep 86400"] - resources: - limits: - nvidia.com/gpu: 2 # requesting 2 vGPUs - nvidia.com/gpumem: 8000 # Each vGPU contains 8000m device memory (Optional,Integer) +```json +{ + "name": "2g.10gb", + "memoryMB": 9984, + "core": 29, + "sliceCount": 2, + "placements": [ + { "start": 0, "size": 2 }, + { "start": 2, "size": 2 }, + { "start": 4, "size": 2 } + ] +} ``` -A task can decide only to use `mig` or `hami-core` by setting `annotations.nvidia.com/vgpu-mode` to corresponding value, as the example below shows: - -```yaml -apiVersion: v1 -kind: Pod -metadata: - name: gpu-pod1 - annotations: - nvidia.com/vgpu-mode: "mig" -spec: - containers: - - name: ubuntu-container1 - image: ubuntu:22.04 - command: ["bash", "-c", "sleep 86400"] - resources: - limits: - nvidia.com/gpu: 2 # requesting 2 vGPUs - nvidia.com/gpumem: 8000 # Each vGPU contains 8000m device memory (Optional,Integer) +| Field | Scheduler use | +| ------------ | ----------------------------------------------------------------- | +| `name` | Stable profile identity across components | +| `memoryMB` | Match `nvidia.com/gpumem` demand against actual reported capacity | +| `core` | Account for the profile's compute share | +| `sliceCount` | Order profile candidates deterministically | +| `placements` | Select a legal, non-overlapping physical slice interval | + +Profile names are policy labels, but their memory, compute, and placement data are node-local facts. A profile that is not allowlisted or that NVML cannot resolve is omitted. Device-plugin-only discovery data, such as the maximum instance count, is not added to the shared wire format when placements already express schedulable capacity. + +## Scheduling and reservation + +For a Pod selecting `nvidia.com/vgpu-mode: "mig"`, scheduling follows this sequence: + +1. Reconstruct each GPU's occupied intervals from active Pod reservations. +2. Sort discovered profile candidates by NVML-reported memory and slice count. +3. Choose the smallest profile that satisfies the container's memory request. +4. Select a deterministic legal placement that does not overlap an existing or newly accepted reservation. +5. Add the reservation to scheduler occupancy immediately, before the Pod is bound. +6. Persist the complete logical reservation in the Pod. + +A placement occupies the half-open interval `[placement.start, placement.start + placement.size)`. Aggregate free slices are not sufficient if no placement reported for the requested profile fits without overlap. In that case the Pod remains Pending. The scheduler does not request a whole-GPU template change and does not move a running GI. + +### Pod reservation contract + +The scheduler writes a JSON array to `hami.io/vgpu-mig-allocations`, with one entry per requested MIG device: + +```json +[ + { + "containerIndex": 0, + "deviceIndex": 0, + "gpuUUID": "GPU-xxxxxxxx", + "profile": "2g.10gb", + "placement": { "start": 2, "size": 2 }, + "migUUID": "MIG-xxxxxxxx", + "gpuInstanceID": 4, + "computeInstanceID": 0 + } +] ``` -## Procedures +The scheduler writes `containerIndex`, `deviceIndex`, `gpuUUID`, `profile`, and `placement`. The runtime identity fields are initially absent. After realization, the device plugin patches the same record with `migUUID`, `gpuInstanceID`, and `computeInstanceID`. -The Procedure of a vGPU task which uses dynamic-mig is shown below: +Container and device indexes disambiguate repeated allocations on the same physical GPU. Runtime identity is valid only when all three runtime fields are present; partial runtime identity fails validation. This annotation is an internal protocol and must not be generated or modified by users. -HAMi dynamic MIG procedure flowchart showing task scheduling process +## Runtime realization -After submitting a task, the deviceshare plugin iterates over templates defined in configMap `hami-scheduler-device` and finds the first available template to fit. You can always change the content of that configMap, and restart vc-scheduler to customize. +During kubelet `Allocate`, the device plugin resolves the entries for the container being started and performs the following operations: -If you submit the example on an empty A100-PCIE-40GB node, then it will select a GPU and choose MIG template below: +1. Reconcile tracked instances against the current active-Pod snapshot. +2. Verify that the reserved profile and placement still exist in NVML capability. +3. Acquire the per-physical-GPU mutation lock. +4. Create the GI at exactly the scheduler-selected placement and create its CI. +5. Resolve the MIG UUID, GI ID, and CI ID and expose the MIG device to the container. +6. Patch the Pod's allocation annotation with that runtime identity. -```yaml -2g.10gb: 3 -1g.5gb: 1 +The manager key is the physical GPU index, profile, placement start, and placement size. Repeated `Allocate` calls for the same reservation return the same managed MIG UUID. If a multi-device allocation fails partway through, instances created by that attempt are rolled back; already adopted instances are not destroyed as part of that rollback. + +The device plugin never retries a different physical placement. Doing so would violate the scheduler's topology accounting and could overlap another accepted reservation. + +## Reconciliation and reclaim + +The device plugin periodically lists Pods assigned to its node. Pods with valid reservations form the desired allocation set if they are not deleting and have not reached Succeeded or Failed. Managed instances absent from that set are released by destroying their exact CI and GI. + +Reconciliation is intentionally conservative. A complete Kubernetes list and valid allocation annotations authorize a cleanup pass. API failures, malformed records, or partial runtime identity cause destructive reconciliation to be skipped rather than allowing the plugin to infer ownership. + +This convergent loop makes a placement reusable after its Pod terminates while leaving unrelated placements on the same GPU intact. + +## Device plugin restart adoption + +Startup recovery uses both Kubernetes allocation state and NVML activity: + +1. Active Pod reservations and GPU process state identify GPUs carrying work. +2. Idle GPUs are prepared in a clean MIG-ready state; old GI/CI instances on those idle GPUs can be removed. +3. For each active v2.10 record, the plugin verifies the annotated profile, placement, MIG UUID, GI ID, and CI ID against NVML. +4. A matching live instance is adopted into the new manager's allocation maps. +5. Normal reconciliation resumes from the reconstructed ownership state. + +If allocation state cannot be read reliably during startup, the plugin preserves GPUs instead of applying destructive idle-GPU cleanup. Legacy `GPU-UUID[template-slot]` identifiers cannot be adopted because they do not prove a physical placement and complete runtime identity. + +## Metrics and observability + +The scheduler exports realized instances through the current metric: + +```text +# HELP hami_node_gpu_mig_instance_info Realized MIG instance identity and scheduler placement +# TYPE hami_node_gpu_mig_instance_info gauge +hami_node_gpu_mig_instance_info{compute_instance_id="0",device_index="0",device_uuid="GPU-xxxxxxxx",gpu_instance_id="4",mig_uuid="MIG-xxxxxxxx",node="MIG-NODE-A",placement_size="2",placement_start="2",profile="2g.10gb"} 1 ``` -Then start the container with 2g.10gb instances \* 2 +Only reservations enriched with complete runtime identity produce this series. Its labels join the scheduler reservation to the physical GPU, placement, MIG UUID, GI ID, and CI ID. `device_uuid` plus `gpu_instance_id` can also correlate with DCGM series carrying `UUID` and `GPU_I_ID`. + +`nodeGPUMigInstance` is a compatibility metric emitted only when `legacyMetrics: true`; the current Chart default is `false`. The standard scheduler endpoint is `:31993/metrics` with the default NodePort service. + +Useful operational signals include: + +- device plugin discovery logs and non-empty per-GPU `migProfiles`; +- scheduler placement decisions and Pods that remain Pending under capacity or fragmentation pressure; +- the `hami.io/vgpu-mig-allocations` transition from logical reservation to runtime identity; +- GI/CI visibility through `nvidia-smi` or NVML; +- reclaim and startup-adoption logs; and +- sustained workload progress before and after a device plugin restart. + +## Migration and ownership boundaries + +### Legacy HAMi geometry implementation + +The v2.9 and v2.10 protocols cannot safely serve MIG workloads as a mixed scheduler/device-plugin pair. A legacy scheduler publishes and consumes `migtemplate` and template/slot identifiers; v2.10 consumes `migProfiles` and requires an exact profile/placement reservation. + +Initial migration therefore requires a controlled handover: + +1. stop new MIG scheduling and drain legacy MIG Pods one node at a time; +2. convert the union of required legacy profiles from `knownMigGeometries` into `migProfileAllowlist`; +3. upgrade the scheduler before the node device plugins; +4. validate capability publication, realization, reclaim, and restart adoption; and +5. uncordon each node only after its lifecycle test passes. + +Routine mixed-profile creation after migration does not require a whole-GPU geometry switch when a legal free placement exists. Hardware fragmentation, MIG mode changes, driver maintenance, rollback, or a layout that requires moving an active GI can still require draining or rebooting. + +### NVIDIA MIG Manager + +MIG Manager applies node- or GPU-level geometries, while HAMi Dynamic MIG creates and destroys GI/CI instances from per-Pod reservations. Both mutate the same hardware state and must not reconcile the same physical GPU concurrently. + +GPU Operator may continue to provide the NVIDIA driver, Container Toolkit, DCGM, and other infrastructure. Before HAMi assumes mutation ownership, stop MIG Manager reconciliation and ensure a controller cannot recreate it or reapply `nvidia.com/mig.config`. Deleting one MIG Manager Pod without changing its controller policy does not establish that boundary. + +See the [Dynamic MIG user guide](../userguide/nvidia-device/dynamic-mig-support) for the current Chart allowlist, migration checklist, workload example, and validation commands. + +## Special thanks + +Thanks to @sailorvii for helping make the original Dynamic MIG feature possible. diff --git a/docs/userguide/nvidia-device/dynamic-mig-support.md b/docs/userguide/nvidia-device/dynamic-mig-support.md index f3e983532..992e87202 100644 --- a/docs/userguide/nvidia-device/dynamic-mig-support.md +++ b/docs/userguide/nvidia-device/dynamic-mig-support.md @@ -2,176 +2,210 @@ title: Enable dynamic MIG feature --- -HAMi now supports dynamic MIG using mig-parted to adjust MIG devices dynamically, including: +HAMi v2.10 uses a reservation-first, topology-aware implementation of dynamic NVIDIA Multi-Instance GPU (MIG). It does not select and switch a predefined whole-GPU geometry. Instead: -- **Dynamic MIG Instance Management**: Users no longer need to operate directly on GPU nodes or use commands like `nvidia-smi -i 0 -mig 1` to manage MIG instances. HAMi-device-plugin will handle this automatically. +1. The device plugin discovers the MIG profiles and legal placements supported by each physical GPU through NVML. +2. `migProfileAllowlist` defines which of those profiles HAMi may expose. +3. The scheduler reserves an exact physical GPU, profile, and placement for each Pod. +4. During kubelet `Allocate`, the device plugin creates that GPU instance (GI) and compute instance (CI). +5. When the Pod finishes, the device plugin reclaims that Pod's CI and GI so the placement can be reused. -- **Dynamic MIG Adjustment**: Each MIG device managed by HAMi will dynamically adjust its MIG template according to the jobs submitted, as needed. - -- **Device MIG Observation**: Each MIG instance generated by HAMi will be displayed in the scheduler monitor, along with job information, providing a clear overview of MIG nodes. - -- **Compatibility with HAMi-Core Nodes**: HAMi can manage a unified GPU pool across both `HAMi-core nodes` and `MIG nodes`. A job can be scheduled to either node unless manually specified using the `nvidia.com/vgpu-mode` annotation. - -- **Unified API with HAMi-Core**: No additional work is required to make jobs compatible with the dynamic MIG feature. +This preserves HAMi's unified `nvidia.com/gpu` and `nvidia.com/gpumem` workload API while creating hardware-isolated MIG instances only when workloads need them. ## Prerequisites -- NVIDIA Blackwell, Hopper™, and Ampere GPUs -- HAMi > v2.5.0 -- nvidia-container-toolkit +- HAMi v2.10.0 or later. The scheduler and NVIDIA device plugin must use the same reservation protocol. +- An NVIDIA Ampere, Hopper, or Blackwell GPU that supports MIG, plus a driver version that exposes the required profiles through NVML. +- NVIDIA Container Toolkit. +- Exclusive ownership of MIG hardware mutation on each target GPU. HAMi Dynamic MIG and NVIDIA MIG Manager must not manage the same physical GPU at the same time. -## Enable dynamic MIG support +The current Chart includes profile mappings for A30, A100, H100, H20, H200, B200, and **RTX PRO 6000 Blackwell Server Edition** GPUs. Actual capability is still determined by NVML on each node. -- Install the chart using helm, See [enabling vGPU support in kubernetes](https://github.com/Project-HAMi/HAMi#enabling-vgpu-support-in-kubernetes). +## Enable dynamic MIG support -- Configure `mode` in device-plugin configMap to `mig` for MIG nodes +### 1. Set the node operating mode to `mig` - ```bash - kubectl describe cm hami-device-plugin -n kube-system - ``` +Install or upgrade the HAMi Chart as described in [the online installation guide](../../installation/online-installation.md), then set `operatingmode` to `mig` for each target node. For example, the `devicePlugin.nodeConfiguration.config` value can contain: - ```json - { - "nodeconfig": [ +```yaml +devicePlugin: + nodeConfiguration: + config: | { - "name": "MIG-NODE-A", - "operatingmode": "mig", - "filterdevices": { - "uuid": [], - "index": [] - } + "nodeconfig": [ + { + "name": "MIG-NODE-A", + "operatingmode": "mig", + "filterdevices": { + "uuid": [], + "index": [] + } + } + ] } - ] - } - ``` - -- Restart the following pods for the change to take effect: - - hami-scheduler - - hami-device-plugin on 'MIG-NODE-A' - -## Custom MIG configuration (optional) +``` -HAMi currently has a [built-in MIG configuration](https://github.com/Project-HAMi/HAMi/blob/master/charts/hami/templates/scheduler/device-configmap.yaml) for MIG. +Changing MIG mode can require a GPU reset or node reboot on some hardware and driver combinations. Cordon and drain a node before its initial conversion if it is already running GPU workloads. -You can customize the MIG configuration by following the steps below: +### 2. Configure the profile allowlist -### Edit `device-configmap.yaml` in charts/hami/templates/scheduler +The [current Chart device configuration](https://github.com/Project-HAMi/HAMi/blob/v2.10.0/charts/hami/templates/scheduler/device-configmap.yaml) uses `migProfileAllowlist`. The v2.10 defaults are: - ```yaml nvidia: - resourceCountName: {{ .Values.resourceName }} - resourceMemoryName: {{ .Values.resourceMem }} - resourceMemoryPercentageName: {{ .Values.resourceMemPercentage }} - resourceCoreName: {{ .Values.resourceCores }} - resourcePriorityName: {{ .Values.resourcePriority }} - overwriteEnv: false - defaultMemory: 0 - defaultCores: 0 - defaultGPUNum: 1 - memoryFactor: 1 - deviceSplitCount: {{ .Values.devicePlugin.deviceSplitCount }} - deviceMemoryScaling: {{ .Values.devicePlugin.deviceMemoryScaling }} - deviceCoreScaling: {{ .Values.devicePlugin.deviceCoreScaling }} - knownMigGeometries: + migProfileAllowlist: - models: ["A30"] - allowedGeometries: - - name: 1g.6gb - memory: 6144 - count: 4 - - name: 2g.12gb - memory: 12288 - count: 2 - - name: 4g.24gb - memory: 24576 - count: 1 - + profiles: ["1g.6gb", "2g.12gb", "4g.24gb"] - models: ["A100-SXM4-40GB", "A100-40GB-PCIe", "A100-PCIE-40GB"] - allowedGeometries: - - name: 1g.5gb - memory: 5120 - count: 7 - - name: 2g.10gb - memory: 10240 - count: 3 - - name: 1g.5gb - memory: 5120 - count: 1 - - name: 3g.20gb - memory: 20480 - count: 2 - - name: 7g.40gb - memory: 40960 - count: 1 - + profiles: ["1g.5gb", "2g.10gb", "3g.20gb", "7g.40gb"] - models: ["A100-SXM4-80GB", "A100-80GB-PCIe", "A100-PCIE-80GB"] - allowedGeometries: - - name: 1g.10gb - memory: 10240 - count: 7 - - name: 2g.20gb - memory: 20480 - count: 3 - - name: 1g.10gb - memory: 10240 - count: 1 - - name: 3g.40gb - memory: 40960 - count: 2 - - name: 7g.79gb - memory: 80896 - count: 1 + profiles: ["1g.10gb", "2g.20gb", "3g.40gb", "7g.79gb"] + - models: ["H100-PCIE-80GB", "H100-SXM5-80GB"] + profiles: ["1g.10gb", "2g.20gb", "3g.40gb", "7g.80gb"] + - models: ["H100-PCIE-94GB", "H100-SXM5-94GB"] + profiles: ["1g.12gb", "2g.24gb", "3g.47gb", "7g.94gb"] + - models: ["H20", "H100 on GH200"] + profiles: ["1g.12gb", "2g.24gb", "3g.48gb", "7g.96gb"] + - models: ["H200 NVL", "H200-SXM5"] + profiles: ["1g.18gb", "2g.35gb", "3g.71gb", "7g.141gb"] + - models: ["B200"] + profiles: ["1g.23gb", "2g.45gb", "3g.90gb", "7g.180gb"] + - models: ["RTX PRO 6000 Blackwell Server Edition"] + profiles: ["1g.24gb", "2g.48gb", "4g.96gb"] ``` -:::note +The allowlist is cluster policy, not a hardware topology description. Do not configure profile memory, compute percentage, instance count, or placement. The device plugin obtains those values from `GetGpuInstanceProfileInfo` and `GetGpuInstancePossiblePlacements` in NVML and publishes only allowlisted, discoverable profiles to the scheduler. -Helm installations and updates will follow the configuration specified in this file, overriding the default Helm settings. +:::warning -HAMi uses the first MIG template that matches the job, in the order defined in this configMap. +If you set `device-config.content` or provide an external scheduler device ConfigMap, that content replaces the Chart's default device configuration. Update the complete custom configuration to use `migProfileAllowlist`. `knownMigGeometries` is a legacy v2.9 field and is not converted automatically. ::: -## Running MIG jobs +After changing the node mode or scheduler device configuration, restart the HAMi scheduler and the NVIDIA device plugin on the affected nodes. Confirm that device plugin logs show profile discovery and that each MIG GPU publishes a non-empty `migProfiles` array in `hami.io/node-nvidia-register`: + +```bash +kubectl get node MIG-NODE-A -o json \ + | jq -r '.metadata.annotations["hami.io/node-nvidia-register"] | fromjson' +``` + +A model or profile that is not allowlisted, or that NVML cannot discover, is not advertised as schedulable MIG capacity. + +## Run a MIG workload -A MIG instance can now be requested by a container in the same way as `hami-core`, by specifying the `nvidia.com/gpu` and `nvidia.com/gpumem` resource types. +Request a MIG-backed vGPU with the same resource names used by HAMi-core. Set `nvidia.com/vgpu-mode: "mig"` when the workload must run on a MIG node: ```yaml apiVersion: v1 kind: Pod metadata: - name: gpu-pod + name: mig-workload annotations: - nvidia.com/vgpu-mode: "mig" #(Optional), if not set, this pod can be assigned to a MIG instance or a hami-core instance + nvidia.com/vgpu-mode: "mig" spec: containers: - - name: ubuntu-container + - name: workload image: ubuntu:22.04 command: ["bash", "-c", "sleep 86400"] resources: limits: - nvidia.com/gpu: 2 + nvidia.com/gpu: 1 nvidia.com/gpumem: 8000 ``` -In this example above, the job allocates two MIG instances, each with at least 8G device memory. +Each unit of `nvidia.com/gpumem` is 1 MiB. The scheduler selects the smallest allowlisted profile whose NVML-reported memory satisfies the request and that has a legal, non-overlapping placement. If no placement is currently possible, the Pod remains Pending; HAMi does not move a running GI or switch the entire GPU to another template. + +Without the `nvidia.com/vgpu-mode` annotation, a workload can be placed in a compatible HAMi-core or MIG pool. HAMi continues to expose the unified `nvidia.com/gpu` resource rather than resources such as `nvidia.com/mig-1g.10gb`. + +## Reservation and instance lifecycle + +The scheduler writes one record per requested MIG device to the Pod annotation `hami.io/vgpu-mig-allocations`. For example, after successful allocation a record can look like this: + +```json +[ + { + "containerIndex": 0, + "deviceIndex": 0, + "gpuUUID": "GPU-xxxxxxxx", + "profile": "2g.10gb", + "placement": { "start": 2, "size": 2 }, + "migUUID": "MIG-xxxxxxxx", + "gpuInstanceID": 4, + "computeInstanceID": 0 + } +] +``` + +The scheduler initially records the container and device indexes, parent GPU, profile, and placement. During `Allocate`, the device plugin: + +1. verifies that reservation against current NVML capability; +2. serializes mutation on the selected physical GPU; +3. creates the GI and CI at exactly the reserved placement; +4. injects the resulting MIG device into the container; and +5. adds `migUUID`, `gpuInstanceID`, and `computeInstanceID` to the annotation. -## Monitor MIG Instance +The physical GPU, profile, and placement form an idempotent reservation key, so a repeated allocation request converges on the same managed instance. The annotation is an internal HAMi contract: users and other controllers must not create, remove, or modify it. -MIG Instance managed by HAMi will be displayed in scheduler monitor (scheduler node ip:31993/metrics), as follows: +Inspect it with: ```bash -# HELP nodeGPUMigInstance GPU Sharing mode. 0 for hami-core, 1 for mig, 2 for mps -# TYPE nodeGPUMigInstance gauge -nodeGPUMigInstance{deviceidx="0",deviceuuid="GPU-936619fc-f6a1-74a8-0bc6-ecf6b3269313",migname="3g.20gb-0",nodeid="aio-node15",zone="vGPU"} 1 -nodeGPUMigInstance{deviceidx="0",deviceuuid="GPU-936619fc-f6a1-74a8-0bc6-ecf6b3269313",migname="3g.20gb-1",nodeid="aio-node15",zone="vGPU"} 0 -nodeGPUMigInstance{deviceidx="1",deviceuuid="GPU-30f90f49-43ab-0a78-bf5c-93ed41ef2da2",migname="3g.20gb-0",nodeid="aio-node15",zone="vGPU"} 1 -nodeGPUMigInstance{deviceidx="1",deviceuuid="GPU-30f90f49-43ab-0a78-bf5c-93ed41ef2da2",migname="3g.20gb-1",nodeid="aio-node15",zone="vGPU"} 1 +kubectl get pod mig-workload -o json \ + | jq -r '.metadata.annotations["hami.io/vgpu-mig-allocations"] | fromjson' ``` -:::note +The device plugin periodically compares managed instances with active Pods on its node. When a Pod is deleted, succeeds, or fails, its exact CI and GI are destroyed without changing instances owned by other Pods. If the Kubernetes API or an annotation cannot be read completely, destructive reconciliation is skipped rather than guessing. -1. No action is required on MIG nodes - everything is managed by `mig-parted` in `hami-device-plugin`. -2. NVIDIA devices older than the Ampere architecture do not support MIG mode. -3. MIG resources (e.g., `nvidia.com/mig-1g.10gb`) won’t be visible on the node. HAMi uses a unified resource name for both MIG and hami-core nodes. +### Device plugin restart recovery -::: +On startup, the device plugin combines active Pod reservations with NVML process state to identify GPUs carrying work. It prepares idle GPUs for dynamic allocation, verifies complete runtime records against NVML, and adopts matching active GI/CI instances into the new manager process. This preserves current v2.10 allocations across a device plugin restart. + +Restart adoption requires the complete profile, placement, MIG UUID, GI ID, and CI ID recorded by v2.10. Legacy template/slot identifiers do not contain enough physical identity to be adopted safely. + +## Monitor realized MIG instances + +The scheduler metrics endpoint exposes one `hami_node_gpu_mig_instance_info` series for each realized allocation that has a complete runtime identity: + +```bash +curl http://:31993/metrics +``` + +```text +# HELP hami_node_gpu_mig_instance_info Realized MIG instance identity and scheduler placement +# TYPE hami_node_gpu_mig_instance_info gauge +hami_node_gpu_mig_instance_info{compute_instance_id="0",device_index="0",device_uuid="GPU-xxxxxxxx",gpu_instance_id="4",mig_uuid="MIG-xxxxxxxx",node="MIG-NODE-A",placement_size="2",placement_start="2",profile="2g.10gb"} 1 +``` + +The parent `device_uuid` and `gpu_instance_id` can be correlated with DCGM metrics that carry `UUID` and `GPU_I_ID` labels. The older `nodeGPUMigInstance` metric is emitted only when `legacyMetrics: true`; the Chart default is `false`. See [Cluster device allocation](../monitoring/device-allocation) for the other scheduler allocation metrics. + +## Migrate from legacy dynamic MIG + +The v2.9 geometry implementation and the v2.10 reservation implementation use incompatible scheduler/device-plugin contracts. The legacy scheduler encodes a template and slot in a device identifier, while v2.10 requires the explicit Pod reservation described above. There is no seamless rolling adoption of legacy MIG Pods. + +For the initial migration: + +1. Inventory the scheduler device ConfigMap, active MIG Pods, node registration annotations, and `nvidia-smi -L` output. +2. Cordon a MIG node and drain or finish its legacy MIG workloads. +3. Replace `knownMigGeometries` with `migProfileAllowlist`. Usually the allowlist is the union of the profile names in the old geometries; remove manually maintained memory, core, count, and layout data. +4. Upgrade the scheduler before upgrading device plugins, so a legacy scheduler cannot send template/slot allocations to a v2.10 plugin. +5. Upgrade device plugins one node at a time. Startup can remove pre-existing GI/CI instances from GPUs that HAMi determines are idle. +6. Validate profile publication, Pod reservation, GI/CI realization, Pod deletion and reclaim, and device plugin restart adoption before uncordoning the node. + +Routine creation and deletion of mixed profiles no longer requires switching a whole-GPU template when a legal free placement exists. Draining can still be required for the initial migration, enabling or disabling MIG mode, driver or GPU reset maintenance, rollback, or a layout change that would have to move a running instance. + +## Migrate from NVIDIA MIG Manager + +NVIDIA MIG Manager and HAMi Dynamic MIG can both create and destroy GI/CI instances, so they must not reconcile the same physical GPU concurrently. Before enabling HAMi `mig` mode on a target node: + +1. cordon the node and move existing GPU workloads; +2. stop MIG Manager reconciliation for the target GPU, including any controller that would recreate it or reapply `nvidia.com/mig.config`; +3. keep the GPU Operator components HAMi still needs, such as the driver, Container Toolkit, and optionally DCGM; and +4. start with one canary Pod, then test mixed profiles, capacity exhaustion, reclaim, and device plugin restart recovery. + +Deleting a MIG Manager Pod once is not enough if its controller immediately recreates it. Assign one controller as the hardware mutation owner unless you have explicit and verified per-GPU ownership isolation. For NVIDIA's static MIG workflow and its reconfiguration constraints, see the [NVIDIA GPU Operator MIG documentation](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-mig.html). + +## Limitations + +- Dynamic MIG obeys NVIDIA's placement rules. Fragmentation can leave enough total slices but no legal contiguous placement for a larger profile. +- HAMi does not relocate an active GI/CI to defragment a GPU. +- GPUs older than Ampere do not support MIG. +- Enabling or disabling MIG mode can still require a reset or reboot depending on the GPU and driver. diff --git a/docs/userguide/nvidia-device/examples/dynamic-mig-example.md b/docs/userguide/nvidia-device/examples/dynamic-mig-example.md index ce96de666..2a60fc723 100644 --- a/docs/userguide/nvidia-device/examples/dynamic-mig-example.md +++ b/docs/userguide/nvidia-device/examples/dynamic-mig-example.md @@ -2,7 +2,7 @@ title: Assign task to MIG instance --- -This example will allocate `2g.10gb * 2` for A100-40GB-PCIE device or `1g.10gb * 2` for A100-80GB-SXM device. +This Pod requests two MIG devices with at least 8,000 MiB each. The `nvidia.com/vgpu-mode: "mig"` annotation requires Dynamic MIG, and the optional `binpack` policy asks HAMi to prefer packing the allocations. ```yaml apiVersion: v1 @@ -11,7 +11,7 @@ metadata: name: gpu-pod annotations: nvidia.com/vgpu-mode: "mig" - hami.io/gpu-scheduler-policy: "binpack" #(Optional) + hami.io/gpu-scheduler-policy: "binpack" # Optional spec: containers: - name: ubuntu-container @@ -22,3 +22,7 @@ spec: nvidia.com/gpu: 2 nvidia.com/gpumem: 8000 ``` + +On an empty A100 40 GB GPU this request normally selects two `2g.10gb` profiles; on an empty A100 80 GB GPU it normally selects two `1g.10gb` profiles. The result is not a fixed template: HAMi chooses the smallest allowlisted, NVML-discovered profile with enough memory and a legal free placement, so driver-reported capacity and existing reservations can change the selected profile or leave the Pod Pending. + +HAMi records the selected GPU, profile, and placement in the internal `hami.io/vgpu-mig-allocations` Pod annotation. The device plugin creates the reserved GI/CI instances during `Allocate`, adds their runtime identities to the annotation, and reclaims them after the Pod terminates. Users must not set or edit this annotation. diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/developers/dynamic-mig.md b/i18n/zh/docusaurus-plugin-content-docs/current/developers/dynamic-mig.md index c86a65278..d8ad9c779 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/developers/dynamic-mig.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/developers/dynamic-mig.md @@ -3,6 +3,18 @@ title: NVIDIA GPU MPS 和 MIG 动态切片插件 translated: true --- +:::important HAMi v2.10 当前实现说明 + +本说明仅描述 v2.10 的变更边界；下方原有的 `knownMigGeometries` 配置、模板遍历流程和架构图保留为 v2.9 及更早实现的参考，不代表 v2.10 的实现。 + +v2.10 采用预留优先架构：节点上的 device plugin 是硬件能力与 GI/CI 变更的权威，通过 NVML 发现 allowlist 中 profile 的 `memoryMB`、`core`、`sliceCount` 和合法 `placements`，并在 `hami.io/node-nvidia-register` 的 `migProfiles` 中发布紧凑能力；调度器是 placement 策略与预留的权威，在绑定前选择 GPU、profile 和不重叠的 placement。 + +调度器将逻辑预留持久化到 Pod 的 `hami.io/vgpu-mig-allocations`。device plugin 在 kubelet `Allocate` 阶段严格按预留创建该 Pod 的 GI/CI，并补充 MIG UUID、GI ID 和 CI ID。协调循环根据活动 Pod 回收已结束工作负载的实例；device plugin 重启后，会通过完整注解和 NVML 校验并接管仍在运行的实例。API 或注解读取不完整时会跳过破坏性清理。 + +当前可观测性使用 `hami_node_gpu_mig_instance_info`；旧 `nodeGPUMigInstance` 仅在启用 `legacyMetrics` 后输出。HAMi 与 NVIDIA MIG Manager 不能同时修改同一块物理 GPU。从旧模板协议迁移必须先 drain 旧 MIG Pod，并先升级调度器再逐节点升级 device plugin。当前 Chart 使用 `migProfileAllowlist`，并包含 RTX PRO 6000 Blackwell Server Edition 的 profile 支持。 + +::: + ## 特别感谢 没有 @sailorvii 的帮助，这个功能将无法实现。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/dynamic-mig-support.md b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/dynamic-mig-support.md index 01911ecc8..2de3ab804 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/dynamic-mig-support.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/dynamic-mig-support.md @@ -3,6 +3,18 @@ title: 启用动态 MIG 功能 translated: true --- +:::important HAMi v2.10 当前行为 + +本说明仅用于界定 v2.10 的当前行为；下方保留的 `knownMigGeometries`、整卡 MIG 模板切换、`mig-parted` 和旧 `nodeGPUMigInstance` 指标内容描述的是 v2.9 及更早实现，不适用于 v2.10。 + +- v2.10 使用预留优先模型。`migProfileAllowlist` 只定义集群允许使用的配置文件名称；显存、算力、切片数量及合法 placement 由每个节点通过 NVML 发现，并以 `migProfiles` 发布给调度器。当前 Chart 还包含 **RTX PRO 6000 Blackwell Server Edition** 的 `1g.24gb`、`2g.48gb` 和 `4g.96gb`。 +- 调度器在绑定 Pod 前预留物理 GPU、profile 和 placement，并写入内部注解 `hami.io/vgpu-mig-allocations`。device plugin 在 `Allocate` 阶段按该预留创建每个 Pod 的 GI/CI，随后补充 MIG UUID、GI ID 和 CI ID；用户不得自行创建或修改此注解。 +- Pod 删除、成功或失败后，device plugin 会回收该 Pod 对应的 CI/GI。重启时，具有完整运行时身份的活动实例会经 NVML 校验后被接管；旧模板/slot 标识无法安全接管。 +- 当前 MIG 指标为 `hami_node_gpu_mig_instance_info`，包含 profile、placement、MIG UUID、GI ID 和 CI ID。仅当 `legacyMetrics: true` 时才额外输出旧指标 `nodeGPUMigInstance`，Chart 默认值为 `false`。 +- 从旧 `knownMigGeometries` 升级时，需要逐节点 cordon、drain，并先升级调度器再升级 device plugin。NVIDIA MIG Manager 与 HAMi 都会修改 GI/CI，不能同时管理同一块物理 GPU；GPU Operator 可继续提供驱动、Container Toolkit 和 DCGM，但目标 GPU 上的 MIG Manager reconciliation 必须停止。 + +::: + ## 介绍 **我们现在支持通过使用 mig-parted 动态调整 mig-devices 来支持 dynamic-mig**，包括： @@ -25,7 +37,7 @@ translated: true ## 启用 Dynamic-mig 支持 -- 使用 Helm 安装 Chart，参见[此处](https://github.com/Project-HAMi/HAMi#enabling-vgpu-support-in-kubernetes)的“在 Kubernetes 中启用 vGPU 支持”部分 +- 使用 Helm 安装 Chart，参见[在线安装指南](../../installation/online-installation.md) - 在 device-plugin ConfigMap 中将 `mode` 配置为 `mig` 以支持 MIG 节点 @@ -54,7 +66,7 @@ kubectl describe cm hami-device-plugin -n kube-system ## 自定义 MIG 配置（可选） -HAMi 目前有一个 [内置的 MIG 配置](https://github.com/Project-HAMi/HAMi/blob/master/charts/hami/templates/scheduler/device-configmap.yaml) 用于 MIG。 +HAMi 目前有一个 [内置的 MIG 配置](https://github.com/Project-HAMi/HAMi/blob/v2.10.0/charts/hami/templates/scheduler/device-configmap.yaml) 用于 MIG。 你可以按照以下步骤自定义 MIG 配置： diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/examples/dynamic-mig-example.md b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/examples/dynamic-mig-example.md index a381f7f67..0af29cfdd 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/examples/dynamic-mig-example.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/userguide/nvidia-device/examples/dynamic-mig-example.md @@ -3,6 +3,12 @@ title: 将任务分配给 MIG 实例 translated: true --- +:::note HAMi v2.10 当前行为 + +下方 A100 profile 是 GPU 空闲时的常见结果，并不是固定模板。v2.10 会为两个设备请求分别选择显存满足 8000 MiB、已列入 `migProfileAllowlist`、由 NVML 发现且具有合法空闲 placement 的最小 profile；现有预留可能改变结果或使 Pod 保持 Pending。HAMi 会管理内部注解 `hami.io/vgpu-mig-allocations`，在 `Allocate` 时按预留创建 GI/CI，并在 Pod 结束后回收；用户不得设置或修改该注解。 + +::: + 此示例将为 A100-40GB-PCIE 设备分配 `2g.10gb * 2` 或为 A100-80GB-SXM 设备分配 `1g.10gb * 2`。 ```yaml