diff --git a/docs/develop/amd-vgpu.md b/docs/develop/amd-vgpu.md index ae570d2fcb..587e6df978 100644 --- a/docs/develop/amd-vgpu.md +++ b/docs/develop/amd-vgpu.md @@ -7,8 +7,10 @@ AMD ROCm GPUs, so multiple pods can share one Instinct GPU. ## 2. Goals -- Fractional allocation by memory (`amd.com/gpumem`, MB) and compute (`amd.com/gpucores`, CU count). +- Fractional allocation by memory (`amd.com/gpumem`, MiB) and compute (`amd.com/gpucores`, percentage). - Exclusive, non-overlapping CU partitioning across pods. +- Support all AMD ROCm GPU types; initial CU masking starts on non-WGP + (CDNA) devices, then extends to WGP-capable (RDNA) with pair-aligned masks. - Following HAMi's existing architecture design to achieve the functionality. ## 3. Approach @@ -20,12 +22,31 @@ Switching to LD_AUDIT (`la_symbind64`), which intercepts only cross-library bind resolved it. The existing NVIDIA LD_PRELOAD path is unchanged. **Advantages of CU masking, compared to hardware partitioning (CPX/NPS).** -Masking (`ROC_GLOBAL_CU_MASK`) assigns an arbitrary and fine-grained per-pod CU partitioning at container start ([AMD Docs](https://rocm.docs.amd.com/en/latest/how-to/setting-cus.html)). -On the other hand, hardware partitioning (CPX/NPS) slices only at fixed XCD granularity and is set per physical GPU. +Masking (`HSA_CU_MASK`) assigns fine-grained, hardware-valid per-pod CU +partitioning at container start ([AMD Docs](https://rocm.docs.amd.com/en/latest/how-to/setting-cus.html)). +On the other hand, hardware partitioning (CPX/NPS) slices only at fixed XCD +granularity and is set per physical GPU. + +**WGP pairing and rollout scope.** ROCm documents that not every CU mask is +valid on every device: on GPUs where two CUs form a Work Group Processor +(WGP) and kernels run in WGP mode, disabling only one CU of a pair is +invalid ([setting CUs](https://rocm.docs.amd.com/en/latest/how-to/setting-cus.html)). +WGP is an **RDNA** construct (GFX10+); HIP describes it under the RDNA hardware +model ([HIP hardware implementation](https://rocm.docs.amd.com/projects/HIP/en/latest/understand/hardware_implementation.html)). +**CDNA** devices (e.g. Instinct MI300X, `gfx942`) keep independent CUs and are +not subject to that pairing rule. + +The design aims to support **all AMD ROCm GPU types**. Initial implementation +starts on **non-WGP** devices (CDNA / Instinct), where CU ranges can be chosen +without pair alignment. Support for WGP-capable (RDNA) devices follows later +and must select CU masks in adjacent pairs so `HSA_CU_MASK` remains +hardware-valid. ## 4. Protocol (scheduler <-> device-plugin) -For AMD, it adds one AMD-specific annotation for the CU bitmap. +For AMD, the scheduler writes AMD-specific pod annotations; the device-plugin +injects `ROCR_VISIBLE_DEVICES`, `HSA_CU_MASK`, and `HIP_DEVICE_MEMORY_LIMIT` +into each container's allocation response. ### 4.1 Node registration (device-plugin -> node annotations) @@ -47,45 +68,46 @@ Registered under `hami.io/node-amd-register`, in JSON format — an array of `De ] ``` -- `devmem` = total device memory in MB (e.g. 196608 for MI300X). +- `devmem` = total device memory in MiB (e.g. 196608 for MI300X). - `devcore` = total CU count (e.g. 304 for MI300X). - `id` = device identifier. ### 4.2 Pod allocation (scheduler -> pod annotations -> Allocate) -The allocation result is written under the **AMD-specific** key (each vendor has its own): +The scheduler allocation result is written under the **AMD-specific** keys: ```text -hami.io/amd-devices-allocated: ,AMDGPU,,:; +hami.io/amd-devices-to-allocate: ,AMDGPU,,:; +hami.io/amd-devices-allocated: ,AMDGPU,,:; ``` -Plus, a dedicated AMD annotation carrying the per-device CU bitmap. Following the -convention other vendors use for allocation data that does not fit the standard -`UUID,Type,mem,cores` encoding (e.g. Ascend's `huawei.com/`), -this is a separate annotation under the **`amd.com/`** namespace with a JSON value, for example: +The type field uses the existing `AMDGPU` constant (see `pkg/device/amd/device.go`). -```json -amd.com/cu-mask: [{"uuid":"","cu_mask":"0x337f"},{"uuid":"","cu_mask":"0x00ff"}] -``` +During `Allocate`, the device-plugin reads `hami.io/amd-devices-allocated` +from the pod annotations, converts each device's `cuCount` into a +non-overlapping CU range, and returns the result in the container's +`ContainerAllocateResponse.Envs` (not as a pod annotation): -`cu_mask` uses ROCm's hex-bitmask form of `CU_list` (`0x[0-F]*`, e.g. `0x337f`; see -). -A JSON value avoids inventing a delimiter scheme (and the :/; collision with -`ROC_GLOBAL_CU_MASK`'s own grammar). +- `ROCR_VISIBLE_DEVICES` — limits which GPUs the container sees; UUID order + defines the container-local GPU index. +- `HSA_CU_MASK` — restricts CUs per visible GPU, using `GPU_list:CU_list` + where the GPU index is resolved **after** `ROCR_VISIBLE_DEVICES` reordering + (index `0` = first visible GPU, `1` = second, …). -The device-plugin translates each entry into the per-GPU `ROC_GLOBAL_CU_MASK` -at container start. +For a multi-GPU pod, the device-plugin pairs each allocated UUID with its +container-local index when building `HSA_CU_MASK`, for example: -Exclusivity of CU bitmaps across pods on a device will be enforced -under a node lock (`AMDDevices.LockNode` and `ReleaseNodeLock`) so multiple pods never receive overlapping masks. -(These are currently stubs; the node-lock enforcement is not yet implemented — TODO.) - -Optional handshake: ```text -amd.com/cu-mask-assigned: "false" -> "true" (set by device-plugin) +# two GPUs: UUID-A (index 0) gets CUs 0-75, UUID-B (index 1) gets CUs 0-75 +HSA_CU_MASK=0:0-75;1:0-75 ``` +Each `CU_list` uses HSA's CU ID-list grammar, for example `0-3,8,10-12`. + +Exclusivity of CU ranges across pods on a device is enforced under the AMD +node lock (`AMDDevices.LockNode` and `ReleaseNodeLock` which are unimplemented now). + ## 5. Resource model and core_limit -> CU mask Example Pod request: @@ -94,39 +116,55 @@ Example Pod request: resources: limits: amd.com/gpu: 1 # number of physical AMD GPUs - amd.com/gpumem: 16384 # MB - amd.com/gpucores: 152 # CU count + amd.com/gpumem: 16384 # MiB + amd.com/gpucores: 25 # percentage of physical CUs ``` ### 5.1 Memory limit -`amd.com/gpumem` (MB) flows through the shared annotation, is -injected by the device plugin as `HIP_DEVICE_MEMORY_LIMIT_` (value `m`). +`amd.com/gpumem` (MiB) flows through the shared annotation and is injected by +the device plugin as `HIP_DEVICE_MEMORY_LIMIT=m`. HAMi's AMD LD_AUDIT +layer enforces this limit at the HIP API boundary; the value must use the +`m` format. The limit is scoped to the container and applies to all GPUs +mounted in that container. Each container receives an independent allocation +response and environment. -### 5.2 Core limit -> CU mask (Discussion needed) +### 5.2 Core limit -> CU mask -`amd.com/gpucores` is a **CU count**, not a percentage (contrast NVIDIA's -SM-utilization %). -And converting a count into a usable `ROC_GLOBAL_CU_MASK` is not a -simple "set N bits" operation: +`amd.com/gpucores` is a **percentage** in the inclusive range `1`–`100` (values +outside this range are rejected at admission), not a CU count. The scheduler +converts the requested percentage into a physical CU count with: -1. **The hard guarantee is non-overlap.** The scheduler - assigns each pod a CU bitmap that does **not overlap** any other pod's bitmap on the same device. - This will be enforced via the bitmap allocator under a node lock (`LockNode` / `ReleaseNodeLock`), which currently return `nil` (TODO: implement). +```text +cuCount = floor(percentage × devcore / 100) +``` -2. **Zero-interference is not guaranteed.** Even with non-overlapping - masks, residual interference remains (as reported in #1707). +where `devcore` is the device's total CU count from node registration. The +result is clamped to `[1, devcore]`. For example, `25` on a 304-CU MI300X +yields `76` CUs; `33` yields `100` CUs (`floor(100.32)`); `67` yields +`203` CUs (`floor(203.68)`). -The count -> bitmap conversion is a **scheduler's** responsibility, because non-overlap requires knowledge of the device's current -allocation state. +The scheduler records this `cuCount` in `hami.io/amd-devices-allocated`. The +device-plugin selects a non-overlapping CU range of that size and injects it +as `HSA_CU_MASK`. On WGP-capable devices, the chosen range must additionally +align to adjacent CU pairs within each WGP. -The device-plugin simply passes the scheduler-decided mask through verbatim as `ROC_GLOBAL_CU_MASK`. +**Zero-interference is not guaranteed.** Even with non-overlapping masks, +residual interference remains (as reported in #1707). ## 6. Known limitations - **No `amd-smi` / `rocm-smi` virtualization.** These read sysfs/drm, not HIP, so LD_AUDIT cannot intercept them; in-container tools may report physical resources. +- **Mixed GPU types are not supported on one node.** + The device plugin derives the GPU type from `amd.com/gpu.product-name`, but + this label cannot describe multiple GPU types. +- **WGP-aware CU allocation is phased.** First ship CU partitioning on + non-WGP devices (CDNA / Instinct). RDNA (GFX10+) devices need WGP pair + alignment when building `HSA_CU_MASK` + ([setting CUs](https://rocm.docs.amd.com/en/latest/how-to/setting-cus.html)) + and land in a follow-up. ## 7. Discussion points -- **Device-plugin layering.** In #1707, the AMD vGPU device-plugin is proposed to be built on ROCm/k8s-device-plugin, which already advertises whole-GPU `amd.com/gpu`. Since kubelet cannot register the same resource from two plugins, the natural path is to **extend the ROCm plugin** so a single plugin owns `amd.com/gpu` and also advertises the fractional `amd.com/gpumem` / `amd.com/gpucores` (to be confirmed). +- **Device-plugin layering.** In #1707, the AMD vGPU device-plugin is proposed to be built on ROCm/k8s-device-plugin, which already advertises whole-GPU `amd.com/gpu`. Since kubelet cannot register the same resource from two plugins, the natural path is to **extend the ROCm plugin** so a single plugin owns `amd.com/gpu` and also advertises the fractional `amd.com/gpumem` / `amd.com/gpucores` (optional, which is not a must-have for HAMi but can be useful for other schedulers).