From f313736e07140f6985a3a1b74c8634528351a030 Mon Sep 17 00:00:00 2001 From: Saiyam Pathak Date: Tue, 7 Jul 2026 11:11:07 +0530 Subject: [PATCH 1/2] Add Lab 7: GPU isolation on k3s without the GPU Operator Adds a new hands-on lab (English + Chinese) that builds a single-node k3s cluster on a cloud GPU VM, installs HAMi without the NVIDIA GPU Operator with nvidia as the default containerd runtime, and proves memory isolation end to end: virtualized nvidia-smi inside each pod, a CUDA allocation refused at the slice by HAMi-core, an oversubscribing pod kept Pending with CardInsufficientMemory, and the libvgpu.so /etc/ld.so.preload injection mechanism. Every output was captured from a live run on a GCP g4-standard-48 Spot VM with one 96 GB NVIDIA RTX PRO 6000 Blackwell (k3s v1.36.2+k3s1, HAMi v2.9.0, driver 610.43.02). The lab design is adapted from Lovedeep Singh's (@ld-singh) AI Factory Operations Lab with his permission, credited in the lab. Closes #559 Signed-off-by: Saiyam Pathak --- .../current/labs/hami-isolation-k3s.md | 385 ++++++++++++++++++ .../current/overview.md | 2 +- sidebars-tutorials.js | 5 + .../oversubscribe-pending.yaml | 28 ++ .../07-hami-isolation-k3s/share-two-pods.yaml | 35 ++ tutorials/labs/hami-isolation-k3s.md | 385 ++++++++++++++++++ tutorials/overview.md | 2 +- 7 files changed, 840 insertions(+), 2 deletions(-) create mode 100644 i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/hami-isolation-k3s.md create mode 100644 tutorials/labs/examples/07-hami-isolation-k3s/oversubscribe-pending.yaml create mode 100644 tutorials/labs/examples/07-hami-isolation-k3s/share-two-pods.yaml create mode 100644 tutorials/labs/hami-isolation-k3s.md diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/hami-isolation-k3s.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/hami-isolation-k3s.md new file mode 100644 index 000000000..ef70a1705 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/hami-isolation-k3s.md @@ -0,0 +1,385 @@ +--- +title: "实验 7: 在 k3s 上不使用 GPU Operator 实现 GPU 隔离" +description: "在单节点 k3s 上让多个 Pod 共享一张非 MIG GPU，并证明 HAMi-core 强制执行显存上限。" +sidebar_label: "实验 7: k3s 隔离" +lab: + level: Intermediate + duration: 约 45 分钟 + environment: 云 GPU 虚拟机上的单节点 k3s（任意非 MIG NVIDIA 显卡） + cost: GPU 费用不到 2 美元 + authors: + - saiyam1814 + verified: "2026-07-07" +tags: + - 隔离 + - k3s +toc_max_heading_level: 2 +--- + +本实验在租用的 GPU 虚拟机上搭建单节点 k3s 集群，**不安装** NVIDIA GPU Operator 直接安装 HAMi，并证明 HAMi 的显存隔离是真实生效的：两个 Pod 共享一张物理显卡，每个 Pod 内的 `nvidia-smi` 只报告自己的切片大小，超出切片的 CUDA 分配会被 HAMi-core 拒绝（此时显卡上仍有几十 GB 空闲显存），而一个会导致超额分配的第三个 Pod 则保持 `Pending` 状态。 + +本实验中的每条命令和输出均采集自 GCP `g4-standard-48` Spot 虚拟机上的真实运行（一张 96 GB NVIDIA RTX PRO 6000 Blackwell，单节点 k3s v1.36.2+k3s1、HAMi v2.9.0、NVIDIA 驱动 610.43.02、Ubuntu 22.04）。任何非 MIG 显卡的行为相同；只需调整切片大小（随显存容量缩放）。RTX PRO 6000 Blackwell 支持 MIG，但出厂默认禁用——本实验验证的正是 HAMi 的软件共享。 + +## 与实验 3 的区别 + +[实验 3](./gpu-partitioning.md) 在由 **GPU Operator** 提供驱动和容器工具包、HAMi 设备插件叠加其上的集群中证明了相同的隔离特性。本实验走另一条受支持的路径：**完全不使用 GPU Operator**。HAMi 自带设备插件，NVIDIA Container Toolkit 直接安装在主机上，并将 `nvidia` 设为 containerd 的**默认**运行时，使 HAMi-core 被注入到每个 GPU Pod 中。这是在边缘节点、裸金属机器或廉价租用 GPU 虚拟机上会采用的更精简的部署方式——并且它能直接展示 Operator 路径所隐藏的底层机制（步骤 7）。 + +## 你将学到什么 + +- 搭建带 NVIDIA Container Toolkit 且以 `nvidia` 为默认运行时的单节点 k3s +- 不使用 GPU Operator 安装 HAMi，并让调度器镜像标签与服务端版本匹配 +- HAMi 在哪里记录可共享的 GPU 显存（`hami.io/node-nvidia-register` 注解，而非节点 allocatable） +- 证明显存上限真实生效：虚拟化的 `nvidia-smi` 与在切片处被拒绝的 CUDA 分配 +- 证明按设备核算：超额分配的 Pod 保持 `Pending`，事件为 `CardInsufficientMemory` +- 上限如何被强制执行：HAMi-core（`libvgpu.so`）注册在 Pod 的 `/etc/ld.so.preload` 中（软件隔离，非 MIG） + +## 实验概览 + +```mermaid +%% title: k3s 隔离实验步骤 +flowchart LR + Step1["步骤 1
主机准备"] --> Step2["步骤 2
k3s + 工具包"] + Step2 --> Step3["步骤 3
nvidia 默认运行时"] + Step3 --> Step4["步骤 4
安装 HAMi"] + Step4 --> Step5["步骤 5
两个 Pod 共享一张 GPU"] + Step5 --> Step6["步骤 6
上限 + 预算测试"] + Step6 --> Step7["步骤 7
底层机制"] +``` + +## 前提条件 + +- 一台全新的云虚拟机（Ubuntu 22.04 或更高版本），配备**一张 NVIDIA GPU** 并具有 root 权限。目标是任意未启用 MIG 的显卡：RTX PRO 6000、RTX A6000、L4、L40/L40S、RTX 4090/3090。对无法（或没有）通过 MIG 分区的显卡进行切分正是 HAMi 的核心使用场景。 +- 主机上的 NVIDIA 驱动可正常工作（`nvidia-smi` 执行成功）。大多数 GPU 虚拟机镜像已内置驱动。 +- 你能控制容器运行时：无法重新配置的受限市场容器不可用，因为 HAMi-core 是通过 NVIDIA 容器运行时注入的。 +- 所有操作**通过 SSH 在虚拟机上**执行——helm 和 kubectl 通过 localhost 与 k3s 通信，避免不稳定的远程链路。 +- 来自 [`tutorials/labs/examples/07-hami-isolation-k3s/`](https://github.com/Project-HAMi/website/tree/master/tutorials/labs/examples/07-hami-isolation-k3s) 的清单文件 + +:::note[费用] + +一次完整的实验会话远低于一小时 GPU 时间。非 MIG 显卡的按需租用价格通常低于每小时 1 美元。GPU 资源紧张——你想要的那张卡经常缺货；选用任何可用的非 MIG 显卡并调整切片大小即可（清单文件中只需改一行）。 + +::: + +:::warning[不要安装 GPU Operator] + +HAMi 自带设备插件，[不能与 NVIDIA 官方设备插件共存](https://project-hami.io/docs/installation/prerequisites)。GPU Operator + HAMi 的集成没有官方文档支持（[HAMi #1708](https://github.com/Project-HAMi/HAMi/issues/1708)）。如果你的集群已经运行了 Operator，请改用实验 3 的路径——或为本实验使用一台全新的虚拟机。 + +::: + +## 步骤 1: 验证主机驱动 + +没有健康的主机驱动，后续步骤都无法进行。以 root 身份（或全程使用 `sudo`）SSH 登录虚拟机并检查： + +```bash +nvidia-smi -L +``` + +```plaintext +GPU 0: NVIDIA RTX PRO 6000 Blackwell Server Edition (UUID: GPU-3c4a3856-fbb4-1425-9679-aed25f4d2977) +``` + +> 一张显卡，主机可见。如果 `nvidia-smi` 失败，请按照 [NVIDIA 驱动安装指南](https://docs.nvidia.com/datacenter/tesla/driver-installation-guide/)安装驱动后重试。在全新的 Ubuntu 虚拟机上，这意味着添加 NVIDIA CUDA apt 仓库并执行 `apt-get install nvidia-open`——Blackwell 一代显卡需要**开源**内核模块（驱动 ≥ 570）；使用闭源模块时 `nvidia-smi` 会报 `No devices found`。 + +## 步骤 2: 先安装 Container Toolkit，再安装 k3s + +顺序很重要：在安装 k3s **之前**先安装 [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)，这样 k3s 在安装时会自动检测到 `nvidia` 容器运行时。 + +```bash +install -m 0755 -d /usr/share/keyrings +curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \ + | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg +curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \ + | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \ + > /etc/apt/sources.list.d/nvidia-container-toolkit.list +apt-get update -y +apt-get install -y nvidia-container-toolkit +``` + +现在安装 k3s： + +```bash +curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--write-kubeconfig-mode 644" sh - +export KUBECONFIG=/etc/rancher/k3s/k3s.yaml +kubectl get nodes -o wide +``` + +```plaintext +NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME +hami-lab-rtx6000 Ready control-plane 65s v1.36.2+k3s1 10.128.0.2 Ubuntu 22.04.5 LTS 6.8.0-1063-gcp (amd64) containerd://2.3.2-k3s2 +``` + +确认 k3s 识别到了 nvidia 运行时，并为 HAMi 打上节点标签： + +```bash +kubectl get runtimeclass nvidia +kubectl label node hami-lab-rtx6000 gpu=on +``` + +```plaintext +NAME HANDLER AGE +nvidia nvidia 55s +node/hami-lab-rtx6000 labeled +``` + +> 当工具包在 k3s 安装时已存在，k3s 会在其 containerd 配置中写入 `nvidia` RuntimeClass。如果缺失，重启 k3s（`systemctl restart k3s`）——参见 [k3s NVIDIA 运行时支持](https://docs.k3s.io/advanced#nvidia-container-runtime-support)。HAMi 的设备插件会调度到带有 `gpu=on` 标签的节点上。 + +## 步骤 3: 将 nvidia 设为 containerd 的默认运行时 + +HAMi 的 Pod 不设置 `runtimeClassName`，因此只有当**默认**运行时是 `nvidia` 时，HAMi-core 才会被注入。使用 k3s 自带的 `default-runtime` 选项——这是与 containerd 配置模式无关的干净做法。**不要**手动编辑 `config.toml`：v2 和 v3 的模式不同，重复声明已有的表会直接破坏 k3s。 + +```bash +echo 'default-runtime: nvidia' >> /etc/rancher/k3s/config.yaml +systemctl restart k3s +``` + +验证配置已生效（此文件由 k3s 重新生成；你从不直接编辑它）： + +```bash +grep default_runtime_name /var/lib/rancher/k3s/agent/etc/containerd/config.toml +``` + +```plaintext +default_runtime_name = "nvidia" +``` + +> 这一行就是 HAMi 强制执行限制与静默不执行之间的区别。如果之后在步骤 7 中 GPU Pod 里看不到 `libvgpu.so`，请先回到这里排查。 + +## 步骤 4: 安装 HAMi + +HAMi 调度器运行一个 kube-scheduler sidecar，其镜像标签必须与集群 Kubernetes **服务端**版本匹配——版本不匹配是最常见的 HAMi 安装失败原因。先检测版本： + +```bash +kubectl version | grep Server +``` + +```plaintext +Server Version: v1.36.2+k3s1 +``` + +去掉 `+k3s1` 后缀，将其作为调度器镜像标签传入： + +```bash +helm repo add hami-charts https://project-hami.github.io/HAMi +helm repo update hami-charts +helm upgrade --install hami hami-charts/hami \ + --version 2.9.0 \ + -n kube-system \ + --set scheduler.kubeScheduler.imageTag=v1.36.2 \ + --wait --timeout 5m +kubectl -n kube-system get pods | grep hami +``` + +```plaintext +hami-device-plugin-62ck6 2/2 Running 0 38s +hami-scheduler-5f5b5589c9-zhgsc 2/2 Running 0 38s +``` + +> 与实验 2（禁用设备插件）不同，此处 **HAMi 设备插件正常运行**——它是集群上唯一的设备插件，独占这张 GPU。 + +验证 HAMi 已注册显卡： + +```bash +kubectl get node hami-lab-rtx6000 -o jsonpath='{.status.allocatable.nvidia\.com/gpu}'; echo +kubectl get node hami-lab-rtx6000 -o jsonpath='{.metadata.annotations.hami\.io/node-nvidia-register}'; echo +``` + +```plaintext +10 +[{"id":"GPU-3c4a3856-fbb4-1425-9679-aed25f4d2977","count":10,"devmem":97887,"devcore":100,"type":"NVIDIA RTX PRO 6000 Blackwell Server Edition","mode":"hami-core","health":true,"devicepairscore":{}}] +``` + +> 有两点值得牢记： +> +> - `nvidia.com/gpu` 的 allocatable 值是 `10`——一张物理 GPU × `deviceSplitCount`（默认 10），即可以共享这张卡的 Pod 数量上限。 +> - 可共享的显存（`devmem: 97887` MiB）记录在 `hami.io/node-nvidia-register` 注解中，**而非**节点 allocatable。`kubectl describe node` 中不会出现 `nvidia.com/gpumem`；HAMi 调度器和 webhook 依据该注解核算 `gpumem`/`gpucores`，由 HAMi-core 按 Pod 强制执行。 + +## 步骤 5: 两个 Pod 共享一张物理 GPU + +`share-two-pods.yaml` 运行两个 Pod，每个请求一张 GPU 和 8000 MiB 的切片——原生 Kubernetes 做不到这一点（它会把整张卡分配给一个 Pod）： + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: hami-share-a +spec: + restartPolicy: Never + containers: + - name: cuda + image: nvidia/cuda:12.4.1-devel-ubuntu22.04 + command: ["bash", "-c", "nvidia-smi; sleep infinity"] + resources: + limits: + nvidia.com/gpu: 1 + nvidia.com/gpumem: 8000 +``` + +> `hami-share-b` 除名称外完全相同。使用 `devel` 镜像是因为步骤 6 需要在 Pod 内用 `nvcc` 编译一个小型 CUDA 分配器；镜像较大、拉取较慢是正常的。在 24 GB 显卡上请改用约 4000 MiB 的切片。 + +```bash +kubectl apply -f share-two-pods.yaml +kubectl wait --for=condition=Ready pod/hami-share-a pod/hami-share-b --timeout=300s +kubectl get pods -o wide +``` + +```plaintext +NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES +hami-share-a 1/1 Running 0 37s 10.42.0.13 hami-lab-rtx6000 +hami-share-b 1/1 Running 0 37s 10.42.0.14 hami-lab-rtx6000 +``` + +> 两个 Pod 都在同一台单 GPU 节点上 `Running`：共驻。它们由 hami-scheduler 放置；每个 Pod 的调度决策在事件中可见： + +```bash +kubectl describe pod hami-share-a | grep -A5 Events: +``` + +```plaintext +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Scheduled 52s hami-scheduler Successfully assigned default/hami-share-a to hami-lab-rtx6000 + Normal FilteringSucceed 53s hami-scheduler find fit node(hami-lab-rtx6000), 0 nodes not fit, 1 nodes fit(hami-lab-rtx6000:0.00) + Normal BindingSucceed 52s hami-scheduler Successfully binding node [hami-lab-rtx6000] to default/hami-share-a +``` + +## 步骤 6: 证明上限真实生效 + +在 `hami-share-a` 中连续做两项检查。首先，容器*认为*这张 GPU 是什么样的？ + +```bash +kubectl exec hami-share-a -- nvidia-smi +``` + +```plaintext +Tue Jul 7 10:25:15 2026 ++-----------------------------------------------------------------------------------------+ +| NVIDIA-SMI 610.43.02 KMD Version: 610.43.02 CUDA UMD Version: 13.3 | ++-----------------------------------------+------------------------+----------------------+ +| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | +| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | +| | | MIG M. | +|=========================================+========================+======================| +| 0 NVIDIA RTX PRO 6000 Blac... Off | 00000000:05:00.0 Off | 0 | +| N/A 34C P0 48W / 600W | 0MiB / 8000MiB | 0% Default | +| | | Disabled | ++-----------------------------------------+------------------------+----------------------+ +``` + +> Memory 列显示 **8000MiB**，而不是显卡真实的 97887MiB。HAMi-core 重写了容器看到的显卡容量。注意 `MIG M.: Disabled`——这里的共享是 HAMi 的软件切片，而非硬件分区。 + +其次，上限真的守得住吗？编译并运行一个每次 `cudaMalloc` 256 MiB、直到被拒绝为止的小型分配器： + +```bash +kubectl exec hami-share-a -- bash -c ' +cat > /tmp/probe.cu < +#include +int main() { + const size_t chunk = (size_t)256 * 1024 * 1024; // 256 MiB + size_t total = 0; + void *p = nullptr; + for (;;) { + cudaError_t e = cudaMalloc(&p, chunk); + if (e != cudaSuccess) { + printf("cudaMalloc refused after %zu MiB allocated: %s\n", + total / (1024 * 1024), cudaGetErrorString(e)); + return 0; + } + total += chunk; + } +} +EOF +nvcc -o /tmp/probe /tmp/probe.cu && /tmp/probe' +``` + +```plaintext +cudaMalloc refused after 7424 MiB allocated: out of memory +[HAMI-core ERROR (pid:54 thread=133390658244608 allocator.c:52)]: Device 0 OOM 8629780480 / 8388608000 +``` + +> 这就是证明，而且是一个矛盾：Pod 在约 7.4 GB 处触发 "out of memory"，而物理显卡上还有约 82 GB 空闲。拒绝来自 **HAMI-core**（见错误行：它试图使用 8629780480 字节，超过了 8388608000 字节的限制——8388608000 字节恰好是 8000 MiB），而非硬件。只有拦截 CUDA 调用的软件上限才能做到这一点。表述时应说"超出切片的分配会被拒绝"——精确边界取决于分配器和 CUDA 上下文开销，不要断言确切的字节数。可以对 `hami-share-b` 重复此操作；行为完全一致。 + +现在证明显卡的显存是一个共享的、有限的预算。`oversubscribe-pending.yaml` 请求一个 90000 MiB 的切片——空的 96 GB 卡放得下，但在已有两个 8000 MiB 切片的情况下放不下（97887 − 16000 ≈ 82000 MiB 空闲）： + +```bash +kubectl apply -f oversubscribe-pending.yaml +sleep 15 +kubectl get pod hami-oversubscribe +kubectl describe pod hami-oversubscribe | grep -A5 Events: +``` + +```plaintext +NAME READY STATUS RESTARTS AGE +hami-oversubscribe 0/1 Pending 0 15s +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Warning FailedScheduling 15s hami-scheduler 0/1 nodes are available: 1 NodeUnfitPod. no new claims to deallocate, preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod. + Warning FilteringFailed 16s (x2 over 16s) hami-scheduler 1 nodes CardInsufficientMemory(hami-lab-rtx6000) +``` + +> `Pending` 且事件为 `CardInsufficientMemory`——与步骤 5 形成对比，同一个调度器在那里记录的是 `FilteringSucceed`。有两点必须正确，否则这个测试会"失败"（被调度而不是保持 Pending）：请求大小必须适配**你的**显卡（大于切片旁的剩余空间、小于整张卡——见清单注释），并且两个共享 Pod 必须仍处于 `Running` 状态（它们持有切片；清单使用 `sleep infinity`，避免它们在实验中途悄然完成）。 + +## 步骤 7: 揭示底层机制 + +上限是如何被强制执行的？在被限制的 Pod 内查看运行时注入了什么： + +```bash +kubectl exec hami-share-a -- bash -c 'env | grep -iE "CUDA_DEVICE|NVIDIA_VISIBLE|LD_PRELOAD" | sort; cat /etc/ld.so.preload; ls -l /usr/local/vgpu/ 2>/dev/null' +``` + +```plaintext +CUDA_DEVICE_MEMORY_LIMIT_0=8000m +CUDA_DEVICE_MEMORY_SHARED_CACHE=/usr/local/vgpu/5af6befa-5ff5-4bd5-9086-c79554723b6f.cache +CUDA_DEVICE_SM_LIMIT=0 +NVIDIA_VISIBLE_DEVICES=GPU-3c4a3856-fbb4-1425-9679-aed25f4d2977 +/usr/local/vgpu/libvgpu.so +total 684 +-rw-rw-rw- 1 root root 2008953 Jul 7 10:25 5af6befa-5ff5-4bd5-9086-c79554723b6f.cache +-rwxr-xr-x 1 root root 684264 Jul 7 10:23 libvgpu.so +``` + +> 整个机制就浓缩在这几行里：`libvgpu.so`（HAMi-core）被挂载进 Pod 并注册在 `/etc/ld.so.preload` 中，因此容器内的**每个**进程都会加载它；HAMi-core 从 `CUDA_DEVICE_MEMORY_LIMIT_0` 读取上限。应用发出的每个 CUDA 驱动调用都会经过这个库，步骤 6 中的拒绝正是来自这里。（`CUDA_DEVICE_SM_LIMIT=0` 表示未请求算力限制——这些 Pod 没有设置 `gpucores`；共享缓存文件是各 HAMi-core 实例跨进程核算用量的方式。）这与 NVIDIA 的 KAI Scheduler 于 [2026 年 6 月采用](https://github.com/NVIDIA/KAI-Scheduler/pull/60)的用于分数 GPU 显存隔离的 `CUDA_DEVICE_MEMORY_LIMIT` 机制完全相同。具体的变量名和路径随 HAMi 版本而异——请记录你实际看到的内容。 + +:::note[这是什么，不是什么] + +这是**软件隔离**：用户态 CUDA 拦截。它不是 MIG 的硬件故障隔离——行为异常的 kernel 受到的是拦截约束，而非硬件分区。应将其视为带运行时强制执行的调度与核算保证，而不是等同于 MIG 的安全边界。本实验也不测量算力限制精度或持续负载下的邻居干扰；`gpucores` 的行为参见实验 3 的步骤 5。 + +::: + +## 清理 + +```bash +kubectl delete pod hami-share-a hami-share-b hami-oversubscribe +helm -n kube-system uninstall hami +``` + +如果你为本实验租用了虚拟机，请保存输出后销毁实例——计费仍在进行。 + +```bash +/usr/local/bin/k3s-uninstall.sh # 可选：完全移除 k3s +``` + +## 本实验证明了什么 + +| 结论 | 证据 | +| --- | --- | +| HAMi 无需 GPU Operator 即可运行 | 集群上只有 HAMi 自己的设备插件；`nvidia.com/gpu` allocatable 正常出现 | +| 两个 Pod 共享一张物理 GPU | `hami-share-a` 和 `hami-share-b` 都在单 GPU 节点上 `Running` | +| 容器只看到自己的切片 | Pod 内 `nvidia-smi` 报告总量 8000 MiB，而非 97887 MiB | +| 显存上限是强制的，不是装饰 | `cudaMalloc` 在约 7424 MiB 处被 HAMI-core 拒绝，此时显卡还有约 82 GB 空闲 | +| 显卡显存是统一核算的预算 | 90000 MiB 的 Pod 保持 `Pending`，事件为 `CardInsufficientMemory` | +| 机制是用户态拦截 | Pod 中注入了 `/etc/ld.so.preload` 里的 `libvgpu.so` 和 `CUDA_DEVICE_MEMORY_LIMIT_0=8000m` | + +## 下一步 + +- 运行 [实验 2: 本地模拟 GPU](./local-fake-gpu.md)，免费、无需 GPU 地学习这个故事的调度部分——模拟证明放置决策，本实验证明强制执行。 +- 与 [实验 3: GPU 分区](./gpu-partitioning.md) 对比，它通过 GPU Operator 路径得出相同的隔离证明，并额外覆盖 `gpucores` 算力限制。 +- 阅读 [HAMi 集群架构](/zh/docs/core-concepts/hami-architecture)，端到端梳理你刚刚验证过的 webhook → 调度器 → 设备插件 → HAMi-core 链路。 + +:::info[致谢] + +本实验改编自 [Lovedeep Singh](https://github.com/ld-singh) 的 [AI Factory Operations Lab](https://ld-singh.github.io/ai-factory-ops-lab/) 中的 HAMi 课程，经其友好授权。该课程将无需 GPU 的调度模拟（本站由 [实验 2](./local-fake-gpu.md) 覆盖）与真实 GPU 隔离证明配对。他最早在一张 48 GB RTX A6000 上验证了这套练习序列（[验证报告](https://github.com/ld-singh/ai-factory-ops-lab/blob/main/portfolio-lab/06-validation-reports/hami-isolation-validation.md)）。 + +::: diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md index 1e088c73a..1bfe5d723 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md @@ -19,4 +19,4 @@ import LabCardGridAuto from '@site/src/components/labs/LabCardGridAuto'; -每个实验都列出了各自的前提条件。实验 3 和 4 直接复用实验 1 搭建的集群，一次开机即可完成全部三个实验；实验 2 可在任意笔记本上运行，无需 GPU。 +每个实验都列出了各自的前提条件。实验 3 和 4 直接复用实验 1 搭建的集群，一次开机即可完成全部三个实验；实验 2 可在任意笔记本上运行，无需 GPU。实验 7 在租用的 GPU 虚拟机上自行搭建单节点 k3s 集群，不使用 GPU Operator。 diff --git a/sidebars-tutorials.js b/sidebars-tutorials.js index 75f5657d2..fedc3dfb9 100644 --- a/sidebars-tutorials.js +++ b/sidebars-tutorials.js @@ -41,6 +41,11 @@ module.exports = { id: "labs/hami-vllm", customProps: { level: "Intermediate", duration: "about 45 minutes" }, }, + { + type: "doc", + id: "labs/hami-isolation-k3s", + customProps: { level: "Intermediate", duration: "about 45 minutes" }, + }, ], }, ], diff --git a/tutorials/labs/examples/07-hami-isolation-k3s/oversubscribe-pending.yaml b/tutorials/labs/examples/07-hami-isolation-k3s/oversubscribe-pending.yaml new file mode 100644 index 000000000..3b465d40a --- /dev/null +++ b/tutorials/labs/examples/07-hami-isolation-k3s/oversubscribe-pending.yaml @@ -0,0 +1,28 @@ +# A third Pod whose memory slice fits the card when it is empty, but NOT alongside +# the two 8000 MiB slices that hami-share-a / hami-share-b already hold. It stays +# Pending with CardInsufficientMemory - proving the device plugin and scheduler +# account the card's memory as one shared, finite budget. +# +# Apply this ONLY after hami-share-a and hami-share-b are Running. +# +# Sizing (IMPORTANT - adjust for your card): set gpumem ABOVE (card_total - 2*slice) +# so it cannot fit beside the two slices, and BELOW card_total so it WOULD fit on an +# empty card. 90000 MiB works for a 96 GB RTX PRO 6000 (97887 MiB) with two 8000 MiB +# slices held (~82000 free). For a 48 GB card use ~45000; for a 24 GB card with two +# 4000 MiB slices, use ~20000. +# Read the card size first: +# kubectl exec hami-share-a -- nvidia-smi --query-gpu=memory.total --format=csv +apiVersion: v1 +kind: Pod +metadata: + name: hami-oversubscribe +spec: + restartPolicy: Never + containers: + - name: cuda + image: nvidia/cuda:12.4.1-devel-ubuntu22.04 + command: ["bash", "-c", "nvidia-smi; sleep infinity"] + resources: + limits: + nvidia.com/gpu: 1 + nvidia.com/gpumem: 90000 diff --git a/tutorials/labs/examples/07-hami-isolation-k3s/share-two-pods.yaml b/tutorials/labs/examples/07-hami-isolation-k3s/share-two-pods.yaml new file mode 100644 index 000000000..092322bd9 --- /dev/null +++ b/tutorials/labs/examples/07-hami-isolation-k3s/share-two-pods.yaml @@ -0,0 +1,35 @@ +# Two Pods, each asking for one physical GPU and an 8000 MiB memory slice. On a +# 96 GB RTX PRO 6000 (or a 48 GB A6000) both fit with plenty of headroom, so both +# land on the one GPU. On a 24 GB card use ~4000 MiB instead. +# +# The image is the CUDA devel image because the memory probe compiles a tiny CUDA +# allocator with nvcc inside the Pod. It is large to pull; that is expected. +apiVersion: v1 +kind: Pod +metadata: + name: hami-share-a +spec: + restartPolicy: Never + containers: + - name: cuda + image: nvidia/cuda:12.4.1-devel-ubuntu22.04 + command: ["bash", "-c", "nvidia-smi; sleep infinity"] + resources: + limits: + nvidia.com/gpu: 1 + nvidia.com/gpumem: 8000 +--- +apiVersion: v1 +kind: Pod +metadata: + name: hami-share-b +spec: + restartPolicy: Never + containers: + - name: cuda + image: nvidia/cuda:12.4.1-devel-ubuntu22.04 + command: ["bash", "-c", "nvidia-smi; sleep infinity"] + resources: + limits: + nvidia.com/gpu: 1 + nvidia.com/gpumem: 8000 diff --git a/tutorials/labs/hami-isolation-k3s.md b/tutorials/labs/hami-isolation-k3s.md new file mode 100644 index 000000000..9888acf44 --- /dev/null +++ b/tutorials/labs/hami-isolation-k3s.md @@ -0,0 +1,385 @@ +--- +title: "Lab 7: GPU Isolation on k3s Without the GPU Operator" +description: "Share one non-MIG GPU between Pods on single-node k3s and prove HAMi-core enforces the memory cap." +sidebar_label: "Lab 7: k3s Isolation" +lab: + level: Intermediate + duration: about 45 minutes + environment: single-node k3s on a cloud GPU VM (any non-MIG NVIDIA card) + cost: under $2 in GPU time + authors: + - saiyam1814 + verified: "2026-07-07" +tags: + - isolation + - k3s +toc_max_heading_level: 2 +--- + +This lab builds a single-node k3s cluster on a rented GPU VM, installs HAMi **without** the NVIDIA GPU Operator, and proves that HAMi's memory isolation is real: two Pods share one physical card, `nvidia-smi` inside each Pod reports only its slice, a CUDA allocation past the slice is refused by HAMi-core while the card still has tens of GB free, and a third Pod that would oversubscribe the card stays `Pending`. + +Every command and output in this lab was captured from a live run on a GCP `g4-standard-48` Spot VM with one 96 GB NVIDIA RTX PRO 6000 Blackwell (single-node k3s v1.36.2+k3s1, HAMi v2.9.0, NVIDIA driver 610.43.02, Ubuntu 22.04). Any non-MIG card works the same; only the slice sizes change (they scale with VRAM). The RTX PRO 6000 Blackwell does support MIG, but it ships with MIG disabled — HAMi's software sharing is what this lab exercises. + +## How This Differs from Lab 3 + +[Lab 3](./gpu-partitioning.md) proves the same isolation property on a cluster where the **GPU Operator** provides the driver and container toolkit, and HAMi's device plugin is layered on top. This lab takes the other supported path: **no GPU Operator at all**. HAMi brings its own device plugin, the NVIDIA Container Toolkit is installed directly on the host, and `nvidia` is made the **default** containerd runtime so HAMi-core is injected into every GPU Pod. This is the leaner setup you would use on edge nodes, bare-metal boxes, or cheap rented GPU VMs — and it surfaces the mechanism (Step 7) that the Operator path hides. + +## What You'll Learn + +- Bringing up single-node k3s with the NVIDIA Container Toolkit and `nvidia` as the default runtime +- Installing HAMi without the GPU Operator, with the scheduler image tag matched to the server version +- Where HAMi records shareable GPU memory (the `hami.io/node-nvidia-register` annotation, not node allocatable) +- Proving the memory cap holds: virtualized `nvidia-smi` and a CUDA allocation refused at the slice +- Proving per-device accounting: an oversubscribing Pod stays `Pending` with `CardInsufficientMemory` +- How the cap is enforced: HAMi-core (`libvgpu.so`) registered in the Pod's `/etc/ld.so.preload` (software isolation, not MIG) + +## Lab Overview + +```mermaid +%% title: k3s Isolation Lab Flow +flowchart LR + Step1["Step 1
Host setup"] --> Step2["Step 2
k3s + toolkit"] + Step2 --> Step3["Step 3
nvidia default runtime"] + Step3 --> Step4["Step 4
Install HAMi"] + Step4 --> Step5["Step 5
Two Pods, one GPU"] + Step5 --> Step6["Step 6
Cap + budget tests"] + Step6 --> Step7["Step 7
The mechanism"] +``` + +## Prerequisites + +- A fresh cloud VM (Ubuntu 22.04 or later) with **one NVIDIA GPU** and root access. Any card without MIG enabled is the intended target: RTX PRO 6000, RTX A6000, L4, L40/L40S, RTX 4090/3090. Fractioning cards that MIG cannot (or does not) partition is exactly HAMi's use case. +- The NVIDIA driver working on the host (`nvidia-smi` succeeds). Most GPU VM images ship it. +- You control the container runtime: a locked-down marketplace container you cannot reconfigure will not work, because HAMi-core is injected through the NVIDIA container runtime. +- Run everything **on the VM over SSH** — helm and kubectl talk to k3s over localhost, so no flaky remote link. +- Manifests from [`tutorials/labs/examples/07-hami-isolation-k3s/`](https://github.com/Project-HAMi/website/tree/master/tutorials/labs/examples/07-hami-isolation-k3s) + +:::note[Cost] + +A working session is well under an hour of GPU time. Non-MIG cards commonly rent for under $1/hour on-demand. GPUs are scarce — the exact card you want is often out of stock; take whichever non-MIG card is available and adjust the slice sizes (a one-line change in the manifests). + +::: + +:::warning[Do not install the GPU Operator] + +HAMi ships its own device plugin and [must not coexist with NVIDIA's official device plugin](https://project-hami.io/docs/installation/prerequisites). GPU Operator + HAMi integration is not officially documented ([HAMi #1708](https://github.com/Project-HAMi/HAMi/issues/1708)). If your cluster already runs the Operator, use Lab 3's path instead — or a fresh VM for this one. + +::: + +## Step 1: Verify the Host Driver + +Nothing below works without a healthy host driver. SSH into the VM as root (or use `sudo` throughout) and check: + +```bash +nvidia-smi -L +``` + +```plaintext +GPU 0: NVIDIA RTX PRO 6000 Blackwell Server Edition (UUID: GPU-3c4a3856-fbb4-1425-9679-aed25f4d2977) +``` + +> One card, visible to the host. If `nvidia-smi` fails, install the driver per the [NVIDIA driver installation guide](https://docs.nvidia.com/datacenter/tesla/driver-installation-guide/) and re-run. On a fresh Ubuntu VM that means the NVIDIA CUDA apt repo and `apt-get install nvidia-open` — Blackwell-generation cards require the **open** kernel modules (driver ≥ 570); with the proprietary module `nvidia-smi` reports `No devices found`. + +## Step 2: Install the Container Toolkit, Then k3s + +Order matters: install the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) **before** k3s, so k3s auto-detects the `nvidia` container runtime at install time. + +```bash +install -m 0755 -d /usr/share/keyrings +curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \ + | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg +curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \ + | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \ + > /etc/apt/sources.list.d/nvidia-container-toolkit.list +apt-get update -y +apt-get install -y nvidia-container-toolkit +``` + +Now install k3s: + +```bash +curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--write-kubeconfig-mode 644" sh - +export KUBECONFIG=/etc/rancher/k3s/k3s.yaml +kubectl get nodes -o wide +``` + +```plaintext +NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME +hami-lab-rtx6000 Ready control-plane 65s v1.36.2+k3s1 10.128.0.2 Ubuntu 22.04.5 LTS 6.8.0-1063-gcp (amd64) containerd://2.3.2-k3s2 +``` + +Confirm k3s picked up the nvidia runtime, and label the node for HAMi: + +```bash +kubectl get runtimeclass nvidia +kubectl label node hami-lab-rtx6000 gpu=on +``` + +```plaintext +NAME HANDLER AGE +nvidia nvidia 55s +node/hami-lab-rtx6000 labeled +``` + +> k3s writes an `nvidia` RuntimeClass into its containerd config when the toolkit is present at install time. If it is missing, restart k3s (`systemctl restart k3s`) — see [k3s NVIDIA runtime support](https://docs.k3s.io/advanced#nvidia-container-runtime-support). HAMi's device plugin schedules onto nodes labelled `gpu=on`. + +## Step 3: Make nvidia the Default Containerd Runtime + +HAMi Pods do not set `runtimeClassName`, so HAMi-core is only injected if the **default** runtime is `nvidia`. Use k3s's own `default-runtime` option — the clean way that works regardless of the containerd config schema. Do **not** hand-edit `config.toml`: the v2 and v3 schemas differ and redeclaring an existing table breaks k3s. + +```bash +echo 'default-runtime: nvidia' >> /etc/rancher/k3s/config.yaml +systemctl restart k3s +``` + +Verify it took (k3s regenerates this file; you never edit it directly): + +```bash +grep default_runtime_name /var/lib/rancher/k3s/agent/etc/containerd/config.toml +``` + +```plaintext +default_runtime_name = "nvidia" +``` + +> This single line is the difference between HAMi enforcing limits and silently not enforcing them. If Step 7 later shows no `libvgpu.so` inside a GPU Pod, come back here first. + +## Step 4: Install HAMi + +The HAMi scheduler runs a kube-scheduler sidecar whose image tag must match the cluster's Kubernetes **server** version — a mismatch is the most common HAMi install failure. Detect it first: + +```bash +kubectl version | grep Server +``` + +```plaintext +Server Version: v1.36.2+k3s1 +``` + +Strip the `+k3s1` suffix and pass it as the scheduler image tag: + +```bash +helm repo add hami-charts https://project-hami.github.io/HAMi +helm repo update hami-charts +helm upgrade --install hami hami-charts/hami \ + --version 2.9.0 \ + -n kube-system \ + --set scheduler.kubeScheduler.imageTag=v1.36.2 \ + --wait --timeout 5m +kubectl -n kube-system get pods | grep hami +``` + +```plaintext +hami-device-plugin-62ck6 2/2 Running 0 38s +hami-scheduler-5f5b5589c9-zhgsc 2/2 Running 0 38s +``` + +> Unlike Lab 2 (which disables it), the HAMi **device plugin runs here** — it is the only device plugin on the cluster and it owns the GPU. + +Verify HAMi registered the card: + +```bash +kubectl get node hami-lab-rtx6000 -o jsonpath='{.status.allocatable.nvidia\.com/gpu}'; echo +kubectl get node hami-lab-rtx6000 -o jsonpath='{.metadata.annotations.hami\.io/node-nvidia-register}'; echo +``` + +```plaintext +10 +[{"id":"GPU-3c4a3856-fbb4-1425-9679-aed25f4d2977","count":10,"devmem":97887,"devcore":100,"type":"NVIDIA RTX PRO 6000 Blackwell Server Edition","mode":"hami-core","health":true,"devicepairscore":{}}] +``` + +> Two things worth internalizing: +> +> - `nvidia.com/gpu` allocatable is `10` — one physical GPU × `deviceSplitCount` (default 10), the number of Pods that may share the card. +> - The shareable memory (`devmem: 97887` MiB) lives in the `hami.io/node-nvidia-register` annotation, **not** in node allocatable. There is no `nvidia.com/gpumem` in `kubectl describe node`; the HAMi scheduler and webhook account `gpumem`/`gpucores` from this annotation, and HAMi-core enforces them per Pod. + +## Step 5: Two Pods Share One Physical GPU + +`share-two-pods.yaml` runs two Pods, each requesting one GPU and an 8000 MiB slice — something stock Kubernetes cannot do (it hands a Pod the whole card): + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: hami-share-a +spec: + restartPolicy: Never + containers: + - name: cuda + image: nvidia/cuda:12.4.1-devel-ubuntu22.04 + command: ["bash", "-c", "nvidia-smi; sleep infinity"] + resources: + limits: + nvidia.com/gpu: 1 + nvidia.com/gpumem: 8000 +``` + +> `hami-share-b` is identical except for the name. The `devel` image is used because Step 6 compiles a small CUDA allocator with `nvcc` inside the Pod; it is large to pull, which is expected. On a 24 GB card use ~4000 MiB slices instead. + +```bash +kubectl apply -f share-two-pods.yaml +kubectl wait --for=condition=Ready pod/hami-share-a pod/hami-share-b --timeout=300s +kubectl get pods -o wide +``` + +```plaintext +NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES +hami-share-a 1/1 Running 0 37s 10.42.0.13 hami-lab-rtx6000 +hami-share-b 1/1 Running 0 37s 10.42.0.14 hami-lab-rtx6000 +``` + +> Both Pods `Running` on the same single-GPU node: co-residency. The hami-scheduler placed them; its per-Pod decision is visible in the events: + +```bash +kubectl describe pod hami-share-a | grep -A5 Events: +``` + +```plaintext +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Scheduled 52s hami-scheduler Successfully assigned default/hami-share-a to hami-lab-rtx6000 + Normal FilteringSucceed 53s hami-scheduler find fit node(hami-lab-rtx6000), 0 nodes not fit, 1 nodes fit(hami-lab-rtx6000:0.00) + Normal BindingSucceed 52s hami-scheduler Successfully binding node [hami-lab-rtx6000] to default/hami-share-a +``` + +## Step 6: Prove the Cap Holds + +Two checks inside `hami-share-a`, back to back. First, what does the container _think_ the GPU is? + +```bash +kubectl exec hami-share-a -- nvidia-smi +``` + +```plaintext +Tue Jul 7 10:25:15 2026 ++-----------------------------------------------------------------------------------------+ +| NVIDIA-SMI 610.43.02 KMD Version: 610.43.02 CUDA UMD Version: 13.3 | ++-----------------------------------------+------------------------+----------------------+ +| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | +| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | +| | | MIG M. | +|=========================================+========================+======================| +| 0 NVIDIA RTX PRO 6000 Blac... Off | 00000000:05:00.0 Off | 0 | +| N/A 34C P0 48W / 600W | 0MiB / 8000MiB | 0% Default | +| | | Disabled | ++-----------------------------------------+------------------------+----------------------+ +``` + +> The Memory column reads **8000MiB**, not the card's real 97887MiB. HAMi-core rewrites the card size the container sees. Note `MIG M.: Disabled` — the sharing here is HAMi's software slice, not a hardware partition. + +Second, does the cap actually hold? Compile and run a tiny allocator that `cudaMalloc`s 256 MiB at a time until refused: + +```bash +kubectl exec hami-share-a -- bash -c ' +cat > /tmp/probe.cu < +#include +int main() { + const size_t chunk = (size_t)256 * 1024 * 1024; // 256 MiB + size_t total = 0; + void *p = nullptr; + for (;;) { + cudaError_t e = cudaMalloc(&p, chunk); + if (e != cudaSuccess) { + printf("cudaMalloc refused after %zu MiB allocated: %s\n", + total / (1024 * 1024), cudaGetErrorString(e)); + return 0; + } + total += chunk; + } +} +EOF +nvcc -o /tmp/probe /tmp/probe.cu && /tmp/probe' +``` + +```plaintext +cudaMalloc refused after 7424 MiB allocated: out of memory +[HAMI-core ERROR (pid:54 thread=133390658244608 allocator.c:52)]: Device 0 OOM 8629780480 / 8388608000 +``` + +> This is the proof, and it is a contradiction: the Pod hit "out of memory" at ~7.4 GB while the physical card still had ~82 GB free. The refusal came from **HAMI-core** (see the error line: it tried to use 8629780480 bytes against the 8388608000-byte limit — 8388608000 B is exactly 8000 MiB), not from the hardware. Only a software cap intercepting CUDA calls can do that. State it as "allocations beyond the slice are refused" — the exact boundary depends on allocator and CUDA context overhead, so do not assert an exact byte count. Repeat for `hami-share-b` if you like; it behaves identically. + +Now prove the card's memory is one shared, finite budget. `oversubscribe-pending.yaml` requests a 90000 MiB slice — it would fit an empty 96 GB card, but not beside the two 8000 MiB slices already held (97887 − 16000 ≈ 82000 MiB free): + +```bash +kubectl apply -f oversubscribe-pending.yaml +sleep 15 +kubectl get pod hami-oversubscribe +kubectl describe pod hami-oversubscribe | grep -A5 Events: +``` + +```plaintext +NAME READY STATUS RESTARTS AGE +hami-oversubscribe 0/1 Pending 0 15s +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Warning FailedScheduling 15s hami-scheduler 0/1 nodes are available: 1 NodeUnfitPod. no new claims to deallocate, preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod. + Warning FilteringFailed 16s (x2 over 16s) hami-scheduler 1 nodes CardInsufficientMemory(hami-lab-rtx6000) +``` + +> `Pending` with `CardInsufficientMemory` — contrast Step 5, where the same scheduler logged `FilteringSucceed`. Two things must be right or this "fails" by scheduling instead of staying Pending: the request must be sized for **your** card (above the free space beside the slices, below the whole card — see the manifest comment), and the two share Pods must still be `Running` (they hold the slices; the manifests use `sleep infinity` so they do not quietly complete mid-lab). + +## Step 7: Surface the Mechanism + +How is the cap enforced? Look inside a capped Pod for what the runtime injected: + +```bash +kubectl exec hami-share-a -- bash -c 'env | grep -iE "CUDA_DEVICE|NVIDIA_VISIBLE|LD_PRELOAD" | sort; cat /etc/ld.so.preload; ls -l /usr/local/vgpu/ 2>/dev/null' +``` + +```plaintext +CUDA_DEVICE_MEMORY_LIMIT_0=8000m +CUDA_DEVICE_MEMORY_SHARED_CACHE=/usr/local/vgpu/5af6befa-5ff5-4bd5-9086-c79554723b6f.cache +CUDA_DEVICE_SM_LIMIT=0 +NVIDIA_VISIBLE_DEVICES=GPU-3c4a3856-fbb4-1425-9679-aed25f4d2977 +/usr/local/vgpu/libvgpu.so +total 684 +-rw-rw-rw- 1 root root 2008953 Jul 7 10:25 5af6befa-5ff5-4bd5-9086-c79554723b6f.cache +-rwxr-xr-x 1 root root 684264 Jul 7 10:23 libvgpu.so +``` + +> The whole mechanism in a few lines: `libvgpu.so` (HAMi-core) is mounted into the Pod and registered in `/etc/ld.so.preload`, so it is loaded into **every** process in the container, and HAMi-core reads its cap from `CUDA_DEVICE_MEMORY_LIMIT_0`. Every CUDA driver call the application makes passes through this library, which is where the Step 6 refusal came from. (`CUDA_DEVICE_SM_LIMIT=0` means no compute throttle was requested — the Pods set no `gpucores`; the shared-cache file is how HAMi-core instances account usage across processes.) This is the same `CUDA_DEVICE_MEMORY_LIMIT` mechanism NVIDIA's KAI Scheduler [adopted in June 2026](https://github.com/NVIDIA/KAI-Scheduler/pull/60) for its fractional-GPU memory isolation. The exact variable names and paths are HAMi-version dependent — record what you actually see. + +:::note[What this is, and is not] + +This is **software isolation**: user-space CUDA interception. It is not the hardware fault isolation of MIG — a misbehaving kernel is constrained by interception, not by a hardware partition. Treat it as a scheduling-and-accounting guarantee with runtime enforcement, not a security boundary. This lab also does not measure compute-throttling accuracy or noisy-neighbour interference under sustained load; see Lab 3's Step 5 for `gpucores` behavior. + +::: + +## Cleanup + +```bash +kubectl delete pod hami-share-a hami-share-b hami-oversubscribe +helm -n kube-system uninstall hami +``` + +If you rented the VM for this lab, capture your outputs and tear the instance down — the meter is running. + +```bash +/usr/local/bin/k3s-uninstall.sh # optional: remove k3s entirely +``` + +## What This Lab Proved + +| Claim | Evidence | +| --- | --- | +| HAMi runs without the GPU Operator | Its own device plugin was the only one on the cluster; `nvidia.com/gpu` allocatable appeared | +| Two Pods share one physical GPU | Both `hami-share-a` and `hami-share-b` `Running` on the single-GPU node | +| The container sees only its slice | In-Pod `nvidia-smi` reported 8000 MiB total, not 97887 MiB | +| The memory cap is enforced, not cosmetic | `cudaMalloc` refused at ~7424 MiB by HAMI-core while the card had ~82 GB free | +| The card's memory is one accounted budget | 90000 MiB Pod stayed `Pending` with `CardInsufficientMemory` | +| The mechanism is user-space interception | `libvgpu.so` in `/etc/ld.so.preload` + `CUDA_DEVICE_MEMORY_LIMIT_0=8000m` injected into the Pod | + +## Next Steps + +- Run [Lab 2: Local Fake GPU](./local-fake-gpu.md) to study the scheduling half of this story for free, with no GPU — the sim proves the placement decision, this lab proves enforcement. +- Compare with [Lab 3: GPU Partitioning](./gpu-partitioning.md), which reaches the same isolation proof via the GPU Operator path and adds `gpucores` compute throttling. +- Read [HAMi Cluster Architecture](/docs/core-concepts/hami-architecture) to map the webhook → scheduler → device plugin → HAMi-core chain you just exercised end to end. + +:::info[Credits] + +This lab is adapted, with his kind permission, from the HAMi lesson in [Lovedeep Singh](https://github.com/ld-singh)'s [AI Factory Operations Lab](https://ld-singh.github.io/ai-factory-ops-lab/), which pairs the no-GPU scheduling sim (covered here by [Lab 2](./local-fake-gpu.md)) with the real-GPU isolation proof. He first validated the exercise sequence on a 48 GB RTX A6000 ([validation report](https://github.com/ld-singh/ai-factory-ops-lab/blob/main/portfolio-lab/06-validation-reports/hami-isolation-validation.md)). + +::: diff --git a/tutorials/overview.md b/tutorials/overview.md index a2dfb0f2d..dbbc9c3b6 100644 --- a/tutorials/overview.md +++ b/tutorials/overview.md @@ -17,4 +17,4 @@ Background knowledge that the labs build on. ## Labs - Each lab lists its own prerequisites. Labs 3 and 4 continue from the cluster Lab 1 builds, so a single session covers all three; Lab 2 runs on any laptop with no GPU required. + Each lab lists its own prerequisites. Labs 3 and 4 continue from the cluster Lab 1 builds, so a single session covers all three; Lab 2 runs on any laptop with no GPU required. Lab 7 brings up its own single-node k3s cluster on a rented GPU VM, without the GPU Operator. From 5e7e2dc623f0e41818b389cda1e2006be0d993ae Mon Sep 17 00:00:00 2001 From: Saiyam Pathak Date: Wed, 8 Jul 2026 13:54:53 +0530 Subject: [PATCH 2/2] Replace em dashes with hyphens per style guide Signed-off-by: Saiyam Pathak --- tutorials/labs/hami-isolation-k3s.md | 40 ++++++++++++++-------------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/tutorials/labs/hami-isolation-k3s.md b/tutorials/labs/hami-isolation-k3s.md index 9888acf44..5941ffe64 100644 --- a/tutorials/labs/hami-isolation-k3s.md +++ b/tutorials/labs/hami-isolation-k3s.md @@ -18,11 +18,11 @@ toc_max_heading_level: 2 This lab builds a single-node k3s cluster on a rented GPU VM, installs HAMi **without** the NVIDIA GPU Operator, and proves that HAMi's memory isolation is real: two Pods share one physical card, `nvidia-smi` inside each Pod reports only its slice, a CUDA allocation past the slice is refused by HAMi-core while the card still has tens of GB free, and a third Pod that would oversubscribe the card stays `Pending`. -Every command and output in this lab was captured from a live run on a GCP `g4-standard-48` Spot VM with one 96 GB NVIDIA RTX PRO 6000 Blackwell (single-node k3s v1.36.2+k3s1, HAMi v2.9.0, NVIDIA driver 610.43.02, Ubuntu 22.04). Any non-MIG card works the same; only the slice sizes change (they scale with VRAM). The RTX PRO 6000 Blackwell does support MIG, but it ships with MIG disabled — HAMi's software sharing is what this lab exercises. +Every command and output in this lab was captured from a live run on a GCP `g4-standard-48` Spot VM with one 96 GB NVIDIA RTX PRO 6000 Blackwell (single-node k3s v1.36.2+k3s1, HAMi v2.9.0, NVIDIA driver 610.43.02, Ubuntu 22.04). Any non-MIG card works the same; only the slice sizes change (they scale with VRAM). The RTX PRO 6000 Blackwell does support MIG, but it ships with MIG disabled - HAMi's software sharing is what this lab exercises. ## How This Differs from Lab 3 -[Lab 3](./gpu-partitioning.md) proves the same isolation property on a cluster where the **GPU Operator** provides the driver and container toolkit, and HAMi's device plugin is layered on top. This lab takes the other supported path: **no GPU Operator at all**. HAMi brings its own device plugin, the NVIDIA Container Toolkit is installed directly on the host, and `nvidia` is made the **default** containerd runtime so HAMi-core is injected into every GPU Pod. This is the leaner setup you would use on edge nodes, bare-metal boxes, or cheap rented GPU VMs — and it surfaces the mechanism (Step 7) that the Operator path hides. +[Lab 3](./gpu-partitioning.md) proves the same isolation property on a cluster where the **GPU Operator** provides the driver and container toolkit, and HAMi's device plugin is layered on top. This lab takes the other supported path: **no GPU Operator at all**. HAMi brings its own device plugin, the NVIDIA Container Toolkit is installed directly on the host, and `nvidia` is made the **default** containerd runtime so HAMi-core is injected into every GPU Pod. This is the leaner setup you would use on edge nodes, bare-metal boxes, or cheap rented GPU VMs - and it surfaces the mechanism (Step 7) that the Operator path hides. ## What You'll Learn @@ -51,18 +51,18 @@ flowchart LR - A fresh cloud VM (Ubuntu 22.04 or later) with **one NVIDIA GPU** and root access. Any card without MIG enabled is the intended target: RTX PRO 6000, RTX A6000, L4, L40/L40S, RTX 4090/3090. Fractioning cards that MIG cannot (or does not) partition is exactly HAMi's use case. - The NVIDIA driver working on the host (`nvidia-smi` succeeds). Most GPU VM images ship it. - You control the container runtime: a locked-down marketplace container you cannot reconfigure will not work, because HAMi-core is injected through the NVIDIA container runtime. -- Run everything **on the VM over SSH** — helm and kubectl talk to k3s over localhost, so no flaky remote link. +- Run everything **on the VM over SSH** - helm and kubectl talk to k3s over localhost, so no flaky remote link. - Manifests from [`tutorials/labs/examples/07-hami-isolation-k3s/`](https://github.com/Project-HAMi/website/tree/master/tutorials/labs/examples/07-hami-isolation-k3s) :::note[Cost] -A working session is well under an hour of GPU time. Non-MIG cards commonly rent for under $1/hour on-demand. GPUs are scarce — the exact card you want is often out of stock; take whichever non-MIG card is available and adjust the slice sizes (a one-line change in the manifests). +A working session is well under an hour of GPU time. Non-MIG cards commonly rent for under $1/hour on-demand. GPUs are scarce - the exact card you want is often out of stock; take whichever non-MIG card is available and adjust the slice sizes (a one-line change in the manifests). ::: :::warning[Do not install the GPU Operator] -HAMi ships its own device plugin and [must not coexist with NVIDIA's official device plugin](https://project-hami.io/docs/installation/prerequisites). GPU Operator + HAMi integration is not officially documented ([HAMi #1708](https://github.com/Project-HAMi/HAMi/issues/1708)). If your cluster already runs the Operator, use Lab 3's path instead — or a fresh VM for this one. +HAMi ships its own device plugin and [must not coexist with NVIDIA's official device plugin](https://project-hami.io/docs/installation/prerequisites). GPU Operator + HAMi integration is not officially documented ([HAMi #1708](https://github.com/Project-HAMi/HAMi/issues/1708)). If your cluster already runs the Operator, use Lab 3's path instead - or a fresh VM for this one. ::: @@ -78,7 +78,7 @@ nvidia-smi -L GPU 0: NVIDIA RTX PRO 6000 Blackwell Server Edition (UUID: GPU-3c4a3856-fbb4-1425-9679-aed25f4d2977) ``` -> One card, visible to the host. If `nvidia-smi` fails, install the driver per the [NVIDIA driver installation guide](https://docs.nvidia.com/datacenter/tesla/driver-installation-guide/) and re-run. On a fresh Ubuntu VM that means the NVIDIA CUDA apt repo and `apt-get install nvidia-open` — Blackwell-generation cards require the **open** kernel modules (driver ≥ 570); with the proprietary module `nvidia-smi` reports `No devices found`. +> One card, visible to the host. If `nvidia-smi` fails, install the driver per the [NVIDIA driver installation guide](https://docs.nvidia.com/datacenter/tesla/driver-installation-guide/) and re-run. On a fresh Ubuntu VM that means the NVIDIA CUDA apt repo and `apt-get install nvidia-open` - Blackwell-generation cards require the **open** kernel modules (driver ≥ 570); with the proprietary module `nvidia-smi` reports `No devices found`. ## Step 2: Install the Container Toolkit, Then k3s @@ -121,11 +121,11 @@ nvidia nvidia 55s node/hami-lab-rtx6000 labeled ``` -> k3s writes an `nvidia` RuntimeClass into its containerd config when the toolkit is present at install time. If it is missing, restart k3s (`systemctl restart k3s`) — see [k3s NVIDIA runtime support](https://docs.k3s.io/advanced#nvidia-container-runtime-support). HAMi's device plugin schedules onto nodes labelled `gpu=on`. +> k3s writes an `nvidia` RuntimeClass into its containerd config when the toolkit is present at install time. If it is missing, restart k3s (`systemctl restart k3s`) - see [k3s NVIDIA runtime support](https://docs.k3s.io/advanced#nvidia-container-runtime-support). HAMi's device plugin schedules onto nodes labelled `gpu=on`. ## Step 3: Make nvidia the Default Containerd Runtime -HAMi Pods do not set `runtimeClassName`, so HAMi-core is only injected if the **default** runtime is `nvidia`. Use k3s's own `default-runtime` option — the clean way that works regardless of the containerd config schema. Do **not** hand-edit `config.toml`: the v2 and v3 schemas differ and redeclaring an existing table breaks k3s. +HAMi Pods do not set `runtimeClassName`, so HAMi-core is only injected if the **default** runtime is `nvidia`. Use k3s's own `default-runtime` option - the clean way that works regardless of the containerd config schema. Do **not** hand-edit `config.toml`: the v2 and v3 schemas differ and redeclaring an existing table breaks k3s. ```bash echo 'default-runtime: nvidia' >> /etc/rancher/k3s/config.yaml @@ -146,7 +146,7 @@ default_runtime_name = "nvidia" ## Step 4: Install HAMi -The HAMi scheduler runs a kube-scheduler sidecar whose image tag must match the cluster's Kubernetes **server** version — a mismatch is the most common HAMi install failure. Detect it first: +The HAMi scheduler runs a kube-scheduler sidecar whose image tag must match the cluster's Kubernetes **server** version - a mismatch is the most common HAMi install failure. Detect it first: ```bash kubectl version | grep Server @@ -174,7 +174,7 @@ hami-device-plugin-62ck6 2/2 Running 0 38s hami-scheduler-5f5b5589c9-zhgsc 2/2 Running 0 38s ``` -> Unlike Lab 2 (which disables it), the HAMi **device plugin runs here** — it is the only device plugin on the cluster and it owns the GPU. +> Unlike Lab 2 (which disables it), the HAMi **device plugin runs here** - it is the only device plugin on the cluster and it owns the GPU. Verify HAMi registered the card: @@ -190,12 +190,12 @@ kubectl get node hami-lab-rtx6000 -o jsonpath='{.metadata.annotations.hami\.io/n > Two things worth internalizing: > -> - `nvidia.com/gpu` allocatable is `10` — one physical GPU × `deviceSplitCount` (default 10), the number of Pods that may share the card. +> - `nvidia.com/gpu` allocatable is `10` - one physical GPU × `deviceSplitCount` (default 10), the number of Pods that may share the card. > - The shareable memory (`devmem: 97887` MiB) lives in the `hami.io/node-nvidia-register` annotation, **not** in node allocatable. There is no `nvidia.com/gpumem` in `kubectl describe node`; the HAMi scheduler and webhook account `gpumem`/`gpucores` from this annotation, and HAMi-core enforces them per Pod. ## Step 5: Two Pods Share One Physical GPU -`share-two-pods.yaml` runs two Pods, each requesting one GPU and an 8000 MiB slice — something stock Kubernetes cannot do (it hands a Pod the whole card): +`share-two-pods.yaml` runs two Pods, each requesting one GPU and an 8000 MiB slice - something stock Kubernetes cannot do (it hands a Pod the whole card): ```yaml apiVersion: v1 @@ -266,7 +266,7 @@ Tue Jul 7 10:25:15 2026 +-----------------------------------------+------------------------+----------------------+ ``` -> The Memory column reads **8000MiB**, not the card's real 97887MiB. HAMi-core rewrites the card size the container sees. Note `MIG M.: Disabled` — the sharing here is HAMi's software slice, not a hardware partition. +> The Memory column reads **8000MiB**, not the card's real 97887MiB. HAMi-core rewrites the card size the container sees. Note `MIG M.: Disabled` - the sharing here is HAMi's software slice, not a hardware partition. Second, does the cap actually hold? Compile and run a tiny allocator that `cudaMalloc`s 256 MiB at a time until refused: @@ -298,9 +298,9 @@ cudaMalloc refused after 7424 MiB allocated: out of memory [HAMI-core ERROR (pid:54 thread=133390658244608 allocator.c:52)]: Device 0 OOM 8629780480 / 8388608000 ``` -> This is the proof, and it is a contradiction: the Pod hit "out of memory" at ~7.4 GB while the physical card still had ~82 GB free. The refusal came from **HAMI-core** (see the error line: it tried to use 8629780480 bytes against the 8388608000-byte limit — 8388608000 B is exactly 8000 MiB), not from the hardware. Only a software cap intercepting CUDA calls can do that. State it as "allocations beyond the slice are refused" — the exact boundary depends on allocator and CUDA context overhead, so do not assert an exact byte count. Repeat for `hami-share-b` if you like; it behaves identically. +> This is the proof, and it is a contradiction: the Pod hit "out of memory" at ~7.4 GB while the physical card still had ~82 GB free. The refusal came from **HAMI-core** (see the error line: it tried to use 8629780480 bytes against the 8388608000-byte limit - 8388608000 B is exactly 8000 MiB), not from the hardware. Only a software cap intercepting CUDA calls can do that. State it as "allocations beyond the slice are refused" - the exact boundary depends on allocator and CUDA context overhead, so do not assert an exact byte count. Repeat for `hami-share-b` if you like; it behaves identically. -Now prove the card's memory is one shared, finite budget. `oversubscribe-pending.yaml` requests a 90000 MiB slice — it would fit an empty 96 GB card, but not beside the two 8000 MiB slices already held (97887 − 16000 ≈ 82000 MiB free): +Now prove the card's memory is one shared, finite budget. `oversubscribe-pending.yaml` requests a 90000 MiB slice - it would fit an empty 96 GB card, but not beside the two 8000 MiB slices already held (97887 − 16000 ≈ 82000 MiB free): ```bash kubectl apply -f oversubscribe-pending.yaml @@ -319,7 +319,7 @@ Events: Warning FilteringFailed 16s (x2 over 16s) hami-scheduler 1 nodes CardInsufficientMemory(hami-lab-rtx6000) ``` -> `Pending` with `CardInsufficientMemory` — contrast Step 5, where the same scheduler logged `FilteringSucceed`. Two things must be right or this "fails" by scheduling instead of staying Pending: the request must be sized for **your** card (above the free space beside the slices, below the whole card — see the manifest comment), and the two share Pods must still be `Running` (they hold the slices; the manifests use `sleep infinity` so they do not quietly complete mid-lab). +> `Pending` with `CardInsufficientMemory` - contrast Step 5, where the same scheduler logged `FilteringSucceed`. Two things must be right or this "fails" by scheduling instead of staying Pending: the request must be sized for **your** card (above the free space beside the slices, below the whole card - see the manifest comment), and the two share Pods must still be `Running` (they hold the slices; the manifests use `sleep infinity` so they do not quietly complete mid-lab). ## Step 7: Surface the Mechanism @@ -340,11 +340,11 @@ total 684 -rwxr-xr-x 1 root root 684264 Jul 7 10:23 libvgpu.so ``` -> The whole mechanism in a few lines: `libvgpu.so` (HAMi-core) is mounted into the Pod and registered in `/etc/ld.so.preload`, so it is loaded into **every** process in the container, and HAMi-core reads its cap from `CUDA_DEVICE_MEMORY_LIMIT_0`. Every CUDA driver call the application makes passes through this library, which is where the Step 6 refusal came from. (`CUDA_DEVICE_SM_LIMIT=0` means no compute throttle was requested — the Pods set no `gpucores`; the shared-cache file is how HAMi-core instances account usage across processes.) This is the same `CUDA_DEVICE_MEMORY_LIMIT` mechanism NVIDIA's KAI Scheduler [adopted in June 2026](https://github.com/NVIDIA/KAI-Scheduler/pull/60) for its fractional-GPU memory isolation. The exact variable names and paths are HAMi-version dependent — record what you actually see. +> The whole mechanism in a few lines: `libvgpu.so` (HAMi-core) is mounted into the Pod and registered in `/etc/ld.so.preload`, so it is loaded into **every** process in the container, and HAMi-core reads its cap from `CUDA_DEVICE_MEMORY_LIMIT_0`. Every CUDA driver call the application makes passes through this library, which is where the Step 6 refusal came from. (`CUDA_DEVICE_SM_LIMIT=0` means no compute throttle was requested - the Pods set no `gpucores`; the shared-cache file is how HAMi-core instances account usage across processes.) This is the same `CUDA_DEVICE_MEMORY_LIMIT` mechanism NVIDIA's KAI Scheduler [adopted in June 2026](https://github.com/NVIDIA/KAI-Scheduler/pull/60) for its fractional-GPU memory isolation. The exact variable names and paths are HAMi-version dependent - record what you actually see. :::note[What this is, and is not] -This is **software isolation**: user-space CUDA interception. It is not the hardware fault isolation of MIG — a misbehaving kernel is constrained by interception, not by a hardware partition. Treat it as a scheduling-and-accounting guarantee with runtime enforcement, not a security boundary. This lab also does not measure compute-throttling accuracy or noisy-neighbour interference under sustained load; see Lab 3's Step 5 for `gpucores` behavior. +This is **software isolation**: user-space CUDA interception. It is not the hardware fault isolation of MIG - a misbehaving kernel is constrained by interception, not by a hardware partition. Treat it as a scheduling-and-accounting guarantee with runtime enforcement, not a security boundary. This lab also does not measure compute-throttling accuracy or noisy-neighbour interference under sustained load; see Lab 3's Step 5 for `gpucores` behavior. ::: @@ -355,7 +355,7 @@ kubectl delete pod hami-share-a hami-share-b hami-oversubscribe helm -n kube-system uninstall hami ``` -If you rented the VM for this lab, capture your outputs and tear the instance down — the meter is running. +If you rented the VM for this lab, capture your outputs and tear the instance down - the meter is running. ```bash /usr/local/bin/k3s-uninstall.sh # optional: remove k3s entirely @@ -374,7 +374,7 @@ If you rented the VM for this lab, capture your outputs and tear the instance do ## Next Steps -- Run [Lab 2: Local Fake GPU](./local-fake-gpu.md) to study the scheduling half of this story for free, with no GPU — the sim proves the placement decision, this lab proves enforcement. +- Run [Lab 2: Local Fake GPU](./local-fake-gpu.md) to study the scheduling half of this story for free, with no GPU - the sim proves the placement decision, this lab proves enforcement. - Compare with [Lab 3: GPU Partitioning](./gpu-partitioning.md), which reaches the same isolation proof via the GPU Operator path and adds `gpucores` compute throttling. - Read [HAMi Cluster Architecture](/docs/core-concepts/hami-architecture) to map the webhook → scheduler → device plugin → HAMi-core chain you just exercised end to end.