diff --git a/blog/volcano-ascend-vnpu-soft-slicing/index.md b/blog/volcano-ascend-vnpu-soft-slicing/index.md new file mode 100644 index 000000000..e786b57b2 --- /dev/null +++ b/blog/volcano-ascend-vnpu-soft-slicing/index.md @@ -0,0 +1,255 @@ +--- +title: "Soft-Slicing Ascend vNPU with Volcano and HAMi-core: How It Works and How We Verified It" +date: "2026-08-17" +description: "How Volcano's HAMi-mode deviceshare combines batch scheduling with HAMi-core runtime isolation on Ascend NPUs, verified end to end on an Ascend 310P3 ARM server, covering source-built images, in-container memory slices, binpack card sharing, and per-container metrics." +authors: [rootsongjc] +tags: ["HAMi", "Volcano", "Ascend", "vNPU", "Soft Slicing", "Kubernetes"] +--- + +[Volcano](https://github.com/volcano-sh/volcano) is the batch scheduler of choice for many AI clusters, and HAMi-core is the runtime that makes shared accelerators behave. This post covers their intersection on Ascend hardware: running **`hami-vnpu-core` soft-sliced vNPUs under the Volcano scheduler**, so batch scheduling semantics (queues, gangs, binpack) and per-container isolation (memory and compute limits enforced at the Ascend API layer) work together. + +We verified the full path on a single-node Kubernetes cluster running on an Ascend 310P3 aarch64 server: built the Volcano images from source, deployed the official [ascend-device-plugin](https://github.com/Project-HAMi/ascend-device-plugin) v1.4.0 image, and confirmed that a container requesting an 8192 MiB slice sees exactly that much memory, while a second Pod binpacks onto the same physical card with its own independent slice and the plugin's Prometheus endpoint reports both limits. The complete step-by-step procedure (including every command and captured output) is [Lab 13: Soft-Slicing Ascend 310P3 vNPU with Volcano and HAMi-core](/tutorials/labs/volcano-ascend-vnpu). + +Because this topic mixes several concepts that are often conflated, the post first separates the layers: what a vNPU is, how hard and soft slicing differ, and what exactly the Volcano integration adds beyond the existing HAMi scheduler path. + +:::note About the captured output + +Every output block in this post was captured from a real run on a physical Ascend 310P3 server, verified as of the time of writing: a Kylin V10 aarch64 node with 2× Ascend 310P3 (driver/npu-smi 25.5.1), Kubernetes v1.28.15, and containerd 1.7.1. UUIDs, IPs, and Pod suffixes will differ in another cluster; compare the component names, placement, and measured values. + +::: + + + +## NPU, vNPU, hard slicing, soft slicing: keep the layers apart + +vNPU, hard slicing, soft slicing, HAMi mode, and Volcano support sit at different layers, and mixing them together makes the whole thing look like one opaque feature. Each answers a different question: + +| Layer | The question it answers | +| :-- | :-- | +| NPU / vNPU | Which device you get: a vNPU is a logical device carved out of a physical NPU | +| Hard / soft slicing | How that virtual device is isolated | +| HAMi-core | What enforces the soft-slicing quota inside the container | +| HAMi / Volcano | Who decides which Pod uses which card, and how much | + +An NPU is the physical device, for example one Ascend 310P3 actually installed in the server; a vNPU is a logical device carved out of a physical NPU and handed to a container (one 21.5 GiB card can become an 8 GiB slice plus another 8 GiB slice, with about 5.5 GiB left over). The name "vNPU" describes the result, not the implementation: the slice can be carved by the Ascend driver's virtualization, or simulated in software by HAMi-core. This is the easiest point to misread in this topic: the "soft-sliced vNPU" in this post is a logical slice from the Kubernetes/HAMi perspective, not a hardware vNPU created with `npu-smi ... create-vnpu`. + +**Hard slicing** is done by the Ascend driver/firmware virtualization, and you can only choose from predefined templates: `vir05_1c_16g`, for example, fixes a number of AI cores, AI CPUs, and 16 GiB of memory, and creating one produces a real vNPU instance at the device level. Query the templates your chip supports with `npu-smi info -t template-info`, and see Huawei's [hard-slicing practice guide](https://www.hiascend.com/developer/techArticles/20251212-1) for a full walkthrough. + +**Soft slicing** creates no hardware vNPU: multiple containers share the same physical NPU, and each container gets `libvnpu.so` injected to intercept and account for the application's calls to the Ascend runtime API: + +```text +application + ↓ Ascend API call +libvnpu.so intercepts and accounts + ↓ only the allocated memory and compute are allowed +Ascend driver + ↓ +physical NPU +``` + +With a quota of 8192 MiB, a device query inside the container reports only 8192 MiB, allocations are accounted by `libvnpu.so`, and over-quota requests are blocked by the interception layer. The [ascend-device-plugin README](https://github.com/Project-HAMi/ascend-device-plugin) documents both modes explicitly. The comparison: + +| | Hard slicing | Soft slicing | +| :-- | :-- | :-- | +| Isolation boundary | enforced by the device virtualization layer, stronger | software runtime interception, not an SR-IOV-class hardware boundary | +| Sizing | vendor templates only, for example 8 GiB or 16 GiB steps | arbitrary MiB and compute ratios | +| Partitioned units | AI cores, AI CPUs, memory, DVPP | memory and compute quotas | +| Requirements | chip and driver support for the templates | `libvnpu.so` injection and driver compatibility, currently ARM-only | + +One analogy: hard slicing builds real walls inside the house; soft slicing keeps one shared house but puts a strict accountant and rate limiter at every door. + +## Two ways Volcano can schedule Ascend vNPUs + +There are **two different ways** Volcano can schedule Ascend virtual NPUs, and they are easy to confuse. Getting this right up front saves hours of debugging: + +| | MindCluster mode | HAMi mode | +| :-- | :-- | :-- | +| Volcano flag | `deviceshare.AscendMindClusterVNPUEnable` | `deviceshare.AscendHAMiVNPUEnable` | +| Provided by | Volcano's native Ascend plugin | [Project-HAMi/ascend-device-plugin](https://github.com/Project-HAMi/ascend-device-plugin) | +| Templates | `vir04_3c_ndvpp` (have a `dvpp` dimension) | `vir05_1c_16g` (fields `memory`/`aiCore`/`aiCPU` only) | +| Slicing modes | Driver templates (hard slicing) | Driver templates (hard slicing) by default, plus `hami-core` soft slicing when the Pod sets `huawei.com/vnpu-mode: hami-core` | +| Resource names | `huawei.com/npu-core` | `huawei.com/Ascend310P`, `-memory` | + +This post is about **HAMi mode** with **`hami-vnpu-core` soft slicing**. Note that HAMi mode is not the same thing as soft-slicing mode: the same ascend-device-plugin supports both template-based hard slicing and hami-vnpu-core soft slicing, and the Pod's annotation selects the path. The lab's Pods set the annotation, so they take the soft path, which is the only one of the two Volcano modes that does runtime interception: instead of pre-cutting the card into fixed virtualization templates, HAMi-core intercepts Ascend calls in user space and enforces per-container memory and compute limits at runtime. Volcano decides which Pod gets which slice; HAMi-core makes that decision stick. + +## What the Volcano integration actually adds + +First, the naming: **HAMi-core** is the umbrella name for this family of in-container runtime isolation, and it originally referred to `libvgpu.so` on NVIDIA GPUs; **hami-vnpu-core** is the Ascend implementation, and what it actually injects is `libvnpu.so`. + +Ascend soft slicing itself is not new. The support landed in April 2026 ([ascend-device-plugin integrated hami-vnpu-core](https://github.com/Project-HAMi/ascend-device-plugin/pull/61), [HAMi added the Ascend ResourceCoreName and soft-slicing scheduling support](https://github.com/Project-HAMi/HAMi/pull/1771)), and through HAMi 2.9 the flow already works: the HAMi scheduler allocates Ascend slices, `ascend-device-plugin` mounts the devices, and `hami-vnpu-core` enforces the memory and compute limits in the container. + +What the Volcano integration in the upcoming HAMi 2.10 adds is the allocator: it swaps the HAMi scheduler for Volcano, while the layers below are reused as they are. + +```text +HAMi 2.9: +HAMi Scheduler → ascend-device-plugin → hami-vnpu-core → NPU + +HAMi 2.10 / Volcano integration: +Volcano Scheduler → ascend-device-plugin → hami-vnpu-core → NPU +``` + +| Layer | HAMi 2.9 path | Volcano integration path | +| :-- | :-- | :-- | +| Scheduler | HAMi Scheduler | Volcano Scheduler | +| Device discovery and mounting | ascend-device-plugin | the same ascend-device-plugin | +| Soft-slicing enforcement | hami-vnpu-core (`libvnpu.so`) | the same hami-vnpu-core | +| Memory and compute isolation | already supported | reused | +| Queues, gang scheduling | not the focus | provided by Volcano | +| binpack / spread | HAMi policy | Volcano deviceshare policy | +| Monitoring and mixed soft/hard management | early stage | further completed in 2.10 | + +Volcano can now understand these HAMi Ascend resources and decide which Pod uses which physical NPU, whether several Pods binpack onto the same card, whether a group of training Pods meets its gang condition, and which queue, priority, and preemption policy applies. The precise statement is therefore: **the Volcano integration completes scheduling of Ascend HAMi-core soft-sliced resources under Volcano, together with monitoring and mixed soft/hard management; the Ascend soft-slicing capability itself existed before.** + +## How the integration works + +The path has three responsibilities: + +- **Volcano's `deviceshare` plugin** reads the vNPU geometries from the `hami-scheduler-device` ConfigMap (with `AscendHAMiVNPUEnable: "true"`) and, together with the `binpack` or `spread` policy, decides which node and card serves each Pod. +- **The `ascend-device-plugin` DaemonSet** registers `huawei.com/Ascend310P` (card count) and `huawei.com/Ascend310P-memory` (MiB) as extended resources, and copies the HAMi-core assets (`libvnpu.so` and `ld.so.preload`) onto the host at `/usr/local/hami-vnpu-core/`. +- **HAMi-core (`libvnpu.so`)** is injected into workload containers through Ascend Docker Runtime's preload mechanism and enforces the slice the scheduler chose. + +```mermaid +%% title: Volcano HAMi-mode Ascend vNPU soft-slicing path +graph TD + CAP["ascend-device-plugin
advertises vNPU capacity"] + CM["hami-scheduler-device ConfigMap
templates + hamiVnpuCore: true"] + VOL["Volcano deviceshare plugin
AscendHAMiVNPUEnable: true"] + SCHD["Volcano Scheduler
binds Pod to a vNPU slice"] + HOOK["hami-vnpu-core
libvnpu.so intercepts Ascend calls"] + ENF["Enforce per-container
memory + compute limits"] + MON[":9395/metrics
per-container HBM + utilization"] + + CAP --> VOL + CM -. "KnownGeometriesCMName" .-> VOL + VOL --> SCHD --> HOOK --> ENF + HOOK -. "shared-memory accounting" .-> MON + + style VOL fill:#d9f99d,stroke:#4f7d00,stroke-width:2px,color:#1f2937 + style HOOK fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#1f2937 + style ENF fill:#dcfce7,stroke:#0b6b3c,stroke-width:2px,color:#1f2937 + style MON fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 +``` + +The Pod contract is minimal: `schedulerName: volcano`, `runtimeClassName: ascend`, the annotation `huawei.com/vnpu-mode: hami-core`, and limits on the two extended resources. Without the annotation, the Pod falls back to the template-based path and can stay Pending on a soft-slicing node. + +The [hami-vnpu-core](https://github.com/Project-HAMi/hami-vnpu-core) library is preloaded via `/etc/ld.so.preload` inside the container, communicates through a shared-memory region at `/hami-shared-region`, and receives the enforced quota through environment variables such as `NPU_MEM_QUOTA`. + +## The basic procedure + +Everything ran on a single-node Kubernetes 1.28 cluster on the Ascend 310P3 server. Lab 13 carries every command and captured output; the essential steps: + +1. **Prepare the node.** Driver/npu-smi ≥ 25.5, ascend-docker-runtime installed, and the node labeled `ascend=on`. +2. **Install Volcano ≥ 1.16.** Soft slicing requires 1.16, and no stable 1.16 existed at the time of writing (latest stable: v1.15.1, plus a `1.16.0-alpha.1` chart), so the verification built Volcano master (`7d9504320`) from source as Lab 13 shows. Once a stable release ships, install it directly with Helm and skip the compile: + + ```bash + helm repo add volcano-sh https://volcano-sh.github.io/helm-charts + helm install volcano volcano-sh/volcano \ + --namespace volcano-system --create-namespace \ + --version 1.16.0 + ``` + +3. **Enable HAMi-mode deviceshare.** Override `volcano-scheduler-configmap` so the `deviceshare` plugin runs with `AscendHAMiVNPUEnable: "true"`, `SchedulePolicy: binpack`, and `KnownGeometriesCMName: hami-scheduler-device`, then restart the scheduler. +4. **Deploy the plugin in soft-slicing mode.** Apply the `ascend` RuntimeClass, set `hamiVnpuCore: true` in the `hami-scheduler-device` ConfigMap plus `hami-vnpu-core: true` for the node in `hami-device-node-config`, and apply the DaemonSet. The node then advertises `huawei.com/Ascend310P` and `huawei.com/Ascend310P-memory`. +5. **Run a soft-sliced Pod.** `schedulerName: volcano`, `runtimeClassName: ascend`, the `huawei.com/vnpu-mode: hami-core` annotation, and limits of one card plus 8192 MiB of memory. +6. **Verify.** `npu-smi info` inside the Pod shows the slice rather than the card, a second Pod binpacks onto the same physical card, and the plugin's `:9395` endpoint exports per-container metrics. + +## What we verified + +The host reported two healthy 310P3 cards: + +```text +$ npu-smi info ++--------------------------------------------------------------------------------------------------------+ +| npu-smi 25.5.1 Version: 25.5.1 | ++-------------------------------+-----------------+------------------------------------------------------+ +| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page) | +| Chip Device | Bus-Id | AICore(%) Memory-Usage(MB) | ++===============================+=================+======================================================+ +| 4 310P3 | OK | NA 37 0 / 0 | +| 0 0 | 0000:81:00.0 | 0 1848 / 21525 | ++===============================+=================+======================================================+ +| 5 310P3 | OK | NA 40 0 / 0 | +| 0 1 | 0000:85:00.0 | 0 1849 / 21525 | ++===============================+=================+======================================================+ +``` + +With the plugin registered, the node advertised 14 vNPUs (2 cards × 7, matching `vDeviceCount: 7`) and 43054 MiB of allocatable memory. The memory figure follows the chip config (`memoryAllocatable: 21527` MB per card), which sits 2 MiB per card above the 21525 MB `npu-smi` displays. Each test Pod requested one vNPU with an 8192 MiB slice: + +```yaml +resources: + limits: + huawei.com/Ascend310P: "1" + huawei.com/Ascend310P-memory: "8192" +``` + +### 1. The container sees only its slice + +Inside the first Pod, `npu-smi info` reported a **0 / 8192 MB** device, not the 1848 / 21525 MB the host sees on the same card: + +```text +$ kubectl exec ascend-vnpu-check -- npu-smi info +[INFO limiter::supervisor] [Supervisor PID:10] won manager election +[INFO limiter::manager] [Manager] Registered as Global Manager #0 (PID: 10). Compute limit: 1, Memory limit: 8192, FixedShare: false +open global registry path is "/hami-shared-region/0_global_registry" +... +| 32768 310P3 | OK | NA 38 0 / 0 | +| 0 0 | 0000:81:00.0 | 0 0 / 8192 | +``` + +`libvnpu.so` had rewritten the device query to the container's quota, and the injected environment confirmed the wiring: `NPU_MEM_QUOTA=8192`, `NPU_GLOBAL_SHM_PATH=/hami-shared-region/0_global_registry`, `ASCEND_VISIBLE_DEVICES=0`. + +### 2. binpack shares one physical card between Pods + +A second, identical Pod landed on the **same Bus-Id `0000:81:00.0`**, each with its own 8192 MiB window, and registered as `Global Manager #1` in the same shared registry as Pod 1's `#0`: + +```text +$ kubectl exec ascend-vnpu-check-2 -- npu-smi info | grep -E "Memory limit|0000" +[INFO limiter::manager] [Manager] Registered as Global Manager #1 (PID: 10). Compute limit: 1, Memory limit: 8192, FixedShare: false +| 0 0 | 0000:81:00.0 | 0 0 / 8192 | +``` + +The node's allocated resources told the same story: `huawei.com/Ascend310P 2` (of 14) and `huawei.com/Ascend310P-memory 16384` (of 43054), two 8192 MiB slices packed onto one 21.5 GiB card instead of spread across two. + +### 3. Per-container metrics are exported + +The plugin (not the workload Pod) serves Prometheus metrics on `:9395`: + +```text +hami_vgpu_memory_limit_bytes{container="npu",...,pod="ascend-vnpu-check",vdevice_index="0"} 8.589934592e+09 +hami_vgpu_memory_limit_bytes{container="npu",...,pod="ascend-vnpu-check-2",vdevice_index="0"} 8.589934592e+09 +hami_host_gpu_memory_used_bytes{device_index="0",device_type="Ascend-Atlas 300I Pro",...} 1.937768448e+09 +``` + +`8.589934592e+09` bytes is exactly 8192 MiB, matching both Pods' requests, and both vdevices carry the same physical-card UUID. The endpoint also exports `hami_vgpu_memory_used_bytes`, `hami_container_device_utilization_ratio`, and `hami_host_gpu_utilization_ratio`. + +## Results + +| Verification | Result | Evidence | +| :-- | :-- | :-- | +| Volcano source build (aarch64) | Pass | 3 images, scheduler `--version` reports commit `7d950432...` | +| Plugin image with matching libvnpu | Pass | `libvnpu.so` asset verified against the release matching driver 25.5.1 | +| Volcano + HAMi-mode deviceshare | Pass | scheduler log loads `AscendHAMiVNPUEnable: "true"` | +| Node resource registration | Pass | `Ascend310P: 14`, `Ascend310P-memory: 43054` | +| Memory slice isolation | Pass | container `0 / 8192` vs host `1848 / 21525` | +| binpack card sharing | Pass | both Pods on Bus-Id `0000:81:00.0`, `Global Manager #0/#1` | +| Resource accounting | Pass | node allocated 2 vNPU, 16384 MiB | +| Monitoring | Pass | `:9395` exports host/container/vdevice metrics | + +## Gotchas worth knowing before you try + +- **`libvnpu.so` must match the NPU driver.** A mismatch does not produce an error; in-container `npu-smi` just hangs at `Initialize SchedulerClient...`. Copy the asset from the official image release that matches your driver and verify the md5. +- **Docker and containerd have separate image stores.** Import with `ctr -n k8s.io images import` or expect `ErrImageNeverPull`. +- **The Helm key is `basic.image_pull_policy`** (underscore), not `scheduler.imagePullPolicy`. With the wrong key, nodes try to pull images that only exist locally. +- **The `-core` resource is not registered in v1.4.0.** A Pod spec only needs `huawei.com/Ascend310P` (count) and `huawei.com/Ascend310P-memory` (MiB); the `resourceCoreName` entry in the config is not reported as a node resource. +- **Metrics live on the plugin Pod.** Curling `:9395` from the workload Pod returns nothing; select the DaemonSet Pod by label. +- **Uninstalling Volcano can leave `volcano-system` Terminating** after the webhooks are gone. Clearing the namespace finalizer unblocks it. + +Soft slicing here is runtime API-level enforcement (software interception via `libvnpu.so`), not an SR-IOV-style hardware security boundary, which is the same caveat that applies to HAMi-core on GPUs. + +## Next steps + +- Full procedure with every command and captured output: [Lab 13: Soft-Slicing Ascend 310P3 vNPU with Volcano and HAMi-core](/tutorials/labs/volcano-ascend-vnpu) +- User guide: [Huawei Ascend devices in Volcano](/docs/installation/how-to-use-volcano-ascend) and [Enable Ascend sharing](/docs/userguide/ascend-device/enable-ascend-sharing) +- Hard slicing on Ascend: [NPU virtualization hard-slicing practice (Huawei)](https://www.hiascend.com/developer/techArticles/20251212-1) +- Components: [Project-HAMi/ascend-device-plugin](https://github.com/Project-HAMi/ascend-device-plugin) · [Project-HAMi/hami-vnpu-core](https://github.com/Project-HAMi/hami-vnpu-core) · [volcano-sh/volcano](https://github.com/volcano-sh/volcano) +- Related: [Lab 8: Volcano vGPU with Gang Scheduling and Queues](/tutorials/labs/volcano-vgpu-gang-queue) applies the same scheduler to NVIDIA GPUs diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/index.md b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/index.md index da9fb154f..0dda7203b 100644 --- a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/index.md +++ b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/index.md @@ -5,6 +5,7 @@ description: "12 月 27 日，HAMi Meetup 北京站近百位技术伙伴齐聚 authors: [hami_community] tags: ["HAMi", "Meetup", "异构算力", "GPU 虚拟化", "云原生"] image: "./hami-meetup-beijing-banner.webp" +unlisted: true --- ![HAMi Meetup 北京站](hami-meetup-beijing-banner.webp) diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/index.md b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/index.md index b1cbd386c..58f0fd28f 100644 --- a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/index.md +++ b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/index.md @@ -6,6 +6,7 @@ image: ./meetup-banner.png tags: ["Meetup", "上海", "异构算力调度", "GPU 虚拟化", "Kubernetes", "国产算力", "AI 训练与推理优化"] authors: [hami_community] +unlisted: true --- 11 月 30 日，首场 HAMi Meetup 在上海圆满结束。本次活动以"不卷算力卷效率"为主题，近百位 AI 开发者、运维工程师、企业 IT 架构师齐聚现场，聚焦异构算力调度的核心命题。 diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shenzhen-2026/index.md b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shenzhen-2026/index.md index 63ab18c91..dc1e8365f 100644 --- a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shenzhen-2026/index.md +++ b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shenzhen-2026/index.md @@ -5,6 +5,7 @@ description: "4 月 25 日，HAMi Meetup 深圳站成功举办。来自 CNCF、 image: ./meetup-shenzhen-group-photo.webp tags: ["HAMi", "Meetup", "深圳", "GPU 虚拟化", "异构算力调度", "DRA", "云原生", "AI 基础设施"] authors: [hami_community] +unlisted: true --- ![HAMi Meetup 深圳站](meetup-shenzhen-group-photo.webp) diff --git a/i18n/zh/docusaurus-plugin-content-blog/tags.yml b/i18n/zh/docusaurus-plugin-content-blog/tags.yml index 623662468..ed1caa5cc 100644 --- a/i18n/zh/docusaurus-plugin-content-blog/tags.yml +++ b/i18n/zh/docusaurus-plugin-content-blog/tags.yml @@ -82,6 +82,18 @@ "NVIDIA": label: "NVIDIA" permalink: "/nvidia" +"Volcano": + label: "Volcano" + permalink: "/volcano" +"Ascend": + label: "Ascend" + permalink: "/ascend" +"vNPU": + label: "vNPU" + permalink: "/v-npu" +"软切分": + label: "软切分" + permalink: "/soft-slicing" "Release": label: "Release" permalink: "/release" diff --git a/i18n/zh/docusaurus-plugin-content-blog/volcano-ascend-vnpu-soft-slicing/index.md b/i18n/zh/docusaurus-plugin-content-blog/volcano-ascend-vnpu-soft-slicing/index.md new file mode 100644 index 000000000..9e191ff25 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-blog/volcano-ascend-vnpu-soft-slicing/index.md @@ -0,0 +1,255 @@ +--- +title: "用 Volcano + HAMi-core 软切分昇腾 vNPU：原理与真机验证" +date: "2026-08-17" +description: "Volcano 的 HAMi 模式 deviceshare 如何把批量调度与 HAMi-core 运行时隔离结合到昇腾 NPU 上，在昇腾 310P3 ARM 服务器上从源码编译镜像开始，端到端验证容器内显存切片、binpack 共卡与容器级监控指标。" +authors: [rootsongjc] +tags: ["HAMi", "Volcano", "Ascend", "vNPU", "软切分", "Kubernetes"] +--- + +[Volcano](https://github.com/volcano-sh/volcano) 是很多 AI 集群的批量调度器首选，HAMi-core 则是让共享加速器"守规矩"的运行时。本文关注两者在昇腾硬件上的交汇点：在 **Volcano 调度器下运行 `hami-vnpu-core` 软切分的 vNPU**，让批量调度语义（队列、Gang、binpack）与容器级隔离（在昇腾 API 层强制生效的显存与算力上限）协同工作。 + +我们在一台昇腾 310P3 aarch64 服务器的单节点 Kubernetes 集群上验证了完整链路：源码编译 Volcano 镜像、部署官方 [ascend-device-plugin](https://github.com/Project-HAMi/ascend-device-plugin) v1.4.0 镜像，并确认申请 8192 MiB 切片的容器恰好只能看到这么多显存；同时第二个 Pod 以 binpack 方式落到同一张物理卡上、拿到独立切片，插件的 Prometheus 端点也如实上报了两个容器的配额。完整步骤（含每条命令与真实输出）见 [实验 13：用 Volcano + HAMi-core 软切分昇腾 310P3 vNPU](/zh/tutorials/labs/volcano-ascend-vnpu)。 + +这个话题里混着好几个经常被混为一谈的概念，所以本文先把层次分开：vNPU 是什么、硬切分和软切分有何不同、Volcano 集成相比已有的 HAMi 调度器路径到底新增了什么。 + +:::note 关于本文中的输出 + +本文所有输出均采集自真实的昇腾 310P3 物理服务器，截至撰写本文时已在真机上验证：麒麟 V10 aarch64 节点、2× 昇腾 310P3（驱动/npu-smi 25.5.1）、Kubernetes v1.28.15、containerd 1.7.1。其他集群中的 UUID、IP、Pod 后缀会不同；请对比组件名、调度位置与测量值。 + +::: + + + +## 先把 NPU、vNPU、硬切分、软切分分清楚 + +vNPU、硬切分、软切分、HAMi 模式、Volcano 支持分别处在不同的层次，混在一起看就会显得像同一件事。它们各自回答不同的问题： + +| 层次 | 回答的问题 | +| :-------------- | :--------------------------------------------------------------- | +| NPU / vNPU | 你拿到的是什么设备：vNPU 是从物理 NPU 划出、提供给容器的逻辑设备 | +| 硬切分 / 软切分 | 这个虚拟设备如何实现隔离 | +| HAMi-core | 谁在容器内执行软切分配额 | +| HAMi / Volcano | 由谁决定 Pod 用哪张卡、用多少资源 | + +NPU 是物理设备，例如服务器里真实存在的一张昇腾 310P3；vNPU 是从物理 NPU 上划出来、提供给容器使用的逻辑设备（一张 21.5 GiB 的卡可以划成 8 GiB 加 8 GiB 两份切片，剩余约 5.5 GiB）。但“vNPU”只表示结果，并不说明实现：切片既可能由昇腾驱动的虚拟化能力切出来，也可能由 HAMi-core 用软件模拟出来。这也是本话题最容易产生歧义的地方：本文所说的“软切分 vNPU”是 Kubernetes/HAMi 视角下的逻辑切片，并不是通过 `npu-smi ... create-vnpu` 创建的昇腾硬件 vNPU。 + +**硬切分**由昇腾驱动/固件的虚拟化能力完成，用户只能从预定义模板中选择：`vir05_1c_16g` 代表固定数量的 AI Core、AI CPU 和 16 GiB 显存，创建后设备层会产生一个真正的 vNPU 实例。可用 `npu-smi info -t template-info` 查询芯片支持的模板，完整实践见昇腾社区的[《NPU 卡虚拟化硬切分参考实践》](https://www.hiascend.com/developer/techArticles/20251212-1)。 + +**软切分**不在硬件里真正创建 vNPU，而是让多个容器共享同一张物理 NPU，在容器内注入 `libvnpu.so` 拦截应用对昇腾运行时 API 的调用并记账： + +```text +应用 + ↓ 昇腾 API 调用 +libvnpu.so 拦截并记账 + ↓ 只允许使用分配到的显存和算力 +昇腾驱动 + ↓ +物理 NPU +``` + +以 8192 MiB 配额为例：应用查询设备时只看到 8192 MiB，申请显存由 `libvnpu.so` 记账，超过配额的请求被运行时拦截层阻止，多个容器可以共享同一张物理卡。官方 [ascend-device-plugin 文档](https://github.com/Project-HAMi/ascend-device-plugin)也明确区分了“模板硬切分”与“运行时拦截的软切分”两种模式。两者对比： + +| | 硬切分 | 软切分 | +| :-- | :-- | :-- | +| 隔离边界 | 设备虚拟化层强制，更强 | 软件运行时拦截，不等于 SR-IOV 级硬件边界 | +| 规格 | 仅厂商模板，例如 8 GiB、16 GiB 档位 | 任意 MiB、算力比例 | +| 切分单元 | AI Core、AI CPU、显存、DVPP | 显存与算力配额 | +| 前置条件 | 芯片/驱动需支持对应模板 | 依赖 `libvnpu.so` 注入与驱动兼容，当前仅支持 ARM | + +可以把两者理解成：**硬切分是在房子里真正砌墙；软切分是大家共用房子，但每个门口都有一个严格记账和限流的管理员。** + +## Volcano 调度昇腾 vNPU 的两种方式 + +Volcano 调度昇腾虚拟 NPU 有**两种不同方式**，很容易混淆。先把它讲清楚，能省掉好几个小时的排错： + +| | MindCluster 模式 | HAMi 模式 | +| :-- | :-- | :-- | +| Volcano 开关 | `deviceshare.AscendMindClusterVNPUEnable` | `deviceshare.AscendHAMiVNPUEnable` | +| 提供方 | Volcano 原生昇腾插件 | [Project-HAMi/ascend-device-plugin](https://github.com/Project-HAMi/ascend-device-plugin) | +| 模板 | `vir04_3c_ndvpp`（带 `dvpp` 维度） | `vir05_1c_16g`（仅 `memory`/`aiCore`/`aiCPU` 字段） | +| 切分方式 | 驱动模板（硬切分） | 默认走驱动模板（硬切分），Pod 设置 `huawei.com/vnpu-mode: hami-core` 后走 `hami-core` 软切分 | +| 资源名 | `huawei.com/npu-core` | `huawei.com/Ascend310P`、`-memory` | + +本文讲的是 **HAMi 模式**下的 **`hami-vnpu-core` 软切分**。注意，HAMi 模式并不等于软切分模式：同一个 ascend-device-plugin 同时支持模板硬切分与 hami-vnpu-core 软切分，由 Pod 的注解选择路径。实验中的 Pod 设置了该注解，因此走的是软切分路径，也是两种 Volcano 模式里唯一做运行时拦截的一种：不是把卡预先切成固定的虚拟化模板，而是在用户态拦截昇腾调用，在运行时按容器强制显存与算力上限。Volcano 决定哪个 Pod 拿到哪个切片，HAMi-core 让这个决定真正生效。 + +## Volcano 集成到底新增了什么 + +先纠正一个名字：**HAMi-core** 是这一类容器内运行时隔离技术的统称，最早主要指 NVIDIA 的 `libvgpu.so`；**hami-vnpu-core** 是专门面向昇腾 NPU 的实现，实际注入的是 `libvnpu.so`。 + +昇腾软切分能力本身并不是新东西。相关支持在 2026 年 4 月已经进入代码（[ascend-device-plugin 集成 hami-vnpu-core](https://github.com/Project-HAMi/ascend-device-plugin/pull/61)、[HAMi 增加昇腾 ResourceCoreName 和软切分调度支持](https://github.com/Project-HAMi/HAMi/pull/1771)），在 HAMi 2.9 中链路已经可用：HAMi 自己的调度器分配昇腾切片，`ascend-device-plugin` 完成设备挂载，`hami-vnpu-core` 在容器内执行显存和算力限制。 + +即将发布的 HAMi 2.10 中的 Volcano 集成没有重新发明软切分，而是把"分配者"从 HAMi Scheduler 换成了 Volcano，底下两层原样复用： + +```text +HAMi 2.9: +HAMi Scheduler → ascend-device-plugin → hami-vnpu-core → NPU + +HAMi 2.10 / Volcano 集成： +Volcano Scheduler → ascend-device-plugin → hami-vnpu-core → NPU +``` + +| 层次 | HAMi 2.9 路径 | Volcano 集成路径 | +| :----------------- | :----------------------------- | :-------------------------- | +| 调度器 | HAMi Scheduler | Volcano Scheduler | +| 设备发现/挂载 | ascend-device-plugin | 同一个 ascend-device-plugin | +| 软切分执行 | hami-vnpu-core（`libvnpu.so`） | 同一个 hami-vnpu-core | +| 显存、算力隔离 | 已支持 | 复用原有能力 | +| 队列、Gang 调度 | 不是重点 | Volcano 提供 | +| binpack / spread | HAMi 策略 | Volcano deviceshare 策略 | +| 监控及软硬混合管理 | 相对早期 | 2.10 进一步补全 | + +Volcano 现在能理解这些 HAMi 昇腾资源，并决定：哪个 Pod 使用哪张物理 NPU、多个 Pod 是否 binpack 到同一张卡、整组训练 Pod 是否满足 Gang 条件、使用哪个队列、优先级和抢占策略。因此准确的说法是：**这次集成完成的是"Volcano 调度昇腾 HAMi-core 软切分资源"，并补充监控及软硬切分混合管理；昇腾软切分能力本身早已存在。** + +## 集成是如何工作的 + +这条链路有三个分工： + +- **Volcano 的 `deviceshare` 插件**从 `hami-scheduler-device` ConfigMap（配合 `AscendHAMiVNPUEnable: "true"`）读取 vNPU 规格，并按 `binpack` 或 `spread` 策略决定每个 Pod 由哪个节点、哪张卡服务。 +- **`ascend-device-plugin` DaemonSet** 向节点注册 `huawei.com/Ascend310P`（卡数）与 `huawei.com/Ascend310P-memory`（MiB）扩展资源，并把 HAMi-core 资产（`libvnpu.so` 和 `ld.so.preload`）拷贝到宿主机 `/usr/local/hami-vnpu-core/`。 +- **HAMi-core（`libvnpu.so`）** 经 Ascend Docker Runtime 的 preload 机制注入业务容器，强制执行调度器选定的切片。 + +```mermaid +%% title: Volcano HAMi 模式昇腾 vNPU 软切分链路 +graph TD + CAP["ascend-device-plugin
上报 vNPU 容量"] + CM["hami-scheduler-device ConfigMap
模板 + hamiVnpuCore: true"] + VOL["Volcano deviceshare 插件
AscendHAMiVNPUEnable: true"] + SCHD["Volcano Scheduler
将 Pod 绑定到某个 vNPU 切片"] + HOOK["hami-vnpu-core
libvnpu.so 拦截昇腾调用"] + ENF["按容器强制
显存 + 算力上限"] + MON[":9395/metrics
容器级 HBM + 利用率"] + + CAP --> VOL + CM -. "KnownGeometriesCMName" .-> VOL + VOL --> SCHD --> HOOK --> ENF + HOOK -. "共享内存记账" .-> MON + + style VOL fill:#d9f99d,stroke:#4f7d00,stroke-width:2px,color:#1f2937 + style HOOK fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#1f2937 + style ENF fill:#dcfce7,stroke:#0b6b3c,stroke-width:2px,color:#1f2937 + style MON fill:#dbeafe,stroke:#1a5fb4,stroke-width:2px,color:#1f2937 +``` + +Pod 侧的契约很简洁：`schedulerName: volcano`、`runtimeClassName: ascend`、注解 `huawei.com/vnpu-mode: hami-core`，以及对两个扩展资源的 limits。少了这个注解，Pod 会退回模板路径，在纯软切分节点上可能一直 Pending。 + +[hami-vnpu-core](https://github.com/Project-HAMi/hami-vnpu-core) 的 `libvnpu.so` 在容器内通过 `/etc/ld.so.preload` 预加载，经 `/hami-shared-region` 共享内存区通信，并通过 `NPU_MEM_QUOTA` 等环境变量拿到强制配额。 + +## 基本步骤 + +以下步骤都在昇腾 310P3 服务器的单节点 Kubernetes 1.28 集群上执行，完整命令与真实输出见实验 13，核心流程如下： + +1. **准备节点。** 驱动/npu-smi ≥ 25.5，安装 ascend-docker-runtime，并为节点打上 `ascend=on` 标签。 +2. **安装 Volcano ≥ 1.16。** 软切分要求 1.16，而撰写本文时还没有稳定的 1.16（最新稳定版为 v1.15.1，仅有 `1.16.0-alpha.1` chart），因此验证时按实验 13 从源码编译了 Volcano master（commit `7d9504320`）。等稳定版本发布后，直接用 Helm 安装即可跳过编译： + + ```bash + helm repo add volcano-sh https://volcano-sh.github.io/helm-charts + helm install volcano volcano-sh/volcano \ + --namespace volcano-system --create-namespace \ + --version 1.16.0 + ``` + +3. **开启 HAMi 模式 deviceshare。** 覆盖 `volcano-scheduler-configmap`，让 `deviceshare` 插件带上 `AscendHAMiVNPUEnable: "true"`、`SchedulePolicy: binpack`、`KnownGeometriesCMName: hami-scheduler-device`，然后重启调度器。 +4. **以软切分模式部署插件。** 应用 `ascend` RuntimeClass，在 `hami-scheduler-device` ConfigMap 中打开 `hamiVnpuCore: true`，并在 `hami-device-node-config` 中为节点打开 `hami-vnpu-core: true`，再部署 DaemonSet。节点随后上报 `huawei.com/Ascend310P` 与 `huawei.com/Ascend310P-memory` 资源。 +5. **运行软切分 Pod。** `schedulerName: volcano`、`runtimeClassName: ascend`、注解 `huawei.com/vnpu-mode: hami-core`，limits 申请 1 张卡加 8192 MiB 显存。 +6. **验证。** Pod 内 `npu-smi info` 显示的是切片而不是整卡，第二个 Pod 会 binpack 到同一张物理卡，插件的 `:9395` 端点导出容器级指标。 + +## 我们验证了什么 + +宿主机上两张 310P3 均为健康状态： + +```text +$ npu-smi info ++--------------------------------------------------------------------------------------------------------+ +| npu-smi 25.5.1 Version: 25.5.1 | ++-------------------------------+-----------------+------------------------------------------------------+ +| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page) | +| Chip Device | Bus-Id | AICore(%) Memory-Usage(MB) | ++===============================+=================+======================================================+ +| 4 310P3 | OK | NA 37 0 / 0 | +| 0 0 | 0000:81:00.0 | 0 1848 / 21525 | ++===============================+=================+======================================================+ +| 5 310P3 | OK | NA 40 0 / 0 | +| 0 1 | 0000:85:00.0 | 0 1849 / 21525 | ++===============================+=================+======================================================+ +``` + +插件注册后，节点上报 14 个 vNPU（2 卡 × 7，与 `vDeviceCount: 7` 一致）与 43054 MiB 可分配显存。显存数字来自芯片配置（每卡 `memoryAllocatable: 21527` MB），比 `npu-smi` 显示的 21525 MB 每卡多 2 MiB。每个测试 Pod 申请 1 个 vNPU、8192 MiB 切片： + +```yaml +resources: + limits: + huawei.com/Ascend310P: "1" + huawei.com/Ascend310P-memory: "8192" +``` + +### 1. 容器只看到自己的切片 + +第一个 Pod 内的 `npu-smi info` 显示的是 **0 / 8192 MB** 的设备，而不是宿主机在同一张卡上看到的 1848 / 21525 MB： + +```text +$ kubectl exec ascend-vnpu-check -- npu-smi info +[INFO limiter::supervisor] [Supervisor PID:10] won manager election +[INFO limiter::manager] [Manager] Registered as Global Manager #0 (PID: 10). Compute limit: 1, Memory limit: 8192, FixedShare: false +open global registry path is "/hami-shared-region/0_global_registry" +... +| 32768 310P3 | OK | NA 38 0 / 0 | +| 0 0 | 0000:81:00.0 | 0 0 / 8192 | +``` + +`libvnpu.so` 已把设备查询改写为容器的配额，注入的环境变量也印证了接线：`NPU_MEM_QUOTA=8192`、`NPU_GLOBAL_SHM_PATH=/hami-shared-region/0_global_registry`、`ASCEND_VISIBLE_DEVICES=0`。 + +### 2. binpack 让多个 Pod 共享一张物理卡 + +第二个同规格的 Pod 落在了**同一个 Bus-Id `0000:81:00.0`** 上，各自拥有独立的 8192 MiB 窗口，并作为 `Global Manager #1` 注册进与 Pod 1 的 `#0` 相同的共享注册表： + +```text +$ kubectl exec ascend-vnpu-check-2 -- npu-smi info | grep -E "Memory limit|0000" +[INFO limiter::manager] [Manager] Registered as Global Manager #1 (PID: 10). Compute limit: 1, Memory limit: 8192, FixedShare: false +| 0 0 | 0000:81:00.0 | 0 0 / 8192 | +``` + +节点的已分配资源说的是同一件事：`huawei.com/Ascend310P 2`（共 14）、`huawei.com/Ascend310P-memory 16384`（共 43054），两个 8192 MiB 切片打包进一张 21.5 GiB 的卡，而不是分散到两张卡。 + +### 3. 容器级监控指标正常导出 + +插件（不是业务 Pod）在 `:9395` 上提供 Prometheus 指标： + +```text +hami_vgpu_memory_limit_bytes{container="npu",...,pod="ascend-vnpu-check",vdevice_index="0"} 8.589934592e+09 +hami_vgpu_memory_limit_bytes{container="npu",...,pod="ascend-vnpu-check-2",vdevice_index="0"} 8.589934592e+09 +hami_host_gpu_memory_used_bytes{device_index="0",device_type="Ascend-Atlas 300I Pro",...} 1.937768448e+09 +``` + +`8.589934592e+09` 字节恰好是 8192 MiB，与两个 Pod 的申请值一致，且两个 vdevice 挂在同一张物理卡的 UUID 上。该端点还导出 `hami_vgpu_memory_used_bytes`、`hami_container_device_utilization_ratio` 和 `hami_host_gpu_utilization_ratio`。 + +## 验证结果 + +| 验证项 | 结果 | 证据 | +| :-- | :-- | :-- | +| Volcano 源码编译（aarch64） | 通过 | 3 个镜像，scheduler `--version` 输出 commit `7d950432...` | +| 插件镜像携带匹配的 libvnpu | 通过 | `libvnpu.so` 资产已对照匹配驱动 25.5.1 的版本核验 | +| Volcano + HAMi 模式 deviceshare | 通过 | 调度器日志加载 `AscendHAMiVNPUEnable: "true"` | +| 节点资源注册 | 通过 | `Ascend310P: 14`、`Ascend310P-memory: 43054` | +| 显存切片隔离 | 通过 | 容器内 `0 / 8192`，宿主机 `1848 / 21525` | +| binpack 共卡 | 通过 | 两个 Pod 同为 Bus-Id `0000:81:00.0`，`Global Manager #0/#1` | +| 资源计量 | 通过 | 节点已分配 2 vNPU、16384 MiB | +| 监控 | 通过 | `:9395` 导出宿主机/容器/vdevice 三层指标 | + +## 动手前值得知道的坑 + +- **`libvnpu.so` 必须与 NPU 驱动匹配。** 不匹配不会报错，容器内 `npu-smi` 只是永远卡在 `Initialize SchedulerClient...`。请从与驱动匹配的官方镜像版本拷贝资产并校验 md5。 +- **Docker 与 containerd 的镜像存储是隔离的。** 用 `ctr -n k8s.io images import` 导入，否则等着收 `ErrImageNeverPull`。 +- **Helm 里镜像拉取策略的 key 是 `basic.image_pull_policy`**（下划线），不是 `scheduler.imagePullPolicy`。key 写错时节点会去拉取只有本地才有的镜像。 +- **v1.4.0 不注册 `-core` 资源。** Pod spec 只需要 `huawei.com/Ascend310P`（卡数）与 `huawei.com/Ascend310P-memory`（MiB）；配置里的 `resourceCoreName` 不会上报为节点资源。 +- **指标在插件 Pod 上。** 在业务 Pod 里 curl `:9395` 毫无响应；要按 label 选中 DaemonSet Pod。 +- **卸载 Volcano 后 `volcano-system` 可能卡在 Terminating**（webhook 已删除之后）。清理 namespace 的 finalizer 即可解除。 + +这里的软切分是运行时 API 层的强制（`libvnpu.so` 软件拦截），不是 SR-IOV 式的硬件安全边界，这一点与 HAMi-core 在 GPU 上的定性相同。 + +## 延伸阅读 + +- 完整流程（每条命令与真实输出）：[实验 13：用 Volcano + HAMi-core 软切分昇腾 310P3 vNPU](/zh/tutorials/labs/volcano-ascend-vnpu) +- 用户指南：[Volcano 中的华为昇腾设备](/zh/docs/installation/how-to-use-volcano-ascend) 与 [启用昇腾共享](/zh/docs/userguide/ascend-device/enable-ascend-sharing) +- 昇腾硬切分：[《NPU 卡虚拟化硬切分参考实践》（昇腾社区）](https://www.hiascend.com/developer/techArticles/20251212-1) +- 组件仓库：[Project-HAMi/ascend-device-plugin](https://github.com/Project-HAMi/ascend-device-plugin) · [Project-HAMi/hami-vnpu-core](https://github.com/Project-HAMi/hami-vnpu-core) · [volcano-sh/volcano](https://github.com/volcano-sh/volcano) +- 相关实验：[实验 8：Volcano vGPU、Gang 调度与队列限制](/zh/tutorials/labs/volcano-vgpu-gang-queue) 把同一套调度器用于 NVIDIA GPU diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/volcano-ascend-vnpu.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/volcano-ascend-vnpu.md new file mode 100644 index 000000000..d6e185ede --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/labs/volcano-ascend-vnpu.md @@ -0,0 +1,702 @@ +--- +title: "实验 13：用 Volcano + HAMi-core 软切分昇腾 310P3 vNPU" +description: "在 aarch64 昇腾服务器上源码编译 Volcano 与 ascend-device-plugin，启用 hami-vnpu-core 软切分，并验证显存隔离、binpack 共卡与容器级监控指标。" +sidebar_label: "实验 13：Volcano 昇腾 vNPU" +lab: + level: Advanced + duration: 约 120 分钟 + environment: 昇腾 310P3 aarch64 服务器上的单节点 Kubernetes 1.28 集群（麒麟 V10、驱动 25.5.1、containerd） + cost: 需要专属的昇腾 310P3 aarch64 硬件；软切分仅支持 ARM + authors: + - rootsongjc + verified: "2026-08-14" +tags: + - volcano + - hami-core + - ascend + - vnpu + - soft-slicing +toc_max_heading_level: 2 +--- + +本实验从昇腾 310P3 aarch64 服务器上的干净 Kubernetes 集群开始，最终让两个 Pod 通过 `hami-vnpu-core` 软切分共享一张物理 NPU：各自锁定独立的 8192 MiB 显存窗口，并且都能被 Prometheus 指标观测到。 + +软切分要求 [Volcano](https://github.com/volcano-sh/volcano) ≥ 1.16，而验证时（最新稳定版 v1.15.1）仅有 `1.16.0-alpha.1` chart、尚无稳定版本，因此本实验从源码编译 Volcano master，插件直接使用官方 `v1.4.0` 镜像部署。如果你运行本实验时已发布稳定的 Volcano 1.16 chart，可以用 chart 安装替代第 3、5 步，其余步骤不变。 + +:::note 关于本文中的输出 + +下述输出均采集自 2026-08-14 的真机验证。节点名、IP、Pod 后缀与 UUID 因环境而异；请对比组件名、就绪状态、调度位置与测量值。 + +::: + +## 你将学到什么 + +- 在宿主机编译 Volcano 并把二进制打包成 containerd 可用的镜像； +- 拉取插件镜像并校验其 `libvnpu.so` 资产与 NPU 驱动匹配； +- 配置 Volcano 的 `deviceshare` 插件以 HAMi 模式 + `binpack` 调度 vNPU； +- 全局打开 `hamiVnpuCore`、按节点打开 `hami-vnpu-core`； +- 用 `npu-smi` 验证容器内显存隔离； +- 通过让两个 Pod 落在同一张物理卡验证 binpack 共享；以及 +- 从插件的 `:9395` 端点读取容器级 vNPU 指标。 + +## 实验概览 + +```mermaid +%% title: Volcano + HAMi-core 昇腾 vNPU 软切分实验流程 +flowchart LR + S1["步骤 1
验证环境"] --> S2["步骤 2
清理集群"] + S2 --> S3["步骤 3
编译 Volcano"] + S3 --> S4["步骤 4
插件镜像"] + S4 --> S5["步骤 5
部署 Volcano"] + S5 --> S6["步骤 6
部署插件"] + S6 --> S7["步骤 7
软切分 Pod"] + S7 --> S8["步骤 8
验证指标"] +``` + +## 前提条件 + +- 一台带昇腾 310P（或 310P3）NPU 的 aarch64 服务器，驱动/npu-smi **≥ 25.5**，并安装了 [ascend-docker-runtime](https://gitcode.com/Ascend/mind-cluster/tree/master/component/ascend-docker-runtime)（软切分仅支持 ARM）。 +- 该服务器上一个使用 containerd 的 Kubernetes ≥ 1.20 集群。验证集群为单节点 kubeadm 集群（节点 `aio-node74-arm`，同时是控制面和工作节点），麒麟 V10、Kubernetes v1.28.15、containerd 1.7.1。 +- 宿主机上：Go 1.26（验证机为 `go1.26.2 linux/arm64`）、带 Buildx 的 Docker 24（仅用于打包镜像；其镜像存储与 containerd 隔离）、Helm 3、`ctr`（随 containerd 提供）。 +- [`tutorials/labs/examples/13-volcano-ascend-vnpu/`](https://github.com/Project-HAMi/website/tree/master/tutorials/labs/examples/13-volcano-ascend-vnpu) 下的实验文件。下文命令中所有 `tutorials/labs/examples/...` 路径均相对 website 仓库检出根目录，请在仓库根目录执行（步骤 3、4 会 `cd` 进 Volcano 与插件源码目录，应用清单前请先切回）。 + +验证机硬件清单供参考： + +| 项目 | 值 | +| :----------------- | :----------------------------------------------------------------- | +| 操作系统 | 麒麟 V10 Lance（aarch64），内核 4.19.90 | +| NPU | 2× 昇腾 310P3，每卡 21525 MB（Bus-Id 0000:81:00.0 / 0000:85:00.0） | +| 驱动 / npu-smi | 25.5.1 | +| Kubernetes | v1.28.15 单节点，Flannel，containerd 1.7.1 | +| Go / Docker / Helm | go1.26.2、Docker 24.0.4 + Buildx v0.27.0、Helm v3.9.0 | +| 测试镜像 | `quay.io/ascend/vllm-ascend:v0.18.0-310p` | + +## 步骤 1：验证环境 + +确认驱动能看到健康的 NPU： + +```bash +npu-smi info +``` + +```text ++--------------------------------------------------------------------------------------------------------+ +| npu-smi 25.5.1 Version: 25.5.1 | ++-------------------------------+-----------------+------------------------------------------------------+ +| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page) | +| Chip Device | Bus-Id | AICore(%) Memory-Usage(MB) | ++===============================+=================+======================================================+ +| 4 310P3 | OK | NA 37 0 / 0 | +| 0 0 | 0000:81:00.0 | 0 1848 / 21525 | ++===============================+=================+======================================================+ +| 5 310P3 | OK | NA 40 0 / 0 | +| 0 1 | 0000:85:00.0 | 0 1849 / 21525 | ++===============================+----------------=+======================================================+ +``` + +HAMi-core 模式要求节点带 `ascend=on` 标签（插件的 DaemonSet 按它选择节点）。检查节点与标签，节点名换成你的： + +```bash +kubectl get nodes -o wide +kubectl get node aio-node74-arm -o jsonpath-as-json='{.metadata.labels}' \ + | python3 -m json.tool | grep -iE "ascend|accelerator|servertype" +``` + +```text +NAME STATUS ROLES AGE VERSION INTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME +aio-node74-arm Ready control-plane,worker 358d v1.28.15 172.26.1.74 Kylin Linux Advanced Server V10 (Lance) 4.19.90-52.48.v2207.ky10.aarch64 containerd://1.7.1 + + "accelerator": "huawei-Ascend310P", + "ascend": "on", + "servertype": "Ascend310P-8", +``` + +如果缺少 `ascend=on`，补上： + +```bash +kubectl label node aio-node74-arm ascend=on --overwrite +``` + +最后确认 containerd 已配置 Ascend 运行时 handler。业务 Pod 会声明 `runtimeClassName: ascend`，经由它做设备注入： + +```bash +grep -A3 'runtimes.ascend' /etc/containerd/config.toml +``` + +```text +[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.ascend] + runtime_type = "io.containerd.runc.v2" + [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.ascend.options] + BinaryName = "/usr/local/Ascend/Ascend-Docker-Runtime/ascend-docker-runtime" +``` + +该配置随昇腾驱动/容器运行时套件安装。如果没有，先安装 ascend-docker-runtime 并重启 containerd 再继续。 + +## 步骤 2：清理已有部署 + +如果机器上已装 Volcano 或 HAMi 昇腾插件，先全部清理，让实验从已知状态开始： + +```bash +helm uninstall volcano -n volcano-system + +kubectl -n kube-system delete ds hami-ascend-device-plugin +kubectl -n kube-system delete cm hami-scheduler-device hami-device-node-config +kubectl delete clusterrole,clusterrolebinding hami-ascend +kubectl -n kube-system delete sa hami-ascend +kubectl delete runtimeclass ascend + +rm -rf /usr/local/hami-vnpu-core/containers/* +rm -rf /usr/local/hami-shared-region/* +``` + +如果卸载后 `volcano-system` 一直卡在 `Terminating`（webhook 删除后的已知现象），先触发删除再清掉 finalizer： + +```bash +kubectl delete ns volcano-system --wait=false +kubectl get ns volcano-system -o json | python3 -c " +import json,sys +ns = json.load(sys.stdin) +ns['spec']['finalizers'] = [] +json.dump(ns, sys.stdout) +" | kubectl replace --raw "/api/v1/namespaces/volcano-system/finalize" -f - +``` + +确认已清理干净： + +```bash +kubectl get clusterrole,clusterrolebinding,validatingwebhookconfiguration,mutatingwebhookconfiguration,crd 2>&1 \ + | grep -iE "volcano|hami" || echo CLEAN +``` + +```text +CLEAN +``` + +## 步骤 3：源码编译 Volcano 镜像 + +克隆 Volcano 并签出验证时的 commit（当时的 master）： + +```bash +git clone https://github.com/volcano-sh/volcano.git /root/temp/volcano +cd /root/temp/volcano +git checkout 7d9504320533a9f4e9bfbb59f79ec5c53a68f3e8 +``` + +Volcano 官方 `make images` 会在 builder 容器内执行 `go mod download`，在受限网络上很不稳定（连 `proxy.golang.org` 超时、换 `goproxy.cn` 也会被断连）。本实验改为**宿主机编译**（模块缓存是热的），Docker 只负责打包静态二进制： + +```bash +make vc-scheduler vc-controller-manager vc-webhook-manager +``` + +```text +$ ls -lh _output/bin/ +total 157M +-rwxr-xr-x 1 root root 51M vc-controller-manager +-rwxr-xr-x 1 root root 60M vc-scheduler +-rwxr-xr-x 1 root root 48M vc-webhook-manager +``` + +Volcano 以 `CGO_ENABLED=0` 编译，任意基础镜像都可用。scheduler 与 controller-manager 只需 alpine + 二进制；webhook-manager 还需要 kubectl（admission init job 生成证书用）与仓库的 `gen-admission-secret.sh`： + +```bash +cat <<'EOF' | docker buildx build -t volcanosh/vc-scheduler:latest -f - . --load +FROM alpine:3.24.1 +COPY _output/bin/vc-scheduler /vc-scheduler +ENTRYPOINT ["/vc-scheduler"] +EOF + +cat <<'EOF' | docker buildx build -t volcanosh/vc-controller-manager:latest -f - . --load +FROM alpine:3.24.1 +COPY _output/bin/vc-controller-manager /vc-controller-manager +ENTRYPOINT ["/vc-controller-manager"] +EOF + +cat <<'EOF' | docker buildx build -t volcanosh/vc-webhook-manager:latest -f - . --load +FROM alpine:3.24.1 +RUN apk add --update ca-certificates && \ + apk add --update openssl && \ + apk add --update -t deps curl && \ + curl -L https://dl.k8s.io/release/v1.28.15/bin/linux/arm64/kubectl -o /usr/local/bin/kubectl && \ + chmod +x /usr/local/bin/kubectl && \ + apk del --purge deps && \ + rm /var/cache/apk/* +COPY _output/bin/vc-webhook-manager /vc-webhook-manager +ADD ./installer/dockerfile/webhook-manager/gen-admission-secret.sh /gen-admission-secret.sh +ENTRYPOINT ["/vc-webhook-manager"] +EOF +``` + +```text +$ docker images --format "{{.Repository}}:{{.Tag}} {{.Size}}" | grep "volcanosh/vc-.*:latest" +volcanosh/vc-webhook-manager:latest 114MB +volcanosh/vc-controller-manager:latest 61.7MB +volcanosh/vc-scheduler:latest 70.8MB +``` + +集群运行时是 containerd，而 Docker 的镜像存储对 kubelet 不可见，因此要把三个镜像全部导入 containerd 的 `k8s.io` namespace： + +```bash +for img in vc-scheduler vc-controller-manager vc-webhook-manager; do + docker save volcanosh/$img:latest | ctr -n k8s.io images import - +done +``` + +```text +unpacking docker.io/volcanosh/vc-scheduler:latest (sha256:63e40eb5...)...done +unpacking docker.io/volcanosh/vc-controller-manager:latest (sha256:1de7438d...)...done +unpacking docker.io/volcanosh/vc-webhook-manager:latest (sha256:afe553d5...)...done +``` + +抽查一个镜像（版本号是经 ldflags 注入的 commit SHA）： + +```bash +docker run --rm volcanosh/vc-scheduler:latest --version +``` + +```text +API Version: v1alpha1 +Version: 7d9504320533a9f4e9bfbb59f79ec5c53a68f3e8 +Git SHA: 7d9504320533a9f4e9bfbb59f79ec5c53a68f3e8 +Built At: 2026-08-14 15:25:14 +Go Version: go1.26.2 +``` + +## 步骤 4：获取 ascend-device-plugin 镜像 + +从 [Project-HAMi/ascend-device-plugin](https://github.com/Project-HAMi/ascend-device-plugin) 拉取官方镜像（多架构，含 arm64）并导入 containerd： + +```bash +docker pull projecthami/ascend-device-plugin:v1.4.0 +docker save projecthami/ascend-device-plugin:v1.4.0 | ctr -n k8s.io images import - +``` + +镜像内置了 [Project-HAMi/hami-vnpu-core](https://github.com/Project-HAMi/hami-vnpu-core) 的 `libvnpu.so` 拦截库（由插件 CI 在 CANN 环境中构建）：插件把它拷贝到宿主机 `/usr/local/hami-vnpu-core/`，再由 Ascend 运行时通过 `ld.so.preload` 注入业务容器。**库的版本必须与 NPU 驱动匹配。** 不匹配不会报错，容器内 `npu-smi` 只会永远卡在 `Initialize SchedulerClient...`。验证时就因两个月前缓存的 `libvnpu` 资产出现过完全相同的故障。如果遇到卡死，请对比镜像内资产与匹配驱动的镜像版本： + +```bash +docker run --rm --entrypoint md5sum projecthami/ascend-device-plugin:v1.4.0 \ + /usr/local/hami-vnpu-core-assets/libvnpu.so +``` + +## 步骤 5：部署 Volcano 并开启 HAMi 模式 deviceshare + +用本地 chart 安装 Volcano。注意镜像拉取策略的 key：是 `basic.image_pull_policy`（下划线）。写成 `scheduler.imagePullPolicy` 会静默无效，节点会去拉取只有本地才有的镜像： + +```bash +helm install volcano /root/temp/volcano/installer/helm/chart/volcano \ + --namespace volcano-system --create-namespace \ + --set basic.image_pull_policy=IfNotPresent \ + --timeout 300s +``` + +```text +NAME: volcano +NAMESPACE: volcano-system +STATUS: deployed +REVISION: 1 +``` + +三个组件全部基于本地编译的镜像运行： + +```bash +kubectl -n volcano-system get pods -o wide +``` + +```text +NAME READY STATUS RESTARTS AGE IP NODE +volcano-admission-5bc7fb6d67-btbfp 1/1 Running 0 20s 10.244.0.86 aio-node74-arm +volcano-admission-init-kgcqb 0/1 Completed 0 25s 10.244.0.84 aio-node74-arm +volcano-controllers-557bd8d995-tz4st 1/1 Running 0 20s 10.244.0.85 aio-node74-arm +volcano-scheduler-ff5d85ffb-k7slw 1/1 Running 0 20s 10.244.0.87 aio-node74-arm +``` + +接着把调度器的 `deviceshare` 插件指向 HAMi 的 vNPU 规格。请在 website 仓库根目录执行下面的清单命令（示例路径相对该根目录）： + +```bash +kubectl apply -f tutorials/labs/examples/13-volcano-ascend-vnpu/01-volcano-scheduler-configmap.yaml +kubectl -n volcano-system rollout restart deploy volcano-scheduler +``` + +所应用的 `volcano-scheduler.conf` 保留 Volcano 标准插件分层，只在 `deviceshare` 上追加 HAMi 模式参数： + +```yaml +actions: "enqueue, allocate, backfill" +tiers: + - plugins: + - name: priority + - name: gang + enablePreemptable: false + - name: conformance + - plugins: + - name: overcommit + - name: drf + enablePreemptable: false + - name: predicates + - name: deviceshare + arguments: + deviceshare.AscendHAMiVNPUEnable: "true" + deviceshare.SchedulePolicy: binpack + deviceshare.KnownGeometriesCMNamespace: kube-system + deviceshare.KnownGeometriesCMName: hami-scheduler-device + - name: proportion + - name: nodeorder + - name: binpack +``` + +验证调度器加载了新配置： + +```bash +kubectl -n volcano-system logs deploy/volcano-scheduler | grep -A4 "name: deviceshare" +``` + +```text +I0814 07:40:47.668217 1 scheduler.go:160] - name: deviceshare +I0814 07:40:47.668222 1 scheduler.go:160] deviceshare.AscendHAMiVNPUEnable: "true" +I0814 07:40:47.668225 1 scheduler.go:160] deviceshare.SchedulePolicy: binpack +I0814 07:40:47.668230 1 scheduler.go:160] deviceshare.KnownGeometriesCMNamespace: kube-system +``` + +如果同一集群里还跑 Volcano vGPU（NVIDIA），需把两边的规格 ConfigMap 合并为一个并让 `KnownGeometriesCMName` 指向它，因为 volcano-vgpu 有自己的一套。 + +## 步骤 6：以 hami-core 模式部署插件 + +先应用插件仓库的 RuntimeClass，再应用打开 `hamiVnpuCore` 的设备配置（模板默认为 `false`）： + +```bash +kubectl apply -f https://raw.githubusercontent.com/Project-HAMi/ascend-device-plugin/v1.4.0/ascend-runtimeclass.yaml + +curl -s https://raw.githubusercontent.com/Project-HAMi/ascend-device-plugin/v1.4.0/ascend-device-configmap.yaml \ + | sed 's/hamiVnpuCore: false/hamiVnpuCore: true/' | kubectl apply -f - +``` + +```text +runtimeclass.node.k8s.io/ascend created +configmap/hami-scheduler-device created +``` + +ConfigMap 中 310P3 的条目就是 Pod 资源与硬件的对应关系（每卡 `memoryAllocatable: 21527` MB、8 个 AI 核心，最小模板 `vir01` 为 3072 MB）： + +```yaml +vnpus: + hamiVnpuCore: true + configs: + - chipName: 310P3 + commonWord: Ascend310P + resourceName: huawei.com/Ascend310P + resourceMemoryName: huawei.com/Ascend310P-memory + memoryAllocatable: 21527 + memoryCapacity: 24576 + aiCore: 8 + aiCPU: 7 +``` + +然后是按节点的开关。`vDeviceCount` 限制每张物理卡的 vNPU 数，插件会直接采用该值（支持由 [ascend-device-plugin PR #100](https://github.com/Project-HAMi/ascend-device-plugin/pull/100) 引入）；`7` 与下面的验证容量一致： + +```bash +kubectl apply -f tutorials/labs/examples/13-volcano-ascend-vnpu/02-hami-device-node-config.yaml +``` + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + labels: + app.kubernetes.io/component: hami-scheduler + app.kubernetes.io/name: hami + app.kubernetes.io/instance: hami + name: hami-device-node-config + namespace: kube-system +data: + node-config.yaml: |- + nodes: + - name: "aio-node74-arm" + hami-vnpu-core: true + vDeviceCount: 7 + filterDevices: + index: [] + uuid: [] +``` + +把 `aio-node74-arm` 换成你的节点名。最后应用 RBAC 与 DaemonSet（清单使用 `projecthami/ascend-device-plugin:v1.4.0` 且 `imagePullPolicy: IfNotPresent`，因此会运行步骤 4 导入的镜像）： + +```bash +kubectl apply -f https://raw.githubusercontent.com/Project-HAMi/ascend-device-plugin/v1.4.0/ascend-device-plugin.yaml +``` + +```text +clusterrole.rbac.authorization.k8s.io/hami-ascend created +clusterrolebinding.rbac.authorization.k8s.io/hami-ascend created +serviceaccount/hami-ascend created +daemonset.apps/hami-ascend-device-plugin created +``` + +:::important 一次性应用完整清单 + +请完整应用 `ascend-device-plugin.yaml`。手工截取清单（比如只复制 DaemonSet 部分）会破坏 selector/label 匹配并产生费解的报错。 + +::: + +等插件就绪后检查日志中 HAMi-core 健康启动的三个标志（节点配置匹配、metrics 服务启动、宿主机资产写入）： + +```bash +kubectl -n kube-system get pods -o wide | grep ascend +kubectl -n kube-system logs ds/hami-ascend-device-plugin | grep -iE "matched|libvnpu|metrics|config file" +``` + +```text +hami-ascend-device-plugin-lnd4c 1/1 Running 0 20s 10.244.0.89 aio-node74-arm + +I0814 07:47:39.795228 1 main.go:124] using config file: /device-config.yaml +I0814 07:47:40.290044 1 manager.go:72] Successfully matched node config for aio-node74-arm: {Name:aio-node74-arm HamiVnpuCore:true VDeviceCount:7} +I0814 07:47:40.391244 1 metrics.go:27] vNPU monitor metrics server starting on :9395 +I0814 07:47:40.396783 1 server.go:192] ✓ Copied /usr/local/hami-vnpu-core-assets/libvnpu.so -> /usr/local/hami-vnpu-core/libvnpu.so +I0814 07:47:40.396900 1 server.go:180] ✓ /usr/local/hami-vnpu-core/ld.so.preload already up-to-date, skipping +``` + +此时节点应上报昇腾扩展资源（2 卡 × 7 vNPU = 14，显存 2 × 21527 MiB）： + +```bash +kubectl describe node aio-node74-arm | grep huawei.com/Ascend310P +``` + +```text + huawei.com/Ascend310P: 14 + huawei.com/Ascend310P-memory: 43054 +``` + +注册显存按芯片配置的每卡 `memoryAllocatable: 21527` MB 计算，比步骤 1 中 `npu-smi` 显示的 21525 MB 每卡多 2 MiB，因此是 43054 而不是 43050。 + +## 步骤 7：运行软切分 Pod 并验证 + +部署第一个测试 Pod，申请 1 个 vNPU、8192 MiB 显存切片： + +```bash +kubectl apply -f tutorials/labs/examples/13-volcano-ascend-vnpu/03-ascend-vnpu-check.yaml +kubectl wait --for=condition=Ready pod/ascend-vnpu-check --timeout=5m +kubectl get pod ascend-vnpu-check -o wide +``` + +清单里四个关键开关： + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: ascend-vnpu-check + annotations: + huawei.com/vnpu-mode: hami-core +spec: + schedulerName: volcano + runtimeClassName: ascend + containers: + - name: npu + image: quay.io/ascend/vllm-ascend:v0.18.0-310p + command: ["sleep", "infinity"] + resources: + limits: + huawei.com/Ascend310P: "1" + huawei.com/Ascend310P-memory: "8192" +``` + +`schedulerName: volcano` 让调度走 `deviceshare`，`runtimeClassName: ascend` 让设备注入走 Ascend 运行时，注解 `huawei.com/vnpu-mode: hami-core` 选择软切分（缺了它 Pod 走模板路径，可能一直 Pending），两个 limits 定义切片大小。 + +```text +NAME READY STATUS RESTARTS AGE IP NODE +ascend-vnpu-check 1/1 Running 0 30s 10.244.0.90 aio-node74-arm +``` + +调度注解记录了 Volcano 分配的结果： + +```bash +kubectl get pod ascend-vnpu-check -o jsonpath-as-json='{.metadata.annotations}' \ + | python3 -m json.tool | grep -iE "ascend|vnpu|bind" +``` + +```text +"hami.io/Ascend310P-devices-allocated": "68496E64-20E05477-92C31323-6E78030A-BD003019,Ascend310P,8192,0:;", +"hami.io/bind-phase": "success", +"huawei.com/Ascend310P": "[{\"UUID\":\"68496E64-...\",\"memory\":8192}]", +"huawei.com/vnpu-mode": "hami-core", +``` + +### 容器只看到自己的切片 + +在 Pod 内执行 `npu-smi`，与步骤 1 中宿主机视角（同一张卡 `1848 / 21525`）对比： + +```bash +kubectl exec ascend-vnpu-check -- npu-smi info +``` + +```text +[INFO limiter::supervisor] [Supervisor PID:10] won manager election +[INFO limiter::manager] [Manager] Registered as Global Manager #0 (PID: 10). Compute limit: 1, Memory limit: 8192, FixedShare: false +open global registry path is "/hami-shared-region/0_global_registry" +[Global] Global Registry not exist, now creating... +connect to global registry ++--------------------------------------------------------------------------------------------------------+ +| npu-smi 25.5.1 Version: 25.5.1 | ++-------------------------------+-----------------+------------------------------------------------------+ +| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page) | +| Chip Device | Bus-Id | AICore(%) Memory-Usage(MB) | ++===============================+=================+======================================================+ +| 32768 310P3 | OK | NA 38 0 / 0 | +| 0 0 | 0000:81:00.0 | 0 0 / 8192 | ++===============================+----------------=+======================================================+ +``` + +容器看到的是 `0 / 8192` MB，即 `libvnpu.so` 强制的显存窗口，而不是物理卡的 21525 MB。注入的环境变量印证了接线（先用 `crictl ps` 拿到容器 ID）： + +```bash +crictl exec env | grep -E "NPU_|ASCEND_VIS" +``` + +```text +ASCEND_VISIBLE_DEVICES=0 +NPU_LOCAL_SHM_PATH=/hami-vnpu-shmem/vnpu_local_shmem +NPU_GLOBAL_SHM_PATH=/hami-shared-region/0_global_registry +NPU_MEM_QUOTA=8192 +``` + +### binpack 让两个 Pod 共享一张卡 + +启动第二个同规格 Pod（`04-ascend-vnpu-check-2.yaml`）： + +```bash +kubectl apply -f tutorials/labs/examples/13-volcano-ascend-vnpu/04-ascend-vnpu-check-2.yaml +kubectl wait --for=condition=Ready pod/ascend-vnpu-check-2 --timeout=5m +kubectl exec ascend-vnpu-check-2 -- npu-smi info | grep -E "Memory limit|0000" +``` + +```text +[INFO limiter::manager] [Manager] Registered as Global Manager #1 (PID: 10). Compute limit: 1, Memory limit: 8192, FixedShare: false +| 0 0 | 0000:81:00.0 | 0 0 / 8192 | +``` + +两个 Pod 的 Bus-Id 都是 `0000:81:00.0`（**同一张物理卡**），各自拥有独立的 8192 MiB 窗口。`Global Manager #0` / `#1` 两行说明两个容器注册进了同一个共享注册表，HAMi-core 由此协调它们在这张卡上的算力调度。再看节点的计量： + +```bash +kubectl describe node aio-node74-arm | grep huawei.com/Ascend310P +``` + +```text + huawei.com/Ascend310P: 14 + huawei.com/Ascend310P-memory: 43054 + huawei.com/Ascend310P 2 2 + huawei.com/Ascend310P-memory 16384 16384 +``` + +已分配 2 个 vNPU、2 × 8192 = 16384 MiB。同规格的第三个 Pod 仍能放进这张卡（3 × 8192 < 21527）；把数量加到总显存超过 `memoryAllocatable` 为止，超出的 Pod 会一直 Pending 直到释放容量，因为 binpack 会先把一张卡填满再碰第二张。 + +## 步骤 8：验证 vNPU 监控指标 + +metrics 端点在**device-plugin Pod**（9395 端口）上，不在业务 Pod 上。按 label 查询： + +```bash +PLUGIN_IP=$(kubectl -n kube-system get pod \ + -l app.kubernetes.io/component=hami-ascend-device-plugin \ + -o jsonpath='{.items[0].status.podIP}') +curl -s "$PLUGIN_IP":9395/metrics | grep -E "^hami" +``` + +```text +hami_container_device_utilization_ratio{container="npu",device_uuid="68496E64-...",namespace="default",pod="ascend-vnpu-check",vdevice_index="0"} 0 +hami_container_device_utilization_ratio{container="npu",device_uuid="68496E64-...",namespace="default",pod="ascend-vnpu-check-2",vdevice_index="0"} 0 +hami_host_gpu_memory_used_bytes{device_index="0",device_type="Ascend-Atlas 300I Pro",device_uuid="68496E64-..."} 1.937768448e+09 +hami_host_gpu_memory_used_bytes{device_index="1",device_type="Ascend-Atlas 300I Pro",device_uuid="D8496E64-..."} 1.938817024e+09 +hami_vgpu_memory_limit_bytes{container="npu",...,pod="ascend-vnpu-check",vdevice_index="0"} 8.589934592e+09 +hami_vgpu_memory_limit_bytes{container="npu",...,pod="ascend-vnpu-check-2",vdevice_index="0"} 8.589934592e+09 +``` + +解读这些数字： + +- `hami_vgpu_memory_limit_bytes = 8.589934592e+09` 字节 = 恰好 8192 MiB，与两个 Pod 的申请值一致； +- 两条 vdevice 序列共享物理卡 0 的 UUID，即步骤 7 的 binpack 结果； +- `hami_host_gpu_memory_used_bytes` 按卡上报宿主机用量（每张空闲卡约 1.9 GB 驱动开销）； +- 利用率指标为 `0`，因为 Pod 只在 `sleep`。 + +该端点还导出 `hami_vgpu_memory_used_bytes` 及 buffer/context/module 细分，以及 `hami_host_gpu_utilization_ratio`。也可以用 port-forward 代替直连 Pod IP： + +```bash +PLUGIN_POD=$(kubectl -n kube-system get pod \ + -l app.kubernetes.io/component=hami-ascend-device-plugin \ + -o jsonpath='{.items[0].metadata.name}') +kubectl -n kube-system port-forward "pod/$PLUGIN_POD" 9395:9395 & +curl -s http://localhost:9395/metrics +``` + +## 故障排查 + +| 症状 | 验证环境中的原因 | 处理 | +| :-- | :-- | :-- | +| 容器内 `npu-smi info` 卡死在 `Initialize SchedulerClient...` | `libvnpu.so` 与 NPU 驱动不匹配（资产来源镜像过期） | 改用 `libvnpu.so` 资产与驱动匹配的发布镜像（校验 md5）；重启插件刷新宿主机副本 | +| Pod 报 `ErrImageNeverPull` | Docker 与 containerd 镜像存储隔离 | `docker save \| ctr -n k8s.io images import -` | +| 节点仍尝试拉取本地才有的镜像 | Helm 拉取策略 key 写错 | 使用 `basic.image_pull_policy`（下划线） | +| 卸载后 `volcano-system` 卡在 `Terminating` | webhook 删除后 namespace finalizer 未释放 | 经 `finalize` 子资源清理 finalizer（步骤 2） | +| 手工复制的 DaemonSet `kubectl apply` 报 selector 错误 | 清单被截断 | 完整应用仓库清单；`sed` 只用于替换镜像 tag | +| 在业务 Pod 里 `curl :9395` 无响应 | metrics 由插件 DaemonSet 提供，不是业务 Pod | 按 label 选中插件 Pod（步骤 8） | +| `make images` 中 `go mod download` 失败 | 容器内下载在受限网络超时 | 宿主机编译；Docker 只打包二进制 | +| Pod 分配失败报 `cannot patch resource "pods"` | ClusterRole 丢了 `pods` 的 `patch`/`update` 权限 | 原样保留仓库 RBAC；插件必须回写 Pod 的分配注解 | + +:::note 三个避免困惑的事实 + +- **`-core` 不会被注册。** v1.4.0 不会把 `huawei.com/Ascend310P-core` 注册为节点资源；配置中的 `resourceCoreName` 不会上报。Pod spec 只需卡数与显存 MiB。 +- **是 `libvnpu.so`，不是 `libvgpu.so`。** HAMi 在 NVIDIA 上的拦截库是 `libvgpu.so`；昇腾 HAMi-core 用的是 `libvnpu.so`，经 `/etc/ld.so.preload` 注入，宿主机资产位于 `/usr/local/hami-vnpu-core/`。 +- **资源名来自 `commonWord`。** 芯片叫 `310P3`，但 Kubernetes 资源是 `huawei.com/Ascend310P`；写成 `huawei.com/Ascend310P3` 或 `huawei.com/Ascend` Pod 会一直 Pending。 + +::: + +## 清理 + +删除测试 Pod： + +```bash +kubectl delete pod ascend-vnpu-check ascend-vnpu-check-2 +``` + +删除插件及其资源： + +```bash +kubectl -n kube-system delete ds hami-ascend-device-plugin +kubectl -n kube-system delete cm hami-scheduler-device hami-device-node-config +kubectl delete clusterrole,clusterrolebinding hami-ascend +kubectl -n kube-system delete sa hami-ascend +kubectl delete runtimeclass ascend +rm -rf /usr/local/hami-vnpu-core/containers/* /usr/local/hami-shared-region/* +``` + +卸载 Volcano（若卡住，按步骤 2 清理 finalizer）： + +```bash +helm uninstall volcano -n volcano-system +``` + +Volcano 镜像仍留在 Docker 与 containerd 中，连同插件镜像一并删除： + +```bash +for img in vc-scheduler vc-controller-manager vc-webhook-manager; do + docker rmi volcanosh/$img:latest + ctr -n k8s.io images remove docker.io/volcanosh/$img:latest +done +docker rmi projecthami/ascend-device-plugin:v1.4.0 +ctr -n k8s.io images remove docker.io/projecthami/ascend-device-plugin:v1.4.0 +``` + +## 本实验证明了什么 + +| 论断 | 证据 | +| :-- | :-- | +| Volcano 以 HAMi 模式调度源码构建镜像的 vNPU | 3 个组件 Running；调度器日志含 `AscendHAMiVNPUEnable: "true"` | +| 插件上报软切分容量 | 节点上报 `Ascend310P: 14`、`Ascend310P-memory: 43054` | +| 容器显存视图被限制，而非仅被调度 | 容器内 `npu-smi` 显示 `0 / 8192`；宿主机显示 `1848 / 21525` | +| 配额按容器下发 | 注入 `NPU_MEM_QUOTA=8192`；两个 Pod 均报 `Memory limit: 8192` | +| binpack 把多个 vNPU 装进一张卡 | 两个 Pod 同为 Bus-Id `0000:81:00.0`；节点分配 2 vNPU / 16384 MiB | +| 切片经同一注册表协调 | `/hami-shared-region/0_global_registry` 中的 `Global Manager #0` 与 `#1` | +| 整条链路可观测 | `:9395` 上报的容器级 limit 恰为 8192 MiB | + +一个如实的边界说明：测试 Pod 只在 `sleep`，因此本实验证明的是配额被下发、按容器生效并在容器内可见，并未实际执行越配额的分配。要证明超过切片的分配会被拒绝，需要运行真正吃显存的负载（例如把显存参数设到 8192 MiB 以上的 vLLM），可作为本实验的扩展；GPU 上等价的越界验证可参考[实验 12](./kai-scheduler-hami-gke.md) 中超配额 `cudaMalloc` 的做法。 + +## 延伸阅读 + +- 架构背景与 Volcano vNPU 两种模式的对比：[用 Volcano + HAMi-core 软切分昇腾 vNPU](/zh/blog/volcano-ascend-vnpu-soft-slicing) +- 对比 [实验 8：Volcano vGPU、Gang 调度与队列限制](./volcano-vgpu-gang-queue.md)：同一套调度器在 NVIDIA GPU 上的用法 +- 把测试 Pod 包进 Volcano `VCJob`（`tasks[].template`），在软切分之上叠加 Gang 调度与队列 +- 参考 [Volcano 中的华为昇腾设备](/zh/docs/installation/how-to-use-volcano-ascend) 用户指南与上游 [ascend-device-plugin Volcano 指南](https://github.com/Project-HAMi/ascend-device-plugin/blob/main/docs/volcano.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md index 80667767d..427edc0f8 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/overview.md @@ -19,4 +19,4 @@ import LabCardGridAuto from '@site/src/components/labs/LabCardGridAuto'; -每个实验都列出了各自的前提条件。实验 3 和 4 直接复用实验 1 搭建的集群，一次开机即可完成全部三个实验；实验 2 可在任意笔记本上运行，无需 GPU。实验 7 在租用的 GPU 虚拟机上自行搭建单节点 k3s 集群，不使用 GPU Operator。实验 8 需要已有的 Volcano GPU 集群，用于验证 Volcano vGPU、Gang 调度和队列级资源限制。实验 9 使用 Kueue 准入控制限制 HAMi vGPU 数量、显存和算力配额。实验 11 将从头搭建完整的 KServe Standard 推理环境，并通过 HAMi 原生 DRA Claim 让两个 vLLM 副本共享一张 GPU。实验 12 在 GKE 1.35/COS/CDI 上部署 KAI Scheduler 与 HAMi-core，并通过 CUDA 分配验证显存上限。 +每个实验都列出了各自的前提条件。实验 3 和 4 直接复用实验 1 搭建的集群，一次开机即可完成全部三个实验；实验 2 可在任意笔记本上运行，无需 GPU。实验 7 在租用的 GPU 虚拟机上自行搭建单节点 k3s 集群，不使用 GPU Operator。实验 8 需要已有的 Volcano GPU 集群，用于验证 Volcano vGPU、Gang 调度和队列级资源限制。实验 9 使用 Kueue 准入控制限制 HAMi vGPU 数量、显存和算力配额。实验 11 将从头搭建完整的 KServe Standard 推理环境，并通过 HAMi 原生 DRA Claim 让两个 vLLM 副本共享一张 GPU。实验 12 在 GKE 1.35/COS/CDI 上部署 KAI Scheduler 与 HAMi-core，并通过 CUDA 分配验证显存上限。实验 13 在昇腾 310P3 ARM 服务器上源码编译 Volcano 与 ascend-device-plugin，验证 hami-vnpu-core 软切分、binpack 共卡与容器级监控指标。 diff --git a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/tags.yml b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/tags.yml index eafe5c2ab..fcc24b4e5 100644 --- a/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/tags.yml +++ b/i18n/zh/docusaurus-plugin-content-docs-tutorials/current/tags.yml @@ -1,6 +1,9 @@ "DRA": label: "DRA" permalink: "/dra" +"ascend": + label: "Ascend" + permalink: "/ascend" "GPU 分区": label: "GPU 分区" permalink: "/gpu-partitioning" @@ -55,6 +58,9 @@ "topology": label: "topology" permalink: "/topology" +"vnpu": + label: "vNPU" + permalink: "/vnpu" "vgpu": label: "vgpu" permalink: "/vgpu" @@ -82,6 +88,9 @@ "资源共享": label: "资源共享" permalink: "/resource-sharing" +"soft-slicing": + label: "软切分" + permalink: "/soft-slicing" "隔离": label: "隔离" permalink: "/isolation" diff --git a/sidebars-tutorials.js b/sidebars-tutorials.js index 9e28572c9..0b44c90a7 100644 --- a/sidebars-tutorials.js +++ b/sidebars-tutorials.js @@ -71,6 +71,11 @@ module.exports = { id: "labs/kai-scheduler-hami-gke", customProps: { level: "Advanced", duration: "about 90 minutes" }, }, + { + type: "doc", + id: "labs/volcano-ascend-vnpu", + customProps: { level: "Advanced", duration: "about 120 minutes" }, + }, ], }, ], diff --git a/src/pages/landing/coscup-2026.js b/src/pages/landing/coscup-2026.js deleted file mode 100644 index 42b18ab48..000000000 --- a/src/pages/landing/coscup-2026.js +++ /dev/null @@ -1,3 +0,0 @@ -import EventLanding from "@site/src/components/EventLanding"; - -export default () => ; diff --git a/src/pages/landing/opensource-summit-korea.js b/src/pages/landing/opensource-summit-korea.js deleted file mode 100644 index be7cb3dca..000000000 --- a/src/pages/landing/opensource-summit-korea.js +++ /dev/null @@ -1,3 +0,0 @@ -import EventLanding from "@site/src/components/EventLanding"; - -export default () => ; diff --git a/tutorials/labs/examples/13-volcano-ascend-vnpu/01-volcano-scheduler-configmap.yaml b/tutorials/labs/examples/13-volcano-ascend-vnpu/01-volcano-scheduler-configmap.yaml new file mode 100644 index 000000000..6fe7b7eaa --- /dev/null +++ b/tutorials/labs/examples/13-volcano-ascend-vnpu/01-volcano-scheduler-configmap.yaml @@ -0,0 +1,30 @@ +# Volcano scheduler configuration with the deviceshare plugin in HAMi vNPU mode. +# Apply into volcano-system, then: kubectl -n volcano-system rollout restart deploy volcano-scheduler +apiVersion: v1 +kind: ConfigMap +metadata: + name: volcano-scheduler-configmap + namespace: volcano-system +data: + volcano-scheduler.conf: | + actions: "enqueue, allocate, backfill" + tiers: + - plugins: + - name: priority + - name: gang + enablePreemptable: false + - name: conformance + - plugins: + - name: overcommit + - name: drf + enablePreemptable: false + - name: predicates + - name: deviceshare + arguments: + deviceshare.AscendHAMiVNPUEnable: "true" + deviceshare.SchedulePolicy: binpack + deviceshare.KnownGeometriesCMNamespace: kube-system + deviceshare.KnownGeometriesCMName: hami-scheduler-device + - name: proportion + - name: nodeorder + - name: binpack diff --git a/tutorials/labs/examples/13-volcano-ascend-vnpu/02-hami-device-node-config.yaml b/tutorials/labs/examples/13-volcano-ascend-vnpu/02-hami-device-node-config.yaml new file mode 100644 index 000000000..7f3627e45 --- /dev/null +++ b/tutorials/labs/examples/13-volcano-ascend-vnpu/02-hami-device-node-config.yaml @@ -0,0 +1,22 @@ +# Per-node HAMi device plugin configuration: enable hami-vnpu-core on one node. +# Replace "aio-node74-arm" with your Ascend node name. vDeviceCount is the +# per-card vNPU count, honored directly by the plugin (since v1.4.0); 7 matches +# the 310P3 capacity verified in this lab. +apiVersion: v1 +kind: ConfigMap +metadata: + labels: + app.kubernetes.io/component: hami-scheduler + app.kubernetes.io/name: hami + app.kubernetes.io/instance: hami + name: hami-device-node-config + namespace: kube-system +data: + node-config.yaml: |- + nodes: + - name: "aio-node74-arm" + hami-vnpu-core: true + vDeviceCount: 7 + filterDevices: + index: [] + uuid: [] diff --git a/tutorials/labs/examples/13-volcano-ascend-vnpu/03-ascend-vnpu-check.yaml b/tutorials/labs/examples/13-volcano-ascend-vnpu/03-ascend-vnpu-check.yaml new file mode 100644 index 000000000..8fe903135 --- /dev/null +++ b/tutorials/labs/examples/13-volcano-ascend-vnpu/03-ascend-vnpu-check.yaml @@ -0,0 +1,18 @@ +# First soft-sliced vNPU Pod: 1 vNPU with an 8192 MiB memory slice. +apiVersion: v1 +kind: Pod +metadata: + name: ascend-vnpu-check + annotations: + huawei.com/vnpu-mode: hami-core +spec: + schedulerName: volcano + runtimeClassName: ascend + containers: + - name: npu + image: quay.io/ascend/vllm-ascend:v0.18.0-310p + command: ["sleep", "infinity"] + resources: + limits: + huawei.com/Ascend310P: "1" + huawei.com/Ascend310P-memory: "8192" diff --git a/tutorials/labs/examples/13-volcano-ascend-vnpu/04-ascend-vnpu-check-2.yaml b/tutorials/labs/examples/13-volcano-ascend-vnpu/04-ascend-vnpu-check-2.yaml new file mode 100644 index 000000000..cd6560a9b --- /dev/null +++ b/tutorials/labs/examples/13-volcano-ascend-vnpu/04-ascend-vnpu-check-2.yaml @@ -0,0 +1,19 @@ +# Second soft-sliced vNPU Pod, identical to the first: with the binpack policy +# it lands on the same physical card as ascend-vnpu-check. +apiVersion: v1 +kind: Pod +metadata: + name: ascend-vnpu-check-2 + annotations: + huawei.com/vnpu-mode: hami-core +spec: + schedulerName: volcano + runtimeClassName: ascend + containers: + - name: npu + image: quay.io/ascend/vllm-ascend:v0.18.0-310p + command: ["sleep", "infinity"] + resources: + limits: + huawei.com/Ascend310P: "1" + huawei.com/Ascend310P-memory: "8192" diff --git a/tutorials/labs/volcano-ascend-vnpu.md b/tutorials/labs/volcano-ascend-vnpu.md new file mode 100644 index 000000000..7efdb963c --- /dev/null +++ b/tutorials/labs/volcano-ascend-vnpu.md @@ -0,0 +1,702 @@ +--- +title: "Lab 13: Soft-Slicing Ascend 310P3 vNPU with Volcano and HAMi-core" +description: "Build Volcano and the ascend-device-plugin from source on an aarch64 Ascend server, enable hami-vnpu-core soft slicing, and verify memory isolation, binpack card sharing, and per-container metrics." +sidebar_label: "Lab 13: Volcano Ascend vNPU" +lab: + level: Advanced + duration: about 120 minutes + environment: single-node Kubernetes 1.28 cluster on an aarch64 Ascend 310P3 server (Kylin V10, driver 25.5.1, containerd) + cost: requires dedicated Ascend 310P3 aarch64 hardware; soft slicing is ARM-only + authors: + - rootsongjc + verified: "2026-08-14" +tags: + - volcano + - hami-core + - ascend + - vnpu + - soft-slicing +toc_max_heading_level: 2 +--- + +This lab starts from a clean Kubernetes cluster on an Ascend 310P3 aarch64 server and ends with two Pods sharing one physical NPU through `hami-vnpu-core` soft slicing, each locked to its own 8192 MiB memory window and both visible to Prometheus metrics. + +Because soft slicing requires [Volcano](https://github.com/volcano-sh/volcano) ≥ 1.16, and no stable 1.16 existed at verification time (latest stable: v1.15.1; only a `1.16.0-alpha.1` chart existed), this lab builds Volcano master from source and deploys the plugin from its official `v1.4.0` image. If a stable Volcano 1.16 chart is published by the time you run this, you can substitute the chart install for Steps 3 and 5 and keep everything else. + +:::note About the output blocks + +The outputs below were captured from the verified run on 2026-08-14. Node names, IPs, Pod suffixes, and UUIDs are environment-specific; compare the component names, readiness, placement, and measured values. + +::: + +## What You'll Learn + +- compile Volcano on the host and package the binaries into images for containerd; +- pull the plugin image and verify its `libvnpu.so` asset matches your NPU driver; +- configure Volcano's `deviceshare` plugin for HAMi-mode vNPU scheduling with `binpack`; +- switch the plugin to `hamiVnpuCore` globally and `hami-vnpu-core` per node; +- prove in-container memory isolation with `npu-smi`; +- prove binpack sharing by landing two Pods on the same physical card; and +- read per-container vNPU metrics from the plugin's `:9395` endpoint. + +## Lab Overview + +```mermaid +%% title: Volcano + HAMi-core Ascend vNPU soft-slicing lab +flowchart LR + S1["Step 1
Verify environment"] --> S2["Step 2
Clean cluster"] + S2 --> S3["Step 3
Build Volcano"] + S3 --> S4["Step 4
Plugin image"] + S4 --> S5["Step 5
Deploy Volcano"] + S5 --> S6["Step 6
Deploy plugin"] + S6 --> S7["Step 7
Soft-sliced Pods"] + S7 --> S8["Step 8
Verify metrics"] +``` + +## Prerequisites + +- An aarch64 server with Ascend 310P (or 310P3) NPUs, driver/npu-smi **≥ 25.5**, and [ascend-docker-runtime](https://gitcode.com/Ascend/mind-cluster/tree/master/component/ascend-docker-runtime) installed (soft slicing is ARM-only). +- A Kubernetes ≥ 1.20 cluster on that server using containerd. The verified cluster was a single-node kubeadm cluster (node `aio-node74-arm`, both control-plane and worker) on Kylin Linux Advanced Server V10, Kubernetes v1.28.15, containerd 1.7.1. +- On the host: Go 1.26 (the verified host used `go1.26.2 linux/arm64`), Docker 24 with Buildx (used only to package images; its image store is separate from containerd's), Helm 3, and `ctr` (ships with containerd). +- The files under [`tutorials/labs/examples/13-volcano-ascend-vnpu/`](https://github.com/Project-HAMi/website/tree/master/tutorials/labs/examples/13-volcano-ascend-vnpu). All `tutorials/labs/examples/...` paths in the commands below are relative to the website repository checkout, so run them from its root (Steps 3 and 4 `cd` into the Volcano and plugin sources; come back before applying manifests). + +The verified host inventory, for reference: + +| Item | Value | +| :----------------- | :------------------------------------------------------------------ | +| OS | Kylin Linux Advanced Server V10 Lance (aarch64), kernel 4.19.90 | +| NPU | 2× Ascend 310P3, 21525 MB each (Bus-Id 0000:81:00.0 / 0000:85:00.0) | +| Driver / npu-smi | 25.5.1 | +| Kubernetes | v1.28.15 single node, Flannel, containerd 1.7.1 | +| Go / Docker / Helm | go1.26.2, Docker 24.0.4 + Buildx v0.27.0, Helm v3.9.0 | +| Test image | `quay.io/ascend/vllm-ascend:v0.18.0-310p` | + +## Step 1: Verify the Environment + +Confirm the driver sees healthy NPUs: + +```bash +npu-smi info +``` + +```text ++--------------------------------------------------------------------------------------------------------+ +| npu-smi 25.5.1 Version: 25.5.1 | ++-------------------------------+-----------------+------------------------------------------------------+ +| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page) | +| Chip Device | Bus-Id | AICore(%) Memory-Usage(MB) | ++===============================+=================+======================================================+ +| 4 310P3 | OK | NA 37 0 / 0 | +| 0 0 | 0000:81:00.0 | 0 1848 / 21525 | ++===============================+=================+======================================================+ +| 5 310P3 | OK | NA 40 0 / 0 | +| 0 1 | 0000:85:00.0 | 0 1849 / 21525 | ++===============================+----------------=+======================================================+ +``` + +HAMi-core mode requires the node to carry `ascend=on` (the plugin's DaemonSet selects on it). Check the node and its labels, replacing the node name with yours: + +```bash +kubectl get nodes -o wide +kubectl get node aio-node74-arm -o jsonpath-as-json='{.metadata.labels}' \ + | python3 -m json.tool | grep -iE "ascend|accelerator|servertype" +``` + +```text +NAME STATUS ROLES AGE VERSION INTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME +aio-node74-arm Ready control-plane,worker 358d v1.28.15 172.26.1.74 Kylin Linux Advanced Server V10 (Lance) 4.19.90-52.48.v2207.ky10.aarch64 containerd://1.7.1 + + "accelerator": "huawei-Ascend310P", + "ascend": "on", + "servertype": "Ascend310P-8", +``` + +If `ascend=on` is missing, add it: + +```bash +kubectl label node aio-node74-arm ascend=on --overwrite +``` + +Finally, confirm containerd has the Ascend runtime handler. Workload Pods will declare `runtimeClassName: ascend`, which routes through it: + +```bash +grep -A3 'runtimes.ascend' /etc/containerd/config.toml +``` + +```text +[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.ascend] + runtime_type = "io.containerd.runc.v2" + [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.ascend.options] + BinaryName = "/usr/local/Ascend/Ascend-Docker-Runtime/ascend-docker-runtime" +``` + +This configuration is installed with the Ascend driver/container-runtime suite. If it is absent, install ascend-docker-runtime and restart containerd before continuing. + +## Step 2: Clean Up Any Existing Deployment + +If Volcano or a HAMi Ascend plugin is already installed, remove it so the lab starts from a known state: + +```bash +helm uninstall volcano -n volcano-system + +kubectl -n kube-system delete ds hami-ascend-device-plugin +kubectl -n kube-system delete cm hami-scheduler-device hami-device-node-config +kubectl delete clusterrole,clusterrolebinding hami-ascend +kubectl -n kube-system delete sa hami-ascend +kubectl delete runtimeclass ascend + +rm -rf /usr/local/hami-vnpu-core/containers/* +rm -rf /usr/local/hami-shared-region/* +``` + +If `volcano-system` sticks in `Terminating` after the uninstall (a known behavior once the admission webhooks are gone), trigger the deletion and clear the finalizer: + +```bash +kubectl delete ns volcano-system --wait=false +kubectl get ns volcano-system -o json | python3 -c " +import json,sys +ns = json.load(sys.stdin) +ns['spec']['finalizers'] = [] +json.dump(ns, sys.stdout) +" | kubectl replace --raw "/api/v1/namespaces/volcano-system/finalize" -f - +``` + +Verify nothing remains: + +```bash +kubectl get clusterrole,clusterrolebinding,validatingwebhookconfiguration,mutatingwebhookconfiguration,crd 2>&1 \ + | grep -iE "volcano|hami" || echo CLEAN +``` + +```text +CLEAN +``` + +## Step 3: Build the Volcano Images from Source + +Clone Volcano and check out the verified commit (master at verification time): + +```bash +git clone https://github.com/volcano-sh/volcano.git /root/temp/volcano +cd /root/temp/volcano +git checkout 7d9504320533a9f4e9bfbb59f79ec5c53a68f3e8 +``` + +Volcano's official `make images` runs `go mod download` inside a builder container, which is unreliable on restricted networks (timeouts against `proxy.golang.org`, resets even via `goproxy.cn`). This lab instead **compiles on the host**, where the Go module cache is warm, and uses Docker only to package the static binaries: + +```bash +make vc-scheduler vc-controller-manager vc-webhook-manager +``` + +```text +$ ls -lh _output/bin/ +total 157M +-rwxr-xr-x 1 root root 51M vc-controller-manager +-rwxr-xr-x 1 root root 60M vc-scheduler +-rwxr-xr-x 1 root root 48M vc-webhook-manager +``` + +Volcano builds with `CGO_ENABLED=0`, so any base image works. The scheduler and controller-manager need only Alpine plus the binary; the webhook-manager also needs kubectl (its admission-init job generates certificates) and the repo's `gen-admission-secret.sh`: + +```bash +cat <<'EOF' | docker buildx build -t volcanosh/vc-scheduler:latest -f - . --load +FROM alpine:3.24.1 +COPY _output/bin/vc-scheduler /vc-scheduler +ENTRYPOINT ["/vc-scheduler"] +EOF + +cat <<'EOF' | docker buildx build -t volcanosh/vc-controller-manager:latest -f - . --load +FROM alpine:3.24.1 +COPY _output/bin/vc-controller-manager /vc-controller-manager +ENTRYPOINT ["/vc-controller-manager"] +EOF + +cat <<'EOF' | docker buildx build -t volcanosh/vc-webhook-manager:latest -f - . --load +FROM alpine:3.24.1 +RUN apk add --update ca-certificates && \ + apk add --update openssl && \ + apk add --update -t deps curl && \ + curl -L https://dl.k8s.io/release/v1.28.15/bin/linux/arm64/kubectl -o /usr/local/bin/kubectl && \ + chmod +x /usr/local/bin/kubectl && \ + apk del --purge deps && \ + rm /var/cache/apk/* +COPY _output/bin/vc-webhook-manager /vc-webhook-manager +ADD ./installer/dockerfile/webhook-manager/gen-admission-secret.sh /gen-admission-secret.sh +ENTRYPOINT ["/vc-webhook-manager"] +EOF +``` + +```text +$ docker images --format "{{.Repository}}:{{.Tag}} {{.Size}}" | grep "volcanosh/vc-.*:latest" +volcanosh/vc-webhook-manager:latest 114MB +volcanosh/vc-controller-manager:latest 61.7MB +volcanosh/vc-scheduler:latest 70.8MB +``` + +The cluster runs containerd, and Docker's image store is invisible to kubelet, so import all three images into containerd's `k8s.io` namespace: + +```bash +for img in vc-scheduler vc-controller-manager vc-webhook-manager; do + docker save volcanosh/$img:latest | ctr -n k8s.io images import - +done +``` + +```text +unpacking docker.io/volcanosh/vc-scheduler:latest (sha256:63e40eb5...)...done +unpacking docker.io/volcanosh/vc-controller-manager:latest (sha256:1de7438d...)...done +unpacking docker.io/volcanosh/vc-webhook-manager:latest (sha256:afe553d5...)...done +``` + +Sanity-check one image (the version is the commit SHA, injected via ldflags): + +```bash +docker run --rm volcanosh/vc-scheduler:latest --version +``` + +```text +API Version: v1alpha1 +Version: 7d9504320533a9f4e9bfbb59f79ec5c53a68f3e8 +Git SHA: 7d9504320533a9f4e9bfbb59f79ec5c53a68f3e8 +Built At: 2026-08-14 15:25:14 +Go Version: go1.26.2 +``` + +## Step 4: Get the ascend-device-plugin Image + +Pull the official image from [Project-HAMi/ascend-device-plugin](https://github.com/Project-HAMi/ascend-device-plugin) (multi-arch, including arm64) and import it into containerd: + +```bash +docker pull projecthami/ascend-device-plugin:v1.4.0 +docker save projecthami/ascend-device-plugin:v1.4.0 | ctr -n k8s.io images import - +``` + +The image bundles the `libvnpu.so` interception library from [Project-HAMi/hami-vnpu-core](https://github.com/Project-HAMi/hami-vnpu-core), built by the plugin's CI in a CANN environment: the plugin copies it onto the host at `/usr/local/hami-vnpu-core/`, and the Ascend runtime injects it into workload containers via `ld.so.preload`. **The library version must match the NPU driver.** A mismatch does not fail loudly; in-container `npu-smi` just hangs forever at `Initialize SchedulerClient...`. During verification, a two-month-old cached `libvnpu` asset produced exactly that failure. If you hit the hang, compare the asset in your image against the image release that matches your driver: + +```bash +docker run --rm --entrypoint md5sum projecthami/ascend-device-plugin:v1.4.0 \ + /usr/local/hami-vnpu-core-assets/libvnpu.so +``` + +## Step 5: Deploy Volcano and Enable HAMi-mode deviceshare + +Install Volcano from the local chart. Note the image pull policy key: it is `basic.image_pull_policy` (underscore). Using `scheduler.imagePullPolicy` silently does nothing, and nodes then try to pull images that exist only locally: + +```bash +helm install volcano /root/temp/volcano/installer/helm/chart/volcano \ + --namespace volcano-system --create-namespace \ + --set basic.image_pull_policy=IfNotPresent \ + --timeout 300s +``` + +```text +NAME: volcano +NAMESPACE: volcano-system +STATUS: deployed +REVISION: 1 +``` + +All three components came up on the locally built images: + +```bash +kubectl -n volcano-system get pods -o wide +``` + +```text +NAME READY STATUS RESTARTS AGE IP NODE +volcano-admission-5bc7fb6d67-btbfp 1/1 Running 0 20s 10.244.0.86 aio-node74-arm +volcano-admission-init-kgcqb 0/1 Completed 0 25s 10.244.0.84 aio-node74-arm +volcano-controllers-557bd8d995-tz4st 1/1 Running 0 20s 10.244.0.85 aio-node74-arm +volcano-scheduler-ff5d85ffb-k7slw 1/1 Running 0 20s 10.244.0.87 aio-node74-arm +``` + +Now point the scheduler's `deviceshare` plugin at the HAMi vNPU geometries. Run the manifest commands from the website repository root (the example paths are relative to it): + +```bash +kubectl apply -f tutorials/labs/examples/13-volcano-ascend-vnpu/01-volcano-scheduler-configmap.yaml +kubectl -n volcano-system rollout restart deploy volcano-scheduler +``` + +The applied `volcano-scheduler.conf` keeps Volcano's standard plugin tiers and adds the HAMi-mode arguments to `deviceshare`: + +```yaml +actions: "enqueue, allocate, backfill" +tiers: + - plugins: + - name: priority + - name: gang + enablePreemptable: false + - name: conformance + - plugins: + - name: overcommit + - name: drf + enablePreemptable: false + - name: predicates + - name: deviceshare + arguments: + deviceshare.AscendHAMiVNPUEnable: "true" + deviceshare.SchedulePolicy: binpack + deviceshare.KnownGeometriesCMNamespace: kube-system + deviceshare.KnownGeometriesCMName: hami-scheduler-device + - name: proportion + - name: nodeorder + - name: binpack +``` + +Verify the scheduler loaded the new configuration: + +```bash +kubectl -n volcano-system logs deploy/volcano-scheduler | grep -A4 "name: deviceshare" +``` + +```text +I0814 07:40:47.668217 1 scheduler.go:160] - name: deviceshare +I0814 07:40:47.668222 1 scheduler.go:160] deviceshare.AscendHAMiVNPUEnable: "true" +I0814 07:40:47.668225 1 scheduler.go:160] deviceshare.SchedulePolicy: binpack +I0814 07:40:47.668230 1 scheduler.go:160] deviceshare.KnownGeometriesCMNamespace: kube-system +``` + +If you also run Volcano vGPU (NVIDIA) in the same cluster, merge both geometry ConfigMaps into one and point `KnownGeometriesCMName` at the merged ConfigMap, because volcano-vgpu uses its own. + +## Step 6: Deploy the Plugin in hami-core Mode + +Apply the RuntimeClass from the plugin repository, then the device config with `hamiVnpuCore` switched on (the template ships with `false`): + +```bash +kubectl apply -f https://raw.githubusercontent.com/Project-HAMi/ascend-device-plugin/v1.4.0/ascend-runtimeclass.yaml + +curl -s https://raw.githubusercontent.com/Project-HAMi/ascend-device-plugin/v1.4.0/ascend-device-configmap.yaml \ + | sed 's/hamiVnpuCore: false/hamiVnpuCore: true/' | kubectl apply -f - +``` + +```text +runtimeclass.node.k8s.io/ascend created +configmap/hami-scheduler-device created +``` + +The 310P3 entry of the ConfigMap is what matches your Pod resources to the hardware (each card: `memoryAllocatable: 21527` MB, 8 AI cores; the smallest template `vir01` reserves 3072 MB): + +```yaml +vnpus: + hamiVnpuCore: true + configs: + - chipName: 310P3 + commonWord: Ascend310P + resourceName: huawei.com/Ascend310P + resourceMemoryName: huawei.com/Ascend310P-memory + memoryAllocatable: 21527 + memoryCapacity: 24576 + aiCore: 8 + aiCPU: 7 +``` + +Next, the per-node override. `vDeviceCount` caps the vNPU count per physical card, and the plugin honors it directly (support landed in [ascend-device-plugin PR #100](https://github.com/Project-HAMi/ascend-device-plugin/pull/100)); `7` matches the capacity verified below: + +```bash +kubectl apply -f tutorials/labs/examples/13-volcano-ascend-vnpu/02-hami-device-node-config.yaml +``` + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + labels: + app.kubernetes.io/component: hami-scheduler + app.kubernetes.io/name: hami + app.kubernetes.io/instance: hami + name: hami-device-node-config + namespace: kube-system +data: + node-config.yaml: |- + nodes: + - name: "aio-node74-arm" + hami-vnpu-core: true + vDeviceCount: 7 + filterDevices: + index: [] + uuid: [] +``` + +Replace `aio-node74-arm` with your node name. Finally, apply the RBAC and DaemonSet (the manifest uses `projecthami/ascend-device-plugin:v1.4.0` with `imagePullPolicy: IfNotPresent`, so it runs the image imported in Step 4): + +```bash +kubectl apply -f https://raw.githubusercontent.com/Project-HAMi/ascend-device-plugin/v1.4.0/ascend-device-plugin.yaml +``` + +```text +clusterrole.rbac.authorization.k8s.io/hami-ascend created +clusterrolebinding.rbac.authorization.k8s.io/hami-ascend created +serviceaccount/hami-ascend created +daemonset.apps/hami-ascend-device-plugin created +``` + +:::important Apply the full manifest file + +Apply the complete `ascend-device-plugin.yaml` in one shot. Truncating the manifest (for example, copying only the DaemonSet portion) breaks selector/label matching and produces confusing apply errors. + +::: + +Wait for the plugin, then check its logs for the three markers of a healthy HAMi-core start: the node config matched, the metrics server started, and the host assets written: + +```bash +kubectl -n kube-system get pods -o wide | grep ascend +kubectl -n kube-system logs ds/hami-ascend-device-plugin | grep -iE "matched|libvnpu|metrics|config file" +``` + +```text +hami-ascend-device-plugin-lnd4c 1/1 Running 0 20s 10.244.0.89 aio-node74-arm + +I0814 07:47:39.795228 1 main.go:124] using config file: /device-config.yaml +I0814 07:47:40.290044 1 manager.go:72] Successfully matched node config for aio-node74-arm: {Name:aio-node74-arm HamiVnpuCore:true VDeviceCount:7} +I0814 07:47:40.391244 1 metrics.go:27] vNPU monitor metrics server starting on :9395 +I0814 07:47:40.396783 1 server.go:192] ✓ Copied /usr/local/hami-vnpu-core-assets/libvnpu.so -> /usr/local/hami-vnpu-core/libvnpu.so +I0814 07:47:40.396900 1 server.go:180] ✓ /usr/local/hami-vnpu-core/ld.so.preload already up-to-date, skipping +``` + +The node should now advertise the Ascend extended resources: 2 cards × 7 vNPUs = 14, and 2 × 21527 MiB: + +```bash +kubectl describe node aio-node74-arm | grep huawei.com/Ascend310P +``` + +```text + huawei.com/Ascend310P: 14 + huawei.com/Ascend310P-memory: 43054 +``` + +The registered memory follows the chip config's `memoryAllocatable` of 21527 MB per card, 2 MiB per card above the 21525 MB `npu-smi` displays in Step 1, hence 43054 rather than 43050. + +## Step 7: Run Soft-Sliced Pods and Verify + +Deploy the first test Pod, which requests one vNPU with an 8192 MiB memory slice: + +```bash +kubectl apply -f tutorials/labs/examples/13-volcano-ascend-vnpu/03-ascend-vnpu-check.yaml +kubectl wait --for=condition=Ready pod/ascend-vnpu-check --timeout=5m +kubectl get pod ascend-vnpu-check -o wide +``` + +The manifest's four essential switches: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: ascend-vnpu-check + annotations: + huawei.com/vnpu-mode: hami-core +spec: + schedulerName: volcano + runtimeClassName: ascend + containers: + - name: npu + image: quay.io/ascend/vllm-ascend:v0.18.0-310p + command: ["sleep", "infinity"] + resources: + limits: + huawei.com/Ascend310P: "1" + huawei.com/Ascend310P-memory: "8192" +``` + +`schedulerName: volcano` routes scheduling through `deviceshare`, `runtimeClassName: ascend` routes device injection through the Ascend runtime, the `huawei.com/vnpu-mode: hami-core` annotation selects soft slicing (without it, the Pod uses the template path and can stay Pending), and the two limits define the slice size. + +```text +NAME READY STATUS RESTARTS AGE IP NODE +ascend-vnpu-check 1/1 Running 0 30s 10.244.0.90 aio-node74-arm +``` + +The scheduling annotations record what Volcano allocated: + +```bash +kubectl get pod ascend-vnpu-check -o jsonpath-as-json='{.metadata.annotations}' \ + | python3 -m json.tool | grep -iE "ascend|vnpu|bind" +``` + +```text +"hami.io/Ascend310P-devices-allocated": "68496E64-20E05477-92C31323-6E78030A-BD003019,Ascend310P,8192,0:;", +"hami.io/bind-phase": "success", +"huawei.com/Ascend310P": "[{\"UUID\":\"68496E64-...\",\"memory\":8192}]", +"huawei.com/vnpu-mode": "hami-core", +``` + +### The container sees only its slice + +Run `npu-smi` inside the Pod and compare with the host view from Step 1 (`1848 / 21525` on the same card): + +```bash +kubectl exec ascend-vnpu-check -- npu-smi info +``` + +```text +[INFO limiter::supervisor] [Supervisor PID:10] won manager election +[INFO limiter::manager] [Manager] Registered as Global Manager #0 (PID: 10). Compute limit: 1, Memory limit: 8192, FixedShare: false +open global registry path is "/hami-shared-region/0_global_registry" +[Global] Global Registry not exist, now creating... +connect to global registry ++--------------------------------------------------------------------------------------------------------+ +| npu-smi 25.5.1 Version: 25.5.1 | ++-------------------------------+-----------------+------------------------------------------------------+ +| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page) | +| Chip Device | Bus-Id | AICore(%) Memory-Usage(MB) | ++===============================+=================+======================================================+ +| 32768 310P3 | OK | NA 38 0 / 0 | +| 0 0 | 0000:81:00.0 | 0 0 / 8192 | ++===============================+=================+======================================================+ +``` + +The container sees `0 / 8192` MB: this is the memory window `libvnpu.so` enforces, not the physical card's 21525 MB. The injected environment variables confirm the wiring (`crictl ps` to find the container ID first): + +```bash +crictl exec env | grep -E "NPU_|ASCEND_VIS" +``` + +```text +ASCEND_VISIBLE_DEVICES=0 +NPU_LOCAL_SHM_PATH=/hami-vnpu-shmem/vnpu_local_shmem +NPU_GLOBAL_SHM_PATH=/hami-shared-region/0_global_registry +NPU_MEM_QUOTA=8192 +``` + +### binpack shares one card between Pods + +Launch a second Pod with an identical spec (`04-ascend-vnpu-check-2.yaml`): + +```bash +kubectl apply -f tutorials/labs/examples/13-volcano-ascend-vnpu/04-ascend-vnpu-check-2.yaml +kubectl wait --for=condition=Ready pod/ascend-vnpu-check-2 --timeout=5m +kubectl exec ascend-vnpu-check-2 -- npu-smi info | grep -E "Memory limit|0000" +``` + +```text +[INFO limiter::manager] [Manager] Registered as Global Manager #1 (PID: 10). Compute limit: 1, Memory limit: 8192, FixedShare: false +| 0 0 | 0000:81:00.0 | 0 0 / 8192 | +``` + +Both Pods report Bus-Id `0000:81:00.0` (the **same physical card**), each with an independent 8192 MiB window. The `Global Manager #0` / `#1` lines show both containers registered into one shared registry, so HAMi-core coordinates their compute scheduling across the card. Check the node's accounting: + +```bash +kubectl describe node aio-node74-arm | grep huawei.com/Ascend310P +``` + +```text + huawei.com/Ascend310P: 14 + huawei.com/Ascend310P-memory: 43054 + huawei.com/Ascend310P 2 2 + huawei.com/Ascend310P-memory 16384 16384 +``` + +Two vNPUs and 2 × 8192 = 16384 MiB allocated. A third Pod of the same size still fits on the card (3 × 8192 < 21527); scale the count until the total would exceed `memoryAllocatable`, and the extra Pods stay Pending until capacity frees, because binpack tries to fill one card before touching the second. + +## Step 8: Verify the vNPU Metrics + +The metrics endpoint lives on the **device-plugin Pod** (port 9395), not on workload Pods. Query it by label: + +```bash +PLUGIN_IP=$(kubectl -n kube-system get pod \ + -l app.kubernetes.io/component=hami-ascend-device-plugin \ + -o jsonpath='{.items[0].status.podIP}') +curl -s "$PLUGIN_IP":9395/metrics | grep -E "^hami" +``` + +```text +hami_container_device_utilization_ratio{container="npu",device_uuid="68496E64-...",namespace="default",pod="ascend-vnpu-check",vdevice_index="0"} 0 +hami_container_device_utilization_ratio{container="npu",device_uuid="68496E64-...",namespace="default",pod="ascend-vnpu-check-2",vdevice_index="0"} 0 +hami_host_gpu_memory_used_bytes{device_index="0",device_type="Ascend-Atlas 300I Pro",device_uuid="68496E64-..."} 1.937768448e+09 +hami_host_gpu_memory_used_bytes{device_index="1",device_type="Ascend-Atlas 300I Pro",device_uuid="D8496E64-..."} 1.938817024e+09 +hami_vgpu_memory_limit_bytes{container="npu",...,pod="ascend-vnpu-check",vdevice_index="0"} 8.589934592e+09 +hami_vgpu_memory_limit_bytes{container="npu",...,pod="ascend-vnpu-check-2",vdevice_index="0"} 8.589934592e+09 +``` + +Read the numbers: + +- `hami_vgpu_memory_limit_bytes = 8.589934592e+09` bytes = exactly 8192 MiB, matching both Pods' requests; +- both vdevice series share the UUID of physical card 0, confirming the binpack placement from Step 7; +- `hami_host_gpu_memory_used_bytes` reports per-card host usage (~1.9 GB of driver overhead on each idle card); +- utilization gauges are `0` because the Pods only `sleep`. + +The endpoint also exports `hami_vgpu_memory_used_bytes` plus the buffer/context/module breakdowns, and `hami_host_gpu_utilization_ratio`. Alternatively, port-forward instead of addressing the Pod IP directly: + +```bash +PLUGIN_POD=$(kubectl -n kube-system get pod \ + -l app.kubernetes.io/component=hami-ascend-device-plugin \ + -o jsonpath='{.items[0].metadata.name}') +kubectl -n kube-system port-forward "pod/$PLUGIN_POD" 9395:9395 & +curl -s http://localhost:9395/metrics +``` + +## Troubleshooting + +| Symptom | Cause in the verified environment | Action | +| :-- | :-- | :-- | +| In-container `npu-smi info` hangs at `Initialize SchedulerClient...` | `libvnpu.so` version does not match the NPU driver (stale asset source image) | Use an image whose `libvnpu.so` asset comes from the release matching your driver (verify the md5); restart the plugin so the host copy refreshes | +| Pod fails with `ErrImageNeverPull` | Docker and containerd image stores are separate | `docker save \| ctr -n k8s.io images import -` | +| Node still tries to pull local-only images | Wrong Helm key for pull policy | Use `basic.image_pull_policy` (underscore) | +| `volcano-system` stuck `Terminating` after uninstall | Namespace finalizer not released once webhooks are gone | Clear the finalizer via the `finalize` subresource (Step 2) | +| `kubectl apply` of a hand-copied DaemonSet fails on selectors | Manifest was truncated when copied | Apply the complete repo manifest; use `sed` only for the image tag | +| `curl :9395` from the workload Pod returns nothing | Metrics are served by the plugin DaemonSet, not workloads | Select the plugin Pod by label (Step 8) | +| `go mod download` fails inside `make images` | Container-side downloads time out on restricted networks | Compile on the host; let Docker only package binaries | +| Pod allocation fails with `cannot patch resource "pods"` | ClusterRole lost `pods` `patch`/`update` permission | Keep the repo's RBAC verbatim; the plugin must erase allocation annotations on the Pod | + +:::note Three facts that prevent confusion + +- **`-core` is not registered.** v1.4.0 does not report `huawei.com/Ascend310P-core` as a node resource; the `resourceCoreName` entry in the config is not uploaded. Pod specs need only the card count and memory MiB. +- **`libvnpu.so`, not `libvgpu.so`.** HAMi's NVIDIA interception library is `libvgpu.so`; the Ascend HAMi-core library is `libvnpu.so`, injected through `/etc/ld.so.preload` with host assets under `/usr/local/hami-vnpu-core/`. +- **Resource names come from `commonWord`.** The chip is `310P3`, but the Kubernetes resource is `huawei.com/Ascend310P`; request `huawei.com/Ascend310P3` or `huawei.com/Ascend` and the Pod stays Pending. + +::: + +## Cleanup + +Remove the test Pods: + +```bash +kubectl delete pod ascend-vnpu-check ascend-vnpu-check-2 +``` + +Remove the plugin and its resources: + +```bash +kubectl -n kube-system delete ds hami-ascend-device-plugin +kubectl -n kube-system delete cm hami-scheduler-device hami-device-node-config +kubectl delete clusterrole,clusterrolebinding hami-ascend +kubectl -n kube-system delete sa hami-ascend +kubectl delete runtimeclass ascend +rm -rf /usr/local/hami-vnpu-core/containers/* /usr/local/hami-shared-region/* +``` + +Uninstall Volcano (clear the namespace finalizer if it hangs, as in Step 2): + +```bash +helm uninstall volcano -n volcano-system +``` + +The Volcano images remain in Docker and containerd; remove them if you no longer need them, together with the plugin image: + +```bash +for img in vc-scheduler vc-controller-manager vc-webhook-manager; do + docker rmi volcanosh/$img:latest + ctr -n k8s.io images remove docker.io/volcanosh/$img:latest +done +docker rmi projecthami/ascend-device-plugin:v1.4.0 +ctr -n k8s.io images remove docker.io/projecthami/ascend-device-plugin:v1.4.0 +``` + +## What This Lab Proved + +| Claim | Evidence | +| :-- | :-- | +| Volcano schedules vNPUs in HAMi mode from source-built images | 3 components Running; scheduler log shows `AscendHAMiVNPUEnable: "true"` | +| The plugin advertises soft-sliced capacity | Node reports `Ascend310P: 14`, `Ascend310P-memory: 43054` | +| The container's memory view is capped, not just scheduled | In-container `npu-smi` shows `0 / 8192`; host shows `1848 / 21525` | +| Each container receives its own quota | `NPU_MEM_QUOTA=8192` injected; both Pods report `Memory limit: 8192` | +| binpack packs multiple vNPUs onto one card | Both Pods on Bus-Id `0000:81:00.0`; node allocates 2 vNPU / 16384 MiB | +| Slices coordinate through one registry | `Global Manager #0` and `#1` in `/hami-shared-region/0_global_registry` | +| The stack is observable | `:9395` exports per-container limits of exactly 8192 MiB | + +One honest limit: the test Pods only `sleep`, so this lab proves that the quota is delivered, applied per container, and visible inside the container. It does not exercise an over-quota allocation. To prove that an allocation beyond the slice fails, run a real memory-allocating workload (for example vLLM with a memory setting above 8192 MiB) as an extension; [Lab 12](./kai-scheduler-hami-gke.md) shows the equivalent proof on GPUs with an over-quota `cudaMalloc`. + +## Next Steps + +- Read [Soft-Slicing Ascend vNPU with Volcano and HAMi-core](/blog/volcano-ascend-vnpu-soft-slicing) for the architecture background and the two Volcano vNPU modes. +- Compare with [Lab 8: Volcano vGPU with Gang Scheduling and Queues](./volcano-vgpu-gang-queue.md), the same scheduler on NVIDIA GPUs. +- Wrap the test Pod into a Volcano `VCJob` (`tasks[].template`) to add gang scheduling and queues on top of soft slicing. +- Consult the [Ascend-in-Volcano user guide](/docs/installation/how-to-use-volcano-ascend) and the upstream [ascend-device-plugin Volcano guide](https://github.com/Project-HAMi/ascend-device-plugin/blob/main/docs/volcano.md). diff --git a/tutorials/overview.md b/tutorials/overview.md index 5cfd995f1..9762308c2 100644 --- a/tutorials/overview.md +++ b/tutorials/overview.md @@ -17,4 +17,4 @@ Background knowledge that the labs build on. ## Labs - Each lab lists its own prerequisites. Labs 3 and 4 continue from the cluster Lab 1 builds, so a single session covers all three; Lab 2 runs on any laptop with no GPU required. Lab 7 brings up its own single-node k3s cluster on a rented GPU VM, without the GPU Operator. Lab 8 requires an existing Volcano GPU cluster and validates Volcano vGPU, Gang scheduling, and queue-level limits. Lab 9 uses Kueue admission control to enforce HAMi vGPU count, memory, and compute quotas. Lab 11 builds a complete KServe Standard inference stack and runs two vLLM replicas on one GPU through native HAMi DRA claims. Lab 12 deploys KAI Scheduler and HAMi-core on GKE 1.35/COS/CDI and proves the memory ceiling with CUDA allocations. + Each lab lists its own prerequisites. Labs 3 and 4 continue from the cluster Lab 1 builds, so a single session covers all three; Lab 2 runs on any laptop with no GPU required. Lab 7 brings up its own single-node k3s cluster on a rented GPU VM, without the GPU Operator. Lab 8 requires an existing Volcano GPU cluster and validates Volcano vGPU, Gang scheduling, and queue-level limits. Lab 9 uses Kueue admission control to enforce HAMi vGPU count, memory, and compute quotas. Lab 11 builds a complete KServe Standard inference stack and runs two vLLM replicas on one GPU through native HAMi DRA claims. Lab 12 deploys KAI Scheduler and HAMi-core on GKE 1.35/COS/CDI and proves the memory ceiling with CUDA allocations. Lab 13 builds Volcano and the ascend-device-plugin from source on an Ascend 310P3 ARM server and verifies hami-vnpu-core soft slicing, binpack card sharing, and per-container metrics.