Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 66 additions & 11 deletions docs/core-concepts/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,29 +2,84 @@
title: Architecture
---

The overall architecture of HAMi is shown as below:
HAMi extends Kubernetes with device-aware scheduling and runtime resource controls for heterogeneous AI accelerators. Its architecture separates cluster-wide placement decisions from node-level device allocation and, where supported, in-container enforcement.

![Architecture](/img/docs/common/core-concepts/architect.jpg)
![HAMi components and the GPU workload scheduling sequence](/img/docs/common/core-concepts/hami-architecture-en.svg)

HAMi consists of the following components:
## What are HAMi's core components?

- HAMi MutatingWebhook
- HAMi scheduler-extender
- Device-plugin (HAMi-device-plugin)
- In-container resource control (HAMi-Core)
HAMi coordinates four components during the lifecycle of an accelerator workload:

| Component | Runs as | Primary responsibility |
| --- | --- | --- |
| HAMi MutatingWebhook | Part of the `hami-scheduler` deployment | Directs Pods that request HAMi-managed resources to `hami-scheduler` |
| HAMi scheduler extender | Part of the `hami-scheduler` deployment | Selects a node and a physical device from a cluster-wide view of available resources |
| HAMi device plugin | DaemonSet on accelerator nodes | Registers devices with kubelet and prepares the selected device for the container |
| HAMi-Core | Library injected into supported containers | Enforces the assigned memory and compute limits at runtime |

The exact resources and enforcement mechanism depend on the device vendor. For example, NVIDIA workloads can request `nvidia.com/gpumem` in MiB and `nvidia.com/gpucores` as a percentage. Other devices expose vendor-specific resources and may support different allocation granularities. See the [FAQ](../faq/faq.md) for the current support matrix.

## How does a workload move through HAMi?

1. **Admission:** When a Pod requests a HAMi-managed device, the MutatingWebhook sets `spec.schedulerName` to `hami-scheduler` unless the Pod already names a scheduler.
2. **Placement:** The HAMi scheduler extender combines the Pod request with device information reported by each node. During scheduling, it filters nodes that cannot satisfy the request and selects a suitable physical device.
3. **Allocation:** The scheduler records the selected device and quota in Pod annotations. On the chosen node, kubelet calls the HAMi device plugin, which reads that result and makes the device available to the container.
4. **Runtime control:** For devices that support in-container control, the device plugin injects the required runtime library and configuration. For NVIDIA virtual GPUs, HAMi-Core intercepts relevant CUDA and NVML calls to apply the assigned memory and compute limits.

This division keeps cluster policy in the control plane, hardware discovery and allocation on each node, and workload-level enforcement close to the application.

## HAMi MutatingWebhook {#hami-mutatingwebhook}

HAMi MutatingWebhook checks if this task can be handled by HAMi. It scans the resource field of each pod submitted. If each resource the pod requires is either 'CPU', 'Memory' or a HAMi-resource, then it will set the schedulerName field of this pod to 'hami-scheduler'.
The MutatingWebhook is the admission entry point. It examines a newly created Pod's resource requests to determine whether HAMi should handle it. For eligible Pods, it sets:

```yaml
spec:
schedulerName: hami-scheduler
```

Pods that do not request HAMi-managed resources continue through the normal Kubernetes scheduling path. Pods that explicitly choose another scheduler are not silently reassigned.

## HAMi Scheduler {#hami-scheduler}

The HAMi scheduler is responsible for assigning tasks to the appropriate nodes and devices. At the same time, the scheduler needs to maintain a global view of heterogeneous computing devices for monitoring.
The scheduler is responsible for choosing both a node and a device. Kubernetes device plugins normally advertise integer resource counts, which are not enough to describe properties such as device model, memory capacity, compute capacity, health, or topology. HAMi device plugins therefore report detailed device information through node annotations, allowing the scheduler to maintain a cluster-wide view.

HAMi participates in the standard scheduling flow as a [scheduler extender](https://github.com/kubernetes/design-proposals-archive/blob/main/scheduling/scheduler_extender.md); it does not replace Kubernetes scheduling. It filters candidates according to the requested resources, applies the configured binpack or spread policy, binds the Pod, and writes the allocation result to annotations such as `hami.io/vgpu-devices-allocated`.

## Device Plugin {#device-plugin}

The device-plugin layer obtains the scheduling result from the annotations field of the task and maps the corresponding device to the container.
The HAMi device plugin runs on each supported accelerator node and implements the Kubernetes [device plugin API](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/). It discovers local devices, registers allocatable resources with kubelet, reports device details for scheduling, and handles kubelet's `Allocate` request after a Pod is bound.

During allocation, the plugin reads the scheduler's result from the Pod annotations and exposes the selected device to the container. Depending on the vendor integration, it may mount device files and runtime libraries or inject environment variables that describe the assigned quota.

## HAMi-Core {#hami-core}

The in-container resource control is responsible for monitoring the resource usage within the container and providing hard isolation capabilities.
HAMi-Core provides runtime control for NVIDIA virtual GPUs through `libvgpu.so`. The device plugin loads the library into the container through `/etc/ld.so.preload`. HAMi-Core then intercepts CUDA memory allocation and kernel launch calls: allocations beyond the assigned memory budget return an out-of-memory error, while compute usage is throttled toward the requested limit. It also adjusts NVML results so applications see their assigned memory rather than the full physical device.

This is user-space enforcement, not a hardware security boundary. Applications that bypass the intercepted libraries—for example, by using direct driver calls or Docker-in-Docker—may bypass these controls. Use [NVIDIA MIG](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/) when a supported GPU and hardware-enforced isolation are required. See [GPU Virtualization Principles](./gpu-virtualization.md) for the full interception and allocation flow.

## How does HAMi compare with time-slicing and MIG?

| Approach | Sharing model | Memory and compute boundary | Best fit |
| --- | --- | --- | --- |
| Time-slicing | Workloads take turns on one GPU | No per-workload GPU memory isolation | Simple concurrency where strict quotas are unnecessary |
| HAMi virtual GPU | Workloads share a GPU with flexible requested quotas | User-space memory enforcement and compute throttling | Fine-grained, dynamic sharing across a broad range of GPUs |
| NVIDIA MIG | A supported GPU is divided into fixed hardware partitions | Hardware-enforced memory and compute isolation | Strong isolation on MIG-capable GPUs |

HAMi also supports dynamic MIG allocation, so these approaches are not always mutually exclusive. The right choice depends on the accelerator, workload, isolation requirement, and desired partition granularity.

## Product architecture and reference deployment

This page describes HAMi's product components and the request lifecycle common to its integrations. A complete cluster can also include a CNI, vendor drivers and runtimes, monitoring, and an optional dashboard. For one NVIDIA-oriented deployment topology and its dependencies, see [HAMi Cluster Architecture After Installation](./hami-architecture.md).

## References

- [Kubernetes device plugins](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/)
- [Kubernetes scheduler extender design](https://github.com/kubernetes/design-proposals-archive/blob/main/scheduling/scheduler_extender.md)
- [HAMi-Core source code](https://github.com/Project-HAMi/HAMi-core)
- [NVIDIA Multi-Instance GPU user guide](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/)

## What's next

- Explore the detailed [GPU virtualization principles](./gpu-virtualization.md)
- Review the [reference cluster architecture](./hami-architecture.md)
- Check the [installation prerequisites](../installation/prerequisites.md)
6 changes: 6 additions & 0 deletions docs/core-concepts/hami-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@ title: "HAMi Cluster Architecture After Installation"

After completing the HAMi installation, the cluster is no longer an ordinary Kubernetes cluster, it becomes an AI infrastructure platform with GPU virtualization capabilities. This document breaks down the responsibilities and dependencies of every layer and every component in the cluster after installation.

:::note

This page describes one NVIDIA-oriented reference deployment after installation, including optional components such as GPU Operator, Prometheus, and WebUI. For HAMi's product components and the request lifecycle, see [Architecture](./architecture.md).

:::

## 5-Layer Architecture Overview

The cluster after installation consists of 5 layers, each providing services to the layer above:
Expand Down
17 changes: 17 additions & 0 deletions docusaurus.config.js
Original file line number Diff line number Diff line change
Expand Up @@ -409,6 +409,23 @@ module.exports = {
},
],
},
{
title: "Legal",
items: [
{
label: "About",
to: "/about",
},
{
label: "Privacy",
to: "/privacy",
},
{
label: "Terms",
to: "/terms",
},
],
},
],
copyright: `
<br />
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,29 +3,84 @@ title: 架构设计
translated: true
---

HAMi 的整体架构如下所示:
HAMi 通过设备感知调度和运行时资源控制,扩展 Kubernetes 对异构 AI 加速器的管理能力。其架构将集群级调度决策、节点级设备分配以及支持场景下的容器内资源控制相互分离。

![HAMi 系统架构图,显示各组件之间的关系和交互](/img/docs/common/core-concepts/architect.jpg)
![HAMi 组件及 GPU 工作负载调度流程](/img/docs/common/core-concepts/hami-architecture.svg)

HAMi 由以下组件组成:
## HAMi 有哪些核心组件?

- HAMi MutatingWebhook
- HAMi scheduler-extender
- 设备插件 (HAMi-device-plugin)
- 容器内资源控制 (HAMi-Core)
在加速器工作负载的整个生命周期中,HAMi 会协调以下四个组件:

| 组件 | 运行方式 | 主要职责 |
| --- | --- | --- |
| HAMi MutatingWebhook | `hami-scheduler` Deployment 的一部分 | 将请求 HAMi 管理资源的 Pod 交给 `hami-scheduler` |
| HAMi 调度器扩展程序 | `hami-scheduler` Deployment 的一部分 | 根据集群范围内的可用资源视图选择节点和物理设备 |
| HAMi 设备插件 | 加速器节点上的 DaemonSet | 向 kubelet 注册设备,并为容器准备选定的设备 |
| HAMi-Core | 注入受支持容器的动态库 | 在运行时执行已分配的显存和算力限制 |

具体资源名称和控制机制取决于设备厂商。例如,NVIDIA 工作负载可以使用 `nvidia.com/gpumem` 请求以 MiB 为单位的显存,并使用 `nvidia.com/gpucores` 请求百分比形式的算力。其他设备使用厂商特定的资源名称,支持的分配粒度也可能不同。当前支持情况请参阅[常见问题](../faq/faq.md)。

## 工作负载如何通过 HAMi 运行?

1. **准入:** 当 Pod 请求 HAMi 管理的设备时,MutatingWebhook 会将 `spec.schedulerName` 设置为 `hami-scheduler`;如果 Pod 已明确指定调度器,则不会改写。
2. **调度:** HAMi 调度器扩展程序将 Pod 请求与各节点上报的设备信息结合起来,在调度过程中排除无法满足请求的节点,并选择合适的物理设备。
3. **分配:** 调度器把选定的设备和配额写入 Pod 注解。在目标节点上,kubelet 调用 HAMi 设备插件;设备插件读取调度结果,并将设备提供给容器。
4. **运行时控制:** 对于支持容器内控制的设备,设备插件会注入所需的运行时库和配置。对于 NVIDIA 虚拟 GPU,HAMi-Core 会拦截相关 CUDA 和 NVML 调用,以执行已分配的显存和算力限制。

这种职责划分让集群策略留在控制平面、硬件发现和分配发生在各节点,并让工作负载级资源控制靠近应用程序。

## HAMi MutatingWebhook {#hami-mutatingwebhook}

HAMi MutatingWebhook 检查该任务是否可以由 HAMi 处理,它扫描每个提交的 Pod 的资源字段,如果这些 Pod 所需的每个资源是 'cpu'、'memory' 或 HAMi 资源,则会将该 Pod 的 schedulerName 字段设置为 'hami-scheduler'。
MutatingWebhook 是准入入口。它检查新建 Pod 的资源请求,以确定是否应由 HAMi 处理。对于符合条件的 Pod,它会设置:

```yaml
spec:
schedulerName: hami-scheduler
```

没有请求 HAMi 管理资源的 Pod 会继续使用常规 Kubernetes 调度流程。明确选择其他调度器的 Pod 不会被静默改写。

## HAMi 调度器 {#hami-scheduler}

HAMi 调度器负责将任务分配给适当的节点和设备。同时,调度器需要维护异构计算设备的全局视图以进行监控。
HAMi 调度器负责同时选择节点和设备。Kubernetes 设备插件通常只能通告整数形式的资源数量,无法完整描述设备型号、显存容量、算力、健康状态或拓扑等属性。因此,HAMi 设备插件通过节点注解上报详细设备信息,使调度器能够维护集群范围的设备视图。

HAMi 以[调度器扩展程序](https://github.com/kubernetes/design-proposals-archive/blob/main/scheduling/scheduler_extender.md)的方式参与标准调度流程,并不会取代 Kubernetes 调度机制。它根据资源请求筛选候选节点,应用配置的 binpack 或 spread 策略,绑定 Pod,并将分配结果写入 `hami.io/vgpu-devices-allocated` 等注解。

## 设备插件 {#device-plugin}

设备插件层从任务的注释字段获取调度结果,并将相应的设备映射到容器。
HAMi 设备插件运行在每个受支持的加速器节点上,并实现 Kubernetes [设备插件 API](https://kubernetes.io/zh-cn/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/)。它发现本地设备、向 kubelet 注册可分配资源、上报调度所需的设备详情,并在 Pod 绑定后处理 kubelet 的 `Allocate` 请求。

分配期间,设备插件从 Pod 注解中读取调度结果,并将选定的设备提供给容器。根据厂商集成方式的不同,它可能会挂载设备文件和运行时库,或注入描述分配配额的环境变量。

## HAMi-Core {#hami-core}

容器内资源控制负责监控容器内的资源使用情况,并提供硬隔离能力。
HAMi-Core 通过 `libvgpu.so` 为 NVIDIA 虚拟 GPU 提供运行时控制。设备插件借助 `/etc/ld.so.preload` 将该库加载到容器中。随后,HAMi-Core 会拦截 CUDA 显存分配和内核启动调用:超出已分配显存配额的申请会收到显存不足错误,算力使用则会被节流至请求的限制。它还会调整 NVML 返回结果,使应用看到已分配的显存,而不是物理设备的全部显存。

这种机制属于用户态控制,并非硬件安全边界。绕过被拦截库的应用(例如使用直接驱动调用或 Docker-in-Docker)可能绕过这些限制。当 GPU 支持 MIG 且需要硬件级隔离时,请使用 [NVIDIA MIG](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/)。完整的拦截和分配流程请参阅 [GPU 虚拟化原理](./gpu-virtualization.md)。

## HAMi 与时间切片和 MIG 有何区别?

| 方案 | 共享方式 | 显存和算力边界 | 适用场景 |
| --- | --- | --- | --- |
| 时间切片 | 多个工作负载轮流使用同一 GPU | 不提供工作负载级 GPU 显存隔离 | 不需要严格配额的简单并发场景 |
| HAMi 虚拟 GPU | 多个工作负载按灵活请求的配额共享 GPU | 用户态显存控制和算力节流 | 在多种 GPU 上进行细粒度、动态共享 |
| NVIDIA MIG | 将受支持的 GPU 划分为固定硬件分区 | 硬件级显存和算力隔离 | 在支持 MIG 的 GPU 上实现强隔离 |

HAMi 也支持动态 MIG 分配,因此这些方案并不总是互斥。具体选择取决于加速器型号、工作负载、隔离要求和期望的分区粒度。

## 产品架构与参考部署

本文介绍 HAMi 的产品组件,以及各类集成共有的请求处理流程。完整集群还可以包括 CNI、厂商驱动和运行时、监控系统及可选的仪表盘。有关一种面向 NVIDIA 的部署拓扑及其依赖关系,请参阅 [HAMi 安装后的集群架构](./hami-architecture.md)。

## 参考资料

- [Kubernetes 设备插件](https://kubernetes.io/zh-cn/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/)
- [Kubernetes 调度器扩展程序设计](https://github.com/kubernetes/design-proposals-archive/blob/main/scheduling/scheduler_extender.md)
- [HAMi-Core 源代码](https://github.com/Project-HAMi/HAMi-core)
- [NVIDIA Multi-Instance GPU 用户指南](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/)

## 下一步

- 深入了解 [GPU 虚拟化原理](./gpu-virtualization.md)
- 查看[参考集群架构](./hami-architecture.md)
- 检查[安装前提条件](../installation/prerequisites.md)
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@ title: "HAMi 安装后的集群架构"

完成 HAMi 安装后,集群不再是一个普通的 Kubernetes 集群,它变成了一个具备 GPU 虚拟化能力的 AI 基础设施平台。本文将拆解安装完成后集群中每一层、每一个组件的职责和依赖关系。

:::note

本文描述一种面向 NVIDIA 的参考部署拓扑(安装完成后),其中可能包含 GPU Operator、Prometheus 和 WebUI 等可选组件。HAMi 的产品组件与请求处理流程见[架构设计](./architecture.md)。

:::

## 5 层架构总览

安装完成后的集群由 5 层组成,每一层为上层提供服务:
Expand Down
Loading