Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/faq/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,8 @@ devicePlugin:

DCGM Exporter is not affected and continues to report physical-level counters normally. HAMi’s per-container virtual metrics are separate; see [GPU Utilization Metrics](../developers/gpu-utilization-metrics).

If the HAMi Device Plugin or workloads fail after upgrading to GPU Operator 25.10+, see [NVIDIA containers fail with GPU Operator 25.10+](../troubleshooting/troubleshooting.md#nvidia-toolkit-gpu-operator-25-10).

## How do I set up Prometheus and Grafana monitoring for HAMi vGPU metrics?

The `hami-device-plugin` pod on each node exposes per-container vGPU metrics on port `31992` (configurable via `devicePlugin.service.httpPort`). See [Grafana Dashboard](../userguide/monitoring/grafana-dashboard) for the full setup including Prometheus scrape config and dashboard import.
127 changes: 127 additions & 0 deletions docs/troubleshooting/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,3 +33,130 @@ If a container exceeds its `nvidia.com/gpumem` limit, check the following causes
```

This will apply the fix automatically.

## NVIDIA containers fail with GPU Operator 25.10+ {#nvidia-toolkit-gpu-operator-25-10}

Use this section when the HAMi Device Plugin or a HAMi-scheduled NVIDIA workload stops starting after GPU Operator is installed or upgraded.

### Problem 1: The HAMi Device Plugin fails to start

#### Identify the cause

Check the Device Plugin logs:

```bash
kubectl logs -n kube-system \
-l app.kubernetes.io/component=hami-device-plugin \
--all-containers --tail=200
```

Match the output to one of these errors:

| Error in the log | Cause |
| --- | --- |
| `Incompatible strategy detected auto` | The Device Plugin cannot discover NVML devices because its container did not receive the NVIDIA driver and devices. |
| `invalid device discovery strategy` | The Device Plugin could not initialize NVIDIA device discovery, usually for the same runtime-injection reason. |
| `failed to locate libcuda.so` or `failed to locate libnvidia-ml.so` | HAMi cannot find the driver libraries under the configured driver root while generating a CDI specification. |

Confirm the runtime and CDI configuration:

```bash
kubectl get clusterpolicy -o yaml | grep -A 5 'cdi:'
kubectl get runtimeclass nvidia

kubectl get pods -n kube-system \
-l app.kubernetes.io/component=hami-device-plugin \
-o custom-columns=NAME:.metadata.name,RUNTIMECLASS:.spec.runtimeClassName
```

With GPU Operator 25.10+, CDI is normally enabled, the `nvidia` RuntimeClass must exist, and the HAMi Device Plugin must show `nvidia` in the `RUNTIMECLASS` column.

#### Solution

Configure the `nvidia` RuntimeClass for HAMi and restart the Device Plugin:

```bash
helm upgrade hami hami-charts/hami \
--namespace kube-system \
--reuse-values \
--set devicePlugin.runtimeClassName=nvidia

kubectl rollout restart daemonset/hami-device-plugin -n kube-system
kubectl rollout status daemonset/hami-device-plugin -n kube-system
```

If the logs report missing driver libraries while HAMi CDI is enabled, use the GPU Operator paths:

```yaml
devicePlugin:
runtimeClassName: nvidia
deviceListStrategy: cdi-annotations
nvidiaDriverRoot: /run/nvidia/driver
nvidiaHookPath: /usr/local/nvidia/toolkit/nvidia-ctk
```

Wait until the NVIDIA driver and Toolkit DaemonSets are ready, then restart the HAMi Device Plugin. For host-installed drivers, set `devicePlugin.nvidiaDriverRoot` to `/` instead.

### Problem 2: A HAMi-scheduled Pod fails to start

#### Identify the cause

Inspect the Pod events and its assigned RuntimeClass:

```bash
kubectl describe pod <pod-name> -n <namespace>
kubectl get pod <pod-name> -n <namespace> \
-o custom-columns=NAME:.metadata.name,RUNTIMECLASS:.spec.runtimeClassName
```

Use the error text to select the correct path:

| Error in the Pod events | Cause |
| --- | --- |
| `libcuda.so.1: cannot open shared object file` | The container started without the NVIDIA driver libraries. |
| `unresolvable CDI devices management.nvidia.com/gpu=GPU-...` | The NVIDIA runtime selected a GPU management CDI device, but the corresponding device could not be generated or resolved. |
| `unresolvable CDI devices k8s.device-plugin.nvidia.com/gpu=GPU-...` | HAMi returned a CDI device, but the runtime cannot find a matching HAMi-generated CDI specification. |

#### Solution

First determine which HAMi injection mode is configured:

```bash
helm get values hami -n kube-system | grep -A 5 'devicePlugin:'
```
Comment thread
FouoF marked this conversation as resolved.
Resolved

- For the default `devicePlugin.deviceListStrategy=envvar` mode, set `devicePlugin.runtimeClassName=nvidia` by using the Helm command from Problem 1. This makes the NVIDIA runtime process the UUID returned through `NVIDIA_VISIBLE_DEVICES`.
- For `devicePlugin.deviceListStrategy=cdi-annotations`, apply all four CDI values shown in Problem 1. Then inspect `/var/run/cdi/k8s.device-plugin.nvidia.com-gpu.json` on the node and verify that it contains the allocated GPU UUID.
- For a host-installed Container Toolkit, confirm that the `nvidia` runtime is present in the active container runtime configuration. Restart the container runtime after correcting its configuration.

Do not mix HAMi CDI annotations with a missing or stale HAMi CDI specification. See [NVIDIA CDI support](../installation/configure-cdi.md) for the complete setup and verification procedure.

### Why this happens

Starting with [GPU Operator 25.10.0](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/25.10/cdi.html), CDI is enabled by default and the Operator no longer makes the `nvidia` runtime the default runtime.

Before 25.10.0, GPU Operator normally configured the NVIDIA runtime as the default. As a result, the NVIDIA runtime hook processed every Pod and injected the devices and driver libraries selected through `NVIDIA_VISIBLE_DEVICES`.

With 25.10.0 and later, `runc` remains the default runtime and the container runtime uses native CDI for standard Device Plugin workloads. GPU management containers that access GPUs through `NVIDIA_VISIBLE_DEVICES`, including the HAMi Device Plugin, must explicitly use `runtimeClassName: nvidia`.

HAMi supports two device-injection paths:

| HAMi mode | Allocation result | Runtime requirement |
| --- | --- | --- |
| `envvar` (default) | HAMi writes the allocated GPU UUID to `NVIDIA_VISIBLE_DEVICES`. | On GPU Operator 25.10+, the Pod must use the `nvidia` RuntimeClass. |
| `cdi-annotations` | HAMi returns a CDI device named `k8s.device-plugin.nvidia.com/gpu=GPU-...` and generates its CDI specification on the node. | The container runtime must have CDI enabled and be able to read the current HAMi specification. |

The HAMi chart applies `devicePlugin.runtimeClassName` both to the Device Plugin and to NVIDIA workloads mutated by the HAMi scheduler. This is why setting it to `nvidia` fixes the management container and keeps the workload runtime path consistent.

For new clusters, GPU Operator is recommended because it provides one entry point for configuring and upgrading the driver, Container Toolkit, and monitoring components. If these components are already installed on the hosts and you maintain their runtime configuration yourself, GPU Operator is optional; follow [Prerequisites](../installation/prerequisites.md).

:::warning

Disable the GPU Operator Device Plugin when using HAMi. Both plugins advertise `nvidia.com/gpu` and must not run on the same nodes.

```yaml
devicePlugin:
enabled: false
```

:::
2 changes: 2 additions & 0 deletions i18n/zh/docusaurus-plugin-content-docs/current/faq/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -220,6 +220,8 @@ devicePlugin:

DCGM Exporter 不受影响,继续正常上报物理级计数器。HAMi 的每容器虚拟指标是独立的;参见 [GPU 利用率指标](../developers/gpu-utilization-metrics)。

如果升级到 GPU Operator 25.10+ 后 HAMi Device Plugin 或工作负载启动失败,请参见[使用 GPU Operator 25.10+ 时 NVIDIA 容器启动失败](../troubleshooting/troubleshooting.md#nvidia-toolkit-gpu-operator-25-10)。

## 如何为 HAMi vGPU 指标设置 Prometheus 和 Grafana 监控?

每个节点上的 `hami-device-plugin` Pod 在端口 `31992`(可通过 `devicePlugin.service.httpPort` 配置)上暴露每容器 vGPU 指标。完整的设置步骤(包括 Prometheus 采集配置和 Dashboard 导入)参见 [Grafana Dashboard](../userguide/monitoring/grafana-dashboard)。
Original file line number Diff line number Diff line change
Expand Up @@ -25,3 +25,130 @@ translated: true
- 从 v2.3.10 起,HAMi 将 `device-plugin` 环境变量名称从 `NodeName` 更改为 `NODE_NAME`。如果你使用早于 v2.3.10 的镜像版本,`device-plugin` 可能无法启动,有两种方法可以解决:
- 手动执行 `kubectl edit daemonset` 修改 `device-plugin` 环境变量从 `NodeName` 为 `NODE_NAME`。
- 使用 Helm 升级到最新版本:执行 `helm upgrade hami hami/hami -n kube-system`,它将自动修复。

## 使用 GPU Operator 25.10+ 时 NVIDIA 容器启动失败 {#nvidia-toolkit-gpu-operator-25-10}

安装或升级 GPU Operator 后,如果 HAMi Device Plugin 或由 HAMi 调度的 NVIDIA 工作负载无法启动,请按照本节排查。

### 问题一:HAMi Device Plugin 无法启动

#### 定位原因

查看 Device Plugin 日志:

```bash
kubectl logs -n kube-system \
-l app.kubernetes.io/component=hami-device-plugin \
--all-containers --tail=200
```

根据错误日志定位原因:

| 错误日志 | 原因 |
| --- | --- |
| `Incompatible strategy detected auto` | Device Plugin 容器没有获得 NVIDIA 驱动和设备,因而无法通过 NVML 发现 GPU。 |
| `invalid device discovery strategy` | Device Plugin 无法初始化 NVIDIA 设备发现,通常也是 runtime 未正确注入设备所致。 |
| `failed to locate libcuda.so` 或 `failed to locate libnvidia-ml.so` | HAMi 生成 CDI spec 时,无法从配置的驱动根目录找到驱动库。 |

确认 runtime 和 CDI 配置:

```bash
kubectl get clusterpolicy -o yaml | grep -A 5 'cdi:'
kubectl get runtimeclass nvidia

kubectl get pods -n kube-system \
-l app.kubernetes.io/component=hami-device-plugin \
-o custom-columns=NAME:.metadata.name,RUNTIMECLASS:.spec.runtimeClassName
```

使用 GPU Operator 25.10+ 时,CDI 通常处于开启状态,集群中必须存在 `nvidia` RuntimeClass,HAMi Device Plugin 的 `RUNTIMECLASS` 列应显示 `nvidia`。

#### 解决方案

为 HAMi 配置 `nvidia` RuntimeClass,然后重启 Device Plugin:

```bash
helm upgrade hami hami-charts/hami \
--namespace kube-system \
--reuse-values \
--set devicePlugin.runtimeClassName=nvidia

kubectl rollout restart daemonset/hami-device-plugin -n kube-system
kubectl rollout status daemonset/hami-device-plugin -n kube-system
```

如果 HAMi 已开启 CDI,且日志显示驱动库缺失,请使用 GPU Operator 对应的路径:

```yaml
devicePlugin:
runtimeClassName: nvidia
deviceListStrategy: cdi-annotations
nvidiaDriverRoot: /run/nvidia/driver
nvidiaHookPath: /usr/local/nvidia/toolkit/nvidia-ctk
```

等待 NVIDIA 驱动和 Toolkit DaemonSet 就绪后,再重启 HAMi Device Plugin。如果驱动直接安装在宿主机,请将 `devicePlugin.nvidiaDriverRoot` 设为 `/`。

### 问题二:HAMi 调度的 Pod 无法启动

#### 定位原因

检查 Pod 事件和分配到的 RuntimeClass:

```bash
kubectl describe pod <pod-name> -n <namespace>
kubectl get pod <pod-name> -n <namespace> \
-o custom-columns=NAME:.metadata.name,RUNTIMECLASS:.spec.runtimeClassName
```

根据错误内容选择对应的排障路径:

| Pod 事件中的错误 | 原因 |
| --- | --- |
| `libcuda.so.1: cannot open shared object file` | 容器启动时没有注入 NVIDIA 驱动库。 |
| `unresolvable CDI devices management.nvidia.com/gpu=GPU-...` | NVIDIA runtime 选择了 GPU 管理 CDI 设备,但无法生成或解析对应设备。 |
| `unresolvable CDI devices k8s.device-plugin.nvidia.com/gpu=GPU-...` | HAMi 返回了 CDI 设备,但容器运行时找不到匹配的 HAMi CDI spec。 |

#### 解决方案

先确认 HAMi 使用的设备注入模式:

```bash
helm get values hami -n kube-system | grep -A 5 'devicePlugin:'
```

- 默认的 `devicePlugin.deviceListStrategy=envvar` 模式:使用问题一中的 Helm 命令设置 `devicePlugin.runtimeClassName=nvidia`,让 NVIDIA runtime 处理通过 `NVIDIA_VISIBLE_DEVICES` 返回的 UUID。
- `devicePlugin.deviceListStrategy=cdi-annotations` 模式:同时应用问题一列出的四个 CDI 参数,然后在节点上检查 `/var/run/cdi/k8s.device-plugin.nvidia.com-gpu.json`,确认其中包含分配到的 GPU UUID。
- Container Toolkit 直接安装在宿主机:确认当前容器运行时配置中存在 `nvidia` runtime;修正配置后重启容器运行时。

不要混用 HAMi CDI annotation 与缺失或过期的 HAMi CDI spec。完整配置和验证步骤参见 [NVIDIA CDI 支持](../installation/configure-cdi.md)。

### 为什么会出现这个问题

从 [GPU Operator 25.10.0](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/25.10/cdi.html) 开始,CDI 默认开启,Operator 不再将 `nvidia` 设为默认 runtime。

在 25.10.0 之前,GPU Operator 通常会把 NVIDIA runtime 配置为默认 runtime。因此,每个 Pod 都会经过 NVIDIA runtime hook,并根据 `NVIDIA_VISIBLE_DEVICES` 注入指定的设备和驱动库。

从 25.10.0 开始,`runc` 保持为默认 runtime,普通 Device Plugin 工作负载由容器运行时通过原生 CDI 注入设备。通过 `NVIDIA_VISIBLE_DEVICES` 访问 GPU 的管理容器必须显式使用 `runtimeClassName: nvidia`,HAMi Device Plugin 就属于此类管理容器。

HAMi 支持两条设备注入路径:

| HAMi 模式 | 分配结果 | runtime 要求 |
| --- | --- | --- |
| `envvar`(默认) | HAMi 将分配到的 GPU UUID 写入 `NVIDIA_VISIBLE_DEVICES`。 | 使用 GPU Operator 25.10+ 时,Pod 必须使用 `nvidia` RuntimeClass。 |
| `cdi-annotations` | HAMi 返回名为 `k8s.device-plugin.nvidia.com/gpu=GPU-...` 的 CDI 设备,并在节点上生成对应 CDI spec。 | 容器运行时必须开启 CDI,并能够读取当前的 HAMi CDI spec。 |

HAMi Chart 会把 `devicePlugin.runtimeClassName` 同时应用到 Device Plugin,以及由 HAMi scheduler 修改的 NVIDIA 工作负载。因此,将其设为 `nvidia` 既能修复管理容器,也能保持工作负载的 runtime 路径一致。

对于新集群,推荐使用 GPU Operator,因为它提供统一的驱动、Container Toolkit 和监控组件配置与升级入口。如果这些组件已经安装在宿主机,并由你自行维护容器运行时配置,也可以不使用 GPU Operator;这种情况请参考[前置条件](../installation/prerequisites.md)。

:::warning

使用 HAMi 时必须关闭 GPU Operator 自带的 Device Plugin。两者都会注册 `nvidia.com/gpu`,不能在同一节点同时运行。

```yaml
devicePlugin:
enabled: false
```

:::