diff --git a/docs/developers/protocol.md b/docs/developers/protocol.md
index 3932766ec..adc84c335 100644
--- a/docs/developers/protocol.md
+++ b/docs/developers/protocol.md
@@ -6,7 +6,7 @@ title: Protocol design
-HAMi needs to know the spec of each AI device in the cluster to schedule properly. During device registration, device-plugin needs to keep patching the spec of each device into node annotations every 30 seconds, in the format of the following:
+HAMi needs to know the spec of each AI device in the cluster to schedule properly. During device registration, device-plugin rescans its devices every 30 seconds and patches the spec of each device into node annotations, in the format of the following:
```text
hami.io/node-handshake-\{device-type\}: Reported_\{device_node_current_timestamp\}
@@ -19,10 +19,18 @@ The definition of each device is in the following format:
\{Device UUID\},\{device split count\},\{device memory limit\},\{device core limit\},\{device type\},\{device numa\},\{healthy\}
```
+:::note Encoding differs by vendor
+
+The colon-separated form above is the legacy encoding, still used by device types decoded with `DecodeNodeDevices` (for example DCU and Iluvatar). NVIDIA and several other device types encode the same fields as a **JSON array** instead, one object per device, decoded with `UnMarshalNodeDevices`. Fields are serialized with `omitempty`, so zero and `false` values are absent rather than written out. See [GPU Virtualization](../core-concepts/gpu-virtualization.md) for a JSON example.
+
+The annotation key is also not fully uniform: NVIDIA uses `hami.io/node-nvidia-register` and `hami.io/node-handshake` (no device-type suffix), while Kunlun uses `hami.io/node-register-xpu`.
+
+:::
+
An example is shown below:
```text
-hami.io/node-handshake-nvidia: Reported 2024-01-23 04:30:04.434037031 +0000 UTC m=+1104711.777756895
+hami.io/node-handshake: Requesting_2024-01-23 04:30:04.434037031 +0000 UTC m=+1104711.777756895
hami.io/node-handshake-mlu: Requesting_2024.01.10 04:06:57
hami.io/node-mlu-register: MLU-45013011-2257-0000-0000-000000000000,10,23308,0,MLU-MLU370-X4,0,false:MLU-54043011-2257-0000-0000-000000000000,10,23308,0,
hami.io/node-nvidia-register: GPU-00552014-5c87-89ac-b1a6-7b53aa24b0ec,10,32768,100,NVIDIA-Tesla V100-PCIE-32GB,0,true:GPU-0fc3eda5-e98b-a25b-5b0d-cf5c855d1448,10,32768,100,NVIDIA-Tesla V100-PCIE-32GB,0,true:
@@ -31,10 +39,18 @@ hami.io/node-nvidia-register: GPU-00552014-5c87-89ac-b1a6-7b53aa24b0ec,10,32768,
In this example, this node has two different AI devices, 2 NVIDIA-V100 GPUs, and 2 Cambricon 370-X4 MLUs
-A device node may become unavailable due to hardware or network failure. If a node hasn't registered in the last 5 minutes, the scheduler marks it as 'unavailable'.
-
-Since system clock on scheduler node and 'device' node may not align properly, scheduler node will patch the following device node annotations every 30s
+A device node may become unavailable due to hardware or network failure. Since the system clock on the scheduler node and on the device node may not align properly, the scheduler owns the timestamp. Whenever the handshake annotation is absent or does not contain `Requesting`, the scheduler stamps it with its own clock.
```text
hami.io/node-handshake-\{device-type\}: Requesting_{scheduler_node_current_timestamp}
```
+
+The scheduler's registration loop runs every 15 seconds, and also on node events and on leader election changes. Only the elected leader performs registration.
+
+A handshake is treated as expired once its timestamp is more than **60 seconds** old. Expiry alone does not remove the node. The scheduler additionally requires the node's allocatable device count to have dropped to zero before it runs node cleanup, which removes the node's devices from the scheduler cache and deletes the handshake annotation. A node whose device-plugin is still reporting to kubelet therefore stays available even with an expired handshake.
+
+:::note The NVIDIA device-plugin does not write the `Reported_` handshake
+
+The `Reported_` side of this protocol is written by device-plugins that implement it. The in-tree NVIDIA device-plugin does not, so on an NVIDIA node the annotation normally stays at `Requesting_` and ages past 60 seconds. That is the expected steady state, not a fault: the effective liveness signal for NVIDIA is the allocatable device count. See [GPU Nodes Not Registering](../troubleshooting/node-registration.md).
+
+:::
diff --git a/docs/troubleshooting/node-registration.md b/docs/troubleshooting/node-registration.md
new file mode 100644
index 000000000..f3890620d
--- /dev/null
+++ b/docs/troubleshooting/node-registration.md
@@ -0,0 +1,384 @@
+---
+title: GPU Nodes Not Registering
+sidebar_label: Node Registration
+---
+
+Before HAMi can schedule anything, a GPU node has to complete registration. This page covers the failures that happen _before_ scheduling: the node advertises no GPUs, the HAMi scheduler does not know the node exists, or a node that used to work silently drops out of the cluster's GPU capacity.
+
+Typical symptoms:
+
+- `kubectl describe node` shows no `nvidia.com/gpu` under `Capacity` and `Allocatable`.
+- A Pod requesting `nvidia.com/gpu` stays `Pending` with `0/N nodes are available`, and **no** `FilteringFailed` event from `hami-scheduler`.
+- `nvidia-smi` works on the host, but the node still contributes nothing to the cluster.
+- A node worked yesterday and stopped being selected today.
+
+If your Pod _does_ get a `FilteringFailed` event from `hami-scheduler`, registration already succeeded and the problem is scheduling instead. See [Troubleshooting](./troubleshooting.md).
+
+## How registration works
+
+Registration is not one action. It is three independent channels, and each can break on its own:
+
+```mermaid
+%% title: HAMi GPU node registration path
+flowchart TB
+ A["NVIDIA driver + NVML on the node"] --> B["hami-device-plugin"]
+ B -->|"channel 1: gRPC ListAndWatch"| C["kubelet"]
+ C -->|"node status"| D["nvidia.com/gpu in Capacity/Allocatable"]
+ B -->|"channel 2: patch node annotation"| E["hami.io/node-nvidia-register"]
+ D --> F["hami-scheduler cache"]
+ E --> F
+ F -->|"channel 3: handshake"| G["hami.io/node-handshake"]
+```
+
+| Channel | Written by | Carries | Where to observe it |
+| --- | --- | --- | --- |
+| Device count | Device Plugin to kubelet | An integer count only | `nvidia.com/gpu` in node `Allocatable` |
+| Device specification | Device Plugin to the API server | UUID, memory, compute, model, NUMA, health | `hami.io/node-nvidia-register` node annotation |
+| Liveness handshake | Scheduler to the API server | A timestamp | `hami.io/node-handshake` node annotation |
+
+The count and the specification travel separately because the Device Plugin API can only report a single integer resource. A node can therefore advertise `nvidia.com/gpu: 10` while the HAMi scheduler still refuses to use it, because the annotation the scheduler actually reads is missing.
+
+The count kubelet advertises is the **inflated** count: physical GPUs multiplied by `devicePlugin.deviceSplitCount` (default `10`). One physical card on a default install shows as `nvidia.com/gpu: 10`, not `1`. See [GPU Virtualization](../core-concepts/gpu-virtualization.md).
+
+## Step 1: Find the broken channel
+
+Run all three checks against the affected node before changing anything:
+
+```bash
+NODE=
+
+# Channel 1: does kubelet advertise the resource?
+kubectl get node $NODE -o jsonpath='{.status.allocatable.nvidia\.com/gpu}{"\n"}'
+
+# Channel 2: did the Device Plugin write the device specification?
+kubectl get node $NODE -o jsonpath='{.metadata.annotations.hami\.io/node-nvidia-register}{"\n"}'
+
+# Channel 3: is a Device Plugin Pod actually running there?
+kubectl get pods -n kube-system \
+ -l app.kubernetes.io/component=hami-device-plugin \
+ -o wide --field-selector spec.nodeName=$NODE
+```
+
+Match the result to the section to read next:
+
+| Result | Broken channel | Go to |
+| -------------------------------------- | ------------------------------- | ----------------- |
+| No Device Plugin Pod on the node | Nothing is registering | [Case 1](#case-1) |
+| Pod exists but is not `Running` | Plugin cannot start | [Case 2](#case-2) |
+| Empty `nvidia.com/gpu`, Pod `Running` | Plugin to kubelet | [Case 3](#case-3) |
+| `nvidia.com/gpu` set, annotation empty | Plugin to API server | [Case 4](#case-4) |
+| Both set, Pod still `Pending` | Scheduler does not see the node | [Case 5](#case-5) |
+
+## Case 1: no Device Plugin Pod on the node {#case-1}
+
+The HAMi NVIDIA Device Plugin DaemonSet carries a node selector. Its chart default is:
+
+```yaml
+devicePlugin:
+ nvidiaNodeSelector:
+ gpu: "on"
+```
+
+A node without that label never receives a Device Plugin Pod, so none of the three channels start. This is the single most common cause of a GPU node contributing nothing.
+
+### Check
+
+```bash
+kubectl get node $NODE --show-labels | tr ',' '\n' | grep gpu
+kubectl get daemonset -n kube-system hami-device-plugin \
+ -o jsonpath='{.spec.template.spec.nodeSelector}{"\n"}'
+```
+
+### Fix
+
+```bash
+kubectl label node $NODE gpu=on --overwrite
+```
+
+Then wait for the DaemonSet to place a Pod:
+
+```bash
+kubectl rollout status daemonset/hami-device-plugin -n kube-system
+```
+
+If the label is already present and correct, check that the node is not tainted in a way the DaemonSet does not tolerate:
+
+```bash
+kubectl describe node $NODE | grep -A 3 Taints
+```
+
+Add matching entries under `devicePlugin.tolerations` if needed. Node labelling is also covered in [Prerequisites](../installation/prerequisites.md).
+
+## Case 2: the Device Plugin Pod does not stay running {#case-2}
+
+```bash
+kubectl logs -n kube-system -l app.kubernetes.io/component=hami-device-plugin \
+ -c device-plugin --tail=100
+kubectl describe pod -n kube-system -l app.kubernetes.io/component=hami-device-plugin
+```
+
+### NVML initialization failure
+
+```plaintext
+nvml Init err: ERROR_LIBRARY_NOT_FOUND
+```
+
+The Device Plugin treats every NVML failure during a device scan as fatal and exits, so the container ends up in `CrashLoopBackOff` rather than running degraded. The same fatal path is taken for `nvml get memory error`, `nvml get name error`, and `nvml new device by index error`.
+
+This almost always means the container did not receive the driver, which in turn means `nvidia-container-runtime` is not the default runtime on the node:
+
+```bash
+containerd config dump | grep default_runtime_name
+```
+
+The output must be `nvidia`. If it is not, follow [Prerequisites](../installation/prerequisites.md), then restart the container runtime. On GPU Operator 25.10 and later the default runtime deliberately stays `runc`; that case needs `devicePlugin.runtimeClassName=nvidia` instead, and is covered in [Troubleshooting](./troubleshooting.md#nvidia-toolkit-gpu-operator-25-10).
+
+### Stuck in `Init`
+
+```plaintext
+Waiting for /run/nvidia/validations/toolkit-ready...
+```
+
+The `toolkit-validation` init container blocks until the NVIDIA Container Toolkit writes a `toolkit-ready` file under `devicePlugin.gpuOperatorToolkitReady.hostPath` (default `/run/nvidia/validations`). The gate is off by default and is meant for GPU Operator clusters. If it was enabled on a cluster without GPU Operator, that file is never created and the init container waits forever:
+
+```bash
+helm upgrade hami hami-charts/hami -n kube-system --reuse-values \
+ --set devicePlugin.gpuOperatorToolkitReady.enabled=false
+```
+
+### Node name not resolved
+
+The Device Plugin patches the node named by its `NODE_NAME` environment variable, which the chart populates from `spec.nodeName`. On charts older than v2.3.10 the variable was called `NodeName`, and a mismatched image and chart pair leaves the plugin unable to identify its own node. Upgrade rather than patching by hand:
+
+```bash
+helm upgrade hami hami-charts/hami -n kube-system --reuse-values
+```
+
+## Case 3: the resource never appears in Allocatable {#case-3}
+
+The Pod is `Running` and NVML works, but `nvidia.com/gpu` is absent. The Device Plugin registered with the API server but not with kubelet.
+
+### Check
+
+```bash
+kubectl logs -n kube-system -l app.kubernetes.io/component=hami-device-plugin \
+ -c device-plugin --tail=200 | grep -i -E "register|socket|kubelet"
+
+ls -l /var/lib/kubelet/device-plugins/ # on the node itself
+```
+
+The plugin registers over `kubelet.sock` in that directory. The chart mounts it from `devicePlugin.pluginPath`, which defaults to `/var/lib/kubelet/device-plugins`. If your distribution relocates the kubelet root, the plugin writes its socket somewhere kubelet never reads and registration silently never completes.
+
+### Fix
+
+Confirm the real path on the node, then point the chart at it:
+
+```bash
+# on the node
+ps aux | grep kubelet | grep -o '\--root-dir=[^ ]*'
+
+helm upgrade hami hami-charts/hami -n kube-system --reuse-values \
+ --set devicePlugin.pluginPath=/device-plugins
+```
+
+Restarting kubelet also forces every Device Plugin to re-register, which is a fast way to confirm the socket path is the problem.
+
+## Case 4: the register annotation is missing or stale {#case-4}
+
+This is the case that most often looks like a scheduler bug. kubelet advertises the GPUs, so `kubectl describe node` looks healthy, but `hami-scheduler` never places a Pod there.
+
+### Read the annotation
+
+```bash
+kubectl get node $NODE \
+ -o jsonpath='{.metadata.annotations.hami\.io/node-nvidia-register}' | jq .
+```
+
+Expected output for one 24 GiB card on a default install:
+
+```json
+[
+ {
+ "id": "GPU-fc28df76-54d2-c387-e52e-5f0a9495968c",
+ "count": 10,
+ "devmem": 24576,
+ "devcore": 100,
+ "type": "NVIDIA-NVIDIA L40S",
+ "mode": "hami-core",
+ "health": true
+ }
+]
+```
+
+| Field | Meaning | Source |
+| --------- | ------------------------------ | ------------------------------------------- |
+| `id` | GPU UUID | NVML |
+| `count` | Logical split count | `devicePlugin.deviceSplitCount` |
+| `devmem` | Schedulable memory in MiB | Physical memory times `deviceMemoryScaling` |
+| `devcore` | Schedulable compute percentage | `deviceCoreScaling` times 100 |
+| `type` | Model, prefixed with `NVIDIA-` | NVML |
+| `numa` | NUMA node | sysfs |
+| `mode` | `hami-core` or `mig` | Plugin operating mode |
+| `health` | Device health | Device Plugin health check |
+
+:::warning Zero-valued fields are omitted
+
+The annotation is serialized with `omitempty`, so any field whose value is zero or `false` is absent rather than shown. An **unhealthy GPU has no `health` key at all**, it does not appear as `"health": false`. The same applies to `"numa": 0` and `"index": 0`. Read a missing `health` key as unhealthy, not as healthy.
+
+:::
+
+### The annotation is only rewritten when it changes
+
+The Device Plugin rescans devices every 30 seconds, but it compares the newly encoded list against the last one it wrote and skips the patch when they are identical:
+
+```plaintext
+Device info unchanged, skipping annotation update
+```
+
+That line appears at verbosity `-v=3`. At the default verbosity, a real update logs:
+
+```plaintext
+Updating node annotations with 1 device(s)
+```
+
+The consequence: an old timestamp on the annotation is normal and is not evidence of a stuck plugin. Conversely, deleting the annotation by hand does **not** get it rewritten within 30 seconds, because the plugin's in-memory cache still matches what it thinks it wrote. Restart the Pod instead. See [Force a re-registration](#force-a-re-registration).
+
+### The patch is rejected
+
+```plaintext
+patch node error nodes "gpu-node-1" is forbidden: User "system:serviceaccount:kube-system:hami-device-plugin" cannot patch resource "nodes"
+```
+
+The ServiceAccount lost permission to patch nodes, usually after a partial upgrade or a hand-edited ClusterRole:
+
+```bash
+kubectl auth can-i patch nodes \
+ --as=system:serviceaccount:kube-system:hami-device-plugin
+```
+
+Reinstalling or upgrading the chart restores the RBAC objects.
+
+## Case 5: the scheduler does not see the node {#case-5}
+
+Both the resource and the annotation are correct, but Pods still do not land. The node is missing from the scheduler's in-memory cache.
+
+### Check the scheduler log
+
+```bash
+kubectl logs -n kube-system deploy/hami-scheduler -c vgpu-scheduler-extender --tail=200
+```
+
+The registration loop runs every 15 seconds, and also on node events and on leader changes. Raise verbosity to `-v=5` to see per-node decisions, which are otherwise silent:
+
+```plaintext
+Using label selector for list nodes
+Listed nodes
+Processing node
+Failed to get node devices
+```
+
+`Failed to get node devices` for your node means the scheduler read the annotation and rejected it. There are three ways that happens: the annotation key is absent, the JSON does not decode, and the decoded list is empty. The second and third also log at the default verbosity:
+
+```plaintext
+failed to decode node devices
+no nvidia gpu device found
+```
+
+A hand-edited annotation is the usual cause of a decode failure.
+
+### Cause A: a node label selector excludes the node
+
+The scheduler can be restricted to a subset of nodes with `scheduler.nodeLabelSelector`. It is commented out in the chart by default, so an unmodified install lists every node. If it was set, the value is echoed on startup:
+
+```bash
+kubectl logs -n kube-system deploy/hami-scheduler -c vgpu-scheduler-extender \
+ | grep "label selector"
+```
+
+Any node missing those labels is never registered, no matter how healthy it is.
+
+### Cause B: the replica you are reading is not the leader
+
+Only the leader performs registration. Every other replica logs:
+
+```plaintext
+Scheduler is not leader yet, skipping ...
+```
+
+With more than one `hami-scheduler` replica, a quiet log is expected on followers and proves nothing. Identify the leader before concluding the loop is dead:
+
+```bash
+kubectl get lease -n kube-system | grep hami
+```
+
+## The handshake annotation
+
+`hami.io/node-handshake` is a liveness marker maintained by the **scheduler**, not by the Device Plugin. Reading it backwards causes a lot of wasted debugging, so it is worth stating the actual behavior:
+
+- When the annotation is absent, or its value does not contain `Requesting`, the scheduler stamps it with `Requesting_` and treats the node as healthy.
+- While the timestamp is less than **60 seconds** old, the node is healthy.
+- Once it is older than 60 seconds, the scheduler checks the node's allocatable `nvidia.com/gpu`. If that is still greater than zero, the node stays healthy and nothing is cleaned up.
+- Only when the handshake has expired **and** allocatable has dropped to zero does the scheduler run node cleanup: it drops the node's devices from its cache and deletes the handshake annotation.
+
+```bash
+kubectl get node $NODE -o jsonpath='{.metadata.annotations.hami\.io/node-handshake}{"\n"}'
+# Requesting_2026-08-15 09:12:44
+```
+
+Two practical consequences:
+
+- **An old `Requesting_` timestamp is not a fault.** It is the normal steady state, and on its own it never removes a node. Do not tune anything based on it.
+- **The real trigger for a node leaving the scheduler's cache is allocatable falling to zero**, which means the Device Plugin stopped reporting to kubelet. Debug that, not the handshake. The scheduler logs the removal:
+
+ ```plaintext
+ Device is unhealthy, cleaning up node
+ ```
+
+:::note NVIDIA uses an unsuffixed key
+
+The NVIDIA handshake key is `hami.io/node-handshake`. Other vendors use a suffixed form such as `hami.io/node-handshake-dcu` or `hami.io/node-handshake-xpu`. A `kubectl get node -o yaml | grep node-handshake-nvidia` returns nothing on an NVIDIA node, which is expected.
+
+:::
+
+## Force a re-registration
+
+Restarting the Device Plugin Pod clears its in-memory device cache and forces a fresh annotation patch on the next scan. This is the correct recovery for a missing, truncated, or hand-edited register annotation:
+
+```bash
+kubectl delete pod -n kube-system \
+ -l app.kubernetes.io/component=hami-device-plugin \
+ --field-selector spec.nodeName=$NODE
+```
+
+Confirm the cycle completed:
+
+```bash
+kubectl logs -n kube-system -l app.kubernetes.io/component=hami-device-plugin \
+ -c device-plugin --tail=50 | grep "Updating node annotations"
+
+kubectl get node $NODE \
+ -o jsonpath='{.metadata.annotations.hami\.io/node-nvidia-register}' | jq length
+```
+
+The scheduler picks the node back up within one registration cycle, so allow about 15 seconds before retesting with a Pod.
+
+If the node still does not register, collect the following before opening an issue:
+
+```bash
+kubectl get node $NODE -o yaml > node.yaml
+kubectl logs -n kube-system -l app.kubernetes.io/component=hami-device-plugin \
+ -c device-plugin --tail=500 > device-plugin.log
+kubectl logs -n kube-system deploy/hami-scheduler \
+ -c vgpu-scheduler-extender --tail=500 > scheduler.log
+```
+
+## Validation environment
+
+The behavior described on this page was verified by reading the HAMi source at **v2.9.0** and on `master`, specifically `pkg/device-plugin/nvidiadevice/nvinternal/plugin/register.go`, `pkg/device/devices.go`, `pkg/device/nvidia/device.go`, `pkg/scheduler/scheduler.go`, and `charts/hami/values.yaml`. Chart defaults quoted here are the NVIDIA values from `charts/hami`. Intervals, the 60-second handshake window, and log strings can change between releases; check the source for the version you run before relying on an exact number.
+
+## Related pages
+
+- [Troubleshooting](./troubleshooting.md) for runtime and GPU Operator issues
+- [Validate HAMi](../get-started/verify-hami.md) for the end-to-end post-install check
+- [Protocol design](../developers/protocol.md) for the registration protocol itself
+- [GPU Virtualization](../core-concepts/gpu-virtualization.md) for how the annotation feeds scheduling
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/developers/protocol.md b/i18n/zh/docusaurus-plugin-content-docs/current/developers/protocol.md
index 7ee6c7c69..077b3248f 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/current/developers/protocol.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/developers/protocol.md
@@ -18,6 +18,26 @@ hami.io/node-handshake: Requesting_2024.05.14 07:07:33
hami.io/node-nvidia-register: "GPU-00552014-5c87-89ac-b1a6-7b53aa24b0ec,10,32768,100,NVIDIA-Tesla V100-PCIE-32GB,0,true:GPU-0fc3eda5-e98b-a25b-5b0d-cf5c855d1448,10,32768,100,NVIDIA-Tesla V100-PCIE-32GB,0,true:"
```
+:::note 编码方式因设备类型而异
+
+上面这种冒号分隔的形式是早期编码，目前仍用于通过 `DecodeNodeDevices` 解码的设备类型（例如 DCU、天数智芯）。NVIDIA 以及其他若干设备类型改为使用 **JSON 数组**编码同样的字段，每个设备一个对象，通过 `UnMarshalNodeDevices` 解码。这些字段使用 `omitempty` 序列化，因此取值为零或 `false` 的字段会被省略而不会写出。JSON 示例参见 [GPU 虚拟化](../core-concepts/gpu-virtualization.md)。
+
+注解键也并非完全统一：NVIDIA 使用 `hami.io/node-nvidia-register` 和 `hami.io/node-handshake`（不带设备类型后缀），而昆仑芯使用 `hami.io/node-register-xpu`。
+
+:::
+
+### 握手与节点存活
+
+时间戳由调度器负责写入：当握手注解不存在，或其值不含 `Requesting` 时，调度器会用自己的时钟写入 `Requesting_<时间戳>`。调度器的注册循环每 15 秒执行一次，节点事件和主节点切换时也会触发，且只有当选为 leader 的副本才会执行注册。
+
+握手时间戳超过 **60 秒**即视为过期。但过期本身不会移除节点：调度器还要求该节点的可分配设备数量降为零，才会执行节点清理，把设备移出调度器缓存并删除握手注解。因此，只要 device-plugin 仍在向 kubelet 上报，节点即使握手过期也依然可用。
+
+:::note NVIDIA device-plugin 不写握手注解
+
+本协议中 `Reported_` 一侧由实现了该协议的 device-plugin 写入。内置的 NVIDIA device-plugin 并未实现，因此在 NVIDIA 节点上该注解通常一直停留在 `Requesting_<时间戳>` 并超过 60 秒。这是预期稳态而非故障：对 NVIDIA 而言，真正的存活信号是可分配设备数量。参见 [GPU 节点未注册](../troubleshooting/node-registration.md)。
+
+:::
+
### 调度决策
kube-scheduler 在 `bind` 过程中调用 device-plugin 挂载设备，但仅向 device-plugin 提供设备的 `UUID`。因此，在设备共享的场景中，device-plugin 无法获取任务请求的相应设备规格，如 `设备显存` 和 `计算核心`。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/node-registration.md b/i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/node-registration.md
new file mode 100644
index 000000000..bd40bf9ad
--- /dev/null
+++ b/i18n/zh/docusaurus-plugin-content-docs/current/troubleshooting/node-registration.md
@@ -0,0 +1,385 @@
+---
+title: GPU 节点未注册
+sidebar_label: 节点注册
+translated: true
+---
+
+在 HAMi 调度任何任务之前，GPU 节点必须先完成注册。本页覆盖发生在调度**之前**的故障：节点没有上报任何 GPU、HAMi 调度器不知道该节点存在，或者一个原本正常的节点悄悄从集群的 GPU 容量中消失。
+
+典型现象：
+
+- `kubectl describe node` 的 `Capacity` 和 `Allocatable` 中没有 `nvidia.com/gpu`。
+- 申请 `nvidia.com/gpu` 的 Pod 一直处于 `Pending`，提示 `0/N nodes are available`，并且**没有** `hami-scheduler` 发出的 `FilteringFailed` 事件。
+- 主机上 `nvidia-smi` 正常，但该节点对集群没有任何贡献。
+- 节点昨天还能用，今天就不再被选中。
+
+如果你的 Pod **确实**收到了 `hami-scheduler` 的 `FilteringFailed` 事件，说明注册已经成功，问题出在调度阶段。请参阅[排障手册](./troubleshooting.md)。
+
+## 注册是如何工作的
+
+注册不是一个动作，而是三条彼此独立的链路，每一条都可能单独出问题：
+
+```mermaid
+%% title: HAMi GPU 节点注册链路
+flowchart TB
+ A["节点上的 NVIDIA 驱动 + NVML"] --> B["hami-device-plugin"]
+ B -->|"链路 1：gRPC ListAndWatch"| C["kubelet"]
+ C -->|"节点状态"| D["Capacity/Allocatable 中的 nvidia.com/gpu"]
+ B -->|"链路 2：patch 节点注解"| E["hami.io/node-nvidia-register"]
+ D --> F["hami-scheduler 缓存"]
+ E --> F
+ F -->|"链路 3：握手"| G["hami.io/node-handshake"]
+```
+
+| 链路 | 写入方 | 携带内容 | 观察位置 |
+| --- | --- | --- | --- |
+| 设备数量 | Device Plugin 到 kubelet | 仅一个整数 | 节点 `Allocatable` 中的 `nvidia.com/gpu` |
+| 设备规格 | Device Plugin 到 API Server | UUID、显存、算力、型号、NUMA、健康状态 | `hami.io/node-nvidia-register` 节点注解 |
+| 存活握手 | 调度器到 API Server | 一个时间戳 | `hami.io/node-handshake` 节点注解 |
+
+数量和规格之所以走两条路，是因为 Device Plugin API 只能上报一个整数类型的资源。因此完全可能出现这种情况：节点上报了 `nvidia.com/gpu: 10`，但 HAMi 调度器仍然拒绝使用它，因为调度器真正读取的那个注解不存在。
+
+kubelet 上报的数量是**放大后**的数量：物理 GPU 数乘以 `devicePlugin.deviceSplitCount`（默认 `10`）。默认安装下一块物理卡显示为 `nvidia.com/gpu: 10`，而不是 `1`。参阅 [GPU 虚拟化](../core-concepts/gpu-virtualization.md)。
+
+## 第 1 步：定位出问题的链路
+
+在做任何改动之前，先对故障节点执行以下三项检查：
+
+```bash
+NODE=
+
+# 链路 1：kubelet 是否上报了该资源？
+kubectl get node $NODE -o jsonpath='{.status.allocatable.nvidia\.com/gpu}{"\n"}'
+
+# 链路 2：Device Plugin 是否写入了设备规格？
+kubectl get node $NODE -o jsonpath='{.metadata.annotations.hami\.io/node-nvidia-register}{"\n"}'
+
+# 链路 3：该节点上是否真的运行着 Device Plugin Pod？
+kubectl get pods -n kube-system \
+ -l app.kubernetes.io/component=hami-device-plugin \
+ -o wide --field-selector spec.nodeName=$NODE
+```
+
+根据结果决定接下来阅读的章节：
+
+| 结果 | 出问题的链路 | 跳转 |
+| ----------------------------------------- | ------------------ | ----------------- |
+| 节点上没有 Device Plugin Pod | 什么都没有在注册 | [情况 1](#case-1) |
+| Pod 存在但不是 `Running` | 插件无法启动 | [情况 2](#case-2) |
+| Pod 为 `Running` 但 `nvidia.com/gpu` 为空 | 插件到 kubelet | [情况 3](#case-3) |
+| `nvidia.com/gpu` 有值但注解为空 | 插件到 API Server | [情况 4](#case-4) |
+| 两者都有值但 Pod 仍 `Pending` | 调度器看不到该节点 | [情况 5](#case-5) |
+
+## 情况 1：节点上没有 Device Plugin Pod {#case-1}
+
+HAMi NVIDIA Device Plugin DaemonSet 带有节点选择器，Chart 默认值为：
+
+```yaml
+devicePlugin:
+ nvidiaNodeSelector:
+ gpu: "on"
+```
+
+没有该标签的节点永远不会被调度 Device Plugin Pod，三条链路也就都不会启动。这是 GPU 节点毫无贡献的最常见单一原因。
+
+### 检查
+
+```bash
+kubectl get node $NODE --show-labels | tr ',' '\n' | grep gpu
+kubectl get daemonset -n kube-system hami-device-plugin \
+ -o jsonpath='{.spec.template.spec.nodeSelector}{"\n"}'
+```
+
+### 修复
+
+```bash
+kubectl label node $NODE gpu=on --overwrite
+```
+
+然后等待 DaemonSet 下发 Pod：
+
+```bash
+kubectl rollout status daemonset/hami-device-plugin -n kube-system
+```
+
+如果标签本来就正确，请检查节点上是否存在 DaemonSet 无法容忍的污点：
+
+```bash
+kubectl describe node $NODE | grep -A 3 Taints
+```
+
+必要时在 `devicePlugin.tolerations` 中补充相应条目。节点打标签的说明另见[前置条件](../installation/prerequisites.md)。
+
+## 情况 2：Device Plugin Pod 无法保持运行 {#case-2}
+
+```bash
+kubectl logs -n kube-system -l app.kubernetes.io/component=hami-device-plugin \
+ -c device-plugin --tail=100
+kubectl describe pod -n kube-system -l app.kubernetes.io/component=hami-device-plugin
+```
+
+### NVML 初始化失败
+
+```plaintext
+nvml Init err: ERROR_LIBRARY_NOT_FOUND
+```
+
+Device Plugin 在扫描设备时将任何 NVML 失败视为致命错误并退出，因此容器会进入 `CrashLoopBackOff`，而不是降级运行。`nvml get memory error`、`nvml get name error` 和 `nvml new device by index error` 同样走这条致命路径。
+
+这几乎总是意味着容器没有拿到驱动，而根因通常是节点上 `nvidia-container-runtime` 不是默认运行时：
+
+```bash
+containerd config dump | grep default_runtime_name
+```
+
+输出必须是 `nvidia`。如果不是，请按[前置条件](../installation/prerequisites.md)配置后重启容器运行时。在 GPU Operator 25.10 及以上版本中，默认运行时会刻意保持为 `runc`；这种情况需要改为设置 `devicePlugin.runtimeClassName=nvidia`，详见[排障手册](./troubleshooting.md#nvidia-toolkit-gpu-operator-25-10)。
+
+### 卡在 `Init` 阶段
+
+```plaintext
+Waiting for /run/nvidia/validations/toolkit-ready...
+```
+
+`toolkit-validation` 初始化容器会一直阻塞，直到 NVIDIA Container Toolkit 在 `devicePlugin.gpuOperatorToolkitReady.hostPath`（默认 `/run/nvidia/validations`）下写入 `toolkit-ready` 文件。该开关默认关闭，仅面向使用 GPU Operator 的集群。如果在没有 GPU Operator 的集群上启用了它，这个文件永远不会出现，初始化容器就会一直等待：
+
+```bash
+helm upgrade hami hami-charts/hami -n kube-system --reuse-values \
+ --set devicePlugin.gpuOperatorToolkitReady.enabled=false
+```
+
+### 节点名未解析
+
+Device Plugin 会 patch 由其 `NODE_NAME` 环境变量指定的节点，该变量由 Chart 从 `spec.nodeName` 注入。在早于 v2.3.10 的 Chart 中该变量名为 `NodeName`，镜像与 Chart 版本不匹配会导致插件无法识别自己所在的节点。请升级而不是手工修改：
+
+```bash
+helm upgrade hami hami-charts/hami -n kube-system --reuse-values
+```
+
+## 情况 3：资源始终不出现在 Allocatable 中 {#case-3}
+
+Pod 处于 `Running`，NVML 也正常，但 `nvidia.com/gpu` 不存在。说明 Device Plugin 向 API Server 完成了注册，却没有向 kubelet 注册。
+
+### 检查
+
+```bash
+kubectl logs -n kube-system -l app.kubernetes.io/component=hami-device-plugin \
+ -c device-plugin --tail=200 | grep -i -E "register|socket|kubelet"
+
+ls -l /var/lib/kubelet/device-plugins/ # 在节点上执行
+```
+
+插件通过该目录下的 `kubelet.sock` 注册。Chart 从 `devicePlugin.pluginPath` 挂载该目录，默认值为 `/var/lib/kubelet/device-plugins`。如果你的发行版改变了 kubelet 根目录，插件会把 socket 写到 kubelet 根本不读取的位置，注册就会悄无声息地一直失败。
+
+### 修复
+
+先在节点上确认真实路径，再让 Chart 指向它：
+
+```bash
+# 在节点上执行
+ps aux | grep kubelet | grep -o '\--root-dir=[^ ]*'
+
+helm upgrade hami hami-charts/hami -n kube-system --reuse-values \
+ --set devicePlugin.pluginPath=/device-plugins
+```
+
+重启 kubelet 也会强制所有 Device Plugin 重新注册，这是快速验证 socket 路径是否为根因的办法。
+
+## 情况 4：注册注解缺失或过期 {#case-4}
+
+这种情况最容易被误判为调度器的 Bug。kubelet 已经上报了 GPU，`kubectl describe node` 看起来一切正常，但 `hami-scheduler` 从不把 Pod 放到这个节点上。
+
+### 读取注解
+
+```bash
+kubectl get node $NODE \
+ -o jsonpath='{.metadata.annotations.hami\.io/node-nvidia-register}' | jq .
+```
+
+默认安装下一块 24 GiB 显卡的预期输出：
+
+```json
+[
+ {
+ "id": "GPU-fc28df76-54d2-c387-e52e-5f0a9495968c",
+ "count": 10,
+ "devmem": 24576,
+ "devcore": 100,
+ "type": "NVIDIA-NVIDIA L40S",
+ "mode": "hami-core",
+ "health": true
+ }
+]
+```
+
+| 字段 | 含义 | 来源 |
+| --------- | ----------------------- | ---------------------------------- |
+| `id` | GPU UUID | NVML |
+| `count` | 逻辑切分数量 | `devicePlugin.deviceSplitCount` |
+| `devmem` | 可调度显存（MiB） | 物理显存乘以 `deviceMemoryScaling` |
+| `devcore` | 可调度算力百分比 | `deviceCoreScaling` 乘以 100 |
+| `type` | 型号，带 `NVIDIA-` 前缀 | NVML |
+| `numa` | NUMA 节点 | sysfs |
+| `mode` | `hami-core` 或 `mig` | 插件运行模式 |
+| `health` | 设备健康状态 | Device Plugin 健康检查 |
+
+:::warning 零值字段会被省略
+
+该注解使用 `omitempty` 序列化，因此任何取值为零或 `false` 的字段都会直接消失，而不是显示出来。**不健康的 GPU 根本不会有 `health` 字段**，而不是显示为 `"health": false`。`"numa": 0` 和 `"index": 0` 同理。请把缺失的 `health` 字段理解为不健康，而不是健康。
+
+:::
+
+### 注解只在内容变化时才会重写
+
+Device Plugin 每 30 秒重新扫描一次设备，但会把新编码出的设备列表与上次写入的内容比较，完全相同时就跳过 patch：
+
+```plaintext
+Device info unchanged, skipping annotation update
+```
+
+这行日志在 `-v=3` 下才可见。在默认日志级别下，一次真正的更新会打印：
+
+```plaintext
+Updating node annotations with 1 device(s)
+```
+
+由此带来两个推论：注解上的时间戳很旧是正常现象，不能作为插件卡死的证据；反过来，手工删除该注解**不会**在 30 秒内被重新写回，因为插件内存中的缓存仍然与它认为已写入的内容一致。此时应重启 Pod，参见[强制重新注册](#force-a-re-registration)。
+
+### patch 被拒绝
+
+```plaintext
+patch node error nodes "gpu-node-1" is forbidden: User "system:serviceaccount:kube-system:hami-device-plugin" cannot patch resource "nodes"
+```
+
+说明该 ServiceAccount 失去了 patch 节点的权限，通常发生在升级不完整或手工改过 ClusterRole 之后：
+
+```bash
+kubectl auth can-i patch nodes \
+ --as=system:serviceaccount:kube-system:hami-device-plugin
+```
+
+重新安装或升级 Chart 即可恢复相关 RBAC 对象。
+
+## 情况 5：调度器看不到该节点 {#case-5}
+
+资源和注解都正确，但 Pod 依然落不下去。此时节点不在调度器的内存缓存中。
+
+### 检查调度器日志
+
+```bash
+kubectl logs -n kube-system deploy/hami-scheduler -c vgpu-scheduler-extender --tail=200
+```
+
+注册循环每 15 秒执行一次，节点事件和主节点切换时也会触发。把日志级别提高到 `-v=5` 才能看到逐节点的判定，否则这些信息是静默的：
+
+```plaintext
+Using label selector for list nodes
+Listed nodes
+Processing node
+Failed to get node devices
+```
+
+你的节点出现 `Failed to get node devices`，说明调度器读到了注解但拒绝了它。有三种可能：注解不存在、JSON 无法解码、解码后的列表为空。后两种在默认日志级别下也会打印：
+
+```plaintext
+failed to decode node devices
+no nvidia gpu device found
+```
+
+解码失败通常是手工编辑过注解导致的。
+
+### 原因 A：节点标签选择器排除了该节点
+
+可以通过 `scheduler.nodeLabelSelector` 把调度器限制在部分节点上。Chart 中该项默认被注释掉，因此未经修改的安装会列出所有节点。如果设置过，启动时会打印出该值：
+
+```bash
+kubectl logs -n kube-system deploy/hami-scheduler -c vgpu-scheduler-extender \
+ | grep "label selector"
+```
+
+任何不满足这些标签的节点都不会被注册，无论它多健康。
+
+### 原因 B：你查看的副本不是主节点
+
+只有主节点（leader）会执行注册，其余副本都会打印：
+
+```plaintext
+Scheduler is not leader yet, skipping ...
+```
+
+当 `hami-scheduler` 有多个副本时，从副本日志安静是正常的，不能说明循环已经停止。下结论之前先确认谁是主节点：
+
+```bash
+kubectl get lease -n kube-system | grep hami
+```
+
+## 关于握手注解
+
+`hami.io/node-handshake` 是由**调度器**维护的存活标记，而不是由 Device Plugin 维护。把它理解反了会浪费大量排查时间，因此这里明确说明其实际行为：
+
+- 当该注解不存在，或其值不含 `Requesting` 时，调度器会写入 `Requesting_<时间戳>` 并将节点视为健康。
+- 时间戳在 **60 秒**以内时，节点健康。
+- 超过 60 秒后，调度器会检查节点的可分配 `nvidia.com/gpu`。只要它仍大于零，节点就保持健康，不会触发任何清理。
+- 只有在握手过期**并且**可分配数量降为零时，调度器才会执行节点清理：把该节点的设备移出缓存，并删除握手注解。
+
+```bash
+kubectl get node $NODE -o jsonpath='{.metadata.annotations.hami\.io/node-handshake}{"\n"}'
+# Requesting_2026-08-15 09:12:44
+```
+
+两个实践结论：
+
+- **`Requesting_` 时间戳很旧不是故障**，这是正常稳态，它本身永远不会导致节点被移除，不要据此调参。
+- **节点离开调度器缓存的真正触发条件是可分配数量降为零**，也就是 Device Plugin 停止向 kubelet 上报。应该排查这一点，而不是握手。调度器会记录这次移除：
+
+ ```plaintext
+ Device is unhealthy, cleaning up node
+ ```
+
+:::note NVIDIA 使用不带后缀的键
+
+NVIDIA 的握手注解键是 `hami.io/node-handshake`。其他厂商使用带后缀的形式，例如 `hami.io/node-handshake-dcu` 或 `hami.io/node-handshake-xpu`。在 NVIDIA 节点上执行 `kubectl get node -o yaml | grep node-handshake-nvidia` 没有任何输出，这是预期行为。
+
+:::
+
+## 强制重新注册 {#force-a-re-registration}
+
+重启 Device Plugin Pod 会清空其内存中的设备缓存，并在下一次扫描时强制重新 patch 注解。对于注解缺失、被截断或被手工改坏的情况，这是正确的恢复手段：
+
+```bash
+kubectl delete pod -n kube-system \
+ -l app.kubernetes.io/component=hami-device-plugin \
+ --field-selector spec.nodeName=$NODE
+```
+
+确认这一轮已经完成：
+
+```bash
+kubectl logs -n kube-system -l app.kubernetes.io/component=hami-device-plugin \
+ -c device-plugin --tail=50 | grep "Updating node annotations"
+
+kubectl get node $NODE \
+ -o jsonpath='{.metadata.annotations.hami\.io/node-nvidia-register}' | jq length
+```
+
+调度器会在一个注册周期内重新纳管该节点，因此请等待约 15 秒后再用 Pod 验证。
+
+如果节点仍未注册，请在提交 Issue 前收集以下信息：
+
+```bash
+kubectl get node $NODE -o yaml > node.yaml
+kubectl logs -n kube-system -l app.kubernetes.io/component=hami-device-plugin \
+ -c device-plugin --tail=500 > device-plugin.log
+kubectl logs -n kube-system deploy/hami-scheduler \
+ -c vgpu-scheduler-extender --tail=500 > scheduler.log
+```
+
+## 验证环境
+
+本页描述的行为通过阅读 HAMi **v2.9.0** 与 `master` 分支的源码验证，具体为 `pkg/device-plugin/nvidiadevice/nvinternal/plugin/register.go`、`pkg/device/devices.go`、`pkg/device/nvidia/device.go`、`pkg/scheduler/scheduler.go` 和 `charts/hami/values.yaml`。文中引用的 Chart 默认值取自 `charts/hami` 中的 NVIDIA 配置。各类间隔、60 秒握手窗口以及日志文本都可能随版本变化，依赖具体数值前请先核对你所运行版本的源码。
+
+## 相关页面
+
+- [排障手册](./troubleshooting.md)：运行时与 GPU Operator 相关问题
+- [验证 HAMi](../get-started/verify-hami.md)：安装后的端到端检查
+- [协议设计](../developers/protocol.md)：注册协议本身
+- [GPU 虚拟化](../core-concepts/gpu-virtualization.md)：该注解如何参与调度
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/developers/protocol.md b/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/developers/protocol.md
index 7ee6c7c69..077b3248f 100644
--- a/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/developers/protocol.md
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/developers/protocol.md
@@ -18,6 +18,26 @@ hami.io/node-handshake: Requesting_2024.05.14 07:07:33
hami.io/node-nvidia-register: "GPU-00552014-5c87-89ac-b1a6-7b53aa24b0ec,10,32768,100,NVIDIA-Tesla V100-PCIE-32GB,0,true:GPU-0fc3eda5-e98b-a25b-5b0d-cf5c855d1448,10,32768,100,NVIDIA-Tesla V100-PCIE-32GB,0,true:"
```
+:::note 编码方式因设备类型而异
+
+上面这种冒号分隔的形式是早期编码，目前仍用于通过 `DecodeNodeDevices` 解码的设备类型（例如 DCU、天数智芯）。NVIDIA 以及其他若干设备类型改为使用 **JSON 数组**编码同样的字段，每个设备一个对象，通过 `UnMarshalNodeDevices` 解码。这些字段使用 `omitempty` 序列化，因此取值为零或 `false` 的字段会被省略而不会写出。JSON 示例参见 [GPU 虚拟化](../core-concepts/gpu-virtualization.md)。
+
+注解键也并非完全统一：NVIDIA 使用 `hami.io/node-nvidia-register` 和 `hami.io/node-handshake`（不带设备类型后缀），而昆仑芯使用 `hami.io/node-register-xpu`。
+
+:::
+
+### 握手与节点存活
+
+时间戳由调度器负责写入：当握手注解不存在，或其值不含 `Requesting` 时，调度器会用自己的时钟写入 `Requesting_<时间戳>`。调度器的注册循环每 15 秒执行一次，节点事件和主节点切换时也会触发，且只有当选为 leader 的副本才会执行注册。
+
+握手时间戳超过 **60 秒**即视为过期。但过期本身不会移除节点：调度器还要求该节点的可分配设备数量降为零，才会执行节点清理，把设备移出调度器缓存并删除握手注解。因此，只要 device-plugin 仍在向 kubelet 上报，节点即使握手过期也依然可用。
+
+:::note NVIDIA device-plugin 不写握手注解
+
+本协议中 `Reported_` 一侧由实现了该协议的 device-plugin 写入。内置的 NVIDIA device-plugin 并未实现，因此在 NVIDIA 节点上该注解通常一直停留在 `Requesting_<时间戳>` 并超过 60 秒。这是预期稳态而非故障：对 NVIDIA 而言，真正的存活信号是可分配设备数量。参见 [GPU 节点未注册](../troubleshooting/node-registration.md)。
+
+:::
+
### 调度决策
kube-scheduler 在 `bind` 过程中调用 device-plugin 挂载设备，但仅向 device-plugin 提供设备的 `UUID`。因此，在设备共享的场景中，device-plugin 无法获取任务请求的相应设备规格，如 `设备显存` 和 `计算核心`。
diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/troubleshooting/node-registration.md b/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/troubleshooting/node-registration.md
new file mode 100644
index 000000000..bd40bf9ad
--- /dev/null
+++ b/i18n/zh/docusaurus-plugin-content-docs/version-v2.9.0/troubleshooting/node-registration.md
@@ -0,0 +1,385 @@
+---
+title: GPU 节点未注册
+sidebar_label: 节点注册
+translated: true
+---
+
+在 HAMi 调度任何任务之前，GPU 节点必须先完成注册。本页覆盖发生在调度**之前**的故障：节点没有上报任何 GPU、HAMi 调度器不知道该节点存在，或者一个原本正常的节点悄悄从集群的 GPU 容量中消失。
+
+典型现象：
+
+- `kubectl describe node` 的 `Capacity` 和 `Allocatable` 中没有 `nvidia.com/gpu`。
+- 申请 `nvidia.com/gpu` 的 Pod 一直处于 `Pending`，提示 `0/N nodes are available`，并且**没有** `hami-scheduler` 发出的 `FilteringFailed` 事件。
+- 主机上 `nvidia-smi` 正常，但该节点对集群没有任何贡献。
+- 节点昨天还能用，今天就不再被选中。
+
+如果你的 Pod **确实**收到了 `hami-scheduler` 的 `FilteringFailed` 事件，说明注册已经成功，问题出在调度阶段。请参阅[排障手册](./troubleshooting.md)。
+
+## 注册是如何工作的
+
+注册不是一个动作，而是三条彼此独立的链路，每一条都可能单独出问题：
+
+```mermaid
+%% title: HAMi GPU 节点注册链路
+flowchart TB
+ A["节点上的 NVIDIA 驱动 + NVML"] --> B["hami-device-plugin"]
+ B -->|"链路 1：gRPC ListAndWatch"| C["kubelet"]
+ C -->|"节点状态"| D["Capacity/Allocatable 中的 nvidia.com/gpu"]
+ B -->|"链路 2：patch 节点注解"| E["hami.io/node-nvidia-register"]
+ D --> F["hami-scheduler 缓存"]
+ E --> F
+ F -->|"链路 3：握手"| G["hami.io/node-handshake"]
+```
+
+| 链路 | 写入方 | 携带内容 | 观察位置 |
+| --- | --- | --- | --- |
+| 设备数量 | Device Plugin 到 kubelet | 仅一个整数 | 节点 `Allocatable` 中的 `nvidia.com/gpu` |
+| 设备规格 | Device Plugin 到 API Server | UUID、显存、算力、型号、NUMA、健康状态 | `hami.io/node-nvidia-register` 节点注解 |
+| 存活握手 | 调度器到 API Server | 一个时间戳 | `hami.io/node-handshake` 节点注解 |
+
+数量和规格之所以走两条路，是因为 Device Plugin API 只能上报一个整数类型的资源。因此完全可能出现这种情况：节点上报了 `nvidia.com/gpu: 10`，但 HAMi 调度器仍然拒绝使用它，因为调度器真正读取的那个注解不存在。
+
+kubelet 上报的数量是**放大后**的数量：物理 GPU 数乘以 `devicePlugin.deviceSplitCount`（默认 `10`）。默认安装下一块物理卡显示为 `nvidia.com/gpu: 10`，而不是 `1`。参阅 [GPU 虚拟化](../core-concepts/gpu-virtualization.md)。
+
+## 第 1 步：定位出问题的链路
+
+在做任何改动之前，先对故障节点执行以下三项检查：
+
+```bash
+NODE=
+
+# 链路 1：kubelet 是否上报了该资源？
+kubectl get node $NODE -o jsonpath='{.status.allocatable.nvidia\.com/gpu}{"\n"}'
+
+# 链路 2：Device Plugin 是否写入了设备规格？
+kubectl get node $NODE -o jsonpath='{.metadata.annotations.hami\.io/node-nvidia-register}{"\n"}'
+
+# 链路 3：该节点上是否真的运行着 Device Plugin Pod？
+kubectl get pods -n kube-system \
+ -l app.kubernetes.io/component=hami-device-plugin \
+ -o wide --field-selector spec.nodeName=$NODE
+```
+
+根据结果决定接下来阅读的章节：
+
+| 结果 | 出问题的链路 | 跳转 |
+| ----------------------------------------- | ------------------ | ----------------- |
+| 节点上没有 Device Plugin Pod | 什么都没有在注册 | [情况 1](#case-1) |
+| Pod 存在但不是 `Running` | 插件无法启动 | [情况 2](#case-2) |
+| Pod 为 `Running` 但 `nvidia.com/gpu` 为空 | 插件到 kubelet | [情况 3](#case-3) |
+| `nvidia.com/gpu` 有值但注解为空 | 插件到 API Server | [情况 4](#case-4) |
+| 两者都有值但 Pod 仍 `Pending` | 调度器看不到该节点 | [情况 5](#case-5) |
+
+## 情况 1：节点上没有 Device Plugin Pod {#case-1}
+
+HAMi NVIDIA Device Plugin DaemonSet 带有节点选择器，Chart 默认值为：
+
+```yaml
+devicePlugin:
+ nvidiaNodeSelector:
+ gpu: "on"
+```
+
+没有该标签的节点永远不会被调度 Device Plugin Pod，三条链路也就都不会启动。这是 GPU 节点毫无贡献的最常见单一原因。
+
+### 检查
+
+```bash
+kubectl get node $NODE --show-labels | tr ',' '\n' | grep gpu
+kubectl get daemonset -n kube-system hami-device-plugin \
+ -o jsonpath='{.spec.template.spec.nodeSelector}{"\n"}'
+```
+
+### 修复
+
+```bash
+kubectl label node $NODE gpu=on --overwrite
+```
+
+然后等待 DaemonSet 下发 Pod：
+
+```bash
+kubectl rollout status daemonset/hami-device-plugin -n kube-system
+```
+
+如果标签本来就正确，请检查节点上是否存在 DaemonSet 无法容忍的污点：
+
+```bash
+kubectl describe node $NODE | grep -A 3 Taints
+```
+
+必要时在 `devicePlugin.tolerations` 中补充相应条目。节点打标签的说明另见[前置条件](../installation/prerequisites.md)。
+
+## 情况 2：Device Plugin Pod 无法保持运行 {#case-2}
+
+```bash
+kubectl logs -n kube-system -l app.kubernetes.io/component=hami-device-plugin \
+ -c device-plugin --tail=100
+kubectl describe pod -n kube-system -l app.kubernetes.io/component=hami-device-plugin
+```
+
+### NVML 初始化失败
+
+```plaintext
+nvml Init err: ERROR_LIBRARY_NOT_FOUND
+```
+
+Device Plugin 在扫描设备时将任何 NVML 失败视为致命错误并退出，因此容器会进入 `CrashLoopBackOff`，而不是降级运行。`nvml get memory error`、`nvml get name error` 和 `nvml new device by index error` 同样走这条致命路径。
+
+这几乎总是意味着容器没有拿到驱动，而根因通常是节点上 `nvidia-container-runtime` 不是默认运行时：
+
+```bash
+containerd config dump | grep default_runtime_name
+```
+
+输出必须是 `nvidia`。如果不是，请按[前置条件](../installation/prerequisites.md)配置后重启容器运行时。在 GPU Operator 25.10 及以上版本中，默认运行时会刻意保持为 `runc`；这种情况需要改为设置 `devicePlugin.runtimeClassName=nvidia`，详见[排障手册](./troubleshooting.md#nvidia-toolkit-gpu-operator-25-10)。
+
+### 卡在 `Init` 阶段
+
+```plaintext
+Waiting for /run/nvidia/validations/toolkit-ready...
+```
+
+`toolkit-validation` 初始化容器会一直阻塞，直到 NVIDIA Container Toolkit 在 `devicePlugin.gpuOperatorToolkitReady.hostPath`（默认 `/run/nvidia/validations`）下写入 `toolkit-ready` 文件。该开关默认关闭，仅面向使用 GPU Operator 的集群。如果在没有 GPU Operator 的集群上启用了它，这个文件永远不会出现，初始化容器就会一直等待：
+
+```bash
+helm upgrade hami hami-charts/hami -n kube-system --reuse-values \
+ --set devicePlugin.gpuOperatorToolkitReady.enabled=false
+```
+
+### 节点名未解析
+
+Device Plugin 会 patch 由其 `NODE_NAME` 环境变量指定的节点，该变量由 Chart 从 `spec.nodeName` 注入。在早于 v2.3.10 的 Chart 中该变量名为 `NodeName`，镜像与 Chart 版本不匹配会导致插件无法识别自己所在的节点。请升级而不是手工修改：
+
+```bash
+helm upgrade hami hami-charts/hami -n kube-system --reuse-values
+```
+
+## 情况 3：资源始终不出现在 Allocatable 中 {#case-3}
+
+Pod 处于 `Running`，NVML 也正常，但 `nvidia.com/gpu` 不存在。说明 Device Plugin 向 API Server 完成了注册，却没有向 kubelet 注册。
+
+### 检查
+
+```bash
+kubectl logs -n kube-system -l app.kubernetes.io/component=hami-device-plugin \
+ -c device-plugin --tail=200 | grep -i -E "register|socket|kubelet"
+
+ls -l /var/lib/kubelet/device-plugins/ # 在节点上执行
+```
+
+插件通过该目录下的 `kubelet.sock` 注册。Chart 从 `devicePlugin.pluginPath` 挂载该目录，默认值为 `/var/lib/kubelet/device-plugins`。如果你的发行版改变了 kubelet 根目录，插件会把 socket 写到 kubelet 根本不读取的位置，注册就会悄无声息地一直失败。
+
+### 修复
+
+先在节点上确认真实路径，再让 Chart 指向它：
+
+```bash
+# 在节点上执行
+ps aux | grep kubelet | grep -o '\--root-dir=[^ ]*'
+
+helm upgrade hami hami-charts/hami -n kube-system --reuse-values \
+ --set devicePlugin.pluginPath=/device-plugins
+```
+
+重启 kubelet 也会强制所有 Device Plugin 重新注册，这是快速验证 socket 路径是否为根因的办法。
+
+## 情况 4：注册注解缺失或过期 {#case-4}
+
+这种情况最容易被误判为调度器的 Bug。kubelet 已经上报了 GPU，`kubectl describe node` 看起来一切正常，但 `hami-scheduler` 从不把 Pod 放到这个节点上。
+
+### 读取注解
+
+```bash
+kubectl get node $NODE \
+ -o jsonpath='{.metadata.annotations.hami\.io/node-nvidia-register}' | jq .
+```
+
+默认安装下一块 24 GiB 显卡的预期输出：
+
+```json
+[
+ {
+ "id": "GPU-fc28df76-54d2-c387-e52e-5f0a9495968c",
+ "count": 10,
+ "devmem": 24576,
+ "devcore": 100,
+ "type": "NVIDIA-NVIDIA L40S",
+ "mode": "hami-core",
+ "health": true
+ }
+]
+```
+
+| 字段 | 含义 | 来源 |
+| --------- | ----------------------- | ---------------------------------- |
+| `id` | GPU UUID | NVML |
+| `count` | 逻辑切分数量 | `devicePlugin.deviceSplitCount` |
+| `devmem` | 可调度显存（MiB） | 物理显存乘以 `deviceMemoryScaling` |
+| `devcore` | 可调度算力百分比 | `deviceCoreScaling` 乘以 100 |
+| `type` | 型号，带 `NVIDIA-` 前缀 | NVML |
+| `numa` | NUMA 节点 | sysfs |
+| `mode` | `hami-core` 或 `mig` | 插件运行模式 |
+| `health` | 设备健康状态 | Device Plugin 健康检查 |
+
+:::warning 零值字段会被省略
+
+该注解使用 `omitempty` 序列化，因此任何取值为零或 `false` 的字段都会直接消失，而不是显示出来。**不健康的 GPU 根本不会有 `health` 字段**，而不是显示为 `"health": false`。`"numa": 0` 和 `"index": 0` 同理。请把缺失的 `health` 字段理解为不健康，而不是健康。
+
+:::
+
+### 注解只在内容变化时才会重写
+
+Device Plugin 每 30 秒重新扫描一次设备，但会把新编码出的设备列表与上次写入的内容比较，完全相同时就跳过 patch：
+
+```plaintext
+Device info unchanged, skipping annotation update
+```
+
+这行日志在 `-v=3` 下才可见。在默认日志级别下，一次真正的更新会打印：
+
+```plaintext
+Updating node annotations with 1 device(s)
+```
+
+由此带来两个推论：注解上的时间戳很旧是正常现象，不能作为插件卡死的证据；反过来，手工删除该注解**不会**在 30 秒内被重新写回，因为插件内存中的缓存仍然与它认为已写入的内容一致。此时应重启 Pod，参见[强制重新注册](#force-a-re-registration)。
+
+### patch 被拒绝
+
+```plaintext
+patch node error nodes "gpu-node-1" is forbidden: User "system:serviceaccount:kube-system:hami-device-plugin" cannot patch resource "nodes"
+```
+
+说明该 ServiceAccount 失去了 patch 节点的权限，通常发生在升级不完整或手工改过 ClusterRole 之后：
+
+```bash
+kubectl auth can-i patch nodes \
+ --as=system:serviceaccount:kube-system:hami-device-plugin
+```
+
+重新安装或升级 Chart 即可恢复相关 RBAC 对象。
+
+## 情况 5：调度器看不到该节点 {#case-5}
+
+资源和注解都正确，但 Pod 依然落不下去。此时节点不在调度器的内存缓存中。
+
+### 检查调度器日志
+
+```bash
+kubectl logs -n kube-system deploy/hami-scheduler -c vgpu-scheduler-extender --tail=200
+```
+
+注册循环每 15 秒执行一次，节点事件和主节点切换时也会触发。把日志级别提高到 `-v=5` 才能看到逐节点的判定，否则这些信息是静默的：
+
+```plaintext
+Using label selector for list nodes
+Listed nodes
+Processing node
+Failed to get node devices
+```
+
+你的节点出现 `Failed to get node devices`，说明调度器读到了注解但拒绝了它。有三种可能：注解不存在、JSON 无法解码、解码后的列表为空。后两种在默认日志级别下也会打印：
+
+```plaintext
+failed to decode node devices
+no nvidia gpu device found
+```
+
+解码失败通常是手工编辑过注解导致的。
+
+### 原因 A：节点标签选择器排除了该节点
+
+可以通过 `scheduler.nodeLabelSelector` 把调度器限制在部分节点上。Chart 中该项默认被注释掉，因此未经修改的安装会列出所有节点。如果设置过，启动时会打印出该值：
+
+```bash
+kubectl logs -n kube-system deploy/hami-scheduler -c vgpu-scheduler-extender \
+ | grep "label selector"
+```
+
+任何不满足这些标签的节点都不会被注册，无论它多健康。
+
+### 原因 B：你查看的副本不是主节点
+
+只有主节点（leader）会执行注册，其余副本都会打印：
+
+```plaintext
+Scheduler is not leader yet, skipping ...
+```
+
+当 `hami-scheduler` 有多个副本时，从副本日志安静是正常的，不能说明循环已经停止。下结论之前先确认谁是主节点：
+
+```bash
+kubectl get lease -n kube-system | grep hami
+```
+
+## 关于握手注解
+
+`hami.io/node-handshake` 是由**调度器**维护的存活标记，而不是由 Device Plugin 维护。把它理解反了会浪费大量排查时间，因此这里明确说明其实际行为：
+
+- 当该注解不存在，或其值不含 `Requesting` 时，调度器会写入 `Requesting_<时间戳>` 并将节点视为健康。
+- 时间戳在 **60 秒**以内时，节点健康。
+- 超过 60 秒后，调度器会检查节点的可分配 `nvidia.com/gpu`。只要它仍大于零，节点就保持健康，不会触发任何清理。
+- 只有在握手过期**并且**可分配数量降为零时，调度器才会执行节点清理：把该节点的设备移出缓存，并删除握手注解。
+
+```bash
+kubectl get node $NODE -o jsonpath='{.metadata.annotations.hami\.io/node-handshake}{"\n"}'
+# Requesting_2026-08-15 09:12:44
+```
+
+两个实践结论：
+
+- **`Requesting_` 时间戳很旧不是故障**，这是正常稳态，它本身永远不会导致节点被移除，不要据此调参。
+- **节点离开调度器缓存的真正触发条件是可分配数量降为零**，也就是 Device Plugin 停止向 kubelet 上报。应该排查这一点，而不是握手。调度器会记录这次移除：
+
+ ```plaintext
+ Device is unhealthy, cleaning up node
+ ```
+
+:::note NVIDIA 使用不带后缀的键
+
+NVIDIA 的握手注解键是 `hami.io/node-handshake`。其他厂商使用带后缀的形式，例如 `hami.io/node-handshake-dcu` 或 `hami.io/node-handshake-xpu`。在 NVIDIA 节点上执行 `kubectl get node -o yaml | grep node-handshake-nvidia` 没有任何输出，这是预期行为。
+
+:::
+
+## 强制重新注册 {#force-a-re-registration}
+
+重启 Device Plugin Pod 会清空其内存中的设备缓存，并在下一次扫描时强制重新 patch 注解。对于注解缺失、被截断或被手工改坏的情况，这是正确的恢复手段：
+
+```bash
+kubectl delete pod -n kube-system \
+ -l app.kubernetes.io/component=hami-device-plugin \
+ --field-selector spec.nodeName=$NODE
+```
+
+确认这一轮已经完成：
+
+```bash
+kubectl logs -n kube-system -l app.kubernetes.io/component=hami-device-plugin \
+ -c device-plugin --tail=50 | grep "Updating node annotations"
+
+kubectl get node $NODE \
+ -o jsonpath='{.metadata.annotations.hami\.io/node-nvidia-register}' | jq length
+```
+
+调度器会在一个注册周期内重新纳管该节点，因此请等待约 15 秒后再用 Pod 验证。
+
+如果节点仍未注册，请在提交 Issue 前收集以下信息：
+
+```bash
+kubectl get node $NODE -o yaml > node.yaml
+kubectl logs -n kube-system -l app.kubernetes.io/component=hami-device-plugin \
+ -c device-plugin --tail=500 > device-plugin.log
+kubectl logs -n kube-system deploy/hami-scheduler \
+ -c vgpu-scheduler-extender --tail=500 > scheduler.log
+```
+
+## 验证环境
+
+本页描述的行为通过阅读 HAMi **v2.9.0** 与 `master` 分支的源码验证，具体为 `pkg/device-plugin/nvidiadevice/nvinternal/plugin/register.go`、`pkg/device/devices.go`、`pkg/device/nvidia/device.go`、`pkg/scheduler/scheduler.go` 和 `charts/hami/values.yaml`。文中引用的 Chart 默认值取自 `charts/hami` 中的 NVIDIA 配置。各类间隔、60 秒握手窗口以及日志文本都可能随版本变化，依赖具体数值前请先核对你所运行版本的源码。
+
+## 相关页面
+
+- [排障手册](./troubleshooting.md)：运行时与 GPU Operator 相关问题
+- [验证 HAMi](../get-started/verify-hami.md)：安装后的端到端检查
+- [协议设计](../developers/protocol.md)：注册协议本身
+- [GPU 虚拟化](../core-concepts/gpu-virtualization.md)：该注解如何参与调度
diff --git a/sidebars.js b/sidebars.js
index 3487c0e9f..340814633 100644
--- a/sidebars.js
+++ b/sidebars.js
@@ -422,6 +422,10 @@ module.exports = {
type: "doc",
id: "troubleshooting/troubleshooting",
},
+ {
+ type: "doc",
+ id: "troubleshooting/node-registration",
+ },
{
type: "doc",
id: "faq/faq",
diff --git a/versioned_docs/version-v2.9.0/developers/protocol.md b/versioned_docs/version-v2.9.0/developers/protocol.md
index 36c77acc7..5b22944c5 100644
--- a/versioned_docs/version-v2.9.0/developers/protocol.md
+++ b/versioned_docs/version-v2.9.0/developers/protocol.md
@@ -6,7 +6,7 @@ title: Protocol design
-HAMi needs to know the spec of each AI device in the cluster to schedule properly. During device registration, device-plugin needs to keep patching the spec of each device into node annotations every 30 seconds, in the format of the following:
+HAMi needs to know the spec of each AI device in the cluster to schedule properly. During device registration, device-plugin rescans its devices every 30 seconds and patches the spec of each device into node annotations, in the format of the following:
```text
hami.io/node-handshake-\{device-type\}: Reported_\{device_node_current_timestamp\}
@@ -19,10 +19,18 @@ The definition of each device is in the following format:
\{Device UUID\},\{device split count\},\{device memory limit\},\{device core limit\},\{device type\},\{device numa\},\{healthy\}
```
+:::note Encoding differs by vendor
+
+The colon-separated form above is the legacy encoding, still used by device types decoded with `DecodeNodeDevices` (for example DCU and Iluvatar). NVIDIA and several other device types encode the same fields as a **JSON array** instead, one object per device, decoded with `UnMarshalNodeDevices`. Fields are serialized with `omitempty`, so zero and `false` values are absent rather than written out. See [GPU Virtualization](../core-concepts/gpu-virtualization.md) for a JSON example.
+
+The annotation key is also not fully uniform: NVIDIA uses `hami.io/node-nvidia-register` and `hami.io/node-handshake` (no device-type suffix), while Kunlun uses `hami.io/node-register-xpu`.
+
+:::
+
An example is shown below:
```text
-hami.io/node-handshake-nvidia: Reported 2024-01-23 04:30:04.434037031 +0000 UTC m=+1104711.777756895
+hami.io/node-handshake: Requesting_2024-01-23 04:30:04.434037031 +0000 UTC m=+1104711.777756895
hami.io/node-handshake-mlu: Requesting_2024.01.10 04:06:57
hami.io/node-mlu-register: MLU-45013011-2257-0000-0000-000000000000,10,23308,0,MLU-MLU370-X4,0,false:MLU-54043011-2257-0000-0000-000000000000,10,23308,0,
hami.io/node-nvidia-register: GPU-00552014-5c87-89ac-b1a6-7b53aa24b0ec,10,32768,100,NVIDIA-Tesla V100-PCIE-32GB,0,true:GPU-0fc3eda5-e98b-a25b-5b0d-cf5c855d1448,10,32768,100,NVIDIA-Tesla V100-PCIE-32GB,0,true:
@@ -31,10 +39,18 @@ hami.io/node-nvidia-register: GPU-00552014-5c87-89ac-b1a6-7b53aa24b0ec,10,32768,
In this example, this node has two different AI devices, 2 Nvidia-V100 GPUs, and 2 Cambricon 370-X4 MLUs
-A device node may become unavailable due to hardware or network failure. If a node hasn't registered in the last 5 minutes, the scheduler marks it as 'unavailable'.
-
-Since system clock on scheduler node and 'device' node may not align properly, scheduler node will patch the following device node annotations every 30s
+A device node may become unavailable due to hardware or network failure. Since the system clock on the scheduler node and on the device node may not align properly, the scheduler owns the timestamp. Whenever the handshake annotation is absent or does not contain `Requesting`, the scheduler stamps it with its own clock.
```text
hami.io/node-handshake-\{device-type\}: Requesting_{scheduler_node_current_timestamp}
```
+
+The scheduler's registration loop runs every 15 seconds, and also on node events and on leader election changes. Only the elected leader performs registration.
+
+A handshake is treated as expired once its timestamp is more than **60 seconds** old. Expiry alone does not remove the node. The scheduler additionally requires the node's allocatable device count to have dropped to zero before it runs node cleanup, which removes the node's devices from the scheduler cache and deletes the handshake annotation. A node whose device-plugin is still reporting to kubelet therefore stays available even with an expired handshake.
+
+:::note The NVIDIA device-plugin does not write the `Reported_` handshake
+
+The `Reported_` side of this protocol is written by device-plugins that implement it. The in-tree NVIDIA device-plugin does not, so on an NVIDIA node the annotation normally stays at `Requesting_` and ages past 60 seconds. That is the expected steady state, not a fault: the effective liveness signal for NVIDIA is the allocatable device count. See [GPU Nodes Not Registering](../troubleshooting/node-registration.md).
+
+:::
diff --git a/versioned_docs/version-v2.9.0/troubleshooting/node-registration.md b/versioned_docs/version-v2.9.0/troubleshooting/node-registration.md
new file mode 100644
index 000000000..f3890620d
--- /dev/null
+++ b/versioned_docs/version-v2.9.0/troubleshooting/node-registration.md
@@ -0,0 +1,384 @@
+---
+title: GPU Nodes Not Registering
+sidebar_label: Node Registration
+---
+
+Before HAMi can schedule anything, a GPU node has to complete registration. This page covers the failures that happen _before_ scheduling: the node advertises no GPUs, the HAMi scheduler does not know the node exists, or a node that used to work silently drops out of the cluster's GPU capacity.
+
+Typical symptoms:
+
+- `kubectl describe node` shows no `nvidia.com/gpu` under `Capacity` and `Allocatable`.
+- A Pod requesting `nvidia.com/gpu` stays `Pending` with `0/N nodes are available`, and **no** `FilteringFailed` event from `hami-scheduler`.
+- `nvidia-smi` works on the host, but the node still contributes nothing to the cluster.
+- A node worked yesterday and stopped being selected today.
+
+If your Pod _does_ get a `FilteringFailed` event from `hami-scheduler`, registration already succeeded and the problem is scheduling instead. See [Troubleshooting](./troubleshooting.md).
+
+## How registration works
+
+Registration is not one action. It is three independent channels, and each can break on its own:
+
+```mermaid
+%% title: HAMi GPU node registration path
+flowchart TB
+ A["NVIDIA driver + NVML on the node"] --> B["hami-device-plugin"]
+ B -->|"channel 1: gRPC ListAndWatch"| C["kubelet"]
+ C -->|"node status"| D["nvidia.com/gpu in Capacity/Allocatable"]
+ B -->|"channel 2: patch node annotation"| E["hami.io/node-nvidia-register"]
+ D --> F["hami-scheduler cache"]
+ E --> F
+ F -->|"channel 3: handshake"| G["hami.io/node-handshake"]
+```
+
+| Channel | Written by | Carries | Where to observe it |
+| --- | --- | --- | --- |
+| Device count | Device Plugin to kubelet | An integer count only | `nvidia.com/gpu` in node `Allocatable` |
+| Device specification | Device Plugin to the API server | UUID, memory, compute, model, NUMA, health | `hami.io/node-nvidia-register` node annotation |
+| Liveness handshake | Scheduler to the API server | A timestamp | `hami.io/node-handshake` node annotation |
+
+The count and the specification travel separately because the Device Plugin API can only report a single integer resource. A node can therefore advertise `nvidia.com/gpu: 10` while the HAMi scheduler still refuses to use it, because the annotation the scheduler actually reads is missing.
+
+The count kubelet advertises is the **inflated** count: physical GPUs multiplied by `devicePlugin.deviceSplitCount` (default `10`). One physical card on a default install shows as `nvidia.com/gpu: 10`, not `1`. See [GPU Virtualization](../core-concepts/gpu-virtualization.md).
+
+## Step 1: Find the broken channel
+
+Run all three checks against the affected node before changing anything:
+
+```bash
+NODE=
+
+# Channel 1: does kubelet advertise the resource?
+kubectl get node $NODE -o jsonpath='{.status.allocatable.nvidia\.com/gpu}{"\n"}'
+
+# Channel 2: did the Device Plugin write the device specification?
+kubectl get node $NODE -o jsonpath='{.metadata.annotations.hami\.io/node-nvidia-register}{"\n"}'
+
+# Channel 3: is a Device Plugin Pod actually running there?
+kubectl get pods -n kube-system \
+ -l app.kubernetes.io/component=hami-device-plugin \
+ -o wide --field-selector spec.nodeName=$NODE
+```
+
+Match the result to the section to read next:
+
+| Result | Broken channel | Go to |
+| -------------------------------------- | ------------------------------- | ----------------- |
+| No Device Plugin Pod on the node | Nothing is registering | [Case 1](#case-1) |
+| Pod exists but is not `Running` | Plugin cannot start | [Case 2](#case-2) |
+| Empty `nvidia.com/gpu`, Pod `Running` | Plugin to kubelet | [Case 3](#case-3) |
+| `nvidia.com/gpu` set, annotation empty | Plugin to API server | [Case 4](#case-4) |
+| Both set, Pod still `Pending` | Scheduler does not see the node | [Case 5](#case-5) |
+
+## Case 1: no Device Plugin Pod on the node {#case-1}
+
+The HAMi NVIDIA Device Plugin DaemonSet carries a node selector. Its chart default is:
+
+```yaml
+devicePlugin:
+ nvidiaNodeSelector:
+ gpu: "on"
+```
+
+A node without that label never receives a Device Plugin Pod, so none of the three channels start. This is the single most common cause of a GPU node contributing nothing.
+
+### Check
+
+```bash
+kubectl get node $NODE --show-labels | tr ',' '\n' | grep gpu
+kubectl get daemonset -n kube-system hami-device-plugin \
+ -o jsonpath='{.spec.template.spec.nodeSelector}{"\n"}'
+```
+
+### Fix
+
+```bash
+kubectl label node $NODE gpu=on --overwrite
+```
+
+Then wait for the DaemonSet to place a Pod:
+
+```bash
+kubectl rollout status daemonset/hami-device-plugin -n kube-system
+```
+
+If the label is already present and correct, check that the node is not tainted in a way the DaemonSet does not tolerate:
+
+```bash
+kubectl describe node $NODE | grep -A 3 Taints
+```
+
+Add matching entries under `devicePlugin.tolerations` if needed. Node labelling is also covered in [Prerequisites](../installation/prerequisites.md).
+
+## Case 2: the Device Plugin Pod does not stay running {#case-2}
+
+```bash
+kubectl logs -n kube-system -l app.kubernetes.io/component=hami-device-plugin \
+ -c device-plugin --tail=100
+kubectl describe pod -n kube-system -l app.kubernetes.io/component=hami-device-plugin
+```
+
+### NVML initialization failure
+
+```plaintext
+nvml Init err: ERROR_LIBRARY_NOT_FOUND
+```
+
+The Device Plugin treats every NVML failure during a device scan as fatal and exits, so the container ends up in `CrashLoopBackOff` rather than running degraded. The same fatal path is taken for `nvml get memory error`, `nvml get name error`, and `nvml new device by index error`.
+
+This almost always means the container did not receive the driver, which in turn means `nvidia-container-runtime` is not the default runtime on the node:
+
+```bash
+containerd config dump | grep default_runtime_name
+```
+
+The output must be `nvidia`. If it is not, follow [Prerequisites](../installation/prerequisites.md), then restart the container runtime. On GPU Operator 25.10 and later the default runtime deliberately stays `runc`; that case needs `devicePlugin.runtimeClassName=nvidia` instead, and is covered in [Troubleshooting](./troubleshooting.md#nvidia-toolkit-gpu-operator-25-10).
+
+### Stuck in `Init`
+
+```plaintext
+Waiting for /run/nvidia/validations/toolkit-ready...
+```
+
+The `toolkit-validation` init container blocks until the NVIDIA Container Toolkit writes a `toolkit-ready` file under `devicePlugin.gpuOperatorToolkitReady.hostPath` (default `/run/nvidia/validations`). The gate is off by default and is meant for GPU Operator clusters. If it was enabled on a cluster without GPU Operator, that file is never created and the init container waits forever:
+
+```bash
+helm upgrade hami hami-charts/hami -n kube-system --reuse-values \
+ --set devicePlugin.gpuOperatorToolkitReady.enabled=false
+```
+
+### Node name not resolved
+
+The Device Plugin patches the node named by its `NODE_NAME` environment variable, which the chart populates from `spec.nodeName`. On charts older than v2.3.10 the variable was called `NodeName`, and a mismatched image and chart pair leaves the plugin unable to identify its own node. Upgrade rather than patching by hand:
+
+```bash
+helm upgrade hami hami-charts/hami -n kube-system --reuse-values
+```
+
+## Case 3: the resource never appears in Allocatable {#case-3}
+
+The Pod is `Running` and NVML works, but `nvidia.com/gpu` is absent. The Device Plugin registered with the API server but not with kubelet.
+
+### Check
+
+```bash
+kubectl logs -n kube-system -l app.kubernetes.io/component=hami-device-plugin \
+ -c device-plugin --tail=200 | grep -i -E "register|socket|kubelet"
+
+ls -l /var/lib/kubelet/device-plugins/ # on the node itself
+```
+
+The plugin registers over `kubelet.sock` in that directory. The chart mounts it from `devicePlugin.pluginPath`, which defaults to `/var/lib/kubelet/device-plugins`. If your distribution relocates the kubelet root, the plugin writes its socket somewhere kubelet never reads and registration silently never completes.
+
+### Fix
+
+Confirm the real path on the node, then point the chart at it:
+
+```bash
+# on the node
+ps aux | grep kubelet | grep -o '\--root-dir=[^ ]*'
+
+helm upgrade hami hami-charts/hami -n kube-system --reuse-values \
+ --set devicePlugin.pluginPath=/device-plugins
+```
+
+Restarting kubelet also forces every Device Plugin to re-register, which is a fast way to confirm the socket path is the problem.
+
+## Case 4: the register annotation is missing or stale {#case-4}
+
+This is the case that most often looks like a scheduler bug. kubelet advertises the GPUs, so `kubectl describe node` looks healthy, but `hami-scheduler` never places a Pod there.
+
+### Read the annotation
+
+```bash
+kubectl get node $NODE \
+ -o jsonpath='{.metadata.annotations.hami\.io/node-nvidia-register}' | jq .
+```
+
+Expected output for one 24 GiB card on a default install:
+
+```json
+[
+ {
+ "id": "GPU-fc28df76-54d2-c387-e52e-5f0a9495968c",
+ "count": 10,
+ "devmem": 24576,
+ "devcore": 100,
+ "type": "NVIDIA-NVIDIA L40S",
+ "mode": "hami-core",
+ "health": true
+ }
+]
+```
+
+| Field | Meaning | Source |
+| --------- | ------------------------------ | ------------------------------------------- |
+| `id` | GPU UUID | NVML |
+| `count` | Logical split count | `devicePlugin.deviceSplitCount` |
+| `devmem` | Schedulable memory in MiB | Physical memory times `deviceMemoryScaling` |
+| `devcore` | Schedulable compute percentage | `deviceCoreScaling` times 100 |
+| `type` | Model, prefixed with `NVIDIA-` | NVML |
+| `numa` | NUMA node | sysfs |
+| `mode` | `hami-core` or `mig` | Plugin operating mode |
+| `health` | Device health | Device Plugin health check |
+
+:::warning Zero-valued fields are omitted
+
+The annotation is serialized with `omitempty`, so any field whose value is zero or `false` is absent rather than shown. An **unhealthy GPU has no `health` key at all**, it does not appear as `"health": false`. The same applies to `"numa": 0` and `"index": 0`. Read a missing `health` key as unhealthy, not as healthy.
+
+:::
+
+### The annotation is only rewritten when it changes
+
+The Device Plugin rescans devices every 30 seconds, but it compares the newly encoded list against the last one it wrote and skips the patch when they are identical:
+
+```plaintext
+Device info unchanged, skipping annotation update
+```
+
+That line appears at verbosity `-v=3`. At the default verbosity, a real update logs:
+
+```plaintext
+Updating node annotations with 1 device(s)
+```
+
+The consequence: an old timestamp on the annotation is normal and is not evidence of a stuck plugin. Conversely, deleting the annotation by hand does **not** get it rewritten within 30 seconds, because the plugin's in-memory cache still matches what it thinks it wrote. Restart the Pod instead. See [Force a re-registration](#force-a-re-registration).
+
+### The patch is rejected
+
+```plaintext
+patch node error nodes "gpu-node-1" is forbidden: User "system:serviceaccount:kube-system:hami-device-plugin" cannot patch resource "nodes"
+```
+
+The ServiceAccount lost permission to patch nodes, usually after a partial upgrade or a hand-edited ClusterRole:
+
+```bash
+kubectl auth can-i patch nodes \
+ --as=system:serviceaccount:kube-system:hami-device-plugin
+```
+
+Reinstalling or upgrading the chart restores the RBAC objects.
+
+## Case 5: the scheduler does not see the node {#case-5}
+
+Both the resource and the annotation are correct, but Pods still do not land. The node is missing from the scheduler's in-memory cache.
+
+### Check the scheduler log
+
+```bash
+kubectl logs -n kube-system deploy/hami-scheduler -c vgpu-scheduler-extender --tail=200
+```
+
+The registration loop runs every 15 seconds, and also on node events and on leader changes. Raise verbosity to `-v=5` to see per-node decisions, which are otherwise silent:
+
+```plaintext
+Using label selector for list nodes
+Listed nodes
+Processing node
+Failed to get node devices
+```
+
+`Failed to get node devices` for your node means the scheduler read the annotation and rejected it. There are three ways that happens: the annotation key is absent, the JSON does not decode, and the decoded list is empty. The second and third also log at the default verbosity:
+
+```plaintext
+failed to decode node devices
+no nvidia gpu device found
+```
+
+A hand-edited annotation is the usual cause of a decode failure.
+
+### Cause A: a node label selector excludes the node
+
+The scheduler can be restricted to a subset of nodes with `scheduler.nodeLabelSelector`. It is commented out in the chart by default, so an unmodified install lists every node. If it was set, the value is echoed on startup:
+
+```bash
+kubectl logs -n kube-system deploy/hami-scheduler -c vgpu-scheduler-extender \
+ | grep "label selector"
+```
+
+Any node missing those labels is never registered, no matter how healthy it is.
+
+### Cause B: the replica you are reading is not the leader
+
+Only the leader performs registration. Every other replica logs:
+
+```plaintext
+Scheduler is not leader yet, skipping ...
+```
+
+With more than one `hami-scheduler` replica, a quiet log is expected on followers and proves nothing. Identify the leader before concluding the loop is dead:
+
+```bash
+kubectl get lease -n kube-system | grep hami
+```
+
+## The handshake annotation
+
+`hami.io/node-handshake` is a liveness marker maintained by the **scheduler**, not by the Device Plugin. Reading it backwards causes a lot of wasted debugging, so it is worth stating the actual behavior:
+
+- When the annotation is absent, or its value does not contain `Requesting`, the scheduler stamps it with `Requesting_` and treats the node as healthy.
+- While the timestamp is less than **60 seconds** old, the node is healthy.
+- Once it is older than 60 seconds, the scheduler checks the node's allocatable `nvidia.com/gpu`. If that is still greater than zero, the node stays healthy and nothing is cleaned up.
+- Only when the handshake has expired **and** allocatable has dropped to zero does the scheduler run node cleanup: it drops the node's devices from its cache and deletes the handshake annotation.
+
+```bash
+kubectl get node $NODE -o jsonpath='{.metadata.annotations.hami\.io/node-handshake}{"\n"}'
+# Requesting_2026-08-15 09:12:44
+```
+
+Two practical consequences:
+
+- **An old `Requesting_` timestamp is not a fault.** It is the normal steady state, and on its own it never removes a node. Do not tune anything based on it.
+- **The real trigger for a node leaving the scheduler's cache is allocatable falling to zero**, which means the Device Plugin stopped reporting to kubelet. Debug that, not the handshake. The scheduler logs the removal:
+
+ ```plaintext
+ Device is unhealthy, cleaning up node
+ ```
+
+:::note NVIDIA uses an unsuffixed key
+
+The NVIDIA handshake key is `hami.io/node-handshake`. Other vendors use a suffixed form such as `hami.io/node-handshake-dcu` or `hami.io/node-handshake-xpu`. A `kubectl get node -o yaml | grep node-handshake-nvidia` returns nothing on an NVIDIA node, which is expected.
+
+:::
+
+## Force a re-registration
+
+Restarting the Device Plugin Pod clears its in-memory device cache and forces a fresh annotation patch on the next scan. This is the correct recovery for a missing, truncated, or hand-edited register annotation:
+
+```bash
+kubectl delete pod -n kube-system \
+ -l app.kubernetes.io/component=hami-device-plugin \
+ --field-selector spec.nodeName=$NODE
+```
+
+Confirm the cycle completed:
+
+```bash
+kubectl logs -n kube-system -l app.kubernetes.io/component=hami-device-plugin \
+ -c device-plugin --tail=50 | grep "Updating node annotations"
+
+kubectl get node $NODE \
+ -o jsonpath='{.metadata.annotations.hami\.io/node-nvidia-register}' | jq length
+```
+
+The scheduler picks the node back up within one registration cycle, so allow about 15 seconds before retesting with a Pod.
+
+If the node still does not register, collect the following before opening an issue:
+
+```bash
+kubectl get node $NODE -o yaml > node.yaml
+kubectl logs -n kube-system -l app.kubernetes.io/component=hami-device-plugin \
+ -c device-plugin --tail=500 > device-plugin.log
+kubectl logs -n kube-system deploy/hami-scheduler \
+ -c vgpu-scheduler-extender --tail=500 > scheduler.log
+```
+
+## Validation environment
+
+The behavior described on this page was verified by reading the HAMi source at **v2.9.0** and on `master`, specifically `pkg/device-plugin/nvidiadevice/nvinternal/plugin/register.go`, `pkg/device/devices.go`, `pkg/device/nvidia/device.go`, `pkg/scheduler/scheduler.go`, and `charts/hami/values.yaml`. Chart defaults quoted here are the NVIDIA values from `charts/hami`. Intervals, the 60-second handshake window, and log strings can change between releases; check the source for the version you run before relying on an exact number.
+
+## Related pages
+
+- [Troubleshooting](./troubleshooting.md) for runtime and GPU Operator issues
+- [Validate HAMi](../get-started/verify-hami.md) for the end-to-end post-install check
+- [Protocol design](../developers/protocol.md) for the registration protocol itself
+- [GPU Virtualization](../core-concepts/gpu-virtualization.md) for how the annotation feeds scheduling
diff --git a/versioned_sidebars/version-v2.9.0-sidebars.json b/versioned_sidebars/version-v2.9.0-sidebars.json
index 6d0cd8678..55c9efe4b 100644
--- a/versioned_sidebars/version-v2.9.0-sidebars.json
+++ b/versioned_sidebars/version-v2.9.0-sidebars.json
@@ -395,6 +395,10 @@
"type": "doc",
"id": "troubleshooting/troubleshooting"
},
+ {
+ "type": "doc",
+ "id": "troubleshooting/node-registration"
+ },
{
"type": "doc",
"id": "faq/faq"