docs: add GPU node registration troubleshooting guide

docs/developers/protocol.md

@@ -6,7 +6,7 @@ title: Protocol design
 
 <img src="/img/docs/common/developers/protocol/protocol-register.png" width="600px" alt="HAMi device registration protocol diagram showing node annotation process" />
 
-HAMi needs to know the spec of each AI device in the cluster to schedule properly. During device registration, device-plugin needs to keep patching the spec of each device into node annotations every 30 seconds, in the format of the following:
+HAMi needs to know the spec of each AI device in the cluster to schedule properly. During device registration, device-plugin rescans its devices every 30 seconds and patches the spec of each device into node annotations, in the format of the following:
 
 ```text
 hami.io/node-handshake-\{device-type\}: Reported_\{device_node_current_timestamp\}
@@ -19,6 +19,14 @@ The definition of each device is in the following format:
 \{Device UUID\},\{device split count\},\{device memory limit\},\{device core limit\},\{device type\},\{device numa\},\{healthy\}
 ```
 
+:::note Encoding differs by vendor
+
+The colon-separated form above is the legacy encoding, still used by device types decoded with `DecodeNodeDevices` (for example DCU and Iluvatar). NVIDIA and several other device types encode the same fields as a **JSON array** instead, one object per device, decoded with `UnMarshalNodeDevices`. Fields are serialized with `omitempty`, so zero and `false` values are absent rather than written out. See [GPU Virtualization](../core-concepts/gpu-virtualization.md) for a JSON example.
+
+The annotation key is also not fully uniform: NVIDIA uses `hami.io/node-nvidia-register` and `hami.io/node-handshake` (no device-type suffix), while Kunlun uses `hami.io/node-register-xpu`.
+
+:::
+
 An example is shown below:
 
 ```text
@@ -31,10 +39,18 @@ hami.io/node-nvidia-register: GPU-00552014-5c87-89ac-b1a6-7b53aa24b0ec,10,32768,
 
 In this example, this node has two different AI devices, 2 NVIDIA-V100 GPUs, and 2 Cambricon 370-X4 MLUs
 
-A device node may become unavailable due to hardware or network failure. If a node hasn't registered in the last 5 minutes, the scheduler marks it as 'unavailable'.
-
-Since system clock on scheduler node and 'device' node may not align properly, scheduler node will patch the following device node annotations every 30s
+A device node may become unavailable due to hardware or network failure. Since the system clock on the scheduler node and on the device node may not align properly, the scheduler owns the timestamp: whenever the handshake annotation is absent or does not contain `Requesting`, the scheduler stamps it with its own clock.
 
 ```text
 hami.io/node-handshake-\{device-type\}: Requesting_{scheduler_node_current_timestamp}
 ```
+
+The scheduler's registration loop runs every 15 seconds, and also on node events and on leader election changes. Only the elected leader performs registration.
+
+A handshake is treated as expired once its timestamp is more than **60 seconds** old. Expiry alone does not remove the node. The scheduler additionally requires the node's allocatable device count to have dropped to zero before it runs node cleanup, which removes the node's devices from the scheduler cache and deletes the handshake annotation. A node whose device-plugin is still reporting to kubelet therefore stays available even with an expired handshake.
+
+:::note The NVIDIA device-plugin does not write the handshake
+
+The `Reported_` side of this protocol is written by device-plugins that implement it. The in-tree NVIDIA device-plugin does not, so on an NVIDIA node the annotation normally stays at `Requesting_<timestamp>` and ages past 60 seconds. That is the expected steady state, not a fault: the effective liveness signal for NVIDIA is the allocatable device count. See [GPU Nodes Not Registering](../troubleshooting/node-registration.md).
+
+:::

i18n/zh/docusaurus-plugin-content-docs/current/developers/protocol.md

@@ -18,6 +18,26 @@ hami.io/node-handshake: Requesting_2024.05.14 07:07:33
 hami.io/node-nvidia-register: "GPU-00552014-5c87-89ac-b1a6-7b53aa24b0ec,10,32768,100,NVIDIA-Tesla V100-PCIE-32GB,0,true:GPU-0fc3eda5-e98b-a25b-5b0d-cf5c855d1448,10,32768,100,NVIDIA-Tesla V100-PCIE-32GB,0,true:"
 ```
 
+:::note 编码方式因设备类型而异
+
+上面这种冒号分隔的形式是早期编码，目前仍用于通过 `DecodeNodeDevices` 解码的设备类型（例如 DCU、天数智芯）。NVIDIA 以及其他若干设备类型改为使用 **JSON 数组**编码同样的字段，每个设备一个对象，通过 `UnMarshalNodeDevices` 解码。这些字段使用 `omitempty` 序列化，因此取值为零或 `false` 的字段会被省略而不会写出。JSON 示例参见 [GPU 虚拟化](../core-concepts/gpu-virtualization.md)。
+
+注解键也并非完全统一：NVIDIA 使用 `hami.io/node-nvidia-register` 和 `hami.io/node-handshake`（不带设备类型后缀），而昆仑芯使用 `hami.io/node-register-xpu`。
+
+:::
+
+### 握手与节点存活
+
+时间戳由调度器负责写入：当握手注解不存在，或其值不含 `Requesting` 时，调度器会用自己的时钟写入 `Requesting_<时间戳>`。调度器的注册循环每 15 秒执行一次，节点事件和主节点切换时也会触发，且只有当选为 leader 的副本才会执行注册。
+
+握手时间戳超过 **60 秒**即视为过期。但过期本身不会移除节点：调度器还要求该节点的可分配设备数量降为零，才会执行节点清理，把设备移出调度器缓存并删除握手注解。因此，只要 device-plugin 仍在向 kubelet 上报，节点即使握手过期也依然可用。
+
+:::note NVIDIA device-plugin 不写握手注解
+
+本协议中 `Reported_` 一侧由实现了该协议的 device-plugin 写入。内置的 NVIDIA device-plugin 并未实现，因此在 NVIDIA 节点上该注解通常一直停留在 `Requesting_<时间戳>` 并超过 60 秒。这是预期稳态而非故障：对 NVIDIA 而言，真正的存活信号是可分配设备数量。参见 [GPU 节点未注册](../troubleshooting/node-registration.md)。
+
+:::
+
 ### 调度决策
 
 kube-scheduler 在 `bind` 过程中调用 device-plugin 挂载设备，但仅向 device-plugin 提供设备的 `UUID`。因此，在设备共享的场景中，device-plugin 无法获取任务请求的相应设备规格，如 `设备显存` 和 `计算核心`。