From 88556ba611f0987a152d2c3a7becd67044a5e25c Mon Sep 17 00:00:00 2001 From: Samkeliso Date: Tue, 18 Aug 2026 11:26:51 +0200 Subject: [PATCH 1/2] docs: add simulated GPU scheduling troubleshooting Signed-off-by: Samkeliso --- docs/troubleshooting/troubleshooting.md | 154 ++++++++++++++++++++++++ 1 file changed, 154 insertions(+) diff --git a/docs/troubleshooting/troubleshooting.md b/docs/troubleshooting/troubleshooting.md index ec4851f41..18f2c32c8 100644 --- a/docs/troubleshooting/troubleshooting.md +++ b/docs/troubleshooting/troubleshooting.md @@ -160,3 +160,157 @@ devicePlugin: ``` ::: + + +## GPU Pod Scheduling Failure with Simulated GPUs + +When using simulated GPUs with HAMi, a GPU workload can remain `Pending` even though Kubernetes advertises the simulated GPU resources. + +### Symptom + +A GPU workload remains unscheduled: + +```bash +kubectl get pod hami-gpu-test -o wide +``` + +Output: +``` +NAME READY STATUS RESTARTS AGE +hami-gpu-test 0/1 Pending 0 5m +``` + +The pod events may show: + +``` +Warning FailedScheduling ... node ... has been locked within 5m0s +``` + + +HAMi scheduler logs may also report: + +``` +failed to decode node devices +``` + +### Cause + +HAMi needs the GPU device information on the node in its expected JSON format. + +In this simulated-GPU setup, the node annotation contained a colon-delimited device string: + +``` +GPU-MOCK-0,0,11441,100,NVIDIA-Tesla-K80,0,true:GPU-MOCK-1,1,11441,100,NVIDIA-Tesla-K80,0,true: +``` + +HAMi expected JSON and therefore failed to decode the node device information: + +``` +failed to decode node devices +err="invalid character 'G' looking for beginning of value" +``` + +### Solution + +Register the simulated NVIDIA devices on the node using the HAMi node registration annotation. + +Apply the annotation with proper JSON format: + +```bash +kubectl annotate node kcna-cluster-worker \ + 'hami.io/node-nvidia-register=[{"id":"GPU-MOCK-0","count":1,"devmem":11441,"devcore":100,"type":"NVIDIA-Tesla-K80","health":true,"numa":0,"mode":"hami-core"},{"id":"GPU-MOCK-1","count":1,"devmem":11441,"devcore":100,"type":"NVIDIA-Tesla-K80","health":true,"numa":0,"mode":"hami-core"}]' \ + --overwrite +``` + +Verify the annotation: + +```bash +kubectl get node kcna-cluster-worker \ + -o jsonpath='{.metadata.annotations.hami\.io/node-nvidia-register}' +echo +``` + +The output should contain a valid JSON array similar to: + +```json +[{"id":"GPU-MOCK-0","count":1,"devmem":11441,"devcore":100,"type":"NVIDIA-Tesla-K80","health":true,"numa":0,"mode":"hami-core"},{"id":"GPU-MOCK-1","count":1,"devmem":11441,"devcore":100,"type":"NVIDIA-Tesla-K80","health":true,"numa":0,"mode":"hami-core"}] +``` + +Then delete and recreate the affected pod so HAMi can attempt scheduling again: + +```bash +kubectl delete pod hami-gpu-test +``` + +Recreate the workload using the GPU resource request appropriate for the test environment. + +### Verification + +Check the pod: + +```bash +kubectl get pod hami-gpu-test -o wide +``` + +A successful result should show the pod running on the registered node: + +``` +NAME READY STATUS RESTARTS AGE +hami-gpu-test 1/1 Running 0 11s +``` + +Check the scheduling events: + +```bash +kubectl describe pod hami-gpu-test | grep -A10 "Events:" +``` + +Successful scheduling should include messages similar to: + +``` +Normal FilteringSucceed ... find fit node(kcna-cluster-worker) +Normal BindingSucceed ... Successfully binding node [kcna-cluster-worker] +Normal Scheduled ... Successfully assigned default/hami-gpu-test to kcna-cluster-worker +``` + +### Troubleshooting + +If the pod remains `Pending`,verify that Kubernetes advertises the simulated GPU resources: + +```bash +kubectl get node kcna-cluster-worker \ + -o jsonpath='{.status.capacity.nvidia\.com/gpu}' +echo +``` + + +Check the HAMi GPU registration annotation: +```bash +kubectl get node kcna-cluster-worker \ + -o jsonpath='{.metadata.annotations.hami\.io/node-nvidia-register}' +echo +``` + +Check the HAMi scheduler logs: +```bash +kubectl logs -n kube-system \ + -l app.kubernetes.io/component=hami-scheduler \ + --tail=200 +``` + +Look specifically for: + +``` +failed to decode node devices +``` + +If this error appears, inspect the `hami.io/node-nvidia-register` annotation and make sure it contains valid JSON. + +### Environment Tested + +* Kubernetes v1.36.1 +* HAMi v2.9.0 +* Kind +* Simulated GPUs +* Two simulated NVIDIA devices on the worker node +* Node: `kcna-cluster-worker` \ No newline at end of file From e3af87aab2af05864ac371ad93a78ac631f04da5 Mon Sep 17 00:00:00 2001 From: Samkeliso Date: Tue, 18 Aug 2026 12:10:07 +0200 Subject: [PATCH 2/2] docs: update simulated GPU scheduling troubleshooting Signed-off-by: Samkeliso --- docs/troubleshooting/troubleshooting.md | 41 +++++++++++++++++++------ 1 file changed, 31 insertions(+), 10 deletions(-) diff --git a/docs/troubleshooting/troubleshooting.md b/docs/troubleshooting/troubleshooting.md index 18f2c32c8..bfb87fb8e 100644 --- a/docs/troubleshooting/troubleshooting.md +++ b/docs/troubleshooting/troubleshooting.md @@ -161,7 +161,6 @@ devicePlugin: ::: - ## GPU Pod Scheduling Failure with Simulated GPUs When using simulated GPUs with HAMi, a GPU workload can remain `Pending` even though Kubernetes advertises the simulated GPU resources. @@ -175,6 +174,7 @@ kubectl get pod hami-gpu-test -o wide ``` Output: + ``` NAME READY STATUS RESTARTS AGE hami-gpu-test 0/1 Pending 0 5m @@ -186,7 +186,6 @@ The pod events may show: Warning FailedScheduling ... node ... has been locked within 5m0s ``` - HAMi scheduler logs may also report: ``` @@ -233,7 +232,28 @@ echo The output should contain a valid JSON array similar to: ```json -[{"id":"GPU-MOCK-0","count":1,"devmem":11441,"devcore":100,"type":"NVIDIA-Tesla-K80","health":true,"numa":0,"mode":"hami-core"},{"id":"GPU-MOCK-1","count":1,"devmem":11441,"devcore":100,"type":"NVIDIA-Tesla-K80","health":true,"numa":0,"mode":"hami-core"}] +[ + { + "id": "GPU-MOCK-0", + "count": 1, + "devmem": 11441, + "devcore": 100, + "type": "NVIDIA-Tesla-K80", + "health": true, + "numa": 0, + "mode": "hami-core" + }, + { + "id": "GPU-MOCK-1", + "count": 1, + "devmem": 11441, + "devcore": 100, + "type": "NVIDIA-Tesla-K80", + "health": true, + "numa": 0, + "mode": "hami-core" + } +] ``` Then delete and recreate the affected pod so HAMi can attempt scheduling again: @@ -283,8 +303,8 @@ kubectl get node kcna-cluster-worker \ echo ``` - Check the HAMi GPU registration annotation: + ```bash kubectl get node kcna-cluster-worker \ -o jsonpath='{.metadata.annotations.hami\.io/node-nvidia-register}' @@ -292,6 +312,7 @@ echo ``` Check the HAMi scheduler logs: + ```bash kubectl logs -n kube-system \ -l app.kubernetes.io/component=hami-scheduler \ @@ -308,9 +329,9 @@ If this error appears, inspect the `hami.io/node-nvidia-register` annotation and ### Environment Tested -* Kubernetes v1.36.1 -* HAMi v2.9.0 -* Kind -* Simulated GPUs -* Two simulated NVIDIA devices on the worker node -* Node: `kcna-cluster-worker` \ No newline at end of file +- Kubernetes v1.36.1 +- HAMi v2.9.0 +- Kind +- Simulated GPUs +- Two simulated NVIDIA devices on the worker node +- Node: `kcna-cluster-worker`