From d224ac056bb403449dd803bd9a4c6f54d9ebcd90 Mon Sep 17 00:00:00 2001 From: Olamide Oladejo Date: Mon, 17 Aug 2026 10:43:22 +0100 Subject: [PATCH 1/3] docs: add pod stuck in pending troubleshooting section Signed-off-by: Olamide Oladejo --- docs/troubleshooting/troubleshooting.md | 52 +++++++++++++++++++ .../troubleshooting/troubleshooting.md | 50 ++++++++++++++++++ 2 files changed, 102 insertions(+) diff --git a/docs/troubleshooting/troubleshooting.md b/docs/troubleshooting/troubleshooting.md index ec4851f41..683d28cff 100644 --- a/docs/troubleshooting/troubleshooting.md +++ b/docs/troubleshooting/troubleshooting.md @@ -34,6 +34,7 @@ If a container exceeds its `nvidia.com/gpumem` limit, check the following causes This will apply the fix automatically. + ## NVIDIA containers fail with GPU Operator 25.10+ {#nvidia-toolkit-gpu-operator-25-10} Use this section when the HAMi Device Plugin or a HAMi-scheduled NVIDIA workload stops starting after GPU Operator is installed or upgraded. @@ -160,3 +161,54 @@ devicePlugin: ``` ::: + +## Pod Stuck in Pending {#pod-stuck-in-pending} + +If a Pod requesting GPU resources stays in `Pending`, check the following before assuming the cluster lacks capacity. + +- **Confirm the Pod is using HAMi's scheduler** - HAMi's mutating webhook only rewrites `schedulerName` for Pods whose resource requests it recognizes as HAMi-manageable. If it doesn't recognize the request, the Pod falls through to the default Kubernetes scheduler silently, and none of the failure reasons below will apply. Check with: + +```bash + kubectl get pod -o jsonpath='{.spec.schedulerName}' +``` + + If this does not return HAMi's scheduler name, confirm your resource requests use HAMi's expected resource names (e.g. `nvidia.com/gpu`), and confirm the webhook itself is running with `kubectl get pods -n kube-system | grep hami-scheduler`. + +- **Confirm the correct device plugin is installed** - HAMi requires its own customized device plugin per accelerator vendor. The stock/official vendor device plugin is not compatible and will produce unexpected behavior rather than a clean error. Check with `kubectl get pods -n kube-system -o wide | grep device-plugin` and confirm the image matches HAMi's documented device plugin for your vendor. + +- **Read the failure reason in Pod events** - as of HAMi v2.7.0, the scheduler reports a specific reason for each rejected node directly through Pod events, instead of only a generic "no available node" message. Check with `kubectl describe pod ` and look under Events. If you're running an earlier version, Pod events show only the generic message; check the scheduler logs directly instead with `kubectl logs -f -n kube-system -c vgpu-scheduler-extender`. + +Common failure reasons and what they mean: + +- **`NodeInsufficientDevice`** - the Pod requested more devices (by count) than the node has at all. +- **`CardTypeMismatch`** - the Pod's requested device type doesn't match the candidate card's actual type. +- **`CardInsufficientMemory`** - the card doesn't have enough free memory for the request. HAMi computes free memory as `Total memory - Used memory` and compares it against the requested amount (set via `gpumem`, or as a percentage of total device memory via `gpumem-percentage` if `gpumem` is unset). +- **`NumaNotFit`** - NUMA-aware scheduling is enabled, and the candidate device's NUMA node doesn't satisfy the Pod's topology requirement. + +### Worked example + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: gpu-pod +spec: + containers: + - name: worker01 + image: ubuntu:22.04 + command: ["bash", "-c", "sleep 86400"] + resources: + limits: + nvidia.com/gpu: "1" + nvidia.com/gpumem: "3000" + nvidia.com/gpucores: "30" +``` + +Apply this Pod with `gpumem` set higher than any node has free, then check its events: + +```bash +kubectl apply -f gpu-pod.yaml +kubectl describe pod gpu-pod +``` + +You should see a `CardInsufficientMemory` event referencing the specific card and its total/used memory. Adjust the request to fit within available capacity and reapply; the Pod should transition to `Running`. diff --git a/versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md b/versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md index ec4851f41..a9f2fd676 100644 --- a/versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md +++ b/versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md @@ -160,3 +160,53 @@ devicePlugin: ``` ::: +## Pod Stuck in Pending {#pod-stuck-in-pending} + +If a Pod requesting GPU resources stays in `Pending`, check the following before assuming the cluster lacks capacity. + +- **Confirm the Pod is using HAMi's scheduler** - HAMi's mutating webhook only rewrites `schedulerName` for Pods whose resource requests it recognizes as HAMi-manageable. If it doesn't recognize the request, the Pod falls through to the default Kubernetes scheduler silently, and none of the failure reasons below will apply. Check with: + +```bash + kubectl get pod -o jsonpath='{.spec.schedulerName}' +``` + + If this does not return HAMi's scheduler name, confirm your resource requests use HAMi's expected resource names (e.g. `nvidia.com/gpu`), and confirm the webhook itself is running with `kubectl get pods -n kube-system | grep hami-scheduler`. + +- **Confirm the correct device plugin is installed** - HAMi requires its own customized device plugin per accelerator vendor. The stock/official vendor device plugin is not compatible and will produce unexpected behavior rather than a clean error. Check with `kubectl get pods -n kube-system -o wide | grep device-plugin` and confirm the image matches HAMi's documented device plugin for your vendor. + +- **Read the failure reason in Pod events** - as of HAMi v2.7.0, the scheduler reports a specific reason for each rejected node directly through Pod events, instead of only a generic "no available node" message. Check with `kubectl describe pod ` and look under Events. If you're running an earlier version, Pod events show only the generic message; check the scheduler logs directly instead with `kubectl logs -f -n kube-system -c vgpu-scheduler-extender`. + +Common failure reasons and what they mean: + +- **`NodeInsufficientDevice`** - the Pod requested more devices (by count) than the node has at all. +- **`CardTypeMismatch`** - the Pod's requested device type doesn't match the candidate card's actual type. +- **`CardInsufficientMemory`** - the card doesn't have enough free memory for the request. HAMi computes free memory as `Total memory - Used memory` and compares it against the requested amount (set via `gpumem`, or as a percentage of total device memory via `gpumem-percentage` if `gpumem` is unset). +- **`NumaNotFit`** - NUMA-aware scheduling is enabled, and the candidate device's NUMA node doesn't satisfy the Pod's topology requirement. + +### Worked example + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: gpu-pod +spec: + containers: + - name: worker01 + image: ubuntu:22.04 + command: ["bash", "-c", "sleep 86400"] + resources: + limits: + nvidia.com/gpu: "1" + nvidia.com/gpumem: "3000" + nvidia.com/gpucores: "30" +``` + +Apply this Pod with `gpumem` set higher than any node has free, then check its events: + +```bash +kubectl apply -f gpu-pod.yaml +kubectl describe pod gpu-pod +``` + +You should see a `CardInsufficientMemory` event referencing the specific card and its total/used memory. Adjust the request to fit within available capacity and reapply; the Pod should transition to `Running`. From 6cdd6830ca2e9ad04cf7f0e2e54031369915dc33 Mon Sep 17 00:00:00 2001 From: Olamide Oladejo Date: Mon, 17 Aug 2026 13:52:25 +0100 Subject: [PATCH 2/3] docs: address CodeRabbit review feedback Signed-off-by: Olamide Oladejo --- docs/troubleshooting/troubleshooting.md | 13 +++++++++---- .../troubleshooting/troubleshooting.md | 13 ++++++++++--- 2 files changed, 19 insertions(+), 7 deletions(-) diff --git a/docs/troubleshooting/troubleshooting.md b/docs/troubleshooting/troubleshooting.md index 683d28cff..f1c63a794 100644 --- a/docs/troubleshooting/troubleshooting.md +++ b/docs/troubleshooting/troubleshooting.md @@ -15,7 +15,7 @@ If a container exceeds its `nvidia.com/gpumem` limit, check the following causes containerd config dump | grep default_runtime_name ``` - The output must show `nvidia`. If not, follow the [Prerequisites](./installation/online-installation) guide. + The output must show `nvidia`. If not, follow the [Prerequisites](../installation/online-installation) guide. - If you don’t explicitly request vGPUs when using the device plugin with NVIDIA images, all GPUs on the host may be exposed to your container. - Currently, A100 MIG can be supported in only "none" and "mixed" modes. @@ -34,7 +34,6 @@ If a container exceeds its `nvidia.com/gpumem` limit, check the following causes This will apply the fix automatically. - ## NVIDIA containers fail with GPU Operator 25.10+ {#nvidia-toolkit-gpu-operator-25-10} Use this section when the HAMi Device Plugin or a HAMi-scheduled NVIDIA workload stops starting after GPU Operator is installed or upgraded. @@ -204,11 +203,17 @@ spec: nvidia.com/gpucores: "30" ``` -Apply this Pod with `gpumem` set higher than any node has free, then check its events: +Replace `3000` with a value higher than the free memory on any candidate node in your cluster, so the request genuinely exceeds capacity. Apply this Pod, then check its events: ```bash kubectl apply -f gpu-pod.yaml kubectl describe pod gpu-pod ``` -You should see a `CardInsufficientMemory` event referencing the specific card and its total/used memory. Adjust the request to fit within available capacity and reapply; the Pod should transition to `Running`. +You should see a `FilteringFailed` event whose message includes `CardInsufficientMemory`. For the specific total/used memory numbers behind the failure, check the scheduler logs: + +```bash +kubectl logs -f -n kube-system -c vgpu-scheduler-extender +``` + +Adjust the request to fit within available capacity and reapply; the Pod should transition to `Running`. diff --git a/versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md b/versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md index a9f2fd676..f1c63a794 100644 --- a/versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md +++ b/versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md @@ -15,7 +15,7 @@ If a container exceeds its `nvidia.com/gpumem` limit, check the following causes containerd config dump | grep default_runtime_name ``` - The output must show `nvidia`. If not, follow the [Prerequisites](./installation/online-installation) guide. + The output must show `nvidia`. If not, follow the [Prerequisites](../installation/online-installation) guide. - If you don’t explicitly request vGPUs when using the device plugin with NVIDIA images, all GPUs on the host may be exposed to your container. - Currently, A100 MIG can be supported in only "none" and "mixed" modes. @@ -160,6 +160,7 @@ devicePlugin: ``` ::: + ## Pod Stuck in Pending {#pod-stuck-in-pending} If a Pod requesting GPU resources stays in `Pending`, check the following before assuming the cluster lacks capacity. @@ -202,11 +203,17 @@ spec: nvidia.com/gpucores: "30" ``` -Apply this Pod with `gpumem` set higher than any node has free, then check its events: +Replace `3000` with a value higher than the free memory on any candidate node in your cluster, so the request genuinely exceeds capacity. Apply this Pod, then check its events: ```bash kubectl apply -f gpu-pod.yaml kubectl describe pod gpu-pod ``` -You should see a `CardInsufficientMemory` event referencing the specific card and its total/used memory. Adjust the request to fit within available capacity and reapply; the Pod should transition to `Running`. +You should see a `FilteringFailed` event whose message includes `CardInsufficientMemory`. For the specific total/used memory numbers behind the failure, check the scheduler logs: + +```bash +kubectl logs -f -n kube-system -c vgpu-scheduler-extender +``` + +Adjust the request to fit within available capacity and reapply; the Pod should transition to `Running`. From 7de405a0e2eb255e1b8b5c1d87776f9c2ee79fa2 Mon Sep 17 00:00:00 2001 From: Olamide Oladejo Date: Mon, 17 Aug 2026 14:20:59 +0100 Subject: [PATCH 3/3] docs: revert Prerequisites link to relative path that Docusaurus resolves correctly Signed-off-by: Olamide Oladejo --- docs/troubleshooting/troubleshooting.md | 2 +- .../version-v2.9.0/troubleshooting/troubleshooting.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/troubleshooting/troubleshooting.md b/docs/troubleshooting/troubleshooting.md index f1c63a794..e6d31dd55 100644 --- a/docs/troubleshooting/troubleshooting.md +++ b/docs/troubleshooting/troubleshooting.md @@ -15,7 +15,7 @@ If a container exceeds its `nvidia.com/gpumem` limit, check the following causes containerd config dump | grep default_runtime_name ``` - The output must show `nvidia`. If not, follow the [Prerequisites](../installation/online-installation) guide. + The output must show `nvidia`. If not, follow the [Prerequisites](./installation/online-installation) guide. - If you don’t explicitly request vGPUs when using the device plugin with NVIDIA images, all GPUs on the host may be exposed to your container. - Currently, A100 MIG can be supported in only "none" and "mixed" modes. diff --git a/versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md b/versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md index f1c63a794..e6d31dd55 100644 --- a/versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md +++ b/versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md @@ -15,7 +15,7 @@ If a container exceeds its `nvidia.com/gpumem` limit, check the following causes containerd config dump | grep default_runtime_name ``` - The output must show `nvidia`. If not, follow the [Prerequisites](../installation/online-installation) guide. + The output must show `nvidia`. If not, follow the [Prerequisites](./installation/online-installation) guide. - If you don’t explicitly request vGPUs when using the device plugin with NVIDIA images, all GPUs on the host may be exposed to your container. - Currently, A100 MIG can be supported in only "none" and "mixed" modes.