diff --git a/docs/troubleshooting/troubleshooting.md b/docs/troubleshooting/troubleshooting.md index ec4851f41..e6d31dd55 100644 --- a/docs/troubleshooting/troubleshooting.md +++ b/docs/troubleshooting/troubleshooting.md @@ -160,3 +160,60 @@ devicePlugin: ``` ::: + +## Pod Stuck in Pending {#pod-stuck-in-pending} + +If a Pod requesting GPU resources stays in `Pending`, check the following before assuming the cluster lacks capacity. + +- **Confirm the Pod is using HAMi's scheduler** - HAMi's mutating webhook only rewrites `schedulerName` for Pods whose resource requests it recognizes as HAMi-manageable. If it doesn't recognize the request, the Pod falls through to the default Kubernetes scheduler silently, and none of the failure reasons below will apply. Check with: + +```bash + kubectl get pod -o jsonpath='{.spec.schedulerName}' +``` + + If this does not return HAMi's scheduler name, confirm your resource requests use HAMi's expected resource names (e.g. `nvidia.com/gpu`), and confirm the webhook itself is running with `kubectl get pods -n kube-system | grep hami-scheduler`. + +- **Confirm the correct device plugin is installed** - HAMi requires its own customized device plugin per accelerator vendor. The stock/official vendor device plugin is not compatible and will produce unexpected behavior rather than a clean error. Check with `kubectl get pods -n kube-system -o wide | grep device-plugin` and confirm the image matches HAMi's documented device plugin for your vendor. + +- **Read the failure reason in Pod events** - as of HAMi v2.7.0, the scheduler reports a specific reason for each rejected node directly through Pod events, instead of only a generic "no available node" message. Check with `kubectl describe pod ` and look under Events. If you're running an earlier version, Pod events show only the generic message; check the scheduler logs directly instead with `kubectl logs -f -n kube-system -c vgpu-scheduler-extender`. + +Common failure reasons and what they mean: + +- **`NodeInsufficientDevice`** - the Pod requested more devices (by count) than the node has at all. +- **`CardTypeMismatch`** - the Pod's requested device type doesn't match the candidate card's actual type. +- **`CardInsufficientMemory`** - the card doesn't have enough free memory for the request. HAMi computes free memory as `Total memory - Used memory` and compares it against the requested amount (set via `gpumem`, or as a percentage of total device memory via `gpumem-percentage` if `gpumem` is unset). +- **`NumaNotFit`** - NUMA-aware scheduling is enabled, and the candidate device's NUMA node doesn't satisfy the Pod's topology requirement. + +### Worked example + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: gpu-pod +spec: + containers: + - name: worker01 + image: ubuntu:22.04 + command: ["bash", "-c", "sleep 86400"] + resources: + limits: + nvidia.com/gpu: "1" + nvidia.com/gpumem: "3000" + nvidia.com/gpucores: "30" +``` + +Replace `3000` with a value higher than the free memory on any candidate node in your cluster, so the request genuinely exceeds capacity. Apply this Pod, then check its events: + +```bash +kubectl apply -f gpu-pod.yaml +kubectl describe pod gpu-pod +``` + +You should see a `FilteringFailed` event whose message includes `CardInsufficientMemory`. For the specific total/used memory numbers behind the failure, check the scheduler logs: + +```bash +kubectl logs -f -n kube-system -c vgpu-scheduler-extender +``` + +Adjust the request to fit within available capacity and reapply; the Pod should transition to `Running`. diff --git a/versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md b/versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md index ec4851f41..e6d31dd55 100644 --- a/versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md +++ b/versioned_docs/version-v2.9.0/troubleshooting/troubleshooting.md @@ -160,3 +160,60 @@ devicePlugin: ``` ::: + +## Pod Stuck in Pending {#pod-stuck-in-pending} + +If a Pod requesting GPU resources stays in `Pending`, check the following before assuming the cluster lacks capacity. + +- **Confirm the Pod is using HAMi's scheduler** - HAMi's mutating webhook only rewrites `schedulerName` for Pods whose resource requests it recognizes as HAMi-manageable. If it doesn't recognize the request, the Pod falls through to the default Kubernetes scheduler silently, and none of the failure reasons below will apply. Check with: + +```bash + kubectl get pod -o jsonpath='{.spec.schedulerName}' +``` + + If this does not return HAMi's scheduler name, confirm your resource requests use HAMi's expected resource names (e.g. `nvidia.com/gpu`), and confirm the webhook itself is running with `kubectl get pods -n kube-system | grep hami-scheduler`. + +- **Confirm the correct device plugin is installed** - HAMi requires its own customized device plugin per accelerator vendor. The stock/official vendor device plugin is not compatible and will produce unexpected behavior rather than a clean error. Check with `kubectl get pods -n kube-system -o wide | grep device-plugin` and confirm the image matches HAMi's documented device plugin for your vendor. + +- **Read the failure reason in Pod events** - as of HAMi v2.7.0, the scheduler reports a specific reason for each rejected node directly through Pod events, instead of only a generic "no available node" message. Check with `kubectl describe pod ` and look under Events. If you're running an earlier version, Pod events show only the generic message; check the scheduler logs directly instead with `kubectl logs -f -n kube-system -c vgpu-scheduler-extender`. + +Common failure reasons and what they mean: + +- **`NodeInsufficientDevice`** - the Pod requested more devices (by count) than the node has at all. +- **`CardTypeMismatch`** - the Pod's requested device type doesn't match the candidate card's actual type. +- **`CardInsufficientMemory`** - the card doesn't have enough free memory for the request. HAMi computes free memory as `Total memory - Used memory` and compares it against the requested amount (set via `gpumem`, or as a percentage of total device memory via `gpumem-percentage` if `gpumem` is unset). +- **`NumaNotFit`** - NUMA-aware scheduling is enabled, and the candidate device's NUMA node doesn't satisfy the Pod's topology requirement. + +### Worked example + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: gpu-pod +spec: + containers: + - name: worker01 + image: ubuntu:22.04 + command: ["bash", "-c", "sleep 86400"] + resources: + limits: + nvidia.com/gpu: "1" + nvidia.com/gpumem: "3000" + nvidia.com/gpucores: "30" +``` + +Replace `3000` with a value higher than the free memory on any candidate node in your cluster, so the request genuinely exceeds capacity. Apply this Pod, then check its events: + +```bash +kubectl apply -f gpu-pod.yaml +kubectl describe pod gpu-pod +``` + +You should see a `FilteringFailed` event whose message includes `CardInsufficientMemory`. For the specific total/used memory numbers behind the failure, check the scheduler logs: + +```bash +kubectl logs -f -n kube-system -c vgpu-scheduler-extender +``` + +Adjust the request to fit within available capacity and reapply; the Pod should transition to `Running`.