diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 95c4f387a11..bc645452167 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -143,6 +143,8 @@ repos: - id: check-case-conflict priority: 10 - id: check-yaml + # Helm templates use Go templating ({{ ... }}) that isn't valid standalone YAML. + exclude: ^deploy/helm/.*/templates/ priority: 10 - id: check-toml priority: 10 diff --git a/deploy/helm/gpu_autoscaling_k8s/.helmignore b/deploy/helm/gpu_autoscaling_k8s/.helmignore new file mode 100644 index 00000000000..7da0ef05be4 --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/.helmignore @@ -0,0 +1,5 @@ +.DS_Store +*.swp +*.bak +*.tmp +.git/ diff --git a/deploy/helm/gpu_autoscaling_k8s/Chart.yaml b/deploy/helm/gpu_autoscaling_k8s/Chart.yaml new file mode 100644 index 00000000000..1d474ab7be7 --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/Chart.yaml @@ -0,0 +1,14 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +apiVersion: v2 +name: nemoclaw-gpu +description: GPU NemoClaw agent pods on Kubernetes with local Ollama inference and HPA +type: application +version: 0.1.0 +appVersion: "2026.05.27" +keywords: + - nemoclaw + - ollama + - gpu +maintainers: + - name: maggiezha diff --git a/deploy/helm/gpu_autoscaling_k8s/README.md b/deploy/helm/gpu_autoscaling_k8s/README.md new file mode 100644 index 00000000000..5166823b806 --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/README.md @@ -0,0 +1,695 @@ + + +# NemoClaw Kubernetes GPU autoscaling + +This Helm chart uses Kubernetes to autoscale NemoClaw and NGINX to balance workloads. +A Kubernetes Horizontal Pod Autoscaler (HPA) scales the pods, and each replica requests one NVIDIA GPU. + +GPU utilization is the chart's only supported HPA scaling signal. +The HPA reads the per-pod `gpu_utilization_percent` custom metric. +The metric pipeline is: + +```text +NVIDIA GPU + → DCGM Exporter: DCGM_FI_DEV_GPU_UTIL + → Prometheus + → Prometheus Adapter: gpu_utilization_percent + → custom.metrics.k8s.io + → Horizontal Pod Autoscaler +``` + +The default deployment uses: + +| Setting | Default | +|---------|---------| +| Namespace | `nemoclaw-gpu` | +| Release | `nemoclaw-gpu` | +| Service port | `8081` | +| Ollama model | `llama3.2:3b` | +| GPUs per pod | `1` | +| Minimum replicas | `1` | +| Maximum replicas | `4` | +| GPU utilization target | `40%` | +| Ingress class | `nginx` | +| Ingress host | `nemoclaw.local` | + +The chart and load test were validated on a single-node MicroK8s cluster with four NVIDIA L40S GPUs. Set the maximum replica count to the number of allocatable GPUs in your cluster. + +Screenshot 2026-06-09 at 7 01 28 PM + +## Prerequisites + +- Kubernetes 1.25 or newer +- One or more allocatable `nvidia.com/gpu` resources +- GPU nodes labeled `nvidia.com/gpu.present=true` +- NVIDIA GPU Operator with DCGM Exporter running +- Metrics Server +- Helm 3 +- `kubectl` configured for the cluster + +Check the cluster before installation: + +```bash +kubectl get nodes +kubectl get nodes \ + -o jsonpath='{range .items[*]}{.metadata.name}{" GPUs="}{.status.allocatable.nvidia\.com/gpu}{"\n"}{end}' +kubectl get nodes -l nvidia.com/gpu.present=true +kubectl get pods -n gpu-operator-resources \ + -l app=nvidia-dcgm-exporter +``` + +For MicroK8s, the installer enables the GPU and Metrics Server add-ons when needed. On other Kubernetes distributions, install those components before running the installer. The installer also installs the ingress-nginx controller when the cluster does not already have an `nginx` IngressClass. + +## Install + +From the NemoClaw repository, enter the chart directory: + +```bash +cd deploy/helm/gpu_autoscaling_k8s +``` + +Before installing (or after editing the chart), check that the HPA/Deployment/Service/ +ServiceMonitor name-and-label contract and the script security contract still hold. +These checks do not require a cluster: + +```bash +./scripts/test-render-contract.sh +./scripts/test-script-security-contract.sh +``` + +The scripts require TLS by default. +Before installation, create the target namespace and its certificate Secret: + +```bash +kubectl create namespace nemoclaw-gpu --dry-run=client -o yaml \ + | kubectl apply -f - +kubectl create secret tls nemoclaw-example-tls \ + --namespace nemoclaw-gpu \ + --cert=/path/to/tls.crt \ + --key=/path/to/tls.key \ + --dry-run=client -o yaml \ + | kubectl apply -f - +``` + +Kubernetes stores the certificate and private key in the `nemoclaw-example-tls` Secret. +The chart does not create, rotate, or delete this Secret. + +Copy the GPU HPA values file and add an `ingress` block that references the Secret: + +```bash +cp values-step2-hpa.yaml /path/to/hpa-tls-values.yaml +``` + +Add this configuration to `/path/to/hpa-tls-values.yaml`: + +```yaml +ingress: + host: nemoclaw.example.com + tls: + - secretName: nemoclaw-example-tls + hosts: + - nemoclaw.example.com +``` + +Export the values file and hostname in the shell that runs the installation and later operational scripts: + +```bash +export HPA_VALUES=/path/to/hpa-tls-values.yaml +export INGRESS_HOST=nemoclaw.example.com +``` + +The installer creates the ingress-nginx controller Service as `ClusterIP` by default. +Use port forwarding to reach it from outside the cluster. +Set `INGRESS_SERVICE_TYPE=NodePort` or `LoadBalancer` only when the cluster network must expose the controller. +Those types can make the Ingress reachable outside the cluster. +Verify the assigned addresses and network access controls before you send credentials or completion traffic: + +```bash +kubectl get service ingress-nginx-controller -n ingress-nginx -o wide +``` + +Install the chart: + +```bash +./scripts/install-hpa.sh +``` + +The installer: + +1. Verifies that the cluster has an allocatable GPU. +2. Waits for Metrics Server to become ready. +3. Installs Prometheus when it is missing. +4. Installs Prometheus Adapter and the GPU metric rule. +5. Installs the ingress-nginx controller when the `nginx` IngressClass is missing. +6. Deploys one Ollama-backed API proxy pod and the NGINX Ingress in front of it. +7. Creates the GPU utilization HPA. +8. Waits for the agent rollout and prints HPA status. + +Set the maximum replica count to the number of available GPUs: + +```bash +MAX_REPLICAS=4 ./scripts/install-hpa.sh +``` + +The first startup downloads the Ollama model and can take several minutes. Increase the rollout timeout when needed: + +```bash +ROLLOUT_TIMEOUT=1200 \ +INFERENCE_MODEL=llama3.2:3b \ +MAX_REPLICAS=4 \ +./scripts/install-hpa.sh +``` + +`hpa-reset.sh` does not persist the release's current Ingress host — it re-applies whatever +`INGRESS_HOST` is set in its own environment (default: unset, which falls back to +`values.yaml`'s `nemoclaw.local`). If you set a custom host, pass the same `INGRESS_HOST` +and `HPA_VALUES` to later script invocations. + +## Verify the deployment + +Check the workload and HPA: + +```bash +kubectl get pods,service,hpa -n nemoclaw-gpu +``` + +The idle state should have: + +- One `Running` agent pod +- Two ready containers in the pod +- One HPA replica +- HPA bounds matching the configured GPU count +- A `current/40` GPU utilization target + +Confirm that the custom metric API returns a value for each agent pod: + +```bash +kubectl get --raw \ + '/apis/custom.metrics.k8s.io/v1beta1/namespaces/nemoclaw-gpu/pods/*/gpu_utilization_percent' +``` + +Inspect readable HPA and per-pod GPU utilization: + +```bash +./scripts/get-hpa.sh -n nemoclaw-gpu +./scripts/get-agent-pods.sh -n nemoclaw-gpu +``` + +Metrics can remain unknown for one or two minutes while Prometheus discovers DCGM Exporter and Prometheus Adapter publishes the custom metric. + +## Call the inference service + +Forward the agent Service to the local machine: + +```bash +kubectl port-forward \ + -n nemoclaw-gpu \ + service/nemoclaw-gpu-agent \ + 8081:8081 +``` + +In another terminal: + +```bash +curl -s http://127.0.0.1:8081/healthz +curl -s http://127.0.0.1:8081/readyz +curl -s http://127.0.0.1:8081/v1/chat/completions \ + -H 'Content-Type: application/json' \ + -d '{ + "messages": [{"role": "user", "content": "Say hello."}], + "max_tokens": 16, + "stream": false + }' +``` + +`/readyz` can return `503` during the initial model download. + +## Watch GPU autoscaling + +Watch the HPA: + +```bash +kubectl get hpa -n nemoclaw-gpu -w +``` + +In another terminal, watch every agent pod and its GPU utilization: + +```bash +./scripts/get-agent-pods.sh -n nemoclaw-gpu -w +``` + +The HPA adds replicas when average GPU utilization remains above the configured target. It removes replicas after load stops and the scale-down stabilization window expires. + +## Test scale-up and scale-down + +`hpa-load-test.sh` generates a synthetic chat-completion workload to verify that the HPA actually autoscales the pod count, rather than relying on organic traffic. The load test sends chat-completion requests across running GPU pods, verifies the HPA replica count increases, removes the load, and verifies the HPA returns to one replica. + +Run the test with `TARGET_PODS` and `SCALE_UP_TARGET` set to your full allocatable GPU count (4 on the reference node). A one-GPU run cannot validate scale-up, and validating at a lower replica count (for example, 2) does not confirm the HPA and load generator can also reach the full count. HPA replica-count success confirms autoscaler behavior, but it does not by itself prove that every new replica completed inference successfully. + +Run the full test: + +```bash +HPA_VALUES=/path/to/hpa-tls-values.yaml \ +INGRESS_HOST=nemoclaw.example.com \ +./scripts/hpa-load-test.sh +``` + +By default, `TARGET_PODS` is the number of allocatable GPUs, so the test already targets your full GPU count without overrides. + +A successful run reports: + +```text +Scale-up OK: 4/4 replicas +Load test complete: scaled to 4/4 GPU replicas and back to 1 +``` + +Screenshot 2026-06-09 at 7 02 00 PM + +Screenshot 2026-06-09 at 7 39 36 PM + +The script exits with a nonzero status if it does not reach the scale-up target or does not return to one replica. + +Restore the normal one-to-four bounds after every load test, including successful runs: + +```bash +HPA_VALUES=/path/to/hpa-tls-values.yaml \ +INGRESS_HOST=nemoclaw.example.com \ +./scripts/hpa-reset.sh +``` + +## Load-test settings + +| Variable | Default | Purpose | +|----------|---------|---------| +| `TARGET_PODS` | Allocatable GPUs | Temporary HPA maximum and test target | +| `SCALE_UP_TARGET` | `TARGET_PODS` | Replica count required for scale-up success | +| `HPA_TARGET_GPU` | `40` | GPU utilization target during the test | +| `DURATION_SEC` | `720` | Load duration | +| `MAX_TOKENS` | `128` | Maximum generated tokens per request | +| `INFLIGHT_PER_GPU` | `64` | Base concurrent load per GPU | +| `LOAD_MULTIPLIER` | `2` | Load multiplier | +| `MAX_INFLIGHT_PER_POD` | `512` | Per-pod concurrency cap | +| `WARMUP_SEC` | `90` | Warm-up period | +| `SCALE_UP_WAIT_LOOPS` | `60` | Scale-up polling limit | +| `SCALE_DOWN_WAIT_LOOPS` | `40` | Scale-down polling limit | + +Override a setting by placing it before the command: + +```bash +HPA_VALUES=/path/to/hpa-tls-values.yaml \ +INGRESS_HOST=nemoclaw.example.com \ +TARGET_PODS=4 MAX_TOKENS=256 DURATION_SEC=600 \ + ./scripts/hpa-load-test.sh +``` + +## Scripts + +All commands below are run from `deploy/helm/gpu_autoscaling_k8s`. + +| Script | Purpose | +|--------|---------| +| `scripts/install-hpa.sh` | Install or refresh the monitoring pipeline, chart, and GPU HPA | +| `scripts/hpa-load-test.sh` | Verify GPU-driven scale-up and scale-down | +| `scripts/hpa-reset.sh` | Remove test resources and restore the idle HPA configuration | +| `scripts/cluster-recover.sh` | Recover from repeated rollout or namespace failures | +| `scripts/get-agent-pods.sh` | Show agent pods with per-pod GPU utilization | +| `scripts/get-hpa.sh` | Show readable HPA GPU utilization | +| `scripts/hpa-watch.sh` | Watch HPA changes | +| `scripts/hpa-common.sh` | Shared script helpers | +| `scripts/test-render-contract.sh` | Static `helm template` check: HPA/Deployment/Service/ServiceMonitor names and labels agree | +| `scripts/test-script-security-contract.sh` | Static recovery-selector and cleartext-ingress security regression check | + +Export the TLS configuration before you run operational scripts: + +```bash +export HPA_VALUES=/path/to/hpa-tls-values.yaml +export INGRESS_HOST=nemoclaw.example.com +``` + +Run these commands as separate activities: + +```text +Install: ./scripts/install-hpa.sh +Watch HPA: ./scripts/hpa-watch.sh +Watch GPU pods: ./scripts/get-agent-pods.sh -w +Test autoscaling: ./scripts/hpa-load-test.sh +Restore idle state: ./scripts/hpa-reset.sh +``` + +Run `./scripts/cluster-recover.sh` only when the selected release needs the destructive recovery described in the next section. + +## Recover the Chart Workload + +`cluster-recover.sh` deletes and recreates the selected release's workload resources during recovery. +It restricts pod cleanup to the selected Helm release and the named load-test Job. +It preserves Helm resources that have the `helm.sh/resource-policy: keep` annotation, including the generated Basic auth Secret. + +The default load-test Job is `nemoclaw-gpu-hpa-load-test`. +Set `JOB_NAME` when the load test used a different name: + +```bash +NAMESPACE=nemoclaw-gpu \ +RELEASE=nemoclaw-gpu \ +JOB_NAME=nemoclaw-gpu-hpa-load-test \ +HPA_VALUES=/path/to/hpa-tls-values.yaml \ +INGRESS_HOST=nemoclaw.example.com \ +./scripts/cluster-recover.sh +``` + +The script performs these destructive operations in `NAMESPACE`: + +- Deletes the Deployment, Service, Horizontal Pod Autoscaler, ReplicaSets, and pods that have the selected release's chart ownership labels. +- Deletes only the Job named by `JOB_NAME` and pods with `job-name=${JOB_NAME}`. +- Clears finalizers only from pods in those two groups before force deletion. +- Uninstalls only the Helm release named by `RELEASE`. + +The script does not delete other Jobs or pods that have a different `job-name` value. + +The script does not restart MicroK8s by default. +Set `RESTART_MICROK8S=1` only when the cluster runtime must restart. +That setting stops every workload in the MicroK8s cluster before recovery continues: + +```bash +RESTART_MICROK8S=1 ./scripts/cluster-recover.sh +``` + +After an opt-in restart, verify all namespaces before you treat the cluster as restored: + +```bash +kubectl get pods --all-namespaces +``` + +After recovery, verify that unrelated resources remain and the selected release is available: + +```bash +kubectl get jobs,pods -n nemoclaw-gpu --show-labels +./scripts/get-agent-pods.sh -n nemoclaw-gpu +./scripts/get-hpa.sh -n nemoclaw-gpu +``` + +The recovery boundary is correct when nonmatching Jobs and pods remain and the selected release reports its agent pods and HPA. + +## Configure Ollama Model Storage + +The base `values.yaml` disables persistence, so each replica uses an independent `emptyDir` volume. +Kubernetes removes that cache when it replaces the pod, and the replacement pod downloads the configured model again. + +The provided `values-step2-hpa.yaml` sets `ollama.persistence.hostPath` for the documented single-node workflow. +All replicas on that node share the directory. +Do not use this mode when replicas can run on different nodes because the same path refers to different node-local directories. + +For storage that remains available when pods move between nodes, configure a storage class that provisions `ReadWriteMany` volumes: + +```yaml +ollama: + persistence: + enabled: true + hostPath: "" + accessMode: ReadWriteMany + storageClass: example-rwx + size: 20Gi +``` + +Every replica mounts this one persistent volume claim and shares the mutable Ollama model cache. +A model downloaded or removed by one replica is visible to the other replicas. +Use a storage backend that supports concurrent access from every GPU node, and avoid changing the configured model while load-test or inference replicas are running. + +The chart rejects PVC persistence when `accessMode` is not `ReadWriteMany` or `storageClass` is empty. +This render validation prevents a `ReadWriteOnce` claim from causing multi-attach failures when the HPA schedules replicas on different nodes. +Kubernetes still determines whether the selected storage class can provision the requested access mode. + +After installing the RWX configuration, verify the claim before generating load: + +```bash +kubectl get pvc -n nemoclaw-gpu +kubectl get pods -n nemoclaw-gpu -o wide +``` + +The check passes when the Ollama claim reports `Bound` with `RWX` access and each agent pod's `READY` column reports `2/2`. + +## Configuration files + +| File | Purpose | +|------|---------| +| `values.yaml` | Base GPU workload and resource settings | +| `values-step2-hpa.yaml` | GPU utilization HPA settings | +| `values-load-test-hpa.yaml` | Faster scale-up policy used by the load test | +| `monitoring/dcgm-servicemonitor.yaml` | Prometheus discovery for DCGM Exporter | +| `monitoring/kube-prometheus-microk8s.yaml` | Prometheus settings for MicroK8s | +| `monitoring/prometheus-adapter-gpu-values.yaml` | Custom GPU metric mapping | +| `templates/hpa.yaml` | HPA resource | +| `templates/deployment.yaml` | Ollama and agent pod | +| `templates/service.yaml` | Agent ClusterIP Service | +| `templates/ingress.yaml` | NGINX ingress route (always created) | + +Change the GPU HPA policy in `values-step2-hpa.yaml`: + +```yaml +autoscaling: + enabled: true + minReplicas: 1 + maxReplicas: 4 + targetGPUUtilizationPercentage: 40 +``` + +The maximum replica count should not exceed the total allocatable GPU count when every pod requests one GPU. + +Scale-up adds one pod per reconcile (`scaleUp.policies: [{type: Pods, value: 1}]`) instead of jumping straight to the replica count the raw GPU-utilization ratio (`ceil(currentReplicas * current/target)`) would otherwise allow in a single step. Increase `value` in `scaleUp.policies` to allow larger jumps. + +## GPU metric details + +| Layer | Value | +|-------|-------| +| DCGM metric | `DCGM_FI_DEV_GPU_UTIL` | +| HPA metric | `gpu_utilization_percent` | +| Kubernetes API | `custom.metrics.k8s.io/v1beta1` | +| HPA target type | `AverageValue` | +| Default target | `40` | +| Scope | One value per agent pod | + +Inspect the HPA conditions and recent events: + +```bash +kubectl describe hpa nemoclaw-gpu-agent -n nemoclaw-gpu +``` + +`ScalingActive=True` and `ValidMetricFound` confirm that the HPA can calculate a desired replica count from GPU utilization. + +## Traffic distribution + +NGINX is the load balancer for this chart — every install creates an Ingress in front of the agent Service, and `install-hpa.sh` installs the ingress-nginx controller automatically when the cluster does not already have one. There is no toggle to disable it. + +```text +HTTPS client + → ingress-nginx + → Service nemoclaw-gpu-agent:8081 (ClusterIP) + → ready agent pod + → local Ollama container + → assigned NVIDIA GPU +``` + +ingress-nginx watches Kubernetes endpoints and adds newly ready HPA replicas to its upstream pool. This chart uses a standard Kubernetes Ingress and keeps the application Service as `ClusterIP`. + +### NGINX ingress + +Verify that ingress-nginx and its `nginx` IngressClass are running after `install-hpa.sh` completes: + +```bash +kubectl get pods -n ingress-nginx +kubectl get ingressclass nginx +kubectl get ingress -n nemoclaw-gpu +``` + +Set a custom hostname at install time, or rerun on an existing release: + +```bash +HPA_VALUES=/path/to/hpa-tls-values.yaml \ +INGRESS_HOST=nemoclaw.example.com \ +./scripts/install-hpa.sh +``` + +The default NGINX annotations allow long inference requests and streaming responses: + +```yaml +ingress: + className: nginx + host: nemoclaw.local + path: / + annotations: + nginx.ingress.kubernetes.io/proxy-connect-timeout: "60" + nginx.ingress.kubernetes.io/proxy-read-timeout: "600" + nginx.ingress.kubernetes.io/proxy-send-timeout: "600" + nginx.ingress.kubernetes.io/proxy-buffering: "off" + nginx.ingress.kubernetes.io/proxy-request-buffering: "off" +``` + +Verify the route without requiring an external load balancer: + +```bash +kubectl port-forward \ + -n ingress-nginx \ + service/ingress-nginx-controller \ + 8443:443 +``` + +In another terminal, retrieve the auto-generated Basic auth password. +Send an HTTPS request with the configured hostname and the certificate authority file that verifies your certificate: + +```bash +PASSWORD=$(kubectl get secret nemoclaw-gpu-agent-ingress-auth -n nemoclaw-gpu \ + -o jsonpath='{.data.password}' | base64 -d) +curl --fail --show-error --silent \ + --cacert /path/to/ca.crt \ + --resolve 'nemoclaw.example.com:8443:127.0.0.1' \ + -u "admin:${PASSWORD}" \ + https://nemoclaw.example.com:8443/healthz +``` + +Rate limiting is disabled by default because it can suppress the load that drives autoscaling. Enable it only when the limit is intentionally part of the deployment policy: + +```yaml +ingress: + annotations: + nginx.ingress.kubernetes.io/limit-rps: "20" + nginx.ingress.kubernetes.io/limit-burst-multiplier: "5" +``` + +### Ingress security + +The completion proxy (`agent-server.ts`) has no authentication of its own, so the chart +enforces two things at the Ingress level: + +- **Basic auth is on by default** (`ingress.auth.enabled: true`). The chart auto-generates a + random password on first install and reads it back from the existing Secret (via Helm's + `lookup`) on every later `helm upgrade`, so it doesn't rotate every time + `install-hpa.sh`/`hpa-load-test.sh`/`hpa-reset.sh` re-runs. Retrieve it with: + + ```bash + kubectl get secret nemoclaw-gpu-agent-ingress-auth -n nemoclaw-gpu \ + -o jsonpath='{.data.password}' | base64 -d + ``` + + Set `ingress.auth.password` yourself, or `ingress.auth.existingSecret` to point at your + own `kubernetes.io/basic-auth`-style secret (must contain an `auth` key in htpasswd + format), to use a specific credential instead. + + The generated credential is in the `nemoclaw-gpu-agent-ingress-auth` Secret in the + `nemoclaw-gpu` namespace. A Kubernetes subject that can read Secrets in that namespace + can retrieve it. The Secret has the Helm `keep` policy, so `helm uninstall` and + `cluster-recover.sh` preserve it for a later reinstall. Delete it explicitly to remove + the credential or force the next install to generate a new value: + + ```bash + kubectl delete secret nemoclaw-gpu-agent-ingress-auth -n nemoclaw-gpu + ``` + + Deleting the namespace also deletes the generated credential. The TLS Secret is + operator-owned and remains until the operator deletes the Secret or its namespace. + +- **TLS is required by default.** The chart refuses to render the Ingress unless + `ingress.tls` references a certificate Secret or you explicitly opt in to cleartext HTTP. + When TLS is configured and `ingress.allowInsecureHttp` is `false`, the chart explicitly + sets `nginx.ingress.kubernetes.io/ssl-redirect: "true"` and overrides a conflicting value + in `ingress.annotations`. The scripts do not enable insecure HTTP during the normal workflow. + +Cleartext HTTP exposes the reusable Basic auth credential and completion traffic to interception on any network path that can observe the request. +Use the cleartext exception only after a firewall, VPN, or equivalent access control restricts the cluster nodes to trusted clients: + +```bash +ALLOW_INSECURE_HTTP=1 ./scripts/install-hpa.sh +``` + +`ALLOW_INSECURE_HTTP=1` acknowledges that Kubernetes cannot verify the surrounding network boundary. +Each script invocation also runs a Kubernetes exposure preflight before it sets `ingress.allowInsecureHttp=true`. +The preflight requires all of these conditions: + +- At least one cluster node has an `InternalIP` address. +- Every node `InternalIP` address is private, loopback, or link-local. +- No cluster node has an `ExternalIP` address. +- At least one managed ingress-nginx controller Service exists. +- Every matching controller Service uses `ClusterIP`, has no `externalIPs`, and has no entry in `.status.loadBalancer.ingress`. +- Managed ingress-nginx controller pods do not use `hostNetwork` or `hostPort`. + +These checks reject Kubernetes-reported exposure paths. +They do not prove that other hosts on a private network cannot reach the cluster. +If the preflight cannot verify every condition, the script exits before enabling cleartext and instructs you to configure TLS. +Set `ALLOW_INSECURE_HTTP=1` separately for each install, reset, recovery, or load-test command that must preserve cleartext operation. + +Do not expose the endpoint on a public network until you've confirmed both of the above are +configured the way you intend. + +NGINX selects a ready backend for each request. It cannot move an inference request that is already running when the HPA adds a new pod, so long-lived requests may still produce temporary utilization differences between GPUs. + +## Grafana workload allocation + +Grafana can compare request traffic and GPU utilization across the HPA replicas. Scraping of the agent `/metrics` endpoint is enabled by default (`metrics.serviceMonitor.enabled: true`), so no extra setup is needed after `install-hpa.sh`. + +Forward the Grafana Service: + +```bash +kubectl port-forward \ + -n monitoring \ + service/kube-prometheus-grafana \ + 3000:80 +``` + +Open `http://127.0.0.1:3000`. Get the login credentials when needed: + +```bash +kubectl get secret kube-prometheus-grafana -n monitoring \ + -o jsonpath='{.data.admin-user}' | base64 -d; echo + +kubectl get secret kube-prometheus-grafana -n monitoring \ + -o jsonpath='{.data.admin-password}' | base64 -d; echo +``` + +In Grafana: + +1. Select **Explore** in the left navigation. +2. Select the **Prometheus** data source. +3. Select **Code** at the upper-right of query row `A`. +4. Paste a PromQL query from below. +5. Select the blue **Run queries** button at the upper-right. +6. Use a time range such as **Last 15 minutes**. + +PromQL is entered in Grafana Explore, not in a shell. + +### GPU utilization by pod + +```promql +avg by (exported_pod) ( + DCGM_FI_DEV_GPU_UTIL{ + exported_namespace="nemoclaw-gpu", + exported_pod=~"nemoclaw-gpu-agent-.*" + } +) +``` + +### Successful inference requests by pod + +```promql +sum by (pod) ( + rate(nemoclaw_llm_requests_total{ + namespace="nemoclaw-gpu", + result="success" + }[5m]) +) +``` + +Add the request-rate and GPU-utilization queries to the same Explore view to see whether traffic and GPU work are distributed across replicas. When the HPA scales up, a new pod should appear after it becomes ready. A `rate(...)` result requires at least two Prometheus scrapes. + +If the request-rate graph shows only one pod (or stops updating) while GPU utilization shows all pods, check `kubectl get servicemonitor -n nemoclaw-gpu`. Anything that runs a plain `helm upgrade` without `--reuse-values` (custom scripts, manual re-installs) resets `metrics.serviceMonitor.enabled` to the chart default, which is `true`; if it was manually forced to `false` re-run `install-hpa.sh` or `helm upgrade` with `--set metrics.serviceMonitor.enabled=true` to restore it. + +## Uninstall + +Remove the NemoClaw release and namespace: + +```bash +helm uninstall nemoclaw-gpu -n nemoclaw-gpu +kubectl delete namespace nemoclaw-gpu --ignore-not-found +``` + +The Prometheus and Prometheus Adapter releases are shared monitoring components and are not removed by these commands. diff --git a/deploy/helm/gpu_autoscaling_k8s/files/agent-metrics.ts b/deploy/helm/gpu_autoscaling_k8s/files/agent-metrics.ts new file mode 100644 index 00000000000..4a83fca2bcb --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/files/agent-metrics.ts @@ -0,0 +1,89 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 +// +// Shared Prometheus helpers for agent /metrics (LLM latency, HTTP counters). + +const configuredLlmLatencyWindow = Number(process.env.LLM_LATENCY_WINDOW_SIZE ?? "128"); +const LLM_LATENCY_WINDOW = + Number.isSafeInteger(configuredLlmLatencyWindow) && configuredLlmLatencyWindow > 0 + ? Math.min(configuredLlmLatencyWindow, 10_000) + : 128; +const llmDurationsMs = []; +let llmDurationSumSec = 0; +let llmDurationCount = 0; +let llmRequestsOk = 0; +let llmRequestsError = 0; +const llmHistogramBucketsSec = [0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120, 300]; +const llmHistogramCounts = Array.from({ length: llmHistogramBucketsSec.length + 1 }, () => 0); + +export function recordLlmLatency(durationMs, ok) { + // Normalize once so the rolling window (p50/p95/avg) and the cumulative + // counters/histogram below always agree on the same finite, non-negative value. + const normalizedMs = Number.isFinite(durationMs) ? Math.max(0, durationMs) : 0; + const sec = normalizedMs / 1000; + llmDurationSumSec += sec; + llmDurationCount += 1; + if (ok) llmRequestsOk += 1; + else llmRequestsError += 1; + + llmDurationsMs.push(normalizedMs); + if (llmDurationsMs.length > LLM_LATENCY_WINDOW) llmDurationsMs.shift(); + + let bucketIdx = llmHistogramBucketsSec.findIndex((bound) => sec <= bound); + if (bucketIdx === -1) bucketIdx = llmHistogramBucketsSec.length; + for (let i = bucketIdx; i < llmHistogramCounts.length; i += 1) { + llmHistogramCounts[i] += 1; + } +} + +function percentileMs(sorted, p) { + if (!sorted.length) return 0; + const idx = Math.ceil(sorted.length * p) - 1; + return sorted[Math.max(0, idx)]; +} + +function llmLatencySnapshotMs() { + if (!llmDurationsMs.length) { + return { p50: 0, p95: 0, avg: 0 }; + } + const sorted = [...llmDurationsMs].sort((a, b) => a - b); + const sum = sorted.reduce((acc, v) => acc + v, 0); + return { + p50: percentileMs(sorted, 0.5), + p95: percentileMs(sorted, 0.95), + avg: sum / sorted.length, + }; +} + +export function llmMetricsLines() { + const { p50, p95, avg } = llmLatencySnapshotMs(); + const lines = [ + "# HELP nemoclaw_llm_requests_total Chat/completions proxied to inference backend", + "# TYPE nemoclaw_llm_requests_total counter", + `nemoclaw_llm_requests_total{result="success"} ${llmRequestsOk}`, + `nemoclaw_llm_requests_total{result="error"} ${llmRequestsError}`, + "# HELP nemoclaw_llm_request_duration_seconds LLM chat/completions end-to-end proxy latency", + "# TYPE nemoclaw_llm_request_duration_seconds histogram", + ]; + + for (let i = 0; i < llmHistogramBucketsSec.length; i += 1) { + lines.push( + `nemoclaw_llm_request_duration_seconds_bucket{le="${llmHistogramBucketsSec[i]}"} ${llmHistogramCounts[i]}`, + ); + } + lines.push( + `nemoclaw_llm_request_duration_seconds_bucket{le="+Inf"} ${llmHistogramCounts[llmHistogramCounts.length - 1]}`, + `nemoclaw_llm_request_duration_seconds_sum ${llmDurationSumSec}`, + `nemoclaw_llm_request_duration_seconds_count ${llmDurationCount}`, + "# HELP nemoclaw_llm_latency_p50_milliseconds Rolling p50 LLM latency (recent window)", + "# TYPE nemoclaw_llm_latency_p50_milliseconds gauge", + `nemoclaw_llm_latency_p50_milliseconds ${Math.round(p50)}`, + "# HELP nemoclaw_llm_latency_p95_milliseconds Rolling p95 LLM latency (recent window)", + "# TYPE nemoclaw_llm_latency_p95_milliseconds gauge", + `nemoclaw_llm_latency_p95_milliseconds ${Math.round(p95)}`, + "# HELP nemoclaw_llm_latency_avg_milliseconds Rolling average LLM latency (recent window)", + "# TYPE nemoclaw_llm_latency_avg_milliseconds gauge", + `nemoclaw_llm_latency_avg_milliseconds ${Math.round(avg)}`, + ); + return lines; +} diff --git a/deploy/helm/gpu_autoscaling_k8s/files/agent-server.ts b/deploy/helm/gpu_autoscaling_k8s/files/agent-server.ts new file mode 100755 index 00000000000..8d9b08c1986 --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/files/agent-server.ts @@ -0,0 +1,254 @@ +#!/usr/bin/env node +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 +// +// GPU agent pod: health + Prometheus metrics + OpenAI-compatible proxy to local Ollama. + +import http from "node:http"; +import { Readable } from "node:stream"; +import { pipeline } from "node:stream/promises"; +import { llmMetricsLines, recordLlmLatency } from "./agent-metrics.ts"; + +const PORT = Number(process.env.PORT || 8081); +const BASE_URL = (process.env.INFERENCE_BASE_URL || "http://127.0.0.1:11434/v1").replace(/\/$/, ""); +const OLLAMA_BASE = (process.env.OLLAMA_BASE_URL || "http://127.0.0.1:11434").replace(/\/$/, ""); +const MODEL = process.env.INFERENCE_MODEL || ""; +// Bound the unauthenticated proxy request path: cap buffered body size and time-to-complete +// so a large or never-ending request body cannot exhaust pod memory or hold connections open. +const MAX_BODY_BYTES = Number(process.env.MAX_BODY_BYTES || 2 * 1024 * 1024); +const REQUEST_BODY_TIMEOUT_MS = Number(process.env.REQUEST_BODY_TIMEOUT_MS || 30_000); + +class PayloadTooLargeError extends Error {} +class RequestBodyTimeoutError extends Error {} + +let inflight = 0; +let totalRequests = 0; +let inferenceReachable = 0; +let inferenceCache = { ok: false, at: 0 }; +const INFERENCE_CACHE_MS = Number(process.env.INFERENCE_READY_CACHE_MS || 15_000); +let inferenceReadyEver = false; +let inferenceFailStreak = 0; +const INFERENCE_FAIL_MAX = Number(process.env.INFERENCE_FAIL_MAX || 8); + +function readBody(req) { + return new Promise((resolve, reject) => { + const chunks = []; + let size = 0; + let done = false; + const onData = (c) => { + size += c.length; + if (size > MAX_BODY_BYTES) { + finish(reject, new PayloadTooLargeError("request body too large")); + return; + } + chunks.push(c); + }; + const onEnd = () => finish(resolve, Buffer.concat(chunks).toString("utf8")); + const onError = (err) => finish(reject, err); + const timer = setTimeout(() => { + finish(reject, new RequestBodyTimeoutError("request body timeout")); + }, REQUEST_BODY_TIMEOUT_MS); + function finish(fn, arg) { + if (done) return; + done = true; + clearTimeout(timer); + req.off("data", onData); + req.off("end", onEnd); + req.off("error", onError); + // Stop reading further bytes from an oversized/stalled request, but leave the + // socket itself open so the caller below can still write a clean HTTP response + // (destroying it here would reset the connection before the response flushes). + req.pause(); + fn(arg); + } + req.on("data", onData); + req.on("end", onEnd); + req.on("error", onError); + }); +} + +async function proxyChatCompletions(req, res) { + let raw; + try { + raw = await readBody(req); + } catch (err) { + if (err instanceof PayloadTooLargeError) { + res.writeHead(413, { "content-type": "text/plain", Connection: "close" }); + res.end("payload too large\n", () => req.destroy()); + } else if (err instanceof RequestBodyTimeoutError) { + res.writeHead(408, { "content-type": "text/plain", Connection: "close" }); + res.end("request timeout\n", () => req.destroy()); + } else { + res.writeHead(400, { "content-type": "text/plain" }); + res.end("bad request\n"); + } + return; + } + let body; + try { + body = raw ? JSON.parse(raw) : {}; + } catch { + res.writeHead(400, { "content-type": "text/plain" }); + res.end("invalid json\n"); + return; + } + if (!body.model) body.model = MODEL; + const llmStart = performance.now(); + let llmOk = false; + try { + const hubRes = await fetch(`${BASE_URL}/chat/completions`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify(body), + signal: AbortSignal.timeout(300_000), + }); + llmOk = hubRes.ok; + // Pipe the upstream body straight through (don't buffer with .text()) so + // "stream": true chat-completions reach the client incrementally, and forward + // its real content-type instead of forcing application/json on SSE responses. + const contentType = hubRes.headers.get("content-type") || "application/json"; + res.writeHead(hubRes.status, { "content-type": contentType }); + if (hubRes.body) { + await pipeline(Readable.fromWeb(hubRes.body), res); + } else { + res.end(); + } + } catch (err) { + // Log the full error server-side only; the client gets a generic message so + // internal details (upstream host/port, stack trace) never leave the pod. + console.error("chat completion proxy error:", err); + if (!res.headersSent) { + res.writeHead(502, { "content-type": "application/json" }); + res.end(JSON.stringify({ error: "upstream inference request failed" })); + } else { + res.destroy(); + } + } finally { + recordLlmLatency(performance.now() - llmStart, llmOk); + } +} + +async function checkInference() { + const now = Date.now(); + if (now - inferenceCache.at < INFERENCE_CACHE_MS) return inferenceCache.ok; + try { + const res = await fetch(`${OLLAMA_BASE}/api/tags`, { + signal: AbortSignal.timeout(10_000), + }); + if (!res.ok) { + inferenceCache = { ok: false, at: now }; + return false; + } + const data = await res.json(); + const names = (data.models || []).map((m) => m.name || m.model || ""); + const ok = MODEL + ? names.some( + (name) => + name === MODEL || + (!MODEL.includes(":") && name.startsWith(`${MODEL}:`)), + ) + : names.length > 0; + if (ok) { + inferenceReadyEver = true; + inferenceFailStreak = 0; + inferenceCache = { ok: true, at: now }; + return true; + } + inferenceFailStreak += 1; + if (inferenceReadyEver && (inflight > 0 || inferenceFailStreak < INFERENCE_FAIL_MAX)) { + inferenceCache = { ok: true, at: now }; + return true; + } + inferenceCache = { ok: false, at: now }; + return false; + } catch { + inferenceFailStreak += 1; + if (inferenceReadyEver && (inflight > 0 || inferenceFailStreak < INFERENCE_FAIL_MAX)) { + inferenceCache = { ok: true, at: now }; + return true; + } + inferenceCache = { ok: false, at: now }; + return false; + } +} + +function metricsText() { + return [ + "# HELP nemoclaw_http_requests_total Total HTTP requests to agent pod", + "# TYPE nemoclaw_http_requests_total counter", + `nemoclaw_http_requests_total ${totalRequests}`, + "# HELP nemoclaw_http_inflight_requests In-flight HTTP requests", + "# TYPE nemoclaw_http_inflight_requests gauge", + `nemoclaw_http_inflight_requests ${inflight}`, + "# HELP nemoclaw_inference_reachable 1 if local Ollama model is ready", + "# TYPE nemoclaw_inference_reachable gauge", + `nemoclaw_inference_reachable ${inferenceReachable}`, + ...llmMetricsLines(), + "", + ].join("\n"); +} + +// Defense-in-depth against slow/never-ending requests, independent of the per-request +// body cap enforced in readBody(). headersTimeout must stay <= requestTimeout (Node requires it). +const REQUEST_TIMEOUT_MS = REQUEST_BODY_TIMEOUT_MS + 5_000; +const HEADERS_TIMEOUT_MS = Math.min(10_000, REQUEST_TIMEOUT_MS); + +const server = http.createServer( + { + requestTimeout: REQUEST_TIMEOUT_MS, + headersTimeout: HEADERS_TIMEOUT_MS, + }, + async (req, res) => { + totalRequests += 1; + inflight += 1; + try { + if (req.url === "/healthz" || req.url === "/health") { + res.writeHead(200, { "content-type": "text/plain" }); + res.end("ok\n"); + return; + } + if (req.url === "/readyz" || req.url === "/ready") { + const ok = await checkInference(); + inferenceReachable = ok ? 1 : 0; + res.writeHead(ok ? 200 : 503, { "content-type": "text/plain" }); + res.end(ok ? "ready\n" : "ollama model not ready\n"); + return; + } + if (req.url === "/metrics") { + res.writeHead(200, { "content-type": "text/plain; version=0.0.4" }); + res.end(metricsText()); + return; + } + const pathOnly = (req.url || "").split("?")[0]; + if ( + (pathOnly === "/v1/chat/completions" || pathOnly === "/chat/completions") && + req.method === "POST" + ) { + await proxyChatCompletions(req, res); + return; + } + if (req.url === "/" && req.method === "GET") { + res.writeHead(200, { "content-type": "application/json" }); + res.end( + JSON.stringify({ + service: "nemoclaw-gpu-agent", + model: MODEL, + inferenceBaseUrl: BASE_URL, + ollamaBaseUrl: OLLAMA_BASE, + endpoints: ["/healthz", "/readyz", "/metrics", "POST /v1/chat/completions"], + note: "Local Ollama on GPU; scale replicas with kubectl or HPA (one pod per GPU)", + }), + ); + return; + } + res.writeHead(404); + res.end("not found\n"); + } finally { + inflight -= 1; + } + }, +); + +server.listen(PORT, () => { + console.log(`nemoclaw-gpu-agent listening on :${PORT} model=${MODEL}`); +}); diff --git a/deploy/helm/gpu_autoscaling_k8s/files/load-generator.ts b/deploy/helm/gpu_autoscaling_k8s/files/load-generator.ts new file mode 100755 index 00000000000..fdf60983e26 --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/files/load-generator.ts @@ -0,0 +1,636 @@ +#!/usr/bin/env node +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 +// +// Drive GPU utilization for HPA: chat completions directly to each agent pod IP. +// Each Running agent pod gets PER_POD_PEAK × compensation concurrent requests. +// Compensation = HPA currentReplicas / loadTargetCount so cold pods at 0% GPU +// do not drag the average to ~42% while a new replica is starting. + +import fs from "node:fs"; +import https from "node:https"; +import { setTimeout as sleep } from "node:timers/promises"; + +const DURATION_SEC = Number(process.env.DURATION_SEC || 720); +const TARGET_PODS = Number(process.env.TARGET_PODS || 4); +const HPA_TARGET_GPU = Number(process.env.HPA_TARGET_GPU || 40); +const JOB_PARALLELISM = Number(process.env.JOB_PARALLELISM || 1); +const AGENT_PORT = Number(process.env.AGENT_PORT || 8081); +const INFLIGHT_PER_GPU = Number(process.env.INFLIGHT_PER_GPU || 384); +const LOAD_MULTIPLIER = Number(process.env.LOAD_MULTIPLIER || 1); +const PER_POD_PEAK = INFLIGHT_PER_GPU * LOAD_MULTIPLIER; +const RAMP_SEC = Number(process.env.RAMP_SEC || 60); +const REQUEST_TIMEOUT_MS = Number(process.env.REQUEST_TIMEOUT_MS || 300_000); +const MAX_TOKENS = Number(process.env.MAX_TOKENS || 512); +const LOG_EVERY_SEC = Number(process.env.LOG_EVERY_SEC || 15); +const TARGET_POLL_SEC = Number(process.env.TARGET_POLL_SEC || 1); +const K8S_NAMESPACE = process.env.K8S_NAMESPACE || "nemoclaw-gpu"; +const AGENT_SERVICE = process.env.AGENT_SERVICE || "nemoclaw-gpu-agent"; +const HPA_NAME = process.env.HPA_NAME || AGENT_SERVICE; +const AGENT_LABEL_SELECTOR = + process.env.AGENT_LABEL_SELECTOR || "app.kubernetes.io/name=nemoclaw-gpu,component=gpu-agent"; +const ESCALATE_INTERVAL_SEC = Number(process.env.ESCALATE_INTERVAL_SEC || 10); +const ESCALATE_FACTOR = Number(process.env.ESCALATE_FACTOR || 0.5); +const ESCALATE_MAX_MULT = Number(process.env.ESCALATE_MAX_MULT || 3); +const REQUEST_RETRIES = Number(process.env.REQUEST_RETRIES || 2); +const LOAD_COMPENSATION_SAFETY = Number(process.env.LOAD_COMPENSATION_SAFETY || 3); +const MAX_COMPENSATION = Number(process.env.MAX_COMPENSATION || 16); +const NEW_POD_RAMP_SEC = Number(process.env.NEW_POD_RAMP_SEC || 0); +const MAX_INFLIGHT_PER_POD = Number(process.env.MAX_INFLIGHT_PER_POD || 6144); +const WARMUP_SEC = Number(process.env.WARMUP_SEC || 45); +const ERROR_BACKOFF_FACTOR = Number(process.env.ERROR_BACKOFF_FACTOR || 0.92); +const ERROR_BACKOFF_MIN = Number(process.env.ERROR_BACKOFF_MIN || 0.4); +const ERROR_BACKOFF_RECOVERY = Number(process.env.ERROR_BACKOFF_RECOVERY || 1.15); +const BOOTSTRAP_INFLIGHT = Number(process.env.BOOTSTRAP_INFLIGHT || 4); +const NEW_POD_WARMUP_PARALLEL = Number(process.env.NEW_POD_WARMUP_PARALLEL || 8); +const NEW_POD_WARMUP_MAX_SEC = Number(process.env.NEW_POD_WARMUP_MAX_SEC || 120); +const CIRCUIT_BREAKER_BACKOFF = Number(process.env.CIRCUIT_BREAKER_BACKOFF || 0.15); +const MIN_INFLIGHT_FLOOR = Number(process.env.MIN_INFLIGHT_FLOOR || 8); +const MIN_RECOVERY_INFLIGHT = Number(process.env.MIN_RECOVERY_INFLIGHT || 4); +const READYZ_GRACE_SEC = Number(process.env.READYZ_GRACE_SEC || 45); +const REQUIRE_CHAT_PROBE = + process.env.REQUIRE_CHAT_PROBE === "1" || process.env.REQUIRE_CHAT_PROBE === "true"; +const PROBE_CHAT_TIMEOUT_MS = Number(process.env.PROBE_CHAT_TIMEOUT_MS || 30_000); + +let podTargets = []; +let podCandidates = []; +let hpaReplicas = 1; +let hpaDesired = 1; +let loadCompensation = 1; +let lastTargetPoll = 0; +const podFirstSeen = new Map(); +const targetBackoff = new Map(); +const targetChatOk = new Set(); +const warmInFlight = new Set(); +const readyzLastOk = new Map(); + +function loadQuestions() { + try { + const lines = fs + .readFileSync(process.env.QUESTIONS_FILE || "/questions/questions.txt", "utf8") + .split("\n") + .map((l) => l.trim()) + .filter(Boolean); + if (lines.length) return lines; + } catch { + /* use fallback */ + } + return [ + "Explain Kubernetes HPA and GPU autoscaling in detail with examples.", + "Write a long summary of transformer inference on NVIDIA GPUs.", + "Describe how Ollama serves models and batches concurrent chat requests.", + ]; +} + +function k8sGet(path) { + const tokenPath = "/var/run/secrets/kubernetes.io/serviceaccount/token"; + const caPath = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"; + if (!fs.existsSync(tokenPath) || !process.env.KUBERNETES_SERVICE_HOST) { + return Promise.resolve(null); + } + // Standard in-cluster Kubernetes auth: the projected service-account token/CA are + // not attacker-controllable, and sending them to the API server is the intended use. + const token = fs.readFileSync(tokenPath, "utf8"); + const ca = fs.readFileSync(caPath); + return new Promise((resolve) => { + // codeql[js/file-access-to-http] -- in-cluster K8s API auth: token/ca come from the + // kubelet-projected service-account files and are sent only to KUBERNETES_SERVICE_HOST. + const req = https.request( + { + hostname: process.env.KUBERNETES_SERVICE_HOST, + port: process.env.KUBERNETES_SERVICE_PORT || 443, + path, + method: "GET", + headers: { Authorization: `Bearer ${token}` }, + ca, + rejectUnauthorized: true, + }, + (res) => { + let data = ""; + res.on("data", (chunk) => { + data += chunk; + }); + res.on("end", () => { + if (res.statusCode && res.statusCode >= 200 && res.statusCode < 300) { + try { + resolve(JSON.parse(data)); + } catch { + resolve(null); + } + } else { + resolve(null); + } + }); + }, + ); + req.on("error", () => resolve(null)); + req.setTimeout(5000, () => { + req.destroy(); + resolve(null); + }); + req.end(); + }); +} + +function ipsFromEndpointSliceList(list) { + const ips = new Set(); + for (const slice of list?.items || []) { + for (const ep of slice.endpoints || []) { + if (ep.conditions?.ready === false) continue; + for (const addr of ep.addresses || []) { + if (addr) ips.add(addr); + } + } + } + return ips; +} + +function ipsFromRunningPods(list) { + const ips = new Set(); + for (const pod of list?.items || []) { + if (pod.status?.phase !== "Running") continue; + const ip = pod.status?.podIP; + if (ip) ips.add(ip); + } + return ips; +} + +async function pollHpaReplicas() { + const hpa = await k8sGet( + `/apis/autoscaling/v2/namespaces/${K8S_NAMESPACE}/horizontalpodautoscalers/${HPA_NAME}`, + ); + if (hpa?.status?.currentReplicas >= 1) { + hpaReplicas = hpa.status.currentReplicas; + } + if (hpa?.status?.desiredReplicas >= 1) { + hpaDesired = hpa.status.desiredReplicas; + } + return hpaReplicas; +} + +async function probeInferenceReady(target) { + const ip = podIpFromTarget(target); + try { + const res = await fetch(`${target}/readyz`, { + signal: AbortSignal.timeout(10_000), + }); + if (res.ok) { + readyzLastOk.set(ip, Date.now()); + return true; + } + } catch { + /* grace below */ + } + const last = readyzLastOk.get(ip); + if (last && (Date.now() - last) / 1000 < READYZ_GRACE_SEC) return true; + return false; +} + +async function pollAgentPodTargets() { + const now = Date.now(); + if (now - lastTargetPoll < TARGET_POLL_SEC * 1000) { + return podTargets; + } + lastTargetPoll = now; + + await pollHpaReplicas(); + + const ips = new Set(); + + const sliceList = await k8sGet( + `/apis/discovery.k8s.io/v1/namespaces/${K8S_NAMESPACE}/endpointslices?labelSelector=${encodeURIComponent(`kubernetes.io/service-name=${AGENT_SERVICE}`)}`, + ); + for (const ip of ipsFromEndpointSliceList(sliceList)) ips.add(ip); + + const podList = await k8sGet( + `/api/v1/namespaces/${K8S_NAMESPACE}/pods?labelSelector=${encodeURIComponent(AGENT_LABEL_SELECTOR)}`, + ); + for (const ip of ipsFromRunningPods(podList)) ips.add(ip); + + if (ips.size) { + const nowMs = Date.now(); + for (const ip of ips) { + if (!podFirstSeen.has(ip)) { + podFirstSeen.set(ip, nowMs); + console.log(JSON.stringify({ event: "newPodDiscovered", ip })); + } + } + const candidates = [...ips].map((ip) => `http://${ip}:${AGENT_PORT}`); + podCandidates = candidates; + const ready = []; + await Promise.all( + candidates.map(async (target) => { + if (await probeInferenceReady(target)) ready.push(target); + }), + ); + podTargets = ready; + for (const target of podCandidates) scheduleWarmTarget(target); + } else { + podCandidates = []; + podTargets = []; + } + + // Compensate for the target pods not yet represented by candidate IPs. + const readyCount = Math.max(1, podCandidates.length); + const hpaCount = Math.max(hpaReplicas, hpaDesired, readyCount, 1); + if (readyCount >= TARGET_PODS) { + loadCompensation = 1; + } else if (readyCount >= hpaCount) { + // Spread the target-pod load across the candidates that are already available. + loadCompensation = Math.min(MAX_COMPENSATION, TARGET_PODS / readyCount); + } else { + // Add a safety multiplier while HPA replicas are still missing candidate IPs. + loadCompensation = Math.min( + MAX_COMPENSATION, + Math.max(1, (TARGET_PODS / readyCount) * LOAD_COMPENSATION_SAFETY), + ); + } + + return podTargets; +} + +function podIpFromTarget(target) { + return target.match(/^http:\/\/([^:/]+)/)?.[1] || target; +} + +function newPodRampMultiplier(ip) { + if (NEW_POD_RAMP_SEC <= 0) return 1; + const seenAt = podFirstSeen.get(ip); + if (!seenAt) return 1; + const ageSec = (Date.now() - seenAt) / 1000; + if (ageSec >= NEW_POD_RAMP_SEC) return 1; + return 0.75 + 0.25 * (ageSec / NEW_POD_RAMP_SEC); +} + +function getTargetBackoff(ip) { + return targetBackoff.get(ip) ?? 1; +} + +function noteTargetResult(ip, ok) { + const cur = targetBackoff.get(ip) ?? 1; + if (!ok) { + targetBackoff.set(ip, Math.max(ERROR_BACKOFF_MIN, cur * ERROR_BACKOFF_FACTOR)); + } else if (cur < 1) { + targetBackoff.set(ip, Math.min(1, cur * ERROR_BACKOFF_RECOVERY)); + } +} + +function activeReplicaCount() { + return Math.max(hpaReplicas, hpaDesired, podCandidates.length, podTargets.length, 1); +} + +function strugglingTargetCount() { + let n = 0; + for (const ip of podFirstSeen.keys()) { + if (getTargetBackoff(ip) <= CIRCUIT_BREAKER_BACKOFF) n += 1; + } + return n; +} + +function healthyBoostMultiplier(ip) { + const n = activeReplicaCount(); + if (n < 2) return 1; + const backoff = getTargetBackoff(ip); + if (backoff <= CIRCUIT_BREAKER_BACKOFF) return 1; + const struggling = strugglingTargetCount(); + if (struggling <= 0) return 1; + return Math.min(2, 1 + struggling / Math.max(1, targetChatOk.size)); +} + +async function probeChatWorks(target) { + try { + const res = await fetch(`${target}/v1/chat/completions`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ + messages: [{ role: "user", content: "Say OK." }], + max_tokens: 8, + stream: false, + }), + signal: AbortSignal.timeout(PROBE_CHAT_TIMEOUT_MS), + }); + if (!res.ok) return false; + await res.json(); + return true; + } catch { + return false; + } +} + +async function requirePodTargets(deadlineMs) { + while (Date.now() < deadlineMs) { + await pollAgentPodTargets(); + const targets = podCandidates.length ? podCandidates : podTargets; + if (targets.length === 0) { + await sleep(1000); + continue; + } + if (!REQUIRE_CHAT_PROBE) { + console.log( + JSON.stringify({ + event: "targetsReady", + loadTargets: targets, + readyzOk: podTargets.length, + }), + ); + return targets; + } + for (const target of podTargets.length ? podTargets : targets) { + if (await probeChatWorks(target)) { + targetChatOk.add(target); + console.log(JSON.stringify({ event: "chatProbeOk", target })); + console.log(JSON.stringify({ event: "targetsReady", loadTargets: podTargets })); + return podTargets; + } + } + console.log(JSON.stringify({ event: "waitingForChatProbe", loadTargets: podTargets })); + await sleep(3000); + } + console.error( + REQUIRE_CHAT_PROBE + ? "FATAL: agent pod(s) found but chat probe never succeeded — wait for Ollama model pull" + : "FATAL: no ready agent pod IPs — check RBAC (pods/endpointslices) and agent pods", + ); + process.exit(1); +} + +function inflightFloor(multiReplica) { + if (multiReplica) return Math.max(MIN_INFLIGHT_FLOOR, 12); + return Math.max(MIN_INFLIGHT_FLOOR, MIN_RECOVERY_INFLIGHT); +} + +function escalationCap() { + let cap = ESCALATE_MAX_MULT; + const hpaN = Math.max(hpaReplicas, hpaDesired); + if (hpaN >= 2 && targetChatOk.size < hpaN) { + cap = Math.min(cap, 1.25); + } + return cap; +} + +function rampMultiplier(elapsedSec) { + const cap = escalationCap(); + if (RAMP_SEC <= 0) { + const steps = Math.floor(elapsedSec / ESCALATE_INTERVAL_SEC); + return Math.min(cap, 1 + steps * ESCALATE_FACTOR); + } + if (elapsedSec < RAMP_SEC) { + const progress = elapsedSec / RAMP_SEC; + return Math.min(cap, 0.75 + 0.25 * progress); + } + const steps = Math.floor((elapsedSec - RAMP_SEC) / ESCALATE_INTERVAL_SEC); + return Math.min(cap, 1 + steps * ESCALATE_FACTOR); +} + +function warmupCompensationScale(startedAt) { + const elapsed = (Date.now() - startedAt) / 1000; + if (WARMUP_SEC <= 0 || elapsed >= WARMUP_SEC) return 1; + return 0.35 + 0.65 * (elapsed / WARMUP_SEC); +} + +function effectiveCompensation(startedAt) { + const warm = warmupCompensationScale(startedAt); + return 1 + (loadCompensation - 1) * warm; +} + +function baseInflightPerPodPerGenerator(startedAt, stats) { + const mult = rampMultiplier((Date.now() - startedAt) / 1000); + const comp = effectiveCompensation(startedAt); + let raw = Math.ceil((PER_POD_PEAK * mult * comp) / JOB_PARALLELISM); + const total = (stats?.chat ?? 0) + (stats?.fail ?? 0); + if (total >= 40) { + const failRate = (stats?.fail ?? 0) / total; + if (failRate > 0.6) raw = Math.ceil(raw * 0.25); + else if (failRate > 0.3) raw = Math.ceil(raw * 0.5); + else if (failRate > 0.15) raw = Math.ceil(raw * 0.75); + } + const cap = Math.max(2, Math.ceil(MAX_INFLIGHT_PER_POD / JOB_PARALLELISM)); + const floor = inflightFloor(activeReplicaCount() >= 2); + return Math.max(floor, Math.min(raw, cap)); +} + +function inflightForTarget(target, startedAt, inferenceReady = true, stats = null) { + const ip = podIpFromTarget(target); + const backoff = getTargetBackoff(ip); + const multiReplica = activeReplicaCount() >= 2; + const floor = inflightFloor(multiReplica || targetChatOk.has(target)); + + if (backoff <= CIRCUIT_BREAKER_BACKOFF) { + return Math.max(floor, MIN_RECOVERY_INFLIGHT); + } + if (!inferenceReady) { + return Math.max(floor, BOOTSTRAP_INFLIGHT); + } + if (!targetChatOk.has(target)) { + const ageSec = (Date.now() - (podFirstSeen.get(ip) || Date.now())) / 1000; + if (ageSec > NEW_POD_WARMUP_MAX_SEC) { + return Math.max( + floor, + BOOTSTRAP_INFLIGHT, + Math.ceil(baseInflightPerPodPerGenerator(startedAt, stats) * 0.25), + ); + } + return Math.max(floor, BOOTSTRAP_INFLIGHT); + } + const base = baseInflightPerPodPerGenerator(startedAt, stats); + let limit = Math.ceil(base * newPodRampMultiplier(ip) * backoff * healthyBoostMultiplier(ip)); + return Math.max(floor, limit); +} + +async function warmTarget(target) { + if (targetChatOk.has(target)) return true; + const probes = []; + for (let i = 0; i < NEW_POD_WARMUP_PARALLEL; i++) { + probes.push(probeChatWorks(target)); + } + const ok = (await Promise.all(probes)).some(Boolean); + if (ok) { + targetChatOk.add(target); + console.log(JSON.stringify({ event: "podWarmed", target })); + } + return ok; +} + +function scheduleWarmTarget(target) { + if (targetChatOk.has(target) || warmInFlight.has(target)) return; + warmInFlight.add(target); + warmTarget(target) + .catch(() => false) + .finally(() => warmInFlight.delete(target)); +} + +async function ask(target, questions, stats) { + const ip = podIpFromTarget(target); + // `questions` comes from the bundled sample file (or QUESTIONS_FILE override) — this + // is the load generator's synthetic workload payload by design, not attacker input. + const q = questions[Math.floor(Math.random() * questions.length)]; + let lastErr; + for (let attempt = 0; attempt <= REQUEST_RETRIES; attempt++) { + try { + // codeql[js/file-access-to-http] -- `q` is the synthetic sample-question payload + // this load generator bundles/reads by design, not attacker-controlled file data. + const res = await fetch(`${target}/v1/chat/completions`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ + messages: [{ role: "user", content: q }], + max_tokens: MAX_TOKENS, + stream: false, + }), + signal: AbortSignal.timeout(REQUEST_TIMEOUT_MS), + }); + if (!res.ok) throw new Error(`chat HTTP ${res.status}`); + await res.json(); + stats.chat += 1; + targetChatOk.add(target); + noteTargetResult(ip, true); + return; + } catch (err) { + lastErr = err; + if (attempt < REQUEST_RETRIES) await sleep(100 * (attempt + 1)); + } + } + noteTargetResult(ip, false); + throw lastErr; +} + +const targetWorkers = new Map(); +const workerPromises = new Map(); + +async function runTargetWorker(target, questions, endAt, stats) { + const state = { limit: 2, tasks: new Set() }; + targetWorkers.set(target, state); + + while (Date.now() < endAt) { + while (state.tasks.size < state.limit && Date.now() < endAt) { + const p = ask(target, questions, stats) + .catch((err) => { + stats.fail += 1; + if (stats.fail <= 10 || stats.fail % 100 === 0) { + console.error(`[gpu-load] ${target} ${err.message}`); + } + }) + .finally(() => state.tasks.delete(p)); + state.tasks.add(p); + } + if (state.tasks.size > 0) await Promise.race(state.tasks); + else await sleep(20); + } + await Promise.all(state.tasks); + targetWorkers.delete(target); +} + +function syncTargetWorkers(targets, startedAt, stats) { + const readySet = new Set(podTargets); + const active = new Set(targets); + for (const target of active) { + scheduleWarmTarget(target); + const inferenceReady = readySet.has(target); + const limit = inflightForTarget(target, startedAt, inferenceReady, stats); + const worker = targetWorkers.get(target); + if (worker) worker.limit = limit; + } + for (const target of [...targetWorkers.keys()]) { + if (!active.has(target)) { + const worker = targetWorkers.get(target); + if (worker) worker.limit = 0; + } + } +} + +async function main() { + const questions = loadQuestions(); + const startedAt = Date.now(); + const endAt = startedAt + DURATION_SEC * 1000; + const stats = { chat: 0, fail: 0 }; + let lastLog = startedAt; + let lastUnevenLog = 0; + + console.log( + JSON.stringify({ + targetPods: TARGET_PODS, + hpaTargetGpu: HPA_TARGET_GPU, + jobParallelism: JOB_PARALLELISM, + perPodPeak: PER_POD_PEAK, + loadMultiplier: LOAD_MULTIPLIER, + loadCompensationSafety: LOAD_COMPENSATION_SAFETY, + maxTokens: MAX_TOKENS, + rampSec: RAMP_SEC, + durationSec: DURATION_SEC, + loadModel: "direct-pod-IP saturation (multi-replica floor, per-target backoff, no full idle)", + maxInflightPerPod: MAX_INFLIGHT_PER_POD, + escalateMaxMult: ESCALATE_MAX_MULT, + }), + ); + + await requirePodTargets(startedAt + 90_000); + + while (Date.now() < endAt) { + await pollAgentPodTargets(); + const targets = podCandidates.length ? podCandidates : podTargets; + if (!targets.length) { + await sleep(1000); + continue; + } + + if (hpaReplicas > podTargets.length && Date.now() - lastUnevenLog >= 30_000) { + console.log( + JSON.stringify({ + event: "unevenReplicas", + hpaReplicas, + hpaDesired, + readyLoadTargets: podTargets.length, + candidatePods: targets.length, + warmedPods: targetChatOk.size, + message: "HPA has more replicas than warmed GPUs — bootstrapping new pods", + }), + ); + lastUnevenLog = Date.now(); + } + + for (const target of targets) { + if (!workerPromises.has(target)) { + workerPromises.set(target, runTargetWorker(target, questions, endAt, stats)); + } + } + syncTargetWorkers(targets, startedAt, stats); + + if (Date.now() - lastLog >= LOG_EVERY_SEC * 1000) { + const sampleTarget = podTargets[0] || targets[0]; + const sampleLimit = sampleTarget + ? inflightForTarget(sampleTarget, startedAt, podTargets.includes(sampleTarget), stats) + : 0; + console.log( + JSON.stringify({ + event: "progress", + hpaReplicas, + hpaDesired, + loadTargets: podTargets.length, + candidatePods: targets.length, + warmedPods: targetChatOk.size, + loadCompensation, + effectiveCompensation: effectiveCompensation(startedAt), + inflightPerPodPerGenerator: sampleLimit, + perPodClusterInflight: sampleLimit * JOB_PARALLELISM, + sampleTarget, + chat: stats.chat, + fail: stats.fail, + elapsedSec: Math.round((Date.now() - startedAt) / 1000), + }), + ); + lastLog = Date.now(); + } + + await sleep(1000); + } + + await Promise.all([...workerPromises.values()]); + + console.log(`done chat=${stats.chat} fail=${stats.fail} lastPodCount=${podTargets.length}`); + process.exit(0); +} + +main().catch((e) => { + console.error(e); + process.exit(1); +}); diff --git a/deploy/helm/gpu_autoscaling_k8s/files/ollama-start.sh b/deploy/helm/gpu_autoscaling_k8s/files/ollama-start.sh new file mode 100755 index 00000000000..df5a3f5ca6a --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/files/ollama-start.sh @@ -0,0 +1,34 @@ +#!/bin/sh +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +set -eu + +MODEL="${OLLAMA_MODEL:?OLLAMA_MODEL required}" +export OLLAMA_HOST="${OLLAMA_HOST:-0.0.0.0:11434}" + +ollama serve & +SERVE_PID=$! + +cleanup() { + kill "${SERVE_PID}" 2>/dev/null || true +} +trap cleanup EXIT INT TERM + +echo "Waiting for Ollama API..." +for _ in $(seq 1 120); do + if ollama list >/dev/null 2>&1; then + break + fi + sleep 2 +done + +# Pull only when the model is not already in /root/.ollama (PVC, hostPath, or emptyDir). +if ollama show "${MODEL}" >/dev/null 2>&1; then + echo "Model ${MODEL} already present — skipping pull" +else + echo "Pulling model ${MODEL} (first time on this volume; may take several minutes)..." + ollama pull "${MODEL}" +fi + +echo "Ollama ready with model ${MODEL}" +wait "${SERVE_PID}" diff --git a/deploy/helm/gpu_autoscaling_k8s/files/package.json b/deploy/helm/gpu_autoscaling_k8s/files/package.json new file mode 100644 index 00000000000..3dbc1ca591c --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/files/package.json @@ -0,0 +1,3 @@ +{ + "type": "module" +} diff --git a/deploy/helm/gpu_autoscaling_k8s/files/questions-sample.txt b/deploy/helm/gpu_autoscaling_k8s/files/questions-sample.txt new file mode 100644 index 00000000000..24ebe2b944d --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/files/questions-sample.txt @@ -0,0 +1,10 @@ +What is Kubernetes HPA and when should you use it? +Explain the difference between CPU requests and limits on a pod. +How does NVIDIA Inference Hub route requests to Nemotron Ultra? +Write a short Python function to compute Fibonacci numbers recursively. +What are the tradeoffs of manual replica scaling vs autoscaling? +Describe how a readiness probe differs from a liveness probe. +What is the CAP theorem in distributed systems? +How would you debug a pod stuck in Pending state? +Summarize how Prometheus metrics feed into Kubernetes HPA. +What is the purpose of a HorizontalPodAutoscaler behavior stabilization window? diff --git a/deploy/helm/gpu_autoscaling_k8s/monitoring/dcgm-servicemonitor.yaml b/deploy/helm/gpu_autoscaling_k8s/monitoring/dcgm-servicemonitor.yaml new file mode 100644 index 00000000000..c976e141a35 --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/monitoring/dcgm-servicemonitor.yaml @@ -0,0 +1,21 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Scrape NVIDIA DCGM exporter (already installed by microk8s enable gpu / GPU operator). +apiVersion: monitoring.coreos.com/v1 +kind: ServiceMonitor +metadata: + name: nvidia-dcgm-exporter + namespace: gpu-operator-resources + labels: + release: kube-prometheus +spec: + namespaceSelector: + matchNames: + - gpu-operator-resources + selector: + matchLabels: + app: nvidia-dcgm-exporter + endpoints: + - port: gpu-metrics + interval: 15s diff --git a/deploy/helm/gpu_autoscaling_k8s/monitoring/kube-prometheus-microk8s.yaml b/deploy/helm/gpu_autoscaling_k8s/monitoring/kube-prometheus-microk8s.yaml new file mode 100644 index 00000000000..0a7172cb5ba --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/monitoring/kube-prometheus-microk8s.yaml @@ -0,0 +1,52 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Slim kube-prometheus-stack for single-node MicroK8s (GPU HPA via DCGM metrics). + +alertmanager: + enabled: false + +grafana: + enabled: true + defaultDashboardsEnabled: false + resources: + requests: + cpu: 100m + memory: 128Mi + limits: + cpu: 500m + memory: 512Mi + +kubeControllerManager: + enabled: false +kubeScheduler: + enabled: false +kubeEtcd: + enabled: false +coreDns: + enabled: false + +prometheusOperator: + resources: + requests: + cpu: 100m + memory: 128Mi + limits: + cpu: 500m + memory: 512Mi + +prometheus: + prometheusSpec: + retention: 6h + scrapeInterval: 30s + evaluationInterval: 30s + serviceMonitorSelectorNilUsesHelmValues: false + podMonitorSelectorNilUsesHelmValues: false + ruleSelectorNilUsesHelmValues: false + resources: + requests: + cpu: 300m + memory: 512Mi + limits: + cpu: "1" + memory: 1536Mi diff --git a/deploy/helm/gpu_autoscaling_k8s/monitoring/prometheus-adapter-gpu-values.yaml b/deploy/helm/gpu_autoscaling_k8s/monitoring/prometheus-adapter-gpu-values.yaml new file mode 100644 index 00000000000..6e4c81c0078 --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/monitoring/prometheus-adapter-gpu-values.yaml @@ -0,0 +1,25 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# prometheus-adapter: expose DCGM GPU utilization to HPA. +# DCGM_FI_DEV_GPU_UTIL is scraped from nvidia-dcgm-exporter (GPU operator). + +prometheus: + url: http://REPLACE_PROMETHEUS_SERVICE.monitoring.svc + port: 9090 + +rules: + default: false + custom: + # DCGM exporter attributes GPU util to workloads via exported_* labels (not pod/namespace). + - seriesQuery: 'DCGM_FI_DEV_GPU_UTIL{exported_namespace!="",exported_pod!=""}' + resources: + overrides: + exported_namespace: + resource: namespace + exported_pod: + resource: pod + name: + matches: "^(.*)$" + as: "gpu_utilization_percent" + metricsQuery: 'avg(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)' diff --git a/deploy/helm/gpu_autoscaling_k8s/scripts/cluster-recover.sh b/deploy/helm/gpu_autoscaling_k8s/scripts/cluster-recover.sh new file mode 100755 index 00000000000..24acb7e9388 --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/scripts/cluster-recover.sh @@ -0,0 +1,46 @@ +#!/usr/bin/env bash +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Kubernetes deletions target only this Helm release and the configured load-test Job. +# RESTART_MICROK8S=1 is a separate, cluster-wide interruption. + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=hpa-common.sh +source "${SCRIPT_DIR}/hpa-common.sh" + +NAMESPACE="${NAMESPACE:-nemoclaw-gpu}" +RELEASE="${RELEASE:-nemoclaw-gpu}" +JOB_NAME="${JOB_NAME:-nemoclaw-gpu-hpa-load-test}" +RESTART_MICROK8S="${RESTART_MICROK8S:-0}" +RUN_INSTALL="${RUN_INSTALL:-1}" + +require_cmd kubectl +require_cmd helm + +RELEASE_SELECTOR="$(RELEASE="${RELEASE}" CHART_NAME=nemoclaw-gpu hpa_common_release_selector)" + +kubectl delete deploy,svc,hpa,rs -n "${NAMESPACE}" -l "${RELEASE_SELECTOR}" --ignore-not-found +kubectl delete job "${JOB_NAME}" -n "${NAMESPACE}" --ignore-not-found +hpa_common_clear_stuck_pods "${NAMESPACE}" "${JOB_NAME}" + +helm uninstall "${RELEASE}" -n "${NAMESPACE}" 2>/dev/null || true +sleep 3 + +kubectl delete deploy,svc,hpa,rs -n "${NAMESPACE}" -l "${RELEASE_SELECTOR}" --ignore-not-found +kubectl delete job "${JOB_NAME}" -n "${NAMESPACE}" --ignore-not-found +hpa_common_clear_stuck_pods "${NAMESPACE}" "${JOB_NAME}" + +if [[ "${RESTART_MICROK8S}" == "1" ]] && command -v microk8s >/dev/null 2>&1; then + echo "RESTART_MICROK8S=1 stops every workload in this MicroK8s cluster." >&2 + microk8s stop + microk8s start + microk8s status --wait-ready + microk8s enable gpu 2>/dev/null || true +fi + +if [[ "${RUN_INSTALL}" == "1" ]]; then + exec "${SCRIPT_DIR}/install-hpa.sh" +fi diff --git a/deploy/helm/gpu_autoscaling_k8s/scripts/get-agent-pods.sh b/deploy/helm/gpu_autoscaling_k8s/scripts/get-agent-pods.sh new file mode 100755 index 00000000000..592a79787e8 --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/scripts/get-agent-pods.sh @@ -0,0 +1,66 @@ +#!/usr/bin/env bash +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Per-pod GPU agent status (READY, GPU UTIL %, load-test generators). +# Use alongside: kubectl get hpa -n nemoclaw-gpu -w +# +# Usage: +# ./scripts/get-agent-pods.sh -n nemoclaw-gpu +# ./scripts/get-agent-pods.sh -n nemoclaw-gpu -w + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=hpa-common.sh +source "${SCRIPT_DIR}/hpa-common.sh" + +NAMESPACE="${NAMESPACE:-nemoclaw-gpu}" +INTERVAL="${INTERVAL:-5}" +WATCH=0 + +while [[ $# -gt 0 ]]; do + case "$1" in + -n | --namespace) + NAMESPACE="${2:?namespace required after -n}" + shift 2 + ;; + -w | --watch) + WATCH=1 + shift + ;; + -h | --help) + cat <&2 + exit 1 + ;; + esac +done + +require_cmd kubectl + +if [[ "${WATCH}" -eq 1 ]]; then + while true; do + clear 2>/dev/null || true + hpa_common_print_agent_pods "${NAMESPACE}" + sleep "${INTERVAL}" + done +fi + +hpa_common_print_agent_pods "${NAMESPACE}" diff --git a/deploy/helm/gpu_autoscaling_k8s/scripts/get-hpa.sh b/deploy/helm/gpu_autoscaling_k8s/scripts/get-hpa.sh new file mode 100755 index 00000000000..a50fed75e6f --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/scripts/get-hpa.sh @@ -0,0 +1,64 @@ +#!/usr/bin/env bash +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# One-shot GPU HPA with readable percentage current/target values +# (30.25%/40%, not Kubernetes Quantity milli-units such as 30250m/40). +# For live updates prefer: kubectl get hpa -n nemoclaw-gpu -w +# +# Usage: +# ./scripts/get-hpa.sh -n nemoclaw-gpu +# ./scripts/get-hpa.sh -n nemoclaw-gpu -w # same as kubectl get hpa -w + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=hpa-common.sh +source "${SCRIPT_DIR}/hpa-common.sh" + +NAMESPACE="${NAMESPACE:-nemoclaw-gpu}" +WATCH=0 + +while [[ $# -gt 0 ]]; do + case "$1" in + -n | --namespace) + NAMESPACE="${2:?namespace required after -n}" + shift 2 + ;; + -w | --watch) + WATCH=1 + shift + ;; + -h | --help) + cat <&2 + exit 1 + ;; + esac +done + +require_cmd kubectl + +if [[ "${WATCH}" -eq 1 ]]; then + exec kubectl get hpa -n "${NAMESPACE}" -w +fi + +hpa_common_print_hpa "${NAMESPACE}" diff --git a/deploy/helm/gpu_autoscaling_k8s/scripts/hpa-common.sh b/deploy/helm/gpu_autoscaling_k8s/scripts/hpa-common.sh new file mode 100755 index 00000000000..f295eda06e7 --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/scripts/hpa-common.sh @@ -0,0 +1,722 @@ +#!/usr/bin/env bash +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# Shared helpers for nemoclaw-gpu HPA scripts + +hpa_common_log() { printf '[%s] %s\n' "$(date -u +%H:%M:%S)" "$*"; } + +# Kubernetes custom metrics use Quantity milli-units (33500m = 33.5). Format as plain % for scripts. +# Style: script (GPU UTIL % column + subtitle) | kubectl (matches kubectl get hpa TARGETS column). +hpa_common_format_hpa() { + local ns="${1:?namespace}" + local headers="${2:-1}" + local style="${3:-script}" + python3 - "${ns}" "${headers}" "${style}" <<'PY' +import json, subprocess, sys +from datetime import datetime, timezone + +ns, headers = sys.argv[1], sys.argv[2] == "1" +style = sys.argv[3] if len(sys.argv) > 3 else "script" + +def qty(raw): + if raw is None: + return None + s = str(raw).strip() + if not s or s == "": + return None + if s.endswith("m"): + return float(s[:-1]) / 1000.0 + return float(s) + +def fmt_pct(n): + if n is None: + return "" + if abs(n - round(n)) < 1e-6: + return f"{int(round(n))}%" + s = f"{n:.2f}".rstrip("0").rstrip(".") + return f"{s}%" + +def age(ts): + if not ts: + return "?" + created = datetime.fromisoformat(ts.replace("Z", "+00:00")) + secs = int((datetime.now(timezone.utc) - created).total_seconds()) + if secs < 60: + return f"{secs}s" + if secs < 3600: + return f"{secs // 60}m" + if secs < 86400: + return f"{secs // 3600}h" + return f"{secs // 86400}d" + +def targets(h): + spec_metrics = h.get("spec", {}).get("metrics") or [] + current = h.get("status", {}).get("currentMetrics") or [] + parts = [] + for i, sm in enumerate(spec_metrics): + mtype = sm.get("type") + cm = current[i] if i < len(current) else {} + if mtype == "Pods": + name = sm["pods"]["metric"]["name"] + target = sm["pods"]["target"] + tgt_raw = target.get("averageValue") or target.get("value") + cur_raw = (cm.get("pods") or {}).get("current", {}) + cur_raw = cur_raw.get("averageValue") or cur_raw.get("value") + if name == "gpu_utilization_percent": + parts.append(f"{fmt_pct(qty(cur_raw))}/{fmt_pct(qty(tgt_raw))}") + return " ".join(parts) if parts else "" + +def print_row(h): + meta = h["metadata"] + spec = h["spec"] + status = h.get("status") or {} + ref = spec["scaleTargetRef"] + ref_str = f"{ref['kind']}/{ref['name']}" + tgt = targets(h) + if style == "kubectl": + print( + f"{meta['name']:<20} " + f"{ref_str:<31} " + f"{tgt:<11} " + f"{spec.get('minReplicas', ''):<9} " + f"{spec.get('maxReplicas', ''):<9} " + f"{status.get('currentReplicas', ''):<10} " + f"{age(meta.get('creationTimestamp'))}" + ) + else: + print( + f"{meta['name']:<22} " + f"{ref_str:<31} " + f"{tgt:<18} " + f"{spec.get('minReplicas', ''):<8} " + f"{spec.get('maxReplicas', ''):<8} " + f"{status.get('currentReplicas', ''):<10} " + f"{age(meta.get('creationTimestamp'))}" + ) + +try: + raw = subprocess.check_output( + ["kubectl", "get", "hpa", "-n", ns, "-o", "json"], + stderr=subprocess.DEVNULL, + text=True, + ) + items = json.loads(raw).get("items") or [] +except (subprocess.CalledProcessError, json.JSONDecodeError, FileNotFoundError): + sys.exit(1) + +if not items: + sys.exit(1) + +if headers: + if style == "kubectl": + print( + f"{'NAME':<20} {'REFERENCE':<31} {'TARGETS':<11} " + f"{'MINPODS':<9} {'MAXPODS':<9} {'REPLICAS':<10} AGE" + ) + else: + print("GPU utilization rate (avg per pod): current / target") + print( + f"{'NAME':<22} {'REFERENCE':<31} {'GPU UTIL %':<18} " + f"{'MINPODS':<8} {'MAXPODS':<8} {'REPLICAS':<10} AGE" + ) + +for h in items: + print_row(h) +PY +} + +# Autoscaling-only stdout: GPU utilization as 30.25%/40% (not kubectl milli-units). +hpa_common_print_hpa() { + local ns="${1:?namespace}" + if ! hpa_common_format_hpa "${ns}" 1 "script"; then + kubectl get hpa -n "${ns}" 2>/dev/null || true + fi +} + +# Agent pods + per-pod GPU % (same namespace as HPA). +hpa_common_print_agent_pods() { + local ns="${1:?namespace}" + python3 - "${ns}" <<'PY' +import json, subprocess, sys + +ns = sys.argv[1] + +def qty(raw): + if raw is None: + return None + s = str(raw).strip() + if not s or s == "": + return None + if s.endswith("m"): + return float(s[:-1]) / 1000.0 + return float(s) + +def fmt_pct(n): + if n is None: + return "" + if abs(n - round(n)) < 1e-6: + return f"{int(round(n))}%" + s = f"{n:.2f}".rstrip("0").rstrip(".") + return f"{s}%" + +def age(ts): + if not ts: + return "?" + from datetime import datetime, timezone + created = datetime.fromisoformat(ts.replace("Z", "+00:00")) + secs = int((datetime.now(timezone.utc) - created).total_seconds()) + if secs < 60: + return f"{secs}s" + if secs < 3600: + return f"{secs // 60}m" + if secs < 86400: + return f"{secs // 3600}h" + return f"{secs // 86400}d" + +gpu = {} +try: + raw = subprocess.check_output( + [ + "kubectl", "get", "--raw", + f"/apis/custom.metrics.k8s.io/v1beta1/namespaces/{ns}/pods/*/gpu_utilization_percent", + ], + stderr=subprocess.DEVNULL, + text=True, + ) + for item in json.loads(raw).get("items") or []: + pod = item.get("describedObject", {}).get("name", "") + gpu[pod] = fmt_pct(qty(item.get("value"))) +except (subprocess.CalledProcessError, json.JSONDecodeError, FileNotFoundError): + pass + +try: + raw = subprocess.check_output( + [ + "kubectl", "get", "pods", "-n", ns, + "-l", "app.kubernetes.io/name=nemoclaw-gpu,component=gpu-agent", + "-o", "json", + ], + stderr=subprocess.DEVNULL, + text=True, + ) + items = json.loads(raw).get("items") or [] +except (subprocess.CalledProcessError, json.JSONDecodeError, FileNotFoundError): + items = [] + +print() +print("Agent pods (avg GPU util per pod):") +if not items: + print(" (no gpu-agent pods)") +else: + print( + f"{'NAME':<42} {'READY':<7} {'STATUS':<11} {'RESTARTS':<9} " + f"{'GPU UTIL':<10} AGE" + ) + for pod in sorted(items, key=lambda p: p["metadata"]["name"]): + meta = pod["metadata"] + status = pod.get("status") or {} + name = meta["name"] + ready = sum( + 1 for c in (status.get("containerStatuses") or []) + if c.get("ready") + ) + total = len(status.get("containerStatuses") or []) + ready_s = f"{ready}/{total}" if total else "?" + phase = status.get("phase") or "?" + restarts = sum( + (c.get("restartCount") or 0) for c in (status.get("containerStatuses") or []) + ) + print( + f"{name:<42} {ready_s:<7} {phase:<11} {restarts:<9} " + f"{gpu.get(name, ''):<10} {age(meta.get('creationTimestamp'))}" + ) + +# Load-test job pods (if running) +try: + raw = subprocess.check_output( + [ + "kubectl", "get", "pods", "-n", ns, + "-l", "job-name=nemoclaw-gpu-hpa-load-test", + "-o", "json", + ], + stderr=subprocess.DEVNULL, + text=True, + ) + load_items = json.loads(raw).get("items") or [] +except (subprocess.CalledProcessError, json.JSONDecodeError, FileNotFoundError): + load_items = [] + +if load_items: + print() + print("Load-test generators:") + print(f"{'NAME':<42} {'READY':<7} {'STATUS':<11} {'RESTARTS':<9} AGE") + for pod in sorted(load_items, key=lambda p: p["metadata"]["name"]): + meta = pod["metadata"] + status = pod.get("status") or {} + name = meta["name"] + ready = sum( + 1 for c in (status.get("containerStatuses") or []) + if c.get("ready") + ) + total = len(status.get("containerStatuses") or []) + ready_s = f"{ready}/{total}" if total else "?" + phase = status.get("phase") or "?" + restarts = sum( + (c.get("restartCount") or 0) for c in (status.get("containerStatuses") or []) + ) + print( + f"{name:<42} {ready_s:<7} {phase:<11} {restarts:<9} " + f"{age(meta.get('creationTimestamp'))}" + ) +PY +} + +# Log one HPA row when TARGETS or REPLICAS change (load-test loops). +# Usage: hpa_common_log_hpa_if_changed +hpa_common_log_hpa_if_changed() { + local ns="${1:?namespace}" + local last_var="${2:?lastLineVar}" + local line last + line="$(hpa_common_format_hpa "${ns}" 0 "script" 2>/dev/null | head -1 || true)" + [[ -z "${line}" ]] && return 0 + last="${!last_var}" + if [[ "${line}" != "${last}" ]]; then + hpa_common_log "${line}" + printf -v "${last_var}" '%s' "${line}" + fi +} + +require_cmd() { + command -v "$1" >/dev/null 2>&1 || { + echo "missing command: $1" >&2 + exit 1 + } +} + +# Match Helm fullname in templates/_helpers.tpl (release name contains chart name → use release only). +# Driven entirely by the RELEASE/CHART_NAME env vars; no caller passes positional args. +hpa_common_release_fullname() { + local release="${RELEASE:-nemoclaw-gpu}" + local chart="${CHART_NAME:-nemoclaw-gpu}" + if [[ "${release}" == *"${chart}"* ]]; then + echo "${release}" + else + echo "${release}-${chart}" + fi +} + +hpa_common_agent_deployment() { + echo "$(hpa_common_release_fullname)-agent" +} + +hpa_common_agent_service() { + echo "$(hpa_common_release_fullname)-agent" +} + +hpa_common_release_selector() { + local release="${RELEASE:-nemoclaw-gpu}" + local chart="${CHART_NAME:-nemoclaw-gpu}" + printf 'app.kubernetes.io/name=%s,app.kubernetes.io/instance=%s' "${chart}" "${release}" +} + +# Reject cleartext when Kubernetes reports a node or ingress-controller exposure path. +# The operator must separately restrict access from other hosts on the private network. +hpa_common_verify_insecure_ingress_isolation() { + local ingress_ns="${INGRESS_NS:-ingress-nginx}" + local ingress_release="${INGRESS_RELEASE:-ingress-nginx}" + + require_cmd kubectl + require_cmd python3 + python3 - "${ingress_ns}" "${ingress_release}" <<'PY' +import ipaddress +import json +import subprocess +import sys + +namespace, release = sys.argv[1:] + + +def kubectl_json(*args): + try: + raw = subprocess.check_output( + ["kubectl", *args, "-o", "json"], + stderr=subprocess.PIPE, + text=True, + ) + return json.loads(raw) + except (subprocess.CalledProcessError, json.JSONDecodeError, FileNotFoundError) as exc: + print(f"cannot verify cleartext ingress isolation: kubectl {' '.join(args)} failed", file=sys.stderr) + raise SystemExit(1) from exc + + +private_ranges = tuple( + ipaddress.ip_network(cidr) + for cidr in ( + "10.0.0.0/8", + "127.0.0.0/8", + "169.254.0.0/16", + "172.16.0.0/12", + "192.168.0.0/16", + "::1/128", + "fc00::/7", + "fe80::/10", + ) +) + +nodes = kubectl_json("get", "nodes").get("items") or [] +internal_ips = [] +external_ips = [] +for node in nodes: + for address in (node.get("status") or {}).get("addresses") or []: + if address.get("type") == "InternalIP": + internal_ips.append(address.get("address", "")) + elif address.get("type") == "ExternalIP": + external_ips.append(address.get("address", "")) + +if not internal_ips: + print("cleartext ingress denied: cluster nodes have no verifiable InternalIP", file=sys.stderr) + raise SystemExit(1) +if external_ips: + print("cleartext ingress denied: cluster nodes expose ExternalIP addresses", file=sys.stderr) + raise SystemExit(1) +for raw in internal_ips: + try: + address = ipaddress.ip_address(raw) + except ValueError as exc: + print(f"cleartext ingress denied: invalid node InternalIP {raw!r}", file=sys.stderr) + raise SystemExit(1) from exc + if not any(address in network for network in private_ranges): + print(f"cleartext ingress denied: node InternalIP {raw} is not private", file=sys.stderr) + raise SystemExit(1) + +selector = ( + "app.kubernetes.io/component=controller," + f"app.kubernetes.io/instance={release}" +) +services = kubectl_json( + "get", "services", "-n", namespace, "-l", selector +).get("items") or [] +if not services: + print("cleartext ingress denied: managed ingress controller Service not found", file=sys.stderr) + raise SystemExit(1) + +for service in services: + name = (service.get("metadata") or {}).get("name", "") + spec = service.get("spec") or {} + status = service.get("status") or {} + if spec.get("type") != "ClusterIP": + print(f"cleartext ingress denied: Service {name} is not ClusterIP", file=sys.stderr) + raise SystemExit(1) + if spec.get("externalIPs"): + print(f"cleartext ingress denied: Service {name} has externalIPs", file=sys.stderr) + raise SystemExit(1) + if ((status.get("loadBalancer") or {}).get("ingress") or []): + print(f"cleartext ingress denied: Service {name} has a load-balancer address", file=sys.stderr) + raise SystemExit(1) + +pods = kubectl_json( + "get", "pods", "-n", namespace, "-l", selector +).get("items") or [] +if not pods: + print("cleartext ingress denied: managed ingress controller pods not found", file=sys.stderr) + raise SystemExit(1) +for pod in pods: + name = (pod.get("metadata") or {}).get("name", "") + spec = pod.get("spec") or {} + if spec.get("hostNetwork"): + print(f"cleartext ingress denied: pod {name} uses hostNetwork", file=sys.stderr) + raise SystemExit(1) + for container in spec.get("containers") or []: + for port in container.get("ports") or []: + if port.get("hostPort"): + print(f"cleartext ingress denied: pod {name} uses hostPort", file=sys.stderr) + raise SystemExit(1) +PY +} + +hpa_common_ingress_allow_insecure_value() { + case "${ALLOW_INSECURE_HTTP:-0}" in + 0) + printf 'false' + ;; + 1) + if ! hpa_common_verify_insecure_ingress_isolation; then + echo "Configure ingress.tls instead, or restrict the reported exposure path before retrying cleartext." >&2 + return 1 + fi + printf 'true' + ;; + *) + echo "ALLOW_INSECURE_HTTP must be 0 or 1" >&2 + return 1 + ;; + esac +} + +# Old releases used component=agent; chart now uses gpu-agent + workload-type (immutable selector). +hpa_common_gpu_stale_workload() { + local ns="${1:?namespace}" + local deploy="${2:?deploy}" + local comp + comp="$(kubectl get "deployment/${deploy}" -n "${ns}" \ + -o jsonpath='{.spec.selector.matchLabels.component}' 2>/dev/null || true)" + [[ "${comp}" == "agent" ]] +} + +hpa_common_gpu_recreate_stale_workload() { + local ns="${1:?namespace}" + local deploy="${2:?deploy}" + local svc="${3:-${deploy}}" + if hpa_common_gpu_stale_workload "${ns}" "${deploy}"; then + kubectl delete "deployment/${deploy}" "service/${svc}" -n "${ns}" \ + --ignore-not-found --wait=false 2>/dev/null || true + sleep 2 + fi +} + +# Idle GPU HPA baseline (no --reuse-values — avoids Service port merge bugs). +hpa_common_gpu_helm_upgrade() { + local release="${1:?release}" + local chart_dir="${2:?chartDir}" + local ns="${3:?namespace}" + local hpa_values="${4:?valuesFile}" + local min="${5:-1}" + local max="${6:-4}" + local gpu_target="${7:-40}" + local inference_model="${8:-llama3.2:3b}" + local ingress_host="${9:-}" + + local allow_insecure_http + allow_insecure_http="$(hpa_common_ingress_allow_insecure_value)" + + local helm_args=( + upgrade --install "${release}" "${chart_dir}" + --namespace "${ns}" + --create-namespace + --set namespace.create=false + -f "${hpa_values}" + --set inference.model="${inference_model}" + --set probes.readinessChecksInference=true + --set autoscaling.enabled=true + --set autoscaling.minReplicas="${min}" + --set autoscaling.maxReplicas="${max}" + --set "autoscaling.targetGPUUtilizationPercentage=${gpu_target}" + --set "ingress.allowInsecureHttp=${allow_insecure_http}" + ) + if [[ -n "${ingress_host}" ]]; then + helm_args+=(--set "ingress.host=${ingress_host}") + fi + + helm "${helm_args[@]}" >/dev/null +} + +hpa_common_cleanup_load_test_resources() { + local ns="${1:?namespace}" + local job_name="${2:?jobName}" + + kubectl delete job "${job_name}" -n "${ns}" --ignore-not-found=true >/dev/null 2>&1 || true + kubectl delete rolebinding "${job_name}-endpoints-reader" -n "${ns}" --ignore-not-found=true >/dev/null 2>&1 || true + kubectl delete role "${job_name}-endpoints-reader" -n "${ns}" --ignore-not-found=true >/dev/null 2>&1 || true + kubectl delete serviceaccount "${job_name}-sa" -n "${ns}" --ignore-not-found=true >/dev/null 2>&1 || true + kubectl delete configmap "${job_name}-scripts" -n "${ns}" --ignore-not-found=true >/dev/null 2>&1 || true +} + +# Recovery touches only pods owned by this Helm release and the named load-test Job. +hpa_common_clear_stuck_pods() { + local ns="${1:?namespace}" + local job_name="${2:-nemoclaw-gpu-hpa-load-test}" + local release_selector + release_selector="$(hpa_common_release_selector)" + local pod + for pod in $(kubectl get pods -n "${ns}" \ + -l "${release_selector}" \ + -o jsonpath='{.items[*].metadata.name}' 2>/dev/null); do + [[ -z "${pod}" ]] && continue + kubectl patch pod "${pod}" -n "${ns}" -p '{"metadata":{"finalizers":null}}' --type=merge \ + >/dev/null 2>&1 || true + done + for pod in $(kubectl get pods -n "${ns}" \ + -l "job-name=${job_name}" \ + -o jsonpath='{.items[*].metadata.name}' 2>/dev/null); do + [[ -z "${pod}" ]] && continue + kubectl patch pod "${pod}" -n "${ns}" -p '{"metadata":{"finalizers":null}}' --type=merge \ + >/dev/null 2>&1 || true + done + kubectl delete pods -n "${ns}" -l "${release_selector}" \ + --force --grace-period=0 >/dev/null 2>&1 || true + kubectl delete pods -n "${ns}" -l "job-name=${job_name}" \ + --force --grace-period=0 >/dev/null 2>&1 || true +} + +hpa_common_ensure_agent_ready() { + local ns="${1:?namespace}" + local release="${2:?release}" + local chart_dir="${3:?chartDir}" + local values_file="${4:-}" + local rollout_timeout="${5:-600}" + local deploy + deploy="$(RELEASE="${release}" hpa_common_agent_deployment)" + + local allow_insecure_http + allow_insecure_http="$(hpa_common_ingress_allow_insecure_value)" + + local helm_args=( + upgrade --install "${release}" "${chart_dir}" -n "${ns}" + --set "namespace.create=false" + --set "autoscaling.enabled=false" + --set "gpuScaling.count=1" + --set "ingress.allowInsecureHttp=${allow_insecure_http}" + ) + if [[ -n "${values_file}" && -f "${values_file}" ]]; then + helm_args+=(-f "${values_file}") + fi + helm "${helm_args[@]}" >/dev/null + + hpa_common_kick_deployment "${ns}" "${deploy}" || helm "${helm_args[@]}" >/dev/null + + if ! kubectl rollout status "deployment/${deploy}" -n "${ns}" --timeout="${rollout_timeout}s" >/dev/null; then + hpa_common_diagnose_rollout "${ns}" "${deploy}" + return 1 + fi + + local ready + ready="$(kubectl get "deployment/${deploy}" -n "${ns}" -o jsonpath='{.status.readyReplicas}' 2>/dev/null || echo 0)" + if [[ "${ready}" != "1" ]]; then + hpa_common_diagnose_rollout "${ns}" "${deploy}" + return 1 + fi + return 0 +} + +hpa_common_wait_rollout() { + local deploy="${1:?deploy}" + local ns="${2:?namespace}" + local timeout="${3:-600}" + kubectl rollout status "deployment/${deploy}" -n "${ns}" --timeout="${timeout}s" >/dev/null +} + +hpa_common_kick_deployment() { + local ns="${1:?namespace}" + local deploy="${2:?deploy}" + local rs + rs="$(kubectl get rs -n "${ns}" -l "app.kubernetes.io/name=nemoclaw-gpu,component=gpu-agent" \ + -o jsonpath='{.items[*].metadata.name}' 2>/dev/null || true)" + if [[ -n "${rs}" ]]; then + return 0 + fi + kubectl rollout restart "deployment/${deploy}" -n "${ns}" >/dev/null 2>&1 || true + sleep 8 + rs="$(kubectl get rs -n "${ns}" -l "app.kubernetes.io/name=nemoclaw-gpu,component=gpu-agent" \ + -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || true)" + [[ -n "${rs}" ]] && return 0 + kubectl delete "deployment/${deploy}" -n "${ns}" --ignore-not-found --wait=false 2>/dev/null || true + sleep 3 + return 1 +} + +hpa_common_diagnose_rollout() { + local ns="${1:?namespace}" + hpa_common_print_hpa "${ns}" + kubectl describe hpa -n "${ns}" 2>/dev/null | tail -20 || true +} + +hpa_common_enforce_replica_floor() { + local ns="${1:?namespace}" + local deploy="${2:?deploy}" + local min="${3:-1}" + local spec + spec="$(kubectl get "deployment/${deploy}" -n "${ns}" -o jsonpath='{.spec.replicas}' 2>/dev/null || echo "")" + if [[ ! "${spec}" =~ ^[0-9]+$ ]] || [[ "${spec}" -lt "${min}" ]]; then + kubectl patch "deployment/${deploy}" -n "${ns}" \ + --type=merge -p "{\"spec\":{\"replicas\":${min}}}" + fi +} + +hpa_common_verify_hpa_bounds() { + local ns="${1:?namespace}" + local deploy="${2:?deploy}" + local hpa_name="${3:-${deploy}}" + local min="${4:-1}" + local max="${5:-4}" + + if ! kubectl get "horizontalpodautoscaler/${hpa_name}" -n "${ns}" >/dev/null 2>&1; then + echo "HPA ${hpa_name} not found" >&2 + return 1 + fi + + local desired + desired="$(kubectl get "horizontalpodautoscaler/${hpa_name}" -n "${ns}" -o jsonpath='{.status.desiredReplicas}' 2>/dev/null || echo "")" + + hpa_common_enforce_replica_floor "${ns}" "${deploy}" "${min}" + + if [[ -n "${desired}" && "${desired}" =~ ^[0-9]+$ && "${desired}" -lt "${min}" ]]; then + kubectl patch "deployment/${deploy}" -n "${ns}" \ + --type=merge -p "{\"spec\":{\"replicas\":${min}}}" + sleep 5 + fi + + return 0 +} + +hpa_common_verify_gpu_nodes() { + local gpu_count + gpu_count="$(hpa_common_allocatable_gpus)" + if [[ "${gpu_count}" -lt 1 ]]; then + echo "No allocatable nvidia.com/gpu — HPA cannot scale GPU pods" >&2 + return 1 + fi + return 0 +} + +hpa_common_allocatable_gpus() { + kubectl get nodes -o jsonpath='{range .items[*]}{.status.allocatable.nvidia\.com/gpu}{"\n"}{end}' 2>/dev/null \ + | awk 'NF && $1+0>0 {s+=$1} END {print s+0}' +} + +hpa_common_verify_gpu_hpa_metric() { + local ns="${1:-${NAMESPACE:-nemoclaw-gpu}}" + if kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/${ns}/pods/*/gpu_utilization_percent" 2>/dev/null \ + | grep -q '"metricName":"gpu_utilization_percent"'; then + return 0 + fi + echo "gpu_utilization_percent not available — HPA cannot scale on GPU util" >&2 + return 1 +} + +# Human-readable HPA metric (optional; VERBOSE=1 for full legend). +hpa_common_hpa_metric_display() { + local ns="${1:?namespace}" + local hpa_name="${2:-}" + if [[ "${VERBOSE:-0}" != "1" ]]; then + return 0 + fi + if [[ -z "${hpa_name}" ]]; then + hpa_name="$(kubectl get hpa -n "${ns}" -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || true)" + fi + [[ -n "${hpa_name}" ]] || return 0 + + local metric spec_target spec_type + spec_type="$(kubectl get "horizontalpodautoscaler/${hpa_name}" -n "${ns}" \ + -o jsonpath='{.spec.metrics[0].type}' 2>/dev/null || true)" + if [[ "${spec_type}" == "Pods" ]]; then + metric="$(kubectl get "horizontalpodautoscaler/${hpa_name}" -n "${ns}" \ + -o jsonpath='{.spec.metrics[0].pods.metric.name}' 2>/dev/null || true)" + spec_target="$(kubectl get "horizontalpodautoscaler/${hpa_name}" -n "${ns}" \ + -o jsonpath='{.spec.metrics[0].pods.target.averageValue}' 2>/dev/null || true)" + elif [[ "${spec_type}" == "Resource" ]]; then + metric="$(kubectl get "horizontalpodautoscaler/${hpa_name}" -n "${ns}" \ + -o jsonpath='{.spec.metrics[0].resource.name}' 2>/dev/null || true)" + spec_target="$(kubectl get "horizontalpodautoscaler/${hpa_name}" -n "${ns}" \ + -o jsonpath='{.spec.metrics[0].resource.target.averageUtilization}' 2>/dev/null || true)" + fi + echo "HPA metric: ${metric:-unknown} target=${spec_target:-?}" +} + +# Default GPU HPA custom metric (prometheus-adapter → custom.metrics.k8s.io). +hpa_common_gpu_hpa_metric_name() { + echo "gpu_utilization_percent" +} + +hpa_common_print_hpa_status() { + hpa_common_print_hpa "${1:?namespace}" +} diff --git a/deploy/helm/gpu_autoscaling_k8s/scripts/hpa-load-test.sh b/deploy/helm/gpu_autoscaling_k8s/scripts/hpa-load-test.sh new file mode 100755 index 00000000000..107d1ab9d3b --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/scripts/hpa-load-test.sh @@ -0,0 +1,410 @@ +#!/usr/bin/env bash +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# HPA scale-up / scale-down test driven by GPU utilization (DCGM → HPA). +# Goal: raise average GPU util above HPA target so replicas grow to TARGET_PODS. +# +# Usage: +# cd deploy/helm/gpu_autoscaling_k8s +# ./scripts/hpa-load-test.sh + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +CHART_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)" +# shellcheck source=hpa-common.sh +source "${SCRIPT_DIR}/hpa-common.sh" +NAMESPACE="${NAMESPACE:-nemoclaw-gpu}" +RELEASE="${RELEASE:-nemoclaw-gpu}" +JOB_NAME="${JOB_NAME:-nemoclaw-gpu-hpa-load-test}" +ALLOC_GPUS="$(hpa_common_allocatable_gpus 2>/dev/null || echo 4)" +TARGET_PODS="${TARGET_PODS:-${ALLOC_GPUS}}" + +# Backoff / floor — never drive all GPUs to 0% when HPA has 2+ replicas (circuit breaker keeps probe load). +ERROR_BACKOFF_FACTOR="${ERROR_BACKOFF_FACTOR:-0.92}" +ERROR_BACKOFF_MIN="${ERROR_BACKOFF_MIN:-0.4}" +ERROR_BACKOFF_RECOVERY="${ERROR_BACKOFF_RECOVERY:-1.15}" +CIRCUIT_BREAKER_BACKOFF="${CIRCUIT_BREAKER_BACKOFF:-0.15}" +MIN_INFLIGHT_FLOOR="${MIN_INFLIGHT_FLOOR:-12}" +MIN_RECOVERY_INFLIGHT="${MIN_RECOVERY_INFLIGHT:-4}" +READYZ_GRACE_SEC="${READYZ_GRACE_SEC:-45}" + +# Steady GPU saturation — scales to all allocatable GPUs (TARGET_PODS defaults to GPU count). +# Defaults avoid overload → 502/503 → 0% GPU → retry spikes (see README.md). +# Override any knob via env, e.g. INFLIGHT_PER_GPU=512 ./scripts/hpa-load-test.sh +# HPA target GPU util % (default 40 — easier scale-up vs 50 while load spreads across GPUs). +if [[ "${TARGET_PODS}" -ge 4 ]]; then + JOB_PARALLELISM="${JOB_PARALLELISM:-4}" + MAX_TOKENS="${MAX_TOKENS:-128}" + HPA_TARGET_GPU="${HPA_TARGET_GPU:-40}" + INFLIGHT_PER_GPU="${INFLIGHT_PER_GPU:-64}" + LOAD_MULTIPLIER="${LOAD_MULTIPLIER:-2}" + LOAD_COMPENSATION_SAFETY="${LOAD_COMPENSATION_SAFETY:-2}" + MAX_COMPENSATION="${MAX_COMPENSATION:-4}" + MAX_INFLIGHT_PER_POD="${MAX_INFLIGHT_PER_POD:-512}" + WARMUP_SEC="${WARMUP_SEC:-90}" + NEW_POD_RAMP_SEC="${NEW_POD_RAMP_SEC:-0}" + BOOTSTRAP_INFLIGHT="${BOOTSTRAP_INFLIGHT:-8}" + NEW_POD_WARMUP_PARALLEL="${NEW_POD_WARMUP_PARALLEL:-8}" + RAMP_SEC="${RAMP_SEC:-45}" + ESCALATE_INTERVAL_SEC="${ESCALATE_INTERVAL_SEC:-15}" + ESCALATE_FACTOR="${ESCALATE_FACTOR:-0.35}" + ESCALATE_MAX_MULT="${ESCALATE_MAX_MULT:-1.5}" + TARGET_POLL_SEC="${TARGET_POLL_SEC:-1}" + SCALE_UP_POLL_SEC="${SCALE_UP_POLL_SEC:-10}" +else + JOB_PARALLELISM="${JOB_PARALLELISM:-2}" + LOAD_MULTIPLIER="${LOAD_MULTIPLIER:-2}" + MAX_TOKENS="${MAX_TOKENS:-128}" + HPA_TARGET_GPU="${HPA_TARGET_GPU:-40}" + INFLIGHT_PER_GPU="${INFLIGHT_PER_GPU:-64}" + LOAD_COMPENSATION_SAFETY="${LOAD_COMPENSATION_SAFETY:-3}" + MAX_COMPENSATION="${MAX_COMPENSATION:-8}" + MAX_INFLIGHT_PER_POD="${MAX_INFLIGHT_PER_POD:-512}" + WARMUP_SEC="${WARMUP_SEC:-90}" + NEW_POD_RAMP_SEC="${NEW_POD_RAMP_SEC:-0}" + BOOTSTRAP_INFLIGHT="${BOOTSTRAP_INFLIGHT:-8}" + NEW_POD_WARMUP_PARALLEL="${NEW_POD_WARMUP_PARALLEL:-8}" + RAMP_SEC="${RAMP_SEC:-20}" + ESCALATE_INTERVAL_SEC="${ESCALATE_INTERVAL_SEC:-15}" + ESCALATE_FACTOR="${ESCALATE_FACTOR:-0.4}" + ESCALATE_MAX_MULT="${ESCALATE_MAX_MULT:-2}" + SCALE_UP_POLL_SEC="${SCALE_UP_POLL_SEC:-10}" +fi + +DURATION_SEC="${DURATION_SEC:-720}" +SCALE_UP_TARGET="${SCALE_UP_TARGET:-${TARGET_PODS}}" +SCALE_UP_WAIT_LOOPS="${SCALE_UP_WAIT_LOOPS:-60}" +HPA_VALUES="${HPA_VALUES:-${CHART_DIR}/values-step2-hpa.yaml}" +LOAD_TEST_HPA_VALUES="${LOAD_TEST_HPA_VALUES:-${CHART_DIR}/values-load-test-hpa.yaml}" +SCALE_DOWN_WAIT_LOOPS="${SCALE_DOWN_WAIT_LOOPS:-40}" +ROLLOUT_TIMEOUT="${ROLLOUT_TIMEOUT:-900}" +DEPLOYMENT="${DEPLOYMENT:-$(RELEASE="${RELEASE}" CHART_NAME=nemoclaw-gpu hpa_common_agent_deployment)}" +SERVICE="${SERVICE:-$(RELEASE="${RELEASE}" CHART_NAME=nemoclaw-gpu hpa_common_agent_service)}" +SERVICE_PORT="${SERVICE_PORT:-8081}" +# shellcheck disable=SC2034 # passed by name to hpa_common_log_hpa_if_changed +LAST_HPA_LINE="" + +require_cmd kubectl +require_cmd helm + +ALLOW_INSECURE_VALUE="$(hpa_common_ingress_allow_insecure_value)" + +kubectl get apiservice v1beta1.metrics.k8s.io 2>/dev/null | grep -q True || { + echo "metrics-server not ready" >&2 + exit 1 +} +hpa_common_verify_gpu_nodes || exit 1 +hpa_common_verify_gpu_hpa_metric "${NAMESPACE}" || exit 1 + +if [[ "${TARGET_PODS}" -gt "${ALLOC_GPUS}" ]]; then + TARGET_PODS="${ALLOC_GPUS}" + SCALE_UP_TARGET="${ALLOC_GPUS}" +fi + +if ! hpa_common_ensure_agent_ready "${NAMESPACE}" "${RELEASE}" "${CHART_DIR}" \ + "${HPA_VALUES}" "${ROLLOUT_TIMEOUT}"; then + echo "Baseline pod not ready — HPA test cannot start" >&2 + exit 1 +fi + +hpa_common_gpu_recreate_stale_workload "${NAMESPACE}" "${DEPLOYMENT}" "${SERVICE}" + +INFERENCE_MODEL="${INFERENCE_MODEL:-llama3.2:3b}" +helm upgrade --install "${RELEASE}" "${CHART_DIR}" \ + --namespace "${NAMESPACE}" \ + --create-namespace \ + --set namespace.create=false \ + -f "${HPA_VALUES}" \ + -f "${LOAD_TEST_HPA_VALUES}" \ + --set inference.model="${INFERENCE_MODEL}" \ + --set probes.readinessChecksInference=true \ + --set autoscaling.enabled=true \ + --set autoscaling.minReplicas=1 \ + --set autoscaling.maxReplicas="${TARGET_PODS}" \ + --set "autoscaling.targetGPUUtilizationPercentage=${HPA_TARGET_GPU}" \ + --set "ingress.allowInsecureHttp=${ALLOW_INSECURE_VALUE}" \ + >/dev/null + +hpa_common_verify_hpa_bounds "${NAMESPACE}" "${DEPLOYMENT}" "${DEPLOYMENT}" 1 "${TARGET_PODS}" || true +hpa_common_wait_rollout "${DEPLOYMENT}" "${NAMESPACE}" "${ROLLOUT_TIMEOUT}" +hpa_common_print_hpa "${NAMESPACE}" + +# Ensure agent pods are Ready (Ollama loaded) before load starts. +kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=nemoclaw-gpu,component=gpu-agent \ + -n "${NAMESPACE}" --timeout=600s >/dev/null 2>&1 || { + echo "Agent pods not Ready — run ./scripts/hpa-reset.sh then retry" >&2 + exit 1 +} + +# Wait for inference ready (Ollama model loaded) before starting load Job. +hpa_common_log "Waiting for agent /readyz (model loaded)..." +READY_OK=0 +for _ in $(seq 1 60); do + AGENT_POD="$(kubectl get pods -n "${NAMESPACE}" -l app.kubernetes.io/name=nemoclaw-gpu,component=gpu-agent \ + --field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || true)" + if [[ -n "${AGENT_POD}" ]] && kubectl exec -n "${NAMESPACE}" "${AGENT_POD}" -c agent -- \ + node -e "fetch('http://127.0.0.1:${SERVICE_PORT}/readyz').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))" \ + >/dev/null 2>&1; then + READY_OK=1 + break + fi + sleep 3 +done +if [[ "${READY_OK}" -ne 1 ]]; then + echo "Agent /readyz not stable — Ollama may still be pulling the model. Run ./scripts/hpa-reset.sh then retry" >&2 + exit 1 +fi + +# Smoke-test one chat completion before load Job starts. +hpa_common_log "Smoke test: chat completion on agent pod..." +SMOKE_OK=0 +for _ in $(seq 1 60); do + AGENT_POD="$(kubectl get pods -n "${NAMESPACE}" -l app.kubernetes.io/name=nemoclaw-gpu,component=gpu-agent \ + --field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || true)" + if [[ -n "${AGENT_POD}" ]] && kubectl exec -n "${NAMESPACE}" "${AGENT_POD}" -c agent -- \ + node -e "fetch('http://127.0.0.1:${SERVICE_PORT}/v1/chat/completions',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({messages:[{role:'user',content:'Say OK.'}],max_tokens:8,stream:false})}).then(r=>r.ok?process.exit(0):process.exit(1)).catch(()=>process.exit(1));" \ + >/dev/null 2>&1; then + SMOKE_OK=1 + break + fi + sleep 5 +done +if [[ "${SMOKE_OK}" -ne 1 ]]; then + echo "Chat smoke test failed — inference not serving yet" >&2 + exit 1 +fi +hpa_common_log "Smoke test OK — starting load generators" + +LOAD_SA="${JOB_NAME}-sa" +cleanup() { + # Remove every resource this script creates (Job, RBAC, ConfigMap) so repeated runs + # don't accumulate unused ServiceAccounts/Roles/RoleBindings/ConfigMaps in the namespace. + hpa_common_cleanup_load_test_resources "${NAMESPACE}" "${JOB_NAME}" +} +trap cleanup EXIT + +kubectl delete job "${JOB_NAME}" -n "${NAMESPACE}" --ignore-not-found=true >/dev/null 2>&1 || true + +kubectl apply -f - >/dev/null </dev/null 2>&1 || true +kubectl create configmap "${JOB_NAME}-scripts" -n "${NAMESPACE}" \ + --from-file=load-generator.ts="${CHART_DIR}/files/load-generator.ts" \ + --from-file=questions.txt="${CHART_DIR}/files/questions-sample.txt" \ + --dry-run=client -o yaml | kubectl apply -f - >/dev/null + +cat </dev/null +apiVersion: batch/v1 +kind: Job +metadata: + name: ${JOB_NAME} + namespace: ${NAMESPACE} + labels: + app.kubernetes.io/name: nemoclaw-gpu + app.kubernetes.io/instance: ${RELEASE} + nemoclaw.ai/workload-type: load-test +spec: + backoffLimit: 0 + parallelism: ${JOB_PARALLELISM} + completions: ${JOB_PARALLELISM} + ttlSecondsAfterFinished: 600 + template: + metadata: + labels: + app.kubernetes.io/name: nemoclaw-gpu + app.kubernetes.io/instance: ${RELEASE} + nemoclaw.ai/workload-type: load-test + spec: + serviceAccountName: ${LOAD_SA} + restartPolicy: Never + containers: + - name: load-generator + image: node:22-bookworm-slim@sha256:8607a9064d4a571140998ae9e52a3b3fcf9cff361d04642d5971e6cd76d39e27 + command: ["node", "/scripts/load-generator.ts"] + env: + - name: TARGET_PODS + value: "${TARGET_PODS}" + - name: HPA_TARGET_GPU + value: "${HPA_TARGET_GPU}" + - name: JOB_PARALLELISM + value: "${JOB_PARALLELISM}" + - name: INFLIGHT_PER_GPU + value: "${INFLIGHT_PER_GPU}" + - name: LOAD_MULTIPLIER + value: "${LOAD_MULTIPLIER}" + - name: LOAD_COMPENSATION_SAFETY + value: "${LOAD_COMPENSATION_SAFETY}" + - name: MAX_COMPENSATION + value: "${MAX_COMPENSATION}" + - name: NEW_POD_RAMP_SEC + value: "${NEW_POD_RAMP_SEC}" + - name: MAX_INFLIGHT_PER_POD + value: "${MAX_INFLIGHT_PER_POD}" + - name: WARMUP_SEC + value: "${WARMUP_SEC}" + - name: BOOTSTRAP_INFLIGHT + value: "${BOOTSTRAP_INFLIGHT}" + - name: NEW_POD_WARMUP_PARALLEL + value: "${NEW_POD_WARMUP_PARALLEL}" + - name: ERROR_BACKOFF_FACTOR + value: "${ERROR_BACKOFF_FACTOR}" + - name: ERROR_BACKOFF_MIN + value: "${ERROR_BACKOFF_MIN}" + - name: ERROR_BACKOFF_RECOVERY + value: "${ERROR_BACKOFF_RECOVERY}" + - name: CIRCUIT_BREAKER_BACKOFF + value: "${CIRCUIT_BREAKER_BACKOFF}" + - name: MIN_INFLIGHT_FLOOR + value: "${MIN_INFLIGHT_FLOOR}" + - name: MIN_RECOVERY_INFLIGHT + value: "${MIN_RECOVERY_INFLIGHT}" + - name: READYZ_GRACE_SEC + value: "${READYZ_GRACE_SEC}" + - name: REQUIRE_CHAT_PROBE + value: "false" + - name: TARGET_POLL_SEC + value: "${TARGET_POLL_SEC:-1}" + - name: K8S_NAMESPACE + value: "${NAMESPACE}" + - name: AGENT_SERVICE + value: "${SERVICE}" + - name: HPA_NAME + value: "${DEPLOYMENT}" + - name: AGENT_PORT + value: "${SERVICE_PORT}" + - name: RAMP_SEC + value: "${RAMP_SEC}" + - name: DURATION_SEC + value: "${DURATION_SEC}" + - name: MAX_TOKENS + value: "${MAX_TOKENS}" + - name: ESCALATE_INTERVAL_SEC + value: "${ESCALATE_INTERVAL_SEC}" + - name: ESCALATE_FACTOR + value: "${ESCALATE_FACTOR}" + - name: ESCALATE_MAX_MULT + value: "${ESCALATE_MAX_MULT}" + - name: QUESTIONS_FILE + value: "/questions/questions.txt" + volumeMounts: + - name: scripts + mountPath: /scripts + readOnly: true + - name: questions + mountPath: /questions + readOnly: true + volumes: + - name: scripts + configMap: + name: ${JOB_NAME}-scripts + items: + - key: load-generator.ts + path: load-generator.ts + - name: questions + configMap: + name: ${JOB_NAME}-scripts + items: + - key: questions.txt + path: questions.txt +EOF + +PER_POD_PEAK=$((INFLIGHT_PER_GPU * LOAD_MULTIPLIER)) +hpa_common_log "GPU load: ${JOB_PARALLELISM} generators × ${MAX_TOKENS} tokens → each Ready agent pod; base ~${PER_POD_PEAK} in-flight/pod (${LOAD_MULTIPLIER}×), cap ${MAX_INFLIGHT_PER_POD}/pod, warmup ${WARMUP_SEC}s, bootstrap ${BOOTSTRAP_INFLIGHT}; HPA target ${HPA_TARGET_GPU}% → max ${TARGET_PODS} replicas" + +kubectl wait --for=condition=ready pod -l "job-name=${JOB_NAME}" -n "${NAMESPACE}" --timeout=120s >/dev/null 2>&1 || { + echo "Load-generator pods not ready — check: kubectl get pods -n ${NAMESPACE} -l job-name=${JOB_NAME}" >&2 +} + +if ! kubectl logs -n "${NAMESPACE}" -l "job-name=${JOB_NAME}" --tail=200 2>/dev/null \ + | grep -q 'targetsReady'; then + hpa_common_log "Waiting for load generators to discover agent pods..." + for _ in $(seq 1 15); do + kubectl logs -n "${NAMESPACE}" -l "job-name=${JOB_NAME}" --tail=200 2>/dev/null \ + | grep -q 'targetsReady' && break + sleep 1 + done +fi + +SCALE_UP_OK=0 +SCALE_UP_POLL_SEC="${SCALE_UP_POLL_SEC:-10}" +for _ in $(seq 1 "${SCALE_UP_WAIT_LOOPS}"); do + hpa_common_log_hpa_if_changed "${NAMESPACE}" LAST_HPA_LINE + REPLICAS="$(kubectl get hpa -n "${NAMESPACE}" -o jsonpath='{.items[0].status.currentReplicas}' 2>/dev/null || echo 0)" + if [[ "${REPLICAS}" -ge "${SCALE_UP_TARGET}" ]]; then + SCALE_UP_OK=1 + hpa_common_log "Scale-up OK: ${REPLICAS}/${SCALE_UP_TARGET} replicas" + break + fi + sleep "${SCALE_UP_POLL_SEC}" +done + +if [[ "${SCALE_UP_OK}" -ne 1 ]]; then + echo "HPA did not scale to ${SCALE_UP_TARGET} replicas" >&2 +fi + +kubectl wait --for=condition=complete "job/${JOB_NAME}" -n "${NAMESPACE}" --timeout="$((DURATION_SEC + 180))s" >/dev/null 2>&1 || true +kubectl delete job "${JOB_NAME}" -n "${NAMESPACE}" --ignore-not-found=true >/dev/null 2>&1 || true + +SCALE_DOWN_OK=0 +for _ in $(seq 1 "${SCALE_DOWN_WAIT_LOOPS}"); do + hpa_common_log_hpa_if_changed "${NAMESPACE}" LAST_HPA_LINE + REPLICAS="$(kubectl get hpa -n "${NAMESPACE}" -o jsonpath='{.items[0].status.currentReplicas}' 2>/dev/null || echo 0)" + if [[ "${REPLICAS}" -le 1 ]]; then + SCALE_DOWN_OK=1 + break + fi + sleep 15 +done + +hpa_common_print_hpa "${NAMESPACE}" + +cleanup +trap - EXIT +if [[ "${SCALE_UP_OK}" -ne 1 ]]; then + echo "HPA load test incomplete: did not reach ${SCALE_UP_TARGET} replicas" >&2 +fi +if [[ "${SCALE_DOWN_OK}" -ne 1 ]]; then + echo "HPA load test incomplete: did not scale down to 1 replica" >&2 +fi +if [[ "${SCALE_UP_OK}" -ne 1 || "${SCALE_DOWN_OK}" -ne 1 ]]; then + exit 1 +fi +hpa_common_log "Load test complete: scaled to ${SCALE_UP_TARGET}/${TARGET_PODS} GPU replicas and back to 1" diff --git a/deploy/helm/gpu_autoscaling_k8s/scripts/hpa-reset.sh b/deploy/helm/gpu_autoscaling_k8s/scripts/hpa-reset.sh new file mode 100755 index 00000000000..d111d7968a7 --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/scripts/hpa-reset.sh @@ -0,0 +1,134 @@ +#!/usr/bin/env bash +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Tear down the configured load-test Job and this release's GPU agent pods, then +# helm upgrade the idle baseline. +# +# Usage: +# cd deploy/helm/gpu_autoscaling_k8s +# ./scripts/hpa-reset.sh + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +CHART_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)" +# shellcheck source=hpa-common.sh +source "${SCRIPT_DIR}/hpa-common.sh" +NAMESPACE="${NAMESPACE:-nemoclaw-gpu}" +RELEASE="${RELEASE:-nemoclaw-gpu}" +JOB_NAME="${JOB_NAME:-nemoclaw-gpu-hpa-load-test}" +DEPLOYMENT="${DEPLOYMENT:-$(RELEASE="${RELEASE}" CHART_NAME=nemoclaw-gpu hpa_common_agent_deployment)}" +HPA_NAME="${HPA_NAME:-${DEPLOYMENT}}" +REINSTALL_HELM="${REINSTALL_HELM:-1}" +SKIP_HELM="${SKIP_HELM:-0}" +DELETE_DEPLOYMENT="${DELETE_DEPLOYMENT:-0}" +DELETE_HPA="${DELETE_HPA:-0}" +RUN_LOAD_TEST="${RUN_LOAD_TEST:-0}" +HPA_VALUES="${HPA_VALUES:-${CHART_DIR}/values-step2-hpa.yaml}" +WAIT_ROLLOUT="${WAIT_ROLLOUT:-1}" +ROLLOUT_TIMEOUT="${ROLLOUT_TIMEOUT:-900}" +MIN_REPLICAS="${MIN_REPLICAS:-1}" +MAX_REPLICAS="${MAX_REPLICAS:-4}" +GPU_TARGET="${GPU_TARGET:-40}" +INFERENCE_MODEL="${INFERENCE_MODEL:-llama3.2:3b}" +# Preserve a previously configured Ingress host across reset — without this, the helm +# upgrade below leaves ingress.host unset and Helm falls back to values.yaml's default, +# silently changing the route clients use to reach the agent. +INGRESS_HOST="${INGRESS_HOST:-}" +SERVICE="${SERVICE:-$(RELEASE="${RELEASE}" CHART_NAME=nemoclaw-gpu hpa_common_agent_service)}" +RELEASE_SELECTOR="$(RELEASE="${RELEASE}" CHART_NAME=nemoclaw-gpu hpa_common_release_selector)" + +require_cmd kubectl + +if [[ "${SKIP_HELM}" != "1" ]] || [[ "${RUN_LOAD_TEST}" == "1" ]]; then + require_cmd helm +fi + +namespace_exists() { + kubectl get namespace "${NAMESPACE}" >/dev/null 2>&1 +} + +# Scoped to this release and the configured load-test Job. +clear_pod_finalizers() { + local pod + for pod in $(kubectl get pods -n "${NAMESPACE}" \ + -l "${RELEASE_SELECTOR}" \ + -o jsonpath='{.items[*].metadata.name}' 2>/dev/null); do + [[ -z "${pod}" ]] && continue + kubectl patch pod "${pod}" -n "${NAMESPACE}" -p '{"metadata":{"finalizers":null}}' --type=merge \ + >/dev/null 2>&1 || true + done + for pod in $(kubectl get pods -n "${NAMESPACE}" \ + -l "job-name=${JOB_NAME}" \ + -o jsonpath='{.items[*].metadata.name}' 2>/dev/null); do + [[ -z "${pod}" ]] && continue + kubectl patch pod "${pod}" -n "${NAMESPACE}" -p '{"metadata":{"finalizers":null}}' --type=merge \ + >/dev/null 2>&1 || true + done +} + +if ! namespace_exists; then + exit 0 +fi + +kubectl delete job "${JOB_NAME}" -n "${NAMESPACE}" --ignore-not-found --wait=false 2>/dev/null || true +kubectl delete configmap "${JOB_NAME}-scripts" -n "${NAMESPACE}" --ignore-not-found 2>/dev/null || true +# Matches the RBAC hpa-load-test.sh creates for pod/HPA discovery — clean it up here too in +# case a load test's own EXIT trap didn't run (e.g. the shell was killed). +kubectl delete rolebinding "${JOB_NAME}-endpoints-reader" -n "${NAMESPACE}" --ignore-not-found 2>/dev/null || true +kubectl delete role "${JOB_NAME}-endpoints-reader" -n "${NAMESPACE}" --ignore-not-found 2>/dev/null || true +kubectl delete serviceaccount "${JOB_NAME}-sa" -n "${NAMESPACE}" --ignore-not-found 2>/dev/null || true + +if [[ "${DELETE_HPA}" == "1" ]]; then + kubectl delete hpa "${HPA_NAME}" -n "${NAMESPACE}" --ignore-not-found --wait=false 2>/dev/null || true + kubectl delete hpa -n "${NAMESPACE}" -l "${RELEASE_SELECTOR}" --ignore-not-found --wait=false 2>/dev/null || true +fi + +if [[ "${DELETE_DEPLOYMENT}" == "1" ]]; then + kubectl delete deployment "${DEPLOYMENT}" -n "${NAMESPACE}" --ignore-not-found --wait=false 2>/dev/null || true +fi + +kubectl delete pods -n "${NAMESPACE}" -l "${RELEASE_SELECTOR}" --force --grace-period=0 2>/dev/null || true +kubectl delete pods -n "${NAMESPACE}" -l "job-name=${JOB_NAME}" --force --grace-period=0 2>/dev/null || true +sleep 2 +clear_pod_finalizers +kubectl delete pods -n "${NAMESPACE}" -l "${RELEASE_SELECTOR}" --force --grace-period=0 2>/dev/null || true +kubectl delete pods -n "${NAMESPACE}" -l "job-name=${JOB_NAME}" --force --grace-period=0 2>/dev/null || true + +kubectl delete rs -n "${NAMESPACE}" -l "${RELEASE_SELECTOR}" --ignore-not-found --wait=false 2>/dev/null || true +hpa_common_clear_stuck_pods "${NAMESPACE}" "${JOB_NAME}" + +if [[ "${SKIP_HELM}" == "1" ]]; then + hpa_common_print_hpa "${NAMESPACE}" + exit 0 +fi + +if [[ "${DELETE_HPA}" == "1" ]]; then + if ! hpa_common_ensure_agent_ready "${NAMESPACE}" "${RELEASE}" "${CHART_DIR}" \ + "${HPA_VALUES}" "${ROLLOUT_TIMEOUT}"; then + echo "HPA reset failed — baseline pod not ready" >&2 + exit 1 + fi +fi + +hpa_common_gpu_recreate_stale_workload "${NAMESPACE}" "${DEPLOYMENT}" "${SERVICE}" + +hpa_common_gpu_helm_upgrade "${RELEASE}" "${CHART_DIR}" "${NAMESPACE}" "${HPA_VALUES}" \ + "${MIN_REPLICAS}" "${MAX_REPLICAS}" "${GPU_TARGET}" "${INFERENCE_MODEL}" "${INGRESS_HOST}" + +hpa_common_kick_deployment "${NAMESPACE}" "${DEPLOYMENT}" || hpa_common_gpu_helm_upgrade "${RELEASE}" "${CHART_DIR}" "${NAMESPACE}" "${HPA_VALUES}" \ + "${MIN_REPLICAS}" "${MAX_REPLICAS}" "${GPU_TARGET}" "${INFERENCE_MODEL}" "${INGRESS_HOST}" + +hpa_common_verify_hpa_bounds "${NAMESPACE}" "${DEPLOYMENT}" "${HPA_NAME}" "${MIN_REPLICAS}" "${MAX_REPLICAS}" || true + +if [[ "${WAIT_ROLLOUT}" == "1" ]]; then + hpa_common_wait_rollout "${DEPLOYMENT}" "${NAMESPACE}" "${ROLLOUT_TIMEOUT}" \ + || hpa_common_diagnose_rollout "${NAMESPACE}" "${DEPLOYMENT}" +fi + +hpa_common_print_hpa "${NAMESPACE}" + +if [[ "${RUN_LOAD_TEST}" == "1" ]]; then + exec "${SCRIPT_DIR}/hpa-load-test.sh" +fi diff --git a/deploy/helm/gpu_autoscaling_k8s/scripts/hpa-watch.sh b/deploy/helm/gpu_autoscaling_k8s/scripts/hpa-watch.sh new file mode 100755 index 00000000000..7901d00d57c --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/scripts/hpa-watch.sh @@ -0,0 +1,11 @@ +#!/usr/bin/env bash +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Live HPA watch (kubectl native streaming). +# Same as: kubectl get hpa -n nemoclaw-gpu -w + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +exec "${SCRIPT_DIR}/get-hpa.sh" -w "$@" diff --git a/deploy/helm/gpu_autoscaling_k8s/scripts/install-hpa.sh b/deploy/helm/gpu_autoscaling_k8s/scripts/install-hpa.sh new file mode 100755 index 00000000000..e055fff207b --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/scripts/install-hpa.sh @@ -0,0 +1,229 @@ +#!/usr/bin/env bash +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Install GPU HPA (DCGM → prometheus-adapter → gpu_utilization_percent) and the +# ingress-nginx controller that load-balances traffic across HPA replicas. +# Script output is HPA-focused only; see ../README.md for full operations. +# +# Usage: +# cd deploy/helm/gpu_autoscaling_k8s +# ./scripts/install-hpa.sh + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +CHART_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)" +# shellcheck source=hpa-common.sh +source "${SCRIPT_DIR}/hpa-common.sh" + +NAMESPACE="${NAMESPACE:-nemoclaw-gpu}" +RELEASE="${RELEASE:-nemoclaw-gpu}" +MONITORING_NS="${MONITORING_NS:-monitoring}" +PROM_RELEASE="${PROM_RELEASE:-kube-prometheus}" +ADAPTER_RELEASE="${ADAPTER_RELEASE:-prometheus-adapter}" +INGRESS_NS="${INGRESS_NS:-ingress-nginx}" +INGRESS_RELEASE="${INGRESS_RELEASE:-ingress-nginx}" +INGRESS_CLASS="${INGRESS_CLASS:-nginx}" +INGRESS_SERVICE_TYPE="${INGRESS_SERVICE_TYPE:-ClusterIP}" +INGRESS_HELM_TIMEOUT="${INGRESS_HELM_TIMEOUT:-5m}" +INGRESS_HOST="${INGRESS_HOST:-}" +# Pinned to reviewed chart versions — installing by name alone (no --version) would let a +# later run silently pull whatever the maintainers most recently published upstream. +# Bump deliberately: `helm search repo / --versions` to pick a new version. +PROM_CHART_VERSION="${PROM_CHART_VERSION:-87.19.0}" +ADAPTER_CHART_VERSION="${ADAPTER_CHART_VERSION:-5.3.0}" +INGRESS_CHART_VERSION="${INGRESS_CHART_VERSION:-4.15.1}" +DEPLOYMENT="${DEPLOYMENT:-$(RELEASE="${RELEASE}" CHART_NAME=nemoclaw-gpu hpa_common_agent_deployment)}" +HPA_NAME="${HPA_NAME:-${DEPLOYMENT}}" +HPA_VALUES="${HPA_VALUES:-${CHART_DIR}/values-step2-hpa.yaml}" +MIN_REPLICAS="${MIN_REPLICAS:-1}" +MAX_REPLICAS="${MAX_REPLICAS:-4}" +ROLLOUT_TIMEOUT="${ROLLOUT_TIMEOUT:-900}" +INFERENCE_MODEL="${INFERENCE_MODEL:-llama3.2:3b}" +GPU_TARGET="${GPU_TARGET:-40}" +PROM_HELM_TIMEOUT="${PROM_HELM_TIMEOUT:-25m}" +PROM_VALUES="${PROM_VALUES:-${CHART_DIR}/monitoring/kube-prometheus-microk8s.yaml}" +ADAPTER_VALUES="${ADAPTER_VALUES:-${CHART_DIR}/monitoring/prometheus-adapter-gpu-values.yaml}" + +require_cmd kubectl +require_cmd helm + +case "${INGRESS_SERVICE_TYPE}" in + ClusterIP | NodePort | LoadBalancer) ;; + *) + echo "INGRESS_SERVICE_TYPE must be ClusterIP, NodePort, or LoadBalancer" >&2 + exit 1 + ;; +esac +if [[ "${ALLOW_INSECURE_HTTP:-0}" == "1" && "${INGRESS_SERVICE_TYPE}" != "ClusterIP" ]]; then + echo "ALLOW_INSECURE_HTTP=1 requires INGRESS_SERVICE_TYPE=ClusterIP" >&2 + exit 1 +fi + +case "${ALLOW_INSECURE_HTTP:-0}" in + 0) + INGRESS_RENDER_ERROR="" + if ! INGRESS_RENDER_ERROR="$(helm template ingress-policy-check "${CHART_DIR}" -f "${HPA_VALUES}" \ + --set ingress.allowInsecureHttp=false 2>&1 >/dev/null)"; then + if [[ "${INGRESS_RENDER_ERROR}" == *"ingress.tls is empty"* ]]; then + echo "TLS is required. Configure ingress.tls in HPA_VALUES, or set ALLOW_INSECURE_HTTP=1 for an isolated cluster." >&2 + else + printf '%s\n' "${INGRESS_RENDER_ERROR}" >&2 + fi + exit 1 + fi + ;; + 1) ;; + *) + echo "ALLOW_INSECURE_HTTP must be 0 or 1" >&2 + exit 1 + ;; +esac + +custom_metrics_ready() { + kubectl get apiservice v1beta1.custom.metrics.k8s.io 2>/dev/null | grep -q True +} + +prometheus_service_name() { + local svc="" + svc="$(kubectl get svc -n "${MONITORING_NS}" \ + -l 'app=kube-prometheus-stack-prometheus' \ + -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || true)" + if [[ -z "${svc}" ]]; then + svc="$(kubectl get svc -n "${MONITORING_NS}" -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' \ + | grep -E 'kube-prome-prometheus$' | head -1 || true)" + fi + [[ -n "${svc}" ]] || return 1 + printf '%s' "${svc}" +} + +ensure_prometheus_stack() { + helm repo add prometheus-community https://prometheus-community.github.io/helm-charts >/dev/null 2>&1 || true + helm repo update prometheus-community >/dev/null 2>&1 || helm repo update >/dev/null 2>&1 + + kubectl create namespace "${MONITORING_NS}" --dry-run=client -o yaml | kubectl apply -f - >/dev/null + + if ! helm status "${PROM_RELEASE}" -n "${MONITORING_NS}" >/dev/null 2>&1; then + helm upgrade --install "${PROM_RELEASE}" prometheus-community/kube-prometheus-stack \ + --namespace "${MONITORING_NS}" \ + --create-namespace \ + --version "${PROM_CHART_VERSION}" \ + -f "${PROM_VALUES}" \ + --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \ + --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \ + --set prometheus.prometheusSpec.ruleSelectorNilUsesHelmValues=false \ + --timeout "${PROM_HELM_TIMEOUT}" \ + --wait >/dev/null 2>&1 || true + fi + + kubectl wait --for=condition=ready pod \ + -l app.kubernetes.io/name=prometheus \ + -n "${MONITORING_NS}" \ + --timeout=600s >/dev/null 2>&1 || true + + kubectl apply -f "${CHART_DIR}/monitoring/dcgm-servicemonitor.yaml" >/dev/null + + PROM_SVC="$(prometheus_service_name)" || { + echo "Prometheus not found — GPU HPA metric pipeline unavailable" >&2 + exit 1 + } + PROM_URL="http://${PROM_SVC}.${MONITORING_NS}.svc" + + helm upgrade --install "${ADAPTER_RELEASE}" prometheus-community/prometheus-adapter \ + --namespace "${MONITORING_NS}" \ + --version "${ADAPTER_CHART_VERSION}" \ + -f "${ADAPTER_VALUES}" \ + --set "prometheus.url=${PROM_URL}" \ + --set prometheus.port=9090 \ + --wait --timeout 10m >/dev/null + + for _ in $(seq 1 36); do + custom_metrics_ready && break + sleep 5 + done + custom_metrics_ready || { + echo "custom.metrics.k8s.io not ready — HPA cannot use gpu_utilization_percent" >&2 + exit 1 + } +} + +INFERENCE_MODEL="${INFERENCE_MODEL:-llama3.2:3b}" + +ensure_ingress_nginx() { + local class_exists=0 + kubectl get ingressclass "${INGRESS_CLASS}" >/dev/null 2>&1 && class_exists=1 + + if ! helm status "${INGRESS_RELEASE}" -n "${INGRESS_NS}" >/dev/null 2>&1 && [[ "${class_exists}" == "1" ]]; then + # IngressClass already provided by something this script does not manage — leave it alone. + return 0 + fi + + helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx >/dev/null 2>&1 || true + helm repo update ingress-nginx >/dev/null 2>&1 || helm repo update >/dev/null 2>&1 + + # controller.metrics.* exposes NGINX's own request/latency stats (nginx_ingress_controller_*) + # to Prometheus via a ServiceMonitor — separate from the per-pod GPU/app metrics. + helm upgrade --install "${INGRESS_RELEASE}" ingress-nginx/ingress-nginx \ + --namespace "${INGRESS_NS}" \ + --create-namespace \ + --version "${INGRESS_CHART_VERSION}" \ + --set controller.ingressClassResource.name="${INGRESS_CLASS}" \ + --set controller.service.type="${INGRESS_SERVICE_TYPE}" \ + --set controller.metrics.enabled=true \ + --set controller.metrics.serviceMonitor.enabled=true \ + --timeout "${INGRESS_HELM_TIMEOUT}" \ + --wait >/dev/null + + kubectl wait --for=condition=ready pod \ + -l "app.kubernetes.io/component=controller,app.kubernetes.io/instance=${INGRESS_RELEASE}" \ + -n "${INGRESS_NS}" \ + --timeout=300s >/dev/null 2>&1 || true + + kubectl get ingressclass "${INGRESS_CLASS}" >/dev/null 2>&1 || { + echo "ingress-nginx installed but IngressClass ${INGRESS_CLASS} not found — Ingress cannot route traffic" >&2 + exit 1 + } +} + +helm_install() { + hpa_common_gpu_helm_upgrade "${RELEASE}" "${CHART_DIR}" "${NAMESPACE}" "${HPA_VALUES}" \ + "${MIN_REPLICAS}" "${MAX_REPLICAS}" "${GPU_TARGET}" "${INFERENCE_MODEL}" "${INGRESS_HOST}" +} + +if command -v microk8s >/dev/null 2>&1; then + microk8s enable gpu 2>/dev/null || true + microk8s enable metrics-server 2>/dev/null || true +fi +for _ in $(seq 1 36); do + kubectl get apiservice v1beta1.metrics.k8s.io 2>/dev/null | grep -q True && break + sleep 5 +done +kubectl get apiservice v1beta1.metrics.k8s.io 2>/dev/null | grep -q True || { + echo "metrics-server not ready — CPU/memory HPA APIs unavailable" >&2 + exit 1 +} +hpa_common_verify_gpu_nodes || exit 1 +kubectl get pods -n gpu-operator-resources -l app=nvidia-dcgm-exporter 2>/dev/null | grep -q Running || { + echo "nvidia-dcgm-exporter not running — GPU HPA metric unavailable" >&2 + exit 1 +} + +ensure_prometheus_stack +ensure_ingress_nginx + +hpa_common_gpu_recreate_stale_workload "${NAMESPACE}" "${DEPLOYMENT}" "${DEPLOYMENT}" + +helm_install +# hpa_common_kick_deployment returns 0 when the Deployment is already healthy (or a +# rollout restart fixed it) and non-zero only after it deletes an unrecoverable +# Deployment — so helm_install must run on failure (to recreate it), not on success. +hpa_common_kick_deployment "${NAMESPACE}" "${DEPLOYMENT}" || helm_install + +if ! hpa_common_wait_rollout "${DEPLOYMENT}" "${NAMESPACE}" "${ROLLOUT_TIMEOUT}"; then + hpa_common_diagnose_rollout "${NAMESPACE}" "${DEPLOYMENT}" + exit 1 +fi + +hpa_common_verify_hpa_bounds "${NAMESPACE}" "${DEPLOYMENT}" "${HPA_NAME}" "${MIN_REPLICAS}" "${MAX_REPLICAS}" || true +hpa_common_print_hpa "${NAMESPACE}" diff --git a/deploy/helm/gpu_autoscaling_k8s/scripts/test-render-contract.sh b/deploy/helm/gpu_autoscaling_k8s/scripts/test-render-contract.sh new file mode 100755 index 00000000000..b385f2c5c0e --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/scripts/test-render-contract.sh @@ -0,0 +1,281 @@ +#!/usr/bin/env bash +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Static (no cluster required) Helm render test for the HPA/Deployment/Service/ +# ServiceMonitor label-and-name contract this chart depends on at runtime: +# - hpa.yaml's scaleTargetRef.name must match deployment.yaml's Deployment name. +# - hpa.yaml must use only gpu_utilization_percent and its target must match +# values.autoscaling.targetGPUUtilizationPercentage. +# - service.yaml's selector and servicemonitor.yaml's selector must both match +# deployment.yaml's pod template labels — otherwise the Service has no +# endpoints, or Prometheus scrapes nothing, while the chart still renders +# valid YAML (the kind of silent breakage a template-only change can cause). +# +# Usage: +# cd deploy/helm/gpu_autoscaling_k8s +# ./scripts/test-render-contract.sh + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +CHART_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)" + +require_cmd() { + command -v "$1" >/dev/null 2>&1 || { + echo "missing command: $1" >&2 + exit 1 + } +} +require_cmd helm +require_cmd python3 +python3 -c 'import yaml' 2>/dev/null || { + echo "missing Python dependency: PyYAML" >&2 + exit 1 +} + +NOTES_FILE="${CHART_DIR}/templates/NOTES.txt" +if grep -q '\./scripts/' "${NOTES_FILE}"; then + echo "FAIL: Helm NOTES contains a chart-directory-relative script command" >&2 + exit 1 +fi +grep -Fq 'Pods: kubectl get pods' "${NOTES_FILE}" || { + echo "FAIL: Helm NOTES does not provide a working-directory-independent pod command" >&2 + exit 1 +} +grep -Fq 'Per-pod GPU metrics: kubectl get --raw' "${NOTES_FILE}" || { + echo "FAIL: Helm NOTES does not provide the per-pod GPU metrics command" >&2 + exit 1 +} + +CLEARTEXT_RENDER_OUTPUT="" +if CLEARTEXT_RENDER_OUTPUT="$(helm template test-release "${CHART_DIR}" \ + -f "${CHART_DIR}/values-step2-hpa.yaml" \ + --set autoscaling.enabled=true 2>&1)"; then + echo "FAIL: chart rendered a cleartext Ingress without explicit opt-in" >&2 + exit 1 +fi +EXPECTED_TLS_POLICY_ERROR='ingress.tls is empty and ingress.allowInsecureHttp is false: refusing to render an Ingress that would expose /v1/chat/completions over plain HTTP. Configure ingress.tls with a real certificate, or set ALLOW_INSECURE_HTTP=1 when running the chart scripts to acknowledge cleartext HTTP after their exposure preflight. See README "Ingress security".' +if [[ "${CLEARTEXT_RENDER_OUTPUT}" != *"${EXPECTED_TLS_POLICY_ERROR}"* ]]; then + echo "FAIL: cleartext Ingress render failed for an unexpected reason" >&2 + printf '%s\n' "${CLEARTEXT_RENDER_OUTPUT}" >&2 + exit 1 +fi + +TLS_RENDERED_FILE="$(mktemp)" +trap 'rm -f "${TLS_RENDERED_FILE}"' EXIT +helm template tls-policy-check "${CHART_DIR}" \ + -f "${CHART_DIR}/values-step2-hpa.yaml" \ + --set ingress.allowInsecureHttp=false \ + --set 'ingress.tls[0].secretName=test-tls' \ + --set 'ingress.tls[0].hosts[0]=nemoclaw.example.com' \ + --set-string 'ingress.annotations.nginx\.ingress\.kubernetes\.io/ssl-redirect=false' \ + >"${TLS_RENDERED_FILE}" + +python3 - "${TLS_RENDERED_FILE}" <<'PYEOF' +import sys +import yaml + +with open(sys.argv[1]) as f: + ingresses = [doc for doc in yaml.safe_load_all(f) if doc and doc.get("kind") == "Ingress"] + +if len(ingresses) != 1: + print(f"FAIL: expected exactly one TLS Ingress, found {len(ingresses)}", file=sys.stderr) + sys.exit(1) + +annotations = ingresses[0].get("metadata", {}).get("annotations", {}) +if annotations.get("nginx.ingress.kubernetes.io/ssl-redirect") != "true": + print("FAIL: TLS Ingress does not enforce ssl-redirect=true", file=sys.stderr) + sys.exit(1) +PYEOF + +assert_persistence_render_rejected() { + local expected_message="${1:?expected message}" + shift + local output + if output="$(helm template persistence-policy-check "${CHART_DIR}" \ + -f "${CHART_DIR}/values-step2-hpa.yaml" \ + --set ingress.allowInsecureHttp=true \ + --set ollama.persistence.enabled=true \ + --set-string ollama.persistence.hostPath= \ + "$@" 2>&1)"; then + echo "FAIL: chart rendered an unsafe shared persistence configuration" >&2 + exit 1 + fi + if [[ "${output}" != *"${expected_message}"* ]]; then + echo "FAIL: persistence validation returned an unexpected error" >&2 + printf '%s\n' "${output}" >&2 + exit 1 + fi +} + +assert_persistence_render_rejected \ + "ollama.persistence.accessMode must be ReadWriteMany" \ + --set ollama.persistence.accessMode=ReadWriteOnce \ + --set ollama.persistence.storageClass=test-rwx +assert_persistence_render_rejected \ + "ollama.persistence.storageClass is required" \ + --set ollama.persistence.accessMode=ReadWriteMany + +if ! helm template hostpath-policy-check "${CHART_DIR}" \ + -f "${CHART_DIR}/values-step2-hpa.yaml" \ + --set ingress.allowInsecureHttp=true \ + --set ollama.persistence.enabled=true \ + --set ollama.persistence.accessMode=ReadWriteOnce \ + --set-string ollama.persistence.storageClass= \ + >/dev/null; then + echo "FAIL: chart rejected the explicit single-node hostPath persistence mode" >&2 + exit 1 +fi + +RENDERED_FILE="$(mktemp)" +trap 'rm -f "${TLS_RENDERED_FILE}" "${RENDERED_FILE}"' EXIT +# A legacy non-GPU metric-name override must not alter the fixed HPA metric. +helm template test-release "${CHART_DIR}" -f "${CHART_DIR}/values-step2-hpa.yaml" \ + --set autoscaling.enabled=true \ + --set-string autoscaling.gpu.metricName=nemoclaw_http_inflight_requests \ + --set ollama.persistence.enabled=true \ + --set-string ollama.persistence.hostPath= \ + --set ollama.persistence.accessMode=ReadWriteMany \ + --set ollama.persistence.storageClass=test-rwx \ + --set ingress.allowInsecureHttp=true >"${RENDERED_FILE}" + +python3 - "${RENDERED_FILE}" <<'PYEOF' +import json +import sys +import yaml + +with open(sys.argv[1]) as f: + docs = [d for d in yaml.safe_load_all(f) if d] +by_kind = {} +for d in docs: + by_kind.setdefault(d.get("kind"), []).append(d) + +failures = [] + + +def get(kind): + items = by_kind.get(kind, []) + if len(items) != 1: + failures.append(f"expected exactly one {kind}, found {len(items)}") + return None + return items[0] + + +deploy = get("Deployment") +config = get("ConfigMap") +hpa = get("HorizontalPodAutoscaler") +svc = get("Service") +svcmon = get("ServiceMonitor") +pvc = get("PersistentVolumeClaim") + +if deploy: + deploy_name = deploy["metadata"]["name"] + pod_labels = deploy["spec"]["template"]["metadata"]["labels"] + deploy_selector = deploy["spec"]["selector"]["matchLabels"] + + agent_containers = [ + c for c in deploy["spec"]["template"]["spec"]["containers"] if c.get("name") == "agent" + ] + if len(agent_containers) != 1: + failures.append(f"expected exactly one agent container, found {len(agent_containers)}") + elif agent_containers[0].get("command") != ["node", "/app/agent-server.ts"]: + failures.append("agent container does not execute the mounted TypeScript entry point") + + app_volumes = [ + v for v in deploy["spec"]["template"]["spec"]["volumes"] if v.get("name") == "app" + ] + if len(app_volumes) != 1: + failures.append(f"expected exactly one app volume, found {len(app_volumes)}") + else: + app_items = app_volumes[0].get("configMap", {}).get("items", []) + package_items = [ + item + for item in app_items + if item.get("key") == "package.json" and item.get("path") == "package.json" + ] + if len(package_items) != 1: + failures.append("app volume does not mount package.json next to agent-server.ts") + + if hpa: + target_name = hpa["spec"]["scaleTargetRef"]["name"] + if target_name != deploy_name: + failures.append( + f"HPA scaleTargetRef.name={target_name!r} != Deployment name={deploy_name!r}" + ) + metrics = hpa["spec"]["metrics"] + if len(metrics) != 1: + failures.append(f"HPA must have exactly one GPU metric, found {len(metrics)}") + elif metrics[0].get("type") != "Pods": + failures.append(f"HPA metric type={metrics[0].get('type')!r}, expected 'Pods'") + else: + gpu_metric = metrics[0]["pods"] + metric_name = gpu_metric["metric"]["name"] + if metric_name != "gpu_utilization_percent": + failures.append( + f"HPA Pods metric={metric_name!r}, expected 'gpu_utilization_percent'" + ) + target_value = gpu_metric["target"]["averageValue"] + if str(target_value) != "40": + failures.append( + f"HPA gpu_utilization_percent averageValue={target_value!r}, expected 40 " + "(values-step2-hpa.yaml default targetGPUUtilizationPercentage)" + ) + hpa_mode = hpa.get("metadata", {}).get("annotations", {}).get("nemoclaw.ai/hpa-mode") + if hpa_mode != "gpu": + failures.append(f"HPA mode annotation={hpa_mode!r}, expected 'gpu'") + + for kind, obj in (("Service", svc), ("ServiceMonitor", svcmon)): + if not obj: + continue + selector = obj["spec"]["selector"] + selector = selector.get("matchLabels", selector) if kind == "ServiceMonitor" else selector + for k, v in selector.items(): + if pod_labels.get(k) != v: + failures.append( + f"{kind} selector {k}={v!r} does not match Deployment pod label {k}={pod_labels.get(k)!r}" + ) + + if pvc: + pvc_name = pvc["metadata"]["name"] + volumes = deploy["spec"]["template"]["spec"]["volumes"] + ollama_volumes = [v for v in volumes if v.get("name") == "ollama-data"] + if len(ollama_volumes) != 1: + failures.append( + f"expected exactly one ollama-data volume, found {len(ollama_volumes)}" + ) + elif ollama_volumes[0].get("persistentVolumeClaim", {}).get("claimName") != pvc_name: + failures.append("Deployment ollama-data volume does not reference the rendered PVC") + +if config: + package_json = config.get("data", {}).get("package.json") + try: + package_metadata = json.loads(package_json or "") + except json.JSONDecodeError: + failures.append("agent ConfigMap package.json is not valid JSON") + else: + if package_metadata != {"type": "module"}: + failures.append( + f"agent ConfigMap package.json={package_metadata!r}, expected ESM metadata" + ) + +if pvc: + if pvc["spec"].get("accessModes") != ["ReadWriteMany"]: + failures.append("Ollama PVC does not request ReadWriteMany access") + if pvc["spec"].get("storageClassName") != "test-rwx": + failures.append("Ollama PVC does not use the configured storage class") + +if failures: + print("FAIL: render contract violations:") + for f in failures: + print(f" - {f}") + sys.exit(1) + +print("OK: HPA/Deployment/Service/ServiceMonitor render contract holds") +PYEOF + +echo "OK: chart rejects cleartext Ingress without explicit opt-in" +echo "OK: Helm NOTES commands do not depend on the chart source directory" +echo "OK: chart enforces ssl-redirect=true when TLS is configured" +echo "OK: chart requires an explicit ReadWriteMany storage class for shared PVC persistence" +echo "OK: chart preserves the explicit single-node hostPath persistence mode" diff --git a/deploy/helm/gpu_autoscaling_k8s/scripts/test-script-security-contract.sh b/deploy/helm/gpu_autoscaling_k8s/scripts/test-script-security-contract.sh new file mode 100755 index 00000000000..f99563553fe --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/scripts/test-script-security-contract.sh @@ -0,0 +1,214 @@ +#!/usr/bin/env bash +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=hpa-common.sh +source "${SCRIPT_DIR}/hpa-common.sh" + +fail() { + echo "FAIL: $*" >&2 + exit 1 +} + +TEST_TMP="$(mktemp -d)" +trap 'rm -f "${TEST_TMP}/kubectl" "${TEST_TMP}/kubectl.log"; rmdir "${TEST_TMP}"' EXIT + +cat >"${TEST_TMP}/kubectl" <<'MOCK' +#!/usr/bin/env bash +set -euo pipefail + +if [[ "$*" == "get hpa -n test-namespace -o json" ]]; then + printf '%s' "${HPA_FORMAT_FIXTURE:?}" +elif [[ "$*" == "get nodes -o json" ]]; then + case "${MOCK_NODE_MODE:-private}" in + private) + printf '%s' '{"items":[{"status":{"addresses":[{"type":"InternalIP","address":"10.1.2.3"}]}}]}' + ;; + external) + printf '%s' '{"items":[{"status":{"addresses":[{"type":"InternalIP","address":"10.1.2.3"},{"type":"ExternalIP","address":"203.0.113.10"}]}}]}' + ;; + esac +elif [[ "$*" == get\ services* ]]; then + case "${MOCK_SERVICE_MODE:-internal}" in + internal) + printf '%s' '{"items":[{"metadata":{"name":"ingress-nginx-controller"},"spec":{"type":"ClusterIP"},"status":{}}]}' + ;; + external) + printf '%s' '{"items":[{"metadata":{"name":"ingress-nginx-controller"},"spec":{"type":"LoadBalancer"},"status":{"loadBalancer":{"ingress":[{"ip":"203.0.113.20"}]}}}]}' + ;; + missing) + printf '%s' '{"items":[]}' + ;; + esac +elif [[ "$*" == get\ pods* ]]; then + case "${MOCK_POD_MODE:-internal}" in + internal) + printf '%s' '{"items":[{"metadata":{"name":"ingress-nginx-controller"},"spec":{"hostNetwork":false,"containers":[{"ports":[{"containerPort":80}]}]}}]}' + ;; + host-network) + printf '%s' '{"items":[{"metadata":{"name":"ingress-nginx-controller"},"spec":{"hostNetwork":true,"containers":[{}]}}]}' + ;; + host-port) + printf '%s' '{"items":[{"metadata":{"name":"ingress-nginx-controller"},"spec":{"hostNetwork":false,"containers":[{"ports":[{"containerPort":80,"hostPort":80}]}]}}]}' + ;; + esac +else + echo "unexpected kubectl call: $*" >&2 + exit 1 +fi +MOCK +chmod +x "${TEST_TMP}/kubectl" + +export MOCK_NODE_MODE=private +export MOCK_SERVICE_MODE=internal +export MOCK_POD_MODE=internal +[[ "$(PATH="${TEST_TMP}:${PATH}" ALLOW_INSECURE_HTTP=1 hpa_common_ingress_allow_insecure_value)" == "true" ]] \ + || fail "isolated ClusterIP ingress did not pass the cleartext preflight" + +export MOCK_NODE_MODE=external +if PATH="${TEST_TMP}:${PATH}" ALLOW_INSECURE_HTTP=1 \ + hpa_common_ingress_allow_insecure_value >/dev/null 2>&1; then + fail "cleartext preflight accepted a node ExternalIP" +fi + +export MOCK_NODE_MODE=private +export MOCK_SERVICE_MODE=external +if PATH="${TEST_TMP}:${PATH}" ALLOW_INSECURE_HTTP=1 \ + hpa_common_ingress_allow_insecure_value >/dev/null 2>&1; then + fail "cleartext preflight accepted an external ingress Service" +fi + +export MOCK_SERVICE_MODE=missing +if PATH="${TEST_TMP}:${PATH}" ALLOW_INSECURE_HTTP=1 \ + hpa_common_ingress_allow_insecure_value >/dev/null 2>&1; then + fail "cleartext preflight accepted an unverifiable ingress controller" +fi + +export MOCK_SERVICE_MODE=internal +export MOCK_POD_MODE=host-network +if PATH="${TEST_TMP}:${PATH}" ALLOW_INSECURE_HTTP=1 \ + hpa_common_ingress_allow_insecure_value >/dev/null 2>&1; then + fail "cleartext preflight accepted ingress controller hostNetwork" +fi + +export MOCK_POD_MODE=host-port +if PATH="${TEST_TMP}:${PATH}" ALLOW_INSECURE_HTTP=1 \ + hpa_common_ingress_allow_insecure_value >/dev/null 2>&1; then + fail "cleartext preflight accepted ingress controller hostPort" +fi + +[[ "$(ALLOW_INSECURE_HTTP=0 hpa_common_ingress_allow_insecure_value)" == "false" ]] \ + || fail "cleartext opt-in default is not false" + +MOCK_ISOLATION_STATUS=17 +hpa_common_verify_insecure_ingress_isolation() { return "${MOCK_ISOLATION_STATUS}"; } +if hpa_common_verify_insecure_ingress_isolation; then + fail "cleartext preflight failure override unexpectedly passed" +else + status=$? + [[ "${status}" == "17" ]] || fail "cleartext preflight failure override returned ${status}" +fi +if ALLOW_INSECURE_HTTP=1 hpa_common_ingress_allow_insecure_value >/dev/null 2>&1; then + fail "cleartext opt-in bypassed the isolation preflight" +else + status=$? + [[ "${status}" == "1" ]] || fail "cleartext opt-in did not return the preflight failure" +fi + +MOCK_ISOLATION_STATUS=0 +[[ "$(ALLOW_INSECURE_HTTP=1 hpa_common_ingress_allow_insecure_value)" == "true" ]] \ + || fail "verified cleartext opt-in did not return true" + +if ALLOW_INSECURE_HTTP=yes hpa_common_ingress_allow_insecure_value >/dev/null 2>&1; then + fail "invalid cleartext opt-in value was accepted" +fi + +assert_hpa_format() { + local fixture="${1:?fixture}" + shift + local output + output="$(HPA_FORMAT_FIXTURE="${fixture}" PATH="${TEST_TMP}:${PATH}" \ + hpa_common_format_hpa test-namespace 1 script)" + local expected + for expected in "$@"; do + [[ "${output}" == *"${expected}"* ]] \ + || fail "HPA output does not contain ${expected}: ${output}" + done +} + +assert_hpa_format \ + '{"items":[{"metadata":{"name":"gpu-hpa"},"spec":{"scaleTargetRef":{"kind":"Deployment","name":"agent"},"metrics":[{"type":"Pods","pods":{"metric":{"name":"gpu_utilization_percent"},"target":{"type":"AverageValue","averageValue":"40"}}}]},"status":{"currentMetrics":[{"type":"Pods","pods":{"current":{"averageValue":"30250m"}}}]}}]}' \ + 'GPU utilization rate (avg per pod): current / target' \ + 'GPU UTIL %' \ + '30.25%/40%' + +KUBECTL_LOG="${TEST_TMP}/kubectl.log" +export KUBECTL_LOG + +kubectl() { + printf '%s\n' "$*" >>"${KUBECTL_LOG}" + if [[ "$*" == *"get deployment/test-agent"* ]]; then + printf '%s' "${MOCK_DEPLOYMENT_REPLICAS:-}" + elif [[ "$*" == *"get pods"*"app.kubernetes.io/instance=test-release"* ]]; then + printf 'agent-pod' + elif [[ "$*" == *"get pods"*"job-name=test-load-job"* ]]; then + printf 'load-pod' + fi +} + +: >"${KUBECTL_LOG}" +MOCK_DEPLOYMENT_REPLICAS=not-a-number \ + hpa_common_enforce_replica_floor test-namespace test-agent 2 +grep -Fq 'patch deployment/test-agent -n test-namespace --type=merge -p {"spec":{"replicas":2}}' \ + "${KUBECTL_LOG}" || fail "malformed Deployment replica count did not trigger the replica floor" + +: >"${KUBECTL_LOG}" +MOCK_DEPLOYMENT_REPLICAS=3 \ + hpa_common_enforce_replica_floor test-namespace test-agent 2 +if grep -Fq 'patch deployment/test-agent' "${KUBECTL_LOG}"; then + fail "valid Deployment replica count above the floor triggered a patch" +fi + +RELEASE=test-release CHART_NAME=nemoclaw-gpu \ + hpa_common_clear_stuck_pods test-namespace test-load-job + +grep -q 'job-name=test-load-job' "${KUBECTL_LOG}" \ + || fail "load-test pod cleanup did not use the exact Job name" +grep -q 'app.kubernetes.io/name=nemoclaw-gpu,app.kubernetes.io/instance=test-release' \ + "${KUBECTL_LOG}" || fail "pod cleanup did not use the Helm release selector" +if grep -Eq -- '(^| )-l job-name( |$)' "${KUBECTL_LOG}"; then + fail "pod cleanup used an existential job-name selector" +fi + +: >"${KUBECTL_LOG}" +hpa_common_cleanup_load_test_resources test-namespace test-load-job +for resource in \ + 'job test-load-job' \ + 'rolebinding test-load-job-endpoints-reader' \ + 'role test-load-job-endpoints-reader' \ + 'serviceaccount test-load-job-sa' \ + 'configmap test-load-job-scripts'; do + grep -Fq "delete ${resource} -n test-namespace" "${KUBECTL_LOG}" \ + || fail "load-test cleanup did not delete ${resource}" +done + +awk ' + /^cleanup$/ { cleanup_line = NR } + /^trap - EXIT$/ && cleanup_line < NR { found = 1 } + END { exit !found } +' "${SCRIPT_DIR}/hpa-load-test.sh" \ + || fail "load test does not run cleanup before disabling its EXIT trap" +if grep -q -- '--all' "${SCRIPT_DIR}/cluster-recover.sh"; then + fail "cluster recovery contains namespace-wide deletion" +fi +# shellcheck disable=SC2016 # Match the literal default expression in the target script. +grep -Fq 'RESTART_MICROK8S="${RESTART_MICROK8S:-0}"' "${SCRIPT_DIR}/cluster-recover.sh" \ + || fail "cluster recovery enables a MicroK8s restart by default" +# shellcheck disable=SC2016 # Match the literal default expression in the target script. +grep -Fq 'INGRESS_SERVICE_TYPE="${INGRESS_SERVICE_TYPE:-ClusterIP}"' "${SCRIPT_DIR}/install-hpa.sh" \ + || fail "installer ingress Service does not default to ClusterIP" + +echo "OK: recovery ownership, cleartext ingress security, and GPU HPA formatting contracts hold" diff --git a/deploy/helm/gpu_autoscaling_k8s/templates/NOTES.txt b/deploy/helm/gpu_autoscaling_k8s/templates/NOTES.txt new file mode 100644 index 00000000000..e83de39375b --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/templates/NOTES.txt @@ -0,0 +1,11 @@ +{{/* SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. */}} +{{/* SPDX-License-Identifier: Apache-2.0 */}} +HPA: kubectl get hpa -n {{ include "nemoclaw-gpu.namespace" . }} -w +Pods: kubectl get pods -n {{ include "nemoclaw-gpu.namespace" . }} -l 'app.kubernetes.io/name={{ include "nemoclaw-gpu.name" . }},app.kubernetes.io/instance={{ .Release.Name }},component=gpu-agent' -w +Per-pod GPU metrics: kubectl get --raw '/apis/custom.metrics.k8s.io/v1beta1/namespaces/{{ include "nemoclaw-gpu.namespace" . }}/pods/*/gpu_utilization_percent' +Ingress: kubectl get ingress {{ include "nemoclaw-gpu.fullname" . }}-agent -n {{ include "nemoclaw-gpu.namespace" . }} +Endpoint: http{{ if .Values.ingress.tls }}s{{ end }}://{{ .Values.ingress.host }}{{ .Values.ingress.path }} +{{- if and .Values.ingress.auth.enabled (not .Values.ingress.auth.existingSecret) }} +Ingress basic-auth username: {{ .Values.ingress.auth.username }} +Ingress basic-auth password: kubectl get secret {{ include "nemoclaw-gpu.ingressAuthSecretName" . }} -n {{ include "nemoclaw-gpu.namespace" . }} -o jsonpath='{.data.password}' | base64 -d +{{- end }} diff --git a/deploy/helm/gpu_autoscaling_k8s/templates/_helpers.tpl b/deploy/helm/gpu_autoscaling_k8s/templates/_helpers.tpl new file mode 100644 index 00000000000..701fe94f944 --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/templates/_helpers.tpl @@ -0,0 +1,108 @@ +{{/* SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. */}} +{{/* SPDX-License-Identifier: Apache-2.0 */}} +{{- define "nemoclaw-gpu.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{- define "nemoclaw-gpu.fullname" -}} +{{- if .Values.fullnameOverride }} +{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- $name := default .Chart.Name .Values.nameOverride }} +{{- if contains $name .Release.Name }} +{{- .Release.Name | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }} +{{- end }} +{{- end }} +{{- end }} + +{{- define "nemoclaw-gpu.chart" -}} +{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{- define "nemoclaw-gpu.labels" -}} +helm.sh/chart: {{ include "nemoclaw-gpu.chart" . }} +{{ include "nemoclaw-gpu.selectorLabels" . }} +{{- if .Chart.AppVersion }} +app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} +{{- end }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- end }} + +{{- define "nemoclaw-gpu.selectorLabels" -}} +app.kubernetes.io/name: {{ include "nemoclaw-gpu.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +component: gpu-agent +nemoclaw.ai/workload-type: gpu +{{- end }} + +{{- define "nemoclaw-gpu.namespace" -}} +{{- .Values.namespace.name }} +{{- end }} + +{{- define "nemoclaw-gpu.ingressAuthSecretName" -}} +{{- printf "%s-agent-ingress-auth" (include "nemoclaw-gpu.fullname" .) | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{- define "nemoclaw-gpu.replicas" -}} +{{- if .Values.gpuScaling.oneReplicaPerGpu -}} +{{- .Values.gpuScaling.count | int }} +{{- else -}} +{{- .Values.replicaCount | int }} +{{- end -}} +{{- end }} + +{{- define "nemoclaw-gpu.ollamaResources" -}} +requests: + cpu: {{ .Values.gpuScaling.perPodCpuRequest | quote }} + memory: {{ .Values.gpuScaling.perPodMemory | quote }} + nvidia.com/gpu: {{ .Values.gpuScaling.perPodGpu | quote }} +limits: + cpu: {{ .Values.gpuScaling.perPodCpuLimit | quote }} + memory: {{ .Values.gpuScaling.perPodMemoryLimit | quote }} + nvidia.com/gpu: {{ .Values.gpuScaling.perPodGpu | quote }} +{{- end }} + +{{- define "nemoclaw-gpu.agentResources" -}} +requests: + cpu: {{ .Values.gpuScaling.agentCpuRequest | quote }} + memory: {{ .Values.gpuScaling.agentMemory | quote }} +limits: + cpu: {{ .Values.gpuScaling.agentCpuLimit | quote }} + memory: {{ .Values.gpuScaling.agentMemoryLimit | quote }} +{{- end }} + +{{- /* +One replica = one GPU in GPU mode, so the HPA must never be allowed to scale past +maxGpus even if maxReplicas is set higher — extra pods would just sit Pending with +no GPU to schedule onto. Use the lower of the two positive limits in that mode. +*/}} +{{- define "nemoclaw-gpu.hpaMaxReplicas" -}} +{{- $maxReplicas := int .Values.autoscaling.maxReplicas -}} +{{- $maxGpus := int .Values.autoscaling.maxGpus -}} +{{- if .Values.gpuScaling.oneReplicaPerGpu -}} +{{- if and (gt $maxReplicas 0) (gt $maxGpus 0) -}} +{{- min $maxReplicas $maxGpus -}} +{{- else if gt $maxGpus 0 -}} +{{- $maxGpus -}} +{{- else if gt $maxReplicas 0 -}} +{{- $maxReplicas -}} +{{- else -}} +{{- 10 -}} +{{- end -}} +{{- else if gt $maxReplicas 0 -}} +{{- $maxReplicas -}} +{{- else -}} +{{- 10 -}} +{{- end -}} +{{- end }} + +{{- define "nemoclaw-gpu.hpaMinReplicas" -}} +{{- $min := int .Values.autoscaling.minReplicas -}} +{{- if lt $min 1 -}} +{{- 1 -}} +{{- else -}} +{{- $min -}} +{{- end -}} +{{- end }} diff --git a/deploy/helm/gpu_autoscaling_k8s/templates/configmap.yaml b/deploy/helm/gpu_autoscaling_k8s/templates/configmap.yaml new file mode 100644 index 00000000000..15125354371 --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/templates/configmap.yaml @@ -0,0 +1,20 @@ +{{/* SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. */}} +{{/* SPDX-License-Identifier: Apache-2.0 */}} +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ include "nemoclaw-gpu.fullname" . }}-agent + namespace: {{ include "nemoclaw-gpu.namespace" . }} + labels: + {{- include "nemoclaw-gpu.labels" . | nindent 4 }} +data: + agent-server.ts: | +{{ .Files.Get "files/agent-server.ts" | indent 4 }} + agent-metrics.ts: | +{{ .Files.Get "files/agent-metrics.ts" | indent 4 }} + package.json: | +{{ .Files.Get "files/package.json" | indent 4 }} + ollama-start.sh: | +{{ .Files.Get "files/ollama-start.sh" | indent 4 }} + INFERENCE_BASE_URL: {{ .Values.inference.baseUrl | quote }} + INFERENCE_MODEL: {{ .Values.inference.model | quote }} diff --git a/deploy/helm/gpu_autoscaling_k8s/templates/deployment.yaml b/deploy/helm/gpu_autoscaling_k8s/templates/deployment.yaml new file mode 100644 index 00000000000..4c5b39f321b --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/templates/deployment.yaml @@ -0,0 +1,183 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "nemoclaw-gpu.fullname" . }}-agent + namespace: {{ include "nemoclaw-gpu.namespace" . }} + labels: + {{- include "nemoclaw-gpu.labels" . | nindent 4 }} +spec: + {{- if .Values.autoscaling.enabled }} + replicas: {{ include "nemoclaw-gpu.hpaMinReplicas" . }} + {{- else }} + replicas: {{ include "nemoclaw-gpu.replicas" . }} + {{- end }} + selector: + matchLabels: + {{- include "nemoclaw-gpu.selectorLabels" . | nindent 6 }} + template: + metadata: + labels: + {{- include "nemoclaw-gpu.selectorLabels" . | nindent 8 }} + annotations: + checksum/agent-config: {{ printf "%s%s%s%s" (.Files.Get "files/agent-server.ts") (.Files.Get "files/agent-metrics.ts") (.Files.Get "files/package.json") (.Files.Get "files/ollama-start.sh") | sha256sum }} + {{- with .Values.podAnnotations }} + {{- toYaml . | nindent 8 }} + {{- end }} + spec: + {{- with .Values.podSecurityContext }} + securityContext: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- if .Values.runtimeClassName }} + runtimeClassName: {{ .Values.runtimeClassName | quote }} + {{- end }} + {{- with .Values.nodeSelector }} + nodeSelector: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.affinity }} + affinity: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.tolerations }} + tolerations: + {{- toYaml . | nindent 8 }} + {{- end }} + containers: + - name: ollama + image: "{{ .Values.ollama.image.repository }}:{{ .Values.ollama.image.tag }}{{ with .Values.ollama.image.digest }}@{{ . }}{{ end }}" + imagePullPolicy: {{ .Values.ollama.image.pullPolicy }} + {{- with .Values.ollamaSecurityContext }} + securityContext: + {{- toYaml . | nindent 12 }} + {{- end }} + command: ["/bin/sh", "/scripts/ollama-start.sh"] + ports: + - name: ollama + containerPort: {{ .Values.ollama.port }} + protocol: TCP + env: + - name: OLLAMA_HOST + value: "0.0.0.0:{{ .Values.ollama.port }}" + - name: OLLAMA_MODEL + valueFrom: + configMapKeyRef: + name: {{ include "nemoclaw-gpu.fullname" . }}-agent + key: INFERENCE_MODEL + - name: OLLAMA_NUM_PARALLEL + value: {{ .Values.ollama.numParallel | default 4 | quote }} + volumeMounts: + - name: scripts + mountPath: /scripts + readOnly: true + - name: ollama-data + mountPath: /root/.ollama + resources: + {{- include "nemoclaw-gpu.ollamaResources" . | nindent 12 }} + livenessProbe: + httpGet: + path: /api/tags + port: ollama + initialDelaySeconds: 30 + periodSeconds: 20 + timeoutSeconds: 5 + failureThreshold: 6 + readinessProbe: + httpGet: + path: /api/tags + port: ollama + initialDelaySeconds: 15 + periodSeconds: 15 + timeoutSeconds: 5 + failureThreshold: 12 + - name: agent + image: "{{ .Values.agent.image.repository }}:{{ .Values.agent.image.tag }}{{ with .Values.agent.image.digest }}@{{ . }}{{ end }}" + imagePullPolicy: {{ .Values.agent.image.pullPolicy }} + {{- with .Values.agentSecurityContext }} + securityContext: + {{- toYaml . | nindent 12 }} + {{- end }} + command: ["node", "/app/agent-server.ts"] + ports: + - name: http + containerPort: {{ .Values.service.port }} + protocol: TCP + env: + - name: PORT + value: {{ .Values.service.port | quote }} + - name: INFERENCE_BASE_URL + valueFrom: + configMapKeyRef: + name: {{ include "nemoclaw-gpu.fullname" . }}-agent + key: INFERENCE_BASE_URL + - name: INFERENCE_MODEL + valueFrom: + configMapKeyRef: + name: {{ include "nemoclaw-gpu.fullname" . }}-agent + key: INFERENCE_MODEL + - name: OLLAMA_BASE_URL + value: "http://127.0.0.1:{{ .Values.ollama.port }}" + volumeMounts: + - name: app + mountPath: /app + readOnly: true + - name: tmp + mountPath: /tmp + resources: + {{- include "nemoclaw-gpu.agentResources" . | nindent 12 }} + startupProbe: + httpGet: + path: {{ if .Values.probes.readinessChecksInference }}/readyz{{ else }}/healthz{{ end }} + port: http + periodSeconds: 10 + timeoutSeconds: 5 + failureThreshold: {{ max (div (int .Values.probes.startupProbeSeconds) 10) 12 }} + livenessProbe: + httpGet: + path: /healthz + port: http + initialDelaySeconds: 10 + periodSeconds: 15 + timeoutSeconds: 3 + failureThreshold: 5 + readinessProbe: + httpGet: + path: {{ if .Values.probes.readinessChecksInference }}/readyz{{ else }}/healthz{{ end }} + port: http + initialDelaySeconds: 5 + periodSeconds: 20 + timeoutSeconds: 5 + failureThreshold: 6 + volumes: + - name: app + configMap: + name: {{ include "nemoclaw-gpu.fullname" . }}-agent + items: + - key: agent-server.ts + path: agent-server.ts + - key: agent-metrics.ts + path: agent-metrics.ts + - key: package.json + path: package.json + - name: scripts + configMap: + name: {{ include "nemoclaw-gpu.fullname" . }}-agent + items: + - key: ollama-start.sh + path: ollama-start.sh + mode: 0755 + - name: tmp + emptyDir: {} + - name: ollama-data + {{- if and .Values.ollama.persistence.enabled .Values.ollama.persistence.hostPath }} + hostPath: + path: {{ .Values.ollama.persistence.hostPath | quote }} + type: DirectoryOrCreate + {{- else if .Values.ollama.persistence.enabled }} + persistentVolumeClaim: + claimName: {{ include "nemoclaw-gpu.fullname" . }}-ollama + {{- else }} + emptyDir: {} + {{- end }} diff --git a/deploy/helm/gpu_autoscaling_k8s/templates/hpa.yaml b/deploy/helm/gpu_autoscaling_k8s/templates/hpa.yaml new file mode 100644 index 00000000000..09b1992d6c8 --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/templates/hpa.yaml @@ -0,0 +1,42 @@ +{{/* SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. */}} +{{/* SPDX-License-Identifier: Apache-2.0 */}} +{{- if .Values.autoscaling.enabled }} +{{- $hpaMin := include "nemoclaw-gpu.hpaMinReplicas" . | int }} +{{- $hpaMax := include "nemoclaw-gpu.hpaMaxReplicas" . | int }} +{{- if lt $hpaMax $hpaMin }} +{{- fail (printf "autoscaling.maxReplicas (%d) must be >= minReplicas (%d)" $hpaMax $hpaMin) }} +{{- end }} +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: {{ include "nemoclaw-gpu.fullname" . }}-agent + namespace: {{ include "nemoclaw-gpu.namespace" . }} + labels: + {{- include "nemoclaw-gpu.labels" . | nindent 4 }} + annotations: + nemoclaw.ai/hpa-policy: "min-{{ $hpaMin }}-max-{{ $hpaMax }}-gpu-target-{{ .Values.autoscaling.targetGPUUtilizationPercentage }}pct" + nemoclaw.ai/hpa-mode: "gpu" + nemoclaw.ai/hpa-metric: "gpu_utilization_percent" + nemoclaw.ai/hpa-metric-display: "GPU utilization % (DCGM / nvidia-smi family)" + nemoclaw.ai/hpa-metric-source: "DCGM_FI_DEV_GPU_UTIL" + nemoclaw.ai/hpa-targets-format: "gpu_utilization_percent: %/% (GPU utilization avg per pod)" +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: {{ include "nemoclaw-gpu.fullname" . }}-agent + minReplicas: {{ $hpaMin }} + maxReplicas: {{ $hpaMax }} + metrics: + - type: Pods + pods: + metric: + name: gpu_utilization_percent + target: + type: AverageValue + averageValue: {{ .Values.autoscaling.targetGPUUtilizationPercentage | quote }} + {{- with .Values.autoscaling.behavior }} + behavior: + {{- toYaml . | nindent 4 }} + {{- end }} +{{- end }} diff --git a/deploy/helm/gpu_autoscaling_k8s/templates/ingress-auth-secret.yaml b/deploy/helm/gpu_autoscaling_k8s/templates/ingress-auth-secret.yaml new file mode 100644 index 00000000000..eb8b69bd329 --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/templates/ingress-auth-secret.yaml @@ -0,0 +1,33 @@ +{{/* SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. */}} +{{/* SPDX-License-Identifier: Apache-2.0 */}} +{{- if and .Values.ingress.auth.enabled (not .Values.ingress.auth.existingSecret) }} +{{- $ns := include "nemoclaw-gpu.namespace" . }} +{{- $secretName := include "nemoclaw-gpu.ingressAuthSecretName" . }} +{{- $existing := lookup "v1" "Secret" $ns $secretName }} +{{- $password := .Values.ingress.auth.password }} +{{- if not $password }} +{{- if $existing }} +{{- $password = index $existing.data "password" | b64dec }} +{{- else }} +{{- $password = randAlphaNum 20 }} +{{- end }} +{{- end }} +apiVersion: v1 +kind: Secret +metadata: + name: {{ $secretName }} + namespace: {{ $ns }} + labels: + {{- include "nemoclaw-gpu.labels" . | nindent 4 }} + annotations: + # Keep the Secret (and the password it holds) across `helm uninstall` so a reinstall + # of the same release doesn't invalidate credentials an operator already saved. + helm.sh/resource-policy: keep +type: Opaque +data: + password: {{ $password | b64enc | quote }} +stringData: + # ingress-nginx auth-secret format: one "user:hash" line per user (nginx supports bcrypt). + auth: | + {{ .Values.ingress.auth.username }}:{{ $password | bcrypt }} +{{- end }} diff --git a/deploy/helm/gpu_autoscaling_k8s/templates/ingress.yaml b/deploy/helm/gpu_autoscaling_k8s/templates/ingress.yaml new file mode 100644 index 00000000000..a0d9b17baa6 --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/templates/ingress.yaml @@ -0,0 +1,49 @@ +{{/* SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. */}} +{{/* SPDX-License-Identifier: Apache-2.0 */}} +{{- if and (not .Values.ingress.tls) (not .Values.ingress.allowInsecureHttp) }} +{{- fail "ingress.tls is empty and ingress.allowInsecureHttp is false: refusing to render an Ingress that would expose /v1/chat/completions over plain HTTP. Configure ingress.tls with a real certificate, or set ALLOW_INSECURE_HTTP=1 when running the chart scripts to acknowledge cleartext HTTP after their exposure preflight. See README \"Ingress security\"." }} +{{- end }} +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: {{ include "nemoclaw-gpu.fullname" . }}-agent + namespace: {{ include "nemoclaw-gpu.namespace" . }} + labels: + {{- include "nemoclaw-gpu.labels" . | nindent 4 }} + annotations: + {{- if .Values.ingress.auth.enabled }} + nginx.ingress.kubernetes.io/auth-type: basic + nginx.ingress.kubernetes.io/auth-secret: {{ .Values.ingress.auth.existingSecret | default (include "nemoclaw-gpu.ingressAuthSecretName" .) }} + nginx.ingress.kubernetes.io/auth-realm: "NemoClaw GPU agent - authentication required" + {{- end }} + {{- if and .Values.ingress.tls (not .Values.ingress.allowInsecureHttp) }} + {{- with omit .Values.ingress.annotations "nginx.ingress.kubernetes.io/ssl-redirect" }} + {{- toYaml . | nindent 4 }} + {{- end }} + nginx.ingress.kubernetes.io/ssl-redirect: "true" + {{- else }} + {{- with .Values.ingress.annotations }} + {{- toYaml . | nindent 4 }} + {{- end }} + {{- end }} +spec: + {{- with .Values.ingress.className }} + ingressClassName: {{ . | quote }} + {{- end }} + rules: + - {{- with .Values.ingress.host }} + host: {{ . | quote }} + {{- end }} + http: + paths: + - path: {{ .Values.ingress.path | quote }} + pathType: {{ .Values.ingress.pathType }} + backend: + service: + name: {{ include "nemoclaw-gpu.fullname" . }}-agent + port: + name: http + {{- with .Values.ingress.tls }} + tls: + {{- toYaml . | nindent 4 }} + {{- end }} diff --git a/deploy/helm/gpu_autoscaling_k8s/templates/namespace.yaml b/deploy/helm/gpu_autoscaling_k8s/templates/namespace.yaml new file mode 100644 index 00000000000..feffee803e5 --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/templates/namespace.yaml @@ -0,0 +1,10 @@ +{{/* SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. */}} +{{/* SPDX-License-Identifier: Apache-2.0 */}} +{{- if .Values.namespace.create }} +apiVersion: v1 +kind: Namespace +metadata: + name: {{ include "nemoclaw-gpu.namespace" . }} + labels: + {{- include "nemoclaw-gpu.labels" . | nindent 4 }} +{{- end }} diff --git a/deploy/helm/gpu_autoscaling_k8s/templates/pvc.yaml b/deploy/helm/gpu_autoscaling_k8s/templates/pvc.yaml new file mode 100644 index 00000000000..ca6b08190d9 --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/templates/pvc.yaml @@ -0,0 +1,24 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +{{- if and .Values.ollama.persistence.enabled (not .Values.ollama.persistence.hostPath) }} +{{- if ne .Values.ollama.persistence.accessMode "ReadWriteMany" }} +{{- fail "ollama.persistence.accessMode must be ReadWriteMany when PVC persistence is enabled because every replica mounts the shared claim" }} +{{- end }} +{{- if not .Values.ollama.persistence.storageClass }} +{{- fail "ollama.persistence.storageClass is required when PVC persistence is enabled; select a class that provisions ReadWriteMany volumes" }} +{{- end }} +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: {{ include "nemoclaw-gpu.fullname" . }}-ollama + namespace: {{ include "nemoclaw-gpu.namespace" . }} + labels: + {{- include "nemoclaw-gpu.labels" . | nindent 4 }} +spec: + accessModes: + - {{ .Values.ollama.persistence.accessMode | quote }} + resources: + requests: + storage: {{ .Values.ollama.persistence.size | quote }} + storageClassName: {{ .Values.ollama.persistence.storageClass | quote }} +{{- end }} diff --git a/deploy/helm/gpu_autoscaling_k8s/templates/service.yaml b/deploy/helm/gpu_autoscaling_k8s/templates/service.yaml new file mode 100644 index 00000000000..5a2c7bafc81 --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/templates/service.yaml @@ -0,0 +1,20 @@ +{{/* SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. */}} +{{/* SPDX-License-Identifier: Apache-2.0 */}} +apiVersion: v1 +kind: Service +metadata: + name: {{ include "nemoclaw-gpu.fullname" . }}-agent + namespace: {{ include "nemoclaw-gpu.namespace" . }} + labels: + {{- include "nemoclaw-gpu.labels" . | nindent 4 }} + annotations: + nemoclaw.ai/agent-port: {{ .Values.service.port | quote }} +spec: + type: {{ .Values.service.type }} + ports: + - port: {{ .Values.service.port }} + targetPort: http + protocol: TCP + name: http + selector: + {{- include "nemoclaw-gpu.selectorLabels" . | nindent 4 }} diff --git a/deploy/helm/gpu_autoscaling_k8s/templates/servicemonitor.yaml b/deploy/helm/gpu_autoscaling_k8s/templates/servicemonitor.yaml new file mode 100644 index 00000000000..316731d01b1 --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/templates/servicemonitor.yaml @@ -0,0 +1,22 @@ +{{/* SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. */}} +{{/* SPDX-License-Identifier: Apache-2.0 */}} +{{- if and .Values.metrics.enabled .Values.metrics.serviceMonitor.enabled }} +apiVersion: monitoring.coreos.com/v1 +kind: ServiceMonitor +metadata: + name: {{ include "nemoclaw-gpu.fullname" . }}-agent + namespace: {{ include "nemoclaw-gpu.namespace" . }} + labels: + {{- include "nemoclaw-gpu.labels" . | nindent 4 }} + {{- with .Values.metrics.serviceMonitor.labels }} + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + selector: + matchLabels: + {{- include "nemoclaw-gpu.selectorLabels" . | nindent 6 }} + endpoints: + - port: http + path: {{ .Values.metrics.path }} + interval: {{ .Values.metrics.serviceMonitor.interval }} +{{- end }} diff --git a/deploy/helm/gpu_autoscaling_k8s/values-load-test-hpa.yaml b/deploy/helm/gpu_autoscaling_k8s/values-load-test-hpa.yaml new file mode 100644 index 00000000000..fc823d74212 --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/values-load-test-hpa.yaml @@ -0,0 +1,17 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Optional overlay for load-test — same 40% target as values-step2-hpa.yaml, with a +# shorter scale-up period so the one-pod-at-a-time steps are visible within the test window. +autoscaling: + targetGPUUtilizationPercentage: 40 + behavior: + # One pod per reconcile (~10s here) — gradual scale-up instead of jumping + # straight to the replica count the raw GPU-utilization ratio would allow. + scaleUp: + stabilizationWindowSeconds: 0 + policies: + - type: Pods + value: 1 + periodSeconds: 10 + selectPolicy: Max diff --git a/deploy/helm/gpu_autoscaling_k8s/values-step2-hpa.yaml b/deploy/helm/gpu_autoscaling_k8s/values-step2-hpa.yaml new file mode 100644 index 00000000000..54c10ace0dc --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/values-step2-hpa.yaml @@ -0,0 +1,68 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# GPU HPA — one pod per GPU, scale on DCGM GPU utilization (%). +# Requires: nvidia-dcgm-exporter + Prometheus + prometheus-adapter (install-hpa.sh sets up). + +autoscaling: + enabled: true + minReplicas: 1 + maxReplicas: 4 + maxGpus: 4 + targetGPUUtilizationPercentage: 40 + behavior: + # One pod per reconcile (~15s sync period) — visible, gradual scale-up + # instead of jumping straight to the replica count the raw GPU-utilization + # ratio would allow (ceil(currentReplicas * current/target)). + scaleUp: + stabilizationWindowSeconds: 0 + policies: + - type: Pods + value: 1 + periodSeconds: 15 + selectPolicy: Max + scaleDown: + stabilizationWindowSeconds: 180 + policies: + - type: Percent + value: 50 + periodSeconds: 60 + - type: Pods + value: 1 + periodSeconds: 60 + selectPolicy: Max + +gpuScaling: + oneReplicaPerGpu: true + count: 1 + perPodGpu: 1 + perPodCpuRequest: "2" + perPodCpuLimit: "4" + perPodMemory: "16Gi" + perPodMemoryLimit: "24Gi" + agentCpuRequest: "250m" + agentCpuLimit: "1" + agentMemory: "512Mi" + agentMemoryLimit: "1Gi" + +loadTest: + chatOnly: true + concurrencyPerPod: 64 + maxTokens: 512 + jobParallelism: 8 + loadMultiplier: 4 + inflightPerGpu: 1024 + rampSec: 30 + +probes: + readinessChecksInference: true + startupProbeSeconds: 600 + +# Single-node model cache: pull once and reuse across HPA replicas and restarts. +# For multi-node scheduling, clear hostPath and configure the RWX PVC settings documented in README.md. +ollama: + persistence: + enabled: true + hostPath: /var/lib/nemoclaw-gpu/ollama + size: 20Gi + accessMode: ReadWriteMany diff --git a/deploy/helm/gpu_autoscaling_k8s/values.yaml b/deploy/helm/gpu_autoscaling_k8s/values.yaml new file mode 100644 index 00000000000..74ec582f64c --- /dev/null +++ b/deploy/helm/gpu_autoscaling_k8s/values.yaml @@ -0,0 +1,208 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Manual scaling: one pod per GPU (see README.md for HPA installation). +# Target deployment: local Ollama on GPU, GPU-util HPA. +# helm install nemoclaw-gpu . -n nemoclaw-gpu --create-namespace --set gpuScaling.count=2 +gpuScaling: + oneReplicaPerGpu: true + count: 1 + perPodGpu: 1 + perPodCpuRequest: "2" + perPodCpuLimit: "4" + perPodMemory: "16Gi" + perPodMemoryLimit: "24Gi" + # Agent sidecar (no GPU) — metrics + OpenAI-compatible proxy to local Ollama + agentCpuRequest: "250m" + agentCpuLimit: "1" + agentMemory: "512Mi" + agentMemoryLimit: "1Gi" + +replicaCount: 1 + +ollama: + image: + repository: ollama/ollama + # Pinned by digest so `helm install/upgrade` always pulls the exact image reviewed with + # this chart, not whatever `latest` currently resolves to. `tag` stays human-readable; + # the digest (not the tag) is what containerd actually resolves. To intentionally pick + # up a newer Ollama release, re-pin both: `docker manifest inspect ollama/ollama:latest` + # (or the registry API) for the new linux/amd64 digest, then bump `tag` to match. + tag: latest + digest: "sha256:f040603c11a11f125660eaf848170780e43b68973fc270e26f168536031ac258" + pullPolicy: IfNotPresent + port: 11434 + # Concurrent GPU batches per model (raise with care — VRAM limited). + numParallel: 4 + # Disabled uses an independent emptyDir for every replica. PVC persistence is shared by all + # replicas and therefore requires ReadWriteMany plus an explicit RWX-capable storage class. + # hostPath is the single-node development alternative and overrides the PVC settings. + persistence: + enabled: false + size: 20Gi + storageClass: "" + # Every scalable replica mounts the same claim, so other access modes are rejected. + accessMode: ReadWriteMany + # Single-node development: host directory shared by all replicas. Overrides PVC. + hostPath: "" + +agent: + image: + repository: node + # Same digest-pinning rationale as ollama.image above. + tag: "22-bookworm-slim" + digest: "sha256:8607a9064d4a571140998ae9e52a3b3fcf9cff361d04642d5971e6cd76d39e27" + pullPolicy: IfNotPresent + +nameOverride: "" +fullnameOverride: "" + +namespace: + create: false + name: nemoclaw-gpu + +service: + type: ClusterIP + # Agent HTTP port (CPU chart uses 8080 — keep charts on different ports) + port: 8081 + +# Layer-7 load balancing through ingress-nginx. install-hpa.sh installs the +# ingress-nginx controller automatically when the cluster does not already have one. +ingress: + className: nginx + annotations: + nginx.ingress.kubernetes.io/proxy-connect-timeout: "60" + nginx.ingress.kubernetes.io/proxy-read-timeout: "600" + nginx.ingress.kubernetes.io/proxy-send-timeout: "600" + nginx.ingress.kubernetes.io/proxy-buffering: "off" + nginx.ingress.kubernetes.io/proxy-request-buffering: "off" + host: nemoclaw.local + path: / + pathType: Prefix + tls: [] + # The chart refuses to render a routable Ingress unless ingress.tls is configured or this + # value is explicitly true. The scripts keep this false unless ALLOW_INSECURE_HTTP=1 passes + # their Kubernetes exposure preflight. See README "Ingress security". + allowInsecureHttp: false + # Basic auth in front of the completion proxy, which has no authentication of its own. + # On by default. Leaving password unset (null, not an empty string — this is intentional, + # not an accidentally-blank credential) auto-generates a random one on first install and + # reuses it across `helm upgrade` re-runs (install-hpa.sh/hpa-load-test.sh/hpa-reset.sh + # never pass --reuse-values) so the credential doesn't rotate underneath you. Retrieve it + # with `kubectl get secret -agent-ingress-auth -o jsonpath='{.data.password}' | + # base64 -d` (see README "Ingress security"). + auth: + enabled: true + username: admin + password: null + existingSecret: "" + +# Local GPU inference via Ollama sidecar (same pattern as NemoClaw GPU E2E / Ollama onboard) +inference: + baseUrl: "http://127.0.0.1:11434/v1" + model: "llama3.2:3b" + # Optional bearer token if fronting Ollama with auth later + apiKey: "" + existingSecret: "" + secretName: nemoclaw-gpu-inference + +resources: + requests: + cpu: "2" + memory: "16Gi" + nvidia.com/gpu: "1" + limits: + cpu: "4" + memory: "24Gi" + nvidia.com/gpu: "1" + +podAnnotations: {} + +podSecurityContext: + seccompProfile: + type: RuntimeDefault + +# The ollama container needs GPU device access via the NVIDIA device plugin; forcing +# non-root/read-only-rootfs here is untested against every driver/container-toolkit +# combination and risks silently breaking GPU access, so this stays minimal (no privilege +# escalation, no extra Linux capabilities) rather than fully locked down. Override via +# --set ollamaSecurityContext.=... once you've validated non-root on your cluster. +ollamaSecurityContext: + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + +# The agent sidecar is a plain Node.js HTTP proxy with no device or host access, so it can +# be fully locked down: non-root, no capabilities, read-only rootfs (it writes nothing; see +# the "tmp" emptyDir mount below for any incidental scratch space Node itself may want). +agentSecurityContext: + allowPrivilegeEscalation: false + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + +# Readiness checks local Ollama (/readyz). First model pull can take several minutes. +probes: + readinessChecksInference: true + startupProbeSeconds: 600 + +runtimeClassName: "" + +nodeSelector: + nvidia.com/gpu.present: "true" + +tolerations: + - key: nvidia.com/gpu + operator: Exists + effect: NoSchedule + +affinity: {} + +autoscaling: + enabled: false + minReplicas: 1 + maxReplicas: 4 + maxGpus: 4 + # DCGM_FI_DEV_GPU_UTIL via Prometheus + prometheus-adapter (see install-hpa.sh) + targetGPUUtilizationPercentage: 40 + behavior: + scaleUp: + stabilizationWindowSeconds: 0 + policies: + - type: Percent + value: 100 + periodSeconds: 15 + - type: Pods + value: 4 + periodSeconds: 15 + selectPolicy: Max + scaleDown: + stabilizationWindowSeconds: 180 + policies: + - type: Percent + value: 50 + periodSeconds: 60 + - type: Pods + value: 1 + periodSeconds: 60 + selectPolicy: Max +loadTest: + chatOnly: true + concurrencyPerPod: 20 + maxTokens: 128 + jobParallelism: 2 + rampSec: 45 + +metrics: + enabled: true + path: /metrics + # Scraped by Prometheus so per-pod request/latency metrics (nemoclaw_llm_*) show up + # in Grafana out of the box. Default true so install/load-test/reset re-runs (plain + # `helm upgrade`, not --reuse-values) don't silently disable this on every re-apply. + serviceMonitor: + enabled: true + interval: 30s + labels: {}