2026 05 27 0hlj

.cursor/skills/nemoclaw-k8s-hpa/SKILL.md

@@ -0,0 +1,178 @@
+---
+name: nemoclaw-k8s-hpa
+description: >-
+  Deploy and validate NemoClaw CPU agent tier on Kubernetes with Helm, CPU-based
+  HPA, metrics-server, and load tests (Inference Hub / Nemotron). Use when the user
+  asks about nemoclaw-cpu Helm chart, Kubernetes autoscaling, HPA scale-up/down,
+  agent pods, load testing for HPA, or K8s packaging separate from VM NemoClaw/Telegram.
+---
+
+# NemoClaw Kubernetes HPA (nemoclaw-cpu)
+
+## Architecture (read first)
+
+```text
+Telegram / OpenShell sandbox (VM)     ≠     K8s agent pods (nemoclaw-cpu chart)
+         │                                              │
+         └─ Inference Hub (Nemotron Ultra) ◄────────────┘ (optional same API)
+```
+
+| Layer | Location | Scales with HPA? |
+|-------|----------|------------------|
+| Full NemoClaw (Telegram, OpenShell) | VM `nemoclaw onboard` | **No** |
+| CPU agent pods | `deploy/helm/nemoclaw-cpu/` | **Yes** |
+
+HPA watches **CPU % of pod CPU requests** on agent pods, not VM processes.
+
+**Load balancing:** `ClusterIP` Service `nemoclaw-nemoclaw-cpu-agent` is the in-cluster LB. No cloud ALB required. Traffic only reaches **Ready** pods (`kubectl get endpoints`).
+
+---
+
+## Repo map
+
+| Path | Purpose |
+|------|---------|
+| `deploy/helm/nemoclaw-cpu/` | Helm chart (agent Deployment, Service, HPA, ConfigMap) |
+| `deploy/helm/nemoclaw-cpu/scripts/install-hpa.sh` | **One-command** CPU HPA (metrics-server) |
+| `deploy/helm/nemoclaw-cpu/scripts/install-performance-hpa.sh` | Optional Prometheus + performance HPA |
+| `deploy/helm/nemoclaw-cpu/scripts/hpa-load-test.sh` | End-to-end HPA load test |
+| `deploy/helm/nemoclaw-cpu/files/load-generator.mjs` | In-cluster load Job |
+| `deploy/helm/nemoclaw-cpu/files/agent-server.mjs` | `/healthz`, `/readyz`, `/bench`, `/v1/chat/completions` |
+| `deploy/helm/nemoclaw-cpu/values-step2-hpa.yaml` | Enable CPU HPA |
+| `deploy/helm/nemoclaw-cpu/values-step2-hpa-saturate.yaml` | HPA + **400m CPU request/pod** (fit 7 on 8 vCPU node) |
+| `deploy/helm/nemoclaw-cpu/scripts/hpa-reset.sh` | Clean Jobs/HPA/pods before a new test |
+
+---
+
+## Prerequisites
+
+- Kubernetes (MicroK8s OK), `helm` 3, `kubectl`
+- `NVIDIA_INFERENCE_HUB_API_KEY` in `~/.nemoclaw/secrets.env` (`sk-*`)
+- **metrics-server** for CPU HPA: `microk8s enable metrics-server`
+- Verify: `kubectl get apiservice v1beta1.metrics.k8s.io` → `AVAILABLE True`
+
+---
+
+## One-command install (recommended)
+
+CPU HPA + metrics-server (no Prometheus):
+
+```bash
+cd deploy/helm/nemoclaw-cpu
+source ~/.nemoclaw/secrets.env
+./scripts/install-hpa.sh
+kubectl get hpa,pods -n nemoclaw -w
+```
+
+Optional Prometheus path: `./scripts/install-performance-hpa.sh`
+
+**Do not** use placeholder `sk-YOUR-INFERENCE-HUB-KEY` — `/readyz` stays 503 and pods never become Ready.
+
+---
+
+## Alternate — CPU HPA only (no Prometheus)
+
+```bash
+helm upgrade --install nemoclaw . -n nemoclaw --create-namespace \
+  --set namespace.create=false \
+  --set inference.apiKey="${NVIDIA_INFERENCE_HUB_API_KEY}"
+
+helm upgrade nemoclaw . -n nemoclaw --reuse-values \
+  -f values-step2-hpa-saturate.yaml \
+  --set inference.apiKey="${NVIDIA_INFERENCE_HUB_API_KEY}"
+```
+
+| Value | Typical |
+|-------|---------|
+| `autoscaling.minReplicas` | 1 |
+| `autoscaling.maxReplicas` | 7 (8 vCPU node) |
+| `autoscaling.targetCPUUtilizationPercentage` | 50–65 (saturate); 65 (1 CPU/pod) |
+| `autoscaling.behavior.scaleDown.stabilizationWindowSeconds` | 90–120 (not 300) |
+| `cpuScaling.perPodRequest` | **400m** (fit 7 pods); **1** only fits ~4–6 |
+
+When `autoscaling.enabled=true`, Helm sets `spec.replicas` to `minReplicas` (≥1); HPA scales between min and max.
+
+---
+
+## Step 3 — Run HPA load test
+
+```bash
+./scripts/hpa-reset.sh      # optional: clean Jobs/HPA/pods first
+./scripts/hpa-load-test.sh
+```
+
+Defaults: `TARGET_PODS=7`, `CONCURRENCY_PER_POD=40`, `/bench` with worker-thread CPU spin, **one** load Job pod (`JOB_PARALLELISM=1`).
+
+**Watch (two terminals — `kubectl get -w` accepts only one resource type):**
+
+```bash
+watch -n 5 'kubectl get hpa,pods -n nemoclaw; kubectl top pods -n nemoclaw 2>/dev/null | grep agent'
+kubectl get endpoints nemoclaw-nemoclaw-cpu-agent -n nemoclaw
+```
+
+**Success:** `REPLICAS` climbs toward 7; multiple endpoint IPs; agent pods `1/1 Ready`; `kubectl top` shows **300m+** CPU per pod under load.
+
+**After test:**
+
+```bash
+helm upgrade nemoclaw . -n nemoclaw --reuse-values --set loadTest.cpuSpinMs=0
+```
+
+---
+
+## Manual scale (no HPA)
+
+```bash
+helm upgrade nemoclaw . -n nemoclaw --reuse-values --set autoscaling.enabled=false --set cpuScaling.count=4
+```
+
+`cpuScaling.count=N` → N pods, each `perPodRequest` CPU (default 1).
+
+---
+
+## Troubleshooting (symptom → cause)
+
+| Symptom | Likely cause |
+|---------|----------------|
+| Helm `namespace already exists` | Chart + `--create-namespace` both create NS → `namespace.create=false` |
+| Pod `0/1`, readiness 503 | Bad Inference Hub key |
+| HPA `cpu: <unknown>` | metrics-server missing or pod unready |
+| Stuck at 3–4 replicas, `cpu: 99%` | **Node CPU requests full** (1 CPU/pod); use `values-step2-hpa-saturate.yaml` (400m) |
+| One endpoint IP only | Only one pod Ready — fix probes / worker-thread `/bench` |
+| Load Job `fetch failed` | Pods unready or overloaded; check endpoints |
+| `7/8` manual scale, pod Pending | Insufficient allocatable CPU |
+| HPA scale-down slow | 5 min stabilization window (by design) |
+
+**Not caused by missing external load balancer** — Service + Endpoints handle distribution.
+
+---
+
+## Agent HTTP API (for load & health)
+
+| Path | Role |
+|------|------|
+| `GET /healthz` | Liveness |
+| `GET /readyz` | Readiness (cached Inference Hub check) |
+| `POST /bench?ms=450&threads=2` | CPU load for HPA tests (worker threads) |
+| `POST /v1/chat/completions` | Proxy to Inference Hub (same model as NemoClaw) |
+| `GET /metrics` | Prometheus metrics |
+
+---
+
+## Agent building checklist
+
+When implementing or extending a **Kubernetes HPA agent** for NemoClaw:
+
+- [ ] Distinguish VM NemoClaw vs K8s agent tier in docs and tests
+- [ ] Confirm metrics-server before HPA
+- [ ] Use saturate values on single-node 8 vCPU clusters
+- [ ] Keep `/healthz` responsive under load (CPU spin off main thread)
+- [ ] Verify `endpoints` has multiple IPs during load test
+- [ ] Size `maxReplicas` to allocatable CPU on node
+- [ ] Store secrets in `~/.nemoclaw/secrets.env`, never commit keys
+
+---
+
+## Additional resources
+
+- Deep dive and env vars: [reference.md](reference.md)

.cursor/skills/nemoclaw-k8s-hpa/reference.md

@@ -0,0 +1,98 @@
+# NemoClaw K8s HPA — Reference
+
+## HPA math (CPU resource metric)
+
+- Target: `averageUtilization` = % of **CPU request** (not limit).
+- Example: request `1` CPU, target `35%` → HPA wants ~350m average per pod.
+- With `perPodRequest: 400m` and target `30%` → ~120m average triggers scale-up.
+
+Desired replicas (simplified):
+
+```text
+desired = ceil(currentReplicas × (currentCPU% / targetCPU%))
+```
+
+Scale-up stabilization: **0s**. Scale-down: **120s** (chart default).
+
+## 8 vCPU node capacity (example)
+
+| Config | 7 pods schedulable? |
+|--------|---------------------|
+| `perPodRequest: 1` | Usually **no** (~7 + system > 8) |
+| `perPodRequest: 400m` | **Often yes** (~2.8 + ~1.5 system) |
+
+Check: `kubectl describe node | grep -A6 "Allocated resources"`
+
+## Load generator env vars
+
+Used by `scripts/hpa-load-test.sh` → Job `nemoclaw-hpa-load-test`:
+
+| Variable | Default | Meaning |
+|----------|---------|---------|
+| `TARGET_PODS` | 7 | HPA max / load target |
+| `CONCURRENCY_PER_POD` | 40 | In-flight requests per pod (via Service) |
+| `BENCH_MS` | 450 | CPU spin per `/bench` |
+| `BENCH_THREADS` | 2 | Worker threads per bench (agent) |
+| `JOB_PARALLELISM` | 1 | Load Job pods (default: one generator) |
+| `DURATION_SEC` | 720 | Test duration |
+| `RAMP_SEC` | 90 | Ramp to full concurrency |
+| `HPA_TARGET_CPU` | 30 | Helm HPA target % |
+| `SCALE_UP_TARGET` | 7 | Script success threshold |
+
+Heavier run:
+
+```bash
+CONCURRENCY_PER_POD=55 BENCH_MS=500 JOB_PARALLELISM=3 HPA_TARGET_CPU=25 \
+  ./scripts/hpa-load-test.sh
+```
+
+## Helm values quick reference
+
+```yaml
+# Manual scale
+cpuScaling:
+  oneReplicaPerCpu: true
+  count: 4
+  perPodRequest: "1"
+
+# HPA
+autoscaling:
+  enabled: true
+  minReplicas: 1
+  maxReplicas: 7
+  targetCPUUtilizationPercentage: 30
+
+# Load test CPU on agent
+loadTest:
+  cpuSpinMs: 450
+```
+
+## Inference: VM vs K8s
+
+| | VM NemoClaw | K8s agent pod |
+|--|-------------|---------------|
+| Telegram | Yes | No |
+| Config | `~/.nemoclaw/`, openclaw.json | Helm values + Secret |
+| Model | Nemotron Ultra via Inference Hub | Same via proxy |
+| Scales with HPA | No | Yes |
+
+## Performance HPA (Step 2b, optional)
+
+Requires Prometheus + prometheus-adapter. Custom metric: `nemoclaw_http_inflight_requests` from `/metrics`. See `values-step2-hpa-performance.yaml` and `deploy/helm/nemoclaw-cpu/observability.md`.
+
+## Reset + load test scripts
+
+| Script | Purpose |
+|--------|---------|
+| `scripts/hpa-reset.sh` | Delete Jobs/HPA/stuck pods; reinstall baseline (min 1 replica) |
+| `scripts/hpa-load-test.sh` | Run load Job; wait for scale-up/down |
+| `scripts/hpa-common.sh` | Shared helpers (never scale to 0) |
+
+## Git / fork notes
+
+- Fork branch work: Inference Hub, Tavily, Telegram fixes live on VM sandbox path.
+- `deploy/helm/nemoclaw-cpu/` may be uncommitted — commit when stabilizing HPA agent.
+
+## Riva HPA pattern (prior art)
+
+NVIDIA blog: autoscaling Riva with K8s HPA + Grafana — same pattern (metrics → HPA → dashboards). CPU HPA is Step 2a; queue/latency metrics are Step 2b.

.gitignore

@@ -43,6 +43,7 @@ vdr-notes/
 .npmrc
 .pypirc
 credentials.json
+secrets.env
 DRAFT-*.md
 key.json
 secrets.json

Dockerfile

@@ -409,9 +409,10 @@ RUN chmod 755 /usr/local/bin/nemoclaw-start /usr/local/bin/nemoclaw-codex-acp \
 
 # Build args for config that varies per deployment.
 # nemoclaw onboard passes these at image build time.
-ARG NEMOCLAW_MODEL=nvidia/nemotron-3-super-120b-a12b
+# Nemotron Ultra — inference-api.nvidia.com (see src/lib/inference/config.ts)
+ARG NEMOCLAW_MODEL=nvidia/nvidia/llama-3.1-nemotron-ultra-253b-v1
 ARG NEMOCLAW_PROVIDER_KEY=inference
-ARG NEMOCLAW_PRIMARY_MODEL_REF=inference/nvidia/nemotron-3-super-120b-a12b
+ARG NEMOCLAW_PRIMARY_MODEL_REF=inference/nvidia/nvidia/llama-3.1-nemotron-ultra-253b-v1
 # Default dashboard port 18789 — override at runtime via NEMOCLAW_DASHBOARD_PORT.
 ARG CHAT_UI_URL=http://127.0.0.1:18789
 ARG NEMOCLAW_INFERENCE_BASE_URL=https://inference.local/v1
@@ -488,6 +489,7 @@ ARG NEMOCLAW_PROXY_PORT=3128
 # The actual API key is injected at runtime via openshell:resolve:env, never
 # baked into the image.
 ARG NEMOCLAW_WEB_SEARCH_ENABLED=0
+ARG NEMOCLAW_WEB_SEARCH_PROVIDER=brave
 
 # SECURITY: Promote build-args to env vars so the Python script reads them
 # via os.environ, never via string interpolation into Python source code.
@@ -515,7 +517,8 @@ ENV NEMOCLAW_MODEL=${NEMOCLAW_MODEL} \
     NEMOCLAW_DISABLE_DEVICE_AUTH=${NEMOCLAW_DISABLE_DEVICE_AUTH} \
     NEMOCLAW_PROXY_HOST=${NEMOCLAW_PROXY_HOST} \
     NEMOCLAW_PROXY_PORT=${NEMOCLAW_PROXY_PORT} \
-    NEMOCLAW_WEB_SEARCH_ENABLED=${NEMOCLAW_WEB_SEARCH_ENABLED}
+    NEMOCLAW_WEB_SEARCH_ENABLED=${NEMOCLAW_WEB_SEARCH_ENABLED} \
+    NEMOCLAW_WEB_SEARCH_PROVIDER=${NEMOCLAW_WEB_SEARCH_PROVIDER}
 
 WORKDIR /sandbox
 USER sandbox