Skip to content

Dttest - #6982

Closed
maggiezha wants to merge 26 commits into
NVIDIA:mainfrom
maggiezha:dttest
Closed

Dttest#6982
maggiezha wants to merge 26 commits into
NVIDIA:mainfrom
maggiezha:dttest

Conversation

@maggiezha

@maggiezha maggiezha commented Jul 16, 2026

Copy link
Copy Markdown

Summary

Related Issue

Changes

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification:
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification:
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification:
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result:
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Your Name your-email@example.com

Summary by CodeRabbit

  • New Features

    • Added Tavily as an alternative web-search provider, including onboarding, policy setup, configuration, and verification support.
    • Added Nemotron Ultra inference routing and updated default model configurations.
    • Added CPU and GPU Kubernetes deployments with Helm-based autoscaling, monitoring, load testing, and recovery workflows.
    • Added a CLI command for creating a secure local secrets file.
  • Documentation

    • Added comprehensive CPU/GPU deployment guides, HPA references, observability guidance, and fork-specific setup instructions.
  • Security

    • Improved credential handling, provider-specific validation, and protection for local secrets.

maggiezha and others added 25 commits May 27, 2026 23:53
Split NVIDIA Build (nvapi-*) and Inference Hub (sk-*) env vars, load keys from
~/.nemoclaw/secrets.env, route Ultra via inference-api, and fix Tavily on rebuild resume.
Keep Tavily/Telegram support alongside upstream Brave egress verification and provider profile registration.
Removed unnecessary details about Dual NVIDIA keys and Secrets file while retaining key features for clarity.
Updated the README.fork.md to include new information and images.
Set toolSearch false for Ultra and document web_search usage so Tavily runs instead of failing tool_search_code.
Introduce the nemoclaw-cpu Helm chart with metrics-server-based autoscaling,
install/recover/reset/load-test scripts, and documentation for running
Inference Hub agents on MicroK8s (min 1, max 7 replicas per 8 vCPU).
Added an image and recovery instructions to README.
Introduce nemoclaw-gpu (Ollama + agent, DCGM gpu_utilization_percent HPA) with load-test and ops scripts tuned for 4× L40S. Split deploy guides into README-gpu/README-cpu and refresh the CPU chart helpers.
Removed 'GPU' from the title of the deployment README.
Updated images in README-gpu.md to include width and height attributes for better display.
Updated image syntax and added additional screenshots in the README for GPU deployment.
Updated README for GPU deployment with HPA details.
Signed-off-by: maggiezha <42832776+maggiezha@users.noreply.github.com>
Added port 9000 for the CPU deployment section.
Removed the third port entry from the table.
Signed-off-by: maggiezha <42832776+maggiezha@users.noreply.github.com>
@copy-pr-bot

copy-pr-bot Bot commented Jul 16, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 2efe58de-d137-42d0-b6a2-df39842d6fa0

📥 Commits

Reviewing files that changed from the base of the PR and between acbe904 and 72a721a.

📒 Files selected for processing (127)
  • .cursor/skills/nemoclaw-k8s-hpa/SKILL.md
  • .cursor/skills/nemoclaw-k8s-hpa/reference.md
  • .gitignore
  • Dockerfile
  • README.fork.md
  • README.md
  • deploy/README-cpu.md
  • deploy/README-gpu.md
  • deploy/README.md
  • deploy/helm/README.md
  • deploy/helm/nemoclaw-cpu/.helmignore
  • deploy/helm/nemoclaw-cpu/Chart.yaml
  • deploy/helm/nemoclaw-cpu/README.md
  • deploy/helm/nemoclaw-cpu/files/agent-metrics.mjs
  • deploy/helm/nemoclaw-cpu/files/agent-server.mjs
  • deploy/helm/nemoclaw-cpu/files/cpu-spin-worker.mjs
  • deploy/helm/nemoclaw-cpu/files/load-generator.mjs
  • deploy/helm/nemoclaw-cpu/files/questions-sample.txt
  • deploy/helm/nemoclaw-cpu/monitoring/kube-prometheus-microk8s.yaml
  • deploy/helm/nemoclaw-cpu/monitoring/prometheus-adapter-values.yaml
  • deploy/helm/nemoclaw-cpu/observability.md
  • deploy/helm/nemoclaw-cpu/scripts/cluster-recover.sh
  • deploy/helm/nemoclaw-cpu/scripts/hpa-common.sh
  • deploy/helm/nemoclaw-cpu/scripts/hpa-load-test.sh
  • deploy/helm/nemoclaw-cpu/scripts/hpa-reset.sh
  • deploy/helm/nemoclaw-cpu/scripts/install-hpa.sh
  • deploy/helm/nemoclaw-cpu/scripts/install-performance-hpa.sh
  • deploy/helm/nemoclaw-cpu/templates/NOTES.txt
  • deploy/helm/nemoclaw-cpu/templates/_helpers.tpl
  • deploy/helm/nemoclaw-cpu/templates/configmap.yaml
  • deploy/helm/nemoclaw-cpu/templates/deployment.yaml
  • deploy/helm/nemoclaw-cpu/templates/hpa.yaml
  • deploy/helm/nemoclaw-cpu/templates/namespace.yaml
  • deploy/helm/nemoclaw-cpu/templates/secret.yaml
  • deploy/helm/nemoclaw-cpu/templates/service.yaml
  • deploy/helm/nemoclaw-cpu/templates/servicemonitor.yaml
  • deploy/helm/nemoclaw-cpu/values-step2-hpa-performance.yaml
  • deploy/helm/nemoclaw-cpu/values-step2-hpa-saturate.yaml
  • deploy/helm/nemoclaw-cpu/values-step2-hpa.yaml
  • deploy/helm/nemoclaw-cpu/values.yaml
  • deploy/helm/nemoclaw-gpu/.helmignore
  • deploy/helm/nemoclaw-gpu/Chart.yaml
  • deploy/helm/nemoclaw-gpu/README.md
  • deploy/helm/nemoclaw-gpu/files/agent-metrics.mjs
  • deploy/helm/nemoclaw-gpu/files/agent-server.mjs
  • deploy/helm/nemoclaw-gpu/files/load-generator.mjs
  • deploy/helm/nemoclaw-gpu/files/ollama-start.sh
  • deploy/helm/nemoclaw-gpu/files/questions-sample.txt
  • deploy/helm/nemoclaw-gpu/monitoring/dcgm-servicemonitor.yaml
  • deploy/helm/nemoclaw-gpu/monitoring/kube-prometheus-microk8s.yaml
  • deploy/helm/nemoclaw-gpu/monitoring/prometheus-adapter-gpu-values.yaml
  • deploy/helm/nemoclaw-gpu/scripts/cluster-recover.sh
  • deploy/helm/nemoclaw-gpu/scripts/get-agent-pods.sh
  • deploy/helm/nemoclaw-gpu/scripts/get-hpa.sh
  • deploy/helm/nemoclaw-gpu/scripts/hpa-common.sh
  • deploy/helm/nemoclaw-gpu/scripts/hpa-load-test.sh
  • deploy/helm/nemoclaw-gpu/scripts/hpa-reset.sh
  • deploy/helm/nemoclaw-gpu/scripts/hpa-watch.sh
  • deploy/helm/nemoclaw-gpu/scripts/install-hpa.sh
  • deploy/helm/nemoclaw-gpu/templates/NOTES.txt
  • deploy/helm/nemoclaw-gpu/templates/_helpers.tpl
  • deploy/helm/nemoclaw-gpu/templates/configmap.yaml
  • deploy/helm/nemoclaw-gpu/templates/deployment.yaml
  • deploy/helm/nemoclaw-gpu/templates/hpa.yaml
  • deploy/helm/nemoclaw-gpu/templates/namespace.yaml
  • deploy/helm/nemoclaw-gpu/templates/pvc.yaml
  • deploy/helm/nemoclaw-gpu/templates/service.yaml
  • deploy/helm/nemoclaw-gpu/templates/servicemonitor.yaml
  • deploy/helm/nemoclaw-gpu/values-load-test-hpa.yaml
  • deploy/helm/nemoclaw-gpu/values-step2-hpa-latency.yaml
  • deploy/helm/nemoclaw-gpu/values-step2-hpa-performance.yaml
  • deploy/helm/nemoclaw-gpu/values-step2-hpa.yaml
  • deploy/helm/nemoclaw-gpu/values.yaml
  • deploy/scripts/install-both.sh
  • deploy/scripts/status-both.sh
  • deploy/scripts/uninstall-both.sh
  • docs/network-policy/integration-policy-examples.mdx
  • docs/security/credential-storage.mdx
  • fibonacci.py
  • nemoclaw-blueprint/blueprint.yaml
  • nemoclaw-blueprint/model-specific-setup/openclaw/nemotron-ultra-managed-inference.json
  • nemoclaw-blueprint/policies/presets/tavily.yaml
  • nemoclaw-blueprint/router/pool-config.yaml
  • nemoclaw/src/index.ts
  • nemoclaw/src/register.test.ts
  • scripts/checks/direct-credential-env.ts
  • scripts/examples/nemotron-ultra-inference.py
  • scripts/generate-openclaw-config.py
  • scripts/setup-tavily-search.sh
  • scripts/test-tavily-flow.sh
  • scripts/verify-tavily-openclaw-config.py
  • secrets.env.example
  • src/commands/credentials/init-secrets.ts
  • src/lib/cli/public-display-defaults.ts
  • src/lib/credentials/command-support.ts
  • src/lib/credentials/secrets-env.test.ts
  • src/lib/credentials/secrets-env.ts
  • src/lib/credentials/store.ts
  • src/lib/inference/config.test.ts
  • src/lib/inference/config.ts
  • src/lib/inference/model-prompts.ts
  • src/lib/inference/provider-models.ts
  • src/lib/inference/web-search.ts
  • src/lib/onboard.ts
  • src/lib/onboard/dashboard.ts
  • src/lib/onboard/dockerfile-patch.ts
  • src/lib/onboard/machine/handlers/finalization.ts
  • src/lib/onboard/machine/handlers/provider-inference.ts
  • src/lib/onboard/machine/handlers/sandbox.ts
  • src/lib/onboard/policy-presets.ts
  • src/lib/onboard/policy-selection.ts
  • src/lib/onboard/providers.test.ts
  • src/lib/onboard/providers.ts
  • src/lib/onboard/summary.test.ts
  • src/lib/onboard/summary.ts
  • src/lib/onboard/web-search-flow.ts
  • src/lib/onboard/web-search-verify.test.ts
  • src/lib/onboard/web-search-verify.ts
  • src/lib/policy/index.ts
  • src/lib/state/onboard-session.ts
  • src/lib/validation.test.ts
  • src/lib/validation.ts
  • test/credentials.test.ts
  • test/generate-openclaw-config.test.ts
  • test/onboard-brave-validation.test.ts
  • test/onboard-policy-suggestions.test.ts
  • test/validate-blueprint.test.ts

📝 Walkthrough

Walkthrough

This PR adds Kubernetes HPA deployment support (CPU and GPU Helm charts, agent servers, monitoring, and operational scripts), reroutes default NVIDIA inference to Nemotron Ultra with new endpoint/credential routing, introduces local secrets file storage, and adds Tavily web-search as an alternative to Brave with corresponding onboarding, config generation, and validation changes.

Changes

Platform features

Layer / File(s) Summary
CPU and GPU Kubernetes HPA deployments
deploy/helm/nemoclaw-cpu/*, deploy/helm/nemoclaw-gpu/*, deploy/README*.md, .cursor/skills/nemoclaw-k8s-hpa/*, deploy/scripts/*-both.sh
New CPU/GPU Helm charts with agent servers, metrics, load generators, HPA templates, monitoring configs, operational scripts, and documentation; deprecated combined install/status/uninstall scripts.
Nemotron Ultra routing and local credentials
src/lib/inference/config.ts, src/lib/credentials/store.ts, src/lib/credentials/secrets-env.ts, src/commands/credentials/init-secrets.ts, Dockerfile, nemoclaw/src/index.ts, nemoclaw-blueprint/*
New NVIDIA model/endpoint routing (resolveNvidiaCloudModelRoute), Inference Hub vs Build credential handling, and a local secrets.env file mechanism with a new CLI command.
Tavily web-search integration
src/lib/onboard/web-search-flow.ts, src/lib/inference/web-search.ts, scripts/generate-openclaw-config.py, scripts/setup-tavily-search.sh, docs/network-policy/*
Web search configuration now supports Brave or Tavily providers throughout onboarding, policy selection, sandbox verification, and OpenClaw config generation.
Supporting documentation and regression validation
docs/security/credential-storage.mdx, test/validate-blueprint.test.ts, fibonacci.py
Credential storage docs and blueprint regression tests updated to reflect new endpoints and models.

Estimated code review effort: 4 (Complex) | ~75 minutes

Possibly related PRs

  • NVIDIA/NemoClaw#6977: Both PRs add the same NemoClaw CPU HPA Cursor skill/reference docs plus the same core web-search/Tavily and Dockerfile/secrets changes.

Suggested labels: feature, area: docs, area: inference, integration: telegram

Suggested reviewers: cv

Poem

🐰 A rabbit hops through pods anew,
HPA scales as CPUs grew,
Nemotron Ultra lights the way,
Tavily searches, come what may,
Secrets tucked in files so tight—
This burrow's deploy is set just right!

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
⚔️ Resolve merge conflicts
  • Resolve merge conflict in branch dttest

Comment @coderabbitai help to get the list of available commands.

Warning

⚠️ This pull request has been flagged as potential spam (other-spam) by CodeRabbit slop detection and should be reviewed carefully.

Signed-off-by: maggiezha <42832776+maggiezha@users.noreply.github.com>
@github-actions

github-actions Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — Blocking findings reported

Advisor assessment: Blockers require maintainer review
Next action: Review the blockers below.
Findings: 6 blockers · 3 warnings · 0 suggestions

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 6 blockers · 3 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Failed

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, credential-sanitization, full-e2e, hermes-e2e, security-posture, inference-routing, network-policy, onboard-repair, onboard-resume

3 optional E2E recommendations
  • gpu-e2e
  • ollama-auth-proxy
  • onboard-negative-paths

Blockers

PRA-1 Blocker — Recompute the endpoint when repairing a resumed NVIDIA model route

  • Location: src/lib/onboard/machine/handlers/provider-inference.ts:190
  • Category: correctness
  • Problem: On the completed-provider resume path for `nvidia-prod`, the code replaces `credentialEnv` with the model-derived value but only assigns the model-derived endpoint when the persisted endpoint is empty. A session retaining an Integrate endpoint and resumed with the Ultra model therefore rebuilds with the Inference Hub credential against the stale Integrate URL.
  • Impact: Resumed onboarding can register or retain an incompatible inference route, causing requests for the selected Ultra model to fail or be sent to the wrong NVIDIA service.
  • Fix: Assign both `credentialEnv` and `endpointUrl` from `resolveNvidiaCloudModelRoute(model)` on this repair path, rather than preserving a persisted endpoint for a model-specific route.
  • Verification: Inspect a resumed `nvidia-prod` session whose persisted endpoint is `https://integrate.api.nvidia.com/v1\` and model is `NVIDIA_NEMOTRON_ULTRA_MODEL`; verify the setup call receives `https://inference-api.nvidia.com/v1\`.
  • Test coverage: Add a provider-inference handler test with a completed `nvidia-prod` resume session containing the stale Integrate endpoint and Ultra model, asserting `setupInference` receives both the Inference Hub credential environment and Inference Hub API base URL.
  • Evidence: src/lib/onboard/machine/handlers/provider-inference.ts:188-190 replaces `credentialEnv` from `resolveNvidiaCloudModelRoute(model)` but uses `endpointUrl = endpointUrl || nvidiaRoute.apiBaseUrl`. src/lib/inference/config.ts defines model-specific API bases and `resolveNvidiaCloudModelRoute()` returns the matching route. src/lib/onboard/machine/handlers/provider-inference.test.ts has recovery coverage but no `nvidia-prod` stale-endpoint resume case.

PRA-2 Blocker — Limit recovery cleanup to resources owned by this chart

  • Location: deploy/helm/nemoclaw-cpu/scripts/cluster-recover.sh:28
  • Category: correctness
  • Problem: The CPU and GPU recovery scripts delete every HPA, Job, Pod and later every Deployment/ReplicaSet/HPA/Job in their configurable namespaces, with no ownership selector on most commands. The documented recovery operation can therefore remove workloads that do not belong to the NemoClaw chart.
  • Impact: Running the advertised recovery command in a shared or misconfigured namespace can forcibly terminate and delete unrelated application workloads and their batch jobs.
  • Fix: Restrict every delete and finalizer-clear operation to the release/chart ownership labels (and the intended release name), or use explicit names for resources created by these scripts; do not use namespace-wide `--all` deletion for recovery.
  • Verification: Read `deploy/helm/nemoclaw-{cpu,gpu}/scripts/cluster-recover.sh` and confirm that the `kubectl delete ... --all` commands operate on all matching resource kinds in `$NAMESPACE` without a chart/release selector.
  • Test coverage: Add a shell-level fixture or static script test that places chart-labeled and unrelated-labeled resources in the target namespace and asserts recovery delete arguments select only the chart-owned resources.
  • Evidence: deploy/helm/nemoclaw-cpu/scripts/cluster-recover.sh:28-30 deletes all HPAs, Jobs, and Pods in `$NAMESPACE`; :36 deletes all Deployments, ReplicaSets, HPAs, and Jobs. deploy/helm/nemoclaw-gpu/scripts/cluster-recover.sh:20-22 and :28 delete the same resource classes namespace-wide. deploy/README-cpu.md and deploy/README-gpu.md document `cluster-recover.sh` as the remedy for rollout failures. The risk-plan lifecycle invariant requires cleanup to preserve unrelated sandboxes and remove only owned resources.

PRA-3 Blocker — Do not pass the Inference Hub key in Helm command arguments

  • Location: deploy/helm/nemoclaw-cpu/scripts/install-hpa.sh:49
  • Category: security
  • Problem: The CPU deployment scripts interpolate `NVIDIA_INFERENCE_HUB_API_KEY` into `helm --set inference.apiKey=...`. Command-line arguments are observable to local users/processes and Helm stores supplied values with release metadata, so the key is exposed outside the intended Secret boundary.
  • Impact: A cluster operator’s Inference Hub credential can be recovered from process inspection or Helm release values and then used to make inference requests.
  • Fix: Create or require a pre-existing Kubernetes Secret and pass only `inference.existingSecret`/its name to Helm; remove secret-valued `--set inference.apiKey` arguments from all install, reset, load-test, and performance scripts.
  • Verification: Inspect `deploy/helm/nemoclaw-cpu/scripts/{install-hpa,hpa-common,hpa-load-test,hpa-reset,install-performance-hpa}.sh` and verify each supplies the credential through `--set inference.apiKey`.
  • Test coverage: Add a static shell-script test asserting CPU chart scripts never interpolate `NVIDIA_INFERENCE_HUB_API_KEY` or an API-key parameter into Helm arguments and instead reference an existing Secret name.
  • Evidence: deploy/helm/nemoclaw-cpu/scripts/install-hpa.sh:49 passes `--set inference.apiKey="${NVIDIA_INFERENCE_HUB_API_KEY}"`. The same secret-valued Helm setting appears in hpa-common.sh, hpa-load-test.sh, hpa-reset.sh, and install-performance-hpa.sh. deploy/helm/nemoclaw-cpu/templates/secret.yaml supports an `inference.existingSecret` alternative.

PRA-4 Blocker — Restrict access to the credential-backed inference proxy

  • Location: deploy/helm/nemoclaw-cpu/templates/service.yaml:1
  • Category: security
  • Problem: The CPU chart exposes unauthenticated chat-completions through a ClusterIP Service while injecting `NVIDIA_INFERENCE_HUB_API_KEY` into the agent pod. The chart supplies neither request authentication nor a NetworkPolicy, so any pod with namespace/network reachability can use that credential-backed proxy.
  • Impact: An untrusted or compromised workload in the cluster can consume the operator’s paid Inference Hub credential and send arbitrary prompts through the proxy.
  • Fix: Add a default-deny ingress policy with explicit intended clients and/or authenticate and authorize proxy requests before forwarding; document the supported caller boundary.
  • Verification: Inspect `deploy/helm/nemoclaw-cpu/templates/service.yaml` and `deployment.yaml`, and confirm the chart has no NetworkPolicy template while `/v1/chat/completions` forwards requests using the injected API key.
  • Test coverage: Render the chart and add a policy/endpoint test demonstrating an allowed client can reach the Service while an unlabeled or unauthorized pod cannot invoke chat completions.
  • Evidence: deploy/helm/nemoclaw-cpu/templates/service.yaml creates a ClusterIP Service selecting the agent pods. deploy/helm/nemoclaw-cpu/templates/deployment.yaml injects `NVIDIA_INFERENCE_HUB_API_KEY` from a Secret. deploy/helm/nemoclaw-cpu/files/agent-server.mjs proxies POST `/v1/chat/completions` with `Authorization: Bearer ${API_KEY}`. No NetworkPolicy files were found under deploy/helm.

PRA-5 Blocker — Pin chart workload images to immutable reviewed digests

  • Location: deploy/helm/nemoclaw-gpu/values.yaml:21
  • Category: security
  • Problem: The GPU chart deploys `ollama/ollama:latest`; the chart workload images use tags instead of immutable digests. A registry tag update can silently change code that runs with GPU/node-adjacent privileges without a reviewed chart change.
  • Impact: A compromised or retagged upstream image can alter deployed workload behavior and compromise inference workloads on the next pull or new node.
  • Fix: Pin CPU, GPU agent, and Ollama images to reviewed SHA256 digests and update them through the repository’s dependency-review process.
  • Verification: Inspect `deploy/helm/nemoclaw-{cpu,gpu}/values.yaml` image settings and confirm they specify tags such as `latest` or `22-bookworm-slim` without `@sha256:` digests.
  • Test coverage: Add a chart lint/static test that rejects image repositories lacking an `@sha256:` digest in deploy Helm values.
  • Evidence: deploy/helm/nemoclaw-gpu/values.yaml:21-24 sets `ollama/ollama:latest`. deploy/helm/nemoclaw-gpu/values.yaml defines the agent as `node:22-bookworm-slim`. deploy/helm/nemoclaw-cpu/values.yaml defines the agent as `node:22-bookworm-slim`.

PRA-6 Blocker — Apply restrictive pod security contexts to chart workloads

  • Location: deploy/helm/nemoclaw-gpu/templates/deployment.yaml:40
  • Category: security
  • Problem: The new CPU and GPU Deployment templates define no pod or container security contexts: there is no non-root requirement, privilege-escalation denial, capability drop, or read-only root filesystem. GPU Ollama additionally runs with its image-default root user and mounts writable storage at `/root/.ollama`.
  • Impact: A compromise in the internet-facing/in-cluster proxy or model server gets unnecessary default container privileges and a larger path to host or cluster escalation.
  • Fix: Set pod/container security contexts to run as non-root where supported, disallow privilege escalation, drop all capabilities, and use read-only root filesystems with narrowly scoped writable volumes; adapt Ollama storage ownership rather than relying on image-default root.
  • Verification: Search both `deploy/helm/nemoclaw-{cpu,gpu}/templates/deployment.yaml` files for `securityContext`, `runAsNonRoot`, `allowPrivilegeEscalation`, and `readOnlyRootFilesystem`; none are present.
  • Test coverage: Add rendered-manifest tests requiring restrictive pod and container securityContext fields for CPU agent, GPU agent, and Ollama containers.
  • Evidence: No securityContext-related fields appear in either chart Deployment template. deploy/helm/nemoclaw-gpu/templates/deployment.yaml mounts writable persistence at `/root/.ollama`. Both charts create new executable workloads from these templates.
3 warnings · 0 suggestions

Warnings

Warnings do not block.

PRA-7 Warning — Bound request bodies before buffering proxy input

  • Location: deploy/helm/nemoclaw-cpu/files/agent-server.mjs:26
  • Category: security
  • Problem: The CPU and GPU agents collect every incoming request chunk and call `Buffer.concat` before parsing JSON, with no byte limit. A reachable caller can submit an arbitrarily large body and exhaust sidecar memory.
  • Impact: A malicious or faulty in-cluster client can repeatedly crash or degrade the proxy pod, disrupting inference availability and HPA behavior.
  • Recommendation: Track received bytes while streaming, destroy/reject requests over a small documented limit with HTTP 413, and avoid retaining oversized buffers.
  • Verification: Inspect `readBody` in both `deploy/helm/nemoclaw-{cpu,gpu}/files/agent-server.mjs`; it pushes every chunk without a length guard.
  • Test coverage: Add agent-server tests that submit a body just below the configured cap and one above it, asserting the latter returns 413 without invoking inference.
  • Evidence: deploy/helm/nemoclaw-cpu/files/agent-server.mjs:26-34 buffers all request chunks then calls `Buffer.concat(chunks)`. deploy/helm/nemoclaw-gpu/files/agent-server.mjs uses the same unbounded readBody implementation. The sidecars expose POST chat endpoints through the chart Services.

PRA-8 Warning — Cover refusal of unsafe secrets.env filesystem inputs

  • Location: src/lib/credentials/secrets-env.test.ts:45
  • Category: tests
  • Problem: The new secrets-file loader relies on symlink-path rejection, `O_NOFOLLOW`, regular-file validation, and a 256 KiB size cap, but its tests cover only normal parsing and successful staging. There is no regression evidence that unsafe filesystem inputs fail closed without staging credentials.
  • Impact: A future change can weaken the local credential-file trust boundary—following an attacker-controlled symlink, accepting a nonregular file, or loading an unexpectedly large file—without a test failure.
  • Recommendation: Add temporary-directory tests for a symlinked `.nemoclaw` path or `secrets.env`, a nonregular file where supported, and a file exceeding 256 KiB; assert no credential is staged in every case.
  • Verification: Compare `src/lib/credentials/secrets-env.ts` path checks, `O_NOFOLLOW`, `stat.isFile()`, and size check with `src/lib/credentials/secrets-env.test.ts`, which only tests parsing and regular-file staging.
  • Test coverage: Use temporary HOME directories to create each unsafe input and assert `stageSecretsEnvFile()` returns an empty staged-key list and leaves all credential environment variables unset.
  • Evidence: src/lib/credentials/secrets-env.ts rejects symlink paths, opens with `fs.constants.O_NOFOLLOW`, requires `stat.isFile()`, and rejects files over 256 KiB. src/lib/credentials/secrets-env.test.ts contains parsing and normal allowlisted-key staging tests only. The credentials-security risk invariant requires invalid security state to fail closed.

PRA-9 Warning — Return failure when the HPA load test misses its scale-up target

  • Location: deploy/helm/nemoclaw-cpu/scripts/hpa-load-test.sh:172
  • Category: workflow
  • Problem: The CPU and GPU load-test scripts only print an error when replicas never reach `SCALE_UP_TARGET`; they continue through cleanup and end without a nonzero exit. Callers and automation therefore receive success even though the script's advertised autoscaling proof failed.
  • Impact: Operational automation can report a successful HPA validation while no scale-up occurred, masking broken metrics, scheduling, or load generation.
  • Recommendation: After cleanup, exit nonzero when `SCALE_UP_OK` is false; also propagate load-generator readiness and completion failures while retaining cleanup in a trap.
  • Verification: Set `SCALE_UP_TARGET` above available capacity or inspect the false `SCALE_UP_OK` branch in both load-test scripts; neither branch calls `exit 1`.
  • Test coverage: Add shell-level tests with mocked `kubectl` that keep current replicas below `SCALE_UP_TARGET` and assert each CPU/GPU script returns nonzero after cleanup.
  • Evidence: deploy/helm/nemoclaw-cpu/scripts/hpa-load-test.sh:172-174 emits `HPA did not scale` but does not exit. deploy/helm/nemoclaw-gpu/scripts/hpa-load-test.sh:366-368 emits the same failure and :383-386 only logs an incomplete result. Both scripts otherwise use `set -euo pipefail` and fail early for unavailable prerequisites.

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

@maggiezha maggiezha closed this Jul 16, 2026
@maggiezha

Copy link
Copy Markdown
Author

close it as it was a testing with CodeRabbit team to make it work with the repo from a forked repo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants