fix: resolve GPU sandbox creation failures on DGX machines

[dnandakumar-nv]

Problem

Running nemoclaw onboard on a DGX machine (or any host with an NVIDIA GPU where nimCapable = true) fails silently in Step 3, then crashes in Step 7 with:

Error: × status: FailedPrecondition, message: "GPU sandbox requested, but the active gateway has no allocatable GPUs."
...
Error: × status: NotFound, message: "sandbox not found"
Command failed (exit 1): openshell policy set --policy "/tmp/nemoclaw-policy-....yaml" --wait "<sandbox-name>"

The user sees "✓ Sandbox created" but the sandbox was never actually created in openshell, leaving them with a broken, half-configured install.

Root Cause Analysis

Three bugs conspire to produce this failure:

Bug 1 — Pipe masks sandbox creation exit code (primary cause of "sandbox not found")

File: bin/lib/onboard.js:204

// BEFORE — awk always exits 0, masking openshell's failure
run(`openshell sandbox create ${createArgs.join(" ")} -- ... 2>&1 | awk '...'`);

Bash pipelines return the last command's exit code — awk's, not openshell's. When openshell sandbox create --gpu fails, awk processes the error output and exits 0. run() checks result.status !== 0 and sees nothing wrong, so execution continues. NemoClaw registers the sandbox in its local registry and prints ✓ Sandbox 'X' created even though openshell never created it. Step 7 then calls openshell policy set ... <sandbox-name> on a sandbox that doesn't exist → "sandbox not found".

Bug 2 — GPU device plugin not ready on DGX (root trigger)

File: bin/lib/onboard.js:109–145

The gateway health check only polls openshell status until it reports "Connected" (HTTP endpoint up), then sleeps 5 seconds for DNS. On DGX machines running k3s-inside-Docker, the NVIDIA GPU device plugin needs additional time to enumerate GPUs and advertise them to the Kubernetes scheduler after the HTTP endpoint becomes healthy. The 5-second sleep is not enough, so sandbox create --gpu is issued while the gateway has no allocatable GPU resources yet → FailedPrecondition.

Bug 3 — Stale port 18789 forward silently blocks new sandbox dashboard

File: bin/lib/onboard.js:207

A prior onboard attempt (e.g. using the default my-assistant name) leaves port 18789 forwarded to the old sandbox. When the new sandbox tries to claim the same port, openshell forward start fails. The ignoreError: true silently swallows the error, and the new sandbox's web dashboard is never reachable on localhost:18789.

Fix

1. set -o pipefail on sandbox create command

// AFTER — pipefail propagates openshell's exit code through the awk pipe
run(`set -o pipefail; openshell sandbox create ${createArgs.join(" ")} -- ... 2>&1 | awk '...'`);

Now if openshell sandbox create exits non-zero, run() detects it and exits with an actionable error instead of continuing into a broken state.

2. Poll for GPU allocatability before sandbox creation

After the gateway health check, if gpu.nimCapable is true, the onboarder now polls openshell gateway info (up to 12 × 10s = 2 minutes) waiting for the GPU device plugin to register resources. If GPUs become allocatable, it proceeds confidently. If the timeout expires, it emits a warning and proceeds anyway (the pipefail fix then catches any subsequent failure cleanly).

3. Stop stale port 18789 forward before claiming it

run(`openshell forward stop 18789 2>/dev/null || true`, { ignoreError: true });
run(`openshell forward start --background 18789 "${sandboxName}"`, { ignoreError: true });

Any port 18789 forward from a previous onboard is released first, ensuring the new sandbox's dashboard is always reachable.

Failure Mode Walkthrough (Before vs After)

Step	Before	After
Gateway healthy	✓	✓
GPU device plugin ready?	Not checked — proceeds immediately	Polled up to 2 min
sandbox create --gpu fails	Exit code masked by awk; "✓ Sandbox created" printed	pipefail surfaces the error; process exits with clear message
Port 18789 forward	Fails silently if port in use	Old forward stopped first; new forward succeeds
Step 7 policy apply	"sandbox not found" crash	Never reached if sandbox creation failed

Testing

Reproduce with a DGX (or any NVIDIA GPU host) by running ./install.sh immediately after gateway start to trigger the device plugin race. With this fix:

• Bug 1: verified by unit-testing that a non-zero openshell exit propagates when piped through awk
• Bug 2: the poll loop prevents the FailedPrecondition error in all observed DGX runs
• Bug 3: openshell forward stop is idempotent (|| true) so it's safe on first-ever runs too

Files Changed

• bin/lib/onboard.js — all three fixes

[ericksoa]

Pushed a follow-up commit (21c98f8) on top of yours.

Your pipefail and stale port forward fixes are solid and I kept them as-is. The additional commit removes --gpu from gateway start and sandbox create entirely, and drops the GPU allocatability poll loop.

Why: The official NVIDIA DGX Spark playbook does not use --gpu on openshell gateway start or openshell sandbox create. On Spark, inference is routed through a host-side provider (Ollama/vLLM/cloud API) via openshell provider create + openshell inference set — the sandbox never needs direct GPU access. The playbook's troubleshooting table even says "No GPU detected → Expected on DGX Spark; proceeds normally."

GPU resources will never become allocatable on Spark because that's not how the architecture works, so the poll loop would always time out (2 min wait for nothing). With your pipefail fix, that timeout would then surface as a hard failure — correct behavior given the premise, but the premise itself is wrong.

Filed #239 to track this.

[kjw3]

e2e complete along with testing race conditions and stale scenarios on main and with fixes.

Using NVIDIA Cloud API with model: nvidia/nemotron-3-super-120b-a12b

  [5/7] Setting up inference provider
  ──────────────────────────────────────────────────
✓ Created provider nvidia-nim
Gateway inference configured:

  Route: inference.local
  Provider: nvidia-nim
  Model: nvidia/nemotron-3-super-120b-a12b
  Version: 1
  ✓ Inference route set: nvidia-nim / nvidia/nemotron-3-super-120b-a12b

  [6/7] Setting up OpenClaw inside sandbox
  ──────────────────────────────────────────────────
  ✓ OpenClaw gateway launched inside sandbox

  [7/7] Policy presets
  ──────────────────────────────────────────────────

  Available policy presets:
    ○ discord — Discord API, gateway, and CDN access
    ○ docker — Docker Hub and NVIDIA container registry access
    ○ huggingface — Hugging Face Hub, LFS, and Inference API access
    ○ jira — Jira and Atlassian Cloud access
    ○ npm — npm and Yarn registry access (suggested)
    ○ outlook — Microsoft Outlook and Graph API access
    ○ pypi — Python Package Index (PyPI) access (suggested)
    ○ slack — Slack API and webhooks access
    ○ telegram — Telegram Bot API access

  Apply suggested presets (pypi, npm)? [Y/n/list]: 
✓ Policy version 2 submitted (hash: 6fa9c625721b)
✓ Policy version 2 loaded (active version: 2)
  Applied preset: pypi
✓ Policy version 3 submitted (hash: 969e5739a280)
✓ Policy version 3 loaded (active version: 3)
  Applied preset: npm
  ✓ Policies applied

  ──────────────────────────────────────────────────
  Sandbox      meme (Landlock + seccomp + netns)
  Model        nvidia/nemotron-3-super-120b-a12b (NVIDIA Cloud API)
  NIM          not running
  ──────────────────────────────────────────────────
  Run:         nemoclaw meme connect
  Status:      nemoclaw meme status
  Logs:        nemoclaw meme logs --follow
  ──────────────────────────────────────────────────

[INFO]  === Installation complete ===
$~/git/nvidia/NemoClaw$ nemoclaw meme connect
sandbox@meme:~$ openclaw nemoclaw status
(node:2468) [UNDICI-EHPA] Warning: EnvHttpProxyAgent is experimental, expect them to change at any time.
(Use `node --trace-warnings ...` to show where the warning was created)
(node:2476) [UNDICI-EHPA] Warning: EnvHttpProxyAgent is experimental, expect them to change at any time.
(Use `node --trace-warnings ...` to show where the warning was created)
20:35:29 [plugins] plugins.allow is empty; discovered non-bundled plugins may auto-load: nemoclaw (/sandbox/.openclaw/extensions/nemoclaw/dist/index.js). Set plugins.allow to explicit trusted ids.
20:35:29 [plugins] 
20:35:29 [plugins]   ┌─────────────────────────────────────────────────────┐
20:35:29 [plugins]   │  NemoClaw registered                                │
20:35:29 [plugins]   │                                                     │
20:35:29 [plugins]   │  Endpoint:  build.nvidia.com                        │
20:35:29 [plugins]   │  Model:     nvidia/nemotron-3-super-120b-a12b       │
20:35:29 [plugins]   │  Commands:  openclaw nemoclaw <command>             │
20:35:29 [plugins]   └─────────────────────────────────────────────────────┘
20:35:29 [plugins] 

🦞 OpenClaw 2026.3.11 (29dc654) — Somewhere between 'hello world' and 'oh god what have I built.'

20:35:29 [plugins] NemoClaw Status
20:35:29 [plugins] ===============
20:35:29 [plugins] 
20:35:29 [plugins] Context: running inside an active OpenShell sandbox
20:35:29 [plugins]   Host sandbox state is not inspectable from inside the sandbox.
20:35:29 [plugins]   Run 'openshell sandbox status' on the host for full details.
20:35:29 [plugins] 
20:35:29 [plugins] Plugin State:
20:35:29 [plugins]   No operations have been performed yet.
20:35:29 [plugins] 
20:35:29 [plugins] Sandbox:
20:35:29 [plugins]   Name:    openclaw
20:35:29 [plugins]   Status:  active (inside sandbox)
20:35:29 [plugins]   Note:    Cannot query host sandbox state from within the sandbox.
20:35:29 [plugins] 
20:35:29 [plugins] Inference:
20:35:29 [plugins]   Status:  unable to query from inside sandbox
20:35:29 [plugins]   Note:    Run 'openshell inference get' on the host to check.