Skip to content

fix: resolve GPU sandbox creation failures on DGX machines #238

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kjw3 merged 2 commits into from
Mar 17, 2026
Merged

fix: resolve GPU sandbox creation failures on DGX machines #238

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4e600af
fix: resolve GPU sandbox creation failures on DGX machines (#235)
Mar 17, 2026
21c98f8
fix: stop passing --gpu to gateway and sandbox on DGX Spark
ericksoa Mar 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 16 additions & 5 deletions bin/lib/onboard.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,11 @@ async function startGateway(gpu) {
run("openshell gateway destroy -g nemoclaw 2>/dev/null || true", { ignoreError: true });

const gwArgs = ["--name", "nemoclaw"];
if (gpu && gpu.nimCapable) gwArgs.push("--gpu");
// Do NOT pass --gpu here. On DGX Spark (and most GPU hosts), inference is
// routed through a host-side provider (Ollama, vLLM, or cloud API) — the
// sandbox itself does not need direct GPU access. Passing --gpu causes
// FailedPrecondition errors when the gateway's k3s device plugin cannot
// allocate GPUs. See: https://build.nvidia.com/spark/nemoclaw/instructions

run(`openshell gateway start ${gwArgs.join(" ")}`, { ignoreError: false });

Expand Down Expand Up @@ -143,6 +147,7 @@ async function startGateway(gpu) {
}
// Give DNS a moment to propagate
require("child_process").spawnSync("sleep", ["5"]);

}

// ── Step 3: Sandbox ──────────────────────────────────────────────
Expand Down Expand Up @@ -193,17 +198,23 @@ async function createSandbox(gpu) {
`--name "${sandboxName}"`,
`--policy "${basePolicyPath}"`,
];
if (gpu && gpu.nimCapable) createArgs.push("--gpu");
// --gpu is intentionally omitted. See comment in startGateway().

console.log(` Creating sandbox '${sandboxName}' (this takes a few minutes on first run)...`);
const chatUiUrl = process.env.CHAT_UI_URL || 'http://127.0.0.1:18789';
const envArgs = [`CHAT_UI_URL=${chatUiUrl}`];
if (process.env.NVIDIA_API_KEY) {
envArgs.push(`NVIDIA_API_KEY=${process.env.NVIDIA_API_KEY}`);
}
run(`openshell sandbox create ${createArgs.join(" ")} -- env ${envArgs.join(" ")} nemoclaw-start 2>&1 | awk '/Sandbox allocated/{if(!seen){print;seen=1}next}1'`);

// Forward dashboard port separately
// set -o pipefail ensures the openshell exit code propagates through the awk pipe.
// Without it, awk's exit code (always 0) would mask a failed sandbox create.
run(`set -o pipefail; openshell sandbox create ${createArgs.join(" ")} -- env ${envArgs.join(" ")} nemoclaw-start 2>&1 | awk '/Sandbox allocated/{if(!seen){print;seen=1}next}1'`);

// Release any stale forward on port 18789 before claiming it for the new sandbox.
// A previous onboard run may have left the port forwarded to a different sandbox,
// which would silently prevent the new sandbox's dashboard from being reachable.
run(`openshell forward stop 18789 2>/dev/null || true`, { ignoreError: true });
// Forward dashboard port to the new sandbox
run(`openshell forward start --background 18789 "${sandboxName}"`, { ignoreError: true });

// Clean up build context
Expand Down