Skip to content
2 changes: 1 addition & 1 deletion ci/onboard-entry-composition-budget.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
"provider": {
"createSandboxWithBaseImageResolution": 15,
"handleNimLocalSelection": 32,
"handleRemoteProviderSelection": 80,
"handleRemoteProviderSelection": 76,
"handleRoutedSelection": 15,
"runOnboard": 8,
"selectAndValidateOllamaModel": 18
Expand Down
22 changes: 19 additions & 3 deletions docs/reference/commands.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1009,6 +1009,14 @@ Use `--gpu` to require GPU passthrough and fail fast if an NVIDIA GPU is not det
Use `--sandbox-gpu` or `--no-sandbox-gpu` to control only direct NVIDIA GPU access inside the sandbox.
Use `--sandbox-gpu --sandbox-gpu-device <device>` to pass a specific OpenShell GPU device selector to `openshell sandbox create`; device selectors require explicit sandbox GPU enablement.
On ordinary native Linux Docker-driver hosts, NemoClaw uses native OpenShell GPU injection by default and never broadens confinement automatically.

<AgentOnly variant="openclaw">

Portable OpenClaw onboarding also requires native OpenShell GPU injection.
It does not use `NEMOCLAW_DOCKER_GPU_PATCH` compatibility routing, so do not set `fallback`, `1`, or another legacy nonzero value for that profile.

</AgentOnly>

Set `NEMOCLAW_DOCKER_GPU_PATCH=fallback` to explicitly authorize one native attempt followed by one compatibility retry.
NemoClaw permits the retry only after it confirms either a trusted host-side GPU routing failure or an explicit driver proof plus exact-container host configuration showing that no GPU was attached.
It then saves redacted diagnostics and removes the incomplete sandbox before retrying.
Expand Down Expand Up @@ -4825,7 +4833,7 @@ Set them before running `$$nemoclaw onboard`.
| `NEMOCLAW_SANDBOX_GPU_DEVICE` | OpenShell GPU device selector | Selects the GPU device passed with `openshell sandbox create --gpu-device`. Requires explicit sandbox GPU enablement with `NEMOCLAW_SANDBOX_GPU=1` (or `--sandbox-gpu` for CLI-driven onboarding); otherwise onboarding rejects the selector instead of treating it as an implicit opt-in. |
| `NEMOCLAW_SANDBOX_BASE_IMAGE_REFRESH` | `1`, `true`, `yes`, or `on` to enable | Bypasses recorded sandbox base-image resolution metadata during onboarding, recreation, and rebuild. NemoClaw reruns candidate resolution but can still use a compatible image from Docker's local image store. Versioned release candidates that exist locally but fail validation are refreshed from the registry once during normal resolution. This setting does not discard onboarding session state. |
| `NEMOCLAW_SANDBOX_BASE_LOCAL_BUILD` | unset or `auto` (default); `1`, `true`, `yes`, or `on` to enable; `0`, `false`, `no`, or `off` to disable | Controls whether base-image resolution may build a compatible image locally. The default allows builds during normal CLI runs and disables them when `NODE_ENV=test` or `VITEST=true`. When source inputs or a missing/incompatible release-version base require a fresh build, disabling local builds makes resolution fail instead of using an unproven image. |
| `NEMOCLAW_DOCKER_GPU_PATCH` | unset, `auto`, `fallback`, `1`, or `0`; other legacy nonzero values remain accepted through `v0.0.x` and will be removed in `v0.1.0` | Selects Linux Docker-driver GPU routing. Unset, `auto`, or `0` uses native OpenShell GPU injection on ordinary native Linux. `fallback` explicitly opts into one native attempt followed by one bounded compatibility retry when trusted host evidence identifies a GPU-routing failure. `1` and legacy nonzero values select the compatibility patch from the outset. Docker Desktop WSL and Jetson/Tegra use the compatibility path by default; Docker Desktop WSL ignores `0`, while Jetson/Tegra accepts `0` only as a troubleshooting override that bypasses device-group propagation. |
| `NEMOCLAW_DOCKER_GPU_PATCH` | unset, `auto`, `fallback`, `1`, or `0`; other legacy nonzero values remain accepted through `v0.0.x` and will be removed in `v0.1.0` | Selects Linux Docker-driver GPU routing. Unset, `auto`, or `0` uses native OpenShell GPU injection on ordinary native Linux. `fallback` explicitly opts into one native attempt followed by one bounded compatibility retry when trusted host evidence identifies a GPU-routing failure. `1` and legacy nonzero values select the compatibility patch from the outset. Docker Desktop WSL and Jetson/Tegra use the compatibility path by default; Docker Desktop WSL ignores `0`, while Jetson/Tegra accepts `0` only as a troubleshooting override that bypasses device-group propagation. Portable OpenClaw onboarding requires native OpenShell GPU injection and does not use this setting. |
| `NEMOCLAW_OPENSHELL_GATEWAY_CONTAINER_PATCH` | `1` to enable; disabled by default | This setting explicitly opts into the Linux gateway compatibility container for an older host ABI or a diagnostic run; use it only on a trusted local host because it uses host networking and mounts the Docker socket read-only even though the socket still exposes the privileged Docker API; prefer OpenShell 0.0.101's directly supported glibc 2.39+ path; see [Gateway Compatibility Container](/user-guide/openclaw/security/security-controls/gateway-authentication-controls#gateway-compatibility-container) for the container boundary and removal conditions. |
| `NEMOCLAW_OPENSHELL_GATEWAY_BIN` | path | Advanced override for the `openshell-gateway` binary used by Linux Docker-driver startup. For the default port, the installer accepts the binary under an absolute `XDG_BIN_HOME` when set, otherwise `~/.local/bin/openshell-gateway`; it also accepts `/usr/local/bin/openshell-gateway` or `/usr/bin/openshell-gateway`. Another path fails service staging. The macOS Homebrew service uses the formula's binary. Defaults to the binary next to `openshell`, then common install paths. |
| `NEMOCLAW_OPENSHELL_SANDBOX_BIN` | path | Advanced override for the `openshell-sandbox` binary used by Linux Docker-driver startup. The macOS Homebrew service uses the formula's driver layout. Defaults to the binary next to `openshell`, then common install paths. |
Expand Down Expand Up @@ -4963,7 +4971,7 @@ Set the onboarding variables before running `$$nemoclaw onboard` if a slow conne
|----------|---------|---------|
| `NEMOCLAW_OLLAMA_PULL_TIMEOUT` | `1800` (30 minutes) | Wall-clock timeout for `ollama pull` during onboard, in seconds. Accepts integer or float values. Already-downloaded layers are kept; re-running the pull resumes them. |
| `NEMOCLAW_LOCAL_INFERENCE_TIMEOUT` | `180` | Wall-clock timeout for the inference-server validation probe during onboard, in seconds. Raise on slow networks or for very large prompts. |
| `NEMOCLAW_SANDBOX_READY_TIMEOUT` | `180` | Wall-clock timeout for post-create readiness, in seconds. Raise when the sandbox image build, gateway upload, or in-sandbox boot exceeds the default (typical on 70B+ models, first-time gateway uploads over slow links, or DGX Station / remote-VM first runs). When the post-create deadline expires, onboarding deletes an orphaned sandbox and prints the retry hint. |
| `NEMOCLAW_SANDBOX_READY_TIMEOUT` | `180` | Wall-clock timeout for post-create readiness, in seconds. Raise when the sandbox image build, gateway upload, or in-sandbox boot exceeds the default (typical on 70B+ models, first-time gateway uploads over slow links, or DGX Station / remote-VM first runs). Ordinary onboarding deletes the partially created sandbox when the deadline expires and prints the retry hint. Portable OpenClaw onboarding instead preserves the sandbox when NemoClaw cannot verify its exact runtime identity. |
| `NEMOCLAW_SANDBOX_READY_ERROR_DEBOUNCE` | `30` | Consecutive `Error`-phase polls the post-create readiness wait tolerates before treating `Error` as terminal. Polling starts at 250ms and backs off to a 2-second cap, while `NEMOCLAW_SANDBOX_READY_TIMEOUT` remains the overall deadline. The gateway can briefly report a just-created sandbox in `Error` while it re-registers the sandbox (seen on DGX Spark); the debounce lets that transient recover to `Ready`. `Failed` and `CrashLoopBackOff` always fail immediately. Set to `1` to restore fast-fail on the first `Error` poll. |
| `NEMOCLAW_GATEWAY_RECOVERY_WAIT_SECONDS` | `30`, `90`, or `120`, depending on the recovery phase | Wall-clock timeout for OpenShell command re-registration after policy application, plus gateway health and re-registration during managed OpenClaw or Hermes recovery. A valid finite, nonnegative value overrides the internal budget for the current recovery phase. |

Expand Down Expand Up @@ -4996,7 +5004,15 @@ $$nemoclaw <sandbox-name> recover

If the Ollama pull or post-create readiness timeout fires, onboarding emits the elapsed budget plus a hint to raise the relevant variable.
The Ollama pull preserves its partial download for the next attempt.
The post-create readiness wait deletes the orphaned sandbox first so the next `$$nemoclaw onboard` starts clean.
The ordinary post-create readiness wait deletes the orphaned sandbox first so the next `$$nemoclaw onboard` starts without that partially created sandbox.

<AgentOnly variant="openclaw">

For portable OpenClaw onboarding, NemoClaw instead leaves the sandbox in place when it cannot verify the exact runtime identity.
Inspect it with `openshell sandbox list` and `$$nemoclaw <name> status`, then follow the recovery guidance from `status`.

</AgentOnly>

A post-policy re-registration failure leaves the sandbox in place and reports that OpenShell did not re-register it.

### Lifecycle Behavior Flags
Expand Down
17 changes: 16 additions & 1 deletion docs/reference/troubleshooting.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2125,7 +2125,15 @@ When the ordinary create deadline expires, NemoClaw tries to delete the partiall
After successful cleanup, the output ends with `Retry: $$nemoclaw onboard`.
If cleanup fails, NemoClaw instead reports that the failed sandbox could not be removed and prints `Manual cleanup: openshell sandbox delete "<name>"`.

The failure path differs when NemoClaw recreates an OpenShell-managed Docker runtime immediately before this wait.
<AgentOnly variant="openclaw">

Portable OpenClaw onboarding preserves the sandbox when NemoClaw cannot verify its exact runtime identity.
It does not start dashboard forwarding on this failure path.
Inspect the preserved sandbox with the status commands below, then follow the recovery guidance from `$$nemoclaw <name> status`.

</AgentOnly>

The failure path also differs when NemoClaw recreates an OpenShell-managed Docker runtime immediately before this wait.
NemoClaw pins the exact OpenShell sandbox ID before recreation.
Within the same deadline, NemoClaw requires two consecutive `Ready` observations that each confirm the exact ID and successful command execution.
It retries only OpenShell's exact `sandbox is not ready` response.
Expand Down Expand Up @@ -3028,6 +3036,13 @@ If GPU passthrough is not required on this host, rerun onboarding with `--no-gpu
The route depends on the host environment and the operator control.
Identify the matching path before applying the recovery guidance.

<AgentOnly variant="openclaw">

Do not apply this compatibility guidance to portable OpenClaw onboarding.
That path requires native OpenShell GPU injection and does not use `NEMOCLAW_DOCKER_GPU_PATCH`, so do not set `fallback`, `1`, or another legacy nonzero value for a portable OpenClaw run.

</AgentOnly>

| Symptom | Route or stage | Recovery |
| --- | --- | --- |
| Native `--gpu` is rejected, host runtime evidence identifies GPU injection failure, or an explicit driver proof fails and host configuration confirms no GPU attachment | Ordinary Linux native attempt | The default native-only route stops. Retry with `NEMOCLAW_DOCKER_GPU_PATCH=fallback` only if you explicitly accept one bounded compatibility retry, or use `=1` to select compatibility before creation. |
Expand Down
19 changes: 8 additions & 11 deletions src/lib/onboard.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1618,6 +1618,7 @@ async function createSandboxWithBaseImageResolution(
const plannedMessagingState =
envMessagingState?.plan.sandboxName === sandboxName ? envMessagingState : undefined;
const managedWorkloadRuntime = managedWorkloadOnboard.createManagedWorkloadOnboardRuntime({ computePlan, managedWorkloadRebuild, tempManagedRuntime, tempManagedRuntimeCatalog, agentName: requestedAgentName, legacyDockerfilePath, customDockerfilePath: fromDockerfile ?? (preparedBuildContext ? preparedBuildContext.stagedDockerfile : null), rootDir: ROOT, model, provider, preferredInferenceApi, endpointUrl: createIntent?.endpointUrl ?? null, startupProfile: { chatUiUrl, effectiveDashboardPort: effectivePort, manageDashboard, dashboardBindAddress: process.env.NEMOCLAW_DASHBOARD_BIND, wslExposure: requestedAgentName === "openclaw" && isWsl(), hermesDashboardState, webSearch: webSearchConfig, toolDisclosure: effectiveToolDisclosure, hermesToolGateways, messagingPlan: plannedMessagingState?.plan ?? null, dcodeAutoApprovalMode: dcodeAutoApprovalPlan.mode, observabilityEnabled: createIntent?.observabilityEnabled === true, environment: process.env }, note, fallbackBuildEstimate: () => process.env.NEMOCLAW_IGNORE_RUNTIME_RESOURCES === "1" ? null : formatSandboxBuildEstimateNote(assessHost()) }, { resolveAgentInferenceApi: inferenceConfig.resolveAgentInferenceApi, getSandboxInferenceConfig });
const ensurePreparedSandboxWorkload = () => managedWorkloadOnboard.prepareSandboxWorkloadForPortableLifecycle(managedWorkloadRuntime, sandboxGpuCreateFlow.resolvePortableLifecycleMode(agent));
// #4614: capture default AFTER prune so a stale registry row isn't read as a live sandbox.
const sandboxWasLiveDefault = liveExists && wasSandboxDefault(registry.getDefault(), sandboxName);

Expand Down Expand Up @@ -1856,6 +1857,9 @@ async function createSandboxWithBaseImageResolution(
for (const hint of recreateJournal.managedMcpRecreateRefusalHints({ sandboxName, cliName: cliName(), toolDisclosure: effectiveToolDisclosure, rebuildFlag: dcodeAutoApprovalPlan.rebuildFlag, observabilityFlag: observabilityCommandFlag.explicitObservabilityFlag(createIntent?.observabilityEnabled === true, createIntent?.observabilityRequestedExplicitly === true) })) console.error(hint);
process.exit(1);
}
// Resolve and validate immutable workload authority before opening a recreate journal or
// mutating a live sandbox.
await ensurePreparedSandboxWorkload();
await hermesApiPortReservationScope.selectAndReserve(hermesApiPortReservationInput);
if (!createIntent?.recreateTransaction) recreateRuntime = openRecreateJournal();
if (recreateRuntime.acceptedTarget) {
Expand All @@ -1867,10 +1871,6 @@ async function createSandboxWithBaseImageResolution(
baseImageResolutionFlow.captureBaseResolution(baseImageResolutionContext, previousEntry?.imageTag);
policyPresetCarry.applyRecreatePolicyCarryForward(sandboxName, isNonInteractive(), note);

// Resolve and validate immutable workload authority before deleting a live sandbox.
const replacementWorkload = await managedWorkloadRuntime.ensurePreparedWorkload();
managedWorkloadRuntime.ensurePreparedProfile(replacementWorkload);

const noRestorePending = pendingStateRestore === null && pendingStateRestoreBackupPath === null;
if (noRestorePending && !notReadyRecreateInProgress && !shouldSkipPreRecreateBackup(process.env)) {
note(" Backing up workspace state before recreating sandbox...");
Expand All @@ -1895,8 +1895,7 @@ async function createSandboxWithBaseImageResolution(
}
if (!liveExists)
await hermesApiPortReservationScope.selectAndReserve(hermesApiPortReservationInput);
const preparedSandboxWorkload = await managedWorkloadRuntime.ensurePreparedWorkload();
managedWorkloadRuntime.ensurePreparedProfile(preparedSandboxWorkload);
const preparedSandboxWorkload = await ensurePreparedSandboxWorkload();
applyExtraProviderReconciliation({
extraProviders: resolvedCreateIntent.extraProviders,
staleExtraProviders: resolvedCreateIntent.staleExtraProviders ?? [],
Expand Down Expand Up @@ -1950,7 +1949,7 @@ async function createSandboxWithBaseImageResolution(
restoreBackupPath,
terminalAgent: agentDefs.isTerminalAgent(agent),
managedBootstrap,
...sandboxGpuCreateFlow.resolveDockerStartupCommandPatch(agent, dockerDriverGateway),
...sandboxGpuCreateFlow.resolveAgentCreateInput(agent, dockerDriverGateway),
},
{
runOpenshell,
Expand Down Expand Up @@ -2854,10 +2853,8 @@ const sandboxCreateIntentResolver = sandboxCreateIntentResolution.createSandboxC
filterEnabledChannelsByAgent,
defaultPolicyPath: path.join(ROOT, "nemoclaw-blueprint", "policies", "openclaw-sandbox.yaml"),
getAgentPolicyPath: (agent) => (agent ? agentOnboard.getAgentPolicyPath(agent) : null),
resolveGpuPlan: (config) =>
dockerGpuSandboxCreate.resolveDockerGpuSandboxCreatePlan(config, {
dockerDriverGateway: isLinuxDockerDriverGatewayEnabled(),
}),
resolveGpuPlan: (config, agent) =>
dockerGpuSandboxCreate.resolveAgentPlan(config, agent, isLinuxDockerDriverGatewayEnabled()),
appendResourceCreateArgs: (args, resourceProfile) =>
appendResourceFlagsForProfile(args, resourceProfile, getOpenshellBinary(), {
isNonInteractive,
Expand Down
31 changes: 28 additions & 3 deletions src/lib/onboard/docker-gpu-sandbox-create-plan.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
// SPDX-License-Identifier: Apache-2.0

import { type DockerGpuRoutePlan, resolveDockerGpuRoutePlan } from "./docker-gpu-route";
import { isPortableExperimentalProfile } from "./experimental/portable-profile";
import { detectWslDockerDesktopStatus } from "./wsl-docker-desktop-gpu";

type DockerGpuSandboxConfig = {
Expand Down Expand Up @@ -31,6 +32,22 @@ export function resetIsDockerDesktopWslRuntimeCache(): void {
cachedDockerDesktopWslRuntime = null;
}

export function resolveAgentPlan(
config: DockerGpuSandboxConfig,
agent: { name?: string | null } | null,
dockerDriverGateway: boolean,
env: NodeJS.ProcessEnv = process.env,
platform: NodeJS.Platform = process.platform,
): DockerGpuSandboxCreatePlan {
return resolveDockerGpuSandboxCreatePlan(config, {
dockerDriverGateway,
portableLifecycle:
isPortableExperimentalProfile(env) && (agent?.name ?? "openclaw") === "openclaw",
env,
platform,
});
}

/**
* SOURCE_OF_TRUTH_REVIEW (GPU create route selection; #6110)
* invalidState: one attempt combines native `--gpu` with compatibility recreation.
Expand All @@ -47,16 +64,24 @@ export function resolveDockerGpuSandboxCreatePlan(
dockerDesktopWsl?: boolean;
detectDockerDesktopWsl?: () => boolean;
env?: NodeJS.ProcessEnv;
portableLifecycle?: boolean;
platform?: NodeJS.Platform;
log?: (message: string) => void;
},
): DockerGpuSandboxCreatePlan {
const env = options.env ?? process.env;
const portableLifecycle = options.portableLifecycle === true;
const dockerDesktopWsl =
options.dockerDesktopWsl ?? (options.detectDockerDesktopWsl ?? isDockerDesktopWslRuntime)();
portableLifecycle
? false
: (options.dockerDesktopWsl ?? (options.detectDockerDesktopWsl ?? isDockerDesktopWslRuntime)());
const gpuRoutePlan = resolveDockerGpuRoutePlan(config, {
dockerDriverGateway: options.dockerDriverGateway,
// The hidden portable profile reaches OpenShell through its Docker-compatible
// endpoint, but rootless Podman owns sandbox lifecycle. Keep Docker-only
// container substitution out of that lifecycle path.
dockerDriverGateway: options.dockerDriverGateway && !portableLifecycle,
dockerDesktopWsl,
env: options.env,
env,
platform: options.platform,
log: options.log,
});
Expand Down
Loading
Loading