diff --git a/docs/deployment/deploy-to-remote-gpu.md b/docs/deployment/deploy-to-remote-gpu.md index a0ca0649df5..0d6c01c4a0b 100644 --- a/docs/deployment/deploy-to-remote-gpu.md +++ b/docs/deployment/deploy-to-remote-gpu.md @@ -127,6 +127,25 @@ When `CHAT_UI_URL` points at a non-loopback origin, NemoClaw disables OpenClaw d Any device that can reach the configured dashboard origin can connect without pairing, so avoid exposing that origin on internet-reachable or shared-network deployments. ::: +## First-Run Readiness Budget + +On a remote GPU host, the first `nemoclaw onboard` typically does the slowest work of the lifecycle: the sandbox image is built locally and uploaded into the OpenShell gateway, which can stream hundreds of MiB over the VM's link before the readiness wait even starts. +The post-create readiness wait defaults to 180 seconds (`NEMOCLAW_SANDBOX_READY_TIMEOUT`), which is sized for warm-cache, workstation-class onboarding and can be exceeded on: + +- DGX Station first runs with large quantised models (70B+ parameter footprints, NVFP4 weights). +- Cloud VMs where the local image-build cache is cold and the upload runs over the public network. +- Hosts onboarding the Brave Web Search preset on the first run (the egress policy stack adds boot work). + +Raise the budget before re-running onboard: + +```console +$ export NEMOCLAW_SANDBOX_READY_TIMEOUT=600 +$ nemoclaw onboard +``` + +If onboard ends with `Sandbox '' was created but did not become ready within 180s`, onboard deletes the partially-created sandbox first, so the next attempt with the raised budget starts from a clean state. +For the inference-probe budget that runs earlier in onboarding, see [`NEMOCLAW_LOCAL_INFERENCE_TIMEOUT`](../inference/use-local-inference.md#timeout-configuration). + ## Proxy Configuration NemoClaw routes sandbox traffic through a gateway proxy that defaults to `10.200.0.1:3128`. diff --git a/docs/inference/use-local-inference.md b/docs/inference/use-local-inference.md index e77a51b5609..bb5f0bc5dd0 100644 --- a/docs/inference/use-local-inference.md +++ b/docs/inference/use-local-inference.md @@ -357,6 +357,18 @@ The value is in seconds. This setting is baked into the sandbox at build time. Changing it after onboarding requires re-running `nemoclaw onboard`. +`NEMOCLAW_LOCAL_INFERENCE_TIMEOUT` only governs the inference-server validation probe. +The post-create readiness wait (image build, gateway upload, in-sandbox boot) has its own budget, `NEMOCLAW_SANDBOX_READY_TIMEOUT`, also defaulting to 180 seconds. +On hosts where the sandbox image takes minutes to build or upload — large quantised models, DGX Station first runs, or remote VMs over a slow link — raise both together: + +```console +$ export NEMOCLAW_LOCAL_INFERENCE_TIMEOUT=300 +$ export NEMOCLAW_SANDBOX_READY_TIMEOUT=600 +$ nemoclaw onboard +``` + +If onboard ends with `Sandbox '' was created but did not become ready within 180s`, refer to [Troubleshooting](../reference/troubleshooting.md#sandbox-onboard-times-out-with-did-not-become-ready-within-ns). + ## Verify the Configuration After onboarding completes, confirm the active provider and model. diff --git a/docs/reference/commands.md b/docs/reference/commands.md index 0a774fec1eb..92dd6ea5380 100644 --- a/docs/reference/commands.md +++ b/docs/reference/commands.md @@ -1183,13 +1183,18 @@ Set them before running `nemoclaw onboard` if a slow connection or large model p | Variable | Default | Purpose | |----------|---------|---------| | `NEMOCLAW_OLLAMA_PULL_TIMEOUT` | `1800` (30 minutes) | Wall-clock timeout for `ollama pull` during onboard, in seconds. Accepts integer or float values. Already-downloaded layers are kept; re-running the pull resumes them. | +| `NEMOCLAW_LOCAL_INFERENCE_TIMEOUT` | `180` | Wall-clock timeout for the inference-server validation probe during onboard, in seconds. Raise on slow networks or for very large prompts. | +| `NEMOCLAW_SANDBOX_READY_TIMEOUT` | `180` | Wall-clock timeout for the post-create readiness wait, in seconds. Raise when the sandbox image build, gateway upload, or in-sandbox boot exceeds the default (typical on 70B+ models, first-time gateway uploads over slow links, or DGX Station / remote-VM first runs). When the deadline expires onboarding deletes the orphaned sandbox and prints the retry hint. | ```console $ export NEMOCLAW_OLLAMA_PULL_TIMEOUT=3600 +$ export NEMOCLAW_SANDBOX_READY_TIMEOUT=600 $ nemoclaw onboard ``` -If the pull exceeds the limit, onboarding emits the timeout in minutes plus a hint to raise this variable, and the partial download is preserved for the next attempt. +If a timeout fires, onboarding emits the elapsed budget plus a hint to raise the relevant variable. +The Ollama pull preserves its partial download for the next attempt. +The readiness wait deletes the orphaned sandbox first so the next `nemoclaw onboard` starts clean. ### Lifecycle Behavior Flags diff --git a/docs/reference/troubleshooting.md b/docs/reference/troubleshooting.md index 7481390179f..ff5216686eb 100644 --- a/docs/reference/troubleshooting.md +++ b/docs/reference/troubleshooting.md @@ -647,6 +647,41 @@ $ nemoclaw onboard For local Ollama and vLLM, onboarding retries the container reachability check and can fall back to the host-side health check when the local backend is healthy. If all attempts fail, the error includes container reachability diagnostics such as HTTP status and host gateway resolution. +`NEMOCLAW_LOCAL_INFERENCE_TIMEOUT` only covers the inference-server validation probe. +The post-create readiness wait has its own budget (`NEMOCLAW_SANDBOX_READY_TIMEOUT`); see [Sandbox onboard times out with "did not become ready within Ns"](#sandbox-onboard-times-out-with-did-not-become-ready-within-ns) for the readiness path. + +### Sandbox onboard times out with "did not become ready within Ns" + +Onboarding ends with: + +```text + Sandbox 'my-assistant' was created but did not become ready within 180s. + The orphaned sandbox has been removed — you can safely retry. +``` + +This is a separate budget from `NEMOCLAW_LOCAL_INFERENCE_TIMEOUT` — it covers the readiness wait that follows sandbox creation (in-sandbox boot, OpenClaw start, policy load), not the inference probe. +The 180-second default fits typical workstations but can be exceeded when: + +- The host is building or uploading the sandbox image for the first time (cold caches, slow link). +- The selected model is large (70B+ parameters or 4-bit/8-bit quantisations that take time to memory-map). +- Onboarding runs on a remote VM where image upload to the gateway streams over the network (for example DGX Station first-run installer). + +Raise the budget before re-running onboard: + +```console +$ export NEMOCLAW_SANDBOX_READY_TIMEOUT=600 +$ nemoclaw onboard +``` + +The variable accepts seconds and applies to the readiness wait only. +When the deadline expires, NemoClaw deletes the partially-created sandbox before printing the retry hint, so the next `nemoclaw onboard` starts from a clean state. +If readiness still fails after the extended budget, inspect the gateway and sandbox status: + +```console +$ openshell sandbox list +$ nemoclaw status +``` + ### Agent fails at runtime after onboarding succeeds with a compatible endpoint Some OpenAI-compatible servers (such as SGLang) expose `/v1/responses` but their