Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions docs/deployment/deploy-to-remote-gpu.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,25 @@ When `CHAT_UI_URL` points at a non-loopback origin, NemoClaw disables OpenClaw d
Any device that can reach the configured dashboard origin can connect without pairing, so avoid exposing that origin on internet-reachable or shared-network deployments.
:::

## First-Run Readiness Budget

On a remote GPU host, the first `nemoclaw onboard` typically does the slowest work of the lifecycle: the sandbox image is built locally and uploaded into the OpenShell gateway, which can stream hundreds of MiB over the VM's link before the readiness wait even starts.
The post-create readiness wait defaults to 180 seconds (`NEMOCLAW_SANDBOX_READY_TIMEOUT`), which is sized for warm-cache, workstation-class onboarding and can be exceeded on:

- DGX Station first runs with large quantised models (70B+ parameter footprints, NVFP4 weights).
- Cloud VMs where the local image-build cache is cold and the upload runs over the public network.
- Hosts onboarding the Brave Web Search preset on the first run (the egress policy stack adds boot work).

Raise the budget before re-running onboard:

```console
$ export NEMOCLAW_SANDBOX_READY_TIMEOUT=600
$ nemoclaw onboard
```

If onboard ends with `Sandbox '<name>' was created but did not become ready within 180s`, the partially-created sandbox is deleted first, so the next attempt with the raised budget starts from a clean state.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
For the inference-probe budget that runs earlier in onboarding, see [`NEMOCLAW_LOCAL_INFERENCE_TIMEOUT`](../inference/use-local-inference.md#timeout-configuration).

## Proxy Configuration

NemoClaw routes sandbox traffic through a gateway proxy that defaults to `10.200.0.1:3128`.
Expand Down
12 changes: 12 additions & 0 deletions docs/inference/use-local-inference.md
Original file line number Diff line number Diff line change
Expand Up @@ -345,6 +345,18 @@ The value is in seconds.
This setting is baked into the sandbox at build time.
Changing it after onboarding requires re-running `nemoclaw onboard`.

`NEMOCLAW_LOCAL_INFERENCE_TIMEOUT` only governs the inference-server validation probe.
The post-create readiness wait (image build, gateway upload, in-sandbox boot) has its own budget, `NEMOCLAW_SANDBOX_READY_TIMEOUT`, also defaulting to 180 seconds.
On hosts where the sandbox image takes minutes to build or upload — large quantised models, DGX Station first runs, or remote VMs over a slow link — raise both together:

```console
$ export NEMOCLAW_LOCAL_INFERENCE_TIMEOUT=300
$ export NEMOCLAW_SANDBOX_READY_TIMEOUT=600
$ nemoclaw onboard
```

If onboard ends with `Sandbox '<name>' was created but did not become ready within 180s`, refer to [Troubleshooting](../reference/troubleshooting.md#sandbox-onboard-times-out-with-did-not-become-ready-within-ns).

## Verify the Configuration

After onboarding completes, confirm the active provider and model.
Expand Down
5 changes: 4 additions & 1 deletion docs/reference/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -1166,13 +1166,16 @@ Set them before running `nemoclaw onboard` if a slow connection or large model p
| Variable | Default | Purpose |
|----------|---------|---------|
| `NEMOCLAW_OLLAMA_PULL_TIMEOUT` | `1800` (30 minutes) | Wall-clock timeout for `ollama pull` during onboard, in seconds. Accepts integer or float values. Already-downloaded layers are kept; re-running the pull resumes them. |
| `NEMOCLAW_LOCAL_INFERENCE_TIMEOUT` | `180` | Wall-clock timeout for the inference-server validation probe during onboard, in seconds. Raise on slow networks or for very large prompts. |
| `NEMOCLAW_SANDBOX_READY_TIMEOUT` | `180` | Wall-clock timeout for the post-create readiness wait, in seconds. Raise when the sandbox image build, gateway upload, or in-sandbox boot exceeds the default (typical on 70B+ models, first-time gateway uploads over slow links, or DGX Station / remote-VM first runs). When the deadline expires onboarding deletes the orphaned sandbox and prints the retry hint. |

```console
$ export NEMOCLAW_OLLAMA_PULL_TIMEOUT=3600
$ export NEMOCLAW_SANDBOX_READY_TIMEOUT=600
$ nemoclaw onboard
```

If the pull exceeds the limit, onboarding emits the timeout in minutes plus a hint to raise this variable, and the partial download is preserved for the next attempt.
If a timeout fires, onboarding emits the elapsed budget plus a hint to raise the relevant variable. The Ollama pull preserves its partial download for the next attempt; the readiness wait deletes the orphaned sandbox first so the next `nemoclaw onboard` starts clean.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

### Lifecycle Behavior Flags

Expand Down
35 changes: 35 additions & 0 deletions docs/reference/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -621,6 +621,41 @@ $ nemoclaw onboard
For local Ollama and vLLM, onboarding retries the container reachability check and can fall back to the host-side health check when the local backend is healthy.
If all attempts fail, the error includes container reachability diagnostics such as HTTP status and host gateway resolution.

`NEMOCLAW_LOCAL_INFERENCE_TIMEOUT` only covers the inference-server validation probe.
The post-create readiness wait has its own budget (`NEMOCLAW_SANDBOX_READY_TIMEOUT`); see [Sandbox onboard times out with "did not become ready within Ns"](#sandbox-onboard-times-out-with-did-not-become-ready-within-ns) for the readiness path.

### Sandbox onboard times out with "did not become ready within Ns"

Onboarding ends with:

```text
Sandbox 'my-assistant' was created but did not become ready within 180s.
The orphaned sandbox has been removed — you can safely retry.
```

This is a separate budget from `NEMOCLAW_LOCAL_INFERENCE_TIMEOUT` — it covers the readiness wait that follows sandbox creation (in-sandbox boot, OpenClaw start, policy load), not the inference probe.
The 180-second default fits typical workstations but can be exceeded when:

- The host is building or uploading the sandbox image for the first time (cold caches, slow link).
- The selected model is large (70B+ parameters or 4-bit/8-bit quantisations that take time to memory-map).
- Onboarding runs on a remote VM where image upload to the gateway streams over the network (for example DGX Station first-run installer).

Raise the budget before re-running onboard:

```console
$ export NEMOCLAW_SANDBOX_READY_TIMEOUT=600
$ nemoclaw onboard
```

The variable accepts seconds and applies to the readiness wait only.
When the deadline expires, NemoClaw deletes the partially-created sandbox before printing the retry hint, so the next `nemoclaw onboard` starts from a clean state.
If readiness still fails after the extended budget, inspect the gateway and sandbox status:

```console
$ openshell sandbox list
$ nemoclaw <name> status
```

### Agent fails at runtime after onboarding succeeds with a compatible endpoint

Some OpenAI-compatible servers (such as SGLang) expose `/v1/responses` but their
Expand Down
Loading