Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/reference/commands-nemohermes.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2103,6 +2103,7 @@ Set them before running `nemohermes onboard` if a slow connection or large model
| `NEMOCLAW_OLLAMA_PULL_TIMEOUT` | `1800` (30 minutes) | Wall-clock timeout for `ollama pull` during onboard, in seconds. Accepts integer or float values. Already-downloaded layers are kept; re-running the pull resumes them. |
| `NEMOCLAW_LOCAL_INFERENCE_TIMEOUT` | `180` | Wall-clock timeout for the inference-server validation probe during onboard, in seconds. Raise on slow networks or for very large prompts. |
| `NEMOCLAW_SANDBOX_READY_TIMEOUT` | `180` | Wall-clock timeout for the post-create readiness wait, in seconds. Raise when the sandbox image build, gateway upload, or in-sandbox boot exceeds the default (typical on 70B+ models, first-time gateway uploads over slow links, or DGX Station / remote-VM first runs). When the deadline expires onboarding deletes the orphaned sandbox and prints the retry hint. |
| `NEMOCLAW_SANDBOX_READY_ERROR_DEBOUNCE` | `30` | Consecutive `Error`-phase polls (2s apart, so ~60s by default) the post-create readiness wait tolerates before treating `Error` as terminal. The gateway can briefly report a just-created sandbox in `Error` while it re-registers the sandbox (seen on DGX Spark); the debounce lets that transient recover to `Ready`. `Failed` and `CrashLoopBackOff` always fail immediately. Set to `1` to restore fast-fail on the first `Error` poll. |

```bash
export NEMOCLAW_OLLAMA_PULL_TIMEOUT=3600
Expand Down
1 change: 1 addition & 0 deletions docs/reference/commands.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2588,6 +2588,7 @@ Set them before running `$$nemoclaw onboard` if a slow connection or large model
| `NEMOCLAW_OLLAMA_PULL_TIMEOUT` | `1800` (30 minutes) | Wall-clock timeout for `ollama pull` during onboard, in seconds. Accepts integer or float values. Already-downloaded layers are kept; re-running the pull resumes them. |
| `NEMOCLAW_LOCAL_INFERENCE_TIMEOUT` | `180` | Wall-clock timeout for the inference-server validation probe during onboard, in seconds. Raise on slow networks or for very large prompts. |
| `NEMOCLAW_SANDBOX_READY_TIMEOUT` | `180` | Wall-clock timeout for the post-create readiness wait, in seconds. Raise when the sandbox image build, gateway upload, or in-sandbox boot exceeds the default (typical on 70B+ models, first-time gateway uploads over slow links, or DGX Station / remote-VM first runs). When the deadline expires onboarding deletes the orphaned sandbox and prints the retry hint. |
| `NEMOCLAW_SANDBOX_READY_ERROR_DEBOUNCE` | `30` | Consecutive `Error`-phase polls (2s apart, so ~60s by default) the post-create readiness wait tolerates before treating `Error` as terminal. The gateway can briefly report a just-created sandbox in `Error` while it re-registers the sandbox (seen on DGX Spark); the debounce lets that transient recover to `Ready`. `Failed` and `CrashLoopBackOff` always fail immediately. Set to `1` to restore fast-fail on the first `Error` poll. |

```bash
export NEMOCLAW_OLLAMA_PULL_TIMEOUT=3600
Expand Down
26 changes: 26 additions & 0 deletions docs/reference/troubleshooting.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -977,6 +977,32 @@ openshell sandbox list
$$nemoclaw <name> status
```

### Sandbox onboard fails with "entered Error phase before it became ready"

Onboarding ends with:

```text
Sandbox 'my-assistant' entered Error phase before it became ready (waited up to 180s).
```

On a fresh onboard the OpenShell gateway can (re)start its supervisor session and re-register the just-created sandbox. During that window `openshell sandbox list` briefly reports the sandbox in the transient `Error` phase before it flips to `Ready` — seen on DGX Spark, where the dashboard port fallback and supervisor restart race the sandbox bootstrap.

NemoClaw tolerates a bounded run of consecutive `Error` polls (default 30 polls / ~60s) so this transient recovers on its own; only `Error` that persists past the debounce window is treated as terminal. `Failed` and `CrashLoopBackOff` are always terminal and fail immediately.

If your host needs a longer window (slower re-registration), raise the debounce; to fail fast on the first `Error` poll, set it to `1`:

```bash
export NEMOCLAW_SANDBOX_READY_ERROR_DEBOUNCE=60 # tolerate ~120s of transient Error
$$nemoclaw onboard
```

If the failure persists after the debounce, the sandbox is genuinely stuck — inspect the retained diagnostics and gateway state:

```bash
openshell sandbox list
$$nemoclaw <name> status
```

### Agent fails at runtime after onboarding succeeds with a compatible endpoint

Some OpenAI-compatible servers (such as SGLang) expose `/v1/responses` but their
Expand Down
31 changes: 3 additions & 28 deletions src/lib/onboard/docker-gpu-patch.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import os from "node:os";
import path from "node:path";

import { describe, expect, it, vi } from "vitest";
import { getSandboxFailurePhase, isSandboxReady } from "../state/gateway";
import { getSandboxFailurePhase } from "../state/gateway";
import {
buildDockerGpuCloneRunArgs,
buildDockerGpuCloneRunOptions,
Expand All @@ -27,7 +27,6 @@ import {
shouldApplyDockerGpuPatch,
waitForOpenShellSupervisorReconnect,
} from "./docker-gpu-patch";
import { waitForCreatedSandboxReadyWithTrace } from "./sandbox-readiness-tracing";

function inspectFixture(): DockerContainerInspect {
return {
Expand Down Expand Up @@ -931,32 +930,8 @@ describe("docker-gpu-patch Error-phase diagnostics (#4316)", () => {
expect(getSandboxFailurePhase("", "my-sandbox")).toBeNull();
});

it("short-circuits the readiness wait when the sandbox enters Error phase", () => {
const outputs = ["my-sandbox Provisioning 1s ago", "my-sandbox Error 3s ago"];
let i = 0;
const runCaptureOpenshell = vi.fn(() => outputs[Math.min(i++, outputs.length - 1)]);
const sleep = vi.fn();

const ready = waitForCreatedSandboxReadyWithTrace({
sandboxName: "my-sandbox",
// 600 / 2 = 300 readyAttempts. Without short-circuit we'd loop 300
// times. With short-circuit we should bail out after the 2nd poll.
timeoutSecs: 600,
runCaptureOpenshell,
isSandboxReady,
getSandboxFailurePhase,
sleep,
});

expect(ready).toEqual({
ready: false,
reason: "terminal_failure_phase",
failurePhase: "Error",
});
expect(runCaptureOpenshell).toHaveBeenCalledTimes(2);
// Should not sleep after detecting the terminal phase.
expect(sleep).toHaveBeenCalledTimes(1);
});
// Create/readiness-wait Error-phase behavior (including the #6043 transient
// debounce and its env contract) lives in sandbox-readiness-tracing.test.ts.

it("short-circuits the supervisor-reconnect wait when the sandbox enters Error phase", () => {
// Without the short-circuit, a patched container that crashes on startup
Expand Down
4 changes: 3 additions & 1 deletion src/lib/onboard/docker-gpu-supervisor-reconnect.ts
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,9 @@ export function waitForOpenShellSupervisorReconnect(
const errorPhaseDebouncePolls =
deps.errorPhaseDebouncePolls == null || !Number.isFinite(deps.errorPhaseDebouncePolls)
? getDockerGpuSupervisorReconnectErrorDebouncePolls()
: Math.max(1, Math.trunc(deps.errorPhaseDebouncePolls));
: // Round (not truncate) to match the env-var path's envInt rounding and
// the sibling create/readiness debounce in sandbox-readiness-tracing.ts.
Math.max(1, Math.round(deps.errorPhaseDebouncePolls));
let consecutiveErrorPolls = 0;
while (Date.now() <= deadline) {
const result = deps.runOpenshell(["sandbox", "exec", "-n", sandboxName, "--", "true"], {
Expand Down
Loading
Loading