Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions docs/get-started/quickstart.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -317,9 +317,11 @@ Use these details when your first-run path needs more control.
The selector can include destinations such as GitHub, Jira, Slack, Telegram, or local inference.
Press `r` to switch a selected preset between read-only and read-write when it supports both modes.

Before it prints the ready summary, NemoClaw checks that the sandbox gateway and dashboard port forward are reachable.
Use the final onboarding summary to verify that the sandbox gateway, dashboard port forward, and `inference.local` route are reachable.
When web search is enabled, it also checks the selected provider configuration and sends a real search request through sandbox egress.
Web search, inference-route, and messaging-bridge checks report warnings instead of aborting onboarding when they need more time or configuration.
Treat an unreachable route or HTTP 5xx response as a failed readiness check: onboarding marks the sandbox not ready and exits non-zero.
Restore the configured endpoint or proxy, run `nemoclaw onboard --resume` to complete the retained onboarding session, then rerun `nemoclaw <sandbox-name> status` to verify the route.
Web search and messaging-bridge checks remain warnings when they need more time or configuration.

```text
──────────────────────────────────────────────────
Expand Down
22 changes: 17 additions & 5 deletions docs/inference/verify-inference-route.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -34,15 +34,27 @@ $$nemoclaw <name> status

The `Inference` row checks the sandbox's `inference.local` path and reports the provider, model, and endpoint with the rest of the sandbox state.
This path includes the OpenShell proxy and its authentication rewrite.
When onboarding prints a dashboard summary, use it to verify that NemoClaw ran the same route-reachability probe from inside the sandbox.
Treat an unreachable route or HTTP 5xx response as a failed readiness check: onboarding marks the sandbox not ready and exits non-zero.
Restore the configured endpoint or proxy, run `$$nemoclaw onboard --resume` to complete the retained onboarding session, then rerun the status command.

## Understand Post-Ready Checks
## Understand Local Provider Post-Ready Checks

For local Ollama and vLLM, onboarding performs an additional check after the sandbox becomes ready.
For local Ollama and vLLM on Docker GPU sandboxes using the compatibility route, onboarding performs an additional check after the sandbox becomes ready.
It requests `https://inference.local/v1/models` from inside the sandbox and accepts only a 2xx response.
When this check fails, onboarding reports the endpoint and recovery steps before the first agent prompt.
When this check fails, onboarding reports the endpoint and local-provider recovery steps before the first agent prompt.

NVIDIA NIM and other compatible endpoints receive their provider validation during onboarding but do not receive this post-ready sandbox-route check.
For those routes, use the status command and a short agent request after onboarding.
NVIDIA NIM and other compatible endpoints receive their provider validation during onboarding but do not receive this local-provider post-ready check.
For those routes, continue to the final route check, then use the status command and a short agent request after onboarding.

## Understand Final Route Checks

When onboarding prints a dashboard summary, it first requests `https://inference.local/v1/models` from inside the sandbox after policy and process recovery.
A transport failure or HTTP 5xx response leaves the onboarding session retryable at final verification instead of completing it.
After restoring the route, resume onboarding to run the check again without rebuilding a healthy sandbox.

Provider setup still performs its own model, credential, and endpoint validation before this final route check.
Use the status command and a short agent request after onboarding to verify ongoing availability and model responses.

## Send a Short Agent Request

Expand Down
4 changes: 2 additions & 2 deletions src/lib/onboard.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4666,7 +4666,7 @@ async function runOnboard(opts: OnboardOptions = {}): Promise<void> {
},
});

await runFinalOnboardFlowSlice({
const finalFlowResult = await runFinalOnboardFlowSlice({
context: finalFlowContext,
runtime: onboardRuntimeBoundary.getRuntime(),
phases: [branchSetupPhase, policiesPhase, finalizationPhase],
Expand All @@ -4681,7 +4681,7 @@ async function runOnboard(opts: OnboardOptions = {}): Promise<void> {
},
});
completed = true;
traceCompleted = true;
traceCompleted = finalFlowResult.session.machine.state === "complete";
} finally {
releaseOnboardLock();
onboardRuntimeBoundary.clear();
Expand Down
4 changes: 3 additions & 1 deletion src/lib/onboard/dashboard.ts
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,7 @@ export interface OnboardDashboardHelpers {
provider: string,
nimContainer?: string | null,
agent?: AgentDefinition | null,
ready?: boolean,
): void;
stopAllDashboardForwards(): void;
}
Expand Down Expand Up @@ -439,6 +440,7 @@ export function createOnboardDashboardHelpers(deps: OnboardDashboardDeps): Onboa
provider: string,
nimContainer: string | null = null,
agent: AgentDefinition | null = null,
ready = true,
): void {
const nimStatus = deps.nimStatus ?? nim.nimStatus;
const nimStatusByName = deps.nimStatusByName ?? nim.nimStatusByName;
Expand Down Expand Up @@ -471,7 +473,7 @@ export function createOnboardDashboardHelpers(deps: OnboardDashboardDeps): Onboa

console.log("");
console.log(` ${"─".repeat(50)}`);
console.log(` ${deps.agentProductName()} is ready`);
console.log(` ${deps.agentProductName()} is ${ready ? "ready" : "not ready"}`);
console.log("");
console.log(` Sandbox: ${sandboxName}`);
console.log(` Model: ${model} (${providerLabel})`);
Expand Down
35 changes: 35 additions & 0 deletions src/lib/onboard/finalization-deps.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

import { afterEach, describe, expect, it } from "vitest";

import type { VerifyDeploymentResult } from "../verify-deployment";
import { finalizationHandlerDeps } from "./finalization-deps";

describe("finalizationHandlerDeps.reportDeploymentReadiness", () => {
const originalExitCode = process.exitCode;
afterEach(() => {
process.exitCode = originalExitCode;
});

it("sets a non-zero exit code when the deployment is not ready", () => {
process.exitCode = 0;
finalizationHandlerDeps.reportDeploymentReadiness(false);
expect(process.exitCode).toBe(1);
});

it("leaves the exit code unchanged when the deployment is ready", () => {
process.exitCode = 0;
finalizationHandlerDeps.reportDeploymentReadiness(true);
expect(process.exitCode).toBe(0);
});
});

describe("finalizationHandlerDeps.isDeploymentHealthy", () => {
it("reports the verification healthy flag", () => {
const healthy = { healthy: true } as unknown as VerifyDeploymentResult;
const unhealthy = { healthy: false } as unknown as VerifyDeploymentResult;
expect(finalizationHandlerDeps.isDeploymentHealthy(healthy)).toBe(true);
expect(finalizationHandlerDeps.isDeploymentHealthy(unhealthy)).toBe(false);
});
});
6 changes: 6 additions & 0 deletions src/lib/onboard/finalization-deps.ts
Original file line number Diff line number Diff line change
Expand Up @@ -29,4 +29,10 @@ export const finalizationHandlerDeps = {
require("../actions/sandbox/auto-pair-warmup");
warmup.runSandboxScopeWarmupRun(name);
},
isDeploymentHealthy(result: import("../verify-deployment").VerifyDeploymentResult): boolean {
return result.healthy;
},
reportDeploymentReadiness(healthy: boolean): void {
if (!healthy) process.exitCode = 1;
},
};
3 changes: 2 additions & 1 deletion src/lib/onboard/lifecycle-contracts.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Related guides: [`README.md`](README.md) describes package placement, [`machine/
| **plan** | Intent plus observed state, ready to apply | `MessagingWorkflowPlanner.buildPlan`; `materializeSandboxCreatePlan` |
| **apply** | Effectful phase that binds credentials and live capabilities | `bindMessagingTokenDefs`; create, rebuild, and mutation executors |
| **checkpoint** | Durable, secret-minimized state from which a later process can continue | onboard session and machine snapshot; registry; backup/recovery manifests |
| **result** | Handler outcome: advance, retry, branch, complete, or fail | `OnboardStateResult`, applied by `OnboardRuntime` through `OnboardRuntimeBoundary` |
| **result** | Handler outcome: advance, retry, branch, pause, complete, or fail | `OnboardStateResult`, applied by `OnboardRuntime` through `OnboardRuntimeBoundary` |
| **compensation** | Effect that undoes or limits a partial apply | failed-create deletion, `cancel-rollback.ts`, `rollbackChannelAdd`, recovery-registry restore |
| **reconcile** | Align recorded and live state without replaying the full journey | sandbox drift checks, `reconcileSandboxMessaging`, `mergeOpenClawRestoredConfig` |

Expand All @@ -31,6 +31,7 @@ inference --retry--> provider_selection
inference --advance--> sandbox
sandbox --branch--> openclaw -> policies -> finalizing -> post_verify -> complete
sandbox --branch--> agent_setup -> policies -> finalizing -> post_verify -> complete
post_verify --pause--> post_verify (retryable handoff without a state transition)
each nonterminal state --failure--> failed
```

Expand Down
7 changes: 4 additions & 3 deletions src/lib/onboard/machine/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,9 @@ The target shape is a machine-driven onboarding runner:
2. Build an onboarding context that contains sanitized operator choices, runtime dependencies, and mutable values returned by states.
3. Enter `runOnboardMachine(context)`.
4. Dispatch the current machine state to a handler.
5. Let the handler return an explicit state result such as advance, retry, branch, complete, or failed.
5. Let the handler return an explicit state result such as advance, retry, branch, pause, complete, or failed.
6. Apply the result through `OnboardRuntime`, which validates the transition, updates the persisted session snapshot, and emits redacted machine events.
7. Continue until the machine reaches `complete` or `failed`.
7. Continue until the machine reaches `complete` or `failed`, or a handler pauses at a retryable non-terminal state.

In that final shape, `src/lib/onboard.ts` should be a thin entrypoint. State handlers should own state-specific prompts, resume validation, repair decisions, and side effects.

Expand Down Expand Up @@ -80,7 +80,8 @@ sequence must declare its source state in `metadata.state`, and that source must
machine's current state when the result is applied. The runner also checks the handler's sequence
ownership allowlist; add a new entry in `DEFAULT_SEQUENCE_OWNERSHIP` before introducing another
composite handler that crosses into a later state. Terminal results (`complete` or `failed`) end
the sequence immediately.
the sequence immediately. A `pause` result persists any supplied safe context and returns control
without a state transition so a later process can resume the same non-terminal state.

## Runtime responsibilities

Expand Down
76 changes: 76 additions & 0 deletions src/lib/onboard/machine/final-flow-phases.runtime.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,26 @@ import {
sessionAt,
} from "../../../../test/helpers/onboard-final-flow-phases";
import { createSession } from "../../state/onboard-session";
import type { VerifyDeploymentResult } from "../../verify-deployment";
import { runFinalOnboardFlowSlice } from "./final-flow-phases";

function deploymentResult(healthy: boolean): VerifyDeploymentResult {
return {
healthy,
verification: {
gatewayReachable: true,
gatewayVersion: "test",
inferenceRouteWorking: healthy,
dashboardReachable: true,
messagingBridgesHealthy: true,
messagingRuntimeChannelsMissing: null,
messagingConfigChannelsMissing: null,
accessMethod: "localhost",
},
diagnostics: [],
};
}

describe("final onboard flow runtime boundary", () => {
it("uses the strict final runner for fresh OpenClaw sessions with a real runtime boundary", async () => {
const order: string[] = [];
Expand Down Expand Up @@ -285,4 +303,62 @@ describe("final onboard flow runtime boundary", () => {
machine: { state: "post_verify" },
});
});

it("keeps an unhealthy final verification retryable and completes after a later resume (#6849)", async () => {
const order: string[] = [];
const harness = createRuntimeHarness(sessionAt("openclaw"));
const recorders = harness.boundary.recorders();
const verifyDeployment = vi
.fn()
.mockResolvedValueOnce(deploymentResult(false))
.mockResolvedValueOnce(deploymentResult(true));
const phases = createPhases("openclaw", order, {
loadSession: harness.getSession,
recordStepSkipped: recorders.recordStepSkipped,
recordStateSkipped: recorders.recordStateSkipped,
startRecordedStep: recorders.startRecordedStep,
recordStepComplete: recorders.recordStepComplete,
recordPostVerifyStarted: recorders.recordPostVerifyStarted,
verifyDeployment,
});

const first = await runFinalOnboardFlowSlice({
context: context({ session: harness.getSession() }),
runtime: harness.boundary.getRuntime(),
phases,
resume: false,
recordStateResult: harness.boundary.recordStateResultWithStepCompatibility.bind(
harness.boundary,
),
recordInvalidatedStateResult: harness.boundary.recordInvalidatedStateResult.bind(
harness.boundary,
),
});

expect(first.session).toMatchObject({
status: "in_progress",
resumable: true,
machine: { state: "post_verify" },
});

const resumed = await runFinalOnboardFlowSlice({
context: context({ resume: true, session: harness.getSession() }),
runtime: harness.boundary.getRuntime(),
phases,
resume: true,
recordStateResult: harness.boundary.recordStateResultWithStepCompatibility.bind(
harness.boundary,
),
recordInvalidatedStateResult: harness.boundary.recordInvalidatedStateResult.bind(
harness.boundary,
),
});

expect(verifyDeployment).toHaveBeenCalledTimes(2);
expect(resumed.session).toMatchObject({
status: "complete",
resumable: false,
machine: { state: "complete" },
});
});
});
6 changes: 3 additions & 3 deletions src/lib/onboard/machine/final-flow-phases.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -71,10 +71,10 @@ describe("final onboard flow phases", () => {
phases,
resume: true,
recordStateResult: async (result) => {
if (result.type === "complete" || result.type === "failed") {
recorded.push(result.type);
} else {
if (result.type === "transition") {
recorded.push(result.next);
} else {
recorded.push(result.type);
}
},
recordInvalidatedStateResult: async (result) => {
Expand Down
4 changes: 2 additions & 2 deletions src/lib/onboard/machine/final-flow-phases.ts
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,7 @@ export async function runFinalOnboardFlowSlice<Context extends OnboardFlowContex
recordInvalidatedStateResult: InvalidatedOnboardStateResultRecorder;
afterPoliciesResultApplied?(): void;
onContextUpdated?(context: Context): void;
}): Promise<void> {
}) {
// Recompute plan for live resume repair when durable machine snapshots
// are already downstream of this slice even though branch setup/readiness,
// policy reconciliation, and final verification must still re-run. Those
Expand All @@ -180,7 +180,7 @@ export async function runFinalOnboardFlowSlice<Context extends OnboardFlowContex
// tests cover ahead-state resume and terminal-state rejection; remove this
// fallback once final-phase repair checks are first-class FSM recovery states
// and legacy machine step mutation is gone.
await runLiveOnboardFlowSlice({
return runLiveOnboardFlowSlice({
context: options.context,
runtime: withAfterPoliciesResultApplied(options.runtime, options.afterPoliciesResultApplied),
phases: withContextObserver(options.phases, options.onContextUpdated),
Expand Down
51 changes: 49 additions & 2 deletions src/lib/onboard/machine/handlers/finalization.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,8 @@ function createDeps(
diagnostics: vi.fn(() => [" ✓ verified"]),
verifyWebSearch: vi.fn(),
dashboard: vi.fn(),
isHealthy: vi.fn(() => true),
reportReadiness: vi.fn(),
error: vi.fn(),
log: vi.fn(),
};
Expand All @@ -63,6 +65,8 @@ function createDeps(
formatVerificationDiagnostics: calls.diagnostics,
verifyWebSearchInsideSandbox: calls.verifyWebSearch,
printDashboard: calls.dashboard,
isDeploymentHealthy: calls.isHealthy,
reportDeploymentReadiness: calls.reportReadiness,
error: calls.error,
log: calls.log,
...overrides,
Expand Down Expand Up @@ -104,7 +108,14 @@ describe("handleFinalizationState", () => {
expect(calls.buildChain).toHaveBeenCalledWith("http://127.0.0.1:18789");
expect(calls.verify).toHaveBeenCalledWith("my-assistant", { port: 18789 });
expect(calls.log).toHaveBeenCalledWith(" ✓ verified");
expect(calls.dashboard).toHaveBeenCalledWith("my-assistant", "model", "provider", null, null);
expect(calls.dashboard).toHaveBeenCalledWith(
"my-assistant",
"model",
"provider",
null,
null,
true,
);
expect(calls.postVerify).toHaveBeenCalledOnce();
expect(result.stateResult).toEqual({
type: "complete",
Expand All @@ -120,6 +131,35 @@ describe("handleFinalizationState", () => {
expect(result.verificationDiagnostics).toEqual([" ✓ verified"]);
});

it("prints a not-ready dashboard and returns a resumable failure when verification is unhealthy", async () => {
const { deps, calls } = createDeps({ isDeploymentHealthy: vi.fn(() => false) });

const result = await handleFinalizationState(baseOptions(deps));

expect(calls.dashboard).toHaveBeenCalledWith(
"my-assistant",
"model",
"provider",
null,
null,
false,
);
expect(calls.reportReadiness).toHaveBeenCalledWith(false);
expect(calls.postVerify).toHaveBeenCalledOnce();
expect(result.deploymentHealthy).toBe(false);
expect(result.stateResult).toEqual({
type: "pause",
updates: {
sandboxName: "my-assistant",
provider: "provider",
model: "model",
hermesAuthMethod: null,
hermesToolGateways: [],
},
metadata: { state: "finalizing", reason: "deployment_not_ready" },
});
});

it("ensures agent dashboard forwarding before completion for non-OpenClaw agents", async () => {
const { deps, calls } = createDeps();
const agent = { name: "hermes" };
Expand All @@ -130,7 +170,14 @@ describe("handleFinalizationState", () => {
expect(calls.ensureAgentDashboard.mock.invocationCallOrder[0]).toBeLessThan(
calls.dashboard.mock.invocationCallOrder[0],
);
expect(calls.dashboard).toHaveBeenCalledWith("my-assistant", "model", "provider", null, agent);
expect(calls.dashboard).toHaveBeenCalledWith(
"my-assistant",
"model",
"provider",
null,
agent,
true,
);
});

it("skips dashboard and gateway verification for terminal agents without forwards", async () => {
Expand Down
Loading
Loading