Skip to content
6 changes: 5 additions & 1 deletion docs/manage-sandboxes/recover-rebuild-sandboxes.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -262,7 +262,7 @@ $$nemoclaw <sandbox-name> rebuild

### Resolve Rebuild Preflight Stops

Before it backs up or deletes the existing sandbox, `rebuild` validates the recorded sandbox, gateway, policy, MCP, agent, and operation-lock state.
Before it backs up or deletes the existing sandbox, `rebuild` validates the recorded sandbox, gateway, inference route, policy, MCP, agent, and operation-lock state.
When one of these checks fails, NemoClaw prints `Rebuild preflight failed`, explains how to recover, and ends with `Aborting rebuild`.
At this boundary, the existing sandbox is unchanged and no sandbox data has been removed.

Expand All @@ -274,6 +274,10 @@ Use the recovery guidance that matches the reported check:
- Resolve an incomplete MCP destroy transaction before retrying.
- Back up the sandbox state and recreate it with `$$nemoclaw onboard` when the record contains multiple agents. Transactional multi-agent rebuild is not supported.
- Wait for another onboarding or rebuild operation to finish before retrying. If verified stale-lock cleanup is still in progress, wait briefly and rerun the command. Do not delete the lock manually.
- Set the live OpenShell inference route to the sandbox's recorded provider and model when rebuild reports route drift.

A gateway that reports no live inference route does not stop the rebuild.
Replacement onboarding configures and verifies the recorded route before it recreates the sandbox.

<AgentOnly variant="openclaw">
The rebuild command preserves the mounted workspace and registered policies while recreating the container.
Expand Down
9 changes: 7 additions & 2 deletions docs/reference/system-readiness.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -289,8 +289,13 @@ Fresh onboarding and authoritative rebuilds use this order:
8. Revalidate gateway authority immediately before gateway selection, recovery, reconciliation, or other lifecycle effects.

The readiness gate runs before model-router cleanup, provider selection, credential registration, policy changes, image builds, or sandbox lifecycle effects.
Host and gateway observations have a 30-second reuse window.
If collection itself takes too long, onboarding rejects the stale composite instead of assigning a fresh timestamp to old facts.
Host and gateway observations have a 30-second reuse window that starts when collection finishes.
The readiness gate does not reject a collection for the time its own probes take.
The `gateway.owner` evidence records the gateway collection duration as `collectionMs`.
The readiness gate rejects an observation set that waits past the window for another collection.
Onboarding then collects that set again instead of assigning a fresh timestamp to old facts.
Bounded evidence for a rejected set appears under `host.probe.stale` or `gateway.probe.stale` with the applied `windowMs` and the measured `ageMs`.
`ageMs` is `null` when the recorded time cannot be parsed or is later than the current time.

The policy permits only these narrow exceptions:

Expand Down
5 changes: 2 additions & 3 deletions src/lib/onboard.ts
Original file line number Diff line number Diff line change
Expand Up @@ -929,6 +929,7 @@ const providerExistsInGateway = (name: string, gatewayName: string = GATEWAY_NAM
const {
verifyInferenceRoute,
isInferenceRouteReady,
readInferenceRouteState,
checkGatewayRouteCompatibility,
preflightGatewayRouteDiscovery,
} = inferenceRouteHelpers.createInferenceRouteHelpers(runCaptureOpenshell);
Expand All @@ -944,8 +945,6 @@ const { inspectSandboxForCreate, confirmRecreateForSelectionDrift, isOpenclawRea

const { ensureValidatedWebSearchCredential, ensureValidatedBraveSearchCredential, configureWebSearch, verifyWebSearchInsideSandbox, webSearchProviderForConfig } = createWebSearchFlowHelpers({ prompt, note, isNonInteractive, cliName, runCaptureOpenshell });

// getSandboxInferenceConfig — moved to onboard-providers.ts
// Inference probes — moved to inference/onboard-probes.ts
const {
hasResponsesToolCall,
hasChatCompletionsToolCall,
Expand Down Expand Up @@ -3020,7 +3019,7 @@ async function preflightAuthoritativeRebuildTarget(
fail(`OpenShell component preflight exited with code ${String(code)}`),
),
assertGatewayReadiness: onboardPreflightGatewayAuthority.collectGatewayReadiness,
inferenceRouteReady: (p, m) => isInferenceRouteReady(authoritativeGateway.name, p, m),
inferenceRouteState: (p, m) => readInferenceRouteState(authoritativeGateway.name, p, m),
captureForwardList: () => runCaptureOpenshell(["forward", "list"], { ignoreError: true }),
checkPort: (port) => checkPortAvailable(port),
},
Expand Down
33 changes: 23 additions & 10 deletions src/lib/onboard/authoritative-rebuild-target.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import {
rebuildProviderFlowOptions,
resolveAuthoritativeOnboardGatewayBinding,
} from "./authoritative-rebuild-target";
import type { InferenceRouteState } from "./inference-route";
import {
mintProviderRecoveryReceipt,
type ProviderRecoveryReceiptTarget,
Expand Down Expand Up @@ -65,7 +66,7 @@ function deps(overrides: Partial<AuthoritativeRebuildTargetDeps> = {}) {
runFatalRuntimePreflight: vi.fn(),
ensureOpenshell: vi.fn(),
assertGatewayReadiness: vi.fn(),
inferenceRouteReady: vi.fn(() => true),
inferenceRouteState: vi.fn((): InferenceRouteState => "matched"),
captureForwardList: vi.fn(() => "alpha 127.0.0.1 18789 42 active"),
checkPort: vi.fn(async () => ({ ok: true })),
...overrides,
Expand Down Expand Up @@ -265,7 +266,7 @@ describe("authoritative rebuild target preflight", () => {
expect(targetDeps.bindGatewayAuthority).not.toHaveBeenCalled();
expect(targetDeps.ensureOpenshell).not.toHaveBeenCalled();
expect(targetDeps.assertGatewayReadiness).not.toHaveBeenCalled();
expect(targetDeps.inferenceRouteReady).not.toHaveBeenCalled();
expect(targetDeps.inferenceRouteState).not.toHaveBeenCalled();
});

it("pins the requested gateway for route and forward checks, then restores it", async () => {
Expand All @@ -275,9 +276,9 @@ describe("authoritative rebuild target preflight", () => {
await preflightAuthoritativeRebuildTarget(
target,
deps({
inferenceRouteReady: vi.fn(() => {
inferenceRouteState: vi.fn((): InferenceRouteState => {
seen.push(`route:${process.env.OPENSHELL_GATEWAY}`);
return true;
return "matched";
}),
captureForwardList: vi.fn(() => {
seen.push(`forward:${process.env.OPENSHELL_GATEWAY}`);
Expand All @@ -296,13 +297,25 @@ describe("authoritative rebuild target preflight", () => {
await expect(
preflightAuthoritativeRebuildTarget(
target,
deps({ inferenceRouteReady: vi.fn(() => false) }),
deps({ inferenceRouteState: vi.fn((): InferenceRouteState => "mismatched") }),
),
).rejects.toThrow("inference route does not match");
});

it("proceeds when the gateway cannot answer the route query (#9310)", async () => {
const targetDeps = deps({
inferenceRouteState: vi.fn((): InferenceRouteState => "unanswered"),
});

await expect(preflightAuthoritativeRebuildTarget(target, targetDeps)).resolves.toBeUndefined();

expect(targetDeps.inferenceRouteState).toHaveBeenCalledOnce();
});

it("defers route validation for prepared recovery until authoritative onboard (#6114)", async () => {
const targetDeps = deps({ inferenceRouteReady: vi.fn(() => false) });
const targetDeps = deps({
inferenceRouteState: vi.fn((): InferenceRouteState => "mismatched"),
});

await expect(
preflightAuthoritativeRebuildTarget(
Expand All @@ -311,7 +324,7 @@ describe("authoritative rebuild target preflight", () => {
),
).resolves.toBeUndefined();

expect(targetDeps.inferenceRouteReady).not.toHaveBeenCalled();
expect(targetDeps.inferenceRouteState).not.toHaveBeenCalled();
expect(targetDeps.runFatalRuntimePreflight).toHaveBeenCalledOnce();
expect(targetDeps.ensureOpenshell).toHaveBeenCalledOnce();
});
Expand Down Expand Up @@ -366,9 +379,9 @@ describe("authoritative rebuild target preflight", () => {
}),
ensureOpenshell: vi.fn(() => calls.push("openshell")),
assertGatewayReadiness: vi.fn(() => calls.push("gateway")),
inferenceRouteReady: vi.fn(() => {
inferenceRouteState: vi.fn((): InferenceRouteState => {
calls.push("route");
return true;
return "matched";
}),
});

Expand All @@ -393,6 +406,6 @@ describe("authoritative rebuild target preflight", () => {
);
expect(targetDeps.ensureOpenshell).not.toHaveBeenCalled();
expect(targetDeps.assertGatewayReadiness).not.toHaveBeenCalled();
expect(targetDeps.inferenceRouteReady).not.toHaveBeenCalled();
expect(targetDeps.inferenceRouteState).not.toHaveBeenCalled();
});
});
9 changes: 6 additions & 3 deletions src/lib/onboard/authoritative-rebuild-target.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@

import { findDashboardForwardOwner } from "./dashboard-port";
import { resolveGatewayName } from "./gateway-binding";
import type { InferenceRouteState } from "./inference-route";
import type { PortProbeResult } from "./preflight";
import { assertDashboardPortNotReserved } from "./preflight-ports";
import {
Expand Down Expand Up @@ -195,7 +196,7 @@ export type AuthoritativeRebuildTargetDeps = {
runFatalRuntimePreflight(): unknown | Promise<unknown>;
ensureOpenshell(): unknown;
assertGatewayReadiness(): unknown | Promise<unknown>;
inferenceRouteReady(provider: string, model: string): boolean;
inferenceRouteState(provider: string, model: string): InferenceRouteState;
captureForwardList(): string | null;
checkPort(port: number): Promise<PortProbeResult>;
env?: NodeJS.ProcessEnv;
Expand Down Expand Up @@ -223,10 +224,12 @@ export async function preflightAuthoritativeRebuildTarget(
// Prepared-backup recovery can run after the installer has replaced a
// legacy gateway. That fresh gateway has no inference route to validate
// yet; authoritative onboarding configures and verifies the pinned route
// before recreating the sandbox. Normal rebuilds must still match here.
// before recreating the sandbox. A gateway that cannot answer at all leaves
// the route unknown, which onboarding resolves the same way. Only a gateway
// that answers with a different route contradicts the rebuild target.
if (
target.deferInferenceRouteUntilOnboard !== true &&
!deps.inferenceRouteReady(target.provider, target.model)
deps.inferenceRouteState(target.provider, target.model) === "mismatched"
) {
fail(
`OpenShell inference route does not match provider '${target.provider}' and model '${target.model}'.`,
Expand Down
Loading
Loading