diff --git a/docs/reference/commands.mdx b/docs/reference/commands.mdx index 443058f6292..1d286aec2db 100644 --- a/docs/reference/commands.mdx +++ b/docs/reference/commands.mdx @@ -1452,6 +1452,9 @@ For Portable Hermes, `status` reports `Portable lifecycle phase: pending`, `conf +For a non-Portable sandbox, after `stop` completes, status reports `phase: "Stopped"`, keeps `failureLayer` null, skips runtime and inference probes, and exits `0`. +A non-Portable stopped runtime without recorded stop intent remains a `sandbox_container_stopped` failure. + For a `compatible-endpoint` route that uses `openai-completions`, the text output prints `Reasoning effort` as `low`, `medium`, `high`, or `endpoint-default`. The line is omitted for another provider or API family. Pass `--json` to emit a structured per-sandbox report instead of the text renderer. The JSON output includes at least `schemaVersion`, `name`, `found`, `agent`, `agentDisplayName`, `agentRuntime`, `dcodeAutoApprovalMode`, `model`, `provider`, `recordedRoute`, `liveRoute`, `routeDrift`, `phase`, `gatewayState`, `inferenceHealth`, `rpcIssue`, `hostGpuDetected`, `sandboxGpuEnabled`, `sandboxGpuMode`, `sandboxGpuDevice`, `openshellDriver`, `openshellVersion`, `policies`, `policiesAvailable`, `llamaCpp`, `failureLayer`, `terminalRuntimeHealth`, `servingProcessHealth`, and `dockerPaused`. `policies` is derived from the current OpenShell policy; NemoClaw does not persist a second preset list or baseline-exclusion ledger. `policiesAvailable` is `false` when that live policy cannot be read or parsed, distinguishing an unavailable result from a verified empty `policies` array; text status prints `Policies: unavailable` for the same state. When the live gateway route matches the sandbox's recorded llama.cpp route, `llamaCpp.kind` is `attached`, `managed`, or `unavailable`. An attached route also includes its fixed loopback `llamaCpp.endpointUrl`. An unavailable result means NemoClaw could not safely verify the managed ownership receipt; it does not include an endpoint and provides a secret-free diagnostic and recovery action. During route drift, `status` omits llama.cpp ownership attribution; inspect `routeDrift` before acting on ownership or endpoint information. The schema-version `1` `model` and `provider` fields keep their established live-route meaning when the gateway route is readable. Use `recordedRoute` for the sandbox's durable provider and model and `liveRoute` for the gateway-global route. When the live shared route differs, text output prints both routes and JSON output sets `routeDrift.live`, `routeDrift.recorded`, and `routeDrift.canConnect`. When `routeDrift.canConnect` is `false`, `connect` cannot safely restore the recorded route because provider-global identity differs or required route or gateway metadata is incomplete. Refer to [Use Shared Gateway Routes](../inference/manage-inference/use-shared-gateway-routes) for the route-sharing workflow. `openshellDriver` and `openshellVersion` are always strings (falling back to `"unknown"` when the registry has no value), so consumers can rely on `typeof` checks. `agent` is always a string and reports `openclaw` when the registry records no agent for the sandbox. `failureLayer` is `null` when no preflight failure was detected and otherwise one of `docker_unreachable`, `sandbox_container_stopped`, or `sandbox_dashboard_port_conflict`; when set, `inferenceHealth` is suppressed to `null` so automation does not see a stale remote-provider healthy status during a local outage. `inferenceHealth.ok` reports whether the inference route returned a structurally valid result for one request sent from inside the sandbox. The result must match Chat Completions, Responses, or Anthropic Messages for the selected route. An empty body, malformed JSON, provider-error envelope, or wrong response shape reports `unhealthy`, even with a 2xx status. The probe captures at most 64 KiB and does not include the response body in diagnostics. The route probe treats any final HTTP status from `200` through `499` as reachable, so a route with an invalidated provider credential answers HTTP `401` while the route is up. The request uses the live gateway route's provider and model, and falls back to the recorded values when the live route is unreadable. When the live provider matches the recorded provider, the request uses the sandbox's recorded API family, even when only the model differs. This includes `openai-responses`. When the live provider differs, NemoClaw does not carry the recorded API family to the live provider. An ordinary run sends one 16-token request per attempt through the stored provider credential, with a 30-second timeout for each request, and consumes provider tokens on a hosted route. When the inference request returns HTTP `429`, `502`, `503`, or `504`, `status` retries the route and inference request together up to three total attempts, with a two-second delay between failed attempts, because those statuses are transient gateway and availability answers rather than evidence that the route is broken. Before each retry, it writes the failed probe boundary, an HTTP status when one is available, the next attempt, total attempts, and delay to stderr; `--json` keeps stdout machine-readable. On an ordinary run, every other failure is final on the first attempt with no delay: HTTP `401`, `403`, `404`, and `500`, an invalid 2xx response body, a request that returned no HTTP status, and a failing `/v1/models` route probe. When the same run recovers a managed gateway, `status` retries any failed route or inference probe on that schedule instead, while the restarted delivery chain settles. Each inference request can consume another 16 tokens on a hosted route. Each route probe has a 10-second timeout and each inference request has a 30-second timeout, so three complete probe pairs and two delays can take about 124 seconds. When NemoClaw sends an inference request, `inferenceHealth.subprobes` reports the route probe result as the `route reachability` hop, so a failing verdict still shows that the route itself answered. `inferenceHealth.failureLabel` reports why the inference request failed: diff --git a/src/lib/actions/sandbox/gateway-state-observe-mode.test.ts b/src/lib/actions/sandbox/gateway-state-observe-mode.test.ts index f784f3bfbac..00416277dfb 100644 --- a/src/lib/actions/sandbox/gateway-state-observe-mode.test.ts +++ b/src/lib/actions/sandbox/gateway-state-observe-mode.test.ts @@ -5,6 +5,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import * as gatewayRuntime from "../../gateway-runtime-action"; import * as openshellRuntime from "../../adapters/openshell/runtime"; +import * as dockerDriverRecovery from "../../onboard/docker-driver-sandbox-recovery"; import * as portableAgentLifecycle from "../../onboard/experimental/portable-agent-lifecycle"; import * as registry from "../../state/registry"; import * as gatewaySelect from "./gateway-select"; @@ -104,6 +105,19 @@ describe("getReconciledSandboxGatewayState observe mode", () => { expect(result).toMatchObject({ state: "present" }); }); + it("does not restore a missing sandbox in observe mode (#11025)", async () => { + const recover = vi.spyOn(dockerDriverRecovery, "recoverDockerDriverSandbox"); + const getState = vi.fn().mockResolvedValue({ state: "missing", output: "not found" }); + + const result = await getReconciledSandboxGatewayState("beta", { + getState, + gatewayRecovery: "observe", + }); + + expect(recover).not.toHaveBeenCalled(); + expect(result).toMatchObject({ state: "missing", output: "not found" }); + }); + it("keeps receipt-owned observation scoped without changing global gateway selection (#9203)", async () => { const getState = vi.fn().mockResolvedValue({ state: "present", output: "Phase: Ready" }); diff --git a/src/lib/actions/sandbox/gateway-state.ts b/src/lib/actions/sandbox/gateway-state.ts index ecb8b5c7815..0891f77e7c9 100644 --- a/src/lib/actions/sandbox/gateway-state.ts +++ b/src/lib/actions/sandbox/gateway-state.ts @@ -929,6 +929,7 @@ export async function getReconciledSandboxGatewayState( return lookup; } if (lookup.state === "missing") { + if (gatewayRecovery === "observe") return lookup; return reconcileMissingAgainstNamedGateway(sandboxName, lookup, targetGatewayName); } diff --git a/src/lib/actions/sandbox/rebuild-flow-lifecycle.test.ts b/src/lib/actions/sandbox/rebuild-flow-lifecycle.test.ts index 90a7f04da2a..30d02d78404 100644 --- a/src/lib/actions/sandbox/rebuild-flow-lifecycle.test.ts +++ b/src/lib/actions/sandbox/rebuild-flow-lifecycle.test.ts @@ -220,6 +220,7 @@ describe("rebuildSandbox flow: lifecycle", () => { hermesAuthMethod: null, }), ); + expect(harness.registryUpdateSpy).toHaveBeenCalledWith("alpha", { stopped: false }); const deleteCall = harness.runOpenshellSpy.mock.calls.findIndex( (call) => Array.isArray(call[0]) && diff --git a/src/lib/actions/sandbox/rebuild-pipeline.ts b/src/lib/actions/sandbox/rebuild-pipeline.ts index bea0cf499be..90e21736317 100644 --- a/src/lib/actions/sandbox/rebuild-pipeline.ts +++ b/src/lib/actions/sandbox/rebuild-pipeline.ts @@ -77,7 +77,10 @@ import { recordRebuildRecoveryBackup, } from "./rebuild-recreate-journal"; import { runRebuildRecreatePhase } from "./rebuild-recreate-phase"; -import { createRebuildRegistryRollback } from "./rebuild-registry-rollback"; +import { + createRebuildRegistryRollback, + persistSandboxStopIntent, +} from "./rebuild-registry-rollback"; import { runRebuildRestorePhase } from "./rebuild-restore-phase"; export { buildRefreshMutableOpenClawConfigHashCommand, stageMessagingManifestPlanForRebuild }; @@ -837,6 +840,11 @@ async function rebuildSandboxUnlocked( } retireRemovedImmutabilityStateRecord(sandboxName, "mutable-rebuild"); } + if (!persistSandboxStopIntent(sandboxName, false)) { + return bail( + `Sandbox '${sandboxName}' was rebuilt, but NemoClaw could not clear its intentional-stop record. Run 'nemoclaw ${sandboxName} status' before another lifecycle command.`, + ); + } if (backup.backupManifest) { if (!completePolicyHandoffCleanup(recreateJournal.id, backup.backupManifest)) return; if (!clearRecoveryMarker(recreateJournal.id, backup.backupManifest)) return; diff --git a/src/lib/actions/sandbox/rebuild-registry-rollback.ts b/src/lib/actions/sandbox/rebuild-registry-rollback.ts index 209e77d5e48..9229050e4ac 100644 --- a/src/lib/actions/sandbox/rebuild-registry-rollback.ts +++ b/src/lib/actions/sandbox/rebuild-registry-rollback.ts @@ -16,6 +16,10 @@ export interface RebuildRegistryRollback { restoreForRetry(): void; } +export function persistSandboxStopIntent(name: string, stopped: boolean): boolean { + return registry.recordSandboxStopIntent(name, stopped, registry.updateSandbox); +} + interface RebuildRegistryRollbackDeps { restoreSandboxEntry?: typeof registry.restoreSandboxEntry; restoreSandboxEntryIfMissing?: typeof registry.restoreSandboxEntryIfMissing; diff --git a/src/lib/actions/sandbox/runtime/hermes-cron-restore-recovery.test.ts b/src/lib/actions/sandbox/runtime/hermes-cron-restore-recovery.test.ts index 230e7899ed4..329588ae706 100644 --- a/src/lib/actions/sandbox/runtime/hermes-cron-restore-recovery.test.ts +++ b/src/lib/actions/sandbox/runtime/hermes-cron-restore-recovery.test.ts @@ -102,7 +102,6 @@ describe("sandbox recovery with a Hermes cron restore gate", () => { await expect(recoverSandboxWithHermesCronRestore("alpha")).rejects.toThrow( "recovery authority is unsafe", ); - expect(mocks.connectSandbox).not.toHaveBeenCalled(); expect(mocks.recoverHermesCronRestore).not.toHaveBeenCalled(); }); diff --git a/src/lib/actions/sandbox/start.test.ts b/src/lib/actions/sandbox/start.test.ts index 8557ff91f38..570345ccc76 100644 --- a/src/lib/actions/sandbox/start.test.ts +++ b/src/lib/actions/sandbox/start.test.ts @@ -32,7 +32,8 @@ const FAILED_RECOVERY = { ...SUCCESSFUL_RECOVERY, wasRunning: false } as const; const REDACTED_TOKEN = "opaque-token-8662"; function harness(overrides: Partial = {}) { - const getSandbox = vi.fn>(() => sandbox()); + let storedSandbox = sandbox({ stopped: true }); + const getSandbox = vi.fn>(() => storedSandbox); const isDockerRuntimeDown = vi.fn( () => false, ); @@ -73,6 +74,10 @@ function harness(overrides: Partial = {}) { NonNullable >(() => false); const log = vi.fn<(message: string) => void>(); + const updateSandbox = vi.fn>((_name, updates) => { + storedSandbox = { ...storedSandbox, ...updates }; + return true; + }); const runtimeProviders = createRuntimeProviderBundleRegistry([ [ "docker", @@ -94,6 +99,7 @@ function harness(overrides: Partial = {}) { restoreStartupState, waitForManagedGatewaySupervisor, verifyGateway, + updateSandbox, log, withLifecycleLock: async (_sandboxName, operation) => operation(), ...overrides, @@ -110,6 +116,7 @@ function harness(overrides: Partial = {}) { recoverDockerDriverSandbox, recoverPortableSandbox, restoreStartupState, + updateSandbox, waitForManagedGatewaySupervisor, verifyGateway, }; @@ -244,6 +251,24 @@ describe("startSandbox", () => { ); }); + it("records stopped: false in the sandbox registry on successful start (#11025)", async () => { + const h = harness(); + + const result = await startSandbox("my-sandbox", h.deps); + + expect(result.exitCode).toBe(0); + expect(h.updateSandbox).toHaveBeenCalledWith("my-sandbox", { stopped: false }); + expect(h.getSandbox("my-sandbox")?.stopped).toBe(false); + }); + + it("reports a partial success when the running state cannot be recorded (#11025)", async () => { + const h = harness({ updateSandbox: vi.fn(() => false) }); + + await expect(startSandbox("my-sandbox", h.deps)).rejects.toThrow( + "started, but NemoClaw could not clear its intentional-stop record", + ); + }); + it( "retries startup after a structured recovery failure (#8662)", testTimeoutOptions(30_000), @@ -770,6 +795,7 @@ describe("startSandbox", () => { const result = await startSandbox("my-sandbox", h.deps); expect(result.exitCode).toBe(1); + expect(h.updateSandbox).toHaveBeenCalledWith("my-sandbox", { stopped: false }); const output = h.log.mock.calls.map(([line]) => line).join("\n"); expect(output).toContain("HTTP 401"); expect(output).toContain("doctor"); diff --git a/src/lib/actions/sandbox/start.ts b/src/lib/actions/sandbox/start.ts index f70fa1bc97b..178a463e9ec 100644 --- a/src/lib/actions/sandbox/start.ts +++ b/src/lib/actions/sandbox/start.ts @@ -71,6 +71,7 @@ export interface SandboxStartDeps { observer?: OpenShellSandboxObserver; environment?: NodeJS.ProcessEnv; getSandbox?: typeof registry.getSandbox; + updateSandbox?: typeof registry.updateSandbox; restoreProcessState?: (sandboxName: string) => SandboxStartupRecoveryResult; runtimeProviders?: RuntimeProviderBundleRegistry; restoreStartupState?: ( @@ -238,6 +239,17 @@ async function startSandboxWithinLifecycleFence( await (deps.verifyGateway ?? verifyGateway)(name); readiness.inference = checkStartedSandboxInference(name, resolved.sandbox, deps, log); }); + if ( + !registry.recordSandboxStopIntent( + sandboxName, + false, + deps.updateSandbox ?? registry.updateSandbox, + ) + ) { + throw new Error( + `Sandbox '${sandboxName}' started, but NemoClaw could not clear its intentional-stop record. Run '${cliName()} ${sandboxName} status' before another lifecycle command.`, + ); + } if (readiness.inference && !readiness.inference.ok) { log(` The sandbox started but inference is not usable: ${readiness.inference.detail}.`); log(` Run the sandbox doctor command for '${sandboxName}' to identify the failing hop.`); diff --git a/src/lib/actions/sandbox/status-inference.test.ts b/src/lib/actions/sandbox/status-inference.test.ts index 247077d1307..01669215448 100644 --- a/src/lib/actions/sandbox/status-inference.test.ts +++ b/src/lib/actions/sandbox/status-inference.test.ts @@ -12,6 +12,8 @@ import { describe("sandbox status inference.local route health (#6192)", () => { function snapshotDeps(options: { agent?: string; + confirmedStopped?: boolean; + stopped?: boolean; lookupState?: "present" | "missing"; provider?: string; liveProvider?: string; @@ -35,6 +37,7 @@ describe("sandbox status inference.local route health (#6192)", () => { model: "nvidia/nemotron", provider, preferredInferenceApi: options.preferredInferenceApi, + ...(options.stopped !== undefined ? { stopped: options.stopped } : {}), }; return { getSandbox: () => sandbox, @@ -42,12 +45,23 @@ describe("sandbox status inference.local route health (#6192)", () => { reconcile: async () => options.lookupState === "missing" ? { state: "missing" as const, output: "sandbox alpha not found" } - : { state: "present" as const, output: "Name: alpha\nPhase: Ready\n" }, + : { + state: "present" as const, + phase: "Ready", + output: "Name: alpha\nPhase: Ready\n", + }, captureOpenshellForStatusImpl: async () => ({ status: 0, output: `Gateway inference:\n Provider: ${options.liveProvider ?? provider}\n Model: ${options.liveModel ?? "nvidia/nemotron"}\n`, }) as never, + getSandboxStatusPreflightImpl: vi.fn(async () => ({ + failure: null, + failureLayer: null, + intentionalStopConfirmed: options.confirmedStopped === true, + suppressInferenceProbe: options.confirmedStopped === true, + exitCode: 0 as const, + })), probeProviderHealthImpl: vi.fn( options.providerProbeThrows ? () => { @@ -123,6 +137,44 @@ describe("sandbox status inference.local route health (#6192)", () => { expect(report.servingProcessHealth).toBeNull(); }); + it("does not probe terminal runtime health when the sandbox is stopped (#11025)", async () => { + const deps = snapshotDeps({ + agent: "langchain-deepagents-code", + confirmedStopped: true, + stopped: true, + routeHealth: { + ok: true, + endpoint: "https://inference.local/v1/models", + httpStatus: 200, + detail: "route reachable", + }, + }); + + const snapshot = await collectSandboxStatusSnapshot("alpha", { deps }); + + expect(snapshot.terminalRuntimeHealth).toBeNull(); + expect(deps.probeTerminalRuntimeHealth).not.toHaveBeenCalled(); + expect(deps.probeProviderHealthImpl).not.toHaveBeenCalled(); + expect(deps.probeSandboxInferenceGatewayHealthImpl).not.toHaveBeenCalled(); + }); + + it("probes a live sandbox when its persisted stop marker is stale (#11025)", async () => { + const deps = snapshotDeps({ + stopped: true, + routeHealth: { + ok: true, + endpoint: "https://inference.local/v1/models", + httpStatus: 200, + detail: "route reachable", + }, + }); + + const report = await getSandboxStatusReport("alpha", deps); + + expect(report.phase).toBe("Ready"); + expect(deps.probeSandboxInferenceGatewayHealthImpl).toHaveBeenCalled(); + }); + it("does not invent serving-process health when the gateway is unavailable (#7003)", async () => { const deps = snapshotDeps({ lookupState: "missing", diff --git a/src/lib/actions/sandbox/status-lookup-rendering.test.ts b/src/lib/actions/sandbox/status-lookup-rendering.test.ts index 962e31c2a3b..0aacd5d89ad 100644 --- a/src/lib/actions/sandbox/status-lookup-rendering.test.ts +++ b/src/lib/actions/sandbox/status-lookup-rendering.test.ts @@ -92,6 +92,23 @@ describe("printNonReadySandboxPhaseGuidance (#7222)", () => { expect(text).not.toContain("rebuild --yes"); }); + it("renders Phase: Stopped and clean stopped guidance when phase is Stopped (#11025)", async () => { + const cap = captureConsoleLog(); + await printGuidance({ + phase: "Stopped", + dockerRuntime: null, + }); + const text = cap.lines(); + cap.restore(); + + expect(text).toContain("Phase: Stopped"); + expect(text).toContain("Sandbox 'beta' is stopped."); + expect(text).toContain("Workspace state is preserved."); + expect(text).toContain("nemoclaw beta start"); + expect(text).not.toContain("is stuck"); + expect(text).not.toContain("rebuild --yes"); + }); + it("keeps the unpause hint for a paused container and never suggests start/rebuild (#4495)", async () => { const cap = captureConsoleLog(); await printGuidance({ diff --git a/src/lib/actions/sandbox/status-lookup-rendering.ts b/src/lib/actions/sandbox/status-lookup-rendering.ts index 0785c151930..e27892bac86 100644 --- a/src/lib/actions/sandbox/status-lookup-rendering.ts +++ b/src/lib/actions/sandbox/status-lookup-rendering.ts @@ -133,7 +133,12 @@ function printPresentSandboxGatewayLookupStatus({ ); console.log(""); } - console.log(lookup.output); + const isStopped = phase === "Stopped"; + const renderedOutput = + isStopped && lookup.output + ? lookup.output.replace(/^(\s*Phase:\s*)\S+\s*$/gmu, "$1Stopped") + : lookup.output; + if (renderedOutput) console.log(renderedOutput); printNonReadySandboxPhaseGuidance({ sandboxName, phase, dockerRuntime }); } @@ -237,7 +242,10 @@ function printNonReadySandboxPhaseGuidance({ dockerRuntime: ReturnType | null; }): void { if (!phase || phase === "Ready") return; - if (dockerRuntime?.containerName && !dockerRuntime.running && !dockerRuntime.paused) { + if ( + phase === "Stopped" || + (dockerRuntime?.containerName && !dockerRuntime.running && !dockerRuntime.paused) + ) { console.log(""); console.log(` Sandbox '${sandboxName}' is stopped.`); console.log(" Workspace state is preserved."); diff --git a/src/lib/actions/sandbox/status-preflight.ts b/src/lib/actions/sandbox/status-preflight.ts index b4f772bcb39..8faf642a34b 100644 --- a/src/lib/actions/sandbox/status-preflight.ts +++ b/src/lib/actions/sandbox/status-preflight.ts @@ -49,6 +49,8 @@ export interface SandboxStatusPreflightFailure { export interface SandboxStatusPreflightResult { failure: SandboxStatusPreflightFailure | null; failureLayer: SandboxStatusFailureLayer | null; + /** True only when persisted stop intent agrees with the provider-owned live observation. */ + intentionalStopConfirmed?: boolean; suppressInferenceProbe: boolean; exitCode: 0 | 1; } @@ -184,14 +186,27 @@ export async function getSandboxStatusPreflight( deps: ClassifySandboxStatusPreflightFailureDeps = {}, ): Promise { const failure = await classifySandboxStatusPreflightFailure(sb, deps); + const intentionalStopConfirmed = Boolean( + sb?.stopped && failure?.layer === "sandbox_container_stopped", + ); + const effectiveFailure = intentionalStopConfirmed ? null : failure; return { - failure, - failureLayer: failure ? failure.layer : null, + failure: effectiveFailure, + failureLayer: effectiveFailure ? effectiveFailure.layer : null, + intentionalStopConfirmed, suppressInferenceProbe: failure !== null, - exitCode: failure ? 1 : 0, + exitCode: effectiveFailure ? 1 : 0, }; } +/** Project a provider-confirmed intentional stop onto the OpenShell phase. */ +export function resolveSandboxStatusPhase( + observedPhase: string | null, + preflight: SandboxStatusPreflightResult, +): string | null { + return preflight.intentionalStopConfirmed ? "Stopped" : observedPhase; +} + /** * Preserve terminal OpenShell sandbox phases as the primary user-facing cause * only for host-wide Docker daemon outages. A terminal `Failed`/`Error` phase @@ -209,6 +224,7 @@ export function withoutTerminalPhasePreflight( return { failure: null, failureLayer: null, + intentionalStopConfirmed: preflight.intentionalStopConfirmed, suppressInferenceProbe: preflight.suppressInferenceProbe, exitCode: 0, }; diff --git a/src/lib/actions/sandbox/status-snapshot.ts b/src/lib/actions/sandbox/status-snapshot.ts index 6fc3cb16221..f4e16570a75 100644 --- a/src/lib/actions/sandbox/status-snapshot.ts +++ b/src/lib/actions/sandbox/status-snapshot.ts @@ -50,6 +50,7 @@ import { import { getSandboxStatusPreflight, hasLegacyStatusRuntimeObservation, + resolveSandboxStatusPhase, type SandboxStatusFailureLayer, type SandboxStatusPreflightResult, usesManagedProviderGateway, @@ -422,12 +423,6 @@ export async function collectSandboxStatusSnapshot( deps?: CollectSandboxStatusSnapshotDeps; } = {}, ): Promise { - const reconcile = - opts.deps?.reconcile ?? - ((name: string) => - getReconciledSandboxGatewayState(name, { - getState: getSandboxGatewayStateForStatus, - })); const getSandbox = opts.deps?.getSandbox ?? ((name: string) => { @@ -435,6 +430,20 @@ export async function collectSandboxStatusSnapshot( return entry && registry.isPublishedSandboxRegistration(entry) ? entry : null; }); const sb = getSandbox(sandboxName); + const initialPreflight = + opts.preflight ?? + (sb?.stopped + ? await (opts.deps?.getSandboxStatusPreflightImpl ?? getSandboxStatusPreflight)(sb) + : undefined); + const reconcile = + opts.deps?.reconcile ?? + ((name: string) => + getReconciledSandboxGatewayState(name, { + getState: getSandboxGatewayStateForStatus, + ...(initialPreflight?.intentionalStopConfirmed + ? { gatewayRecovery: "observe" as const } + : {}), + })); let lookup: SandboxGatewayState; try { lookup = await reconcile(sandboxName); @@ -452,7 +461,7 @@ export async function collectSandboxStatusSnapshot( usesManagedProviderGateway(sb) && (sb.agent ?? "openclaw") === "openclaw" && lookup.phase === "Ready" && - !opts.preflight?.failure; + !initialPreflight?.failure; let recoveredManagedGateway = false; if ( lookup.state === "present" && @@ -489,15 +498,15 @@ export async function collectSandboxStatusSnapshot( } } const postRecoveryPreflight = - dockerRecovered && opts.preflight + dockerRecovered && initialPreflight ? await refreshPreflightAfterDockerRecovery( sb, - opts.preflight, + initialPreflight, opts.deps?.getSandboxStatusPreflightImpl ?? getSandboxStatusPreflight, ) : undefined; const suppressInferenceProbe = - (postRecoveryPreflight ?? opts.preflight)?.suppressInferenceProbe ?? + (postRecoveryPreflight ?? initialPreflight)?.suppressInferenceProbe ?? opts.suppressInferenceProbe === true; let liveResult: Awaited> | null = null; let gatewayName: string | null = null; @@ -695,7 +704,7 @@ export async function collectSandboxStatusSnapshot( ); const statusAgent = resolveSandboxStatusAgent(sb?.agent || "openclaw"); const terminalRuntimeHealth = - lookup.state === "present" && statusAgent.agentRuntime === "terminal" + lookup.state === "present" && !suppressInferenceProbe && statusAgent.agentRuntime === "terminal" ? (opts.deps?.probeTerminalRuntimeHealth ?? probeTerminalRuntimeCgroupOom)(sandboxName) : null; // The serving-process leg is only meaningful when the gateway is up. A @@ -768,7 +777,11 @@ async function buildSandboxStatusReport( lookup.state === "present" && hasLegacyStatusRuntimeObservation(sb) ? getSandboxDockerRuntime(sandboxName) : null; - const phase = lookup.state === "present" ? (lookup.phase ?? null) : null; + const observedPhase = lookup.state === "present" ? (lookup.phase ?? null) : null; + const phase = resolveSandboxStatusPhase( + observedPhase, + snapshot.postRecoveryPreflight ?? preflight, + ); const effectivePreflight = withoutTerminalPhasePreflight( snapshot.postRecoveryPreflight ?? preflight, phase, diff --git a/src/lib/actions/sandbox/status-text.ts b/src/lib/actions/sandbox/status-text.ts index 11cc1c4ddfb..76e6c0fcf48 100644 --- a/src/lib/actions/sandbox/status-text.ts +++ b/src/lib/actions/sandbox/status-text.ts @@ -46,6 +46,7 @@ export interface SandboxStatusTextContext > { sandboxName: string; statusAgent: SandboxStatusAgentInfo; + phase: string | null; } export interface SandboxStatusTextOutcome { @@ -55,11 +56,14 @@ export interface SandboxStatusTextOutcome { /** Returns true when status can validate an agent version against the running sandbox. */ function shouldProbeSandboxRuntimeVersion( lookup: SandboxGatewayState, + phase: string | null, sandbox: SandboxEntry, agentRuntimeKind: string, ): boolean { return ( - lookup.state === "present" && (Boolean(sandbox.agentVersion) || agentRuntimeKind === "terminal") + phase !== "Stopped" && + lookup.state === "present" && + (Boolean(sandbox.agentVersion) || agentRuntimeKind === "terminal") ); } @@ -229,7 +233,12 @@ function printActiveSessions(sandboxName: string): void { function printAgentVersion(context: SandboxStatusTextContext, sandbox: SandboxEntry): void { try { const { lookup, sandboxName, statusAgent } = context; - const shouldProbe = shouldProbeSandboxRuntimeVersion(lookup, sandbox, statusAgent.agentRuntime); + const shouldProbe = shouldProbeSandboxRuntimeVersion( + lookup, + context.phase, + sandbox, + statusAgent.agentRuntime, + ); const versionCheck = sandboxVersion.checkAgentVersion(sandboxName, { forceProbe: shouldProbe, skipProbe: !shouldProbe, @@ -380,7 +389,7 @@ async function printGatewayProcessStatus(context: SandboxStatusTextContext): Pro /** Render the live agent process status after the gateway lookup is shown. */ export async function printAgentProcessStatus(context: SandboxStatusTextContext): Promise { - if (context.lookup.state !== "present") return; + if (context.lookup.state !== "present" || context.phase === "Stopped") return; if (context.statusAgent.agentRuntime === "gateway") { await printGatewayProcessStatus(context); return; diff --git a/src/lib/actions/sandbox/status.test.ts b/src/lib/actions/sandbox/status.test.ts index a1d434d04cc..04468c14789 100644 --- a/src/lib/actions/sandbox/status.test.ts +++ b/src/lib/actions/sandbox/status.test.ts @@ -9,6 +9,7 @@ import type { ProviderHealthProbeOptions } from "../../inference/health"; import { classifySandboxContainerFailureForStatus, classifySandboxStatusPreflightFailure, + getSandboxStatusPreflight, getSandboxStatusInferenceHealth, getSandboxStatusReport, isDockerDaemonUnreachableForStatus, @@ -384,6 +385,56 @@ describe("classifySandboxStatusPreflightFailure", () => { expect(result).toBeNull(); }); + it("keeps the stopped observation available for intentional-stop classification (#11025)", async () => { + const result = await classifySandboxStatusPreflightFailure( + { name: "alpha", openshellDriver: "docker", stopped: true } as never, + { + dockerProbe: () => true, + sandboxContainerProbe: async () => ({ + layer: "sandbox_container_stopped", + detail: "stub stopped container", + }), + }, + ); + expect(result).toEqual({ + layer: "sandbox_container_stopped", + dockerUnreachable: false, + }); + }); + + it("reports a clean stop only when provider observation confirms persisted intent (#11025)", async () => { + const stopped = await getSandboxStatusPreflight( + { name: "alpha", openshellDriver: "docker", stopped: true } as never, + { + dockerProbe: () => true, + sandboxContainerProbe: async () => ({ + layer: "sandbox_container_stopped", + detail: "stub stopped container", + }), + }, + ); + const running = await getSandboxStatusPreflight( + { name: "alpha", openshellDriver: "docker", stopped: true } as never, + { + dockerProbe: () => true, + sandboxContainerProbe: async () => null, + }, + ); + + expect(stopped).toMatchObject({ + intentionalStopConfirmed: true, + failureLayer: null, + suppressInferenceProbe: true, + exitCode: 0, + }); + expect(running).toMatchObject({ + intentionalStopConfirmed: false, + failureLayer: null, + suppressInferenceProbe: false, + exitCode: 0, + }); + }); + it("returns null when the sandbox is not on the docker driver", async () => { let dockerCalled = false; let sandboxCalled = false; diff --git a/src/lib/actions/sandbox/status.ts b/src/lib/actions/sandbox/status.ts index 1daede89b2b..2bf5687af6a 100644 --- a/src/lib/actions/sandbox/status.ts +++ b/src/lib/actions/sandbox/status.ts @@ -17,6 +17,7 @@ import { printSandboxGatewayLookupStatus } from "./status-lookup-rendering"; import { getSandboxStatusPreflight, printSandboxStatusPreflightHeader, + resolveSandboxStatusPhase, withoutTerminalPhasePreflight, } from "./status-preflight"; import { @@ -42,6 +43,7 @@ export { isDockerDaemonUnreachableForStatus, printGatewayFailureLayerHeader, printSandboxStatusPreflightHeader, + resolveSandboxStatusPhase, type SandboxStatusFailureLayer, type SandboxStatusPreflightFailure, type SandboxStatusPreflightResult, @@ -197,7 +199,11 @@ async function showLegacySandboxStatus(sandboxName: string): Promise { // Resolve the docker-driver container once: reused for the paused-container // recovery hint (#4495) and the Docker health line below (#3975). const dockerRuntime = lookup.state === "present" ? getSandboxDockerRuntime(sandboxName) : null; - const phase = lookup.state === "present" ? (lookup.phase ?? null) : null; + const observedPhase = lookup.state === "present" ? (lookup.phase ?? null) : null; + const phase = resolveSandboxStatusPhase( + observedPhase, + snapshot.postRecoveryPreflight ?? preflight, + ); const effectivePreflight = withoutTerminalPhasePreflight( snapshot.postRecoveryPreflight ?? preflight, phase, @@ -227,6 +233,7 @@ async function showLegacySandboxStatus(sandboxName: string): Promise { terminalRuntimeHealth, servingProcessHealth, statusAgent, + phase, }; const textOutcome = printSandboxDetails(textContext); if (textOutcome.exitCode && (!process.exitCode || process.exitCode === 0)) { diff --git a/src/lib/actions/sandbox/stop.test.ts b/src/lib/actions/sandbox/stop.test.ts index 5d04551f92a..c84cb459c1f 100644 --- a/src/lib/actions/sandbox/stop.test.ts +++ b/src/lib/actions/sandbox/stop.test.ts @@ -81,7 +81,8 @@ function harness(overrides: StopHarnessOverrides = {}) { findLabeledSandboxContainers: findContainersOverride, ...actionOverrides } = overrides; - const getSandbox = vi.fn>(() => sandbox()); + let storedSandbox = sandbox(); + const getSandbox = vi.fn>(() => storedSandbox); const isDockerRuntimeDown = vi.fn( () => false, ); @@ -102,6 +103,10 @@ function harness(overrides: StopHarnessOverrides = {}) { ); const teardownSandboxDashboardForward = vi.fn>(); + const updateSandbox = vi.fn>((_name, updates) => { + storedSandbox = { ...storedSandbox, ...updates }; + return true; + }); const log = vi.fn<(message: string) => void>(); const warn = vi.fn<(message: string) => void>(); const runtimeProviders = createRuntimeProviderBundleRegistry([ @@ -133,12 +138,14 @@ function harness(overrides: StopHarnessOverrides = {}) { }), withOllamaModelOwnershipLock: (operation) => operation(), withLifecycleLockSync: (_sandboxName, operation) => operation(), + updateSandbox, ...actionOverrides, }; return { deps, dockerStop, teardownSandboxDashboardForward, + updateSandbox, findLabeledSandboxContainers, getSandbox, hasPortableLifecycleReceipt, @@ -333,6 +340,50 @@ describe("stopSandbox", () => { expect(h.teardownSandboxDashboardForward).not.toHaveBeenCalled(); }); + it("records stopped: true in the sandbox registry on successful stop (#11025)", () => { + const h = harness(); + + const result = stopSandbox("my-sandbox", h.deps); + + expect(result.exitCode).toBe(0); + expect(h.updateSandbox).toHaveBeenCalledWith("my-sandbox", { stopped: true }); + expect(h.getSandbox("my-sandbox")?.stopped).toBe(true); + }); + + it("does not record stopped: true when container stop fails (#11025)", () => { + const h = harness({ dockerStop: vi.fn(() => ({ status: 1 })) }); + + const result = stopSandbox("my-sandbox", h.deps); + + expect(result.exitCode).toBe(1); + expect(h.updateSandbox).not.toHaveBeenCalled(); + expect(h.getSandbox("my-sandbox")?.stopped).toBeUndefined(); + }); + + it("returns retryable error and still runs cleanup when updateSandbox throws (#11025)", () => { + const teardownSandboxDashboardForward = vi.fn(); + const updateSandbox = vi.fn(() => { + throw new Error("disk full"); + }); + const h = harness({ teardownSandboxDashboardForward, updateSandbox }); + + const result = stopSandbox("my-sandbox", h.deps); + + expect(result.exitCode).toBe(1); + expect(result.message).toContain("could not record the intentional stop"); + expect(result.message).toContain("Retry 'nemoclaw my-sandbox stop'"); + expect(teardownSandboxDashboardForward).toHaveBeenCalledWith("my-sandbox"); + }); + + it("returns a retryable error when the registry row disappears after stop (#11025)", () => { + const h = harness({ updateSandbox: vi.fn(() => false) }); + + const result = stopSandbox("my-sandbox", h.deps); + + expect(result.exitCode).toBe(1); + expect(result.message).toContain("could not record the intentional stop"); + }); + it("releases a leftover dashboard forward for an already-stopped sandbox — idempotent (#7227)", () => { const h = harness({ findLabeledSandboxContainers: vi.fn(() => [container("openshell-my-sandbox", false)]), diff --git a/src/lib/actions/sandbox/stop.ts b/src/lib/actions/sandbox/stop.ts index 6b0a27d7f3d..c8056818653 100644 --- a/src/lib/actions/sandbox/stop.ts +++ b/src/lib/actions/sandbox/stop.ts @@ -238,6 +238,7 @@ export type { SandboxLifecycleResult } from "./runtime/lifecycle-runtime"; export interface SandboxStopDeps { environment?: NodeJS.ProcessEnv; getSandbox?: typeof registry.getSandbox; + updateSandbox?: typeof registry.updateSandbox; runtimeProviders?: RuntimeProviderBundleRegistry; stopSandboxChannels?: typeof stopSandboxChannels; teardownSandboxDashboardForward?: typeof teardownSandboxDashboardForward; @@ -312,6 +313,13 @@ function stopSandboxWithinLifecycleFence( if (outcome.exitCode !== 0) return outcome; const hermesPortableVerified = "hermesPortableVerified" in outcome && outcome.hermesPortableVerified === true; + const stopIntentRecorded = + hermesPortableVerified || + registry.recordSandboxStopIntent( + sandboxName, + true, + deps.updateSandbox ?? registry.updateSandbox, + ); const ollamaRelease = releaseStoppedSandboxOllamaModel(resolved.sandbox, deps, log); if (!hermesPortableVerified) { teardownDashboardForwardBestEffort( @@ -320,6 +328,14 @@ function stopSandboxWithinLifecycleFence( warn, ); } + if (!stopIntentRecorded) { + return { + exitCode: 1, + message: + `Sandbox '${sandboxName}' stopped, but NemoClaw could not record the intentional stop. ` + + `Retry '${CLI_NAME} ${sandboxName} stop'.`, + }; + } if (!ollamaRelease.ok) return { exitCode: 1, message: ollamaRelease.message }; if (hermesPortableVerified) { log( diff --git a/src/lib/state/registry.ts b/src/lib/state/registry.ts index 294ab10b422..79a60f8f0a3 100644 --- a/src/lib/state/registry.ts +++ b/src/lib/state/registry.ts @@ -775,6 +775,19 @@ export function updateSandbox(name: string, updates: Partial): boo }); } +/** Persist intentional-stop state while containing registry write failures. */ +export function recordSandboxStopIntent( + name: string, + stopped: boolean, + update: typeof updateSandbox, +): boolean { + try { + return update(name, { stopped }); + } catch { + return false; + } +} + /** Publish a missing gateway port only while the complete qualified row remains current. */ export function compareAndSetSandboxGatewayPort( name: string, diff --git a/src/lib/state/registry/types.ts b/src/lib/state/registry/types.ts index 14f371a6d11..34a81c46875 100644 --- a/src/lib/state/registry/types.ts +++ b/src/lib/state/registry/types.ts @@ -155,6 +155,8 @@ export interface SandboxEntry extends Partial { // different NEMOCLAW_GATEWAY_PORT no longer recreates/kills the first (#4422). gatewayName?: string | null; gatewayPort?: number | null; + /** Whether the sandbox was intentionally stopped via the stop command (#11025). */ + stopped?: boolean; } export type SandboxWorkloadReceipt = diff --git a/test/cli/sandbox-status-text.test.ts b/test/cli/sandbox-status-text.test.ts index 961aaf5fef9..c741b94d082 100644 --- a/test/cli/sandbox-status-text.test.ts +++ b/test/cli/sandbox-status-text.test.ts @@ -588,4 +588,123 @@ describe("CLI sandbox status text output", () => { expect(parsed.dockerPaused).toBe(true); }, ); + + it( + "sandbox status reports clean Stopped state without failureLayer when stopped (#11025)", + testTimeoutOptions(30_000), + () => { + const home = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-cli-status-stopped-")); + const localBin = path.join(home, "bin"); + const stoppedState = path.join(home, "docker-stopped"); + fs.mkdirSync(localBin, { recursive: true }); + writeSandboxRegistry(home, "alpha", { + openshellDriver: "docker", + openshellVersion: "0.0.44", + }); + fs.writeFileSync( + path.join(localBin, "openshell"), + [ + "#!/usr/bin/env bash", + 'if [ "$1" = "sandbox" ] && [ "$2" = "get" ] && { [ "$3" = "alpha" ] || [ "$5" = "alpha" ]; }; then', + " echo 'Sandbox:'", + " echo", + " echo ' Id: abc'", + " echo ' Name: alpha'", + " echo ' Namespace: openshell'", + " echo ' Phase: Provisioning'", + " exit 0", + "fi", + 'if [ "$1" = "inference" ] && [ "$2" = "get" ]; then', + " echo ' Provider: nvidia-prod'", + " echo ' Model: nvidia/nemotron'", + " exit 0", + "fi", + 'if [ "$1" = "status" ]; then', + " echo 'Gateway: nemoclaw'", + " echo 'Status: Connected'", + " exit 0", + "fi", + 'if [ "$1" = "gateway" ] && [ "$2" = "info" ]; then', + " echo 'Gateway: nemoclaw'", + " exit 0", + "fi", + "exit 0", + ].join("\n"), + { mode: 0o755 }, + ); + fs.writeFileSync( + path.join(localBin, "docker"), + [ + "#!/usr/bin/env bash", + 'if [ "$1" = "info" ]; then echo "24.0.0"; exit 0; fi', + `if [ "$1" = "stop" ]; then touch ${JSON.stringify(stoppedState)}; exit 0; fi`, + 'if [ "$1" = "ps" ]; then', + ` if [ ! -f ${JSON.stringify(stoppedState)} ]; then echo "openshell-alpha-abc123"; exit 0; fi`, + ' for a in "$@"; do [ "$a" = "-a" ] && { echo "openshell-alpha-abc123"; exit 0; }; done', + ' for a in "$@"; do', + ' case "$a" in', + ' *Status*) printf "openshell-alpha-abc123\\tExited (0) 2 hours ago\\n"; exit 0 ;;', + " esac", + " done", + ' echo ""', + " exit 0", + "fi", + 'if [ "$1" = "inspect" ]; then', + ' for a in "$@"; do', + ' case "$a" in', + ` *Running*) if [ -f ${JSON.stringify(stoppedState)} ]; then echo "false"; else echo "true"; fi; exit 0 ;;`, + ' *Paused*) echo "false"; exit 0 ;;', + ' *Health*) echo "none"; exit 0 ;;', + " esac", + " done", + ' echo ""; exit 0', + "fi", + "exit 0", + ].join("\n"), + { mode: 0o755 }, + ); + + const stopped = runWithEnv( + "alpha stop", + { + HOME: home, + PATH: `${localBin}:${process.env.PATH || ""}`, + }, + 30_000, + ); + expect(stopped.code).toBe(0); + + const r = runWithEnv( + "alpha status", + { + HOME: home, + PATH: `${localBin}:${process.env.PATH || ""}`, + }, + 30000, + ); + + expect(r.code, r.out).toBe(0); + expect(r.out).not.toContain("Failure layer:"); + expect(r.out).toContain("Phase: Stopped"); + expect(r.out).not.toContain("Phase: Provisioning"); + expect(r.out).toContain("Sandbox 'alpha' is stopped."); + expect(r.out).toContain("Workspace state is preserved."); + expect(r.out).toContain("Start it again with `nemoclaw alpha start`."); + expect(r.out).not.toContain("rebuild --yes"); + expect(r.out).not.toContain("The sandbox is alive but the"); + + const j = runWithEnv( + "alpha status --json", + { + HOME: home, + PATH: `${localBin}:${process.env.PATH || ""}`, + }, + 30000, + ); + expect(j.code).toBe(0); + const parsed = JSON.parse(j.out); + expect(parsed.phase).toBe("Stopped"); + expect(parsed.failureLayer).toBeNull(); + }, + ); });