fix(sandbox): report clean Stopped phase without failure layer for stopped sandboxes (#11025)

docs/reference/commands.mdx

@@ -1452,6 +1452,9 @@ For Portable Hermes, `status` reports `Portable lifecycle phase: pending`, `conf
 
 </AgentOnly>
 
+For a non-Portable sandbox, after `stop` completes, status reports `phase: "Stopped"`, keeps `failureLayer` null, skips runtime and inference probes, and exits `0`.
+A non-Portable stopped runtime without recorded stop intent remains a `sandbox_container_stopped` failure.
+
 For a `compatible-endpoint` route that uses `openai-completions`, the text output prints `Reasoning effort` as `low`, `medium`, `high`, or `endpoint-default`. The line is omitted for another provider or API family.
 
 Pass `--json` to emit a structured per-sandbox report instead of the text renderer. The JSON output includes at least `schemaVersion`, `name`, `found`, `agent`, `agentDisplayName`, `agentRuntime`, `dcodeAutoApprovalMode`, `model`, `provider`, `recordedRoute`, `liveRoute`, `routeDrift`, `phase`, `gatewayState`, `inferenceHealth`, `rpcIssue`, `hostGpuDetected`, `sandboxGpuEnabled`, `sandboxGpuMode`, `sandboxGpuDevice`, `openshellDriver`, `openshellVersion`, `policies`, `policiesAvailable`, `llamaCpp`, `failureLayer`, `terminalRuntimeHealth`, `servingProcessHealth`, and `dockerPaused`. `policies` is derived from the current OpenShell policy; NemoClaw does not persist a second preset list or baseline-exclusion ledger. `policiesAvailable` is `false` when that live policy cannot be read or parsed, distinguishing an unavailable result from a verified empty `policies` array; text status prints `Policies: unavailable` for the same state. When the live gateway route matches the sandbox's recorded llama.cpp route, `llamaCpp.kind` is `attached`, `managed`, or `unavailable`. An attached route also includes its fixed loopback `llamaCpp.endpointUrl`. An unavailable result means NemoClaw could not safely verify the managed ownership receipt; it does not include an endpoint and provides a secret-free diagnostic and recovery action. During route drift, `status` omits llama.cpp ownership attribution; inspect `routeDrift` before acting on ownership or endpoint information. The schema-version `1` `model` and `provider` fields keep their established live-route meaning when the gateway route is readable. Use `recordedRoute` for the sandbox's durable provider and model and `liveRoute` for the gateway-global route. When the live shared route differs, text output prints both routes and JSON output sets `routeDrift.live`, `routeDrift.recorded`, and `routeDrift.canConnect`. When `routeDrift.canConnect` is `false`, `connect` cannot safely restore the recorded route because provider-global identity differs or required route or gateway metadata is incomplete. Refer to [Use Shared Gateway Routes](../inference/manage-inference/use-shared-gateway-routes) for the route-sharing workflow. `openshellDriver` and `openshellVersion` are always strings (falling back to `"unknown"` when the registry has no value), so consumers can rely on `typeof` checks. `agent` is always a string and reports `openclaw` when the registry records no agent for the sandbox. `failureLayer` is `null` when no preflight failure was detected and otherwise one of `docker_unreachable`, `sandbox_container_stopped`, or `sandbox_dashboard_port_conflict`; when set, `inferenceHealth` is suppressed to `null` so automation does not see a stale remote-provider healthy status during a local outage. `inferenceHealth.ok` reports whether the inference route returned a structurally valid result for one request sent from inside the sandbox. The result must match Chat Completions, Responses, or Anthropic Messages for the selected route. An empty body, malformed JSON, provider-error envelope, or wrong response shape reports `unhealthy`, even with a 2xx status. The probe captures at most 64 KiB and does not include the response body in diagnostics. The route probe treats any final HTTP status from `200` through `499` as reachable, so a route with an invalidated provider credential answers HTTP `401` while the route is up. The request uses the live gateway route's provider and model, and falls back to the recorded values when the live route is unreadable. When the live provider matches the recorded provider, the request uses the sandbox's recorded API family, even when only the model differs. This includes `openai-responses`. When the live provider differs, NemoClaw does not carry the recorded API family to the live provider. An ordinary run sends one 16-token request per attempt through the stored provider credential, with a 30-second timeout for each request, and consumes provider tokens on a hosted route. When the inference request returns HTTP `429`, `502`, `503`, or `504`, `status` retries the route and inference request together up to three total attempts, with a two-second delay between failed attempts, because those statuses are transient gateway and availability answers rather than evidence that the route is broken. Before each retry, it writes the failed probe boundary, an HTTP status when one is available, the next attempt, total attempts, and delay to stderr; `--json` keeps stdout machine-readable. On an ordinary run, every other failure is final on the first attempt with no delay: HTTP `401`, `403`, `404`, and `500`, an invalid 2xx response body, a request that returned no HTTP status, and a failing `/v1/models` route probe. When the same run recovers a managed gateway, `status` retries any failed route or inference probe on that schedule instead, while the restarted delivery chain settles. Each inference request can consume another 16 tokens on a hosted route. Each route probe has a 10-second timeout and each inference request has a 30-second timeout, so three complete probe pairs and two delays can take about 124 seconds. When NemoClaw sends an inference request, `inferenceHealth.subprobes` reports the route probe result as the `route reachability` hop, so a failing verdict still shows that the route itself answered. `inferenceHealth.failureLabel` reports why the inference request failed:

src/lib/actions/sandbox/gateway-state-observe-mode.test.ts

@@ -5,6 +5,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
 
 import * as gatewayRuntime from "../../gateway-runtime-action";
 import * as openshellRuntime from "../../adapters/openshell/runtime";
+import * as dockerDriverRecovery from "../../onboard/docker-driver-sandbox-recovery";
 import * as portableAgentLifecycle from "../../onboard/experimental/portable-agent-lifecycle";
 import * as registry from "../../state/registry";
 import * as gatewaySelect from "./gateway-select";
@@ -104,6 +105,19 @@ describe("getReconciledSandboxGatewayState observe mode", () => {
     expect(result).toMatchObject({ state: "present" });
   });
 
+  it("does not restore a missing sandbox in observe mode (#11025)", async () => {
+    const recover = vi.spyOn(dockerDriverRecovery, "recoverDockerDriverSandbox");
+    const getState = vi.fn().mockResolvedValue({ state: "missing", output: "not found" });
+
+    const result = await getReconciledSandboxGatewayState("beta", {
+      getState,
+      gatewayRecovery: "observe",
+    });
+
+    expect(recover).not.toHaveBeenCalled();
+    expect(result).toMatchObject({ state: "missing", output: "not found" });
+  });
+
   it("keeps receipt-owned observation scoped without changing global gateway selection (#9203)", async () => {
     const getState = vi.fn().mockResolvedValue({ state: "present", output: "Phase: Ready" });
 

src/lib/actions/sandbox/gateway-state.ts

@@ -929,6 +929,7 @@ export async function getReconciledSandboxGatewayState(
     return lookup;
   }
   if (lookup.state === "missing") {
+    if (gatewayRecovery === "observe") return lookup;
     return reconcileMissingAgainstNamedGateway(sandboxName, lookup, targetGatewayName);
   }
 

src/lib/actions/sandbox/rebuild-flow-lifecycle.test.ts

@@ -220,6 +220,7 @@ describe("rebuildSandbox flow: lifecycle", () => {
         hermesAuthMethod: null,
       }),
     );
+    expect(harness.registryUpdateSpy).toHaveBeenCalledWith("alpha", { stopped: false });
     const deleteCall = harness.runOpenshellSpy.mock.calls.findIndex(
       (call) =>
         Array.isArray(call[0]) &&

src/lib/actions/sandbox/rebuild-pipeline.ts

@@ -77,7 +77,10 @@ import {
   recordRebuildRecoveryBackup,
 } from "./rebuild-recreate-journal";
 import { runRebuildRecreatePhase } from "./rebuild-recreate-phase";
-import { createRebuildRegistryRollback } from "./rebuild-registry-rollback";
+import {
+  createRebuildRegistryRollback,
+  persistSandboxStopIntent,
+} from "./rebuild-registry-rollback";
 import { runRebuildRestorePhase } from "./rebuild-restore-phase";
 
 export { buildRefreshMutableOpenClawConfigHashCommand, stageMessagingManifestPlanForRebuild };
@@ -837,6 +840,11 @@ async function rebuildSandboxUnlocked(
         }
         retireRemovedImmutabilityStateRecord(sandboxName, "mutable-rebuild");
       }
+      if (!persistSandboxStopIntent(sandboxName, false)) {
+        return bail(
+          `Sandbox '${sandboxName}' was rebuilt, but NemoClaw could not clear its intentional-stop record. Run 'nemoclaw ${sandboxName} status' before another lifecycle command.`,
+        );
+      }
       if (backup.backupManifest) {
         if (!completePolicyHandoffCleanup(recreateJournal.id, backup.backupManifest)) return;
         if (!clearRecoveryMarker(recreateJournal.id, backup.backupManifest)) return;

src/lib/actions/sandbox/rebuild-registry-rollback.ts

@@ -16,6 +16,10 @@ export interface RebuildRegistryRollback {
   restoreForRetry(): void;
 }
 
+export function persistSandboxStopIntent(name: string, stopped: boolean): boolean {
+  return registry.recordSandboxStopIntent(name, stopped, registry.updateSandbox);
+}
+
 interface RebuildRegistryRollbackDeps {
   restoreSandboxEntry?: typeof registry.restoreSandboxEntry;
   restoreSandboxEntryIfMissing?: typeof registry.restoreSandboxEntryIfMissing;

src/lib/actions/sandbox/runtime/hermes-cron-restore-recovery.test.ts

@@ -102,7 +102,6 @@ describe("sandbox recovery with a Hermes cron restore gate", () => {
     await expect(recoverSandboxWithHermesCronRestore("alpha")).rejects.toThrow(
       "recovery authority is unsafe",
     );
-
     expect(mocks.connectSandbox).not.toHaveBeenCalled();
     expect(mocks.recoverHermesCronRestore).not.toHaveBeenCalled();
   });

src/lib/actions/sandbox/start.test.ts

@@ -32,7 +32,8 @@ const FAILED_RECOVERY = { ...SUCCESSFUL_RECOVERY, wasRunning: false } as const;
 const REDACTED_TOKEN = "opaque-token-8662";
 
 function harness(overrides: Partial<SandboxStartDeps> = {}) {
-  const getSandbox = vi.fn<NonNullable<SandboxStartDeps["getSandbox"]>>(() => sandbox());
+  let storedSandbox = sandbox({ stopped: true });
+  const getSandbox = vi.fn<NonNullable<SandboxStartDeps["getSandbox"]>>(() => storedSandbox);
   const isDockerRuntimeDown = vi.fn<DockerRuntimeProviderDependencies["isRuntimeDown"]>(
     () => false,
   );
@@ -73,6 +74,10 @@ function harness(overrides: Partial<SandboxStartDeps> = {}) {
     NonNullable<SandboxStartDeps["waitForManagedGatewaySupervisor"]>
   >(() => false);
   const log = vi.fn<(message: string) => void>();
+  const updateSandbox = vi.fn<NonNullable<SandboxStartDeps["updateSandbox"]>>((_name, updates) => {
+    storedSandbox = { ...storedSandbox, ...updates };
+    return true;
+  });
   const runtimeProviders = createRuntimeProviderBundleRegistry([
     [
       "docker",
@@ -94,6 +99,7 @@ function harness(overrides: Partial<SandboxStartDeps> = {}) {
     restoreStartupState,
     waitForManagedGatewaySupervisor,
     verifyGateway,
+    updateSandbox,
     log,
     withLifecycleLock: async (_sandboxName, operation) => operation(),
     ...overrides,
@@ -110,6 +116,7 @@ function harness(overrides: Partial<SandboxStartDeps> = {}) {
     recoverDockerDriverSandbox,
     recoverPortableSandbox,
     restoreStartupState,
+    updateSandbox,
     waitForManagedGatewaySupervisor,
     verifyGateway,
   };
@@ -244,6 +251,24 @@ describe("startSandbox", () => {
     );
   });
 
+  it("records stopped: false in the sandbox registry on successful start (#11025)", async () => {
+    const h = harness();
+
+    const result = await startSandbox("my-sandbox", h.deps);
+
+    expect(result.exitCode).toBe(0);
+    expect(h.updateSandbox).toHaveBeenCalledWith("my-sandbox", { stopped: false });
+    expect(h.getSandbox("my-sandbox")?.stopped).toBe(false);
+  });
+
+  it("reports a partial success when the running state cannot be recorded (#11025)", async () => {
+    const h = harness({ updateSandbox: vi.fn(() => false) });
+
+    await expect(startSandbox("my-sandbox", h.deps)).rejects.toThrow(
+      "started, but NemoClaw could not clear its intentional-stop record",
+    );
+  });
+
   it(
     "retries startup after a structured recovery failure (#8662)",
     testTimeoutOptions(30_000),
@@ -770,6 +795,7 @@ describe("startSandbox", () => {
     const result = await startSandbox("my-sandbox", h.deps);
 
     expect(result.exitCode).toBe(1);
+    expect(h.updateSandbox).toHaveBeenCalledWith("my-sandbox", { stopped: false });
     const output = h.log.mock.calls.map(([line]) => line).join("\n");
     expect(output).toContain("HTTP 401");
     expect(output).toContain("doctor");

src/lib/actions/sandbox/start.ts

@@ -71,6 +71,7 @@ export interface SandboxStartDeps {
   observer?: OpenShellSandboxObserver;
   environment?: NodeJS.ProcessEnv;
   getSandbox?: typeof registry.getSandbox;
+  updateSandbox?: typeof registry.updateSandbox;
   restoreProcessState?: (sandboxName: string) => SandboxStartupRecoveryResult;
   runtimeProviders?: RuntimeProviderBundleRegistry;
   restoreStartupState?: (
@@ -238,6 +239,17 @@ async function startSandboxWithinLifecycleFence(
     await (deps.verifyGateway ?? verifyGateway)(name);
     readiness.inference = checkStartedSandboxInference(name, resolved.sandbox, deps, log);
   });
+  if (
+    !registry.recordSandboxStopIntent(
+      sandboxName,
+      false,
+      deps.updateSandbox ?? registry.updateSandbox,
+    )
+  ) {
+    throw new Error(
+      `Sandbox '${sandboxName}' started, but NemoClaw could not clear its intentional-stop record. Run '${cliName()} ${sandboxName} status' before another lifecycle command.`,
+    );
+  }
   if (readiness.inference && !readiness.inference.ok) {
     log(`  The sandbox started but inference is not usable: ${readiness.inference.detail}.`);
     log(`  Run the sandbox doctor command for '${sandboxName}' to identify the failing hop.`);

src/lib/actions/sandbox/status-inference.test.ts

@@ -12,6 +12,8 @@ import {
 describe("sandbox status inference.local route health (#6192)", () => {
   function snapshotDeps(options: {
     agent?: string;
+    confirmedStopped?: boolean;
+    stopped?: boolean;
     lookupState?: "present" | "missing";
     provider?: string;
     liveProvider?: string;
@@ -35,19 +37,31 @@ describe("sandbox status inference.local route health (#6192)", () => {
       model: "nvidia/nemotron",
       provider,
       preferredInferenceApi: options.preferredInferenceApi,
+      ...(options.stopped !== undefined ? { stopped: options.stopped } : {}),
     };
     return {
       getSandbox: () => sandbox,
       listSandboxes: () => ({ sandboxes: [sandbox], defaultSandbox: "alpha" }),
       reconcile: async () =>
         options.lookupState === "missing"
           ? { state: "missing" as const, output: "sandbox alpha not found" }
-          : { state: "present" as const, output: "Name: alpha\nPhase: Ready\n" },
+          : {
+              state: "present" as const,
+              phase: "Ready",
+              output: "Name: alpha\nPhase: Ready\n",
+            },
       captureOpenshellForStatusImpl: async () =>
         ({
           status: 0,
           output: `Gateway inference:\n  Provider: ${options.liveProvider ?? provider}\n  Model: ${options.liveModel ?? "nvidia/nemotron"}\n`,
         }) as never,
+      getSandboxStatusPreflightImpl: vi.fn(async () => ({
+        failure: null,
+        failureLayer: null,
+        intentionalStopConfirmed: options.confirmedStopped === true,
+        suppressInferenceProbe: options.confirmedStopped === true,
+        exitCode: 0 as const,
+      })),
       probeProviderHealthImpl: vi.fn(
         options.providerProbeThrows
           ? () => {
@@ -123,6 +137,44 @@ describe("sandbox status inference.local route health (#6192)", () => {
     expect(report.servingProcessHealth).toBeNull();
   });
 
+  it("does not probe terminal runtime health when the sandbox is stopped (#11025)", async () => {
+    const deps = snapshotDeps({
+      agent: "langchain-deepagents-code",
+      confirmedStopped: true,
+      stopped: true,
+      routeHealth: {
+        ok: true,
+        endpoint: "https://inference.local/v1/models",
+        httpStatus: 200,
+        detail: "route reachable",
+      },
+    });
+
+    const snapshot = await collectSandboxStatusSnapshot("alpha", { deps });
+
+    expect(snapshot.terminalRuntimeHealth).toBeNull();
+    expect(deps.probeTerminalRuntimeHealth).not.toHaveBeenCalled();
+    expect(deps.probeProviderHealthImpl).not.toHaveBeenCalled();
+    expect(deps.probeSandboxInferenceGatewayHealthImpl).not.toHaveBeenCalled();
+  });
+
+  it("probes a live sandbox when its persisted stop marker is stale (#11025)", async () => {
+    const deps = snapshotDeps({
+      stopped: true,
+      routeHealth: {
+        ok: true,
+        endpoint: "https://inference.local/v1/models",
+        httpStatus: 200,
+        detail: "route reachable",
+      },
+    });
+
+    const report = await getSandboxStatusReport("alpha", deps);
+
+    expect(report.phase).toBe("Ready");
+    expect(deps.probeSandboxInferenceGatewayHealthImpl).toHaveBeenCalled();
+  });
+
   it("does not invent serving-process health when the gateway is unavailable (#7003)", async () => {
     const deps = snapshotDeps({
       lookupState: "missing",

src/lib/actions/sandbox/status-lookup-rendering.test.ts

@@ -92,6 +92,23 @@ describe("printNonReadySandboxPhaseGuidance (#7222)", () => {
     expect(text).not.toContain("rebuild --yes");
   });
 
+  it("renders Phase: Stopped and clean stopped guidance when phase is Stopped (#11025)", async () => {
+    const cap = captureConsoleLog();
+    await printGuidance({
+      phase: "Stopped",
+      dockerRuntime: null,
+    });
+    const text = cap.lines();
+    cap.restore();
+
+    expect(text).toContain("Phase: Stopped");
+    expect(text).toContain("Sandbox 'beta' is stopped.");
+    expect(text).toContain("Workspace state is preserved.");
+    expect(text).toContain("nemoclaw beta start");
+    expect(text).not.toContain("is stuck");
+    expect(text).not.toContain("rebuild --yes");
+  });
+
   it("keeps the unpause hint for a paused container and never suggests start/rebuild (#4495)", async () => {
     const cap = captureConsoleLog();
     await printGuidance({

src/lib/actions/sandbox/status-lookup-rendering.ts

@@ -133,7 +133,12 @@ function printPresentSandboxGatewayLookupStatus({
     );
     console.log("");
   }
-  console.log(lookup.output);
+  const isStopped = phase === "Stopped";
+  const renderedOutput =
+    isStopped && lookup.output
+      ? lookup.output.replace(/^(\s*Phase:\s*)\S+\s*$/gmu, "$1Stopped")
+      : lookup.output;
+  if (renderedOutput) console.log(renderedOutput);
   printNonReadySandboxPhaseGuidance({ sandboxName, phase, dockerRuntime });
 }
 
@@ -237,7 +242,10 @@ function printNonReadySandboxPhaseGuidance({
   dockerRuntime: ReturnType<typeof getSandboxDockerRuntime> | null;
 }): void {
   if (!phase || phase === "Ready") return;
-  if (dockerRuntime?.containerName && !dockerRuntime.running && !dockerRuntime.paused) {
+  if (
+    phase === "Stopped" ||
+    (dockerRuntime?.containerName && !dockerRuntime.running && !dockerRuntime.paused)
+  ) {
     console.log("");
     console.log(`  Sandbox '${sandboxName}' is stopped.`);
     console.log("  Workspace state is preserved.");