fix(sandbox): keep retained destroy fail closed

docs/reference/troubleshooting.mdx

@@ -1925,14 +1925,15 @@ Onboarding ends with:
 
 ```text
   Sandbox 'my-assistant' was created but did not become ready within 180s.
-  Retry: $$nemoclaw onboard
+  terminal_resolution: timed_out_retained
+  Recovery remains blocked while sandbox 'my-assistant' exists.
 ```
 
 This is a separate budget from `NEMOCLAW_LOCAL_INFERENCE_TIMEOUT`. It covers the readiness wait that follows sandbox creation, including in-sandbox boot, OpenClaw start, and policy load. It does not cover the inference probe.
 
 <AgentOnly variant="openclaw,hermes">
 
-For a newly created OpenClaw or Hermes sandbox, `Ready` is not the final acceptance signal. Within this same budget, NemoClaw also requires OpenShell to return a durable sandbox ID and accept `openshell sandbox exec --name <sandbox> -- true`. NemoClaw keeps waiting only when OpenShell returns its exact `sandbox is not ready` response. A missing or malformed ID, or another command failure, stops the wait. Ordinary onboarding then follows the failed-creation cleanup path. Portable OpenClaw onboarding preserves the sandbox as described below.
+For a newly created OpenClaw or Hermes sandbox, `Ready` is not the final acceptance signal. Within this same budget, NemoClaw also requires OpenShell to return a durable sandbox ID and accept `openshell sandbox exec --name <sandbox> -- true`. NemoClaw keeps waiting only when OpenShell returns its exact `sandbox is not ready` response. A missing or malformed ID, or another command failure, stops the wait and preserves the sandbox. Inspect the gateway, then run `$$nemoclaw <sandbox-name> destroy` to check whether authoritative absence permits cleanup. Do not retry same-name onboarding while the retained sandbox exists.
 
 </AgentOnly>
 
@@ -1949,7 +1950,7 @@ export NEMOCLAW_SANDBOX_READY_TIMEOUT=600
 $$nemoclaw onboard
 ```
 
-The variable accepts seconds and applies to the readiness wait only. When the ordinary create deadline expires, NemoClaw tries to delete the partially created sandbox. After successful cleanup, the output ends with `Retry: $$nemoclaw onboard`. If cleanup fails, NemoClaw instead reports that the failed sandbox could not be removed and prints `Manual cleanup: openshell sandbox delete "<name>"`.
+The variable accepts seconds and applies to the readiness wait only. When the ordinary create deadline expires, NemoClaw preserves the partially created sandbox because OpenShell deletion targets its mutable name. Same-name onboarding remains blocked while that sandbox exists. Run `$$nemoclaw <sandbox-name> destroy` to check retained recovery. Do not delete the sandbox by mutable name.
 
 <AgentOnly variant="openclaw">
 
@@ -2742,13 +2743,7 @@ Starting with NemoClaw v0.0.43, the standard installer handles the `/proc/<pid>/
 If an older release fails direct GPU proof with that path and `Permission denied`, upgrade NemoClaw and rerun onboarding.
 </Note>
 
-When inspection confirms that the failed sandbox remains, delete it with a command such as:
-
-```bash
-openshell sandbox delete <sandbox-name>
-```
-
-Fix the NVIDIA Container Toolkit or CDI configuration reported in the diagnostics. Run the deletion command only after confirming that the pre-patch sandbox was not restored, then rerun onboarding. If you do not need GPU access inside the sandbox, rerun with `--no-sandbox-gpu`.
+When the failed sandbox remains, recovery stays blocked because OpenShell deletion targets its mutable name. Inspection is diagnostic only and does not authorize deletion. Run `$$nemoclaw <sandbox-name> destroy` to check for authoritative absence. Fix the NVIDIA Container Toolkit or CDI configuration reported in the diagnostics. Rerun onboarding only after retained recovery completes. If you do not need GPU access inside the sandbox, use `--no-sandbox-gpu` when onboarding can continue.
 
 If sandbox creation fails with `CDI device injection failed: unresolvable CDI devices nvidia.com/gpu=all`, the OpenShell gateway tried `docker create --device nvidia.com/gpu=all` and Docker could not resolve the CDI spec. This injection happens inside the gateway, so `NEMOCLAW_DOCKER_GPU_PATCH=0` does not bypass it. Rerun with `--no-gpu`, or set `NEMOCLAW_SANDBOX_GPU=0` and resume onboarding.
 

src/lib/actions/sandbox/destroy-preflight.ts

@@ -24,7 +24,10 @@ import type {
 import type { SandboxEntry } from "../../state/registry";
 import * as registry from "../../state/registry";
 import { type DestroyRunOpenshell, selectGatewayForSandboxDestroy } from "./destroy-gateway";
-import { classifyDestroySandboxPresence } from "./destroy-presence";
+import {
+  classifyDestroySandboxPresence,
+  type DestroySandboxPresence,
+} from "./destroy-presence";
 import {
   getPersistedSandboxTargetGatewayName,
   getSandboxTargetGatewayName,
@@ -40,6 +43,7 @@ export type SandboxDestroyPreflight = {
   selectedRunOpenshell: DestroyRunOpenshell;
   sandbox: SandboxEntry | null;
   sandboxConfirmedAbsent: boolean;
+  sandboxPresence?: DestroySandboxPresence;
 };
 
 export function resolveSandboxDestroyRuntimeSelection(
@@ -350,13 +354,14 @@ export function prepareSandboxDestroy(
       timeout: OPENSHELL_PROBE_TIMEOUT_MS,
     }),
   );
-  const sandboxConfirmedAbsent = sandboxPresence === "absent";
+
   return {
     cleanupGatewayName,
     runOpenshell,
     selectedRunOpenshell,
     sandbox,
-    sandboxConfirmedAbsent,
+    sandboxConfirmedAbsent: sandboxPresence === "absent",
+    sandboxPresence,
     ...(selectedCaptureOpenshell ? { selectedCaptureOpenshell } : {}),
     ...(runtimeSelection ? { runtimeSelection } : {}),
   };

src/lib/actions/sandbox/destroy-retained-recovery-flow.test.ts

@@ -32,6 +32,14 @@ function retainedRecoveryRecord(sandboxId = "sb-alpha"): RetainedSandboxRecovery
   };
 }
 
+function retainedRecoveryRecordWithoutIdentity(): RetainedSandboxRecoveryRecord {
+  return {
+    ...retainedRecoveryRecord(),
+    sandboxIdentityFingerprint: null,
+    identityWasUnavailable: true,
+  };
+}
+
 describe("destroySandbox retained recovery flow", () => {
   let exitSpy: MockInstance;
   let originalGatewayEnv: string | undefined;
@@ -217,7 +225,7 @@ describe("destroySandbox retained recovery flow", () => {
       );
 
       expect(harness.errorSpy).toHaveBeenCalledWith(
-        expect.stringContaining("delete command accepts only the mutable sandbox name"),
+        expect.stringContaining("cannot bind a mutable-name delete to the retained record"),
       );
       expect(harness.runOpenshellSpy).not.toHaveBeenCalledWith(
         ["sandbox", "delete", "alpha"],
@@ -251,12 +259,169 @@ describe("destroySandbox retained recovery flow", () => {
       );
 
       expect(harness.errorSpy).toHaveBeenCalledWith(
-        expect.stringContaining("cannot bind that deletion to the retained immutable identity"),
+        expect.stringContaining("cannot bind a mutable-name delete to the retained record"),
+      );
+      expect(harness.runOpenshellSpy).not.toHaveBeenCalledWith(
+        ["sandbox", "delete", "alpha"],
+        expect.anything(),
+      );
+      expect(harness.resolveRetainedSandboxRecoverySpy).not.toHaveBeenCalled();
+      expect(harness.removeSandboxSpy).not.toHaveBeenCalled();
+    },
+  );
+
+  it(
+    "still fails closed for a live retained sandbox even with a proven-matching OpenShell identity (#10863)",
+    { timeout: 30_000 },
+    async () => {
+      // OpenShell exposes no atomic delete-by-identity primitive, so no
+      // amount of identity proof inside NemoClaw can close the window where
+      // another OpenShell client replaces the sandbox under the same name
+      // between the last read and OpenShell processing the delete. Automatic
+      // deletion of a live retained sandbox is therefore always fail-closed,
+      // even when the live OpenShell id matches the retained record.
+      const recovery = retainedRecoveryRecord("sandbox-alpha");
+      const containerId = "a".repeat(64);
+      const harness = createDestroyHarness({
+        dockerRunResult: {
+          status: 0,
+          stdout: `${containerId}\topenshell\tdefault\tsandbox-alpha`,
+        },
+        registryEntryOverrides: {
+          lifecycleGeneration: recovery.lifecycleGeneration!,
+          lifecycleLiveIdentityFingerprint: recovery.sandboxIdentityFingerprint!,
+        },
+        retainedRecoveryRecords: [recovery],
+      });
+
+      await expect(harness.destroySandbox("alpha", { yes: true })).rejects.toThrow(
+        "process.exit(1)",
+      );
+
+      expect(harness.errorSpy).toHaveBeenCalledWith(
+        expect.stringContaining("cannot bind a mutable-name delete to the retained record"),
+      );
+      expect(harness.errorSpy.mock.calls.flat().join("\n")).not.toContain(
+        "openshell sandbox delete",
+      );
+      expect(harness.errorSpy).toHaveBeenCalledWith(
+        expect.stringContaining(`openshell sandbox list -g ${recovery.gatewayName} -o json`),
+      );
+      expect(harness.runOpenshellSpy).not.toHaveBeenCalledWith(
+        ["sandbox", "delete", "alpha"],
+        expect.anything(),
+      );
+      expect(harness.resolveRetainedSandboxRecoverySpy).not.toHaveBeenCalled();
+    },
+  );
+
+  it(
+    "reconciles a lone recovery record without identity after OpenShell confirms absence (#10863)",
+    { timeout: 30_000 },
+    async () => {
+      const recovery = retainedRecoveryRecordWithoutIdentity();
+      const harness = createDestroyHarness({
+        sandboxPresent: false,
+        dockerRunResult: { status: 0, stdout: "" },
+        registryEntryOverrides: {
+          lifecycleGeneration: recovery.lifecycleGeneration!,
+        },
+        retainedRecoveryRecords: [recovery],
+      });
+
+      await expect(harness.destroySandbox("alpha", { yes: true })).resolves.toBeUndefined();
+
+      expect(harness.runOpenshellSpy).not.toHaveBeenCalledWith(
+        ["sandbox", "delete", "alpha"],
+        expect.anything(),
       );
+      expect(harness.resolveRetainedSandboxRecoverySpy).toHaveBeenCalledWith(recovery);
+      expect(harness.removeSandboxSpy).toHaveBeenCalledWith("alpha");
+      expect(exitSpy).not.toHaveBeenCalled();
+    },
+  );
+
+  it.each([
+    ["present", undefined, "reports a sandbox present"],
+    [
+      "a failed list command",
+      { status: 1, stdout: "", stderr: "" },
+      "could not determine",
+    ],
+    [
+      "list diagnostics",
+      { status: 0, stdout: "[]", stderr: "gateway unavailable" },
+      "could not determine",
+    ],
+    [
+      "malformed list output",
+      { status: 0, stdout: "not-json", stderr: "" },
+      "could not determine",
+    ],
+  ])(
+    "preserves a recovery record without identity when OpenShell reports %s (#10863)",
+    { timeout: 30_000 },
+    async (_case, sandboxListResult, expectedPresence) => {
+      const recovery = retainedRecoveryRecordWithoutIdentity();
+      const harness = createDestroyHarness({
+        dockerRunResult: { status: 0, stdout: "" },
+        registryEntryOverrides: {
+          lifecycleGeneration: recovery.lifecycleGeneration!,
+        },
+        retainedRecoveryRecords: [recovery],
+        ...(sandboxListResult ? { sandboxListResult } : {}),
+      });
+
+      await expect(harness.destroySandbox("alpha", { yes: true })).rejects.toThrow(
+        "process.exit(1)",
+      );
+
       expect(harness.runOpenshellSpy).not.toHaveBeenCalledWith(
         ["sandbox", "delete", "alpha"],
         expect.anything(),
       );
+      expect(harness.dockerRunSpy).not.toHaveBeenCalledWith(
+        ["rm", "-f", expect.any(String)],
+        expect.anything(),
+      );
+      expect(harness.resolveRetainedSandboxRecoverySpy).not.toHaveBeenCalled();
+      expect(harness.removeSandboxSpy).not.toHaveBeenCalled();
+      const output = harness.errorSpy.mock.calls.flat().join("\n");
+      expect(output).not.toContain("openshell sandbox delete");
+      expect(output).toContain("Preserve");
+      expect(output).toContain(expectedPresence);
+    },
+  );
+
+  it(
+    "preserves a recovery record without identity when a residual container exists (#10863)",
+    { timeout: 30_000 },
+    async () => {
+      const recovery = retainedRecoveryRecordWithoutIdentity();
+      const containerId = "a".repeat(64);
+      const harness = createDestroyHarness({
+        sandboxPresent: false,
+        dockerRunResult: {
+          status: 0,
+          stdout: `${containerId}\topenshell\tdefault\tsb-alpha`,
+        },
+        registryEntryOverrides: {
+          lifecycleGeneration: recovery.lifecycleGeneration!,
+        },
+        retainedRecoveryRecords: [recovery],
+      });
+
+      await expect(harness.destroySandbox("alpha", { yes: true })).rejects.toThrow(
+        "process.exit(1)",
+      );
+
+      expect(harness.errorSpy).toHaveBeenCalledWith(
+        expect.stringContaining("has no durable sandbox identity"),
+      );
+      expect(harness.dockerRunSpy).not.toHaveBeenCalledWith(
+        ["rm", "-f", containerId],
+        expect.anything(),
+      );
       expect(harness.resolveRetainedSandboxRecoverySpy).not.toHaveBeenCalled();
       expect(harness.removeSandboxSpy).not.toHaveBeenCalled();
     },

src/lib/actions/sandbox/destroy.ts

@@ -100,9 +100,7 @@ function selectRetainedSandboxRecoveryAuthority(
   sandbox: registry.SandboxEntry | null,
   records: readonly onboardSession.RetainedSandboxRecoveryRecord[],
 ): onboardSession.RetainedSandboxRecoveryRecord | null {
-  const candidates = records.filter(
-    (record) => record.sandboxName === sandboxName && record.sandboxIdentityFingerprint !== null,
-  );
+  const candidates = records.filter((record) => record.sandboxName === sandboxName);
   if (!sandbox) {
     // Once resource cleanup has removed the registry row, a retry must still
     // select the lone durable record so the later Docker proof can confirm
@@ -112,10 +110,11 @@ function selectRetainedSandboxRecoveryAuthority(
     if (candidates.length === 0) return null;
     const observation = observeDestroyContainerIdentity(sandboxName);
     const observedMatches = candidates.filter((record) => {
+      if (record.sandboxIdentityFingerprint === null) return false;
       const verdict = classifyDestroyContainerIdentity(
         sandboxName,
         observation,
-        record.sandboxIdentityFingerprint!,
+        record.sandboxIdentityFingerprint,
       );
       return (
         verdict.status === "recovery" || (verdict.status === "clear" && verdict.identity !== null)
@@ -133,7 +132,8 @@ function selectRetainedSandboxRecoveryAuthority(
         record.gatewayName === pending.gatewayName &&
         record.gatewayPort === pending.gatewayPort &&
         record.lifecycleGeneration === pending.lifecycleGeneration &&
-        record.sandboxIdentityFingerprint === pending.sandboxIdentityFingerprint &&
+        (record.sandboxIdentityFingerprint === null ||
+          record.sandboxIdentityFingerprint === pending.sandboxIdentityFingerprint) &&
         (pending.createAttemptNonce === undefined ||
           record.createAttemptNonce === pending.createAttemptNonce)
       );
@@ -142,7 +142,8 @@ function selectRetainedSandboxRecoveryAuthority(
       record.gatewayName === sandbox.gatewayName &&
       record.gatewayPort === sandbox.gatewayPort &&
       record.lifecycleGeneration === sandbox.lifecycleGeneration &&
-      record.sandboxIdentityFingerprint === sandbox.lifecycleLiveIdentityFingerprint
+      (record.sandboxIdentityFingerprint === null ||
+        record.sandboxIdentityFingerprint === sandbox.lifecycleLiveIdentityFingerprint)
     );
   };
   const matching = candidates.filter(matchesRegistryAuthority);
@@ -675,6 +676,16 @@ async function destroySandboxUnlocked(
   if (initialIdentity === false) {
     requestSandboxDestroyExit(1);
   }
+  if (
+    retainedRecoveryAuthority?.sandboxIdentityFingerprint === null &&
+    initialIdentity?.identities !== undefined &&
+    initialIdentity.identities.length > 0
+  ) {
+    console.error(
+      `  Refusing to destroy retained sandbox '${sandboxName}': the recovery record has no durable sandbox identity, so NemoClaw cannot qualify a residual container for deletion. No sandbox resources were removed. Preserve the recovery record and resolve the container identity conflict before retrying.`,
+    );
+    requestSandboxDestroyExit(1);
+  }
   const initialContainerIdentities = initialIdentity?.identities;
 
   let preparedManagedLlamaCppCleanup: ReturnType<
@@ -743,10 +754,25 @@ async function destroySandboxUnlocked(
     selectedRunOpenshell: cleanupRunOpenshell,
     sandbox,
     sandboxConfirmedAbsent,
+    sandboxPresence = sandboxConfirmedAbsent ? "absent" : "unknown",
   } = destroyPreflight;
-  if (retainedRecoveryAuthority && !sandboxConfirmedAbsent) {
+  if (retainedRecoveryAuthority && sandboxPresence !== "absent") {
+    // OpenShell has no atomic delete-by-identity primitive: it exposes no
+    // way to bind a mutable-name delete to the retained record's immutable
+    // sandbox id/resource version. Even a fresh identity read immediately
+    // before the delete command cannot close the window where another
+    // OpenShell client removes the retained sandbox and creates a
+    // replacement under the same name between that read and OpenShell
+    // processing the delete (#10863). Automatic deletion of a live retained
+    // sandbox is therefore always fail-closed. Inspection cannot authorize a
+    // later mutable-name delete, so the recovery record remains unresolved
+    // until OpenShell can prove absence through the owning gateway.
+    const presenceDetail =
+      sandboxPresence === "present"
+        ? "OpenShell reports a sandbox present under this name."
+        : "OpenShell could not determine whether a sandbox is present under this name.";
     console.error(
-      `  Refusing to automatically delete retained sandbox '${sandboxName}': OpenShell still reports it present, but its delete command accepts only the mutable sandbox name. NemoClaw cannot bind that deletion to the retained immutable identity. No sandbox resources were removed. Ask an OpenShell administrator to resolve create-attempt label '${retainedRecoveryAuthority.createAttemptNonce}' to the exact sandbox and use an identity-bound removal procedure. After OpenShell confirms the retained sandbox is absent, rerun '${CLI_NAME} ${sandboxName} destroy --yes' to reconcile its verified Docker containers and recovery record.`,
+      `  Refusing to delete retained sandbox '${sandboxName}': ${presenceDetail} NemoClaw cannot bind a mutable-name delete to the retained record (create-attempt label '${retainedRecoveryAuthority.createAttemptNonce}') without an atomic OpenShell delete-by-identity primitive. No sandbox resources were removed. Preserve the recovery record. Inspect 'openshell sandbox list -g ${retainedRecoveryAuthority.gatewayName} -o json' for diagnosis only; do not run mutable-name deletion. Recovery remains blocked until the owning gateway reports the sandbox absent. Then rerun '${CLI_NAME} ${sandboxName} destroy --yes' to reconcile verified residual resources and the recovery record.`,
     );
     preparedManagedLlamaCppCleanup?.abort();
     requestSandboxDestroyExit(1);

src/lib/actions/sandbox/rebuild-preflight-confirmation.test.ts

@@ -234,7 +234,10 @@ describe("rebuild preflight guards", () => {
       listRecovery.mock.invocationCallOrder[0]!,
     );
     expect(bail).toHaveBeenCalledWith("Retained sandbox recovery blocks rebuild for 'alpha'.", 1);
-    expect(error.mock.calls.flat().join("\n")).toContain("nemoclaw alpha destroy --yes");
+    const output = error.mock.calls.flat().join("\n");
+    expect(output).toContain("nemoclaw alpha destroy --yes");
+    expect(output).toContain("reports the sandbox present or cannot determine presence");
+    expect(output).not.toContain("administrator");
   });
 
   it("stops after a failed onboard-lock acquisition without releasing another run's lock (#7794)", () => {