fix(onboard): recover retained sandbox cleanup (#10547)

docs/reference/commands.mdx

@@ -452,18 +452,32 @@ For a new or fresh session, `--yes` and `NEMOCLAW_YES=1` accept supported confir
 If onboarding returns without reaching the final `complete` state, the command exits with status `1`.
 When that result is resumable, NemoClaw keeps the session `in_progress` at its last checkpoint instead of marking it failed, so correct the reported condition and run `$$nemoclaw onboard --resume`.
 
-If onboarding cannot complete after sandbox creation, NemoClaw preserves the sandbox and records its create-attempt label.
+#### Recover a retained sandbox
+
+If onboarding cannot complete after sandbox creation, NemoClaw preserves the sandbox.
+When available, NemoClaw records and prints the create-attempt label as the exact `ai.nvidia.nemoclaw.create-attempt=<value>` selector.
 When available, NemoClaw also records a durable identity fingerprint and verified policy evidence for recovery.
 Automatic and explicit resume, reuse, recreation, and fresh onboarding with that sandbox name remain blocked.
-Do not destroy the retained sandbox by name.
-Ask an OpenShell administrator to verify the exact live durable ID before using an identity-bound removal procedure.
-NemoClaw does not provide an operation to clear the recovery record, even after external removal.
-Start another onboarding run with a different name:
+When the recovery record contains a durable identity fingerprint, run `$$nemoclaw <sandbox-name> destroy` to attempt identity-bound recovery.
+
+Use the result from `destroy` to choose the next action:
+
+- If OpenShell still reports the sandbox present, `destroy` preserves the record and removes no resources. Do not delete the sandbox manually by mutable name. Give the displayed create-attempt label to an OpenShell administrator when present and ask them to remove that exact sandbox through an identity-bound procedure. Without a label, preserve the terminal output and ask the administrator to identify the exact sandbox from gateway or controller evidence.
+- After the administrator removes the exact sandbox, rerun `$$nemoclaw <sandbox-name> destroy`.
+- If OpenShell confirms the sandbox is absent, `destroy` verifies one retained recovery record and its immutable runtime identity. For Docker-backed sandboxes, it also verifies the immutable Docker sandbox identity and removes only qualified residual containers. It clears the matching recovery record only after verified cleanup.
+- If the recovery record is ambiguous or immutable runtime-identity verification fails, cleanup stops and the record remains. For Docker-backed sandboxes, a foreign container, changed Docker identity, or failed Docker probe has the same fail-closed result.
+
+If OpenShell did not return a durable identity fingerprint, `destroy` cannot complete recovery. A create-attempt label can help an OpenShell administrator identify and remove the exact sandbox, but it does not let NemoClaw retire the record without immutable identity authority. If neither a fingerprint nor a label is available, preserve the terminal output and ask the administrator to identify the exact sandbox from gateway or controller evidence. If NemoClaw reports that it could not save recovery evidence, preserve the terminal output and follow the same escalation.
+
+This fail-closed record keeps only the affected sandbox name unavailable. It is not retired from mutable-name absence alone, and this command does not accept administrator-supplied identity authority.
+To onboard another sandbox while the record remains unresolved, supply a different explicit name:
 
 ```bash
-$$nemoclaw onboard --fresh --name <new-sandbox-name>
+$$nemoclaw onboard --name <new-sandbox-name>
 ```
 
+`--fresh` alone does not clear the recovery record or permit reuse of the retained sandbox name.
+
 <AgentOnly variant="openclaw">
 
 OpenClaw sessions also record the web search selection, messaging selection and non-secret settings, and resource profile.
@@ -530,12 +544,9 @@ $$nemoclaw onboard --fresh --apf-interceptor --name my-apf-sandbox
 
 <Warning title="Recover an Incomplete APF Creation">
 If post-create verification or native GPU fallback fails after OpenShell may have created the sandbox, NemoClaw preserves the incomplete sandbox because automatic deletion would use its mutable name.
-Do not destroy that sandbox by name.
-Retain the reported sandbox name, create-attempt label, and durable identity fingerprint for comparison only.
-If OpenShell did not return the fingerprint, recovery remains blocked until an administrator resolves the create-attempt label to one exact sandbox.
-Ask an OpenShell administrator to obtain the exact live durable ID, verify it against the fingerprint, and use an identity-bound removal procedure.
+Follow the [retained-sandbox recovery procedure](#recover-a-retained-sandbox); its OpenShell-absence, immutable runtime-identity, and administrator requirements also apply to APF creation. For Docker-backed APF creation, its Docker container-identity requirement also applies.
 This onboarding mode does not support `--resume` or `--recreate-sandbox`, regardless of whether sandbox creation began.
-After the administrator confirms identity-bound removal, repeat the original command with `--fresh` and a new name.
+After destroy completes, repeat the original command with `--fresh`.
 </Warning>
 
 </AgentOnly>
@@ -938,18 +949,8 @@ Pairing and `TELEGRAM_ALLOWED_IDS` still govern direct messages.
 If you cancel a brand-new onboarding run at the policy-tier selector or either policy-preset selector after sandbox creation, NemoClaw preserves the incomplete sandbox, registry entry, and onboarding session for identity-bound recovery.
 NemoClaw reports the durable sandbox identity fingerprint when it is available.
 It does not run OpenShell's mutable-name deletion command because the name may now identify a replacement sandbox.
-Do not delete the sandbox by mutable name.
-Shared inference providers are gateway configuration, not sandbox cleanup targets.
-Sandbox-scoped provider registrations or gateway-bound credentials may remain when the durable recovery record lists them.
-Ask an OpenShell administrator to inspect the exact sandbox identity and remove only sandbox-scoped resources whose ownership is confirmed for the retained sandbox.
-A credential environment name in the recovery record does not prove that its value was exposed.
-Rotate a credential only when identity-bound inspection proves that it was exposed or attached to a retained sandbox-scoped resource.
-NemoClaw stores the recovery record independently from the active onboarding session.
-A fresh run with a different name can proceed without clearing that record, but automatic resume, explicit `--resume`, reuse, recreation, and fresh onboarding with the retained name remain blocked.
-NemoClaw has no supported operation in this release to clear the recovery record, so the retained name remains unavailable even after external recovery or removal.
-Preserve the record as evidence.
-Start fresh onboarding with `$$nemoclaw onboard --fresh --name <new-sandbox-name>`.
-Select the required provider, model, agent, policy, and environment inputs again because `--fresh` does not retain them.
+Follow the [retained-sandbox recovery procedure](#recover-a-retained-sandbox) to reconcile this cancellation.
+A fresh run with a different explicit name can continue while the cancelled sandbox name remains blocked.
 
 If you run onboarding again with the same sandbox name and choose a different inference provider or model, NemoClaw detects the drift and recreates the sandbox so the running agent config matches your selection.
 In interactive mode, the wizard asks for confirmation before delete and recreate.
@@ -2659,7 +2660,8 @@ For one matching container, the command continues only when all these labels hav
 - A nonempty `openshell.ai/sandbox-workspace`
 - A nonempty `openshell.ai/sandbox-id`
 
-If the initial inspection cannot complete, more than one container matches, a matching container has conflicting or incomplete labels, or Docker returns malformed identity data, `destroy` exits before changing sandbox resources.
+In an ordinary destroy flow, if the initial inspection cannot complete, more than one container matches, a matching container has conflicting or incomplete labels, or Docker returns malformed identity data, `destroy` exits before changing sandbox resources.
+Retained-sandbox recovery accepts multiple managed containers only when every immutable sandbox ID matches the retained recovery fingerprint.
 The identity checks still apply with `--force`, `--yes`, or `NEMOCLAW_NON_INTERACTIVE=1`; those controls authorize confirmation but do not authorize an unproven container identity.
 NemoClaw rechecks the identity after read-only preflight, before provider cleanup, and synchronously at the sandbox-deletion boundary.
 If a later recheck detects drift or fails, `destroy` refuses sandbox deletion, restores managed MCP preparation when possible, preserves local ownership state, and reports any earlier cleanup already performed.

src/lib/actions/sandbox/destroy-container-identity.test.ts

@@ -3,6 +3,7 @@
 
 import { describe, expect, it, vi } from "vitest";
 
+import { fingerprintOpenShellSandboxId } from "../../adapters/openshell/sandbox-identity";
 import {
   classifyDestroyContainerIdentity,
   type DestroyContainerIdentityVerdict,
@@ -126,6 +127,35 @@ describe("classifyDestroyContainerIdentity", () => {
     expect(verdict.reason).toContain("2 managed containers");
   });
 
+  it("accepts every managed container bound to one retained sandbox identity (#10547)", () => {
+    const sandboxIdentityFingerprint = fingerprintOpenShellSandboxId(MANAGED.sandboxId)!;
+    const identities = [MANAGED, { ...MANAGED, id: "dddd000000000000" }];
+
+    expect(
+      classifyDestroyContainerIdentity(
+        "destroytest",
+        observeRows(identities),
+        sandboxIdentityFingerprint,
+      ),
+    ).toEqual({ status: "recovery", identities });
+  });
+
+  it("refuses a retained recovery set that contains another sandbox identity (#10547)", () => {
+    const sandboxIdentityFingerprint = fingerprintOpenShellSandboxId(MANAGED.sandboxId)!;
+    const verdict = expectAmbiguous(
+      classifyDestroyContainerIdentity(
+        "destroytest",
+        observeRows([
+          MANAGED,
+          { ...MANAGED, id: "dddd000000000000", sandboxId: "sb-replacement" },
+        ]),
+        sandboxIdentityFingerprint,
+      ),
+    );
+
+    expect(verdict.reason).toContain("retained sandbox identity");
+  });
+
   it.each([
     ["workspace", { ...MANAGED, workspace: "" }],
     ["sandbox ID", { ...MANAGED, sandboxId: "" }],

src/lib/actions/sandbox/destroy-execution.ts

@@ -33,11 +33,12 @@ import { readTimerMarker } from "../../shields/timer-control";
 import type { SandboxEntry } from "../../state/registry";
 import {
   classifyDestroyContainerIdentity,
-  isSameDestroyContainerIdentity,
+  isSameDestroyContainerIdentityProof,
   observeDestroyContainerIdentity,
-  removeExactDestroyContainerIdentity,
+  type DestroyContainerIdentityProof,
   type SandboxNameLabeledContainer,
 } from "./destroy-presence";
+import { removeExactOpenShellDockerSandboxContainers } from "../../onboard/openshell-docker-sandbox-containers";
 import { type DestroyRunOpenshell, SANDBOX_DESTROY_TIMEOUT_MS } from "./destroy-gateway";
 import {
   finalizeMcpBridgesAfterSandboxDelete,
@@ -69,10 +70,11 @@ type SandboxDestroyExecutionInput = {
   sandbox: SandboxEntry | null;
   sandboxConfirmedAbsent: boolean;
   sandboxName: string;
-  // `undefined` delegates identity gating to the runtime provider.
-  // `null` records confirmed absence; an object records the one managed
-  // container observed by the pre-destroy guard.
-  expectedContainerIdentity?: SandboxNameLabeledContainer | null;
+  // `undefined` delegates identity gating to the runtime provider. An empty
+  // array records confirmed absence; other arrays contain the immutable
+  // Docker IDs qualified before destroy preparation.
+  expectedContainerIdentities?: readonly SandboxNameLabeledContainer[];
+  expectedContainerIdentityFingerprint?: string;
   portableContainerAuthority?: PreparedPortableDemoSandboxDestroyAuthority;
   stopInferenceResources: () => void;
   runtimeProviders?: RuntimeProviderBundleRegistry;
@@ -299,7 +301,8 @@ export async function executeSandboxDestroy({
   sandbox,
   sandboxConfirmedAbsent,
   sandboxName,
-  expectedContainerIdentity,
+  expectedContainerIdentities,
+  expectedContainerIdentityFingerprint,
   portableContainerAuthority,
   stopInferenceResources,
   runtimeProviders = CURRENT_RUNTIME_PROVIDER_BUNDLES,
@@ -312,6 +315,17 @@ export async function executeSandboxDestroy({
       | { status: "ambiguous"; detail: string; subject?: string }
       | { status: "probe-failed"; detail: string; subject?: string };
     const pendingPolicyVerification = sandbox?.pendingPolicyVerification;
+    const expectedContainerProof: DestroyContainerIdentityProof =
+      expectedContainerIdentities === undefined ? {} : { identities: expectedContainerIdentities };
+    const proofFromVerdict = (
+      verdict: ReturnType<typeof classifyDestroyContainerIdentity>,
+    ): DestroyContainerIdentityProof | null => {
+      if (verdict.status === "clear") {
+        return { identities: verdict.identity === null ? [] : [verdict.identity] };
+      }
+      if (verdict.status === "recovery") return { identities: verdict.identities };
+      return null;
+    };
     const inspectPendingPolicyVerificationContinuity = (): IdentityContinuity => {
       if (!pendingPolicyVerification) return { status: "match" };
       if (!getSandbox) {
@@ -326,6 +340,16 @@ export async function executeSandboxDestroy({
         if (!isDeepStrictEqual(readCurrentCheckpoint(), pendingPolicyVerification)) {
           return { status: "changed", subject: "Pending policy verification authority" };
         }
+        if (
+          sandboxConfirmedAbsent &&
+          expectedContainerIdentities !== undefined &&
+          expectedContainerIdentityFingerprint ===
+            pendingPolicyVerification.sandboxIdentityFingerprint
+        ) {
+          return isDeepStrictEqual(readCurrentCheckpoint(), pendingPolicyVerification)
+            ? { status: "match" }
+            : { status: "changed", subject: "Pending policy verification authority" };
+        }
         const inspectIdentity =
           deps.inspectOpenShellSandboxIdentityFingerprint ??
           inspectOpenShellSandboxIdentityFingerprint;
@@ -362,12 +386,17 @@ export async function executeSandboxDestroy({
           return { status: "probe-failed", detail: redactDestroyError(error) };
         }
       }
-      if (expectedContainerIdentity === undefined) return { status: "match" };
+      if (expectedContainerIdentities === undefined) return { status: "match" };
       const verdict = classifyDestroyContainerIdentity(
         sandboxName,
         observeDestroyContainerIdentity(sandboxName),
+        expectedContainerIdentityFingerprint,
       );
-      if (isSameDestroyContainerIdentity(expectedContainerIdentity, verdict)) {
+      const actualContainerProof = proofFromVerdict(verdict);
+      if (
+        actualContainerProof &&
+        isSameDestroyContainerIdentityProof(expectedContainerProof, actualContainerProof)
+      ) {
         return { status: "match" };
       }
       if (verdict.status === "probe-failed") {
@@ -510,14 +539,14 @@ export async function executeSandboxDestroy({
         " Managed inference cleanup may already be partial; inspect or restart its resources before retrying.",
       );
     }
-    // `expectedContainerIdentity === null` is a completed Docker identity
+    // An empty `expectedContainerIdentities` is a completed Docker identity
     // probe with zero matching containers. `undefined` means this runtime
     // does not use that probe (or Portable owns identity); skip hardening
     // only when OpenShell already proved absence. A live labeled Docker
     // identity still hardens even if the OpenShell list says absent.
     const sandboxRuntimeConfirmedAbsent =
-      expectedContainerIdentity === null ||
-      (expectedContainerIdentity === undefined && sandboxConfirmedAbsent);
+      expectedContainerIdentities?.length === 0 ||
+      (expectedContainerIdentities === undefined && sandboxConfirmedAbsent);
     let hardened: HardenedDeleteState;
     try {
       hardened = wipeAndHardenLiveSandbox(
@@ -585,18 +614,24 @@ export async function executeSandboxDestroy({
     const deleteArgs = pendingPolicyVerification
       ? ["sandbox", "delete", "-g", pendingPolicyVerification.gatewayName, sandboxName]
       : ["sandbox", "delete", sandboxName];
-    const deleteResult = runOpenshell(deleteArgs, {
-      ignoreError: true,
-      killSignal: "SIGKILL",
-      stdio: ["ignore", "pipe", "pipe"],
-      timeout: SANDBOX_DESTROY_TIMEOUT_MS,
-    });
+    // A successful preflight absence is already the required OpenShell
+    // lifecycle proof. Do not issue a later mutable-name delete that could
+    // target a same-name replacement created after that observation.
+    const deleteResult: ReturnType<DestroyRunOpenshell> = sandboxConfirmedAbsent
+      ? { status: 0, stdout: "", stderr: "" }
+      : runOpenshell(deleteArgs, {
+          ignoreError: true,
+          killSignal: "SIGKILL",
+          stdio: ["ignore", "pipe", "pipe"],
+          timeout: SANDBOX_DESTROY_TIMEOUT_MS,
+        });
     const {
       output: capturedDeleteOutput,
-      alreadyGone,
+      alreadyGone: deleteReportedAlreadyGone,
       gatewayUnreachable,
       timedOut,
     } = getSandboxDeleteOutcome(deleteResult);
+    const alreadyGone = sandboxConfirmedAbsent || deleteReportedAlreadyGone;
     const deleteOutput = timedOut
       ? `OpenShell sandbox delete timed out after ${String(SANDBOX_DESTROY_TIMEOUT_MS / 1000)} seconds. Deletion could not be confirmed.`
       : capturedDeleteOutput;
@@ -638,12 +673,19 @@ export async function executeSandboxDestroy({
       };
     }
 
-    if (!forcedLocalCleanup && (portableContainerAuthority || expectedContainerIdentity)) {
+    if (
+      !forcedLocalCleanup &&
+      (portableContainerAuthority || expectedContainerIdentities !== undefined)
+    ) {
       try {
         if (portableContainerAuthority) {
           portableContainerAuthority.verifyAbsent();
-        } else if (expectedContainerIdentity) {
-          removeExactDestroyContainerIdentity(sandboxName, expectedContainerIdentity, console.log);
+        } else if (expectedContainerIdentities !== undefined) {
+          removeExactOpenShellDockerSandboxContainers(
+            sandboxName,
+            expectedContainerIdentities.map(({ id }) => id),
+            console.log,
+          );
         }
       } catch (error) {
         const detail = redactDestroyError(error);

src/lib/actions/sandbox/destroy-flow.test.ts

@@ -121,6 +121,7 @@ describe("destroySandbox flow", () => {
         switch (`${String(argv[0])}:${String(argv[1])}`) {
           case "sandbox:delete":
             trace.push("delete");
+            harness.setSandboxPresent(false);
             return { status: 0, stdout: "", stderr: "" };
           case "sandbox:list":
             trace.push("list");
@@ -177,6 +178,7 @@ describe("destroySandbox flow", () => {
           switch (`${String(argv[0])}:${String(argv[1])}`) {
             case "sandbox:delete":
               crossedDeleteBoundary = true;
+              harness.setSandboxPresent(false);
               return { status: 0, stdout: "", stderr: "" };
             case "sandbox:list":
               return {
@@ -992,7 +994,13 @@ describe("destroySandbox flow", () => {
 
     await expect(harness.destroySandbox("alpha", { yes: true })).resolves.toBeUndefined();
 
-    expect(trace.slice(-2)).toEqual([`probe:${String(identityProbeCalls)}`, "delete"]);
+    const deleteIndex = trace.indexOf("delete");
+    expect(deleteIndex).toBeGreaterThan(0);
+    expect(trace[deleteIndex - 1]).toMatch(/^probe:/u);
+    expect(trace.slice(deleteIndex + 1)).toEqual([
+      `probe:${String(identityProbeCalls - 1)}`,
+      `probe:${String(identityProbeCalls)}`,
+    ]);
   });
 
   it("preserves provider and registry ownership when runtime authority is unknown", async () => {