From e448e6e4c55c2a48c7278053871c55a9f7240afd Mon Sep 17 00:00:00 2001 From: Yanyun Liao Date: Wed, 2 Sep 2026 16:49:04 +0800 Subject: [PATCH 01/12] fix(sandbox): prove live identity before refusing retained destroy Interrupting onboarding after sandbox creation (e.g. Ctrl-C at the Policy tier or preset selector) leaves a retained sandbox. destroy then refused to remove it whenever OpenShell still reported it present, unconditionally, regardless of whether the live sandbox was actually the exact retained one. Its remediation told the user to "ask an OpenShell administrator" for an "identity-bound removal procedure" that does not exist in the OpenShell CLI -- the only delete verb is `openshell sandbox delete `, exactly what an ordinary user can (and, per the report, successfully did) run themselves. destroy-preflight.ts already read the live sandbox's exact OpenShell id from `sandbox list -o json` but discarded it once presence was classified present/absent. Since a retained recovery record already carries a durable identity fingerprint of the exact sandbox it is protecting (the same mechanism used for Hermes Portable lifecycle verification), destroy can compare the two: fingerprint the live sandbox's id and check it against the recovery record's fingerprint before deciding to refuse. When they match, destroy has proven the live sandbox is the retained one, not a replacement that reused the name, and proceeds through the same delete path used for any other present sandbox -- the same outcome the manual `openshell sandbox delete` workaround produces. When they don't match (or the live id can't be read), destroy still refuses and removes nothing, but its message now names the exact `openshell sandbox delete ` command as something the user can run themselves after manually confirming identity, instead of pointing at a nonexistent administrator role. Reproduced on our Ubuntu 24.04 x86_64 test host: interrupting an interactive onboard's Policy tier selection with Ctrl-C after sandbox creation left a retained sandbox; `destroy --yes` refused unconditionally and exited nonzero. After the fix, the same repro's `destroy --yes` deletes the sandbox automatically and exits 0, matching the manual workaround's outcome without requiring it. The exit-code-0-on-refusal and `nemoclaw list` vs `openshell sandbox list` visibility-gap symptoms also reported in this issue no longer reproduce on current main; only the misdirected remediation guidance was still present, and this fix addresses that. Fixes #10863 Signed-off-by: Yanyun Liao --- docs/reference/commands.mdx | 8 ++--- src/lib/actions/sandbox/destroy-preflight.ts | 30 ++++++++++++++--- .../destroy-retained-recovery-flow.test.ts | 33 +++++++++++++++++-- src/lib/actions/sandbox/destroy.ts | 32 ++++++++++++++---- 4 files changed, 87 insertions(+), 16 deletions(-) diff --git a/docs/reference/commands.mdx b/docs/reference/commands.mdx index de3a33321eb..718651d84e0 100644 --- a/docs/reference/commands.mdx +++ b/docs/reference/commands.mdx @@ -365,14 +365,14 @@ When the recovery record contains a durable identity fingerprint, run `$$nemocla Use the result from `destroy` to choose the next action: -- If OpenShell still reports the sandbox present, `destroy` preserves the record and removes no resources. Do not delete the sandbox manually by mutable name. Give the displayed create-attempt label to an OpenShell administrator when present and ask them to remove that exact sandbox through an identity-bound procedure. Without a label, preserve the terminal output and ask the administrator to identify the exact sandbox from gateway or controller evidence. -- After the administrator removes the exact sandbox, run `$$nemoclaw destroy --yes` to reconcile the recovery record. +- If OpenShell still reports the sandbox present, `destroy` compares the live sandbox's exact OpenShell id against the retained recovery record's durable identity fingerprint. When they match, `destroy` has proven the live sandbox is the exact retained one — not a different sandbox that has since reused the same name — and proceeds to delete it and clear the matching recovery record, the same outcome as running `openshell sandbox delete ` yourself. +- If OpenShell reports the sandbox present but `destroy` cannot prove that match (for example, a mismatched or unreadable live id), `destroy` preserves the record and removes no resources. Its output names the exact `openshell sandbox delete ` command; compare `openshell sandbox list -o json` against the displayed create-attempt label yourself, and once you are confident it is the same sandbox, running that command is safe. After OpenShell confirms the sandbox is absent, run `$$nemoclaw destroy --yes` to reconcile the recovery record. - If OpenShell confirms the sandbox is absent, `destroy` verifies one retained recovery record and its immutable runtime identity. For Docker-backed sandboxes, it also verifies the immutable Docker sandbox identity and removes only qualified residual containers. It clears the matching recovery record only after verified cleanup. - If the recovery record is ambiguous or immutable runtime-identity verification fails, cleanup stops and the record remains. For Docker-backed sandboxes, a foreign container, changed Docker identity, or failed Docker probe has the same fail-closed result. -If OpenShell did not return a durable identity fingerprint, `destroy` cannot complete recovery. A create-attempt label can help an OpenShell administrator identify and remove the exact sandbox, but it does not let NemoClaw retire the record without immutable identity authority. If neither a fingerprint nor a label is available, preserve the terminal output and ask the administrator to identify the exact sandbox from gateway or controller evidence. If NemoClaw reports that it could not save recovery evidence, preserve the terminal output and follow the same escalation. +If OpenShell did not return a durable identity fingerprint, `destroy` cannot complete recovery or prove a match for a live sandbox. A create-attempt label narrows the search, but you must identify and remove the exact sandbox yourself — inspect `openshell sandbox list -o json` and compare it against the displayed create-attempt label and any other available evidence. If NemoClaw reports that it could not save recovery evidence, preserve the terminal output and follow the same manual-inspection path. -This fail-closed record keeps only the affected sandbox name unavailable. It is not retired from mutable-name absence alone, and this command does not accept administrator-supplied identity authority. +This fail-closed record keeps only the affected sandbox name unavailable. It is not retired from mutable-name absence alone, and this command does not accept externally supplied identity authority. To onboard another sandbox while the record remains unresolved, supply a different explicit name: ```bash diff --git a/src/lib/actions/sandbox/destroy-preflight.ts b/src/lib/actions/sandbox/destroy-preflight.ts index ce0b0565300..ef4c72da767 100644 --- a/src/lib/actions/sandbox/destroy-preflight.ts +++ b/src/lib/actions/sandbox/destroy-preflight.ts @@ -3,6 +3,8 @@ import os from "node:os"; +import { fingerprintOpenShellSandboxId } from "../../adapters/openshell/sandbox-identity"; +import { observeOpenShellSandboxIdentity } from "../../adapters/openshell/sandbox-presence"; import { OPENSHELL_PROBE_TIMEOUT_MS } from "../../adapters/openshell/timeouts"; import { withModelRouterPortLifecycleLock } from "../../inference/gateway-route-mutation-lock"; import { DEFAULT_MODEL_ROUTER_PORT, isRoutedInferenceProvider } from "../../onboard/model-router"; @@ -22,7 +24,6 @@ import type { import type { SandboxEntry } from "../../state/registry"; import * as registry from "../../state/registry"; import { type DestroyRunOpenshell, selectGatewayForSandboxDestroy } from "./destroy-gateway"; -import { classifyDestroySandboxPresence } from "./destroy-presence"; import { getPersistedSandboxTargetGatewayName, getSandboxTargetGatewayName, @@ -34,6 +35,12 @@ export type SandboxDestroyPreflight = { runOpenshell: DestroyRunOpenshell; sandbox: SandboxEntry | null; sandboxConfirmedAbsent: boolean; + /** + * Fingerprint of the exact live OpenShell sandbox id when one is present, + * comparable to a retained recovery record's `sandboxIdentityFingerprint` + * (#10863). Null when absent or the observation could not be trusted. + */ + presentSandboxIdentityFingerprint: string | null; }; export function stopSandboxInferenceResources( @@ -295,7 +302,12 @@ export function prepareSandboxDestroy( selectGatewayForSandboxDestroy(sandboxName, cleanupGatewayName, runOpenshell); process.env.OPENSHELL_GATEWAY = cleanupGatewayName; - const sandboxPresence = classifyDestroySandboxPresence( + // Read the exact live OpenShell sandbox id (when present) alongside its + // presence, not just present/absent, so a caller can prove the live + // sandbox is the exact retained one before treating it as safe to delete + // by mutable name (#10863) — mirrors the identity comparison already used + // for Hermes Portable lifecycle verification. + const sandboxIdentityObservation = observeOpenShellSandboxIdentity( sandboxName, runOpenshell(["sandbox", "list", "-o", "json"], { ignoreError: true, @@ -303,7 +315,11 @@ export function prepareSandboxDestroy( timeout: OPENSHELL_PROBE_TIMEOUT_MS, }), ); - const sandboxConfirmedAbsent = sandboxPresence === "absent"; + const sandboxConfirmedAbsent = sandboxIdentityObservation.kind === "absent"; + const presentSandboxIdentityFingerprint = + sandboxIdentityObservation.kind === "present" + ? fingerprintOpenShellSandboxId(sandboxIdentityObservation.id) + : null; const mcpEntriesRequiringConfigMutation = Object.values(sandbox?.mcp?.bridges ?? {}).filter( (entry) => entry.addState !== "prepared", ); @@ -323,5 +339,11 @@ export function prepareSandboxDestroy( assertMcpAdapterConfigMutationsAllowed(sandboxName, sandbox, mcpEntriesRequiringConfigMutation); } - return { cleanupGatewayName, runOpenshell, sandbox, sandboxConfirmedAbsent }; + return { + cleanupGatewayName, + runOpenshell, + sandbox, + sandboxConfirmedAbsent, + presentSandboxIdentityFingerprint, + }; } diff --git a/src/lib/actions/sandbox/destroy-retained-recovery-flow.test.ts b/src/lib/actions/sandbox/destroy-retained-recovery-flow.test.ts index 2d5af68dd8d..0baff383a46 100644 --- a/src/lib/actions/sandbox/destroy-retained-recovery-flow.test.ts +++ b/src/lib/actions/sandbox/destroy-retained-recovery-flow.test.ts @@ -168,7 +168,7 @@ describe("destroySandbox retained recovery flow", () => { ); expect(harness.errorSpy).toHaveBeenCalledWith( - expect.stringContaining("delete command accepts only the mutable sandbox name"), + expect.stringContaining("could not prove it is the exact retained sandbox"), ); expect(harness.runOpenshellSpy).not.toHaveBeenCalledWith( ["sandbox", "delete", "alpha"], @@ -202,7 +202,7 @@ describe("destroySandbox retained recovery flow", () => { ); expect(harness.errorSpy).toHaveBeenCalledWith( - expect.stringContaining("cannot bind that deletion to the retained immutable identity"), + expect.stringContaining("could not prove it is the exact retained sandbox"), ); expect(harness.runOpenshellSpy).not.toHaveBeenCalledWith( ["sandbox", "delete", "alpha"], @@ -213,6 +213,35 @@ describe("destroySandbox retained recovery flow", () => { }, ); + it( + "issues mutable-name deletion for a live retained sandbox with a proven-matching OpenShell identity (#10863)", + { timeout: 30_000 }, + async () => { + const recovery = retainedRecoveryRecord("sandbox-alpha"); + const containerId = "a".repeat(64); + const harness = createDestroyHarness({ + dockerRunResult: { + status: 0, + stdout: `${containerId}\topenshell\tdefault\tsandbox-alpha`, + }, + registryEntryOverrides: { + lifecycleGeneration: recovery.lifecycleGeneration!, + lifecycleLiveIdentityFingerprint: recovery.sandboxIdentityFingerprint!, + }, + retainedRecoveryRecords: [recovery], + }); + + await expect(harness.destroySandbox("alpha", { yes: true })).resolves.toBeUndefined(); + + expect(harness.runOpenshellSpy).toHaveBeenCalledWith( + ["sandbox", "delete", "alpha"], + expect.anything(), + ); + expect(harness.resolveRetainedSandboxRecoverySpy).toHaveBeenCalledWith(recovery); + expect(exitSpy).not.toHaveBeenCalled(); + }, + ); + it( "finishes retained cleanup after OpenShell already removed the sandbox (#10547)", { timeout: 30_000 }, diff --git a/src/lib/actions/sandbox/destroy.ts b/src/lib/actions/sandbox/destroy.ts index a63595ab2d1..d6d75b3968c 100644 --- a/src/lib/actions/sandbox/destroy.ts +++ b/src/lib/actions/sandbox/destroy.ts @@ -798,13 +798,33 @@ async function destroySandboxUnlocked( retainedRecoveryGatewayName: retainedRecoveryAuthority?.gatewayName, }), ); - const { cleanupGatewayName, runOpenshell, sandbox, sandboxConfirmedAbsent } = destroyPreflight; + const { + cleanupGatewayName, + runOpenshell, + sandbox, + sandboxConfirmedAbsent, + presentSandboxIdentityFingerprint, + } = destroyPreflight; if (retainedRecoveryAuthority && !sandboxConfirmedAbsent) { - console.error( - ` Refusing to automatically delete retained sandbox '${sandboxName}': OpenShell still reports it present, but its delete command accepts only the mutable sandbox name. NemoClaw cannot bind that deletion to the retained immutable identity. No sandbox resources were removed. Ask an OpenShell administrator to resolve create-attempt label '${retainedRecoveryAuthority.createAttemptNonce}' to the exact sandbox and use an identity-bound removal procedure. After OpenShell confirms the retained sandbox is absent, rerun '${CLI_NAME} ${sandboxName} destroy --yes' to reconcile its verified Docker containers and recovery record.`, - ); - preparedManagedLlamaCppCleanup?.abort(); - requestSandboxDestroyExit(1); + // The live OpenShell sandbox with this mutable name may be the exact + // retained sandbox recovery is waiting on, or (rarely) a replacement + // sandbox that has since reused the same name. Prove which one this is + // from the live sandbox id fingerprint before deciding (#10863): a + // confirmed match is provably the retained sandbox, so destroy proceeds + // exactly as it would for any other present sandbox — an ordinary user + // running `openshell sandbox delete ` themselves reaches the same + // outcome, so NemoClaw can safely do it once identity is proven. + const identityConfirmedMatch = + retainedRecoveryAuthority.sandboxIdentityFingerprint !== null && + presentSandboxIdentityFingerprint !== null && + retainedRecoveryAuthority.sandboxIdentityFingerprint === presentSandboxIdentityFingerprint; + if (!identityConfirmedMatch) { + console.error( + ` Refusing to automatically delete retained sandbox '${sandboxName}': OpenShell reports a sandbox present under this name, but NemoClaw could not prove it is the exact retained sandbox (create-attempt label '${retainedRecoveryAuthority.createAttemptNonce}') rather than a different sandbox that has since reused the same name. No sandbox resources were removed. Compare 'openshell sandbox list -o json' against the create-attempt label above; if you confirm it is the same sandbox, 'openshell sandbox delete ${sandboxName}' is the exact command NemoClaw would otherwise run, and is safe to run yourself. After OpenShell confirms the retained sandbox is absent, rerun '${CLI_NAME} ${sandboxName} destroy --yes' to reconcile its verified Docker containers and recovery record.`, + ); + preparedManagedLlamaCppCleanup?.abort(); + requestSandboxDestroyExit(1); + } } // Recheck identity after pre-delete qualification and recoverable journal // publication reconciliation, before any sandbox runtime mutation. From c21c193e36c701dd2f4383e07210be84a25f3d13 Mon Sep 17 00:00:00 2001 From: Yanyun Liao Date: Wed, 2 Sep 2026 17:34:20 +0800 Subject: [PATCH 02/12] Address CodeRabbit review: TOCTOU-safe retained-identity re-check + docs fix Re-prove the retained sandbox's OpenShell identity fingerprint immediately before the delete command fires, not just during preflight, so a same-name replacement sandbox created in between cannot be deleted under the retained recovery record's authority. Also fixes a self-contradictory docs sentence that implied manual `openshell sandbox delete` clears the recovery record. Co-Authored-By: Claude Sonnet 5 --- docs/reference/commands.mdx | 2 +- src/lib/actions/sandbox/destroy-execution.ts | 30 +++++++++++++++ .../destroy-retained-recovery-flow.test.ts | 37 +++++++++++++++++++ src/lib/actions/sandbox/destroy.ts | 10 +++++ test/helpers/destroy-flow-test-harness.ts | 14 +++++++ 5 files changed, 92 insertions(+), 1 deletion(-) diff --git a/docs/reference/commands.mdx b/docs/reference/commands.mdx index 061f763e870..deacf9a1840 100644 --- a/docs/reference/commands.mdx +++ b/docs/reference/commands.mdx @@ -365,7 +365,7 @@ When the recovery record contains a durable identity fingerprint, run `$$nemocla Use the result from `destroy` to choose the next action: -- If OpenShell still reports the sandbox present, `destroy` compares the live sandbox's exact OpenShell id against the retained recovery record's durable identity fingerprint. When they match, `destroy` has proven the live sandbox is the exact retained one — not a different sandbox that has since reused the same name — and proceeds to delete it and clear the matching recovery record, the same outcome as running `openshell sandbox delete ` yourself. +- If OpenShell still reports the sandbox present, `destroy` compares the live sandbox's exact OpenShell id against the retained recovery record's durable identity fingerprint. When they match, `destroy` has proven the live sandbox is the exact retained one — not a different sandbox that has since reused the same name — and proceeds to delete it and clear the matching recovery record. This matches the sandbox-deletion result of running `openshell sandbox delete ` yourself, but the manual command does not clear the recovery record. - If OpenShell reports the sandbox present but `destroy` cannot prove that match (for example, a mismatched or unreadable live id), `destroy` preserves the record and removes no resources. Its output names the exact `openshell sandbox delete ` command; compare `openshell sandbox list -o json` against the displayed create-attempt label yourself, and once you are confident it is the same sandbox, running that command is safe. After OpenShell confirms the sandbox is absent, run `$$nemoclaw destroy --yes` to reconcile the recovery record. - If OpenShell confirms the sandbox is absent, `destroy` verifies one retained recovery record and its immutable runtime identity. For Docker-backed sandboxes, it also verifies the immutable Docker sandbox identity and removes only qualified residual containers. It clears the matching recovery record only after verified cleanup. - If the recovery record is ambiguous or immutable runtime-identity verification fails, cleanup stops and the record remains. For Docker-backed sandboxes, a foreign container, changed Docker identity, or failed Docker probe has the same fail-closed result. diff --git a/src/lib/actions/sandbox/destroy-execution.ts b/src/lib/actions/sandbox/destroy-execution.ts index 2a1ef90c100..41b82947844 100644 --- a/src/lib/actions/sandbox/destroy-execution.ts +++ b/src/lib/actions/sandbox/destroy-execution.ts @@ -78,6 +78,15 @@ type SandboxDestroyExecutionInput = { expectedContainerIdentities?: readonly SandboxNameLabeledContainer[]; expectedContainerIdentityFingerprint?: string; expectedRuntimeProviderIdentity?: RuntimeProviderDestroyIdentityReceipt; + // Set only when a retained-sandbox recovery record's identity was proven + // against the live OpenShell sandbox during preflight (#10863). Re-proven + // immediately before the delete command fires, so a same-name replacement + // sandbox created after preflight cannot be deleted under the recovery + // record's authority. + expectedRetainedSandboxIdentity?: { + readonly gatewayName: string; + readonly sandboxIdentityFingerprint: string; + }; portableContainerAuthority?: PreparedPortableDemoSandboxDestroyAuthority; stopInferenceResources: () => void; runtimeProviders?: RuntimeProviderBundleRegistry; @@ -311,6 +320,7 @@ export async function executeSandboxDestroy({ expectedContainerIdentities, expectedContainerIdentityFingerprint, expectedRuntimeProviderIdentity, + expectedRetainedSandboxIdentity, portableContainerAuthority, stopInferenceResources, runtimeProviders = CURRENT_RUNTIME_PROVIDER_BUNDLES, @@ -389,9 +399,29 @@ export async function executeSandboxDestroy({ }; } }; + const inspectRetainedSandboxIdentityContinuity = (): IdentityContinuity => { + if (!expectedRetainedSandboxIdentity) return { status: "match" }; + const subject = "Retained sandbox identity"; + try { + const inspectIdentity = + deps.inspectOpenShellSandboxIdentityFingerprint ?? + inspectOpenShellSandboxIdentityFingerprint; + const liveFingerprint = inspectIdentity({ + sandboxName, + gatewayName: expectedRetainedSandboxIdentity.gatewayName, + }); + return liveFingerprint === expectedRetainedSandboxIdentity.sandboxIdentityFingerprint + ? { status: "match" } + : { status: "changed", subject }; + } catch (error) { + return { status: "probe-failed", subject, detail: redactDestroyError(error) }; + } + }; const inspectIdentityContinuity = (): IdentityContinuity => { const pendingContinuity = inspectPendingCreateVerificationContinuity(); if (pendingContinuity.status !== "match") return pendingContinuity; + const retainedContinuity = inspectRetainedSandboxIdentityContinuity(); + if (retainedContinuity.status !== "match") return retainedContinuity; if (portableContainerAuthority) { try { portableContainerAuthority.revalidate(); diff --git a/src/lib/actions/sandbox/destroy-retained-recovery-flow.test.ts b/src/lib/actions/sandbox/destroy-retained-recovery-flow.test.ts index 0baff383a46..6b962367b10 100644 --- a/src/lib/actions/sandbox/destroy-retained-recovery-flow.test.ts +++ b/src/lib/actions/sandbox/destroy-retained-recovery-flow.test.ts @@ -9,6 +9,7 @@ import { createDestroyHarness, resetDestroyModuleCache, } from "../../../../test/helpers/destroy-flow-test-harness"; +import { executeSandboxDestroy } from "./destroy-execution"; import type { RetainedSandboxRecoveryRecord } from "../../state/onboard-session/retained-sandbox-recovery"; function retainedRecoveryRecord(sandboxId = "sb-alpha"): RetainedSandboxRecoveryRecord { @@ -242,6 +243,42 @@ describe("destroySandbox retained recovery flow", () => { }, ); + it( + "aborts the delete when the retained sandbox's live identity no longer matches at the delete boundary (#10863)", + { timeout: 30_000 }, + async () => { + const recovery = retainedRecoveryRecord("sandbox-alpha"); + const runOpenshell = vi.fn(() => ({ status: 0, stdout: "", stderr: "" })); + + const result = await executeSandboxDestroy({ + cleanupShieldsArtifacts: () => undefined, + force: false, + runOpenshell, + sandbox: null, + sandboxConfirmedAbsent: false, + sandboxName: "alpha", + stopInferenceResources: () => undefined, + expectedRetainedSandboxIdentity: { + gatewayName: recovery.gatewayName, + sandboxIdentityFingerprint: recovery.sandboxIdentityFingerprint!, + }, + deps: { + // Simulates a same-name replacement sandbox that appeared after the + // destroy.ts preflight proof but before this delete-boundary re-check. + inspectOpenShellSandboxIdentityFingerprint: () => "b".repeat(64), + readTimerMarker: () => null, + wipeSandboxState: () => undefined, + }, + }); + + expect(result).toMatchObject({ + ok: false, + deleteOutput: expect.stringContaining("Retained sandbox identity"), + }); + expect(runOpenshell).not.toHaveBeenCalled(); + }, + ); + it( "finishes retained cleanup after OpenShell already removed the sandbox (#10547)", { timeout: 30_000 }, diff --git a/src/lib/actions/sandbox/destroy.ts b/src/lib/actions/sandbox/destroy.ts index 0e21e6bc13f..27596749312 100644 --- a/src/lib/actions/sandbox/destroy.ts +++ b/src/lib/actions/sandbox/destroy.ts @@ -810,6 +810,7 @@ async function destroySandboxUnlocked( sandboxConfirmedAbsent, presentSandboxIdentityFingerprint, } = destroyPreflight; + let confirmedRetainedSandboxIdentityFingerprint: string | null = null; if (retainedRecoveryAuthority && !sandboxConfirmedAbsent) { // The live OpenShell sandbox with this mutable name may be the exact // retained sandbox recovery is waiting on, or (rarely) a replacement @@ -830,6 +831,7 @@ async function destroySandboxUnlocked( preparedManagedLlamaCppCleanup?.abort(); requestSandboxDestroyExit(1); } + confirmedRetainedSandboxIdentityFingerprint = retainedRecoveryAuthority.sandboxIdentityFingerprint; } // Recheck identity after pre-delete qualification and recoverable journal // publication reconciliation, before any sandbox runtime mutation. @@ -880,6 +882,14 @@ async function destroySandboxUnlocked( ? { expectedRuntimeProviderIdentity: initialIdentity.providerIdentity } : {}), ...(portableContainerAuthority ? { portableContainerAuthority } : {}), + ...(retainedRecoveryAuthority && confirmedRetainedSandboxIdentityFingerprint !== null + ? { + expectedRetainedSandboxIdentity: { + gatewayName: retainedRecoveryAuthority.gatewayName, + sandboxIdentityFingerprint: confirmedRetainedSandboxIdentityFingerprint, + }, + } + : {}), stopInferenceResources: () => stopSandboxInferenceResources(sandboxName, sandbox), }); } catch (error) { diff --git a/test/helpers/destroy-flow-test-harness.ts b/test/helpers/destroy-flow-test-harness.ts index 290bbb9e0d2..be5e9c858cf 100644 --- a/test/helpers/destroy-flow-test-harness.ts +++ b/test/helpers/destroy-flow-test-harness.ts @@ -473,6 +473,20 @@ export function createDestroyHarness(options: DestroyHarnessOptions = {}): Destr status: 0, output: options.liveListOutput ?? "", }); + // Backs the delete-boundary retained-sandbox identity re-check + // (`inspectOpenShellSandboxIdentityFingerprint`, #10863). Mirrors the same + // `sandbox-${name}` id convention `sandboxListJson` uses for `sandbox list`. + vi.spyOn(runtime, "captureResolvedOpenshell").mockImplementation((args: unknown) => { + const argv = Array.isArray(args) ? args.map(String) : []; + if (argv[0] === "sandbox" && argv[1] === "get") { + if (!sandboxPresent) { + return { status: 1, stdout: "", stderr: "sandbox not found" }; + } + const name = argv.at(-1) ?? ""; + return { status: 0, stdout: `Id: sandbox-${name}\n`, stderr: "" }; + } + return { status: 0, stdout: "", stderr: "" }; + }); const dockerCaptureSpy = vi .spyOn(dockerRun, "dockerCapture") .mockImplementation((args: unknown) => { From a5a973ea0775eb908ac5e1409086a6aefbbb1fc2 Mon Sep 17 00:00:00 2001 From: Yanyun Liao Date: Wed, 2 Sep 2026 17:44:30 +0800 Subject: [PATCH 03/12] Fix retained-identity re-check mock after main's OpenShell identity refactor Main moved inspectOpenShellSandboxIdentityFingerprint's transport from runtime.captureResolvedOpenshell to sandbox-identity-cli's captureSanitizedResolvedOpenshell, and it binds that capture function by value at module load time rather than through a live namespace lookup. The harness now spies on the new capture function and force-refreshes the modules that bind it before each test, so the #10863 delete-boundary re-check exercises the mock instead of the real OpenShell binary. Co-Authored-By: Claude Sonnet 5 --- test/helpers/destroy-flow-test-harness.ts | 47 ++++++++++++++++------- 1 file changed, 33 insertions(+), 14 deletions(-) diff --git a/test/helpers/destroy-flow-test-harness.ts b/test/helpers/destroy-flow-test-harness.ts index be5e9c858cf..066a0ee09e1 100644 --- a/test/helpers/destroy-flow-test-harness.ts +++ b/test/helpers/destroy-flow-test-harness.ts @@ -207,6 +207,39 @@ export function createDestroyHarness(options: DestroyHarnessOptions = {}): Destr const resolve = requireSource("../../adapters/openshell/resolve.js"); const runtime = requireSource("../../adapters/openshell/runtime.js"); + const sanitizedCapture = requireSource("../../adapters/openshell/sanitized-capture.js"); + // Backs the delete-boundary retained-sandbox identity re-check + // (`inspectOpenShellSandboxIdentityFingerprint`, #10863). Mirrors the same + // `sandbox-${name}` id convention `sandboxListJson` uses for `sandbox list`. + // `sandbox-identity-cli.js` binds this by value at its own module load time + // (not a live namespace lookup), so it — and everything that transitively + // requires it — must be re-required fresh after the spy is installed. + vi.spyOn(sanitizedCapture, "captureSanitizedResolvedOpenshell").mockImplementation( + (args: unknown) => { + const argv = Array.isArray(args) ? args.map(String) : []; + if (argv[0] === "sandbox" && argv[1] === "get") { + if (!sandboxPresent) { + return { + status: 1, + output: "", + stdout: "", + stderr: "sandbox not found", + error: new Error("sandbox not found"), + }; + } + const name = argv.at(-1) ?? ""; + return { + status: 0, + output: `Id: sandbox-${name}\n`, + stdout: `Id: sandbox-${name}\n`, + stderr: "", + }; + } + return { status: 0, output: "", stdout: "", stderr: "" }; + }, + ); + delete require.cache[requireSource.resolve("../../adapters/openshell/sandbox-identity-cli.js")]; + delete require.cache[requireSource.resolve("./destroy-execution.js")]; const destroyGateway = requireSource("./destroy-gateway.js"); const credentialStore = requireSource("../../credentials/store.js"); const sandboxProviderCleanup = requireSource("../../onboard/sandbox-provider-cleanup.js"); @@ -473,20 +506,6 @@ export function createDestroyHarness(options: DestroyHarnessOptions = {}): Destr status: 0, output: options.liveListOutput ?? "", }); - // Backs the delete-boundary retained-sandbox identity re-check - // (`inspectOpenShellSandboxIdentityFingerprint`, #10863). Mirrors the same - // `sandbox-${name}` id convention `sandboxListJson` uses for `sandbox list`. - vi.spyOn(runtime, "captureResolvedOpenshell").mockImplementation((args: unknown) => { - const argv = Array.isArray(args) ? args.map(String) : []; - if (argv[0] === "sandbox" && argv[1] === "get") { - if (!sandboxPresent) { - return { status: 1, stdout: "", stderr: "sandbox not found" }; - } - const name = argv.at(-1) ?? ""; - return { status: 0, stdout: `Id: sandbox-${name}\n`, stderr: "" }; - } - return { status: 0, stdout: "", stderr: "" }; - }); const dockerCaptureSpy = vi .spyOn(dockerRun, "dockerCapture") .mockImplementation((args: unknown) => { From 15acba8e7da96f15f7b256e7ca6b6b1fb0332b1c Mon Sep 17 00:00:00 2001 From: Yanyun Liao Date: Wed, 2 Sep 2026 18:18:37 +0800 Subject: [PATCH 04/12] Address CodeRabbit follow-up: also refresh destroy-preflight.js module cache Keeps the harness's forced module refresh consistent across every module in the destroy identity-check chain, so a future preflight change that binds a capture function by value (as sandbox-identity-cli.js does today) can't silently pick up a stale reference from an earlier test's harness instance. Co-Authored-By: Claude Sonnet 5 --- test/helpers/destroy-flow-test-harness.ts | 1 + 1 file changed, 1 insertion(+) diff --git a/test/helpers/destroy-flow-test-harness.ts b/test/helpers/destroy-flow-test-harness.ts index 066a0ee09e1..830881ac684 100644 --- a/test/helpers/destroy-flow-test-harness.ts +++ b/test/helpers/destroy-flow-test-harness.ts @@ -240,6 +240,7 @@ export function createDestroyHarness(options: DestroyHarnessOptions = {}): Destr ); delete require.cache[requireSource.resolve("../../adapters/openshell/sandbox-identity-cli.js")]; delete require.cache[requireSource.resolve("./destroy-execution.js")]; + delete require.cache[requireSource.resolve("./destroy-preflight.js")]; const destroyGateway = requireSource("./destroy-gateway.js"); const credentialStore = requireSource("../../credentials/store.js"); const sandboxProviderCleanup = requireSource("../../onboard/sandbox-provider-cleanup.js"); From c997d3fa11822db4571e64fe72452974540efe3e Mon Sep 17 00:00:00 2001 From: Yanyun Liao Date: Mon, 7 Sep 2026 11:20:42 +0800 Subject: [PATCH 05/12] Rework #10863 fix to fail-closed: never auto-delete a live retained sandbox A maintainer review found that even the delete-boundary identity re-check cannot close the race: OpenShell has no atomic delete-by-identity primitive, so another OpenShell client can still replace the retained sandbox under the same mutable name between NemoClaw's last identity read and OpenShell processing the delete. Automatic deletion of a live retained sandbox is now always fail-closed, regardless of identity match, with guidance pointing to the owner-scoped 'openshell sandbox delete -g ' command for the operator to run themselves after independently confirming identity. Removes the now-unnecessary identity-proof machinery (destroy-preflight.ts's presentSandboxIdentityFingerprint, destroy-execution.ts's expectedRetainedSandboxIdentity re-check, and the test harness mocking it required) since it no longer gates any auto-delete decision. Also brings the onboarding-cancel-time recovery guidance (cancel-rollback.ts) in line: it no longer points to a nonexistent "OpenShell administrator identity-bound removal procedure" and instead tells the operator to rerun destroy (which now prints the exact manual command) or inspect 'openshell sandbox list -o json' themselves when no identity fingerprint was ever recorded. Co-Authored-By: Claude Sonnet 5 --- docs/reference/commands.mdx | 3 +- src/lib/actions/sandbox/destroy-execution.ts | 30 --------- src/lib/actions/sandbox/destroy-preflight.ts | 17 ----- .../destroy-retained-recovery-flow.test.ts | 64 ++++++------------- src/lib/actions/sandbox/destroy.ts | 44 ++++--------- src/lib/onboard/cancel-rollback.test.ts | 2 +- src/lib/onboard/cancel-rollback.ts | 8 +-- test/helpers/destroy-flow-test-harness.ts | 34 ---------- 8 files changed, 40 insertions(+), 162 deletions(-) diff --git a/docs/reference/commands.mdx b/docs/reference/commands.mdx index df3717481cc..70b0e1c8e63 100644 --- a/docs/reference/commands.mdx +++ b/docs/reference/commands.mdx @@ -362,8 +362,7 @@ When the recovery record contains a durable identity fingerprint, run `$$nemocla Use the result from `destroy` to choose the next action: -- If OpenShell still reports the sandbox present, `destroy` compares the live sandbox's exact OpenShell id against the retained recovery record's durable identity fingerprint. When they match, `destroy` has proven the live sandbox is the exact retained one — not a different sandbox that has since reused the same name — and proceeds to delete it and clear the matching recovery record. This matches the sandbox-deletion result of running `openshell sandbox delete ` yourself, but the manual command does not clear the recovery record. -- If OpenShell reports the sandbox present but `destroy` cannot prove that match (for example, a mismatched or unreadable live id), `destroy` preserves the record and removes no resources. Its output names the exact `openshell sandbox delete ` command; compare `openshell sandbox list -o json` against the displayed create-attempt label yourself, and once you are confident it is the same sandbox, running that command is safe. After OpenShell confirms the sandbox is absent, run `$$nemoclaw destroy --yes` to reconcile the recovery record. +- If OpenShell still reports the sandbox present, `destroy` never deletes it automatically, even when the live sandbox's exact OpenShell id matches the retained recovery record's durable identity fingerprint. OpenShell exposes no atomic delete-by-identity primitive, so NemoClaw cannot bind a mutable-name delete to that identity evidence without a residual race: another OpenShell client could remove the retained sandbox and create a replacement under the same name between NemoClaw's last identity read and OpenShell processing the delete. `destroy` preserves the record and removes no resources; its output names the owner-scoped `openshell sandbox delete -g '' ` command. Compare `openshell sandbox list -o json` against the displayed create-attempt label yourself, and only once you have independently confirmed it is the same sandbox, run that command. After OpenShell confirms the sandbox is absent, run `$$nemoclaw destroy --yes` to reconcile the recovery record — the manual delete alone does not clear it. - If OpenShell confirms the sandbox is absent, `destroy` verifies one retained recovery record and its immutable runtime identity. For Docker-backed sandboxes, it also verifies the immutable Docker sandbox identity and removes only qualified residual containers. It clears the matching recovery record only after verified cleanup. - If the recovery record is ambiguous or immutable runtime-identity verification fails, cleanup stops and the record remains. For Docker-backed sandboxes, a foreign container, changed Docker identity, or failed Docker probe has the same fail-closed result. diff --git a/src/lib/actions/sandbox/destroy-execution.ts b/src/lib/actions/sandbox/destroy-execution.ts index c399d44875f..891d6c82a40 100644 --- a/src/lib/actions/sandbox/destroy-execution.ts +++ b/src/lib/actions/sandbox/destroy-execution.ts @@ -78,15 +78,6 @@ type SandboxDestroyExecutionInput = { expectedContainerIdentities?: readonly SandboxNameLabeledContainer[]; expectedContainerIdentityFingerprint?: string; expectedRuntimeProviderIdentity?: RuntimeProviderDestroyIdentityReceipt; - // Set only when a retained-sandbox recovery record's identity was proven - // against the live OpenShell sandbox during preflight (#10863). Re-proven - // immediately before the delete command fires, so a same-name replacement - // sandbox created after preflight cannot be deleted under the recovery - // record's authority. - expectedRetainedSandboxIdentity?: { - readonly gatewayName: string; - readonly sandboxIdentityFingerprint: string; - }; portableContainerAuthority?: PreparedPortableDemoSandboxDestroyAuthority; verifyForwardPortsReleased?: () => boolean; stopInferenceResources: () => void; @@ -220,7 +211,6 @@ export async function executeSandboxDestroy({ expectedContainerIdentities, expectedContainerIdentityFingerprint, expectedRuntimeProviderIdentity, - expectedRetainedSandboxIdentity, portableContainerAuthority, verifyForwardPortsReleased = () => true, stopInferenceResources, @@ -302,29 +292,9 @@ export async function executeSandboxDestroy({ }; } }; - const inspectRetainedSandboxIdentityContinuity = (): IdentityContinuity => { - if (!expectedRetainedSandboxIdentity) return { status: "match" }; - const subject = "Retained sandbox identity"; - try { - const inspectIdentity = - deps.inspectOpenShellSandboxIdentityFingerprint ?? - inspectOpenShellSandboxIdentityFingerprint; - const liveFingerprint = inspectIdentity({ - sandboxName, - gatewayName: expectedRetainedSandboxIdentity.gatewayName, - }); - return liveFingerprint === expectedRetainedSandboxIdentity.sandboxIdentityFingerprint - ? { status: "match" } - : { status: "changed", subject }; - } catch (error) { - return { status: "probe-failed", subject, detail: redactDestroyError(error) }; - } - }; const inspectIdentityContinuity = (): IdentityContinuity => { const pendingContinuity = inspectPendingCreateVerificationContinuity(); if (pendingContinuity.status !== "match") return pendingContinuity; - const retainedContinuity = inspectRetainedSandboxIdentityContinuity(); - if (retainedContinuity.status !== "match") return retainedContinuity; if (portableContainerAuthority) { try { portableContainerAuthority.revalidate(); diff --git a/src/lib/actions/sandbox/destroy-preflight.ts b/src/lib/actions/sandbox/destroy-preflight.ts index 33a50c589a9..c654bea5d97 100644 --- a/src/lib/actions/sandbox/destroy-preflight.ts +++ b/src/lib/actions/sandbox/destroy-preflight.ts @@ -4,7 +4,6 @@ import os from "node:os"; import { buildSelectedOpenShellSubprocessEnv } from "../../adapters/openshell/command-argv"; -import { fingerprintOpenShellSandboxId } from "../../adapters/openshell/sandbox-identity"; import { observeOpenShellSandboxIdentity } from "../../adapters/openshell/sandbox-presence"; import type { OpenShellRuntimeSelection } from "../../adapters/openshell/runtime-selection"; import { OPENSHELL_PROBE_TIMEOUT_MS } from "../../adapters/openshell/timeouts"; @@ -41,12 +40,6 @@ export type SandboxDestroyPreflight = { selectedRunOpenshell: DestroyRunOpenshell; sandbox: SandboxEntry | null; sandboxConfirmedAbsent: boolean; - /** - * Fingerprint of the exact live OpenShell sandbox id when one is present, - * comparable to a retained recovery record's `sandboxIdentityFingerprint` - * (#10863). Null when absent or the observation could not be trusted. - */ - presentSandboxIdentityFingerprint: string | null; }; export function resolveSandboxDestroyRuntimeSelection( @@ -349,11 +342,6 @@ export function prepareSandboxDestroy( selectGatewayForSandboxDestroy(sandboxName, cleanupGatewayName, selectedRunOpenshell); process.env.OPENSHELL_GATEWAY = cleanupGatewayName; - // Read the exact live OpenShell sandbox id (when present) alongside its - // presence, not just present/absent, so a caller can prove the live - // sandbox is the exact retained one before treating it as safe to delete - // by mutable name (#10863) — mirrors the identity comparison already used - // for Hermes Portable lifecycle verification. const sandboxIdentityObservation = observeOpenShellSandboxIdentity( sandboxName, selectedRunOpenshell(["sandbox", "list", "-o", "json"], { @@ -363,10 +351,6 @@ export function prepareSandboxDestroy( }), ); const sandboxConfirmedAbsent = sandboxIdentityObservation.kind === "absent"; - const presentSandboxIdentityFingerprint = - sandboxIdentityObservation.kind === "present" - ? fingerprintOpenShellSandboxId(sandboxIdentityObservation.id) - : null; return { cleanupGatewayName, @@ -374,7 +358,6 @@ export function prepareSandboxDestroy( selectedRunOpenshell, sandbox, sandboxConfirmedAbsent, - presentSandboxIdentityFingerprint, ...(selectedCaptureOpenshell ? { selectedCaptureOpenshell } : {}), ...(runtimeSelection ? { runtimeSelection } : {}), }; diff --git a/src/lib/actions/sandbox/destroy-retained-recovery-flow.test.ts b/src/lib/actions/sandbox/destroy-retained-recovery-flow.test.ts index 6b962367b10..ffacbadf6b6 100644 --- a/src/lib/actions/sandbox/destroy-retained-recovery-flow.test.ts +++ b/src/lib/actions/sandbox/destroy-retained-recovery-flow.test.ts @@ -9,7 +9,6 @@ import { createDestroyHarness, resetDestroyModuleCache, } from "../../../../test/helpers/destroy-flow-test-harness"; -import { executeSandboxDestroy } from "./destroy-execution"; import type { RetainedSandboxRecoveryRecord } from "../../state/onboard-session/retained-sandbox-recovery"; function retainedRecoveryRecord(sandboxId = "sb-alpha"): RetainedSandboxRecoveryRecord { @@ -169,7 +168,7 @@ describe("destroySandbox retained recovery flow", () => { ); expect(harness.errorSpy).toHaveBeenCalledWith( - expect.stringContaining("could not prove it is the exact retained sandbox"), + expect.stringContaining("cannot safely bind an automatic delete to the retained record"), ); expect(harness.runOpenshellSpy).not.toHaveBeenCalledWith( ["sandbox", "delete", "alpha"], @@ -203,7 +202,7 @@ describe("destroySandbox retained recovery flow", () => { ); expect(harness.errorSpy).toHaveBeenCalledWith( - expect.stringContaining("could not prove it is the exact retained sandbox"), + expect.stringContaining("cannot safely bind an automatic delete to the retained record"), ); expect(harness.runOpenshellSpy).not.toHaveBeenCalledWith( ["sandbox", "delete", "alpha"], @@ -215,9 +214,15 @@ describe("destroySandbox retained recovery flow", () => { ); it( - "issues mutable-name deletion for a live retained sandbox with a proven-matching OpenShell identity (#10863)", + "still fails closed for a live retained sandbox even with a proven-matching OpenShell identity (#10863)", { timeout: 30_000 }, async () => { + // OpenShell exposes no atomic delete-by-identity primitive, so no + // amount of identity proof inside NemoClaw can close the window where + // another OpenShell client replaces the sandbox under the same name + // between the last read and OpenShell processing the delete. Automatic + // deletion of a live retained sandbox is therefore always fail-closed, + // even when the live OpenShell id matches the retained record. const recovery = retainedRecoveryRecord("sandbox-alpha"); const containerId = "a".repeat(64); const harness = createDestroyHarness({ @@ -232,50 +237,21 @@ describe("destroySandbox retained recovery flow", () => { retainedRecoveryRecords: [recovery], }); - await expect(harness.destroySandbox("alpha", { yes: true })).resolves.toBeUndefined(); + await expect(harness.destroySandbox("alpha", { yes: true })).rejects.toThrow( + "process.exit(1)", + ); - expect(harness.runOpenshellSpy).toHaveBeenCalledWith( + expect(harness.errorSpy).toHaveBeenCalledWith( + expect.stringContaining("cannot safely bind an automatic delete to the retained record"), + ); + expect(harness.errorSpy).toHaveBeenCalledWith( + expect.stringContaining(`openshell sandbox delete -g ${recovery.gatewayName} alpha`), + ); + expect(harness.runOpenshellSpy).not.toHaveBeenCalledWith( ["sandbox", "delete", "alpha"], expect.anything(), ); - expect(harness.resolveRetainedSandboxRecoverySpy).toHaveBeenCalledWith(recovery); - expect(exitSpy).not.toHaveBeenCalled(); - }, - ); - - it( - "aborts the delete when the retained sandbox's live identity no longer matches at the delete boundary (#10863)", - { timeout: 30_000 }, - async () => { - const recovery = retainedRecoveryRecord("sandbox-alpha"); - const runOpenshell = vi.fn(() => ({ status: 0, stdout: "", stderr: "" })); - - const result = await executeSandboxDestroy({ - cleanupShieldsArtifacts: () => undefined, - force: false, - runOpenshell, - sandbox: null, - sandboxConfirmedAbsent: false, - sandboxName: "alpha", - stopInferenceResources: () => undefined, - expectedRetainedSandboxIdentity: { - gatewayName: recovery.gatewayName, - sandboxIdentityFingerprint: recovery.sandboxIdentityFingerprint!, - }, - deps: { - // Simulates a same-name replacement sandbox that appeared after the - // destroy.ts preflight proof but before this delete-boundary re-check. - inspectOpenShellSandboxIdentityFingerprint: () => "b".repeat(64), - readTimerMarker: () => null, - wipeSandboxState: () => undefined, - }, - }); - - expect(result).toMatchObject({ - ok: false, - deleteOutput: expect.stringContaining("Retained sandbox identity"), - }); - expect(runOpenshell).not.toHaveBeenCalled(); + expect(harness.resolveRetainedSandboxRecoverySpy).not.toHaveBeenCalled(); }, ); diff --git a/src/lib/actions/sandbox/destroy.ts b/src/lib/actions/sandbox/destroy.ts index 02b4fc10b34..7759769b8a7 100644 --- a/src/lib/actions/sandbox/destroy.ts +++ b/src/lib/actions/sandbox/destroy.ts @@ -742,30 +742,22 @@ async function destroySandboxUnlocked( selectedRunOpenshell: cleanupRunOpenshell, sandbox, sandboxConfirmedAbsent, - presentSandboxIdentityFingerprint, } = destroyPreflight; - let confirmedRetainedSandboxIdentityFingerprint: string | null = null; if (retainedRecoveryAuthority && !sandboxConfirmedAbsent) { - // The live OpenShell sandbox with this mutable name may be the exact - // retained sandbox recovery is waiting on, or (rarely) a replacement - // sandbox that has since reused the same name. Prove which one this is - // from the live sandbox id fingerprint before deciding (#10863): a - // confirmed match is provably the retained sandbox, so destroy proceeds - // exactly as it would for any other present sandbox — an ordinary user - // running `openshell sandbox delete ` themselves reaches the same - // outcome, so NemoClaw can safely do it once identity is proven. - const identityConfirmedMatch = - retainedRecoveryAuthority.sandboxIdentityFingerprint !== null && - presentSandboxIdentityFingerprint !== null && - retainedRecoveryAuthority.sandboxIdentityFingerprint === presentSandboxIdentityFingerprint; - if (!identityConfirmedMatch) { - console.error( - ` Refusing to automatically delete retained sandbox '${sandboxName}': OpenShell reports a sandbox present under this name, but NemoClaw could not prove it is the exact retained sandbox (create-attempt label '${retainedRecoveryAuthority.createAttemptNonce}') rather than a different sandbox that has since reused the same name. No sandbox resources were removed. Compare 'openshell sandbox list -o json' against the create-attempt label above; if you confirm it is the same sandbox, 'openshell sandbox delete ${sandboxName}' is the exact command NemoClaw would otherwise run, and is safe to run yourself. After OpenShell confirms the retained sandbox is absent, rerun '${CLI_NAME} ${sandboxName} destroy --yes' to reconcile its verified Docker containers and recovery record.`, - ); - preparedManagedLlamaCppCleanup?.abort(); - requestSandboxDestroyExit(1); - } - confirmedRetainedSandboxIdentityFingerprint = retainedRecoveryAuthority.sandboxIdentityFingerprint; + // OpenShell has no atomic delete-by-identity primitive: it exposes no + // way to bind a mutable-name delete to the retained record's immutable + // sandbox id/resource version. Even a fresh identity read immediately + // before the delete command cannot close the window where another + // OpenShell client removes the retained sandbox and creates a + // replacement under the same name between that read and OpenShell + // processing the delete (#10863). Automatic deletion of a live retained + // sandbox is therefore always fail-closed; a human must confirm identity + // out-of-band and run the exact OpenShell command themselves. + console.error( + ` Refusing to automatically delete retained sandbox '${sandboxName}': OpenShell reports a sandbox present under this name, but NemoClaw cannot safely bind an automatic delete to the retained record (create-attempt label '${retainedRecoveryAuthority.createAttemptNonce}') without an atomic OpenShell delete-by-identity primitive. No sandbox resources were removed. Compare 'openshell sandbox list -o json' against the create-attempt label above; once you have confirmed it is the same sandbox, run 'openshell sandbox delete -g ${retainedRecoveryAuthority.gatewayName} ${sandboxName}' yourself. After OpenShell confirms the sandbox is absent, rerun '${CLI_NAME} ${sandboxName} destroy --yes' to reconcile its verified Docker containers and recovery record.`, + ); + preparedManagedLlamaCppCleanup?.abort(); + requestSandboxDestroyExit(1); } // Recheck identity after pre-delete qualification and recoverable journal // publication reconciliation, before any sandbox runtime mutation. @@ -815,14 +807,6 @@ async function destroySandboxUnlocked( ? { expectedRuntimeProviderIdentity: initialIdentity.providerIdentity } : {}), ...(portableContainerAuthority ? { portableContainerAuthority } : {}), - ...(retainedRecoveryAuthority && confirmedRetainedSandboxIdentityFingerprint !== null - ? { - expectedRetainedSandboxIdentity: { - gatewayName: retainedRecoveryAuthority.gatewayName, - sandboxIdentityFingerprint: confirmedRetainedSandboxIdentityFingerprint, - }, - } - : {}), verifyForwardPortsReleased: () => teardownSandboxDashboardForward(sandboxName), stopInferenceResources: () => stopSandboxInferenceResources(sandboxName, sandbox), }); diff --git a/src/lib/onboard/cancel-rollback.test.ts b/src/lib/onboard/cancel-rollback.test.ts index d8201d13efe..f4ff3db0d51 100644 --- a/src/lib/onboard/cancel-rollback.test.ts +++ b/src/lib/onboard/cancel-rollback.test.ts @@ -282,7 +282,7 @@ describe("installSandboxCancelRollback", () => { const guidance = log.mock.calls.flat().join("\n"); expect(guidance).toContain("identity fingerprint is unavailable"); - expect(guidance).toContain("OpenShell administrator"); + expect(guidance).toContain("openshell sandbox list -o json"); }); }); diff --git a/src/lib/onboard/cancel-rollback.ts b/src/lib/onboard/cancel-rollback.ts index 90d7e9d2192..288e0dcc612 100644 --- a/src/lib/onboard/cancel-rollback.ts +++ b/src/lib/onboard/cancel-rollback.ts @@ -73,7 +73,7 @@ export function buildCancelRollbackMessage( ] : [ " Its durable identity fingerprint is unavailable; preserve the registry and onboarding recovery state.", - " Ask an OpenShell administrator to establish the exact sandbox identity before recovery or removal.", + " Inspect 'openshell sandbox list -o json' yourself to identify and remove the sandbox; NemoClaw cannot verify its identity without a recorded fingerprint.", ]), " NemoClaw did not run OpenShell's mutable-name deletion command because the name may now identify a replacement sandbox.", " Do not delete the sandbox by mutable sandbox name.", @@ -82,11 +82,11 @@ export function buildCancelRollbackMessage( ? [ ` Run '${cliName()} ${sandboxName} destroy'. If OpenShell confirms the retained sandbox absent, destroy removes only verified residual containers and can clear the matching recovery record.`, recoveryContext - ? " If it is still live, give the displayed create-attempt label to an OpenShell administrator for identity-bound removal." - : " If it is still live, preserve the displayed fingerprint and ask an OpenShell administrator for identity-bound removal.", + ? " If it is still live, destroy refuses automatic deletion and prints the exact manual removal command once you confirm identity against the create-attempt label above yourself." + : " If it is still live, preserve the displayed fingerprint; destroy refuses automatic deletion and prints manual removal guidance once you confirm identity yourself.", ] : [ - " NemoClaw cannot clear this recovery record until an OpenShell administrator establishes the exact sandbox identity.", + " NemoClaw cannot clear this recovery record until you manually confirm and remove the sandbox; inspect 'openshell sandbox list -o json' yourself.", ]), ]; } diff --git a/test/helpers/destroy-flow-test-harness.ts b/test/helpers/destroy-flow-test-harness.ts index b894b8799c1..4465230710a 100644 --- a/test/helpers/destroy-flow-test-harness.ts +++ b/test/helpers/destroy-flow-test-harness.ts @@ -210,40 +210,6 @@ export function createDestroyHarness(options: DestroyHarnessOptions = {}): Destr const resolve = requireSource("../../adapters/openshell/resolve.js"); const runtime = requireSource("../../adapters/openshell/runtime.js"); - const sanitizedCapture = requireSource("../../adapters/openshell/sanitized-capture.js"); - // Backs the delete-boundary retained-sandbox identity re-check - // (`inspectOpenShellSandboxIdentityFingerprint`, #10863). Mirrors the same - // `sandbox-${name}` id convention `sandboxListJson` uses for `sandbox list`. - // `sandbox-identity-cli.js` binds this by value at its own module load time - // (not a live namespace lookup), so it — and everything that transitively - // requires it — must be re-required fresh after the spy is installed. - vi.spyOn(sanitizedCapture, "captureSanitizedResolvedOpenshell").mockImplementation( - (args: unknown) => { - const argv = Array.isArray(args) ? args.map(String) : []; - if (argv[0] === "sandbox" && argv[1] === "get") { - if (!sandboxPresent) { - return { - status: 1, - output: "", - stdout: "", - stderr: "sandbox not found", - error: new Error("sandbox not found"), - }; - } - const name = argv.at(-1) ?? ""; - return { - status: 0, - output: `Id: sandbox-${name}\n`, - stdout: `Id: sandbox-${name}\n`, - stderr: "", - }; - } - return { status: 0, output: "", stdout: "", stderr: "" }; - }, - ); - delete require.cache[requireSource.resolve("../../adapters/openshell/sandbox-identity-cli.js")]; - delete require.cache[requireSource.resolve("./destroy-execution.js")]; - delete require.cache[requireSource.resolve("./destroy-preflight.js")]; const destroyGateway = requireSource("./destroy-gateway.js"); const credentialStore = requireSource("../../credentials/store.js"); const sandboxProviderCleanup = requireSource("../../onboard/sandbox-provider-cleanup.js"); From 88dc49fe0d98f6bb951e3e4fcf1145c72a6d5c46 Mon Sep 17 00:00:00 2001 From: Julie Yaunches Date: Tue, 8 Sep 2026 10:45:41 -0400 Subject: [PATCH 06/12] fix(sandbox): keep retained recovery fail closed Signed-off-by: Julie Yaunches --- docs/reference/commands.mdx | 17 ++-- docs/reference/troubleshooting.mdx | 10 +- src/lib/actions/sandbox/destroy-preflight.ts | 12 ++- .../destroy-retained-recovery-flow.test.ts | 97 ++++++++++++++++++- src/lib/actions/sandbox/destroy.ts | 27 ++++-- src/lib/onboard/cancel-rollback.test.ts | 7 +- src/lib/onboard/cancel-rollback.ts | 13 +-- .../onboard/created-sandbox-failure.test.ts | 38 ++------ src/lib/onboard/created-sandbox-failure.ts | 56 ++++------- .../created-sandbox-finalization.test.ts | 15 +-- .../onboard/created-sandbox-finalization.ts | 9 +- src/lib/onboard/dashboard.ts | 4 +- src/lib/onboard/entry-options.ts | 2 +- src/lib/onboard/lifecycle-contracts.md | 2 +- .../sandbox-create/orchestration.test.ts | 11 ++- .../onboard/sandbox-create/orchestration.ts | 12 +-- .../onboard/sandbox-gpu-create-flow.test.ts | 6 +- src/lib/onboard/sandbox-gpu-create-flow.ts | 8 +- .../sandbox-gpu-create-identity-gate.test.ts | 11 ++- .../onboard/sandbox-gpu-create-run-attempt.ts | 8 +- test/helpers/destroy-flow-test-harness.ts | 13 ++- 21 files changed, 226 insertions(+), 152 deletions(-) diff --git a/docs/reference/commands.mdx b/docs/reference/commands.mdx index 98750f3d6f7..86ff146f9d9 100644 --- a/docs/reference/commands.mdx +++ b/docs/reference/commands.mdx @@ -385,17 +385,18 @@ If onboarding cannot complete after sandbox creation, NemoClaw preserves the san When available, NemoClaw records and prints the create-attempt label as the exact `ai.nvidia.nemoclaw.create-attempt=` selector. When available, NemoClaw also records a durable identity fingerprint for recovery. Automatic and explicit resume, reuse, recreation, and fresh onboarding with that sandbox name remain blocked. -When the recovery record contains a durable identity fingerprint, run `$$nemoclaw destroy` to attempt identity-bound recovery. +Run `$$nemoclaw destroy` to check retained recovery and reconcile verified residual state. Use the result from `destroy` to choose the next action: -- If OpenShell still reports the sandbox present, `destroy` never deletes it automatically, even when the live sandbox's exact OpenShell id matches the retained recovery record's durable identity fingerprint. OpenShell exposes no atomic delete-by-identity primitive, so NemoClaw cannot bind a mutable-name delete to that identity evidence without a residual race: another OpenShell client could remove the retained sandbox and create a replacement under the same name between NemoClaw's last identity read and OpenShell processing the delete. `destroy` preserves the record and removes no resources; its output names the owner-scoped `openshell sandbox delete -g '' ` command. Compare `openshell sandbox list -o json` against the displayed create-attempt label yourself, and only once you have independently confirmed it is the same sandbox, run that command. After OpenShell confirms the sandbox is absent, run `$$nemoclaw destroy --yes` to reconcile the recovery record — the manual delete alone does not clear it. -- If OpenShell confirms the sandbox is absent, `destroy` verifies one retained recovery record and its immutable runtime identity. For Docker-backed sandboxes, it also verifies the immutable Docker sandbox identity and removes only qualified residual containers. It clears the matching recovery record only after verified cleanup. +- If OpenShell reports the sandbox present, `destroy` preserves the record and removes no resources. OpenShell exposes no atomic delete-by-identity primitive. Neither NemoClaw nor manual inspection can bind a later mutable-name delete to the durable identity fingerprint. You can inspect the owning gateway for diagnosis, but do not delete the sandbox by mutable name. Recovery remains blocked until the owning gateway reports the sandbox absent. +- If OpenShell cannot determine whether the sandbox is present, `destroy` reports unknown presence, removes no resources, and preserves the recovery record. Correct the OpenShell gateway or list failure, then rerun `destroy`. +- If OpenShell confirms the sandbox is absent, `destroy` selects one retained record from its registry generation, gateway, create-attempt evidence, and available identity fingerprint. Provider and Docker cleanup then apply their own immutable identity checks. `destroy` removes only qualified residual resources and clears the matching recovery record only after verified cleanup. - If the recovery record is ambiguous or immutable runtime-identity verification fails, cleanup stops and the record remains. For Docker-backed sandboxes, a foreign container, changed Docker identity, or failed Docker probe has the same fail-closed result. -If OpenShell did not return a durable identity fingerprint, `destroy` cannot complete recovery or prove a match for a live sandbox. A create-attempt label narrows the search, but you must identify and remove the exact sandbox yourself — inspect `openshell sandbox list -o json` and compare it against the displayed create-attempt label and any other available evidence. The label does not let NemoClaw retire the record without immutable identity authority. If neither a fingerprint nor a label is available, preserve the terminal output and ask an OpenShell administrator to identify the exact sandbox from gateway or controller evidence. If NemoClaw reports that it could not save recovery evidence, rerun `$$nemoclaw destroy --yes`; if recovery remains blocked, preserve the terminal output and follow the same escalation. +If OpenShell did not return a durable identity fingerprint, `destroy` cannot authorize deletion of a live sandbox. It can still select one recovery record that matches the immutable registry generation, gateway, and create-attempt evidence. It clears that record only after the owning gateway reports absence and verified residual cleanup succeeds. If the sandbox is present or presence is unknown, preserve the record and do not delete by mutable name. If NemoClaw could not save recovery evidence, preserve the registry entry and terminal output. -This fail-closed record keeps only the affected sandbox name unavailable. It is not retired from mutable-name absence alone, and this command does not accept externally supplied identity authority. +This fail-closed record keeps only the affected sandbox name unavailable. This command does not accept externally supplied identity authority. To onboard another sandbox while the record remains unresolved, supply a different explicit name: ```bash @@ -442,7 +443,7 @@ $$nemoclaw onboard --fresh --apf-interceptor --name my-apf-sandbox If post-create verification or native GPU fallback fails after OpenShell may have created the sandbox, NemoClaw preserves the incomplete sandbox because automatic deletion would use its mutable name. -Follow the [retained-sandbox recovery procedure](#recover-a-retained-sandbox); its OpenShell-absence, immutable runtime-identity, and administrator requirements also apply to APF creation. For Docker-backed APF creation, its Docker container-identity requirement also applies. +Follow the [retained-sandbox recovery procedure](#recover-a-retained-sandbox); its OpenShell-absence and immutable runtime-identity requirements also apply to APF creation. For Docker-backed APF creation, its Docker container-identity requirement also applies. This onboarding mode does not support `--resume` or `--recreate-sandbox`, regardless of whether sandbox creation began. After destroy completes, repeat the original command with `--fresh`. @@ -736,7 +737,7 @@ If you enable Telegram during onboarding, the wizard can also prompt for whether -If you cancel a brand-new onboarding run at the policy-tier selector or either policy-preset selector after sandbox creation, NemoClaw preserves the incomplete sandbox, registry entry, and onboarding session for identity-bound recovery. +If you cancel a brand-new onboarding run at the policy-tier selector or either policy-preset selector after sandbox creation, NemoClaw preserves the incomplete sandbox, registry entry, and onboarding session for retained recovery. NemoClaw reports the durable sandbox identity fingerprint when it is available. It does not run OpenShell's mutable-name deletion command because the name may now identify a replacement sandbox. Follow the [retained-sandbox recovery procedure](#recover-a-retained-sandbox) to reconcile this cancellation. @@ -744,7 +745,7 @@ A fresh run with a different explicit name can continue while the cancelled sand If you run onboarding again with the same sandbox name and choose a different inference provider or model, NemoClaw detects the drift and recreates the sandbox so the running agent config matches your selection. In interactive mode, the wizard asks for confirmation before delete and recreate. In non-interactive mode, NemoClaw recreates automatically when the stored selection is readable and differs. For managed Deep Agents Code sandboxes, NemoClaw also recreates when the live `dcode identity` selection is unreadable; other agent paths continue to reuse by default when their stored selection cannot be read. Set `NEMOCLAW_RECREATE_SANDBOX=1` to force recreation even when no drift is detected. -Before deleting an existing sandbox during recreation, NemoClaw backs up the workspace state declared by the selected agent profile and restores it into the new sandbox once it is live. This applies whether the existing sandbox is ready or marked not-ready, so cross-version upgrades that pass `NEMOCLAW_RECREATE_SANDBOX=1` no longer drop user files from the selected agent workspace. The behaviour matches `$$nemoclaw rebuild --force`. NemoClaw aborts the recreate when the backup cannot complete in full, including when individual state directories or files fail mid-backup, so failed entries are not silently dropped on delete. If restoration into the replacement does not complete, NemoClaw reports the failed directories, files, and restore reason that are available, preserves the snapshot, leaves the replacement unregistered, and exits nonzero. Run the owner-scoped `openshell sandbox delete -g '' ''` command that onboarding prints, then rerun the original onboarding command. If NemoClaw cannot identify the owning gateway, do not delete a same-name sandbox. Preserve the snapshot for manual recovery. Set `NEMOCLAW_RECREATE_WITHOUT_BACKUP=1` to skip the pre-recreate backup. The destination sandbox starts with a fresh workspace. +Before deleting an existing sandbox during recreation, NemoClaw backs up the workspace state declared by the selected agent profile and restores it into the new sandbox once it is live. This applies whether the existing sandbox is ready or marked not-ready, so cross-version upgrades that pass `NEMOCLAW_RECREATE_SANDBOX=1` no longer drop user files from the selected agent workspace. The behaviour matches `$$nemoclaw rebuild --force`. NemoClaw aborts the recreate when the backup cannot complete in full, including when individual state directories or files fail mid-backup, so failed entries are not silently dropped on delete. If restoration into the replacement does not complete, NemoClaw reports the failed directories, files, and restore reason that are available, preserves the snapshot, leaves the replacement unregistered, and exits nonzero. Recovery remains blocked while that unregistered sandbox exists. Do not delete it by mutable name. Preserve the snapshot, and run `$$nemoclaw destroy` to check for authoritative absence before you retry onboarding. Set `NEMOCLAW_RECREATE_WITHOUT_BACKUP=1` to skip the pre-recreate backup. The destination sandbox starts with a fresh workspace. Before deletion, onboarding prints a `Journaled replacement` diagnostic with the replacement identifier, recorded OpenShell gateway, and current phase. If the process stops after this point, a later same-target onboarding run continues the active replacement without requiring `--resume`. It accepts a ready same-name replacement only when the live identity and sandbox registry generation match the journal. It fails closed if the gateway, source, target, durable source registry fields, or replacement settings changed. diff --git a/docs/reference/troubleshooting.mdx b/docs/reference/troubleshooting.mdx index 8dcc5c1a2fd..20f8d082da1 100644 --- a/docs/reference/troubleshooting.mdx +++ b/docs/reference/troubleshooting.mdx @@ -1949,7 +1949,7 @@ export NEMOCLAW_SANDBOX_READY_TIMEOUT=600 $$nemoclaw onboard ``` -The variable accepts seconds and applies to the readiness wait only. When the ordinary create deadline expires, NemoClaw tries to delete the partially created sandbox. After successful cleanup, the output ends with `Retry: $$nemoclaw onboard`. If cleanup fails, NemoClaw instead reports that the failed sandbox could not be removed and prints `Manual cleanup: openshell sandbox delete ""`. +The variable accepts seconds and applies to the readiness wait only. When the ordinary create deadline expires, NemoClaw preserves the partially created sandbox because OpenShell deletion targets its mutable name. Same-name onboarding remains blocked while that sandbox exists. Run `$$nemoclaw destroy` to check retained recovery. Do not delete the sandbox by mutable name. @@ -2742,13 +2742,7 @@ Starting with NemoClaw v0.0.43, the standard installer handles the `/proc// If an older release fails direct GPU proof with that path and `Permission denied`, upgrade NemoClaw and rerun onboarding. -When inspection confirms that the failed sandbox remains, delete it with a command such as: - -```bash -openshell sandbox delete -``` - -Fix the NVIDIA Container Toolkit or CDI configuration reported in the diagnostics. Run the deletion command only after confirming that the pre-patch sandbox was not restored, then rerun onboarding. If you do not need GPU access inside the sandbox, rerun with `--no-sandbox-gpu`. +When the failed sandbox remains, recovery stays blocked because OpenShell deletion targets its mutable name. Inspection is diagnostic only and does not authorize deletion. Run `$$nemoclaw destroy` to check for authoritative absence. Fix the NVIDIA Container Toolkit or CDI configuration reported in the diagnostics. Rerun onboarding only after retained recovery completes. If you do not need GPU access inside the sandbox, use `--no-sandbox-gpu` when onboarding can continue. If sandbox creation fails with `CDI device injection failed: unresolvable CDI devices nvidia.com/gpu=all`, the OpenShell gateway tried `docker create --device nvidia.com/gpu=all` and Docker could not resolve the CDI spec. This injection happens inside the gateway, so `NEMOCLAW_DOCKER_GPU_PATCH=0` does not bypass it. Rerun with `--no-gpu`, or set `NEMOCLAW_SANDBOX_GPU=0` and resume onboarding. diff --git a/src/lib/actions/sandbox/destroy-preflight.ts b/src/lib/actions/sandbox/destroy-preflight.ts index c654bea5d97..ca1f95d874e 100644 --- a/src/lib/actions/sandbox/destroy-preflight.ts +++ b/src/lib/actions/sandbox/destroy-preflight.ts @@ -4,7 +4,6 @@ import os from "node:os"; import { buildSelectedOpenShellSubprocessEnv } from "../../adapters/openshell/command-argv"; -import { observeOpenShellSandboxIdentity } from "../../adapters/openshell/sandbox-presence"; import type { OpenShellRuntimeSelection } from "../../adapters/openshell/runtime-selection"; import { OPENSHELL_PROBE_TIMEOUT_MS } from "../../adapters/openshell/timeouts"; import { withModelRouterPortLifecycleLock } from "../../inference/gateway-route-mutation-lock"; @@ -25,6 +24,10 @@ import type { import type { SandboxEntry } from "../../state/registry"; import * as registry from "../../state/registry"; import { type DestroyRunOpenshell, selectGatewayForSandboxDestroy } from "./destroy-gateway"; +import { + classifyDestroySandboxPresence, + type DestroySandboxPresence, +} from "./destroy-presence"; import { getPersistedSandboxTargetGatewayName, getSandboxTargetGatewayName, @@ -40,6 +43,7 @@ export type SandboxDestroyPreflight = { selectedRunOpenshell: DestroyRunOpenshell; sandbox: SandboxEntry | null; sandboxConfirmedAbsent: boolean; + sandboxPresence?: DestroySandboxPresence; }; export function resolveSandboxDestroyRuntimeSelection( @@ -342,7 +346,7 @@ export function prepareSandboxDestroy( selectGatewayForSandboxDestroy(sandboxName, cleanupGatewayName, selectedRunOpenshell); process.env.OPENSHELL_GATEWAY = cleanupGatewayName; - const sandboxIdentityObservation = observeOpenShellSandboxIdentity( + const sandboxPresence = classifyDestroySandboxPresence( sandboxName, selectedRunOpenshell(["sandbox", "list", "-o", "json"], { ignoreError: true, @@ -350,14 +354,14 @@ export function prepareSandboxDestroy( timeout: OPENSHELL_PROBE_TIMEOUT_MS, }), ); - const sandboxConfirmedAbsent = sandboxIdentityObservation.kind === "absent"; return { cleanupGatewayName, runOpenshell, selectedRunOpenshell, sandbox, - sandboxConfirmedAbsent, + sandboxConfirmedAbsent: sandboxPresence === "absent", + sandboxPresence, ...(selectedCaptureOpenshell ? { selectedCaptureOpenshell } : {}), ...(runtimeSelection ? { runtimeSelection } : {}), }; diff --git a/src/lib/actions/sandbox/destroy-retained-recovery-flow.test.ts b/src/lib/actions/sandbox/destroy-retained-recovery-flow.test.ts index 2d8f6aed98e..51db963d657 100644 --- a/src/lib/actions/sandbox/destroy-retained-recovery-flow.test.ts +++ b/src/lib/actions/sandbox/destroy-retained-recovery-flow.test.ts @@ -32,6 +32,14 @@ function retainedRecoveryRecord(sandboxId = "sb-alpha"): RetainedSandboxRecovery }; } +function retainedRecoveryRecordWithoutIdentity(): RetainedSandboxRecoveryRecord { + return { + ...retainedRecoveryRecord(), + sandboxIdentityFingerprint: null, + identityWasUnavailable: true, + }; +} + describe("destroySandbox retained recovery flow", () => { let exitSpy: MockInstance; let originalGatewayEnv: string | undefined; @@ -217,7 +225,7 @@ describe("destroySandbox retained recovery flow", () => { ); expect(harness.errorSpy).toHaveBeenCalledWith( - expect.stringContaining("cannot safely bind an automatic delete to the retained record"), + expect.stringContaining("cannot bind a mutable-name delete to the retained record"), ); expect(harness.runOpenshellSpy).not.toHaveBeenCalledWith( ["sandbox", "delete", "alpha"], @@ -251,7 +259,7 @@ describe("destroySandbox retained recovery flow", () => { ); expect(harness.errorSpy).toHaveBeenCalledWith( - expect.stringContaining("cannot safely bind an automatic delete to the retained record"), + expect.stringContaining("cannot bind a mutable-name delete to the retained record"), ); expect(harness.runOpenshellSpy).not.toHaveBeenCalledWith( ["sandbox", "delete", "alpha"], @@ -291,10 +299,13 @@ describe("destroySandbox retained recovery flow", () => { ); expect(harness.errorSpy).toHaveBeenCalledWith( - expect.stringContaining("cannot safely bind an automatic delete to the retained record"), + expect.stringContaining("cannot bind a mutable-name delete to the retained record"), + ); + expect(harness.errorSpy.mock.calls.flat().join("\n")).not.toContain( + "openshell sandbox delete", ); expect(harness.errorSpy).toHaveBeenCalledWith( - expect.stringContaining(`openshell sandbox delete -g ${recovery.gatewayName} alpha`), + expect.stringContaining(`openshell sandbox list -g ${recovery.gatewayName} -o json`), ); expect(harness.runOpenshellSpy).not.toHaveBeenCalledWith( ["sandbox", "delete", "alpha"], @@ -304,6 +315,84 @@ describe("destroySandbox retained recovery flow", () => { }, ); + it( + "reconciles a lone recovery record without identity after OpenShell confirms absence (#10863)", + { timeout: 30_000 }, + async () => { + const recovery = retainedRecoveryRecordWithoutIdentity(); + const harness = createDestroyHarness({ + sandboxPresent: false, + dockerRunResult: { status: 0, stdout: "" }, + registryEntryOverrides: { + lifecycleGeneration: recovery.lifecycleGeneration!, + }, + retainedRecoveryRecords: [recovery], + }); + + await expect(harness.destroySandbox("alpha", { yes: true })).resolves.toBeUndefined(); + + expect(harness.runOpenshellSpy).not.toHaveBeenCalledWith( + ["sandbox", "delete", "alpha"], + expect.anything(), + ); + expect(harness.resolveRetainedSandboxRecoverySpy).toHaveBeenCalledWith(recovery); + expect(harness.removeSandboxSpy).toHaveBeenCalledWith("alpha"); + expect(exitSpy).not.toHaveBeenCalled(); + }, + ); + + it.each([ + ["present", undefined, "reports a sandbox present"], + [ + "a failed list command", + { status: 1, stdout: "", stderr: "" }, + "could not determine", + ], + [ + "list diagnostics", + { status: 0, stdout: "[]", stderr: "gateway unavailable" }, + "could not determine", + ], + [ + "malformed list output", + { status: 0, stdout: "not-json", stderr: "" }, + "could not determine", + ], + ])( + "preserves a recovery record without identity when OpenShell reports %s (#10863)", + { timeout: 30_000 }, + async (_case, sandboxListResult, expectedPresence) => { + const recovery = retainedRecoveryRecordWithoutIdentity(); + const harness = createDestroyHarness({ + dockerRunResult: { status: 0, stdout: "" }, + registryEntryOverrides: { + lifecycleGeneration: recovery.lifecycleGeneration!, + }, + retainedRecoveryRecords: [recovery], + ...(sandboxListResult ? { sandboxListResult } : {}), + }); + + await expect(harness.destroySandbox("alpha", { yes: true })).rejects.toThrow( + "process.exit(1)", + ); + + expect(harness.runOpenshellSpy).not.toHaveBeenCalledWith( + ["sandbox", "delete", "alpha"], + expect.anything(), + ); + expect(harness.dockerRunSpy).not.toHaveBeenCalledWith( + ["rm", "-f", expect.any(String)], + expect.anything(), + ); + expect(harness.resolveRetainedSandboxRecoverySpy).not.toHaveBeenCalled(); + expect(harness.removeSandboxSpy).not.toHaveBeenCalled(); + const output = harness.errorSpy.mock.calls.flat().join("\n"); + expect(output).not.toContain("openshell sandbox delete"); + expect(output).toContain("Preserve"); + expect(output).toContain(expectedPresence); + }, + ); + it( "finishes retained cleanup after OpenShell already removed the sandbox (#10547)", { timeout: 30_000 }, diff --git a/src/lib/actions/sandbox/destroy.ts b/src/lib/actions/sandbox/destroy.ts index 5b0d01db296..9fe649a25b9 100644 --- a/src/lib/actions/sandbox/destroy.ts +++ b/src/lib/actions/sandbox/destroy.ts @@ -100,9 +100,7 @@ function selectRetainedSandboxRecoveryAuthority( sandbox: registry.SandboxEntry | null, records: readonly onboardSession.RetainedSandboxRecoveryRecord[], ): onboardSession.RetainedSandboxRecoveryRecord | null { - const candidates = records.filter( - (record) => record.sandboxName === sandboxName && record.sandboxIdentityFingerprint !== null, - ); + const candidates = records.filter((record) => record.sandboxName === sandboxName); if (!sandbox) { // Once resource cleanup has removed the registry row, a retry must still // select the lone durable record so the later Docker proof can confirm @@ -112,10 +110,11 @@ function selectRetainedSandboxRecoveryAuthority( if (candidates.length === 0) return null; const observation = observeDestroyContainerIdentity(sandboxName); const observedMatches = candidates.filter((record) => { + if (record.sandboxIdentityFingerprint === null) return false; const verdict = classifyDestroyContainerIdentity( sandboxName, observation, - record.sandboxIdentityFingerprint!, + record.sandboxIdentityFingerprint, ); return ( verdict.status === "recovery" || (verdict.status === "clear" && verdict.identity !== null) @@ -133,7 +132,8 @@ function selectRetainedSandboxRecoveryAuthority( record.gatewayName === pending.gatewayName && record.gatewayPort === pending.gatewayPort && record.lifecycleGeneration === pending.lifecycleGeneration && - record.sandboxIdentityFingerprint === pending.sandboxIdentityFingerprint && + (record.sandboxIdentityFingerprint === null || + record.sandboxIdentityFingerprint === pending.sandboxIdentityFingerprint) && (pending.createAttemptNonce === undefined || record.createAttemptNonce === pending.createAttemptNonce) ); @@ -142,7 +142,8 @@ function selectRetainedSandboxRecoveryAuthority( record.gatewayName === sandbox.gatewayName && record.gatewayPort === sandbox.gatewayPort && record.lifecycleGeneration === sandbox.lifecycleGeneration && - record.sandboxIdentityFingerprint === sandbox.lifecycleLiveIdentityFingerprint + (record.sandboxIdentityFingerprint === null || + record.sandboxIdentityFingerprint === sandbox.lifecycleLiveIdentityFingerprint) ); }; const matching = candidates.filter(matchesRegistryAuthority); @@ -743,8 +744,9 @@ async function destroySandboxUnlocked( selectedRunOpenshell: cleanupRunOpenshell, sandbox, sandboxConfirmedAbsent, + sandboxPresence = sandboxConfirmedAbsent ? "absent" : "unknown", } = destroyPreflight; - if (retainedRecoveryAuthority && !sandboxConfirmedAbsent) { + if (retainedRecoveryAuthority && sandboxPresence !== "absent") { // OpenShell has no atomic delete-by-identity primitive: it exposes no // way to bind a mutable-name delete to the retained record's immutable // sandbox id/resource version. Even a fresh identity read immediately @@ -752,10 +754,15 @@ async function destroySandboxUnlocked( // OpenShell client removes the retained sandbox and creates a // replacement under the same name between that read and OpenShell // processing the delete (#10863). Automatic deletion of a live retained - // sandbox is therefore always fail-closed; a human must confirm identity - // out-of-band and run the exact OpenShell command themselves. + // sandbox is therefore always fail-closed. Inspection cannot authorize a + // later mutable-name delete, so the recovery record remains unresolved + // until OpenShell can prove absence through the owning gateway. + const presenceDetail = + sandboxPresence === "present" + ? "OpenShell reports a sandbox present under this name." + : "OpenShell could not determine whether a sandbox is present under this name."; console.error( - ` Refusing to automatically delete retained sandbox '${sandboxName}': OpenShell reports a sandbox present under this name, but NemoClaw cannot safely bind an automatic delete to the retained record (create-attempt label '${retainedRecoveryAuthority.createAttemptNonce}') without an atomic OpenShell delete-by-identity primitive. No sandbox resources were removed. Compare 'openshell sandbox list -o json' against the create-attempt label above; once you have confirmed it is the same sandbox, run 'openshell sandbox delete -g ${retainedRecoveryAuthority.gatewayName} ${sandboxName}' yourself. After OpenShell confirms the sandbox is absent, rerun '${CLI_NAME} ${sandboxName} destroy --yes' to reconcile its verified Docker containers and recovery record.`, + ` Refusing to delete retained sandbox '${sandboxName}': ${presenceDetail} NemoClaw cannot bind a mutable-name delete to the retained record (create-attempt label '${retainedRecoveryAuthority.createAttemptNonce}') without an atomic OpenShell delete-by-identity primitive. No sandbox resources were removed. Preserve the recovery record. Inspect 'openshell sandbox list -g ${retainedRecoveryAuthority.gatewayName} -o json' for diagnosis only; do not run mutable-name deletion. Recovery remains blocked until the owning gateway reports the sandbox absent. Then rerun '${CLI_NAME} ${sandboxName} destroy --yes' to reconcile verified residual resources and the recovery record.`, ); preparedManagedLlamaCppCleanup?.abort(); requestSandboxDestroyExit(1); diff --git a/src/lib/onboard/cancel-rollback.test.ts b/src/lib/onboard/cancel-rollback.test.ts index f4ff3db0d51..e172b080ced 100644 --- a/src/lib/onboard/cancel-rollback.test.ts +++ b/src/lib/onboard/cancel-rollback.test.ts @@ -282,7 +282,9 @@ describe("installSandboxCancelRollback", () => { const guidance = log.mock.calls.flat().join("\n"); expect(guidance).toContain("identity fingerprint is unavailable"); - expect(guidance).toContain("openshell sandbox list -o json"); + expect(guidance).toContain("can clear the recovery record only after OpenShell confirms"); + expect(guidance).not.toContain("identify and remove"); + expect(guidance).not.toContain("openshell sandbox delete"); }); }); @@ -309,7 +311,8 @@ describe("buildCancelRollbackMessage", () => { expect(message).toContain("preserved incomplete sandbox 'sb'"); expect(message).toContain(SANDBOX_FINGERPRINT); expect(message).toContain(RECOVERY_CONTEXT.createAttemptNonce); - expect(message).toContain("identity-bound inspection, recovery, or removal"); + expect(message).toContain("retained recovery evidence"); + expect(message).toContain("does not authorize deletion by mutable name"); expect(message).not.toContain("openshell sandbox delete"); expect(message).not.toContain("cannot delete it by immutable identity"); }); diff --git a/src/lib/onboard/cancel-rollback.ts b/src/lib/onboard/cancel-rollback.ts index 288e0dcc612..320a3236903 100644 --- a/src/lib/onboard/cancel-rollback.ts +++ b/src/lib/onboard/cancel-rollback.ts @@ -69,11 +69,11 @@ export function buildCancelRollbackMessage( ...(sandboxIdentityFingerprint ? [ ` Durable sandbox identity fingerprint: ${sandboxIdentityFingerprint}`, - " Preserve this fingerprint for identity-bound inspection, recovery, or removal.", + " Preserve this fingerprint as retained recovery evidence; it does not authorize deletion by mutable name.", ] : [ " Its durable identity fingerprint is unavailable; preserve the registry and onboarding recovery state.", - " Inspect 'openshell sandbox list -o json' yourself to identify and remove the sandbox; NemoClaw cannot verify its identity without a recorded fingerprint.", + " NemoClaw cannot verify this sandbox identity or authorize its removal without a recorded fingerprint.", ]), " NemoClaw did not run OpenShell's mutable-name deletion command because the name may now identify a replacement sandbox.", " Do not delete the sandbox by mutable sandbox name.", @@ -82,11 +82,12 @@ export function buildCancelRollbackMessage( ? [ ` Run '${cliName()} ${sandboxName} destroy'. If OpenShell confirms the retained sandbox absent, destroy removes only verified residual containers and can clear the matching recovery record.`, recoveryContext - ? " If it is still live, destroy refuses automatic deletion and prints the exact manual removal command once you confirm identity against the create-attempt label above yourself." - : " If it is still live, preserve the displayed fingerprint; destroy refuses automatic deletion and prints manual removal guidance once you confirm identity yourself.", + ? " If it is still live or presence is unknown, destroy refuses deletion. Inspect the owning gateway for diagnosis only; do not delete by mutable name." + : " If it is still live or presence is unknown, preserve the displayed fingerprint; destroy refuses deletion and preserves the recovery record.", ] : [ - " NemoClaw cannot clear this recovery record until you manually confirm and remove the sandbox; inspect 'openshell sandbox list -o json' yourself.", + ` Run '${cliName()} ${sandboxName} destroy'. It can clear the recovery record only after OpenShell confirms the sandbox absent and residual cleanup succeeds.`, + " If the sandbox is present or presence is unknown, preserve the recovery record and do not delete by mutable name.", ]), ]; } @@ -208,7 +209,7 @@ export function createSandboxCancelRollback( guidanceReported = true; if (recoveryPersistenceFailed) { deps.log( - " NemoClaw could not save the onboarding recovery record; preserve the registry entry and exact sandbox identity for administrator recovery.", + " NemoClaw could not save the onboarding recovery record; preserve the registry entry and terminal output. Do not delete the sandbox by mutable name.", ); } for (const line of buildCancelRollbackMessage( diff --git a/src/lib/onboard/created-sandbox-failure.test.ts b/src/lib/onboard/created-sandbox-failure.test.ts index 69fe3edc5d2..fd2f629ae10 100644 --- a/src/lib/onboard/created-sandbox-failure.test.ts +++ b/src/lib/onboard/created-sandbox-failure.test.ts @@ -163,7 +163,6 @@ function readinessDeps( printReadinessFailure: vi.fn(), printCreateFailureDiagnostics: vi.fn(), printDockerGpuReadinessFailure: vi.fn(), - deleteSandbox: vi.fn(() => ({ status: 0 })), cliName: vi.fn(() => "nemoclaw"), error: vi.fn(), exitProcess: vi.fn((code: number): never => { @@ -202,33 +201,12 @@ function expectReceiptBlock( } describe("reportSandboxReadinessFailure", () => { - it("deletes the failed sandbox on the non-GPU path and exits 1", () => { + it("preserves the failed sandbox on the non-GPU path and exits 1", () => { const deps = readinessDeps(); expect(() => reportSandboxReadinessFailure(readinessOptions(), deps)).toThrow(ExitSignal); expect(deps.printReadinessFailure).toHaveBeenCalledWith(NOT_READY, "alpha", 300); expect(deps.printCreateFailureDiagnostics).toHaveBeenCalledWith("alpha", { backupPath: null }); - expect(deps.deleteSandbox).toHaveBeenCalledWith("alpha"); expect(deps.printDockerGpuReadinessFailure).not.toHaveBeenCalled(); - expectReceiptBlock(deps, [ - " Sandbox lifecycle receipt:", - " state: created_but_not_ready", - " sandbox: alpha", - " readiness_gate: sandbox_list:not_ready_timeout", - " readiness_reason: timeout", - " create_stream_status: 0", - " timeout_seconds: 300", - " terminal_resolution: timed_out_deleted", - ]); - expect(deps.error).toHaveBeenCalledWith( - " Deleted sandbox 'alpha' after the readiness gate failed; retry will recreate it.", - ); - expect(deps.error).toHaveBeenCalledWith(" Retry: nemoclaw onboard"); - expect(deps.exitProcess).toHaveBeenCalledWith(1); - }); - - it("surfaces manual cleanup when deletion fails", () => { - const deps = readinessDeps({ deleteSandbox: vi.fn(() => ({ status: 1 })) }); - expect(() => reportSandboxReadinessFailure(readinessOptions(), deps)).toThrow(ExitSignal); expectReceiptBlock(deps, [ " Sandbox lifecycle receipt:", " state: created_but_not_ready", @@ -240,9 +218,10 @@ describe("reportSandboxReadinessFailure", () => { " terminal_resolution: timed_out_retained", ]); expect(deps.error).toHaveBeenCalledWith( - " Could not remove the failed sandbox. Manual cleanup:", + " Recovery remains blocked while sandbox 'alpha' exists. Do not delete it by mutable name; run 'nemoclaw alpha destroy' to check for authoritative absence.", ); - expect(deps.error).toHaveBeenCalledWith(' openshell sandbox delete "alpha"'); + expect(errorLines(deps)).not.toContain(" Retry: nemoclaw onboard"); + expect(deps.exitProcess).toHaveBeenCalledWith(1); }); it("defers cleanup to the Docker-GPU patch and never deletes the sandbox", () => { @@ -251,7 +230,6 @@ describe("reportSandboxReadinessFailure", () => { reportSandboxReadinessFailure(readinessOptions({ useDockerGpuPatch: true }), deps), ).toThrow(ExitSignal); expect(deps.printDockerGpuReadinessFailure).toHaveBeenCalledTimes(1); - expect(deps.deleteSandbox).not.toHaveBeenCalled(); expectReceiptBlock(deps, [ " Sandbox lifecycle receipt:", " state: created_but_not_ready", @@ -287,12 +265,12 @@ describe("reportSandboxReadinessFailure", () => { " readiness_reason: terminal_failure_phase", " create_stream_status: 0", " timeout_seconds: 300", - " terminal_resolution: terminal_failure_deleted", + " terminal_resolution: terminal_failure_retained", ]); }); - it("reports retained cleanup for terminal readiness failures when delete fails", () => { - const deps = readinessDeps({ deleteSandbox: vi.fn(() => ({ status: 1 })) }); + it("reports retained cleanup for terminal readiness failures", () => { + const deps = readinessDeps(); expect(() => reportSandboxReadinessFailure( readinessOptions({ @@ -342,7 +320,7 @@ describe("reportSandboxReadinessFailure", () => { " readiness_reason: terminal_failure_phase", " create_stream_status: 0", " timeout_seconds: 300", - " terminal_resolution: terminal_failure_deleted", + " terminal_resolution: terminal_failure_retained", ]); }); diff --git a/src/lib/onboard/created-sandbox-failure.ts b/src/lib/onboard/created-sandbox-failure.ts index f0c136cd1fd..8b47c4f24c9 100644 --- a/src/lib/onboard/created-sandbox-failure.ts +++ b/src/lib/onboard/created-sandbox-failure.ts @@ -18,7 +18,7 @@ export function formatRetainedSandboxRecoveryMessage(input: { createAttemptEvidence + `Sandbox '${input.sandboxName}' reached Ready before OpenShell returned one exact durable create identity. Gateway '${input.gatewayName}'. ` + "OpenShell did not return one exact durable sandbox identity for this create attempt. " + - "Do not delete a sandbox by mutable name; preserve it until an OpenShell administrator resolves the create-attempt label to one sandbox." + `Do not delete the sandbox by mutable name. Run '${cliName()} ${input.sandboxName} destroy'; it can clear retained recovery only after OpenShell confirms absence.` ); } return ( @@ -26,9 +26,9 @@ export function formatRetainedSandboxRecoveryMessage(input: { `Durable sandbox identity fingerprint: ${input.sandboxIdentityFingerprint}. ` + `NemoClaw stopped before owning-gateway publication and identity verification completed for sandbox '${input.sandboxName}' through gateway '${input.gatewayName}'. ` + `Do not delete the sandbox by mutable name. Run '${cliName()} ${input.sandboxName} destroy'. ` + - "If OpenShell reports the sandbox present, the command removes nothing and preserves the recovery record. " + - "Give the create-attempt label to an OpenShell administrator for identity-bound removal. " + - `After OpenShell confirms removal, run '${cliName()} ${input.sandboxName} destroy --yes' to reconcile the recovery record.` + "If OpenShell reports the sandbox present or cannot determine presence, the command removes nothing and preserves the recovery record. " + + "Inspection is diagnostic only and does not authorize mutable-name deletion. " + + `Recovery remains blocked while the sandbox is present or presence is unknown. Rerun '${cliName()} ${input.sandboxName} destroy --yes' after the owning gateway reports absence.` ); } @@ -109,7 +109,6 @@ export type SandboxReadinessFailureReportDeps = { ): void; printCreateFailureDiagnostics(sandboxName: string, options: { backupPath: string | null }): void; printDockerGpuReadinessFailure(): void; - deleteSandbox(sandboxName: string): { status: number | null }; cliName(): string; error(message: string): void; exitProcess(code: number): never; @@ -171,8 +170,8 @@ function formatCreatedSandboxReadinessReceipt(options: { /** * Report a sandbox that never reached Ready: print the readiness failure and - * create diagnostics, then either defer cleanup to the Docker-GPU patch or - * delete the failed sandbox so a same-name retry does not collide, and exit. + * create diagnostics, then preserve the failed sandbox for retained recovery + * unless the Docker-GPU patch owns its identity-bound cleanup, and exit. */ export function reportSandboxReadinessFailure( options: SandboxReadinessFailureReportOptions, @@ -195,37 +194,22 @@ export function reportSandboxReadinessFailure( } deps.printDockerGpuReadinessFailure(); } else { - // Clean up non-GPU failures after preserving local diagnostics so the - // next onboard retry with the same name does not fail on "sandbox already exists". - const delResult = deps.deleteSandbox(options.sandboxName); - if (delResult.status === 0) { - for (const line of formatCreatedSandboxReadinessReceipt({ - sandboxName: options.sandboxName, - readiness: options.readiness, - createStatus: options.createStatus, - timeoutSecs: options.timeoutSecs, - terminalResolution: readinessTerminalResolution(options.readiness, true), - })) { - deps.error(line); - } - deps.error( - ` Deleted sandbox '${options.sandboxName}' after the readiness gate failed; retry will recreate it.`, - ); - } else { - for (const line of formatCreatedSandboxReadinessReceipt({ - sandboxName: options.sandboxName, - readiness: options.readiness, - createStatus: options.createStatus, - timeoutSecs: options.timeoutSecs, - terminalResolution: readinessTerminalResolution(options.readiness, false), - })) { - deps.error(line); - } - deps.error(" Could not remove the failed sandbox. Manual cleanup:"); - deps.error(` openshell sandbox delete "${options.sandboxName}"`); + for (const line of formatCreatedSandboxReadinessReceipt({ + sandboxName: options.sandboxName, + readiness: options.readiness, + createStatus: options.createStatus, + timeoutSecs: options.timeoutSecs, + terminalResolution: readinessTerminalResolution(options.readiness, false), + })) { + deps.error(line); } + deps.error( + ` Recovery remains blocked while sandbox '${options.sandboxName}' exists. Do not delete it by mutable name; run '${deps.cliName()} ${options.sandboxName} destroy' to check for authoritative absence.`, + ); + } + if (options.useDockerGpuPatch) { + deps.error(` Retry: ${deps.cliName()} onboard`); } - deps.error(` Retry: ${deps.cliName()} onboard`); const exitCode = options.createStatus === 0 ? 1 : options.createStatus; return deps.exitProcess(exitCode); } diff --git a/src/lib/onboard/created-sandbox-finalization.test.ts b/src/lib/onboard/created-sandbox-finalization.test.ts index 10b28c20d04..a682f60b084 100644 --- a/src/lib/onboard/created-sandbox-finalization.test.ts +++ b/src/lib/onboard/created-sandbox-finalization.test.ts @@ -108,8 +108,9 @@ describe("new sandbox cancellation recovery", () => { expect(guidance).toContain("Sandbox 'new-sandbox' was created on gateway 'nemoclaw'"); expect(guidance).toContain("registry entry and onboarding session were preserved"); expect(guidance).toContain("Do not delete the sandbox by mutable sandbox name"); - expect(guidance).toContain("establish the exact live durable identity before removal"); - expect(guidance).toContain("add --fresh, and use a new sandbox name"); + expect(guidance).toContain("can clear retained recovery only after OpenShell confirms"); + expect(guidance).toContain("use a different explicit sandbox name"); + expect(guidance).not.toContain("administrator"); expect(runFile).not.toHaveBeenCalled(); expect(armCancelRollback).not.toHaveBeenCalled(); expect(markCancellationRecovery).toHaveBeenCalledOnce(); @@ -649,7 +650,7 @@ describe("created DCode sandbox finalization", () => { expect(register).not.toHaveBeenCalled(); expect(error).toHaveBeenCalledWith(expect.stringContaining("sandbox still exists")); expect(error).toHaveBeenCalledWith(expect.stringContaining("rebuild is unsafe")); - expect(error).toHaveBeenCalledWith(expect.stringContaining("Verify its durable identity")); + expect(error).toHaveBeenCalledWith(expect.stringContaining("Recovery remains blocked while this sandbox exists")); expect(error.mock.calls.flat().join("\n")).not.toContain("openshell sandbox delete"); expect(error).toHaveBeenCalledWith(expect.stringContaining("nemoclaw onboard")); }); @@ -710,7 +711,7 @@ describe("created DCode sandbox finalization", () => { " NemoClaw left unregistered sandbox 'dcode' in place because OpenShell can delete it only by mutable name.", ); expect(error).toHaveBeenCalledWith( - " Verify its durable identity before manual cleanup; do not act by name alone.", + " Recovery remains blocked while this sandbox exists. Do not delete it by mutable name; run 'nemoclaw dcode destroy' to check for authoritative absence.", ); expect(error.mock.calls.flat().join("\n")).not.toContain("openshell sandbox delete"); expect(error).toHaveBeenCalledWith( @@ -1054,7 +1055,7 @@ describe("created OpenClaw sandbox finalization", () => { expect(error).toHaveBeenCalledWith( " State was not restored and registry metadata was not updated.", ); - expect(error).toHaveBeenCalledWith(expect.stringContaining("Verify its durable identity")); + expect(error).toHaveBeenCalledWith(expect.stringContaining("Recovery remains blocked while this sandbox exists")); expect(error.mock.calls.flat().join("\n")).not.toContain("openshell sandbox delete"); expect(error).toHaveBeenCalledWith(" Manual recovery: /tmp/managed-openclaw-backup"); }); @@ -1101,7 +1102,7 @@ describe("created OpenClaw sandbox finalization", () => { expect(error).toHaveBeenCalledWith( " State was not restored and registry metadata was not updated.", ); - expect(error).toHaveBeenCalledWith(expect.stringContaining("Verify its durable identity")); + expect(error).toHaveBeenCalledWith(expect.stringContaining("Recovery remains blocked while this sandbox exists")); expect(error.mock.calls.flat().join("\n")).not.toContain("openshell sandbox delete"); expect(error).toHaveBeenCalledWith( " Then rerun the original `nemoclaw onboard --from ` command.", @@ -1157,7 +1158,7 @@ describe("created OpenClaw sandbox finalization", () => { expect(error).toHaveBeenCalledWith( expect.stringContaining(sandboxState.OPENCLAW_IMAGE_PLUGIN_PROVENANCE_RESTORE_ERROR), ); - expect(error).toHaveBeenCalledWith(expect.stringContaining("Verify its durable identity")); + expect(error).toHaveBeenCalledWith(expect.stringContaining("Recovery remains blocked while this sandbox exists")); expect(error.mock.calls.flat().join("\n")).not.toContain("openshell sandbox delete"); expect(error).toHaveBeenCalledWith( " Then rerun the original `nemoclaw onboard --from ` command.", diff --git a/src/lib/onboard/created-sandbox-finalization.ts b/src/lib/onboard/created-sandbox-finalization.ts index 818c45cbcf9..de81ac8c984 100644 --- a/src/lib/onboard/created-sandbox-finalization.ts +++ b/src/lib/onboard/created-sandbox-finalization.ts @@ -22,6 +22,7 @@ import { type RecreatedSandboxRestoreOptions, type RestoreResult, } from "../state/sandbox"; +import { cliName } from "./branding"; import { createDcodeSelectionDriftReader } from "./dcode-selection-drift"; import { restoreDefaultAfterRecreate } from "./default-preservation"; import * as dockerGpuLocalInference from "./docker-gpu-local-inference"; @@ -300,8 +301,8 @@ export function completeOrdinaryOnboardSandboxCreation( ` Sandbox '${input.sandboxName}' was created on gateway '${deps.gatewayName}', but NemoClaw could not verify its durable identity.`, " The sandbox registry entry and onboarding session were preserved for recovery.", " Do not delete the sandbox by mutable sandbox name.", - " Ask an OpenShell administrator to establish the exact live durable identity before removal.", - " After confirmed identity-bound removal, rerun the original onboarding command with the same required inputs, add --fresh, and use a new sandbox name.", + ` Run '${cliName()} ${input.sandboxName} destroy'. It can clear retained recovery only after OpenShell confirms the sandbox absent.`, + " Until recovery completes, use a different explicit sandbox name for new onboarding.", ]) { console.error(line); } @@ -808,7 +809,9 @@ export function finalizeCreatedSandbox( deps.error( ` NemoClaw left unregistered sandbox '${options.sandboxName}' in place because OpenShell can delete it only by mutable name.`, ); - deps.error(" Verify its durable identity before manual cleanup; do not act by name alone."); + deps.error( + ` Recovery remains blocked while this sandbox exists. Do not delete it by mutable name; run '${cliName()} ${options.sandboxName} destroy' to check for authoritative absence.`, + ); }; let freshOpenClawImagePluginInstalls: readonly OpenClawImagePluginInstall[] | undefined; if (options.discoverOpenClawImagePluginInstalls === true) { diff --git a/src/lib/onboard/dashboard.ts b/src/lib/onboard/dashboard.ts index 4062356bb4b..e29213aa714 100644 --- a/src/lib/onboard/dashboard.ts +++ b/src/lib/onboard/dashboard.ts @@ -373,8 +373,8 @@ export function createOnboardDashboardHelpers(deps: OnboardDashboardDeps): Onboa ]; if (owningGateway) { lines.push( - " Verify the sandbox identity, then clean up manually:", - ` openshell sandbox delete -g ${JSON.stringify(owningGateway)} ${JSON.stringify(sandboxName)}`, + ` Recovery remains blocked while gateway ${JSON.stringify(owningGateway)} reports this sandbox present.`, + ` Do not delete it by mutable name; run 'nemoclaw ${sandboxName} destroy' to check for authoritative absence.`, ); } else { lines.push(" The owning OpenShell gateway is unknown. Do not delete a same-name sandbox."); diff --git a/src/lib/onboard/entry-options.ts b/src/lib/onboard/entry-options.ts index 3b81e4dd41d..648c0f54fb2 100644 --- a/src/lib/onboard/entry-options.ts +++ b/src/lib/onboard/entry-options.ts @@ -477,7 +477,7 @@ export function resolveOnboardEntryOptions( " Onboarding cannot replace the recovery-only session because its independent retained sandbox recovery record is unavailable.", ); deps.error( - " Preserve the session and registry state for identity-bound administrator recovery.", + " Preserve the session, registry state, and terminal output. Do not delete the sandbox by mutable name.", ); deps.exitProcess(1); } diff --git a/src/lib/onboard/lifecycle-contracts.md b/src/lib/onboard/lifecycle-contracts.md index 912e3e18d46..b7906341ef7 100644 --- a/src/lib/onboard/lifecycle-contracts.md +++ b/src/lib/onboard/lifecycle-contracts.md @@ -123,7 +123,7 @@ runtime mutation | Journey and entry | Desired state, planning, and assembly | Visible and destructive boundaries | Checkpoint and secret boundary | Compensation, coverage, and gaps | |---|---|---|---|---| -| **New interactive or non-interactive onboard** — `onboard()` and `resolveOnboardEntryOptions` | Current flags, environment, and prompts. `MessagingWorkflowPlanner.buildPlan`, `prepareSandboxMessagingPreflight`, resource-profile selection, `resolveSandboxCreateIntent`, and `materializeSandboxCreatePlan` assemble policy, provider, package, resource, host-forward, and runtime-setup contributions. Non-interactive mode replaces prompts with defaults or hard aborts. | Consent/session/lock setup and preflight can persist local state, install OpenShell, or clean stale gateway artifacts before the gateway handler. Gateway reuse/recovery/start is the first provider-routing effect; inference-provider upserts follow. For OpenClaw, messaging selection and plan reconciliation complete before web-search or messaging provider registration. Each validated provider group is then created or updated and checkpointed before resource selection. A name with no live sandbox has no sandbox-destructive boundary; an existing target enters the recreate contract below. | Whole-step session plus machine snapshot. OpenClaw adds narrow checkpoints after each completed secret-free sandbox prompt group; sandbox registry registration is deferred until readiness and live validation. The session stores credential environment names, redacted endpoint metadata, legacy-value digests, and non-secret names of web-search and messaging providers registered for resume; real values remain process- or gateway-bound. | Readiness, post-create policy verification, dashboard forwarding, and cancellation failures preserve the live sandbox and an independent identity-bound recovery record. A later `destroy` refuses mutable-name deletion while that sandbox is live. After administrator identity-bound removal, destroy uses the record to qualify immutable runtime identity and, for Docker-backed sandboxes, exact container identities before residual cleanup and record retirement. A different explicit sandbox name starts a fresh session without changing the retained record. Exact provider-owned GPU cleanup can proceed through its owner receipt. NemoClaw attempts to remove temporary policy and build-context sources and reports cleanup failures with the onboarding error; post-create failures retain recovery state. Cancellation before sandbox creation can leave the session resumable. Shared inference providers remain gateway configuration and are not sandbox cleanup targets. Coverage: `transition-traces.test.ts`, `sandbox-create-intent-boundary.test.ts`, `sandbox-create-plan.test.ts`, and the focused cancellation, readiness, GPU cleanup, dashboard, policy-authority, destroy, and retained-recovery tests. Gap: gateway upserts can outlive a failed or interrupted create. | +| **New interactive or non-interactive onboard** — `onboard()` and `resolveOnboardEntryOptions` | Current flags, environment, and prompts. `MessagingWorkflowPlanner.buildPlan`, `prepareSandboxMessagingPreflight`, resource-profile selection, `resolveSandboxCreateIntent`, and `materializeSandboxCreatePlan` assemble policy, provider, package, resource, host-forward, and runtime-setup contributions. Non-interactive mode replaces prompts with defaults or hard aborts. | Consent/session/lock setup and preflight can persist local state, install OpenShell, or clean stale gateway artifacts before the gateway handler. Gateway reuse/recovery/start is the first provider-routing effect; inference-provider upserts follow. For OpenClaw, messaging selection and plan reconciliation complete before web-search or messaging provider registration. Each validated provider group is then created or updated and checkpointed before resource selection. A name with no live sandbox has no sandbox-destructive boundary; an existing target enters the recreate contract below. | Whole-step session plus machine snapshot. OpenClaw adds narrow checkpoints after each completed secret-free sandbox prompt group; sandbox registry registration is deferred until readiness and live validation. The session stores credential environment names, redacted endpoint metadata, legacy-value digests, and non-secret names of web-search and messaging providers registered for resume; real values remain process- or gateway-bound. | Readiness, post-create policy verification, dashboard forwarding, and cancellation failures preserve the live sandbox and an independent identity-bound recovery record. A later `destroy` refuses mutable-name deletion while that sandbox is live. Recovery remains blocked while the sandbox is present or its presence is unknown. After OpenShell authoritatively reports absence, destroy uses the record to qualify immutable runtime identity and, for Docker-backed sandboxes, exact container identities before residual cleanup and record retirement. A different explicit sandbox name starts a fresh session without changing the retained record. Exact provider-owned GPU cleanup can proceed through its owner receipt. NemoClaw attempts to remove temporary policy and build-context sources and reports cleanup failures with the onboarding error; post-create failures retain recovery state. Cancellation before sandbox creation can leave the session resumable. Shared inference providers remain gateway configuration and are not sandbox cleanup targets. Coverage: `transition-traces.test.ts`, `sandbox-create-intent-boundary.test.ts`, `sandbox-create-plan.test.ts`, and the focused cancellation, readiness, GPU cleanup, dashboard, policy-authority, destroy, and retained-recovery tests. Gap: gateway upserts can outlive a failed or interrupted create. | | **`--fresh` onboard** — `resolveOnboardEntryOptions`, `prepareFreshSession`, `createBaseImageResolutionContext` | Current flags/environment/prompts replace resumable intent. `--fresh` disables auto-resume and forces base-image resolution; it does not prove that the selected sandbox name is unused. | The first destructive effect is local: the prior onboard session is cleared before a new session is saved. A matching live sandbox can later reuse or recreate through the normal sandbox decision; `--fresh` does not itself delete it. | The new session and machine snapshot replace the old resume checkpoint. Credential and effect boundaries then match new onboard or live recreate. | The discarded resume checkpoint is not restored on later failure. Covered by `entry-options.test.ts`, `session-bootstrap.test.ts`, and base-image resolution tests. | | **Resume, re-onboard, or recreate** — `onboard()`, `prepareOnboardSession`, `decideSandboxResume`, live-sandbox handling in `createSandbox` | For `--resume`, the recorded session is authoritative and conflicting current name/provider/model/image/tool-disclosure hints are rejected. A new re-onboard run takes current flags, environment, and prompts as intent while registry/gateway state provides drift evidence. The machine resolves a complete secret-free create intent, including policy, messaging/provider, GPU, resource, disabled-channel, and agent inputs, before repair/removal or live recreation. | Ordinary live recreation conditionally backs up before provider cleanup, **delete**, and image removal. The recreate journal preserves the source registry row after deletion. Replacement registration commits the new row after readiness and validation. A selected pre-upgrade backup suppresses a new one; an explicit override permits recreation without backup. Resume registry removal and `repair-and-recreate` occur only after complete intent validation. Temporary policy/build artifacts remain materialization effects after the delete boundary. | Resume continues the recorded session/machine snapshot; non-resume re-onboard writes a new session first. OpenClaw records completed sandbox name, web search, messaging, and resource choices with explicit progress markers, including explicit `null` choices, while the complete create intent stays process-local and is not persisted or emitted. Raw credential values remain outside the session. A missing process value can be rebound only when the same OpenClaw session recorded successfully registering that provider and its live provider name, provider type, and credential key still match; otherwise interactive resume requests it again and non-interactive resume exits with environment-variable guidance. Credentials are checked before mutation and again immediately before materialization. | A failed replacement keeps the source registry row. Restore failures warn and can still publish the replacement; managed-DCode live-selection failure leaves a running, unregistered sandbox with manual-delete guidance. Checkpoint replay reuses an exact live sandbox after an interrupted create and backfills missing create/register receipts. Cancel rollback is not armed and there is no rebuild-style receipt rollback. Coverage: transition traces, create-intent characterization, checkpoint replay and resume guards, and sandbox-handler crash recovery. Gaps: early backup asymmetry and no rebuild-style cross-effect rollback. | | **Rebuild or installer-driven upgrade** — `rebuildSandbox` in `rebuild-pipeline.ts`; `upgradeSandboxes` | Registry state is authoritative. A matching session may fill guarded legacy gaps only when its selection agrees; an unrelated/global session is never used. Ambient provider/model selection is quarantined by `isolateAmbientRecreateEnv`, apart from narrowly scoped legacy recovery. Legacy and custom-image rebuilds retain and fingerprint a prepared build context. Managed-image rebuilds instead stage an immutable image and startup-profile handoff, skip Dockerfile image preflight, and revalidate provider-bound workload authority before each deletion boundary. | Consent persistence, target-gateway selection/recovery, and target-preflight registry updates can precede disposable image build/probes. Backup is the first durable recovery checkpoint when available. MCP detach/scrub and NIM stop are destructive in-place effects before the **sandbox delete** boundary. Legacy and custom-image paths recheck prepared context and mutation-edge conditions before delete. Managed-image paths revalidate the provider-bound handoff before delete. | Durable checkpoints are the backup/recovery manifest when one exists and the rewritten recreate session; stale recovery can reach deletion without a manifest, making that session its first new durable checkpoint. Rollback receipts/snapshots are process-local. Credential metadata comes from the target or guarded fallback; raw credentials/providers are checked against current process/gateway state, while prepared installer recovery may reconstruct a missing gateway provider from a validated host credential. | In-process rollback best-effort restores registry/MCP retry metadata, but process death after non-MCP delete can still lose it. The inner onboarding consumes the managed-workload handoff or selects the legacy resource profile after deletion. Covered by rebuild, managed-workload authority, image-preflight, DCode, and messaging tests. Gaps: health-before-delete and atomic swap. Closed issue #5801 records the original gap; #6835 fixed only the printed recovery path. | diff --git a/src/lib/onboard/sandbox-create/orchestration.test.ts b/src/lib/onboard/sandbox-create/orchestration.test.ts index 8ca432a08b4..47202416d6f 100644 --- a/src/lib/onboard/sandbox-create/orchestration.test.ts +++ b/src/lib/onboard/sandbox-create/orchestration.test.ts @@ -544,11 +544,12 @@ describe("deferred provider effect authority", () => { expect(error).toBeInstanceOf(Error); expect((error as Error).message).toContain( - "Do not delete it by mutable sandbox name. Ask an OpenShell administrator to remove the retained sandbox through an identity-bound procedure.", + "Do not delete it by mutable sandbox name. Run 'nemoclaw alpha destroy'.", ); expect((error as Error).message).toContain( - "After OpenShell confirms the retained sandbox is absent, run 'nemoclaw alpha destroy --yes' to reconcile its verified Docker containers and recovery record.", + "can reconcile verified residual resources and the recovery record only after OpenShell confirms absence", ); + expect((error as Error).message).not.toContain("administrator"); expect(runOpenshell).not.toHaveBeenCalledWith( expect.arrayContaining(["sandbox", "provider", "attach"]), @@ -851,7 +852,7 @@ describe("sandbox create identity checks", () => { expect(error).toBeInstanceOf(AggregateError); expect((error as AggregateError).message).toMatch( new RegExp( - `Create-attempt label: ai\\.nvidia\\.nemoclaw\\.create-attempt=${createAttemptNonce}.*left sandbox 'alpha' in place.*identity fingerprint: ${exactIdentity}.*did not run OpenShell's mutable-name deletion command.*Do not delete the sandbox by mutable sandbox name.*OpenShell administrator.*identity-bound recovery or removal procedure`, + `Create-attempt label: ai\\.nvidia\\.nemoclaw\\.create-attempt=${createAttemptNonce}.*left sandbox 'alpha' in place.*identity fingerprint: ${exactIdentity}.*did not run OpenShell's mutable-name deletion command.*Do not delete the sandbox by mutable sandbox name.*Inspection is diagnostic only and does not authorize deletion`, "u", ), ); @@ -860,7 +861,7 @@ describe("sandbox create identity checks", () => { expect.objectContaining({ message: expect.stringMatching( new RegExp( - `Create-attempt label: ai\\.nvidia\\.nemoclaw\\.create-attempt=${createAttemptNonce}.*left sandbox 'alpha' in place.*identity fingerprint: ${exactIdentity}.*did not run OpenShell's mutable-name deletion command.*Do not delete the sandbox by mutable sandbox name.*OpenShell administrator.*identity-bound recovery or removal procedure`, + `Create-attempt label: ai\\.nvidia\\.nemoclaw\\.create-attempt=${createAttemptNonce}.*left sandbox 'alpha' in place.*identity fingerprint: ${exactIdentity}.*did not run OpenShell's mutable-name deletion command.*Do not delete the sandbox by mutable sandbox name.*Inspection is diagnostic only and does not authorize deletion`, "u", ), ), @@ -1388,7 +1389,7 @@ describe("sandbox create identity checks", () => { expect(error).toBeInstanceOf(AggregateError); expect((error as AggregateError).message).toMatch( - /left sandbox 'alpha' in place.*did not return a durable sandbox identity fingerprint.*Do not delete the sandbox by mutable sandbox name.*identity-bound recovery or removal procedure/u, + /left sandbox 'alpha' in place.*did not return a durable sandbox identity fingerprint.*Do not delete the sandbox by mutable sandbox name.*Inspection is diagnostic only and does not authorize deletion/u, ); expect((error as AggregateError).errors).toEqual( expect.arrayContaining([ diff --git a/src/lib/onboard/sandbox-create/orchestration.ts b/src/lib/onboard/sandbox-create/orchestration.ts index 98d7b84c43a..22cd04e9c62 100644 --- a/src/lib/onboard/sandbox-create/orchestration.ts +++ b/src/lib/onboard/sandbox-create/orchestration.ts @@ -373,7 +373,7 @@ export function installPostCreateRecoveryRetryOwner( } catch (error) { if (propagateFailure) throw error; log( - " NemoClaw still could not save the retained sandbox recovery record; the recovery-only session remains blocked for administrator recovery.", + " NemoClaw still could not save the retained sandbox recovery record. Preserve the registry entry and terminal output; do not delete the sandbox by mutable name.", ); } }; @@ -427,7 +427,7 @@ export function persistPostCreateRecovery(input: { `Create-attempt label: ${NEMOCLAW_CREATE_ATTEMPT_LABEL}=${input.recoveryContext.createAttemptNonce}. ` + `Sandbox '${input.sandboxName}' was retained after ${input.stage} failed. ` + `Gateway '${input.gatewayName}'. Lifecycle generation '${input.lifecycleGeneration}'. ` + - "Do not delete the sandbox by mutable name; preserve it for identity-bound administrator recovery."; + `Do not delete the sandbox by mutable name. Run '${cliName()} ${input.sandboxName} destroy'; it can clear retained recovery only after OpenShell confirms absence.`; console.error(` ${message}`); let persisted = false; try { @@ -849,8 +849,8 @@ export async function runSandboxCreateWithIdentityVerification< createAttemptGuidance + `NemoClaw left sandbox '${input.sandboxName}' in place after post-create verification or finalization failed. ` + `${identityGuidance} NemoClaw did not run OpenShell's mutable-name deletion command because the name may now identify a replacement sandbox. ` + - `Do not delete the sandbox by mutable sandbox name. Run '${cliName()} ${input.sandboxName} destroy' to use the retained identity. ` + - "If destroy cannot prove that identity, stop. Ask the OpenShell administrator to inspect the surviving sandbox and use an identity-bound recovery or removal procedure."; + `Do not delete the sandbox by mutable sandbox name. Run '${cliName()} ${input.sandboxName} destroy'. ` + + "If OpenShell reports the sandbox present or cannot determine presence, destroy removes nothing and preserves the recovery record. Inspection is diagnostic only and does not authorize deletion."; const compensationErrors: unknown[] = []; if (input.persistRetainedSandboxRecovery) { try { @@ -1056,8 +1056,8 @@ export function createProviderEffectBoundary(input: { throw new Error( `OpenShell cannot attach providers to the immutable identity of sandbox '${input.sandboxName}'. ` + `NemoClaw retained the incomplete sandbox on gateway '${input.gatewayName}'. ` + - `Do not delete it by mutable sandbox name. Ask an OpenShell administrator to remove the retained sandbox through an identity-bound procedure. ` + - `After OpenShell confirms the retained sandbox is absent, run '${cliName()} ${input.sandboxName} destroy --yes' to reconcile its verified Docker containers and recovery record.`, + `Do not delete it by mutable sandbox name. Run '${cliName()} ${input.sandboxName} destroy'. ` + + `The command preserves a present or unknown sandbox and can reconcile verified residual resources and the recovery record only after OpenShell confirms absence.`, ); }, }; diff --git a/src/lib/onboard/sandbox-gpu-create-flow.test.ts b/src/lib/onboard/sandbox-gpu-create-flow.test.ts index 9b46b929cfb..0519c45d5d6 100644 --- a/src/lib/onboard/sandbox-gpu-create-flow.test.ts +++ b/src/lib/onboard/sandbox-gpu-create-flow.test.ts @@ -970,7 +970,7 @@ describe("runSandboxGpuCreateFlow native failure and readiness", () => { expect(mocks.streamSandboxCreate).toHaveBeenCalledOnce(); expect(mocks.verifyGpuSandboxAccessAfterReady).not.toHaveBeenCalled(); expect(deps.runOpenshell).not.toHaveBeenCalled(); - expect(errorOutput()).toContain("Verify the sandbox identity before manual cleanup"); + expect(errorOutput()).toContain("Recovery remains blocked while this sandbox exists"); expect(mocks.streamSandboxCreate).toHaveBeenCalledOnce(); }); @@ -1384,7 +1384,7 @@ describe("runSandboxGpuCreateFlow cleanup and provenance", () => { const output = vi.mocked(console.error).mock.calls.flat().join("\n"); expect(deps.runOpenshell).not.toHaveBeenCalled(); expect(output).toContain("left sandbox 'alpha' in place"); - expect(output).toContain("Verify the sandbox identity before manual cleanup"); + expect(output).toContain("Recovery remains blocked while this sandbox exists"); expect(output).not.toContain("openshell sandbox delete"); expect(output).not.toContain("Retry: nemoclaw onboard"); }); @@ -1401,7 +1401,7 @@ describe("runSandboxGpuCreateFlow cleanup and provenance", () => { const output = vi.mocked(console.error).mock.calls.flat().join("\n"); expect(deps.runOpenshell).not.toHaveBeenCalled(); expect(output).toContain("left sandbox 'alpha' in place"); - expect(output).toContain("Verify the sandbox identity before manual cleanup"); + expect(output).toContain("Recovery remains blocked while this sandbox exists"); expect(output).not.toContain("Retry: nemoclaw onboard"); }); diff --git a/src/lib/onboard/sandbox-gpu-create-flow.ts b/src/lib/onboard/sandbox-gpu-create-flow.ts index 72a8b7719df..e68c688c31c 100644 --- a/src/lib/onboard/sandbox-gpu-create-flow.ts +++ b/src/lib/onboard/sandbox-gpu-create-flow.ts @@ -559,7 +559,7 @@ export async function runSandboxGpuCreateFlow( ? ` Managed bootstrap retained exact owner-cleanup authority for sandbox '${input.sandboxName}'. Do not delete a runtime by mutable sandbox name; preserve it for identity-bound recovery.` : hermesPortableLifecycle ? ` Hermes portable sandbox '${input.sandboxName}' did not complete receipt-owned creation. Preserve its lifecycle receipt and resume onboarding after correcting the reported failure.` - : ` Sandbox '${input.sandboxName}' may still exist. Verify its durable identity before manual cleanup; do not act by mutable name alone.`, + : ` Sandbox '${input.sandboxName}' may still exist. Recovery remains blocked while it exists; do not delete it by mutable name. Run 'nemoclaw ${input.sandboxName} destroy' to check for authoritative absence.`, ); if (input.requirePolicylessCreate) { const persistRetainedSandboxRecovery = input.persistRetainedSandboxRecovery; @@ -574,13 +574,13 @@ export async function runSandboxGpuCreateFlow( } else { const identityGuidance = evidence.liveIdentityFingerprint ? "Use that fingerprint only to compare the surviving sandbox with this create attempt." - : "OpenShell did not return one exact durable sandbox identity for this create attempt. Recovery is blocked until an OpenShell administrator resolves the create-attempt label to one sandbox."; + : "OpenShell did not return one exact durable sandbox identity for this create attempt."; const message = `Create-attempt label: ${NEMOCLAW_CREATE_ATTEMPT_LABEL}=${evidence.createAttemptNonce}. ` + `${evidence.liveIdentityFingerprint ? `Durable sandbox identity fingerprint: ${evidence.liveIdentityFingerprint}. ` : ""}` + `APF sandbox '${input.sandboxName}' may have been retained after native GPU fallback stopped. ` + `Gateway '${input.gatewayName}'. ${identityGuidance} ` + - "Do not delete a sandbox by mutable name; use an identity-bound administrator recovery procedure."; + `Do not delete the sandbox by mutable name. Run 'nemoclaw ${input.sandboxName} destroy'; it can clear retained recovery only after OpenShell confirms absence.`; let persisted = false; try { persisted = evidence.liveIdentityFingerprint @@ -596,7 +596,7 @@ export async function runSandboxGpuCreateFlow( console.error(` ${message}`); if (!persisted) { console.error( - " APF recovery is blocked because NemoClaw could not save this create-attempt evidence. Preserve the terminal output for an OpenShell administrator.", + " APF recovery is blocked because NemoClaw could not save this create-attempt evidence. Preserve the registry entry and terminal output; do not delete the sandbox by mutable name.", ); } } diff --git a/src/lib/onboard/sandbox-gpu-create-identity-gate.test.ts b/src/lib/onboard/sandbox-gpu-create-identity-gate.test.ts index b6e97d0e618..7371c66555a 100644 --- a/src/lib/onboard/sandbox-gpu-create-identity-gate.test.ts +++ b/src/lib/onboard/sandbox-gpu-create-identity-gate.test.ts @@ -515,9 +515,11 @@ describe("created sandbox identity gate", () => { expect(output).toContain(`Durable sandbox identity fingerprint: ${fingerprint}`); expect(output).toContain("Run 'nemoclaw alpha destroy'"); expect(output).toContain("the command removes nothing and preserves the recovery record"); - expect(output).toContain("Give the create-attempt label to an OpenShell administrator"); - expect(output).toContain("After OpenShell confirms removal"); + expect(output).toContain("Inspection is diagnostic only"); + expect(output).toContain("Recovery remains blocked while the sandbox is present"); + expect(output).toContain("after the owning gateway reports absence"); expect(output).toContain("run 'nemoclaw alpha destroy --yes'"); + expect(output).not.toContain("administrator"); expect(output).not.toContain("alpha-sandbox-id"); expect(output).not.toContain("Recovery:"); expect(output).not.toContain("Or: nemoclaw onboard"); @@ -1171,7 +1173,7 @@ describe("created sandbox identity gate", () => { expect(input.persistRetainedSandboxRecovery).toHaveBeenCalledExactlyOnceWith( expect.stringMatching( new RegExp( - `^Create-attempt label: ${NEMOCLAW_CREATE_ATTEMPT_LABEL}=${nonce}\\..*Recovery is blocked until an OpenShell administrator resolves the create-attempt label`, + `^Create-attempt label: ${NEMOCLAW_CREATE_ATTEMPT_LABEL}=${nonce}\\..*Do not delete the sandbox by mutable name.*can clear retained recovery only after OpenShell confirms absence`, "u", ), ), @@ -1180,7 +1182,8 @@ describe("created sandbox identity gate", () => { ); const output = vi.mocked(console.error).mock.calls.flat().join("\n"); expect(output).toContain(`${NEMOCLAW_CREATE_ATTEMPT_LABEL}=${nonce}`); - expect(output).toContain("Recovery is blocked"); + expect(output).toContain("can clear retained recovery only after OpenShell confirms absence"); + expect(output).not.toContain("administrator"); expect(deps.runOpenshell).not.toHaveBeenCalledWith( ["sandbox", "delete", "alpha"], expect.anything(), diff --git a/src/lib/onboard/sandbox-gpu-create-run-attempt.ts b/src/lib/onboard/sandbox-gpu-create-run-attempt.ts index 3e4c44b21c0..49adf64aeef 100644 --- a/src/lib/onboard/sandbox-gpu-create-run-attempt.ts +++ b/src/lib/onboard/sandbox-gpu-create-run-attempt.ts @@ -546,7 +546,7 @@ export function createSandboxGpuCreateAttemptRunner( const persistenceFailureMessage = "NemoClaw could not save the retained sandbox recovery record for this create attempt."; console.error( - ` ${persistenceFailureMessage} Preserve the terminal output for an OpenShell administrator.`, + ` ${persistenceFailureMessage} Preserve the registry entry and terminal output; do not delete the sandbox by mutable name.`, ); throw new Error(persistenceFailureMessage, { cause: persistenceCause }); } @@ -774,7 +774,7 @@ export function createSandboxGpuCreateAttemptRunner( throw new Error( createAttemptNonce ? `OpenShell create client did not exit after Ready for sandbox '${input.sandboxName}'. NemoClaw retained the sandbox and blocked post-create effects. Follow the retained recovery action above.` - : `OpenShell create client did not exit after Ready for sandbox '${input.sandboxName}'. NemoClaw blocked post-create effects. No create-attempt identity was available for retained recovery. Preserve the sandbox for identity-bound OpenShell administrator recovery; do not delete it by mutable name.`, + : `OpenShell create client did not exit after Ready for sandbox '${input.sandboxName}'. NemoClaw blocked post-create effects. No create-attempt identity was available for retained recovery. Preserve the registry entry and terminal output; do not delete the sandbox by mutable name.`, ); } return createResult; @@ -1158,7 +1158,9 @@ export function createSandboxGpuCreateAttemptRunner( console.error( ` NemoClaw left sandbox '${input.sandboxName}' in place because OpenShell can delete it only by mutable name.`, ); - console.error(" Verify the sandbox identity before manual cleanup."); + console.error( + ` Recovery remains blocked while this sandbox exists. Do not delete it by mutable name; run 'nemoclaw ${input.sandboxName} destroy' to check for authoritative absence.`, + ); } failAfterCreatedSandboxVerification( `Sandbox '${input.sandboxName}' did not become ready after verified creation.`, diff --git a/test/helpers/destroy-flow-test-harness.ts b/test/helpers/destroy-flow-test-harness.ts index a91197149b2..6103c892850 100644 --- a/test/helpers/destroy-flow-test-harness.ts +++ b/test/helpers/destroy-flow-test-harness.ts @@ -129,6 +129,7 @@ type DestroyHarnessOptions = { replaceSessionAfterRegistryRemoval?: boolean; removeSandboxResult?: boolean; restoreMcpError?: string; + sandboxListResult?: { status: number | null; stdout?: string; stderr?: string }; sandboxPresent?: boolean; sessionRouterPid?: number; stopInferenceError?: string; @@ -477,11 +478,13 @@ export function createDestroyHarness(options: DestroyHarnessOptions = {}): Destr }; case "sandbox:list": gatewayPinsAtSandboxList.push(process.env.OPENSHELL_GATEWAY); - return { - status: 0, - stdout: sandboxListJson(sandboxPresent ? ["alpha"] : []), - stderr: "", - }; + return ( + options.sandboxListResult ?? { + status: 0, + stdout: sandboxListJson(sandboxPresent ? ["alpha"] : []), + stderr: "", + } + ); case "sandbox:delete": events.push("delete"); sandboxPresent = false; From e72f06e06b39fae82dae2726e152e2f43a48a7cc Mon Sep 17 00:00:00 2001 From: Julie Yaunches Date: Tue, 8 Sep 2026 10:58:08 -0400 Subject: [PATCH 07/12] fix(sandbox): close retained cleanup gaps Signed-off-by: Julie Yaunches --- .../destroy-retained-recovery-flow.test.ts | 34 +++++++++++++++++++ src/lib/actions/sandbox/destroy.ts | 10 ++++++ src/lib/onboard/created-sandbox-failure.ts | 14 +++----- 3 files changed, 49 insertions(+), 9 deletions(-) diff --git a/src/lib/actions/sandbox/destroy-retained-recovery-flow.test.ts b/src/lib/actions/sandbox/destroy-retained-recovery-flow.test.ts index 51db963d657..8fb160696ae 100644 --- a/src/lib/actions/sandbox/destroy-retained-recovery-flow.test.ts +++ b/src/lib/actions/sandbox/destroy-retained-recovery-flow.test.ts @@ -393,6 +393,40 @@ describe("destroySandbox retained recovery flow", () => { }, ); + it( + "preserves a recovery record without identity when a residual container exists (#10863)", + { timeout: 30_000 }, + async () => { + const recovery = retainedRecoveryRecordWithoutIdentity(); + const containerId = "a".repeat(64); + const harness = createDestroyHarness({ + sandboxPresent: false, + dockerRunResult: { + status: 0, + stdout: `${containerId}\topenshell\tdefault\tsb-alpha`, + }, + registryEntryOverrides: { + lifecycleGeneration: recovery.lifecycleGeneration!, + }, + retainedRecoveryRecords: [recovery], + }); + + await expect(harness.destroySandbox("alpha", { yes: true })).rejects.toThrow( + "process.exit(1)", + ); + + expect(harness.errorSpy).toHaveBeenCalledWith( + expect.stringContaining("has no durable sandbox identity"), + ); + expect(harness.dockerRunSpy).not.toHaveBeenCalledWith( + ["rm", "-f", containerId], + expect.anything(), + ); + expect(harness.resolveRetainedSandboxRecoverySpy).not.toHaveBeenCalled(); + expect(harness.removeSandboxSpy).not.toHaveBeenCalled(); + }, + ); + it( "finishes retained cleanup after OpenShell already removed the sandbox (#10547)", { timeout: 30_000 }, diff --git a/src/lib/actions/sandbox/destroy.ts b/src/lib/actions/sandbox/destroy.ts index 9fe649a25b9..71ba1867cbc 100644 --- a/src/lib/actions/sandbox/destroy.ts +++ b/src/lib/actions/sandbox/destroy.ts @@ -676,6 +676,16 @@ async function destroySandboxUnlocked( if (initialIdentity === false) { requestSandboxDestroyExit(1); } + if ( + retainedRecoveryAuthority?.sandboxIdentityFingerprint === null && + initialIdentity?.identities !== undefined && + initialIdentity.identities.length > 0 + ) { + console.error( + ` Refusing to destroy retained sandbox '${sandboxName}': the recovery record has no durable sandbox identity, so NemoClaw cannot qualify a residual container for deletion. No sandbox resources were removed. Preserve the recovery record and resolve the container identity conflict before retrying.`, + ); + requestSandboxDestroyExit(1); + } const initialContainerIdentities = initialIdentity?.identities; let preparedManagedLlamaCppCleanup: ReturnType< diff --git a/src/lib/onboard/created-sandbox-failure.ts b/src/lib/onboard/created-sandbox-failure.ts index 8b47c4f24c9..5050b124b2a 100644 --- a/src/lib/onboard/created-sandbox-failure.ts +++ b/src/lib/onboard/created-sandbox-failure.ts @@ -116,20 +116,16 @@ export type SandboxReadinessFailureReportDeps = { export type SandboxReadinessTerminalResolution = | "deferred_to_docker_gpu_patch" - | "terminal_failure_deleted" | "terminal_failure_retained" - | "timed_out_deleted" | "timed_out_retained"; -/** Map the readiness reason and cleanup outcome into the receipt terminal state. */ +/** Map the readiness reason into the retained receipt terminal state. */ function readinessTerminalResolution( readiness: CreatedSandboxReadinessResult, - deleted: boolean, ): SandboxReadinessTerminalResolution { - if (readiness.reason === "terminal_failure_phase") { - return deleted ? "terminal_failure_deleted" : "terminal_failure_retained"; - } - return deleted ? "timed_out_deleted" : "timed_out_retained"; + return readiness.reason === "terminal_failure_phase" + ? "terminal_failure_retained" + : "timed_out_retained"; } /** Name the readiness gate that blocked the created sandbox from becoming Ready. */ @@ -199,7 +195,7 @@ export function reportSandboxReadinessFailure( readiness: options.readiness, createStatus: options.createStatus, timeoutSecs: options.timeoutSecs, - terminalResolution: readinessTerminalResolution(options.readiness, false), + terminalResolution: readinessTerminalResolution(options.readiness), })) { deps.error(line); } From c952c0007c97ddb242e10248a58b14d7d656f507 Mon Sep 17 00:00:00 2001 From: Julie Yaunches Date: Tue, 8 Sep 2026 11:30:29 -0400 Subject: [PATCH 08/12] fix(sandbox): align retained recovery messages Signed-off-by: Julie Yaunches --- .../rebuild-preflight-confirmation.test.ts | 5 ++++- .../sandbox/rebuild-preflight-guards.ts | 2 +- src/lib/onboard/exit-step-failure.test.ts | 4 ++++ src/lib/state/onboard-session.ts | 2 +- test/onboarding/onboard-rollback.test.ts | 19 ++++++++++++++----- 5 files changed, 24 insertions(+), 8 deletions(-) diff --git a/src/lib/actions/sandbox/rebuild-preflight-confirmation.test.ts b/src/lib/actions/sandbox/rebuild-preflight-confirmation.test.ts index 10cd6e19bcc..d7372094365 100644 --- a/src/lib/actions/sandbox/rebuild-preflight-confirmation.test.ts +++ b/src/lib/actions/sandbox/rebuild-preflight-confirmation.test.ts @@ -234,7 +234,10 @@ describe("rebuild preflight guards", () => { listRecovery.mock.invocationCallOrder[0]!, ); expect(bail).toHaveBeenCalledWith("Retained sandbox recovery blocks rebuild for 'alpha'.", 1); - expect(error.mock.calls.flat().join("\n")).toContain("nemoclaw alpha destroy --yes"); + const output = error.mock.calls.flat().join("\n"); + expect(output).toContain("nemoclaw alpha destroy --yes"); + expect(output).toContain("reports the sandbox present or cannot determine presence"); + expect(output).not.toContain("administrator"); }); it("stops after a failed onboard-lock acquisition without releasing another run's lock (#7794)", () => { diff --git a/src/lib/actions/sandbox/rebuild-preflight-guards.ts b/src/lib/actions/sandbox/rebuild-preflight-guards.ts index 49056e8a6e0..d9fcc6ef14d 100644 --- a/src/lib/actions/sandbox/rebuild-preflight-guards.ts +++ b/src/lib/actions/sandbox/rebuild-preflight-guards.ts @@ -339,7 +339,7 @@ export function blockRebuildOnRetainedSandboxRecovery( ` Rebuild cannot use retained sandbox '${sandboxName}' while recovery record '${retainedRecovery.recordId}' is unresolved. No sandbox or Docker resources were removed.`, ); console.error( - ` Run '${CLI_NAME} ${sandboxName} destroy --yes'. If OpenShell still reports the sandbox present, follow destroy's create-attempt label guidance for identity-bound administrator removal.`, + ` Run '${CLI_NAME} ${sandboxName} destroy --yes'. If the owning gateway reports the sandbox present or cannot determine presence, destroy removes nothing and preserves the recovery record.`, ); bail(`Retained sandbox recovery blocks rebuild for '${sandboxName}'.`, 1); return true; diff --git a/src/lib/onboard/exit-step-failure.test.ts b/src/lib/onboard/exit-step-failure.test.ts index 1c267781144..837e96fe8c1 100644 --- a/src/lib/onboard/exit-step-failure.test.ts +++ b/src/lib/onboard/exit-step-failure.test.ts @@ -262,6 +262,10 @@ describe("incomplete-onboard --resume backstop (#6003)", () => { createAttemptNonce: "c".repeat(62), }); const beforeExit = requireLoadedSession(); + expect(beforeExit.failure?.message).toContain( + "retained recovery blocks this sandbox name until destroy confirms absence", + ); + expect(beforeExit.failure?.message).not.toContain("administrator"); const output = runExitHandler(1); diff --git a/src/lib/state/onboard-session.ts b/src/lib/state/onboard-session.ts index fed13ed38ca..aa7503e948b 100644 --- a/src/lib/state/onboard-session.ts +++ b/src/lib/state/onboard-session.ts @@ -2148,7 +2148,7 @@ export function markCancellationRecovery( session.failure = { step: session.lastStepStarted, message: - "Onboarding was cancelled after sandbox creation; administrator recovery is required.", + "Onboarding was cancelled after sandbox creation; retained recovery blocks this sandbox name until destroy confirms absence and completes cleanup.", recordedAt, interrupted: true, }; diff --git a/test/onboarding/onboard-rollback.test.ts b/test/onboarding/onboard-rollback.test.ts index 956315c86e5..2641e851f60 100644 --- a/test/onboarding/onboard-rollback.test.ts +++ b/test/onboarding/onboard-rollback.test.ts @@ -35,7 +35,7 @@ if (!isOnboardRollbackInternals(onboardInternals)) { const { buildOrphanedSandboxRollbackMessage } = onboardInternals; describe("ghost-sandbox rollback message (#2174)", () => { - it("reports the surviving sandbox and manual identity-checked cleanup", () => { + it("reports the surviving sandbox and fail-closed recovery", () => { const lines = buildOrphanedSandboxRollbackMessage( "alpha", new Error("All dashboard ports in range 18789-18798 are occupied"), @@ -47,8 +47,13 @@ describe("ghost-sandbox rollback message (#2174)", () => { expect(lines).toContain( " NemoClaw left the sandbox running because OpenShell deletion targets a mutable name.", ); - expect(lines).toContain(" Verify the sandbox identity, then clean up manually:"); - expect(lines).toContain(' openshell sandbox delete -g "nemoclaw-18080" "alpha"'); + expect(lines).toContain( + ' Recovery remains blocked while gateway "nemoclaw-18080" reports this sandbox present.', + ); + expect(lines).toContain( + " Do not delete it by mutable name; run 'nemoclaw alpha destroy' to check for authoritative absence.", + ); + expect(lines.join("\n")).not.toContain("openshell sandbox delete"); }); it("renders non-Error throwables via String coercion", () => { @@ -56,15 +61,19 @@ describe("ghost-sandbox rollback message (#2174)", () => { expect(lines).toContain(" raw string failure"); }); - it("escapes the sandbox name into the manual-cleanup command exactly", () => { + it("escapes the gateway and preserves the sandbox name in recovery guidance", () => { const lines = buildOrphanedSandboxRollbackMessage( 'weird-name_42"', new Error("oops"), 'gateway"name', ); expect(lines).toContain( - ' openshell sandbox delete -g "gateway\\\"name" "weird-name_42\\\""', + ' Recovery remains blocked while gateway "gateway\\\"name" reports this sandbox present.', + ); + expect(lines).toContain( + " Do not delete it by mutable name; run 'nemoclaw weird-name_42\" destroy' to check for authoritative absence.", ); + expect(lines.join("\n")).not.toContain("openshell sandbox delete"); }); it("does not suggest deletion when the owning gateway is unknown", () => { From e2a455e7a730d14d67217c80386057719d79a46a Mon Sep 17 00:00:00 2001 From: Julie Yaunches Date: Tue, 8 Sep 2026 12:06:46 -0400 Subject: [PATCH 09/12] docs(sandbox): state retained recovery limitation Signed-off-by: Julie Yaunches --- docs/reference/commands.mdx | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/reference/commands.mdx b/docs/reference/commands.mdx index 86ff146f9d9..4e781032bfe 100644 --- a/docs/reference/commands.mdx +++ b/docs/reference/commands.mdx @@ -379,7 +379,7 @@ OpenClaw onboarding does not enter the `complete` state until NemoClaw proves th Before the configuration review, NemoClaw records the sandbox name and the selected provider and model as an incomplete choice. If onboarding stops at the review prompt, an interactive `--resume` run shows the prompt again. A non-interactive `--resume` run reuses the recorded choice and continues to inference setup. After you choose **Apply configuration**, NemoClaw records the choice before inference setup starts. If inference setup fails, `--resume` reuses the accepted provider, model, and sandbox name. If you choose **Exit onboarding**, onboarding exits with a nonzero status and clears those recorded choices. Run `$$nemoclaw onboard` to make new choices after exit. During a resume without terminal input, `--yes` or `NEMOCLAW_YES=1` also selects non-interactive resume behavior. For a new or fresh session, `--yes` and `NEMOCLAW_YES=1` accept supported confirmations but do not replace `--non-interactive`. If onboarding returns without reaching the final `complete` state, the command exits with status `1`. When that result is resumable, NemoClaw keeps the session `in_progress` at its last checkpoint instead of marking it failed, so correct the reported condition and run `$$nemoclaw onboard --resume`. -#### Recover a retained sandbox +#### Inspect retained sandbox recovery If onboarding cannot complete after sandbox creation, NemoClaw preserves the sandbox. When available, NemoClaw records and prints the create-attempt label as the exact `ai.nvidia.nemoclaw.create-attempt=` selector. @@ -389,7 +389,7 @@ Run `$$nemoclaw destroy` to check retained recovery and reconcile Use the result from `destroy` to choose the next action: -- If OpenShell reports the sandbox present, `destroy` preserves the record and removes no resources. OpenShell exposes no atomic delete-by-identity primitive. Neither NemoClaw nor manual inspection can bind a later mutable-name delete to the durable identity fingerprint. You can inspect the owning gateway for diagnosis, but do not delete the sandbox by mutable name. Recovery remains blocked until the owning gateway reports the sandbox absent. +- If OpenShell reports the sandbox present, `destroy` preserves the record and removes no resources. OpenShell exposes no atomic delete-by-identity primitive. Neither NemoClaw nor manual inspection can bind a later mutable-name delete to the durable identity fingerprint. You can inspect the owning gateway for diagnosis, but do not delete the sandbox by mutable name. Live retained-sandbox removal is not supported yet and remains tracked by the linked OpenShell capability issue. Use a different explicit sandbox name while this record remains blocked. - If OpenShell cannot determine whether the sandbox is present, `destroy` reports unknown presence, removes no resources, and preserves the recovery record. Correct the OpenShell gateway or list failure, then rerun `destroy`. - If OpenShell confirms the sandbox is absent, `destroy` selects one retained record from its registry generation, gateway, create-attempt evidence, and available identity fingerprint. Provider and Docker cleanup then apply their own immutable identity checks. `destroy` removes only qualified residual resources and clears the matching recovery record only after verified cleanup. - If the recovery record is ambiguous or immutable runtime-identity verification fails, cleanup stops and the record remains. For Docker-backed sandboxes, a foreign container, changed Docker identity, or failed Docker probe has the same fail-closed result. @@ -443,7 +443,7 @@ $$nemoclaw onboard --fresh --apf-interceptor --name my-apf-sandbox If post-create verification or native GPU fallback fails after OpenShell may have created the sandbox, NemoClaw preserves the incomplete sandbox because automatic deletion would use its mutable name. -Follow the [retained-sandbox recovery procedure](#recover-a-retained-sandbox); its OpenShell-absence and immutable runtime-identity requirements also apply to APF creation. For Docker-backed APF creation, its Docker container-identity requirement also applies. +Follow the [retained-sandbox recovery procedure](#inspect-retained-sandbox-recovery); its OpenShell-absence and immutable runtime-identity requirements also apply to APF creation. For Docker-backed APF creation, its Docker container-identity requirement also applies. This onboarding mode does not support `--resume` or `--recreate-sandbox`, regardless of whether sandbox creation began. After destroy completes, repeat the original command with `--fresh`. @@ -740,7 +740,7 @@ If you enable Telegram during onboarding, the wizard can also prompt for whether If you cancel a brand-new onboarding run at the policy-tier selector or either policy-preset selector after sandbox creation, NemoClaw preserves the incomplete sandbox, registry entry, and onboarding session for retained recovery. NemoClaw reports the durable sandbox identity fingerprint when it is available. It does not run OpenShell's mutable-name deletion command because the name may now identify a replacement sandbox. -Follow the [retained-sandbox recovery procedure](#recover-a-retained-sandbox) to reconcile this cancellation. +Follow the [retained-sandbox recovery procedure](#inspect-retained-sandbox-recovery) to reconcile this cancellation. A fresh run with a different explicit name can continue while the cancelled sandbox name remains blocked. If you run onboarding again with the same sandbox name and choose a different inference provider or model, NemoClaw detects the drift and recreates the sandbox so the running agent config matches your selection. In interactive mode, the wizard asks for confirmation before delete and recreate. In non-interactive mode, NemoClaw recreates automatically when the stored selection is readable and differs. For managed Deep Agents Code sandboxes, NemoClaw also recreates when the live `dcode identity` selection is unreadable; other agent paths continue to reuse by default when their stored selection cannot be read. Set `NEMOCLAW_RECREATE_SANDBOX=1` to force recreation even when no drift is detected. From 84b525b3b74e24f44a75260a4a267a44fdcb5684 Mon Sep 17 00:00:00 2001 From: Julie Yaunches Date: Tue, 8 Sep 2026 12:32:28 -0400 Subject: [PATCH 10/12] docs(sandbox): remove unresolved tracking claim Signed-off-by: Julie Yaunches --- docs/reference/commands.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/commands.mdx b/docs/reference/commands.mdx index 4e781032bfe..549c5bada8f 100644 --- a/docs/reference/commands.mdx +++ b/docs/reference/commands.mdx @@ -389,7 +389,7 @@ Run `$$nemoclaw destroy` to check retained recovery and reconcile Use the result from `destroy` to choose the next action: -- If OpenShell reports the sandbox present, `destroy` preserves the record and removes no resources. OpenShell exposes no atomic delete-by-identity primitive. Neither NemoClaw nor manual inspection can bind a later mutable-name delete to the durable identity fingerprint. You can inspect the owning gateway for diagnosis, but do not delete the sandbox by mutable name. Live retained-sandbox removal is not supported yet and remains tracked by the linked OpenShell capability issue. Use a different explicit sandbox name while this record remains blocked. +- If OpenShell reports the sandbox present, `destroy` preserves the record and removes no resources. OpenShell exposes no atomic delete-by-identity primitive. Neither NemoClaw nor manual inspection can bind a later mutable-name delete to the durable identity fingerprint. You can inspect the owning gateway for diagnosis, but do not delete the sandbox by mutable name. Live retained-sandbox removal is not supported yet. Use a different explicit sandbox name while this record remains blocked. - If OpenShell cannot determine whether the sandbox is present, `destroy` reports unknown presence, removes no resources, and preserves the recovery record. Correct the OpenShell gateway or list failure, then rerun `destroy`. - If OpenShell confirms the sandbox is absent, `destroy` selects one retained record from its registry generation, gateway, create-attempt evidence, and available identity fingerprint. Provider and Docker cleanup then apply their own immutable identity checks. `destroy` removes only qualified residual resources and clears the matching recovery record only after verified cleanup. - If the recovery record is ambiguous or immutable runtime-identity verification fails, cleanup stops and the record remains. For Docker-backed sandboxes, a foreign container, changed Docker identity, or failed Docker probe has the same fail-closed result. From d1ef3aa5e0347371f8f462ef7405cbb6d826b4cf Mon Sep 17 00:00:00 2001 From: Julie Yaunches Date: Tue, 8 Sep 2026 12:45:12 -0400 Subject: [PATCH 11/12] docs(sandbox): align readiness timeout recovery Signed-off-by: Julie Yaunches --- docs/reference/commands.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/commands.mdx b/docs/reference/commands.mdx index 549c5bada8f..dd4c506c1b5 100644 --- a/docs/reference/commands.mdx +++ b/docs/reference/commands.mdx @@ -4061,7 +4061,7 @@ The following environment variables tune onboard-time and recovery wall-clock li | `NEMOCLAW_HF_DOWNLOAD_STALL_TIMEOUT` | `600` (10 minutes) | Maximum silence between Hugging Face download output during onboard. A positive finite value in seconds overrides the default, up to the Node.js timer limit of about 24.8 days. Blank, invalid, non-positive, sub-millisecond, and oversized values use the default. This is not a total download limit. Increase it only when a working download can produce no output for ten minutes. | | `NEMOCLAW_ONBOARD_VALIDATION_TIMEOUT_SECONDS` | Probe-specific | Connection and total deadlines for applicable OpenAI-compatible provider validation during onboard, including DeepSeek V4 Pro streaming validation. Positive finite seconds round up and cap at `600`; the value only raises existing deadlines. It does not extend the separate fixed five-second streaming-event probe. | | `NEMOCLAW_LOCAL_INFERENCE_TIMEOUT` | `180` | Wall-clock timeout for the inference-server validation probe during onboard, in seconds. Raise on slow networks or for very large prompts. | -| `NEMOCLAW_SANDBOX_READY_TIMEOUT` | `180` | Wall-clock timeout for post-create readiness, in seconds. Raise the timeout when the managed-image pull, explicit custom image build, gateway upload, or in-sandbox boot exceeds the default (typical on 70B+ models, first-time gateway uploads over slow links, or DGX Station / remote-VM first runs). Ordinary onboarding deletes the partially created sandbox when the deadline expires and prints the retry hint. Portable OpenClaw onboarding instead preserves the sandbox when NemoClaw cannot verify its runtime identity. | +| `NEMOCLAW_SANDBOX_READY_TIMEOUT` | `180` | Wall-clock timeout for post-create readiness, in seconds. Raise the timeout when the managed-image pull, explicit custom image build, gateway upload, or in-sandbox boot exceeds the default (typical on 70B+ models, first-time gateway uploads over slow links, or DGX Station / remote-VM first runs). Onboarding preserves a partially created sandbox when the deadline expires. Follow [Inspect retained sandbox recovery](#inspect-retained-sandbox-recovery); do not delete the sandbox by mutable name. | | `NEMOCLAW_SANDBOX_READY_ERROR_DEBOUNCE` | `30` | Consecutive `Error`-phase polls the post-create readiness wait tolerates before treating `Error` as terminal. Polling starts at 250ms and backs off to a 2-second cap, while `NEMOCLAW_SANDBOX_READY_TIMEOUT` remains the overall deadline. The gateway can briefly report a just-created sandbox in `Error` while it re-registers the sandbox (seen on DGX Spark); the debounce lets that transient recover to `Ready`. Every terminal observation outside the `Error` phase, including one with no reported phase, fails immediately. Set to `1` to restore fast-fail on the first `Error` poll. | | `NEMOCLAW_GATEWAY_RECOVERY_WAIT_SECONDS` | `30`, `90`, or `120`, depending on the recovery phase | Wall-clock timeout for OpenShell command re-registration after policy application, plus gateway health and re-registration during managed OpenClaw or Hermes recovery. A valid finite, nonnegative value overrides the internal budget for the current recovery phase. | @@ -4096,7 +4096,7 @@ $$nemoclaw recover -If the Ollama pull or post-create readiness timeout fires, onboarding emits the elapsed budget plus a hint to raise the relevant variable. The Ollama pull preserves its partial download for the next attempt. The ordinary post-create readiness wait deletes the orphaned sandbox first so the next `$$nemoclaw onboard` starts without that partially created sandbox. +If the Ollama pull or post-create readiness timeout fires, onboarding emits the elapsed budget plus a hint to raise the relevant variable. The Ollama pull preserves its partial download for the next attempt. A post-create readiness timeout preserves the sandbox. Follow [Inspect retained sandbox recovery](#inspect-retained-sandbox-recovery), and retry onboarding only after retained recovery completes. From 6beb7fe3a9aaed00314de0ecdf2ddbbda3822fa3 Mon Sep 17 00:00:00 2001 From: Julie Yaunches Date: Tue, 8 Sep 2026 12:56:36 -0400 Subject: [PATCH 12/12] docs(sandbox): show retained timeout result Signed-off-by: Julie Yaunches --- docs/reference/troubleshooting.mdx | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/reference/troubleshooting.mdx b/docs/reference/troubleshooting.mdx index 20f8d082da1..1be70340d25 100644 --- a/docs/reference/troubleshooting.mdx +++ b/docs/reference/troubleshooting.mdx @@ -1925,14 +1925,15 @@ Onboarding ends with: ```text Sandbox 'my-assistant' was created but did not become ready within 180s. - Retry: $$nemoclaw onboard + terminal_resolution: timed_out_retained + Recovery remains blocked while sandbox 'my-assistant' exists. ``` This is a separate budget from `NEMOCLAW_LOCAL_INFERENCE_TIMEOUT`. It covers the readiness wait that follows sandbox creation, including in-sandbox boot, OpenClaw start, and policy load. It does not cover the inference probe. -For a newly created OpenClaw or Hermes sandbox, `Ready` is not the final acceptance signal. Within this same budget, NemoClaw also requires OpenShell to return a durable sandbox ID and accept `openshell sandbox exec --name -- true`. NemoClaw keeps waiting only when OpenShell returns its exact `sandbox is not ready` response. A missing or malformed ID, or another command failure, stops the wait. Ordinary onboarding then follows the failed-creation cleanup path. Portable OpenClaw onboarding preserves the sandbox as described below. +For a newly created OpenClaw or Hermes sandbox, `Ready` is not the final acceptance signal. Within this same budget, NemoClaw also requires OpenShell to return a durable sandbox ID and accept `openshell sandbox exec --name -- true`. NemoClaw keeps waiting only when OpenShell returns its exact `sandbox is not ready` response. A missing or malformed ID, or another command failure, stops the wait and preserves the sandbox. Inspect the gateway, then run `$$nemoclaw destroy` to check whether authoritative absence permits cleanup. Do not retry same-name onboarding while the retained sandbox exists.