diff --git a/docs/reference/commands.mdx b/docs/reference/commands.mdx index f85fd1597cc..dd4c506c1b5 100644 --- a/docs/reference/commands.mdx +++ b/docs/reference/commands.mdx @@ -379,24 +379,24 @@ OpenClaw onboarding does not enter the `complete` state until NemoClaw proves th Before the configuration review, NemoClaw records the sandbox name and the selected provider and model as an incomplete choice. If onboarding stops at the review prompt, an interactive `--resume` run shows the prompt again. A non-interactive `--resume` run reuses the recorded choice and continues to inference setup. After you choose **Apply configuration**, NemoClaw records the choice before inference setup starts. If inference setup fails, `--resume` reuses the accepted provider, model, and sandbox name. If you choose **Exit onboarding**, onboarding exits with a nonzero status and clears those recorded choices. Run `$$nemoclaw onboard` to make new choices after exit. During a resume without terminal input, `--yes` or `NEMOCLAW_YES=1` also selects non-interactive resume behavior. For a new or fresh session, `--yes` and `NEMOCLAW_YES=1` accept supported confirmations but do not replace `--non-interactive`. If onboarding returns without reaching the final `complete` state, the command exits with status `1`. When that result is resumable, NemoClaw keeps the session `in_progress` at its last checkpoint instead of marking it failed, so correct the reported condition and run `$$nemoclaw onboard --resume`. -#### Recover a retained sandbox +#### Inspect retained sandbox recovery If onboarding cannot complete after sandbox creation, NemoClaw preserves the sandbox. When available, NemoClaw records and prints the create-attempt label as the exact `ai.nvidia.nemoclaw.create-attempt=` selector. When available, NemoClaw also records a durable identity fingerprint for recovery. Automatic and explicit resume, reuse, recreation, and fresh onboarding with that sandbox name remain blocked. -When the recovery record contains a durable identity fingerprint, run `$$nemoclaw destroy` to attempt identity-bound recovery. +Run `$$nemoclaw destroy` to check retained recovery and reconcile verified residual state. Use the result from `destroy` to choose the next action: -- If OpenShell still reports the sandbox present, `destroy` preserves the record and removes no resources. Do not delete the sandbox manually by mutable name. Give the displayed create-attempt label to an OpenShell administrator when present and ask them to remove that exact sandbox through an identity-bound procedure. Without a label, preserve the terminal output and ask the administrator to identify the exact sandbox from gateway or controller evidence. -- After the administrator removes the exact sandbox, run `$$nemoclaw destroy --yes` to reconcile the recovery record. -- If OpenShell confirms the sandbox is absent, `destroy` verifies one retained recovery record and its immutable runtime identity. For Docker-backed sandboxes, it also verifies the immutable Docker sandbox identity and removes only qualified residual containers. It clears the matching recovery record only after verified cleanup. +- If OpenShell reports the sandbox present, `destroy` preserves the record and removes no resources. OpenShell exposes no atomic delete-by-identity primitive. Neither NemoClaw nor manual inspection can bind a later mutable-name delete to the durable identity fingerprint. You can inspect the owning gateway for diagnosis, but do not delete the sandbox by mutable name. Live retained-sandbox removal is not supported yet. Use a different explicit sandbox name while this record remains blocked. +- If OpenShell cannot determine whether the sandbox is present, `destroy` reports unknown presence, removes no resources, and preserves the recovery record. Correct the OpenShell gateway or list failure, then rerun `destroy`. +- If OpenShell confirms the sandbox is absent, `destroy` selects one retained record from its registry generation, gateway, create-attempt evidence, and available identity fingerprint. Provider and Docker cleanup then apply their own immutable identity checks. `destroy` removes only qualified residual resources and clears the matching recovery record only after verified cleanup. - If the recovery record is ambiguous or immutable runtime-identity verification fails, cleanup stops and the record remains. For Docker-backed sandboxes, a foreign container, changed Docker identity, or failed Docker probe has the same fail-closed result. -If OpenShell did not return a durable identity fingerprint, `destroy` cannot complete recovery. A create-attempt label can help an OpenShell administrator identify and remove the exact sandbox, but it does not let NemoClaw retire the record without immutable identity authority. If neither a fingerprint nor a label is available, preserve the terminal output and ask the administrator to identify the exact sandbox from gateway or controller evidence. If NemoClaw reports that it could not save recovery evidence, rerun `$$nemoclaw destroy --yes`; if recovery remains blocked, preserve the terminal output and follow the same escalation. +If OpenShell did not return a durable identity fingerprint, `destroy` cannot authorize deletion of a live sandbox. It can still select one recovery record that matches the immutable registry generation, gateway, and create-attempt evidence. It clears that record only after the owning gateway reports absence and verified residual cleanup succeeds. If the sandbox is present or presence is unknown, preserve the record and do not delete by mutable name. If NemoClaw could not save recovery evidence, preserve the registry entry and terminal output. -This fail-closed record keeps only the affected sandbox name unavailable. It is not retired from mutable-name absence alone, and this command does not accept administrator-supplied identity authority. +This fail-closed record keeps only the affected sandbox name unavailable. This command does not accept externally supplied identity authority. To onboard another sandbox while the record remains unresolved, supply a different explicit name: ```bash @@ -443,7 +443,7 @@ $$nemoclaw onboard --fresh --apf-interceptor --name my-apf-sandbox If post-create verification or native GPU fallback fails after OpenShell may have created the sandbox, NemoClaw preserves the incomplete sandbox because automatic deletion would use its mutable name. -Follow the [retained-sandbox recovery procedure](#recover-a-retained-sandbox); its OpenShell-absence, immutable runtime-identity, and administrator requirements also apply to APF creation. For Docker-backed APF creation, its Docker container-identity requirement also applies. +Follow the [retained-sandbox recovery procedure](#inspect-retained-sandbox-recovery); its OpenShell-absence and immutable runtime-identity requirements also apply to APF creation. For Docker-backed APF creation, its Docker container-identity requirement also applies. This onboarding mode does not support `--resume` or `--recreate-sandbox`, regardless of whether sandbox creation began. After destroy completes, repeat the original command with `--fresh`. @@ -737,15 +737,15 @@ If you enable Telegram during onboarding, the wizard can also prompt for whether -If you cancel a brand-new onboarding run at the policy-tier selector or either policy-preset selector after sandbox creation, NemoClaw preserves the incomplete sandbox, registry entry, and onboarding session for identity-bound recovery. +If you cancel a brand-new onboarding run at the policy-tier selector or either policy-preset selector after sandbox creation, NemoClaw preserves the incomplete sandbox, registry entry, and onboarding session for retained recovery. NemoClaw reports the durable sandbox identity fingerprint when it is available. It does not run OpenShell's mutable-name deletion command because the name may now identify a replacement sandbox. -Follow the [retained-sandbox recovery procedure](#recover-a-retained-sandbox) to reconcile this cancellation. +Follow the [retained-sandbox recovery procedure](#inspect-retained-sandbox-recovery) to reconcile this cancellation. A fresh run with a different explicit name can continue while the cancelled sandbox name remains blocked. If you run onboarding again with the same sandbox name and choose a different inference provider or model, NemoClaw detects the drift and recreates the sandbox so the running agent config matches your selection. In interactive mode, the wizard asks for confirmation before delete and recreate. In non-interactive mode, NemoClaw recreates automatically when the stored selection is readable and differs. For managed Deep Agents Code sandboxes, NemoClaw also recreates when the live `dcode identity` selection is unreadable; other agent paths continue to reuse by default when their stored selection cannot be read. Set `NEMOCLAW_RECREATE_SANDBOX=1` to force recreation even when no drift is detected. -Before deleting an existing sandbox during recreation, NemoClaw backs up the workspace state declared by the selected agent profile and restores it into the new sandbox once it is live. This applies whether the existing sandbox is ready or marked not-ready, so cross-version upgrades that pass `NEMOCLAW_RECREATE_SANDBOX=1` no longer drop user files from the selected agent workspace. The behaviour matches `$$nemoclaw rebuild --force`. NemoClaw aborts the recreate when the backup cannot complete in full, including when individual state directories or files fail mid-backup, so failed entries are not silently dropped on delete. If restoration into the replacement does not complete, NemoClaw reports the failed directories, files, and restore reason that are available, preserves the snapshot, leaves the replacement unregistered, and exits nonzero. Run the owner-scoped `openshell sandbox delete -g '' ''` command that onboarding prints, then rerun the original onboarding command. If NemoClaw cannot identify the owning gateway, do not delete a same-name sandbox. Preserve the snapshot for manual recovery. Set `NEMOCLAW_RECREATE_WITHOUT_BACKUP=1` to skip the pre-recreate backup. The destination sandbox starts with a fresh workspace. +Before deleting an existing sandbox during recreation, NemoClaw backs up the workspace state declared by the selected agent profile and restores it into the new sandbox once it is live. This applies whether the existing sandbox is ready or marked not-ready, so cross-version upgrades that pass `NEMOCLAW_RECREATE_SANDBOX=1` no longer drop user files from the selected agent workspace. The behaviour matches `$$nemoclaw rebuild --force`. NemoClaw aborts the recreate when the backup cannot complete in full, including when individual state directories or files fail mid-backup, so failed entries are not silently dropped on delete. If restoration into the replacement does not complete, NemoClaw reports the failed directories, files, and restore reason that are available, preserves the snapshot, leaves the replacement unregistered, and exits nonzero. Recovery remains blocked while that unregistered sandbox exists. Do not delete it by mutable name. Preserve the snapshot, and run `$$nemoclaw destroy` to check for authoritative absence before you retry onboarding. Set `NEMOCLAW_RECREATE_WITHOUT_BACKUP=1` to skip the pre-recreate backup. The destination sandbox starts with a fresh workspace. Before deletion, onboarding prints a `Journaled replacement` diagnostic with the replacement identifier, recorded OpenShell gateway, and current phase. If the process stops after this point, a later same-target onboarding run continues the active replacement without requiring `--resume`. It accepts a ready same-name replacement only when the live identity and sandbox registry generation match the journal. It fails closed if the gateway, source, target, durable source registry fields, or replacement settings changed. @@ -4061,7 +4061,7 @@ The following environment variables tune onboard-time and recovery wall-clock li | `NEMOCLAW_HF_DOWNLOAD_STALL_TIMEOUT` | `600` (10 minutes) | Maximum silence between Hugging Face download output during onboard. A positive finite value in seconds overrides the default, up to the Node.js timer limit of about 24.8 days. Blank, invalid, non-positive, sub-millisecond, and oversized values use the default. This is not a total download limit. Increase it only when a working download can produce no output for ten minutes. | | `NEMOCLAW_ONBOARD_VALIDATION_TIMEOUT_SECONDS` | Probe-specific | Connection and total deadlines for applicable OpenAI-compatible provider validation during onboard, including DeepSeek V4 Pro streaming validation. Positive finite seconds round up and cap at `600`; the value only raises existing deadlines. It does not extend the separate fixed five-second streaming-event probe. | | `NEMOCLAW_LOCAL_INFERENCE_TIMEOUT` | `180` | Wall-clock timeout for the inference-server validation probe during onboard, in seconds. Raise on slow networks or for very large prompts. | -| `NEMOCLAW_SANDBOX_READY_TIMEOUT` | `180` | Wall-clock timeout for post-create readiness, in seconds. Raise the timeout when the managed-image pull, explicit custom image build, gateway upload, or in-sandbox boot exceeds the default (typical on 70B+ models, first-time gateway uploads over slow links, or DGX Station / remote-VM first runs). Ordinary onboarding deletes the partially created sandbox when the deadline expires and prints the retry hint. Portable OpenClaw onboarding instead preserves the sandbox when NemoClaw cannot verify its runtime identity. | +| `NEMOCLAW_SANDBOX_READY_TIMEOUT` | `180` | Wall-clock timeout for post-create readiness, in seconds. Raise the timeout when the managed-image pull, explicit custom image build, gateway upload, or in-sandbox boot exceeds the default (typical on 70B+ models, first-time gateway uploads over slow links, or DGX Station / remote-VM first runs). Onboarding preserves a partially created sandbox when the deadline expires. Follow [Inspect retained sandbox recovery](#inspect-retained-sandbox-recovery); do not delete the sandbox by mutable name. | | `NEMOCLAW_SANDBOX_READY_ERROR_DEBOUNCE` | `30` | Consecutive `Error`-phase polls the post-create readiness wait tolerates before treating `Error` as terminal. Polling starts at 250ms and backs off to a 2-second cap, while `NEMOCLAW_SANDBOX_READY_TIMEOUT` remains the overall deadline. The gateway can briefly report a just-created sandbox in `Error` while it re-registers the sandbox (seen on DGX Spark); the debounce lets that transient recover to `Ready`. Every terminal observation outside the `Error` phase, including one with no reported phase, fails immediately. Set to `1` to restore fast-fail on the first `Error` poll. | | `NEMOCLAW_GATEWAY_RECOVERY_WAIT_SECONDS` | `30`, `90`, or `120`, depending on the recovery phase | Wall-clock timeout for OpenShell command re-registration after policy application, plus gateway health and re-registration during managed OpenClaw or Hermes recovery. A valid finite, nonnegative value overrides the internal budget for the current recovery phase. | @@ -4096,7 +4096,7 @@ $$nemoclaw recover -If the Ollama pull or post-create readiness timeout fires, onboarding emits the elapsed budget plus a hint to raise the relevant variable. The Ollama pull preserves its partial download for the next attempt. The ordinary post-create readiness wait deletes the orphaned sandbox first so the next `$$nemoclaw onboard` starts without that partially created sandbox. +If the Ollama pull or post-create readiness timeout fires, onboarding emits the elapsed budget plus a hint to raise the relevant variable. The Ollama pull preserves its partial download for the next attempt. A post-create readiness timeout preserves the sandbox. Follow [Inspect retained sandbox recovery](#inspect-retained-sandbox-recovery), and retry onboarding only after retained recovery completes. diff --git a/docs/reference/troubleshooting.mdx b/docs/reference/troubleshooting.mdx index 8dcc5c1a2fd..1be70340d25 100644 --- a/docs/reference/troubleshooting.mdx +++ b/docs/reference/troubleshooting.mdx @@ -1925,14 +1925,15 @@ Onboarding ends with: ```text Sandbox 'my-assistant' was created but did not become ready within 180s. - Retry: $$nemoclaw onboard + terminal_resolution: timed_out_retained + Recovery remains blocked while sandbox 'my-assistant' exists. ``` This is a separate budget from `NEMOCLAW_LOCAL_INFERENCE_TIMEOUT`. It covers the readiness wait that follows sandbox creation, including in-sandbox boot, OpenClaw start, and policy load. It does not cover the inference probe. -For a newly created OpenClaw or Hermes sandbox, `Ready` is not the final acceptance signal. Within this same budget, NemoClaw also requires OpenShell to return a durable sandbox ID and accept `openshell sandbox exec --name -- true`. NemoClaw keeps waiting only when OpenShell returns its exact `sandbox is not ready` response. A missing or malformed ID, or another command failure, stops the wait. Ordinary onboarding then follows the failed-creation cleanup path. Portable OpenClaw onboarding preserves the sandbox as described below. +For a newly created OpenClaw or Hermes sandbox, `Ready` is not the final acceptance signal. Within this same budget, NemoClaw also requires OpenShell to return a durable sandbox ID and accept `openshell sandbox exec --name -- true`. NemoClaw keeps waiting only when OpenShell returns its exact `sandbox is not ready` response. A missing or malformed ID, or another command failure, stops the wait and preserves the sandbox. Inspect the gateway, then run `$$nemoclaw destroy` to check whether authoritative absence permits cleanup. Do not retry same-name onboarding while the retained sandbox exists. @@ -1949,7 +1950,7 @@ export NEMOCLAW_SANDBOX_READY_TIMEOUT=600 $$nemoclaw onboard ``` -The variable accepts seconds and applies to the readiness wait only. When the ordinary create deadline expires, NemoClaw tries to delete the partially created sandbox. After successful cleanup, the output ends with `Retry: $$nemoclaw onboard`. If cleanup fails, NemoClaw instead reports that the failed sandbox could not be removed and prints `Manual cleanup: openshell sandbox delete ""`. +The variable accepts seconds and applies to the readiness wait only. When the ordinary create deadline expires, NemoClaw preserves the partially created sandbox because OpenShell deletion targets its mutable name. Same-name onboarding remains blocked while that sandbox exists. Run `$$nemoclaw destroy` to check retained recovery. Do not delete the sandbox by mutable name. @@ -2742,13 +2743,7 @@ Starting with NemoClaw v0.0.43, the standard installer handles the `/proc// If an older release fails direct GPU proof with that path and `Permission denied`, upgrade NemoClaw and rerun onboarding. -When inspection confirms that the failed sandbox remains, delete it with a command such as: - -```bash -openshell sandbox delete -``` - -Fix the NVIDIA Container Toolkit or CDI configuration reported in the diagnostics. Run the deletion command only after confirming that the pre-patch sandbox was not restored, then rerun onboarding. If you do not need GPU access inside the sandbox, rerun with `--no-sandbox-gpu`. +When the failed sandbox remains, recovery stays blocked because OpenShell deletion targets its mutable name. Inspection is diagnostic only and does not authorize deletion. Run `$$nemoclaw destroy` to check for authoritative absence. Fix the NVIDIA Container Toolkit or CDI configuration reported in the diagnostics. Rerun onboarding only after retained recovery completes. If you do not need GPU access inside the sandbox, use `--no-sandbox-gpu` when onboarding can continue. If sandbox creation fails with `CDI device injection failed: unresolvable CDI devices nvidia.com/gpu=all`, the OpenShell gateway tried `docker create --device nvidia.com/gpu=all` and Docker could not resolve the CDI spec. This injection happens inside the gateway, so `NEMOCLAW_DOCKER_GPU_PATCH=0` does not bypass it. Rerun with `--no-gpu`, or set `NEMOCLAW_SANDBOX_GPU=0` and resume onboarding. diff --git a/src/lib/actions/sandbox/destroy-preflight.ts b/src/lib/actions/sandbox/destroy-preflight.ts index 644287b2679..ca1f95d874e 100644 --- a/src/lib/actions/sandbox/destroy-preflight.ts +++ b/src/lib/actions/sandbox/destroy-preflight.ts @@ -24,7 +24,10 @@ import type { import type { SandboxEntry } from "../../state/registry"; import * as registry from "../../state/registry"; import { type DestroyRunOpenshell, selectGatewayForSandboxDestroy } from "./destroy-gateway"; -import { classifyDestroySandboxPresence } from "./destroy-presence"; +import { + classifyDestroySandboxPresence, + type DestroySandboxPresence, +} from "./destroy-presence"; import { getPersistedSandboxTargetGatewayName, getSandboxTargetGatewayName, @@ -40,6 +43,7 @@ export type SandboxDestroyPreflight = { selectedRunOpenshell: DestroyRunOpenshell; sandbox: SandboxEntry | null; sandboxConfirmedAbsent: boolean; + sandboxPresence?: DestroySandboxPresence; }; export function resolveSandboxDestroyRuntimeSelection( @@ -350,13 +354,14 @@ export function prepareSandboxDestroy( timeout: OPENSHELL_PROBE_TIMEOUT_MS, }), ); - const sandboxConfirmedAbsent = sandboxPresence === "absent"; + return { cleanupGatewayName, runOpenshell, selectedRunOpenshell, sandbox, - sandboxConfirmedAbsent, + sandboxConfirmedAbsent: sandboxPresence === "absent", + sandboxPresence, ...(selectedCaptureOpenshell ? { selectedCaptureOpenshell } : {}), ...(runtimeSelection ? { runtimeSelection } : {}), }; diff --git a/src/lib/actions/sandbox/destroy-retained-recovery-flow.test.ts b/src/lib/actions/sandbox/destroy-retained-recovery-flow.test.ts index e30d8085284..8fb160696ae 100644 --- a/src/lib/actions/sandbox/destroy-retained-recovery-flow.test.ts +++ b/src/lib/actions/sandbox/destroy-retained-recovery-flow.test.ts @@ -32,6 +32,14 @@ function retainedRecoveryRecord(sandboxId = "sb-alpha"): RetainedSandboxRecovery }; } +function retainedRecoveryRecordWithoutIdentity(): RetainedSandboxRecoveryRecord { + return { + ...retainedRecoveryRecord(), + sandboxIdentityFingerprint: null, + identityWasUnavailable: true, + }; +} + describe("destroySandbox retained recovery flow", () => { let exitSpy: MockInstance; let originalGatewayEnv: string | undefined; @@ -217,7 +225,7 @@ describe("destroySandbox retained recovery flow", () => { ); expect(harness.errorSpy).toHaveBeenCalledWith( - expect.stringContaining("delete command accepts only the mutable sandbox name"), + expect.stringContaining("cannot bind a mutable-name delete to the retained record"), ); expect(harness.runOpenshellSpy).not.toHaveBeenCalledWith( ["sandbox", "delete", "alpha"], @@ -251,12 +259,169 @@ describe("destroySandbox retained recovery flow", () => { ); expect(harness.errorSpy).toHaveBeenCalledWith( - expect.stringContaining("cannot bind that deletion to the retained immutable identity"), + expect.stringContaining("cannot bind a mutable-name delete to the retained record"), + ); + expect(harness.runOpenshellSpy).not.toHaveBeenCalledWith( + ["sandbox", "delete", "alpha"], + expect.anything(), + ); + expect(harness.resolveRetainedSandboxRecoverySpy).not.toHaveBeenCalled(); + expect(harness.removeSandboxSpy).not.toHaveBeenCalled(); + }, + ); + + it( + "still fails closed for a live retained sandbox even with a proven-matching OpenShell identity (#10863)", + { timeout: 30_000 }, + async () => { + // OpenShell exposes no atomic delete-by-identity primitive, so no + // amount of identity proof inside NemoClaw can close the window where + // another OpenShell client replaces the sandbox under the same name + // between the last read and OpenShell processing the delete. Automatic + // deletion of a live retained sandbox is therefore always fail-closed, + // even when the live OpenShell id matches the retained record. + const recovery = retainedRecoveryRecord("sandbox-alpha"); + const containerId = "a".repeat(64); + const harness = createDestroyHarness({ + dockerRunResult: { + status: 0, + stdout: `${containerId}\topenshell\tdefault\tsandbox-alpha`, + }, + registryEntryOverrides: { + lifecycleGeneration: recovery.lifecycleGeneration!, + lifecycleLiveIdentityFingerprint: recovery.sandboxIdentityFingerprint!, + }, + retainedRecoveryRecords: [recovery], + }); + + await expect(harness.destroySandbox("alpha", { yes: true })).rejects.toThrow( + "process.exit(1)", + ); + + expect(harness.errorSpy).toHaveBeenCalledWith( + expect.stringContaining("cannot bind a mutable-name delete to the retained record"), + ); + expect(harness.errorSpy.mock.calls.flat().join("\n")).not.toContain( + "openshell sandbox delete", + ); + expect(harness.errorSpy).toHaveBeenCalledWith( + expect.stringContaining(`openshell sandbox list -g ${recovery.gatewayName} -o json`), + ); + expect(harness.runOpenshellSpy).not.toHaveBeenCalledWith( + ["sandbox", "delete", "alpha"], + expect.anything(), + ); + expect(harness.resolveRetainedSandboxRecoverySpy).not.toHaveBeenCalled(); + }, + ); + + it( + "reconciles a lone recovery record without identity after OpenShell confirms absence (#10863)", + { timeout: 30_000 }, + async () => { + const recovery = retainedRecoveryRecordWithoutIdentity(); + const harness = createDestroyHarness({ + sandboxPresent: false, + dockerRunResult: { status: 0, stdout: "" }, + registryEntryOverrides: { + lifecycleGeneration: recovery.lifecycleGeneration!, + }, + retainedRecoveryRecords: [recovery], + }); + + await expect(harness.destroySandbox("alpha", { yes: true })).resolves.toBeUndefined(); + + expect(harness.runOpenshellSpy).not.toHaveBeenCalledWith( + ["sandbox", "delete", "alpha"], + expect.anything(), ); + expect(harness.resolveRetainedSandboxRecoverySpy).toHaveBeenCalledWith(recovery); + expect(harness.removeSandboxSpy).toHaveBeenCalledWith("alpha"); + expect(exitSpy).not.toHaveBeenCalled(); + }, + ); + + it.each([ + ["present", undefined, "reports a sandbox present"], + [ + "a failed list command", + { status: 1, stdout: "", stderr: "" }, + "could not determine", + ], + [ + "list diagnostics", + { status: 0, stdout: "[]", stderr: "gateway unavailable" }, + "could not determine", + ], + [ + "malformed list output", + { status: 0, stdout: "not-json", stderr: "" }, + "could not determine", + ], + ])( + "preserves a recovery record without identity when OpenShell reports %s (#10863)", + { timeout: 30_000 }, + async (_case, sandboxListResult, expectedPresence) => { + const recovery = retainedRecoveryRecordWithoutIdentity(); + const harness = createDestroyHarness({ + dockerRunResult: { status: 0, stdout: "" }, + registryEntryOverrides: { + lifecycleGeneration: recovery.lifecycleGeneration!, + }, + retainedRecoveryRecords: [recovery], + ...(sandboxListResult ? { sandboxListResult } : {}), + }); + + await expect(harness.destroySandbox("alpha", { yes: true })).rejects.toThrow( + "process.exit(1)", + ); + expect(harness.runOpenshellSpy).not.toHaveBeenCalledWith( ["sandbox", "delete", "alpha"], expect.anything(), ); + expect(harness.dockerRunSpy).not.toHaveBeenCalledWith( + ["rm", "-f", expect.any(String)], + expect.anything(), + ); + expect(harness.resolveRetainedSandboxRecoverySpy).not.toHaveBeenCalled(); + expect(harness.removeSandboxSpy).not.toHaveBeenCalled(); + const output = harness.errorSpy.mock.calls.flat().join("\n"); + expect(output).not.toContain("openshell sandbox delete"); + expect(output).toContain("Preserve"); + expect(output).toContain(expectedPresence); + }, + ); + + it( + "preserves a recovery record without identity when a residual container exists (#10863)", + { timeout: 30_000 }, + async () => { + const recovery = retainedRecoveryRecordWithoutIdentity(); + const containerId = "a".repeat(64); + const harness = createDestroyHarness({ + sandboxPresent: false, + dockerRunResult: { + status: 0, + stdout: `${containerId}\topenshell\tdefault\tsb-alpha`, + }, + registryEntryOverrides: { + lifecycleGeneration: recovery.lifecycleGeneration!, + }, + retainedRecoveryRecords: [recovery], + }); + + await expect(harness.destroySandbox("alpha", { yes: true })).rejects.toThrow( + "process.exit(1)", + ); + + expect(harness.errorSpy).toHaveBeenCalledWith( + expect.stringContaining("has no durable sandbox identity"), + ); + expect(harness.dockerRunSpy).not.toHaveBeenCalledWith( + ["rm", "-f", containerId], + expect.anything(), + ); expect(harness.resolveRetainedSandboxRecoverySpy).not.toHaveBeenCalled(); expect(harness.removeSandboxSpy).not.toHaveBeenCalled(); }, diff --git a/src/lib/actions/sandbox/destroy.ts b/src/lib/actions/sandbox/destroy.ts index d960c0b646b..71ba1867cbc 100644 --- a/src/lib/actions/sandbox/destroy.ts +++ b/src/lib/actions/sandbox/destroy.ts @@ -100,9 +100,7 @@ function selectRetainedSandboxRecoveryAuthority( sandbox: registry.SandboxEntry | null, records: readonly onboardSession.RetainedSandboxRecoveryRecord[], ): onboardSession.RetainedSandboxRecoveryRecord | null { - const candidates = records.filter( - (record) => record.sandboxName === sandboxName && record.sandboxIdentityFingerprint !== null, - ); + const candidates = records.filter((record) => record.sandboxName === sandboxName); if (!sandbox) { // Once resource cleanup has removed the registry row, a retry must still // select the lone durable record so the later Docker proof can confirm @@ -112,10 +110,11 @@ function selectRetainedSandboxRecoveryAuthority( if (candidates.length === 0) return null; const observation = observeDestroyContainerIdentity(sandboxName); const observedMatches = candidates.filter((record) => { + if (record.sandboxIdentityFingerprint === null) return false; const verdict = classifyDestroyContainerIdentity( sandboxName, observation, - record.sandboxIdentityFingerprint!, + record.sandboxIdentityFingerprint, ); return ( verdict.status === "recovery" || (verdict.status === "clear" && verdict.identity !== null) @@ -133,7 +132,8 @@ function selectRetainedSandboxRecoveryAuthority( record.gatewayName === pending.gatewayName && record.gatewayPort === pending.gatewayPort && record.lifecycleGeneration === pending.lifecycleGeneration && - record.sandboxIdentityFingerprint === pending.sandboxIdentityFingerprint && + (record.sandboxIdentityFingerprint === null || + record.sandboxIdentityFingerprint === pending.sandboxIdentityFingerprint) && (pending.createAttemptNonce === undefined || record.createAttemptNonce === pending.createAttemptNonce) ); @@ -142,7 +142,8 @@ function selectRetainedSandboxRecoveryAuthority( record.gatewayName === sandbox.gatewayName && record.gatewayPort === sandbox.gatewayPort && record.lifecycleGeneration === sandbox.lifecycleGeneration && - record.sandboxIdentityFingerprint === sandbox.lifecycleLiveIdentityFingerprint + (record.sandboxIdentityFingerprint === null || + record.sandboxIdentityFingerprint === sandbox.lifecycleLiveIdentityFingerprint) ); }; const matching = candidates.filter(matchesRegistryAuthority); @@ -675,6 +676,16 @@ async function destroySandboxUnlocked( if (initialIdentity === false) { requestSandboxDestroyExit(1); } + if ( + retainedRecoveryAuthority?.sandboxIdentityFingerprint === null && + initialIdentity?.identities !== undefined && + initialIdentity.identities.length > 0 + ) { + console.error( + ` Refusing to destroy retained sandbox '${sandboxName}': the recovery record has no durable sandbox identity, so NemoClaw cannot qualify a residual container for deletion. No sandbox resources were removed. Preserve the recovery record and resolve the container identity conflict before retrying.`, + ); + requestSandboxDestroyExit(1); + } const initialContainerIdentities = initialIdentity?.identities; let preparedManagedLlamaCppCleanup: ReturnType< @@ -743,10 +754,25 @@ async function destroySandboxUnlocked( selectedRunOpenshell: cleanupRunOpenshell, sandbox, sandboxConfirmedAbsent, + sandboxPresence = sandboxConfirmedAbsent ? "absent" : "unknown", } = destroyPreflight; - if (retainedRecoveryAuthority && !sandboxConfirmedAbsent) { + if (retainedRecoveryAuthority && sandboxPresence !== "absent") { + // OpenShell has no atomic delete-by-identity primitive: it exposes no + // way to bind a mutable-name delete to the retained record's immutable + // sandbox id/resource version. Even a fresh identity read immediately + // before the delete command cannot close the window where another + // OpenShell client removes the retained sandbox and creates a + // replacement under the same name between that read and OpenShell + // processing the delete (#10863). Automatic deletion of a live retained + // sandbox is therefore always fail-closed. Inspection cannot authorize a + // later mutable-name delete, so the recovery record remains unresolved + // until OpenShell can prove absence through the owning gateway. + const presenceDetail = + sandboxPresence === "present" + ? "OpenShell reports a sandbox present under this name." + : "OpenShell could not determine whether a sandbox is present under this name."; console.error( - ` Refusing to automatically delete retained sandbox '${sandboxName}': OpenShell still reports it present, but its delete command accepts only the mutable sandbox name. NemoClaw cannot bind that deletion to the retained immutable identity. No sandbox resources were removed. Ask an OpenShell administrator to resolve create-attempt label '${retainedRecoveryAuthority.createAttemptNonce}' to the exact sandbox and use an identity-bound removal procedure. After OpenShell confirms the retained sandbox is absent, rerun '${CLI_NAME} ${sandboxName} destroy --yes' to reconcile its verified Docker containers and recovery record.`, + ` Refusing to delete retained sandbox '${sandboxName}': ${presenceDetail} NemoClaw cannot bind a mutable-name delete to the retained record (create-attempt label '${retainedRecoveryAuthority.createAttemptNonce}') without an atomic OpenShell delete-by-identity primitive. No sandbox resources were removed. Preserve the recovery record. Inspect 'openshell sandbox list -g ${retainedRecoveryAuthority.gatewayName} -o json' for diagnosis only; do not run mutable-name deletion. Recovery remains blocked until the owning gateway reports the sandbox absent. Then rerun '${CLI_NAME} ${sandboxName} destroy --yes' to reconcile verified residual resources and the recovery record.`, ); preparedManagedLlamaCppCleanup?.abort(); requestSandboxDestroyExit(1); diff --git a/src/lib/actions/sandbox/rebuild-preflight-confirmation.test.ts b/src/lib/actions/sandbox/rebuild-preflight-confirmation.test.ts index 10cd6e19bcc..d7372094365 100644 --- a/src/lib/actions/sandbox/rebuild-preflight-confirmation.test.ts +++ b/src/lib/actions/sandbox/rebuild-preflight-confirmation.test.ts @@ -234,7 +234,10 @@ describe("rebuild preflight guards", () => { listRecovery.mock.invocationCallOrder[0]!, ); expect(bail).toHaveBeenCalledWith("Retained sandbox recovery blocks rebuild for 'alpha'.", 1); - expect(error.mock.calls.flat().join("\n")).toContain("nemoclaw alpha destroy --yes"); + const output = error.mock.calls.flat().join("\n"); + expect(output).toContain("nemoclaw alpha destroy --yes"); + expect(output).toContain("reports the sandbox present or cannot determine presence"); + expect(output).not.toContain("administrator"); }); it("stops after a failed onboard-lock acquisition without releasing another run's lock (#7794)", () => { diff --git a/src/lib/actions/sandbox/rebuild-preflight-guards.ts b/src/lib/actions/sandbox/rebuild-preflight-guards.ts index 49056e8a6e0..d9fcc6ef14d 100644 --- a/src/lib/actions/sandbox/rebuild-preflight-guards.ts +++ b/src/lib/actions/sandbox/rebuild-preflight-guards.ts @@ -339,7 +339,7 @@ export function blockRebuildOnRetainedSandboxRecovery( ` Rebuild cannot use retained sandbox '${sandboxName}' while recovery record '${retainedRecovery.recordId}' is unresolved. No sandbox or Docker resources were removed.`, ); console.error( - ` Run '${CLI_NAME} ${sandboxName} destroy --yes'. If OpenShell still reports the sandbox present, follow destroy's create-attempt label guidance for identity-bound administrator removal.`, + ` Run '${CLI_NAME} ${sandboxName} destroy --yes'. If the owning gateway reports the sandbox present or cannot determine presence, destroy removes nothing and preserves the recovery record.`, ); bail(`Retained sandbox recovery blocks rebuild for '${sandboxName}'.`, 1); return true; diff --git a/src/lib/onboard/cancel-rollback.test.ts b/src/lib/onboard/cancel-rollback.test.ts index d8201d13efe..e172b080ced 100644 --- a/src/lib/onboard/cancel-rollback.test.ts +++ b/src/lib/onboard/cancel-rollback.test.ts @@ -282,7 +282,9 @@ describe("installSandboxCancelRollback", () => { const guidance = log.mock.calls.flat().join("\n"); expect(guidance).toContain("identity fingerprint is unavailable"); - expect(guidance).toContain("OpenShell administrator"); + expect(guidance).toContain("can clear the recovery record only after OpenShell confirms"); + expect(guidance).not.toContain("identify and remove"); + expect(guidance).not.toContain("openshell sandbox delete"); }); }); @@ -309,7 +311,8 @@ describe("buildCancelRollbackMessage", () => { expect(message).toContain("preserved incomplete sandbox 'sb'"); expect(message).toContain(SANDBOX_FINGERPRINT); expect(message).toContain(RECOVERY_CONTEXT.createAttemptNonce); - expect(message).toContain("identity-bound inspection, recovery, or removal"); + expect(message).toContain("retained recovery evidence"); + expect(message).toContain("does not authorize deletion by mutable name"); expect(message).not.toContain("openshell sandbox delete"); expect(message).not.toContain("cannot delete it by immutable identity"); }); diff --git a/src/lib/onboard/cancel-rollback.ts b/src/lib/onboard/cancel-rollback.ts index 90d7e9d2192..320a3236903 100644 --- a/src/lib/onboard/cancel-rollback.ts +++ b/src/lib/onboard/cancel-rollback.ts @@ -69,11 +69,11 @@ export function buildCancelRollbackMessage( ...(sandboxIdentityFingerprint ? [ ` Durable sandbox identity fingerprint: ${sandboxIdentityFingerprint}`, - " Preserve this fingerprint for identity-bound inspection, recovery, or removal.", + " Preserve this fingerprint as retained recovery evidence; it does not authorize deletion by mutable name.", ] : [ " Its durable identity fingerprint is unavailable; preserve the registry and onboarding recovery state.", - " Ask an OpenShell administrator to establish the exact sandbox identity before recovery or removal.", + " NemoClaw cannot verify this sandbox identity or authorize its removal without a recorded fingerprint.", ]), " NemoClaw did not run OpenShell's mutable-name deletion command because the name may now identify a replacement sandbox.", " Do not delete the sandbox by mutable sandbox name.", @@ -82,11 +82,12 @@ export function buildCancelRollbackMessage( ? [ ` Run '${cliName()} ${sandboxName} destroy'. If OpenShell confirms the retained sandbox absent, destroy removes only verified residual containers and can clear the matching recovery record.`, recoveryContext - ? " If it is still live, give the displayed create-attempt label to an OpenShell administrator for identity-bound removal." - : " If it is still live, preserve the displayed fingerprint and ask an OpenShell administrator for identity-bound removal.", + ? " If it is still live or presence is unknown, destroy refuses deletion. Inspect the owning gateway for diagnosis only; do not delete by mutable name." + : " If it is still live or presence is unknown, preserve the displayed fingerprint; destroy refuses deletion and preserves the recovery record.", ] : [ - " NemoClaw cannot clear this recovery record until an OpenShell administrator establishes the exact sandbox identity.", + ` Run '${cliName()} ${sandboxName} destroy'. It can clear the recovery record only after OpenShell confirms the sandbox absent and residual cleanup succeeds.`, + " If the sandbox is present or presence is unknown, preserve the recovery record and do not delete by mutable name.", ]), ]; } @@ -208,7 +209,7 @@ export function createSandboxCancelRollback( guidanceReported = true; if (recoveryPersistenceFailed) { deps.log( - " NemoClaw could not save the onboarding recovery record; preserve the registry entry and exact sandbox identity for administrator recovery.", + " NemoClaw could not save the onboarding recovery record; preserve the registry entry and terminal output. Do not delete the sandbox by mutable name.", ); } for (const line of buildCancelRollbackMessage( diff --git a/src/lib/onboard/created-sandbox-failure.test.ts b/src/lib/onboard/created-sandbox-failure.test.ts index 69fe3edc5d2..fd2f629ae10 100644 --- a/src/lib/onboard/created-sandbox-failure.test.ts +++ b/src/lib/onboard/created-sandbox-failure.test.ts @@ -163,7 +163,6 @@ function readinessDeps( printReadinessFailure: vi.fn(), printCreateFailureDiagnostics: vi.fn(), printDockerGpuReadinessFailure: vi.fn(), - deleteSandbox: vi.fn(() => ({ status: 0 })), cliName: vi.fn(() => "nemoclaw"), error: vi.fn(), exitProcess: vi.fn((code: number): never => { @@ -202,33 +201,12 @@ function expectReceiptBlock( } describe("reportSandboxReadinessFailure", () => { - it("deletes the failed sandbox on the non-GPU path and exits 1", () => { + it("preserves the failed sandbox on the non-GPU path and exits 1", () => { const deps = readinessDeps(); expect(() => reportSandboxReadinessFailure(readinessOptions(), deps)).toThrow(ExitSignal); expect(deps.printReadinessFailure).toHaveBeenCalledWith(NOT_READY, "alpha", 300); expect(deps.printCreateFailureDiagnostics).toHaveBeenCalledWith("alpha", { backupPath: null }); - expect(deps.deleteSandbox).toHaveBeenCalledWith("alpha"); expect(deps.printDockerGpuReadinessFailure).not.toHaveBeenCalled(); - expectReceiptBlock(deps, [ - " Sandbox lifecycle receipt:", - " state: created_but_not_ready", - " sandbox: alpha", - " readiness_gate: sandbox_list:not_ready_timeout", - " readiness_reason: timeout", - " create_stream_status: 0", - " timeout_seconds: 300", - " terminal_resolution: timed_out_deleted", - ]); - expect(deps.error).toHaveBeenCalledWith( - " Deleted sandbox 'alpha' after the readiness gate failed; retry will recreate it.", - ); - expect(deps.error).toHaveBeenCalledWith(" Retry: nemoclaw onboard"); - expect(deps.exitProcess).toHaveBeenCalledWith(1); - }); - - it("surfaces manual cleanup when deletion fails", () => { - const deps = readinessDeps({ deleteSandbox: vi.fn(() => ({ status: 1 })) }); - expect(() => reportSandboxReadinessFailure(readinessOptions(), deps)).toThrow(ExitSignal); expectReceiptBlock(deps, [ " Sandbox lifecycle receipt:", " state: created_but_not_ready", @@ -240,9 +218,10 @@ describe("reportSandboxReadinessFailure", () => { " terminal_resolution: timed_out_retained", ]); expect(deps.error).toHaveBeenCalledWith( - " Could not remove the failed sandbox. Manual cleanup:", + " Recovery remains blocked while sandbox 'alpha' exists. Do not delete it by mutable name; run 'nemoclaw alpha destroy' to check for authoritative absence.", ); - expect(deps.error).toHaveBeenCalledWith(' openshell sandbox delete "alpha"'); + expect(errorLines(deps)).not.toContain(" Retry: nemoclaw onboard"); + expect(deps.exitProcess).toHaveBeenCalledWith(1); }); it("defers cleanup to the Docker-GPU patch and never deletes the sandbox", () => { @@ -251,7 +230,6 @@ describe("reportSandboxReadinessFailure", () => { reportSandboxReadinessFailure(readinessOptions({ useDockerGpuPatch: true }), deps), ).toThrow(ExitSignal); expect(deps.printDockerGpuReadinessFailure).toHaveBeenCalledTimes(1); - expect(deps.deleteSandbox).not.toHaveBeenCalled(); expectReceiptBlock(deps, [ " Sandbox lifecycle receipt:", " state: created_but_not_ready", @@ -287,12 +265,12 @@ describe("reportSandboxReadinessFailure", () => { " readiness_reason: terminal_failure_phase", " create_stream_status: 0", " timeout_seconds: 300", - " terminal_resolution: terminal_failure_deleted", + " terminal_resolution: terminal_failure_retained", ]); }); - it("reports retained cleanup for terminal readiness failures when delete fails", () => { - const deps = readinessDeps({ deleteSandbox: vi.fn(() => ({ status: 1 })) }); + it("reports retained cleanup for terminal readiness failures", () => { + const deps = readinessDeps(); expect(() => reportSandboxReadinessFailure( readinessOptions({ @@ -342,7 +320,7 @@ describe("reportSandboxReadinessFailure", () => { " readiness_reason: terminal_failure_phase", " create_stream_status: 0", " timeout_seconds: 300", - " terminal_resolution: terminal_failure_deleted", + " terminal_resolution: terminal_failure_retained", ]); }); diff --git a/src/lib/onboard/created-sandbox-failure.ts b/src/lib/onboard/created-sandbox-failure.ts index f0c136cd1fd..5050b124b2a 100644 --- a/src/lib/onboard/created-sandbox-failure.ts +++ b/src/lib/onboard/created-sandbox-failure.ts @@ -18,7 +18,7 @@ export function formatRetainedSandboxRecoveryMessage(input: { createAttemptEvidence + `Sandbox '${input.sandboxName}' reached Ready before OpenShell returned one exact durable create identity. Gateway '${input.gatewayName}'. ` + "OpenShell did not return one exact durable sandbox identity for this create attempt. " + - "Do not delete a sandbox by mutable name; preserve it until an OpenShell administrator resolves the create-attempt label to one sandbox." + `Do not delete the sandbox by mutable name. Run '${cliName()} ${input.sandboxName} destroy'; it can clear retained recovery only after OpenShell confirms absence.` ); } return ( @@ -26,9 +26,9 @@ export function formatRetainedSandboxRecoveryMessage(input: { `Durable sandbox identity fingerprint: ${input.sandboxIdentityFingerprint}. ` + `NemoClaw stopped before owning-gateway publication and identity verification completed for sandbox '${input.sandboxName}' through gateway '${input.gatewayName}'. ` + `Do not delete the sandbox by mutable name. Run '${cliName()} ${input.sandboxName} destroy'. ` + - "If OpenShell reports the sandbox present, the command removes nothing and preserves the recovery record. " + - "Give the create-attempt label to an OpenShell administrator for identity-bound removal. " + - `After OpenShell confirms removal, run '${cliName()} ${input.sandboxName} destroy --yes' to reconcile the recovery record.` + "If OpenShell reports the sandbox present or cannot determine presence, the command removes nothing and preserves the recovery record. " + + "Inspection is diagnostic only and does not authorize mutable-name deletion. " + + `Recovery remains blocked while the sandbox is present or presence is unknown. Rerun '${cliName()} ${input.sandboxName} destroy --yes' after the owning gateway reports absence.` ); } @@ -109,7 +109,6 @@ export type SandboxReadinessFailureReportDeps = { ): void; printCreateFailureDiagnostics(sandboxName: string, options: { backupPath: string | null }): void; printDockerGpuReadinessFailure(): void; - deleteSandbox(sandboxName: string): { status: number | null }; cliName(): string; error(message: string): void; exitProcess(code: number): never; @@ -117,20 +116,16 @@ export type SandboxReadinessFailureReportDeps = { export type SandboxReadinessTerminalResolution = | "deferred_to_docker_gpu_patch" - | "terminal_failure_deleted" | "terminal_failure_retained" - | "timed_out_deleted" | "timed_out_retained"; -/** Map the readiness reason and cleanup outcome into the receipt terminal state. */ +/** Map the readiness reason into the retained receipt terminal state. */ function readinessTerminalResolution( readiness: CreatedSandboxReadinessResult, - deleted: boolean, ): SandboxReadinessTerminalResolution { - if (readiness.reason === "terminal_failure_phase") { - return deleted ? "terminal_failure_deleted" : "terminal_failure_retained"; - } - return deleted ? "timed_out_deleted" : "timed_out_retained"; + return readiness.reason === "terminal_failure_phase" + ? "terminal_failure_retained" + : "timed_out_retained"; } /** Name the readiness gate that blocked the created sandbox from becoming Ready. */ @@ -171,8 +166,8 @@ function formatCreatedSandboxReadinessReceipt(options: { /** * Report a sandbox that never reached Ready: print the readiness failure and - * create diagnostics, then either defer cleanup to the Docker-GPU patch or - * delete the failed sandbox so a same-name retry does not collide, and exit. + * create diagnostics, then preserve the failed sandbox for retained recovery + * unless the Docker-GPU patch owns its identity-bound cleanup, and exit. */ export function reportSandboxReadinessFailure( options: SandboxReadinessFailureReportOptions, @@ -195,37 +190,22 @@ export function reportSandboxReadinessFailure( } deps.printDockerGpuReadinessFailure(); } else { - // Clean up non-GPU failures after preserving local diagnostics so the - // next onboard retry with the same name does not fail on "sandbox already exists". - const delResult = deps.deleteSandbox(options.sandboxName); - if (delResult.status === 0) { - for (const line of formatCreatedSandboxReadinessReceipt({ - sandboxName: options.sandboxName, - readiness: options.readiness, - createStatus: options.createStatus, - timeoutSecs: options.timeoutSecs, - terminalResolution: readinessTerminalResolution(options.readiness, true), - })) { - deps.error(line); - } - deps.error( - ` Deleted sandbox '${options.sandboxName}' after the readiness gate failed; retry will recreate it.`, - ); - } else { - for (const line of formatCreatedSandboxReadinessReceipt({ - sandboxName: options.sandboxName, - readiness: options.readiness, - createStatus: options.createStatus, - timeoutSecs: options.timeoutSecs, - terminalResolution: readinessTerminalResolution(options.readiness, false), - })) { - deps.error(line); - } - deps.error(" Could not remove the failed sandbox. Manual cleanup:"); - deps.error(` openshell sandbox delete "${options.sandboxName}"`); + for (const line of formatCreatedSandboxReadinessReceipt({ + sandboxName: options.sandboxName, + readiness: options.readiness, + createStatus: options.createStatus, + timeoutSecs: options.timeoutSecs, + terminalResolution: readinessTerminalResolution(options.readiness), + })) { + deps.error(line); } + deps.error( + ` Recovery remains blocked while sandbox '${options.sandboxName}' exists. Do not delete it by mutable name; run '${deps.cliName()} ${options.sandboxName} destroy' to check for authoritative absence.`, + ); + } + if (options.useDockerGpuPatch) { + deps.error(` Retry: ${deps.cliName()} onboard`); } - deps.error(` Retry: ${deps.cliName()} onboard`); const exitCode = options.createStatus === 0 ? 1 : options.createStatus; return deps.exitProcess(exitCode); } diff --git a/src/lib/onboard/created-sandbox-finalization.test.ts b/src/lib/onboard/created-sandbox-finalization.test.ts index 10b28c20d04..a682f60b084 100644 --- a/src/lib/onboard/created-sandbox-finalization.test.ts +++ b/src/lib/onboard/created-sandbox-finalization.test.ts @@ -108,8 +108,9 @@ describe("new sandbox cancellation recovery", () => { expect(guidance).toContain("Sandbox 'new-sandbox' was created on gateway 'nemoclaw'"); expect(guidance).toContain("registry entry and onboarding session were preserved"); expect(guidance).toContain("Do not delete the sandbox by mutable sandbox name"); - expect(guidance).toContain("establish the exact live durable identity before removal"); - expect(guidance).toContain("add --fresh, and use a new sandbox name"); + expect(guidance).toContain("can clear retained recovery only after OpenShell confirms"); + expect(guidance).toContain("use a different explicit sandbox name"); + expect(guidance).not.toContain("administrator"); expect(runFile).not.toHaveBeenCalled(); expect(armCancelRollback).not.toHaveBeenCalled(); expect(markCancellationRecovery).toHaveBeenCalledOnce(); @@ -649,7 +650,7 @@ describe("created DCode sandbox finalization", () => { expect(register).not.toHaveBeenCalled(); expect(error).toHaveBeenCalledWith(expect.stringContaining("sandbox still exists")); expect(error).toHaveBeenCalledWith(expect.stringContaining("rebuild is unsafe")); - expect(error).toHaveBeenCalledWith(expect.stringContaining("Verify its durable identity")); + expect(error).toHaveBeenCalledWith(expect.stringContaining("Recovery remains blocked while this sandbox exists")); expect(error.mock.calls.flat().join("\n")).not.toContain("openshell sandbox delete"); expect(error).toHaveBeenCalledWith(expect.stringContaining("nemoclaw onboard")); }); @@ -710,7 +711,7 @@ describe("created DCode sandbox finalization", () => { " NemoClaw left unregistered sandbox 'dcode' in place because OpenShell can delete it only by mutable name.", ); expect(error).toHaveBeenCalledWith( - " Verify its durable identity before manual cleanup; do not act by name alone.", + " Recovery remains blocked while this sandbox exists. Do not delete it by mutable name; run 'nemoclaw dcode destroy' to check for authoritative absence.", ); expect(error.mock.calls.flat().join("\n")).not.toContain("openshell sandbox delete"); expect(error).toHaveBeenCalledWith( @@ -1054,7 +1055,7 @@ describe("created OpenClaw sandbox finalization", () => { expect(error).toHaveBeenCalledWith( " State was not restored and registry metadata was not updated.", ); - expect(error).toHaveBeenCalledWith(expect.stringContaining("Verify its durable identity")); + expect(error).toHaveBeenCalledWith(expect.stringContaining("Recovery remains blocked while this sandbox exists")); expect(error.mock.calls.flat().join("\n")).not.toContain("openshell sandbox delete"); expect(error).toHaveBeenCalledWith(" Manual recovery: /tmp/managed-openclaw-backup"); }); @@ -1101,7 +1102,7 @@ describe("created OpenClaw sandbox finalization", () => { expect(error).toHaveBeenCalledWith( " State was not restored and registry metadata was not updated.", ); - expect(error).toHaveBeenCalledWith(expect.stringContaining("Verify its durable identity")); + expect(error).toHaveBeenCalledWith(expect.stringContaining("Recovery remains blocked while this sandbox exists")); expect(error.mock.calls.flat().join("\n")).not.toContain("openshell sandbox delete"); expect(error).toHaveBeenCalledWith( " Then rerun the original `nemoclaw onboard --from ` command.", @@ -1157,7 +1158,7 @@ describe("created OpenClaw sandbox finalization", () => { expect(error).toHaveBeenCalledWith( expect.stringContaining(sandboxState.OPENCLAW_IMAGE_PLUGIN_PROVENANCE_RESTORE_ERROR), ); - expect(error).toHaveBeenCalledWith(expect.stringContaining("Verify its durable identity")); + expect(error).toHaveBeenCalledWith(expect.stringContaining("Recovery remains blocked while this sandbox exists")); expect(error.mock.calls.flat().join("\n")).not.toContain("openshell sandbox delete"); expect(error).toHaveBeenCalledWith( " Then rerun the original `nemoclaw onboard --from ` command.", diff --git a/src/lib/onboard/created-sandbox-finalization.ts b/src/lib/onboard/created-sandbox-finalization.ts index 818c45cbcf9..de81ac8c984 100644 --- a/src/lib/onboard/created-sandbox-finalization.ts +++ b/src/lib/onboard/created-sandbox-finalization.ts @@ -22,6 +22,7 @@ import { type RecreatedSandboxRestoreOptions, type RestoreResult, } from "../state/sandbox"; +import { cliName } from "./branding"; import { createDcodeSelectionDriftReader } from "./dcode-selection-drift"; import { restoreDefaultAfterRecreate } from "./default-preservation"; import * as dockerGpuLocalInference from "./docker-gpu-local-inference"; @@ -300,8 +301,8 @@ export function completeOrdinaryOnboardSandboxCreation( ` Sandbox '${input.sandboxName}' was created on gateway '${deps.gatewayName}', but NemoClaw could not verify its durable identity.`, " The sandbox registry entry and onboarding session were preserved for recovery.", " Do not delete the sandbox by mutable sandbox name.", - " Ask an OpenShell administrator to establish the exact live durable identity before removal.", - " After confirmed identity-bound removal, rerun the original onboarding command with the same required inputs, add --fresh, and use a new sandbox name.", + ` Run '${cliName()} ${input.sandboxName} destroy'. It can clear retained recovery only after OpenShell confirms the sandbox absent.`, + " Until recovery completes, use a different explicit sandbox name for new onboarding.", ]) { console.error(line); } @@ -808,7 +809,9 @@ export function finalizeCreatedSandbox( deps.error( ` NemoClaw left unregistered sandbox '${options.sandboxName}' in place because OpenShell can delete it only by mutable name.`, ); - deps.error(" Verify its durable identity before manual cleanup; do not act by name alone."); + deps.error( + ` Recovery remains blocked while this sandbox exists. Do not delete it by mutable name; run '${cliName()} ${options.sandboxName} destroy' to check for authoritative absence.`, + ); }; let freshOpenClawImagePluginInstalls: readonly OpenClawImagePluginInstall[] | undefined; if (options.discoverOpenClawImagePluginInstalls === true) { diff --git a/src/lib/onboard/dashboard.ts b/src/lib/onboard/dashboard.ts index 4062356bb4b..e29213aa714 100644 --- a/src/lib/onboard/dashboard.ts +++ b/src/lib/onboard/dashboard.ts @@ -373,8 +373,8 @@ export function createOnboardDashboardHelpers(deps: OnboardDashboardDeps): Onboa ]; if (owningGateway) { lines.push( - " Verify the sandbox identity, then clean up manually:", - ` openshell sandbox delete -g ${JSON.stringify(owningGateway)} ${JSON.stringify(sandboxName)}`, + ` Recovery remains blocked while gateway ${JSON.stringify(owningGateway)} reports this sandbox present.`, + ` Do not delete it by mutable name; run 'nemoclaw ${sandboxName} destroy' to check for authoritative absence.`, ); } else { lines.push(" The owning OpenShell gateway is unknown. Do not delete a same-name sandbox."); diff --git a/src/lib/onboard/entry-options.ts b/src/lib/onboard/entry-options.ts index 3b81e4dd41d..648c0f54fb2 100644 --- a/src/lib/onboard/entry-options.ts +++ b/src/lib/onboard/entry-options.ts @@ -477,7 +477,7 @@ export function resolveOnboardEntryOptions( " Onboarding cannot replace the recovery-only session because its independent retained sandbox recovery record is unavailable.", ); deps.error( - " Preserve the session and registry state for identity-bound administrator recovery.", + " Preserve the session, registry state, and terminal output. Do not delete the sandbox by mutable name.", ); deps.exitProcess(1); } diff --git a/src/lib/onboard/exit-step-failure.test.ts b/src/lib/onboard/exit-step-failure.test.ts index 1c267781144..837e96fe8c1 100644 --- a/src/lib/onboard/exit-step-failure.test.ts +++ b/src/lib/onboard/exit-step-failure.test.ts @@ -262,6 +262,10 @@ describe("incomplete-onboard --resume backstop (#6003)", () => { createAttemptNonce: "c".repeat(62), }); const beforeExit = requireLoadedSession(); + expect(beforeExit.failure?.message).toContain( + "retained recovery blocks this sandbox name until destroy confirms absence", + ); + expect(beforeExit.failure?.message).not.toContain("administrator"); const output = runExitHandler(1); diff --git a/src/lib/onboard/lifecycle-contracts.md b/src/lib/onboard/lifecycle-contracts.md index 912e3e18d46..b7906341ef7 100644 --- a/src/lib/onboard/lifecycle-contracts.md +++ b/src/lib/onboard/lifecycle-contracts.md @@ -123,7 +123,7 @@ runtime mutation | Journey and entry | Desired state, planning, and assembly | Visible and destructive boundaries | Checkpoint and secret boundary | Compensation, coverage, and gaps | |---|---|---|---|---| -| **New interactive or non-interactive onboard** — `onboard()` and `resolveOnboardEntryOptions` | Current flags, environment, and prompts. `MessagingWorkflowPlanner.buildPlan`, `prepareSandboxMessagingPreflight`, resource-profile selection, `resolveSandboxCreateIntent`, and `materializeSandboxCreatePlan` assemble policy, provider, package, resource, host-forward, and runtime-setup contributions. Non-interactive mode replaces prompts with defaults or hard aborts. | Consent/session/lock setup and preflight can persist local state, install OpenShell, or clean stale gateway artifacts before the gateway handler. Gateway reuse/recovery/start is the first provider-routing effect; inference-provider upserts follow. For OpenClaw, messaging selection and plan reconciliation complete before web-search or messaging provider registration. Each validated provider group is then created or updated and checkpointed before resource selection. A name with no live sandbox has no sandbox-destructive boundary; an existing target enters the recreate contract below. | Whole-step session plus machine snapshot. OpenClaw adds narrow checkpoints after each completed secret-free sandbox prompt group; sandbox registry registration is deferred until readiness and live validation. The session stores credential environment names, redacted endpoint metadata, legacy-value digests, and non-secret names of web-search and messaging providers registered for resume; real values remain process- or gateway-bound. | Readiness, post-create policy verification, dashboard forwarding, and cancellation failures preserve the live sandbox and an independent identity-bound recovery record. A later `destroy` refuses mutable-name deletion while that sandbox is live. After administrator identity-bound removal, destroy uses the record to qualify immutable runtime identity and, for Docker-backed sandboxes, exact container identities before residual cleanup and record retirement. A different explicit sandbox name starts a fresh session without changing the retained record. Exact provider-owned GPU cleanup can proceed through its owner receipt. NemoClaw attempts to remove temporary policy and build-context sources and reports cleanup failures with the onboarding error; post-create failures retain recovery state. Cancellation before sandbox creation can leave the session resumable. Shared inference providers remain gateway configuration and are not sandbox cleanup targets. Coverage: `transition-traces.test.ts`, `sandbox-create-intent-boundary.test.ts`, `sandbox-create-plan.test.ts`, and the focused cancellation, readiness, GPU cleanup, dashboard, policy-authority, destroy, and retained-recovery tests. Gap: gateway upserts can outlive a failed or interrupted create. | +| **New interactive or non-interactive onboard** — `onboard()` and `resolveOnboardEntryOptions` | Current flags, environment, and prompts. `MessagingWorkflowPlanner.buildPlan`, `prepareSandboxMessagingPreflight`, resource-profile selection, `resolveSandboxCreateIntent`, and `materializeSandboxCreatePlan` assemble policy, provider, package, resource, host-forward, and runtime-setup contributions. Non-interactive mode replaces prompts with defaults or hard aborts. | Consent/session/lock setup and preflight can persist local state, install OpenShell, or clean stale gateway artifacts before the gateway handler. Gateway reuse/recovery/start is the first provider-routing effect; inference-provider upserts follow. For OpenClaw, messaging selection and plan reconciliation complete before web-search or messaging provider registration. Each validated provider group is then created or updated and checkpointed before resource selection. A name with no live sandbox has no sandbox-destructive boundary; an existing target enters the recreate contract below. | Whole-step session plus machine snapshot. OpenClaw adds narrow checkpoints after each completed secret-free sandbox prompt group; sandbox registry registration is deferred until readiness and live validation. The session stores credential environment names, redacted endpoint metadata, legacy-value digests, and non-secret names of web-search and messaging providers registered for resume; real values remain process- or gateway-bound. | Readiness, post-create policy verification, dashboard forwarding, and cancellation failures preserve the live sandbox and an independent identity-bound recovery record. A later `destroy` refuses mutable-name deletion while that sandbox is live. Recovery remains blocked while the sandbox is present or its presence is unknown. After OpenShell authoritatively reports absence, destroy uses the record to qualify immutable runtime identity and, for Docker-backed sandboxes, exact container identities before residual cleanup and record retirement. A different explicit sandbox name starts a fresh session without changing the retained record. Exact provider-owned GPU cleanup can proceed through its owner receipt. NemoClaw attempts to remove temporary policy and build-context sources and reports cleanup failures with the onboarding error; post-create failures retain recovery state. Cancellation before sandbox creation can leave the session resumable. Shared inference providers remain gateway configuration and are not sandbox cleanup targets. Coverage: `transition-traces.test.ts`, `sandbox-create-intent-boundary.test.ts`, `sandbox-create-plan.test.ts`, and the focused cancellation, readiness, GPU cleanup, dashboard, policy-authority, destroy, and retained-recovery tests. Gap: gateway upserts can outlive a failed or interrupted create. | | **`--fresh` onboard** — `resolveOnboardEntryOptions`, `prepareFreshSession`, `createBaseImageResolutionContext` | Current flags/environment/prompts replace resumable intent. `--fresh` disables auto-resume and forces base-image resolution; it does not prove that the selected sandbox name is unused. | The first destructive effect is local: the prior onboard session is cleared before a new session is saved. A matching live sandbox can later reuse or recreate through the normal sandbox decision; `--fresh` does not itself delete it. | The new session and machine snapshot replace the old resume checkpoint. Credential and effect boundaries then match new onboard or live recreate. | The discarded resume checkpoint is not restored on later failure. Covered by `entry-options.test.ts`, `session-bootstrap.test.ts`, and base-image resolution tests. | | **Resume, re-onboard, or recreate** — `onboard()`, `prepareOnboardSession`, `decideSandboxResume`, live-sandbox handling in `createSandbox` | For `--resume`, the recorded session is authoritative and conflicting current name/provider/model/image/tool-disclosure hints are rejected. A new re-onboard run takes current flags, environment, and prompts as intent while registry/gateway state provides drift evidence. The machine resolves a complete secret-free create intent, including policy, messaging/provider, GPU, resource, disabled-channel, and agent inputs, before repair/removal or live recreation. | Ordinary live recreation conditionally backs up before provider cleanup, **delete**, and image removal. The recreate journal preserves the source registry row after deletion. Replacement registration commits the new row after readiness and validation. A selected pre-upgrade backup suppresses a new one; an explicit override permits recreation without backup. Resume registry removal and `repair-and-recreate` occur only after complete intent validation. Temporary policy/build artifacts remain materialization effects after the delete boundary. | Resume continues the recorded session/machine snapshot; non-resume re-onboard writes a new session first. OpenClaw records completed sandbox name, web search, messaging, and resource choices with explicit progress markers, including explicit `null` choices, while the complete create intent stays process-local and is not persisted or emitted. Raw credential values remain outside the session. A missing process value can be rebound only when the same OpenClaw session recorded successfully registering that provider and its live provider name, provider type, and credential key still match; otherwise interactive resume requests it again and non-interactive resume exits with environment-variable guidance. Credentials are checked before mutation and again immediately before materialization. | A failed replacement keeps the source registry row. Restore failures warn and can still publish the replacement; managed-DCode live-selection failure leaves a running, unregistered sandbox with manual-delete guidance. Checkpoint replay reuses an exact live sandbox after an interrupted create and backfills missing create/register receipts. Cancel rollback is not armed and there is no rebuild-style receipt rollback. Coverage: transition traces, create-intent characterization, checkpoint replay and resume guards, and sandbox-handler crash recovery. Gaps: early backup asymmetry and no rebuild-style cross-effect rollback. | | **Rebuild or installer-driven upgrade** — `rebuildSandbox` in `rebuild-pipeline.ts`; `upgradeSandboxes` | Registry state is authoritative. A matching session may fill guarded legacy gaps only when its selection agrees; an unrelated/global session is never used. Ambient provider/model selection is quarantined by `isolateAmbientRecreateEnv`, apart from narrowly scoped legacy recovery. Legacy and custom-image rebuilds retain and fingerprint a prepared build context. Managed-image rebuilds instead stage an immutable image and startup-profile handoff, skip Dockerfile image preflight, and revalidate provider-bound workload authority before each deletion boundary. | Consent persistence, target-gateway selection/recovery, and target-preflight registry updates can precede disposable image build/probes. Backup is the first durable recovery checkpoint when available. MCP detach/scrub and NIM stop are destructive in-place effects before the **sandbox delete** boundary. Legacy and custom-image paths recheck prepared context and mutation-edge conditions before delete. Managed-image paths revalidate the provider-bound handoff before delete. | Durable checkpoints are the backup/recovery manifest when one exists and the rewritten recreate session; stale recovery can reach deletion without a manifest, making that session its first new durable checkpoint. Rollback receipts/snapshots are process-local. Credential metadata comes from the target or guarded fallback; raw credentials/providers are checked against current process/gateway state, while prepared installer recovery may reconstruct a missing gateway provider from a validated host credential. | In-process rollback best-effort restores registry/MCP retry metadata, but process death after non-MCP delete can still lose it. The inner onboarding consumes the managed-workload handoff or selects the legacy resource profile after deletion. Covered by rebuild, managed-workload authority, image-preflight, DCode, and messaging tests. Gaps: health-before-delete and atomic swap. Closed issue #5801 records the original gap; #6835 fixed only the printed recovery path. | diff --git a/src/lib/onboard/sandbox-create/orchestration.test.ts b/src/lib/onboard/sandbox-create/orchestration.test.ts index 8ca432a08b4..47202416d6f 100644 --- a/src/lib/onboard/sandbox-create/orchestration.test.ts +++ b/src/lib/onboard/sandbox-create/orchestration.test.ts @@ -544,11 +544,12 @@ describe("deferred provider effect authority", () => { expect(error).toBeInstanceOf(Error); expect((error as Error).message).toContain( - "Do not delete it by mutable sandbox name. Ask an OpenShell administrator to remove the retained sandbox through an identity-bound procedure.", + "Do not delete it by mutable sandbox name. Run 'nemoclaw alpha destroy'.", ); expect((error as Error).message).toContain( - "After OpenShell confirms the retained sandbox is absent, run 'nemoclaw alpha destroy --yes' to reconcile its verified Docker containers and recovery record.", + "can reconcile verified residual resources and the recovery record only after OpenShell confirms absence", ); + expect((error as Error).message).not.toContain("administrator"); expect(runOpenshell).not.toHaveBeenCalledWith( expect.arrayContaining(["sandbox", "provider", "attach"]), @@ -851,7 +852,7 @@ describe("sandbox create identity checks", () => { expect(error).toBeInstanceOf(AggregateError); expect((error as AggregateError).message).toMatch( new RegExp( - `Create-attempt label: ai\\.nvidia\\.nemoclaw\\.create-attempt=${createAttemptNonce}.*left sandbox 'alpha' in place.*identity fingerprint: ${exactIdentity}.*did not run OpenShell's mutable-name deletion command.*Do not delete the sandbox by mutable sandbox name.*OpenShell administrator.*identity-bound recovery or removal procedure`, + `Create-attempt label: ai\\.nvidia\\.nemoclaw\\.create-attempt=${createAttemptNonce}.*left sandbox 'alpha' in place.*identity fingerprint: ${exactIdentity}.*did not run OpenShell's mutable-name deletion command.*Do not delete the sandbox by mutable sandbox name.*Inspection is diagnostic only and does not authorize deletion`, "u", ), ); @@ -860,7 +861,7 @@ describe("sandbox create identity checks", () => { expect.objectContaining({ message: expect.stringMatching( new RegExp( - `Create-attempt label: ai\\.nvidia\\.nemoclaw\\.create-attempt=${createAttemptNonce}.*left sandbox 'alpha' in place.*identity fingerprint: ${exactIdentity}.*did not run OpenShell's mutable-name deletion command.*Do not delete the sandbox by mutable sandbox name.*OpenShell administrator.*identity-bound recovery or removal procedure`, + `Create-attempt label: ai\\.nvidia\\.nemoclaw\\.create-attempt=${createAttemptNonce}.*left sandbox 'alpha' in place.*identity fingerprint: ${exactIdentity}.*did not run OpenShell's mutable-name deletion command.*Do not delete the sandbox by mutable sandbox name.*Inspection is diagnostic only and does not authorize deletion`, "u", ), ), @@ -1388,7 +1389,7 @@ describe("sandbox create identity checks", () => { expect(error).toBeInstanceOf(AggregateError); expect((error as AggregateError).message).toMatch( - /left sandbox 'alpha' in place.*did not return a durable sandbox identity fingerprint.*Do not delete the sandbox by mutable sandbox name.*identity-bound recovery or removal procedure/u, + /left sandbox 'alpha' in place.*did not return a durable sandbox identity fingerprint.*Do not delete the sandbox by mutable sandbox name.*Inspection is diagnostic only and does not authorize deletion/u, ); expect((error as AggregateError).errors).toEqual( expect.arrayContaining([ diff --git a/src/lib/onboard/sandbox-create/orchestration.ts b/src/lib/onboard/sandbox-create/orchestration.ts index 98d7b84c43a..22cd04e9c62 100644 --- a/src/lib/onboard/sandbox-create/orchestration.ts +++ b/src/lib/onboard/sandbox-create/orchestration.ts @@ -373,7 +373,7 @@ export function installPostCreateRecoveryRetryOwner( } catch (error) { if (propagateFailure) throw error; log( - " NemoClaw still could not save the retained sandbox recovery record; the recovery-only session remains blocked for administrator recovery.", + " NemoClaw still could not save the retained sandbox recovery record. Preserve the registry entry and terminal output; do not delete the sandbox by mutable name.", ); } }; @@ -427,7 +427,7 @@ export function persistPostCreateRecovery(input: { `Create-attempt label: ${NEMOCLAW_CREATE_ATTEMPT_LABEL}=${input.recoveryContext.createAttemptNonce}. ` + `Sandbox '${input.sandboxName}' was retained after ${input.stage} failed. ` + `Gateway '${input.gatewayName}'. Lifecycle generation '${input.lifecycleGeneration}'. ` + - "Do not delete the sandbox by mutable name; preserve it for identity-bound administrator recovery."; + `Do not delete the sandbox by mutable name. Run '${cliName()} ${input.sandboxName} destroy'; it can clear retained recovery only after OpenShell confirms absence.`; console.error(` ${message}`); let persisted = false; try { @@ -849,8 +849,8 @@ export async function runSandboxCreateWithIdentityVerification< createAttemptGuidance + `NemoClaw left sandbox '${input.sandboxName}' in place after post-create verification or finalization failed. ` + `${identityGuidance} NemoClaw did not run OpenShell's mutable-name deletion command because the name may now identify a replacement sandbox. ` + - `Do not delete the sandbox by mutable sandbox name. Run '${cliName()} ${input.sandboxName} destroy' to use the retained identity. ` + - "If destroy cannot prove that identity, stop. Ask the OpenShell administrator to inspect the surviving sandbox and use an identity-bound recovery or removal procedure."; + `Do not delete the sandbox by mutable sandbox name. Run '${cliName()} ${input.sandboxName} destroy'. ` + + "If OpenShell reports the sandbox present or cannot determine presence, destroy removes nothing and preserves the recovery record. Inspection is diagnostic only and does not authorize deletion."; const compensationErrors: unknown[] = []; if (input.persistRetainedSandboxRecovery) { try { @@ -1056,8 +1056,8 @@ export function createProviderEffectBoundary(input: { throw new Error( `OpenShell cannot attach providers to the immutable identity of sandbox '${input.sandboxName}'. ` + `NemoClaw retained the incomplete sandbox on gateway '${input.gatewayName}'. ` + - `Do not delete it by mutable sandbox name. Ask an OpenShell administrator to remove the retained sandbox through an identity-bound procedure. ` + - `After OpenShell confirms the retained sandbox is absent, run '${cliName()} ${input.sandboxName} destroy --yes' to reconcile its verified Docker containers and recovery record.`, + `Do not delete it by mutable sandbox name. Run '${cliName()} ${input.sandboxName} destroy'. ` + + `The command preserves a present or unknown sandbox and can reconcile verified residual resources and the recovery record only after OpenShell confirms absence.`, ); }, }; diff --git a/src/lib/onboard/sandbox-gpu-create-flow.test.ts b/src/lib/onboard/sandbox-gpu-create-flow.test.ts index 9b46b929cfb..0519c45d5d6 100644 --- a/src/lib/onboard/sandbox-gpu-create-flow.test.ts +++ b/src/lib/onboard/sandbox-gpu-create-flow.test.ts @@ -970,7 +970,7 @@ describe("runSandboxGpuCreateFlow native failure and readiness", () => { expect(mocks.streamSandboxCreate).toHaveBeenCalledOnce(); expect(mocks.verifyGpuSandboxAccessAfterReady).not.toHaveBeenCalled(); expect(deps.runOpenshell).not.toHaveBeenCalled(); - expect(errorOutput()).toContain("Verify the sandbox identity before manual cleanup"); + expect(errorOutput()).toContain("Recovery remains blocked while this sandbox exists"); expect(mocks.streamSandboxCreate).toHaveBeenCalledOnce(); }); @@ -1384,7 +1384,7 @@ describe("runSandboxGpuCreateFlow cleanup and provenance", () => { const output = vi.mocked(console.error).mock.calls.flat().join("\n"); expect(deps.runOpenshell).not.toHaveBeenCalled(); expect(output).toContain("left sandbox 'alpha' in place"); - expect(output).toContain("Verify the sandbox identity before manual cleanup"); + expect(output).toContain("Recovery remains blocked while this sandbox exists"); expect(output).not.toContain("openshell sandbox delete"); expect(output).not.toContain("Retry: nemoclaw onboard"); }); @@ -1401,7 +1401,7 @@ describe("runSandboxGpuCreateFlow cleanup and provenance", () => { const output = vi.mocked(console.error).mock.calls.flat().join("\n"); expect(deps.runOpenshell).not.toHaveBeenCalled(); expect(output).toContain("left sandbox 'alpha' in place"); - expect(output).toContain("Verify the sandbox identity before manual cleanup"); + expect(output).toContain("Recovery remains blocked while this sandbox exists"); expect(output).not.toContain("Retry: nemoclaw onboard"); }); diff --git a/src/lib/onboard/sandbox-gpu-create-flow.ts b/src/lib/onboard/sandbox-gpu-create-flow.ts index 72a8b7719df..e68c688c31c 100644 --- a/src/lib/onboard/sandbox-gpu-create-flow.ts +++ b/src/lib/onboard/sandbox-gpu-create-flow.ts @@ -559,7 +559,7 @@ export async function runSandboxGpuCreateFlow( ? ` Managed bootstrap retained exact owner-cleanup authority for sandbox '${input.sandboxName}'. Do not delete a runtime by mutable sandbox name; preserve it for identity-bound recovery.` : hermesPortableLifecycle ? ` Hermes portable sandbox '${input.sandboxName}' did not complete receipt-owned creation. Preserve its lifecycle receipt and resume onboarding after correcting the reported failure.` - : ` Sandbox '${input.sandboxName}' may still exist. Verify its durable identity before manual cleanup; do not act by mutable name alone.`, + : ` Sandbox '${input.sandboxName}' may still exist. Recovery remains blocked while it exists; do not delete it by mutable name. Run 'nemoclaw ${input.sandboxName} destroy' to check for authoritative absence.`, ); if (input.requirePolicylessCreate) { const persistRetainedSandboxRecovery = input.persistRetainedSandboxRecovery; @@ -574,13 +574,13 @@ export async function runSandboxGpuCreateFlow( } else { const identityGuidance = evidence.liveIdentityFingerprint ? "Use that fingerprint only to compare the surviving sandbox with this create attempt." - : "OpenShell did not return one exact durable sandbox identity for this create attempt. Recovery is blocked until an OpenShell administrator resolves the create-attempt label to one sandbox."; + : "OpenShell did not return one exact durable sandbox identity for this create attempt."; const message = `Create-attempt label: ${NEMOCLAW_CREATE_ATTEMPT_LABEL}=${evidence.createAttemptNonce}. ` + `${evidence.liveIdentityFingerprint ? `Durable sandbox identity fingerprint: ${evidence.liveIdentityFingerprint}. ` : ""}` + `APF sandbox '${input.sandboxName}' may have been retained after native GPU fallback stopped. ` + `Gateway '${input.gatewayName}'. ${identityGuidance} ` + - "Do not delete a sandbox by mutable name; use an identity-bound administrator recovery procedure."; + `Do not delete the sandbox by mutable name. Run 'nemoclaw ${input.sandboxName} destroy'; it can clear retained recovery only after OpenShell confirms absence.`; let persisted = false; try { persisted = evidence.liveIdentityFingerprint @@ -596,7 +596,7 @@ export async function runSandboxGpuCreateFlow( console.error(` ${message}`); if (!persisted) { console.error( - " APF recovery is blocked because NemoClaw could not save this create-attempt evidence. Preserve the terminal output for an OpenShell administrator.", + " APF recovery is blocked because NemoClaw could not save this create-attempt evidence. Preserve the registry entry and terminal output; do not delete the sandbox by mutable name.", ); } } diff --git a/src/lib/onboard/sandbox-gpu-create-identity-gate.test.ts b/src/lib/onboard/sandbox-gpu-create-identity-gate.test.ts index b6e97d0e618..7371c66555a 100644 --- a/src/lib/onboard/sandbox-gpu-create-identity-gate.test.ts +++ b/src/lib/onboard/sandbox-gpu-create-identity-gate.test.ts @@ -515,9 +515,11 @@ describe("created sandbox identity gate", () => { expect(output).toContain(`Durable sandbox identity fingerprint: ${fingerprint}`); expect(output).toContain("Run 'nemoclaw alpha destroy'"); expect(output).toContain("the command removes nothing and preserves the recovery record"); - expect(output).toContain("Give the create-attempt label to an OpenShell administrator"); - expect(output).toContain("After OpenShell confirms removal"); + expect(output).toContain("Inspection is diagnostic only"); + expect(output).toContain("Recovery remains blocked while the sandbox is present"); + expect(output).toContain("after the owning gateway reports absence"); expect(output).toContain("run 'nemoclaw alpha destroy --yes'"); + expect(output).not.toContain("administrator"); expect(output).not.toContain("alpha-sandbox-id"); expect(output).not.toContain("Recovery:"); expect(output).not.toContain("Or: nemoclaw onboard"); @@ -1171,7 +1173,7 @@ describe("created sandbox identity gate", () => { expect(input.persistRetainedSandboxRecovery).toHaveBeenCalledExactlyOnceWith( expect.stringMatching( new RegExp( - `^Create-attempt label: ${NEMOCLAW_CREATE_ATTEMPT_LABEL}=${nonce}\\..*Recovery is blocked until an OpenShell administrator resolves the create-attempt label`, + `^Create-attempt label: ${NEMOCLAW_CREATE_ATTEMPT_LABEL}=${nonce}\\..*Do not delete the sandbox by mutable name.*can clear retained recovery only after OpenShell confirms absence`, "u", ), ), @@ -1180,7 +1182,8 @@ describe("created sandbox identity gate", () => { ); const output = vi.mocked(console.error).mock.calls.flat().join("\n"); expect(output).toContain(`${NEMOCLAW_CREATE_ATTEMPT_LABEL}=${nonce}`); - expect(output).toContain("Recovery is blocked"); + expect(output).toContain("can clear retained recovery only after OpenShell confirms absence"); + expect(output).not.toContain("administrator"); expect(deps.runOpenshell).not.toHaveBeenCalledWith( ["sandbox", "delete", "alpha"], expect.anything(), diff --git a/src/lib/onboard/sandbox-gpu-create-run-attempt.ts b/src/lib/onboard/sandbox-gpu-create-run-attempt.ts index 3e4c44b21c0..49adf64aeef 100644 --- a/src/lib/onboard/sandbox-gpu-create-run-attempt.ts +++ b/src/lib/onboard/sandbox-gpu-create-run-attempt.ts @@ -546,7 +546,7 @@ export function createSandboxGpuCreateAttemptRunner( const persistenceFailureMessage = "NemoClaw could not save the retained sandbox recovery record for this create attempt."; console.error( - ` ${persistenceFailureMessage} Preserve the terminal output for an OpenShell administrator.`, + ` ${persistenceFailureMessage} Preserve the registry entry and terminal output; do not delete the sandbox by mutable name.`, ); throw new Error(persistenceFailureMessage, { cause: persistenceCause }); } @@ -774,7 +774,7 @@ export function createSandboxGpuCreateAttemptRunner( throw new Error( createAttemptNonce ? `OpenShell create client did not exit after Ready for sandbox '${input.sandboxName}'. NemoClaw retained the sandbox and blocked post-create effects. Follow the retained recovery action above.` - : `OpenShell create client did not exit after Ready for sandbox '${input.sandboxName}'. NemoClaw blocked post-create effects. No create-attempt identity was available for retained recovery. Preserve the sandbox for identity-bound OpenShell administrator recovery; do not delete it by mutable name.`, + : `OpenShell create client did not exit after Ready for sandbox '${input.sandboxName}'. NemoClaw blocked post-create effects. No create-attempt identity was available for retained recovery. Preserve the registry entry and terminal output; do not delete the sandbox by mutable name.`, ); } return createResult; @@ -1158,7 +1158,9 @@ export function createSandboxGpuCreateAttemptRunner( console.error( ` NemoClaw left sandbox '${input.sandboxName}' in place because OpenShell can delete it only by mutable name.`, ); - console.error(" Verify the sandbox identity before manual cleanup."); + console.error( + ` Recovery remains blocked while this sandbox exists. Do not delete it by mutable name; run 'nemoclaw ${input.sandboxName} destroy' to check for authoritative absence.`, + ); } failAfterCreatedSandboxVerification( `Sandbox '${input.sandboxName}' did not become ready after verified creation.`, diff --git a/src/lib/state/onboard-session.ts b/src/lib/state/onboard-session.ts index fed13ed38ca..aa7503e948b 100644 --- a/src/lib/state/onboard-session.ts +++ b/src/lib/state/onboard-session.ts @@ -2148,7 +2148,7 @@ export function markCancellationRecovery( session.failure = { step: session.lastStepStarted, message: - "Onboarding was cancelled after sandbox creation; administrator recovery is required.", + "Onboarding was cancelled after sandbox creation; retained recovery blocks this sandbox name until destroy confirms absence and completes cleanup.", recordedAt, interrupted: true, }; diff --git a/test/helpers/destroy-flow-test-harness.ts b/test/helpers/destroy-flow-test-harness.ts index a91197149b2..6103c892850 100644 --- a/test/helpers/destroy-flow-test-harness.ts +++ b/test/helpers/destroy-flow-test-harness.ts @@ -129,6 +129,7 @@ type DestroyHarnessOptions = { replaceSessionAfterRegistryRemoval?: boolean; removeSandboxResult?: boolean; restoreMcpError?: string; + sandboxListResult?: { status: number | null; stdout?: string; stderr?: string }; sandboxPresent?: boolean; sessionRouterPid?: number; stopInferenceError?: string; @@ -477,11 +478,13 @@ export function createDestroyHarness(options: DestroyHarnessOptions = {}): Destr }; case "sandbox:list": gatewayPinsAtSandboxList.push(process.env.OPENSHELL_GATEWAY); - return { - status: 0, - stdout: sandboxListJson(sandboxPresent ? ["alpha"] : []), - stderr: "", - }; + return ( + options.sandboxListResult ?? { + status: 0, + stdout: sandboxListJson(sandboxPresent ? ["alpha"] : []), + stderr: "", + } + ); case "sandbox:delete": events.push("delete"); sandboxPresent = false; diff --git a/test/onboarding/onboard-rollback.test.ts b/test/onboarding/onboard-rollback.test.ts index 956315c86e5..2641e851f60 100644 --- a/test/onboarding/onboard-rollback.test.ts +++ b/test/onboarding/onboard-rollback.test.ts @@ -35,7 +35,7 @@ if (!isOnboardRollbackInternals(onboardInternals)) { const { buildOrphanedSandboxRollbackMessage } = onboardInternals; describe("ghost-sandbox rollback message (#2174)", () => { - it("reports the surviving sandbox and manual identity-checked cleanup", () => { + it("reports the surviving sandbox and fail-closed recovery", () => { const lines = buildOrphanedSandboxRollbackMessage( "alpha", new Error("All dashboard ports in range 18789-18798 are occupied"), @@ -47,8 +47,13 @@ describe("ghost-sandbox rollback message (#2174)", () => { expect(lines).toContain( " NemoClaw left the sandbox running because OpenShell deletion targets a mutable name.", ); - expect(lines).toContain(" Verify the sandbox identity, then clean up manually:"); - expect(lines).toContain(' openshell sandbox delete -g "nemoclaw-18080" "alpha"'); + expect(lines).toContain( + ' Recovery remains blocked while gateway "nemoclaw-18080" reports this sandbox present.', + ); + expect(lines).toContain( + " Do not delete it by mutable name; run 'nemoclaw alpha destroy' to check for authoritative absence.", + ); + expect(lines.join("\n")).not.toContain("openshell sandbox delete"); }); it("renders non-Error throwables via String coercion", () => { @@ -56,15 +61,19 @@ describe("ghost-sandbox rollback message (#2174)", () => { expect(lines).toContain(" raw string failure"); }); - it("escapes the sandbox name into the manual-cleanup command exactly", () => { + it("escapes the gateway and preserves the sandbox name in recovery guidance", () => { const lines = buildOrphanedSandboxRollbackMessage( 'weird-name_42"', new Error("oops"), 'gateway"name', ); expect(lines).toContain( - ' openshell sandbox delete -g "gateway\\\"name" "weird-name_42\\\""', + ' Recovery remains blocked while gateway "gateway\\\"name" reports this sandbox present.', + ); + expect(lines).toContain( + " Do not delete it by mutable name; run 'nemoclaw weird-name_42\" destroy' to check for authoritative absence.", ); + expect(lines.join("\n")).not.toContain("openshell sandbox delete"); }); it("does not suggest deletion when the owning gateway is unknown", () => {