Skip to content
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion docs/reference/commands.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -394,7 +394,12 @@ Use the result from `destroy` to choose the next action:
- If OpenShell confirms the sandbox is absent, `destroy` verifies one retained recovery record and its immutable runtime identity. For Docker-backed sandboxes, it also verifies the immutable Docker sandbox identity and removes only qualified residual containers. It clears the matching recovery record only after verified cleanup.
- If the recovery record is ambiguous or immutable runtime-identity verification fails, cleanup stops and the record remains. For Docker-backed sandboxes, a foreign container, changed Docker identity, or failed Docker probe has the same fail-closed result.

If OpenShell did not return a durable identity fingerprint, `destroy` cannot complete recovery. A create-attempt label can help an OpenShell administrator identify and remove the exact sandbox, but it does not let NemoClaw retire the record without immutable identity authority. If neither a fingerprint nor a label is available, preserve the terminal output and ask the administrator to identify the exact sandbox from gateway or controller evidence. If NemoClaw reports that it could not save recovery evidence, preserve the terminal output and follow the same escalation.
If OpenShell did not return a durable identity fingerprint, `destroy` cannot complete recovery.
A create-attempt label can help an OpenShell administrator identify and remove the exact sandbox, but it does not let NemoClaw retire the record without immutable identity authority.
If neither a fingerprint nor a label is available, preserve the terminal output and ask the administrator to identify the exact sandbox from gateway or controller evidence.
If NemoClaw reports that it could not save recovery evidence, rerun `$$nemoclaw <sandbox-name> destroy --yes`.
NemoClaw reconstructs the recovery record only when the registry preserves a matching `verified-create` checkpoint and lifecycle authority.
If reconstruction fails, the authority conflicts, or immutable identity evidence is missing, preserve the terminal output and follow the same escalation.

This fail-closed record keeps only the affected sandbox name unavailable. It is not retired from mutable-name absence alone, and this command does not accept administrator-supplied identity authority.
To onboard another sandbox while the record remains unresolved, supply a different explicit name:
Expand Down
44 changes: 44 additions & 0 deletions src/lib/actions/sandbox/destroy-retained-recovery-flow.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,50 @@ describe("destroySandbox retained recovery flow", () => {
resetDestroyModuleCache();
});

it(
"reconstructs a missing recovery record from the verified-create registry checkpoint (#11096)",
{ timeout: 30_000 },
async () => {
const recovery = retainedRecoveryRecord();
const pendingCreateIdentity = {
schemaVersion: 1 as const,
state: "verified-create" as const,
gatewayName: recovery.gatewayName,
gatewayPort: recovery.gatewayPort,
sandboxName: recovery.sandboxName,
lifecycleGeneration: recovery.lifecycleGeneration!,
sandboxIdentityFingerprint: recovery.sandboxIdentityFingerprint!,
createAttemptNonce: recovery.createAttemptNonce,
route: "native" as const,
};
const harness = createDestroyHarness({
sandboxPresent: false,
dockerRunResult: { status: 0, stdout: "" },
registryEntryOverrides: {
pendingRouteReservation: true,
reservationSessionId: "failed-create-session",
lifecycleGeneration: recovery.lifecycleGeneration!,
lifecycleLiveIdentityFingerprint: recovery.sandboxIdentityFingerprint!,
pendingCreateIdentity,
},
reconstructRetainedRecoveryRecord: recovery,
});

await expect(harness.destroySandbox("alpha", { yes: true })).resolves.toBeUndefined();

expect(harness.reconstructRetainedSandboxRecoverySpy).toHaveBeenCalledWith(
expect.objectContaining({ pendingCreateIdentity }),
);
expect(harness.runOpenshellSpy).not.toHaveBeenCalledWith(
["sandbox", "delete", "alpha"],
expect.anything(),
);
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
expect(harness.resolveRetainedSandboxRecoverySpy).toHaveBeenCalledWith(recovery);
expect(harness.removeSandboxSpy).toHaveBeenCalledWith("alpha");
expect(exitSpy).not.toHaveBeenCalled();
},
);

it(
"removes every container after OpenShell confirms the retained sandbox absent (#10547)",
{ timeout: 30_000 },
Expand Down
41 changes: 24 additions & 17 deletions src/lib/actions/sandbox/destroy.ts
Original file line number Diff line number Diff line change
Expand Up @@ -620,6 +620,9 @@ async function destroySandboxUnlocked(
const registeredSandbox = registry.getSandbox(sandboxName);
const operationRuntimeSelection = resolveSandboxDestroyRuntimeSelection(registeredSandbox);
if (!(await confirmSandboxDestroy(sandboxName, normalized, operationRuntimeSelection))) return;
if (registeredSandbox) {
onboardSession.reconstructRetainedSandboxRecoveryFromPendingCreate(registeredSandbox);
}
const destroySession = onboardSession.loadSession();
const retainedRecoveryRecords = onboardSession.listRetainedSandboxRecoveryRecords();
const retainedRecoveryAuthority = selectRetainedSandboxRecoveryAuthority(
Expand Down Expand Up @@ -665,9 +668,7 @@ async function destroySandboxUnlocked(
: normalizeRuntimeProviderIdentity(null),
redact: redactDestroyError,
sandbox: registeredSandbox,
...(retainedSandboxIdentityFingerprint
? { retainedSandboxIdentityFingerprint }
: {}),
...(retainedSandboxIdentityFingerprint ? { retainedSandboxIdentityFingerprint } : {}),
});
};
const initialIdentity = portableContainerAuthority ? null : inspectContainerIdentity();
Expand Down Expand Up @@ -877,14 +878,13 @@ async function destroySandboxUnlocked(
runtimeSelection: destroyRuntimeSelection,
} = destructiveResult;

if (
destroyRuntimeSelection &&
cleanupGatewayName !== destroyRuntimeSelection.gatewayName
) {
if (destroyRuntimeSelection && cleanupGatewayName !== destroyRuntimeSelection.gatewayName) {
console.error(
` Sandbox '${sandboxName}' was deleted, but its cleanup target changed from '${destroyRuntimeSelection.gatewayName}' to '${cleanupGatewayName}'.`,
);
console.error(" Local ownership state was preserved. Restore the recorded gateway binding and retry destroy.");
console.error(
" Local ownership state was preserved. Restore the recorded gateway binding and retry destroy.",
);
preparedManagedLlamaCppCleanup?.abort();
requestSandboxDestroyExit(1);
}
Expand Down Expand Up @@ -966,11 +966,15 @@ async function destroySandboxUnlocked(
registeredSandboxCount: registry.listSandboxes().sandboxes.length,
sandboxStillRegistered: !!registry.getSandbox(sandboxName),
});
cleanupSandboxServices(sandboxName, {
stopHostServices: shouldStopHostServices,
}, {
runOpenshell: cleanupRunOpenshell,
});
cleanupSandboxServices(
sandboxName,
{
stopHostServices: shouldStopHostServices,
},
{
runOpenshell: cleanupRunOpenshell,
},
);
});
if (deleteSucceededOrAlreadyGone && commonLlamaCppAuthorityRetired === true) {
preparedManagedLlamaCppCleanup?.abort();
Expand Down Expand Up @@ -1143,10 +1147,13 @@ async function destroySandboxUnlocked(
}
}
if (
shouldCleanupGatewayAfterConfirmedFinalDestroy({
deleteSucceededOrAlreadyGone,
removedRegistryEntry: removed,
}, cleanupCaptureOpenshell ? { captureOpenshell: cleanupCaptureOpenshell } : {})
shouldCleanupGatewayAfterConfirmedFinalDestroy(
{
deleteSucceededOrAlreadyGone,
removedRegistryEntry: removed,
},
cleanupCaptureOpenshell ? { captureOpenshell: cleanupCaptureOpenshell } : {},
)
) {
const shouldCleanupGateway = await resolveCleanupGatewayDecision(normalized);
if (shouldCleanupGateway) {
Expand Down
66 changes: 65 additions & 1 deletion src/lib/actions/sandbox/rebuild-preflight-confirmation.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ import {
} from "./rebuild-preflight-confirmation";
import {
acquireRebuildOnboardLock,
blockRebuildOnRetainedSandboxRecovery,
isSingleAgentRebuildSupported,
} from "./rebuild-preflight-guards";

Expand Down Expand Up @@ -163,12 +164,75 @@ describe("createRebuildCommandContext bail behaviour (#6376)", () => {
// What reached stderr is the redacted form, with the two-space prefix ...
expect(errorSpy).toHaveBeenCalledWith(` ${redacted}`);
// ... and the raw secret never surfaced.
expect(errorSpy.mock.calls.every((call) => !String(call[0]).includes("SUPERSECRETTOKEN123"))).toBe(true);
expect(
errorSpy.mock.calls.every((call) => !String(call[0]).includes("SUPERSECRETTOKEN123")),
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
).toBe(true);
expect(exitSpy).toHaveBeenCalledWith(1);
});
});

describe("rebuild preflight guards", () => {
it("reconstructs registry-only recovery before rebuild can probe live state (#11096)", () => {
const fingerprint = "b".repeat(64);
const createAttemptNonce = "c".repeat(62);
const lifecycleGeneration = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa";
const recoveryRecord: onboardSession.RetainedSandboxRecoveryRecord = {
schemaVersion: 1,
sandboxName: "alpha",
recordId: "a".repeat(64),
sandboxIdentityFingerprint: fingerprint,
identityWasUnavailable: false,
gatewayName: "nemoclaw",
gatewayPort: 8080,
lifecycleGeneration,
createAttemptNonce,
resources: {
sharedInferenceProviders: [],
sandboxScopedProviders: [],
credentialEnvironmentVariables: [],
},
reason: "retained_after_sandbox_creation_failure",
recordedAt: "2026-09-07T00:00:00.000Z",
};
const reconstruct = vi
.spyOn(onboardSession, "reconstructRetainedSandboxRecoveryFromPendingCreate")
.mockReturnValue(recoveryRecord);
const listRecovery = vi
.spyOn(onboardSession, "listRetainedSandboxRecoveryRecords")
.mockReturnValue([recoveryRecord]);
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
const error = vi.spyOn(console, "error").mockImplementation(() => undefined);
const bail = vi.fn() as unknown as (message: string, code?: number) => never;
const sandbox = {
name: "alpha",
pendingRouteReservation: true,
reservationSessionId: "failed-create-session",
gatewayName: "nemoclaw",
gatewayPort: 8080,
lifecycleGeneration,
lifecycleLiveIdentityFingerprint: fingerprint,
pendingCreateIdentity: {
schemaVersion: 1,
state: "verified-create",
gatewayName: "nemoclaw",
gatewayPort: 8080,
sandboxName: "alpha",
lifecycleGeneration,
sandboxIdentityFingerprint: fingerprint,
createAttemptNonce,
route: "native",
},
} satisfies import("./rebuild-flow-helpers").RebuildSandboxEntry;

expect(blockRebuildOnRetainedSandboxRecovery(sandbox, bail)).toBe(true);

expect(reconstruct).toHaveBeenCalledExactlyOnceWith(sandbox);
expect(reconstruct.mock.invocationCallOrder[0]).toBeLessThan(
listRecovery.mock.invocationCallOrder[0]!,
);
expect(bail).toHaveBeenCalledWith("Retained sandbox recovery blocks rebuild for 'alpha'.", 1);
expect(error.mock.calls.flat().join("\n")).toContain("nemoclaw alpha destroy --yes");
});

it("stops after a failed onboard-lock acquisition without releasing another run's lock (#7794)", () => {
vi.spyOn(onboardSession, "acquireOnboardLock").mockReturnValue({
acquired: false,
Expand Down
4 changes: 3 additions & 1 deletion src/lib/actions/sandbox/rebuild-preflight-guards.ts
Original file line number Diff line number Diff line change
Expand Up @@ -325,9 +325,11 @@ export function getRebuildSandboxEntryOrBail(

/** Block rebuild before any live-state probe or cleanup can bypass retained recovery. */
export function blockRebuildOnRetainedSandboxRecovery(
sandboxName: string,
sandbox: RebuildSandboxEntry,
bail: RebuildBail,
): boolean {
const sandboxName = sandbox.name;
onboardSession.reconstructRetainedSandboxRecoveryFromPendingCreate(sandbox);
const retainedRecovery = onboardSession
.listRetainedSandboxRecoveryRecords()
.find((record) => record.sandboxName === sandboxName);
Expand Down
2 changes: 1 addition & 1 deletion src/lib/actions/sandbox/rebuild-preflight-phase.ts
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ export async function runRebuildPreflightPhase(
} = createRebuildCommandContext(options, opts);
const sandboxEntry = getRebuildSandboxEntryOrBail(sandboxName, bail);
if (!sandboxEntry) return null;
if (blockRebuildOnRetainedSandboxRecovery(sandboxName, bail)) return null;
if (blockRebuildOnRetainedSandboxRecovery(sandboxEntry, bail)) return null;
// #6376: refuse a stuck MCP destroy transaction up front — before backup,
// image prep, or the old-sandbox delete. The only MCP marker check used to
// live inside the destroy phase, which runs AFTER the backup phase, so a
Expand Down
4 changes: 2 additions & 2 deletions src/lib/onboard.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2653,8 +2653,8 @@ async function runOnboard(opts: OnboardOptions = {}): Promise<void> {
setOnboardBrandingAgent(opts.agent || process.env.NEMOCLAW_AGENT || null);
AUTO_YES = opts.autoYes === true || process.env.NEMOCLAW_YES === "1";
const resolveEntryOptions = () =>
onboardEntryOptions.resolveDefaultRunEntryOptionsFromState(opts, validateName, onboardSession);
const initialEntryOptions = resolveEntryOptions();
onboardEntryOptions.resolveEntryOptions(opts, validateName, onboardSession, registry);
const initialEntryOptions = onboardEntryOptions.readOptions(opts, validateName, onboardSession);
NON_INTERACTIVE = initialEntryOptions.nonInteractive;
RECREATE_SANDBOX = opts.recreateSandbox || process.env.NEMOCLAW_RECREATE_SANDBOX === "1";
_preflightDashboardPort =
Expand Down
Loading
Loading