Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
b8744e0
fix(shields): recover completed auto-restore locks
prekshivyas Aug 29, 2026
294dfb3
merge: refresh shields containment fix from main
prekshivyas Aug 29, 2026
35db80f
fix(shields): structure containment recovery evidence
prekshivyas Aug 29, 2026
308ad85
test(shields): settle reproduction child failures
prekshivyas Aug 29, 2026
51a777a
fix(shields): recover commands after completed restore
prekshivyas Aug 29, 2026
2258db0
fix(shields): complete auto-restore recovery
prekshivyas Aug 29, 2026
a646df3
fix(shields): reconcile recovery artifacts
prekshivyas Aug 29, 2026
9bab26b
fix(shields): retain durability retry state
prekshivyas Aug 29, 2026
1fdb0aa
merge: update from upstream main
prekshivyas Aug 29, 2026
2f1ecc5
fix(shields): preserve unproven containment
prekshivyas Aug 30, 2026
b8d47ba
merge: refresh upstream main
prekshivyas Aug 30, 2026
560c832
test(shields): cover public status recovery
prekshivyas Aug 30, 2026
1c95ddd
refactor(shields): remove unused recovery fields
prekshivyas Aug 30, 2026
44f1d1c
refactor(shields): share timer marker retirement
prekshivyas Aug 30, 2026
5d84853
test(shields): share completed recovery fixture
prekshivyas Aug 30, 2026
51d4f94
fix(shields): defer completed marker parsing
prekshivyas Aug 30, 2026
4fc8d2a
fix(shields): recover before interactive connect
prekshivyas Aug 30, 2026
5380ced
fix(shields): reject incomplete timer retirement
prekshivyas Aug 30, 2026
ee219f9
Merge remote-tracking branch 'origin/main' into codex/fix-10094-shiel…
prekshivyas Aug 30, 2026
1627a97
test(shields): keep timer recovery coverage focused
prekshivyas Aug 30, 2026
580d61b
fix(shields): preserve completed cleanup retries
senthilr-nv Aug 30, 2026
82700f2
fix(shields): fail closed on recovery inspection
senthilr-nv Aug 30, 2026
123c223
fix(shields): neutralize recovery display controls
senthilr-nv Aug 30, 2026
757948e
fix(shields): harden recovery marker reads
senthilr-nv Aug 30, 2026
d3274af
fix(shields): preserve invalid-name routing
senthilr-nv Aug 30, 2026
620deaa
fix(shields): harden timer proof reads
senthilr-nv Aug 30, 2026
561845d
docs(shields): clarify recovery removal guidance
senthilr-nv Aug 30, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 15 additions & 5 deletions docs/manage-sandboxes/runtime-controls.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -159,11 +159,21 @@ Correct the reported state-directory write failure, then run `$$nemoclaw <name>
When recovery cannot complete, an interactive command returns an error, or the detached timer exits with a failure status.

NemoClaw also records durable containment when an owner exits before it can prove that the owner's descendants stopped, or when ownership becomes ambiguous.
Durable containment, retained gates, or the fail-closed state-directory error keeps new mutations blocked until you complete generation operator recovery.
A `--dry-run` run of a `channels` or `policy` command takes no mutation lock, so you can still preview the change while mutations are blocked.

Stop all NemoClaw processes for the sandbox, then follow the paths, identities, tokens, and removal order in the reported error.
Verify each recorded generation is unchanged, remove only the stale generations first, and remove the containment generation last.
When Shields are already UP after an abandoned auto-restore completes, run `$$nemoclaw <name> shields status`.
NemoClaw removes the timer artifacts and lifecycle gates only when they all identify that completed process generation, the recorded owners are gone, and no Shields transition remains.
Running the command again after successful cleanup is safe because the retired generation no longer supplies recovery authority.
A live or ambiguous owner, a replaced or unrelated generation, conflicting artifacts, or a containment record with no remaining owner generation to verify remains fail-closed.
Those conditions and retained gates keep new mutations blocked until the reported recovery completes.
Do not retry the original sandbox-scoped command while recovery is active, including a `channels` or `policy` command with `--dry-run`.
Run `$$nemoclaw <name> shields status` to complete or inspect recovery, and retry the original command only after `shields status` completes successfully.

If the error reports a live or ambiguous owner, let that process finish or stop it through normal process controls, then rerun `$$nemoclaw <name> shields status` without modifying lifecycle-lock or timer files.
If the error says that completed recovery authority changed, rerun `$$nemoclaw <name> shields status` without modifying those files so NemoClaw can evaluate the current authority.
If the error reports invalid or conflicting recovery artifacts, stop all NemoClaw processes for the sandbox and inspect each reported artifact.
Retain every artifact unless its exact process generation is proven obsolete; remove only an artifact with that proof, then rerun `$$nemoclaw <name> shields status`.
For a replaced or unrelated lifecycle generation, retain the files and follow the reported error.
Apply the owner-generation removal order only when an error explicitly reports committed containment and supplies that order.
Verify every reported identity is unchanged, remove only the exact stale owner generations first and the exact containment generation last, then rerun `$$nemoclaw <name> shields status`.

### Preserve Managed MCP Policy

Expand Down
22 changes: 5 additions & 17 deletions docs/reference/commands.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1852,23 +1852,11 @@ Before `shields down` opens a new window, NemoClaw must revoke any stale auto-re
If marker cleanup fails, the command reports `Cannot revoke stale auto-restore timer authority` and stops before policy capture, state writes, config unlock, replacement-timer startup, or audit writes.
The sandbox retains its existing configuration and policy posture, and the stale timer authority remains.
Resolve the reported timer-marker error on the trusted host, then retry `shields down`.
When a timed shields-down window reaches its deadline, auto-restore closes the per-sandbox lifecycle deadline gate.
The gate blocks new mutations and waits for the recorded live owner to release its lock generation before auto-restore restores lockdown.
NemoClaw does not signal that process because portable process inspection cannot prove that all descendants stopped.
An interactive command can take over an expired timer.
Interactive recovery has separate transition-takeover and restoration phases.
Each phase makes up to 7 attempts and waits 5 seconds between failures, for up to 30 seconds of retry delay per phase.
Detached recovery uses one 7-attempt budget across deadline setup, main-generation publication, and restoration.
The deadline gate remains closed during those attempts.
If restoration cannot commit, NemoClaw attempts to record durable containment.
If that containment commit also fails, NemoClaw retains any lifecycle and deadline gates it already owns.
A state-directory failure that prevented gate publication also prevents normal mutation-lock acquisition.
Correct the reported state-directory write failure, then run `$$nemoclaw <name> shields status` to resume recovery or receive generation recovery guidance.
When recovery cannot complete, an interactive command returns an error, or the detached timer exits with a failure status.
NemoClaw also records durable containment when an owner exits before it can prove that the owner's descendants stopped, or when ownership becomes ambiguous.
Durable containment, retained gates, or the fail-closed state-directory error blocks new mutations until you complete generation operator recovery.
Stop all NemoClaw processes for the sandbox, then follow the paths, identities, tokens, and removal order in the reported error.
Verify each recorded generation is unchanged, remove only the stale generations first, and remove the containment generation last.
When a timed Shields window reaches its deadline, auto-restore closes the per-sandbox lifecycle gate before it restores lockdown.
`$$nemoclaw <name> shields status` can finish cleanup for a completed, abandoned auto-restore, but live or ambiguous ownership remains fail-closed.
<AgentOnly variant="openclaw,hermes">
For retry budgets, containment behavior, and safe remediation for each retained-artifact state, see [Timed Shields Windows](../manage-sandboxes/configure-sandboxes/understand-runtime-changes#timed-shields-windows).
</AgentOnly>

Before a manual Shields transition replaces a policy, NemoClaw requires agreement among the sandbox registry, generated-policy record, and live gateway policy.
`shields down` carries the proven managed MCP policy entries into the relaxed policy.
Expand Down
22 changes: 21 additions & 1 deletion src/commands/sandbox/snapshot.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,17 @@
import { beforeEach, describe, expect, it, vi } from "vitest";

const runSandboxSnapshot = vi.hoisted(() => vi.fn().mockResolvedValue(undefined));
const hasShieldsTimerRecoveryArtifact = vi.hoisted(() => vi.fn(() => false));

vi.mock("../../lib/actions/sandbox/snapshot", () => ({
runSandboxSnapshot,
}));

vi.mock("../../lib/state/mcp-lifecycle-lock/shields-timer-authority", async (importOriginal) => ({
...(await importOriginal()),
hasShieldsTimerRecoveryArtifact,
}));

import SnapshotCommand from "./snapshot";
import SnapshotCreateCommand from "./snapshot/create";
import SnapshotListCommand from "./snapshot/list";
Expand All @@ -19,9 +25,10 @@ const rootDir = process.cwd();
describe("snapshot oclif commands", () => {
beforeEach(() => {
runSandboxSnapshot.mockClear();
hasShieldsTimerRecoveryArtifact.mockClear();
});

it("shows parent snapshot usage through the action", async () => {
it("shows parent snapshot usage through the action", { timeout: 30_000 }, async () => {
await SnapshotCommand.run(["alpha"], rootDir);

expect(runSandboxSnapshot).toHaveBeenCalledWith("alpha", { kind: "help" });
Expand Down Expand Up @@ -51,6 +58,19 @@ describe("snapshot oclif commands", () => {
});
});

it("leaves multi-sandbox recovery entirely to the snapshot action", async () => {
await SnapshotRestoreCommand.run(["alpha", "--to", "beta"], rootDir);

expect(hasShieldsTimerRecoveryArtifact).not.toHaveBeenCalled();
expect(runSandboxSnapshot).toHaveBeenCalledWith("alpha", {
kind: "restore",
selector: undefined,
to: "beta",
force: undefined,
yes: undefined,
});
});

it("threads --force and --yes into the typed restore action (#3756)", async () => {
await SnapshotRestoreCommand.run(["alpha", "--to", "beta", "--force", "--yes"], rootDir);

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,7 @@ vi.mock("../../shields", () => ({
get isShieldsDown() {
return true;
},
recoverCompletedAutoRestoreBeforeCommand: vi.fn(),
repairMutableConfigPerms: vi.fn(() => ({ applied: true, verified: true, errors: [] })),
shieldsUp: vi.fn(),
}));
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@ vi.mock("../../runtime-recovery", () => ({

vi.mock("../../shields", () => ({
isShieldsDown: vi.fn(() => true),
recoverCompletedAutoRestoreBeforeCommand: vi.fn(),
repairMutableConfigPerms: vi.fn(() => ({ applied: true, verified: true, errors: [] })),
}));

Expand Down
5 changes: 5 additions & 0 deletions src/lib/actions/sandbox/snapshot-restore-test-fixture.ts
Original file line number Diff line number Diff line change
Expand Up @@ -124,11 +124,13 @@ const shieldsMock = vi.hoisted(() => {
verified: true,
errors: [],
}));
const recoverCompletedAutoRestoreBeforeCommandMock = vi.fn(() => false);
const shieldsUpMock = vi.fn();
let isShieldsDownExport: unknown = isShieldsDownMock;
return {
isShieldsDownMock,
repairMutableConfigPermsMock,
recoverCompletedAutoRestoreBeforeCommandMock,
shieldsUpMock,
getIsShieldsDownExport: () => isShieldsDownExport,
setIsShieldsDownExport: (value: unknown) => {
Expand Down Expand Up @@ -301,6 +303,8 @@ vi.mock("../../shields", () => ({
return shieldsMock.getIsShieldsDownExport();
},
repairMutableConfigPerms: shieldsMock.repairMutableConfigPermsMock,
recoverCompletedAutoRestoreBeforeCommand:
shieldsMock.recoverCompletedAutoRestoreBeforeCommandMock,
shieldsUp: shieldsMock.shieldsUpMock,
}));

Expand Down Expand Up @@ -385,6 +389,7 @@ export function resetSnapshotRestoreMocks(): void {
});
shieldsMock.setIsShieldsDownExport(shieldsMock.isShieldsDownMock);
shieldsMock.isShieldsDownMock.mockReturnValue(true);
shieldsMock.recoverCompletedAutoRestoreBeforeCommandMock.mockReturnValue(false);
shieldsMock.shieldsUpMock.mockImplementation(() => lifecycleMock.events.push("harden"));
lifecycleMock.events.length = 0;
lifecycleMock.readTimerMarkerMock.mockReturnValue(null);
Expand Down
13 changes: 12 additions & 1 deletion src/lib/actions/sandbox/snapshot.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1318,7 +1318,7 @@ async function runSnapshotRestore(
if (targetSandbox !== sandboxName) {
assertSandboxSnapshotCommandAvailable(targetSandbox, "sandbox:snapshot:restore");
}
const orderedNames = [...new Set(lockNames)].sort();
const orderedNames = recoverCompletedAutoRestoreForSnapshotRestore(lockNames);
const acquire = (index: number): Promise<void> =>
index === orderedNames.length
? Promise.resolve().then(() => {
Expand All @@ -1332,6 +1332,17 @@ async function runSnapshotRestore(
return acquire(0);
}

export function recoverCompletedAutoRestoreForSnapshotRestore(
sandboxNames: readonly string[],
stateDir?: string,
): string[] {
const orderedNames = [...new Set(sandboxNames)].sort();
for (const name of orderedNames) {
shields.recoverCompletedAutoRestoreBeforeCommand(name, stateDir);
}
return orderedNames;
}

async function runSnapshotRestoreUnlocked(
sandboxName: string,
request: Extract<SnapshotRequest, { kind: "restore" }>,
Expand Down
21 changes: 17 additions & 4 deletions src/lib/cli/nemoclaw-oclif-command.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ import { hasHermesPortableReceiptCandidate } from "../onboard/experimental/herme
import { defaultPortableDemoStateDir } from "../onboard/experimental/portable-runtime-receipt-readiness";
import { redactForLog } from "../security/redact";
import { isDeferredShieldsExit } from "../shields/deferred-exit";
import { resolveShieldsStateDir } from "../shields/transition-lock";
import { hasShieldsTimerRecoveryArtifact } from "../state/mcp-lifecycle-lock/shields-timer-authority";
import {
assertNoHermesPortableHostAuthority,
withCurrentPortableHostFence,
Expand Down Expand Up @@ -105,19 +107,30 @@ export abstract class NemoClawCommand extends Command {
if (portablePolicy?.ownsLifecycleFence) return await super._run<T>();
const sandboxName = await this.resolveLifecycleSandboxName(portablePolicy);
if (!sandboxName) return await super._run<T>();
if (this.isInteractiveConnect(commandId)) return await super._run<T>();
const recoverCompletedAutoRestore = async () => {
if (hasShieldsTimerRecoveryArtifact(sandboxName, resolveShieldsStateDir())) {
const { recoverCompletedAutoRestoreBeforeCommand } = await import("../shields");
recoverCompletedAutoRestoreBeforeCommand(sandboxName);
}
};
if (this.isInteractiveConnect(commandId)) {
await recoverCompletedAutoRestore();
return await super._run<T>();
}
const runLocked = () => {
if (typeof commandId === "string" && portablePolicy?.rawSandboxName) {
assertHermesPortableCommandSupported(commandId, sandboxName, this.argv);
}
return super._run<T>();
};
const runWithLifecycleFence = () =>
commandId === "sandbox:destroy"
const runWithLifecycleFence = async () => {
await recoverCompletedAutoRestore();
return await (commandId === "sandbox:destroy"
? withMcpLifecycleLock(sandboxName, runLocked, {
recoverAbandonedExpiredTimer: true,
})
: withMcpLifecycleLock(sandboxName, runLocked);
: withMcpLifecycleLock(sandboxName, runLocked));
};
if (
this.isProbeOnlyConnect(commandId) &&
hasHermesPortableReceiptCandidate(sandboxName, defaultPortableDemoStateDir(process.env))
Expand Down
Loading
Loading