Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions ci/env-var-doc-allowlist.json
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,14 @@
"name": "NEMOCLAW_TEST_NO_SLEEP",
"reason": "Test sentinel that bypasses real-time sleep() calls in onboard inference probes. Set to '1' only by Vitest tests; never user-set."
},
{
"name": "NEMOCLAW_TEST_BASE_HOME",
"reason": "Internal Vitest-only baseline used with NEMOCLAW_TEST_STATE_DIR so tests that explicitly replace HOME retain their fixture paths. Never user-set in production."
},
{
"name": "NEMOCLAW_TEST_STATE_DIR",
"reason": "Internal Vitest-only state root that keeps lifecycle locks and Shields artifacts out of the caller's real NemoClaw state. The production resolver honors it only while Vitest is active."
},
{
"name": "NEMOCLAW_TELEGRAM_STARTUP_GRACE_MS",
"reason": "Internal Vitest-only override that shortens the Telegram diagnostics startup-grace timer. Production uses the built-in default."
Expand Down
2 changes: 1 addition & 1 deletion ci/source-architecture-budget.json
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
"src/lib/adapters/openshell/runtime.ts": 52,
"src/lib/adapters/openshell/timeouts.ts": 37,
"src/lib/agent/defs.ts": 32,
"src/lib/cli/branding.ts": 85,
"src/lib/cli/branding.ts": 86,
"src/lib/cli/nemoclaw-oclif-command.ts": 103,
"src/lib/cli/terminal-style.ts": 45,
"src/lib/core/json-types.ts": 37,
Expand Down
5 changes: 5 additions & 0 deletions ci/source-shape-test-budget.json
Original file line number Diff line number Diff line change
Expand Up @@ -746,6 +746,11 @@
"test": "wires cleanup into root and standalone plugin test runs",
"category": "compatibility"
},
{
"file": "test/vitest-temp-root.test.ts",
"test": "isolates stateful non-live projects without redirecting live E2E state",
"category": "security"
},
{
"file": "test/vitest-watch-triggers.test.ts",
"test": "registers the focused mappings at the root configuration boundary (#6692)",
Expand Down
19 changes: 13 additions & 6 deletions docs/manage-sandboxes/backup-restore.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -91,8 +91,14 @@ NemoClaw computes versions (`v1`, `v2`, through `vN`) from timestamp order, so `
`snapshot create` requires shields to be down.
Snapshot creation and restore share the per-sandbox transition lock with the shields auto-restore timer.

If a timed shields-down window expires during snapshot work, auto-restore can interrupt the operation and restore lockdown instead of allowing state or policy changes to continue past the deadline.
Retry the snapshot in a new shields-down window if the deadline interrupts it.
If a timed shields-down window expires during snapshot work, auto-restore closes the per-sandbox lifecycle deadline gate.
The gate blocks new mutations and waits for the recorded live owner to release its exact lock generation.
NemoClaw does not signal the snapshot process.
If ownership becomes ambiguous, NemoClaw attempts to record durable containment and reports exact-generation recovery guidance.
If the containment commit fails, NemoClaw retains any exact lifecycle and deadline gates it already owns.
A state-directory failure that prevented gate publication also prevents normal mutation-lock acquisition.
Correct the reported state-directory write failure, then run `$$nemoclaw <name> shields status` to resume recovery or receive exact-generation recovery guidance.
If the command reports exact-generation recovery guidance, stop all NemoClaw processes for the sandbox, then follow that guidance before you retry the snapshot.

Tag a snapshot with a human-readable label:

Expand Down Expand Up @@ -207,11 +213,12 @@ If a registered docker-driver sandbox's container is stopped, `backup-all` start
If the container cannot be returned to the stopped state, the backup run fails and reports that the container was left running.
If a sandbox is not running and its container cannot be started this way, start the sandbox or its container and rerun `$$nemoclaw backup-all`.

When Shields are UP for an eligible sandbox, `backup-all` opens a 30-minute shields-down window before it creates that sandbox's snapshot.
For each eligible sandbox, `backup-all` holds one lifecycle transaction through the complete backup.
Within that transaction, it starts a stopped container when required, opens a 30-minute shields-down window when the sandbox starts with Shields up, copies sandbox state, restores the previous Shields state, and returns any container it started to the stopped state.
A sandbox that starts with Shields down remains down.
An unlock failure marks that sandbox as failed, and `backup-all` continues with the next sandbox.
Because the timer does not defer to the backup process, it can restore lockdown when the 30-minute deadline expires.
NemoClaw always attempts to restore Shields lockdown before it processes the next sandbox, including when the backup fails.
If the timer expires during the transaction, the deadline gate blocks new mutations and waits for the exact backup owner to finish without signaling it.
An initial lock or unlock failure marks that sandbox as failed, and `backup-all` continues with the next sandbox.
NemoClaw attempts to restore the previous Shields state before it processes the next sandbox, including when the backup fails.
If lockdown cannot be restored, `backup-all` stops and does not process the remaining sandboxes.
Correct the reported issue, run the printed `$$nemoclaw <name> shields up` command, and rerun `$$nemoclaw backup-all`.

Expand Down
20 changes: 18 additions & 2 deletions docs/manage-sandboxes/runtime-controls.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -108,10 +108,26 @@ Run `$$nemoclaw <name> shields down` before the change, then restore lockdown wi
NemoClaw serializes host-side config and inference writes, snapshot mutation, sandbox destruction, and shields transitions for each sandbox.
When `shields down --timeout` is active, each mutation binds to that exact timer generation so a replaced or expired timer cannot race a later command or a new sandbox that reuses the same name.

If the timeout expires while a mutation is still changing sandbox state, auto-restore can stop that exact process tree, reclaim the transition, and restore the restrictive policy and config posture.
If the timeout expires while a mutation is changing sandbox state, auto-restore closes the per-sandbox lifecycle deadline gate.
The gate blocks new mutations and waits for the recorded live owner to release its exact lock generation.
NemoClaw does not signal that process because portable process inspection cannot prove that all descendants stopped.
After the owner releases the lock, auto-restore restores the restrictive policy and configuration posture.
The ownership check includes both the process ID and process start identity so PID reuse does not grant control over an unrelated process.

Retry a command that the auto-restore deadline interrupts after you open a new shields-down window.
An interactive command can take over an expired timer.
Interactive recovery has separate transition-takeover and restoration phases.
Each phase makes up to 7 attempts and waits 5 seconds between failures, for up to 30 seconds of retry delay per phase.
Detached recovery uses one 7-attempt budget across deadline setup, main-generation publication, and restoration.
The deadline gate remains closed during those attempts.
If restoration cannot commit, NemoClaw attempts to record durable containment.
If that containment commit also fails, NemoClaw retains any exact lifecycle and deadline gates it already owns.
A state-directory failure that prevented gate publication also prevents normal mutation-lock acquisition.
Correct the reported state-directory write failure, then run `$$nemoclaw <name> shields status` to resume recovery or receive exact-generation recovery guidance.
When recovery cannot complete, an interactive command returns an error, or the detached timer exits with a failure status.
NemoClaw also records durable containment when an owner exits before it can prove that the owner's descendants stopped, or when ownership becomes ambiguous.
Durable containment, retained exact gates, or the fail-closed state-directory error keeps new mutations blocked until you complete exact-generation operator recovery.
Stop all NemoClaw processes for the sandbox, then follow the paths, identities, tokens, and removal order in the reported error.
Verify each recorded generation is unchanged, remove only the exact stale generations first, and remove the exact containment generation last.

## Related Topics

Expand Down
39 changes: 35 additions & 4 deletions docs/reference/commands.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1173,8 +1173,23 @@ Follow [Deep Agents Config Lock Failure Recovery](troubleshooting#deep-agents-co
</AgentOnly>

Host-side config and inference writes, snapshot mutation, sandbox destruction, and shields transitions serialize per sandbox.
When a timed shields-down window reaches its deadline, auto-restore can interrupt the exact process tree holding that transition and restore lockdown.
Retry an interrupted command in a new shields-down window.
When a timed shields-down window reaches its deadline, auto-restore closes the per-sandbox lifecycle deadline gate.
The gate blocks new mutations and waits for the recorded live owner to release its exact lock generation before auto-restore restores lockdown.
NemoClaw does not signal that process because portable process inspection cannot prove that all descendants stopped.
An interactive command can take over an expired timer.
Interactive recovery has separate transition-takeover and restoration phases.
Each phase makes up to 7 attempts and waits 5 seconds between failures, for up to 30 seconds of retry delay per phase.
Detached recovery uses one 7-attempt budget across deadline setup, main-generation publication, and restoration.
The deadline gate remains closed during those attempts.
If restoration cannot commit, NemoClaw attempts to record durable containment.
If that containment commit also fails, NemoClaw retains any exact lifecycle and deadline gates it already owns.
A state-directory failure that prevented gate publication also prevents normal mutation-lock acquisition.
Correct the reported state-directory write failure, then run `$$nemoclaw <name> shields status` to resume recovery or receive exact-generation recovery guidance.
When recovery cannot complete, an interactive command returns an error, or the detached timer exits with a failure status.
NemoClaw also records durable containment when an owner exits before it can prove that the owner's descendants stopped, or when ownership becomes ambiguous.
Durable containment, retained exact gates, or the fail-closed state-directory error blocks new mutations until you complete exact-generation operator recovery.
Stop all NemoClaw processes for the sandbox, then follow the paths, identities, tokens, and removal order in the reported error.
Verify each recorded generation is unchanged, remove only the exact stale generations first, and remove the exact containment generation last.

<AgentOnly variant="openclaw,hermes">

Expand Down Expand Up @@ -2938,6 +2953,12 @@ A registered docker-driver sandbox whose container is stopped is started for the
If the container cannot be returned to the stopped state, the command fails and reports that the container was left running.
Sandboxes that are not running and cannot be started this way are skipped with remediation guidance.

For each eligible sandbox, `backup-all` holds one lifecycle transaction through the complete backup.
Within that transaction, it starts a stopped container when required, opens a 30-minute shields-down window when the sandbox starts with Shields up, copies sandbox state, restores the previous Shields state, and returns any container it started to the stopped state.
If the timer expires during the transaction, the deadline gate blocks new mutations and waits for the exact backup owner to finish without signaling it.
An initial lock or unlock failure marks that sandbox as failed, and `backup-all` continues with the next sandbox.
A failure to restore the previous Shields state stops `backup-all` before it processes another sandbox.

```bash
$$nemoclaw backup-all
```
Expand All @@ -2957,7 +2978,12 @@ A skipped sandbox's uncommitted state is not included in its last successful bac
Create a timestamped snapshot of sandbox state.
Snapshots are stored in `~/.nemoclaw/rebuild-backups/<name>/`.
The command requires shields to be down and keeps the shields check and backup under one per-sandbox transition.
An expired auto-restore timer can interrupt a long-running backup and restore lockdown.
If the timer expires during a long-running backup, the deadline gate blocks new mutations and waits for the exact backup owner to finish.
Auto-restore does not signal the backup process.
If ownership becomes ambiguous, NemoClaw attempts to record durable containment and reports exact-generation recovery guidance.
If the containment commit fails, NemoClaw retains any exact lifecycle and deadline gates it already owns.
A state-directory failure that prevented gate publication also prevents normal mutation-lock acquisition.
Correct the reported state-directory write failure, then run `$$nemoclaw <name> shields status` to resume recovery or receive exact-generation recovery guidance.
When the sandbox has active baseline exclusions, successful output lists their keys and repeats that excluded egress leaves dependent agent features unsupported for that sandbox.

```bash
Expand Down Expand Up @@ -2991,7 +3017,12 @@ If no selector is provided, the latest snapshot is used.
Restore removes files added after the snapshot only from state directories selected for cleanup.
It preserves directories that exist only in the target manifest or whose backup failed.
The state replacement, mutable-config permission repair, and policy reconciliation run under the same per-sandbox transition.
An expired auto-restore timer can interrupt that work and restore lockdown.
If the timer expires during that work, the deadline gate blocks new mutations and waits for the exact restore owner to finish.
Auto-restore does not signal the restore process.
If ownership becomes ambiguous, NemoClaw attempts to record durable containment and reports exact-generation recovery guidance.
If the containment commit fails, NemoClaw retains any exact lifecycle and deadline gates it already owns.
A state-directory failure that prevented gate publication also prevents normal mutation-lock acquisition.
Correct the reported state-directory write failure, then run `$$nemoclaw <name> shields status` to resume recovery or receive exact-generation recovery guidance.

The selector accepts any of:

Expand Down
Loading
Loading