fix(backup): preserve Shields during backup-all

[apurvvkumaria]

Summary

backup-all now backs up intentionally protected state by opening one bounded Shields window per eligible sandbox and restoring the original locked posture before it processes the next sandbox.
The change follows @Hokonoken's live root-cause analysis in #6455, keeps incomplete backups as hard failures, and now has protected live coverage for root-owned credential state.
Corrupt persisted Shields state fails closed before policy, config, timer, transition, state, or audit mutation; recovery requires restoring trusted host state.

Related Issue

Fixes #6455

Changes

• Extract the existing rebuild Shields window into a shared helper while keeping the legacy Hermes compatibility path and owner-liveness lease restricted to rebuild.
• Open a 30-minute auto-restore window around each backup-all attempt only when the sandbox starts with Shields UP.
• Restore Shields lockdown in finally after success, a reported backup failure, or an exception.
• Abort the remaining backup batch when lockdown cannot be restored, and print the sandbox-specific shields up recovery command before the retry command.
• Preserve the current mutable posture for a sandbox that starts with Shields down.
• Reject corrupt persisted Shields state before shields up can skip restrictive-policy restoration or destroy timer and transition recovery authority.
• Keep corrupt-state recovery guidance fail closed: restore trusted host state; neither shields up nor ordinary rebuild replaces corrupt state.
• Add protected snapshot-commands coverage for root-owned 0700/0600 credentials, sanitized backup contents, Shields relock, restored permissions, and timer removal.
• Document the bounded backup window, recovery behavior, and corrupt-state refusal on the canonical backup and troubleshooting pages.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification:
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Independent exact-head review at 10954f50ec56bff21e6b829e977b70983995bb90 passed product scope and all nine security categories. The ten PR files are byte-identical to the prior passing security review. Corrupt state is rejected before config resolution, policy commands, state writes, timer kill, transition clear, or audit append. The 30-minute no-owner-deferral window and batch abort on relock failure remain intentional documented contracts. Trusted exact-head snapshot-commands E2E is still required before merge.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: docs-updated
• Evidence: At exact head 10954f50ec56bff21e6b829e977b70983995bb90 against base 0b185498155a0a51a3f682a3e2b57f80c95eeaaa, reviewed the complete ten-file diff and generated OpenClaw, Hermes, and Deep Agents variants. The ten PR files are byte-unchanged from passing reviewed head b2c642425. docs/manage-sandboxes/backup-restore.mdx accurately documents the bounded per-sandbox backup-all Shields window and failure behavior. docs/reference/troubleshooting.mdx documents fail-closed corrupt-state preservation and trusted-state recovery for all three documented Shields variants; each generated troubleshooting page contains the section exactly once. Exact-tree npm run docs passed with 0 errors and two existing warnings; targeted tests passed 73/73; CLI and plugin typechecks and the merge pre-push CLI typecheck passed. GitHub reports the exact-head commit Verified.
• Agent: Codex Desktop

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit:
• Station profile/scenario:
• Result:
• Supporting evidence:

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — justification: Shields, maintenance, and rebuild suites passed 73/73; E2E-support passed 1,715 with 17 intentional skips; CLI type-checking and exact-branch CLI/plugin builds passed.
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: Exact-head GitHub CI and protected snapshot-commands E2E are in progress; no result is waived.
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only) — 0 errors; two pre-existing hidden Fern warnings remain.
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Apurv Kumaria akumaria@nvidia.com
Signed-off-by: Carlos Villela cvillela@nvidia.com

Summary by CodeRabbit

• New Features
  • backup-all now performs a timed Shields-down window per eligible sandbox, then restores lockdown, while tracking whether the window opened successfully.
  • Backup lifecycle more consistently handles orphan-manifest cases and preserves backup/restore error context.
• Bug Fixes
  • Fail-closed behavior for corrupt Shields state: NemoClaw preserves the recorded state and refuses to unlock/relock, with recovery guidance.
  • When Shields restoration fails after backup, the run aborts remaining sandboxes and reports actionable commands.
• Documentation
  • Updated backup/restore and troubleshooting docs for Shields windows, timers, and corrupt-state recovery.
• Tests
  • Expanded unit and protected live coverage for shields window behavior, error aggregation, and protected credentials redaction/permissions.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

backup-all now manages Shields windows around each sandbox backup, restores lockdown before continuing, and aborts when restoration fails. Shared window logic is reused by rebuild operations, with corrupt-state protection, tests, and documentation covering unlock, relock, failure, and sequencing behavior.

Changes

Backup shields lifecycle

Layer / File(s)	Summary
Shared shields-window helpers src/lib/actions/sandbox/backup-shields-window.ts, src/lib/actions/sandbox/rebuild-shields.ts, test/rebuild-shields-window.test.ts	Shared types and helpers handle Shields unlock/relock behavior, while rebuild operations delegate to them with rebuild-specific options.
Backup-all shields orchestration src/lib/actions/maintenance.ts, src/lib/actions/maintenance.test.ts, docs/manage-sandboxes/backup-restore.mdx	Each backup runs within a Shields window, restores lockdown in cleanup, handles unlock and relock failures, preserves combined errors, and documents the resulting control flow.
Corrupt Shields state protection src/lib/shields/index.ts, src/lib/shields/index.test.ts, docs/reference/troubleshooting.mdx	Corrupt persisted state blocks Shields mutations, preserves state files, emits recovery guidance, and documents trusted-state restoration.
Backup behavior and protected data validation test/e2e/live/snapshot-commands.test.ts	The live snapshot flow validates sanitized protected-credential backups, permissions, Shields restoration, and timer cleanup.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related PRs

• NVIDIA/NemoClaw#7290 — Both changes modify backupAll() per-sandbox control flow around orphan and early-continue handling.
• NVIDIA/NemoClaw#7421 — Both changes affect Shields failure handling through failShieldsCommand.
• NVIDIA/NemoClaw#7568 — Both changes modify Shields transition logic used by backup window handling.

Suggested labels: bug-fix, area: security

Suggested reviewers: cv, prekshivyas

Sequence Diagram(s)

sequenceDiagram
  participant backupAll
  participant BackupShieldsWindow
  participant SandboxBackup
  participant Shields
  backupAll->>BackupShieldsWindow: open window
  BackupShieldsWindow->>Shields: shieldsDown
  backupAll->>SandboxBackup: create snapshot
  backupAll->>BackupShieldsWindow: relock window
  BackupShieldsWindow->>Shields: shieldsUp

Loading

🚥 Pre-merge checks | ✅ 2 | ❌ 3

❌ Failed checks (3 warnings)

Check name	Status	Explanation	Resolution
Linked Issues check	⚠️ Warning	The PR does not implement the linked issue's cloned-sandbox skip behavior; cloned audit failures still count as backup failures.	Add the cloned-sandbox handling from #6455 so incomplete identity metadata is skipped with a warning instead of failing backup-all.
Out of Scope Changes check	⚠️ Warning	Most changes focus on Shields windows, corrupt-state handling, rebuild logic, and docs, which are unrelated to the linked backup-all clone failure.	Trim the PR to the clone-audit backup-all fix and move Shields/rebuild/corrupt-state work into separate changes.
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: preserving Shields state during backup-all.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/6455-backup-all-shields-window

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 10954f5 in the codex/6455-backup-al... branch remains at 96%, unchanged from commit 0b18549 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 10954f5 in the codex/6455-backup-al... branch is 80%. The coverage in commit a02a3b9 in the main branch is 81%.

Show a code coverage summary of the most impacted files.

File	main a02a3b9	codex/6455-backup-al... 10954f5	+/-
src/lib/onboard/docker-cdi.ts	80%	70%	-10%
src/lib/shields/audit.ts	98%	89%	-9%
src/lib/shields/index.ts	71%	68%	-3%
src/lib/sandbox/config.ts	75%	72%	-3%
src/lib/shields...nsition-lock.ts	87%	86%	-1%
src/lib/actions...ocker-health.ts	65%	82%	+17%
src/lib/onboard...box-prebuild.ts	73%	92%	+19%
src/lib/actions...ields-window.ts	0%	78%	+78%
src/lib/actions...ol-discovery.ts	0%	82%	+82%
src/lib/actions...time-command.ts	0%	100%	+100%

---

_{Updated July 27, 2026 04:54 UTC}

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-7557.docs.buildwithfern.com/nemoclaw

[github-actions]

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

• GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 1 warning · 0 suggestions
• Model comparison: normalized findings differ; normalized E2E selections differ; Nemotron reported the same number of blockers, 1 more warning, the same number of suggestions.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, credential-sanitization, security-posture, onboard-repair, onboard-resume, snapshot-commands, state-backup-restore, upgrade-stale-sandbox

2 optional E2E recommendations

• shields-config
• docs-validation

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/manage-sandboxes/backup-restore.mdx`:
- Around line 187-192: Update the backup-all behavior description in
backup-restore.mdx to state that an initial Shields-unlock failure marks the
sandbox as failed and processing continues with the next sandbox. Also document
that the 30-minute Shields-down window remains fixed without owner-aware
deferral, so a long-running backup may trigger automatic lockdown before
completion.

In `@src/lib/actions/maintenance.ts`:
- Around line 91-119: Update the backup flow around the try/catch/finally block
so relock failure does not throw from finally or replace an existing backup
failure. Capture the relock failure, preserve and propagate the original
backupError or orphanManifestMessage when present, and only report the relock
error directly when backup succeeded; ensure both failure contexts remain
available when both operations fail.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 3e2f4d55-d076-43df-bfc2-41a3f5d2ab42

📥 Commits

Reviewing files that changed from the base of the PR and between 4fdbb79 and 669147d.

📒 Files selected for processing (6)

• docs/manage-sandboxes/backup-restore.mdx
• src/lib/actions/maintenance.test.ts
• src/lib/actions/maintenance.ts
• src/lib/actions/sandbox/backup-shields-window.ts
• src/lib/actions/sandbox/rebuild-shields.ts
• test/rebuild-shields-window.test.ts

[apurvvkumaria]

Addressed the valid exact-head review findings in 68319c3c6:

• Removed the unused import reported by code scanning.
• Stopped throwing from finally. A combined backup and relock failure now raises an AggregateError whose errors preserve both the original backup or orphan-manifest failure and the relock failure.
• Added regressions for both combined-failure variants. The focused suite now passes 4 files and 45 tests.
• Documented unlock-failure continuation and the fixed 30-minute deadline without owner-aware deferral.

On PRA-1, the proposed clone-only skip is intentionally not implemented. The later investigation in #6455 changed the supported direction:

• The live reproduction withdrew the skip proposal because it would turn an incomplete pre-upgrade backup into a false success.
• The recommended remediation was to read legitimate root-owned state with the privilege that wrote it.
• The later release-blocker reproduction showed the same root-owned credentials failure on primary sandboxes, not only clones.

This PR implements that remediation through the bounded Shields window, which makes the protected credentials state readable during backup and restores lockdown afterward. The hard-failure regression applies only if the backup still fails after the window opens. Incomplete backups therefore remain nonzero by design.

[apurvvkumaria]

The exact-head static failure is a clean-checkout CI prerequisite regression on current main, not a backup-flow failure: semantic E2E collection imports nemoclaw/dist/shared/openshell-policy-boundary.cjs before that generated artifact exists. The narrow signed fix is #7561. Keep this PR blocked until #7561 lands and this head receives a fresh normal CI/E2E run; no failed-job rerun or gate waiver is proposed.

[cv]

Security blocker: backup-all derives authorization to unlock from !isShieldsDown(), but isShieldsDown() returns false for corrupt persisted Shields state. That lets a corrupt or tampered posture enter shieldsDown(), overwrite corruption markers, and later reseal an untrusted baseline. Fail closed before mutation when Shields state is corrupt (or establish authoritative physical posture), and add corrupt+locked regression coverage. I disabled auto-merge pending an exact-head repair and fresh gates.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/reference/troubleshooting.mdx`:
- Around line 1687-1694: Move the “A Shields command reports corrupt persisted
state” troubleshooting section out of the OpenClaw-only content block so it is
shared by both OpenClaw and Hermes pages. Preserve the existing recovery
guidance and placement within the general troubleshooting content.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d47cd31c-0bf6-421f-b94a-535a799d3c8a

📥 Commits

Reviewing files that changed from the base of the PR and between 7aa0fd9 and 91867c1.

📒 Files selected for processing (4)

• docs/reference/troubleshooting.mdx
• src/lib/shields/index.test.ts
• src/lib/shields/index.ts
• test/e2e/live/snapshot-commands.test.ts

[prekshivyas]

Exact-head maintainer review at 10954f50ec56bff21e6b829e977b70983995bb90: the implementation, focused tests, documentation review, ordinary CI, self-hosted image checks, security review, and automated review are clean. Protected E2E cannot validate this change yet: snapshot-commands failed in the existing cross-sandbox clone-restore phase before reaching the new protected-backup phase. State restore completed, but authenticated gateway pairing verification failed (test/e2e/live/snapshot-commands.test.ts:432; child run 30238525114, job 89890933833). This is the gateway self-dialback defect tracked by #7215 and addressed by excluded PR #7230. Per the protected merge gate, this assertion failure is not retry-eligible on the same SHA, so I cannot approve this head. Once #7230 lands, refresh this branch and rerun exact-head CI/E2E; approval can follow if the protected backup phase and full selected suite pass.