fix(state): restore preserved backups into replacement sandboxes

[laitingsheng]

Summary

A preserved backup could not be recovered into a replacement sandbox once the original sandbox was gone, because cross-sandbox restore required the source to still be running. For Docker- and VM-driver sandboxes, restore now resolves the clone image from the source registry entry, so a stopped source with a complete durable inference route stays restorable. Restoring a SQLite state file also reported success from the transport exit status alone; it now opens a write transaction against the replaced database and fails when the database cannot be written.

Related Issue

Resolves #7807
Resolves #7800
Fixes #7799

Changes

• src/lib/actions/sandbox/snapshot.ts: for Docker- and VM-driver sandboxes, cross-sandbox restore no longer aborts when the source is absent from the gateway listing. It uses the recorded image and requires a complete durable inference route; a missing image stops with nemoclaw onboard guidance. Kubernetes sources still require their pod image to be resolvable through the gateway.
• src/lib/state/state-file-restore.ts: the sqlite_backup restore opens BEGIN IMMEDIATE and rolls back against the replaced database, under the same umask as the restore so its own sidecars stay group-writable. It drops both the stale sidecars and the ones the check creates, and extends the cleanup trap to the staged sidecars.
• src/lib/actions/sandbox/snapshot-restore-offline-source.test.ts: a stopped source with a registered image restores into an auto-created replacement; a stopped source without an image stops before creating or deleting anything.
• src/lib/state/state-file-sqlite-restore-behavior.test.ts: runs the generated restore command in a real shell against real SQLite and asserts the restored database accepts a write, that stale sidecars are gone, and that an unwritable target fails the restore.
• docs/manage-sandboxes/backup-restore.mdx: documents restoring into a replacement from a stopped source, and the write check.

The reported non-writable WAL and SHM files in #7799 were already fixed by #7344, which is tagged in v0.0.95 and v0.0.96; the report is against v0.0.93. This PR adds the remaining part of that report, the write check the restore never performed.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification:
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Codex Desktop nine-category review at exact head 3eb94e62c against base 9b1fbd897: PASS; no security findings; product scope remains bounded to the accepted snapshot-restore bugs.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: docs-updated
• Evidence: docs/manage-sandboxes/backup-restore.mdx and docs/reference/commands.mdx; exact-head review confirmed driver scope, image and inference-route prerequisites, in-place requirements, Kubernetes behavior, SQLite write verification, and pre-delete guarantees. Focused tests (2 files, 4 tests), CLI typecheck, npm run docs (0 errors; existing Fern warnings), and npm run check:diff (compatibility alias invoking npm run validate:pr) passed.
• Agent: Codex Desktop documentation-writer subagent

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run validate:pr passed after refreshing origin/main when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: npx vitest run --project cli src/lib/actions/sandbox/snapshot-restore-offline-source.test.ts src/lib/state/state-file-sqlite-restore-behavior.test.ts — 2 files, 4 tests passed; npm run typecheck:cli passed on 3eb94e62c.
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result:
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Tinson Lai tinsonl@nvidia.com

Summary by CodeRabbit

• Bug Fixes

  • Cross-sandbox snapshot restore now resolves the source registry image upfront and validates prerequisites before any destination overwrite/deletion.
  • Improved restore abort behavior for incomplete durable inference routes and clearer failure handling when image resolution isn’t possible.
  • sqlite_backup-based restores now verify the restored database is writable and fail safely when it can’t be opened for writes.

• Documentation

  • Expanded Hermes snapshot contents/restore failure details, plus clarified cross-sandbox restore requirements and stopping rules.

• Tests

  • Added coverage for offline-source restore nullability and for SQLite writeability restore success/failure scenarios.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 707bb1bb-8818-4a68-acca-4e1516728dec

📥 Commits

Reviewing files that changed from the base of the PR and between 3eb94e6 and 20e5a55.

📒 Files selected for processing (3)

• docs/manage-sandboxes/backup-restore.mdx
• docs/reference/commands.mdx
• src/lib/actions/sandbox/snapshot-restore-offline-source.test.ts

🚧 Files skipped from review as they are similar to previous changes (3)

• src/lib/actions/sandbox/snapshot-restore-offline-source.test.ts
• docs/manage-sandboxes/backup-restore.mdx
• docs/reference/commands.mdx

---

📝 Walkthrough

Walkthrough

Cross-sandbox snapshot restores now validate source image and inference-route metadata before destination changes. SQLite state restores now remove stale sidecars and verify writable transactions, with tests and documentation covering both behaviors.

Changes

Sandbox restore recovery

Layer / File(s)	Summary
Offline-source snapshot restoration src/lib/actions/sandbox/snapshot.ts, src/lib/actions/sandbox/snapshot-restore-offline-source.test.ts, docs/reference/commands.mdx, docs/manage-sandboxes/backup-restore.mdx	Cross-sandbox restores resolve source images and durable inference routes for stopped sources, reject incomplete metadata before creating or replacing destinations, and document Docker/VM and Kubernetes prerequisites.
SQLite restore writeability validation src/lib/state/state-file-restore.ts, src/lib/state/state-file-sqlite-restore-behavior.test.ts, docs/manage-sandboxes/backup-restore.mdx	SQLite restores remove stale sidecars, atomically swap the database, verify a writable transaction, return failure code 12 when validation fails, and document Hermes restore behavior.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Suggested labels: integration: hermes

Sequence Diagram(s)

Offline source replacement flow

sequenceDiagram
  participant SnapshotRestore
  participant SourceRegistry
  participant ReplacementSandbox
  SnapshotRestore->>SourceRegistry: resolve source image and inference route
  SourceRegistry-->>SnapshotRestore: return complete or incomplete metadata
  SnapshotRestore->>ReplacementSandbox: create destination from recorded image

Loading

SQLite restore validation flow

sequenceDiagram
  participant RestoreCommand
  participant RestoreScript
  participant SQLiteDatabase
  RestoreCommand->>RestoreScript: restore backup and swap database
  RestoreScript->>SQLiteDatabase: remove stale WAL and SHM sidecars
  RestoreCommand->>SQLiteDatabase: begin immediate transaction and roll back
  SQLiteDatabase-->>RestoreCommand: return writable or failure status

Loading

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: restoring preserved backups into replacement sandboxes.
Linked Issues check	✅ Passed	The restore flow, SQLite write validation, rollback cleanup, and stopped-source handling address the linked recovery and writable-state requirements.
Out of Scope Changes check	✅ Passed	The changes stay focused on restore behavior, validation, tests, and docs with no clear unrelated additions.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/portable-backup-restore

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-7814.docs.buildwithfern.com/nemoclaw

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 20e5a55 in the fix/portable-backup-... branch remains at 96%, unchanged from commit 376beb5 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 20e5a55 in the fix/portable-backup-... branch remains at 81%, unchanged from commit 376beb5 in the main branch.

Show a code coverage summary of the most impacted files.

File	main 376beb5	fix/portable-backup-... 20e5a55	+/-
src/lib/domain/.../connect-env.ts	97%	89%	-8%
src/lib/state/s...file-restore.ts	97%	97%	0%
src/lib/sandbox...rce-identity.ts	88%	88%	0%
src/lib/actions...box/snapshot.ts	81%	83%	+2%
src/lib/onboard...ndbox-create.ts	83%	91%	+8%
src/lib/onboard...-create-plan.ts	75%	88%	+13%
src/lib/onboard...ndbox-create.ts	33%	83%	+50%

---

_{Updated July 30, 2026 03:18 UTC}

[github-actions]

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: Review the warnings below.
Findings: 0 blockers · 1 warning · 0 suggestions

Model lanes

• GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 1 warning · 0 suggestions
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 1 suggestion
• Model comparison: normalized findings differ; normalized E2E selections differ; Nemotron reported the same number of blockers, 1 fewer warning, 1 more suggestion.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: onboard-repair, onboard-resume, state-backup-restore, upgrade-stale-sandbox

2 optional E2E recommendations

• snapshot-commands
• rebuild-hermes

1 warning · 0 suggestions

Warnings

Warnings do not block.

PRA-1 Warning — Cover incomplete-route failure before forced destination deletion

• Location: src/lib/actions/sandbox/snapshot-restore-offline-source.test.ts:84
• Category: tests
• Problem: The incomplete inference-route test restores to a destination that does not exist. It does not cover the forced replacement path, where an existing destination must remain intact when route validation fails.
• Impact: A later change can move or bypass the route check after destination deletion without failing the current tests, allowing an invalid source record to destroy an existing sandbox.
• Recommendation: Add a test with an existing destination and `force: true`, an incomplete source route, and assertions that restore rejects before destination deletion or sandbox creation.
• Verification: Inspect the new test cases and `snapshot.ts` to confirm that the incomplete-route case has no existing `--force` destination and that `deleteSandboxForRestore` is not asserted.
• Test coverage: A mocked restore test with a registered or live destination, `force: true`, and an incomplete source inference route must reject and assert that `deleteSandboxForRestore` and sandbox creation were not called.
• Evidence: src/lib/actions/sandbox/snapshot-restore-offline-source.test.ts:84-101 invokes the incomplete-route case without creating or registering `beta`. src/lib/actions/sandbox/snapshot.ts:1037-1105 requires a complete durable route before `deleteSandboxForRestore` on the forced-destination path.

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/state/state-file-sqlite-restore-behavior.test.ts`:
- Around line 77-87: Update the test “reports failure when the restored database
cannot be written” so staging and swap can complete while an exclusive SQLite
lock blocks only the final write check, rather than making the runtime directory
unwritable. Ensure the lock/setup is released and any permission restoration
runs in a finally block so cleanup occurs even when assertions or restore
execution fail.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 482a0ca7-f506-4b8a-bcc4-8601440f564f

📥 Commits

Reviewing files that changed from the base of the PR and between eeab81c and 8aa98a6.

📒 Files selected for processing (5)

• docs/manage-sandboxes/backup-restore.mdx
• src/lib/actions/sandbox/snapshot-restore-offline-source.test.ts
• src/lib/actions/sandbox/snapshot.ts
• src/lib/state/state-file-restore.ts
• src/lib/state/state-file-sqlite-restore-behavior.test.ts

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/reference/commands.mdx`:
- Around line 2900-2911: Update the snapshot restore prerequisites section to
explicitly cover absent sources, stating the required condition for restoring
when the source no longer exists, or clearly define “stopped” to include absent
Docker/VM sources. Keep the existing Kubernetes gateway requirement and
destination validation behavior unchanged.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e4a8ac9a-7896-4a4c-bfc7-242467136ac3

📥 Commits

Reviewing files that changed from the base of the PR and between 12dec40 and cd77c32.

📒 Files selected for processing (2)

• docs/manage-sandboxes/backup-restore.mdx
• docs/reference/commands.mdx

🚧 Files skipped from review as they are similar to previous changes (1)

• docs/manage-sandboxes/backup-restore.mdx

[cjagwani]

Maintainer follow-up (no changes requested): exact head 3eb94e62c has green CI, the selected E2E gate passed, and the exact-head security/docs receipts are present. The patch also merges cleanly in a local merge-tree check against current main (795de982e).

The PR is still based on 9b1fbd897, and maintainer edits are disabled, so I cannot perform the guarded refresh. Please merge current main into this branch. I’ll babysit the refreshed head through its replacement CI/E2E and final exact-head review.

[cjagwani]

Exact-head review after the contributor refresh onto current main. Deterministic gate passes with all 46 current checks green, clean mergeability, all 15 commits Verified, and no unresolved major findings. Focused exact-head tests pass 5 of 5. Security review PASS: secrets and credentials; input validation; authentication and authorization; dependencies and supply chain; error handling and exposure; cryptography; configuration and infrastructure; security tests; and system security. Source and destination mutations remain locked, registry image and complete inference route are re-read with gateway compatibility before deletion, Kubernetes still requires a live gateway image probe, and SQLite replacement remains staged and atomic with a fail-closed write transaction check. The accepted snapshot-restore bug scope is unchanged.

[cjagwani]

Current-base handoff after main advanced to da1b103: exact head 20e5a55 remains approved and its reviewed product patch is unchanged, but maintainer edits are disabled and the branch now lacks current-base CI/gate evidence. Please refresh it; I will re-gate the next head after the one-hour quiet window. This is a plain status comment, not Changes Requested.

[cjagwani]

Correction to my prior handoff: conflict-free base refreshes are explicitly waived. Please do not merge main solely for base currency; preserving exact-head CI/E2E and documentation receipts is preferred unless GitHub reports a real conflict or reviewed behavior requires a change. The existing approval/review evidence remains in force, and I will continue monitoring mergeability. This is a plain coordination comment, not Changes Requested.