Skip to content

fix(installer): skip unreachable running sandboxes in pre-upgrade backup - #6199

Merged
cv merged 11 commits into
mainfrom
fix/preupgrade-backup-skip-unreachable
Jul 2, 2026
Merged

fix(installer): skip unreachable running sandboxes in pre-upgrade backup#6199
cv merged 11 commits into
mainfrom
fix/preupgrade-backup-skip-unreachable

Conversation

@laitingsheng

@laitingsheng laitingsheng commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Pre-upgrade backup-all aborted the curl | bash installer whenever a running sandbox's in-sandbox SSH endpoint did not answer, with no override, looping the upgrade forever. This classifies such a sandbox as unreachable and adds NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1 to skip it so the upgrade proceeds and onboarding recovers it from its latest validated backup.

Related Issue

Fixes #6188

Changes

  • src/lib/state/sandbox.ts: add unreachable to BackupResult and set it on an SSH transport-level dir-check failure (exit 255, timeout, spawn error) via a new isSshTransportFailure predicate.
  • src/lib/actions/maintenance.ts: an unreachable running sandbox is skipped when NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1, otherwise it still fails but prints actionable guidance before exit.
  • scripts/install.sh: reword the pre-upgrade backup abort to name the override and the recovery path.
  • docs/reference/commands.mdx: document the flag in the backup-all section.
  • Tests: maintenance.test.ts gains skip-with-flag, fail-with-guidance, and flag truth-table cases; new sandbox.test.ts covers isSshTransportFailure.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification:
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification:
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Verification

  • PR description includes the DCO sign-off declaration and every commit appears as Verified in GitHub
  • Git hooks passed during commit and push, or npx prek run --from-ref main --to-ref HEAD passes
  • Targeted tests pass for changed behavior
  • Full npm test passes (broad runtime changes only)
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Tinson Lai tinsonl@nvidia.com

Summary by CodeRabbit

  • New Features
    • backup-all now detects running sandboxes with an unreachable in-sandbox SSH endpoint and can skip them when NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1.
    • Skipped sandboxes are recovered during onboarding from the latest validated backup; any uncommitted state since then is not preserved.
  • Bug Fixes
    • Improved failure handling and remediation when SSH transport/unreachability occurs, including clearer guidance and installer/upgrade retry behavior.
  • Documentation
    • Updated nemoclaw and nemohermes backup-all docs to state the default abort behavior and the skip flag’s exact =1 requirement.

Classify a running sandbox whose SSH endpoint is unreachable as such, and let NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1 skip it so the pre-upgrade backup no longer loops the installer. Onboarding then recovers it from its latest validated backup.

Fixes #6188

Signed-off-by: Tinson Lai <tinsonl@nvidia.com>
@coderabbitai

coderabbitai Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Adds SSH-unreachable sandbox handling in backup state, an opt-in skip path for backup-all, installer retry guidance, and matching documentation and tests.

Changes

Unreachable sandbox backup skip

Layer / File(s) Summary
SSH transport failure detection
src/lib/state/sandbox.ts, src/lib/state/sandbox.test.ts
Adds SSH transport failure classification, marks backup results as unreachable, and tests the helper behavior.
backupAll skip logic and reporting
src/lib/actions/maintenance.ts, src/lib/actions/maintenance.test.ts
Adds the environment flag helper, skips unreachable sandboxes when enabled, tracks unreachable-running failures otherwise, and tests both backup-all paths.
Installer retry guidance and docs
scripts/install.sh, test/install-openshell-upgrade-prompt.test.ts, docs/reference/commands.mdx, docs/reference/commands-nemohermes.mdx
Updates pre-upgrade backup retry behavior, fatal error messaging, installer tests, and backup-all documentation for the skip flag and recovery path.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant install.sh
  participant backupAll
  participant backupSandboxState

  install.sh->>backupAll: run pre-upgrade backup
  loop running sandboxes
    backupAll->>backupSandboxState: backupSandboxState()
    backupSandboxState-->>backupAll: backup result
    alt unreachable and skip flag set
      backupAll->>backupAll: skip sandbox
    else unreachable and skip flag unset
      backupAll->>backupAll: record unreachable failure
    end
  end
  backupAll-->>install.sh: success or failure guidance
Loading

Suggested labels: chore
Suggested reviewers: jyaunches, ericksoa, prekshivyas

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main installer change: skipping unreachable running sandboxes during pre-upgrade backup.
Linked Issues check ✅ Passed The PR adds a documented skip flag and clearer remediation for unreachable running sandboxes, which satisfies the issue's requirement to avoid blocking upgrades.
Out of Scope Changes check ✅ Passed The code, docs, and tests all directly support unreachable-sandbox backup handling; no unrelated changes stand out.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/preupgrade-backup-skip-unreachable

Comment @coderabbitai help to get the list of available commands.

@laitingsheng laitingsheng added NV QA Bugs found by the NVIDIA QA Team area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery integration: dcode LangChain Deep Code integration behavior labels Jul 2, 2026
@github-code-quality

github-code-quality Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in the fix/preupgrade-backu... branch is 96%. Coverage data for the main branch is not yet available.

Show a code coverage summary of the most covered files.
File main fix/preupgrade-backu... ad744cf +/-
nemoclaw/src/se...cret-scanner.ts 100%
nemoclaw/src/commands/slash.ts 100%
nemoclaw/src/bl...eprint/state.ts 98%
nemoclaw/src/onboard/config.ts 98%
nemoclaw/src/bl...int/snapshot.ts 97%
nemoclaw/src/blueprint/ssrf.ts 97%
nemoclaw/src/bl...print/runner.ts 95%
nemoclaw/src/co...ration-state.ts 94%
nemoclaw/src/bl...ate-networks.ts 94%
nemoclaw/src/index.ts 94%

TypeScript / code-coverage/cli

The overall coverage in the fix/preupgrade-backu... branch is 69%. Coverage data for the main branch is not yet available.

Show a code coverage summary of the most covered files.
File main fix/preupgrade-backu... ad744cf +/-
src/lib/actions...dbox/rebuild.ts 82%
src/lib/actions...all/run-plan.ts 80%
src/lib/state/o...oard-session.ts 79%
src/lib/shields/index.ts 75%
src/lib/state/sandbox.ts 73%
src/lib/onboard...er-gpu-patch.ts 69%
src/lib/onboard/preflight.ts 69%
src/lib/actions...licy-channel.ts 59%
src/lib/policy/index.ts 56%
src/lib/onboard.ts 20%

Updated July 02, 2026 20:12 UTC
Code Coverage is in Public Preview. Learn more and provide us with your feedback.

@laitingsheng laitingsheng added the bug-fix PR fixes a bug or regression label Jul 2, 2026
@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor (Nemotron Ultra) — Blocked

Merge posture: Do not merge until addressed
Primary next action: Fix PRA-4: isSshTransportFailure overclassifies spawn errors (ENOENT, EACCES) as transport failures; then add or justify PRA-T1.
Open items: 2 required · 9 warnings · 2 suggestions · 8 test follow-ups
Since last review: 4 prior items resolved · 6 still apply · 3 new items found

Action checklist

  • PRA-4 Fix: isSshTransportFailure overclassifies spawn errors (ENOENT, EACCES) as transport failures in src/lib/state/ssh-transport.ts:20
  • PRA-5 Fix: Missing installer integration test for NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1 with unreachable sandbox in test/install-openshell-upgrade-prompt.test.ts:1
  • PRA-1 Resolve or justify: Source-of-truth review needed: src/lib/actions/maintenance.ts:102-115 (skip-unreachable logic)
  • PRA-2 Resolve or justify: Source-of-truth review needed: src/lib/state/ssh-transport.ts:20 (isSshTransportFailure classifier)
  • PRA-3 Resolve or justify: Source-of-truth review needed: src/lib/state/sandbox.ts:1042 (re-export for backwards compatibility)
  • PRA-6 Resolve or justify: shouldSkipUnreachableSandboxBackup exported but only used internally in src/lib/actions/maintenance.ts:20
  • PRA-7 Resolve or justify: Variable name skipUnreachable inconsistent with function shouldSkipUnreachableSandboxBackup in src/lib/actions/maintenance.ts:102
  • PRA-8 Resolve or justify: Test env var manipulation uses manual process.env instead of vi.stubEnv in src/lib/actions/maintenance.test.ts:215
  • PRA-9 Resolve or justify: Missing test for state-file SSH transport failure propagation to BackupResult.unreachable in src/lib/state/ssh-transport.test.ts:1
  • PRA-10 Resolve or justify: Source-of-truth comment incomplete for NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1 fallback in src/lib/actions/maintenance.ts:102
  • PRA-12 Resolve or justify: Data loss warning not prominent in installer error message in scripts/install.sh:1910
  • PRA-T1 Add or justify test follow-up: Runtime validation
  • PRA-T2 Add or justify test follow-up: Runtime validation
  • PRA-T3 Add or justify test follow-up: Runtime validation
  • PRA-T4 Add or justify test follow-up: Runtime validation
  • PRA-T5 Add or justify test follow-up: Missing installer integration test for NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1 with unreachable sandbox
  • PRA-T6 Add or justify test follow-up: Test env var manipulation uses manual process.env instead of vi.stubEnv
  • PRA-T7 Add or justify test follow-up: Missing test for state-file SSH transport failure propagation to BackupResult.unreachable
  • PRA-T8 Add or justify test follow-up: Missing installer integration test for successful skip-flag path
  • PRA-11 In-scope improvement: Unnecessary re-export of isSshTransportFailure from sandbox.ts in src/lib/state/sandbox.ts:1042
  • PRA-13 In-scope improvement: Missing installer integration test for successful skip-flag path in test/install-openshell-upgrade-prompt.test.ts:133

Findings index

ID Severity Category Location Required action
PRA-1 Resolve/justify architecture Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
PRA-2 Resolve/justify architecture Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
PRA-3 Resolve/justify architecture Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
PRA-4 Required security src/lib/state/ssh-transport.ts:20 Check result.error?.code against network transport codes only (ETIMEDOUT, ECONNREFUSED, ECONNRESET, EHOSTUNREACH, ENETUNREACH, EPIPE). Return false for ENOENT, EACCES, ENOTDIR, ENOTEMPTY, etc. Add audit logging of error.code for traceability.
PRA-5 Required tests test/install-openshell-upgrade-prompt.test.ts:1 Add test case in install-openshell-upgrade-prompt.test.ts that sets NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1, mocks an unreachable sandbox during pre-upgrade backup (backupSucceeds: false), and verifies the installer succeeds with RESTORE=1.
PRA-6 Resolve/justify architecture src/lib/actions/maintenance.ts:20 Remove the export keyword. If testing needs direct access, test via backupAll() with mocked env, or move to a test utilities file.
PRA-7 Resolve/justify architecture src/lib/actions/maintenance.ts:102 Rename const skipUnreachable → const shouldSkipUnreachable.
PRA-8 Resolve/justify tests src/lib/actions/maintenance.test.ts:215 Replace with `vi.stubEnv('NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP', '1')` in test setup and `vi.unstubAllEnvs()` in afterEach.
PRA-9 Resolve/justify tests src/lib/state/ssh-transport.test.ts:1 Add test case where SSH transport fails during state file backup (backupStateFile returns {outcome: 'failed', unreachable: true}) and verify BackupResult.unreachable === true.
PRA-10 Resolve/justify architecture src/lib/actions/maintenance.ts:102 Add a source-of-truth comment near the skip-unreachable logic (around line 102-115) documenting: invalid state (running sandbox with dead SSH endpoint), source boundary (installer → backupAll → backupSandboxState), source-fix constraint (SSH unreachable is runtime condition; sandbox gateway crash-loops are external), regression test (maintenance.test.ts unit tests + installer integration test), removal condition (when pre-backup health check added).
PRA-11 Improvement correctness src/lib/state/sandbox.ts:1042 Remove the re-export 'export { isSshTransportFailure };' from sandbox.ts. Update any external consumers (none found in codebase) to import from ./ssh-transport.
PRA-12 Resolve/justify security scripts/install.sh:1910 Add a dedicated 'WARNING: This will lose uncommitted state since the last successful backup.' line before the skip flag guidance, as a separate console.error block (or warn/error call in shell).
PRA-13 Improvement tests test/install-openshell-upgrade-prompt.test.ts:133 Add a test case that sets NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1, mocks backup failure due to unreachable sandbox (backupSucceeds: false), and verifies the installer succeeds with RESTORE=1.

🚨 Required before merge

Address these before merging unless a maintainer explicitly overrides the advisor with rationale.

PRA-4 Required — isSshTransportFailure overclassifies spawn errors (ENOENT, EACCES) as transport failures

  • Location: src/lib/state/ssh-transport.ts:20
  • Category: security
  • Problem: Line 20 returns `true` for ANY `result.error` without checking `error.code`. This treats missing ssh binary (ENOENT), permission denied (EACCES), ENOTDIR, ENOTEMPTY as 'unreachable sandbox', allowing NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1 to silently skip backup for fixable config issues. User loses uncommitted state thinking sandbox is dead.
  • Impact: Silent data loss: config errors (missing ssh, wrong permissions) mark sandbox as 'unreachable', skip flag bypasses backup, user loses uncommitted state without realizing it's a fixable configuration issue.
  • Required action: Check result.error?.code against network transport codes only (ETIMEDOUT, ECONNREFUSED, ECONNRESET, EHOSTUNREACH, ENETUNREACH, EPIPE). Return false for ENOENT, EACCES, ENOTDIR, ENOTEMPTY, etc. Add audit logging of error.code for traceability.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: Run: node -e "const {isSshTransportFailure}=require('./src/lib/state/ssh-transport'); console.log(isSshTransportFailure({error:{code:'ENOENT'}}))" — should be false, currently returns true.
  • Missing regression test: Add unit tests: isSshTransportFailure({error:{code:'ENOENT'}}) === false; isSshTransportFailure({error:{code:'EACCES'}}) === false; isSshTransportFailure({error:{code:'ETIMEDOUT'}}) === true; isSshTransportFailure({error:{code:'ECONNREFUSED'}}) === true.
  • Done when: The required change is committed and verification passes: Run: node -e "const {isSshTransportFailure}=require('./src/lib/state/ssh-transport'); console.log(isSshTransportFailure({error:{code:'ENOENT'}}))" — should be false, currently returns true.
  • Evidence: src/lib/state/ssh-transport.ts:20 returns true for any result.error without checking error.code; prior review PRA-2 flagged this exact issue in sandbox.ts:1039, not fixed in extracted module

PRA-5 Required — Missing installer integration test for NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1 with unreachable sandbox

  • Location: test/install-openshell-upgrade-prompt.test.ts:1
  • Category: tests
  • Problem: Test file covers other upgrade flows but has no test case that sets NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1, mocks an unreachable sandbox during pre-upgrade backup, and verifies the installer continues and sets RESTORE=1. New test at line 252 verifies env var propagation but not the full success path.
  • Impact: No automated verification that the skip flag works end-to-end through the installer flow. Regression risk if installer retry logic or env var handling changes.
  • Required action: Add test case in install-openshell-upgrade-prompt.test.ts that sets NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1, mocks an unreachable sandbox during pre-upgrade backup (backupSucceeds: false), and verifies the installer succeeds with RESTORE=1.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: Read test/install-openshell-upgrade-prompt.test.ts and verify a test case exists for the skip flag with unreachable sandbox that asserts result.status === 0 and result.stdout contains 'RESTORE=1'.
  • Missing regression test: Installer integration test: NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1 with unreachable sandbox → installer continues, RESTORE=1 set.
  • Done when: The required change is committed and verification passes: Read test/install-openshell-upgrade-prompt.test.ts and verify a test case exists for the skip flag with unreachable sandbox that asserts result.status === 0 and result.stdout contains 'RESTORE=1'.
  • Evidence: Prior review PRA-4 identified this gap. Current test file has 10 blocks covering other installer flows but none for the skip flag with unreachable sandbox succeeding end-to-end.
Review findings by urgency: 2 required fixes, 9 items to resolve/justify, 2 in-scope improvements

⚠️ Resolve or justify before merge

Investigate these in the current review; either fix them, explain why they are not applicable, or document the accepted risk.

PRA-1 Resolve/justify — Source-of-truth review needed: src/lib/actions/maintenance.ts:102-115 (skip-unreachable logic)

  • Location: not file-specific
  • Category: architecture
  • Problem: The advisor marked localized patch analysis as needs_followup.
  • Impact: A localized workaround can preserve or hide an invalid state when the source boundary is unclear.
  • Recommended action: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Missing regression test: maintenance.test.ts unit tests (skip with flag, fail with guidance, truth table) + installer integration test (missing PRA-12)
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Evidence: maintenance.ts:102-115 has skip-unreachable logic but no source-of-truth comment; lines 26-50 only covers orphan-manifest catch

PRA-2 Resolve/justify — Source-of-truth review needed: src/lib/state/ssh-transport.ts:20 (isSshTransportFailure classifier)

  • Location: not file-specific
  • Category: architecture
  • Problem: The advisor marked localized patch analysis as needs_followup.
  • Impact: A localized workaround can preserve or hide an invalid state when the source boundary is unclear.
  • Recommended action: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Missing regression test: ssh-transport.test.ts classifier tests (missing ENOENT/EACCES negative tests — PRA-3/PRA-8)
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Evidence: ssh-transport.ts:20 returns true for any result.error without checking error.code; header comment exists but not full 5-element source-of-truth

PRA-3 Resolve/justify — Source-of-truth review needed: src/lib/state/sandbox.ts:1042 (re-export for backwards compatibility)

  • Location: not file-specific
  • Category: architecture
  • Problem: The advisor marked localized patch analysis as needs_followup.
  • Impact: A localized workaround can preserve or hide an invalid state when the source boundary is unclear.
  • Recommended action: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Missing regression test: None needed — grep confirms no external imports from sandbox.ts
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Evidence: sandbox.ts:1042 re-exports; ssh-transport.ts:20 is canonical; no external imports from sandbox.ts found

PRA-6 Resolve/justify — shouldSkipUnreachableSandboxBackup exported but only used internally

  • Location: src/lib/actions/maintenance.ts:20
  • Category: architecture
  • Problem: Function is exported at line 20 but only called internally by backupAll() at line 102. No external consumers found in codebase.
  • Impact: Unnecessary public API surface; testing should go through backupAll() with mocked env or use a test utilities file.
  • Recommended action: Remove the export keyword. If testing needs direct access, test via backupAll() with mocked env, or move to a test utilities file.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: grep -r 'shouldSkipUnreachableSandboxBackup' --include='*.ts' --exclude-dir=node_modules — should only show maintenance.ts and maintenance.test.ts
  • Missing regression test: None needed — removal is a cleanup. Existing maintenance.test.ts tests cover the behavior via backupAll().
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: grep -r 'shouldSkipUnreachableSandboxBackup' --include='*.ts' --exclude-dir=node_modules — should only show maintenance.ts and maintenance.test.ts.
  • Evidence: src/lib/actions/maintenance.ts:20 exports the function; line 102 calls it internally; no other imports found in codebase

PRA-7 Resolve/justify — Variable name skipUnreachable inconsistent with function shouldSkipUnreachableSandboxBackup

  • Location: src/lib/actions/maintenance.ts:102
  • Category: architecture
  • Problem: Line 102 declares `const skipUnreachable = shouldSkipUnreachableSandboxBackup(process.env);` — variable name doesn't match function name pattern.
  • Impact: Reduced readability; inconsistency with function naming convention.
  • Recommended action: Rename const skipUnreachable → const shouldSkipUnreachable.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Read src/lib/actions/maintenance.ts:102 and verify variable name matches function name pattern.
  • Missing regression test: None — purely cosmetic rename.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Read src/lib/actions/maintenance.ts:102 and verify variable name matches function name pattern.
  • Evidence: src/lib/actions/maintenance.ts:102 shows `const skipUnreachable = shouldSkipUnreachableSandboxBackup(process.env);`

PRA-8 Resolve/justify — Test env var manipulation uses manual process.env instead of vi.stubEnv

  • Location: src/lib/actions/maintenance.test.ts:215
  • Category: tests
  • Problem: Lines 215, 228, 251 use manual `process.env.NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP = '1'` and `delete process.env.NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP` instead of `vi.stubEnv`/`vi.unstubAllEnvs`. This can leak env between tests.
  • Impact: Test pollution risk; env var changes may affect other tests running in same process.
  • Recommended action: Replace with `vi.stubEnv('NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP', '1')` in test setup and `vi.unstubAllEnvs()` in afterEach.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Read src/lib/actions/maintenance.test.ts lines 215, 228, 251 and verify vi.stubEnv/vi.unstubAllEnvs usage.
  • Missing regression test: None — test infrastructure improvement. Existing tests should continue passing after migration.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Read src/lib/actions/maintenance.test.ts lines 215, 228, 251 and verify vi.stubEnv/vi.unstubAllEnvs usage.
  • Evidence: src/lib/actions/maintenance.test.ts:215, 228, 251 show manual process.env manipulation

PRA-9 Resolve/justify — Missing test for state-file SSH transport failure propagation to BackupResult.unreachable

  • Location: src/lib/state/ssh-transport.test.ts:1
  • Category: tests
  • Problem: No test covers backupStateFile returning unreachable: true propagating to sandbox-level BackupResult.unreachable. The classifier tests exist but not the integration through backupStateFile → backupSandboxState.
  • Impact: Regression risk: state-file SSH transport failures (ssh-config lookup, dir check, audit, tar download, state files) may not correctly set unreachable flag, breaking skip flag activation for those failure modes.
  • Recommended action: Add test case where SSH transport fails during state file backup (backupStateFile returns {outcome: 'failed', unreachable: true}) and verify BackupResult.unreachable === true.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Read src/lib/state/ssh-transport.test.ts and verify a test exists that mocks backupStateFile transport failure and asserts BackupResult.unreachable.
  • Missing regression test: Unit test: backupStateFile transport failure → BackupResult.unreachable === true.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Read src/lib/state/ssh-transport.test.ts and verify a test exists that mocks backupStateFile transport failure and asserts BackupResult.unreachable.
  • Evidence: ssh-transport.test.ts has 7 classifier tests but no integration test for state-file failure path; sandbox.ts:886 shows backupStateFile returns unreachable from isSshTransportFailure

PRA-10 Resolve/justify — Source-of-truth comment incomplete for NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1 fallback

  • Location: src/lib/actions/maintenance.ts:102
  • Category: architecture
  • Problem: maintenance.ts has detailed source-of-truth comment for orphan-manifest catch (lines 26-50) but skip-unreachable logic (lines 102-115) lacks equivalent 5-element documentation (invalid state, source boundary, source-fix constraint, regression test, removal condition).
  • Impact: Future maintainers cannot evaluate whether this workaround is still needed or can be removed. Risk of workaround becoming permanent technical debt.
  • Recommended action: Add a source-of-truth comment near the skip-unreachable logic (around line 102-115) documenting: invalid state (running sandbox with dead SSH endpoint), source boundary (installer → backupAll → backupSandboxState), source-fix constraint (SSH unreachable is runtime condition; sandbox gateway crash-loops are external), regression test (maintenance.test.ts unit tests + installer integration test), removal condition (when pre-backup health check added).
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Read src/lib/actions/maintenance.ts lines 102-115 and verify a 5-element source-of-truth comment exists.
  • Missing regression test: Documentation only — no test needed. Existing maintenance.test.ts covers unit behavior.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Read src/lib/actions/maintenance.ts lines 102-115 and verify a 5-element source-of-truth comment exists.
  • Evidence: src/lib/actions/maintenance.ts:102-115 has skip-unreachable logic but no source-of-truth comment; lines 26-50 only covers orphan-manifest catch

PRA-12 Resolve/justify — Data loss warning not prominent in installer error message

  • Location: scripts/install.sh:1910
  • Category: security
  • Problem: Error message at line 1910 includes 'any uncommitted state since the last successful backup will be lost' as inline text, not a dedicated console.error block. Could be missed by users scanning error output.
  • Impact: Users may not notice the data loss warning when deciding whether to use the skip flag, leading to unexpected state loss.
  • Recommended action: Add a dedicated 'WARNING: This will lose uncommitted state since the last successful backup.' line before the skip flag guidance, as a separate console.error block (or warn/error call in shell).
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Run installer with failing backup and verify error output contains a prominent standalone WARNING line about data loss before the skip flag guidance.
  • Missing regression test: Installer test verifying error message contains prominent WARNING block (can be added to install-openshell-upgrade-prompt.test.ts).
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Run installer with failing backup and verify error output contains a prominent standalone WARNING line about data loss before the skip flag guidance.
  • Evidence: scripts/install.sh:1910-1915 includes data loss text inline but not as dedicated warning block; prior review PRA-11 requested separate console.error block

💡 In-scope improvements

These are lower-risk, not throwaway. Prefer fixing them in this PR when they are local to changed code; defer only with rationale or a linked follow-up.

PRA-11 Improvement — Unnecessary re-export of isSshTransportFailure from sandbox.ts

  • Location: src/lib/state/sandbox.ts:1042
  • Category: correctness
  • Problem: Line 1042 re-exports `export { isSshTransportFailure };` for backwards compatibility, but no external consumers found in codebase. The function lives in ./ssh-transport now and new code imports from there.
  • Impact: Unnecessary public API surface; adds confusion about canonical import path.
  • Suggested action: Remove the re-export 'export { isSshTransportFailure };' from sandbox.ts. Update any external consumers (none found in codebase) to import from ./ssh-transport.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: grep -r 'isSshTransportFailure' --include='*.ts' --exclude-dir=node_modules | grep -v 'ssh-transport' — should only show imports from ./ssh-transport
  • Missing regression test: None — removal is a cleanup. Existing tests import from ./ssh-transport directly.
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: src/lib/state/sandbox.ts:1042 re-exports; ssh-transport.ts:20 is the canonical definition; no external imports from sandbox.ts found

PRA-13 Improvement — Missing installer integration test for successful skip-flag path

  • Location: test/install-openshell-upgrade-prompt.test.ts:133
  • Category: tests
  • Problem: No test verifies the full success path: NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1 with unreachable sandbox → installer completes with RESTORE=1. New test at line 252 only verifies env var propagation to current-CLI child.
  • Impact: No end-to-end verification that skipped sandboxes are actually recovered during onboarding (RESTORE=1 path).
  • Suggested action: Add a test case that sets NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1, mocks backup failure due to unreachable sandbox (backupSucceeds: false), and verifies the installer succeeds with RESTORE=1.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Read test/install-openshell-upgrade-prompt.test.ts and verify a test case exists for the skip flag with unreachable sandbox that asserts result.status === 0 and result.stdout contains 'RESTORE=1'.
  • Missing regression test: Installer integration test: NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1 with unreachable sandbox → installer succeeds, RESTORE=1 set.
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: Test file has 10 blocks; line 252 tests env var propagation but not full success path; PRA-12 identified this gap
Simplification opportunities: 1 possible cut, net -1 lines possible

These are safe simplification checks only. Do not remove validation, security controls, data-loss prevention, or required tests.

  • PRA-11 delete (src/lib/state/sandbox.ts:1042): export { isSshTransportFailure }; at line 1042
    • Replacement: Use the simpler existing path.
    • Net: -1 lines
    • Safety boundary: No external consumers; all internal uses already import from ./ssh-transport (line 42)
Test follow-ups to resolve or justify

If these cover changed behavior, prefer adding them in this PR; otherwise state why existing coverage is enough or link the follow-up.

  • PRA-T1 Runtime validation — Installer E2E: NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1 with unreachable sandbox → installer completes, RESTORE=1 set. Runtime/sandbox/infrastructure paths need behavioral runtime validation: scripts/install.sh (shell script, has 10 integration tests), src/lib/state/sandbox.ts (SSH/tar integration, unit tests only), src/lib/actions/maintenance.ts (CLI entry point, unit tests with mocks).
  • PRA-T2 Runtime validation — CLI E2E: nemoclaw backup-all with unreachable sandbox + flag → exits 0, skips sandbox. Runtime/sandbox/infrastructure paths need behavioral runtime validation: scripts/install.sh (shell script, has 10 integration tests), src/lib/state/sandbox.ts (SSH/tar integration, unit tests only), src/lib/actions/maintenance.ts (CLI entry point, unit tests with mocks).
  • PRA-T3 Runtime validation — State-file failure: backupStateFile with SSH transport error → BackupResult.unreachable=true. Runtime/sandbox/infrastructure paths need behavioral runtime validation: scripts/install.sh (shell script, has 10 integration tests), src/lib/state/sandbox.ts (SSH/tar integration, unit tests only), src/lib/actions/maintenance.ts (CLI entry point, unit tests with mocks).
  • PRA-T4 Runtime validation — Classifier negative tests: isSshTransportFailure({error:{code:'ENOENT'}}) === false, isSshTransportFailure({error:{code:'EACCES'}}) === false, isSshTransportFailure({error:{code:'ETIMEDOUT'}}) === true, isSshTransportFailure({error:{code:'ECONNREFUSED'}}) === true. Runtime/sandbox/infrastructure paths need behavioral runtime validation: scripts/install.sh (shell script, has 10 integration tests), src/lib/state/sandbox.ts (SSH/tar integration, unit tests only), src/lib/actions/maintenance.ts (CLI entry point, unit tests with mocks).
  • PRA-T5 Missing installer integration test for NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1 with unreachable sandbox — Add test case in install-openshell-upgrade-prompt.test.ts that sets NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1, mocks an unreachable sandbox during pre-upgrade backup (backupSucceeds: false), and verifies the installer succeeds with RESTORE=1.
  • PRA-T6 Test env var manipulation uses manual process.env instead of vi.stubEnv — Replace with `vi.stubEnv('NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP', '1')` in test setup and `vi.unstubAllEnvs()` in afterEach.
  • PRA-T7 Missing test for state-file SSH transport failure propagation to BackupResult.unreachable — Add test case where SSH transport fails during state file backup (backupStateFile returns {outcome: 'failed', unreachable: true}) and verify BackupResult.unreachable === true.
  • PRA-T8 Missing installer integration test for successful skip-flag path — Add a test case that sets NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1, mocks backup failure due to unreachable sandbox (backupSucceeds: false), and verifies the installer succeeds with RESTORE=1.
Since last review details

Current findings, using the urgency labels above:

PRA-1 Resolve/justify — Source-of-truth review needed: src/lib/actions/maintenance.ts:102-115 (skip-unreachable logic)

  • Location: not file-specific
  • Category: architecture
  • Problem: The advisor marked localized patch analysis as needs_followup.
  • Impact: A localized workaround can preserve or hide an invalid state when the source boundary is unclear.
  • Recommended action: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Missing regression test: maintenance.test.ts unit tests (skip with flag, fail with guidance, truth table) + installer integration test (missing PRA-12)
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Evidence: maintenance.ts:102-115 has skip-unreachable logic but no source-of-truth comment; lines 26-50 only covers orphan-manifest catch

PRA-2 Resolve/justify — Source-of-truth review needed: src/lib/state/ssh-transport.ts:20 (isSshTransportFailure classifier)

  • Location: not file-specific
  • Category: architecture
  • Problem: The advisor marked localized patch analysis as needs_followup.
  • Impact: A localized workaround can preserve or hide an invalid state when the source boundary is unclear.
  • Recommended action: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Missing regression test: ssh-transport.test.ts classifier tests (missing ENOENT/EACCES negative tests — PRA-3/PRA-8)
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Evidence: ssh-transport.ts:20 returns true for any result.error without checking error.code; header comment exists but not full 5-element source-of-truth

PRA-3 Resolve/justify — Source-of-truth review needed: src/lib/state/sandbox.ts:1042 (re-export for backwards compatibility)

  • Location: not file-specific
  • Category: architecture
  • Problem: The advisor marked localized patch analysis as needs_followup.
  • Impact: A localized workaround can preserve or hide an invalid state when the source boundary is unclear.
  • Recommended action: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Missing regression test: None needed — grep confirms no external imports from sandbox.ts
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Evidence: sandbox.ts:1042 re-exports; ssh-transport.ts:20 is canonical; no external imports from sandbox.ts found

PRA-4 Required — isSshTransportFailure overclassifies spawn errors (ENOENT, EACCES) as transport failures

  • Location: src/lib/state/ssh-transport.ts:20
  • Category: security
  • Problem: Line 20 returns `true` for ANY `result.error` without checking `error.code`. This treats missing ssh binary (ENOENT), permission denied (EACCES), ENOTDIR, ENOTEMPTY as 'unreachable sandbox', allowing NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1 to silently skip backup for fixable config issues. User loses uncommitted state thinking sandbox is dead.
  • Impact: Silent data loss: config errors (missing ssh, wrong permissions) mark sandbox as 'unreachable', skip flag bypasses backup, user loses uncommitted state without realizing it's a fixable configuration issue.
  • Required action: Check result.error?.code against network transport codes only (ETIMEDOUT, ECONNREFUSED, ECONNRESET, EHOSTUNREACH, ENETUNREACH, EPIPE). Return false for ENOENT, EACCES, ENOTDIR, ENOTEMPTY, etc. Add audit logging of error.code for traceability.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: Run: node -e "const {isSshTransportFailure}=require('./src/lib/state/ssh-transport'); console.log(isSshTransportFailure({error:{code:'ENOENT'}}))" — should be false, currently returns true.
  • Missing regression test: Add unit tests: isSshTransportFailure({error:{code:'ENOENT'}}) === false; isSshTransportFailure({error:{code:'EACCES'}}) === false; isSshTransportFailure({error:{code:'ETIMEDOUT'}}) === true; isSshTransportFailure({error:{code:'ECONNREFUSED'}}) === true.
  • Done when: The required change is committed and verification passes: Run: node -e "const {isSshTransportFailure}=require('./src/lib/state/ssh-transport'); console.log(isSshTransportFailure({error:{code:'ENOENT'}}))" — should be false, currently returns true.
  • Evidence: src/lib/state/ssh-transport.ts:20 returns true for any result.error without checking error.code; prior review PRA-2 flagged this exact issue in sandbox.ts:1039, not fixed in extracted module

PRA-5 Required — Missing installer integration test for NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1 with unreachable sandbox

  • Location: test/install-openshell-upgrade-prompt.test.ts:1
  • Category: tests
  • Problem: Test file covers other upgrade flows but has no test case that sets NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1, mocks an unreachable sandbox during pre-upgrade backup, and verifies the installer continues and sets RESTORE=1. New test at line 252 verifies env var propagation but not the full success path.
  • Impact: No automated verification that the skip flag works end-to-end through the installer flow. Regression risk if installer retry logic or env var handling changes.
  • Required action: Add test case in install-openshell-upgrade-prompt.test.ts that sets NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1, mocks an unreachable sandbox during pre-upgrade backup (backupSucceeds: false), and verifies the installer succeeds with RESTORE=1.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: Read test/install-openshell-upgrade-prompt.test.ts and verify a test case exists for the skip flag with unreachable sandbox that asserts result.status === 0 and result.stdout contains 'RESTORE=1'.
  • Missing regression test: Installer integration test: NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1 with unreachable sandbox → installer continues, RESTORE=1 set.
  • Done when: The required change is committed and verification passes: Read test/install-openshell-upgrade-prompt.test.ts and verify a test case exists for the skip flag with unreachable sandbox that asserts result.status === 0 and result.stdout contains 'RESTORE=1'.
  • Evidence: Prior review PRA-4 identified this gap. Current test file has 10 blocks covering other installer flows but none for the skip flag with unreachable sandbox succeeding end-to-end.

PRA-6 Resolve/justify — shouldSkipUnreachableSandboxBackup exported but only used internally

  • Location: src/lib/actions/maintenance.ts:20
  • Category: architecture
  • Problem: Function is exported at line 20 but only called internally by backupAll() at line 102. No external consumers found in codebase.
  • Impact: Unnecessary public API surface; testing should go through backupAll() with mocked env or use a test utilities file.
  • Recommended action: Remove the export keyword. If testing needs direct access, test via backupAll() with mocked env, or move to a test utilities file.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: grep -r 'shouldSkipUnreachableSandboxBackup' --include='*.ts' --exclude-dir=node_modules — should only show maintenance.ts and maintenance.test.ts
  • Missing regression test: None needed — removal is a cleanup. Existing maintenance.test.ts tests cover the behavior via backupAll().
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: grep -r 'shouldSkipUnreachableSandboxBackup' --include='*.ts' --exclude-dir=node_modules — should only show maintenance.ts and maintenance.test.ts.
  • Evidence: src/lib/actions/maintenance.ts:20 exports the function; line 102 calls it internally; no other imports found in codebase

PRA-7 Resolve/justify — Variable name skipUnreachable inconsistent with function shouldSkipUnreachableSandboxBackup

  • Location: src/lib/actions/maintenance.ts:102
  • Category: architecture
  • Problem: Line 102 declares `const skipUnreachable = shouldSkipUnreachableSandboxBackup(process.env);` — variable name doesn't match function name pattern.
  • Impact: Reduced readability; inconsistency with function naming convention.
  • Recommended action: Rename const skipUnreachable → const shouldSkipUnreachable.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Read src/lib/actions/maintenance.ts:102 and verify variable name matches function name pattern.
  • Missing regression test: None — purely cosmetic rename.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Read src/lib/actions/maintenance.ts:102 and verify variable name matches function name pattern.
  • Evidence: src/lib/actions/maintenance.ts:102 shows `const skipUnreachable = shouldSkipUnreachableSandboxBackup(process.env);`

PRA-8 Resolve/justify — Test env var manipulation uses manual process.env instead of vi.stubEnv

  • Location: src/lib/actions/maintenance.test.ts:215
  • Category: tests
  • Problem: Lines 215, 228, 251 use manual `process.env.NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP = '1'` and `delete process.env.NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP` instead of `vi.stubEnv`/`vi.unstubAllEnvs`. This can leak env between tests.
  • Impact: Test pollution risk; env var changes may affect other tests running in same process.
  • Recommended action: Replace with `vi.stubEnv('NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP', '1')` in test setup and `vi.unstubAllEnvs()` in afterEach.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Read src/lib/actions/maintenance.test.ts lines 215, 228, 251 and verify vi.stubEnv/vi.unstubAllEnvs usage.
  • Missing regression test: None — test infrastructure improvement. Existing tests should continue passing after migration.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Read src/lib/actions/maintenance.test.ts lines 215, 228, 251 and verify vi.stubEnv/vi.unstubAllEnvs usage.
  • Evidence: src/lib/actions/maintenance.test.ts:215, 228, 251 show manual process.env manipulation

PRA-9 Resolve/justify — Missing test for state-file SSH transport failure propagation to BackupResult.unreachable

  • Location: src/lib/state/ssh-transport.test.ts:1
  • Category: tests
  • Problem: No test covers backupStateFile returning unreachable: true propagating to sandbox-level BackupResult.unreachable. The classifier tests exist but not the integration through backupStateFile → backupSandboxState.
  • Impact: Regression risk: state-file SSH transport failures (ssh-config lookup, dir check, audit, tar download, state files) may not correctly set unreachable flag, breaking skip flag activation for those failure modes.
  • Recommended action: Add test case where SSH transport fails during state file backup (backupStateFile returns {outcome: 'failed', unreachable: true}) and verify BackupResult.unreachable === true.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Read src/lib/state/ssh-transport.test.ts and verify a test exists that mocks backupStateFile transport failure and asserts BackupResult.unreachable.
  • Missing regression test: Unit test: backupStateFile transport failure → BackupResult.unreachable === true.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Read src/lib/state/ssh-transport.test.ts and verify a test exists that mocks backupStateFile transport failure and asserts BackupResult.unreachable.
  • Evidence: ssh-transport.test.ts has 7 classifier tests but no integration test for state-file failure path; sandbox.ts:886 shows backupStateFile returns unreachable from isSshTransportFailure

PRA-10 Resolve/justify — Source-of-truth comment incomplete for NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1 fallback

  • Location: src/lib/actions/maintenance.ts:102
  • Category: architecture
  • Problem: maintenance.ts has detailed source-of-truth comment for orphan-manifest catch (lines 26-50) but skip-unreachable logic (lines 102-115) lacks equivalent 5-element documentation (invalid state, source boundary, source-fix constraint, regression test, removal condition).
  • Impact: Future maintainers cannot evaluate whether this workaround is still needed or can be removed. Risk of workaround becoming permanent technical debt.
  • Recommended action: Add a source-of-truth comment near the skip-unreachable logic (around line 102-115) documenting: invalid state (running sandbox with dead SSH endpoint), source boundary (installer → backupAll → backupSandboxState), source-fix constraint (SSH unreachable is runtime condition; sandbox gateway crash-loops are external), regression test (maintenance.test.ts unit tests + installer integration test), removal condition (when pre-backup health check added).
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Read src/lib/actions/maintenance.ts lines 102-115 and verify a 5-element source-of-truth comment exists.
  • Missing regression test: Documentation only — no test needed. Existing maintenance.test.ts covers unit behavior.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Read src/lib/actions/maintenance.ts lines 102-115 and verify a 5-element source-of-truth comment exists.
  • Evidence: src/lib/actions/maintenance.ts:102-115 has skip-unreachable logic but no source-of-truth comment; lines 26-50 only covers orphan-manifest catch

PRA-11 Improvement — Unnecessary re-export of isSshTransportFailure from sandbox.ts

  • Location: src/lib/state/sandbox.ts:1042
  • Category: correctness
  • Problem: Line 1042 re-exports `export { isSshTransportFailure };` for backwards compatibility, but no external consumers found in codebase. The function lives in ./ssh-transport now and new code imports from there.
  • Impact: Unnecessary public API surface; adds confusion about canonical import path.
  • Suggested action: Remove the re-export 'export { isSshTransportFailure };' from sandbox.ts. Update any external consumers (none found in codebase) to import from ./ssh-transport.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: grep -r 'isSshTransportFailure' --include='*.ts' --exclude-dir=node_modules | grep -v 'ssh-transport' — should only show imports from ./ssh-transport
  • Missing regression test: None — removal is a cleanup. Existing tests import from ./ssh-transport directly.
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: src/lib/state/sandbox.ts:1042 re-exports; ssh-transport.ts:20 is the canonical definition; no external imports from sandbox.ts found

PRA-12 Resolve/justify — Data loss warning not prominent in installer error message

  • Location: scripts/install.sh:1910
  • Category: security
  • Problem: Error message at line 1910 includes 'any uncommitted state since the last successful backup will be lost' as inline text, not a dedicated console.error block. Could be missed by users scanning error output.
  • Impact: Users may not notice the data loss warning when deciding whether to use the skip flag, leading to unexpected state loss.
  • Recommended action: Add a dedicated 'WARNING: This will lose uncommitted state since the last successful backup.' line before the skip flag guidance, as a separate console.error block (or warn/error call in shell).
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Run installer with failing backup and verify error output contains a prominent standalone WARNING line about data loss before the skip flag guidance.
  • Missing regression test: Installer test verifying error message contains prominent WARNING block (can be added to install-openshell-upgrade-prompt.test.ts).
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Run installer with failing backup and verify error output contains a prominent standalone WARNING line about data loss before the skip flag guidance.
  • Evidence: scripts/install.sh:1910-1915 includes data loss text inline but not as dedicated warning block; prior review PRA-11 requested separate console.error block

PRA-13 Improvement — Missing installer integration test for successful skip-flag path

  • Location: test/install-openshell-upgrade-prompt.test.ts:133
  • Category: tests
  • Problem: No test verifies the full success path: NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1 with unreachable sandbox → installer completes with RESTORE=1. New test at line 252 only verifies env var propagation to current-CLI child.
  • Impact: No end-to-end verification that skipped sandboxes are actually recovered during onboarding (RESTORE=1 path).
  • Suggested action: Add a test case that sets NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1, mocks backup failure due to unreachable sandbox (backupSucceeds: false), and verifies the installer succeeds with RESTORE=1.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Read test/install-openshell-upgrade-prompt.test.ts and verify a test case exists for the skip flag with unreachable sandbox that asserts result.status === 0 and result.stdout contains 'RESTORE=1'.
  • Missing regression test: Installer integration test: NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1 with unreachable sandbox → installer succeeds, RESTORE=1 set.
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: Test file has 10 blocks; line 252 tests env var propagation but not full success path; PRA-12 identified this gap

Workflow run details

This is an automated, non-binding review; it still expects maintainers and agents to respond to each required or warning item. Treat suggestions as current-PR improvements when they touch changed code; defer only with maintainer rationale or a linked follow-up. A human maintainer must make the final merge decision.

@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

E2E Advisor Recommendation

Required E2E: openshell-gateway-upgrade, sandbox-rebuild
Optional E2E: upgrade-stale-sandbox, state-backup-restore

Dispatch hint: openshell-gateway-upgrade,sandbox-rebuild

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

  • openshell-gateway-upgrade (high): Required because the PR changes install.sh pre-upgrade backup behavior used before OpenShell gateway retirement. This live job installs an old NemoClaw/OpenShell gateway, creates a real sandbox, runs the current installer upgrade path, and verifies the survivor sandbox is restored/reachable.
  • sandbox-rebuild (high): Required because src/lib/state/sandbox.ts changes backupSandboxState unreachable classification used by rebuild/state-preservation flows. This job exercises a real onboarded sandbox, writes durable state, runs nemoclaw <sandbox> rebuild --yes, and verifies state preservation and backup hygiene.

Optional E2E

  • upgrade-stale-sandbox (high): Optional adjacent coverage for stale sandbox upgrade/rebuild behavior. The PR touches installer upgrade guidance and state backup code used by rebuilds, but does not directly change upgrade-sandboxes detection logic.
  • state-backup-restore (high): Optional adjacent confidence for real backup/restore state preservation. It is less directly targeted than sandbox-rebuild because the changed backupSandboxState path is primarily used by rebuild/backup-all, not the legacy backup-workspace script boundary.

New E2E recommendations

  • installer pre-upgrade backup-all unreachable-sandbox skip (high): No existing live E2E appears to intentionally create a registered running sandbox whose in-sandbox SSH endpoint is unreachable, seed a validated previous backup, run the installer with NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1, and verify the OpenShell upgrade continues and the sandbox is restored from the latest backup. The new behavior is covered by unit/script tests but not at the real installer/OpenShell/sandbox boundary.
    • Suggested test: Add a live E2E target for backup-all -- skip unreachable during installer OpenShell upgrade that breaks only the sandbox SSH endpoint, runs the current installer with the exact env opt-in, asserts backup-all skips that sandbox rather than aborting, and verifies post-upgrade restore behavior and user-visible data-loss warning.

Dispatch hint

  • Workflow: e2e.yaml
  • jobs input: openshell-gateway-upgrade,sandbox-rebuild

@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

E2E Target Recommendation

Required E2E targets: openshell-gateway-upgrade, rebuild-openclaw
Optional E2E targets: snapshot-commands

Dispatch required E2E targets:

  • gh workflow run e2e.yaml --ref <pr-head-ref> --field jobs=openshell-gateway-upgrade
  • gh workflow run e2e.yaml --ref <pr-head-ref> --field jobs=rebuild-openclaw

Workflow run

Full E2E target advisor summary

E2E Target Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E targets

  • openshell-gateway-upgrade: Installer pre-upgrade backup retry and backup-all skip semantics changed. The openshell-gateway-upgrade job is the discrete live E2E path for the old-installer/current-installer gateway upgrade boundary, pre-upgrade backup, and survivor restore checks.
    • Dispatch: gh workflow run e2e.yaml --ref <pr-head-ref> --field jobs=openshell-gateway-upgrade
  • rebuild-openclaw: Core sandbox backup result handling and SSH transport failure classification changed in src/lib/state/sandbox.ts. The rebuild-openclaw job exercises real backup/restore through the OpenClaw rebuild lifecycle using the affected backupSandboxState path.
    • Dispatch: gh workflow run e2e.yaml --ref <pr-head-ref> --field jobs=rebuild-openclaw

Optional E2E targets

  • snapshot-commands: Adjacent coverage for the same sandbox state backup machinery through snapshot create/restore CLI flows; useful if extra confidence is needed beyond rebuild and installer-upgrade coverage.
    • Dispatch: gh workflow run e2e.yaml --ref <pr-head-ref> --field jobs=snapshot-commands

Relevant changed files

  • scripts/install.sh
  • src/lib/actions/maintenance.ts
  • src/lib/state/sandbox.ts
  • src/lib/state/ssh-transport.ts

@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — Blocked

Merge posture: Do not merge until addressed
Primary next action: Fix PRA-3: Skipped Ready/Running unreachable sandboxes are not scheduled for validated recovery; then add or justify PRA-T1.
Open items: 1 required · 5 warnings · 0 suggestions · 8 test follow-ups
Since last review: 1 prior item resolved · 5 still apply · 0 new items found

Action checklist

  • PRA-3 Fix: Skipped Ready/Running unreachable sandboxes are not scheduled for validated recovery in src/lib/actions/upgrade-sandboxes.ts:201
  • PRA-1 Resolve or justify: Source-of-truth review needed: Pre-upgrade skip for running but SSH-unreachable sandbox backups
  • PRA-2 Resolve or justify: Source-of-truth review needed: `getSshConfig()` failure classification for backup unreachability
  • PRA-4 Resolve or justify: `ssh-config` local failures are classified as skippable unreachable backups in src/lib/state/sandbox.ts:1157
  • PRA-5 Resolve or justify: State-file SSH transport propagation is not tested through `backupSandboxState()` in src/lib/state/sandbox.ts:1384
  • PRA-6 Resolve or justify: Security-critical sandbox state hotspot still grows after this change in src/lib/state/sandbox.ts:1040
  • PRA-T1 Add or justify test follow-up: Runtime validation
  • PRA-T2 Add or justify test follow-up: Runtime validation
  • PRA-T3 Add or justify test follow-up: Runtime validation
  • PRA-T4 Add or justify test follow-up: Runtime validation
  • PRA-T5 Add or justify test follow-up: Runtime validation
  • PRA-T6 Add or justify test follow-up: State-file SSH transport propagation is not tested through `backupSandboxState()`
  • PRA-T7 Add or justify test follow-up: Acceptance clause
  • PRA-T8 Add or justify test follow-up: Acceptance clause

Findings index

ID Severity Category Location Required action
PRA-1 Resolve/justify architecture Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
PRA-2 Resolve/justify architecture Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
PRA-3 Required acceptance src/lib/actions/upgrade-sandboxes.ts:201 Either make skipped unreachable sandbox names an explicit handoff into the installer/onboarding recovery path and validate each latest backup with `validateRebuildRecoveryManifest()` before any destructive recreate, or narrow `scripts/install.sh`, `backupAll()` guidance, and both docs pages so they do not promise automatic recovery for skipped Ready/Running sandboxes. Include the source-of-truth note at the skip/handoff boundary: invalid state, source boundary, why the source cannot be fixed here, regression proof, and removal condition.
PRA-4 Resolve/justify security src/lib/state/sandbox.ts:1157 Preserve the cause from `getSshConfig()` or the underlying `captureSandboxSshConfigCommand()` result so only true transport-level sandbox reachability failures set `BackupResult.unreachable`. Keep missing OpenShell binary, local command errors, and other non-transport failures non-skippable unless a test proves an earlier preflight makes them impossible here.
PRA-5 Resolve/justify tests src/lib/state/sandbox.ts:1384 Add focused `backupSandboxState()` tests that reach the state-file backup loop and assert the returned `BackupResult.unreachable` contract for both transport and non-transport state-file failures.
PRA-6 Resolve/justify architecture src/lib/state/sandbox.ts:1040 Offset the remaining growth in this PR by extracting a cohesive helper around SSH backup result shaping or state-file backup result propagation, or otherwise shrink nearby duplicated result-construction code. Do not weaken existing symlink/hardlink audit, tar-entry validation, backup sanitization, recovery-manifest validation, or exact env-flag handling.

🚨 Required before merge

Address these before merging unless a maintainer explicitly overrides the advisor with rationale.

PRA-3 Required — Skipped Ready/Running unreachable sandboxes are not scheduled for validated recovery

  • Location: src/lib/actions/upgrade-sandboxes.ts:201
  • Category: acceptance
  • Problem: `backupAll()` can now skip a running sandbox when `BackupResult.unreachable` is set and `NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1`, and the installer then exports `NEMOCLAW_RESTORE_LATEST_BACKUP_ON_RECREATE=1`. However, the recovery consumer still builds `backupRecoveryAssessments` only from `nonReadyLiveNames`, and `nonReadyLiveNames` explicitly filters out `Ready` and `Running`. The changed docs, CLI guidance, and installer error text promise that skipped running sandboxes recover from their latest validated backup during post-upgrade onboarding, but the implementation does not hand those skipped Ready/Running names to `prepareBackupRecovery()` or `rebuildSandbox(..., { recoveryManifest })`.
  • Impact: A user can opt into skipping a fresh backup for a running but SSH-dead sandbox, pass the pre-upgrade backup gate, and believe a validated latest-backup recovery is scheduled. For the Ready/Running failure mode described by the linked issue, the sandbox is excluded from prepared recovery, weakening the data-loss safety boundary for installer-driven OpenShell/sandbox upgrades.
  • Required action: Either make skipped unreachable sandbox names an explicit handoff into the installer/onboarding recovery path and validate each latest backup with `validateRebuildRecoveryManifest()` before any destructive recreate, or narrow `scripts/install.sh`, `backupAll()` guidance, and both docs pages so they do not promise automatic recovery for skipped Ready/Running sandboxes. Include the source-of-truth note at the skip/handoff boundary: invalid state, source boundary, why the source cannot be fixed here, regression proof, and removal condition.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: Read `src/lib/actions/maintenance.ts` around the `result.unreachable` branch and confirm it logs `Skipped` and continues; then read `src/lib/actions/upgrade-sandboxes.ts` around `nonReadyLiveNames` and `backupRecoveryAssessments` and confirm recovery is built from `sandboxes.filter((sandbox) => nonReadyLiveNames.has(sandbox.name))` while `nonReadyLiveNames` filters out `Ready` and `Running`.
  • Missing regression test: Add an installer/upgrade-flow test where a Ready or Running sandbox is skipped by `NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1`, has a latest backup, and is passed to `rebuildSandbox(..., { recoveryManifest })` only after `validateRebuildRecoveryManifest()` succeeds. Add negative cases where no latest backup or failed manifest validation aborts before destructive recreate. If the intended behavior is only to skip the backup, update tests to assert the docs/error text no longer claim post-upgrade recovery for skipped Ready/Running sandboxes.
  • Done when: The required change is committed and verification passes: Read `src/lib/actions/maintenance.ts` around the `result.unreachable` branch and confirm it logs `Skipped` and continues; then read `src/lib/actions/upgrade-sandboxes.ts` around `nonReadyLiveNames` and `backupRecoveryAssessments` and confirm recovery is built from `sandboxes.filter((sandbox) => nonReadyLiveNames.has(sandbox.name))` while `nonReadyLiveNames` filters out `Ready` and `Running`.
  • Evidence: `maintenance.ts` logs `Skipped '${sb.name}' (running but SSH-unreachable...)` and continues. `install.sh` exports `NEMOCLAW_RESTORE_LATEST_BACKUP_ON_RECREATE=1` after backup success. `upgrade-sandboxes.ts` excludes `Ready`/`Running` from `nonReadyLiveNames` and only maps those non-Ready names through `prepareBackupRecovery()`. The docs and installer guidance state that skipped sandboxes recover from their latest validated backup after the upgrade.
Review findings by urgency: 1 required fix, 5 items to resolve/justify, 0 in-scope improvements

⚠️ Resolve or justify before merge

Investigate these in the current review; either fix them, explain why they are not applicable, or document the accepted risk.

PRA-1 Resolve/justify — Source-of-truth review needed: Pre-upgrade skip for running but SSH-unreachable sandbox backups

  • Location: not file-specific
  • Category: architecture
  • Problem: The advisor marked localized patch analysis as needs_followup.
  • Impact: A localized workaround can preserve or hide an invalid state when the source boundary is unclear.
  • Recommended action: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Missing regression test: Partial only: tests cover exact env parsing, mocked `BackupResult.unreachable`, installer guidance, and the SSH classifier. No test proves skipped Ready/Running sandboxes are validated and recovered, and no test proves only true transport failures are skippable.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Evidence: `maintenance.ts` skips unreachable results under the env flag; `install.sh` and docs promise latest validated backup recovery; `upgrade-sandboxes.ts` still limits prepared recovery to non-Ready live names. This is represented by the recovery-handoff finding.

PRA-2 Resolve/justify — Source-of-truth review needed: `getSshConfig()` failure classification for backup unreachability

  • Location: not file-specific
  • Category: architecture
  • Problem: The advisor marked localized patch analysis as needs_followup.
  • Impact: A localized workaround can preserve or hide an invalid state when the source boundary is unclear.
  • Recommended action: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Missing regression test: Missing: no `backupSandboxState()` test distinguishes local `resolveOpenshell()` or ssh-config non-transport failures from true transport failures.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Evidence: `getSshConfig()` returns `null` for missing OpenShell and any non-zero ssh-config status, and `backupSandboxState()` maps all `null` results to `unreachable: true`. This is represented by the security finding.

PRA-4 Resolve/justify — `ssh-config` local failures are classified as skippable unreachable backups

  • Location: src/lib/state/sandbox.ts:1157
  • Category: security
  • Problem: `getSshConfig()` returns only `string | null`: it returns `null` when `resolveOpenshell()` fails and for any non-zero `openshell sandbox ssh-config` result. The new `backupSandboxState()` branch treats every `null` as `unreachable: true`. With `NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1`, local OpenShell resolution/configuration failures can be masked as an in-sandbox SSH endpoint failure.
  • Impact: A local tool/config failure could bypass the pre-upgrade backup gate under the new skip flag even though the documented escape hatch is for true sandbox SSH transport failures. In the installer path this compounds the recovery-handoff gap and can increase data-loss risk.
  • Recommended action: Preserve the cause from `getSshConfig()` or the underlying `captureSandboxSshConfigCommand()` result so only true transport-level sandbox reachability failures set `BackupResult.unreachable`. Keep missing OpenShell binary, local command errors, and other non-transport failures non-skippable unless a test proves an earlier preflight makes them impossible here.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Read `src/lib/state/sandbox.ts:getSshConfig()` and confirm it collapses `resolveOpenshell()` failure and all `ssh-config` non-zero statuses to `null`; then read the `if (!sshConfig)` branch in `backupSandboxState()` and confirm it returns `unreachable: true` without checking the underlying cause.
  • Missing regression test: Add `backupSandboxState()` tests with mocked `resolveOpenshell()` and `captureSandboxSshConfigCommand()` boundaries proving that missing/local OpenShell failures are non-skippable, while a true transport-level `ssh-config` failure is the only `!sshConfig` path that sets `unreachable`.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Read `src/lib/state/sandbox.ts:getSshConfig()` and confirm it collapses `resolveOpenshell()` failure and all `ssh-config` non-zero statuses to `null`; then read the `if (!sshConfig)` branch in `backupSandboxState()` and confirm it returns `unreachable: true` without checking the underlying cause.
  • Evidence: `getSshConfig()` returns `null` on missing OpenShell and any non-zero ssh-config command. The new failure return at `backupSandboxState()` sets `unreachable: true` for all `!sshConfig` cases.

PRA-5 Resolve/justify — State-file SSH transport propagation is not tested through `backupSandboxState()`

  • Location: src/lib/state/sandbox.ts:1384
  • Category: tests
  • Problem: The implementation now promotes state-file SSH transport failures through `backupStateFile()` into the sandbox-level `BackupResult.unreachable` flag, but the added tests only cover the pure `isSshTransportFailure()` classifier and mocked `backupAll()` behavior. They do not drive `backupSandboxState()` through the state-file loop with mocked registry/agent/SSH boundaries.
  • Impact: A regression in the caller/callee contract between `backupStateFile()` and `backupSandboxState()` could silently stop the skip flag from activating for state-file transport failures, or could mark reachable remote validation failures as skippable unreachable backups.
  • Recommended action: Add focused `backupSandboxState()` tests that reach the state-file backup loop and assert the returned `BackupResult.unreachable` contract for both transport and non-transport state-file failures.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Read `src/lib/state/ssh-transport.test.ts` and `src/lib/actions/maintenance.test.ts`; confirm they test the classifier and a mocked `BackupResult.unreachable`, not a full `backupSandboxState()` state-file failure path.
  • Missing regression test: Add one test where a state-file SSH spawn returns status 255 or null and `backupSandboxState()` returns `success: false` with `unreachable: true`; add a negative test where a reachable remote exit such as status 10/11 fails the state-file backup but leaves `unreachable` unset.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Read `src/lib/state/ssh-transport.test.ts` and `src/lib/actions/maintenance.test.ts`; confirm they test the classifier and a mocked `BackupResult.unreachable`, not a full `backupSandboxState()` state-file failure path.
  • Evidence: `backupStateFile()` returns `{ outcome: 'failed', unreachable: isSshTransportFailure(result) }`, and `backupSandboxState()` copies that into the top-level `unreachable` flag. Existing changed tests do not exercise that integration path.

PRA-6 Resolve/justify — Security-critical sandbox state hotspot still grows after this change

  • Location: src/lib/state/sandbox.ts:1040
  • Category: architecture
  • Problem: `src/lib/state/sandbox.ts` remains a large security-critical backup/restore module and grows further in this PR. The new helper extraction to `ssh-transport.ts` is useful, but the changed state-file result shaping and repeated `BackupResult` construction still add complexity to the monolith.
  • Impact: Continued growth in this file makes future changes to symlink/hardlink audit, tar extraction, credential sanitization, manifest validation, and SSH failure classification harder to review safely.
  • Recommended action: Offset the remaining growth in this PR by extracting a cohesive helper around SSH backup result shaping or state-file backup result propagation, or otherwise shrink nearby duplicated result-construction code. Do not weaken existing symlink/hardlink audit, tar-entry validation, backup sanitization, recovery-manifest validation, or exact env-flag handling.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Compare the changed `backupStateFile()` and `backupSandboxState()` failure-result construction in `src/lib/state/sandbox.ts` and identify whether a small helper can remove duplicate success/failure object shaping without changing behavior.
  • Missing regression test: Existing tests around backup classification, tar safety, and state-file restore should remain green after refactor; if a helper is extracted, keep or add focused tests for the transport vs non-transport result contract.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Compare the changed `backupStateFile()` and `backupSandboxState()` failure-result construction in `src/lib/state/sandbox.ts` and identify whether a small helper can remove duplicate success/failure object shaping without changing behavior.
  • Evidence: The drift context reports `src/lib/state/sandbox.ts` grew from 1832 to 1869 lines. This file is on the sandbox lifecycle/data-loss/security boundary.

💡 In-scope improvements

These are lower-risk, not throwaway. Prefer fixing them in this PR when they are local to changed code; defer only with rationale or a linked follow-up.

  • None.
Simplification opportunities: 1 possible cut

These are safe simplification checks only. Do not remove validation, security controls, data-loss prevention, or required tests.

  • PRA-6 shrink (src/lib/state/sandbox.ts:1040): Duplicated inline `BackupResult` and state-file failure-result construction in `backupSandboxState()`.
    • Replacement: A small local helper that constructs failed backup results while preserving `unreachable`, `error`, failed dirs/files, and manifest fields.
    • Safety boundary: Do not remove or weaken symlink/hardlink audit, tar-entry validation, credential sanitization, recovery-manifest validation, or exact env-flag parsing.
Test follow-ups to resolve or justify

If these cover changed behavior, prefer adding them in this PR; otherwise state why existing coverage is enough or link the follow-up.

  • PRA-T1 Runtime validation — Add an installer/upgrade-flow test where a Ready or Running sandbox skipped by `NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1` is later passed to `rebuildSandbox(..., { recoveryManifest })` only after `validateRebuildRecoveryManifest()` succeeds.. The changed behavior crosses installer, OpenShell, sandbox SSH, backup, and recovery boundaries. Unit tests cover several local branches, but behavioral confidence still depends on integrated recovery handoff and failure-classification coverage.
  • PRA-T2 Runtime validation — Add negative recovery tests where a skipped Ready/Running sandbox has no latest backup or a failed manifest validation, and assert the flow aborts before destructive recreate or that docs/guidance no longer promise recovery.. The changed behavior crosses installer, OpenShell, sandbox SSH, backup, and recovery boundaries. Unit tests cover several local branches, but behavioral confidence still depends on integrated recovery handoff and failure-classification coverage.
  • PRA-T3 Runtime validation — Add `backupSandboxState()` tests proving local OpenShell resolution and local `ssh-config` command failures do not set `unreachable`, while true transport-level SSH config failures do.. The changed behavior crosses installer, OpenShell, sandbox SSH, backup, and recovery boundaries. Unit tests cover several local branches, but behavioral confidence still depends on integrated recovery handoff and failure-classification coverage.
  • PRA-T4 Runtime validation — Add `backupSandboxState()` state-file loop tests proving status 255/null state-file SSH failures promote top-level `unreachable`, while reachable remote validation exits remain non-skippable.. The changed behavior crosses installer, OpenShell, sandbox SSH, backup, and recovery boundaries. Unit tests cover several local branches, but behavioral confidence still depends on integrated recovery handoff and failure-classification coverage.
  • PRA-T5 Runtime validation — Add or identify targeted runtime validation for a real or high-fidelity OpenShell SSH banner-timeout path where backup retry propagates `NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1` to the current CLI.. The changed behavior crosses installer, OpenShell, sandbox SSH, backup, and recovery boundaries. Unit tests cover several local branches, but behavioral confidence still depends on integrated recovery handoff and failure-classification coverage.
  • PRA-T6 State-file SSH transport propagation is not tested through `backupSandboxState()` — Add focused `backupSandboxState()` tests that reach the state-file backup loop and assert the returned `BackupResult.unreachable` contract for both transport and non-transport state-file failures.
  • PRA-T7 Acceptance clause — Upgrading NemoClaw via `curl | bash` runs a pre-upgrade `backup-all` before the OpenShell upgrade. If any registered sandbox that is *running* (present in the gateway's Ready list) cannot be backed up because its in-sandbox SSH endpoint does not answer (SSH `Connection timed out during banner exchange`, ~10s), the whole installer aborts with exit 1. — add test evidence or identify existing coverage. `backupSandboxState()` now sets `BackupResult.unreachable` for transport-like backup failures and `backupAll()` recognizes it. The default still aborts, but now prints guidance; the skip path is opt-in.
  • PRA-T8 Acceptance clause — This gate runs **before** the sandbox-recovery logic added in PR fix(installer): recover sandboxes before onboarding #6132 (`upgrade-sandboxes --auto`, which recovers non-Ready sandboxes from their latest validated backup), so fix(installer): recover sandboxes before onboarding #6132's recovery never runs for exactly the kind of degraded sandbox it was designed to handle. — add test evidence or identify existing coverage. The installer can now pass the backup gate when the skip flag is set, but `upgrade-sandboxes.ts` still only prepares recovery for `nonReadyLiveNames`, excluding `Ready` and `Running`; the running skipped sandbox is not routed to validated recovery.
Since last review details

Current findings, using the urgency labels above:

PRA-1 Resolve/justify — Source-of-truth review needed: Pre-upgrade skip for running but SSH-unreachable sandbox backups

  • Location: not file-specific
  • Category: architecture
  • Problem: The advisor marked localized patch analysis as needs_followup.
  • Impact: A localized workaround can preserve or hide an invalid state when the source boundary is unclear.
  • Recommended action: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Missing regression test: Partial only: tests cover exact env parsing, mocked `BackupResult.unreachable`, installer guidance, and the SSH classifier. No test proves skipped Ready/Running sandboxes are validated and recovered, and no test proves only true transport failures are skippable.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Evidence: `maintenance.ts` skips unreachable results under the env flag; `install.sh` and docs promise latest validated backup recovery; `upgrade-sandboxes.ts` still limits prepared recovery to non-Ready live names. This is represented by the recovery-handoff finding.

PRA-2 Resolve/justify — Source-of-truth review needed: `getSshConfig()` failure classification for backup unreachability

  • Location: not file-specific
  • Category: architecture
  • Problem: The advisor marked localized patch analysis as needs_followup.
  • Impact: A localized workaround can preserve or hide an invalid state when the source boundary is unclear.
  • Recommended action: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Missing regression test: Missing: no `backupSandboxState()` test distinguishes local `resolveOpenshell()` or ssh-config non-transport failures from true transport failures.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Evidence: `getSshConfig()` returns `null` for missing OpenShell and any non-zero ssh-config status, and `backupSandboxState()` maps all `null` results to `unreachable: true`. This is represented by the security finding.

PRA-3 Required — Skipped Ready/Running unreachable sandboxes are not scheduled for validated recovery

  • Location: src/lib/actions/upgrade-sandboxes.ts:201
  • Category: acceptance
  • Problem: `backupAll()` can now skip a running sandbox when `BackupResult.unreachable` is set and `NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1`, and the installer then exports `NEMOCLAW_RESTORE_LATEST_BACKUP_ON_RECREATE=1`. However, the recovery consumer still builds `backupRecoveryAssessments` only from `nonReadyLiveNames`, and `nonReadyLiveNames` explicitly filters out `Ready` and `Running`. The changed docs, CLI guidance, and installer error text promise that skipped running sandboxes recover from their latest validated backup during post-upgrade onboarding, but the implementation does not hand those skipped Ready/Running names to `prepareBackupRecovery()` or `rebuildSandbox(..., { recoveryManifest })`.
  • Impact: A user can opt into skipping a fresh backup for a running but SSH-dead sandbox, pass the pre-upgrade backup gate, and believe a validated latest-backup recovery is scheduled. For the Ready/Running failure mode described by the linked issue, the sandbox is excluded from prepared recovery, weakening the data-loss safety boundary for installer-driven OpenShell/sandbox upgrades.
  • Required action: Either make skipped unreachable sandbox names an explicit handoff into the installer/onboarding recovery path and validate each latest backup with `validateRebuildRecoveryManifest()` before any destructive recreate, or narrow `scripts/install.sh`, `backupAll()` guidance, and both docs pages so they do not promise automatic recovery for skipped Ready/Running sandboxes. Include the source-of-truth note at the skip/handoff boundary: invalid state, source boundary, why the source cannot be fixed here, regression proof, and removal condition.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: Read `src/lib/actions/maintenance.ts` around the `result.unreachable` branch and confirm it logs `Skipped` and continues; then read `src/lib/actions/upgrade-sandboxes.ts` around `nonReadyLiveNames` and `backupRecoveryAssessments` and confirm recovery is built from `sandboxes.filter((sandbox) => nonReadyLiveNames.has(sandbox.name))` while `nonReadyLiveNames` filters out `Ready` and `Running`.
  • Missing regression test: Add an installer/upgrade-flow test where a Ready or Running sandbox is skipped by `NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1`, has a latest backup, and is passed to `rebuildSandbox(..., { recoveryManifest })` only after `validateRebuildRecoveryManifest()` succeeds. Add negative cases where no latest backup or failed manifest validation aborts before destructive recreate. If the intended behavior is only to skip the backup, update tests to assert the docs/error text no longer claim post-upgrade recovery for skipped Ready/Running sandboxes.
  • Done when: The required change is committed and verification passes: Read `src/lib/actions/maintenance.ts` around the `result.unreachable` branch and confirm it logs `Skipped` and continues; then read `src/lib/actions/upgrade-sandboxes.ts` around `nonReadyLiveNames` and `backupRecoveryAssessments` and confirm recovery is built from `sandboxes.filter((sandbox) => nonReadyLiveNames.has(sandbox.name))` while `nonReadyLiveNames` filters out `Ready` and `Running`.
  • Evidence: `maintenance.ts` logs `Skipped '${sb.name}' (running but SSH-unreachable...)` and continues. `install.sh` exports `NEMOCLAW_RESTORE_LATEST_BACKUP_ON_RECREATE=1` after backup success. `upgrade-sandboxes.ts` excludes `Ready`/`Running` from `nonReadyLiveNames` and only maps those non-Ready names through `prepareBackupRecovery()`. The docs and installer guidance state that skipped sandboxes recover from their latest validated backup after the upgrade.

PRA-4 Resolve/justify — `ssh-config` local failures are classified as skippable unreachable backups

  • Location: src/lib/state/sandbox.ts:1157
  • Category: security
  • Problem: `getSshConfig()` returns only `string | null`: it returns `null` when `resolveOpenshell()` fails and for any non-zero `openshell sandbox ssh-config` result. The new `backupSandboxState()` branch treats every `null` as `unreachable: true`. With `NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1`, local OpenShell resolution/configuration failures can be masked as an in-sandbox SSH endpoint failure.
  • Impact: A local tool/config failure could bypass the pre-upgrade backup gate under the new skip flag even though the documented escape hatch is for true sandbox SSH transport failures. In the installer path this compounds the recovery-handoff gap and can increase data-loss risk.
  • Recommended action: Preserve the cause from `getSshConfig()` or the underlying `captureSandboxSshConfigCommand()` result so only true transport-level sandbox reachability failures set `BackupResult.unreachable`. Keep missing OpenShell binary, local command errors, and other non-transport failures non-skippable unless a test proves an earlier preflight makes them impossible here.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Read `src/lib/state/sandbox.ts:getSshConfig()` and confirm it collapses `resolveOpenshell()` failure and all `ssh-config` non-zero statuses to `null`; then read the `if (!sshConfig)` branch in `backupSandboxState()` and confirm it returns `unreachable: true` without checking the underlying cause.
  • Missing regression test: Add `backupSandboxState()` tests with mocked `resolveOpenshell()` and `captureSandboxSshConfigCommand()` boundaries proving that missing/local OpenShell failures are non-skippable, while a true transport-level `ssh-config` failure is the only `!sshConfig` path that sets `unreachable`.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Read `src/lib/state/sandbox.ts:getSshConfig()` and confirm it collapses `resolveOpenshell()` failure and all `ssh-config` non-zero statuses to `null`; then read the `if (!sshConfig)` branch in `backupSandboxState()` and confirm it returns `unreachable: true` without checking the underlying cause.
  • Evidence: `getSshConfig()` returns `null` on missing OpenShell and any non-zero ssh-config command. The new failure return at `backupSandboxState()` sets `unreachable: true` for all `!sshConfig` cases.

PRA-5 Resolve/justify — State-file SSH transport propagation is not tested through `backupSandboxState()`

  • Location: src/lib/state/sandbox.ts:1384
  • Category: tests
  • Problem: The implementation now promotes state-file SSH transport failures through `backupStateFile()` into the sandbox-level `BackupResult.unreachable` flag, but the added tests only cover the pure `isSshTransportFailure()` classifier and mocked `backupAll()` behavior. They do not drive `backupSandboxState()` through the state-file loop with mocked registry/agent/SSH boundaries.
  • Impact: A regression in the caller/callee contract between `backupStateFile()` and `backupSandboxState()` could silently stop the skip flag from activating for state-file transport failures, or could mark reachable remote validation failures as skippable unreachable backups.
  • Recommended action: Add focused `backupSandboxState()` tests that reach the state-file backup loop and assert the returned `BackupResult.unreachable` contract for both transport and non-transport state-file failures.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Read `src/lib/state/ssh-transport.test.ts` and `src/lib/actions/maintenance.test.ts`; confirm they test the classifier and a mocked `BackupResult.unreachable`, not a full `backupSandboxState()` state-file failure path.
  • Missing regression test: Add one test where a state-file SSH spawn returns status 255 or null and `backupSandboxState()` returns `success: false` with `unreachable: true`; add a negative test where a reachable remote exit such as status 10/11 fails the state-file backup but leaves `unreachable` unset.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Read `src/lib/state/ssh-transport.test.ts` and `src/lib/actions/maintenance.test.ts`; confirm they test the classifier and a mocked `BackupResult.unreachable`, not a full `backupSandboxState()` state-file failure path.
  • Evidence: `backupStateFile()` returns `{ outcome: 'failed', unreachable: isSshTransportFailure(result) }`, and `backupSandboxState()` copies that into the top-level `unreachable` flag. Existing changed tests do not exercise that integration path.

PRA-6 Resolve/justify — Security-critical sandbox state hotspot still grows after this change

  • Location: src/lib/state/sandbox.ts:1040
  • Category: architecture
  • Problem: `src/lib/state/sandbox.ts` remains a large security-critical backup/restore module and grows further in this PR. The new helper extraction to `ssh-transport.ts` is useful, but the changed state-file result shaping and repeated `BackupResult` construction still add complexity to the monolith.
  • Impact: Continued growth in this file makes future changes to symlink/hardlink audit, tar extraction, credential sanitization, manifest validation, and SSH failure classification harder to review safely.
  • Recommended action: Offset the remaining growth in this PR by extracting a cohesive helper around SSH backup result shaping or state-file backup result propagation, or otherwise shrink nearby duplicated result-construction code. Do not weaken existing symlink/hardlink audit, tar-entry validation, backup sanitization, recovery-manifest validation, or exact env-flag handling.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Compare the changed `backupStateFile()` and `backupSandboxState()` failure-result construction in `src/lib/state/sandbox.ts` and identify whether a small helper can remove duplicate success/failure object shaping without changing behavior.
  • Missing regression test: Existing tests around backup classification, tar safety, and state-file restore should remain green after refactor; if a helper is extracted, keep or add focused tests for the transport vs non-transport result contract.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Compare the changed `backupStateFile()` and `backupSandboxState()` failure-result construction in `src/lib/state/sandbox.ts` and identify whether a small helper can remove duplicate success/failure object shaping without changing behavior.
  • Evidence: The drift context reports `src/lib/state/sandbox.ts` grew from 1832 to 1869 lines. This file is on the sandbox lifecycle/data-loss/security boundary.

Workflow run details

This is an automated, non-binding review; it still expects maintainers and agents to respond to each required or warning item. Treat suggestions as current-PR improvements when they touch changed code; defer only with maintainer rationale or a linked follow-up. A human maintainer must make the final merge decision.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (3)
src/lib/state/sandbox.ts (1)

124-124: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Document when unreachable is set.

Sibling optional fields (manifest, error) have comments explaining when they're populated; unreachable has none. Given it's only set on one of several SSH-failure paths (see the completeness comment above), a short comment clarifying its scope would help avoid future gaps.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/state/sandbox.ts` at line 124, Add a brief doc comment for the
unreachable field in the Sandbox state type so its population scope is clear,
matching the style used for manifest and error. Update the Sandbox interface
near unreachable to note that it is only set on specific SSH failure paths and
not for all connectivity errors, using the surrounding completeness/comment
context to place it correctly.
docs/reference/commands.mdx (1)

1762-1762: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Split into one sentence per line.

This line packs two sentences together. As per coding guidelines, docs/**/*.{md,mdx} files should "Keep one sentence per line in Markdown and MDX source files."

📝 Proposed fix
-A running sandbox whose in-sandbox SSH endpoint does not answer fails its backup and aborts the run. Set `NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1` to skip such sandboxes and continue the upgrade; they are recovered from their latest validated backup during onboarding.
+A running sandbox whose in-sandbox SSH endpoint does not answer fails its backup and aborts the run.
+Set `NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1` to skip such sandboxes and continue the upgrade.
+They are recovered from their latest validated backup during onboarding.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/reference/commands.mdx` at line 1762, The markdown/MDX text in the
referenced command docs currently combines two sentences on one line, which
violates the one-sentence-per-line guideline. Split the content into separate
lines so each sentence stands alone, keeping the same wording and updating the
relevant prose near the sandbox backup note in the commands reference.

Source: Coding guidelines

src/lib/actions/maintenance.ts (1)

34-36: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Move the backup-skip decision out of actions.

shouldSkipUnreachableSandboxBackup is a pure decision function with no I/O — exactly the kind of helper the layering doc calls out for state/domain, not actions. As per path instructions, src/lib/README.md states: "When adding behavior like SSH-unreachable classification and backup-skip decisions for the installer, place orchestration/user-facing flow in actions (e.g., backupAll), and keep classification helpers testable/pure where possible in state/domain (e.g., isSshTransportFailure)."

Consider relocating this predicate alongside isSshTransportFailure (or another domain module) and importing it into backupAll, keeping actions/maintenance.ts focused on orchestration.

#!/bin/bash
# Survey existing domain modules to find the right home for this predicate
fd . src/lib/domain
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/actions/maintenance.ts` around lines 34 - 36, The pure backup-skip
predicate is in the wrong layer and should be moved out of actions. Relocate
shouldSkipUnreachableSandboxBackup from maintenance.ts into a domain/state
helper alongside isSshTransportFailure (or another appropriate testable module),
then import and use it from backupAll so actions/maintenance.ts stays
orchestration-only. Keep the helper pure and preserve its current env-based
decision logic in the new home.

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/state/sandbox.ts`:
- Around line 1022-1026: Propagate the unreachable/skippable state through SSH
failure handling so later audit and SSH+tar paths don’t keep retrying after the
initial dir probe marks the sandbox unreachable. Update isSshTransportFailure
and the related audit/tar failure handling in sandbox.ts to carry forward the
unreachable flag from the probe result, and use that in the SSH retry/skip
decision instead of returning a plain failure.

---

Nitpick comments:
In `@docs/reference/commands.mdx`:
- Line 1762: The markdown/MDX text in the referenced command docs currently
combines two sentences on one line, which violates the one-sentence-per-line
guideline. Split the content into separate lines so each sentence stands alone,
keeping the same wording and updating the relevant prose near the sandbox backup
note in the commands reference.

In `@src/lib/actions/maintenance.ts`:
- Around line 34-36: The pure backup-skip predicate is in the wrong layer and
should be moved out of actions. Relocate shouldSkipUnreachableSandboxBackup from
maintenance.ts into a domain/state helper alongside isSshTransportFailure (or
another appropriate testable module), then import and use it from backupAll so
actions/maintenance.ts stays orchestration-only. Keep the helper pure and
preserve its current env-based decision logic in the new home.

In `@src/lib/state/sandbox.ts`:
- Line 124: Add a brief doc comment for the unreachable field in the Sandbox
state type so its population scope is clear, matching the style used for
manifest and error. Update the Sandbox interface near unreachable to note that
it is only set on specific SSH failure paths and not for all connectivity
errors, using the surrounding completeness/comment context to place it
correctly.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 8f7cd8eb-38a8-4bcb-85c8-82631ee48981

📥 Commits

Reviewing files that changed from the base of the PR and between 9fe4536 and 9186fdd.

📒 Files selected for processing (6)
  • docs/reference/commands.mdx
  • scripts/install.sh
  • src/lib/actions/maintenance.test.ts
  • src/lib/actions/maintenance.ts
  • src/lib/state/sandbox.test.ts
  • src/lib/state/sandbox.ts

Comment thread src/lib/state/sandbox.ts Outdated
@wscurran
wscurran requested a review from cjagwani July 2, 2026 16:41
@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
test/install-openshell-upgrade-prompt.test.ts (1)

221-234: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Sequence assertions don't actually check order.

This test is meant to verify the ordered sequence old:backup-allprepare-currentcurrent:backup-all, but .toContain on the split array only checks membership, so it would still pass even if the calls happened out of order (e.g. if a future regression called prepare-current before the old backup attempt).

🧪 Proposed fix to assert actual order
-    expect(cliLog.split(/\r?\n/)).toContain("old:backup-all");
-    expect(cliLog.split(/\r?\n/)).toContain("prepare-current");
-    expect(cliLog.split(/\r?\n/)).toContain("current:backup-all");
+    const cliLogLines = cliLog.split(/\r?\n/).filter(Boolean);
+    expect(cliLogLines.indexOf("old:backup-all")).toBeGreaterThanOrEqual(0);
+    expect(cliLogLines.indexOf("old:backup-all")).toBeLessThan(cliLogLines.indexOf("prepare-current"));
+    expect(cliLogLines.indexOf("prepare-current")).toBeLessThan(cliLogLines.indexOf("current:backup-all"));

As per path instructions, "Flag copied production algorithms, broad mocks that bypass the behavior under test, and conditionals that make a test pass without exercising its claim."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/install-openshell-upgrade-prompt.test.ts` around lines 221 - 234, The
sequence check in the retry test does not verify the call order, so a regression
could still pass even if the backup steps happen out of order. Update the
assertions in the test that uses runPreinstallUpgradeGuard, cliLog, and the
expected old:backup-all → prepare-current → current:backup-all flow so it
compares indices or otherwise enforces ordering, not just membership. Keep the
same symbols and log markers, but make the test fail unless the commands occur
in the intended sequence.

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/reference/commands-nemohermes.mdx`:
- Around line 1402-1403: Split the text in the affected MDX paragraph so each
sentence is on its own line, keeping the wording the same. Update the sentence
in the command reference content near the sandbox backup note so the statement
about skipping unreachable sandboxes and the statement about recovery from the
latest validated backup are separated. Preserve the surrounding markdown
structure and ensure the one-sentence-per-line guideline is followed.

---

Nitpick comments:
In `@test/install-openshell-upgrade-prompt.test.ts`:
- Around line 221-234: The sequence check in the retry test does not verify the
call order, so a regression could still pass even if the backup steps happen out
of order. Update the assertions in the test that uses runPreinstallUpgradeGuard,
cliLog, and the expected old:backup-all → prepare-current → current:backup-all
flow so it compares indices or otherwise enforces ordering, not just membership.
Keep the same symbols and log markers, but make the test fail unless the
commands occur in the intended sequence.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: af67cd1a-90cc-4f75-a403-62777823ab4b

📥 Commits

Reviewing files that changed from the base of the PR and between 9186fdd and d86d3f7.

📒 Files selected for processing (4)
  • docs/reference/commands-nemohermes.mdx
  • docs/reference/commands.mdx
  • scripts/install.sh
  • test/install-openshell-upgrade-prompt.test.ts
✅ Files skipped from review due to trivial changes (1)
  • docs/reference/commands.mdx
🚧 Files skipped from review as they are similar to previous changes (1)
  • scripts/install.sh

Comment thread docs/reference/commands-nemohermes.mdx Outdated
Signed-off-by: Tinson Lai <tinsonl@nvidia.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
src/lib/state/sandbox.ts (1)

1128-1141: 🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift

Mark SSH-config failures as unreachable in src/lib/state/sandbox.ts:1129-1141
getSshConfig() collapses transport failures into null, so this return loses the SSH-unreachable signal. backupAll() won’t honor NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP for sandboxes that die at the ssh-config step.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/state/sandbox.ts` around lines 1128 - 1141, The SSH config lookup
path in backupAll currently treats getSshConfig(sandboxName) returning null as a
generic failure, which drops the unreachable signal. Update the SSH+tar download
branch to preserve this as an SSH-unreachable case so the existing
skip-unreachable behavior still applies, using the backupAll flow and the
getSshConfig result handling in src/lib/state/sandbox.ts.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@src/lib/state/sandbox.ts`:
- Around line 1128-1141: The SSH config lookup path in backupAll currently
treats getSshConfig(sandboxName) returning null as a generic failure, which
drops the unreachable signal. Update the SSH+tar download branch to preserve
this as an SSH-unreachable case so the existing skip-unreachable behavior still
applies, using the backupAll flow and the getSshConfig result handling in
src/lib/state/sandbox.ts.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 3957ccbc-20f0-4a81-bb5a-2ea52f3b155f

📥 Commits

Reviewing files that changed from the base of the PR and between d86d3f7 and cf042c3.

📒 Files selected for processing (4)
  • docs/reference/commands-nemohermes.mdx
  • docs/reference/commands.mdx
  • src/lib/state/sandbox.ts
  • test/install-openshell-upgrade-prompt.test.ts
✅ Files skipped from review due to trivial changes (2)
  • docs/reference/commands.mdx
  • docs/reference/commands-nemohermes.mdx
🚧 Files skipped from review as they are similar to previous changes (1)
  • test/install-openshell-upgrade-prompt.test.ts

@cv cv added v0.0.73 and removed v0.0.74 labels Jul 2, 2026
cjagwani and others added 7 commits July 2, 2026 12:45
…docs (#6188)

Address advisor findings on #6199:

- PRA-5/PRA-8: isSshTransportFailure now checks result.signal for SIGHUP
  and SIGPIPE explicitly (matches connect.ts). Signal-killed SSH probes
  (common when the sandbox gateway dies mid-connection) are classified
  as transport-level, so the skip flag activates.
- PRA-2: warning + guidance messages at maintenance.ts:134 and 158, and
  the installer error at scripts/install.sh, now explicitly state that
  any uncommitted state since the last successful backup will be lost.
- PRA-3/PRA-4: docs at commands.mdx and commands-nemohermes.mdx call
  out that only the exact value '1' is accepted (true/yes/0 are not).
- CodeRabbit minor: split the two-clause sentence per line at both mdx
  sites while updating.

Adds two sandbox.test.ts cases for the new signal handling.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
… paths (#6188)

Address advisor PRA-1 / PRA-7 and CodeRabbit Major #2:

- backupStateFile now returns { outcome, unreachable } instead of a
  bare string enum. Transport-level SSH failures (exit 255, signal
  kill, spawn error) during state-file backup set unreachable=true.
  The caller in backupSandboxState propagates this to the outer
  BackupResult.unreachable so NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1
  activates for state-file failures, not only the initial dir probe.

- getSshConfig-null branch in backupSandboxState now returns
  unreachable=true. For a sandbox already confirmed running (the
  maintenance.ts loop only reaches BackupResult.unreachable for
  running sandboxes), an ssh-config lookup failure is transport-level.
  Addresses CodeRabbit's Major finding at sandbox.ts:1128-1141.

Both branches now feed the same skip path already exercised by the
initial dir probe. Propagation is covered by manual E2E for the
running-sandbox-goes-unreachable-mid-loop case; unit tests would
require heavy spawnSync mocking that this PR does not introduce.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
…ycle-flags row (#6188)

Close remaining GPT-5.5 advisor items:

- PRA-2: the docs promised 'recovered from their latest validated
  backup during onboarding' without distinguishing between the
  installer's automatic pre-upgrade backup-all (which exports
  NEMOCLAW_RESTORE_LATEST_BACKUP_ON_RECREATE=1 at install.sh:1912)
  and standalone 'nemoclaw backup-all' runs (which do not).
  Split the paragraph in both mdx files so the automatic restore is
  clearly installer-scoped; standalone runs only skip the failure.

- PRA-4: add a Lifecycle Behavior Flags table row for
  NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP in both commands.mdx and
  commands-nemohermes.mdx so it shows up alongside the other
  sandbox-scoped lifecycle env vars.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
#6188)

Address advisor PRA-6 and PRA-10:

- Move isSshTransportFailure from src/lib/state/sandbox.ts into a new
  src/lib/state/ssh-transport.ts. The function was already a pure
  utility with no sandbox.ts internals; extraction keeps sandbox.ts
  smaller and gives future SSH-transport helpers a natural home. Re-
  exported from sandbox.ts for backwards compatibility.

- Rename src/lib/state/sandbox.test.ts to
  src/lib/state/ssh-transport.test.ts so the test file name matches
  its actual scope.

No behavior change.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
…ion (#6188)

The re-export syntax 'export { isSshTransportFailure } from "./ssh-transport"'
doesn't bind the name into local module scope in TS, so internal call
sites in sandbox.ts stopped resolving after the previous extraction commit.
Add a real import so both external re-export and local uses resolve.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
…retry (#6188)

Add advisor PRA-9 coverage: prove the installer passes
NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1 through unchanged when it
retries pre-upgrade backup with the current CLI. The env var is
consumed by the CLI's backup-all path (maintenance.ts); the
installer's contract is just to propagate it.

- Extend the current-CLI mock to echo the skip env var into cli.log
  so the test can assert propagation without relying on install.sh
  internals.
- Add a new test case where NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1
  is set, the old CLI fails, the current CLI is used for retry, and
  cli.log records skip-env=1 alongside current:backup-all.
- Update the pre-existing 'aborts current-gateway upgrades' assertion
  to match the new install.sh error text (which now includes the
  data-loss disclosure per PRA-2).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>

@cjagwani cjagwani left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @laitingsheng, pushed a bunch of follow-ups tackling everything the advisors flagged. Walked through each one:

  • backupStateFile now returns a typed result so transport failures during state-file backup carry the unreachable flag up; same treatment on the getSshConfig null path. Both feed the existing skip flow.
  • Data-loss disclosure added to all three warning/error sites, and the docs now spell out that only exactly =1 counts (not true/yes) plus a row in the Lifecycle Flags table.
  • isSshTransportFailure moved into its own module and the test file renamed to match. Also added SIGHUP/SIGPIPE handling since spawnSync surfaces those with status=null and we want the diagnostic explicit.
  • Installer integration test now proves the skip env var actually propagates through the current-CLI retry, not just that install.sh mentions it.

Left shouldSkipUnreachableSandboxBackup exported since the unit test imports it directly. CodeRabbit came back clean on the re-run, cloud-onboard E2E already green, gateway-upgrade and state-backup-restore still running.

Approving pending the last CI checks. Nice fix.

@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Vitest E2E Target Results — ✅ All requested jobs passed

Run: 28617740925
Workflow ref: fix/preupgrade-backup-skip-unreachable
Requested targets: (default — all supported)
Requested jobs: openshell-gateway-upgrade,state-backup-restore,cloud-onboard
Summary: 3 passed, 0 failed, 0 cancelled, 0 skipped

Job Result
cloud-onboard ✅ success
openshell-gateway-upgrade ✅ success
state-backup-restore ✅ success

@cv
cv merged commit 2c447b7 into main Jul 2, 2026
43 checks passed
@cv
cv deleted the fix/preupgrade-backup-skip-unreachable branch July 2, 2026 20:31
@cv cv mentioned this pull request Jul 2, 2026
21 tasks
cv added a commit that referenced this pull request Jul 2, 2026
<!-- markdownlint-disable MD041 -->
## Summary
This PR prepares the user-facing documentation for v0.0.73 before the
release plan is frozen.
It adds release notes for the merged runtime changes and closes
documentation gaps around DNS-backed HTTPS endpoint validation and
LangChain Deep Agents Code proxy recovery.

## Changes
- Add the `v0.0.73` release-note section with links to the detailed
command, inference, recovery, lifecycle, platform, and setup
documentation.
- Correct the custom endpoint guidance so DNS-backed HTTPS rejection and
the supported alternatives match the fail-closed runtime behavior.
- Document the managed `inference.local` proxy boundary and rebuild
requirement for existing LangChain Deep Agents Code sandboxes.
- Add troubleshooting guidance for the DNS-backed HTTPS validation
error.
- Validate with `npm run docs:sync-agent-variants` and `npm run docs`;
Fern completed with 0 errors and 2 existing warnings.
- Source summary:
- [#6139](#6139) ->
`docs/about/release-notes.mdx`, `docs/inference/inference-options.mdx`,
`docs/reference/commands.mdx`, `docs/reference/commands-nemohermes.mdx`,
and `docs/reference/troubleshooting.mdx`: Document fail-closed
DNS-backed HTTPS endpoint handling and recovery options.
- [#6142](#6142) ->
`docs/about/release-notes.mdx`: Summarize native OpenShell GPU injection
and compatibility-path diagnostics.
- [#6197](#6197) ->
`docs/about/release-notes.mdx`: Summarize agent-aware messaging preset
rejection.
- [#6199](#6199) ->
`docs/about/release-notes.mdx`: Summarize the unreachable-sandbox backup
opt-in, restore behavior, and data-loss boundary.
- [#6204](#6204) and
[#6206](#6206) ->
`docs/about/release-notes.mdx` and
`docs/get-started/quickstart-langchain-deepagents-code.mdx`: Document
the corrected managed proxy contract and required sandbox rebuild.
- [#6213](#6213) ->
`docs/about/release-notes.mdx`: Summarize the merged setup, recovery,
and host-state documentation.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [x] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates
<!-- Check all that apply. For any "covered by existing tests", "not
applicable", or waiver entry, add a brief justification on the same line
or in the Changes section. -->
- [ ] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [x] Tests not applicable — justification: documentation-only release
preparation; the Fern docs build validates the changed pages and routes.
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [ ] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification:
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification
<!-- Check each item you ran and confirmed. Leave unchecked items you
skipped. Doc-only changes do not require npm test unless you ran it. -->
- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Git hooks passed during commit and push, or `npx prek run
--from-ref main --to-ref HEAD` passes
- [ ] Targeted tests pass for changed behavior
- [ ] Full `npm test` passes (broad runtime changes only)
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
<!-- DCO sign-off is required in this PR description, and every commit
must appear as Verified in GitHub. Run: git config user.name && git
config user.email -->
Signed-off-by: Carlos Villela <cvillela@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
* Added a new **v0.0.73** release notes section with six highlights at
the top of the changelog.
* Expanded **Custom Endpoint URL Validation** guidance in inference
option docs, including explicit acceptance/rejection rules for HTTP vs
DNS-backed HTTPS and how validated IPs are stored.
* Updated command references (`nemohermes inference set`, `$$nemoclaw
inference set`) to match the new validation behavior.
* Added troubleshooting documentation for unsupported **DNS-backed HTTPS
endpoints**, plus clarified Deep Agents Code routing and post-upgrade
sandbox rebuild guidance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Hadar301 pushed a commit to Hadar301/NemoClaw-OpenShift that referenced this pull request Jul 12, 2026
…kup (NVIDIA#6199)

<!-- markdownlint-disable MD041 -->
## Summary
Pre-upgrade `backup-all` aborted the `curl | bash` installer whenever a
running sandbox's in-sandbox SSH endpoint did not answer, with no
override, looping the upgrade forever. This classifies such a sandbox as
unreachable and adds `NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1` to
skip it so the upgrade proceeds and onboarding recovers it from its
latest validated backup.

## Related Issue
Fixes NVIDIA#6188

## Changes
- `src/lib/state/sandbox.ts`: add `unreachable` to `BackupResult` and
set it on an SSH transport-level dir-check failure (exit 255, timeout,
spawn error) via a new `isSshTransportFailure` predicate.
- `src/lib/actions/maintenance.ts`: an unreachable running sandbox is
skipped when `NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1`, otherwise it
still fails but prints actionable guidance before exit.
- `scripts/install.sh`: reword the pre-upgrade backup abort to name the
override and the recovery path.
- `docs/reference/commands.mdx`: document the flag in the `backup-all`
section.
- Tests: `maintenance.test.ts` gains skip-with-flag, fail-with-guidance,
and flag truth-table cases; new `sandbox.test.ts` covers
`isSshTransportFailure`.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates
- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification:
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification
- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Git hooks passed during commit and push, or `npx prek run
--from-ref main --to-ref HEAD` passes
- [x] Targeted tests pass for changed behavior
- [ ] Full `npm test` passes (broad runtime changes only)
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
Signed-off-by: Tinson Lai <tinsonl@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* `backup-all` now detects running sandboxes with an unreachable
in-sandbox SSH endpoint and can skip them when
`NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP=1`.
* Skipped sandboxes are recovered during onboarding from the latest
validated backup; any uncommitted state since then is not preserved.
* **Bug Fixes**
* Improved failure handling and remediation when SSH
transport/unreachability occurs, including clearer guidance and
installer/upgrade retry behavior.
* **Documentation**
* Updated `nemoclaw` and `nemohermes` `backup-all` docs to state the
default abort behavior and the skip flag’s exact `=1` requirement.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Tinson Lai <tinsonl@nvidia.com>
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
Co-authored-by: Charan Jagwani <cjagwani@nvidia.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Hadar301 pushed a commit to Hadar301/NemoClaw-OpenShift that referenced this pull request Jul 12, 2026
<!-- markdownlint-disable MD041 -->
## Summary
This PR prepares the user-facing documentation for v0.0.73 before the
release plan is frozen.
It adds release notes for the merged runtime changes and closes
documentation gaps around DNS-backed HTTPS endpoint validation and
LangChain Deep Agents Code proxy recovery.

## Changes
- Add the `v0.0.73` release-note section with links to the detailed
command, inference, recovery, lifecycle, platform, and setup
documentation.
- Correct the custom endpoint guidance so DNS-backed HTTPS rejection and
the supported alternatives match the fail-closed runtime behavior.
- Document the managed `inference.local` proxy boundary and rebuild
requirement for existing LangChain Deep Agents Code sandboxes.
- Add troubleshooting guidance for the DNS-backed HTTPS validation
error.
- Validate with `npm run docs:sync-agent-variants` and `npm run docs`;
Fern completed with 0 errors and 2 existing warnings.
- Source summary:
- [NVIDIA#6139](NVIDIA#6139) ->
`docs/about/release-notes.mdx`, `docs/inference/inference-options.mdx`,
`docs/reference/commands.mdx`, `docs/reference/commands-nemohermes.mdx`,
and `docs/reference/troubleshooting.mdx`: Document fail-closed
DNS-backed HTTPS endpoint handling and recovery options.
- [NVIDIA#6142](NVIDIA#6142) ->
`docs/about/release-notes.mdx`: Summarize native OpenShell GPU injection
and compatibility-path diagnostics.
- [NVIDIA#6197](NVIDIA#6197) ->
`docs/about/release-notes.mdx`: Summarize agent-aware messaging preset
rejection.
- [NVIDIA#6199](NVIDIA#6199) ->
`docs/about/release-notes.mdx`: Summarize the unreachable-sandbox backup
opt-in, restore behavior, and data-loss boundary.
- [NVIDIA#6204](NVIDIA#6204) and
[NVIDIA#6206](NVIDIA#6206) ->
`docs/about/release-notes.mdx` and
`docs/get-started/quickstart-langchain-deepagents-code.mdx`: Document
the corrected managed proxy contract and required sandbox rebuild.
- [NVIDIA#6213](NVIDIA#6213) ->
`docs/about/release-notes.mdx`: Summarize the merged setup, recovery,
and host-state documentation.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [x] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates
<!-- Check all that apply. For any "covered by existing tests", "not
applicable", or waiver entry, add a brief justification on the same line
or in the Changes section. -->
- [ ] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [x] Tests not applicable — justification: documentation-only release
preparation; the Fern docs build validates the changed pages and routes.
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [ ] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification:
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification
<!-- Check each item you ran and confirmed. Leave unchecked items you
skipped. Doc-only changes do not require npm test unless you ran it. -->
- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Git hooks passed during commit and push, or `npx prek run
--from-ref main --to-ref HEAD` passes
- [ ] Targeted tests pass for changed behavior
- [ ] Full `npm test` passes (broad runtime changes only)
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
<!-- DCO sign-off is required in this PR description, and every commit
must appear as Verified in GitHub. Run: git config user.name && git
config user.email -->
Signed-off-by: Carlos Villela <cvillela@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
* Added a new **v0.0.73** release notes section with six highlights at
the top of the changelog.
* Expanded **Custom Endpoint URL Validation** guidance in inference
option docs, including explicit acceptance/rejection rules for HTTP vs
DNS-backed HTTPS and how validated IPs are stored.
* Updated command references (`nemohermes inference set`, `$$nemoclaw
inference set`) to match the new validation behavior.
* Added troubleshooting documentation for unsupported **DNS-backed HTTPS
endpoints**, plus clarified Deep Agents Code routing and post-upgrade
sandbox rebuild guidance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery bug-fix PR fixes a bug or regression integration: dcode LangChain Deep Code integration behavior NV QA Bugs found by the NVIDIA QA Team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[macOS][Upgrade] v0.0.72 curl|bash upgrade loops forever — pre-upgrade backup-all aborts when a running sandbox's SSH backup times out

4 participants