fix(onboard): recover dead sandbox forwards

[apurvvkumaria]

Summary

OpenShell can retain the expected dashboard forward as an exact dead row after a sandbox rebuild. NemoClaw previously waited the full 180-second registration timeout and then failed because the port was still associated with the stale forward. This change gives that exact row a bounded recovery grace period, terminates the failed detached attempt, and performs exactly one sandbox-scoped cleanup and replacement attempt.

Related Issue

Refs #7140

Changes

• Distinguish a persistent exact sandbox + port + dead row from an untracked occupied port.
• Give a matching dead row two seconds to recover before cleanup.
• Terminate the detached forward child before invoking the existing sandbox-scoped cleanup callback.
• Keep the one-use dead-forward recovery budget independent from the existing port-conflict and listener-failure retry limit.
• Preserve transient recovery, foreign-sandbox and wrong-port ownership boundaries, listener diagnostics, and redaction.
• Cover the observed ANSI-colourised dead status as well as recovery and ownership-boundary cases.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification: this is an internal bounded recovery improvement with no new command, option, configuration, migration, or user action; existing forward troubleshooting remains accurate.
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: focused review caught that the first version reused the three-retry budget; commit 32ecbcdd4 replaces it with one independent dead-forward recovery and adds a default-options persistent-dead regression. Exact sandbox/port matching, sandbox-scoped cleanup, SIGTERM-before-cleanup ordering, listener ownership, and redacted diagnostics remain intact.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: no-docs-needed
• Evidence: Reviewed exact head 35f5ba5ca; the PR diff still only changes src/lib/onboard/forward-start.ts and its tests. The bounded exact-row recovery changes no command, flag, configuration, output contract, or user action, and existing troubleshooting/sandbox docs already cover automatic stale-forward cleanup and repair.
• Agent: Codex Desktop

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit:
• Station profile/scenario:
• Result:
• Supporting evidence:

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — npx vitest run --project cli src/lib/onboard/forward-start.test.ts src/lib/onboard/agent-dashboard-forward.test.ts src/lib/onboard/dashboard-preflight-ports.test.ts src/lib/onboard/hermes-dashboard.test.ts passed 64 tests after the mainline refresh.
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — not applicable to this two-file focused recovery change; npm run build:cli, npm run typecheck:cli, npm run source-shape:check, npm run test:titles:check, and npm run test:projects:check passed.
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Apurv Kumaria akumaria@nvidia.com

[coderabbitai]

📝 Walkthrough

Walkthrough

Detached forward startup now detects exact dead forward rows, waits through a grace period, and returns a dead-forward outcome. Retry handling permits one recovery attempt for this outcome while preserving existing standard retry and cleanup behavior, with tests covering recovery, cleanup, and row matching.

Changes

Dead Forward Recovery

Layer / File(s)	Summary
Dead forward detection and grace period src/lib/onboard/forward-start.ts, src/lib/onboard/forward-start.test.ts	Forward polling parses exact sandbox-port rows, waits two seconds for dead status recovery, terminates persistent dead attempts, and returns dead-forward; diagnostics coverage validates this path.
Dead forward retry policy src/lib/onboard/forward-start.ts, src/lib/onboard/forward-start.test.ts	Retry orchestration allows one dead-forward recovery attempt, limits cleanup to the prior attempt, preserves standard retry limits, and tests recovery and sandbox/port matching.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related PRs

• NVIDIA/NemoClaw#7267: Updates the same forward-start retry flow with listener-start-failure classification and retry handling.

Suggested labels: area: onboarding, area: sandbox, area: networking, bug-fix

Suggested reviewers: cv, prekshivyas

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: recovering onboarding forwards that remain marked dead.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/7140-dead-forward-recovery

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 427078d in the codex/7140-dead-forw... branch remains at 96%, unchanged from commit 584ed60 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 427078d in the codex/7140-dead-forw... branch is 80%. The coverage in commit 584ed60 in the main branch is 81%.

Show a code coverage summary of the most impacted files.

File	main 584ed60	codex/7140-dead-forw... 427078d	+/-
src/lib/actions...-add-restart.ts	19%	10%	-9%
src/lib/actions...on-readiness.ts	100%	91%	-9%
src/lib/actions...x/mcp-bridge.ts	43%	36%	-7%
src/lib/actions...lution-probe.ts	93%	88%	-5%
src/lib/actions...e-validation.ts	84%	81%	-3%
src/lib/onboard...shboard-port.ts	93%	90%	-3%
src/lib/actions...dbox/destroy.ts	95%	93%	-2%
src/lib/shields/index.ts	72%	71%	-1%
src/lib/onboard...eway-service.ts	82%	81%	-1%
src/lib/platform.ts	84%	89%	+5%

---

_{Updated July 26, 2026 11:40 UTC}

[github-actions]

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

• GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Model comparison: normalized findings match; normalized E2E selections differ; severity counts match.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: onboard-repair, onboard-resume, cloud-onboard

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[coderabbitai]

🧹 Nitpick comments (1)

src/lib/onboard/forward-start.ts (1)

391-446: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Dead-forward detection logic is correct; consider extracting for readability.

Verified the priority ordering (owner-match → port-conflict → dead-grace → listener-start-failure/untracked-forward) and the grace-timer reset semantics are sound and match the PR's stated intent. This block adds meaningful branching to an already large function (flagged as a complexity hotspot). Extracting the dead-row match into a small helper (mirroring the existing classifyListenerStartDiagnostic pattern) would keep the polling loop's top-level flow easier to scan.

♻️ Proposed extraction

+function isExpectedForwardDead(
+  list: string,
+  expect: { port: number; sandboxName: string },
+): boolean {
+  return parseForwardList(list).some(
+    (entry) =>
+      entry.sandboxName === expect.sandboxName &&
+      entry.port === String(expect.port) &&
+      entry.status === "dead",
+  );
+}
+
 // ... inside the polling loop ...
-      const expectedForwardIsDead = parseForwardList(list).some(
-        (entry) =>
-          entry.sandboxName === expect.sandboxName &&
-          entry.port === String(expect.port) &&
-          entry.status === "dead",
-      );
+      const expectedForwardIsDead = isExpectedForwardDead(list, expect);

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/onboard/forward-start.ts` around lines 391 - 446, Extract the
dead-forward row matching logic from the polling loop into a small helper,
analogous to classifyListenerStartDiagnostic. The helper should parse the
forward list and determine whether an entry matches expect.sandboxName,
expect.port, and status "dead"; replace the inline expectedForwardIsDead
expression with a call while preserving the existing grace-timer and branch
ordering.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/lib/onboard/forward-start.ts`:
- Around line 391-446: Extract the dead-forward row matching logic from the
polling loop into a small helper, analogous to classifyListenerStartDiagnostic.
The helper should parse the forward list and determine whether an entry matches
expect.sandboxName, expect.port, and status "dead"; replace the inline
expectedForwardIsDead expression with a call while preserving the existing
grace-timer and branch ordering.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 06a8ab0c-f760-48b8-a8b2-b2b7096e0d4e

📥 Commits

Reviewing files that changed from the base of the PR and between 29bd189 and 32ecbcd.

📒 Files selected for processing (2)

• src/lib/onboard/forward-start.test.ts
• src/lib/onboard/forward-start.ts

[apurvvkumaria]

Exact-head CI disposition for 35f5ba5ca:

• All eight CLI shards and aggregate, plugin, build/typecheck, static, installer, WeChat, macOS, CodeQL, both sandbox image builds, and all four self-hosted sandbox tests passed.
• The primary failure is external provenance verification: npm audit signatures --omit=dev reported invalid attestations for unchanged dependencies @clack/core@1.4.2, @clack/prompts@1.6.0, and @mistralai/mistralai@2.4.0. The aggregate checks failure is derivative.
• Terra also failed before analysis with HTTP 403; Nemotron and CodeRabbit passed, and there are no review threads.
• No credential-bearing E2E was dispatched because prerequisite CI failed.

These are fail-closed external audit/advisor outcomes, not failures in the forward-recovery change. I have not blindly rerun them.

[cv]

Approved exact head 427078d after product-scope, correctness, nine-category security, compliance, documentation, collision, focused-test, fresh CI/advisor, and selected E2E review. Canonical gate passed with all 51 current checks green.