fix(status): recommend start for crashed-sandbox recovery (#7222)

[yanyunl1991]

Summary

When a sandbox container crashes (Exited 137), the Error-phase recovery hint recommended rebuild --yes ("workspace state will be preserved"), but that command aborts on the pre-rebuild backup of the dead container and pushes the user toward rebuild --force (which discards state). The command that actually recovers the sandbox without data loss — <sandbox> start — was never surfaced. This PR points the crashed-container hint at start.

Closes #7222.

Reproduction

On our Ubuntu 24.04 x86_64 test host (no GPU). The reporter is on DGX Station aarch64; the crashed-sandbox recovery hint is CLI logic and is arch/GPU-independent.

1. Onboard an OpenClaw sandbox, confirm Phase: Ready.
2. Write a workspace marker: nemoclaw <sb> exec -- bash -c 'echo MARK > /sandbox/marker.txt'.
3. Crash it: docker kill <the openshell-<sb>-... container> (→ Exited 137).
4. nemoclaw <sb> status — observe the recovery hint.
5. Run the hinted command; then run nemoclaw <sb> start.

Observed on main (before fix)

# step 4 (status):
  Sandbox '<sb>' is stuck in 'Error' phase.
  Run `nemoclaw <sb> rebuild --yes` to recreate the sandbox (... workspace state will be preserved).
# step 5 (rebuild --yes):
  Failed to back up sandbox state.  Failed files: openclaw.json
  Aborting rebuild to prevent data loss.
  Hint: use 'nemoclaw <sb> rebuild --force' only if you accept losing state ...
  -> Phase stays Error (not recovered).
# `nemoclaw <sb> start` DOES recover: Phase: Ready, marker preserved — but the hint never mentions it.

Observed on fix/... (after fix)

# step 4 (status):
  Sandbox '<sb>' is stuck in 'Error' phase.
  Run `nemoclaw <sb> start` to restart the crashed container and recover the sandbox with workspace state preserved.
  (`nemoclaw <sb> rebuild --yes` recreates the sandbox instead, but its pre-rebuild backup cannot snapshot a stopped container, so start it first.)
# following the hint -> `nemoclaw <sb> start`: Phase: Ready, marker preserved (verified).

Analysis

For a crashed sandbox the container is still present but its phase is terminal (Error). Two places print an identical recovery hint recommending rebuild --yes: the status renderer printNonReadySandboxPhaseGuidance (src/lib/actions/sandbox/status-lookup-rendering.ts) and the shared non-Ready-phase guidance in ensureLiveSandboxOrExit (src/lib/actions/sandbox/gateway-state.ts). But rebuild first snapshots the sandbox, and the pre-rebuild backup cannot snapshot a stopped container (openclaw.json backup fails), so the backup gate aborts to prevent data loss. startSandbox (src/lib/actions/sandbox/start.ts) is the crashed-container recovery path — it restarts the stopped container (or unpauses a paused one) and brings the gateway back up "exactly as recover", preserving /sandbox. So the promised recovery command could not deliver, while the working one was hidden.

Fix

Point the crashed-container hint (container present + terminal phase) at nemoclaw <sb> start in both print sites. rebuild --yes is kept only as the recreate alternative, with a note that its pre-rebuild backup cannot snapshot a stopped container so start must come first. The missing-container branch in gateway-state.ts — where start cannot help because the container is gone — deliberately keeps the rebuild/onboard guidance. The paused-container branch (#4495, docker unpause) is unchanged.

Tests (status-lookup-rendering.test.ts) pin the contract: a crashed (Error, non-paused) sandbox is steered to start and no longer to the rebuild --yes ... workspace state will be preserved promise; the paused branch still recommends docker unpause and never start/rebuild; a Ready sandbox prints no guidance.

Changes

• src/lib/actions/sandbox/status-lookup-rendering.ts: crashed-container status hint recommends start; export the guidance helper for testing.
• src/lib/actions/sandbox/gateway-state.ts: same hint change in the shared non-Ready-phase guidance.
• src/lib/actions/sandbox/status-lookup-rendering.test.ts: new — pins the hint contract and its boundaries.

Type of Change

• Code change (feature, bug fix, or refactor)

Verification

• Unit tests pass (status-lookup-rendering.test.ts, 3 tests)
• npm run typecheck:cli and npm run build:cli pass
• Reproduced end-to-end on our Ubuntu 24.04 x86_64 test host: before the fix the hint recommended rebuild --yes (which aborts on the dead container); after the fix it recommends start, which recovers to Phase: Ready with the workspace marker preserved.
• No secrets, API keys, or credentials committed

AI Disclosure

• AI-assisted — tool: Claude Code

Signed-off-by: Yanyun Liao yanyunl@nvidia.com

Summary by CodeRabbit

• Bug Fixes

  • Improved sandbox recovery guidance for non-ready phases, with more accurate handling of Error cases using Docker runtime state when available.
  • When a crashed container can be recovered, guidance now recommends restarting with start to preserve workspace state.
  • Added clearer rebuild --yes caveat when backups can’t snapshot stopped containers, and added docker unpause <containerName> instructions for paused containers.

• Tests

  • Expanded console-output assertions across Error, paused, Failed, and Ready scenarios.
  • Updated Docker health/runtime test coverage to resolve containers using an all-states listing.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Sandbox recovery guidance now distinguishes paused, crashed, and unrecoverable containers, recommending docker unpause, start, or rebuild --yes as appropriate. Docker runtime lookup includes exited containers, with expanded phase-specific tests.

Changes

Sandbox recovery guidance

Layer / File(s)	Summary
Resolve Docker runtime across container states src/lib/actions/sandbox/docker-health.ts, src/lib/actions/sandbox/docker-health.test.ts	Uses docker ps -a to resolve exited sandbox containers and tests all-state runtime detection.
Update sandbox recovery guidance src/lib/actions/sandbox/status-lookup-rendering.ts, src/lib/actions/sandbox/gateway-state.ts	Inspects Docker runtime state for Error phases and provides paused-container unpause, crash-recovery start, or rebuild instructions.
Validate phase-specific guidance src/lib/actions/sandbox/*test.ts	Mocks Docker runtime state and tests guidance for crashed, paused, unrecoverable, Failed, and Ready sandbox states.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related issues

• [All Platforms][Sandbox] Crashed-sandbox recovery hint recommends rebuild --yes which aborts on the backup gate; working recovery is <sandbox> start #7222 — Covers changing Error-phase recovery guidance from rebuild --yes to start while preserving workspace state.

Possibly related PRs

• NVIDIA/NemoClaw#6017 — Introduces terminal styling helpers used by the updated sandbox guidance.

Sequence Diagram(s)

sequenceDiagram
  participant SandboxGateway
  participant DockerRuntime
  participant Operator
  SandboxGateway->>DockerRuntime: inspect Error-phase container
  DockerRuntime-->>SandboxGateway: return paused or exited container metadata
  SandboxGateway->>Operator: print docker unpause, start, or rebuild command

Loading

Suggested labels: area: sandbox, bug-fix

Suggested reviewers: cv, apurvvkumaria, cjagwani

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly reflects the main change: crashed-sandbox recovery now recommends start instead of rebuild --yes.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/7222-crashed-sandbox-recovery-hint-start

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 9ac77e6 in the fix/7222-crashed-san... branch remains at 96%, unchanged from commit 5b547cd in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 9ac77e6 in the fix/7222-crashed-san... branch remains at 80%, unchanged from commit 5b547cd in the main branch.

Show a code coverage summary of the most impacted files.

File	main 5b547cd	fix/7222-crashed-san... 9ac77e6	+/-
src/lib/actions...ocker-health.ts	89%	65%	-24%
src/lib/sandbox...rce-identity.ts	91%	91%	0%
src/lib/state/m...lock-storage.ts	97%	97%	0%
src/lib/inferen...er-lifecycle.ts	65%	71%	+6%
src/lib/actions...ateway-state.ts	65%	76%	+11%
src/lib/actions...up-rendering.ts	34%	55%	+21%
src/lib/inferen...lama/process.ts	50%	100%	+50%

---

_{Updated July 20, 2026 16:41 UTC}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

src/lib/actions/sandbox/status-lookup-rendering.test.ts (1)

19-25: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Use a present, non-paused runtime fixture for the crashed case.

dockerRuntime: null does not establish the comment’s “container present but not paused” precondition; it exercises the generic no-runtime input. A regression affecting an actually present stopped container could therefore pass this test. Supply a typed fixture with paused: false and a container name, keeping a separate null-runtime test only if that behavior is intentional.

As per path instructions, tests under **/*.test.ts must provide behavioral confidence for the claimed scenario.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/actions/sandbox/status-lookup-rendering.test.ts` around lines 19 -
25, Update the crashed-sandbox test for printNonReadySandboxPhaseGuidance to
pass a typed dockerRuntime fixture representing a present, non-paused container,
including paused: false and a container name. Preserve or add a separate
null-runtime test only if generic no-runtime behavior is intentionally covered.

Source: Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/actions/sandbox/gateway-state.ts`:
- Around line 592-595: Update ensureLiveSandboxOrExit to detect paused Docker
containers before emitting the generic start/rebuild guidance, and reuse the
existing paused-container guidance helper or runtime query used by
status-lookup-rendering.ts. Preserve the docker unpause instructions for paused
containers, and add a regression test covering a paused container reporting
Phase: Error through this entrypoint.

---

Nitpick comments:
In `@src/lib/actions/sandbox/status-lookup-rendering.test.ts`:
- Around line 19-25: Update the crashed-sandbox test for
printNonReadySandboxPhaseGuidance to pass a typed dockerRuntime fixture
representing a present, non-paused container, including paused: false and a
container name. Preserve or add a separate null-runtime test only if generic
no-runtime behavior is intentionally covered.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 327cf876-0d3d-4d32-acdb-f56891eea294

📥 Commits

Reviewing files that changed from the base of the PR and between 3c4ea5a and 7ada26c.

📒 Files selected for processing (3)

• src/lib/actions/sandbox/gateway-state.ts
• src/lib/actions/sandbox/status-lookup-rendering.test.ts
• src/lib/actions/sandbox/status-lookup-rendering.ts

[github-actions]

PR Review Advisor — Informational

Advisor assessment: Informational / high confidence
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions
Status: No actionable findings remain in the canonical review ledger.

Model lanes

• GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
• Model comparison: normalized findings match; normalized E2E selections differ; severity counts match.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: onboard-repair, onboard-resume

2 optional E2E recommendations

• sandbox-operations
• ubuntu-repo-docker-post-reboot-recovery

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[coderabbitai]

🧹 Nitpick comments (1)

src/lib/actions/sandbox/gateway-state-hints.test.ts (1)

168-168: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Assert the recovery outcome rather than the helper call.

The guidance output and exit already prove this behavior; the spy assertion unnecessarily locks the test to getSandboxDockerRuntime.

As per path instructions, tests should “prefer observable outcomes through the public boundary over ... mock-call assertions.”

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/actions/sandbox/gateway-state-hints.test.ts` at line 168, Update the
test around the recovery scenario to remove the getSandboxDockerRuntimeSpy call
assertion and retain assertions on the observable guidance output and exit
behavior, using the public boundary rather than mocking implementation details.

Source: Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/lib/actions/sandbox/gateway-state-hints.test.ts`:
- Line 168: Update the test around the recovery scenario to remove the
getSandboxDockerRuntimeSpy call assertion and retain assertions on the
observable guidance output and exit behavior, using the public boundary rather
than mocking implementation details.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 03148d10-8495-4e16-bee3-799c2b6f5df6

📥 Commits

Reviewing files that changed from the base of the PR and between 7ada26c and ed44af5.

📒 Files selected for processing (3)

• src/lib/actions/sandbox/gateway-state-hints.test.ts
• src/lib/actions/sandbox/gateway-state.ts
• src/lib/actions/sandbox/status-lookup-rendering.test.ts

🚧 Files skipped from review as they are similar to previous changes (1)

• src/lib/actions/sandbox/status-lookup-rendering.test.ts

[ericksoa]

Maintainer follow-up on exact head 9a69741: I inspected the getSandboxDockerRuntime contract flagged by the Nemotron warning. Container resolution catches registry errors; missing/non-Docker containers return paused=false with no container name; Docker health/paused inspect errors are caught; and unrecognized paused output normalizes to false, so this path fails closed rather than inventing an unpause hint. Exact-head focused validation including docker-health, gateway-state-hints, and status-lookup-rendering passed 37/37 tests. No code change is needed for that warning; the advisor job itself failed on its internal canonical-ledger consistency check.

[ericksoa]

Approved on exact head 26ee70d against current main a5ccdca. Reviewed and remediated the crashed-versus-missing sandbox guidance and exited-container detection paths. Focused local validation: 56/56 tests plus CLI build/typecheck. Both review-advisor passes reported no findings, CodeRabbit has no unresolved major/critical findings, all current CI checks and the selected onboard-repair/onboard-resume E2E jobs are green.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-7225.docs.buildwithfern.com/nemoclaw

[wscurran]

✨
Related open issues:

• #7222 [All Platforms][Sandbox] Crashed-sandbox recovery hint recommends rebuild --yes which aborts on the backup gate; working recovery is <sandbox> start

---

Related open issues:

• #7222 [All Platforms][Sandbox] Crashed-sandbox recovery hint recommends rebuild --yes which aborts on the backup gate; working recovery is <sandbox> start

[cjagwani]

Approved exact head 9ac77e6 on current main 5b547cd. The current recovery recommendation preserves label-based ownership, skips Docker discovery for non-Docker sandboxes, and is covered by 44 focused tests. All 52 current checks and the selected onboard-repair and onboard-resume E2E targets passed; CodeRabbit and the canonical Advisor review are clean; all commits are GitHub Verified.