fix(cli): fail closed on OpenShell gateway drift

[yimoj]

Summary

Fail closed when OpenShell sandbox-state RPCs may be schema-broken by gateway image drift or protobuf invalid-wire-type output. This prevents backup, status, rebuild, recover, and sandbox upgrade flows from trusting stale OpenShell state and misclassifying registered running sandboxes as stopped.

Related Issue

Fixes #3399

Changes

• Added shared gateway cluster image drift/protobuf mismatch detection under src/lib/adapters/openshell/gateway-drift.ts.
• Guarded backup-all, upgrade-sandboxes, sandbox status, rebuild, and recover/connect liveness checks before trusting OpenShell state RPCs.
• Made installer pre-upgrade backup failure fatal so upgrade does not proceed after unsafe gateway-state failures.
• Added focused tests for drift preflight, protobuf mismatch classification, schema-mismatch propagation, and rebuild/maintenance guards.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Verification

• npx prek run --all-files passes
• npm test passes
• Tests added or updated for new or changed behavior
• No secrets, API keys, or credentials committed
• Docs updated for user-facing behavior changes
• make docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

npm test -- --reporter=dot was run and did not complete cleanly in this worktree because of existing/unrelated host-sensitive failures observed in test/install-preflight.test.ts and test/fetch-guard-patch-regression.test.ts. Targeted gateway-drift tests, CLI typecheck, CLI build, shell formatting/linting for scripts/install.sh, and Codex review passed.

---

Signed-off-by: Yimo Jiang yimoj@nvidia.com

Summary by CodeRabbit

• Bug Fixes

  • Backup, upgrade, rebuild, and status commands now detect gateway version/protocol mismatches and abort early with clearer, actionable error messages; pre-upgrade backup failures are now treated as fatal.

• New Features

  • Expanded detection and user-facing remediation for gateway image drift and protobuf/schema mismatches, with targeted CLI guidance.

• Tests

  • Substantially increased test coverage for drift/mismatch scenarios and updated end-to-end harnesses.

• Chores

  • Test runtime sets a sentinel to disable drift preflight; added a new environment-variable allowlist entry.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 95de3619-f47f-4205-9f9d-4288c02afbf7

📥 Commits

Reviewing files that changed from the base of the PR and between c7e1f76 and 8d7d81f.

📒 Files selected for processing (2)

• scripts/install.sh
• src/lib/onboard.ts

🚧 Files skipped from review as they are similar to previous changes (1)

• src/lib/onboard.ts

---

📝 Walkthrough

Walkthrough

Adds OpenShell gateway-image-drift and protobuf/schema-mismatch detection, formatted issue printing, integrates preflight/result validation into maintenance and sandbox flows, adds adapter and tests, sets a Vitest sentinel, updates installer/onboard/e2e/test helpers, and threads a runtime timeout option.

Changes

Gateway Drift Detection Infrastructure

Layer / File(s)	Summary
Gateway drift adapter: types and detection helpers src/lib/adapters/openshell/gateway-drift.ts	New exported types and helpers to parse gateway cluster image versions, inspect container image/ports, detect active gateway, compute image drift, detect protobuf/schema-result issues, compact/format output, and print remediation guidance.
Gateway drift adapter tests src/lib/adapters/openshell/gateway-drift.test.ts	Unit and integration tests for parsing, drift detection, active-gateway filtering, protobuf-result detection, Vitest sentinel bypass, and formatted guidance samples.
Runtime timeout & test config src/lib/adapters/openshell/runtime.ts, vitest.config.ts, ci/env-var-doc-allowlist.json	getInstalledOpenshellVersionOrNull accepts an optional timeout; Vitest config sets NEMOCLAW_DISABLE_GATEWAY_DRIFT_PREFLIGHT=1; allowlist documents that env var for tests.

Maintenance Actions Gateway Drift Integration

Layer / File(s)	Summary
Maintenance actions: preflight and result validation src/lib/actions/maintenance.ts, src/lib/actions/upgrade-sandboxes.ts	backupAll() and upgradeSandboxes() run preflight drift checks and validate openshell sandbox list results, printing formatted issues and exiting with code 1 when preflight or result issues are detected; removed reliance on ignoreError.
Maintenance actions tests src/lib/actions/gateway-drift-preflight.test.ts	Tests assert image-drift preflight causes exit-before-list and protobuf-result mismatch causes exit-after-list, verifying printed command context and absence of later side effects.

Sandbox Operations Gateway Drift Integration

Layer / File(s)	Summary
Sandbox gateway state detection and propagation src/lib/actions/sandbox/gateway-state.ts	getSandboxGatewayState and getSandboxGatewayStateForStatus perform preflight and result validation; on issues return gateway_schema_mismatch with formatted output and propagate this through reconciliation and ensure-live logic.
Sandbox gateway state tests src/lib/actions/sandbox/gateway-state-drift.test.ts	Validates preflight early-exit, timeout threading, ensureLiveSandboxOrExit exit-on-mismatch without registry removal, and reconciliation propagation semantics.
Sandbox rebuild preflight and liveness validation src/lib/actions/sandbox/rebuild.ts, src/lib/actions/sandbox/rebuild-gateway-drift.test.ts	rebuildSandbox() aborts on preflight issues, reworks sandbox-list capture to validate result and non-zero status; tests assert exit(1), printed guidance, and avoided agent-version checks when drift detected.
Sandbox status inference RPC validation src/lib/actions/sandbox/status.ts	showSandboxStatus captures inference RPC without ignoreError, detects result issues, prints formatted issue and exits(1); handles gateway_schema_mismatch reconciler state by printing output and exiting.

Error Classification and Recovery

Layer / File(s)	Summary
Protobuf mismatch detection and classification src/lib/runtime-recovery.ts, src/lib/runtime-recovery.test.ts	Adds isOpenShellProtobufSchemaMismatch, skips mismatch lines when parsing live sandbox names, and classifies mismatch output as unavailable; tests added to validate parsing/classification behavior.

Supporting Infrastructure and Cleanup

Layer / File(s)	Summary
Installer Git detection and backup failure handling; onboard dedup scripts/install.sh, src/lib/onboard.ts	.git presence checks now accept file or directory; pre-upgrade backup-all failure is escalated to fatal with explicit remediation text; onboarding removes duplicate local drift logic and imports adapter helpers.
Install test helper and E2E harness test/install-openshell-upgrade-prompt.test.ts, test/e2e/test-gateway-drift-preflight.sh, test/e2e/docs/parity-inventory.generated.json	Adds openshellVersion option to install test helper and new test; refactors e2e fake openshell/docker scripts to be configurable, passes control env vars (including the new sentinel) into run cases, sets per-case TMPDIR, and refreshes generated parity assertions.

Sequence Diagram

sequenceDiagram
  participant CLI as NemoClaw CLI
  participant OpenShell as OpenShell runtime
  participant Docker as Docker engine
  CLI->>OpenShell: capture RPC (e.g. sandbox list / inference get)
  CLI->>Docker: inspect gateway cluster container (image, ports)
  Docker-->>CLI: container inspect JSON (image, published ports)
  OpenShell-->>CLI: RPC result { status, output }
  CLI->>CLI: detectOpenShellStateRpcPreflightIssue (uses Docker/OpenShell info)
  CLI->>CLI: detectOpenShellStateRpcResultIssue (analyze result output/status)
  CLI->>CLI: formatOpenShellStateRpcIssue -> printOpenShellStateRpcIssue (console.error)
  CLI-->>CLI: exit(1) on detected issue

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~75 minutes

Possibly related PRs

• NVIDIA/NemoClaw#3463: e2e regression guard and drift-preflight test adjustments that align with the new drift preflight/result validation logic.

Suggested reviewers

• jyaunches
• cv

"🐰 I nibble drift where versions roam,
I sniff protobuf wires and guide them home,
Tests snug in fields of Vitest light,
Preflight guards keep backups tight,
A tiny hop — the cluster sleeps tonight."

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 13.89% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'fix(cli): fail closed on OpenShell gateway drift' directly and concisely describes the main change: implementing fail-closed behavior when OpenShell gateway drift is detected, preventing the CLI from trusting stale RPC state.
Linked Issues check	✅ Passed	The PR successfully implements all key coding requirements from issue #3399: (1) Detects gateway image drift via new gateway-drift.ts helpers; (2) Detects protobuf schema mismatches; (3) Guards critical CLI flows (backup-all, upgrade-sandboxes, sandbox status, rebuild, recover) to fail closed on untrusted state; (4) Makes pre-upgrade backup fatal; (5) Adds comprehensive test coverage for drift detection and propagation.
Out of Scope Changes check	✅ Passed	All changes are directly scoped to addressing issue #3399: gateway drift/schema-mismatch detection and guarding CLI flows. Helper function signature updates (e.g., getInstalledOpenshellVersionOrNull timeout parameter) and installer logic changes (pre-upgrade backup fatality) are necessary supporting changes, not out-of-scope additions.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 ESLint

If the error stems from missing dependencies, add them to the package.json file. For unrecoverable errors (e.g., due to private dependencies), disable the tool in the CodeRabbit configuration.

ESLint skipped: no ESLint configuration detected in root package.json. To enable, add eslint to devDependencies.

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/adapters/openshell/gateway-drift.ts`:
- Around line 258-292: The user-visible guidance always says "preflight failed
before ${action}" and hardcodes the default gateway volume name
(DEFAULT_GATEWAY_NAME); update the message to be phase-aware and gateway-name
aware: change the opening line built around action to choose wording based on
whether the issue was detected during preflight versus at runtime (e.g. consult
issue.phase or issue.detectedAt / issue.kind === "protobuf_mismatch" to emit
"preflight failed before ${action}" vs "detected while ${action} was running"),
and replace the hardcoded DEFAULT_GATEWAY_NAME reference with the actual gateway
name variable used elsewhere in this module (use the gateway name variable
instead of DEFAULT_GATEWAY_NAME) so the recovery text and volume name reflect
custom gateways; keep all other context lines (drift, OpenShell output via
compactOutput, CLI_DISPLAY_NAME, CLI_NAME, command) unchanged.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 280a86e2-4450-4928-a7fe-c385c68c5fcf

📥 Commits

Reviewing files that changed from the base of the PR and between fc4a227 and 63e0f13.

📒 Files selected for processing (17)

• ci/env-var-doc-allowlist.json
• scripts/install.sh
• src/lib/actions/gateway-drift-preflight.test.ts
• src/lib/actions/maintenance.ts
• src/lib/actions/sandbox/gateway-state-drift.test.ts
• src/lib/actions/sandbox/gateway-state.ts
• src/lib/actions/sandbox/rebuild-gateway-drift.test.ts
• src/lib/actions/sandbox/rebuild.ts
• src/lib/actions/sandbox/status.ts
• src/lib/actions/upgrade-sandboxes.ts
• src/lib/adapters/openshell/gateway-drift.test.ts
• src/lib/adapters/openshell/gateway-drift.ts
• src/lib/adapters/openshell/runtime.ts
• src/lib/onboard.ts
• src/lib/runtime-recovery.test.ts
• src/lib/runtime-recovery.ts
• vitest.config.ts

[jyaunches]

Coverage guard landed; PR needs rebase/fix

Coverage guard PR #3463 has merged to main in regression-e2e.yaml (not scheduled nightly). It adds gateway-drift-preflight-e2e, which is now the acceptance criterion for this fix.

I merged current origin/main into a local preview of this branch and ran:

bash test/e2e/test-gateway-drift-preflight.sh

Result: the protobuf mismatch half now passes, but the patched-image drift half still fails:

=== Patched stale gateway image fails before sandbox list ===
PASS: backup-all exits non-zero on stale patched gateway image
FAIL: gateway image drift preflight is surfaced
  missing pattern: schema preflight failed|gateway schema preflight failed|image.*does not match|Running gateway image

Please rebase/merge main to pick up #3463, then update the gateway image parser so it recognizes patched local cluster images like:

nemoclaw-cluster:0.0.36-fuse-overlayfs-aa8b8487

as OpenShell cluster version 0.0.36, not only upstream refs like ghcr.io/nvidia/openshell/cluster:0.0.36. After that, dispatch the targeted guard:

gh workflow run regression-e2e.yaml -f jobs=gateway-drift-preflight-e2e --ref fix/3399-gateway-drift-preflight

Expected final state: gateway-drift-preflight-e2e flips green on this branch.

[jyaunches]

✅ Coverage guard verified

gateway-drift-preflight-e2e (added in #3463) is:

• 🔴 Failing on unfixed/main-equivalent code — run 25808832110
• 🟢 Passing on this branch's current head (66881d2f2) — run 25813711718

Note: GitHub Actions can only dispatch workflow_dispatch against refs in NVIDIA/NemoClaw, so the green run used temporary mirror branch pr-3423-verify-green, pushed from the PR branch head after merging main and applying the patched-image parser fix.

This PR now has a regression test proving the fix. The regression guard stays in regression-e2e.yaml for explicit dispatch and later nightly-promotion review.

[wscurran]

✨ Thanks for submitting this detailed PR to address the OpenShell gateway drift issue. This change aims to improve the robustness of the sandbox environment by failing closed when OpenShell sandbox-state RPCs may be schema-broken.

---

Related open PRs:

• #3463 test(e2e): add gateway drift preflight guard for #3423

---

Related open issues:

• #3399 [Ubuntu 24.04][Upgrade] v0.0.38 → v0.0.39 in-place upgrade breaks CLI ↔ cluster RPC with protobuf "invalid wire type" decode error