Skip to content

perf(test): reduce sandbox lifecycle subprocess isolation - #6280

Merged
cv merged 3 commits into
mainfrom
codex/perf-test-io-6245-batch-3
Jul 5, 2026
Merged

perf(test): reduce sandbox lifecycle subprocess isolation#6280
cv merged 3 commits into
mainfrom
codex/perf-test-io-6245-batch-3

Conversation

@cv

@cv cv commented Jul 5, 2026

Copy link
Copy Markdown
Collaborator

Summary

Reduce unnecessary process isolation across sandbox status, gateway reconciliation, connect recovery, and inference route repair tests while retaining representative CLI, timeout, listener, security, and cross-command contracts. The five affected process suites now launch 20 first-level processes instead of 101, with unchanged runtime behavior and no production-code changes.

Related Issue

Part of #6245

Changes

  • Move status routing and gateway lifecycle branches into direct public-dispatch, status-flow, and gateway-state tests while retaining CLI help/parser, unsafe-token, hanging-pipe timeout, and healthy-ordering contracts.
  • Move route-repair branches into the direct connect harness while retaining route-swap plumbing, local Ollama proxy/secret isolation, and WSL fallback contracts.
  • Move gateway reconciliation scenarios 1–12 into direct gateway lifecycle, status rendering, and skill liveness seams while retaining the real connectrebuild scenario 14 contract.
  • Move connect recovery branches into argv, Oclif adapter, registry recovery, and process-recovery seams while retaining successful and failed privileged-Docker recovery plus real session-backed registry recovery contracts.
  • Reset registry-recovery dependency mocks deterministically and cover requested-sandbox recovery through public dispatch under shuffled test order.
  • Extract focused status, connect-route, and seeded-registry test surfaces so existing large test files do not grow past the advisor policy.
  • Reduce outer CLI loaders from 83 to 14 (−83.1%) and explicit test-owned first-level processes from 101 to 20 (−80.2%).

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: all 23 changed paths are tests or test support; the final docs-writer review found no runtime command, flag, configuration, API, policy, or workflow change.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: independent local review covered assertion migration, retained process/security boundaries, mock and module-cache isolation, environment restoration, shuffled-order determinism, and the CI/advisor follow-up; all findings were fixed and re-reviewed with no remaining findings.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Verification

  • PR description includes the DCO sign-off declaration and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — 217/217 changed-file tests passed under shuffled order (seed 6245) in 39.76s; CLI typecheck, Biome, project-overlap, source-shape, test-size, conditional-growth, and monolith-growth checks passed.
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — npm test completed in 15m25.22s with 13,079 passing and 36 skipped tests. Seven mode assertions inherited the host's 077 umask and passed 68/68 under 022; one E2E source-hygiene failure is byte-identical on origin/main.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Benchmark

  • Same five process suites on origin/main: 76 tests in 57.10s.
  • Final branch: 13 retained process tests in 20.19s, 64.6% faster; migrated decision coverage remains in direct tests.
  • Outer CLI loaders: 83 → 14; explicit first-level processes including listeners: 101 → 20.
  • Final-head coverage artifacts measured the five migrated files at 32.42s versus 107.73s on main (−69.9%); aggregate blob execution fell 7.4% and summed shard job time fell 6.2%, while critical shard wall time was 8m39s versus 8m13s on main.
  • gateway-state-drift now executes its tests in 26ms, but its warmed source graph still costs 10.13s during collection and remains a shard-3 hotspot for the next batch.
  • The single full clean-build npm test run was 15m25.22s on Node 25.9.0 versus the issue's 14m19.65s reference, so this PR does not claim a repo-wide local wall-time win yet. The remaining untouched corpus still dominates and needs additional batches.

Signed-off-by: Carlos Villela cvillela@nvidia.com

Summary by CodeRabbit

  • Tests
    • Added and strengthened CLI and sandbox connect/recovery contract coverage, including safer failure behavior, deterministic guidance, and stricter validation of unsafe/privileged flows.
    • Expanded status and lifecycle classification scenarios (including inference and gateway-state edge cases) and tightened assertions around when sandbox teardown does or does not occur.
    • Introduced reusable status/connect-flow test harnesses and improved mocking/teardown isolation; added seeded registry recovery tests and additional timeout/probe-only orchestration checks.

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@coderabbitai

coderabbitai Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

This PR expands and tightens sandbox CLI test coverage across connect, gateway state, status, registry recovery, dispatch, and process recovery flows. It also updates shared test harnesses and removes some older assertions.

Changes

Sandbox test coverage expansion

Layer / File(s) Summary
Connect flow harness and route checks
test/support/connect-flow-test-harness.ts, src/lib/actions/sandbox/connect-flow.test.ts, src/lib/actions/sandbox/connect-route-lifecycle.test.ts, src/lib/actions/sandbox/connect-route-repair.test.ts, src/commands/sandbox/oclif-command-adapters.test.ts, src/lib/cli/argv-normalizer.test.ts, src/lib/actions/sandbox/skill-install.test.ts
The connect harness adds new spies/options and the connect suite gains coverage for route lifecycle/repair, permission-flag rejection, help normalization, and a skill-install liveness guard.
Gateway state drift and hints
src/lib/actions/sandbox/gateway-state-drift.test.ts, src/lib/actions/sandbox/gateway-state-hints.test.ts, src/lib/gateway-runtime-action.test.ts
Gateway state tests add docker-driver recovery mocking, lifecycle classification cases, ANSI hint parsing, and guidance/output assertions for drift and unreachable-gateway states.
Status harness and flow assertions
test/support/status-flow-test-harness.ts, src/lib/actions/sandbox/status-flow.test.ts, test/cli/status-gateway-lifecycle.test.ts, test/cli/status-routing.test.ts, test/cli-oclif-compatibility.test.ts
The shared status harness is extracted and status-flow tests add broader rendering and guidance coverage for wrong gateway selection, outages, shields posture, version precedence, and verification failures.
Registry recovery and seeded paths
src/lib/registry-recovery-action.test.ts, src/lib/registry-recovery-seeded-paths.test.ts
Registry recovery tests add a shared dependency-reset helper, seeded recovery path coverage, requested-sandbox handling, and a removed phantom-session case.
CLI recovery and state reconcile contracts
test/cli/connect-recovery.test.ts, test/gateway-state-reconcile-2276.test.ts
Connect recovery assertions are tightened and renamed, and the gateway reconcile test removes call-log plumbing while adding a stale-state recovery scenario that carries preserved state from connect into rebuild.
Process recovery and timeout checks
test/process-recovery-managed-controller.test.ts, test/process-recovery-primitives.test.ts, test/process-recovery.test.ts
Process recovery tests add spawned-command capture, SSH-absence assertions, sandbox-exec timeout handling, and a forward-start absence check.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related PRs

Suggested labels: chore

Suggested reviewers: jyaunches

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the PR’s main change: reducing sandbox lifecycle test subprocess isolation for performance.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/perf-test-io-6245-batch-3

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in the codex/perf-test-io-6... branch is 96%. Coverage data for the main branch is not yet available.

Show a code coverage summary of the most covered files.
File main codex/perf-test-io-6... cab00c6 +/-
nemoclaw/src/se...cret-scanner.ts 100%
nemoclaw/src/commands/slash.ts 100%
nemoclaw/src/li...bprocess-env.ts 100%
nemoclaw/src/bl...eprint/state.ts 98%
nemoclaw/src/onboard/config.ts 98%
nemoclaw/src/bl...int/snapshot.ts 97%
nemoclaw/src/bl...print/runner.ts 95%
nemoclaw/src/co...ration-state.ts 94%
nemoclaw/src/bl...ate-networks.ts 94%
nemoclaw/src/index.ts 94%

TypeScript / code-coverage/cli

The overall coverage in the codex/perf-test-io-6... branch is 71%. Coverage data for the main branch is not yet available.

Show a code coverage summary of the most covered files.
File main codex/perf-test-io-6... cab00c6 +/-
src/lib/shields...nsition-lock.ts 87%
src/lib/actions...all/run-plan.ts 81%
src/lib/state/o...oard-session.ts 81%
src/lib/onboard/preflight.ts 71%
src/lib/state/sandbox.ts 71%
src/lib/onboard...er-gpu-patch.ts 69%
src/lib/shields/index.ts 68%
src/lib/policy/index.ts 66%
src/lib/actions...licy-channel.ts 60%
src/lib/onboard.ts 24%

Updated July 05, 2026 01:54 UTC
Code Coverage is in Public Preview. Learn more and provide us with your feedback.

@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

E2E Advisor Recommendation

Required E2E: None
Optional E2E: None

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

  • None. No merge-blocking E2E is recommended because this PR is tests-only: every changed file is a test spec or test support harness, with no production installer, onboarding, sandbox lifecycle, credentials, security, network policy, inference routing, deployment, or user-flow implementation changes.

Optional E2E

  • None.

New E2E recommendations

  • None.

@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

E2E Target Recommendation

Required E2E targets: None
Optional E2E targets: None

Workflow run

Full E2E target advisor summary

E2E Target Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E targets

  • None. This PR changes unit/integration test files and non-E2E test support outside test/e2e, with no changes to the E2E target registry, e2e.yaml, live E2E files, test/e2e fixtures/support, or target-relevant production paths that require a NemoClaw E2E target dispatch.

Optional E2E targets

  • None.

Relevant changed files

  • None.

@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor (Nemotron Ultra) — Changes requested

Merge posture: Do not merge yet
Primary next action: Fix PRA-3: Mock drift risk: dispatch harnesses mock 5+ modules without integration smoke test; then add or justify PRA-T1.
Open items: 4 required · 3 warnings · 8 suggestions · 7 test follow-ups
Since last review: 0 prior items resolved · 5 still apply · 8 new items found

Action checklist

  • PRA-3 Fix: Mock drift risk: dispatch harnesses mock 5+ modules without integration smoke test in test/cli-oclif-compatibility.test.ts:300
  • PRA-4 Fix: Missing env isolation regression test for NEMOCLAW_FORCE_VM_DNS_MONKEYPATCH in src/lib/actions/sandbox/connect-route-lifecycle.test.ts:83
  • PRA-5 Fix: E2E→unit security assertion mapping comment missing in src/lib/actions/sandbox/connect-flow.test.ts:1
  • PRA-6 Fix: Overlapping PR fix(mcp): reconcile Hermes runtime state #6261 modifies same test harness (merge conflict risk) in test/support/connect-flow-test-harness.ts:1
  • PRA-1 Resolve or justify: Source-of-truth review needed: NEMOCLAW_FORCE_VM_DNS_MONKEYPATCH env monkeypatch for VM DNS
  • PRA-2 Resolve or justify: Source-of-truth review needed: E2E security assertions mapped to unit tests
  • PRA-7 Resolve or justify: Source-of-truth workaround: validateName filters invalid sandbox names from external gateway in src/lib/registry-recovery-seeded-paths.test.ts:58
  • PRA-T1 Add or justify test follow-up: Missing env isolation regression test for NEMOCLAW_FORCE_VM_DNS_MONKEYPATCH
  • PRA-T2 Add or justify test follow-up: Expand malformed gateway output classification cases
  • PRA-T3 Add or justify test follow-up: Integration smoke test for real dispatchCli with real recovery modules
  • PRA-T4 Add or justify test follow-up: Acceptance clause
  • PRA-T5 Add or justify test follow-up: validateName filters invalid sandbox names from external gateway
  • PRA-T6 Add or justify test follow-up: NEMOCLAW_FORCE_VM_DNS_MONKEYPATCH env monkeypatch for VM DNS
  • PRA-T7 Add or justify test follow-up: E2E security assertions mapped to unit tests
  • PRA-8 In-scope improvement: Expand malformed gateway output classification cases in src/lib/gateway-runtime-action.test.ts:100
  • PRA-9 In-scope improvement: Centralize module cache isolation utilities in test/support/connect-flow-test-harness.ts:1
  • PRA-10 In-scope improvement: Security regression test maintained: rejects removed --dangerously-skip-permissions flag in src/commands/sandbox/oclif-command-adapters.test.ts:65
  • PRA-11 In-scope improvement: Registry preservation tests prevent data loss on gateway drift in src/lib/actions/sandbox/gateway-state-drift.test.ts:70
  • PRA-12 In-scope improvement: Identity drift detection hardens gateway trust boundary in src/lib/actions/sandbox/gateway-state-hints.test.ts:100
  • PRA-13 In-scope improvement: Hermes agent preservation prevents state path corruption in src/lib/registry-recovery-action.test.ts:50
  • PRA-14 In-scope improvement: Integration smoke test for real dispatchCli with real recovery modules in test/cli-oclif-compatibility.test.ts:384
  • PRA-15 In-scope improvement: Replace single-use NEMOCLAW_TEST_NO_SLEEP with vi.useFakeTimers in src/lib/actions/sandbox/connect-route-lifecycle.test.ts:17

Findings index

ID Severity Category Location Required action
PRA-1 Resolve/justify architecture Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
PRA-2 Resolve/justify architecture Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
PRA-3 Required correctness test/cli-oclif-compatibility.test.ts:300 Add one integration smoke test in cli-oclif-compatibility.test.ts using real require (no require.cache mocking for registry/registry-recovery) that calls dispatchCli(['alpha', 'connect']) and verifies connect dispatched after recovery. Keep existing harnesses for unit speed.
PRA-4 Required tests src/lib/actions/sandbox/connect-route-lifecycle.test.ts:83 Add a test immediately after the monkeypatch test that asserts process.env.NEMOCLAW_FORCE_VM_DNS_MONKEYPATCH === undefined.
PRA-5 Required scope src/lib/actions/sandbox/connect-flow.test.ts:1 Add header comment in connect-flow.test.ts mapping: LD_PRELOAD sanitization → gateway-runtime-action.test.ts 'keeps probes fatal' test; SSH not invoked → connect-flow.test.ts 'probe-only mode reports recovered gateways'; docker recovery env vars → connect-flow.test.ts 'wires forced VM DNS monkeypatch'.
PRA-6 Required scope test/support/connect-flow-test-harness.ts:1 Coordinate with PR #6261 author. Review both PRs' changes to connect-flow-test-harness.ts together. Ensure Hermes agent recovery logic is consistent.
PRA-7 Resolve/justify architecture src/lib/registry-recovery-seeded-paths.test.ts:58 Accept workaround. Add code comment documenting when this can be removed (when OpenShell validates names upstream). Test provides regression coverage.
PRA-8 Improvement tests src/lib/gateway-runtime-action.test.ts:100 Add 2-3 more parameterized cases: ANSI-decorated connected output, timeout stderr, permission denied.
PRA-9 Improvement architecture test/support/connect-flow-test-harness.ts:1 Create test/support/module-cache.ts with withFreshModule and withMockedModules helpers. Refactor gateway-state-hints.test.ts and cli-oclif-compatibility.test.ts to use them.
PRA-10 Improvement security src/commands/sandbox/oclif-command-adapters.test.ts:65 Keep this test as-is. It correctly validates the security boundary.
PRA-11 Improvement security src/lib/actions/sandbox/gateway-state-drift.test.ts:70 Maintain these tests. They encode the critical invariant for gateway drift handling.
PRA-12 Improvement security src/lib/actions/sandbox/gateway-state-hints.test.ts:100 Keep identity drift tests. They detect the critical case where gateway trust material rotated after restart.
PRA-13 Improvement security src/lib/registry-recovery-action.test.ts:50 Maintain agent preservation tests. They prevent rebuild-time agent resolution bugs.
PRA-14 Improvement tests test/cli-oclif-compatibility.test.ts:384 Add test using real require for registry/recovery that calls dispatchCli(['alpha', 'connect']) and verifies connect dispatched after recovery.
PRA-15 Improvement architecture src/lib/actions/sandbox/connect-route-lifecycle.test.ts:17 Replace with vi.useFakeTimers() in beforeEach and vi.useRealTimers() in afterEach, or remove if sleeps are already mocked.

🚨 Required before merge

Address these before merging unless a maintainer explicitly overrides the advisor with rationale.

PRA-3 Required — Mock drift risk: dispatch harnesses mock 5+ modules without integration smoke test

  • Location: test/cli-oclif-compatibility.test.ts:300
  • Category: correctness
  • Problem: withDirectStatusDispatch and withDirectSandboxRecoveryDispatch harnesses mock public-dispatch, oclif-runner, sandbox-connect, registry, and registry-recovery modules. No test exercises real dispatchCli with real registry/recovery modules, creating drift risk between mocked unit behavior and real integration behavior.
  • Impact: Unit tests may pass while real CLI dispatch fails due to module interaction bugs not caught by mocks.
  • Required action: Add one integration smoke test in cli-oclif-compatibility.test.ts using real require (no require.cache mocking for registry/registry-recovery) that calls dispatchCli(['alpha', 'connect']) and verifies connect dispatched after recovery. Keep existing harnesses for unit speed.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: Search cli-oclif-compatibility.test.ts for any test that imports real registry/recovery modules without require.cache mocking
  • Missing regression test: Add test: 'dispatches connect after real registry recovery' using real require for registry/recovery modules
  • Done when: The required change is committed and verification passes: Search cli-oclif-compatibility.test.ts for any test that imports real registry/recovery modules without require.cache mocking.
  • Evidence: withDirectStatusDispatch and withDirectSandboxRecoveryDispatch mock 5 modules each; no test uses real modules end-to-end

PRA-4 Required — Missing env isolation regression test for NEMOCLAW_FORCE_VM_DNS_MONKEYPATCH

  • Location: src/lib/actions/sandbox/connect-route-lifecycle.test.ts:83
  • Category: tests
  • Problem: Test 'wires the forced VM DNS monkeypatch into connect route repair' uses vi.stubEnv('NEMOCLAW_FORCE_VM_DNS_MONKEYPATCH', '1') with try/finally vi.unstubAllEnvs() but no test asserts process.env.NEMOCLAW_FORCE_VM_DNS_MONKEYPATCH === undefined after test completes.
  • Impact: Env leak between tests could cause flaky failures or mask real env-dependent bugs.
  • Required action: Add a test immediately after the monkeypatch test that asserts process.env.NEMOCLAW_FORCE_VM_DNS_MONKEYPATCH === undefined.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: grep -n 'NEMOCLAW_FORCE_VM_DNS_MONKEYPATCH' src/lib/actions/sandbox/connect-route-lifecycle.test.ts src/lib/actions/sandbox/connect-flow.test.ts
  • Missing regression test: Add test 'cleans up NEMOCLAW_FORCE_VM_DNS_MONKEYPATCH env after monkeypatch test' asserting env var is undefined
  • Done when: The required change is committed and verification passes: grep -n 'NEMOCLAW_FORCE_VM_DNS_MONKEYPATCH' src/lib/actions/sandbox/connect-route-lifecycle.test.ts src/lib/actions/sandbox/connect-flow.test.ts.
  • Evidence: Test at line 83-109 uses vi.stubEnv with try/finally vi.unstubAllEnvs() but no post-test assertion

PRA-5 Required — E2E→unit security assertion mapping comment missing

  • Location: src/lib/actions/sandbox/connect-flow.test.ts:1
  • Category: scope
  • Problem: connect-recovery.test.ts reduced from 1350 to 482 lines (E2E→unit migration). Security assertions from E2E (LD_PRELOAD sanitization, SSH avoidance, docker recovery env vars) should be explicitly mapped to their new unit test locations.
  • Impact: Future maintainers may not know which unit tests cover which security properties, risking accidental removal.
  • Required action: Add header comment in connect-flow.test.ts mapping: LD_PRELOAD sanitization → gateway-runtime-action.test.ts 'keeps probes fatal' test; SSH not invoked → connect-flow.test.ts 'probe-only mode reports recovered gateways'; docker recovery env vars → connect-flow.test.ts 'wires forced VM DNS monkeypatch'.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: Check connect-flow.test.ts header for mapping comment; grep gateway-runtime-action.test.ts for 'keeps probes fatal'
  • Missing regression test: Add header comment in connect-flow.test.ts with E2E→unit security assertion mapping
  • Done when: The required change is committed and verification passes: Check connect-flow.test.ts header for mapping comment; grep gateway-runtime-action.test.ts for 'keeps probes fatal'.
  • Evidence: PR body claims security assertions retained; connect-recovery.test.ts reduced 80%; no mapping comment exists

PRA-6 Required — Overlapping PR #6261 modifies same test harness (merge conflict risk)

  • Location: test/support/connect-flow-test-harness.ts:1
  • Category: scope
  • Problem: PR fix(mcp): reconcile Hermes runtime state #6261 (fix(mcp): reconcile Hermes runtime state) also modifies test/support/connect-flow-test-harness.ts. This PR changes 39 lines in that file. Behavioral overlap on Hermes agent handling creates merge conflict risk and potential inconsistency.
  • Impact: Merge conflicts or silent behavioral divergence in Hermes agent recovery logic between PRs.
  • Required action: Coordinate with PR fix(mcp): reconcile Hermes runtime state #6261 author. Review both PRs' changes to connect-flow-test-harness.ts together. Ensure Hermes agent recovery logic is consistent.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: Compare diff of test/support/connect-flow-test-harness.ts in this PR vs PR fix(mcp): reconcile Hermes runtime state #6261
  • Missing regression test: N/A — coordination required
  • Done when: The required change is committed and verification passes: Compare diff of test/support/connect-flow-test-harness.ts in this PR vs PR fix(mcp): reconcile Hermes runtime state #6261.
  • Evidence: Both PRs modify connect-flow-test-harness.ts; openPrOverlaps shows sameFiles includes this file
Review findings by urgency: 4 required fixes, 3 items to resolve/justify, 8 in-scope improvements

⚠️ Resolve or justify before merge

Investigate these in the current review; either fix them, explain why they are not applicable, or document the accepted risk.

PRA-1 Resolve/justify — Source-of-truth review needed: NEMOCLAW_FORCE_VM_DNS_MONKEYPATCH env monkeypatch for VM DNS

  • Location: not file-specific
  • Category: architecture
  • Problem: The advisor marked localized patch analysis as missing.
  • Impact: A localized workaround can preserve or hide an invalid state when the source boundary is unclear.
  • Recommended action: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Missing regression test: Test 'wires the forced VM DNS monkeypatch into connect route repair' at connect-route-lifecycle.test.ts:83; missing env cleanup assertion test
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Evidence: vi.stubEnv at line 83 with try/finally cleanup but no post-test assertion

PRA-2 Resolve/justify — Source-of-truth review needed: E2E security assertions mapped to unit tests

  • Location: not file-specific
  • Category: architecture
  • Problem: The advisor marked localized patch analysis as missing.
  • Impact: A localized workaround can preserve or hide an invalid state when the source boundary is unclear.
  • Recommended action: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Missing regression test: Missing header comment in connect-flow.test.ts mapping security assertions to unit test locations
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Evidence: No header comment in connect-flow.test.ts mapping LD_PRELOAD → gateway-runtime-action.test.ts, SSH → connect-flow.test.ts, docker env vars → connect-route-lifecycle.test.ts

PRA-7 Resolve/justify — Source-of-truth workaround: validateName filters invalid sandbox names from external gateway

  • Location: src/lib/registry-recovery-seeded-paths.test.ts:58
  • Category: architecture
  • Problem: Test 'skips invalid session and live sandbox names during seeded recovery' validates sandbox names via validateName regex because OpenShell gateway doesn't validate names on creation. Test provides regression coverage but no code comment documents when this workaround can be removed.
  • Impact: Workaround may persist indefinitely without clear removal condition, masking upstream fix opportunity.
  • Recommended action: Accept workaround. Add code comment documenting when this can be removed (when OpenShell validates names upstream). Test provides regression coverage.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Check registry-recovery-seeded-paths.test.ts for comment documenting removal condition
  • Missing regression test: N/A — test already exists as regression coverage
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Check registry-recovery-seeded-paths.test.ts for comment documenting removal condition.
  • Evidence: Test at line 133 validates names 'Alpha' and 'Bad_Name' are skipped; validateName imported from runner.js at line 58

💡 In-scope improvements

These are lower-risk, not throwaway. Prefer fixing them in this PR when they are local to changed code; defer only with rationale or a linked follow-up.

PRA-8 Improvement — Expand malformed gateway output classification cases

  • Location: src/lib/gateway-runtime-action.test.ts:100
  • Category: tests
  • Problem: PR added 3 new parameterized cases (failed gateway metadata, empty output, malformed output) but more edge cases would improve robustness of getNamedGatewayLifecycleState classification.
  • Impact: Edge case gateway output could be misclassified, affecting drift detection and recovery decisions.
  • Suggested action: Add 2-3 more parameterized cases: ANSI-decorated connected output, timeout stderr, permission denied.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Review gateway-runtime-action.test.ts getNamedGatewayLifecycleState describe block for classify cases
  • Missing regression test: Add parameterized test cases for ANSI-decorated output, timeout stderr, permission denied
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: Existing it.each block at line 100 has 3 cases; room for more edge cases

PRA-9 Improvement — Centralize module cache isolation utilities

  • Location: test/support/connect-flow-test-harness.ts:1
  • Category: architecture
  • Problem: Both cli-oclif-compatibility.test.ts harnesses (withDirectStatusDispatch, withDirectSandboxRecoveryDispatch) and status-flow-test-harness.ts duplicate require.cache manipulation logic.
  • Impact: Duplicated cache manipulation increases maintenance burden and risk of inconsistent restoration.
  • Suggested action: Create test/support/module-cache.ts with withFreshModule and withMockedModules helpers. Refactor gateway-state-hints.test.ts and cli-oclif-compatibility.test.ts to use them.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Compare cache manipulation patterns in cli-oclif-compatibility.test.ts, status-flow-test-harness.ts, gateway-state-hints.test.ts
  • Missing regression test: N/A — refactoring only
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: cli-oclif-compatibility.test.ts lines 36-155, gateway-state-hints.test.ts lines 21,48, status-flow-test-harness.ts all manipulate require.cache

PRA-10 Improvement — Security regression test maintained: rejects removed --dangerously-skip-permissions flag

  • Location: src/commands/sandbox/oclif-command-adapters.test.ts:65
  • Category: security
  • Problem: Test 'rejects the removed connect permission bypass before dispatch' correctly validates the security boundary by rejecting the removed flag before any action dispatch.
  • Impact: Confirms security boundary holds; prevents regression of dangerous permission bypass.
  • Suggested action: Keep this test as-is. It correctly validates the security boundary.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Verify test 'rejects the removed connect permission bypass before dispatch' exists and passes
  • Missing regression test: N/A — test already present
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: Test added in this PR at oclif-command-adapters.test.ts:65-100

PRA-11 Improvement — Registry preservation tests prevent data loss on gateway drift

  • Location: src/lib/actions/sandbox/gateway-state-drift.test.ts:70
  • Category: security
  • Problem: Tests 'preserves a local registry entry when a healthy named gateway still lacks the sandbox' and 'preserves registry state when gateway selection cannot expose the sandbox' encode critical invariant for gateway drift handling.
  • Impact: Prevents accidental registry deletion when gateway state diverges from local registry.
  • Suggested action: Maintain these tests. They encode the critical invariant for gateway drift handling.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Verify tests for gateway drift registry preservation exist and pass
  • Missing regression test: N/A — tests already present
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: gateway-state-drift.test.ts lines 150-190 test registry preservation on drift

PRA-12 Improvement — Identity drift detection hardens gateway trust boundary

  • Location: src/lib/actions/sandbox/gateway-state-hints.test.ts:100
  • Category: security
  • Problem: Tests for 'identity_drift' state (handshake verification failed after recovery) detect critical case where gateway trust material rotated after restart.
  • Impact: Prevents unsafe reattachment to gateway with rotated trust material, which could expose sandbox to wrong gateway.
  • Suggested action: Keep identity drift tests. They detect the critical case where gateway trust material rotated after restart.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Verify gateway-state-hints.test.ts has 'classifies a failed post-recovery handshake as identity drift' test
  • Missing regression test: N/A — tests already present
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: gateway-state-hints.test.ts line 114 tests identity_drift classification

PRA-13 Improvement — Hermes agent preservation prevents state path corruption

  • Location: src/lib/registry-recovery-action.test.ts:50
  • Category: security
  • Problem: Tests ensure persisted 'agent: hermes' not clobbered by session re-seed, preventing rebuild-time agent resolution bugs (/sandbox/.hermes-data vs /sandbox/.openclaw-data).
  • Impact: Prevents agent type corruption that would cause wrong state paths and data loss on rebuild.
  • Suggested action: Maintain agent preservation tests. They prevent rebuild-time agent resolution bugs.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Verify registry-recovery-action.test.ts has 'preserves a persisted Hermes agent when session re-seeds same sandbox' and 'does not clobber persisted agent when session metadata omits it' tests
  • Missing regression test: N/A — tests already present
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: registry-recovery-action.test.ts lines 50-90 test Hermes agent preservation

PRA-14 Improvement — Integration smoke test for real dispatchCli with real recovery modules

  • Location: test/cli-oclif-compatibility.test.ts:384
  • Category: tests
  • Problem: Same as PRA-1: integration smoke test using real require for registry/recovery modules still missing.
  • Impact: Mock drift risk remains unmitigated without real-module integration test.
  • Suggested action: Add test using real require for registry/recovery that calls dispatchCli(['alpha', 'connect']) and verifies connect dispatched after recovery.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Search cli-oclif-compatibility.test.ts for any test that imports real registry/recovery modules without require.cache mocking
  • Missing regression test: Add test: 'dispatches connect after real registry recovery' using real require for registry/recovery modules
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: Harnesses mock 5 modules each; no real-module integration test exists

PRA-15 Improvement — Replace single-use NEMOCLAW_TEST_NO_SLEEP with vi.useFakeTimers

  • Location: src/lib/actions/sandbox/connect-route-lifecycle.test.ts:17
  • Category: architecture
  • Problem: NEMOCLAW_TEST_NO_SLEEP env var is set in beforeEach and deleted in afterEach but only used to speed up tests. No user/CI sets this externally.
  • Impact: Unnecessary env var manipulation; fake timers are more idiomatic in Vitest.
  • Suggested action: Replace with vi.useFakeTimers() in beforeEach and vi.useRealTimers() in afterEach, or remove if sleeps are already mocked.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Check if connectSandbox or its dependencies use setTimeout that would need fake timers
  • Missing regression test: N/A — test infrastructure only
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: Lines 17 and 31 set/delete NEMOCLAW_TEST_NO_SLEEP; no other references in file
Simplification opportunities: 1 possible cut, net -2 lines possible

These are safe simplification checks only. Do not remove validation, security controls, data-loss prevention, or required tests.

  • PRA-15 stdlib (src/lib/actions/sandbox/connect-route-lifecycle.test.ts:17): process.env.NEMOCLAW_TEST_NO_SLEEP = "1" and delete process.env.NEMOCLAW_TEST_NO_SLEEP
    • Replacement: vi.useFakeTimers() / vi.useRealTimers() or remove if sleeps already mocked
    • Net: -2 lines
    • Safety boundary: Must not affect real timing-dependent behavior in connectSandbox
Test follow-ups to resolve or justify

If these cover changed behavior, prefer adding them in this PR; otherwise state why existing coverage is enough or link the follow-up.

  • PRA-T1 Missing env isolation regression test for NEMOCLAW_FORCE_VM_DNS_MONKEYPATCH — Add a test immediately after the monkeypatch test that asserts process.env.NEMOCLAW_FORCE_VM_DNS_MONKEYPATCH === undefined.
  • PRA-T2 Expand malformed gateway output classification cases — Add 2-3 more parameterized cases: ANSI-decorated connected output, timeout stderr, permission denied.
  • PRA-T3 Integration smoke test for real dispatchCli with real recovery modules — Add test using real require for registry/recovery that calls dispatchCli(['alpha', 'connect']) and verifies connect dispatched after recovery.
  • PRA-T4 Acceptance clause — address lifecycle isolation review findings — add test evidence or identify existing coverage. PRA-1, PRA-2, PRA-3, PRA-4, PRA-5 from previous review partially addressed; PRA-1, PRA-2, PRA-3, PRA-4 remain open
  • PRA-T5 validateName filters invalid sandbox names from external gateway — Test 'skips invalid session and live sandbox names during seeded recovery' at registry-recovery-seeded-paths.test.ts:133. Test validates names 'Alpha' and 'Bad_Name' are skipped; validateName imported from runner.js
  • PRA-T6 NEMOCLAW_FORCE_VM_DNS_MONKEYPATCH env monkeypatch for VM DNS — Test 'wires the forced VM DNS monkeypatch into connect route repair' at connect-route-lifecycle.test.ts:83; missing env cleanup assertion test. vi.stubEnv at line 83 with try/finally cleanup but no post-test assertion
  • PRA-T7 E2E security assertions mapped to unit tests — Missing header comment in connect-flow.test.ts mapping security assertions to unit test locations. No header comment in connect-flow.test.ts mapping LD_PRELOAD → gateway-runtime-action.test.ts, SSH → connect-flow.test.ts, docker env vars → connect-route-lifecycle.test.ts
Since last review details

Current findings, using the urgency labels above:

PRA-1 Resolve/justify — Source-of-truth review needed: NEMOCLAW_FORCE_VM_DNS_MONKEYPATCH env monkeypatch for VM DNS

  • Location: not file-specific
  • Category: architecture
  • Problem: The advisor marked localized patch analysis as missing.
  • Impact: A localized workaround can preserve or hide an invalid state when the source boundary is unclear.
  • Recommended action: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Missing regression test: Test 'wires the forced VM DNS monkeypatch into connect route repair' at connect-route-lifecycle.test.ts:83; missing env cleanup assertion test
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Evidence: vi.stubEnv at line 83 with try/finally cleanup but no post-test assertion

PRA-2 Resolve/justify — Source-of-truth review needed: E2E security assertions mapped to unit tests

  • Location: not file-specific
  • Category: architecture
  • Problem: The advisor marked localized patch analysis as missing.
  • Impact: A localized workaround can preserve or hide an invalid state when the source boundary is unclear.
  • Recommended action: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Missing regression test: Missing header comment in connect-flow.test.ts mapping security assertions to unit test locations
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Evidence: No header comment in connect-flow.test.ts mapping LD_PRELOAD → gateway-runtime-action.test.ts, SSH → connect-flow.test.ts, docker env vars → connect-route-lifecycle.test.ts

PRA-3 Required — Mock drift risk: dispatch harnesses mock 5+ modules without integration smoke test

  • Location: test/cli-oclif-compatibility.test.ts:300
  • Category: correctness
  • Problem: withDirectStatusDispatch and withDirectSandboxRecoveryDispatch harnesses mock public-dispatch, oclif-runner, sandbox-connect, registry, and registry-recovery modules. No test exercises real dispatchCli with real registry/recovery modules, creating drift risk between mocked unit behavior and real integration behavior.
  • Impact: Unit tests may pass while real CLI dispatch fails due to module interaction bugs not caught by mocks.
  • Required action: Add one integration smoke test in cli-oclif-compatibility.test.ts using real require (no require.cache mocking for registry/registry-recovery) that calls dispatchCli(['alpha', 'connect']) and verifies connect dispatched after recovery. Keep existing harnesses for unit speed.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: Search cli-oclif-compatibility.test.ts for any test that imports real registry/recovery modules without require.cache mocking
  • Missing regression test: Add test: 'dispatches connect after real registry recovery' using real require for registry/recovery modules
  • Done when: The required change is committed and verification passes: Search cli-oclif-compatibility.test.ts for any test that imports real registry/recovery modules without require.cache mocking.
  • Evidence: withDirectStatusDispatch and withDirectSandboxRecoveryDispatch mock 5 modules each; no test uses real modules end-to-end

PRA-4 Required — Missing env isolation regression test for NEMOCLAW_FORCE_VM_DNS_MONKEYPATCH

  • Location: src/lib/actions/sandbox/connect-route-lifecycle.test.ts:83
  • Category: tests
  • Problem: Test 'wires the forced VM DNS monkeypatch into connect route repair' uses vi.stubEnv('NEMOCLAW_FORCE_VM_DNS_MONKEYPATCH', '1') with try/finally vi.unstubAllEnvs() but no test asserts process.env.NEMOCLAW_FORCE_VM_DNS_MONKEYPATCH === undefined after test completes.
  • Impact: Env leak between tests could cause flaky failures or mask real env-dependent bugs.
  • Required action: Add a test immediately after the monkeypatch test that asserts process.env.NEMOCLAW_FORCE_VM_DNS_MONKEYPATCH === undefined.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: grep -n 'NEMOCLAW_FORCE_VM_DNS_MONKEYPATCH' src/lib/actions/sandbox/connect-route-lifecycle.test.ts src/lib/actions/sandbox/connect-flow.test.ts
  • Missing regression test: Add test 'cleans up NEMOCLAW_FORCE_VM_DNS_MONKEYPATCH env after monkeypatch test' asserting env var is undefined
  • Done when: The required change is committed and verification passes: grep -n 'NEMOCLAW_FORCE_VM_DNS_MONKEYPATCH' src/lib/actions/sandbox/connect-route-lifecycle.test.ts src/lib/actions/sandbox/connect-flow.test.ts.
  • Evidence: Test at line 83-109 uses vi.stubEnv with try/finally vi.unstubAllEnvs() but no post-test assertion

PRA-5 Required — E2E→unit security assertion mapping comment missing

  • Location: src/lib/actions/sandbox/connect-flow.test.ts:1
  • Category: scope
  • Problem: connect-recovery.test.ts reduced from 1350 to 482 lines (E2E→unit migration). Security assertions from E2E (LD_PRELOAD sanitization, SSH avoidance, docker recovery env vars) should be explicitly mapped to their new unit test locations.
  • Impact: Future maintainers may not know which unit tests cover which security properties, risking accidental removal.
  • Required action: Add header comment in connect-flow.test.ts mapping: LD_PRELOAD sanitization → gateway-runtime-action.test.ts 'keeps probes fatal' test; SSH not invoked → connect-flow.test.ts 'probe-only mode reports recovered gateways'; docker recovery env vars → connect-flow.test.ts 'wires forced VM DNS monkeypatch'.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: Check connect-flow.test.ts header for mapping comment; grep gateway-runtime-action.test.ts for 'keeps probes fatal'
  • Missing regression test: Add header comment in connect-flow.test.ts with E2E→unit security assertion mapping
  • Done when: The required change is committed and verification passes: Check connect-flow.test.ts header for mapping comment; grep gateway-runtime-action.test.ts for 'keeps probes fatal'.
  • Evidence: PR body claims security assertions retained; connect-recovery.test.ts reduced 80%; no mapping comment exists

PRA-6 Required — Overlapping PR #6261 modifies same test harness (merge conflict risk)

  • Location: test/support/connect-flow-test-harness.ts:1
  • Category: scope
  • Problem: PR fix(mcp): reconcile Hermes runtime state #6261 (fix(mcp): reconcile Hermes runtime state) also modifies test/support/connect-flow-test-harness.ts. This PR changes 39 lines in that file. Behavioral overlap on Hermes agent handling creates merge conflict risk and potential inconsistency.
  • Impact: Merge conflicts or silent behavioral divergence in Hermes agent recovery logic between PRs.
  • Required action: Coordinate with PR fix(mcp): reconcile Hermes runtime state #6261 author. Review both PRs' changes to connect-flow-test-harness.ts together. Ensure Hermes agent recovery logic is consistent.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: Compare diff of test/support/connect-flow-test-harness.ts in this PR vs PR fix(mcp): reconcile Hermes runtime state #6261
  • Missing regression test: N/A — coordination required
  • Done when: The required change is committed and verification passes: Compare diff of test/support/connect-flow-test-harness.ts in this PR vs PR fix(mcp): reconcile Hermes runtime state #6261.
  • Evidence: Both PRs modify connect-flow-test-harness.ts; openPrOverlaps shows sameFiles includes this file

PRA-7 Resolve/justify — Source-of-truth workaround: validateName filters invalid sandbox names from external gateway

  • Location: src/lib/registry-recovery-seeded-paths.test.ts:58
  • Category: architecture
  • Problem: Test 'skips invalid session and live sandbox names during seeded recovery' validates sandbox names via validateName regex because OpenShell gateway doesn't validate names on creation. Test provides regression coverage but no code comment documents when this workaround can be removed.
  • Impact: Workaround may persist indefinitely without clear removal condition, masking upstream fix opportunity.
  • Recommended action: Accept workaround. Add code comment documenting when this can be removed (when OpenShell validates names upstream). Test provides regression coverage.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Check registry-recovery-seeded-paths.test.ts for comment documenting removal condition
  • Missing regression test: N/A — test already exists as regression coverage
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Check registry-recovery-seeded-paths.test.ts for comment documenting removal condition.
  • Evidence: Test at line 133 validates names 'Alpha' and 'Bad_Name' are skipped; validateName imported from runner.js at line 58

PRA-8 Improvement — Expand malformed gateway output classification cases

  • Location: src/lib/gateway-runtime-action.test.ts:100
  • Category: tests
  • Problem: PR added 3 new parameterized cases (failed gateway metadata, empty output, malformed output) but more edge cases would improve robustness of getNamedGatewayLifecycleState classification.
  • Impact: Edge case gateway output could be misclassified, affecting drift detection and recovery decisions.
  • Suggested action: Add 2-3 more parameterized cases: ANSI-decorated connected output, timeout stderr, permission denied.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Review gateway-runtime-action.test.ts getNamedGatewayLifecycleState describe block for classify cases
  • Missing regression test: Add parameterized test cases for ANSI-decorated output, timeout stderr, permission denied
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: Existing it.each block at line 100 has 3 cases; room for more edge cases

PRA-9 Improvement — Centralize module cache isolation utilities

  • Location: test/support/connect-flow-test-harness.ts:1
  • Category: architecture
  • Problem: Both cli-oclif-compatibility.test.ts harnesses (withDirectStatusDispatch, withDirectSandboxRecoveryDispatch) and status-flow-test-harness.ts duplicate require.cache manipulation logic.
  • Impact: Duplicated cache manipulation increases maintenance burden and risk of inconsistent restoration.
  • Suggested action: Create test/support/module-cache.ts with withFreshModule and withMockedModules helpers. Refactor gateway-state-hints.test.ts and cli-oclif-compatibility.test.ts to use them.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Compare cache manipulation patterns in cli-oclif-compatibility.test.ts, status-flow-test-harness.ts, gateway-state-hints.test.ts
  • Missing regression test: N/A — refactoring only
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: cli-oclif-compatibility.test.ts lines 36-155, gateway-state-hints.test.ts lines 21,48, status-flow-test-harness.ts all manipulate require.cache

PRA-10 Improvement — Security regression test maintained: rejects removed --dangerously-skip-permissions flag

  • Location: src/commands/sandbox/oclif-command-adapters.test.ts:65
  • Category: security
  • Problem: Test 'rejects the removed connect permission bypass before dispatch' correctly validates the security boundary by rejecting the removed flag before any action dispatch.
  • Impact: Confirms security boundary holds; prevents regression of dangerous permission bypass.
  • Suggested action: Keep this test as-is. It correctly validates the security boundary.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Verify test 'rejects the removed connect permission bypass before dispatch' exists and passes
  • Missing regression test: N/A — test already present
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: Test added in this PR at oclif-command-adapters.test.ts:65-100

PRA-11 Improvement — Registry preservation tests prevent data loss on gateway drift

  • Location: src/lib/actions/sandbox/gateway-state-drift.test.ts:70
  • Category: security
  • Problem: Tests 'preserves a local registry entry when a healthy named gateway still lacks the sandbox' and 'preserves registry state when gateway selection cannot expose the sandbox' encode critical invariant for gateway drift handling.
  • Impact: Prevents accidental registry deletion when gateway state diverges from local registry.
  • Suggested action: Maintain these tests. They encode the critical invariant for gateway drift handling.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Verify tests for gateway drift registry preservation exist and pass
  • Missing regression test: N/A — tests already present
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: gateway-state-drift.test.ts lines 150-190 test registry preservation on drift

PRA-12 Improvement — Identity drift detection hardens gateway trust boundary

  • Location: src/lib/actions/sandbox/gateway-state-hints.test.ts:100
  • Category: security
  • Problem: Tests for 'identity_drift' state (handshake verification failed after recovery) detect critical case where gateway trust material rotated after restart.
  • Impact: Prevents unsafe reattachment to gateway with rotated trust material, which could expose sandbox to wrong gateway.
  • Suggested action: Keep identity drift tests. They detect the critical case where gateway trust material rotated after restart.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Verify gateway-state-hints.test.ts has 'classifies a failed post-recovery handshake as identity drift' test
  • Missing regression test: N/A — tests already present
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: gateway-state-hints.test.ts line 114 tests identity_drift classification

PRA-13 Improvement — Hermes agent preservation prevents state path corruption

  • Location: src/lib/registry-recovery-action.test.ts:50
  • Category: security
  • Problem: Tests ensure persisted 'agent: hermes' not clobbered by session re-seed, preventing rebuild-time agent resolution bugs (/sandbox/.hermes-data vs /sandbox/.openclaw-data).
  • Impact: Prevents agent type corruption that would cause wrong state paths and data loss on rebuild.
  • Suggested action: Maintain agent preservation tests. They prevent rebuild-time agent resolution bugs.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Verify registry-recovery-action.test.ts has 'preserves a persisted Hermes agent when session re-seeds same sandbox' and 'does not clobber persisted agent when session metadata omits it' tests
  • Missing regression test: N/A — tests already present
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: registry-recovery-action.test.ts lines 50-90 test Hermes agent preservation

PRA-14 Improvement — Integration smoke test for real dispatchCli with real recovery modules

  • Location: test/cli-oclif-compatibility.test.ts:384
  • Category: tests
  • Problem: Same as PRA-1: integration smoke test using real require for registry/recovery modules still missing.
  • Impact: Mock drift risk remains unmitigated without real-module integration test.
  • Suggested action: Add test using real require for registry/recovery that calls dispatchCli(['alpha', 'connect']) and verifies connect dispatched after recovery.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Search cli-oclif-compatibility.test.ts for any test that imports real registry/recovery modules without require.cache mocking
  • Missing regression test: Add test: 'dispatches connect after real registry recovery' using real require for registry/recovery modules
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: Harnesses mock 5 modules each; no real-module integration test exists

PRA-15 Improvement — Replace single-use NEMOCLAW_TEST_NO_SLEEP with vi.useFakeTimers

  • Location: src/lib/actions/sandbox/connect-route-lifecycle.test.ts:17
  • Category: architecture
  • Problem: NEMOCLAW_TEST_NO_SLEEP env var is set in beforeEach and deleted in afterEach but only used to speed up tests. No user/CI sets this externally.
  • Impact: Unnecessary env var manipulation; fake timers are more idiomatic in Vitest.
  • Suggested action: Replace with vi.useFakeTimers() in beforeEach and vi.useRealTimers() in afterEach, or remove if sleeps are already mocked.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Check if connectSandbox or its dependencies use setTimeout that would need fake timers
  • Missing regression test: N/A — test infrastructure only
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: Lines 17 and 31 set/delete NEMOCLAW_TEST_NO_SLEEP; no other references in file

Workflow run details

This is an automated, non-binding review; it still expects maintainers and agents to respond to each required or warning item. Treat suggestions as current-PR improvements when they touch changed code; defer only with maintainer rationale or a linked follow-up. A human maintainer must make the final merge decision.

@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings

Merge posture: No blocking advisor findings
Primary next action: Add or justify PRA-T1 and any related test follow-ups.
Open items: 0 required · 0 warnings · 0 suggestions · 5 test follow-ups
Since last review: 0 prior items resolved · 2 still apply · 0 new items found

Action checklist

  • PRA-T1 Add or justify test follow-up: Acceptance clause
  • PRA-T2 Add or justify test follow-up: Acceptance clause
  • PRA-T3 Add or justify test follow-up: Acceptance clause
  • PRA-T4 Add or justify test follow-up: Acceptance clause
  • PRA-T5 Add or justify test follow-up: Acceptance clause
Test follow-ups to resolve or justify

If these cover changed behavior, prefer adding them in this PR; otherwise state why existing coverage is enough or link the follow-up.

  • PRA-T1 Acceptance clause — Linked issue clauses and issue comments — add test evidence or identify existing coverage. The deterministic validation context reported linkedIssues: [], so no authoritative issue clauses or issue comments were available to extract literally. The PR body says "Part of perf(test): reduce test I/O to restore 2–5 minute full-suite runs #6245", but PR body text is untrusted evidence and cannot substitute for linked issue/comment clauses.
  • PRA-T2 Acceptance clause — Reduce unnecessary process isolation across sandbox status, gateway reconciliation, connect recovery, and inference route repair tests while retaining representative CLI, timeout, listener, security, and cross-command contracts. — add test evidence or identify existing coverage. The retained representative contracts are visible in changed tests: status timeout/healthy-ordering in test/cli/status-gateway-lifecycle.test.ts, unsafe status token routing in test/cli/status-routing.test.ts, privileged connect recovery success/failure and no SSH fallback in test/cli/connect-recovery.test.ts, route-swap/proxy/WSL contracts in test/sandbox-connect-inference/route-swap-repair.test.ts, and connect-then-rebuild cross-command recovery in test/gateway-state-reconcile-2276.test.ts. The exact process-isolation reduction was not independently measured in this read-only review.
  • PRA-T3 Acceptance clause — The five affected process suites now launch 20 first-level processes instead of 101, with unchanged runtime behavior and no production-code changes. — add test evidence or identify existing coverage. No production implementation files are in changedFiles; all 23 changed paths are tests or test support. The exact process-count claim is PR-provided benchmark text and was not independently executed or derivable from static diff alone.
  • PRA-T4 Acceptance clause — Move gateway reconciliation scenarios 1–12 into direct gateway lifecycle, status rendering, and skill liveness seams while retaining the real `connect` → `rebuild` scenario 14 contract. — add test evidence or identify existing coverage. Direct lifecycle/status/skill seams are covered in src/lib/actions/sandbox/gateway-state-drift.test.ts, src/lib/actions/sandbox/gateway-state-hints.test.ts, src/lib/actions/sandbox/status-flow.test.ts, src/lib/gateway-runtime-action.test.ts, and src/lib/actions/sandbox/skill-install.test.ts. The retained connect-to-rebuild process contract is visible in test/gateway-state-reconcile-2276.test.ts. Without authoritative issue clauses/comments, the exact mapping of scenarios 1–12 cannot be independently verified.
  • PRA-T5 Acceptance clause — Reduce outer CLI loaders from 83 to 14 (−83.1%) and explicit test-owned first-level processes from 101 to 20 (−80.2%). — add test evidence or identify existing coverage. This is PR-provided benchmark text. The static diff shows process suites were shrunk and direct harness tests added, but the exact loader/process counts were not independently executed or checked by a read-only repository inspection.

Workflow run details

This is an automated, non-binding review; it still expects maintainers and agents to respond to each required or warning item. Treat suggestions as current-PR improvements when they touch changed code; defer only with maintainer rationale or a linked follow-up. A human maintainer must make the final merge decision.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (5)
src/lib/actions/sandbox/connect-flow.test.ts (1)

420-459: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Minor redundancy: two assertions confirm the same exit.

Line 432 already asserts connectSandbox rejects with "process.exit(1)"; line 458's expect(exitSpy).toHaveBeenCalledWith(1) is redundant given the mocked process.exit throw already encodes the exit code. Not incorrect, just belt-and-suspenders.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/actions/sandbox/connect-flow.test.ts` around lines 420 - 459, This
test is asserting the same exit behavior twice in connectSandbox; keep the
rejection expectation that captures the mocked process.exit throw and remove the
extra exitSpy assertion in the connect-flow test. Use the existing
harness.connectSandbox, exitSpy, and process.exit mock setup to leave only one
assertion covering the failure path.
src/lib/actions/sandbox/skill-install.test.ts (1)

196-213: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Test logic matches guard-then-capture ordering contract.

Assertions correctly verify ensureLiveSandboxOrExit short-circuits before captureSandboxSshConfig/uploadDirectory, consistent with installSandboxSkill's ordering in skill-install.ts.

One minor note: the console.log spy at line 199 is not restored (no mockRestore()/afterEach). If the suite doesn't have global restoreMocks: true in its vitest config, this could leak into subsequent tests in the same file.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/actions/sandbox/skill-install.test.ts` around lines 196 - 213, The
test in skill-install.test.ts leaves a console.log spy active without cleanup,
which can leak into later tests. Update the "stops skill installation at the
shared gateway liveness guard" test to restore the console.log spy after the
assertion, or move the cleanup into a shared afterEach if that pattern is used
in this suite. Use the existing vi.spyOn(console, "log") setup in the test as
the anchor for the fix.
test/process-recovery-primitives.test.ts (1)

292-292: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Test title missing required issue-ref suffix.

New test title "honors the sandbox-exec timeout without falling back to SSH" doesn't include a local issue ref in a final (#1234) suffix.

As per coding guidelines, test/**/*.test.{js,ts}: "use behavior-oriented titles with local issue refs in a final (#1234) suffix."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/process-recovery-primitives.test.ts` at line 292, Update the test case
title in the affected test block to follow the required behavior-oriented format
and append a local issue reference suffix like "(`#1234`)". Make the change on the
it() call for the sandbox-exec timeout test in
process-recovery-primitives.test.ts, keeping the existing behavior text but
adding the final issue-ref suffix to the title.

Source: Coding guidelines

test/cli-oclif-compatibility.test.ts (1)

384-758: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

New test titles are missing the required local issue-ref suffix.

None of the new tests here (e.g. "recovers a requested sandbox, rereads the registry, and dispatches connect", "corrects a single sandbox-like global status argument without a CLI subprocess", etc.) include a trailing (#1234)-style issue reference.

As per coding guidelines, test/**/*.test.{js,ts}: "Root-level integration tests under test/ should import source code, use ESM imports, and use behavior-oriented titles with local issue refs in a final (#1234) suffix."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/cli-oclif-compatibility.test.ts` around lines 384 - 758, The new
integration test titles in this spec are missing the required local issue-ref
suffix. Update each newly added `it(...)` title in the test file to end with a
`(`#1234`)`-style reference, keeping the existing behavior-oriented wording
intact; use the test names around `withDirectSandboxRecoveryDispatch`,
`withDirectStatusDispatch`, and the direct `dispatchCli` cases as the places to
fix.

Source: Coding guidelines

test/cli/status-routing.test.ts (1)

12-37: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

New test titles are missing the required local issue-ref suffix.

As with the other test/ files in this cohort, none of the three new tests include a trailing (#1234)-style issue reference.

As per coding guidelines, test/**/*.test.{js,ts}: "use behavior-oriented titles with local issue refs in a final (#1234) suffix."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/cli/status-routing.test.ts` around lines 12 - 37, Update the three new
test case titles in status-routing.test.ts to include the required local issue
reference suffix, matching the pattern used by other test files. Keep the
behavior-oriented wording intact in each it(...) block, and append a final
“(`#1234`)”-style issue ref to each title in the test definitions for status
--help, sandbox-first status rejects unexpected positional arguments through
command-id dispatch, and never emits an unsafe sandbox token in a copy-paste
status command.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/lib/actions/sandbox/connect-flow.test.ts`:
- Around line 420-459: This test is asserting the same exit behavior twice in
connectSandbox; keep the rejection expectation that captures the mocked
process.exit throw and remove the extra exitSpy assertion in the connect-flow
test. Use the existing harness.connectSandbox, exitSpy, and process.exit mock
setup to leave only one assertion covering the failure path.

In `@src/lib/actions/sandbox/skill-install.test.ts`:
- Around line 196-213: The test in skill-install.test.ts leaves a console.log
spy active without cleanup, which can leak into later tests. Update the "stops
skill installation at the shared gateway liveness guard" test to restore the
console.log spy after the assertion, or move the cleanup into a shared afterEach
if that pattern is used in this suite. Use the existing vi.spyOn(console, "log")
setup in the test as the anchor for the fix.

In `@test/cli-oclif-compatibility.test.ts`:
- Around line 384-758: The new integration test titles in this spec are missing
the required local issue-ref suffix. Update each newly added `it(...)` title in
the test file to end with a `(`#1234`)`-style reference, keeping the existing
behavior-oriented wording intact; use the test names around
`withDirectSandboxRecoveryDispatch`, `withDirectStatusDispatch`, and the direct
`dispatchCli` cases as the places to fix.

In `@test/cli/status-routing.test.ts`:
- Around line 12-37: Update the three new test case titles in
status-routing.test.ts to include the required local issue reference suffix,
matching the pattern used by other test files. Keep the behavior-oriented
wording intact in each it(...) block, and append a final “(`#1234`)”-style issue
ref to each title in the test definitions for status --help, sandbox-first
status rejects unexpected positional arguments through command-id dispatch, and
never emits an unsafe sandbox token in a copy-paste status command.

In `@test/process-recovery-primitives.test.ts`:
- Line 292: Update the test case title in the affected test block to follow the
required behavior-oriented format and append a local issue reference suffix like
"(`#1234`)". Make the change on the it() call for the sandbox-exec timeout test in
process-recovery-primitives.test.ts, keeping the existing behavior text but
adding the final issue-ref suffix to the title.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 09947a1e-d376-46b0-8924-6562012a2918

📥 Commits

Reviewing files that changed from the base of the PR and between 7044a03 and aa2d52b.

📒 Files selected for processing (20)
  • src/commands/sandbox/oclif-command-adapters.test.ts
  • src/lib/actions/sandbox/connect-flow.test.ts
  • src/lib/actions/sandbox/connect-route-repair.test.ts
  • src/lib/actions/sandbox/gateway-state-drift.test.ts
  • src/lib/actions/sandbox/gateway-state-hints.test.ts
  • src/lib/actions/sandbox/skill-install.test.ts
  • src/lib/actions/sandbox/status-flow.test.ts
  • src/lib/cli/argv-normalizer.test.ts
  • src/lib/gateway-runtime-action.test.ts
  • src/lib/registry-recovery-action.test.ts
  • test/cli-oclif-compatibility.test.ts
  • test/cli/connect-recovery.test.ts
  • test/cli/status-gateway-lifecycle.test.ts
  • test/cli/status-routing.test.ts
  • test/gateway-state-reconcile-2276.test.ts
  • test/process-recovery-managed-controller.test.ts
  • test/process-recovery-primitives.test.ts
  • test/process-recovery.test.ts
  • test/sandbox-connect-inference/route-swap-repair.test.ts
  • test/support/connect-flow-test-harness.ts

Signed-off-by: Carlos Villela <cvillela@nvidia.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
test/cli/connect-recovery.test.ts (1)

367-460: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Large hand-written onboard-session JSON duplicates schema knowledge inline.

This test constructs the full onboard-session.json payload by hand (all step statuses, metadata, etc.) rather than via a shared helper (the file already has writeSandboxRegistry from test/cli/helpers.ts for the analogous registry case). If the onboard-session schema changes elsewhere, this inline literal can silently drift and either break or mask the intended "recover from last session" behavior.

Consider extracting a writeOnboardSession(home, overrides) helper alongside writeSandboxRegistry for reuse and to keep the schema definition in one place.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/cli/connect-recovery.test.ts` around lines 367 - 460, The `connect
recovers a named sandbox from the last onboard session when the registry is
empty` test is duplicating the full onboard-session schema inline, which can
drift from the real shape. Add a shared `writeOnboardSession(home, overrides)`
helper next to `writeSandboxRegistry` in the CLI test helpers, and use it here
to generate `onboard-session.json` from one central schema definition. Keep the
test-specific values (`sandboxName`, `provider`, `metadata`, step statuses,
etc.) as overrides so the recovery scenario stays explicit without hardcoding
the whole payload.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/registry-recovery-seeded-paths.test.ts`:
- Around line 56-63: The mock for validateName in
registry-recovery-seeded-paths.test.ts is duplicating production validation
logic instead of exercising the real implementation, which can let tests drift
from src/lib/runner.js. Update the vi.mock("./runner.js") setup to import the
actual validateName via vi.importActual (or otherwise delegate to the real
function) and only wrap it for the test’s needs, so the assertions about skipped
names stay aligned with the production validator.

In `@test/support/status-flow-test-harness.ts`:
- Line 60: The status-flow test harness is collapsing an explicit null value for
processRunning into false, so null-specific behavior cannot be exercised. Update
the option resolution in status-flow-test-harness around the processRunning
handling so it preserves an explicitly passed null instead of defaulting it
away, while still falling back to false only when the option is undefined. Check
the processRunning field usage in the harness setup and any related helper logic
so tests can pass through boolean | null unchanged.

---

Nitpick comments:
In `@test/cli/connect-recovery.test.ts`:
- Around line 367-460: The `connect recovers a named sandbox from the last
onboard session when the registry is empty` test is duplicating the full
onboard-session schema inline, which can drift from the real shape. Add a shared
`writeOnboardSession(home, overrides)` helper next to `writeSandboxRegistry` in
the CLI test helpers, and use it here to generate `onboard-session.json` from
one central schema definition. Keep the test-specific values (`sandboxName`,
`provider`, `metadata`, step statuses, etc.) as overrides so the recovery
scenario stays explicit without hardcoding the whole payload.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 5bf8de47-df16-458c-9598-789287dfeb5e

📥 Commits

Reviewing files that changed from the base of the PR and between aa2d52b and 144847c.

📒 Files selected for processing (9)
  • src/lib/actions/sandbox/connect-flow.test.ts
  • src/lib/actions/sandbox/connect-route-lifecycle.test.ts
  • src/lib/actions/sandbox/gateway-state-hints.test.ts
  • src/lib/actions/sandbox/status-flow.test.ts
  • src/lib/registry-recovery-action.test.ts
  • src/lib/registry-recovery-seeded-paths.test.ts
  • test/cli-oclif-compatibility.test.ts
  • test/cli/connect-recovery.test.ts
  • test/support/status-flow-test-harness.ts
💤 Files with no reviewable changes (2)
  • src/lib/actions/sandbox/connect-flow.test.ts
  • src/lib/registry-recovery-action.test.ts
🚧 Files skipped from review as they are similar to previous changes (3)
  • src/lib/actions/sandbox/gateway-state-hints.test.ts
  • src/lib/actions/sandbox/status-flow.test.ts
  • test/cli-oclif-compatibility.test.ts

Comment thread src/lib/registry-recovery-seeded-paths.test.ts Outdated
lookup?: SandboxGatewayState;
lookupState?: "present" | "missing";
preflight?: SandboxStatusPreflightResult;
processRunning?: boolean | null;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

?? collapses null into false, making the processRunning: null option unusable.

options.processRunning is typed boolean | null, but options.processRunning ?? false treats null the same as undefined, so a caller passing processRunning: null will always get false resolved instead of null. Any future test intending to exercise null-specific gateway-running behavior via this harness silently can't.

🐛 Proposed fix
-  vi.spyOn(processRecovery, "isSandboxGatewayRunningForStatus").mockResolvedValue(
-    options.processRunning ?? false,
-  );
+  vi.spyOn(processRecovery, "isSandboxGatewayRunningForStatus").mockResolvedValue(
+    options.processRunning === undefined ? false : options.processRunning,
+  );

Also applies to: 168-170

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/support/status-flow-test-harness.ts` at line 60, The status-flow test
harness is collapsing an explicit null value for processRunning into false, so
null-specific behavior cannot be exercised. Update the option resolution in
status-flow-test-harness around the processRunning handling so it preserves an
explicitly passed null instead of defaulting it away, while still falling back
to false only when the option is undefined. Check the processRunning field usage
in the harness setup and any related helper logic so tests can pass through
boolean | null unchanged.

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@cv cv added the v0.0.75 label Jul 5, 2026
@cv

cv commented Jul 5, 2026

Copy link
Copy Markdown
Collaborator Author

Final automated-review disposition for cab00c69:

Addressed in this PR:

  • Growth/monolith findings: status, connect, and registry direct tests were split/extracted; resulting files are 294, 390, and 426 lines, with the registry file net-negative versus main.
  • Retained real process boundaries: privileged Docker recovery failure now proves fail-closed/no-SSH behavior and sanitized root argv; session-backed recovery uses the real CLI/registry path and asserts session-derived metadata persists.
  • CodeRabbit findings: seeded recovery imports the production validateName; the unused nullable processRunning harness option was removed.
  • Coverage-hook timeout: gateway-state dependencies now cold-load outside beforeEach; the coverage-enabled focused run passed 12/12 with an 8ms test body, and final shard 3 passed.

No further code change is warranted for the remaining Nemotron notes:

  • The real dispatch/recovery smoke is the retained session-backed CLI process test.
  • VM DNS env cleanup is explicit in both try/finally and suite teardown.
  • Scrubbed privileged Docker argv and no-SSH behavior are asserted directly in retained process tests, so an additional mapping comment would not add coverage.
  • PR fix(mcp): reconcile Hermes runtime state #6261 touches independent additive harness fields and presents no behavioral contradiction.
  • Extra malformed-output cases and a generalized module-cache utility are optional follow-up scope, not demonstrated defects.

Final evidence: 217/217 shuffled changed-file tests, all normal hooks, 43 CI checks passed, 3 expected skips, and no failed/pending checks. CodeRabbit reports no actionable comments; GPT-5.5 recommends merge_as_is; E2E advisor recommends no targets.

@cv
cv merged commit 34ac134 into main Jul 5, 2026
50 checks passed
@cv
cv deleted the codex/perf-test-io-6245-batch-3 branch July 5, 2026 02:05
Hadar301 pushed a commit to Hadar301/NemoClaw-OpenShift that referenced this pull request Jul 12, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Reduce unnecessary process isolation across sandbox status, gateway
reconciliation, connect recovery, and inference route repair tests while
retaining representative CLI, timeout, listener, security, and
cross-command contracts. The five affected process suites now launch 20
first-level processes instead of 101, with unchanged runtime behavior
and no production-code changes.

## Related Issue

Part of NVIDIA#6245

## Changes

- Move status routing and gateway lifecycle branches into direct
public-dispatch, status-flow, and gateway-state tests while retaining
CLI help/parser, unsafe-token, hanging-pipe timeout, and
healthy-ordering contracts.
- Move route-repair branches into the direct connect harness while
retaining route-swap plumbing, local Ollama proxy/secret isolation, and
WSL fallback contracts.
- Move gateway reconciliation scenarios 1–12 into direct gateway
lifecycle, status rendering, and skill liveness seams while retaining
the real `connect` → `rebuild` scenario 14 contract.
- Move connect recovery branches into argv, Oclif adapter, registry
recovery, and process-recovery seams while retaining successful and
failed privileged-Docker recovery plus real session-backed registry
recovery contracts.
- Reset registry-recovery dependency mocks deterministically and cover
requested-sandbox recovery through public dispatch under shuffled test
order.
- Extract focused status, connect-route, and seeded-registry test
surfaces so existing large test files do not grow past the advisor
policy.
- Reduce outer CLI loaders from 83 to 14 (−83.1%) and explicit
test-owned first-level processes from 101 to 20 (−80.2%).

## Type of Change

- [x] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: all 23 changed paths are
tests or test support; the final docs-writer review found no runtime
command, flag, configuration, API, policy, or workflow change.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: independent local
review covered assertion migration, retained process/security
boundaries, mock and module-cache isolation, environment restoration,
shuffled-order determinism, and the CI/advisor follow-up; all findings
were fixed and re-reviewed with no remaining findings.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification

- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — 217/217 changed-file tests passed
under shuffled order (seed 6245) in 39.76s; CLI typecheck, Biome,
project-overlap, source-shape, test-size, conditional-growth, and
monolith-growth checks passed.
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — `npm test` completed in 15m25.22s with
13,079 passing and 36 skipped tests. Seven mode assertions inherited the
host's `077` umask and passed 68/68 under `022`; one E2E source-hygiene
failure is byte-identical on `origin/main`.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

## Benchmark

- Same five process suites on `origin/main`: 76 tests in 57.10s.
- Final branch: 13 retained process tests in 20.19s, 64.6% faster;
migrated decision coverage remains in direct tests.
- Outer CLI loaders: 83 → 14; explicit first-level processes including
listeners: 101 → 20.
- Final-head coverage artifacts measured the five migrated files at
32.42s versus 107.73s on `main` (−69.9%); aggregate blob execution fell
7.4% and summed shard job time fell 6.2%, while critical shard wall time
was 8m39s versus 8m13s on `main`.
- `gateway-state-drift` now executes its tests in 26ms, but its warmed
source graph still costs 10.13s during collection and remains a shard-3
hotspot for the next batch.
- The single full clean-build `npm test` run was 15m25.22s on Node
25.9.0 versus the issue's 14m19.65s reference, so this PR does not claim
a repo-wide local wall-time win yet. The remaining untouched corpus
still dominates and needs additional batches.

---
Signed-off-by: Carlos Villela <cvillela@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Added and strengthened CLI and sandbox connect/recovery contract
coverage, including safer failure behavior, deterministic guidance, and
stricter validation of unsafe/privileged flows.
* Expanded status and lifecycle classification scenarios (including
inference and gateway-state edge cases) and tightened assertions around
when sandbox teardown does or does not occur.
* Introduced reusable status/connect-flow test harnesses and improved
mocking/teardown isolation; added seeded registry recovery tests and
additional timeout/probe-only orchestration checks.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@wscurran wscurran added area: ci CI workflows, checks, release automation, or GitHub Actions area: e2e End-to-end tests, nightly failures, or validation infrastructure chore Build, CI, dependency, or tooling maintenance labels Aug 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ci CI workflows, checks, release automation, or GitHub Actions area: e2e End-to-end tests, nightly failures, or validation infrastructure chore Build, CI, dependency, or tooling maintenance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants