Skip to content

perf(test): reduce onboarding subprocess isolation - #6276

Merged
cv merged 16 commits into
mainfrom
codex/perf-test-io-6245
Jul 4, 2026
Merged

perf(test): reduce onboarding subprocess isolation#6276
cv merged 16 commits into
mainfrom
codex/perf-test-io-6245

Conversation

@cv

@cv cv commented Jul 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

Replace unit-shaped test subprocesses with direct typed seams while retaining meaningful real-process contracts. The final onboarding pilot is 20.5% faster, and the OpenClaw config target is 40.5% faster, without changing production defaults or CLI behavior.

Related Issue

Part of #6245.

Changes

  • Extract a typed createSetupInference test seam into a focused module while preserving the production dependency wiring and reducing src/lib/onboard.ts by 65 net lines.
  • Rewrite unit-shaped subprocess fixtures in the onboarding, remote-provider-selection, and service-environment suites while retaining representative process-boundary, fail-closed, and production Bash coverage.
  • Replace 46 service-environment harness child calls with equivalent Node filesystem operations and cache the three-proxy fixture input.
  • Expand the direct dependency-failure suite to 25 cases, including remote/Bedrock exit boundaries, credential/upsert/apply failures, falsey status fallbacks, local providers, Ollama proxy recovery, routed reconciliation/upsert/route-apply failures, Hermes provider-store/credential/lookup failures, and a focused real Responses-to-Chat-Completions probe fallback test.
  • Complete injected exit, error, and log wiring across remote, Bedrock, Hermes, Ollama, vLLM, routed-provider, local-route-application, and Hermes-auth paths while retaining production defaults and real setupNim boundaries for all five native-Docker Windows-provider rejection scenarios.
  • Make provider dependency ownership explicit, document the local route recovery source boundary and removal condition, and add three focused local-route recovery tests.
  • Require explicit Bedrock and Hermes auth failure boundaries, cover positive Hermes auth navigation, and use scanner-safe runtime redaction canaries.
  • Retain a production-exported setupInference/OpenShell process boundary proving raw credentials never enter argv and only the provider update child receives the explicitly scoped credential environment.
  • Invoke the exported messaging post-install phase directly in OpenClaw config tests, eliminating 98 redundant outer applier launches while preserving all 20 real openclaw doctor launches and generator/applier executable contracts.
  • Restore checked-JavaScript validation for test/generate-openclaw-config.test.ts by removing its file-wide @ts-nocheck directive.
  • Tighten legacy test-file size ratchets to 6,146 lines for test/onboard-selection.test.ts, 4,057 for test/onboard.test.ts, and 1,945 for test/generate-openclaw-config.test.ts.
  • Record a final-head onboarding median improvement from 44.34s to 35.24s (20.5%) and reduce aggregate execve attempts from 7,654 to 6,444 (15.8%).
  • Record an OpenClaw config median improvement from 19.83s to 11.80s (40.5%), with successful execve calls reduced from 294 to 196.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: Internal test seams and test-harness rewrites only; production CLI behavior, output, configuration, and public interfaces are unchanged.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Independent focused reviews confirmed production dependency wiring, secret containment, environment restoration, and retained process boundaries; automated advisor findings were addressed or explicitly dispositioned.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Verification

  • PR description includes the DCO sign-off declaration and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: current head: onboarding (65/65), dependency failures (25/25), local-route/rebuild source tests (5/5), Hermes auth (6/6), Bedrock source (4/4), OpenClaw config (128/128), and provider/source targets (26/26); earlier focused probe/provider targets (28/28), selection (69/69), final onboarding benchmark target (134/134 in each run), messaging-applier support (33/33), and service environment (39/39); CLI and checked-JavaScript type-checks plus project-membership, title/size/conditional/source-shape/Biome/diff checks pass.
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: broad pilot gate: npm test (1,128 files, 12,621 tests); npm run test:coverage:cli (1,004 files, 11,063 tests, all ratchets); current-head pre-commit and pre-push repository/type-check gates passed.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Carlos Villela cvillela@nvidia.com

Summary by CodeRabbit

  • New Features

    • Expanded onboarding support for multiple inference providers, including improved handling for remote, local, routed, Bedrock Runtime, Hermes, Ollama, and Windows host detection flows.
    • Added clearer fallback behavior when OpenAI-compatible endpoints need to switch from /responses to chat completions.
  • Bug Fixes

    • Improved failure handling and recovery messaging during onboarding, including better exit behavior in non-interactive flows.
    • Reduced the chance of leaking sensitive values in error output.
  • Tests

    • Added broader coverage for onboarding, provider selection, and inference-route fallback scenarios.

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@cv cv self-assigned this Jul 4, 2026
@copy-pr-bot

copy-pr-bot Bot commented Jul 4, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

This PR refactors onboarding inference provider setup (Bedrock Runtime, Hermes auth/provider, Ollama, remote, routed, vLLM-local, local-inference-route, Windows host Ollama detection) to use injected error/log/exitProcess dependencies instead of direct console/process calls, introduces a setup-inference.ts factory module, adds extensive test harnesses and failure-path tests, and includes unrelated test-only refactors (curl fallback probe test, service-env.test.ts fs API sync migration, in-process messaging build in config test).

Changes

Onboarding inference DI boundary refactor

Layer / File(s) Summary
Shared dependency type contracts
src/lib/onboard/inference-providers/types.ts
CommonDeps, RoutedDeps, PromptValidationRecovery, and Registry.updateSandbox types are extended/tightened to include exitProcess, error, log, and redact/compactText.
Bedrock Runtime and Hermes auth DI boundaries
src/lib/onboard/bedrock-runtime.ts, src/lib/onboard/bedrock-runtime.test.ts, src/lib/onboard/hermes-auth.ts, src/lib/onboard/hermes-auth.test.ts
Bedrock and Hermes auth functions accept injected error/exitProcess/log for failure reporting and termination, replacing console.*/process.exit, with corresponding tests.
Provider-specific inference setup DI boundaries
src/lib/onboard/inference-providers/hermes.ts, .../ollama-local.ts, .../remote.ts, .../routed.ts, .../vllm-local.ts, src/lib/onboard/local-inference-route.ts, plus tests
All provider setup functions route failure logging/termination through injected dependencies instead of console/process.exit, validated by unit tests.
Windows host Ollama detection DI
src/lib/onboard/windows-host-ollama.ts, .../windows-host-ollama.test.ts
detectWindowsHostOllama and probing helpers accept injectable isWsl/runCapture overrides.
Setup-inference factory orchestration
src/lib/onboard/setup-inference.ts, src/lib/onboard.ts
New createSetupInference factory dispatches to provider setup routines; onboard.ts replaces its inline implementation with factory wiring.
Direct/production test harness utilities
test/support/setup-inference-test-harness.ts, test/support/onboard-selection-test-helpers.ts, src/lib/actions/sandbox/rebuild-local-provider-recreate.test.ts
New harness utilities for direct command routing and credential-boundary verification of setupInference.
Onboard test suite migration and new failure-path coverage
test/onboard.test.ts, test/onboard-inference-failure-paths.test.ts, ci/test-file-size-budget.json
Migrates onboarding tests from spawn-based fixtures to direct harness assertions and adds a new failure-path test suite; adjusts line-count budgets.

Responses-fallback curl probe test support

Layer / File(s) Summary
Fake curl fallback harness and test
src/lib/inference/onboard-probes-curl-harness.ts, src/lib/inference/onboard-probes-responses-fallback.test.ts
Adds a fake curl script generator and a test verifying /responses fallback to /chat/completions.

service-env.test.ts filesystem API refactor

Layer / File(s) Summary
Sync fs API and snippet extraction refactor
test/service-env.test.ts
Replaces shell-based rm/chmod/sed/stat/ln calls with node:fs sync APIs and caches proxy-variable extraction results.

In-process messaging build phase for config test

Layer / File(s) Summary
In-process messaging applier invocation
test/generate-openclaw-config.test.ts
Runs the messaging build phase in-process via direct function calls instead of spawning a subprocess.

Estimated code review effort: 4 (Complex) | ~75 minutes

Sequence Diagram(s)

sequenceDiagram
  participant OnboardFlow
  participant setupInferenceWithDeps
  participant ProviderSetup
  participant exitProcess

  OnboardFlow->>setupInferenceWithDeps: setupInference(sandbox, model, provider)
  setupInferenceWithDeps->>ProviderSetup: dispatch (Hermes/remote/vllm/ollama/routed/bedrock)
  alt failure
    ProviderSetup->>ProviderSetup: error(message)
    ProviderSetup->>exitProcess: exitProcess(status)
    exitProcess-->>OnboardFlow: terminate / reject
  else success
    ProviderSetup-->>setupInferenceWithDeps: outcome.done
    setupInferenceWithDeps-->>OnboardFlow: { ok: true }
  end
Loading

Possibly related issues

Suggested labels: chore

Suggested reviewers: ericksoa

Poem

A rabbit hopped through onboarding code,
Swapping console.log for a cleaner mode,
Exit calls now injected with care,
Tests harnessed direct, no spawns to spare,
Hop, hop — the setup factory's alive tonight! 🐇

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: reducing onboarding test subprocess isolation for performance.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/perf-test-io-6245

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in the codex/perf-test-io-6... branch is 96%. Coverage data for the main branch is not yet available.

Show a code coverage summary of the most covered files.
File main codex/perf-test-io-6... 1d9d09b +/-
nemoclaw/src/se...cret-scanner.ts 100%
nemoclaw/src/commands/slash.ts 100%
nemoclaw/src/li...bprocess-env.ts 100%
nemoclaw/src/bl...eprint/state.ts 98%
nemoclaw/src/onboard/config.ts 98%
nemoclaw/src/bl...int/snapshot.ts 97%
nemoclaw/src/bl...print/runner.ts 95%
nemoclaw/src/co...ration-state.ts 94%
nemoclaw/src/bl...ate-networks.ts 94%
nemoclaw/src/index.ts 94%

TypeScript / code-coverage/cli

The overall coverage in the codex/perf-test-io-6... branch is 70%. Coverage data for the main branch is not yet available.

Show a code coverage summary of the most covered files.
File main codex/perf-test-io-6... 1d9d09b +/-
src/lib/shields...nsition-lock.ts 87%
src/lib/actions...all/run-plan.ts 81%
src/lib/state/o...oard-session.ts 81%
src/lib/onboard/preflight.ts 71%
src/lib/state/sandbox.ts 71%
src/lib/shields/index.ts 69%
src/lib/onboard...er-gpu-patch.ts 69%
src/lib/policy/index.ts 66%
src/lib/actions...licy-channel.ts 60%
src/lib/onboard.ts 24%

Updated July 04, 2026 19:28 UTC
Code Coverage is in Public Preview. Learn more and provide us with your feedback.

@github-actions

github-actions Bot commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor (Nemotron Ultra) — Changes requested

Merge posture: Do not merge yet
Primary next action: Fix PRA-3: ProviderBranchDeps intersection type masks missing provider-specific dependencies; then add or justify PRA-T1.
Open items: 2 required · 7 warnings · 4 suggestions · 8 test follow-ups
Since last review: 4 prior items resolved · 5 still apply · 2 new items found

Action checklist

  • PRA-3 Fix: ProviderBranchDeps intersection type masks missing provider-specific dependencies in src/lib/onboard/setup-inference.ts:18
  • PRA-4 Fix: New test file not listed in test-file-size-budget.json in ci/test-file-size-budget.json:1
  • PRA-1 Resolve or justify: Source-of-truth review needed: src/lib/onboard/bedrock-runtime.ts
  • PRA-2 Resolve or justify: Source-of-truth review needed: src/lib/onboard/hermes-auth.ts
  • PRA-6 Resolve or justify: Hermes auth module missing interactive positive-path tests for injected boundaries in src/lib/onboard/hermes-auth.test.ts:1
  • PRA-7 Resolve or justify: Bedrock setup has optional ensureAdapter/updateSandbox parameters creating footgun in src/lib/onboard/bedrock-runtime.ts:164
  • PRA-8 Resolve or justify: getRequestedHermesAuthMethod reads process.env directly, reducing testability in src/lib/onboard/hermes-auth.ts:48
  • PRA-9 Resolve or justify: Bedrock runtime module missing interactive/non-interactive failure boundary tests in src/lib/onboard/bedrock-runtime.test.ts:1
  • PRA-12 Resolve or justify: Missing routed provider reconciliation failure test in failure-paths suite in test/onboard-inference-failure-paths.test.ts:1
  • PRA-T1 Add or justify test follow-up: Runtime validation
  • PRA-T2 Add or justify test follow-up: Runtime validation
  • PRA-T3 Add or justify test follow-up: Runtime validation
  • PRA-T4 Add or justify test follow-up: Runtime validation
  • PRA-T5 Add or justify test follow-up: Runtime validation
  • PRA-T6 Add or justify test follow-up: New test file not listed in test-file-size-budget.json
  • PRA-T7 Add or justify test follow-up: Hermes auth module missing interactive positive-path tests for injected boundaries
  • PRA-T8 Add or justify test follow-up: Bedrock runtime module missing interactive/non-interactive failure boundary tests
  • PRA-5 In-scope improvement: Simplify ProviderBranchDeps using discriminated union instead of intersection in src/lib/onboard/setup-inference.ts:18
  • PRA-10 In-scope improvement: Missing routed provider integration test with real subprocess in test/onboard.test.ts:1
  • PRA-11 In-scope improvement: Local inference route retry wrapper workaround documented but removal depends on external OpenShell changes in src/lib/onboard/local-inference-route.ts:10
  • PRA-13 In-scope improvement: Verify generate-openclaw-config.test.ts compiles cleanly without @ts-nocheck in test/generate-openclaw-config.test.ts:1

Findings index

ID Severity Category Location Required action
PRA-1 Resolve/justify architecture Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
PRA-2 Resolve/justify architecture Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
PRA-3 Required architecture src/lib/onboard/setup-inference.ts:18 Split into a base CommonDeps and per-provider extension types (e.g., RemoteProviderDeps extends CommonDeps, HermesDeps extends CommonDeps), then use a discriminated union keyed by provider in createSetupInference. This preserves type safety as interfaces diverge.
PRA-4 Required tests ci/test-file-size-budget.json:1 Add an entry for test/onboard-inference-failure-paths.test.ts in ci/test-file-size-budget.json legacyMaxLines with current line count (1029) or a reasonable ceiling (e.g., 1500).
PRA-5 Improvement architecture src/lib/onboard/setup-inference.ts:18 Replace the intersection with a base CommonDeps interface and per-provider extensions (RemoteProviderDeps extends CommonDeps, etc.), then use a discriminated union in createSetupInference keyed by provider.
PRA-6 Resolve/justify tests src/lib/onboard/hermes-auth.test.ts:1 Add 2 tests to hermes-auth.test.ts: (1) interactive promptHermesAuthMethod selects API key with injected boundaries, (2) interactive promptHermesAuthMethod selects OAuth with injected boundaries. Both should verify deps.prompt, deps.getNavigationChoice, and that error/exitProcess are NOT called.
PRA-7 Resolve/justify security src/lib/onboard/bedrock-runtime.ts:164 Make ensureAdapter and updateSandbox required in the function signature (remove ?). Tests already inject mocks as they do for other providers.
PRA-8 Resolve/justify security src/lib/onboard/hermes-auth.ts:48 Accept the raw auth method string as a parameter instead of reading process.env directly. The caller (promptHermesAuthMethod) already has access to the environment and can pass the value.
PRA-9 Resolve/justify tests src/lib/onboard/bedrock-runtime.test.ts:1 Add 3-4 tests to bedrock-runtime.test.ts using injected boundaries for interactive failure paths: (1) interactive adapter startup failure, (2) interactive provider upsert failure with status preservation, (3) interactive inference-set failure with redaction and status preservation, (4) interactive success path with logging verification.
PRA-10 Improvement tests test/onboard.test.ts:1 Add an integration test in test/onboard.test.ts (or new file) that spawns a real OpenShell gateway subprocess, configures a routed provider, triggers model router reconciliation failure, and verifies the exit boundary behavior. Use runProductionSetupInferenceCredentialBoundary pattern from setup-inference-test-harness.ts.
PRA-11 Improvement architecture src/lib/onboard/local-inference-route.ts:10 Accept the workaround with the documented boundary. Track the removal condition in an issue linked to OpenShell roadmap.
PRA-12 Resolve/justify tests test/onboard-inference-failure-paths.test.ts:1 Add a test case for routed provider reconciliation failure that verifies the injected exitProcess boundary is hit with the correct error message.
PRA-13 Improvement correctness test/generate-openclaw-config.test.ts:1 Verify the file compiles cleanly without @ts-nocheck. If any @ts-ignore or @ts-expect-error comments exist, evaluate if they can be removed with proper typing.

🚨 Required before merge

Address these before merging unless a maintainer explicitly overrides the advisor with rationale.

PRA-3 Required — ProviderBranchDeps intersection type masks missing provider-specific dependencies

  • Location: src/lib/onboard/setup-inference.ts:18
  • Category: architecture
  • Problem: ProviderBranchDeps uses an intersection of Pick types across 6 provider dep types (CommonDeps, HermesDeps, RemoteProviderDeps, VllmDeps, OllamaDeps, RoutedDeps). This forces every provider branch to satisfy all dependencies from all other providers. Adding a dep to one provider interface (e.g., HermesDeps.lookup) makes TypeScript require it for remote/vLLM/Ollama branches too, or if a new provider-specific dep is added but not provided for other providers, the type won't catch it because the intersection is satisfied by the union of all deps.
  • Impact: Compile-time safety degraded: missing provider-specific dependencies may not be caught. Maintainers must manually verify all deps are provided for all providers when adding new ones. Refactoring risk increases as provider interfaces diverge.
  • Required action: Split into a base CommonDeps and per-provider extension types (e.g., RemoteProviderDeps extends CommonDeps, HermesDeps extends CommonDeps), then use a discriminated union keyed by provider in createSetupInference. This preserves type safety as interfaces diverge.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: Add a new required field to HermesDeps only (e.g., hermesSpecificDep: string), then call createSetupInference with a remote provider — TypeScript should error but currently won't because the intersection is satisfied by other deps.
  • Missing regression test: Add a compile-fail test in the test harness that adds a provider-specific required field to one Deps type and verifies TypeScript errors when the orchestrator doesn't provide it for other provider branches.
  • Done when: The required change is committed and verification passes: Add a new required field to HermesDeps only (e.g., hermesSpecificDep: string), then call createSetupInference with a remote provider — TypeScript should error but currently won't because the intersection is satisfied by other deps.
  • Evidence: setup-inference.ts:18-66 defines ProviderBranchDeps as intersection of Pick types from 6 provider dep types. getSetupInferenceDeps() in onboard.ts provides all deps for all providers, masking the issue at runtime.

PRA-4 Required — New test file not listed in test-file-size-budget.json

  • Location: ci/test-file-size-budget.json:1
  • Category: tests
  • Problem: New test file test/onboard-inference-failure-paths.test.ts (1029 lines) is not listed in ci/test-file-size-budget.json legacyMaxLines. The budget file tracks legacy test files that exceed the default 1500-line limit; this new file will exceed it and needs an entry.
  • Impact: CI budget check will fail for the new file, potentially blocking merges or requiring emergency budget increases.
  • Required action: Add an entry for test/onboard-inference-failure-paths.test.ts in ci/test-file-size-budget.json legacyMaxLines with current line count (1029) or a reasonable ceiling (e.g., 1500).
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: Run the test-file-size-budget check (likely part of CI) — it should fail for the new file.
  • Missing regression test: CI budget check should catch this automatically once the entry is added.
  • Done when: The required change is committed and verification passes: Run the test-file-size-budget check (likely part of CI) — it should fail for the new file.
  • Evidence: ci/test-file-size-budget.json has entries for other large test files but not for onboard-inference-failure-paths.test.ts.
Review findings by urgency: 2 required fixes, 7 items to resolve/justify, 4 in-scope improvements

⚠️ Resolve or justify before merge

Investigate these in the current review; either fix them, explain why they are not applicable, or document the accepted risk.

PRA-1 Resolve/justify — Source-of-truth review needed: src/lib/onboard/bedrock-runtime.ts

  • Location: not file-specific
  • Category: architecture
  • Problem: The advisor marked localized patch analysis as needs_followup.
  • Impact: A localized workaround can preserve or hide an invalid state when the source boundary is unclear.
  • Recommended action: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Missing regression test: TypeScript compilation should error if caller omits required parameters
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Evidence: bedrock-runtime.ts:164-165 uses (options.ensureAdapter ?? ensureBedrockRuntimeAdapter) and (options.updateSandbox ?? registry.updateSandbox)

PRA-2 Resolve/justify — Source-of-truth review needed: src/lib/onboard/hermes-auth.ts

  • Location: not file-specific
  • Category: architecture
  • Problem: The advisor marked localized patch analysis as needs_followup.
  • Impact: A localized workaround can preserve or hide an invalid state when the source boundary is unclear.
  • Recommended action: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Missing regression test: Test getRequestedHermesAuthMethod with various raw strings without stubbing process.env
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Evidence: hermes-auth.ts:48-58 reads process.env.NEMOCLAW_HERMES_AUTH_METHOD || process.env.NEMOCLAW_HERMES_AUTH || process.env.NEMOCLAW_NOUS_AUTH_METHOD directly

PRA-6 Resolve/justify — Hermes auth module missing interactive positive-path tests for injected boundaries

  • Location: src/lib/onboard/hermes-auth.test.ts:1
  • Category: tests
  • Problem: Hermes auth module has 6 tests total (2 exit-boundary, 2 positive non-interactive selection, 2 back-navigation). Missing: interactive positive-path tests for promptHermesAuthMethod selecting API key and OAuth with injected boundaries, verifying deps.prompt, deps.getNavigationChoice called, and error/exitProcess NOT called.
  • Impact: Interactive success paths for Hermes auth method selection are not verified to work with injected error/exitProcess/log boundaries. Regression risk if boundary wiring breaks in interactive mode.
  • Recommended action: Add 2 tests to hermes-auth.test.ts: (1) interactive promptHermesAuthMethod selects API key with injected boundaries, (2) interactive promptHermesAuthMethod selects OAuth with injected boundaries. Both should verify deps.prompt, deps.getNavigationChoice, and that error/exitProcess are NOT called.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Run hermes-auth.test.ts and verify 8+ tests pass including the new interactive success paths.
  • Missing regression test: Interactive success path tests for promptHermesAuthMethod with injected boundaries.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Run hermes-auth.test.ts and verify 8+ tests pass including the new interactive success paths.
  • Evidence: hermes-auth.test.ts has 6 tests: 2 exit-boundary, 2 non-interactive positive, 2 back-navigation. No tests with isNonInteractive: () => false that verify successful interactive selection with injected boundaries.

PRA-7 Resolve/justify — Bedrock setup has optional ensureAdapter/updateSandbox parameters creating footgun

  • Location: src/lib/onboard/bedrock-runtime.ts:164
  • Category: security
  • Problem: setupBedrockRuntimeInference has optional ensureAdapter and updateSandbox parameters (with ?? fallback). The only production caller (setup-inference.ts) already provides them. Optional params create a footgun: a future caller could omit them and silently get the real registry/adapter, bypassing test injection.
  • Impact: Future callers could omit parameters and silently bypass test injection, using real adapter/registry in tests or production paths unexpectedly.
  • Recommended action: Make ensureAdapter and updateSandbox required in the function signature (remove ?). Tests already inject mocks as they do for other providers.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Verify setup-inference.ts passes both parameters (it does). Check that no other callers exist.
  • Missing regression test: TypeScript should error if a caller omits the now-required parameters.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Verify setup-inference.ts passes both parameters (it does). Check that no other callers exist.
  • Evidence: bedrock-runtime.ts:164-165 shows adapter = await (options.ensureAdapter ?? ensureBedrockRuntimeAdapter)({...}) and (options.updateSandbox ?? registry.updateSandbox)(...).

PRA-8 Resolve/justify — getRequestedHermesAuthMethod reads process.env directly, reducing testability

  • Location: src/lib/onboard/hermes-auth.ts:48
  • Category: security
  • Problem: getRequestedHermesAuthMethod still reads process.env directly (NEMOCLAW_HERMES_AUTH_METHOD, NEMOCLAW_HERMES_AUTH, NEMOCLAW_NOUS_AUTH_METHOD) instead of accepting the raw auth method string as a parameter. This reduces testability and introduces global state in a pure function.
  • Impact: Tests must stub process.env to test different auth method values; function is not pure and couples to global state.
  • Recommended action: Accept the raw auth method string as a parameter instead of reading process.env directly. The caller (promptHermesAuthMethod) already has access to the environment and can pass the value.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Check that promptHermesAuthMethod calls getRequestedHermesAuthMethod with the env value, and that tests can inject different values without stubbing process.env.
  • Missing regression test: Test that getRequestedHermesAuthMethod returns the correct normalized method when passed various raw strings, without touching process.env.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Check that promptHermesAuthMethod calls getRequestedHermesAuthMethod with the env value, and that tests can inject different values without stubbing process.env.
  • Evidence: hermes-auth.ts:48-58 reads process.env.NEMOCLAW_HERMES_AUTH_METHOD || process.env.NEMOCLAW_HERMES_AUTH || process.env.NEMOCLAW_NOUS_AUTH_METHOD directly.

PRA-9 Resolve/justify — Bedrock runtime module missing interactive/non-interactive failure boundary tests

  • Location: src/lib/onboard/bedrock-runtime.test.ts:1
  • Category: tests
  • Problem: Bedrock runtime module has 4 tests (1 new non-interactive exit boundary, 1 interactive credential prompt success, 1 interactive back navigation, 1 non-interactive AWS profile success). Missing: interactive failure boundary tests for (1) adapter startup failure, (2) provider upsert failure with status preservation, (3) inference-set failure with redaction and status preservation, (4) interactive success path with logging verification.
  • Impact: Interactive failure paths for Bedrock adapter startup, provider upsert, and inference-set are not verified to work with injected error/exitProcess/log boundaries. Regression risk if boundary wiring breaks in interactive mode.
  • Recommended action: Add 3-4 tests to bedrock-runtime.test.ts using injected boundaries for interactive failure paths: (1) interactive adapter startup failure, (2) interactive provider upsert failure with status preservation, (3) interactive inference-set failure with redaction and status preservation, (4) interactive success path with logging verification.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Run bedrock-runtime.test.ts and verify 7-8 tests pass including the new interactive failure/success paths.
  • Missing regression test: Interactive failure boundary tests for Bedrock adapter startup, provider upsert, and inference-set with injected error/exitProcess/log verification.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Run bedrock-runtime.test.ts and verify 7-8 tests pass including the new interactive failure/success paths.
  • Evidence: bedrock-runtime.test.ts has 4 tests but none cover interactive failure paths with injected boundaries.

PRA-12 Resolve/justify — Missing routed provider reconciliation failure test in failure-paths suite

  • Location: test/onboard-inference-failure-paths.test.ts:1
  • Category: tests
  • Problem: The new failure-path test file (1029 lines) comprehensively covers remote, local vLLM, Ollama, Hermes, and Bedrock failure boundaries with injected error/exitProcess/log. However, it does not test the routed provider (nvidia-router) failure path for model router reconciliation failure.
  • Impact: Routed provider reconciliation failure exit boundary not covered in the dedicated failure-path test suite.
  • Recommended action: Add a test case for routed provider reconciliation failure that verifies the injected exitProcess boundary is hit with the correct error message.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Run the test file and verify a routed provider reconciliation failure test exists and passes.
  • Missing regression test: Routed provider reconciliation failure test in onboard-inference-failure-paths.test.ts.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Run the test file and verify a routed provider reconciliation failure test exists and passes.
  • Evidence: onboard-inference-failure-paths.test.ts has 20 test cases covering all providers except routed provider reconciliation failure.

💡 In-scope improvements

These are lower-risk, not throwaway. Prefer fixing them in this PR when they are local to changed code; defer only with rationale or a linked follow-up.

PRA-5 Improvement — Simplify ProviderBranchDeps using discriminated union instead of intersection

  • Location: src/lib/onboard/setup-inference.ts:18
  • Category: architecture
  • Problem: Same root cause as PRA-1 but framed as simplification: the intersection-of-Picks pattern is fragile and hard to read. A discriminated union with a base CommonDeps and per-provider extensions would be cleaner and safer.
  • Impact: Code readability and maintainability reduced; type safety issues as described in PRA-1.
  • Suggested action: Replace the intersection with a base CommonDeps interface and per-provider extensions (RemoteProviderDeps extends CommonDeps, etc.), then use a discriminated union in createSetupInference keyed by provider.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Same as PRA-1 — add a provider-specific required field and verify TypeScript catches missing deps for other providers.
  • Missing regression test: Same as PRA-1.
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: setup-inference.ts:18-66 uses intersection of Pick types across 6 provider dep types.

PRA-10 Improvement — Missing routed provider integration test with real subprocess

  • Location: test/onboard.test.ts:1
  • Category: tests
  • Problem: No integration test for routed provider (nvidia-router) with real subprocess that triggers model router reconciliation failure and verifies exit boundary behavior. The test harness in setup-inference-test-harness.ts supports production subprocess tests.
  • Impact: Routed provider reconciliation failure exit boundary not verified end-to-end with real OpenShell gateway subprocess.
  • Suggested action: Add an integration test in test/onboard.test.ts (or new file) that spawns a real OpenShell gateway subprocess, configures a routed provider, triggers model router reconciliation failure, and verifies the exit boundary behavior. Use runProductionSetupInferenceCredentialBoundary pattern from setup-inference-test-harness.ts.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Run the new integration test and verify it exercises the routed provider path with real subprocess and injected boundaries.
  • Missing regression test: Routed provider integration test with real subprocess covering reconciliation failure and exit boundary.
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: test/support/setup-inference-test-harness.ts provides runProductionSetupInferenceCredentialBoundary for real subprocess testing; no routed provider test uses it.

PRA-11 Improvement — Local inference route retry wrapper workaround documented but removal depends on external OpenShell changes

  • Location: src/lib/onboard/local-inference-route.ts:10
  • Category: architecture
  • Problem: Local inference route retry adapter is a documented workaround for OpenShell's terminating inference set command. The source-of-truth comment correctly identifies the boundary and removal condition. This is acknowledged as acceptable with tracking.
  • Impact: Workaround adds complexity but is necessary until OpenShell provides non-terminating interactive recovery.
  • Suggested action: Accept the workaround with the documented boundary. Track the removal condition in an issue linked to OpenShell roadmap.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Verify the source-of-truth comment exists and accurately describes the invalid state, source, why source can't be fixed here, regression test location, and removal condition.
  • Missing regression test: Already covered by local-inference-route.test.ts and [NemoClaw][Onboard][Windows] nemoclaw onboard stops after step [4/8]; no sandbox created even though Ollama is responsive #4257 onboarding integration tests.
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: local-inference-route.ts:10-18 source-of-truth comment documents invalid state (failed inference set), source boundary (OpenShell), why not fixed here (OpenShell owns command), regression test (local-inference-route.test.ts + [NemoClaw][Onboard][Windows] nemoclaw onboard stops after step [4/8]; no sandbox created even though Ollama is responsive #4257), removal condition (OpenShell exposes non-terminating recovery).

PRA-13 Improvement — Verify generate-openclaw-config.test.ts compiles cleanly without @ts-nocheck

  • Location: test/generate-openclaw-config.test.ts:1
  • Category: correctness
  • Problem: The @ts-nocheck directive was removed (good). However, the test file still uses vi.mock with hoisted mocks and complex env manipulation. Consider if any remaining type suppression is needed.
  • Impact: Potential TypeScript errors suppressed; file may not compile cleanly.
  • Suggested action: Verify the file compiles cleanly without @ts-nocheck. If any @ts-ignore or @ts-expect-error comments exist, evaluate if they can be removed with proper typing.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Run tsc --noEmit on the test file and check for errors.
  • Missing regression test: TypeScript compilation should pass without suppression comments.
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: @ts-nocheck removed from file header; complex vi.mock and env manipulation remain.
Simplification opportunities: 1 possible cut, net -10 lines possible

These are safe simplification checks only. Do not remove validation, security controls, data-loss prevention, or required tests.

  • PRA-5 shrink (src/lib/onboard/setup-inference.ts:18): ProviderBranchDeps intersection of Pick<CommonDeps, ...> & Pick<HermesDeps, ...> & Pick<RemoteProviderDeps, ...> & Pick<VllmDeps, ...> & Pick<OllamaDeps, ...> & Pick<RoutedDeps, ...>
    • Replacement: Base CommonDeps interface with per-provider extension interfaces (RemoteProviderDeps extends CommonDeps, etc.) and a discriminated union keyed by provider in createSetupInference
    • Net: -10 lines
    • Safety boundary: Must preserve explicit failure boundary injection (error, exitProcess, log) for all provider branches
Test follow-ups to resolve or justify

If these cover changed behavior, prefer adding them in this PR; otherwise state why existing coverage is enough or link the follow-up.

  • PRA-T1 Runtime validation — test/onboard-inference-failure-paths.test.ts: fails through injected exit boundary when routed provider model router reconciliation fails. Runtime/sandbox/infrastructure paths need behavioral runtime validation: routed provider with real OpenShell gateway subprocess (model router reconciliation failure), production credential boundary verification. Unit coverage is strong for failure boundaries (25 cases in onboard-inference-failure-paths.test.ts) but routed provider integration test is missing.
  • PRA-T2 Runtime validation — src/lib/onboard/hermes-auth.test.ts: interactive promptHermesAuthMethod selects API key with injected boundaries. Runtime/sandbox/infrastructure paths need behavioral runtime validation: routed provider with real OpenShell gateway subprocess (model router reconciliation failure), production credential boundary verification. Unit coverage is strong for failure boundaries (25 cases in onboard-inference-failure-paths.test.ts) but routed provider integration test is missing.
  • PRA-T3 Runtime validation — src/lib/onboard/hermes-auth.test.ts: interactive promptHermesAuthMethod selects OAuth with injected boundaries. Runtime/sandbox/infrastructure paths need behavioral runtime validation: routed provider with real OpenShell gateway subprocess (model router reconciliation failure), production credential boundary verification. Unit coverage is strong for failure boundaries (25 cases in onboard-inference-failure-paths.test.ts) but routed provider integration test is missing.
  • PRA-T4 Runtime validation — src/lib/onboard/bedrock-runtime.test.ts: interactive adapter startup failure uses injected exit boundary. Runtime/sandbox/infrastructure paths need behavioral runtime validation: routed provider with real OpenShell gateway subprocess (model router reconciliation failure), production credential boundary verification. Unit coverage is strong for failure boundaries (25 cases in onboard-inference-failure-paths.test.ts) but routed provider integration test is missing.
  • PRA-T5 Runtime validation — src/lib/onboard/bedrock-runtime.test.ts: interactive provider upsert failure preserves status through injected exit. Runtime/sandbox/infrastructure paths need behavioral runtime validation: routed provider with real OpenShell gateway subprocess (model router reconciliation failure), production credential boundary verification. Unit coverage is strong for failure boundaries (25 cases in onboard-inference-failure-paths.test.ts) but routed provider integration test is missing.
  • PRA-T6 New test file not listed in test-file-size-budget.json — Add an entry for test/onboard-inference-failure-paths.test.ts in ci/test-file-size-budget.json legacyMaxLines with current line count (1029) or a reasonable ceiling (e.g., 1500).
  • PRA-T7 Hermes auth module missing interactive positive-path tests for injected boundaries — Add 2 tests to hermes-auth.test.ts: (1) interactive promptHermesAuthMethod selects API key with injected boundaries, (2) interactive promptHermesAuthMethod selects OAuth with injected boundaries. Both should verify deps.prompt, deps.getNavigationChoice, and that error/exitProcess are NOT called.
  • PRA-T8 Bedrock runtime module missing interactive/non-interactive failure boundary tests — Add 3-4 tests to bedrock-runtime.test.ts using injected boundaries for interactive failure paths: (1) interactive adapter startup failure, (2) interactive provider upsert failure with status preservation, (3) interactive inference-set failure with redaction and status preservation, (4) interactive success path with logging verification.
Since last review details

Current findings, using the urgency labels above:

PRA-1 Resolve/justify — Source-of-truth review needed: src/lib/onboard/bedrock-runtime.ts

  • Location: not file-specific
  • Category: architecture
  • Problem: The advisor marked localized patch analysis as needs_followup.
  • Impact: A localized workaround can preserve or hide an invalid state when the source boundary is unclear.
  • Recommended action: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Missing regression test: TypeScript compilation should error if caller omits required parameters
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Evidence: bedrock-runtime.ts:164-165 uses (options.ensureAdapter ?? ensureBedrockRuntimeAdapter) and (options.updateSandbox ?? registry.updateSandbox)

PRA-2 Resolve/justify — Source-of-truth review needed: src/lib/onboard/hermes-auth.ts

  • Location: not file-specific
  • Category: architecture
  • Problem: The advisor marked localized patch analysis as needs_followup.
  • Impact: A localized workaround can preserve or hide an invalid state when the source boundary is unclear.
  • Recommended action: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Missing regression test: Test getRequestedHermesAuthMethod with various raw strings without stubbing process.env
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Evidence: hermes-auth.ts:48-58 reads process.env.NEMOCLAW_HERMES_AUTH_METHOD || process.env.NEMOCLAW_HERMES_AUTH || process.env.NEMOCLAW_NOUS_AUTH_METHOD directly

PRA-3 Required — ProviderBranchDeps intersection type masks missing provider-specific dependencies

  • Location: src/lib/onboard/setup-inference.ts:18
  • Category: architecture
  • Problem: ProviderBranchDeps uses an intersection of Pick types across 6 provider dep types (CommonDeps, HermesDeps, RemoteProviderDeps, VllmDeps, OllamaDeps, RoutedDeps). This forces every provider branch to satisfy all dependencies from all other providers. Adding a dep to one provider interface (e.g., HermesDeps.lookup) makes TypeScript require it for remote/vLLM/Ollama branches too, or if a new provider-specific dep is added but not provided for other providers, the type won't catch it because the intersection is satisfied by the union of all deps.
  • Impact: Compile-time safety degraded: missing provider-specific dependencies may not be caught. Maintainers must manually verify all deps are provided for all providers when adding new ones. Refactoring risk increases as provider interfaces diverge.
  • Required action: Split into a base CommonDeps and per-provider extension types (e.g., RemoteProviderDeps extends CommonDeps, HermesDeps extends CommonDeps), then use a discriminated union keyed by provider in createSetupInference. This preserves type safety as interfaces diverge.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: Add a new required field to HermesDeps only (e.g., hermesSpecificDep: string), then call createSetupInference with a remote provider — TypeScript should error but currently won't because the intersection is satisfied by other deps.
  • Missing regression test: Add a compile-fail test in the test harness that adds a provider-specific required field to one Deps type and verifies TypeScript errors when the orchestrator doesn't provide it for other provider branches.
  • Done when: The required change is committed and verification passes: Add a new required field to HermesDeps only (e.g., hermesSpecificDep: string), then call createSetupInference with a remote provider — TypeScript should error but currently won't because the intersection is satisfied by other deps.
  • Evidence: setup-inference.ts:18-66 defines ProviderBranchDeps as intersection of Pick types from 6 provider dep types. getSetupInferenceDeps() in onboard.ts provides all deps for all providers, masking the issue at runtime.

PRA-4 Required — New test file not listed in test-file-size-budget.json

  • Location: ci/test-file-size-budget.json:1
  • Category: tests
  • Problem: New test file test/onboard-inference-failure-paths.test.ts (1029 lines) is not listed in ci/test-file-size-budget.json legacyMaxLines. The budget file tracks legacy test files that exceed the default 1500-line limit; this new file will exceed it and needs an entry.
  • Impact: CI budget check will fail for the new file, potentially blocking merges or requiring emergency budget increases.
  • Required action: Add an entry for test/onboard-inference-failure-paths.test.ts in ci/test-file-size-budget.json legacyMaxLines with current line count (1029) or a reasonable ceiling (e.g., 1500).
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: Run the test-file-size-budget check (likely part of CI) — it should fail for the new file.
  • Missing regression test: CI budget check should catch this automatically once the entry is added.
  • Done when: The required change is committed and verification passes: Run the test-file-size-budget check (likely part of CI) — it should fail for the new file.
  • Evidence: ci/test-file-size-budget.json has entries for other large test files but not for onboard-inference-failure-paths.test.ts.

PRA-5 Improvement — Simplify ProviderBranchDeps using discriminated union instead of intersection

  • Location: src/lib/onboard/setup-inference.ts:18
  • Category: architecture
  • Problem: Same root cause as PRA-1 but framed as simplification: the intersection-of-Picks pattern is fragile and hard to read. A discriminated union with a base CommonDeps and per-provider extensions would be cleaner and safer.
  • Impact: Code readability and maintainability reduced; type safety issues as described in PRA-1.
  • Suggested action: Replace the intersection with a base CommonDeps interface and per-provider extensions (RemoteProviderDeps extends CommonDeps, etc.), then use a discriminated union in createSetupInference keyed by provider.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Same as PRA-1 — add a provider-specific required field and verify TypeScript catches missing deps for other providers.
  • Missing regression test: Same as PRA-1.
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: setup-inference.ts:18-66 uses intersection of Pick types across 6 provider dep types.

PRA-6 Resolve/justify — Hermes auth module missing interactive positive-path tests for injected boundaries

  • Location: src/lib/onboard/hermes-auth.test.ts:1
  • Category: tests
  • Problem: Hermes auth module has 6 tests total (2 exit-boundary, 2 positive non-interactive selection, 2 back-navigation). Missing: interactive positive-path tests for promptHermesAuthMethod selecting API key and OAuth with injected boundaries, verifying deps.prompt, deps.getNavigationChoice called, and error/exitProcess NOT called.
  • Impact: Interactive success paths for Hermes auth method selection are not verified to work with injected error/exitProcess/log boundaries. Regression risk if boundary wiring breaks in interactive mode.
  • Recommended action: Add 2 tests to hermes-auth.test.ts: (1) interactive promptHermesAuthMethod selects API key with injected boundaries, (2) interactive promptHermesAuthMethod selects OAuth with injected boundaries. Both should verify deps.prompt, deps.getNavigationChoice, and that error/exitProcess are NOT called.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Run hermes-auth.test.ts and verify 8+ tests pass including the new interactive success paths.
  • Missing regression test: Interactive success path tests for promptHermesAuthMethod with injected boundaries.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Run hermes-auth.test.ts and verify 8+ tests pass including the new interactive success paths.
  • Evidence: hermes-auth.test.ts has 6 tests: 2 exit-boundary, 2 non-interactive positive, 2 back-navigation. No tests with isNonInteractive: () => false that verify successful interactive selection with injected boundaries.

PRA-7 Resolve/justify — Bedrock setup has optional ensureAdapter/updateSandbox parameters creating footgun

  • Location: src/lib/onboard/bedrock-runtime.ts:164
  • Category: security
  • Problem: setupBedrockRuntimeInference has optional ensureAdapter and updateSandbox parameters (with ?? fallback). The only production caller (setup-inference.ts) already provides them. Optional params create a footgun: a future caller could omit them and silently get the real registry/adapter, bypassing test injection.
  • Impact: Future callers could omit parameters and silently bypass test injection, using real adapter/registry in tests or production paths unexpectedly.
  • Recommended action: Make ensureAdapter and updateSandbox required in the function signature (remove ?). Tests already inject mocks as they do for other providers.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Verify setup-inference.ts passes both parameters (it does). Check that no other callers exist.
  • Missing regression test: TypeScript should error if a caller omits the now-required parameters.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Verify setup-inference.ts passes both parameters (it does). Check that no other callers exist.
  • Evidence: bedrock-runtime.ts:164-165 shows adapter = await (options.ensureAdapter ?? ensureBedrockRuntimeAdapter)({...}) and (options.updateSandbox ?? registry.updateSandbox)(...).

PRA-8 Resolve/justify — getRequestedHermesAuthMethod reads process.env directly, reducing testability

  • Location: src/lib/onboard/hermes-auth.ts:48
  • Category: security
  • Problem: getRequestedHermesAuthMethod still reads process.env directly (NEMOCLAW_HERMES_AUTH_METHOD, NEMOCLAW_HERMES_AUTH, NEMOCLAW_NOUS_AUTH_METHOD) instead of accepting the raw auth method string as a parameter. This reduces testability and introduces global state in a pure function.
  • Impact: Tests must stub process.env to test different auth method values; function is not pure and couples to global state.
  • Recommended action: Accept the raw auth method string as a parameter instead of reading process.env directly. The caller (promptHermesAuthMethod) already has access to the environment and can pass the value.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Check that promptHermesAuthMethod calls getRequestedHermesAuthMethod with the env value, and that tests can inject different values without stubbing process.env.
  • Missing regression test: Test that getRequestedHermesAuthMethod returns the correct normalized method when passed various raw strings, without touching process.env.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Check that promptHermesAuthMethod calls getRequestedHermesAuthMethod with the env value, and that tests can inject different values without stubbing process.env.
  • Evidence: hermes-auth.ts:48-58 reads process.env.NEMOCLAW_HERMES_AUTH_METHOD || process.env.NEMOCLAW_HERMES_AUTH || process.env.NEMOCLAW_NOUS_AUTH_METHOD directly.

PRA-9 Resolve/justify — Bedrock runtime module missing interactive/non-interactive failure boundary tests

  • Location: src/lib/onboard/bedrock-runtime.test.ts:1
  • Category: tests
  • Problem: Bedrock runtime module has 4 tests (1 new non-interactive exit boundary, 1 interactive credential prompt success, 1 interactive back navigation, 1 non-interactive AWS profile success). Missing: interactive failure boundary tests for (1) adapter startup failure, (2) provider upsert failure with status preservation, (3) inference-set failure with redaction and status preservation, (4) interactive success path with logging verification.
  • Impact: Interactive failure paths for Bedrock adapter startup, provider upsert, and inference-set are not verified to work with injected error/exitProcess/log boundaries. Regression risk if boundary wiring breaks in interactive mode.
  • Recommended action: Add 3-4 tests to bedrock-runtime.test.ts using injected boundaries for interactive failure paths: (1) interactive adapter startup failure, (2) interactive provider upsert failure with status preservation, (3) interactive inference-set failure with redaction and status preservation, (4) interactive success path with logging verification.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Run bedrock-runtime.test.ts and verify 7-8 tests pass including the new interactive failure/success paths.
  • Missing regression test: Interactive failure boundary tests for Bedrock adapter startup, provider upsert, and inference-set with injected error/exitProcess/log verification.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Run bedrock-runtime.test.ts and verify 7-8 tests pass including the new interactive failure/success paths.
  • Evidence: bedrock-runtime.test.ts has 4 tests but none cover interactive failure paths with injected boundaries.

PRA-10 Improvement — Missing routed provider integration test with real subprocess

  • Location: test/onboard.test.ts:1
  • Category: tests
  • Problem: No integration test for routed provider (nvidia-router) with real subprocess that triggers model router reconciliation failure and verifies exit boundary behavior. The test harness in setup-inference-test-harness.ts supports production subprocess tests.
  • Impact: Routed provider reconciliation failure exit boundary not verified end-to-end with real OpenShell gateway subprocess.
  • Suggested action: Add an integration test in test/onboard.test.ts (or new file) that spawns a real OpenShell gateway subprocess, configures a routed provider, triggers model router reconciliation failure, and verifies the exit boundary behavior. Use runProductionSetupInferenceCredentialBoundary pattern from setup-inference-test-harness.ts.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Run the new integration test and verify it exercises the routed provider path with real subprocess and injected boundaries.
  • Missing regression test: Routed provider integration test with real subprocess covering reconciliation failure and exit boundary.
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: test/support/setup-inference-test-harness.ts provides runProductionSetupInferenceCredentialBoundary for real subprocess testing; no routed provider test uses it.

PRA-11 Improvement — Local inference route retry wrapper workaround documented but removal depends on external OpenShell changes

  • Location: src/lib/onboard/local-inference-route.ts:10
  • Category: architecture
  • Problem: Local inference route retry adapter is a documented workaround for OpenShell's terminating inference set command. The source-of-truth comment correctly identifies the boundary and removal condition. This is acknowledged as acceptable with tracking.
  • Impact: Workaround adds complexity but is necessary until OpenShell provides non-terminating interactive recovery.
  • Suggested action: Accept the workaround with the documented boundary. Track the removal condition in an issue linked to OpenShell roadmap.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Verify the source-of-truth comment exists and accurately describes the invalid state, source, why source can't be fixed here, regression test location, and removal condition.
  • Missing regression test: Already covered by local-inference-route.test.ts and [NemoClaw][Onboard][Windows] nemoclaw onboard stops after step [4/8]; no sandbox created even though Ollama is responsive #4257 onboarding integration tests.
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: local-inference-route.ts:10-18 source-of-truth comment documents invalid state (failed inference set), source boundary (OpenShell), why not fixed here (OpenShell owns command), regression test (local-inference-route.test.ts + [NemoClaw][Onboard][Windows] nemoclaw onboard stops after step [4/8]; no sandbox created even though Ollama is responsive #4257), removal condition (OpenShell exposes non-terminating recovery).

PRA-12 Resolve/justify — Missing routed provider reconciliation failure test in failure-paths suite

  • Location: test/onboard-inference-failure-paths.test.ts:1
  • Category: tests
  • Problem: The new failure-path test file (1029 lines) comprehensively covers remote, local vLLM, Ollama, Hermes, and Bedrock failure boundaries with injected error/exitProcess/log. However, it does not test the routed provider (nvidia-router) failure path for model router reconciliation failure.
  • Impact: Routed provider reconciliation failure exit boundary not covered in the dedicated failure-path test suite.
  • Recommended action: Add a test case for routed provider reconciliation failure that verifies the injected exitProcess boundary is hit with the correct error message.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Run the test file and verify a routed provider reconciliation failure test exists and passes.
  • Missing regression test: Routed provider reconciliation failure test in onboard-inference-failure-paths.test.ts.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Run the test file and verify a routed provider reconciliation failure test exists and passes.
  • Evidence: onboard-inference-failure-paths.test.ts has 20 test cases covering all providers except routed provider reconciliation failure.

PRA-13 Improvement — Verify generate-openclaw-config.test.ts compiles cleanly without @ts-nocheck

  • Location: test/generate-openclaw-config.test.ts:1
  • Category: correctness
  • Problem: The @ts-nocheck directive was removed (good). However, the test file still uses vi.mock with hoisted mocks and complex env manipulation. Consider if any remaining type suppression is needed.
  • Impact: Potential TypeScript errors suppressed; file may not compile cleanly.
  • Suggested action: Verify the file compiles cleanly without @ts-nocheck. If any @ts-ignore or @ts-expect-error comments exist, evaluate if they can be removed with proper typing.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Run tsc --noEmit on the test file and check for errors.
  • Missing regression test: TypeScript compilation should pass without suppression comments.
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: @ts-nocheck removed from file header; complex vi.mock and env manipulation remain.

Workflow run details

This is an automated, non-binding review; it still expects maintainers and agents to respond to each required or warning item. Treat suggestions as current-PR improvements when they touch changed code; defer only with maintainer rationale or a linked follow-up. A human maintainer must make the final merge decision.

@github-actions

github-actions Bot commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

E2E Advisor Recommendation

Required E2E: onboard-resume, onboard-repair, cloud-onboard, inference-routing, model-router-provider-routed-inference, ollama-auth-proxy, hermes-e2e, bedrock-runtime-compatible-anthropic
Optional E2E: hermes-inference-switch, messaging-compatible-endpoint, rebuild-hermes

Dispatch hint: onboard-resume,onboard-repair,cloud-onboard,inference-routing,model-router-provider-routed-inference,ollama-auth-proxy,hermes-e2e,bedrock-runtime-compatible-anthropic

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

  • onboard-resume (medium): Required by the onboarding resume rule: setupInference and provider setup changes can affect resumed onboarding inference state transitions and resumed local-provider recreation.
  • onboard-repair (medium): Required by the onboarding resume/repair policy because the PR changes provider/inference setup used by repair and recovery paths.
  • cloud-onboard (high): The common hosted onboarding provider setup path was refactored; run the full non-interactive hosted onboarding flow with real OpenShell sandbox lifecycle and inference smoke.
  • inference-routing (medium): Provider inference route application, verification, and failure handling changed across remote/local providers; this job directly validates live inference route behavior.
  • model-router-provider-routed-inference (medium): src/lib/onboard/inference-providers/routed.ts changed the routed provider setup and now checks inference set failure handling; run the routed inference/model-router live contract.
  • ollama-auth-proxy (medium): Ollama local provider setup, proxy credential flow, and local inference route dependencies changed; run the live Ollama auth proxy route/credential boundary test.
  • hermes-e2e (high): Hermes auth and Hermes Provider inference setup changed, including credential preparation, provider-store checks, exit boundaries, and inference route finalization.
  • bedrock-runtime-compatible-anthropic (medium): Bedrock Runtime compatible Anthropic selection/setup code changed, including auth detection, adapter setup injection, provider upsert, route application, and registry update.

Optional E2E

  • hermes-inference-switch (high): Useful adjacent confidence for Hermes provider switching and compatible Anthropic route behavior after the inference setup refactor, but less central than first-onboard Hermes setup.
  • messaging-compatible-endpoint (medium): Useful coverage for fake OpenAI-compatible endpoint assistant flow after probe fallback and remote compatible endpoint setup changes.
  • rebuild-hermes (high): The PR includes a resumed local-provider recreation test and changes provider setup dependencies; rebuild-hermes gives extra live coverage of rebuild plus provider reuse/reconfiguration.

New E2E recommendations

  • local vLLM provider setup (medium): The PR changes vllm-local setup dependencies and exit/error handling, but the existing required local-provider E2E coverage is stronger for Ollama than vLLM.
    • Suggested test: Add a live vllm-local inference setup smoke target that validates provider upsert, local route application, and inference route verification without requiring a large GPU model when possible.

Dispatch hint

  • Workflow: .github/workflows/e2e.yaml
  • jobs input: onboard-resume,onboard-repair,cloud-onboard,inference-routing,model-router-provider-routed-inference,ollama-auth-proxy,hermes-e2e,bedrock-runtime-compatible-anthropic

@github-actions

github-actions Bot commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

E2E Target Recommendation

Required E2E targets: onboard-resume, onboard-repair, ubuntu-repo-cloud-openclaw, ollama-auth-proxy, model-router-provider-routed-inference
Optional E2E targets: gpu-e2e, onboard-negative-paths

Dispatch required E2E targets:

  • gh workflow run e2e.yaml --ref <pr-head-ref> --field jobs=onboard-resume
  • gh workflow run e2e.yaml --ref <pr-head-ref> --field jobs=onboard-repair
  • gh workflow run e2e.yaml --ref <pr-head-ref> --field targets=ubuntu-repo-cloud-openclaw
  • gh workflow run e2e.yaml --ref <pr-head-ref> --field jobs=ollama-auth-proxy
  • gh workflow run e2e.yaml --ref <pr-head-ref> --field jobs=model-router-provider-routed-inference

Workflow run

Full E2E target advisor summary

E2E Target Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E targets

  • onboard-resume: Changes refactor setupInference and provider setup boundaries inside the onboarding flow. The onboarding resume rule requires the live onboard-resume job for changes that can affect resumed onboarding state, session bootstrap, or provider setup after an interrupted session.
    • Dispatch: gh workflow run e2e.yaml --ref <pr-head-ref> --field jobs=onboard-resume
  • onboard-repair: The same setupInference/provider changes can affect repair/backstop execution from persisted onboarding sessions, including provider recreation and route repair paths. The onboarding resume rule makes onboard-repair required for these state-machine resume/repair surfaces.
    • Dispatch: gh workflow run e2e.yaml --ref <pr-head-ref> --field jobs=onboard-repair
  • ubuntu-repo-cloud-openclaw: Primary live-supported registry target for normal cloud OpenClaw onboarding. It exercises the refactored setupInference dispatcher and remote provider setup/finalization on the standard Ubuntu Docker path.
    • Dispatch: gh workflow run e2e.yaml --ref <pr-head-ref> --field targets=ubuntu-repo-cloud-openclaw
  • ollama-auth-proxy: Local Ollama setup, local inference route application, and auth-proxy error handling changed. The ollama-auth-proxy job is the wired live E2E coverage for that local-provider surface.
    • Dispatch: gh workflow run e2e.yaml --ref <pr-head-ref> --field jobs=ollama-auth-proxy
  • model-router-provider-routed-inference: Routed inference setup now checks inference route application failures through injected boundaries. The model-router-provider-routed-inference job directly exercises the routed/model-router provider path.
    • Dispatch: gh workflow run e2e.yaml --ref <pr-head-ref> --field jobs=model-router-provider-routed-inference

Optional E2E targets

  • gpu-e2e: Optional special-runner coverage for local GPU/vLLM-adjacent inference behavior touched by the setupInference refactor. Run if the PR owner wants hardware-backed local inference evidence beyond the primary Ubuntu jobs.
    • Dispatch: gh workflow run e2e.yaml --ref <pr-head-ref> --field jobs=gpu-e2e
  • onboard-negative-paths: Optional additional live coverage for non-interactive onboarding failure boundaries; useful because several changed provider helpers now route errors through injected exit/error boundaries.
    • Dispatch: gh workflow run e2e.yaml --ref <pr-head-ref> --field jobs=onboard-negative-paths

Relevant changed files

  • src/lib/onboard.ts
  • src/lib/onboard/bedrock-runtime.ts
  • src/lib/onboard/hermes-auth.ts
  • src/lib/onboard/inference-providers/hermes.ts
  • src/lib/onboard/inference-providers/ollama-local.ts
  • src/lib/onboard/inference-providers/remote.ts
  • src/lib/onboard/inference-providers/routed.ts
  • src/lib/onboard/inference-providers/types.ts
  • src/lib/onboard/inference-providers/vllm-local.ts
  • src/lib/onboard/local-inference-route.ts
  • src/lib/onboard/setup-inference.ts
  • src/lib/onboard/windows-host-ollama.ts

@github-actions

github-actions Bot commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings

Merge posture: No blocking advisor findings
Primary next action: Add or justify PRA-T1 and any related test follow-ups.
Open items: 0 required · 0 warnings · 0 suggestions · 8 test follow-ups
Since last review: 1 prior item resolved · 0 still apply · 0 new items found

Action checklist

  • PRA-T1 Add or justify test follow-up: Runtime validation
  • PRA-T2 Add or justify test follow-up: Runtime validation
  • PRA-T3 Add or justify test follow-up: Runtime validation
  • PRA-T4 Add or justify test follow-up: Runtime validation
  • PRA-T5 Add or justify test follow-up: Acceptance clause
  • PRA-T6 Add or justify test follow-up: Acceptance clause
  • PRA-T7 Add or justify test follow-up: Acceptance clause
  • PRA-T8 Add or justify test follow-up: Acceptance clause
Test follow-ups to resolve or justify

If these cover changed behavior, prefer adding them in this PR; otherwise state why existing coverage is enough or link the follow-up.

  • PRA-T1 Runtime validation — Identify or run the existing fake-OpenShell production-boundary test that exercises exported `setupInference` for a remote provider and verifies raw credentials stay out of argv while only the provider mutation receives the scoped credential env.. Static and process-boundary coverage is strong for sensitive onboarding/inference changes, including fake-OpenShell credential scoping and direct failure-boundary tests. Because these paths depend on OpenShell command behavior and local provider networking, targeted runtime validation remains useful to corroborate the static evidence.
  • PRA-T2 Runtime validation — Identify or run the local-provider route failure coverage that exercises `createLocalInferenceRouteApplier` for Ollama or vLLM and verifies redacted diagnostics plus non-interactive exit status preservation.. Static and process-boundary coverage is strong for sensitive onboarding/inference changes, including fake-OpenShell credential scoping and direct failure-boundary tests. Because these paths depend on OpenShell command behavior and local provider networking, targeted runtime validation remains useful to corroborate the static evidence.
  • PRA-T3 Runtime validation — Identify or run the routed-provider route-apply failure coverage that makes `openshell inference set` fail and verifies redaction, injected exit status, and no shared finalization (`verifyInferenceRoute`, smoke, registry update) after failure.. Static and process-boundary coverage is strong for sensitive onboarding/inference changes, including fake-OpenShell credential scoping and direct failure-boundary tests. Because these paths depend on OpenShell command behavior and local provider networking, targeted runtime validation remains useful to corroborate the static evidence.
  • PRA-T4 Runtime validation — If benchmark acceptance is required for this PR, attach or identify the artifact that records the 44.34s-to-35.24s onboarding median and 7,654-to-6,444 aggregate `execve` reduction.. Static and process-boundary coverage is strong for sensitive onboarding/inference changes, including fake-OpenShell credential scoping and direct failure-boundary tests. Because these paths depend on OpenShell command behavior and local provider networking, targeted runtime validation remains useful to corroborate the static evidence.
  • PRA-T5 Acceptance clause — Part of perf(test): reduce test I/O to restore 2–5 minute full-suite runs #6245. — add test evidence or identify existing coverage. The deterministic validation context reported linkedIssues: [], so issue perf(test): reduce test I/O to restore 2–5 minute full-suite runs #6245 body/comments and literal issue acceptance clauses were not available for independent mapping.
  • PRA-T6 Acceptance clause — Replace 46 service-environment harness child calls with equivalent Node filesystem operations and cache the three-proxy fixture input. — add test evidence or identify existing coverage. `test/service-env.test.ts` now caches `defaultProxyVars`, `hostOverrideProxyVars`, and `portOverrideProxyVars` in a `beforeAll()` block and reuses them across proxy assertions. This read-only review did not independently count the exact 46 removed child calls.
  • PRA-T7 Acceptance clause — Expand the direct dependency-failure suite to 25 cases, including remote/Bedrock exit boundaries, credential/upsert/apply failures, falsey status fallbacks, local providers, Ollama proxy recovery, routed reconciliation/upsert/route-apply failures, Hermes provider-store/credential/lookup failures, and a focused real Responses-to-Chat-Completions probe fallback test. — add test evidence or identify existing coverage. `test/onboard-inference-failure-paths.test.ts` covers remote missing config/credential/upsert/apply paths, local vLLM/Ollama failures, Hermes provider-store/prep/DNS paths, Bedrock auth/adapter/upsert/apply paths, and routed reconciliation/upsert/apply/finalization. `src/lib/inference/onboard-probes-responses-fallback.test.ts` covers Responses-to-Chat-Completions fallback ordering. The exact count of 25 cases was not independently counted.
  • PRA-T8 Acceptance clause — Invoke the exported messaging post-install phase directly in OpenClaw config tests, eliminating 98 redundant outer applier launches while preserving all 20 real `openclaw doctor` launches and generator/applier executable contracts. — add test evidence or identify existing coverage. `test/generate-openclaw-config.test.ts::runMessagingPostInstall()` now directly calls `applyMessagingBuildPhase(readMessagingBuildPlanFromEnv(...), "post-agent-install", env)`. `runConfigSubprocess()` and the `runs as a node --experimental-strip-types executable` test retain generator executable coverage. This read-only review did not independently verify the exact 98/20 launch counts.

Workflow run details

This is an automated, non-binding review; it still expects maintainers and agents to respond to each required or warning item. Treat suggestions as current-PR improvements when they touch changed code; defer only with maintainer rationale or a linked follow-up. A human maintainer must make the final merge decision.

cv added 12 commits July 4, 2026 09:36
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@cv

cv commented Jul 4, 2026

Copy link
Copy Markdown
Collaborator Author

Final-head advisor disposition (256067f3b):

  • Routed-provider inference set failure handling is a valid adjacent hardening opportunity, but it is not introduced by this PR. This PR retains the existing routed command behavior while adding deterministic coverage for reconciliation and provider-upsert failures. A focused follow-up should add ignoreError, redacted/compacted diagnostics, injected exit handling, and direct status-preservation coverage without extending this already validated performance PR.
  • ProviderBranchDeps intersection/Omit cleanup is a compile-time architecture refactor, not a runtime correctness or security fix. Current per-provider dependency objects remain checked against their concrete dependency types. Defer the larger discriminated-union redesign to a focused follow-up.
  • The local-route retry wrapper predates this PR; this PR only moves its construction behind the merged dependency seam while preserving its existing recovery policy and tests. A broader source-of-truth/recovery-policy review belongs in a separate design-focused change.

There are no required findings on the final head. We are freezing this PR after the batched fixes and treating these warnings as follow-up work to avoid another full push/CI cycle.

cv added 3 commits July 4, 2026 12:06
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Make provider dependency ownership explicit so new requirements must be wired deliberately.

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@cv

cv commented Jul 4, 2026

Copy link
Copy Markdown
Collaborator Author

Final-head advisor disposition (1d9d09be6):

  • GPT-5.5 reports merge_as_is with no required findings, warnings, or suggestions.
  • Nemotron PRA-1/PRA-3 is not a missing-dependency defect. The blanket merged Omit was replaced by explicit provider-owned Pick composition, commonDeps satisfies CommonDeps, an explicitly typed production default set, and concrete *Deps checking at every provider call. A discriminated dependency union cannot describe this factory cleanly because one created setupInference function intentionally handles different provider values on successive invocations.
  • Nemotron PRA-2 is factually incorrect. test/onboard-inference-failure-paths.test.ts is 1,029 lines and is covered by defaultMaxLines: 1500; npm run test-size:check passes. legacyMaxLines is reserved for files already above the default, so adding this file there would weaken the normal policy rather than fix CI.
  • Hermes interactive selection is already exercised through the prompt/back cases and broader onboarding coverage; the added focused tests cover both non-interactive choices and both navigation boundaries. Additional permutations would duplicate decision-table coverage.
  • Bedrock adapter/registry defaults are intentional production implementations, unlike the now-required error/log/exit boundaries. Focused and direct setup tests already cover interactive adapter recovery, provider status/fallback, inference-set status/redaction/fallback, and success logging across the source and direct-seam suites.
  • Reading the Hermes auth-method environment is the intended CLI configuration boundary, not an untestable pure helper; all relevant values are covered with scoped environment stubs.
  • A new routed real-subprocess failure test would duplicate the direct routed failure cases and the existing live routed E2E while adding the process overhead perf(test): reduce test I/O to restore 2–5 minute full-suite runs #6245 is removing.

No unresolved final-head advisor item requires another code change. The branch remains frozen while required CI completes.

@cv
cv marked this pull request as ready for review July 4, 2026 19:28

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (4)
src/lib/onboard/inference-providers/types.ts (1)

71-77: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Consider tightening classification/ClassifyApplyFailure typing.

classification: any and ClassifyApplyFailure's any return weaken the otherwise-improved type safety in this same edit. Not blocking, but since the surrounding types were just hardened, it'd be a small consistency win to type classification as the actual classification shape.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/onboard/inference-providers/types.ts` around lines 71 - 77, Tighten
the typings in the shared inference-provider types by replacing the loose `any`
usage in `ClassifyApplyFailure` and the `classification` parameter with the
actual classification shape used by this flow. Update the relevant type alias
and any related function signatures in `types.ts` so `classification` is
explicitly typed instead of `any`, and make `ClassifyApplyFailure` return the
concrete failure/classification result type rather than `any`.
test/service-env.test.ts (1)

71-98: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Consider consolidating the four near-identical snippet extractors.

extractToolRedirectsSnippet and extractProxyVarsSnippet repeat the same read/indexOf/throw scaffolding as the pre-existing extractRuntimeShellEnvSnippet/extractRuntimeShellEnvShimSnippet. A shared extractScriptSnippet(startMarker, endMarker, errorContext) helper (handling the 2-marker case, with the 3-marker tool-redirects case as a thin wrapper) would remove the duplication without changing behavior.

♻️ Example shared helper
+function extractScriptSnippet(startMarker: string, endMarker: string, errorContext: string, from = 0) {
+  const src = readFileSync(NEMOCLAW_START_SCRIPT, "utf-8");
+  const start = src.indexOf(startMarker, from);
+  const end = src.indexOf(endMarker, start);
+  if (start === -1 || end === -1 || end <= start) {
+    throw new Error(`Failed to extract ${errorContext} from scripts/nemoclaw-start.sh`);
+  }
+  return { src, start, end };
+}
+
 function extractProxyVarsSnippet() {
-  const src = readFileSync(NEMOCLAW_START_SCRIPT, "utf-8");
-  const start = src.indexOf("PROXY_HOST=");
-  const endMarker = 'export no_proxy="$_NO_PROXY_VAL"';
-  const end = src.indexOf(endMarker, start);
-  if (start === -1 || end === -1 || end <= start) {
-    throw new Error(
-      "Failed to extract proxy configuration from scripts/nemoclaw-start.sh — " +
-        "the PROXY_HOST..no_proxy block may have been moved or renamed",
-    );
-  }
-  return src.slice(start, end + endMarker.length);
+  const endMarker = 'export no_proxy="$_NO_PROXY_VAL"';
+  const { src, start, end } = extractScriptSnippet("PROXY_HOST=", endMarker, "proxy configuration");
+  return src.slice(start, end + endMarker.length);
 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/service-env.test.ts` around lines 71 - 98, The two new snippet
extractors duplicate the same read/indexOf/error scaffolding already used by
extractRuntimeShellEnvSnippet and extractRuntimeShellEnvShimSnippet. Refactor
these helpers in test/service-env.test.ts to use a shared
extractScriptSnippet-style utility that takes start/end markers and an error
context, with a thin wrapper for the three-marker _TOOL_REDIRECTS case in
extractToolRedirectsSnippet. Keep the existing behavior and error messages
equivalent while removing the repeated logic in extractProxyVarsSnippet and the
other snippet extractors.
src/lib/onboard/inference-providers/vllm-local.ts (1)

35-44: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

console.warn left un-migrated in this DI boundary.

The failure branch (lines 45-49) was migrated to injected error/exitProcess, but the healthy-host warning branch still calls console.warn directly and log isn't part of this function's destructured deps. This leaves a gap in the DI boundary this PR is establishing, and this warning path can't be asserted/redirected by the new test harness.

♻️ Proposed fix
-  const { upsertProvider, validateLocalProvider, getLocalProviderHealthCheck, getLocalProviderBaseUrl, applyLocalInferenceRoute, run, VLLM_LOCAL_CREDENTIAL_ENV, exitProcess, error } = deps;
+  const { upsertProvider, validateLocalProvider, getLocalProviderHealthCheck, getLocalProviderBaseUrl, applyLocalInferenceRoute, run, VLLM_LOCAL_CREDENTIAL_ENV, exitProcess, error, log } = deps;
   ...
     if (hostResponding) {
-      console.warn(`  ⚠ ${validation.message}`);
+      log(`  ⚠ ${validation.message}`);
       if (validation.diagnostic) {
-        console.warn(`  Diagnostic: ${validation.diagnostic}`);
+        log(`  Diagnostic: ${validation.diagnostic}`);
       }
-      console.warn(
+      log(
         "  The server is healthy on the host — continuing. " +
           "The sandbox uses a different network path and may work correctly.",
       );
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/onboard/inference-providers/vllm-local.ts` around lines 35 - 44, The
healthy-host warning path in vllm-local’s validation flow still uses
console.warn directly, leaving the new dependency-injected boundary incomplete.
Update the same branch that handles hostResponding to use the injected logging
dependency instead of console.warn, and make sure the function’s destructured
deps include the logger used there so the warning can be captured in tests
alongside the existing injected error/exitProcess path.
test/onboard-inference-failure-paths.test.ts (1)

12-16: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Minor: bedrockRuntimeOnboard could use a plain ESM import.

Unlike ../src/lib/onboard (documented to need require() due to its bottom-of-file module.exports pattern), bedrock-runtime.ts doesn't have that constraint, so import * as bedrockRuntimeOnboard from "../src/lib/onboard/bedrock-runtime.js" would align with the coding guideline for root-level integration tests to "use ESM imports." Not functionally impactful under the current TS/vitest interop config.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/onboard-inference-failure-paths.test.ts` around lines 12 - 16,
`bedrockRuntimeOnboard` is using `require()` even though `bedrock-runtime.ts`
does not need CommonJS interop. Update the root-level test setup in the
onboarding failure-paths spec to use a plain ESM namespace import for
`../src/lib/onboard/bedrock-runtime.js`, while keeping the existing `onboard`
require only if needed for its `module.exports` pattern. This keeps the import
style aligned with the guideline and the existing
`createSetupInference`/`bedrockRuntimeOnboard` references should make the change
easy to locate.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/onboard/inference-providers/routed.ts`:
- Around line 47-57: Add a failure-path test for setupRoutedInference covering
the non-zero runOpenshell path in routed.ts. Exercise the branch where the
"inference set" command returns a non-zero status, then assert that the error
handling path logs the failure message and calls exitProcess with the returned
status (or 1 fallback). Use setupRoutedInference, runOpenshell, and exitProcess
as the key symbols to locate and cover this behavior.

---

Nitpick comments:
In `@src/lib/onboard/inference-providers/types.ts`:
- Around line 71-77: Tighten the typings in the shared inference-provider types
by replacing the loose `any` usage in `ClassifyApplyFailure` and the
`classification` parameter with the actual classification shape used by this
flow. Update the relevant type alias and any related function signatures in
`types.ts` so `classification` is explicitly typed instead of `any`, and make
`ClassifyApplyFailure` return the concrete failure/classification result type
rather than `any`.

In `@src/lib/onboard/inference-providers/vllm-local.ts`:
- Around line 35-44: The healthy-host warning path in vllm-local’s validation
flow still uses console.warn directly, leaving the new dependency-injected
boundary incomplete. Update the same branch that handles hostResponding to use
the injected logging dependency instead of console.warn, and make sure the
function’s destructured deps include the logger used there so the warning can be
captured in tests alongside the existing injected error/exitProcess path.

In `@test/onboard-inference-failure-paths.test.ts`:
- Around line 12-16: `bedrockRuntimeOnboard` is using `require()` even though
`bedrock-runtime.ts` does not need CommonJS interop. Update the root-level test
setup in the onboarding failure-paths spec to use a plain ESM namespace import
for `../src/lib/onboard/bedrock-runtime.js`, while keeping the existing
`onboard` require only if needed for its `module.exports` pattern. This keeps
the import style aligned with the guideline and the existing
`createSetupInference`/`bedrockRuntimeOnboard` references should make the change
easy to locate.

In `@test/service-env.test.ts`:
- Around line 71-98: The two new snippet extractors duplicate the same
read/indexOf/error scaffolding already used by extractRuntimeShellEnvSnippet and
extractRuntimeShellEnvShimSnippet. Refactor these helpers in
test/service-env.test.ts to use a shared extractScriptSnippet-style utility that
takes start/end markers and an error context, with a thin wrapper for the
three-marker _TOOL_REDIRECTS case in extractToolRedirectsSnippet. Keep the
existing behavior and error messages equivalent while removing the repeated
logic in extractProxyVarsSnippet and the other snippet extractors.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a93554a8-ec09-4358-82b2-6dc978ba2545

📥 Commits

Reviewing files that changed from the base of the PR and between 1162e89 and 1d9d09b.

📒 Files selected for processing (28)
  • ci/test-file-size-budget.json
  • src/lib/actions/sandbox/rebuild-local-provider-recreate.test.ts
  • src/lib/inference/onboard-probes-curl-harness.ts
  • src/lib/inference/onboard-probes-responses-fallback.test.ts
  • src/lib/onboard.ts
  • src/lib/onboard/bedrock-runtime.test.ts
  • src/lib/onboard/bedrock-runtime.ts
  • src/lib/onboard/hermes-auth.test.ts
  • src/lib/onboard/hermes-auth.ts
  • src/lib/onboard/inference-providers/hermes.test.ts
  • src/lib/onboard/inference-providers/hermes.ts
  • src/lib/onboard/inference-providers/ollama-local.ts
  • src/lib/onboard/inference-providers/remote.ts
  • src/lib/onboard/inference-providers/routed.ts
  • src/lib/onboard/inference-providers/types.ts
  • src/lib/onboard/inference-providers/vllm-local.ts
  • src/lib/onboard/local-inference-route.test.ts
  • src/lib/onboard/local-inference-route.ts
  • src/lib/onboard/setup-inference.ts
  • src/lib/onboard/windows-host-ollama.test.ts
  • src/lib/onboard/windows-host-ollama.ts
  • test/generate-openclaw-config.test.ts
  • test/onboard-inference-failure-paths.test.ts
  • test/onboard-selection.test.ts
  • test/onboard.test.ts
  • test/service-env.test.ts
  • test/support/onboard-selection-test-helpers.ts
  • test/support/setup-inference-test-harness.ts

Comment thread src/lib/onboard/inference-providers/routed.ts
@cv
cv merged commit 002ac62 into main Jul 4, 2026
55 of 56 checks passed
@cv
cv deleted the codex/perf-test-io-6245 branch July 4, 2026 21:09
@cv cv added the v0.0.75 label Jul 5, 2026
cv added a commit that referenced this pull request Jul 7, 2026
<!-- markdownlint-disable MD041 -->
## Summary
Run the integration project as a bounded four-worker phase during the
canonical local `npm test`, while keeping CI, coverage, focused
integration, and direct Vitest runs serialized. Isolate two onboarding
fixtures from host-global dashboard ports so the parallel suite remains
deterministic. This is the final cumulative #6245 step after the named
onboarding conversions, representative process-contract work, and
sequenced loader cleanup already merged; the final clean-build Node 22
suite passes in 3:52.03.

## Related Issue
Closes #6245.

## Changes
- Replace the dashboard-exhaustion fixture's real host listeners with a
fake `lsof` while retaining the real CLI, preflight, diagnostic, and
non-zero exit contract.
- Give the restore-intent fixture an explicit existing dashboard forward
so unrelated host port occupancy cannot divert the behavior under test.
- Resolve integration scheduling from npm lifecycle, CI, coverage, and
worker-cap inputs: local `npm test` uses at most four workers in group
1, while every safety-sensitive route stays serial.
- Add a behavior matrix covering local, CI, coverage, focused, direct,
and explicit worker-throttle modes.
- Complete the cumulative #6245 acceptance path after #6276/#6336/#6383
converted the named onboarding hotspots, #6285/#6417 retained
representative process contracts, and #6286/#6299/#6388/#6415 sequenced
loader cleanup after process removal.
- Record the final host-specific timings, hotspot disposition, and
retained process-contract inventory in `test/README.md` as an advisory
acceptance snapshot rather than a permanent CI budget.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates
<!-- Check exactly one tests line and one docs line. Check other lines
when applicable. Add every requested justification or approval
reference. -->
- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: Test fixtures and local
test-runner scheduling changed; NemoClaw commands, configuration,
runtime behavior, and CI/coverage workflows are unchanged.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: Independent final-diff
review confirmed that the fake `lsof` preserves the real
CLI/preflight/exit contract, the restore-intent assertions remain
intact, and resolved CI/coverage configurations remain serialized.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification
<!-- Check each applicable item only when supported by the requested
evidence. Run targeted tests once per relevant change set and rerun
after later edits or hook autofixes that can affect the tested behavior.
Do not rerun hook-covered checks. -->
- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — command/result or justification: Real
CLI exhaustion contract passed; restore-intent passed with all 11
dashboard ports deliberately occupied; scheduling matrix passed 14/14
through the lifecycle-triggered config; `npm run test:projects:check`
reported 1,327 files disjoint across 8 projects.
- [x] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: Clean-build Node 22 `npm
test -- --reporter=blob` under the normal `umask 022` passed 1,251 files
and 13,879 tests with 39 skipped, 1 todo, and zero failures in 3:52.03,
down 73% from the issue's 14:19.65 baseline despite a larger suite. The
matching diff-scoped routine pre-commit stage passed in 13.95s. #6270
separately removed full coverage from routine pre-commit while
preserving manual and authoritative CI gates.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
<!-- DCO sign-off is required in this PR description, and every commit
must appear as Verified in GitHub. Run: git config user.name && git
config user.email -->
Signed-off-by: Carlos Villela <cvillela@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Integration test runs now use adaptive scheduling to speed up local
execution while keeping CI/focused runs serialized.

* **Bug Fixes**
* Improved reliability of onboarding regression coverage by simulating
dashboard port exhaustion in a hermetic way.
* Updated onboarding-related fixtures to better match the intended
readiness/exit behavior.

* **Tests**
* Added coverage for integration scheduling behavior (local caps,
invalid inputs, and CI/coverage scenarios).

* **Documentation**
* Added test-suite documentation with a local performance snapshot and
key test hotspots.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
jyaunches added a commit that referenced this pull request Jul 9, 2026
…e step into modules (#6444)

<!-- markdownlint-disable MD041 -->
## Summary

Extracts cohesive units of the sandbox create/register orchestration out
of the ~4,900-line `src/lib/onboard.ts` entrypoint into focused modules
under `src/lib/onboard/`, following the injected-deps boundary style
established by `created-sandbox-finalization.ts` (#6332). The primary
goal is maintainability and independent unit-test coverage for the
create path, with intentional safety/behavior refinements discovered
during review: redacted create-output failure echoing, preservation of
non-zero create-stream status when readiness fails, direct argv spawning
for trusted create paths, fail-closed Docker-GPU create-poll side-effect
handling, and redacted trace reporting for poll/readiness errors.

Note on the issue's premise: #6258 cites a `+27 net lines` growth-guard
violation from #6166. That premise is stale — the merged #6166 left
`onboard.ts` net-smaller, and #6276/#6332 shrank it further, so the
guard (`.github/workflows/codebase-growth-guardrails.yaml`, a per-PR
net-neutral diff gate) is not currently red. This PR is therefore
incremental maintainability work; it keeps `onboard.ts`
net-neutral-or-smaller so the gate stays green.

## Related Issue

Refs #6258

<!-- Refs (not Fixes): this PR lands two increments of a larger
extraction; the remaining create/finalize wiring is left for follow-ups,
so the issue should stay open. -->

## Changes

- Add `src/lib/onboard/created-sandbox-failure.ts`:
`reportSandboxCreateFailure` (warns-and-continues on an incomplete
create; otherwise prints diagnostics + recovery hints and exits) and
`reportSandboxReadinessFailure` (prints the readiness failure, defers
cleanup to the Docker-GPU patch or deletes the failed sandbox, then
exits). `onboard.ts` replaces the two inline blocks with module calls —
net −2 lines.
- Add `src/lib/onboard/sandbox-create-step.ts`: `runSandboxCreateStep`
encapsulates the BuildKit prebuild handoff → Docker-GPU create-patch
provisioning → create-stream behind a context + injected-deps boundary.
This move is **line-neutral** on `onboard.ts`; its value is a named,
unit-testable boundary (the `prepare → patch → stream` sequence is now
testable without standing up the entrypoint), not a size reduction.
Build-context and exit-listener cleanup stay with the caller that armed
them.
- Add focused unit tests across `created-sandbox-failure.test.ts`,
`sandbox-create-step.test.ts`, `sandbox-create-launch.test.ts`,
`create-stream.test.ts`, `create-stream-argv.test.ts`, and
`create-stream-ready-gate.test.ts` covering failure branches, redaction,
exit-code preservation, GPU vs non-GPU readiness cleanup,
prebuild/patch/stream wiring, direct argv spawn boundaries,
terminal-agent/default-driver ready-check gating, fail-closed `onPoll`
error handling, and redacted poll/readiness trace behavior.

## Type of Change

- [x] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates
- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: no CLI flags, commands,
configuration, or documented user workflow changed. The user-visible
differences are limited to safer failure-path diagnostics (credential
redaction), more accurate readiness-failure exit status, and fail-closed
create-poll error handling.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: onboarding/sandbox
path. Extraction boundaries were verified against the pre-extraction
source; intentional safety refinements are explicitly covered by tests:
create output is redacted before failure logging, readiness failure
preserves a non-zero create-stream status instead of flattening to `1`,
Docker-GPU during-create polling is isolated from readiness detection
via `onPoll`, escaping poll errors abort create with classified/generic
failure text plus redacted trace emission, and ready-check exceptions
emit redacted trace evidence without falsely forcing Ready. Requesting
maintainer sensitive-path review.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification
- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — command/result: `npx vitest run
--project cli src/lib/actions/sandbox/snapshot.test.ts
src/lib/onboard/created-sandbox-failure.test.ts
src/lib/onboard/sandbox-create-launch.test.ts
src/lib/onboard/sandbox-create-step.test.ts
src/lib/sandbox/create-stream.test.ts
src/lib/sandbox/create-stream-ready-gate.test.ts
src/lib/sandbox/create-stream-argv.test.ts` → 107 passed; `npx tsc
--noEmit --pretty false --project tsconfig.cli.json` → clean; `npm run
test-size:check` → passed; `npx prek run --all-files --stage pre-commit
--skip source-shape-test-budget --skip test-skills-yaml` → passed; `npm
run test-conditionals:scan -- --top 25` → no new changed-file
conditional failures; `git diff --check` → clean.
- [x] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: Required live E2E run
[29021019725](https://github.com/NVIDIA/NemoClaw/actions/runs/29021019725)
passed on exact head `82518f7e6f69c39d9bc04f63a365753aac7b1b6d`:
`cloud-onboard`, `onboard-repair`, `onboard-resume`,
`state-backup-restore`, `upgrade-stale-sandbox`, and `snapshot-commands`
all succeeded. PR CI checks also passed on the same head after rerunning
flaky `policy-channel-list.test.ts` shard timeout (`cli-test-shards
(3)`).
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
Signed-off-by: Dongni Yang <dongniy@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Improved sandbox onboarding using a dedicated creation step that
centralizes prebuild handoff, streaming create execution, GPU patch
wiring, and readiness capture.
* **Bug Fixes**
* More consistent, centralized handling for both create failures and
readiness failures, including redacted output, clearer diagnostics, and
reliable retry guidance.
* Safer cleanup on readiness failures to avoid same-name collisions,
with correct behavior for GPU-enabled flows.
* **Tests**
* Added coverage for create/readiness failure reporting, exit-code
fallback, cleanup/command messaging, and early-detach behavior in
create-stream.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Dongni Yang <dongniy@nvidia.com>
Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Co-authored-by: Julie Yaunches <jyaunches@nvidia.com>
Co-authored-by: Carlos Villela <cvillela@nvidia.com>
Hadar301 pushed a commit to Hadar301/NemoClaw-OpenShift that referenced this pull request Jul 12, 2026
<!-- markdownlint-disable MD041 -->
## Summary
Replace unit-shaped test subprocesses with direct typed seams while
retaining meaningful real-process contracts. The final onboarding pilot
is 20.5% faster, and the OpenClaw config target is 40.5% faster, without
changing production defaults or CLI behavior.

## Related Issue
Part of NVIDIA#6245.

## Changes
- Extract a typed `createSetupInference` test seam into a focused module
while preserving the production dependency wiring and reducing
`src/lib/onboard.ts` by 65 net lines.
- Rewrite unit-shaped subprocess fixtures in the onboarding,
remote-provider-selection, and service-environment suites while
retaining representative process-boundary, fail-closed, and production
Bash coverage.
- Replace 46 service-environment harness child calls with equivalent
Node filesystem operations and cache the three-proxy fixture input.
- Expand the direct dependency-failure suite to 25 cases, including
remote/Bedrock exit boundaries, credential/upsert/apply failures, falsey
status fallbacks, local providers, Ollama proxy recovery, routed
reconciliation/upsert/route-apply failures, Hermes
provider-store/credential/lookup failures, and a focused real
Responses-to-Chat-Completions probe fallback test.
- Complete injected exit, error, and log wiring across remote, Bedrock,
Hermes, Ollama, vLLM, routed-provider, local-route-application, and
Hermes-auth paths while retaining production defaults and real
`setupNim` boundaries for all five native-Docker Windows-provider
rejection scenarios.
- Make provider dependency ownership explicit, document the local route
recovery source boundary and removal condition, and add three focused
local-route recovery tests.
- Require explicit Bedrock and Hermes auth failure boundaries, cover
positive Hermes auth navigation, and use scanner-safe runtime redaction
canaries.
- Retain a production-exported `setupInference`/OpenShell process
boundary proving raw credentials never enter argv and only the provider
update child receives the explicitly scoped credential environment.
- Invoke the exported messaging post-install phase directly in OpenClaw
config tests, eliminating 98 redundant outer applier launches while
preserving all 20 real `openclaw doctor` launches and generator/applier
executable contracts.
- Restore checked-JavaScript validation for
`test/generate-openclaw-config.test.ts` by removing its file-wide
`@ts-nocheck` directive.
- Tighten legacy test-file size ratchets to 6,146 lines for
`test/onboard-selection.test.ts`, 4,057 for `test/onboard.test.ts`, and
1,945 for `test/generate-openclaw-config.test.ts`.
- Record a final-head onboarding median improvement from 44.34s to
35.24s (20.5%) and reduce aggregate `execve` attempts from 7,654 to
6,444 (15.8%).
- Record an OpenClaw config median improvement from 19.83s to 11.80s
(40.5%), with successful `execve` calls reduced from 294 to 196.

## Type of Change

- [x] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates
<!-- Check exactly one tests line and one docs line. Check other lines
when applicable. Add every requested justification or approval
reference. -->
- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: Internal test seams and
test-harness rewrites only; production CLI behavior, output,
configuration, and public interfaces are unchanged.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: Independent focused
reviews confirmed production dependency wiring, secret containment,
environment restoration, and retained process boundaries; automated
advisor findings were addressed or explicitly dispositioned.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification
<!-- Check each applicable item only when supported by the requested
evidence. Run targeted tests once per relevant change set and rerun
after later edits or hook autofixes that can affect the tested behavior.
Do not rerun hook-covered checks. -->
- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — command/result or justification:
current head: onboarding (65/65), dependency failures (25/25),
local-route/rebuild source tests (5/5), Hermes auth (6/6), Bedrock
source (4/4), OpenClaw config (128/128), and provider/source targets
(26/26); earlier focused probe/provider targets (28/28), selection
(69/69), final onboarding benchmark target (134/134 in each run),
messaging-applier support (33/33), and service environment (39/39); CLI
and checked-JavaScript type-checks plus project-membership,
title/size/conditional/source-shape/Biome/diff checks pass.
- [x] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: broad pilot gate: `npm
test` (1,128 files, 12,621 tests); `npm run test:coverage:cli` (1,004
files, 11,063 tests, all ratchets); current-head pre-commit and pre-push
repository/type-check gates passed.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
<!-- DCO sign-off is required in this PR description, and every commit
must appear as Verified in GitHub. Run: git config user.name && git
config user.email -->
Signed-off-by: Carlos Villela <cvillela@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Expanded onboarding support for multiple inference providers,
including improved handling for remote, local, routed, Bedrock Runtime,
Hermes, Ollama, and Windows host detection flows.
* Added clearer fallback behavior when OpenAI-compatible endpoints need
to switch from `/responses` to chat completions.

* **Bug Fixes**
* Improved failure handling and recovery messaging during onboarding,
including better exit behavior in non-interactive flows.
  * Reduced the chance of leaking sensitive values in error output.

* **Tests**
* Added broader coverage for onboarding, provider selection, and
inference-route fallback scenarios.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Hadar301 pushed a commit to Hadar301/NemoClaw-OpenShift that referenced this pull request Jul 12, 2026
<!-- markdownlint-disable MD041 -->
## Summary
Run the integration project as a bounded four-worker phase during the
canonical local `npm test`, while keeping CI, coverage, focused
integration, and direct Vitest runs serialized. Isolate two onboarding
fixtures from host-global dashboard ports so the parallel suite remains
deterministic. This is the final cumulative NVIDIA#6245 step after the named
onboarding conversions, representative process-contract work, and
sequenced loader cleanup already merged; the final clean-build Node 22
suite passes in 3:52.03.

## Related Issue
Closes NVIDIA#6245.

## Changes
- Replace the dashboard-exhaustion fixture's real host listeners with a
fake `lsof` while retaining the real CLI, preflight, diagnostic, and
non-zero exit contract.
- Give the restore-intent fixture an explicit existing dashboard forward
so unrelated host port occupancy cannot divert the behavior under test.
- Resolve integration scheduling from npm lifecycle, CI, coverage, and
worker-cap inputs: local `npm test` uses at most four workers in group
1, while every safety-sensitive route stays serial.
- Add a behavior matrix covering local, CI, coverage, focused, direct,
and explicit worker-throttle modes.
- Complete the cumulative NVIDIA#6245 acceptance path after NVIDIA#6276/NVIDIA#6336/NVIDIA#6383
converted the named onboarding hotspots, NVIDIA#6285/NVIDIA#6417 retained
representative process contracts, and NVIDIA#6286/NVIDIA#6299/NVIDIA#6388/NVIDIA#6415 sequenced
loader cleanup after process removal.
- Record the final host-specific timings, hotspot disposition, and
retained process-contract inventory in `test/README.md` as an advisory
acceptance snapshot rather than a permanent CI budget.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates
<!-- Check exactly one tests line and one docs line. Check other lines
when applicable. Add every requested justification or approval
reference. -->
- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: Test fixtures and local
test-runner scheduling changed; NemoClaw commands, configuration,
runtime behavior, and CI/coverage workflows are unchanged.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: Independent final-diff
review confirmed that the fake `lsof` preserves the real
CLI/preflight/exit contract, the restore-intent assertions remain
intact, and resolved CI/coverage configurations remain serialized.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification
<!-- Check each applicable item only when supported by the requested
evidence. Run targeted tests once per relevant change set and rerun
after later edits or hook autofixes that can affect the tested behavior.
Do not rerun hook-covered checks. -->
- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — command/result or justification: Real
CLI exhaustion contract passed; restore-intent passed with all 11
dashboard ports deliberately occupied; scheduling matrix passed 14/14
through the lifecycle-triggered config; `npm run test:projects:check`
reported 1,327 files disjoint across 8 projects.
- [x] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: Clean-build Node 22 `npm
test -- --reporter=blob` under the normal `umask 022` passed 1,251 files
and 13,879 tests with 39 skipped, 1 todo, and zero failures in 3:52.03,
down 73% from the issue's 14:19.65 baseline despite a larger suite. The
matching diff-scoped routine pre-commit stage passed in 13.95s. NVIDIA#6270
separately removed full coverage from routine pre-commit while
preserving manual and authoritative CI gates.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
<!-- DCO sign-off is required in this PR description, and every commit
must appear as Verified in GitHub. Run: git config user.name && git
config user.email -->
Signed-off-by: Carlos Villela <cvillela@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Integration test runs now use adaptive scheduling to speed up local
execution while keeping CI/focused runs serialized.

* **Bug Fixes**
* Improved reliability of onboarding regression coverage by simulating
dashboard port exhaustion in a hermetic way.
* Updated onboarding-related fixtures to better match the intended
readiness/exit behavior.

* **Tests**
* Added coverage for integration scheduling behavior (local caps,
invalid inputs, and CI/coverage scenarios).

* **Documentation**
* Added test-suite documentation with a local performance snapshot and
key test hotspots.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Hadar301 pushed a commit to Hadar301/NemoClaw-OpenShift that referenced this pull request Jul 12, 2026
…e step into modules (NVIDIA#6444)

<!-- markdownlint-disable MD041 -->
## Summary

Extracts cohesive units of the sandbox create/register orchestration out
of the ~4,900-line `src/lib/onboard.ts` entrypoint into focused modules
under `src/lib/onboard/`, following the injected-deps boundary style
established by `created-sandbox-finalization.ts` (NVIDIA#6332). The primary
goal is maintainability and independent unit-test coverage for the
create path, with intentional safety/behavior refinements discovered
during review: redacted create-output failure echoing, preservation of
non-zero create-stream status when readiness fails, direct argv spawning
for trusted create paths, fail-closed Docker-GPU create-poll side-effect
handling, and redacted trace reporting for poll/readiness errors.

Note on the issue's premise: NVIDIA#6258 cites a `+27 net lines` growth-guard
violation from NVIDIA#6166. That premise is stale — the merged NVIDIA#6166 left
`onboard.ts` net-smaller, and NVIDIA#6276/NVIDIA#6332 shrank it further, so the
guard (`.github/workflows/codebase-growth-guardrails.yaml`, a per-PR
net-neutral diff gate) is not currently red. This PR is therefore
incremental maintainability work; it keeps `onboard.ts`
net-neutral-or-smaller so the gate stays green.

## Related Issue

Refs NVIDIA#6258

<!-- Refs (not Fixes): this PR lands two increments of a larger
extraction; the remaining create/finalize wiring is left for follow-ups,
so the issue should stay open. -->

## Changes

- Add `src/lib/onboard/created-sandbox-failure.ts`:
`reportSandboxCreateFailure` (warns-and-continues on an incomplete
create; otherwise prints diagnostics + recovery hints and exits) and
`reportSandboxReadinessFailure` (prints the readiness failure, defers
cleanup to the Docker-GPU patch or deletes the failed sandbox, then
exits). `onboard.ts` replaces the two inline blocks with module calls —
net −2 lines.
- Add `src/lib/onboard/sandbox-create-step.ts`: `runSandboxCreateStep`
encapsulates the BuildKit prebuild handoff → Docker-GPU create-patch
provisioning → create-stream behind a context + injected-deps boundary.
This move is **line-neutral** on `onboard.ts`; its value is a named,
unit-testable boundary (the `prepare → patch → stream` sequence is now
testable without standing up the entrypoint), not a size reduction.
Build-context and exit-listener cleanup stay with the caller that armed
them.
- Add focused unit tests across `created-sandbox-failure.test.ts`,
`sandbox-create-step.test.ts`, `sandbox-create-launch.test.ts`,
`create-stream.test.ts`, `create-stream-argv.test.ts`, and
`create-stream-ready-gate.test.ts` covering failure branches, redaction,
exit-code preservation, GPU vs non-GPU readiness cleanup,
prebuild/patch/stream wiring, direct argv spawn boundaries,
terminal-agent/default-driver ready-check gating, fail-closed `onPoll`
error handling, and redacted poll/readiness trace behavior.

## Type of Change

- [x] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates
- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: no CLI flags, commands,
configuration, or documented user workflow changed. The user-visible
differences are limited to safer failure-path diagnostics (credential
redaction), more accurate readiness-failure exit status, and fail-closed
create-poll error handling.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: onboarding/sandbox
path. Extraction boundaries were verified against the pre-extraction
source; intentional safety refinements are explicitly covered by tests:
create output is redacted before failure logging, readiness failure
preserves a non-zero create-stream status instead of flattening to `1`,
Docker-GPU during-create polling is isolated from readiness detection
via `onPoll`, escaping poll errors abort create with classified/generic
failure text plus redacted trace emission, and ready-check exceptions
emit redacted trace evidence without falsely forcing Ready. Requesting
maintainer sensitive-path review.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification
- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — command/result: `npx vitest run
--project cli src/lib/actions/sandbox/snapshot.test.ts
src/lib/onboard/created-sandbox-failure.test.ts
src/lib/onboard/sandbox-create-launch.test.ts
src/lib/onboard/sandbox-create-step.test.ts
src/lib/sandbox/create-stream.test.ts
src/lib/sandbox/create-stream-ready-gate.test.ts
src/lib/sandbox/create-stream-argv.test.ts` → 107 passed; `npx tsc
--noEmit --pretty false --project tsconfig.cli.json` → clean; `npm run
test-size:check` → passed; `npx prek run --all-files --stage pre-commit
--skip source-shape-test-budget --skip test-skills-yaml` → passed; `npm
run test-conditionals:scan -- --top 25` → no new changed-file
conditional failures; `git diff --check` → clean.
- [x] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: Required live E2E run
[29021019725](https://github.com/NVIDIA/NemoClaw/actions/runs/29021019725)
passed on exact head `82518f7e6f69c39d9bc04f63a365753aac7b1b6d`:
`cloud-onboard`, `onboard-repair`, `onboard-resume`,
`state-backup-restore`, `upgrade-stale-sandbox`, and `snapshot-commands`
all succeeded. PR CI checks also passed on the same head after rerunning
flaky `policy-channel-list.test.ts` shard timeout (`cli-test-shards
(3)`).
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
Signed-off-by: Dongni Yang <dongniy@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Improved sandbox onboarding using a dedicated creation step that
centralizes prebuild handoff, streaming create execution, GPU patch
wiring, and readiness capture.
* **Bug Fixes**
* More consistent, centralized handling for both create failures and
readiness failures, including redacted output, clearer diagnostics, and
reliable retry guidance.
* Safer cleanup on readiness failures to avoid same-name collisions,
with correct behavior for GPU-enabled flows.
* **Tests**
* Added coverage for create/readiness failure reporting, exit-code
fallback, cleanup/command messaging, and early-detach behavior in
create-stream.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Dongni Yang <dongniy@nvidia.com>
Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Co-authored-by: Julie Yaunches <jyaunches@nvidia.com>
Co-authored-by: Carlos Villela <cvillela@nvidia.com>
@wscurran wscurran added area: ci CI workflows, checks, release automation, or GitHub Actions area: e2e End-to-end tests, nightly failures, or validation infrastructure area: onboarding Onboarding FSM, provider setup, sandbox launch, or first-run flow chore Build, CI, dependency, or tooling maintenance labels Aug 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ci CI workflows, checks, release automation, or GitHub Actions area: e2e End-to-end tests, nightly failures, or validation infrastructure area: onboarding Onboarding FSM, provider setup, sandbox launch, or first-run flow chore Build, CI, dependency, or tooling maintenance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants