Skip to content

perf(test): reduce provider-selection process isolation - #6336

Merged
cv merged 3 commits into
mainfrom
codex/perf-onboard-selection-direct-tests
Jul 6, 2026
Merged

perf(test): reduce provider-selection process isolation#6336
cv merged 3 commits into
mainfrom
codex/perf-onboard-selection-direct-tests

Conversation

@cv

@cv cv commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

Extracts onboarding provider-selection orchestration into a lightweight typed module so unit-shaped selection tests can run directly instead of spawning Node processes. Controlled base/head runs reduced the selection target's median wall time from 20.57s to 18.74s (8.9%) while retaining process-isolated coverage for boundary-sensitive paths.

Related Issue

Part of #6245.

Changes

  • Extract the provider-selection coordinator into setup-nim-flow.ts while preserving provider branches, recovery precedence, fail-closed behavior, and agent/inference API coercion.
  • Add five direct coordinator tests and convert six unit-shaped selection fixtures from child-process execution.
  • Reduce child-process launches in test/onboard-selection.test.ts from 46 to 40; controlled median Vitest duration fell from 20.29s to 18.39s and median test-body time from 19.37s to 17.45s.
  • Retain 40 process-boundary cases for credentials, Ollama, NIM, vLLM, Windows, and fail-closed behavior.
  • Ratchet the selection test file-size budget from 6,146 to 5,835 lines.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: Internal test-performance refactor only; no user-facing command, configuration, prompt, output, or documentation contract changed.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Independent local security review found no issues; fail-closed exits, credential isolation, recovery precedence, and agent/inference API coercion remain intact. CodeRabbit has no open threads, and the automated advisor findings are resolved or evidence-backed in the PR follow-up comments.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Verification

  • PR description includes the DCO sign-off declaration and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: setup-nim-flow passed 5/5 on the final head; the unchanged onboard-selection and Anthropic-compatible OpenAI-agent compatibility suites passed 68/68 and 7/7. npm run typecheck:cli, npm run checks, and npm run test-size:check also passed.
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: Final-head CI passed 40 checks, including all five CLI shards and the merged CLI coverage ratchet: statements 72.34%, branches 65.47%, functions 74.73%, lines 73.05%. One unrelated order-sensitive shard failure passed 7/7 in local isolation and on the single-job CI rerun; the dependent aggregate coverage and checks jobs are green.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Carlos Villela cvillela@nvidia.com

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@coderabbitai

coderabbitai Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

Refactors onboarding setupNim into a dependency-injected setup-nim-flow module, adds focused flow tests, updates onboarding selection tests around shared prompt helpers and featured-model fixtures, and adjusts the test-file size budget.

Changes

setupNim extraction and onboarding test refactor

Layer / File(s) Summary
SetupNimFlowDeps contract and type definitions
src/lib/onboard/setup-nim-flow.ts
Defines exported setup types, the SetupNimFlowDeps interface, and internal selection helpers.
createSetupNim orchestration implementation
src/lib/onboard/setup-nim-flow.ts
Implements host detection, provider-menu construction, selection dispatch, retry handling, and result finalization in createSetupNim.
onboard.ts wiring to setup-nim-flow
src/lib/onboard.ts
Removes the inline setupNim implementation and wires setupNimFlow.createSetupNim(getSetupNimDeps()) into the module exports.
setup-nim-flow test suite
src/lib/onboard/setup-nim-flow.test.ts
Adds tests for remote selection, retry selection, sandbox recovery, rebuild-route handling, and managed vLLM continuation.
Onboard-selection test refactor and shared provider menu builder
test/onboard-selection.test.ts, ci/test-file-size-budget.json
Adds featured-model fixtures, refactors provider-menu helpers and prompt tests, and updates the CI line-budget entry.

Estimated code review effort: 4 (Complex) | ~60 minutes

Possibly related PRs

  • NVIDIA/NemoClaw#1515: Shares onboarding prompt-helper and selection-flow changes used by the refactored setup flow and tests.
  • NVIDIA/NemoClaw#2862: Also changes setupNim-adjacent wiring and provider-selection behavior in src/lib/onboard.ts.

Suggested labels: refactor

Suggested reviewers: jyaunches, ericksoa, prekshivyas

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title matches the main change: reducing provider-selection test process isolation for performance.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/perf-onboard-selection-direct-tests

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in the codex/perf-onboard-s... branch is 96%. Coverage data for the main branch is not yet available.

Show a code coverage summary of the most covered files.
File main codex/perf-onboard-s... 8b01553 +/-
nemoclaw/src/se...cret-scanner.ts 100%
nemoclaw/src/commands/slash.ts 100%
nemoclaw/src/li...bprocess-env.ts 100%
nemoclaw/src/bl...eprint/state.ts 98%
nemoclaw/src/onboard/config.ts 98%
nemoclaw/src/bl...int/snapshot.ts 97%
nemoclaw/src/bl...print/runner.ts 95%
nemoclaw/src/co...ration-state.ts 94%
nemoclaw/src/bl...ate-networks.ts 94%
nemoclaw/src/index.ts 94%

TypeScript / code-coverage/cli

The overall coverage in the codex/perf-onboard-s... branch is 73%. Coverage data for the main branch is not yet available.

Show a code coverage summary of the most covered files.
File main codex/perf-onboard-s... 8b01553 +/-
src/lib/shields...nsition-lock.ts 87%
src/lib/onboard/preflight.ts 83%
src/lib/actions...all/run-plan.ts 81%
src/lib/state/o...oard-session.ts 80%
src/lib/state/sandbox.ts 71%
src/lib/onboard...er-gpu-patch.ts 70%
src/lib/shields/index.ts 67%
src/lib/policy/index.ts 66%
src/lib/actions...licy-channel.ts 63%
src/lib/onboard.ts 29%

Updated July 06, 2026 18:51 UTC
Code Coverage is in Public Preview. Learn more and provide us with your feedback.

@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor (Nemotron Ultra) — Blocked

Merge posture: Do not merge until addressed
Primary next action: Fix PRA-1: No unit test for rejectWindowsHostOllama — sandbox escape vector if WSL with native Docker incorrectly allows Windows-host Ollama; then add or justify PRA-T1.
Open items: 4 required · 4 warnings · 2 suggestions · 8 test follow-ups
Since last review: 0 prior items resolved · 4 still apply · 3 new items found

Action checklist

  • PRA-1 Fix: No unit test for rejectWindowsHostOllama — sandbox escape vector if WSL with native Docker incorrectly allows Windows-host Ollama in src/lib/onboard/setup-nim-flow.test.ts:1
  • PRA-2 Fix: No unit tests for custom endpoint SSRF validation flow (invalid URL, fragment/query stripping, host.docker.internal rejection) in src/lib/onboard/setup-nim-flow.test.ts:1
  • PRA-3 Fix: No unit tests for non-interactive credential validation failure paths (missing NVIDIA_INFERENCE_API_KEY, invalid nvapi- prefix) in src/lib/onboard/setup-nim-flow.test.ts:1
  • PRA-4 Fix: New module setup-nim-flow.ts (559 lines) exceeds 20-line monolith growth threshold — not listed in ci/test-file-size-budget.json legacyMaxLines in src/lib/onboard/setup-nim-flow.ts:1
  • PRA-5 Resolve or justify: SetupNimFlowDeps interface exposes 40+ individual function dependencies — large DI surface increases maintenance burden and incomplete mocking risk in src/lib/onboard/setup-nim-flow.ts:45
  • PRA-6 Resolve or justify: Integration test matrix coverage unverified — need at least 1 spawnSync test per provider type (11 types) in test/onboard-selection.test.ts:633
  • PRA-7 Resolve or justify: No wiring validation test for production getSetupNimDeps() — 40+ dependencies could have missing/wrong signatures in src/lib/onboard.ts:3700
  • PRA-8 Resolve or justify: Net code increase of +206 lines due to DI boilerplate — extraction doesn't reduce overall complexity in src/lib/onboard/setup-nim-flow.ts:1
  • PRA-T1 Add or justify test follow-up: Runtime validation
  • PRA-T2 Add or justify test follow-up: Runtime validation
  • PRA-T3 Add or justify test follow-up: Runtime validation
  • PRA-T4 Add or justify test follow-up: Runtime validation
  • PRA-T5 Add or justify test follow-up: Runtime validation
  • PRA-T6 Add or justify test follow-up: Integration test matrix coverage unverified — need at least 1 spawnSync test per provider type (11 types)
  • PRA-T7 Add or justify test follow-up: Add full-flow unit test with minimal mocks covering provider selection → model selection → validation → result
  • PRA-T8 Add or justify test follow-up: Acceptance clause
  • PRA-9 In-scope improvement: Add full-flow unit test with minimal mocks covering provider selection → model selection → validation → result in src/lib/onboard/setup-nim-flow.test.ts:1
  • PRA-10 In-scope improvement: Verify TypeScript compilation passes without errors for new module in src/lib/onboard/setup-nim-flow.ts:1

Findings index

ID Severity Category Location Required action
PRA-1 Required security src/lib/onboard/setup-nim-flow.test.ts:1 Add unit test in setup-nim-flow.test.ts mocking detectInferenceProviderHostState with isWsl=true, isWindowsHostOllama=true, windowsHostOllamaDockerRequirement.supported=false, and verify selectionLoop continues (retry-selection) when user selects ollama/start-windows-ollama/install-windows-ollama.
PRA-2 Required security src/lib/onboard/setup-nim-flow.test.ts:1 Add unit tests for: (1) custom OpenAI endpoint with invalid URL → handleRemoteProviderSelection returns retry-selection, (2) custom Anthropic endpoint with fragment/query params → normalizeProviderBaseUrl strips them, (3) SSRF attempt via host.docker.internal or host.openshell.internal → rejected by probe layer with retry-selection.
PRA-3 Required security src/lib/onboard/setup-nim-flow.test.ts:1 Add unit tests for non-interactive mode with: (1) missing NVIDIA_INFERENCE_API_KEY env var, (2) invalid nvapi- prefix credential. Verify abortNonInteractive is called with appropriate error message via the deps dependency.
PRA-4 Required architecture src/lib/onboard/setup-nim-flow.ts:1 Either: (a) Split setup-nim-flow.ts into smaller modules (provider-menu-building, selection-loop, recovery-logic, result-assembly) per existing patterns, or (b) Add setup-nim-flow.ts to legacyMaxLines in ci/test-file-size-budget.json with explicit justification for the large module size.
PRA-5 Resolve/justify architecture src/lib/onboard/setup-nim-flow.ts:45 Group DI dependencies into cohesive capability interfaces. Can be done in follow-up PR since current mocks work, but should be tracked as technical debt.
PRA-6 Resolve/justify tests test/onboard-selection.test.ts:633 Audit remaining spawnSync-based tests in test/onboard-selection.test.ts and confirm coverage matrix. Document which provider types have integration coverage.
PRA-7 Resolve/justify correctness src/lib/onboard.ts:3700 Add a unit test that imports getSetupNimDeps, calls it, and asserts all SetupNimFlowDeps keys are present and are functions.
PRA-8 Resolve/justify architecture src/lib/onboard/setup-nim-flow.ts:1 Consider whether the extraction achieves its testability goal without further decomposition. If keeping as single module, add to legacyMaxLines with justification. If decomposing, do it in this PR to avoid technical debt.
PRA-9 Improvement tests src/lib/onboard/setup-nim-flow.test.ts:1 Add an integration-style unit test that exercises the full flow from provider menu through model selection and validation to final result, using minimal mocks that only implement the happy path.
PRA-10 Improvement correctness src/lib/onboard/setup-nim-flow.ts:1 Run tsc --noEmit on the new module to verify no type errors introduced by the extraction.

🚨 Required before merge

Address these before merging unless a maintainer explicitly overrides the advisor with rationale.

PRA-1 Required — No unit test for rejectWindowsHostOllama — sandbox escape vector if WSL with native Docker incorrectly allows Windows-host Ollama

  • Location: src/lib/onboard/setup-nim-flow.test.ts:1
  • Category: security
  • Problem: The rejectWindowsHostOllama security boundary (enforces Docker Desktop WSL integration requirement for Windows-host Ollama) has no unit test in the new test file. If WSL with native Docker incorrectly allows Windows-host Ollama, sandbox containers cannot reach the Ollama endpoint, breaking inference routing. The fail-closed logic exists in local-inference-topology.ts:126 but is only tested indirectly via error reporting in provider-selection-failure.test.ts, not the actual selectionLoop continuation behavior.
  • Impact: Regression in Windows-host Ollama rejection would allow onboarding to proceed with an unreachable inference endpoint, causing silent sandbox inference failures at runtime.
  • Required action: Add unit test in setup-nim-flow.test.ts mocking detectInferenceProviderHostState with isWsl=true, isWindowsHostOllama=true, windowsHostOllamaDockerRequirement.supported=false, and verify selectionLoop continues (retry-selection) when user selects ollama/start-windows-ollama/install-windows-ollama.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: grep -r rejectUnsupportedWindowsHostOllama test/ — no matches found. The function is only tested indirectly via reportProviderSelectionFailure in provider-selection-failure.test.ts.
  • Missing regression test: Unit test for rejectWindowsHostOllama returning true → selectionLoop continues (retry-selection) for unsupported WSL Docker runtime.
  • Done when: The required change is committed and verification passes: grep -r rejectUnsupportedWindowsHostOllama test/ — no matches found. The function is only tested indirectly via reportProviderSelectionFailure in provider-selection-failure.test.ts.
  • Evidence: setup-nim-flow.ts:240-242 defines rejectWindowsHostOllama; setup-nim-flow.ts:370, 402 call it; setup-nim-flow.test.ts has zero tests exercising this branch.

PRA-2 Required — No unit tests for custom endpoint SSRF validation flow (invalid URL, fragment/query stripping, host.docker.internal rejection)

  • Location: src/lib/onboard/setup-nim-flow.test.ts:1
  • Category: security
  • Problem: Custom endpoint validation (resolveCompatibleEndpointInput + validateCustomOpenAiLikeSelection/validateCustomAnthropicSelection) has no unit tests in setup-nim-flow.test.ts. The probe layer in onboard-probes.ts has isHijackedDockerInternalUrl and isSandboxInternalUrl checks, and url-utils.ts normalizeProviderBaseUrl strips fragments/query params, but the flow orchestration's handling of invalid/custom endpoints (URL validation, retry-selection on failure) is untested at unit level.
  • Impact: Regression in custom endpoint SSRF protection (host.docker.internal, host.openshell.internal, invalid URLs) would only be caught by slow subprocess-spawning integration tests, reducing feedback speed and increasing flakiness risk.
  • Required action: Add unit tests for: (1) custom OpenAI endpoint with invalid URL → handleRemoteProviderSelection returns retry-selection, (2) custom Anthropic endpoint with fragment/query params → normalizeProviderBaseUrl strips them, (3) SSRF attempt via host.docker.internal or host.openshell.internal → rejected by probe layer with retry-selection.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: setup-nim-flow.test.ts has no tests mocking handleRemoteProviderSelection for custom endpoint validation failures. Compare with test/onboard-selection.test.ts:2546-2781 which test resolveCompatibleEndpointInput and validation via subprocess.
  • Missing regression test: Unit tests for custom endpoint validation failure → retry-selection, and URL normalization (fragment/query stripping) for both OpenAI-compatible and Anthropic-compatible endpoints.
  • Done when: The required change is committed and verification passes: setup-nim-flow.test.ts has no tests mocking handleRemoteProviderSelection for custom endpoint validation failures. Compare with test/onboard-selection.test.ts:2546-2781 which test resolveCompatibleEndpointInput and validation via subprocess.
  • Evidence: setup-nim-flow.ts imports resolveCompatibleEndpointInput but no test exercises it; onboard-probes.ts:608-612 has SSRF guards; url-utils.ts:27-66 normalizes URLs.

PRA-3 Required — No unit tests for non-interactive credential validation failure paths (missing NVIDIA_INFERENCE_API_KEY, invalid nvapi- prefix)

  • Location: src/lib/onboard/setup-nim-flow.test.ts:1
  • Category: security
  • Problem: Non-interactive mode fails closed on missing/invalid credentials via inference-selection-validation.ts:65-70 (exitNonInteractiveValidationFailure), but the setupNim flow's integration with this behavior is untested at unit level. A regression here could allow onboarding to proceed with invalid credentials.
  • Impact: Onboarding could succeed with missing or malformed API keys in CI/CD pipelines, causing downstream inference failures that are harder to diagnose.
  • Required action: Add unit tests for non-interactive mode with: (1) missing NVIDIA_INFERENCE_API_KEY env var, (2) invalid nvapi- prefix credential. Verify abortNonInteractive is called with appropriate error message via the deps dependency.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: inference-selection-validation.test.ts:10-35 tests non-interactive exit on probe failure, but setup-nim-flow.test.ts has no tests for non-interactive credential validation at the flow level.
  • Missing regression test: Unit tests for non-interactive setupNim with missing/invalid credentials → abortNonInteractive called with appropriate error message.
  • Done when: The required change is committed and verification passes: inference-selection-validation.test.ts:10-35 tests non-interactive exit on probe failure, but setup-nim-flow.test.ts has no tests for non-interactive credential validation at the flow level.
  • Evidence: setup-nim-flow.ts:277 passes abortNonInteractive to reportProviderSelectionFailure; inference-selection-validation.ts exits process on non-interactive validation failure.

PRA-4 Required — New module setup-nim-flow.ts (559 lines) exceeds 20-line monolith growth threshold — not listed in ci/test-file-size-budget.json legacyMaxLines

  • Location: src/lib/onboard/setup-nim-flow.ts:1
  • Category: architecture
  • Problem: The extracted module is 559 lines, exceeding the monolith budget blocker threshold. While onboard.ts reduced by 353 lines (5282→4929), the net codebase increase is +206 lines due to DI boilerplate and type exports. The module is not added to legacyMaxLines in ci/test-file-size-budget.json, so CI will fail or the budget is being violated.
  • Impact: Large modules reduce maintainability and testability. Existing onboard/ pattern uses small focused modules (setup-nim-ollama.ts ~200 lines, provider-selection.ts ~150 lines, provider-selection-prompt.ts ~100 lines). CI budget enforcement may block merge.
  • Required action: Either: (a) Split setup-nim-flow.ts into smaller modules (provider-menu-building, selection-loop, recovery-logic, result-assembly) per existing patterns, or (b) Add setup-nim-flow.ts to legacyMaxLines in ci/test-file-size-budget.json with explicit justification for the large module size.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: Check ci/test-file-size-budget.json — setup-nim-flow.ts not listed; wc -l src/lib/onboard/setup-nim-flow.ts shows 559 lines; monolith budget flags 20+ line growth as blocker.
  • Missing regression test: None — structural concern. Add setup-nim-flow.ts to legacyMaxLines if keeping as-is, or create follow-up PR for decomposition.
  • Done when: The required change is committed and verification passes: Check ci/test-file-size-budget.json — setup-nim-flow.ts not listed; wc -l src/lib/onboard/setup-nim-flow.ts shows 559 lines; monolith budget flags 20+ line growth as blocker.
  • Evidence: Diff shows setup-nim-flow.ts: 559 lines added; ci/test-file-size-budget.json updated for test/onboard-selection.test.ts (6146→5835) but not for setup-nim-flow.ts.
Review findings by urgency: 4 required fixes, 4 items to resolve/justify, 2 in-scope improvements

⚠️ Resolve or justify before merge

Investigate these in the current review; either fix them, explain why they are not applicable, or document the accepted risk.

PRA-5 Resolve/justify — SetupNimFlowDeps interface exposes 40+ individual function dependencies — large DI surface increases maintenance burden and incomplete mocking risk

  • Location: src/lib/onboard/setup-nim-flow.ts:45
  • Category: architecture
  • Problem: The DI interface (lines 45-105) lists 40+ individual dependencies including process.exit wrappers and internal handlers. This increases coupling, test maintenance burden (makeDeps() mocks all 40+ individually), and risk of incomplete mocking. Previous review suggested grouping into capability interfaces: HostStateDetection, ProviderPersistence, UIInteraction, ProviderHandlers.
  • Impact: Each new dependency requires updating makeDeps(), all tests, and production wiring. Incomplete mocks could mask wiring bugs.
  • Recommended action: Group DI dependencies into cohesive capability interfaces. Can be done in follow-up PR since current mocks work, but should be tracked as technical debt.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Review SetupNimFlowDeps interface (lines 45-105) and makeDeps() in setup-nim-flow.test.ts:88-132 — mocks all 40+ deps individually.
  • Missing regression test: No test validates that all DI dependencies are correctly wired in production getSetupNimDeps() — consider adding a wiring validation test.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Review SetupNimFlowDeps interface (lines 45-105) and makeDeps() in setup-nim-flow.test.ts:88-132 — mocks all 40+ deps individually.
  • Evidence: SetupNimFlowDeps lists 40+ individual function dependencies; makeDeps() in test mocks each with unexpected() thrower.

PRA-6 Resolve/justify — Integration test matrix coverage unverified — need at least 1 spawnSync test per provider type (11 types)

  • Location: test/onboard-selection.test.ts:633
  • Category: tests
  • Problem: Many subprocess-spawning integration tests were replaced with direct unit tests of individual functions (promptCloudModel, promptRemoteModel, validateSelectedRemoteModel). This improves speed but reduces end-to-end coverage of full setupNim flow with real CLI interactions. Need to verify at least one integration test per provider type remains: NVIDIA Endpoints, OpenAI, Anthropic, Gemini, Custom OpenAI, Custom Anthropic, Ollama, vLLM, Local NIM, Model Router, Hermes.
  • Impact: Wiring regressions between flow orchestration and handler functions could go undetected if integration test matrix becomes incomplete.
  • Recommended action: Audit remaining spawnSync-based tests in test/onboard-selection.test.ts and confirm coverage matrix. Document which provider types have integration coverage.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: grep -c 'spawnSync' test/onboard-selection.test.ts — count remaining integration tests. Search for provider-specific test names (NVIDIA, OpenAI, Anthropic, Gemini, Ollama, vLLM, NIM, Router, Hermes).
  • Missing regression test: Integration test matrix covering each provider type through the full setupNim flow with real subprocess spawning.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: grep -c 'spawnSync' test/onboard-selection.test.ts — count remaining integration tests. Search for provider-specific test names (NVIDIA, OpenAI, Anthropic, Gemini, Ollama, vLLM, NIM, Router, Hermes).
  • Evidence: PR body claims 46→40 child-process launches; test/onboard-selection.test.ts reduced from 6146 to 5835 lines. No explicit provider-type coverage audit in PR.

PRA-7 Resolve/justify — No wiring validation test for production getSetupNimDeps() — 40+ dependencies could have missing/wrong signatures

  • Location: src/lib/onboard.ts:3700
  • Category: correctness
  • Problem: The getSetupNimDeps() function in onboard.ts (lines 3700-3790) wires 40+ dependencies for production use. No test validates this wiring — if a dependency is missing or has wrong signature, it would only fail at runtime during onboarding.
  • Impact: Runtime onboarding failures due to DI wiring bugs that could be caught at test time.
  • Recommended action: Add a unit test that imports getSetupNimDeps, calls it, and asserts all SetupNimFlowDeps keys are present and are functions.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Read src/lib/onboard.ts:3700-3790 — getSetupNimDeps() returns SetupNimFlowDeps. No test imports and invokes it.
  • Missing regression test: Unit test that imports getSetupNimDeps, calls it, and asserts all SetupNimFlowDeps keys are present and are functions.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Read src/lib/onboard.ts:3700-3790 — getSetupNimDeps() returns SetupNimFlowDeps. No test imports and invokes it.
  • Evidence: getSetupNimDeps() in onboard.ts wires 40+ deps; setup-nim-flow.test.ts uses makeDeps() with mocks but never tests production wiring.

PRA-8 Resolve/justify — Net code increase of +206 lines due to DI boilerplate — extraction doesn't reduce overall complexity

  • Location: src/lib/onboard/setup-nim-flow.ts:1
  • Category: architecture
  • Problem: setup-nim-flow.ts (559 lines) + onboard.ts (4929 lines) = 5488 lines total vs original 5282 lines. Net increase of 206 lines due to DI boilerplate, type exports, and factory pattern. The monolith budget reduction for onboard.ts (-353) is offset by the new module.
  • Impact: Overall codebase complexity increased despite extraction. Testability goal achieved but at cost of more total code to maintain.
  • Recommended action: Consider whether the extraction achieves its testability goal without further decomposition. If keeping as single module, add to legacyMaxLines with justification. If decomposing, do it in this PR to avoid technical debt.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: wc -l src/lib/onboard.ts src/lib/onboard/setup-nim-flow.ts — total lines across both files.
  • Missing regression test: None — architectural concern.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: wc -l src/lib/onboard.ts src/lib/onboard/setup-nim-flow.ts — total lines across both files.
  • Evidence: Diff stat: onboard.ts -353, setup-nim-flow.ts +559 = net +206 lines.

💡 In-scope improvements

These are lower-risk, not throwaway. Prefer fixing them in this PR when they are local to changed code; defer only with rationale or a linked follow-up.

PRA-9 Improvement — Add full-flow unit test with minimal mocks covering provider selection → model selection → validation → result

  • Location: src/lib/onboard/setup-nim-flow.test.ts:1
  • Category: tests
  • Problem: Test file uses makeDeps() with unexpected() throwers for 40+ deps, but only 5 tests override 3-5 deps each. No test exercises the full selectionLoop with a minimal but complete set of mocks, leaving wiring issues undetected.
  • Impact: Integration gaps between flow branches and handler functions could exist despite passing unit tests.
  • Suggested action: Add an integration-style unit test that exercises the full flow from provider menu through model selection and validation to final result, using minimal mocks that only implement the happy path.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Review setup-nim-flow.test.ts:88-132 makeDeps() — all 40+ deps mocked with unexpected() defaults. Only 5 tests override specific deps.
  • Missing regression test: Full-flow unit test with minimal mocks covering provider selection → model selection → validation → result.
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: Current tests cover: Ollama detection+prompt, retry-selection loop, non-interactive recovery, rebuild route, vLLM install — all partial flows.

PRA-10 Improvement — Verify TypeScript compilation passes without errors for new module

  • Location: src/lib/onboard/setup-nim-flow.ts:1
  • Category: correctness
  • Problem: The @ts-nocheck directive is NOT present in setup-nim-flow.ts (good), but the module imports from ./machine/handlers/provider-inference which may have type issues. Verify all imports resolve cleanly under strict TypeScript.
  • Impact: Type errors in the new module would indicate incomplete extraction or missing type exports.
  • Suggested action: Run tsc --noEmit on the new module to verify no type errors introduced by the extraction.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: npx tsc --noEmit src/lib/onboard/setup-nim-flow.ts — check for type errors.
  • Missing regression test: TypeScript compilation should pass without errors.
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: setup-nim-flow.ts has no @ts-nocheck; imports from ./machine/handlers/provider-inference, ./provider-menu, ./provider-selection, etc.
Test follow-ups to resolve or justify

If these cover changed behavior, prefer adding them in this PR; otherwise state why existing coverage is enough or link the follow-up.

  • PRA-T1 Runtime validation — rejectWindowsHostOllama returns true for unsupported WSL Docker → selectionLoop continues. Runtime/sandbox/infrastructure paths (onboard.ts, setup-nim-flow.ts) need behavioral runtime validation. The 68 integration tests in test/onboard-selection.test.ts provide runtime validation but are subprocess-spawning (slower, flakier). The 5 new unit tests are fast but don't cover security branches. Adding unit tests for security-critical flow branches would improve feedback speed.
  • PRA-T2 Runtime validation — Custom OpenAI endpoint with invalid URL → handleRemoteProviderSelection returns retry-selection. Runtime/sandbox/infrastructure paths (onboard.ts, setup-nim-flow.ts) need behavioral runtime validation. The 68 integration tests in test/onboard-selection.test.ts provide runtime validation but are subprocess-spawning (slower, flakier). The 5 new unit tests are fast but don't cover security branches. Adding unit tests for security-critical flow branches would improve feedback speed.
  • PRA-T3 Runtime validation — Custom Anthropic endpoint with fragment/query params → normalizeProviderBaseUrl strips them. Runtime/sandbox/infrastructure paths (onboard.ts, setup-nim-flow.ts) need behavioral runtime validation. The 68 integration tests in test/onboard-selection.test.ts provide runtime validation but are subprocess-spawning (slower, flakier). The 5 new unit tests are fast but don't cover security branches. Adding unit tests for security-critical flow branches would improve feedback speed.
  • PRA-T4 Runtime validation — Non-interactive missing NVIDIA_INFERENCE_API_KEY → abortNonInteractive called. Runtime/sandbox/infrastructure paths (onboard.ts, setup-nim-flow.ts) need behavioral runtime validation. The 68 integration tests in test/onboard-selection.test.ts provide runtime validation but are subprocess-spawning (slower, flakier). The 5 new unit tests are fast but don't cover security branches. Adding unit tests for security-critical flow branches would improve feedback speed.
  • PRA-T5 Runtime validation — Non-interactive invalid nvapi- prefix → abortNonInteractive called. Runtime/sandbox/infrastructure paths (onboard.ts, setup-nim-flow.ts) need behavioral runtime validation. The 68 integration tests in test/onboard-selection.test.ts provide runtime validation but are subprocess-spawning (slower, flakier). The 5 new unit tests are fast but don't cover security branches. Adding unit tests for security-critical flow branches would improve feedback speed.
  • PRA-T6 Integration test matrix coverage unverified — need at least 1 spawnSync test per provider type (11 types) — Audit remaining spawnSync-based tests in test/onboard-selection.test.ts and confirm coverage matrix. Document which provider types have integration coverage.
  • PRA-T7 Add full-flow unit test with minimal mocks covering provider selection → model selection → validation → result — Add an integration-style unit test that exercises the full flow from provider menu through model selection and validation to final result, using minimal mocks that only implement the happy path.
  • PRA-T8 Acceptance clause — Preserve provider branches, recovery precedence, fail-closed behavior, agent/inference API coercion — add test evidence or identify existing coverage. Code moved but security-critical branches (rejectWindowsHostOllama, custom endpoint SSRF, non-interactive credential failure) untested at unit level
Since last review details

Current findings, using the urgency labels above:

PRA-1 Required — No unit test for rejectWindowsHostOllama — sandbox escape vector if WSL with native Docker incorrectly allows Windows-host Ollama

  • Location: src/lib/onboard/setup-nim-flow.test.ts:1
  • Category: security
  • Problem: The rejectWindowsHostOllama security boundary (enforces Docker Desktop WSL integration requirement for Windows-host Ollama) has no unit test in the new test file. If WSL with native Docker incorrectly allows Windows-host Ollama, sandbox containers cannot reach the Ollama endpoint, breaking inference routing. The fail-closed logic exists in local-inference-topology.ts:126 but is only tested indirectly via error reporting in provider-selection-failure.test.ts, not the actual selectionLoop continuation behavior.
  • Impact: Regression in Windows-host Ollama rejection would allow onboarding to proceed with an unreachable inference endpoint, causing silent sandbox inference failures at runtime.
  • Required action: Add unit test in setup-nim-flow.test.ts mocking detectInferenceProviderHostState with isWsl=true, isWindowsHostOllama=true, windowsHostOllamaDockerRequirement.supported=false, and verify selectionLoop continues (retry-selection) when user selects ollama/start-windows-ollama/install-windows-ollama.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: grep -r rejectUnsupportedWindowsHostOllama test/ — no matches found. The function is only tested indirectly via reportProviderSelectionFailure in provider-selection-failure.test.ts.
  • Missing regression test: Unit test for rejectWindowsHostOllama returning true → selectionLoop continues (retry-selection) for unsupported WSL Docker runtime.
  • Done when: The required change is committed and verification passes: grep -r rejectUnsupportedWindowsHostOllama test/ — no matches found. The function is only tested indirectly via reportProviderSelectionFailure in provider-selection-failure.test.ts.
  • Evidence: setup-nim-flow.ts:240-242 defines rejectWindowsHostOllama; setup-nim-flow.ts:370, 402 call it; setup-nim-flow.test.ts has zero tests exercising this branch.

PRA-2 Required — No unit tests for custom endpoint SSRF validation flow (invalid URL, fragment/query stripping, host.docker.internal rejection)

  • Location: src/lib/onboard/setup-nim-flow.test.ts:1
  • Category: security
  • Problem: Custom endpoint validation (resolveCompatibleEndpointInput + validateCustomOpenAiLikeSelection/validateCustomAnthropicSelection) has no unit tests in setup-nim-flow.test.ts. The probe layer in onboard-probes.ts has isHijackedDockerInternalUrl and isSandboxInternalUrl checks, and url-utils.ts normalizeProviderBaseUrl strips fragments/query params, but the flow orchestration's handling of invalid/custom endpoints (URL validation, retry-selection on failure) is untested at unit level.
  • Impact: Regression in custom endpoint SSRF protection (host.docker.internal, host.openshell.internal, invalid URLs) would only be caught by slow subprocess-spawning integration tests, reducing feedback speed and increasing flakiness risk.
  • Required action: Add unit tests for: (1) custom OpenAI endpoint with invalid URL → handleRemoteProviderSelection returns retry-selection, (2) custom Anthropic endpoint with fragment/query params → normalizeProviderBaseUrl strips them, (3) SSRF attempt via host.docker.internal or host.openshell.internal → rejected by probe layer with retry-selection.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: setup-nim-flow.test.ts has no tests mocking handleRemoteProviderSelection for custom endpoint validation failures. Compare with test/onboard-selection.test.ts:2546-2781 which test resolveCompatibleEndpointInput and validation via subprocess.
  • Missing regression test: Unit tests for custom endpoint validation failure → retry-selection, and URL normalization (fragment/query stripping) for both OpenAI-compatible and Anthropic-compatible endpoints.
  • Done when: The required change is committed and verification passes: setup-nim-flow.test.ts has no tests mocking handleRemoteProviderSelection for custom endpoint validation failures. Compare with test/onboard-selection.test.ts:2546-2781 which test resolveCompatibleEndpointInput and validation via subprocess.
  • Evidence: setup-nim-flow.ts imports resolveCompatibleEndpointInput but no test exercises it; onboard-probes.ts:608-612 has SSRF guards; url-utils.ts:27-66 normalizes URLs.

PRA-3 Required — No unit tests for non-interactive credential validation failure paths (missing NVIDIA_INFERENCE_API_KEY, invalid nvapi- prefix)

  • Location: src/lib/onboard/setup-nim-flow.test.ts:1
  • Category: security
  • Problem: Non-interactive mode fails closed on missing/invalid credentials via inference-selection-validation.ts:65-70 (exitNonInteractiveValidationFailure), but the setupNim flow's integration with this behavior is untested at unit level. A regression here could allow onboarding to proceed with invalid credentials.
  • Impact: Onboarding could succeed with missing or malformed API keys in CI/CD pipelines, causing downstream inference failures that are harder to diagnose.
  • Required action: Add unit tests for non-interactive mode with: (1) missing NVIDIA_INFERENCE_API_KEY env var, (2) invalid nvapi- prefix credential. Verify abortNonInteractive is called with appropriate error message via the deps dependency.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: inference-selection-validation.test.ts:10-35 tests non-interactive exit on probe failure, but setup-nim-flow.test.ts has no tests for non-interactive credential validation at the flow level.
  • Missing regression test: Unit tests for non-interactive setupNim with missing/invalid credentials → abortNonInteractive called with appropriate error message.
  • Done when: The required change is committed and verification passes: inference-selection-validation.test.ts:10-35 tests non-interactive exit on probe failure, but setup-nim-flow.test.ts has no tests for non-interactive credential validation at the flow level.
  • Evidence: setup-nim-flow.ts:277 passes abortNonInteractive to reportProviderSelectionFailure; inference-selection-validation.ts exits process on non-interactive validation failure.

PRA-4 Required — New module setup-nim-flow.ts (559 lines) exceeds 20-line monolith growth threshold — not listed in ci/test-file-size-budget.json legacyMaxLines

  • Location: src/lib/onboard/setup-nim-flow.ts:1
  • Category: architecture
  • Problem: The extracted module is 559 lines, exceeding the monolith budget blocker threshold. While onboard.ts reduced by 353 lines (5282→4929), the net codebase increase is +206 lines due to DI boilerplate and type exports. The module is not added to legacyMaxLines in ci/test-file-size-budget.json, so CI will fail or the budget is being violated.
  • Impact: Large modules reduce maintainability and testability. Existing onboard/ pattern uses small focused modules (setup-nim-ollama.ts ~200 lines, provider-selection.ts ~150 lines, provider-selection-prompt.ts ~100 lines). CI budget enforcement may block merge.
  • Required action: Either: (a) Split setup-nim-flow.ts into smaller modules (provider-menu-building, selection-loop, recovery-logic, result-assembly) per existing patterns, or (b) Add setup-nim-flow.ts to legacyMaxLines in ci/test-file-size-budget.json with explicit justification for the large module size.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: Check ci/test-file-size-budget.json — setup-nim-flow.ts not listed; wc -l src/lib/onboard/setup-nim-flow.ts shows 559 lines; monolith budget flags 20+ line growth as blocker.
  • Missing regression test: None — structural concern. Add setup-nim-flow.ts to legacyMaxLines if keeping as-is, or create follow-up PR for decomposition.
  • Done when: The required change is committed and verification passes: Check ci/test-file-size-budget.json — setup-nim-flow.ts not listed; wc -l src/lib/onboard/setup-nim-flow.ts shows 559 lines; monolith budget flags 20+ line growth as blocker.
  • Evidence: Diff shows setup-nim-flow.ts: 559 lines added; ci/test-file-size-budget.json updated for test/onboard-selection.test.ts (6146→5835) but not for setup-nim-flow.ts.

PRA-5 Resolve/justify — SetupNimFlowDeps interface exposes 40+ individual function dependencies — large DI surface increases maintenance burden and incomplete mocking risk

  • Location: src/lib/onboard/setup-nim-flow.ts:45
  • Category: architecture
  • Problem: The DI interface (lines 45-105) lists 40+ individual dependencies including process.exit wrappers and internal handlers. This increases coupling, test maintenance burden (makeDeps() mocks all 40+ individually), and risk of incomplete mocking. Previous review suggested grouping into capability interfaces: HostStateDetection, ProviderPersistence, UIInteraction, ProviderHandlers.
  • Impact: Each new dependency requires updating makeDeps(), all tests, and production wiring. Incomplete mocks could mask wiring bugs.
  • Recommended action: Group DI dependencies into cohesive capability interfaces. Can be done in follow-up PR since current mocks work, but should be tracked as technical debt.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Review SetupNimFlowDeps interface (lines 45-105) and makeDeps() in setup-nim-flow.test.ts:88-132 — mocks all 40+ deps individually.
  • Missing regression test: No test validates that all DI dependencies are correctly wired in production getSetupNimDeps() — consider adding a wiring validation test.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Review SetupNimFlowDeps interface (lines 45-105) and makeDeps() in setup-nim-flow.test.ts:88-132 — mocks all 40+ deps individually.
  • Evidence: SetupNimFlowDeps lists 40+ individual function dependencies; makeDeps() in test mocks each with unexpected() thrower.

PRA-6 Resolve/justify — Integration test matrix coverage unverified — need at least 1 spawnSync test per provider type (11 types)

  • Location: test/onboard-selection.test.ts:633
  • Category: tests
  • Problem: Many subprocess-spawning integration tests were replaced with direct unit tests of individual functions (promptCloudModel, promptRemoteModel, validateSelectedRemoteModel). This improves speed but reduces end-to-end coverage of full setupNim flow with real CLI interactions. Need to verify at least one integration test per provider type remains: NVIDIA Endpoints, OpenAI, Anthropic, Gemini, Custom OpenAI, Custom Anthropic, Ollama, vLLM, Local NIM, Model Router, Hermes.
  • Impact: Wiring regressions between flow orchestration and handler functions could go undetected if integration test matrix becomes incomplete.
  • Recommended action: Audit remaining spawnSync-based tests in test/onboard-selection.test.ts and confirm coverage matrix. Document which provider types have integration coverage.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: grep -c 'spawnSync' test/onboard-selection.test.ts — count remaining integration tests. Search for provider-specific test names (NVIDIA, OpenAI, Anthropic, Gemini, Ollama, vLLM, NIM, Router, Hermes).
  • Missing regression test: Integration test matrix covering each provider type through the full setupNim flow with real subprocess spawning.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: grep -c 'spawnSync' test/onboard-selection.test.ts — count remaining integration tests. Search for provider-specific test names (NVIDIA, OpenAI, Anthropic, Gemini, Ollama, vLLM, NIM, Router, Hermes).
  • Evidence: PR body claims 46→40 child-process launches; test/onboard-selection.test.ts reduced from 6146 to 5835 lines. No explicit provider-type coverage audit in PR.

PRA-7 Resolve/justify — No wiring validation test for production getSetupNimDeps() — 40+ dependencies could have missing/wrong signatures

  • Location: src/lib/onboard.ts:3700
  • Category: correctness
  • Problem: The getSetupNimDeps() function in onboard.ts (lines 3700-3790) wires 40+ dependencies for production use. No test validates this wiring — if a dependency is missing or has wrong signature, it would only fail at runtime during onboarding.
  • Impact: Runtime onboarding failures due to DI wiring bugs that could be caught at test time.
  • Recommended action: Add a unit test that imports getSetupNimDeps, calls it, and asserts all SetupNimFlowDeps keys are present and are functions.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Read src/lib/onboard.ts:3700-3790 — getSetupNimDeps() returns SetupNimFlowDeps. No test imports and invokes it.
  • Missing regression test: Unit test that imports getSetupNimDeps, calls it, and asserts all SetupNimFlowDeps keys are present and are functions.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Read src/lib/onboard.ts:3700-3790 — getSetupNimDeps() returns SetupNimFlowDeps. No test imports and invokes it.
  • Evidence: getSetupNimDeps() in onboard.ts wires 40+ deps; setup-nim-flow.test.ts uses makeDeps() with mocks but never tests production wiring.

PRA-8 Resolve/justify — Net code increase of +206 lines due to DI boilerplate — extraction doesn't reduce overall complexity

  • Location: src/lib/onboard/setup-nim-flow.ts:1
  • Category: architecture
  • Problem: setup-nim-flow.ts (559 lines) + onboard.ts (4929 lines) = 5488 lines total vs original 5282 lines. Net increase of 206 lines due to DI boilerplate, type exports, and factory pattern. The monolith budget reduction for onboard.ts (-353) is offset by the new module.
  • Impact: Overall codebase complexity increased despite extraction. Testability goal achieved but at cost of more total code to maintain.
  • Recommended action: Consider whether the extraction achieves its testability goal without further decomposition. If keeping as single module, add to legacyMaxLines with justification. If decomposing, do it in this PR to avoid technical debt.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: wc -l src/lib/onboard.ts src/lib/onboard/setup-nim-flow.ts — total lines across both files.
  • Missing regression test: None — architectural concern.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: wc -l src/lib/onboard.ts src/lib/onboard/setup-nim-flow.ts — total lines across both files.
  • Evidence: Diff stat: onboard.ts -353, setup-nim-flow.ts +559 = net +206 lines.

PRA-9 Improvement — Add full-flow unit test with minimal mocks covering provider selection → model selection → validation → result

  • Location: src/lib/onboard/setup-nim-flow.test.ts:1
  • Category: tests
  • Problem: Test file uses makeDeps() with unexpected() throwers for 40+ deps, but only 5 tests override 3-5 deps each. No test exercises the full selectionLoop with a minimal but complete set of mocks, leaving wiring issues undetected.
  • Impact: Integration gaps between flow branches and handler functions could exist despite passing unit tests.
  • Suggested action: Add an integration-style unit test that exercises the full flow from provider menu through model selection and validation to final result, using minimal mocks that only implement the happy path.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Review setup-nim-flow.test.ts:88-132 makeDeps() — all 40+ deps mocked with unexpected() defaults. Only 5 tests override specific deps.
  • Missing regression test: Full-flow unit test with minimal mocks covering provider selection → model selection → validation → result.
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: Current tests cover: Ollama detection+prompt, retry-selection loop, non-interactive recovery, rebuild route, vLLM install — all partial flows.

PRA-10 Improvement — Verify TypeScript compilation passes without errors for new module

  • Location: src/lib/onboard/setup-nim-flow.ts:1
  • Category: correctness
  • Problem: The @ts-nocheck directive is NOT present in setup-nim-flow.ts (good), but the module imports from ./machine/handlers/provider-inference which may have type issues. Verify all imports resolve cleanly under strict TypeScript.
  • Impact: Type errors in the new module would indicate incomplete extraction or missing type exports.
  • Suggested action: Run tsc --noEmit on the new module to verify no type errors introduced by the extraction.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: npx tsc --noEmit src/lib/onboard/setup-nim-flow.ts — check for type errors.
  • Missing regression test: TypeScript compilation should pass without errors.
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: setup-nim-flow.ts has no @ts-nocheck; imports from ./machine/handlers/provider-inference, ./provider-menu, ./provider-selection, etc.

Workflow run details

This is an automated, non-binding review; it still expects maintainers and agents to respond to each required or warning item. Treat suggestions as current-PR improvements when they touch changed code; defer only with maintainer rationale or a linked follow-up. A human maintainer must make the final merge decision.

@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

E2E Advisor Recommendation

Required E2E: cloud-onboard, onboard-resume, onboard-repair, inference-routing
Optional E2E: cloud-inference, kimi-inference-compat, openclaw-inference-switch

Dispatch hint: cloud-onboard,onboard-resume,onboard-repair,inference-routing

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

  • cloud-onboard (high): Required because the PR changes the full hosted onboarding provider-selection path and the provider/model/API contracts used to configure inference during onboarding.
  • onboard-resume (medium): Required because setupNim recovery behavior for recorded provider/model state and non-interactive resumed onboarding was refactored; unit tests should not be the only guard for these resume paths.
  • onboard-repair (high): Required because the same setupNim provider recovery and rebuild route handoff contracts are used during repair/recovery flows and can affect whether repaired onboarding preserves or reconstructs inference configuration correctly.
  • inference-routing (medium): Required because the refactored flow returns provider, endpoint, credential, and preferredInferenceApi values that drive inference route setup and can break routing even if onboarding completes.

Optional E2E

  • cloud-inference (medium): Useful adjacent confidence for hosted NVIDIA inference behavior after provider/model selection refactoring, but cloud-onboard plus inference-routing are the merge-blocking coverage for this PR.
  • kimi-inference-compat (medium): Optional because the changed tests touch NVIDIA featured model selection/Kimi compatibility, but the runtime change is mainly setupNim orchestration rather than Kimi-specific inference compatibility.
  • openclaw-inference-switch (high): Optional end-to-end confidence for switching OpenClaw providers after onboarding because provider/API return contracts changed, but it is broader and more expensive than the required routing/onboarding jobs.

New E2E recommendations

  • None.

Dispatch hint

  • Workflow: .github/workflows/e2e.yaml
  • jobs input: cloud-onboard,onboard-resume,onboard-repair,inference-routing

@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

E2E Target Recommendation

Required E2E targets: ubuntu-repo-cloud-openclaw, ubuntu-repo-cloud-langchain-deepagents-code, onboard-resume, onboard-repair
Optional E2E targets: None

Dispatch required E2E targets:

  • gh workflow run e2e.yaml --ref <pr-head-ref> --field targets=ubuntu-repo-cloud-openclaw
  • gh workflow run e2e.yaml --ref <pr-head-ref> --field targets=ubuntu-repo-cloud-langchain-deepagents-code
  • gh workflow run e2e.yaml --ref <pr-head-ref> --field jobs=onboard-resume
  • gh workflow run e2e.yaml --ref <pr-head-ref> --field jobs=onboard-repair

Workflow run

Full E2E target advisor summary

E2E Target Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E targets

  • ubuntu-repo-cloud-openclaw: The PR extracts and rewires setupNim provider-selection flow used by standard cloud OpenClaw onboarding; this live-supported typed target exercises the primary hosted NVIDIA onboarding path.
    • Dispatch: gh workflow run e2e.yaml --ref <pr-head-ref> --field targets=ubuntu-repo-cloud-openclaw
  • ubuntu-repo-cloud-langchain-deepagents-code: The changed setupNim flow preserves agent inference API coercion and rebuild route/credential-reuse contracts; this live-supported DCode target exercises agent-specific onboarding plus the dcode rebuild invalid-credential lifecycle.
    • Dispatch: gh workflow run e2e.yaml --ref <pr-head-ref> --field targets=ubuntu-repo-cloud-langchain-deepagents-code
  • onboard-resume: The setupNim flow changes non-interactive recovered provider/model handling from persisted sandbox state, so the onboarding resume rule requires the onboard-resume live job.
    • Dispatch: gh workflow run e2e.yaml --ref <pr-head-ref> --field jobs=onboard-resume
  • onboard-repair: The same recovered provider/rebuild-route changes can affect repair/backstop execution from persisted sessions; repair is required for these state-machine resume-adjacent paths.
    • Dispatch: gh workflow run e2e.yaml --ref <pr-head-ref> --field jobs=onboard-repair

Optional E2E targets

  • None.

Relevant changed files

  • src/lib/onboard.ts
  • src/lib/onboard/setup-nim-flow.ts

@cv cv self-assigned this Jul 6, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (2)
test/onboard-selection.test.ts (2)

189-225: 🎯 Functional Correctness | 🔵 Trivial

Good DRY consolidation of provider-menu defaults.

Centralizing shared menu defaults into buildProviderMenu and having buildWindowsProviderMenu delegate to it removes duplicated inline construction. Please confirm windowsHostStartLabel's function-typed default (Line 204) matches the real buildInferenceProviderMenu parameter contract used by production code, since a signature drift here would let tests pass while diverging from src/lib/onboard.ts's actual usage.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/onboard-selection.test.ts` around lines 189 - 225, Verify that the
function-typed default passed as windowsHostStartLabel in buildProviderMenu
matches the real buildInferenceProviderMenu contract used by production code,
and update the test helper if the expected signature differs. Check
buildProviderMenu, buildWindowsProviderMenu, and buildInferenceProviderMenu
together so the default callback shape stays aligned with src/lib/onboard.ts and
does not mask a signature drift in tests.

653-702: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Mock-call assertions layered on top of observable-state checks.

expect(probeOpenAiLikeEndpoint).toHaveBeenCalledWith(...) (Lines 697-702, 849-853) and expect(validateNvidiaEndpointModelFn).toHaveBeenCalledWith(...) (Lines 757-760) assert on mock invocation arguments in addition to the observable state/validated outcomes. Since the state assertions (state.preferredInferenceApi, state.model, validated.result) already confirm the behavior, the call-argument assertions add implementation lock-in without additional confidence about the public boundary.

As per path instructions for **/*.test.{ts,js,mts,mjs,cts,cjs}: "Prefer observable outcomes through the public boundary over source-text, private-shape, or mock-call assertions."

Also applies to: 709-761, 802-853

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/onboard-selection.test.ts` around lines 653 - 702, The test cases are
asserting on mock invocation arguments in addition to already verified
observable outcomes, which adds unnecessary implementation coupling. Remove the
`toHaveBeenCalledWith(...)` expectations from `probeOpenAiLikeEndpoint` and
`validateNvidiaEndpointModelFn` in the affected tests, and keep the assertions
on public-facing results like `state`, `validated.result`, and emitted output so
the checks stay focused on observable behavior through the test boundary.

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/onboard/setup-nim-flow.ts`:
- Around line 248-535: The single-option provider flow bypasses the selection
dispatcher, so setup can ավարտ without calling the provider-specific handler.
Update setup-nim-flow.ts so the same dispatch path used inside selectionLoop
also runs when options.length <= 1, ensuring functions like
handleRemoteProviderSelection, handleNimLocalSelection,
handleRunningOllamaSelection, and handleVllmSelection still execute and populate
state. Add a regression test with only one available provider to verify the
selected provider is dispatched and the handler runs.

In `@test/onboard-selection.test.ts`:
- Line 633: The new test titles in the onboarding selection spec are missing the
required local issue-ref suffix. Update each affected test case in
test/onboard-selection.test.ts (the NVIDIA Endpoints, Kimi K2.6, manual model
entry, reprompt, and Gemini model tests) so their behavior-oriented titles end
with the appropriate final "(`#1234`)"-style issue reference, matching the test
naming convention used throughout the suite.

---

Nitpick comments:
In `@test/onboard-selection.test.ts`:
- Around line 189-225: Verify that the function-typed default passed as
windowsHostStartLabel in buildProviderMenu matches the real
buildInferenceProviderMenu contract used by production code, and update the test
helper if the expected signature differs. Check buildProviderMenu,
buildWindowsProviderMenu, and buildInferenceProviderMenu together so the default
callback shape stays aligned with src/lib/onboard.ts and does not mask a
signature drift in tests.
- Around line 653-702: The test cases are asserting on mock invocation arguments
in addition to already verified observable outcomes, which adds unnecessary
implementation coupling. Remove the `toHaveBeenCalledWith(...)` expectations
from `probeOpenAiLikeEndpoint` and `validateNvidiaEndpointModelFn` in the
affected tests, and keep the assertions on public-facing results like `state`,
`validated.result`, and emitted output so the checks stay focused on observable
behavior through the test boundary.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 3c770fba-3b08-4376-a3f7-da83a84f50e4

📥 Commits

Reviewing files that changed from the base of the PR and between 748a788 and 6111da4.

📒 Files selected for processing (5)
  • ci/test-file-size-budget.json
  • src/lib/onboard.ts
  • src/lib/onboard/setup-nim-flow.test.ts
  • src/lib/onboard/setup-nim-flow.ts
  • test/onboard-selection.test.ts

Comment thread src/lib/onboard/setup-nim-flow.ts
Comment thread test/onboard-selection.test.ts Outdated
@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings

Merge posture: No blocking advisor findings
Primary next action: Add or justify PRA-T1 and any related test follow-ups.
Open items: 0 required · 0 warnings · 1 suggestion · 3 test follow-ups
Since last review: 2 prior items resolved · 1 still applies · 0 new items found

Action checklist

  • PRA-T1 Add or justify test follow-up: Runtime validation
  • PRA-T2 Add or justify test follow-up: Runtime validation
  • PRA-T3 Add or justify test follow-up: Acceptance clause
  • PRA-1 In-scope improvement: Shrink repeated selection-state construction in the extracted coordinator in src/lib/onboard/setup-nim-flow.ts:303

Findings index

ID Severity Category Location Required action
PRA-1 Improvement architecture src/lib/onboard/setup-nim-flow.ts:303 If it remains readable, add a narrow local helper pair for making and applying the shared selection state, keeping branch-specific fields explicit. Do not hide the trust-boundary decisions: provider selection, Windows-host Ollama rejection, fail-closed exits, credential-reuse flags, and compatible-endpoint reasoning cleanup should remain visible.
Review findings by urgency: 0 required fixes, 0 items to resolve/justify, 1 in-scope improvement

⚠️ Resolve or justify before merge

Investigate these in the current review; either fix them, explain why they are not applicable, or document the accepted risk.

  • None.

💡 In-scope improvements

These are lower-risk, not throwaway. Prefer fixing them in this PR when they are local to changed code; defer only with rationale or a linked follow-up.

PRA-1 Improvement — Shrink repeated selection-state construction in the extracted coordinator

  • Location: src/lib/onboard/setup-nim-flow.ts:303
  • Category: architecture
  • Problem: The new coordinator repeats nearly identical SetupNimSelectionState object construction and state write-back in each provider branch. The extraction reduces src/lib/onboard.ts, but the new module is 559 lines and the duplicated branch plumbing makes future field drift easier.
  • Impact: A later edit can update one provider branch while forgetting shared fields such as preferredInferenceApi, nimContainer, allowToolsIncompatible, hermesToolGateways, or compatibleEndpointReasoning. In onboarding/provider host glue, that kind of drift can affect provider validation confidence or credential-reuse behavior.
  • Suggested action: If it remains readable, add a narrow local helper pair for making and applying the shared selection state, keeping branch-specific fields explicit. Do not hide the trust-boundary decisions: provider selection, Windows-host Ollama rejection, fail-closed exits, credential-reuse flags, and compatible-endpoint reasoning cleanup should remain visible.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Read src/lib/onboard/setup-nim-flow.ts around the remote-provider, nim-local, ollama, Windows-host Ollama, install-ollama, vLLM, and routed branches; each branch constructs and then destructures a similar SetupNimSelectionState.
  • Missing regression test: No new behavior test is required for a pure helper extraction. If the helper changes field propagation, add a focused test that a handler-mutated preferredInferenceApi, nimContainer, allowToolsIncompatible, and hermesToolGateways survive to the returned ProviderSelectionResult.
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: The repeated state blocks begin at the remote-provider branch around line 303 and recur in the local NIM, Ollama, Windows-host Ollama, install-ollama, vLLM, and routed branches. Previous advisor PRA-3 still applies.
Simplification opportunities: 1 possible cut, net -40 lines possible

These are safe simplification checks only. Do not remove validation, security controls, data-loss prevention, or required tests.

  • PRA-1 shrink (src/lib/onboard/setup-nim-flow.ts:303): Repeated SetupNimSelectionState object literals and duplicated destructuring/write-back blocks in each provider branch.
    • Replacement: A small local makeSelectionState/applySelectionState helper, with an option for branch-only fields such as nvidiaFeaturedModels and with security-sensitive branch decisions left explicit.
    • Net: -40 lines
    • Safety boundary: Do not simplify away provider allowlisting, Windows-host Ollama rejection, fail-closed exits, credential-reuse return flags, compatible-endpoint reasoning cleanup, or the existing coordinator contract tests.
Test follow-ups to resolve or justify

If these cover changed behavior, prefer adding them in this PR; otherwise state why existing coverage is enough or link the follow-up.

  • PRA-T1 Runtime validation — Identify or add a targeted exported setupNim runtime/import-level test for the default NVIDIA Endpoints flow when Ollama is detected, proving getSetupNimDeps connects provider prompt, model selection, remote validation, maybePromptForInferenceInputCapability, and returned preferredInferenceApi.. The changed source is onboarding/provider host glue. The new direct unit tests cover the coordinator contracts well, and many process-boundary tests remain, but a small amount of runtime/import-level validation is still useful because the PR intentionally replaces some subprocess coverage with direct helper tests.
  • PRA-T2 Runtime validation — Identify or add a targeted runtime/import-level test that exercises exported setupNim with a matching rebuildRegistryInferenceRoute through the onboard.ts wiring, proving the handoff is preserved outside the direct createSetupNim unit test.. The changed source is onboarding/provider host glue. The new direct unit tests cover the coordinator contracts well, and many process-boundary tests remain, but a small amount of runtime/import-level validation is still useful because the PR intentionally replaces some subprocess coverage with direct helper tests.
  • PRA-T3 Acceptance clause — Linked issue clauses from perf(test): reduce test I/O to restore 2–5 minute full-suite runs #6245 — add test evidence or identify existing coverage. The deterministic validation context did not include linked issue bodies or comments, and linkedIssues was empty, so no literal issue clauses were available to verify.
Since last review details

Current findings, using the urgency labels above:

PRA-1 Improvement — Shrink repeated selection-state construction in the extracted coordinator

  • Location: src/lib/onboard/setup-nim-flow.ts:303
  • Category: architecture
  • Problem: The new coordinator repeats nearly identical SetupNimSelectionState object construction and state write-back in each provider branch. The extraction reduces src/lib/onboard.ts, but the new module is 559 lines and the duplicated branch plumbing makes future field drift easier.
  • Impact: A later edit can update one provider branch while forgetting shared fields such as preferredInferenceApi, nimContainer, allowToolsIncompatible, hermesToolGateways, or compatibleEndpointReasoning. In onboarding/provider host glue, that kind of drift can affect provider validation confidence or credential-reuse behavior.
  • Suggested action: If it remains readable, add a narrow local helper pair for making and applying the shared selection state, keeping branch-specific fields explicit. Do not hide the trust-boundary decisions: provider selection, Windows-host Ollama rejection, fail-closed exits, credential-reuse flags, and compatible-endpoint reasoning cleanup should remain visible.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Read src/lib/onboard/setup-nim-flow.ts around the remote-provider, nim-local, ollama, Windows-host Ollama, install-ollama, vLLM, and routed branches; each branch constructs and then destructures a similar SetupNimSelectionState.
  • Missing regression test: No new behavior test is required for a pure helper extraction. If the helper changes field propagation, add a focused test that a handler-mutated preferredInferenceApi, nimContainer, allowToolsIncompatible, and hermesToolGateways survive to the returned ProviderSelectionResult.
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: The repeated state blocks begin at the remote-provider branch around line 303 and recur in the local NIM, Ollama, Windows-host Ollama, install-ollama, vLLM, and routed branches. Previous advisor PRA-3 still applies.

Workflow run details

This is an automated, non-binding review; it still expects maintainers and agents to respond to each required or warning item. Treat suggestions as current-PR improvements when they touch changed code; defer only with maintainer rationale or a linked follow-up. A human maintainer must make the final merge decision.

cv added 2 commits July 6, 2026 11:30
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@cv

cv commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator Author

Automated review follow-up (8b015535a)

  • Fixed the issue-reference convention in 7439b97d2: all eight new/directly converted test titles now end in (#6245).
  • Added two focused 14 ms coordinator tests in 8b015535a. Together they cover matching rebuild-route precedence and handler handoff, both gateway-credential reuse return flags, non-compatible reasoning cleanup, managed-vLLM install fallthrough, and agent inference-API coercion. The updated direct suite passes 5/5.
  • The single-option CodeRabbit finding is not a regression: origin/main has the same options.length > 1 gate, and buildInferenceProviderMenu currently initializes six base provider options. Changing that path would be a product behavior change outside this performance refactor.
  • The 559-line coordinator and explicit DI/state propagation are a deliberate one-to-one extraction from the monolith. Further grouping would add architectural churn to a behavior-preserving performance batch; the focused provider handlers remain separate, and the retained 40 process-boundary cases continue to cover credential, fail-closed, Ollama, NIM, vLLM, and Windows behavior.

@cv

cv commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator Author

Final-head Nemotron triage

I reviewed the needs_rework items against the actual module boundary and retained coverage:

  • PRA-1 (Windows-host Ollama): already covered at the full flow boundary. The intentionally retained process cases rejects Windows-host Ollama providers on native Docker WSL before launching Ollama and rejects reachable Windows-host Ollama on native Docker WSL through generic and fallback paths exercise the exact fail-closed selection path (test/onboard-selection.test.ts:5501,5524). The 14 ms direct retry test separately locks down coordinator re-entry. Duplicating the security journey in a mock-only test would not add boundary confidence.
  • PRA-2/PRA-3 (SSRF and credentials): wrong ownership for this coordinator. setup-nim-flow.ts does not import endpoint normalization, probes, or credential validation; those remain behind handleRemoteProviderSelection. Their focused/direct tests and retained process cases cover custom OpenAI/Anthropic validation (:2544,2612,2703,2759,2898), URL normalization (src/lib/core/url-utils.test.ts), invalid/missing NVIDIA credentials (:3481,3577), and provider credential retries (:3884,3962,4040,4118,4212). Mocking the injected handler cannot test those controls.
  • PRA-4 (file budget): not applicable. ci/test-file-size-budget.json scans only *.test/*.spec files. The growth workflow explicitly says “This check allows src/lib/onboard/** to grow” while requiring the top-level src/lib/onboard.ts to shrink; that entrypoint shrank by 400 lines and the final-head growth guard is green. Adding a production source file to a test-file budget would violate the budget schema's intent.
  • PRA-5/PRA-8 (DI size/net lines): accepted design tradeoff. The flat typed dependency seam is what removes module-cache/process isolation while keeping the moved branch logic one-to-one and reviewable. It delivered the measured 8.9% target reduction; capability regrouping or state-helper abstraction would be architectural churn without another process removal.
  • PRA-6 (matrix): retained coverage remains broad. There are 40 spawnSync(process.execPath) boundaries in onboard-selection, covering NVIDIA, OpenAI, Anthropic, Gemini, both custom endpoint families, Ollama/Windows Ollama, Local NIM, Model Router, and Hermes; vLLM has its dedicated six-case process suite in test/onboard-selection-vllm.test.ts.
  • PRA-7/PRA-10 (production wiring/types): covered by static typing and gates. getSetupNimDeps(): SetupNimFlowDeps statically checks every production dependency, and normal pre-push plus final-head build-typecheck are green. A runtime assertion over private object keys would duplicate TypeScript while coupling tests to implementation shape.
  • PRA-9: the detected-Ollama direct case already runs menu construction → interactive selection → remote handler → final result/input-capability boundary. The two new contract tests add recovery/security-field finalization and managed-vLLM fallthrough.

No production change or additional push is warranted for these items. The original independent security review also found no behavior/security delta, and CodeRabbit’s final-head review has no open threads.

@cv
cv merged commit dc2ae9f into main Jul 6, 2026
61 of 64 checks passed
@cv
cv deleted the codex/perf-onboard-selection-direct-tests branch July 6, 2026 19:03
cv added a commit that referenced this pull request Jul 7, 2026
<!-- markdownlint-disable MD041 -->
## Summary
Run the integration project as a bounded four-worker phase during the
canonical local `npm test`, while keeping CI, coverage, focused
integration, and direct Vitest runs serialized. Isolate two onboarding
fixtures from host-global dashboard ports so the parallel suite remains
deterministic. This is the final cumulative #6245 step after the named
onboarding conversions, representative process-contract work, and
sequenced loader cleanup already merged; the final clean-build Node 22
suite passes in 3:52.03.

## Related Issue
Closes #6245.

## Changes
- Replace the dashboard-exhaustion fixture's real host listeners with a
fake `lsof` while retaining the real CLI, preflight, diagnostic, and
non-zero exit contract.
- Give the restore-intent fixture an explicit existing dashboard forward
so unrelated host port occupancy cannot divert the behavior under test.
- Resolve integration scheduling from npm lifecycle, CI, coverage, and
worker-cap inputs: local `npm test` uses at most four workers in group
1, while every safety-sensitive route stays serial.
- Add a behavior matrix covering local, CI, coverage, focused, direct,
and explicit worker-throttle modes.
- Complete the cumulative #6245 acceptance path after #6276/#6336/#6383
converted the named onboarding hotspots, #6285/#6417 retained
representative process contracts, and #6286/#6299/#6388/#6415 sequenced
loader cleanup after process removal.
- Record the final host-specific timings, hotspot disposition, and
retained process-contract inventory in `test/README.md` as an advisory
acceptance snapshot rather than a permanent CI budget.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates
<!-- Check exactly one tests line and one docs line. Check other lines
when applicable. Add every requested justification or approval
reference. -->
- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: Test fixtures and local
test-runner scheduling changed; NemoClaw commands, configuration,
runtime behavior, and CI/coverage workflows are unchanged.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: Independent final-diff
review confirmed that the fake `lsof` preserves the real
CLI/preflight/exit contract, the restore-intent assertions remain
intact, and resolved CI/coverage configurations remain serialized.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification
<!-- Check each applicable item only when supported by the requested
evidence. Run targeted tests once per relevant change set and rerun
after later edits or hook autofixes that can affect the tested behavior.
Do not rerun hook-covered checks. -->
- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — command/result or justification: Real
CLI exhaustion contract passed; restore-intent passed with all 11
dashboard ports deliberately occupied; scheduling matrix passed 14/14
through the lifecycle-triggered config; `npm run test:projects:check`
reported 1,327 files disjoint across 8 projects.
- [x] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: Clean-build Node 22 `npm
test -- --reporter=blob` under the normal `umask 022` passed 1,251 files
and 13,879 tests with 39 skipped, 1 todo, and zero failures in 3:52.03,
down 73% from the issue's 14:19.65 baseline despite a larger suite. The
matching diff-scoped routine pre-commit stage passed in 13.95s. #6270
separately removed full coverage from routine pre-commit while
preserving manual and authoritative CI gates.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
<!-- DCO sign-off is required in this PR description, and every commit
must appear as Verified in GitHub. Run: git config user.name && git
config user.email -->
Signed-off-by: Carlos Villela <cvillela@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Integration test runs now use adaptive scheduling to speed up local
execution while keeping CI/focused runs serialized.

* **Bug Fixes**
* Improved reliability of onboarding regression coverage by simulating
dashboard port exhaustion in a hermetic way.
* Updated onboarding-related fixtures to better match the intended
readiness/exit behavior.

* **Tests**
* Added coverage for integration scheduling behavior (local caps,
invalid inputs, and CI/coverage scenarios).

* **Documentation**
* Added test-suite documentation with a local performance snapshot and
key test hotspots.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Hadar301 pushed a commit to Hadar301/NemoClaw-OpenShift that referenced this pull request Jul 12, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Extracts onboarding provider-selection orchestration into a lightweight
typed module so unit-shaped selection tests can run directly instead of
spawning Node processes. Controlled base/head runs reduced the selection
target's median wall time from 20.57s to 18.74s (8.9%) while retaining
process-isolated coverage for boundary-sensitive paths.

## Related Issue

Part of NVIDIA#6245.

## Changes

- Extract the provider-selection coordinator into `setup-nim-flow.ts`
while preserving provider branches, recovery precedence, fail-closed
behavior, and agent/inference API coercion.
- Add five direct coordinator tests and convert six unit-shaped
selection fixtures from child-process execution.
- Reduce child-process launches in `test/onboard-selection.test.ts` from
46 to 40; controlled median Vitest duration fell from 20.29s to 18.39s
and median test-body time from 19.37s to 17.45s.
- Retain 40 process-boundary cases for credentials, Ollama, NIM, vLLM,
Windows, and fail-closed behavior.
- Ratchet the selection test file-size budget from 6,146 to 5,835 lines.

## Type of Change

- [x] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates
<!-- Check exactly one tests line and one docs line. Check other lines
when applicable. Add every requested justification or approval
reference. -->
- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: Internal test-performance
refactor only; no user-facing command, configuration, prompt, output, or
documentation contract changed.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: Independent local
security review found no issues; fail-closed exits, credential
isolation, recovery precedence, and agent/inference API coercion remain
intact. CodeRabbit has no open threads, and the automated advisor
findings are resolved or evidence-backed in the PR follow-up comments.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification
<!-- Check each applicable item only when supported by the requested
evidence. Run targeted tests once per relevant change set and rerun
after later edits or hook autofixes that can affect the tested behavior.
Do not rerun hook-covered checks. -->
- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — command/result or justification:
`setup-nim-flow` passed 5/5 on the final head; the unchanged
`onboard-selection` and Anthropic-compatible OpenAI-agent compatibility
suites passed 68/68 and 7/7. `npm run typecheck:cli`, `npm run checks`,
and `npm run test-size:check` also passed.
- [x] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: Final-head CI passed 40
checks, including all five CLI shards and the merged CLI coverage
ratchet: statements 72.34%, branches 65.47%, functions 74.73%, lines
73.05%. One unrelated order-sensitive shard failure passed 7/7 in local
isolation and on the single-job CI rerun; the dependent aggregate
coverage and checks jobs are green.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
<!-- DCO sign-off is required in this PR description, and every commit
must appear as Verified in GitHub. Run: git config user.name && git
config user.email -->
Signed-off-by: Carlos Villela <cvillela@nvidia.com>

---------

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Hadar301 pushed a commit to Hadar301/NemoClaw-OpenShift that referenced this pull request Jul 12, 2026
<!-- markdownlint-disable MD041 -->
## Summary
Run the integration project as a bounded four-worker phase during the
canonical local `npm test`, while keeping CI, coverage, focused
integration, and direct Vitest runs serialized. Isolate two onboarding
fixtures from host-global dashboard ports so the parallel suite remains
deterministic. This is the final cumulative NVIDIA#6245 step after the named
onboarding conversions, representative process-contract work, and
sequenced loader cleanup already merged; the final clean-build Node 22
suite passes in 3:52.03.

## Related Issue
Closes NVIDIA#6245.

## Changes
- Replace the dashboard-exhaustion fixture's real host listeners with a
fake `lsof` while retaining the real CLI, preflight, diagnostic, and
non-zero exit contract.
- Give the restore-intent fixture an explicit existing dashboard forward
so unrelated host port occupancy cannot divert the behavior under test.
- Resolve integration scheduling from npm lifecycle, CI, coverage, and
worker-cap inputs: local `npm test` uses at most four workers in group
1, while every safety-sensitive route stays serial.
- Add a behavior matrix covering local, CI, coverage, focused, direct,
and explicit worker-throttle modes.
- Complete the cumulative NVIDIA#6245 acceptance path after NVIDIA#6276/NVIDIA#6336/NVIDIA#6383
converted the named onboarding hotspots, NVIDIA#6285/NVIDIA#6417 retained
representative process contracts, and NVIDIA#6286/NVIDIA#6299/NVIDIA#6388/NVIDIA#6415 sequenced
loader cleanup after process removal.
- Record the final host-specific timings, hotspot disposition, and
retained process-contract inventory in `test/README.md` as an advisory
acceptance snapshot rather than a permanent CI budget.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates
<!-- Check exactly one tests line and one docs line. Check other lines
when applicable. Add every requested justification or approval
reference. -->
- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: Test fixtures and local
test-runner scheduling changed; NemoClaw commands, configuration,
runtime behavior, and CI/coverage workflows are unchanged.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: Independent final-diff
review confirmed that the fake `lsof` preserves the real
CLI/preflight/exit contract, the restore-intent assertions remain
intact, and resolved CI/coverage configurations remain serialized.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification
<!-- Check each applicable item only when supported by the requested
evidence. Run targeted tests once per relevant change set and rerun
after later edits or hook autofixes that can affect the tested behavior.
Do not rerun hook-covered checks. -->
- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — command/result or justification: Real
CLI exhaustion contract passed; restore-intent passed with all 11
dashboard ports deliberately occupied; scheduling matrix passed 14/14
through the lifecycle-triggered config; `npm run test:projects:check`
reported 1,327 files disjoint across 8 projects.
- [x] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: Clean-build Node 22 `npm
test -- --reporter=blob` under the normal `umask 022` passed 1,251 files
and 13,879 tests with 39 skipped, 1 todo, and zero failures in 3:52.03,
down 73% from the issue's 14:19.65 baseline despite a larger suite. The
matching diff-scoped routine pre-commit stage passed in 13.95s. NVIDIA#6270
separately removed full coverage from routine pre-commit while
preserving manual and authoritative CI gates.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
<!-- DCO sign-off is required in this PR description, and every commit
must appear as Verified in GitHub. Run: git config user.name && git
config user.email -->
Signed-off-by: Carlos Villela <cvillela@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Integration test runs now use adaptive scheduling to speed up local
execution while keeping CI/focused runs serialized.

* **Bug Fixes**
* Improved reliability of onboarding regression coverage by simulating
dashboard port exhaustion in a hermetic way.
* Updated onboarding-related fixtures to better match the intended
readiness/exit behavior.

* **Tests**
* Added coverage for integration scheduling behavior (local caps,
invalid inputs, and CI/coverage scenarios).

* **Documentation**
* Added test-suite documentation with a local performance snapshot and
key test hotspots.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
@wscurran wscurran added area: ci CI workflows, checks, release automation, or GitHub Actions chore Build, CI, dependency, or tooling maintenance labels Aug 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ci CI workflows, checks, release automation, or GitHub Actions chore Build, CI, dependency, or tooling maintenance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants