perf(test): run model router onboarding tests in-process - #6411
Merged
Conversation
Contributor
📝 WalkthroughWalkthroughModel Router command provisioning is extracted into a new module, router startup now uses injected dependencies, and the onboarding tests were rewritten to run in-process with new fixture and process helpers. ChangesModel Router provisioner extraction and DI refactor
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Contributor
Code Coverage OverviewLanguages: TypeScript TypeScript / code-coverage/pluginThe overall coverage in the Show a code coverage summary of the most covered files.
TypeScript / code-coverage/cliThe overall coverage in the Show a code coverage summary of the most covered files.
Updated |
Contributor
E2E Advisor RecommendationRequired E2E: Dispatch hint: Full advisor summaryE2E Recommendation AdvisorBase: Required E2E
Optional E2E
New E2E recommendations
Dispatch hint
|
Contributor
E2E Target RecommendationRequired E2E targets: Dispatch required E2E targets:
Full E2E target advisor summaryE2E Target AdvisorBase: Required E2E targets
Optional E2E targets
Relevant changed files
|
Contributor
PR Review Advisor (Nemotron Ultra) — BlockedMerge posture: Do not merge until addressed Action checklist
Findings index
🚨 Required before mergeAddress these before merging unless a maintainer explicitly overrides the advisor with rationale.
|
Contributor
PR Review Advisor — Changes requestedMerge posture: Do not merge yet Action checklist
Findings index
Review findings by urgency: 0 required fixes, 2 items to resolve/justify, 0 in-scope improvements
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
test/support/model-router-process-test-helpers.ts (1)
22-38: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueDuplicated polling loop (100 attempts × 10ms) across both functions.
stopTestProcessandreadRouterLaunchLogrepeat the same attempt-count/interval pattern. Extracting a small sharedpollUntil(predicate, { attempts, intervalMs })helper would remove the duplication and make the timeout tunable in one place.♻️ Suggested extraction
+async function pollUntil( + predicate: () => boolean, + { attempts = 100, intervalMs = 10 }: { attempts?: number; intervalMs?: number } = {}, +): Promise<boolean> { + for (let i = 0; i < attempts; i++) { + if (predicate()) return true; + await new Promise((resolve) => setTimeout(resolve, intervalMs)); + } + return false; +}Also applies to: 40-57
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/support/model-router-process-test-helpers.ts` around lines 22 - 38, The polling loop pattern is duplicated between stopTestProcess and readRouterLaunchLog, so extract the shared retry logic into a small helper such as pollUntil that accepts a predicate plus attempts and intervalMs options. Update stopTestProcess to use this helper for the isProcessAlive check, and reuse the same helper in readRouterLaunchLog so the timeout behavior is defined in one place and easy to tune.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@test/support/model-router-process-test-helpers.ts`:
- Around line 22-38: The polling loop pattern is duplicated between
stopTestProcess and readRouterLaunchLog, so extract the shared retry logic into
a small helper such as pollUntil that accepts a predicate plus attempts and
intervalMs options. Update stopTestProcess to use this helper for the
isProcessAlive check, and reuse the same helper in readRouterLaunchLog so the
timeout behavior is defined in one place and easy to tune.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 7d23b578-432a-4fba-8608-8a42e53b2360
📒 Files selected for processing (2)
test/onboard-model-router.test.tstest/support/model-router-process-test-helpers.ts
Contributor
E2E Target Results — ✅ All requested jobs passedRun: 28883204039
|
Contributor
E2E Target Results — ✅ All requested jobs passedRun: 28883447759
|
Contributor
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
test/support/model-router-process-test-helpers.ts (1)
156-173: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick winGuard against partial writes when reading the launch log.
The log file is appended to by a live subprocess while this function polls it. If a read lands mid-write, the trailing line can be incomplete, and
JSON.parsewill throw aSyntaxErrorthat propagates out of the retry loop instead of triggering a retry — causing intermittent test failures rather than a clean timeout.🔧 Proposed fix: tolerate partial trailing lines and retry
export async function readRouterLaunchLog( logPath: string, expectedEntries: number, ): Promise<RouterLaunchLog[]> { for (let attempt = 0; attempt < 100; attempt++) { if (fs.existsSync(logPath)) { - const entries = fs - .readFileSync(logPath, "utf8") - .trim() - .split("\n") - .filter(Boolean) - .map((line) => JSON.parse(line) as RouterLaunchLog); - if (entries.length >= expectedEntries) return entries; + try { + const entries = fs + .readFileSync(logPath, "utf8") + .trim() + .split("\n") + .filter(Boolean) + .map((line) => JSON.parse(line) as RouterLaunchLog); + if (entries.length >= expectedEntries) return entries; + } catch { + // Partial write in progress; retry on the next attempt. + } } await new Promise((resolve) => setTimeout(resolve, 10)); } throw new Error(`Timed out waiting for ${expectedEntries} Model Router launch log entries`); }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/support/model-router-process-test-helpers.ts` around lines 156 - 173, The readRouterLaunchLog helper currently parses every line immediately, so a partial trailing write from the live subprocess can throw and escape the retry loop. Update readRouterLaunchLog to tolerate incomplete JSON while polling by catching parse failures (or skipping the last non-empty line when it looks incomplete) and retrying until enough valid RouterLaunchLog entries are available, instead of letting a SyntaxError abort the function.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@test/support/model-router-process-test-helpers.ts`:
- Around line 156-173: The readRouterLaunchLog helper currently parses every
line immediately, so a partial trailing write from the live subprocess can throw
and escape the retry loop. Update readRouterLaunchLog to tolerate incomplete
JSON while polling by catching parse failures (or skipping the last non-empty
line when it looks incomplete) and retrying until enough valid RouterLaunchLog
entries are available, instead of letting a SyntaxError abort the function.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: deb45d15-c6cc-4aa2-91b7-7962436a13f1
📒 Files selected for processing (3)
src/lib/onboard/model-router.tstest/onboard-model-router.test.tstest/support/model-router-process-test-helpers.ts
Collaborator
Author
|
Final-head verification for
Automated-review disposition:
Independent final-diff review and final-head CodeRabbit review found no remaining actionable correctness, security, credential-handling, typing, performance, or test-isolation issues. |
wscurran
added
area: ci
Hadar301
pushed a commit
to Hadar301/NemoClaw-OpenShift
that referenced
this pull request
Jul 12, 2026
<!-- markdownlint-disable MD041 --> ## Summary Replace five unit-shaped Model Router onboarding child-process tests with direct typed boundaries while retaining a lightweight real process contract. Extract managed-command provisioning so route, installation, reuse, fingerprint, and launch behavior can be tested without repeatedly loading the full onboarding graph or waiting on real health intervals. ## Related Issue Refs NVIDIA#6245 ## Changes - Extract Model Router command discovery, managed-venv installation, and source fingerprinting into a dependency-light provisioner while preserving the existing `model-router.ts` exports. - Give `startModelRouter` an immutable per-call dependency seam with unchanged production defaults. - Replace five source-loader child drivers with eight direct tests covering routed-provider wiring, managed command installation/reuse/refresh, fallback fingerprints, production adapter composition, and real proxy launch arguments. - Retain one lightweight real fake-router process contract for config generation, detached launch, credential environment filtering, PID liveness, and cleanup. - Reduce the matched CI file result from 14,005.160ms on current main to 404.958ms on the final head (97.11% lower, 34.58× faster); collection plus tests fell from 14,054.999ms to 961.167ms (93.16% lower). ## Type of Change - [x] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates <!-- Check exactly one tests line and one docs line. Check other lines when applicable. Add every requested justification or approval reference. --> - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [ ] Docs updated for user-facing behavior changes - [x] Docs not applicable — justification: internal module extraction and test dependency seams only; Model Router provider selection, credentials, ports, managed virtual environment, startup behavior, and failure semantics are unchanged. - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [x] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: independent final-diff review found no actionable correctness, security, credential-handling, typing, or test-isolation findings; the retained process contract verifies filtered credential delivery and cleanup. - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Verification <!-- Check each applicable item only when supported by the requested evidence. Run targeted tests once per relevant change set and rerun after later edits or hook autofixes that can affect the tested behavior. Do not rerun hook-covered checks. --> - [x] PR description includes the DCO sign-off declaration and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — The target file passes 8/8 locally and on final-head CI; related Model Router/routed-provider suites and 2/2 Model Router E2E-support tests also passed. - [x] Required live E2E targets passed — `model-router-provider-routed-inference` and `cloud-onboard` both passed in run 28883447759; the final follow-up changes only test-log parsing. - [ ] Applicable broad gate passed — `npm test` for broad runtime/test-harness changes; `npm run check` for repo-wide validation/coverage changes — command/result: - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) - [ ] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) --- <!-- DCO sign-off is required in this PR description, and every commit must appear as Verified in GitHub. Run: git config user.name && git config user.email --> Signed-off-by: Carlos Villela <cvillela@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added managed Model Router command provisioning that reuses an existing managed venv install when it matches the current router source, and provisions it when needed. * Refactored router startup to support dependency-injected health checks and proxy spawning. * **Bug Fixes** * Improved “currentness” detection using source fingerprints, with a safe fallback when git-based fingerprinting isn’t available. * More consistent readiness, termination, and retry behavior during startup. * **Tests** * Reworked Model Router onboarding tests to run in-process with a command harness, plus new process/log test helpers and fixtures. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replace five unit-shaped Model Router onboarding child-process tests with direct typed boundaries while retaining a lightweight real process contract. Extract managed-command provisioning so route, installation, reuse, fingerprint, and launch behavior can be tested without repeatedly loading the full onboarding graph or waiting on real health intervals.
Related Issue
Refs #6245
Changes
model-router.tsexports.startModelRouteran immutable per-call dependency seam with unchanged production defaults.Type of Change
Quality Gates
Verification
Verifiedin GitHubpre-commit,commit-msg, andpre-pushhooks passed, ornpm run check:diffpassed when hooks were skipped or unavailablemodel-router-provider-routed-inferenceandcloud-onboardboth passed in run 28883447759; the final follow-up changes only test-log parsing.npm testfor broad runtime/test-harness changes;npm run checkfor repo-wide validation/coverage changes — command/result:npm run docsbuilds without warnings (doc changes only)Signed-off-by: Carlos Villela cvillela@nvidia.com
Summary by CodeRabbit