test(e2e): centralize fake OpenAI-compatible server - #5373
Merged
Conversation
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
✅ Files skipped from review due to trivial changes (1)
📝 WalkthroughWalkthroughConsolidates a reusable fake OpenAI-compatible server (Node + TS fixture + shell helpers), migrates tests and E2E scripts to use it, updates PR-review advisor overlap/comment logic, adjusts nightly E2E provisioning to explicit Node setup, and applies formatting-only refactors across tools. ChangesE2E fake-server consolidation and advisor updates
Sequence Diagram(s)sequenceDiagram
participant TestCase as Test Case
participant Fixture as startFakeOpenAiCompatibleServer (fixture)
participant APIServer as fake-openai-compatible-api.mts (Node server)
participant RequestLog as requests.jsonl
TestCase->>Fixture: start({port, auth, chatContent, ...})
Fixture->>APIServer: spawn child process with FAKE_OPENAI_* env
Fixture->>APIServer: poll GET /v1/models until ready
APIServer-->>Fixture: READY + bound port
Fixture-->>TestCase: return baseUrl, requests(), close()
TestCase->>APIServer: POST /v1/chat/completions or /v1/responses
APIServer->>RequestLog: append JSONL request metadata
TestCase->>Fixture: fake.requests()
Fixture->>RequestLog: parse JSONL
Fixture-->>TestCase: array of typed requests
TestCase->>Fixture: fake.close()
Fixture->>APIServer: SIGTERM
APIServer-->>Fixture: exit
Fixture->>Fixture: cleanup temp files
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
tools/pr-review-advisor/analyze.mts (1)
773-808: ⚡ Quick winBound overlap file-fetch concurrency to avoid GitHub API burst throttling.
Line 773 launches up to 80 overlap checks concurrently, and each may call another paginated files API. This burst can hit secondary rate limits and silently under-report overlaps when the inner
catchfalls back tosameFiles = [].Suggested refactor
- const overlaps = await Promise.all( - openPulls - .filter((pull) => getPath<number>(pull, ["number"]) !== currentPrNumber) - .slice(0, 80) - .map(async (pull): Promise<OpenPrOverlap | null> => { + const overlaps: Array<OpenPrOverlap | null> = []; + for (const pull of openPulls + .filter((item) => getPath<number>(item, ["number"]) !== currentPrNumber) + .slice(0, 80)) { + overlaps.push( + await (async (): Promise<OpenPrOverlap | null> => { // existing body unchanged - }), - ); + })(), + ); + }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tools/pr-review-advisor/analyze.mts` around lines 773 - 808, The overlap checks currently run up to 80 concurrent tasks (the Promise.all over openPulls.slice(0, 80)) and each task may call githubRestPaginated, which can trigger GitHub secondary rate limits and cause sameFiles to silently fall back to empty; change the implementation to bound concurrency (e.g., use a concurrency limiter like p-limit or an explicit worker pool) when mapping openPulls to the overlap check so only a small number (e.g., 4–8) of githubRestPaginated calls run at once; update the code around overlaps/openPulls mapping (the async mapper that computes number, title, body, labels, linkedIssues, duplicateLinkedIssues, sameFiles) to schedule each mapper through the limiter and await Promise.all on the limited tasks instead of launching all 80 at once.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/e2e-scenario/fixtures/fake-openai-compatible.ts`:
- Around line 57-87: The readiness probe currently hardcodes "127.0.0.1" inside
canReachModels which ignores the configured host from
startFakeOpenAiCompatibleServer; modify canReachModels to accept a host
parameter and use that host instead of "127.0.0.1", update waitForReady to pass
the configured host through when calling canReachModels (thread the host from
wherever waitForReady is invoked, e.g., startFakeOpenAiCompatibleServer), and
ensure the http.get options use the provided host so readiness checks honor
non-loopback/IPv6 bindings.
In `@test/e2e/test-issue-2478-crash-loop-recovery.sh`:
- Around line 142-151: The preflight in
test/e2e/test-issue-2478-crash-loop-recovery.sh is incorrectly requiring python3
even though the compatible mock is started by node in
start_fake_openai_compatible_api (in
test/e2e/lib/openai-compatible-api-proof.sh); update the preflight checks to
verify the presence of node (and optionally that node supports the required
flag, e.g., by running `node --version` or a no-op with
`--experimental-strip-types`) instead of python3, leave the existing curl
readiness check intact, and ensure any environment-variable gating for
USE_COMPAT_MOCK=1 still triggers this node check.
---
Nitpick comments:
In `@tools/pr-review-advisor/analyze.mts`:
- Around line 773-808: The overlap checks currently run up to 80 concurrent
tasks (the Promise.all over openPulls.slice(0, 80)) and each task may call
githubRestPaginated, which can trigger GitHub secondary rate limits and cause
sameFiles to silently fall back to empty; change the implementation to bound
concurrency (e.g., use a concurrency limiter like p-limit or an explicit worker
pool) when mapping openPulls to the overlap check so only a small number (e.g.,
4–8) of githubRestPaginated calls run at once; update the code around
overlaps/openPulls mapping (the async mapper that computes number, title, body,
labels, linkedIssues, duplicateLinkedIssues, sameFiles) to schedule each mapper
through the limiter and await Promise.all on the limited tasks instead of
launching all 80 at once.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 1d82198e-a9aa-4f7a-9d34-8d5e98823ab3
📒 Files selected for processing (27)
biome.jsonscripts/generate-openclaw-config.mtsscripts/patch-openclaw-slack-deny-feedback.mtssrc/lib/messaging/applier/build/messaging-build-applier.mtstest/e2e-scenario/fixtures/fake-openai-compatible.tstest/e2e-scenario/live/double-onboard.test.tstest/e2e-scenario/live/token-rotation.test.tstest/e2e/lib/fake-openai-compatible-api.mtstest/e2e/lib/openai-compatible-api-proof.shtest/e2e/test-concurrent-gateway-ports.shtest/e2e/test-double-onboard.shtest/e2e/test-issue-2478-crash-loop-recovery.shtest/e2e/test-openshell-gateway-upgrade.shtools/advisors/git.mtstools/advisors/github.mtstools/advisors/io.mtstools/advisors/json.mtstools/advisors/session.mtstools/e2e-advisor/analyze.mtstools/e2e-advisor/comment.mtstools/e2e-advisor/dispatch.mtstools/e2e-advisor/scenario-comment.mtstools/e2e-advisor/scenarios.mtstools/e2e-scenarios/workflow-boundary.mtstools/pr-review-advisor/analyze.mtstools/pr-review-advisor/comment.mtstools/pr-review-advisor/workflow-boundary.mts
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Contributor
E2E Advisor RecommendationRequired E2E: None Full advisor summaryE2E Recommendation AdvisorFailed: Could not parse JSON from advisor output; see /home/runner/work/NemoClaw/NemoClaw/artifacts/e2e-advisor/e2e-advisor-raw-output.txt |
Contributor
Vitest E2E Scenario RecommendationRequired Vitest E2E scenarios: None Full Vitest E2E advisor summaryVitest E2E Scenario AdvisorFailed: Could not parse JSON from advisor output; see /home/runner/work/NemoClaw/NemoClaw/artifacts/e2e-advisor/e2e-scenario-advisor-raw-output.txt |
Contributor
PR Review AdvisorFindings: 0 needs attention, 1 worth checking, 0 nice ideas Review findings🛠️ Needs attention
🔎 Worth checking
🌱 Nice ideas
Consider writing more tests for
This is an automated advisory review. A human maintainer must make the final merge decision. |
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
…patible-e2e # Conflicts: # tools/e2e-scenarios/workflow-boundary.mts
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Contributor
Selective E2E Results — ❌ Some jobs failedRun: 27454997874
|
Contributor
Selective E2E Results —
|
| Job | Result |
|---|---|
| concurrent-gateway-ports-e2e | |
| double-onboard-e2e |
…' into codex/fake-openai-compatible-e2e
Contributor
Vitest E2E Scenario Results — ✅ All jobs passedRun: 27454997804
|
Contributor
Selective E2E Results — ✅ All requested jobs passedRun: 27455188119
|
This was referenced Jun 24, 2026
jyaunches
added a commit
that referenced
this pull request
Jun 25, 2026
## Summary Restore issue #5800 parity package `P0-A` for merged bash-suite inference-helper deltas only. This is a focused support-test parity PR: the helper/product behavior already exists on `main`; this adds missing Vitest assertions so shell retirement keeps the hosted/hermetic inference contracts covered. ## Related Issues Refs #5800 Refs #5098 Refs #5373 Refs #5374 Refs #5385 Refs #5395 Refs #5399 Refs #5400 Refs #5411 Refs #5751 Refs #5672 Refs #5757 ## Scope gate - Package: `P0-A — Hosted/hermetic inference helper parity` - Included PRs all merged and touched `test/e2e`: yes - Out of scope: unmerged/non-bash PRs; product cleanup; shell lane retirement / PR #5756 cleanup ## Parity map | ID | Source PR | Contract | Inference classification | Vitest assertion / waiver | Status | | --- | --- | --- | --- | --- | --- | | A1 | #5373 | Fake OpenAI-compatible helper supports `/models`, chat completions, responses API, auth checking, and request capture. | `hermetic-default` | `test/e2e-scenario/support-tests/hosted-inference.test.ts` starts `startFakeOpenAiCompatibleServer` and asserts models/chat/responses/request log behavior. | covered | | A2 | #5374, #5385, #5395 | Hosted CI inference stages `NVIDIA_INFERENCE_API_KEY` as `COMPATIBLE_API_KEY`, routes as `custom`/`compatible-endpoint`, and prefers `openai-completions`. | `hosted-compatible capable` | Existing workflow/helper assertions plus new shell helper staging assertion in `hosted-inference.test.ts`. | covered | | A3 | #5399, #5751, #5672, #5757 | Hosted model default remains the Inference Hub provider/namespace/model ID `nvidia/nvidia/nemotron-3-ultra`; explicit `NEMOCLAW_MODEL` takes precedence over `NEMOCLAW_COMPAT_MODEL`, which takes precedence over helper options/default. | `hosted-compatible capable` | New `requireHostedInferenceConfig` model precedence/default assertion; existing workflow/model namespace tests remain green. | covered | | A4 | #5400, #5411 | Hosted reachability probe is bounded and low-cost: no `/models`, chat completions, auth header, or bearer token spend. | `hosted-compatible capable` | Existing probe tests retained and revalidated. | covered | | A5 | #5385 | Public NVIDIA/nvapi shell mode remains distinct from hosted-compatible mode and is not restaged as compatible inference. | `public-nvidia required` | New shell helper assertion checks `nvapi-*` + `cloud` keeps `nvidia-prod`, leaves `COMPATIBLE_API_KEY` unset. | covered | ## Inference mode support - Default mode for touched live targets: none touched; this PR only changes support tests. - Real inference support preserved: yes, by asserting hosted-compatible and public-NVIDIA helper boundaries without invoking real inference. - Modes validated in this PR: hermetic fake endpoint and shell helper mocked hosted-compatible/public boundary. - If not validated with real inference: not required; no live target or hosted secret path changed. ## Validation - [x] `npx vitest run --project e2e-vitest-support test/e2e-scenario/support-tests/hosted-inference.test.ts` - [x] `npx vitest run test/e2e-script-workflow.test.ts test/issue-5667-hosted-inference-model-namespace.test.ts src/lib/inference/onboard-probes.test.ts src/lib/onboard/providers.test.ts` - [x] `git diff --check` - [ ] hosted/public selective E2E workflow, if required by classification: not required; support-test-only PR, no live/workflow behavior changed. ## Follow-ups / waivers - Pre-push full `Test (CLI)` / `Test (plugin)` hooks were not clean on local macOS after the commit: CLI run hit existing macOS/stat/OOM-style failures; plugin run could not import package `json5`. Focused target tests above passed. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Expanded end-to-end coverage for hosted inference compatibility and model ID/model precedence behavior. * Added validations for environment variable staging rules and shell mode behavior across NVAPI key scenarios. * Introduced a fake OpenAI-compatible server and added contract checks for `/models`, auth-required flows, and streamed responses on chat/response endpoints, including cleanup after runs. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
This was referenced Jul 10, 2026
Hadar301
pushed a commit
to Hadar301/NemoClaw-OpenShift
that referenced
this pull request
Jul 12, 2026
## Summary Restore issue NVIDIA#5800 parity package `P0-A` for merged bash-suite inference-helper deltas only. This is a focused support-test parity PR: the helper/product behavior already exists on `main`; this adds missing Vitest assertions so shell retirement keeps the hosted/hermetic inference contracts covered. ## Related Issues Refs NVIDIA#5800 Refs NVIDIA#5098 Refs NVIDIA#5373 Refs NVIDIA#5374 Refs NVIDIA#5385 Refs NVIDIA#5395 Refs NVIDIA#5399 Refs NVIDIA#5400 Refs NVIDIA#5411 Refs NVIDIA#5751 Refs NVIDIA#5672 Refs NVIDIA#5757 ## Scope gate - Package: `P0-A — Hosted/hermetic inference helper parity` - Included PRs all merged and touched `test/e2e`: yes - Out of scope: unmerged/non-bash PRs; product cleanup; shell lane retirement / PR NVIDIA#5756 cleanup ## Parity map | ID | Source PR | Contract | Inference classification | Vitest assertion / waiver | Status | | --- | --- | --- | --- | --- | --- | | A1 | NVIDIA#5373 | Fake OpenAI-compatible helper supports `/models`, chat completions, responses API, auth checking, and request capture. | `hermetic-default` | `test/e2e-scenario/support-tests/hosted-inference.test.ts` starts `startFakeOpenAiCompatibleServer` and asserts models/chat/responses/request log behavior. | covered | | A2 | NVIDIA#5374, NVIDIA#5385, NVIDIA#5395 | Hosted CI inference stages `NVIDIA_INFERENCE_API_KEY` as `COMPATIBLE_API_KEY`, routes as `custom`/`compatible-endpoint`, and prefers `openai-completions`. | `hosted-compatible capable` | Existing workflow/helper assertions plus new shell helper staging assertion in `hosted-inference.test.ts`. | covered | | A3 | NVIDIA#5399, NVIDIA#5751, NVIDIA#5672, NVIDIA#5757 | Hosted model default remains the Inference Hub provider/namespace/model ID `nvidia/nvidia/nemotron-3-ultra`; explicit `NEMOCLAW_MODEL` takes precedence over `NEMOCLAW_COMPAT_MODEL`, which takes precedence over helper options/default. | `hosted-compatible capable` | New `requireHostedInferenceConfig` model precedence/default assertion; existing workflow/model namespace tests remain green. | covered | | A4 | NVIDIA#5400, NVIDIA#5411 | Hosted reachability probe is bounded and low-cost: no `/models`, chat completions, auth header, or bearer token spend. | `hosted-compatible capable` | Existing probe tests retained and revalidated. | covered | | A5 | NVIDIA#5385 | Public NVIDIA/nvapi shell mode remains distinct from hosted-compatible mode and is not restaged as compatible inference. | `public-nvidia required` | New shell helper assertion checks `nvapi-*` + `cloud` keeps `nvidia-prod`, leaves `COMPATIBLE_API_KEY` unset. | covered | ## Inference mode support - Default mode for touched live targets: none touched; this PR only changes support tests. - Real inference support preserved: yes, by asserting hosted-compatible and public-NVIDIA helper boundaries without invoking real inference. - Modes validated in this PR: hermetic fake endpoint and shell helper mocked hosted-compatible/public boundary. - If not validated with real inference: not required; no live target or hosted secret path changed. ## Validation - [x] `npx vitest run --project e2e-vitest-support test/e2e-scenario/support-tests/hosted-inference.test.ts` - [x] `npx vitest run test/e2e-script-workflow.test.ts test/issue-5667-hosted-inference-model-namespace.test.ts src/lib/inference/onboard-probes.test.ts src/lib/onboard/providers.test.ts` - [x] `git diff --check` - [ ] hosted/public selective E2E workflow, if required by classification: not required; support-test-only PR, no live/workflow behavior changed. ## Follow-ups / waivers - Pre-push full `Test (CLI)` / `Test (plugin)` hooks were not clean on local macOS after the commit: CLI run hit existing macOS/stat/OOM-style failures; plugin run could not import package `json5`. Focused target tests above passed. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Expanded end-to-end coverage for hosted inference compatibility and model ID/model precedence behavior. * Added validations for environment variable staging rules and shell mode behavior across NVAPI key scenarios. * Introduced a fake OpenAI-compatible server and added contract checks for `/models`, auth-required flows, and streamed responses on chat/response endpoints, including cleanup after runs. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
wscurran
added
the
area: e2e
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Centralizes the generic OpenAI-compatible fake provider used by E2E tests so shell and Vitest scenarios share the same canned inference behavior. Also broadens Biome coverage to include
.mtshelpers and formats the existing.mtstool scripts.Changes
.mtsfake OpenAI-compatible API server with models, chat completions, responses, auth, streaming, and request logging support.**/*.mtsin Biome and apply formatting/linting to existing.mtsfiles.Type of Change
Verification
npx prek run --all-filespassesnpm testpassesnpm run docsbuilds without warnings (doc changes only)Signed-off-by: Carlos Villela cvillela@nvidia.com
Summary by CodeRabbit
Chores
Tests
CI