test(e2e): align release fixtures with readiness checks

[prekshivyas]

Summary

Align the v0.0.84 live E2E fixtures with gateway-network reachability and inventory-only readiness checks. The affected tests previously used runner loopback addresses or asserted superseded completion and deployment text, producing deterministic release-gate failures despite the intended runtime behavior.

Changes

• Advertise the double-onboard and concurrent-gateway mock inference endpoints through host.openshell.internal while binding their authenticated fixture servers for gateway access.
• Assert the new Hermes deployment verification text that includes inference-route health.
• Treat an authenticated /v1/models inventory request as Hermes baseline readiness evidence instead of requiring a token-consuming chat completion.
• Add focused support coverage that rejects the superseded chat-completion assertion.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification: Fixture and assertion corrections only; supported user behavior and configuration are unchanged, and the canonical v0.0.84 changelog already documents inventory-only readiness.
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Pending PR review; the diff changes live E2E fixtures only and does not alter production paths.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — npx vitest run --project e2e-support test/e2e/support/hermes-inference-switch-command-shape.test.ts (19 passed)
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result:
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Prekshi Vyas prekshiv@nvidia.com

Summary by CodeRabbit

• Bug Fixes
  • Improved live E2E connectivity by explicitly configuring the fake OpenAI-compatible server to be reachable from the gateway network namespace.
  • Strengthened Hermes deployment verification to require healthy gateway, dashboard, and inference route status.
• Tests
  • Updated Hermes inference baseline checks to use authenticated model-inventory (/v1/models) requests instead of chat-completions readiness.
  • Added assertions covering “baseline readiness evidence” behavior for accepted GET vs rejected authenticated POST request shapes.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 9d393cc1-eee6-47cc-9bef-5022582ba0e5

📥 Commits

Reviewing files that changed from the base of the PR and between f1050c1 and ad73cf5.

📒 Files selected for processing (2)

• test/e2e/live/hermes-inference-switch-helpers.ts
• test/e2e/support/hermes-inference-switch-command-shape.test.ts

🚧 Files skipped from review as they are similar to previous changes (1)

• test/e2e/live/hermes-inference-switch-helpers.ts

---

📝 Walkthrough

Walkthrough

The E2E tests configure fake OpenAI-compatible servers for gateway-network access and update Hermes readiness checks to use authenticated model inventory requests. Dashboard-enabled installation assertions now also require inference-route health.

Changes

Fake server network addressing

Layer / File(s)	Summary
Gateway-reachable fake server configuration test/e2e/live/concurrent-gateway-ports.test.ts, test/e2e/live/double-onboard.test.ts	Fake servers bind to 0.0.0.0 and advertise host.openshell.internal while retaining configurable ports.

Hermes inference verification

Layer / File(s)	Summary
Authenticated inventory readiness assertion test/e2e/live/hermes-inference-switch-helpers.ts, test/e2e/live/hermes-inference-switch.test.ts, test/e2e/support/hermes-inference-switch-command-shape.test.ts	Baseline verification now requires an authenticated GET /v1/models request, with updated call sites and coverage for accepted and rejected request shapes.
Inference route health verification test/e2e/live/hermes-e2e.test.ts	The dashboard-enabled installation assertion now expects gateway, dashboard, and inference-route health in the deployment verification message.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• NVIDIA/NemoClaw#6989: Both PRs configure fake OpenAI-compatible servers with explicit gateway-reachable networking.

Suggested labels: integration: hermes, area: e2e, chore

Suggested reviewers: cv, apurvvkumaria

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is concise and accurately reflects the main change: updating E2E release fixtures to match new readiness checks.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/fix-v0.0.84-e2e-fixtures

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage remains at 96%, unchanged from the main branch.

TypeScript / code-coverage/cli

The overall coverage in the codex/fix-v0.0.84-e2... branch remains at 80%, unchanged from the main branch.

Show a code coverage summary of the most impacted files.

File	main 3bc9620	codex/fix-v0.0.84-e2... ad73cf5	+/-
src/lib/state/gateway.ts	91%	93%	+2%

---

_{Updated July 16, 2026 04:42 UTC
Code Coverage is in Public Preview. Learn more and provide us with your feedback.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e/live/hermes-inference-switch-helpers.ts`:
- Around line 87-98: The expectAuthenticatedBaselineInventoryRequest matcher
must require the readiness request to be an authenticated GET /v1/models call.
Add method: "GET" to its expected request shape, and update the command-shape
test to cover that a POST /v1/models does not satisfy the matcher.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b90c2396-476f-48e3-8407-1ca25dabb734

📥 Commits

Reviewing files that changed from the base of the PR and between 3bc9620 and f1050c1.

📒 Files selected for processing (6)

• test/e2e/live/concurrent-gateway-ports.test.ts
• test/e2e/live/double-onboard.test.ts
• test/e2e/live/hermes-e2e.test.ts
• test/e2e/live/hermes-inference-switch-helpers.ts
• test/e2e/live/hermes-inference-switch.test.ts
• test/e2e/support/hermes-inference-switch-command-shape.test.ts

[github-actions]

PR Review Advisor — Informational

Advisor assessment: Informational / high confidence
Next action: Review the warnings below.
Findings: 0 blockers · 1 warning · 0 suggestions
Status: Canonical ledger: 0 blocker(s), 1 warning(s), 0 suggestion(s).

Model lanes

• GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 1 warning · 0 suggestions
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 2 warnings · 0 suggestions
• Model comparison: normalized findings differ; normalized E2E selections match; Nemotron reported the same number of blockers, 1 more warning, the same number of suggestions.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, credential-sanitization, security-posture, concurrent-gateway-ports, double-onboard, hermes-dashboard, hermes-e2e, hermes-inference-switch, inference-routing, network-policy

1 warning · 0 suggestions

Warnings

Warnings do not block.

PRA-1 Warning — Enforce authentication on the inventory readiness route

• Location: test/e2e/live/hermes-inference-switch.test.ts:131
• Category: correctness
• Problem: The new baseline assertion treats a GET /v1/models record with auth: "ok" as proof of authenticated readiness, but this fixture is created with requireAuth only. The fake server applies requireAuthModels independently, so its model route accepts a request bearing an incorrect credential and records it as auth: "ok" whenever requireAuthModels is unset. Thus the changed check does not prove that the baseline credential is accepted for the inventory request.
• Impact: A regression that supplies an invalid baseline credential specifically to model-inventory readiness can pass this E2E assertion, weakening coverage of the intended authenticated readiness contract.
• Recommendation: Set requireAuthModels: true when constructing mockBaseline, so /v1/models validates the bearer token as the assertion intends.
• Verification: Read test/e2e/lib/fake-openai-compatible-api.mts around the GET /v1/models handler and confirm that requireAuthModels controls whether the header must equal the configured bearer token.
• Test coverage: Add a fixture or integration test showing missing or incorrect credentials on GET /v1/models receive 401 and cannot satisfy expectAuthenticatedBaselineInventoryRequest readiness evidence.
• Evidence: test/e2e/live/hermes-inference-switch.test.ts:126-132 creates mockBaseline with requireAuth: true but omits requireAuthModels. test/e2e/live/hermes-inference-switch-helpers.ts:79-92 accepts a recorded auth: "ok" and authorizationSent inventory request as readiness evidence. test/e2e/lib/fake-openai-compatible-api.mts:145-171 sets modelsAuthOk from requireAuthModels and only validates the bearer token when that separate option is enabled.

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.