Skip to content

test(e2e): make inference switch baselines sandbox-reachable - #6989

Merged
cv merged 1 commit into
mainfrom
codex/fix-inference-switch-baseline-route
Jul 16, 2026
Merged

test(e2e): make inference switch baselines sandbox-reachable#6989
cv merged 1 commit into
mainfrom
codex/fix-inference-switch-baseline-route

Conversation

@prekshivyas

@prekshivyas prekshivyas commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

Summary

Make the mock OpenAI-compatible baselines in the OpenClaw and Hermes inference-switch live E2E targets reachable from the OpenShell gateway. Previously the runner-side smoke probe passed against loopback, but sandbox deployment verification returned HTTP 503 because the gateway could not reach the runner's loopback address.

Changes

  • Bind both mock baseline servers to 0.0.0.0 while advertising host.openshell.internal to OpenShell.
  • Document the gateway network-namespace boundary at both fixture call sites.
  • Keep the production compatible-endpoint rewrite and its policy-approved port allowlist unchanged.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: This only corrects live E2E fixture reachability; CLI behavior, defaults, APIs, policies, supported integrations, and user workflows are unchanged. Existing docs already describe host.openshell.internal for sandbox-to-host inference routes.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: The change is confined to mock baseline setup in two live E2E tests. Production inference routing, network policies, credentials, and sandbox implementation are unchanged; both call sites now use the established host-bridge pattern already exercised by other live E2E targets.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — npx vitest run --project e2e-support test/e2e/support/e2e-cleanup-resources.test.ts test/e2e/support/e2e-clients.test.ts (2 files, 70 tests passed). The exact live targets require external state and are left to the E2E workflow.
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: Not applicable to this focused live E2E configuration change. A broader e2e-support run was attempted; the mapped tests passed, while 12 unrelated workflow-boundary subprocess timeout failures occurred across five unchanged files.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Prekshi Vyas prekshiv@nvidia.com

Summary by CodeRabbit

  • Tests
    • Updated live inference-switch test scenarios to use explicit network routing for baseline compatibility servers.
    • Improved test reliability when running with mocked Anthropic-compatible endpoints.

@prekshivyas prekshivyas self-assigned this Jul 16, 2026
@coderabbitai

coderabbitai Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f6b2a243-60d1-41a3-bccf-d4d09181582f

📥 Commits

Reviewing files that changed from the base of the PR and between 07ed842 and 0af6203.

📒 Files selected for processing (2)
  • test/e2e/live/hermes-inference-switch.test.ts
  • test/e2e/live/openclaw-inference-switch.test.ts

📝 Walkthrough

Walkthrough

The Hermes and OpenClaw live inference-switch tests now configure baseline fake OpenAI-compatible servers with explicit binding and public host values for gateway network-namespace reachability.

Changes

Inference fixture networking

Layer / File(s) Summary
Configure gateway-reachable fake servers
test/e2e/live/hermes-inference-switch.test.ts, test/e2e/live/openclaw-inference-switch.test.ts
Baseline fake servers now bind to 0.0.0.0 and use host.openshell.internal as their public host, with comments documenting the gateway routing constraint.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Suggested labels: bug-fix, area: e2e, area: networking

Suggested reviewers: ericksoa, cv

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: making inference-switch test baselines reachable from the sandbox.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/fix-inference-switch-baseline-route

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage remains at 96%, unchanged from the main branch.

TypeScript / code-coverage/cli

The overall coverage in the codex/fix-inference-... branch remains at 80%, unchanged from the main branch.

Show a code coverage summary of the most impacted files.
File main 710d2b3 codex/fix-inference-... 0af6203 +/-
src/lib/state/config-io.ts 94% 90% -4%
src/lib/agent/dashboard-ui.ts 89% 85% -4%
src/lib/agent/m...fest-readers.ts 97% 95% -2%
src/lib/adapter...tp/curl-args.ts 97% 98% +1%
src/lib/state/gateway.ts 91% 93% +2%
src/lib/messagi.../persistence.ts 86% 89% +3%
src/lib/messagi...n-validation.ts 96% 100% +4%
src/lib/adapter...shell/client.ts 83% 88% +5%
src/lib/agent/aliases.ts 94% 100% +6%
src/lib/state/r...y-entry-view.ts 80% 100% +20%

Updated July 16, 2026 03:24 UTC
Code Coverage is in Public Preview. Learn more and provide us with your feedback.

@prekshivyas
prekshivyas requested a review from cv July 16, 2026 03:16
@github-actions

github-actions Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — Informational

Advisor assessment: Informational / medium confidence
Next action: Review the warnings below.
Findings: 0 blockers · 1 warning · 0 suggestions
Status: Canonical ledger: 0 blocker(s), 1 warning(s), 0 suggestion(s).

Model lanes

  • GPT-5.6 Terra (primary): Completed · medium confidence · 0 blockers · 1 warning · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Model comparison: normalized findings differ; normalized E2E selections match; Nemotron reported the same number of blockers, 1 fewer warning, the same number of suggestions.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, credential-sanitization, security-posture, hermes-inference-switch, inference-routing, network-policy, openclaw-inference-switch

1 warning · 0 suggestions

Warnings

Warnings do not block.

PRA-1 Warning — Keep the mock inference listener scoped to the OpenShell bridge

  • Location: test/e2e/live/hermes-inference-switch.test.ts:127
  • Category: security
  • Problem: The mock baseline now binds its authenticated HTTP server to `0.0.0.0`, which listens on every runner interface for the duration of the live test. The same broad bind is added to the OpenClaw target. `host.openshell.internal` is the intended narrow sandbox-to-host route, but the fixture does not restrict the listener to the corresponding bridge address or otherwise demonstrate that other runner-network interfaces cannot reach it.
  • Impact: A network peer able to reach the runner during either live test can attempt requests to the temporary mock inference service, expanding the exposure from loopback-only beyond the required sandbox gateway boundary.
  • Recommendation: Bind the fixture to the specific OpenShell/Docker bridge address that backs `host.openshell.internal`, or apply an equivalent interface-level restriction, while retaining the alias as the advertised sandbox endpoint.
  • Verification: Inspect the fixture process listener during the live-target setup and compare its bound address with the interface resolved for `host.openshell.internal`; confirm no non-bridge runner interface accepts a connection.
  • Test coverage: Add a fixture or live-boundary regression proving the advertised `host.openshell.internal` route is reachable from the gateway namespace while a connection through a non-bridge runner interface is refused.
  • Evidence: `test/e2e/live/hermes-inference-switch.test.ts:127` adds `host: "0.0.0.0"` for the baseline provider. `test/e2e/live/openclaw-inference-switch.test.ts:942` adds the same all-interface bind. `docs/inference/custom-endpoint-security.mdx:43-60` describes `host.openshell.internal:<port>` as a narrow sandbox-host exception, rather than a general private-endpoint bypass. The fixture uses `server.listen(port, host)` in `test/e2e/lib/fake-openai-compatible-api.mts:234`, so `0.0.0.0` directly controls listener exposure.

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

@prekshivyas

Copy link
Copy Markdown
Collaborator Author

Reviewed PRA-1; no code change is planned for this PR.

host.openshell.internal is injected through Docker's host-gateway routing, whose concrete host address differs between native Linux bridges and Docker Desktop/VM-backed Docker. Binding these fixtures to a discovered interface address would make the live targets platform-specific and would not match the repository's established E2E convention: channels-add-remove, device-auth-health, onboard-repair, token-rotation, the shared inference adapter, and other live targets all bind temporary fake services to 0.0.0.0 while advertising host.openshell.internal.

These two servers are short-lived CI fixtures, require a fixed non-secret test credential for inference, and expose no user data or production credential. This PR does not change any production listener or policy. If the project wants interface-specific binding, it should be designed once as a cross-platform fixture hardening change for all existing host-bridge test servers rather than applied inconsistently at these two call sites.

@cv
cv merged commit 3bc9620 into main Jul 16, 2026
81 of 84 checks passed
@cv
cv deleted the codex/fix-inference-switch-baseline-route branch July 16, 2026 03:39
@wscurran wscurran added area: ci CI workflows, checks, release automation, or GitHub Actions area: e2e End-to-end tests, nightly failures, or validation infrastructure area: networking DNS, proxy, TLS, ports, host aliases, or connectivity bug-fix PR fixes a bug or regression integration: hermes Hermes integration behavior integration: openclaw OpenClaw integration behavior labels Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ci CI workflows, checks, release automation, or GitHub Actions area: e2e End-to-end tests, nightly failures, or validation infrastructure area: networking DNS, proxy, TLS, ports, host aliases, or connectivity bug-fix PR fixes a bug or regression integration: hermes Hermes integration behavior integration: openclaw OpenClaw integration behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants