test: point router/completion/triton tests at the local fake OpenAI endpoint

[mateo-berri]

Relevant issues

Follow-up to #30695, which replaced the shared Railway-hosted fake OpenAI mock (exampleopenaiendpoint-production.up.railway.app) with a job-local server for the proxy E2E jobs. That PR migrated the mounted YAML configs but left the api_base literals that several unit/integration tests hardcode in Python still pointing at the hosted host. When Railway is unreachable those tests fail with 404 "Application not found", which is exactly what reddened litellm_router_testing, local_testing_part1, local_testing_part2 and llm_translation_testing on #30894 even though that PR only rebuilds UI artifacts

Linear ticket

N/A

Pre-Submission checklist

• I have added meaningful tests
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

CI (LiteLLM team)

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Screenshots / Proof of Fix

The fix is that the previously-failing tests no longer touch any external host. With nothing pre-started, the test session brings up the local server itself (the local_testing and llm_translation conftests call ensure_fake_openai_endpoint, which reuses a CI-started server when one is already healthy). The two behaviors the migrated tests depend on are the Triton embeddings shape and the slow-endpoint delay; here is the local server serving both, plus the regular chat path, with Railway entirely out of the loop:

$ uv run --no-sync python tests/_fake_openai_endpoint_server.py --host 127.0.0.1 --port 8190 &
$ curl -s http://127.0.0.1:8190/health
ok

$ curl -s -X POST http://127.0.0.1:8190/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -d '{"model":"gpt-4","messages":[{"role":"user","content":"hi"}]}'
{"id":"chatcmpl-...","object":"chat.completion","model":"gpt-4","choices":[{"index":0,"message":{"role":"assistant","content":"Hello! This is a mock response from the fake OpenAI endpoint."},"finish_reason":"stop"}],"usage":{"prompt_tokens":20,"completion_tokens":20,"total_tokens":40}}

$ curl -s -X POST http://127.0.0.1:8190/triton/embeddings -d '{"inputs":[]}'
{"model_name":"my-triton-model","outputs":[{"name":"output","datatype":"FP32","shape":[1,2],"data":[0.1,0.2]}]}

$ time curl -s -m 1 -o /dev/null -w 'exit=%{exitcode}\n' -X POST http://127.0.0.1:8190/v1/chat/completions \
    -d '{"model":"slow-endpoint","messages":[{"role":"user","content":"hi"}]}'
exit=28        # client-side timeout: slow-endpoint sleeps past the 1s router timeout
real    0m1.0s

The end-to-end proof is the branch CI run: litellm_router_testing, local_testing_part1, local_testing_part2 and llm_translation_testing should go green with the Railway host never contacted

Type

🐛 Bug Fix
✅ Test

Changes

Adds tests/fake_openai_endpoint.py, a small shared helper that exposes FAKE_OPENAI_API_BASE (resolved from the env var, default http://127.0.0.1:8190) and an idempotent ensure_fake_openai_endpoint() that reuses an already-healthy server (via its /health check) or spawns tests/_fake_openai_endpoint_server.py detached for the session. It deliberately registers no per-process teardown: under pytest-xdist the spawn happens in whichever worker wins the race and workers exit independently, so an atexit hook would tear the shared server down while siblings are still calling it. CI containers are ephemeral and a cold local re-run just reuses the still-healthy server, so the leak is harmless. The local_testing and llm_translation conftests start it via a session-scoped autouse fixture, so those jobs no longer need the endpoint pre-wired in CI and a cold local run works the same as CI

Replaces the hardcoded Railway api_base literals in the eight tests that actually made live calls (test_triton, test_router, test_router_custom_routing, test_router_fallback_handlers, test_router_fallbacks, test_secret_detect_hook, test_lowest_latency_routing, test_completion) with FAKE_OPENAI_API_BASE. The respx/mock/cassette-backed references in other files are left alone since they never hit the network. The deliberately broken ...railway.appzzzzz fallback URL is also left as-is so fallback handling still has a failing upstream

Extends the canned server with a /triton/embeddings route returning the Triton output shape the embedding transform expects, and a slow-endpoint model that sleeps past the latency test's 1s timeout so the timeout/penalty path runs deterministically offline. Both are inert for the existing proxy E2E jobs, which never send those

Adds tests/local_testing/test_fake_openai_endpoint.py covering the chat shape, the Triton route, the slow-endpoint delay, and a guard that fails if any migrated file reintroduces a live railway.app api_base

---

Note

Low Risk
Test-only infrastructure and URL swaps; no production LiteLLM runtime paths change.

Overview
CI and local runs no longer depend on the shared Railway fake OpenAI host. Router, completion, Triton, and related tests that used to hardcode exampleopenaiendpoint-production.up.railway.app now point api_base at FAKE_OPENAI_API_BASE via a new tests/fake_openai_endpoint.py helper.

That helper resolves the base URL (loopback-only if FAKE_OPENAI_API_BASE is set; remote env values are ignored so CI does not try to bind locally to a hosted URL), health-checks /health, and spawns _fake_openai_endpoint_server.py when needed. The subprocess is detached and not torn down on interpreter exit so pytest-xdist workers do not kill a shared mock mid-session. local_testing and llm_translation conftests start it with a session-scoped autouse fixture.

The canned server gains slow-endpoint (3s delay for latency/timeout tests) and POST /triton/embeddings with the embedding shape Triton tests expect. test_fake_openai_endpoint.py covers those behaviors and fails if migrated files reintroduce live railway.app URLs. VCR is told to skip that file since it hits loopback only.

^{Reviewed by Cursor Bugbot for commit d03f857. Bugbot is set up for automated code reviews on this repo. Configure here.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[mateo-berri]

@greptileai

---

Generated by Claude Code

[greptile-apps]

Greptile Summary

This PR decouples router, completion, Triton, and related tests from the shared Railway-hosted fake OpenAI endpoint by introducing a local mock server helper and pointing all hardcoded api_base literals at FAKE_OPENAI_API_BASE. No production LiteLLM code is modified.

• tests/fake_openai_endpoint.py — new shared helper that resolves FAKE_OPENAI_API_BASE, health-checks it, and spawns _fake_openai_endpoint_server.py detached when needed; lru_cache prevents redundant spawns within a process, and the deliberate lack of atexit teardown is correct for pytest-xdist.
• tests/_fake_openai_endpoint_server.py — gains a /triton/embeddings route and a slow-endpoint model that sleeps 3 s, enabling the Triton and latency-routing tests to run fully offline.
• Eight test files — have their Railway URL literals replaced with FAKE_OPENAI_API_BASE; both conftests gain a session-scoped autouse fixture; test_fake_openai_endpoint.py adds a regex guard against Railway URLs re-appearing.

Confidence Score: 5/5

Safe to merge — all changes are test infrastructure only; no production LiteLLM runtime paths are touched.

Every change is isolated to the test layer. The helper design decisions (no atexit teardown, loopback-only env honoring, lru_cache per process) are sound and well-documented. The one minor gap is the Railway-URL regex guard not covering single-quoted or comma-terminated strings, but this does not affect CI correctness.

No files require special attention.

Important Files Changed

Filename	Overview
tests/fake_openai_endpoint.py	New shared helper that resolves FAKE_OPENAI_API_BASE, health-checks it, and spawns the canned server when needed; well-documented design choices
tests/_fake_openai_endpoint_server.py	Adds /triton/embeddings route and slow-endpoint sleep logic; both are inert for existing proxy E2E tests
tests/local_testing/test_fake_openai_endpoint.py	New regression test covering chat shape, Triton route, slow-endpoint delay, and a regex guard against Railway URLs re-appearing; guard regex has a minor gap with single-quoted strings
tests/local_testing/conftest.py	Adds session-scoped autouse fixture to start the fake endpoint and excludes the new test file from VCR recording
tests/llm_translation/conftest.py	Same session-scoped autouse fixture pattern as local_testing/conftest.py
tests/local_testing/test_lowest_latency_routing.py	Replaces Railway URLs with FAKE_OPENAI_API_BASE; slow-endpoint behavior now served locally via the 3s sleep in the canned server
tests/local_testing/test_router_fallback_handlers.py	URL swap and updates the assertion on _hidden_params["api_base"] to match the new local base
tests/llm_translation/test_triton.py	Points Triton embedding test at the new /triton/embeddings route on the local server
tests/local_testing/test_router.py	Replaces Railway URL literals with the FAKE_OPENAI_API_BASE constant
tests/local_testing/test_router_fallbacks.py	URL swap for two router configurations; test logic and assertions unchanged
tests/local_testing/test_router_custom_routing.py	URL swap in _create_router(); no logic changes
tests/local_testing/test_secret_detect_hook.py	URL swap in module-level router; Router is created at import time but connects only when tests run
tests/local_testing/test_completion.py	URL swap for test_lm_studio_completion; straightforward

_{Reviews (4): Last reviewed commit: "fix(tests): keep fake OpenAI mock alive ..." | Re-trigger Greptile}

[mateo-berri]

@greptileai

---

Generated by Claude Code

[mateo-berri]

bugbot run

---

Generated by Claude Code

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: xdist worker kills shared mock
  • Removed the per-worker atexit hook and spawned the mock with start_new_session=True so the subprocess detaches from the spawning xdist worker and survives until the host or container exits, while the existing /health reuse path covers subsequent runs.

_{You can send follow-ups to the cloud agent here.}

[CLAassistant]

All committers have signed the CLA.

[mateo-berri]

@greptileai

---

Generated by Claude Code

[mateo-berri]

bugbot run

---

Generated by Claude Code

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit d03f857. Configure here.}