feat(e2e): in-network mock provider (OpenAI + Anthropic + Gemini) - #34
Merged
Conversation
Adds a stdlib-only mock OpenAI+Anthropic+Gemini provider container that
runs alongside the litellm e2e proxy under a docker-compose profile
(`--profile mock`). Reachable via `e2e/tools/proxy start --with-mock`.
Three model_list entries are auto-rendered when the flag is set:
- mock-openai -> openai/mock-model
- mock-anthropic -> anthropic/mock-claude
- mock-gemini -> gemini/mock-gemini-pro
Capabilities (see e2e/_config/mock_provider.py header for full contract):
* Streaming + non-streaming for all three provider shapes, including
the Anthropic SSE event sequence (message_start, content_block_*,
message_delta, message_stop) and Gemini's
`/v1beta/models/<m>:streamGenerateContent`.
* Per-request controls via X-Mock-* HTTP headers — these survive
litellm transforms (request body fields don't on Anthropic/Gemini
paths, since those transforms strip unknown JSON fields):
X-Mock-Chunks, X-Mock-Chunk-Chars, X-Mock-Full-Chars
X-Mock-TTFT-Ms, X-Mock-TPS
X-Mock-Fail (force 503|429|401|400|500, provider-shaped envelope)
X-Mock-Tool-Call (`name(k1=v1,k2=v2)` emits tool_calls / tool_use /
functionCall)
* Process-level defaults via env: MOCK_TTFT_MS, MOCK_TPS, MOCK_CHUNKS,
CALLBACK_DELAY, FAIL_RATE. The rendered config opts into
`general_settings.forward_client_headers_to_llm_api: true` only when
--with-mock is set so X-Mock-* headers reach the upstream.
* Callback sinks: Langfuse `/api/public/ingestion` + GenericAPILogger
`/api/hooks/*`, both honouring CALLBACK_DELAY for slow-consumer
queue-retention testing.
* Introspection: `GET /__mock__/state` returns a JSON snapshot of all
counters (requests by provider/mode/status, chunks emitted,
in-flight, errors injected). `GET /__mock__/reset` zeroes them.
`GET /metrics` exposes the same counters in Prometheus text format
so test code can scrape both proxy and mock sides for end-to-end
reconciliation.
* Request validation: rejects malformed bodies with provider-shaped
400 errors (empty messages list, missing role, etc.) — needed for
case 21-style 4xx propagation tests.
* Thread-local RNG (eliminates lock contention + nondeterministic
output across ThreadingHTTPServer worker threads).
* Monotonic-paced TTFT/TPS — avoids drift from cumulative time.sleep
error in long streaming windows (~5% drift over 50 seconds with
naive sleep, ~0% with monotonic scheduling).
* Compact JSON encoding for SSE chunks (`json.dumps(..., separators=
(',',':'))`) — matches real provider wire format so test regexes
that look for `"type":"message_delta"` (no whitespace) hit.
New runner flag: `e2e/tools/run-all-cases --mock-only` skips Tier=real
cases (1, 2, 3, 5, 8, 9, 19 — prompt-caching token assertions,
thinking-signature blocks, label-coverage paths that depend on real
provider semantics) and runs the other 14 against the mock. Zero
provider cost; ~30 s end-to-end.
New case: `e2e/cases/23_mock_memory_pressure.md` documents how to
reproduce the production 12 GB OOM math:
- 5 × 40 MB concurrent body → peak Δ +900 MB / worker (4.5× body amp)
- `num_retries=2` × 30% upstream 503 → peak +1.5 GB / worker
- GenericAPILogger StandardLoggingPayload size: 0.4–3.7 MB / event
The case is the canonical reproducer for the analysis that drove the
production rollout of `MALLOC_ARENA_MAX=2` and body-size triage.
Updates `cases/README.md` with a Tier column (mock / real / both) on
every case and a "Mock-only mode" section documenting the env overlay
(`ANTHROPIC_API_BASE` / `OPENAI_API_BASE` / `GEMINI_API_BASE` →
`http://mock:8080`) needed before `--mock-only` runs.
No `litellm/` source changes. `mock_provider.py` is stdlib-only and
mounted into a `python:3.13-slim` sidecar — no install step, no
provider key, no network egress.
Test plan:
* `e2e/tools/proxy start --with-mock` boots cleanly, `/v1/models`
lists mock-openai / mock-anthropic / mock-gemini alongside the
existing real-provider deployments.
* `e2e/tools/proxy start` (no flag) is unchanged — mock service does
not start; rendered config has no mock model_list entries.
* `e2e/tools/run-all-cases --mock-only` against the mock-redirected
env: 14 PASS, 7 SKIP (Tier=real), 0 FAIL.
8 tasks
songkuan-zheng
added a commit
that referenced
this pull request
Jun 2, 2026
…ls/call (#36) Follow-up to #34. Three best-practice gaps the original PR documented but didn't actually implement: 1. `--mock-only` claimed to "auto-start the proxy with --with-mock and a mock-redirected .env overlay" but the code only skipped Tier=real cases — the env setup was the caller's job. Now the runner preflights three things before running anything: a. proxy reachable at $PROXY_URL, b. litellm-e2e-mock sidecar container running, c. ANTHROPIC_API_BASE inside the proxy container actually points at http://mock:8080. Any failure prints the exact fix commands (with the .env block to paste) and exits 2. README updated to match real behaviour. 2. Case 23 (mock memory pressure) was a Markdown-only runbook with no executable fixture, so `run-all-cases --mock-only` silently skipped it. Added e2e/cases/data/23_mock_memory_pressure.sh — a CI-light version (5 concurrent 200 KB streams, ~10 s) that asserts the two regressions the production OOM investigation identified as load-bearing: a. exactly N upstream calls per N client requests via the mock's /__mock__/state endpoint (regression test for retry amplification — the doc'd default num_retries=0 must stay 0), b. post-burst residual RSS bounded by 80 MB above baseline (catches a real Python-level retention regression). Peak RSS magnitude is reported in the PASS line for runbook context but not asserted — timing-dependent on slow CI hosts. Wired into run-all-cases, fires only under --mock-only. 3. `e2e/tools/call --provider` was hardcoded to the real-provider model aliases (anthropic / anthropic-haiku / openai), so writing new mock-driven cases required raw curl. Added mock-anthropic, mock-openai, mock-gemini choices that map to the model_list entries proxy renders under --with-mock. The Anthropic-shape cache_control transform path now extends to mock-anthropic so callers can exercise that codepath against the deterministic mock (the mock accepts the marker, doesn't account for it). After these fixes `e2e/tools/run-all-cases --mock-only` is: Total: 22 Pass: 15 Fail: 0 Skip: 7 (was 14/0/7 in #34 because case 23's fixture was missing).
This was referenced Jun 4, 2026
songkuan-zheng
added a commit
that referenced
this pull request
Jun 4, 2026
* feat(e2e): in-network mock provider for memory/retry/callback testing (#34) Adds a stdlib-only mock OpenAI+Anthropic+Gemini provider container that runs alongside the litellm e2e proxy under a docker-compose profile (`--profile mock`). Reachable via `e2e/tools/proxy start --with-mock`. Three model_list entries are auto-rendered when the flag is set: - mock-openai -> openai/mock-model - mock-anthropic -> anthropic/mock-claude - mock-gemini -> gemini/mock-gemini-pro Capabilities (see e2e/_config/mock_provider.py header for full contract): * Streaming + non-streaming for all three provider shapes, including the Anthropic SSE event sequence (message_start, content_block_*, message_delta, message_stop) and Gemini's `/v1beta/models/<m>:streamGenerateContent`. * Per-request controls via X-Mock-* HTTP headers — these survive litellm transforms (request body fields don't on Anthropic/Gemini paths, since those transforms strip unknown JSON fields): X-Mock-Chunks, X-Mock-Chunk-Chars, X-Mock-Full-Chars X-Mock-TTFT-Ms, X-Mock-TPS X-Mock-Fail (force 503|429|401|400|500, provider-shaped envelope) X-Mock-Tool-Call (`name(k1=v1,k2=v2)` emits tool_calls / tool_use / functionCall) * Process-level defaults via env: MOCK_TTFT_MS, MOCK_TPS, MOCK_CHUNKS, CALLBACK_DELAY, FAIL_RATE. The rendered config opts into `general_settings.forward_client_headers_to_llm_api: true` only when --with-mock is set so X-Mock-* headers reach the upstream. * Callback sinks: Langfuse `/api/public/ingestion` + GenericAPILogger `/api/hooks/*`, both honouring CALLBACK_DELAY for slow-consumer queue-retention testing. * Introspection: `GET /__mock__/state` returns a JSON snapshot of all counters (requests by provider/mode/status, chunks emitted, in-flight, errors injected). `GET /__mock__/reset` zeroes them. `GET /metrics` exposes the same counters in Prometheus text format so test code can scrape both proxy and mock sides for end-to-end reconciliation. * Request validation: rejects malformed bodies with provider-shaped 400 errors (empty messages list, missing role, etc.) — needed for case 21-style 4xx propagation tests. * Thread-local RNG (eliminates lock contention + nondeterministic output across ThreadingHTTPServer worker threads). * Monotonic-paced TTFT/TPS — avoids drift from cumulative time.sleep error in long streaming windows (~5% drift over 50 seconds with naive sleep, ~0% with monotonic scheduling). * Compact JSON encoding for SSE chunks (`json.dumps(..., separators= (',',':'))`) — matches real provider wire format so test regexes that look for `"type":"message_delta"` (no whitespace) hit. New runner flag: `e2e/tools/run-all-cases --mock-only` skips Tier=real cases (1, 2, 3, 5, 8, 9, 19 — prompt-caching token assertions, thinking-signature blocks, label-coverage paths that depend on real provider semantics) and runs the other 14 against the mock. Zero provider cost; ~30 s end-to-end. New case: `e2e/cases/23_mock_memory_pressure.md` documents how to reproduce the production 12 GB OOM math: - 5 × 40 MB concurrent body → peak Δ +900 MB / worker (4.5× body amp) - `num_retries=2` × 30% upstream 503 → peak +1.5 GB / worker - GenericAPILogger StandardLoggingPayload size: 0.4–3.7 MB / event The case is the canonical reproducer for the analysis that drove the production rollout of `MALLOC_ARENA_MAX=2` and body-size triage. Updates `cases/README.md` with a Tier column (mock / real / both) on every case and a "Mock-only mode" section documenting the env overlay (`ANTHROPIC_API_BASE` / `OPENAI_API_BASE` / `GEMINI_API_BASE` → `http://mock:8080`) needed before `--mock-only` runs. No `litellm/` source changes. `mock_provider.py` is stdlib-only and mounted into a `python:3.13-slim` sidecar — no install step, no provider key, no network egress. Test plan: * `e2e/tools/proxy start --with-mock` boots cleanly, `/v1/models` lists mock-openai / mock-anthropic / mock-gemini alongside the existing real-provider deployments. * `e2e/tools/proxy start` (no flag) is unchanged — mock service does not start; rendered config has no mock model_list entries. * `e2e/tools/run-all-cases --mock-only` against the mock-redirected env: 14 PASS, 7 SKIP (Tier=real), 0 FAIL. * fix(e2e): mock-only preflight, case 23 fixture, mock providers in tools/call (#36) Follow-up to #34. Three best-practice gaps the original PR documented but didn't actually implement: 1. `--mock-only` claimed to "auto-start the proxy with --with-mock and a mock-redirected .env overlay" but the code only skipped Tier=real cases — the env setup was the caller's job. Now the runner preflights three things before running anything: a. proxy reachable at $PROXY_URL, b. litellm-e2e-mock sidecar container running, c. ANTHROPIC_API_BASE inside the proxy container actually points at http://mock:8080. Any failure prints the exact fix commands (with the .env block to paste) and exits 2. README updated to match real behaviour. 2. Case 23 (mock memory pressure) was a Markdown-only runbook with no executable fixture, so `run-all-cases --mock-only` silently skipped it. Added e2e/cases/data/23_mock_memory_pressure.sh — a CI-light version (5 concurrent 200 KB streams, ~10 s) that asserts the two regressions the production OOM investigation identified as load-bearing: a. exactly N upstream calls per N client requests via the mock's /__mock__/state endpoint (regression test for retry amplification — the doc'd default num_retries=0 must stay 0), b. post-burst residual RSS bounded by 80 MB above baseline (catches a real Python-level retention regression). Peak RSS magnitude is reported in the PASS line for runbook context but not asserted — timing-dependent on slow CI hosts. Wired into run-all-cases, fires only under --mock-only. 3. `e2e/tools/call --provider` was hardcoded to the real-provider model aliases (anthropic / anthropic-haiku / openai), so writing new mock-driven cases required raw curl. Added mock-anthropic, mock-openai, mock-gemini choices that map to the model_list entries proxy renders under --with-mock. The Anthropic-shape cache_control transform path now extends to mock-anthropic so callers can exercise that codepath against the deterministic mock (the mock accepts the marker, doesn't account for it). After these fixes `e2e/tools/run-all-cases --mock-only` is: Total: 22 Pass: 15 Fail: 0 Skip: 7 (was 14/0/7 in #34 because case 23's fixture was missing).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--with-mock. Zero provider cost, deterministic, in-network only.--mock-onlytoe2e/tools/run-all-cases: skips 7 Tier=real cases (prompt-caching tokens, thinking-signature blocks, label coverage) and runs 14 cases against the mock in ~30 s.23_mock_memory_pressure.md) as the canonical reproducer for the recent production 12 GB OOM analysis — 5 × 40 MB concurrent bodies peak at +900 MB / worker.What the mock supports
POST /v1/chat/completionsPOST /v1/messagesmessage_start…message_stop)POST /v1beta/models/<m>:{streamGenerateContent,generateContent}POST /api/public/ingestionPOST /api/hooks/*GET /__mock__/stateGET /__mock__/resetGET /metricsPer-request controls go in HTTP headers (so they survive litellm's request transforms on Anthropic/Gemini paths):
Why now
The recent production 12 GB OOM investigation needed a way to A/B test allocator behaviour, retry amplification, and callback queue retention against deterministic streaming traffic. Real-provider e2e cases cost money and bring network noise. The mock makes that work safe to run in CI as
--mock-only, and case 23 makes the OOM math reproducible.What does NOT change
litellm/.e2e/tools/proxy start(no flag) is unchanged — mock service hasprofiles: [mock]and only boots with--with-mock.Test plan
e2e/tools/proxy start --with-mockboots cleanly;/v1/modelslistsmock-openai,mock-anthropic,mock-geminialongside the existing real-provider deployments.e2e/tools/proxy start(no flag) — mock service does not start, rendered config has no mock entries.e2e/tools/run-all-cases --mock-only: 14 PASS, 7 SKIP (Tier=real), 0 FAIL.black --check e2e/_config/mock_provider.pyclean.python3 -c "import ast; ast.parse(open('e2e/_config/mock_provider.py').read())"clean.