feat(v1.87.0): port in-network mock provider (Wave 2.5) - #50
Merged
Conversation
…#34) Adds a stdlib-only mock OpenAI+Anthropic+Gemini provider container that runs alongside the litellm e2e proxy under a docker-compose profile (`--profile mock`). Reachable via `e2e/tools/proxy start --with-mock`. Three model_list entries are auto-rendered when the flag is set: - mock-openai -> openai/mock-model - mock-anthropic -> anthropic/mock-claude - mock-gemini -> gemini/mock-gemini-pro Capabilities (see e2e/_config/mock_provider.py header for full contract): * Streaming + non-streaming for all three provider shapes, including the Anthropic SSE event sequence (message_start, content_block_*, message_delta, message_stop) and Gemini's `/v1beta/models/<m>:streamGenerateContent`. * Per-request controls via X-Mock-* HTTP headers — these survive litellm transforms (request body fields don't on Anthropic/Gemini paths, since those transforms strip unknown JSON fields): X-Mock-Chunks, X-Mock-Chunk-Chars, X-Mock-Full-Chars X-Mock-TTFT-Ms, X-Mock-TPS X-Mock-Fail (force 503|429|401|400|500, provider-shaped envelope) X-Mock-Tool-Call (`name(k1=v1,k2=v2)` emits tool_calls / tool_use / functionCall) * Process-level defaults via env: MOCK_TTFT_MS, MOCK_TPS, MOCK_CHUNKS, CALLBACK_DELAY, FAIL_RATE. The rendered config opts into `general_settings.forward_client_headers_to_llm_api: true` only when --with-mock is set so X-Mock-* headers reach the upstream. * Callback sinks: Langfuse `/api/public/ingestion` + GenericAPILogger `/api/hooks/*`, both honouring CALLBACK_DELAY for slow-consumer queue-retention testing. * Introspection: `GET /__mock__/state` returns a JSON snapshot of all counters (requests by provider/mode/status, chunks emitted, in-flight, errors injected). `GET /__mock__/reset` zeroes them. `GET /metrics` exposes the same counters in Prometheus text format so test code can scrape both proxy and mock sides for end-to-end reconciliation. * Request validation: rejects malformed bodies with provider-shaped 400 errors (empty messages list, missing role, etc.) — needed for case 21-style 4xx propagation tests. * Thread-local RNG (eliminates lock contention + nondeterministic output across ThreadingHTTPServer worker threads). * Monotonic-paced TTFT/TPS — avoids drift from cumulative time.sleep error in long streaming windows (~5% drift over 50 seconds with naive sleep, ~0% with monotonic scheduling). * Compact JSON encoding for SSE chunks (`json.dumps(..., separators= (',',':'))`) — matches real provider wire format so test regexes that look for `"type":"message_delta"` (no whitespace) hit. New runner flag: `e2e/tools/run-all-cases --mock-only` skips Tier=real cases (1, 2, 3, 5, 8, 9, 19 — prompt-caching token assertions, thinking-signature blocks, label-coverage paths that depend on real provider semantics) and runs the other 14 against the mock. Zero provider cost; ~30 s end-to-end. New case: `e2e/cases/23_mock_memory_pressure.md` documents how to reproduce the production 12 GB OOM math: - 5 × 40 MB concurrent body → peak Δ +900 MB / worker (4.5× body amp) - `num_retries=2` × 30% upstream 503 → peak +1.5 GB / worker - GenericAPILogger StandardLoggingPayload size: 0.4–3.7 MB / event The case is the canonical reproducer for the analysis that drove the production rollout of `MALLOC_ARENA_MAX=2` and body-size triage. Updates `cases/README.md` with a Tier column (mock / real / both) on every case and a "Mock-only mode" section documenting the env overlay (`ANTHROPIC_API_BASE` / `OPENAI_API_BASE` / `GEMINI_API_BASE` → `http://mock:8080`) needed before `--mock-only` runs. No `litellm/` source changes. `mock_provider.py` is stdlib-only and mounted into a `python:3.13-slim` sidecar — no install step, no provider key, no network egress. Test plan: * `e2e/tools/proxy start --with-mock` boots cleanly, `/v1/models` lists mock-openai / mock-anthropic / mock-gemini alongside the existing real-provider deployments. * `e2e/tools/proxy start` (no flag) is unchanged — mock service does not start; rendered config has no mock model_list entries. * `e2e/tools/run-all-cases --mock-only` against the mock-redirected env: 14 PASS, 7 SKIP (Tier=real), 0 FAIL.
…ls/call (#36) Follow-up to #34. Three best-practice gaps the original PR documented but didn't actually implement: 1. `--mock-only` claimed to "auto-start the proxy with --with-mock and a mock-redirected .env overlay" but the code only skipped Tier=real cases — the env setup was the caller's job. Now the runner preflights three things before running anything: a. proxy reachable at $PROXY_URL, b. litellm-e2e-mock sidecar container running, c. ANTHROPIC_API_BASE inside the proxy container actually points at http://mock:8080. Any failure prints the exact fix commands (with the .env block to paste) and exits 2. README updated to match real behaviour. 2. Case 23 (mock memory pressure) was a Markdown-only runbook with no executable fixture, so `run-all-cases --mock-only` silently skipped it. Added e2e/cases/data/23_mock_memory_pressure.sh — a CI-light version (5 concurrent 200 KB streams, ~10 s) that asserts the two regressions the production OOM investigation identified as load-bearing: a. exactly N upstream calls per N client requests via the mock's /__mock__/state endpoint (regression test for retry amplification — the doc'd default num_retries=0 must stay 0), b. post-burst residual RSS bounded by 80 MB above baseline (catches a real Python-level retention regression). Peak RSS magnitude is reported in the PASS line for runbook context but not asserted — timing-dependent on slow CI hosts. Wired into run-all-cases, fires only under --mock-only. 3. `e2e/tools/call --provider` was hardcoded to the real-provider model aliases (anthropic / anthropic-haiku / openai), so writing new mock-driven cases required raw curl. Added mock-anthropic, mock-openai, mock-gemini choices that map to the model_list entries proxy renders under --with-mock. The Anthropic-shape cache_control transform path now extends to mock-anthropic so callers can exercise that codepath against the deterministic mock (the mock accepts the marker, doesn't account for it). After these fixes `e2e/tools/run-all-cases --mock-only` is: Total: 22 Pass: 15 Fail: 0 Skip: 7 (was 14/0/7 in #34 because case 23's fixture was missing).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tier classification
litellm_extras/only)litellm/corelitellm/coreIf Tier C or D, did you try upstream first?
Summary
Deferred from Wave 2 (#47). Now that cases 14/16/18/22 are ported,
the mock provider's index/runner edits resolve cleanly. Adds the
in-network mock provider used by
--mock-onlymode and case 23(memory pressure regression).
Cherry-picks (2 commits)
60b40e610f1e57b8cd3dWhat ships
e2e/_config/mock_provider.py(1373 lines) — FastAPI mock thatspeaks OpenAI / Anthropic / Gemini envelopes.
e2e/_config/docker-compose.yml—mockservice under--profile mock.e2e/.env.example—MOCK_TTFT_MS,MOCK_TPS,MOCK_CHUNKSenv vars.e2e/cases/23_mock_memory_pressure.md+ data fixture — reproducesthe prod 12 GB OOM math (peak Δ +900 MB for 5×40 MB concurrent;
+1.5 GB with
num_retries:2+ 30% 503).e2e/tools/run-all-cases—--mock-onlyflag,REAL_ONLY_CASESlist, preflight,
case_23()definition + invocation.e2e/tools/proxy—--with-mockflag.e2e/tools/call— mock-aware behavior.e2e/cases/README.md— Tier column added; case 23 listed.Conflict resolutions
e2e/cases/README.md: adopted the new Tier-column structure fromupstream-of-cherry-pick. Kept ONLY entries for cases we've actually
ported (01-12, 14, 16, 18, 22, 23). Deferred case rows (13, 15, 17,
19, 20, 21) are NOT in the index — they'll be added back as their
feature waves merge.
e2e/tools/run-all-cases(713dfd859e): same — keptcase_23()function definition + invocation, dropped
case_19/20/21(deferred).e2e/tools/proxy: clean auto-merge.Note on deferred cases
After this PR, the runner invokes:
_dispatch/case_*for: 01-12, 13, 14, 16, 18, 22, 23case_13is still invoked but its data fixture (13_passthrough_streaming_ttft.sh)won't land until Wave 6 (passthrough TTFT impl). Running
case_13before then will FAIL noisily — this is pre-existing behavior from
Wave 2 (
07e65a92fdsticky-routing commit), not introduced here.The
--mock-onlyflag works end-to-end on this branch for the casesthat are present (Tier=mock|both excluding 13).
Verification
```bash
Spin up mock
e2e/tools/proxy start --with-mock
Run mock-only suite
e2e/tools/run-all-cases --mock-only
Expected: PASS for 04, 06, 07, 10, 11, 12, 14, 16, 18, 22, 23
(case 13 attempted; FAILs because data fixture missing — expected)
```
Pre-Submission checklist
litellm/core code touched.--with-mock/--mock-only.Type
🚄 Infrastructure / ✅ Test