Extend the record/replay proxy to chat, embeddings, moderations, rerank, and Anthropic

[mateo-berri]

Relevant issues

Follows the gpt-image-1 record/replay work in #29802; generalizes that proxy so it covers the other real-provider E2Es.

Linear ticket

Pre-Submission checklist

• I have added meaningful tests
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

What this does

The record/replay proxy (tests/_openai_record_replay_proxy.py) that took the gpt-image-1 spend E2E off the live OpenAI path now fronts every provider, so the other real-provider E2Es stop paying for and depending on live calls on every commit.

One recorder process fronts all upstreams. The default upstream stays api.openai.com; a non-OpenAI model selects its provider with a /__recorder_upstream/<host>/ path prefix carried on the model's api_base. Routing rides api_base rather than a custom request header because some litellm handlers (cohere rerank) drop unknown headers, so a header-based scheme replays inconsistently across providers. The upstream is folded into the cache key so two providers that share a path can't collide.

Wired in via a reusable start_openai_record_replay_proxy CircleCI command:

• build_and_test (proxy_server_config.yaml): chat (gpt-4), embeddings (text-embedding-ada-002), a new text-moderation-stable model, and the existing image model, all pointed at the recorder for OpenAI
• proxy_logging_guardrails_model_info_tests (otel, otel_test_config.yaml): Cohere rerank, via the /__recorder_upstream/api.cohere.com prefix
• proxy_e2e_anthropic_messages_tests (test_config.yaml): the real-Anthropic claude-sonnet-4-5-20250929 model, via /__recorder_upstream/api.anthropic.com

The api_base values resolve from os.environ/RECORDER_*_BASE_URL, which is only set inside the CI containers; unset everywhere else they fall back to None and the proxy talks to the real provider, so local dev and prod are unaffected.

Cache-buster removal and the response-cache check

The build_and_test chat tests injected time.time()/uuid into the prompt, which makes the body unique and so unrecordable. proxy_server_config.yaml has the proxy response cache off (the cache/cache_params block is commented out), so those busters were defeating only the recorder, not a live cache; I dropped them. To keep that assumption honest, test_key_info_spend_values_image_generation now makes an identical second call: the recorder serves it for free, but the proxy must still bill it, so the key's spend has to grow. If the response cache is ever switched on, the repeat becomes a $0 cache hit and that assertion fails loudly instead of silently zeroing recorded-call spend.

The otel config deliberately keeps its cache: true on; I did not disable it, since that path is what exercises cache-hit cost and metrics. The recorder simply sits behind the proxy cache there with a deterministic rerank body, so it still prevents live Cohere calls across runs without changing what the test exercises.

What is intentionally left live

• /responses (e2e_openai_endpoints): the only real-OpenAI test there runs create -> retrieve(id) -> delete(id) -> retrieve(id)-expects-404, and a body-keyed cache would replay the pre-delete 200 on the second retrieve (even within one run) and fail it; the rest of that job is batches/files/fine-tuning, which are stateful
• the anthropic passthrough test (pass_through job): it asserts api_base == "https://api.anthropic.com/v1/messages" in the spend logs, which routing through the recorder would break, and the rest is streaming/bedrock
• bedrock and the fake-railway-endpoint configs: different upstream, or no real call to cache

These are the same class as the stateful endpoints a stateless body-keyed cache can't serve.

Type

🧪 Test / 🚄 Infrastructure

Changes

• tests/_openai_record_replay_proxy.py: per-request upstream resolution via the /__recorder_upstream/<host>/ prefix, upstream folded into the cache key, docstring
• tests/llm_translation/test_openai_record_replay_proxy.py: unit tests for upstream resolution, per-upstream key separation, prefix routing, and default fallback
• proxy_server_config.yaml, otel_test_config.yaml, test_config.yaml: point the relevant models at the recorder; rename IMAGE_GEN_RECORDER_BASE_URL -> RECORDER_OPENAI_BASE_URL
• tests/test_keys.py, tests/test_openai_endpoints.py: drop the prompt cache-busters, point the moderation request at a model, add the repeat-call spend observability
• .circleci/config.yml: reusable start_openai_record_replay_proxy command, wired into three jobs with the per-provider recorder base URLs

Screenshots / Proof of Fix

Local run: the recorder on :8091 (real cassette redis), the litellm proxy on :4001 with the relevant models' api_base pointed at the recorder, then each endpoint curled twice. Round 1 records live, round 2 replays; every call is 200 and hits real OpenAI, Cohere, and Anthropic (real spend).

=== round 1 ===
  chat=200
  embed=200
  rerank=200
  anthropic=200
=== round 2 ===
  chat=200
  embed=200
  rerank=200
  anthropic=200
--- recorder HIT/MISS log ---
MISS forwarded live and recorded: POST /v1/chat/completions
MISS forwarded live and recorded: POST /v1/embeddings
MISS forwarded live and recorded: POST /v2/rerank
MISS forwarded live and recorded: POST /v1/messages
HIT replayed from cassette: POST /v1/chat/completions
HIT replayed from cassette: POST /v1/embeddings
HIT replayed from cassette: POST /v2/rerank
HIT replayed from cassette: POST /v1/messages

Moderations routes through the recorder the same way once the request names a recorder-backed model:

moderations WITH model:
  round1=200
  round2=200
MISS forwarded live and recorded: POST /v1/moderations
HIT replayed from cassette: POST /v1/moderations

---

Note

Low Risk
Changes are confined to test infrastructure, CI config, and example proxy YAML; production routing is unchanged when recorder env vars are unset.

Overview
Generalizes the CI record/replay proxy so dockerized E2Es can replay OpenAI, Cohere, and Anthropic traffic from Redis cassettes instead of hitting live APIs every commit.

The proxy now resolves per-request upstreams: default api.openai.com, or https://<host> when the model’s api_base uses the /__recorder_upstream/<host>/ path prefix. Cache keys include upstream so identical paths to different providers cannot collide.

CircleCI adds reusable start_openai_record_replay_proxy and wires it into build_and_test, OTEL/guardrails, and Anthropic messages jobs with RECORDER_OPENAI_BASE_URL, RECORDER_COHERE_BASE_URL, and RECORDER_ANTHROPIC_BASE_URL (replacing IMAGE_GEN_RECORDER_BASE_URL).

Proxy test configs point chat, embeddings, image, moderation, Cohere rerank, and Anthropic models at those env-backed bases (unset outside CI → direct to providers).

E2E tweaks: remove prompt time/uuid cache-busters so bodies are recordable; moderation tests target text-moderation-stable; image spend test repeats an identical call and asserts spend still increases (guards against proxy response cache zeroing billed spend on replays).

Unit tests cover upstream resolution, multi-provider keys, and HTTP routing/auth passthrough.

^{Reviewed by Cursor Bugbot for commit 8ed5203. Bugbot is set up for automated code reviews on this repo. Configure here.}

[mateo-berri]

@greptileai

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR generalizes the existing gpt-image-1 record/replay proxy so it fronts OpenAI chat, embeddings, moderation, Cohere rerank, and Anthropic messages — eliminating live provider calls (and their cost and flakiness) across several more CI jobs. Routing between providers uses a /__recorder_upstream/<host>/ path prefix carried on api_base, with the upstream folded into the SHA-256 cache key to prevent cross-provider collisions.

• Core proxy (tests/_openai_record_replay_proxy.py): new _resolve_upstream function strips the path prefix and selects the correct provider; record_key and handle gain an upstream_base_url parameter so recordings are namespaced per provider.
• CI (.circleci/config.yml): a reusable start_openai_record_replay_proxy command wires the proxy into three jobs; per-provider recorder base URLs are passed as env vars; IMAGE_GEN_RECORDER_BASE_URL is renamed RECORDER_OPENAI_BASE_URL consistently.
• Test hardening (tests/test_keys.py, tests/test_openai_endpoints.py): deterministic prompts replace uuid/time cache-busters (enabling recording), and a second identical image call with a spend-growth assertion guards against the proxy response cache being silently switched on.

Confidence Score: 5/5

Safe to merge — all changes are confined to the test/CI layer with no production code paths touched.

The upstream resolution, cache key namespacing, and proxy routing logic are all correct and covered by new unit tests using fakeredis and mock HTTP clients. Modifications to existing tests are intentional and compensated by stronger assertions. Config changes are non-breaking (env vars fall back to None, preserving live-provider behaviour outside CI).

No files require special attention.

Important Files Changed

Filename	Overview
tests/_openai_record_replay_proxy.py	Core change: adds _resolve_upstream for per-request provider routing via path prefix, folds upstream into the SHA-256 cache key to prevent cross-provider collisions, and updates handle/record_key signatures accordingly. Logic is correct and well-tested.
tests/llm_translation/test_openai_record_replay_proxy.py	Adds unit tests for upstream resolution, per-upstream key separation, prefix routing, and default fallback; all use fakeredis/mock clients with no real network calls, consistent with rule 07fb23f7.
.circleci/config.yml	Extracts the record/replay proxy startup into a reusable start_openai_record_replay_proxy command; wires it into three CI jobs with per-provider base URL env vars; env var rename IMAGE_GEN_RECORDER_BASE_URL → RECORDER_OPENAI_BASE_URL is consistent across all usages.
tests/test_keys.py	Removes the time.time() prompt cache-buster (necessary for recording determinism) and adds a second identical image call with a spend-growth assertion to guard against the proxy response cache being silently enabled.
tests/test_openai_endpoints.py	Drops uuid cache-buster from chat_completion (enables recording), adds model field to the moderation request so it routes to the recorder-backed text-moderation-stable model, and removes now-unused litellm._uuid import.
proxy_server_config.yaml	Adds api_base: os.environ/RECORDER_OPENAI_BASE_URL to chat and embedding models; adds new text-moderation-stable model pointing at the recorder; renames old IMAGE_GEN_RECORDER_BASE_URL reference.
litellm/proxy/example_config_yaml/otel_test_config.yaml	Adds api_base: os.environ/RECORDER_COHERE_BASE_URL to the cohere-rerank model; unset locally so the config falls through to direct Cohere in dev.
tests/proxy_e2e_anthropic_messages_tests/test_config.yaml	Adds api_base: os.environ/RECORDER_ANTHROPIC_BASE_URL to route Anthropic messages through the recorder in CI; no-op when the env var is unset.

_{Reviews (3): Last reviewed commit: "docs(ci): note the recorder command's Op..." | Re-trigger Greptile}

[greptile-apps]

Greptile Summary

This PR generalises the record/replay proxy introduced for gpt-image-1 spend tests so it fronts every real-provider E2E: OpenAI chat, embeddings, moderations, Cohere rerank, and Anthropic messages. A single recorder process serves all providers; non-OpenAI providers are selected via a /__recorder_upstream/<host>/ path prefix carried on the model's api_base, with the upstream URL folded into the cache key to prevent cross-provider collisions.

• tests/_openai_record_replay_proxy.py gains _resolve_upstream for per-request provider routing and the upstream is now a parameter of record_key; new unit tests in tests/llm_translation/test_openai_record_replay_proxy.py cover prefix resolution, per-upstream key isolation, and app-level routing.
• proxy_server_config.yaml, otel_test_config.yaml, and test_config.yaml add api_base: os.environ/RECORDER_*_BASE_URL to the relevant models; env vars are only set in CI containers, so local and production paths are unaffected.
• test_keys.py and test_openai_endpoints.py drop UUID/timestamp cache-busters (which were defeating the recorder without exercising any real assertion) and add a repeat-call spend assertion to guard against the proxy's own response cache silently zeroing recorded-call spend.

Confidence Score: 4/5

Safe to merge; changes are confined to test infrastructure and CI configuration with no production code paths affected.

The proxy logic, routing, cache-key construction, and header sanitisation are all correct and well-covered by the new unit tests. The only advisory note is that the shared start_openai_record_replay_proxy command hardcodes the OpenAI default upstream for all three jobs, which works today because Cohere and Anthropic traffic uses prefix routing, but could silently misdirect requests if a future caller omits the prefix.

.circleci/config.yml — the shared recorder command's hardcoded default upstream is worth a second look when wiring in future providers.

Important Files Changed

Filename	Overview
tests/_openai_record_replay_proxy.py	Core record/replay proxy extended with per-request upstream resolution via /__recorder_upstream/<host>/ prefix and upstream folded into cache key to prevent cross-provider collisions; logic is correct and well-tested.
tests/llm_translation/test_openai_record_replay_proxy.py	New unit tests covering upstream resolution, per-upstream key separation, prefix routing, and default fallback; uses fakeredis throughout so no real network calls.
.circleci/config.yml	Inlines old inline recorder startup into a reusable start_openai_record_replay_proxy command and wires it into three jobs; the command hardcodes RECORDER_UPSTREAM_BASE_URL=https://api.openai.com for all three jobs even though two of them route non-OpenAI providers via prefix paths — functionally harmless but potentially surprising.
tests/test_keys.py	Removes unused time import, replaces time-based cache-buster in chat_completion_streaming, and adds a repeat-call spend assertion to test_key_info_spend_values_image_generation to guard against proxy response cache silently zeroing recorded spend.
tests/test_openai_endpoints.py	Removes uuid-based cache-buster from chat_completion, adds model field to moderation request to route through the recorder-backed text-moderation-stable model; assertions remain status-code-based so fixed prompts don't weaken coverage.
proxy_server_config.yaml	Renames IMAGE_GEN_RECORDER_BASE_URL → RECORDER_OPENAI_BASE_URL, adds the same env-var api_base to gpt-4 and embedding models, and introduces a new text-moderation-stable model backed by omni-moderation-latest.

_{Reviews (2): Last reviewed commit: "test(ci): extend record/replay proxy to ..." | Re-trigger Greptile}

[mateo-berri]

Documented the OpenAI default-upstream assumption on the shared start_openai_record_replay_proxy command in 8ed5203, since a non-OpenAI model relies on the /__recorder_upstream/<host>/ prefix to override it.

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 8ed5203. Configure here.}