test(ci): record/replay OpenAI image gen so the spend E2E isn't outage-bound

[mateo-berri]

Relevant issues

The dockerized spend test tests/test_keys.py::test_key_info_spend_values_image_generation curls the proxy for a gpt-image-1 image and asserts a nonzero tracked spend. In proxy_server_config.yaml gpt-image-1 has no explicit route, so it falls through the model_name: "*" wildcard to openai/* on the real OPENAI_API_KEY; every commit run hit api.openai.com for a paid image and depended on OpenAI being up. That is the same class of failure as the OpenAI 401 outage that reddened unrelated PRs.

Linear ticket

Pre-Submission checklist

• I have added meaningful tests
• My PR passes all unit tests
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

What this does

Adds an in-repo record/replay reverse proxy (tests/_openai_record_replay_proxy.py) that sits between the litellm proxy and api.openai.com. The proxy keeps its full real separate-process HTTP topology; the test still curls localhost:4000/v1/images/generations and the proxy still runs auth, routing, the openai provider, a real HTTP round-trip, and the whole spend-accounting -> Redis buffer -> Postgres pipeline unmocked. The only hop that changes is proxy -> "OpenAI": in CI the image model's api_base is pointed at the recorder via IMAGE_GEN_RECORDER_BASE_URL. The first run, and the first after the recording lapses, forwards live to OpenAI and records; subsequent runs replay the recorded response, so the per-commit run no longer depends on OpenAI's uptime and no longer pays per image.

IMAGE_GEN_RECORDER_BASE_URL is only set in the build_and_test CircleCI job. Unset everywhere else it resolves to None, so the shared config falls back to api.openai.com and local/dev behavior is unchanged.

Recordings live in the same Redis cassette store as the VCR persister (CASSETTE_REDIS_URL) and expire 24h after their last write, never refreshed on read, so a recording goes stale a day after capture and the next run re-records live and catches provider contract drift. This matches the lapse-after-write contract in tests/_vcr_redis_persister.py. If CASSETTE_REDIS_URL is not present in the job, the recorder degrades to a pure live passthrough (current behavior), so this is safe to land before the secret is wired.

Replayed responses drop upstream framing/server headers (content-length, transfer-encoding, content-encoding, date, server) so the re-serving layer recomputes them; replaying a stale content-length is exactly the Bedrock content-length: 0 regression, so that lesson is baked into a test here.

Why an in-repo recorder instead of mitmproxy

Same point in the topology, same record/replay behavior, but no third-party binary added to CI (no supply-chain pinning/checksums) and no TLS-MITM CA trust to install in the container. The image path builds an httpx.AsyncClient whose api_base is overridable per-model, so a plain-HTTP reverse recorder is enough; on a replay run neither approach exercises the literal TLS handshake to OpenAI anyway.

CI (LiteLLM team)

• Branch creation CI run
  Link:
• CI run for the last commit
  Link:
• Merge / cherry-pick CI run
  Links:

To actually cache across runs, CASSETTE_REDIS_URL needs to be present in the build_and_test job's context (same dedicated cassette Redis the VCR persister uses). Without it the job still works, just always live.

Screenshots / Proof of Fix

Local end-to-end, no real OpenAI needed to show record then replay. Run in order:

1. Start a stub that stands in for api.openai.com and counts hits, plus a Redis:

docker run -d --name rr-redis -p 6380:6379 redis:7
cat > /tmp/stub_openai.py <<'PY'
from starlette.applications import Starlette
from starlette.responses import JSONResponse
from starlette.routing import Route
hits = {"n": 0}
async def gen(r):
    hits["n"] += 1
    print("UPSTREAM HIT", hits["n"], flush=True)
    return JSONResponse({"data":[{"b64_json":"aW1n"}],"usage":{"total_tokens":7}},
                        headers={"transfer-encoding":"chunked","x-request-id":"req_1"})
app = Starlette(routes=[Route("/v1/images/generations", gen, methods=["POST"])])
PY
python -m uvicorn --app-dir /tmp stub_openai:app --host 127.0.0.1 --port 9099 &

2. Start the recorder pointed at the stub, backed by Redis:

CASSETTE_REDIS_URL=redis://127.0.0.1:6380/0 \
RECORDER_UPSTREAM_BASE_URL=http://127.0.0.1:9099 \
  python tests/_openai_record_replay_proxy.py --host 127.0.0.1 --port 8090 &

3. First call records (watch the stub log print one UPSTREAM HIT); second identical call replays from Redis with no new UPSTREAM HIT, and the returned headers show a single freshly computed content-length, no transfer-encoding:

for i in 1 2; do
  curl -s -D - -o /dev/null -X POST http://127.0.0.1:8090/v1/images/generations \
    -H 'content-type: application/json' \
    -d '{"model":"gpt-image-1","prompt":"A cute baby sea otter"}'
done

For the real run, set RECORDER_UPSTREAM_BASE_URL=https://api.openai.com, point a local proxy's gpt-image-1 api_base at http://127.0.0.1:8090/v1, then run test_key_info_spend_values_image_generation twice; the second run hits no OpenAI quota and still asserts spend > 0.

Type

✅ Test

Changes

• tests/_openai_record_replay_proxy.py: record-once/replay-from-Redis reverse proxy (DI-friendly core class plus a Starlette app and a __main__ runner)
• tests/llm_translation/test_openai_record_replay_proxy.py: unit tests for miss/hit, key canonicalization, 24h lapse-after-write (no refresh-on-read), framing-header stripping, non-2xx not cached, and Redis-outage passthrough
• .circleci/config.yml: start the recorder in build_and_test, wait for readiness, point the container's image model at it
• proxy_server_config.yaml: explicit gpt-image-1 route whose api_base is the env-injected recorder, falling back to OpenAI when unset

---

Generated by Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR inserts a thin record/replay reverse proxy between the litellm CI proxy container and api.openai.com so the image-gen spend E2E test no longer makes a paid live call on every commit. The first run forwards to OpenAI and stores the response in Redis with a 24-hour lapse-after-write TTL; subsequent runs within the window replay from the cassette.

• tests/_openai_record_replay_proxy.py implements the recorder as a standalone Starlette process with DI-friendly constructor and framing-header stripping (guarding against the Bedrock content-length: 0 regression class).
• tests/llm_translation/test_openai_record_replay_proxy.py covers miss/hit, TTL lapse-not-refresh, framing-header stripping, non-2xx not cached, Redis-outage passthrough, and injected-client lifecycle — all with fakeredis and no real network calls.
• .circleci/config.yml starts the recorder before the Docker container and injects IMAGE_GEN_RECORDER_BASE_URL; proxy_server_config.yaml adds an explicit gpt-image-1 route that reads that variable, falling back to api.openai.com when unset.

Confidence Score: 5/5

Safe to merge — changes are confined to test infrastructure and CI config with a clean opt-out when the cassette Redis secret is absent.

All four changed files are test/CI infrastructure with no impact on the production SDK or proxy request path. The recorder degrades to a transparent passthrough when CASSETTE_REDIS_URL is not set, so the change cannot break existing CI jobs that lack the secret. The core record/replay logic is fully unit-tested with fakeredis covering every branch (miss, hit, TTL contract, framing-header stripping, non-2xx not cached, Redis outage, and injected-client lifecycle). No production code is modified.

No files require special attention.

Important Files Changed

Filename	Overview
tests/_openai_record_replay_proxy.py	New record/replay proxy; correct cache key design (canonical JSON body, lapse-after-write TTL), proper framing-header stripping, injected redis/http client for testability, and lifespan correctly closes only self-owned clients.
tests/llm_translation/test_openai_record_replay_proxy.py	Comprehensive unit tests using fakeredis and stubs; covers all key behaviors with no real network calls, consistent with the folder's mock-only convention.
.circleci/config.yml	Adds proxy startup step with 30-second readiness poll and injects IMAGE_GEN_RECORDER_BASE_URL into the Docker container via host.docker.internal; degrades gracefully when CASSETTE_REDIS_URL is absent.
proxy_server_config.yaml	Adds explicit gpt-image-1 route with env-overridable api_base; includes inline comment explaining the None fallback path to api.openai.com.

_{Reviews (2): Last reviewed commit: "test(ci): close recorder http client on ..." | Re-trigger Greptile}

[greptile-apps]

Custom HTTP handler outside litellm's existing handler layer

create_app builds a standalone Starlette ASGI application that intercepts and proxies HTTP traffic. Per the team's convention, custom HTTP handlers should reuse litellm's existing request-handling infrastructure rather than be written from scratch. While this is test infrastructure with no litellm equivalent to reuse, it's worth confirming the team is comfortable with this pattern living in tests/ outside the proxy-tier boundary.

Rule Used: What: Block any PR which writes it's own custom ht... (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[mateo-berri]

This is intentional. The recorder is a test-only transparent reverse proxy that sits between the litellm container and api.openai.com; the traffic it records is litellm's own egress to OpenAI, so routing it back through litellm's request-handling layer would be circular. It has to behave as a byte-faithful HTTP intermediary (capture the raw upstream response, then replay it with framing headers stripped so the serving layer recomputes content-length and sets its own date/server), which is a different job than the provider-translation handlers. It lives entirely in tests/ and no production path imports it, so the proxy-tier boundary is intact. Safe to remember this as an allowed exception for test-only network shims

---

Generated by Claude Code

[greptile-apps]

Got it — the circular-dependency argument is sound and the tests/-only boundary is clear. I've saved this as an exception: test-only network shims (record/replay proxies, stubs, interceptors) confined to tests/ and never imported by production code are exempt from the custom HTTP handler rule.

View rule

[mateo-berri]

@greptileai