feat(e2e): record and replay streamed provider responses chunk-for-chunk

[mateo-berri]

TLDR

Problem this solves:

• Record/replay stored a streamed response as one buffered body
• The /v1/messages streaming test couldn't be replayed offline

How it solves it:

• Record keeps each SSE transfer chunk in order, split points intact
• Replay reproduces the same chunks; a mid-stream error replays too

User Flow

Before: a developer trying to certify the streamed /v1/messages path offline can't, because a streamed response is stored as one coalesced body and the streaming test always calls the provider live

1. They run the messages suite in record mode (E2E_FIXTURE_MODE=record), which does POST /model/new to register anthropic/claude-haiku-4-5, then POST /v1/messages with "stream": true
2. The non-streaming cases record and later replay, but the streaming case's deployment points straight at https://api.anthropic.com, so it spends real money in every mode and its response never lands in the recorded fixture
3. Even a streamed body that did land would be saved as a single blob, so replay would hand the proxy one coalesced chunk instead of the provider's own SSE frames
4. Re-running the suite in replay mode (E2E_FIXTURE_MODE=replay) with bogus provider credentials fails: the streaming case still reaches https://api.anthropic.com and comes back 401

After: the same developer records once against the real provider, then replays offline with bogus credentials and the streamed chunk boundaries reproduce exactly

1. They run the messages suite in record mode: POST /model/new now registers the deployment against the local edge, POST /v1/messages with "stream": true streams back, and each SSE transfer chunk is recorded in the order the provider sent it
2. The recorded fixture for the streaming case stores the response as a stream (kind: "streamed") with more than one chunk, the provider's real split points, not one coalesced body
3. They point the proxy at a bogus ANTHROPIC_API_KEY and run replay mode: POST /v1/messages is answered from the fixture, replaying the same chunk sequence, so the client sees at least two incremental content_block_delta events, the usage event between the last delta and message_stop, and the call spends nothing and never leaves the process
4. A recorded mid-stream upstream error replays as the same mid-stream error, delivering the chunks that arrived and then cutting off, rather than a clean buffered body

Relevant issues

Linear ticket

Resolves LIT-5742

Pre-Submission checklist

• I have added meaningful tests
• The handful of test files covering my change pass locally
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Shared setup: a proxy booted from this branch on port 4823 with general_settings.store_model_in_db: true, pointed at a native Postgres (dedicated QA DB) and Redis. The one test under proof registers an anthropic/claude-haiku-4-5 deployment via POST /model/new and streams POST /v1/messages.

Before (d0da90e)

The streamed /v1/messages path can't be recorded or replayed: the harness has no streamed bundle shape, and the streaming test stays on the live provider in every mode.

1. git show d0da90ee6d:tests/e2e/fixture_bundle.py | grep BUNDLE_FORMAT_VERSION prints BUNDLE_FORMAT_VERSION: Final = 3, and the recorded-response model there is a single buffered body with no chunk-preserving shape, so a streamed response would be stored coalesced

2. git show d0da90ee6d:tests/e2e/llm_translation/test_messages_e2e.py shows test_messages_streams_completion registering its deployment with no api_base, and its own docstring reads "Stays on a live Anthropic deployment in every mode: the edge buffers a streamed response into one body, so chunk fidelity waits on LIT-5742." The assertions only check that some content_block_delta and a message_stop appear, not that the deltas stayed incremental

3. git show d0da90ee6d:tests/e2e/CLAUDE.md | grep -n 5742 prints the standing limit: "streaming chunk fidelity is LIT-5742 (a streamed response records as one buffered body)." The After run's bundle is format_version 4, which this version-3 loader refuses outright, so there is no offline streamed replay to capture before this change

After (b2f7216)

Records each chunk in order, then replays offline with bogus provider credentials and zero provider calls.

1. RECORD leg, proxy env holding the real ANTHROPIC_API_KEY:

   LITELLM_PROXY_URL=http://localhost:4737 E2E_FIXTURE_MODE=record E2E_FIXTURE_DIR=/tmp/e2e-fixtures-5742-v2 \
     uv run pytest tests/e2e/llm_translation/test_messages_e2e.py::TestAnthropicMessages::test_messages_streams_completion -v
   

   ============================== 1 passed in 16.86s ==============================

2. Inspect the recorded interaction /tmp/e2e-fixtures-5742-v2/test_messages_streams_completion-09c84255/0000-post-anthropic-v1-messages.json: "kind": "streamed", status_code 200, content-type: text/event-stream; charset=utf-8, chunks_b64 holds 5 entries (the provider's own split points, not one coalesced body), truncated: null. manifest.json format_version is 4

3. REPLAY leg, proxy restarted with a bogus ANTHROPIC_API_KEY=sk-ant-bogus-xxxx, everything else identical:

   LITELLM_PROXY_URL=http://localhost:4737 E2E_FIXTURE_MODE=replay E2E_FIXTURE_DIR=/tmp/e2e-fixtures-5742-v2 \
     uv run pytest tests/e2e/llm_translation/test_messages_e2e.py::TestAnthropicMessages::test_messages_streams_completion -v
   

   ============================== 1 passed in 15.40s ==============================

4. The replay proxy log shows the inbound POST /v1/messages returned 200 with 0 mentions of api.anthropic.com and 0 auth errors, and the deployment's stored api_base was the local edge (http://127.0.0.1:50526/anthropic), not the real provider. The edge served the recorded chunks, so the call spent nothing and never left the process, even with an invalid Anthropic key

Type

🆕 New Feature
✅ Test

Caveats (if any)

• Publishing recorded bundles for CI is still LIT-5748

QA runbook

• tests/e2e/llm_translation/test_messages_e2e.py::TestAnthropicMessages::test_messages_streams_completion - a streamed /v1/messages response records chunk-by-chunk and replays with the same split points, staying incremental rather than coalescing
  • Bring up a proxy with store_model_in_db: true and a Postgres/Redis it can reach, plus a real ANTHROPIC_API_KEY
  • Record: E2E_FIXTURE_MODE=record E2E_FIXTURE_DIR=/tmp/e2e-fixtures uv run pytest tests/e2e/llm_translation/test_messages_e2e.py::TestAnthropicMessages::test_messages_streams_completion -v
  • Inspect the recorded interaction under /tmp/e2e-fixtures/<slug>/: expect "kind": "streamed" with more than one entry in chunks_b64 and truncated null
  • Restart the proxy with a bogus ANTHROPIC_API_KEY, then replay: E2E_FIXTURE_MODE=replay E2E_FIXTURE_DIR=/tmp/e2e-fixtures uv run pytest ...::test_messages_streams_completion -v and expect it to pass with zero provider calls
  • Sanity check: the test asserts on incremental deltas (>=2 content_block_delta and usage between the last delta and message_stop), so a replay that coalesced the body would fail it, not pass hand-wavily

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

• b2f7216 passes /live-pr-risk

---

Note

Low Risk
Scope is limited to e2e fixture format, provider-edge relay logic, and tests; no production gateway or auth paths are modified.

Overview
E2e record/replay no longer collapses text/event-stream provider responses into one buffered body. Fixture bundles bump to format version 4 with a discriminated kind: streamed shape (chunks_b64 plus optional truncation metadata); older bundles are rejected on load instead of partially replayed.

The provider edge relays SSE upstream one HTTP transfer chunk at a time in record and replay (forward_stream in e2e_http.py, chunked EdgeStream writes in provider_edge.py), so the proxy sees the same split points—including mid-event splits and mid-stream disconnects—as a live provider call.

The Anthropic /v1/messages streaming e2e is edge-wired for record/replay, with stricter assertions on incremental content_block_delta events and usage ordering. Harness unit tests pin transfer-layer fidelity, bundle round-trips, and format-version gates; docs drop the LIT-5742 streaming limitation.

^{Reviewed by Cursor Bugbot for commit b2f7216. Bugbot is set up for automated code reviews on this repo. Configure here.}

[greptile-apps]

Greptile Summary

The PR extends the E2E provider-edge fixture format to preserve streamed SSE transfer chunks and replay upstream truncation.

• Adds streamed response recording and replay with chunk boundaries intact.
• Moves the Anthropic messages streaming test onto the provider edge and strengthens incremental-event assertions.
• Rejects fixture bundles written with incompatible format versions.
• Updates provider-edge and fixture-bundle tests and contributor documentation.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
tests/e2e/e2e_http.py	Adds lazy upstream transfer-chunk iteration and represents mid-stream transport failures as terminal truncation steps.
tests/e2e/fixture_bundle.py	Introduces the version-4 discriminated streamed-response format and enforces manifest compatibility during loading.
tests/e2e/provider_edge.py	Records and replays SSE responses chunk by chunk; the previous delivery-order issue is fixed, while the terminator concern was clarified as outside the capture-completeness contract.
tests/e2e/llm_translation/test_messages_e2e.py	Edge-wires the Anthropic streaming test and verifies incremental deltas, usage ordering, and clean completion.
tests/e2e/test_provider_edge.py	Adds transfer-level coverage for streamed recording, replay, truncation, downstream closure, and ordinary chunked JSON buffering.
tests/e2e/test_fixture_bundle.py	Covers streamed-response serialization and rejection of foreign bundle versions.

_{Reviews (3): Last reviewed commit: "fix(e2e): record a streamed chunk only a..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit b2f7216. Configure here.}

[tin-berri]

Approving. chunk_size=None is the whole trick and the docstring says so — one piece per wire chunk is what makes a recording reproduce the provider's split points instead of re-slicing them, and a recording that coalesces the body can't tell you anything about streaming behaviour. Good closure on LIT-5742.

Things I liked:

• StreamTruncation as a terminal step rather than a raised exception. A stream that delivered six chunks and then died is a different thing from a request that never streamed, and modelling it as data is what preserves that difference into the bundle.
• The truncated reason carrying which side ended it (upstream: / downstream:), even though replay treats both identically. That's for the human reading the bundle, and it's the sort of thing that gets dropped and then wished for.
• StreamHead as a dataclass rather than a BaseModel with the reason given (it owns a live socket, steps is consumed once). Worth the explanation.
• Making load_bundle go through _supported_manifest too. Previously only check_freshness checked the version, so load_bundle would happily read a foreign-version bundle and then miss on every call — this is a quiet bug fix riding along, not just plumbing for the bump.

Two things:

1. The v3 → v4 bump invalidates every existing recording. That's the correct call given the stored shape changed, and the error message pointing at E2E_FIXTURE_MODE=record is the right way to land it. Just confirm nothing committed is still on v3 — if any bundle is checked in rather than recorded locally, it needs the re-record in this PR, or replay is hard-red on day one for whoever runs it next.

2. _stream_steps only closes on exhaustion or GC. The finally: resp.close() fires when the generator completes or is closed, so a consumer that abandons steps part way through (a test failing mid-replay, an early break) leaves the response open until collection. Fine at e2e scale and the docstring warns that closing it closes the response — mentioning it so it's a known property rather than a surprise if the edge ever holds many concurrent streams.

code-quality is the recursive_detector red on llm_request_utils.py, which is base drift and gets fixed by #38149.