fix(proxy): forward Bedrock event-stream content-type on unbuffered passthrough - #33767
Conversation
…assthrough Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
Greptile SummaryThis PR fixes a missing
Confidence Score: 5/5
|
| Filename | Overview |
|---|---|
| litellm/proxy/common_request_processing.py | Adds media_type=self._passthrough_event_stream_media_type() to the unbuffered StreamingResponse branch, mirroring the already-existing call in the buffered Response branch. Change is minimal and correct. |
| tests/test_litellm/proxy/test_common_request_processing.py | Adds two new mock-only tests: one pinning the Bedrock unbuffered stream to application/vnd.amazon.eventstream, and one confirming non-Bedrock providers emit no content-type header. No real network calls; assertions are precise. |
Reviews (3): Last reviewed commit: "docs(proxy): pre-fix passthrough streams..." | Re-trigger Greptile
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
|
Before/after proof against two live proxies hitting real AWS Bedrock ( BEFORE (base branch, port 4001) No AFTER (this PR, port 4000) Body is still the raw AWS event-stream, unchanged by setting the header Session: https://app.devin.ai/sessions/835c506325e2454d80dc4434ae74f7b3 |
…itellm_lit_4561_bedrock_passthrough_content_type
|
❌ Cannot revive Devin session - the session is too old. Please start a new session instead. |
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit d5a4c14. Configure here.
TLDR
Problem this solves:
How it solves it:
User Flow
Before: a developer running Claude Code against the proxy's Bedrock passthrough sees every streaming turn rejected or silently doubled
CLAUDE_CODE_USE_BEDROCK=1andANTHROPIC_BEDROCK_BASE_URL=https://litellm-domain/bedrock, then ask Claude Code a questionAPI Error: Bedrock streaming response has content-type "application/octet-stream"; expected "application/vnd.amazon.eventstream", and versions past 2.1.227 instead silently refetch the same turn withstream: false, so every turn costs two full model invocationsAfter: the same turn streams once and renders normally
CLAUDE_CODE_USE_BEDROCK=1andANTHROPIC_BEDROCK_BASE_URL=https://litellm-domain/bedrock, then ask Claude Code a questioncontent-type: application/vnd.amazon.eventstreamRelevant issues
Linear ticket
Resolves LIT-4561
Resolves LIT-5674
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Two live proxies against real AWS Bedrock (
bedrock/us.anthropic.claude-opus-5, us-east-1, real spend), each booted from its own worktree and venv with its own fresh Postgres, model_list router deploymentus.anthropic.claude-opus-5(custom_llm_provider: bedrock), no mocks. Before runs the merge base f04b1da on port 53571, After runs this PR's tip d5a4c14 on port 33945. Same payloads both sides. The Claude Code case runs the realclaudeCLI 2.1.234 interactively under tmux withCLAUDE_CODE_USE_BEDROCK=1,ANTHROPIC_BEDROCK_BASE_URL=http://127.0.0.1:{port}/bedrock,CLAUDE_CODE_SKIP_BEDROCK_AUTH=1,ANTHROPIC_AUTH_TOKEN=$VKEY(a per-leg virtual key fromPOST /key/generate),ANTHROPIC_MODEL=us.anthropic.claude-opus-5, and the same prompt, and the double invocation shows up through the end-user spend surfaces (GET /spend/logs?api_key=$VKEY,GET /key/info?key=$VKEY)Before (f04b1da)
invoke-with-response-stream
content-typeheader at all, which is what makes Claude Code treat the stream as corruptedconverse-stream
content-typeheader on the Converse stream eitherinvoke (non-streaming control)
content-type: application/jsonon both sides, so the non-streaming path is untouchedClaude Code 2.1.234 turn (Bedrock gateway mode)
curl -s -X POST http://127.0.0.1:53571/key/generate -H "Authorization: Bearer $LITELLM_KEY" -H "Content-Type: application/json" -d '{"key_alias":"claude-code-before"}'and export itskeyas$VKEYCLAUDE_CODE_USE_BEDROCK=1 ANTHROPIC_BEDROCK_BASE_URL=http://127.0.0.1:53571/bedrock CLAUDE_CODE_SKIP_BEDROCK_AUTH=1 ANTHROPIC_AUTH_TOKEN=$VKEY ANTHROPIC_MODEL=us.anthropic.claude-opus-5 ANTHROPIC_SMALL_FAST_MODEL=us.anthropic.claude-opus-5 ANTHROPIC_DEFAULT_HAIKU_MODEL=us.anthropic.claude-opus-5 CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 claude --model us.anthropic.claude-opus-5, accept the folder trust prompt, typereply with exactly: STREAMING OK BEFOREand press Entertmux capture-pane -p)stream: false), and the key was billed for all fourAfter (d5a4c14)
invoke-with-response-stream
content-type: application/vnd.amazon.eventstreamconverse-stream
content-type: application/vnd.amazon.eventstreamtoo, so the fix is not tied to one Bedrock streaming routeinvoke (non-streaming control)
content-type: application/jsonon both sides, so the non-streaming path is untouchedClaude Code 2.1.234 turn (Bedrock gateway mode)
curl -s -X POST http://127.0.0.1:33945/key/generate -H "Authorization: Bearer $LITELLM_KEY" -H "Content-Type: application/json" -d '{"key_alias":"claude-code-after"}'and export itskeyas$VKEYCLAUDE_CODE_USE_BEDROCK=1 ANTHROPIC_BEDROCK_BASE_URL=http://127.0.0.1:33945/bedrock CLAUDE_CODE_SKIP_BEDROCK_AUTH=1 ANTHROPIC_AUTH_TOKEN=$VKEY ANTHROPIC_MODEL=us.anthropic.claude-opus-5 ANTHROPIC_SMALL_FAST_MODEL=us.anthropic.claude-opus-5 ANTHROPIC_DEFAULT_HAIKU_MODEL=us.anthropic.claude-opus-5 CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 claude --model us.anthropic.claude-opus-5, accept the folder trust prompt, typereply with exactly: STREAMING OK AFTERand press Entertmux capture-pane -p)Observations from the run (this PR leaves them alone):
x-litellm-response-cost*headers read 0.0, spend logs correctType
🐛 Bug Fix
Caveats (if any)
Final Attestation
Link to Devin session: https://app.devin.ai/sessions/835c506325e2454d80dc4434ae74f7b3
Requested by: @shivamrawat1