Skip to content

fix(proxy): forward Bedrock event-stream content-type on unbuffered passthrough - #33767

Merged
mateo-berri merged 4 commits into
litellm_internal_stagingfrom
litellm_lit_4561_bedrock_passthrough_content_type
Aug 18, 2026
Merged

fix(proxy): forward Bedrock event-stream content-type on unbuffered passthrough#33767
mateo-berri merged 4 commits into
litellm_internal_stagingfrom
litellm_lit_4561_bedrock_passthrough_content_type

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • Bedrock passthrough streaming responses carried no content-type header
  • Claude Code past 2.1.207 rejects such streams as gateway-corrupted
  • Newer versions silently refetch non-streaming, doubling cost and latency

How it solves it:

  • The unbuffered passthrough stream now sets media_type from the provider handler
  • Same helper the buffered guardrail branch already passes
  • Bedrock streams return application/vnd.amazon.eventstream
  • Providers with no registered event-stream media type stay unchanged

User Flow

Before: a developer running Claude Code against the proxy's Bedrock passthrough sees every streaming turn rejected or silently doubled

  1. They set CLAUDE_CODE_USE_BEDROCK=1 and ANTHROPIC_BEDROCK_BASE_URL=https://litellm-domain/bedrock, then ask Claude Code a question
  2. Claude Code sends POST https://litellm-domain/bedrock/model/{model}/invoke-with-response-stream
  3. The response streams valid AWS event-stream bytes but carries no content-type header
  4. Claude Code errors with API Error: Bedrock streaming response has content-type "application/octet-stream"; expected "application/vnd.amazon.eventstream", and versions past 2.1.227 instead silently refetch the same turn with stream: false, so every turn costs two full model invocations

After: the same turn streams once and renders normally

  1. They set CLAUDE_CODE_USE_BEDROCK=1 and ANTHROPIC_BEDROCK_BASE_URL=https://litellm-domain/bedrock, then ask Claude Code a question
  2. Claude Code sends POST https://litellm-domain/bedrock/model/{model}/invoke-with-response-stream
  3. The response streams the same bytes with content-type: application/vnd.amazon.eventstream
  4. Claude Code renders the streamed answer in one pass, with no guard error and no duplicate non-streaming call

Relevant issues

Linear ticket

Resolves LIT-4561
Resolves LIT-5674

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • My PR passes all CI/CD checks (e.g., lint, format, unit tests)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Two live proxies against real AWS Bedrock (bedrock/us.anthropic.claude-opus-5, us-east-1, real spend), each booted from its own worktree and venv with its own fresh Postgres, model_list router deployment us.anthropic.claude-opus-5 (custom_llm_provider: bedrock), no mocks. Before runs the merge base f04b1da on port 53571, After runs this PR's tip d5a4c14 on port 33945. Same payloads both sides. The Claude Code case runs the real claude CLI 2.1.234 interactively under tmux with CLAUDE_CODE_USE_BEDROCK=1, ANTHROPIC_BEDROCK_BASE_URL=http://127.0.0.1:{port}/bedrock, CLAUDE_CODE_SKIP_BEDROCK_AUTH=1, ANTHROPIC_AUTH_TOKEN=$VKEY (a per-leg virtual key from POST /key/generate), ANTHROPIC_MODEL=us.anthropic.claude-opus-5, and the same prompt, and the double invocation shows up through the end-user spend surfaces (GET /spend/logs?api_key=$VKEY, GET /key/info?key=$VKEY)

Before (f04b1da)

invoke-with-response-stream

  1. Customer's header check
$ curl -s -D headers.txt -o body.bin -X POST http://127.0.0.1:53571/bedrock/model/us.anthropic.claude-opus-5/invoke-with-response-stream \
    -H "Authorization: Bearer $LITELLM_KEY" -H "Content-Type: application/json" \
    -d '{"anthropic_version":"bedrock-2023-05-31","max_tokens":32,"messages":[{"role":"user","content":"Say hi in three words"}]}'
$ grep -i '^HTTP\|^content-type\|^x-litellm-model-name' headers.txt
HTTP/1.1 200 OK
x-litellm-model-name: bedrock/us.anthropic.claude-opus-5
$ grep -ci '^content-type' headers.txt
0
$ xxd body.bin | head -2
00000000: 0000 02f4 0000 004b 7450 10e7 0b3a 6576  .......KtP...:ev
00000010: 656e 742d 7479 7065 0700 0563 6875 6e6b  ent-type...chunk
  1. Streamed AWS event-stream bytes arrive with no content-type header at all, which is what makes Claude Code treat the stream as corrupted

converse-stream

  1. Same check on the Converse streaming route
$ curl -s -D headers.txt -o body.bin -X POST http://127.0.0.1:53571/bedrock/model/us.anthropic.claude-opus-5/converse-stream \
    -H "Authorization: Bearer $LITELLM_KEY" -H "Content-Type: application/json" \
    -d '{"messages":[{"role":"user","content":[{"text":"Say hi in three words"}]}],"inferenceConfig":{"maxTokens":32}}'
$ grep -i '^HTTP\|^content-type\|^x-litellm-model-name' headers.txt
HTTP/1.1 200 OK
x-litellm-model-name: bedrock/us.anthropic.claude-opus-5
$ grep -ci '^content-type' headers.txt
0
$ xxd body.bin | head -2
00000000: 0000 00b0 0000 0052 f080 e7a5 0b3a 6576  .......R.....:ev
00000010: 656e 742d 7479 7065 0700 0c6d 6573 7361  ent-type...messa
  1. No content-type header on the Converse stream either

invoke (non-streaming control)

  1. Non-streaming route, unchanged by this PR
$ curl -s -D headers.txt -o body.json -X POST http://127.0.0.1:53571/bedrock/model/us.anthropic.claude-opus-5/invoke \
    -H "Authorization: Bearer $LITELLM_KEY" -H "Content-Type: application/json" \
    -d '{"anthropic_version":"bedrock-2023-05-31","max_tokens":32,"messages":[{"role":"user","content":"Say hi in three words"}]}'
$ grep -i '^HTTP\|^content-type\|^x-litellm-model-name' headers.txt
HTTP/1.1 200 OK
content-type: application/json
x-litellm-model-name: bedrock/us.anthropic.claude-opus-5
$ head -c 200 body.json
{"model":"claude-opus-5","id":"msg_bdrk_mqtio6tzsztbnpdh2pmzqn7og542inwylmxvkfqlllftjrcwnara","type":"message","role":"assistant","content":[{"type":"text","text":"Hi there, friend!"}],"stop_reason":"
  1. content-type: application/json on both sides, so the non-streaming path is untouched

Claude Code 2.1.234 turn (Bedrock gateway mode)

  1. Mint a virtual key for the session: curl -s -X POST http://127.0.0.1:53571/key/generate -H "Authorization: Bearer $LITELLM_KEY" -H "Content-Type: application/json" -d '{"key_alias":"claude-code-before"}' and export its key as $VKEY
  2. Launch the CLI interactively in tmux: CLAUDE_CODE_USE_BEDROCK=1 ANTHROPIC_BEDROCK_BASE_URL=http://127.0.0.1:53571/bedrock CLAUDE_CODE_SKIP_BEDROCK_AUTH=1 ANTHROPIC_AUTH_TOKEN=$VKEY ANTHROPIC_MODEL=us.anthropic.claude-opus-5 ANTHROPIC_SMALL_FAST_MODEL=us.anthropic.claude-opus-5 ANTHROPIC_DEFAULT_HAIKU_MODEL=us.anthropic.claude-opus-5 CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 claude --model us.anthropic.claude-opus-5, accept the folder trust prompt, type reply with exactly: STREAMING OK BEFORE and press Enter
  3. Pane capture (tmux capture-pane -p)
│   Opus 5 with xhigh effort · Amazon Bedrock   │
❯ reply with exactly: STREAMING OK BEFORE
⏺ STREAMING OK BEFORE
✻ Cooked for 5s
  1. What the proxy billed that one turn to the key, through the end-user spend routes
$ curl -s "http://127.0.0.1:53571/spend/logs?api_key=$VKEY" -H "Authorization: Bearer $LITELLM_KEY" \
    | jq '[.[] | select(.status=="success")] | map({call_type, model, prompt_tokens, completion_tokens, spend})'
[
 {
  "call_type": "allm_passthrough_route",
  "model": "bedrock/us.anthropic.claude-opus-5",
  "prompt_tokens": 31076,
  "completion_tokens": 14,
  "spend": 0.0174867
 },
 {
  "call_type": "allm_passthrough_route",
  "model": "bedrock/us.anthropic.claude-opus-5",
  "prompt_tokens": 993,
  "completion_tokens": 28,
  "spend": 0.0062315
 },
 {
  "call_type": "allm_passthrough_route",
  "model": "bedrock/us.anthropic.claude-opus-5",
  "prompt_tokens": 31076,
  "completion_tokens": 14,
  "spend": 0.21402975
 },
 {
  "call_type": "allm_passthrough_route",
  "model": "bedrock/us.anthropic.claude-opus-5",
  "prompt_tokens": 993,
  "completion_tokens": 16,
  "spend": 0.0059015
 }
]
$ curl -s "http://127.0.0.1:53571/key/info?key=$VKEY" -H "Authorization: Bearer $LITELLM_KEY" | jq '{key_alias: .info.key_alias, spend: .info.spend}'
{
 "key_alias": "claude-code-before",
 "spend": 0.24364945
}
  1. Claude Code 2.1.234 renders the answer but only because it silently refetched: the single turn produced two ~31k-token main invocations and two ~1k-token side calls (each request run once as the rejected stream and once again as stream: false), and the key was billed for all four

After (d5a4c14)

invoke-with-response-stream

  1. Customer's header check
$ curl -s -D headers.txt -o body.bin -X POST http://127.0.0.1:33945/bedrock/model/us.anthropic.claude-opus-5/invoke-with-response-stream \
    -H "Authorization: Bearer $LITELLM_KEY" -H "Content-Type: application/json" \
    -d '{"anthropic_version":"bedrock-2023-05-31","max_tokens":32,"messages":[{"role":"user","content":"Say hi in three words"}]}'
$ grep -i '^HTTP\|^content-type\|^x-litellm-model-name' headers.txt
HTTP/1.1 200 OK
x-litellm-model-name: bedrock/us.anthropic.claude-opus-5
content-type: application/vnd.amazon.eventstream
$ grep -ci '^content-type' headers.txt
1
$ xxd body.bin | head -2
00000000: 0000 02d3 0000 004b 07b1 e3f3 0b3a 6576  .......K.....:ev
00000010: 656e 742d 7479 7065 0700 0563 6875 6e6b  ent-type...chunk
  1. The same event-stream bytes now arrive with content-type: application/vnd.amazon.eventstream

converse-stream

  1. Same check on the Converse streaming route
$ curl -s -D headers.txt -o body.bin -X POST http://127.0.0.1:33945/bedrock/model/us.anthropic.claude-opus-5/converse-stream \
    -H "Authorization: Bearer $LITELLM_KEY" -H "Content-Type: application/json" \
    -d '{"messages":[{"role":"user","content":[{"text":"Say hi in three words"}]}],"inferenceConfig":{"maxTokens":32}}'
$ grep -i '^HTTP\|^content-type\|^x-litellm-model-name' headers.txt
HTTP/1.1 200 OK
x-litellm-model-name: bedrock/us.anthropic.claude-opus-5
content-type: application/vnd.amazon.eventstream
$ grep -ci '^content-type' headers.txt
1
$ xxd body.bin | head -2
00000000: 0000 0083 0000 0052 1601 25f3 0b3a 6576  .......R..%..:ev
00000010: 656e 742d 7479 7065 0700 0c6d 6573 7361  ent-type...messa
  1. The Converse stream carries content-type: application/vnd.amazon.eventstream too, so the fix is not tied to one Bedrock streaming route

invoke (non-streaming control)

  1. Non-streaming route, unchanged by this PR
$ curl -s -D headers.txt -o body.json -X POST http://127.0.0.1:33945/bedrock/model/us.anthropic.claude-opus-5/invoke \
    -H "Authorization: Bearer $LITELLM_KEY" -H "Content-Type: application/json" \
    -d '{"anthropic_version":"bedrock-2023-05-31","max_tokens":32,"messages":[{"role":"user","content":"Say hi in three words"}]}'
$ grep -i '^HTTP\|^content-type\|^x-litellm-model-name' headers.txt
HTTP/1.1 200 OK
content-type: application/json
x-litellm-model-name: bedrock/us.anthropic.claude-opus-5
$ head -c 200 body.json
{"model":"claude-opus-5","id":"msg_bdrk_nx2bcw7z5cdwcecgvmfxgrfb23blyg2usfzovba6f4rz6uds24hq","type":"message","role":"assistant","content":[{"type":"text","text":"Hi there, friend!"}],"stop_reason":"
  1. content-type: application/json on both sides, so the non-streaming path is untouched

Claude Code 2.1.234 turn (Bedrock gateway mode)

  1. Mint a virtual key for the session: curl -s -X POST http://127.0.0.1:33945/key/generate -H "Authorization: Bearer $LITELLM_KEY" -H "Content-Type: application/json" -d '{"key_alias":"claude-code-after"}' and export its key as $VKEY
  2. Launch the CLI interactively in tmux: CLAUDE_CODE_USE_BEDROCK=1 ANTHROPIC_BEDROCK_BASE_URL=http://127.0.0.1:33945/bedrock CLAUDE_CODE_SKIP_BEDROCK_AUTH=1 ANTHROPIC_AUTH_TOKEN=$VKEY ANTHROPIC_MODEL=us.anthropic.claude-opus-5 ANTHROPIC_SMALL_FAST_MODEL=us.anthropic.claude-opus-5 ANTHROPIC_DEFAULT_HAIKU_MODEL=us.anthropic.claude-opus-5 CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 claude --model us.anthropic.claude-opus-5, accept the folder trust prompt, type reply with exactly: STREAMING OK AFTER and press Enter
  3. Pane capture (tmux capture-pane -p)
│   Opus 5 with xhigh effort · Amazon Bedrock   │
❯ reply with exactly: STREAMING OK AFTER
⏺ STREAMING OK AFTER
✻ Worked for 3s
  1. What the proxy billed that one turn to the key, through the end-user spend routes
$ curl -s "http://127.0.0.1:33945/spend/logs?api_key=$VKEY" -H "Authorization: Bearer $LITELLM_KEY" \
    | jq '[.[] | select(.status=="success")] | map({call_type, model, prompt_tokens, completion_tokens, spend})'
[
 {
  "call_type": "allm_passthrough_route",
  "model": "bedrock/us.anthropic.claude-opus-5",
  "prompt_tokens": 31075,
  "completion_tokens": 13,
  "spend": 0.118184275
 },
 {
  "call_type": "allm_passthrough_route",
  "model": "bedrock/us.anthropic.claude-opus-5",
  "prompt_tokens": 992,
  "completion_tokens": 27,
  "spend": 0.0061985
 }
]
$ curl -s "http://127.0.0.1:33945/key/info?key=$VKEY" -H "Authorization: Bearer $LITELLM_KEY" | jq '{key_alias: .info.key_alias, spend: .info.spend}'
{
 "key_alias": "claude-code-after",
 "spend": 0.124382775
}
  1. The same turn produces exactly one ~31k-token main invocation and one ~1k-token side call, and the key is billed once per request; the streamed answer renders in one pass with no guard error and no non-streaming refetch

Observations from the run (this PR leaves them alone):

  • Both legs: startup model-discovery probes fail, models absent from model_list
  • Passthrough x-litellm-response-cost* headers read 0.0, spend logs correct

Type

🐛 Bug Fix

Caveats (if any)

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/835c506325e2454d80dc4434ae74f7b3
Requested by: @shivamrawat1

…assthrough

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@shivamrawat1 shivamrawat1 self-assigned this Jul 17, 2026
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@greptile-apps

greptile-apps Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes a missing Content-Type header on the unbuffered Bedrock passthrough streaming path by forwarding the provider's event-stream media type on the StreamingResponse, exactly as the buffered guardrail branch already did. The change is a single-line addition that reuses the existing _passthrough_event_stream_media_type() helper, keeping the fix provider-agnostic.

  • The one-liner media_type=self._passthrough_event_stream_media_type() is added to the unbuffered StreamingResponse (line 2042), matching the call already present on the buffered Response branch (line 2033).
  • Two focused mock-only regression tests are added: one pinning application/vnd.amazon.eventstream for Bedrock, and one confirming non-Bedrock providers continue to emit no content-type header.

Confidence Score: 5/5

  • Safe to merge — the change is a one-liner that applies an existing helper to a second code path, backed by concrete before/after evidence and passing mock tests.
  • The fix is minimal and surgical: one helper call added to the unbuffered stream path to match behaviour that was already correct on the buffered path. No logic is changed, no new abstractions are introduced, and the existing _passthrough_event_stream_media_type() already had test coverage for the buffered branch. The two new tests cover both the positive (Bedrock gets the correct header) and negative (non-Bedrock is unchanged) cases without real network calls, and the PR description includes live curl output confirming the fix against real AWS Bedrock endpoints.
  • No files require special attention.

Important Files Changed

Filename Overview
litellm/proxy/common_request_processing.py Adds media_type=self._passthrough_event_stream_media_type() to the unbuffered StreamingResponse branch, mirroring the already-existing call in the buffered Response branch. Change is minimal and correct.
tests/test_litellm/proxy/test_common_request_processing.py Adds two new mock-only tests: one pinning the Bedrock unbuffered stream to application/vnd.amazon.eventstream, and one confirming non-Bedrock providers emit no content-type header. No real network calls; assertions are precise.

Reviews (3): Last reviewed commit: "docs(proxy): pre-fix passthrough streams..." | Re-trigger Greptile

Comment thread tests/test_litellm/proxy/test_common_request_processing.py Outdated
@codecov

codecov Bot commented Jul 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 4 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
litellm/proxy/common_request_processing.py 0.00% 4 Missing ⚠️

📢 Thoughts on this report? Let us know!

@codspeed-hq

codspeed-hq Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing litellm_lit_4561_bedrock_passthrough_content_type (d5a4c14) with litellm_internal_staging (2bc87ec)1

Open in CodSpeed

Footnotes

  1. No successful run was found on litellm_internal_staging (cde5465) during the generation of this report, so 2bc87ec was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

Before/after proof against two live proxies hitting real AWS Bedrock (global.anthropic.claude-opus-4-8), same invoke-with-response-stream request to each. Port 4001 runs the base branch (before), port 4000 runs this PR (after)

BEFORE (base branch, port 4001)

$ curl -s -D - -o /tmp/before.bin -X POST \
  "http://localhost:4001/bedrock/model/global.anthropic.claude-opus-4-8/invoke-with-response-stream" \
  -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
  -d '{"anthropic_version":"bedrock-2023-05-31","max_tokens":30,"messages":[{"role":"user","content":"say hello in 3 words"}]}' | grep -iE '^(HTTP|content-type)'
HTTP/1.1 200 OK

No content-type header, which is what trips the Claude Code >2.1.207 guard

AFTER (this PR, port 4000)

$ curl -s -D - -o /tmp/after.bin -X POST \
  "http://localhost:4000/bedrock/model/global.anthropic.claude-opus-4-8/invoke-with-response-stream" \
  -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
  -d '{"anthropic_version":"bedrock-2023-05-31","max_tokens":30,"messages":[{"role":"user","content":"say hello in 3 words"}]}' | grep -iE '^(HTTP|content-type)'
HTTP/1.1 200 OK
content-type: application/vnd.amazon.eventstream

Body is still the raw AWS event-stream, unchanged by setting the header

$ xxd /tmp/after.bin | head -4
00000000: 0000 02e9 0000 004b ec20 43d4 0b3a 6576  .......K. C..:ev
00000010: 656e 742d 7479 7065 0700 0563 6875 6e6b  ent-type...chunk
00000020: 0d3a 636f 6e74 656e 742d 7479 7065 0700  .:content-type..
00000030: 1061 7070 6c69 6361 7469 6f6e 2f6a 736f  .application/jso

Session: https://app.devin.ai/sessions/835c506325e2454d80dc4434ae74f7b3

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

❌ Cannot revive Devin session - the session is too old. Please start a new session instead.

View session

@mateo-berri

Copy link
Copy Markdown
Contributor

@greptileai

@mateo-berri

Copy link
Copy Markdown
Contributor

@greptileai

@mateo-berri

Copy link
Copy Markdown
Contributor

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit d5a4c14. Configure here.

@mateo-berri mateo-berri left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mateo-berri
mateo-berri merged commit 035bd76 into litellm_internal_staging Aug 18, 2026
75 of 76 checks passed
@mateo-berri
mateo-berri deleted the litellm_lit_4561_bedrock_passthrough_content_type branch August 18, 2026 20:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants