fix(passthrough): log upstream 4xx/5xx error bodies and carry them into the failure hook

[devin-ai-integration]

TLDR

Problem this solves:

• Passthrough upstream 4xx/5xx bodies never reach proxy logs, even at --detailed_debug
• Failure hooks and spend logs only see "failed with status 404"
• Operators cannot tell a bad model name from a permissions problem

How it solves it:

• Read at most 4096 bytes of the upstream error body and log it at WARNING with method, URL, status
• Streaming errors replay that prefix plus the live stream to the client, so nothing is buffered in full
• Collapse newlines and control characters so a provider body cannot forge log lines
• Append the body (capped at 4096 chars) to the synthetic failure exception detail
• Strip the query string from the logged URL so provider keys stay out of logs
• Let the passthrough prefix win in error normalization so body text cannot reclassify the row
• Keep the upstream status and the bytes that did arrive when the peek itself fails mid-read, instead of a 500
• Replace the body with redacted-by-litellm when message logging is turned off (global litellm.turn_off_message_logging or the key/team turn_off_message_logging callback var), so an echoed prompt never lands in logs or callbacks

User Flow

Before: an operator debugging a failing Vertex passthrough call sees only a status code and cannot tell what the provider rejected

1. They send POST http://localhost:4000/vertex_ai/v1/projects/$PROJECT/locations/global/publishers/anthropic/models/claude-nope-9:streamRawPredict with a normal Anthropic messages body
2. They get back HTTP 404 with Google's NOT_FOUND body saying the publisher model was not found
3. They open the proxy log started with --detailed_debug and find only the uvicorn access line POST ... 404 Not Found, no provider message anywhere
4. They open http://localhost:4000/ui/?page=logs, click the failed request, and the error message reads 404: Upstream passthrough request failed with status 404

After: the same request leaves the provider's own explanation in the log and in the request's error details

1. They send the same POST http://localhost:4000/vertex_ai/v1/projects/$PROJECT/locations/global/publishers/anthropic/models/claude-nope-9:streamRawPredict
2. They get back the same HTTP 404 with the same NOT_FOUND body, unchanged
3. The proxy log now has a WARNING line pass_through_endpoint: upstream POST https://aiplatform.googleapis.com/v1/projects/.../claude-nope-9:streamRawPredict returned 404: {"error": {"code": 404, "message": "Publisher model ... was not found ...", "status": "NOT_FOUND"}}, with any ?key= query removed from the URL
4. http://localhost:4000/ui/?page=logs shows the failed request with the error message 404: Upstream passthrough request failed with status 404: {"error": {... "was not found ..."}}

Relevant issues

Supersedes #39687, which targeted the retired staging branch and had a merge conflict; the fix here was re-derived from the ticket on current main with the query-string redaction and the normalization guard added

Affected release

Linear ticket

Resolves LIT-6644

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Real Google APIs (Vertex AI and Gemini), no mocks. Proxy launched with python -m litellm.proxy.proxy_cli --config <cfg> --port <port> --num_workers 1. Config: model_list: [], a generated master_key ($KEY), database_url: os.environ/DATABASE_URL, store_model_in_db: true, proxy_batch_write_at: 1, and default_vertex_config with vertex_project: os.environ/VERTEXAI_PROJECT, vertex_location: global, vertex_credentials: os.environ/VERTEXAI_CREDENTIALS. $PROJECT is the Vertex project id

Before (40ec84c)

Vertex passthrough streamRawPredict, nonexistent model

1. curl -s -w "\nHTTP %{http_code}\n" http://localhost:4011/vertex_ai/v1/projects/$PROJECT/locations/global/publishers/anthropic/models/claude-nope-9:streamRawPredict -H "Authorization: Bearer $KEY" -H "content-type: application/json" -d '{"anthropic_version":"vertex-2023-10-16","max_tokens":16,"messages":[{"role":"user","content":"hi"}],"stream":true}'
2. Output:

{
  "error": {
    "code": 404,
    "message": "Publisher model `projects/$PROJECT/locations/global/publishers/anthropic/models/claude-nope-9` was not found or your project does not have access to it. Ensure you are using a valid model name and that the model is available in the specified region. For more information, see: https://docs.cloud.google.com/gemini-enterprise-agent-platform/resources/locations.",
    "status": "NOT_FOUND"
  }
}

HTTP 404

3. grep -c "was not found" /tmp/before.log -> 0, grep "pass_through_endpoint: upstream" /tmp/before.log -> no output. Only the uvicorn access line mentions the 404

Gemini passthrough 404

1. curl -s -w "\nHTTP %{http_code}\n" http://localhost:4011/gemini/v1beta/models/gemini-nope-9:generateContent -H "Authorization: Bearer $KEY" -H "x-goog-api-key: $KEY" -H "content-type: application/json" -d '{"contents":[{"role":"user","parts":[{"text":"hi"}]}]}'
2. Output:

{
  "error": {
    "code": 404,
    "message": "models/gemini-nope-9 is not found for API version v1beta, or is not supported for generateContent. Call ModelService.ListModels to see the list of available models and their supported methods.",
    "status": "NOT_FOUND"
  }
}

HTTP 404

3. Proxy log: no pass_through_endpoint: upstream line, upstream body absent
4. SELECT status, spend, metadata->'error_information' FROM "LiteLLM_SpendLogs" WHERE request_id='<x-litellm-call-id>':

status: failure
spend: 0
error_class: HTTPException
error_code: "404"
error_message: "404: Upstream passthrough request failed with status 404"
normalized_error: 500_UPSTREAM_PASSTHROUGH

Gemini passthrough success (gemini-3.8-flash)

1. curl -s -w "\nHTTP %{http_code}\n" http://localhost:4011/gemini/v1beta/models/gemini-3.8-flash:generateContent -H "Authorization: Bearer $KEY" -H "x-goog-api-key: $KEY" -H "content-type: application/json" -d '{"contents":[{"role":"user","parts":[{"text":"hi"}]}]}'
2. HTTP 200, normal candidates body ("Hello! How can I help you today?"), modelVersion: gemini-3.8-flash

Gemini passthrough 404 with message logging turned off for the key

Same request as the After leg below with a turn_off_message_logging key. At the base the body is never logged in either mode, so the log carries no upstream line and the spend row error_message is 404: Upstream passthrough request failed with status 404, the same as the plain 404 case

Gemini passthrough streaming 404 (streamGenerateContent?alt=sse)

1. curl -s -w "\nHTTP %{http_code}\n" "http://localhost:4011/gemini/v1beta/models/gemini-nope-9:streamGenerateContent?alt=sse" -H "Authorization: Bearer $KEY" -H "x-goog-api-key: $KEY" -H "content-type: application/json" -d '{"contents":[{"role":"user","parts":[{"text":"hi"}]}]}'
2. Output: the same NOT_FOUND body as the non-streaming case, HTTP 404, 275 bytes
3. Proxy log: grep -c "pass_through_endpoint: upstream" -> 0, grep -c NOT_FOUND -> 0

After (a947ba7)

Vertex passthrough streamRawPredict, nonexistent model

1. curl -s -w "\nHTTP %{http_code}\n" http://localhost:4010/vertex_ai/v1/projects/$PROJECT/locations/global/publishers/anthropic/models/claude-nope-9:streamRawPredict -H "Authorization: Bearer $KEY" -H "content-type: application/json" -d '{"anthropic_version":"vertex-2023-10-16","max_tokens":16,"messages":[{"role":"user","content":"hi"}],"stream":true}'
2. Output, byte-identical to Before:

{
  "error": {
    "code": 404,
    "message": "Publisher model `projects/$PROJECT/locations/global/publishers/anthropic/models/claude-nope-9` was not found or your project does not have access to it. Ensure you are using a valid model name and that the model is available in the specified region. For more information, see: https://docs.cloud.google.com/gemini-enterprise-agent-platform/resources/locations.",
    "status": "NOT_FOUND"
  }
}

HTTP 404

3. Proxy log now carries the body:

07:21:01 - LiteLLM Proxy:WARNING: pass_through_endpoints.py:883 - pass_through_endpoint: upstream POST https://aiplatform.googleapis.com/v1/projects/$PROJECT/locations/global/publishers/anthropic/models/claude-nope-9:streamRawPredict returned 404: {
  "error": {
    "code": 404,
    "message": "Publisher model `projects/$PROJECT/locations/global/publishers/anthropic/models/claude-nope-9` was not found or your project does not have access to it. Ensure you are using a valid model name and that the model is available in the specified region. For more information, see: https://docs.cloud.google.com/gemini-enterprise-agent-platform/resources/locations.",
    "status": "NOT_FOUND"
  }
}

Gemini passthrough 404

1. curl -s -w "\nHTTP %{http_code}\n" http://localhost:4010/gemini/v1beta/models/gemini-nope-9:generateContent -H "Authorization: Bearer $KEY" -H "x-goog-api-key: $KEY" -H "content-type: application/json" -d '{"contents":[{"role":"user","parts":[{"text":"hi"}]}]}'
2. Output, byte-identical to Before (x-litellm-call-id: a0171a2b-5ff9-4548-863a-5511d4a946a8):

{
  "error": {
    "code": 404,
    "message": "models/gemini-nope-9 is not found for API version v1beta, or is not supported for generateContent. Call ModelService.ListModels to see the list of available models and their supported methods.",
    "status": "NOT_FOUND"
  }
}

HTTP 404

3. Proxy log; the gemini route appends ?key=<GEMINI_API_KEY> upstream and the logged URL has no query (grep -c "key=" /tmp/after_tip.log -> 0):

07:21:01 - LiteLLM Proxy:WARNING: pass_through_endpoints.py:883 - pass_through_endpoint: upstream POST https://generativelanguage.googleapis.com/v1beta/models/gemini-nope-9:generateContent returned 404: {
  "error": {
    "code": 404,
    "message": "models/gemini-nope-9 is not found for API version v1beta, or is not supported for generateContent. Call ModelService.ListModels to see the list of available models and their supported methods.",
    "status": "NOT_FOUND"
  }
}

4. SELECT status, spend, metadata->'error_information' FROM "LiteLLM_SpendLogs" WHERE request_id='a0171a2b-5ff9-4548-863a-5511d4a946a8':

status: failure
spend: 0
error_class: HTTPException
error_code: "404"
error_message: "404: Upstream passthrough request failed with status 404: {\n  \"error\": {\n    \"code\": 404,\n    \"message\": \"models/gemini-nope-9 is not found for API version v1beta, or is not supported for generateContent. Call ModelService.ListModels to see the list of available models and their supported methods.\",\n    \"status\": \"NOT_FOUND\"\n  }\n}\n"
normalized_error: 500_UPSTREAM_PASSTHROUGH

Gemini passthrough success (gemini-3.8-flash)

1. curl -s -w "\nHTTP %{http_code}\n" http://localhost:4010/gemini/v1beta/models/gemini-3.8-flash:generateContent -H "Authorization: Bearer $KEY" -H "x-goog-api-key: $KEY" -H "content-type: application/json" -d '{"contents":[{"role":"user","parts":[{"text":"hi"}]}]}'
2. HTTP 200, normal candidates body, modelVersion: gemini-3.8-flash, no pass_through_endpoint: upstream WARNING

Gemini passthrough 404 with message logging turned off for the key

Key generated with curl -s http://localhost:4010/key/generate -H "Authorization: Bearer $KEY" -H "content-type: application/json" -d '{"metadata":{"logging":[{"callback_name":"prometheus","callback_type":"success_and_failure","callback_vars":{"turn_off_message_logging":"true"}}]}}', used below as $EKEY

1. curl -s -w "\nHTTP %{http_code}\n" http://localhost:4010/gemini/v1beta/models/gemini-nope-9:generateContent -H "Authorization: Bearer $EKEY" -H "x-goog-api-key: $EKEY" -H "content-type: application/json" -d '{"contents":[{"role":"user","parts":[{"text":"hi"}]}]}'
2. Output: the same NOT_FOUND body as above, HTTP 404, client response unchanged (x-litellm-call-id: e7845dc2-2ede-454a-955b-e65f3fb2f40a)
3. Proxy log:

07:59:44 - LiteLLM Proxy:WARNING: pass_through_endpoints.py:890 - pass_through_endpoint: upstream POST https://generativelanguage.googleapis.com/v1beta/models/gemini-nope-9:generateContent returned 404: redacted-by-litellm

4. Spend row:

status: failure
error_message: "404: Upstream passthrough request failed with status 404: redacted-by-litellm"
normalized_error: 500_UPSTREAM_PASSTHROUGH

Gemini passthrough streaming 404 (streamGenerateContent?alt=sse, prefix-replay path)

1. curl -s -w "\nHTTP %{http_code}\n" "http://localhost:4010/gemini/v1beta/models/gemini-nope-9:streamGenerateContent?alt=sse" -H "Authorization: Bearer $KEY" -H "x-goog-api-key: $KEY" -H "content-type: application/json" -d '{"contents":[{"role":"user","parts":[{"text":"hi"}]}]}'
2. Output: HTTP 404, 275 bytes, cmp against the Before response body reports no difference (x-litellm-call-id: c6c00e01-1310-45a2-9f75-835819942ece)
3. Proxy log:

08:28:13 - LiteLLM Proxy:WARNING: pass_through_endpoints.py:946 - pass_through_endpoint: upstream POST https://generativelanguage.googleapis.com/v1beta/models/gemini-nope-9:streamGenerateContent returned 404: { "error": { "code": 404, "message": "models/gemini-nope-9 is not found for API version v1beta, or is not supported for generateContent. Call ModelService.ListModels to see the list of available models and their supported methods.", "status": "NOT_FOUND" } }

4. Spend row: status failure, error_message carries the same body, normalized_error: 500_UPSTREAM_PASSTHROUGH

Additional live A/B at base vs head: a scripted upstream serving a 6055-byte chunked 500 body through streamGenerateContent?alt=sse delivered all 6055 bytes to the client while the WARNING line ends with ... (truncated at 4096 chars). A scripted upstream returning 400 "no deployments available for this model" normalized to 429_NO_HEALTHY_DEPLOYMENTS on head before the pattern reorder and to 500_UPSTREAM_PASSTHROUGH after it

The same seven live legs were rerun at the tip 23a7284 against the merge base e26a645 after main was merged in (main dropped tests/integration/contracts.json and the covers check, so the merge deletes the manifest and the tip strips the retired markers from the new tests): every client status and body is byte identical between base and tip (the 200 success leg differs only in generated content), and every tip WARNING line and spend row carries the body, the redacted-by-litellm placeholder for the redacted key, or the 4096 marker for the 6055-byte body, exactly as shown above

After the mid-read guard landed (8618621, 5d57d25 which only closes the upstream response in the new unit test, and c3c5676 which relays the decoded pieces rather than raw compressed bytes on a mid-read failure, so a gzip body cannot reach the client corrupted), the streaming legs C, F and CHUNK were rerun at c3c5676 against e26a645: byte identical to the client, head spend rows carry the body (/home/ubuntu/scratch/lit6644_live_pr_risk_c3c56762e2.md). The mid-read failure path itself cannot be reached from a live provider, so it is covered by the mutation-checked unit test (removing the catch fails with httpx.ReadError: peer reset, and the gzip variant fails on 5d57d25 because the client received compressed bytes). The audit head legs were also rerun at c3c5676, 57 nodes, all 20 observability cells green on both runs with no skips (audit_report_c3c56762e2.md); the base leg below still applies since no observability test file changed

Admin UI Logs page (head only, the Before rows carry no body to show)

1. Open http://localhost:4010/ui/?page=logs, log in as admin with the master key
2. Click the Failure row for the Gemini gemini-nope-9:generateContent request (x-litellm-call-id: 25bd8931-c638-48b0-8da8-518367358f26)
3. The Request Failed panel shows Error Code: 404 and the message now carries the upstream body: models/gemini-nope-9 is not found for API version v1beta ... "status": "NOT_FOUND"

Audit matrix (deterministic tests/integration, tip 23a7284, head legs repeated at c3c5676)

Every row is a checked-in test in tests/integration/observability/test_passthrough_upstream_error_visibility.py or test_passthrough_upstream_error_chaos.py, run through tests/integration/run.py extensions against a real proxy with two workers, real Postgres, real Redis and the wire_server scripted upstream, no mock inside the proxy, no real provider. Base leg is the merge base e26a645 with the tip's test tree copied in, the head legs are the tip run twice with an identical collected selection of 57 nodes and no skips. Body-visibility cells are red on base and green on both head legs, the cells the fix must not touch (H3, H4, N1) are green on all three legs

id	scenario	client	base	head x2
H1, H1a	/gemini generateContent 404 JSON: client body unchanged, WARNING and spend row carry the body, error_code 404	httpx sync, httpx async	FAIL	PASS
H2	/gemini streamGenerateContent 500, 6055 bytes chunked: all bytes relayed, WARNING and row end with the 4096 marker	httpx stream	FAIL	PASS
H3, H4	/gemini 200 non streaming and 200 SSE: body relayed, no upstream WARNING, success row	httpx	PASS	PASS
H5	config pass_through_endpoints route 403 with max budget reached in the body and a query string on the target: body logged, query stripped, normalized_error stays 500_UPSTREAM_PASSTHROUGH	httpx	FAIL	PASS
H6, H6a	/openai passthrough 404 through the OpenAI SDK: NotFoundError carries the message, WARNING and row carry the body	openai sync, openai async	FAIL	PASS
S1	502 text/html body with newlines, ANSI escapes and NUL: one log line, marker present	httpx	FAIL	PASS
S2	404 with empty body: WARNING still logged, row has error_code 404, next request still served	httpx	FAIL	PASS
S3, S4	400 gzip body and 403 gzip body in 3 chunks: client sees the decoded JSON, WARNING has the decoded text	httpx	FAIL	PASS
E1	turn_off_message_logging: true: WARNING and row show redacted-by-litellm, never the body	httpx	FAIL	PASS
E2, E3	body of exactly 4096 bytes logs without the marker, 4097 bytes logs with it	httpx	FAIL	PASS
E4	404 body streamed one byte per chunk: reassembled for the client and the log	httpx stream	FAIL	PASS
E5	the same 404 twice: two rows and exactly two WARNING lines	httpx	FAIL	PASS
N1	/v1/chat/completions budget rejection still normalizes to BUDGET_EXCEEDED after the pattern reorder	httpx	PASS	PASS
C1	30 concurrent mixed passthrough calls with the upstream stopped after 10 and restarted on the same port: every call answered, one spend row per call id, error rows carry the body, both workers took traffic	httpx async	FAIL	PASS
C2	20 concurrent 404 calls with one uvicorn worker SIGKILLed mid burst: sibling keeps serving, follow up 404 carries the body, one row per completed call	httpx async	FAIL	PASS

Type

🐛 Bug Fix

Caveats (if any)

Medium

• For a streaming error the relayed response is a replacement object wrapping the original stream with content-encoding and content-length dropped, since the replayed bytes are already decoded; the client-facing headers already excluded both, so nothing visible changes
• The x-litellm-enable-message-redaction request header does not turn redaction on for passthrough requests, because passthrough stores request headers under proxy_server_request.headers, which should_redact_message_logging does not read. That is preexisting and applies to the whole passthrough logging path; the global setting and the key/team turn_off_message_logging callback var do apply
• Spend logs already cap error_message at 2048 chars, so the tail of a long body is kept in the log line only
• The 4096 cap is applied in bytes on the streaming path and in characters after decoding, so a multibyte body may show slightly fewer than 4096 characters before the marker

Low

• The exception detail now ends with the provider body, so any consumer matching the exact old sentence sees a longer string
• Upstream bodies are subject to the existing prompt redaction in spend logs when store_prompts_in_spend_logs is off

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

ran /live-pr-risk and found no regressions/backward incompatible risks

REVIEWER MUST KNOW BEFORE APPROVING

Failure hook exception detail, spend log error_message and the Admin UI Logs error text: before Upstream passthrough request failed with status 404, after the same sentence followed by : and up to 4096 chars of the upstream body, or : redacted-by-litellm when message logging is off. Requested by LIT-6644, approved by yucheng-berri by assigning the ticket

Proxy WARNING log: before nothing for an upstream 4xx/5xx, after one pass_through_endpoint: upstream <METHOD> <url without query> returned <status>: <body> line per failure. Requested by LIT-6644, approved by yucheng-berri by assigning the ticket

Streaming upstream error whose body stream fails inside the first 4096 bytes (live A/B with a scripted upstream that sends 15 bytes of a 502 then resets, both legs run twice, transcripts in /home/ubuntu/scratch/lit6644_midread_legs.md): before the client gets the 502 status line and then a reset, curl: (18) transfer closed with outstanding read data remaining, 0 body bytes, spend row with the bare status; after the client gets the 502 with the 15 bytes that arrived and a clean end of body, curl exit 0, spend row carrying those bytes and a read failed after 15 bytes: ReadError warning. Raised by Bugbot on this PR, fixed in 8618621 and c3c5676, not yet approved by a human: yucheng-berri decides on this PR

Link to Devin session: https://app.devin.ai/sessions/9943a4e7d8ef47638394734d69c2946e
Open in Devin Desktop: https://app.devin.ai/desktop/session/9943a4e7d8ef47638394734d69c2946e?variant=devin
Requested by: @yucheng-berri

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge; no outstanding previous findings or new actionable issues remain.

Summary

This PR improves passthrough error observability while preserving upstream responses.

• Captures a bounded, sanitized upstream error-body preview for warning logs and failure hooks.
• Redacts previews when message logging is disabled and removes query strings from logged URLs.
• Replays decoded streaming prefixes without buffering complete error responses.
• Preserves passthrough error normalization and adds unit, integration, streaming, compression, concurrency, and redaction coverage.

_{Reviews (11) · Last reviewed commit: "fix(passthrough): relay decoded partial ..."}

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_passthrough_upstream_error_body_logging (c3c5676) with main (e0af991)}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[devin-ai-integration]

@greptileai

[devin-ai-integration]

Devin Review found 3 potential issues.

[devin-ai-integration]

@greptileai

[devin-ai-integration]

@greptileai

[devin-ai-integration]

UI proof at a947ba7: the Logs page detail for a Gemini passthrough 404 now shows the upstream body in the error message

[devin-ai-integration]

@greptileai

[devin-ai-integration]

@greptileai

[devin-ai-integration]

Greptile's matching redaction thread stood down after 23c8a9c, and integration cell E1 proves the redacted path at ddcef2e

[devin-ai-integration]

@greptileai

[yucheng-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[devin-ai-integration]

@greptileai

[devin-ai-integration]

@greptileai

[devin-ai-integration]

@greptileai

[devin-ai-integration]

@greptileai

[yucheng-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit c3c5676. Configure here.}