fix(passthrough): relay upstream error streams without waiting for the log preview and keep passthrough normalization

[devin-ai-integration]

TLDR

Problem this solves:

• fix(passthrough): log upstream 4xx/5xx error bodies and carry them into the failure hook #42695 buffers the upstream error body before relaying, so a held-open 429 stream stalls the client 12 s
• Quota wording in the upstream body flips normalized_error to 429_BUDGET_EXCEEDED
• The upstream body reaches failure hooks without the proxy's secret masking

How it solves it:

• Relay every error chunk at once, keep a 4 KB preview for the report
• Passthrough prefix wins over the budget detector in normalize_error
• Preview goes through redact_secrets before logs, hooks and spend rows
• Report is a named task the relay stream dispatches once the 4 KB preview is full (or at EOF, abort, close, or any cancel before the StreamingResponse exists, which also closes the upstream connection and logs "abandoned before reaching the client" rather than a client disconnect, and a failing upstream close is logged instead of masking the exception that abandoned the relay, 5beef60327); the response's BackgroundTask and lifespan shutdown (10 s wait) only await it
• The log preview is built from a bounded 64 KiB scan: sanitize whitespace first, then redact the first 4096 chars in their original coordinates (a secret straddling the cut is masked, nothing past the cut is pulled in), then mask a dangling PEM header only when key material follows it, so a 5 MB buffered error body no longer blocks the loop and a bare -----BEGIN PRIVATE KEY----- mention in an error message stays readable

User Flow

Before: a developer streaming through a passthrough route gets nothing while the provider holds a 429 open, and the log misfiles it as a budget error

1. They send POST https://litellm-domain/gemini/v1beta/models/gemini-2.5-pro:streamGenerateContent?alt=sse and the provider answers 429 with one SSE frame, then holds the connection 12 s
2. A client with a 5 s read timeout gets no status line and no bytes; a patient client sees the 429 and the frame after 12 s
3. On a plain 429 whose body says "exceeded your current quota", https://litellm-domain/ui/?page=logs files it under 429_BUDGET_EXCEEDED

After: the 429 and its first frame arrive at once and the log entry stays a passthrough error

1. They send the same POST with the same provider behavior
2. The 429 and the first frame arrive in under 50 ms, the rest as the provider sends it
3. https://litellm-domain/ui/?page=logs lists it under 500_UPSTREAM_PASSTHROUGH with key-shaped fragments masked
4. If the client hangs up mid-stream the log entry is still written

Behavior changes

Streamed post_call_failure_hook for an upstream error now runs after post_call_response_headers_hook and after the last body byte. A response header derived from failure-hook state is no longer present on streamed error responses (base: x-failure-for-this-request: 500, tip: none), pinned by test_passthrough_streamed_error_headers_do_not_carry_failure_hook_state and the ASGI order test in test_pass_through_endpoints.py. Non-streamed errors are unchanged

Relevant issues

Follow-up to #42695

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Two proxies on real Postgres (proxy_batch_write_at: 1), base at the merge base f6882246d4 and tip at 58d02d5f84 (6a4a17f013 and 3da1a16060 only change tests), forwarding /ab/... to a scripted upstream. /sse-hang answers 429 text/event-stream, sends one frame and holds 12 s. /json-429 answers 429 {"error": {"message": "You exceeded your current quota ... Budget for Key=sk-<40 chars> is spent"}}. /abort-N streams N bytes of a 500 then closes abnormally

curl -sS -N -o body.bin -w "code=%{http_code} ttfb=%{time_starttransfer} total=%{time_total}\n" --max-time 20 \
  -X POST http://127.0.0.1:<port>/ab/<route> -H "Authorization: Bearer $KEY" -H "content-type: application/json" -d '<body>'

Cell	Base	Tip
/sse-hang {"stream":true}	code=429 ttfb=12.076 total=12.076	code=429 ttfb=0.008 total=12.110
/sse-hang {} (SSE detected from content-type)	ttfb=12.063	ttfb=0.006
/sse-hang with --max-time 5	curl: (28) ... 0 bytes received, code=000	curl: (28) ... 32 bytes received, code=429, body is the frame
/json-429, then select normalized_error from "LiteLLM_SpendLogs" where request_id=<call id>	429_BUDGET_EXCEEDED	500_UPSTREAM_PASSTHROUGH, error_message has Key=REDACTED
/sse-hang with --max-time 2 (disconnect after first frame)	n/a, base logs before the first byte	exactly one spend row, 500_UPSTREAM_PASSTHROUGH, error_code 429
/abort-5000 (past the 4 KB preview)	size=5000, curl: (18)	size=5000, curl: (18)
/abort-3000 (inside the preview)	size=3000, exit 0	size=3000, exit 0
200 concurrent streamed 500s, 3 s failure hook, max completion	8.57 s	9.30 s (no shared limiter, no 3/6/9/12 s buckets)
fast streamed request behind 70 slow-hook requests, first body byte	3.016 s	0.006 s
20 disconnects, 15 s failure hook, SIGTERM	20/20 rows	20/20 rows
never-returning failure hook, uvicorn --timeout-graceful-shutdown 3, SIGTERM once the hook is parked	request starved (21/32 bytes at 4.6 s), proxy exit 3.18 s	request answered in 0.034 s, proxy exit 3.22 s, the cancelled task trace ends in the hook via starlette/background.py
sse_keepalive_ping_interval_seconds: 0.2, 2 s failure hook, 500 SSE body	body identical, ttfb 2.048 s	body identical, ttfb 0.034 s, zero : ping frames
x-failure-for-this-request header set from failure-hook state	500	none (see Behavior changes)

Byte-identical on both legs: 6 KB streamed 500 in 512 B chunks, non-streaming 404, 200 SSE success, gzip 500, turn_off_message_logging: true

Audit at the tip 5beef60327 (merge base 14f4c34c61), two proxies with two workers each on real Postgres and Redis, scripted upstream only at the edge, base leg run id 95652898eaa848a38519886505721871, head leg run id 92a5193b18b6415f86b8159fd9c6a8c1. The head's tests/integration/observability/test_passthrough_upstream_error_visibility.py and test_passthrough_upstream_error_chaos.py (54 nodes) ran against the merge base: 19 red, 34 green, 1 skipped. The same files at the tip ran twice inside the full extensions group with --hypothesis-seed=42 --integration-order-seed=42: 53 green and 1 skipped both times, identical collected and passed selections for every passthrough node

cells (node ids under tests/integration/observability/)	merge base	tip run 1	tip run 2
test_gemini_passthrough_streaming_429_first_frame_reaches_client_while_upstream_holds, ..._async_streaming_429_first_frame..., test_provider_passthrough_streaming_429_first_frame_reaches_client_while_upstream_holds[anthropic,azure,cohere,mistral,openai,config-route]	red x8	green	green
test_gemini_passthrough_quota_wording_in_upstream_body_keeps_passthrough_normalized_error, ..._async_quota_wording_keeps_passthrough_normalized_error	red	green	green
test_gemini_passthrough_streaming_429_client_disconnect_still_logs_failure, ..._streaming_429_upstream_abort_after_first_frame_still_logs_once, test_upstream_abort_after_preview_budget_reaches_client_as_truncated, test_passthrough_keepalive_pings_never_follow_the_upstream_error_body, test_passthrough_streamed_error_headers_do_not_carry_failure_hook_state, ..._streaming_error_with_keepalive_and_slow_headers_logs_failure_once	red	green	green
chaos test_passthrough_disconnect_burst_logs_every_failure_once, test_passthrough_sigterm_drains_reports_parked_on_a_slow_failure_hook, test_passthrough_sigterm_with_graceful_timeout_exits_and_flushes_spend, test_passthrough_abort_after_budget_with_disconnect_during_hook_logs_once, test_passthrough_sigterm_exits_when_late_relay_background_never_returns	red	green	green
unchanged cells: test_llm_endpoint_upstream_quota_429_normalized_error_unchanged[*] (6), test_budget_rejected_call_keeps_budget_normalized_error, 200 SSE, gzip, 4096/4097, control chars, empty body, streaming 500, 404, sync and async SDK errors, message logging off, repeated errors, 64 KB near miss, test_passthrough_upstream_outage_mid_burst_still_logs_errors_once, test_passthrough_error_report_survives_sigterm_after_full_response[hypercorn,uvicorn], ..._sigterm_graceful_window_reports_dispatched_at_preview_budget, ..._sigterm_during_headers_hook_still_dispatches_the_report, test_passthrough_buffered_error_body_redaction_stays_bounded	green	green	green
test_passthrough_worker_sigkill_leaves_sibling_serving_and_logging	flaky red	skip (BUG:, see REVIEWER MUST KNOW)	skip

Two cells outside this PR fail in the same extensions group on the merge base and the tip alike or flake between the two tip runs: test_redis_outage_keeps_serving_in_memory_hits (the box's Redis 6.0.16 rejects the set-proc-title directive the owned-redis helper passes) and one test_presidio_streaming_output cell per run (a periodic GET /v1/models poll hitting the scripted wire). Neither touches this diff and both are green on this PR's CircleCI. Unit tests in the mapped tests/test_litellm files cover the 64/65 boundary, exactly-once report, disconnect warning, post-budget abort reporting before the re-raise, the BackgroundTask handoff, the hook order over a real ASGI call and the abandon close failure; fifteen mutation checks each turn exactly their guard red

Type

🐛 Bug Fix

Caveats (if any)

Medium

• Key fragments shorter than 16 chars (MINIMUM_CUSTOM_KEY_LENGTH) still pass the redactor, same as the base and every other error log

Low

• Bedrock and Vertex passthrough routes have no env-only wire target, so they are covered by the shared stream class and unit tests only

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

REVIEWER MUST KNOW BEFORE APPROVING

• Failure callbacks and the spend row now run after the last byte instead of before the first. A worker SIGKILLed between the last byte and the logging queue flush drops that row: chaos cell test_passthrough_worker_sigkill_leaves_sibling_serving_and_logging green at the base, 3/3 red at the tip, now pytest.skip("BUG: ...") per tests/integration/AGENTS.md. Accepted as-is by yucheng-berri; the fix would put the hook back on the body tail, which is the keepalive-ping regression
• At normal stream end the client gets EOF at once and the failure hook runs afterwards as the response's background task, so that connection is only reusable once the hook finishes; the base held the first byte on the same hook. A connected request waits only on its own hook, never on another request's
• The report runs in a named asyncio task the relay stream dispatches as soon as the 4 KB preview is full (the base awaited the hook at that same point, before headers), so a held-open stream no longer delays it; lifespan shutdown waits up to 10 s for those tasks before the spend flushes and warns if any are still running, a hook longer than that at shutdown writes no row
• Keepalive late relay awaits the report at most 10 s per teardown before leaving it to the shutdown wait. A never-returning hook therefore delays SIGTERM exit about 30 s (two teardown awaits plus the drain) instead of pinning it forever, pinned by test_passthrough_sigterm_exits_when_late_relay_background_never_returns
• Redaction sees the first 4096 chars of the sanitized body plus a 256 char margin: the head is redacted in its own coordinates, a secret straddling the 4 KB cut is replaced by REDACTED from its last word boundary, and text past the cut never enters the output even when an earlier redaction shortens the head. A private key whose END marker falls past the window is cut to REDACTED from its BEGIN header only when base64-shaped key material follows it: after optional whitespace or JSON-escaped \n literals, one 40+ char base64 token with no spaces, two or more consecutive 16+ char base64 tokens (a key wrapped into short lines), a 16+ char token that ends the head, or the header itself ending the head. Prose that merely quotes the header (Vertex "must start with -----BEGIN PRIVATE KEY-----", "header is missing", a URL) reaches the log unchanged; a key wrapped into lines shorter than 16 chars would not be caught (7267f32954, 0005428a76, b84ae93341, 2cace25930)
• The truncation marker is appended whenever the sanitized body exceeded 4096 chars or the raw body exceeded the 64 KiB scan, so it reflects dropped content, not the length of the redacted string
• After an upstream abort past the 4 KB preview the report is awaited inline before the httpx.HTTPError re-raise (the ASGI call raises, so no background would run). With keepalive pings enabled, a slow hook there can emit pings before the abnormal close
• A client disconnect before the 4 KB preview is complete reports only the bytes collected so far
• Every httpx.HTTPError on an error stream logs one read failed warning
• See Behavior changes: failure-hook state is no longer visible to post_call_response_headers_hook on streamed errors

Link to Devin session: https://app.devin.ai/sessions/84cc1558c71a4a789c9bd0dfcef7ff23
Open in Devin Desktop: https://app.devin.ai/desktop/session/84cc1558c71a4a789c9bd0dfcef7ff23?variant=devin
Requested by: @yucheng-berri

---

Note

Medium Risk
Changes core proxy passthrough error streaming, failure-hook timing relative to headers/body, and shutdown/report lifecycle, which affects observability and spend rows under client disconnect and graceful shutdown.

Overview
Fixes passthrough routes that blocked the client while reading up to 4 KB of an upstream error before relaying, and misclassified quota-style upstream bodies as budget errors.

Streaming relay: Upstream 4xx/5xx bodies are forwarded chunk-by-chunk immediately via _PreviewReportingStream. A separate named background report collects a preview for warnings, post_call_failure_hook, and spend metadata—after the stream ends, on preview budget, on disconnect, or via abandon if building the response fails. StreamingResponse gets a BackgroundTask; proxy shutdown waits up to 10s for in-flight report tasks.

Safety and classification: Log/hook text is built from a 64 KiB scan with bounded redact_secrets, PEM dangling-key masking, and truncation markers. normalize_error treats an "upstream passthrough request failed" prefix as 500_UPSTREAM_PASSTHROUGH before quota/budget heuristics. Keepalive late-relay teardown now runs response background work with a bounded wait.

Integration chaos tests and unit coverage exercise disconnect bursts, SIGTERM, slow hooks, and large error bodies; one worker-SIGKILL spend-row test is skipped as a known gap.

^{Reviewed by Cursor Bugbot for commit 5beef60. Bugbot is set up for automated code reviews on this repo. Configure here.}

---

Note

Medium Risk
Changes core proxy passthrough error streaming, failure-hook timing relative to headers/body, and shutdown/report lifecycle, which affects observability and spend rows under client disconnect and graceful shutdown.

Overview
Fixes passthrough routes that blocked the client while reading up to 4 KB of an upstream error before relaying, and misclassified quota-style upstream bodies as budget errors.

Streaming relay: Upstream 4xx/5xx bodies are forwarded chunk-by-chunk immediately via _PreviewReportingStream. A separate named background report collects a preview for warnings, post_call_failure_hook, and spend metadata—after the stream ends, on preview budget, on disconnect, or via abandon if building the response fails. StreamingResponse gets a BackgroundTask; proxy shutdown waits up to 10s for in-flight report tasks.

Safety and classification: Log/hook text is built from a 64 KiB scan with bounded redact_secrets, PEM dangling-key masking, and truncation markers. normalize_error treats an "upstream passthrough request failed" prefix as 500_UPSTREAM_PASSTHROUGH before quota/budget heuristics. Keepalive late-relay teardown now runs response background work with a bounded wait.

Integration chaos tests and unit coverage exercise disconnect bursts, SIGTERM, slow hooks, and large error bodies; one worker-SIGKILL spend-row test is skipped as a known gap.

^{Reviewed by Cursor Bugbot for commit 5beef60. Bugbot is set up for automated code reviews on this repo. Configure here.}

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[greptile-apps]

Confidence Score: 5/5

[Medium risk] Refactors passthrough error handling and logging flow.

The PR appears safe to merge based on the reviewed changes and current thread states.

Summary

The PR relays upstream passthrough error bytes without waiting for a log preview, preserves passthrough error classification, and bounds and redacts reported error text. The latest change prevents an upstream close failure from replacing the exception that abandoned a relay.

_{Reviews (26) · Last reviewed commit: "fix(passthrough): keep the original exce..."}

[devin-ai-integration]

Devin Review found 2 potential issues.

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_passthrough_error_preview_stream (5beef60) with main (d18fcb0)}

[codecov]

Codecov Report

❌ Patch coverage is 94.23077% with 9 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...y/pass_through_endpoints/pass_through_endpoints.py	95.93%	5 Missing ⚠️
litellm/proxy/common_request_processing.py	84.21%	3 Missing ⚠️
litellm/proxy/proxy_server.py	87.50%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[devin-ai-integration]

@greptileai

[yucheng-berri]

@greptileai review latest head

[yucheng-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[yucheng-berri]

@greptileai review latest head

[yucheng-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 5beef60. Configure here.}

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai