fix(responses): surface provider errors and run fallbacks on the completion streaming bridge

[devin-ai-integration]

TLDR

Problem this solves:

• Streaming /v1/responses bridge errors masked the provider error
• Configured fallbacks silently never ran on those errors

How it solves it:

• Initialize _hidden_params on the bridge iterator so the router keeps it unwrapped
• Unwrapped, it passes the Responses fallback check and mid-stream fallbacks run

User Flow

Before: a developer streaming /v1/responses against a Claude deployment with a fallback configured gets an internal error when Anthropic is overloaded, and the fallback never kicks in

1. They send POST https://litellm-domain/v1/responses with {"model":"bridge-claude","input":"hi","stream":true}
2. response.created and response.in_progress arrive
3. The stream ends with {"error": {"message": "litellm.MidStreamFallbackError: litellm.InternalServerError: AnthropicError - Overloaded", "code": "500"}} and the configured fallback model is never called

After: the same request fails over to the configured fallback

1. They send the same POST https://litellm-domain/v1/responses with {"model":"bridge-claude","input":"hi","stream":true}
2. response.created arrives with "metadata":{"model_group":"fallback-echo"}, then response.output_text.delta events with the fallback's text
3. response.completed reports "model":"fallback-echo" and the stream ends with [DONE]
4. With no fallback configured, the stream ends with the clean provider error litellm.InternalServerError: AnthropicError - Overloaded instead of the internal MidStreamFallbackError

Relevant issues

Linear ticket

Resolves LIT-4912

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

LiteLLMCompletionStreamingIterator is the bridge iterator used when a chat-completions provider (Anthropic, Bedrock, Vertex) serves a streaming Responses API request. It subclasses ResponsesAPIStreamingIterator but never runs the base constructor. Staging has since added completed_response = None to it, which fixes the AttributeError half of the ticket, but _hidden_params is still absent. Without it Router.set_response_headers wraps the iterator in HiddenParamsAsyncIteratorWrapper, the wrapped object fails the isinstance(response, BaseResponsesAPIStreamingIterator) check in _aresponses_with_streaming_fallbacks, and the Responses mid-stream fallback path is skipped entirely

Reproducing a real provider emitting Overloaded before the first chunk on demand is not feasible, so the upstream is a local server that deterministically emits an Anthropic message_start followed by an error event; everything else (proxy, router, bridge, fallback selection) is the real code path end to end. bridge-claude has a fallback configured to fallback-echo; bridge-claude-nofallback has none

Before (cd9c410)

Fallback configured

1. curl -sS -N -X POST http://localhost:4000/v1/responses -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{"model":"bridge-claude","input":"hi","stream":true}'
2. Output ends with data: {"error": {"message": "litellm.MidStreamFallbackError: litellm.InternalServerError: AnthropicError - Overloaded", "type": null, "param": null, "code": "500"}} and no HELLO FROM FALLBACK

After (a634cb9)

Fallback configured

1. Same curl with "model":"bridge-claude"
2. Output: response.created with "metadata":{"model_group":"fallback-echo"}, response.output_text.delta with "delta":"HELLO FROM FALLBACK", response.completed with "model":"fallback-echo", then data: [DONE]

No fallback configured

1. Same curl with "model":"bridge-claude-nofallback"
2. Output ends with data: {"error": {"message": "litellm.InternalServerError: AnthropicError - Overloaded", "type": null, "param": null, "code": "500"}}

Type

🐛 Bug Fix

Changes

LiteLLMCompletionStreamingIterator.__init__ now sets self._hidden_params, copied from the underlying CustomStreamWrapper when present. The branch was merged with litellm_internal_staging, which already carries the completed_response = None initialization, so that part of the original fix dropped out of the diff. The other base attributes are read by FallbackResponsesStreamWrapper through getattr(..., default), so their absence was already tolerated

Regression coverage builds a real LiteLLMCompletionStreamingIterator (not a MagicMock standing in for it) with no collected chunks and asserts Router._extract_partial_responses_usage returns None without raising, plus that _hidden_params is present so the router does not pre-wrap and skip the fallback path. The rest of the test file diff is ruff format output at the repo's 120 column limit

Caveats (if any)

Low

• Proof of Fix was captured at a634cb9b62, before the staging merge; the runtime change is identical

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/35369f075dcf46f5b132e31de9cb0124
Open in Devin Desktop: https://app.devin.ai/desktop/session/35369f075dcf46f5b132e31de9cb0124?variant=devin
Requested by: @yassin-berriai

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[greptile-apps]

Greptile Summary

The PR restores required state on the chat-completions-to-Responses streaming bridge.

• Initializes completed_response so pre-first-chunk provider failures preserve the original error path.
• Copies wrapper hidden parameters onto the bridge so routing and fallback wrappers can consume expected metadata.
• Adds regression coverage using a real bridge iterator with no collected chunks.

Confidence Score: 5/5

The PR appears safe to merge with no blocking or non-blocking issues identified.

The added iterator state satisfies the existing router consumers, hidden parameters are copied consistently with sibling fallback wrappers, and the bridge’s normal iteration path does not rely on the other base-constructor fields.

Important Files Changed

Filename	Overview
litellm/responses/litellm_completion_transformation/streaming_iterator.py	Initializes the two bridge attributes required by partial-usage extraction and fallback metadata handling; no actionable defect found.
tests/router_unit_tests/test_router_aresponses_streaming_fallback.py	Adds focused regression coverage for the real bridge iterator’s pre-first-chunk state and hidden-parameter propagation.

_{Reviews (1): Last reviewed commit: "fix(responses): initialize completed_res..." | Re-trigger Greptile}

[devin-ai-integration]

E2E verified on a live proxy that the Responses mid-stream fallback path is restored. Ran the LiteLLM proxy against fake Anthropic upstreams where the primary returns Overloaded after message_start and the fallback returns a full response, exercising streaming POST /v1/responses.

T1, the fix, request {"model":"bridge-claude","input":"hi","stream":true}

"delta":"HELLO FROM FALLBACK"
response.completed ... "model":"fallback-echo"
data: [DONE]

The overloaded primary fails over to the configured fallback, response.completed reports fallback-echo, the stream ends with [DONE], and there is no MidStreamFallbackError or AttributeError leak

T2, no fallback configured, request {"model":"bridge-claude-nofallback","input":"hi","stream":true}

data: {"error": {"message": "litellm.InternalServerError: AnthropicError - Overloaded", ..., "code": "500"}}

The client receives the clean provider error with no internal AttributeError or MidStreamFallbackError

Contrast on base commit cd9c410ae2, broken: the same T1 request leaks litellm.MidStreamFallbackError: ... AnthropicError - Overloaded and never emits HELLO FROM FALLBACK, which confirms this PR is what enables the fallback path

The provider Overloaded is simulated by a local fake upstream since a real provider cannot be coerced into emitting it before the first chunk on demand; the proxy, router, completion-to-responses bridge, and fallback selection are all the real code path

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_lit4912_bridge_iterator_base_state (658b778) with litellm_internal_staging (2ce4e3f)}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ shivamrawat1
❌ yassin-berriai
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[tin-berri]

Closing as stale: no activity in the last 30 days. Feel free to reopen if this is still relevant.