fix(responses): route mid-stream error events through exception_type so content_policy_fallbacks fire

[devin-ai-integration]

TLDR

Problem this solves:

• Mid-stream Responses API errors were all classified as APIError
• So a mid-stream content_policy_violation never reached content_policy_fallbacks
• The client got the raw error instead of a fallback answer

How it solves it:

• Route each error event's code and status through the existing exception_type mapping
• Each event surfaces as the same typed exception the non-streaming path already raises
• Only content_policy_violation, 429, and 5xx stay fallback-eligible mid-stream
• The router sends a refusal to content_policy_fallbacks; other client errors raise their typed exception directly

Design notes

1. No existing PR covered this, so this starts from scratch on a new branch
2. The error event is turned into an SDK-shaped provider exception (status code plus a flat {"message","type","code"} body) and passed through exception_type, so the streaming path sees exactly what the non-streaming path sees instead of a second mapping table
3. Mid-stream fallback eligibility is decided from the mapped exception: ContentPolicyViolationError, or a status of 429 or 5xx, is wrapped in MidStreamFallbackError with the typed original attached (preserving generated_content and is_pre_first_chunk) so the router's mid-stream machinery can retry; every other exception, meaning non-retriable 4xx like context-window and invalid_prompt, is raised directly with no wrapping
4. The router unwraps that typed original only for the content-policy fallback dispatch, matching the existing Anthropic precedent; the dispatch branch keys on the trigger's own type, so a refusal must arrive as ContentPolicyViolationError rather than inside its envelope
5. Design decision (why context-window is not fallback-eligible): the merge base never fell back on a mid-stream context-window error, and issue [Bug]: ErrorEvent ValidationError when OpenAI Responses API returns nested error structure #15785 expects a direct-SDK 400 with the provider message preserved and no wrapping. Wrapping it would be new behavior and out of scope for this ticket, so a context-window event raises ContextWindowExceededError(400) directly, exactly like the non-streaming path. The only end-user-observable behavior change versus the merge base is that content_policy_violation now falls back; the other rows keep their status code and only swap the class prefix in the error message
6. context_length_exceeded maps by message substring, exactly like the non-streaming path, since the bare code alone is not treated as a context-window error there either

Test expectations updated for the typed-exception behavior

Routing each event through exception_type narrows the exception the mid-stream path raises from a bare litellm.APIError to the specific typed exception the non-streaming path raises, so three existing assertions were updated to the now-correct type (each is a regression assertion for the new contract, not a weakened test):

• tests/test_litellm/responses/test_streaming_iterator_error_events.py: a context_length_exceeded event is a non-retriable client error, now raised directly as ContextWindowExceededError(400) and no longer wrapped in MidStreamFallbackError, preserving the issue [Bug]: ErrorEvent ValidationError when OpenAI Responses API returns nested error structure #15785 direct-SDK 400 contract instead of adding context-window fallback scope
• tests/router_unit_tests/test_router_aresponses_streaming_fallback.py: a rate_limit_exceeded event now routes through exception_type, so the wrapped original_exception is the typed litellm.RateLimitError(429) the non-streaming path raises rather than a bare litellm.APIError; status stays 429 and the fallback still fires
• tests/llm_responses_api_testing/test_openai_responses_api.py: an oversized-input context-window event now surfaces the typed litellm.BadRequestError(400) the non-streaming path raises rather than a bare litellm.APIError; still 400, still no MidStreamFallbackError wrapping, message preserved per issue [Bug]: ErrorEvent ValidationError when OpenAI Responses API returns nested error structure #15785

User Flow

Before: a developer whose app streams from the Responses API loses the whole answer when the model flags the content mid-stream, even with a content-policy fallback model configured

1. The proxy admin configures content-policy fallbacks so requests to gpt-responses-primary fall back to a Gemini model
2. The developer sends POST https://litellm-domain/v1/responses with {"model":"gpt-responses-primary","input":"Write a one sentence greeting.","stream":true}
3. The stream opens with response.created then response.in_progress
4. Before any text arrives, the upstream model sends a mid-stream error event with code content_policy_violation ("This content was flagged for possible cybersecurity risk...")
5. The proxy forwards that error to the client as data: {"error": {"code": "400", ...}} and the stream ends
6. The developer gets no answer at all, even though a content-policy fallback model was configured

After: the same request quietly continues on the configured content-policy fallback model, so the developer gets a complete answer

1. Same content-policy fallback config
2. The developer sends the same POST https://litellm-domain/v1/responses with stream:true
3. The stream opens with response.created then response.in_progress
4. The upstream model sends the same mid-stream error event with code content_policy_violation
5. Instead of surfacing the error, the proxy restarts the request on the Gemini fallback model
6. The developer receives the fallback model's real streamed answer (response.created for the Gemini model, response.output_text.delta, response.completed) then data: [DONE]

Relevant issues

Affected release

Linear ticket

Resolves LIT-7403

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Shared setup (same for both legs), a two-worker proxy (--num_workers 2) with content-policy fallbacks configured. The primary points at a local stand-in for the OpenAI Responses endpoint that replays the exact mid-stream error event the customer saw (a genuine on-demand mid-stream refusal cannot be triggered reliably); the fallback is a real Gemini deployment, so the After leg is a real provider call.

model_list:
  - model_name: gpt-responses-primary
    litellm_params:
      model: openai/gpt-5.4
      api_key: sk-shim-dummy-key
      api_base: http://127.0.0.1:58122/v1
  - model_name: gemini-content-fallback
    litellm_params:
      model: gemini/gemini-2.5-flash
      api_key: os.environ/GEMINI_API_KEY

router_settings:
  content_policy_fallbacks:
    - gpt-responses-primary: ["gemini-content-fallback"]

The upstream stand-in emits, mid-stream:

data: {"type": "response.created", ...}
data: {"type": "response.in_progress", ...}
data: {"type": "error", "error": {"type": "invalid_request_error", "code": "content_policy_violation", "message": "This content was flagged for possible cybersecurity risk. The response was halted mid-stream."}, "sequence_number": 2}

Before (30f33a9)

1. Send the streaming request:

curl -sS -N http://127.0.0.1:PROXY/v1/responses \
  -H "Authorization: Bearer sk-lit7403-repro" -H "Content-Type: application/json" \
  -d '{"model":"gpt-responses-primary","input":"Write a one sentence greeting.","stream":true}'

2. The client gets response.created, response.in_progress, then the raw error, and no fallback runs:

data: {"type":"response.created","response":{"id":"resp_[...]","model":"gpt-shim","status":"in_progress",...},"sequence_number":0,"model":"gpt-responses-primary"}

data: {"type":"response.in_progress","response":{"id":"resp_[...]","model":"gpt-shim","status":"in_progress",...},"sequence_number":1,"model":"gpt-responses-primary"}

data: {"error": {"message": "litellm.APIError: This content was flagged for possible cybersecurity risk. The response was halted mid-stream.", "type": null, "param": null, "code": "400"}}

HTTP 200

After (fff7a2c)

1. Send the identical request to the fixed proxy:

curl -sS -N http://127.0.0.1:PROXY/v1/responses \
  -H "Authorization: Bearer sk-lit7403-repro" -H "Content-Type: application/json" \
  -d '{"model":"gpt-responses-primary","input":"Write a one sentence greeting.","stream":true}'

2. After the same two primary events and the same mid-stream error, the request continues on the Gemini fallback and the client receives a complete answer, no error event:

data: {"type":"response.created","response":{"id":"resp_[...]","model":"gemini-2.5-flash","status":"in_progress",...},"model":"gpt-responses-primary"}

data: {"type":"response.output_text.delta","item_id":"msg_[...]","delta":"Hello!","model":"gpt-responses-primary"}

data: {"type":"response.output_text.done","text":"Hello!","model":"gpt-responses-primary"}

data: {"type":"response.completed","response":{"id":"resp_[...]","model":"gemini-2.5-flash","status":"completed","output":[{"type":"message","role":"assistant","content":[{"type":"output_text","text":"Hello!"}]}],"usage":{"input_tokens":7,"output_tokens":25,"total_tokens":32,"cost":0.0000646}},"model":"gemini-content-fallback"}

data: [DONE]

HTTP 200

Live A/B for the other surfaces on a second rig of the same shape (two workers, stand-in upstream, real Gemini gemini-3.8-flash fallback): base 30f33a9 vs head fff7a2c, plus the head merged with current main (8481bc2), which behaved identically to the head on every row

• /v1/responses context-window (context_length_exceeded, "exceeds the context window"): base surfaces litellm.APIError 400, head surfaces litellm.BadRequestError 400; both end the stream with no fallback, provider text preserved under the new typed prefix
• /v1/responses rate limit (rate_limit_exceeded): base surfaces litellm.APIError 429, head surfaces litellm.RateLimitError 429; both end the stream with no fallback under this config, provider text preserved under the new typed prefix
• /v1/responses non-content-policy 400 (invalid_prompt): base surfaces litellm.APIError 400, head surfaces litellm.BadRequestError 400; both end the stream with no fallback, provider text preserved under the new typed prefix
• /v1/chat/completions streaming over the Responses bridge (openai/responses/gpt-5.4) with the same content-policy event and a content-policy fallback configured: HTTP 400 on both sides, litellm.APIError on base and litellm.BadRequestError: litellm.ContentPolicyViolationError: ... on head, no fallback on either side (LIT-7757)
• /v1/chat/completions streaming over the bridge with rate_limit_exceeded: HTTP 429 on both sides and both bodies carry the litellm.MidStreamFallbackError: prefix, pre-existing
• /v1/messages streaming over the bridge with the content-policy event: HTTP 200, event: message_start, then the stream ends with no error event on both sides, pre-existing (LIT-6871)
• /v1/chat/completions non-streaming over the bridge with the content-policy event: both sides fall back to Gemini with HTTP 200, unchanged
• Failure callback (async_log_failure_event reading kwargs["exception"]): base gets APIError 400 for content-policy, context-window, and invalid_prompt and MidStreamFallbackError 429 for rate limit; head gets MidStreamFallbackError 400 for content-policy (the typed ContentPolicyViolationError inside it), BadRequestError 400 for the other two, and MidStreamFallbackError 429 unchanged; status codes match on every row

Type

🐛 Bug Fix

Caveats (if any)

Medium

• /v1/chat/completions and /v1/messages share the same mid-stream gap, out of scope; this PR leaves them unchanged
  • Chat over the Responses bridge still skips content_policy_fallbacks: LIT-7757
  • /v1/messages over the bridge ends silently on any mid-stream error: LIT-6871

Low

• Mid-stream Responses errors now raise the typed exception the non-streaming path raises, so caller code that wrote except litellm.APIError around a Responses stream no longer catches a mid-stream context-window, rate-limit, or other client error, since those typed exceptions do not subclass litellm.APIError. Catch openai.APIError (or the specific type) instead; the HTTP status is unchanged and the provider text stays in the message under the new typed prefix
• Client-visible error message now reads litellm.BadRequestError: OpenAIException - <provider text> (429: litellm.RateLimitError: RateLimitError: OpenAIException - <provider text>, the non-streaming path's own wording) instead of litellm.APIError: <provider text>, and type is populated (invalid_request_error, throttling_error) instead of null; code and the provider text are unchanged, so only a client string-matching the old litellm.APIError: prefix notices
• Failure callbacks and the Prometheus exception_class label now see the typed class (BadRequestError, MidStreamFallbackError carrying a ContentPolicyViolationError) instead of APIError; status codes unchanged
• Providers whose exception mapping does not classify content_policy_violation (openrouter, databricks, manus, bedrock_mantle) still raise BadRequestError mid-stream with no fallback, the same as the merge base and as their non-streaming path; openai, azure, xai, litellm_proxy, hosted_vllm, fireworks_ai, volcengine, github_copilot, and chatgpt all map it to ContentPolicyViolationError
• Chat-bridge 429 streams keep the litellm.MidStreamFallbackError: prefix, pre-existing
• Responses exceeds the context window wording still maps to BadRequestError: LIT-6511
• Proof injects the provider error event at the network boundary because a genuine mid-stream refusal cannot be triggered on demand; the two-worker proxy, the router, and the Gemini fallback are live, and the injected event uses the {"type": "error", "error": {"type", "code", "message"}} shape captured from a real OpenAI Responses stream
• The customer runs v1.100.0; a backport to that stable line will follow

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Medium Risk
Changes mid-stream error classification and router fallback dispatch for Responses streaming; behavior shifts for callers catching only litellm.APIError, while HTTP status and messages for most cases stay the same.

Overview
Mid-stream Responses API errors now use the same typed exceptions as the non-streaming path, via a shared _map_error_event_exception that feeds provider error fields through litellm.exception_type instead of always raising litellm.APIError.

Fallback eligibility is driven by _mid_stream_fallback_eligible: ContentPolicyViolationError, HTTP 429, and 5xx are wrapped in MidStreamFallbackError (with generated_content preserved); other client 4xx types raise directly with no wrap—so context-window and similar errors stay non-fallback, matching issue #15785.

The router unwraps MidStreamFallbackError.original_exception when it is a ContentPolicyViolationError before async_function_with_fallbacks_common_utils, so content_policy_fallbacks dispatch matches on the refusal type instead of the envelope.

Failed-stream logging uses the same mapper, so integrations see typed failures (e.g. RateLimitError, ContentPolicyViolationError) rather than generic API errors. Tests were updated/added for typed exceptions and content-policy mid-stream fallback routing.

^{Reviewed by Cursor Bugbot for commit fff7a2c. Bugbot is set up for automated code reviews on this repo. Configure here.}

• fff7a2c passes /live-pr-risk

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_responses_stream_error_exception_mapping (fff7a2c) with main (30f33a9)}

[greptile-apps]

Greptile Summary

This PR maps Responses API mid-stream error events to normalized LiteLLM exception types and lets content-policy failures reach their configured fallback path

• Converts error-event fields into the same typed exceptions used by non-streaming requests
• Unwraps content-policy exceptions before fallback selection
• Preserves direct context-window error behavior for SDK callers
• Adds focused iterator, router, logging, and integration regression coverage

Confidence Score: 5/5

The PR appears safe to merge, with the content-policy fallback path covered and no outstanding findings

Mid-stream Responses errors now use established exception mapping, content-policy errors reach the dedicated fallback dispatch, and context-window errors retain the tested direct SDK contract. The earlier docstring finding was withdrawn after Greptile accepted mateo-berri's explanation, and the mutable collector thread was resolved

Important Files Changed

Filename	Overview
litellm/responses/streaming_iterator.py	Maps streamed provider errors through normalized exception handling while wrapping only fallback-eligible failures
litellm/router.py	Unwraps content-policy failures so the router selects the dedicated content-policy fallback list
tests/test_litellm/responses/test_streaming_iterator_error_events.py	Covers typed exception mapping, fallback eligibility, partial output, and failure logging
tests/test_litellm/test_router.py	Verifies an actual Responses error event selects the configured content-policy fallback
tests/router_unit_tests/test_router_aresponses_streaming_fallback.py	Updates fallback assertions to require the normalized rate-limit exception
tests/llm_responses_api_testing/test_openai_responses_api.py	Updates integration expectations to the normalized bad-request exception contract

_{Reviews (3): Last reviewed commit: "fix(responses): keep context-window even..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Unneeded test docstrings

These docstrings restate straightforward test behavior, violating the repository’s comment restrictions. Remove them before merging; the pattern appears in related tests

Context Used: CLAUDE.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[mateo-berri]

Each docstring documents the exact regression mechanism the test guards, the complex business logic case CLAUDE.md allows comments for, not a restatement of behavior

[greptile-apps]

You're right. Both docstrings explain non-obvious regression mechanisms and fit CLAUDE.md's exception for concise complex business logic; I withdraw this comment

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit fff7a2c. Configure here.}

[mateo-berri]

LGTM