fix(responses): backport mid-stream content_policy_violation fallback routing to stable/1.100.x

[devin-ai-integration]

TLDR

Problem this solves:

• The customer runs v1.100.0, which does not carry the fix(responses): route mid-stream error events through exception_type so content_policy_fallbacks fire #40988 fix
• On that line a mid-stream Responses content_policy_violation is a bare APIError
• So it never reaches content_policy_fallbacks and the client gets the raw error

How it solves it:

• Cherry-picks fix(responses): route mid-stream error events through exception_type so content_policy_fallbacks fire #40988's three commits onto stable/1.100.x with -x
• Each mid-stream error event maps through exception_type, same as non-streaming
• Only content_policy_violation, 429, and 5xx stay fallback-eligible mid-stream
• The router hands a refusal to content_policy_fallbacks; other client errors raise typed

Backport notes

1. Cherry-picked from main in order with -x: 073d4fe, c246372, and fff7a2c, the three commits of fix(responses): route mid-stream error events through exception_type so content_policy_fallbacks fire #40988
2. litellm/responses/streaming_iterator.py and litellm/router.py applied clean, so the source hunks are the same lines as fix(responses): route mid-stream error events through exception_type so content_policy_fallbacks fire #40988
3. tests/test_litellm/test_router.py conflicted because main's copy had drifted past this line; the resolution keeps only the two new tests plus the LiteLLMLogging import they need
4. One extra test-only commit (40f4fa2) adds from importlib import import_module to tests/test_litellm/responses/test_streaming_iterator_error_events.py, since this line's copy of the file did not import it yet and the cherry-picked test raised NameError without it
5. The design carries over unchanged from fix(responses): route mid-stream error events through exception_type so content_policy_fallbacks fire #40988: the error event becomes an SDK-shaped provider exception and goes through exception_type; ContentPolicyViolationError, 429, and 5xx are wrapped in MidStreamFallbackError with the typed original attached; every other 4xx (context window, invalid_prompt) raises directly; the router unwraps the typed original only for the content-policy dispatch
6. Context-window events stay out of mid-stream fallback on purpose: the merge base never fell back on them and issue [Bug]: ErrorEvent ValidationError when OpenAI Responses API returns nested error structure #15785 expects a direct 400 with the provider message preserved, so this backport changes only what fix(responses): route mid-stream error events through exception_type so content_policy_fallbacks fire #40988 changed
7. The same three assertion updates as fix(responses): route mid-stream error events through exception_type so content_policy_fallbacks fire #40988 come along (typed exception instead of bare APIError) in test_streaming_iterator_error_events.py, test_router_aresponses_streaming_fallback.py, and test_openai_responses_api.py; status codes and messages in those tests are unchanged, only the class narrows

User Flow

Before: a developer whose app streams from the Responses API through a v1.100.x proxy loses the whole answer when the model flags the content mid-stream, even with a content-policy fallback model configured

1. The proxy admin configures content-policy fallbacks so requests to gpt-responses-primary fall back to a Gemini model
2. The developer sends POST https://litellm-domain/v1/responses with {"model":"gpt-responses-primary","input":"Write a one sentence greeting.","stream":true}
3. The stream opens with response.created then response.in_progress
4. Before any text arrives, the upstream model sends a mid-stream error event with code content_policy_violation ("This content was flagged for possible cybersecurity risk...")
5. The proxy forwards that error to the client as data: {"error": {"code": "400", ...}} and the stream ends
6. The developer gets no answer at all, even though a content-policy fallback model was configured

After: the same request quietly continues on the configured content-policy fallback model, so the developer gets a complete answer

1. Same content-policy fallback config
2. The developer sends the same POST https://litellm-domain/v1/responses with stream:true
3. The stream opens with response.created then response.in_progress
4. The upstream model sends the same mid-stream error event with code content_policy_violation
5. Instead of surfacing the error, the proxy restarts the request on the Gemini fallback model
6. The developer receives the fallback model's real streamed answer (response.created for the Gemini model, response.output_text.delta, response.completed) then data: [DONE]

Relevant issues

Backport of #40988 (main). Design context for the context-window row: #15785

Affected release

Linear ticket

Resolves LIT-7759

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Shared setup (same for both legs): a two-worker proxy (--num_workers 2) per side, the Before one booted from the stable/1.100.x tip and the After one from this PR's tip, both on the same config. The primaries point at a local stand-in for the OpenAI Responses endpoint that replays the exact mid-stream error event the customer saw (a genuine on-demand mid-stream refusal cannot be triggered reliably), one path per scenario; the fallback is a real Gemini deployment, so every fallback in the After leg is a real provider call. The bridge deployment and the extra scenario deployments only serve the live A/B rows below the main case

model_list:
  - model_name: gpt-responses-primary
    litellm_params:
      model: openai/gpt-5.4
      api_key: sk-shim-dummy-key
      api_base: http://127.0.0.1:43608/cpv/v1
  - model_name: gpt-responses-ctx
    litellm_params:
      model: openai/gpt-5.4
      api_key: sk-shim-dummy-key
      api_base: http://127.0.0.1:43608/ctx/v1
  - model_name: gpt-responses-rl
    litellm_params:
      model: openai/gpt-5.4
      api_key: sk-shim-dummy-key
      api_base: http://127.0.0.1:43608/rl/v1
  - model_name: gpt-responses-ip
    litellm_params:
      model: openai/gpt-5.4
      api_key: sk-shim-dummy-key
      api_base: http://127.0.0.1:43608/ip/v1
  - model_name: gpt-bridge-primary
    litellm_params:
      model: openai/responses/gpt-5.4
      api_key: sk-shim-dummy-key
      api_base: http://127.0.0.1:43608/cpv/v1
  - model_name: gemini-content-fallback
    litellm_params:
      model: gemini/gemini-3.8-flash
      api_key: os.environ/GEMINI_API_KEY

router_settings:
  content_policy_fallbacks:
    - gpt-responses-primary: ["gemini-content-fallback"]
    - gpt-bridge-primary: ["gemini-content-fallback"]

litellm_settings:
  drop_params: true
  callbacks: ["lit7759_cb.instance"]

general_settings:
  master_key: sk-lit7759-proof

The upstream stand-in emits, on the /cpv path:

data: {"type": "response.created", ...}
data: {"type": "response.in_progress", ...}
data: {"type": "error", "error": {"type": "invalid_request_error", "code": "content_policy_violation", "message": "This content was flagged for possible cybersecurity risk. The response was halted mid-stream.", "param": null}, "sequence_number": 2}
data: [DONE]

lit7759_cb.instance is a 12-line CustomLogger whose async_log_failure_event appends the class of kwargs["exception"], its status_code, and the class of its original_exception to a file, so the failure-callback row below is observed rather than reasoned

Before (1dba17b)

1. Send the streaming request to the proxy booted from the stable/1.100.x tip:

curl -sS -N http://127.0.0.1:51773/v1/responses \
  -H "Authorization: Bearer sk-lit7759-proof" -H "Content-Type: application/json" \
  -d '{"model":"gpt-responses-primary","input":"Write a one sentence greeting.","stream":true}'

2. The client gets response.created, response.in_progress, then the raw error, and no fallback runs:

data: {"type":"response.created","response":{"id":"resp_[...]","created_at":1789462079,"model":"gpt-shim","object":"response","output":[],"status":"in_progress"},"sequence_number":0,"model":"gpt-responses-primary"}

data: {"type":"response.in_progress","response":{"id":"resp_[...]","created_at":1789462079,"model":"gpt-shim","object":"response","output":[],"status":"in_progress"},"sequence_number":1,"model":"gpt-responses-primary"}

data: {"error": {"message": "litellm.APIError: This content was flagged for possible cybersecurity risk. The response was halted mid-stream.", "type": null, "param": null, "code": "400"}}

HTTP 200 time 0.313257s

After (40f4fa2)

1. Send the identical request to the proxy booted from this PR's tip:

curl -sS -N http://127.0.0.1:42939/v1/responses \
  -H "Authorization: Bearer sk-lit7759-proof" -H "Content-Type: application/json" \
  -d '{"model":"gpt-responses-primary","input":"Write a one sentence greeting.","stream":true}'

2. After the same two primary events and the same mid-stream error, the request continues on the Gemini fallback and the client receives a complete answer, no error event:

data: {"type":"response.created","response":{"id":"resp_[...]","created_at":1789462079,"model":"gpt-shim","object":"response","output":[],"status":"in_progress"},"sequence_number":0,"model":"gpt-responses-primary"}

data: {"type":"response.in_progress","response":{"id":"resp_[...]","created_at":1789462079,"model":"gpt-shim","object":"response","output":[],"status":"in_progress"},"sequence_number":1,"model":"gpt-responses-primary"}

data: {"type":"response.created","response":{"id":"resp_[...]","created_at":1789462081,"model":"gemini-3.8-flash","object":"response","output":[],...,"status":"in_progress",...},"model":"gpt-responses-primary"}

data: {"type":"response.in_progress","response":{"id":"resp_[...]","created_at":1789462081,"model":"gemini-3.8-flash","object":"response","output":[],...,"status":"in_progress",...},"model":"gpt-responses-primary"}

data: {"type":"response.output_item.added","output_index":0,"item":{"id":"msg_[...]","type":"message","role":"assistant","status":"in_progress","content":[]},"model":"gpt-responses-primary"}

data: {"type":"response.content_part.added","item_id":"msg_[...]","output_index":0,"content_index":0,"part":{"type":"output_text","text":"","annotations":[]},"model":"gpt-responses-primary"}

data: {"type":"response.output_text.delta","item_id":"msg_[...]","output_index":0,"content_index":0,"delta":"Hello, I hope you are having a wonderful day!","model":"gpt-responses-primary"}

data: {"type":"response.output_text.done","item_id":"msg_[...]","output_index":0,"content_index":0,"text":"Hello, I hope you are having a wonderful day!","model":"gpt-responses-primary"}

data: {"type":"response.content_part.done","item_id":"msg_[...]","output_index":0,"content_index":0,"part":{"type":"output_text","text":"Hello, I hope you are having a wonderful day!","annotations":[]},"model":"gpt-responses-primary"}

data: {"type":"response.output_item.done","output_index":0,"sequence_number":1,"item":{"id":"msg_[...]","status":"completed","type":"message","role":"assistant","content":[{"type":"output_text","text":"Hello, I hope you are having a wonderful day!","annotations":[]}]},"model":"gpt-responses-primary"}

data: {"type":"response.completed","response":{"id":"resp_[...]","created_at":1789462081,"metadata":{},"model":"gemini-3.8-flash","object":"response","output":[{"type":"message","id":"msg_[...]","status":"completed","role":"assistant","content":[{"type":"output_text","text":"Hello, I hope you are having a wonderful day!",...}]}],...}}

data: [DONE]

HTTP 200 time 1.821504s

Live A/B for the other surfaces on the same two proxies, run in the same order on both sides: base 1dba17b (the current stable/1.100.x tip, so there is no merge-ref drift to re-run) vs head 40f4fa2. Each row matches the corresponding row #40988 recorded on main

• /v1/responses context-window (context_length_exceeded, "exceeds the context window"): base surfaces litellm.APIError 400 with "type": null, head surfaces litellm.BadRequestError: OpenAIException - Your input exceeds the context window of this model. ... 400 with "type": "invalid_request_error"; both end the stream with no fallback, provider text preserved under the new typed prefix
• /v1/responses rate limit (rate_limit_exceeded): base surfaces litellm.APIError 429 with "type": null, head surfaces litellm.RateLimitError: RateLimitError: OpenAIException - Rate limit reached for gpt-5.4 ... 429 with "type": "throttling_error"; both end the stream with no fallback under this config, provider text preserved
• /v1/responses non-content-policy 400 (invalid_prompt): base surfaces litellm.APIError 400, head surfaces litellm.BadRequestError: OpenAIException - Invalid prompt: your prompt was rejected by the server. 400; both end the stream with no fallback
• /v1/chat/completions streaming over the Responses bridge (openai/responses/gpt-5.4) with the same content-policy event and a content-policy fallback configured: HTTP 400 on both sides, litellm.APIError: This content was flagged ... on base and litellm.BadRequestError: litellm.ContentPolicyViolationError: ContentPolicyViolationError: OpenAIException - This content was flagged ... on head, no fallback on either side (LIT-7757)
• /v1/chat/completions non-streaming over the bridge with the content-policy event: both sides fall back to gemini-3.8-flash with HTTP 200 and a real greeting (10.77s on base, 2.08s on head, both Gemini latency), unchanged
• /v1/messages streaming over the bridge with the content-policy event: HTTP 200, event: message_start with "model": "responses/gpt-5.4", then the stream ends with no error event on both sides, pre-existing (LIT-6871). On the head the first pass of this row landed 2s after the non-streaming bridge refusal and got the router's 5s single-deployment cooldown (429 No deployments available ... Try again in 5 seconds ... cooldown_list=[...]); the base logs the same "Attempting to add ... to cooldown list" for the same refusal but its slower Gemini call put this row 11s later, outside the window. Re-run 60s later on both sides, both attempt 1: HTTP 200 message_start then silent end, identical
• Failure callback (async_log_failure_event reading kwargs["exception"]): base gets APIError 400 for content-policy, context-window, and invalid_prompt, MidStreamFallbackError 429 wrapping APIError for rate limit, and APIError 400 for the chat-bridge stream; head gets MidStreamFallbackError 400 wrapping ContentPolicyViolationError for content-policy on /v1/responses and on the chat and messages bridge streams, BadRequestError 400 for context-window and invalid_prompt, and MidStreamFallbackError 429 wrapping RateLimitError for rate limit; the non-streaming chat-bridge row logs ContentPolicyViolationError on both sides; status codes match on every row

Type

🐛 Bug Fix

Caveats (if any)

Medium

• /v1/chat/completions and /v1/messages share the same mid-stream gap on this line too, out of scope; this backport leaves them as fix(responses): route mid-stream error events through exception_type so content_policy_fallbacks fire #40988 did on main
  • Chat over the Responses bridge still skips content_policy_fallbacks: LIT-7757
  • /v1/messages over the bridge ends silently on any mid-stream error: LIT-6871

Low

• Mid-stream Responses errors now raise the typed exception the non-streaming path raises, so caller code that wrote except litellm.APIError around a Responses stream no longer catches a mid-stream context-window, rate-limit, or other client error, since those typed exceptions do not subclass litellm.APIError. Catch openai.APIError (or the specific type) instead; the HTTP status is unchanged and the provider text stays in the message under the new typed prefix
• Client-visible error message now reads litellm.BadRequestError: OpenAIException - <provider text> (429: litellm.RateLimitError: RateLimitError: OpenAIException - <provider text>, the non-streaming path's own wording) instead of litellm.APIError: <provider text>, and type is populated (invalid_request_error, throttling_error) instead of null; code and the provider text are unchanged, so only a client string-matching the old litellm.APIError: prefix notices
• Failure callbacks and the Prometheus exception_class label now see the typed class (BadRequestError, MidStreamFallbackError carrying a ContentPolicyViolationError) instead of APIError; status codes unchanged
• Providers whose exception mapping on this line does not classify content_policy_violation still raise BadRequestError mid-stream with no fallback, the same as the merge base and as their non-streaming path; openai and azure, the mappings the customer's config uses, map it to ContentPolicyViolationError
• The chat-bridge stream logs its failure callback twice per request on both sides, pre-existing
• A refusal on the non-streaming chat bridge puts that single deployment in the router's default 5s cooldown on both sides (the /v1/messages row above hit it once on the head only because that leg ran sooner), pre-existing and unrelated to the mid-stream path
• Chat-bridge 429 streams keep the litellm.MidStreamFallbackError: prefix, pre-existing
• Responses exceeds the context window wording still maps to BadRequestError: LIT-6511
• Proof injects the provider error event at the network boundary because a genuine mid-stream refusal cannot be triggered on demand; the two-worker proxies, the router, and the Gemini fallback are live, and the injected event uses the {"type": "error", "error": {"type", "code", "message"}} shape captured from a real OpenAI Responses stream
• CircleCI at the tip: 41 of 45 checks green, four jobs red on six tests unrelated to this change. stable/1.100.x lacks the main-only test fixes test(bedrock): drop EOL cohere.command-r-plus-v1:0 from local_testing #39608 (Bedrock cohere.command-r-plus-v1:0 retired, three tests, LIT-6876) and test(responses): expect the 404 OpenAI now returns for an unknown model #39457 (test_bad_request_error expects a 400 where OpenAI now answers 404), and test_router_timeout plus test_openai_embedding_timeouts are the known tiny-timeout flakes (LIT-6670, LIT-6675). None touch the Responses stream or router fallback paths, the branch's own test_router.py and test_streaming_iterator.py cases pass in unit-test, and CircleCI is not a required check on this base
• The base is the current stable/1.100.x tip (1dba17b), so the merged tree equals the head tree; if the line moves before this merges, the main case is re-run on the merged tree

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Medium Risk
Changes mid-stream Responses error types and router fallback dispatch; callers that only catch litellm.APIError may miss new typed exceptions, though HTTP status and fallback behavior are intentional.

Overview
Mid-stream Responses API error and response.failed events no longer become generic litellm.APIError. They are mapped through litellm.exception_type (same as non-streaming), then either raised directly or wrapped in MidStreamFallbackError for router fallbacks.

Fallback eligibility is centralized in _mid_stream_fallback_eligible: ContentPolicyViolationError, 429, and 5xx stay eligible for mid-stream fallback; other 4xx (e.g. context length, invalid prompt) raise the typed client error with no wrapper. Failed-stream logging uses the same mapper so callbacks see the typed exception.

On the router, when a streaming aresponses fallback runs, a MidStreamFallbackError whose original_exception is ContentPolicyViolationError is unwrapped before async_function_with_fallbacks_common_utils, so content_policy_fallbacks dispatch matches the refusal type instead of the envelope.

Tests were updated/added for typed exceptions, content-policy mid-stream wrapping, router content-policy fallback routing, and continuation input after partial output.

^{Reviewed by Cursor Bugbot for commit 40f4fa2. Bugbot is set up for automated code reviews on this repo. Configure here.}

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[greptile-apps]

Greptile Summary

This PR maps Responses API stream error events through LiteLLM's standard exception mapping and enables content-policy fallback routing for mid-stream failures

• Content-policy errors are wrapped for mid-stream handling and unwrapped when selecting the dedicated fallback list
• Other error events now expose the same typed exceptions as corresponding non-streaming failures
• Regression tests cover OpenAI and Azure content-policy events, generated-content preservation, fallback selection, and failure logging

Confidence Score: 5/5

The PR appears safe to merge, with regression coverage for the corrected mid-stream content-policy fallback path

No actionable failures remain after checking exception status preservation, fallback dispatch, stream-state handling, logging compatibility, and modified test integrity

Important Files Changed

Filename	Overview
litellm/responses/streaming_iterator.py	Adds typed error-event mapping and fallback eligibility for content-policy, rate-limit, and server failures
litellm/router.py	Passes the inner content-policy exception to fallback dispatch while retaining stream continuation state
tests/test_litellm/responses/test_streaming_iterator_error_events.py	Adds focused coverage for typed error mapping, content-policy fallback wrapping, partial output, and failure logging
tests/test_litellm/test_router.py	Verifies that real transformed Responses events select the configured content-policy fallback
tests/router_unit_tests/test_router_aresponses_streaming_fallback.py	Updates the router fallback assertion to require the mapped rate-limit exception
tests/llm_responses_api_testing/test_openai_responses_api.py	Updates the integration expectation to the typed bad-request contract

_{Reviews (1): Last reviewed commit: "test(responses): import import_module in..." | Re-trigger Greptile}

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 40f4fa2. Configure here.}

[mateo-berri]

LGTM