fix(router): mid-stream fallback 400s on models without assistant prefill (Claude Sonnet 4.6+) - #30242
Conversation
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
ea7f9ac to
de98087
Compare
Greptile SummaryThis PR fixes deterministic mid-stream fallback failures for Claude Sonnet/Opus 4.6+ by detecting when the primary model or any configured fallback target explicitly rejects assistant prefill, then substituting the documented user-message continuation pattern instead of the legacy prefilled-assistant-message resume.
Confidence Score: 5/5Safe to merge — all previously flagged gaps are closed and no new regressions introduced. The registry sweep is comprehensive: all 27 provider-specific sonnet-4-6/opus-4-6 entries in both JSON files now carry the flag, including the previously-missed snowflake entry. The routing logic is conservative (only deviates from the existing prefill path when the flag is explicitly No files require special attention.
|
| Filename | Overview |
|---|---|
| litellm/router_utils/fallback_event_handlers.py | Adds build_mid_stream_continuation_messages helper that routes to user-message continuation for prefill-rejecting models; handles flat-string and dict-format fallback lists correctly with mutation-safe copy for dict path. |
| litellm/router.py | Both sync and async mid-stream fallback injection sites replaced with build_mid_stream_continuation_messages; identical change correctly propagated to both paths. |
| model_prices_and_context_window.json | All 27 sonnet-4-6/opus-4-6 entries across every provider (bedrock, azure_ai, vertex_ai, openrouter, vercel, perplexity, github_copilot, snowflake) now have supports_assistant_prefill: false; snowflake entry newly added. |
| litellm/model_prices_and_context_window_backup.json | Registry backup kept in sync with root cost map; all sonnet-4-6/opus-4-6 entries updated to supports_assistant_prefill: false including newly added snowflake entry. |
| tests/test_litellm/router_utils/test_fallback_event_handlers.py | New test file with 10 offline (mock-pinned) tests covering prefill-rejecting primaries and fallback targets, legacy prefill preservation, non-mutation of live fallback lists, and flat-string rejecter at non-first position. |
| tests/test_litellm/test_claude_sonnet_4_6_config.py | Updates the capability pin assertion from is True to is False — correcting a wrong baseline expectation to match the Anthropic migration reality, not weakening coverage. |
Reviews (9): Last reviewed commit: "fix(lint): use X | None over Optional in..." | Re-trigger Greptile
Greptile SummaryThis PR fixes mid-stream fallback failures on Claude Sonnet 4.6 / Opus 4.6+, where Anthropic's removal of assistant prefill caused every fallback hop to return a hard 400 error. The fix adds a
Confidence Score: 3/5The core routing logic and test coverage are solid. The fix is incomplete in the registry: openrouter/anthropic/claude-sonnet-4.6 still has true in both JSON files, and snowflake/claude-sonnet-4-6 is missing the field — both deployment routes remain broken after this merges. The build_mid_stream_continuation_messages helper and router call-sites are well-implemented and safe. The registry update that makes the whole fix work is incomplete: openrouter/anthropic/claude-sonnet-4.6 retains supports_assistant_prefill: true in both JSON files and snowflake/claude-sonnet-4-6 is missing the field entirely, reproducing the exact 400 errors the PR aims to eliminate for those two deployment routes. model_prices_and_context_window.json and litellm/model_prices_and_context_window_backup.json — the openrouter/anthropic/claude-sonnet-4.6 entry needs supports_assistant_prefill flipped to false in both, and snowflake/claude-sonnet-4-6 needs the field added with false in the primary JSON.
|
| Filename | Overview |
|---|---|
| model_prices_and_context_window.json | 10 of 12 sonnet-4-6 entries updated to supports_assistant_prefill=false; openrouter/anthropic/claude-sonnet-4.6 still has true and snowflake/claude-sonnet-4-6 is missing the field entirely — both leave those routing paths broken. |
| litellm/model_prices_and_context_window_backup.json | 10 entries updated to false; openrouter/anthropic/claude-sonnet-4.6 entry still has supports_assistant_prefill=true (snowflake entry is not present in backup so no issue there). |
| litellm/router_utils/fallback_event_handlers.py | New build_mid_stream_continuation_messages helper correctly routes to user-message continuation when registry explicitly marks supports_assistant_prefill=false; safe fallback to legacy prefill for all other cases. |
| litellm/router.py | Both sync and async mid-stream fallback injection sites now share the new helper, deduplicating the continuation logic cleanly. |
| litellm/utils.py | New supports_assistant_prefill() function added but not exported from litellm/init.py, leaving it unreachable through the public module API. |
| tests/test_litellm/router_utils/test_fallback_event_handlers.py | New test file with 8 cases covering both the user-continuation and legacy-prefill paths; uses monkeypatching to pin the local registry — no real network calls. |
Comments Outside Diff (3)
-
model_prices_and_context_window.json, line 27632 (link)openrouter/anthropic/claude-sonnet-4.6missed in the registry sweepThis entry still has
"supports_assistant_prefill": truewhile every other Sonnet 4.6 variant (bare, regional, Vertex, Azure AI, etc.) was flipped tofalse. Any fallback chain whosemodel_groupisopenrouter/anthropic/claude-sonnet-4.6will still hit the legacy assistant-prefill path and receive the same 400 error this PR is fixing for all other deployment routes. The same entry inlitellm/model_prices_and_context_window_backup.jsonalso needs the corresponding update. -
model_prices_and_context_window.json, line 41975-41989 (link)The
snowflake/claude-sonnet-4-6entry has nosupports_assistant_prefillfield at all. Because_prefill_explicitly_unsupportedreturnsTrueonly when the field is explicitlyfalse(model_info.get("supports_assistant_prefill") is False), a missing field evaluates toNone is False→False, so the legacy prefill path is used. Snowflake-routed Sonnet 4.6 fallbacks would still hit the 400 error this PR is meant to eliminate. -
litellm/utils.py, line 2451-2472 (link)supports_assistant_prefillnot exported from public APIThe new
supports_assistant_prefill()function is added toutils.pybut is not wired intolitellm/__init__.py. All othersupports_*helpers (supports_function_calling,supports_url_context,supports_web_search, etc.) are listed there asCallable[..., bool]entries. Without this export, the function is unreachable through the module's public surface — making the PR description's claim of "public accessor" inaccurate. Additionally,_prefill_explicitly_unsupportedbypasses this function entirely and callsget_model_infodirectly, so the new function serves neither the internal fix nor an accessible external caller in its current state.
Reviews (1): Last reviewed commit: "fix(router): mid-stream fallback 400s on..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
|
@greptileai Both summary findings addressed: the live-fallbacks mutation was fixed in 4fdf8a6 (shallow copy + regression test asserting flat-string and dict formats stay unmutated — your summary reviewed cd53134, one commit earlier), and the three dot-variant registry keys (openrouter/anthropic/claude-sonnet-4.6, openrouter/anthropic/claude-opus-4.6, vercel_ai_gateway/anthropic/claude-opus-4.6) are flipped in 68df6e1 with a dot-variant test param added. |
|
All three findings are verified resolved in the current HEAD:
The implementation is clean. LGTM. |
|
Want your agent to iterate on Greptile's feedback? Try greploops. |
|
@cwang-otto can you please get greptile score to 5/5 and also please share a before and after to understand the fix betterly for maintainers. Thanks! |
@Sameerlite. Ah. thanks. Paste the before and after here first: Verification (real API calls, otto's anthropic→vertex fallback chain)Local harness (
Also the score is 5/5 now ! |
|
Fixed in 81527f1. Re-swept the full 4-6/4-7 family (dash + dot keys, both registry files): the flagged perplexity/anthropic/claude-opus-4-7 plus two more stragglers that landed after the original sweep — perplexity/anthropic/claude-opus-4-6 and github_copilot/claude-opus-4.6-fast (the latter is mode:chat, so real fallback exposure). Zero entries in the family now lack supports_assistant_prefill:false. |
81527f1 to
35a55a8
Compare
a2f00da to
43a3a11
Compare
|
Thanks for the PR! A couple of things to get this over the finish line:
Triggering Greptile for a code review in the meantime: |
Ah looks like transient test infra issue. Re-triggering the CI tests. |
1975037 to
563a562
Compare
…fill support Anthropic removed assistant message prefill starting with Claude Sonnet 4.6 / Opus 4.6 (per the official migration guide it returns a 400: 'This model does not support assistant message prefill'). The mid-stream fallback resume (BerriAI#13149) appends the partial response as a prefixed assistant message, so every mid-stream fallback for these models fails deterministically across the whole fallback chain - converting recoverable stream timeouts into hard failures. - registry: supports_assistant_prefill=false for all *sonnet-4-6* entries in both cost maps (opus-4-6/4-7/4-8/fable entries were already false) - router_utils: build_mid_stream_continuation_messages - when the registry explicitly marks prefill unsupported, the partial response rides a trailing USER message (the continuation pattern Anthropic's migration guide documents); all other models keep the existing prefill-resume behavior unchanged - router: both (sync + async) injection sites now share the helper - utils: public supports_assistant_prefill() accessor (the registry field existed with 240 entries but had no supports_* accessor)
…ic accessor, update capability pin test - Greptile: the same continuation messages go to every fallback target, so the user-message continuation now engages when the primary OR any configured fallback target for the group is explicitly marked prefill-unsupported (via get_fallback_model_group) - Greptile: drop the supports_assistant_prefill() public accessor - _supports_factory's absent-key=False default contradicts this feature's absent-key=legacy-prefill routing; the routing reads get_model_info directly - CI: test_claude_sonnet_4_6_config pinned the stale capability value; prefill was removed in Sonnet 4.6 so the pin is now False
…ps string-format entries)
…_ai_gateway) The sweep matched the dash form (sonnet-4-6) and missed openrouter/anthropic/claude-sonnet-4.6, openrouter/anthropic/claude-opus-4.6, vercel_ai_gateway/anthropic/claude-opus-4.6 (dot form). Test param added for the dot variant.
… github_copilot opus-4.6-fast lack supports_assistant_prefill:false
…ages Swap List[...] -> list[...] in the new helper so the change adds 0 UP006 violations; the litellm_internal_staging strict-rule budget is tighter than litellm_oss_branch and flagged the 4 added annotations.
…ry flag Greptile findings: 1. build_mid_stream_continuation_messages now scans every entry of a flat string-format fallback list directly. get_fallback_model_group pops a single string mid-iteration, so a prefill-rejecting model at a non-first position was never capability-checked; since the continuation is built once and reused across all fallback hops, the chain still 400'd when it reached that model. Dict/standard format keeps using get_fallback_model_group (already returns the full group). 2. snowflake/claude-sonnet-4-6 gets supports_assistant_prefill:false in both the root map and the bundled backup (it existed only in root before; the backup is the offline/test source), completing the 4-6 sweep. Tests: +snowflake registry param, +flat-string non-first-position -> user continuation, +flat-string all-supporting -> legacy prefill.
Rebased onto current litellm_internal_staging, whose strict-rule budget now tracks UP045 (non-pep604 annotations). Convert the 4 Optional[...] annotations the helper added to X | None so the change stays within budget.
563a562 to
fd2fce1
Compare
|
@greptileai Both findings from the 4/5 review are addressed (head 1. Flat string-format fallback gap. 2. All lint gates (strict-rule budget, type-discipline, basedpyright) and the unit suite pass. |
|
@Sameerlite updated the PR. can take another look. Thanks! |
|
@mateo-berri can you review this please? thanks |
|
Taking a look... |
|
hey hey |
|
@mateo-berri bump on this |
Removing the continuation-prompt fallback (retrying with the partial response as a prefixed assistant message) so a stream failing after partial content always re-raises instead was a scope decision beyond what this PR's title/issue (BerriAI#31874) describe, and it directly conflicts with BerriAI#30242/BerriAI#30743, which are already fixing the same code path for Anthropic's removal of assistant-message prefill on Sonnet 4.6+/Opus 4.6+. Landing this PR's version first would delete the branch those PRs are patching; landing theirs first would have this PR undo their fix on rebase. Restores the original prefill-based continuation-resume behavior (including the is_pre_first_chunk guard already in litellm_internal_staging) in both _acompletion_streaming_iterator and _completion_streaming_iterator, and removes _stream_chunks_have_generated_content along with the tests that only existed to cover the guard. This PR now only touches the deferred-stream eager-fetch fix and the header-stripping fixes; the non-text-content re-raise idea becomes a follow-up PR built on top of whichever of BerriAI#30242/BerriAI#30743 lands.
Removing the continuation-prompt fallback (retrying with the partial response as a prefixed assistant message) so a stream failing after partial content always re-raises instead was a scope decision beyond what this PR's title/issue (BerriAI#31874) describe, and it directly conflicts with BerriAI#30242/BerriAI#30743, which are already fixing the same code path for Anthropic's removal of assistant-message prefill on Sonnet 4.6+/Opus 4.6+. Landing this PR's version first would delete the branch those PRs are patching; landing theirs first would have this PR undo their fix on rebase. Restores the original prefill-based continuation-resume behavior (including the is_pre_first_chunk guard already in litellm_internal_staging) in both _acompletion_streaming_iterator and _completion_streaming_iterator, and removes _stream_chunks_have_generated_content along with the tests that only existed to cover the guard. This PR now only touches the deferred-stream eager-fetch fix and the header-stripping fixes; the non-text-content re-raise idea becomes a follow-up PR built on top of whichever of BerriAI#30242/BerriAI#30743 lands.
…errors in _acompletion fallback path (#34627) * fix(router): eagerly fetch deferred stream to surface HTTP errors in fallback path Providers like Vertex AI and Bedrock defer their HTTP call until the first __anext__ on the returned CustomStreamWrapper (completion_stream=None, make_call set). Errors raised inside __anext__ (e.g. 429, 503) escape the _acompletion try/except block, so fail_calls is never incremented, deployment cooldown does not fire, and the standard fallback chain is bypassed. Call fetch_stream() on the wrapper before delegating to _acompletion_streaming_iterator when completion_stream is None and make_call is set. Any HTTP error now propagates through _acompletion's except block, increments fail_calls, and enters the normal retry/fallback chain. Strip Content-Length, Transfer-Encoding, Content-Encoding, and Content-Type from exception headers at the same point to prevent HTTP framing mismatches when LiteLLM builds its own error response body. Add a re-raise guard in _acompletion_streaming_iterator (async and sync paths) so MidStreamFallbackError with already-generated content re-raises to the caller instead of silently injecting a continuation prompt into a fresh request to a fallback model. Apply logging cleanup in async_function_with_fallbacks_common_utils: use %s-style formatting and exc_info=True instead of f-strings with traceback.format_exc(). * fix(router): undo success_calls on deferred-stream fetch failure; broaden header strip * fix(router): extract header-strip helper to keep _acompletion under strict C901 threshold * test(router): add unit tests for _strip_http_framing_headers to satisfy router coverage gate * test(router): add sync _completion_streaming_iterator re-raise test for mid-chunk MidStreamFallbackError * fix(router): restore Fallbacks context in no-fallback log; document update_team mcp_rpm_limit The log and debug message when no fallback model group is found was missing the Fallbacks list, making it hard to understand why routing failed. Also adds the missing mcp_rpm_limit documentation to update_team to fix the documentation_test_api_docs CI check. * fix(router): preserve original traceback in deferred stream fetch error re-raise Using bare `raise` instead of `raise fetch_err` keeps the full inner traceback from fetch_stream() intact so the error origin is visible in logs and debuggers without being anchored to this line. * style(test): restore black-style formatting in test_router.py An earlier commit on this branch collapsed the file's pre-existing multi-line formatting into single lines while adding the deferred-stream tests, producing a diff full of unrelated reformatting noise. Restores the untouched code to its original formatting; the actual new/changed test content is unaffected (verified via AST comparison). * fix(router): re-raise mid-stream fallback on any generated content, not just text The re-raise guard added for MidStreamFallbackError only checked generated_content, which tracks text deltas alone. A stream that emitted a tool-call or reasoning-only chunk before failing had generated_content="" despite already streaming to the client, so the router silently retried and the client saw duplicated/inconsistent output. The guard now also inspects the wrapper's raw chunks for tool_calls/reasoning_content. Also moves the deferred-stream HTTP-framing-header stripping out of Router._acompletion into the proxy's _handle_llm_api_exception: Router is used directly as an SDK as well as by the proxy, and stripping headers there dropped legitimate provider metadata (content-type, proxy-authenticate) for direct SDK callers who never see the proxy's own response construction. schema.d.ts regenerated via make pre-commit; unrelated to this change. * test(router): add direct coverage for _stream_chunks_have_generated_content CI's router_code_coverage check flags any router.py function never referenced by name in a test file; the new helper was only exercised indirectly through the mid-stream re-raise guard tests. * revert(ui): drop incidental schema.d.ts regeneration Committing router.py/common_request_processing.py touched pre_commit_lint.sh's litellm/proxy trigger for the API-type-sync check, which force-regenerated schema.d.ts even though neither file changes any route or model. The regenerated ordering of two unrelated Union/enum fields (stream_timeout, user_role) isn't stable across process invocations even against completely unmodified backend code (confirmed by regenerating twice against the pre-existing committed code and getting the same diff both times), so this reverts to the original committed file rather than chase non-deterministic output. * fix(proxy): strip framing headers on the pre-existing ProxyException branch too _handle_llm_api_exception filtered framing headers into a local `headers` dict, but for an exception that's already a ProxyException, it merged {**e.headers, **headers}: the original e.headers came first, so a framing header present there but absent from the filtered `headers` (because it was just stripped) was never overwritten and survived into the response unfiltered. Filters the merged result instead of relying on the merge order to do it implicitly. * chore: retrigger CI (no GitHub Actions check-suite was created for the previous two pushes) * fix(router): detect thinking_blocks as generated content in mid-stream guard Greptile flagged that a thinking-only delta (Anthropic extended thinking, Delta.thinking_blocks) wasn't recognized as already-streamed content, so a stream that emitted only thinking blocks before failing could still restart via fallback and append an unrelated response after content the client already received. * fix(proxy): strip browser-facing security headers from provider exceptions too veria-ai flagged that the framing-header denylist still let a malicious or misconfigured provider set browser-facing headers (Access-Control-Allow-Origin, Content-Security-Policy, Clear-Site-Data, etc.) on the proxy's own error response. Adds a dedicated _BROWSER_SECURITY_HEADERS set alongside the existing framing one and strips both wherever provider exception headers reach the client response. * refactor(router): address maintainer review mechanicals - List[ModelResponseStream] -> list[ModelResponseStream] in _stream_chunks_have_generated_content (ruff UP006 strict-budget gate) - drop _strip_http_framing_headers and its 3 tests: the proxy inlines the filter directly now, so the helper has had no production caller since the header-stripping was moved out of Router - move HTTP_FRAMING_HEADERS/BROWSER_SECURITY_HEADERS/ UNSAFE_PROXY_RESPONSE_HEADERS from router.py into litellm/constants.py, removing the router.py <-> proxy import path the two CodeQL cyclic-import alerts were pointing at - move the eager fetch_stream() call before success_calls/logging/ _track_deployment_metrics instead of incrementing then compensating with a manual decrement on failure - fix a dead assert message: `mock_fallback.assert_not_called(), "..."` built a tuple, not an assert-with-message; assert_not_called() already raises on its own so this just drops the inert string * revert(router): pull mid-stream continuation-removal out of this PR Removing the continuation-prompt fallback (retrying with the partial response as a prefixed assistant message) so a stream failing after partial content always re-raises instead was a scope decision beyond what this PR's title/issue (#31874) describe, and it directly conflicts with #30242/#30743, which are already fixing the same code path for Anthropic's removal of assistant-message prefill on Sonnet 4.6+/Opus 4.6+. Landing this PR's version first would delete the branch those PRs are patching; landing theirs first would have this PR undo their fix on rebase. Restores the original prefill-based continuation-resume behavior (including the is_pre_first_chunk guard already in litellm_internal_staging) in both _acompletion_streaming_iterator and _completion_streaming_iterator, and removes _stream_chunks_have_generated_content along with the tests that only existed to cover the guard. This PR now only touches the deferred-stream eager-fetch fix and the header-stripping fixes; the non-text-content re-raise idea becomes a follow-up PR built on top of whichever of #30242/#30743 lands. * fix(proxy): re-filter unsafe headers after the response-headers hook merge _handle_llm_api_exception filtered provider/framing headers once, then merged in post_call_response_headers_hook's return value afterward without re-filtering. The ProxyException branch happened to re-filter after its own header merge, but the HTTPException/httpx.HTTPStatusError/ generic-exception branches passed the post-hook headers straight through unfiltered, so a callback hook (any custom guardrail/logging plugin) returning an unsafe header would bypass the strip entirely for those paths. Filters once, right after the hook merge, so every branch gets the same guarantee. * Revert "revert(router): pull mid-stream continuation-removal out of this PR" This reverts commit c5ca101. * fix(router): detect reasoning_items as generated content in mid-stream guard Greptile flagged that a structured reasoning-only delta (Delta.reasoning_items, the OpenAI Responses-API-style reasoning item) wasn't recognized as already-streamed content by _stream_chunks_have_generated_content, alongside the existing thinking_blocks/tool_calls checks, so a stream that emitted only reasoning_items before failing could still restart via fallback. * fix(router): annotate _stream_chunks_have_generated_content with Sequence, not list The type_discipline_gate LIT001 check flags mutable-collection parameter annotations. chunks is only iterated, never mutated, so Sequence is the correct read-only annotation and clears the ratcheted budget ceiling. * fix(router): surface original provider exception, not the internal wrapper, when mid-stream fallback gives up When content has already streamed and MidStreamFallbackError carries original_exception (e.g. RateLimitError), both the async and sync streaming iterators bare-re-raised the wrapper itself, so the client lost the specific error type/code/provider_specific_fields instead of seeing the real provider error. The fallback-failure path a few lines below already unwraps to original_exception for the same reason; apply the same pattern here. Also extend _stream_chunks_have_generated_content to recognize audio, images, and annotations deltas as generated content, matching is_chunk_non_empty's existing annotations check and Delta's treatment of audio/images as first-class content fields — a stream carrying only one of these before failing was not recognized as already-streamed, so the router could still restart it via fallback after the client had received real content. * chore: retrigger CI (frontend-lint cancelled, schema.d.ts flake) frontend-lint's check-run shows conclusion=cancelled on 70e47f4 with no superseding run, and this PR touches no UI files. Verify schema.d.ts matches the proxy OpenAPI spec is on the previously diagnosed stream_timeout/user_role Union-ordering nondeterminism (e9fc5e5). Empty commit to force a fresh CI run for both rather than a manual rerun, which requires repo admin rights this fork PR doesn't have. --------- Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
3 participants
Relevant issues
Mid-stream fallbacks fail deterministically on Claude Sonnet 4.6 / Opus 4.6+: the resume mechanism from #13149 appends the partial response as a prefixed assistant message, but Anthropic removed assistant prefill starting with these models (migration guide) — every fallback hop returns:
so a recoverable stream timeout becomes a hard failure for the entire fallback chain (anthropic -> vertex -> anthropic -> ... all 400). Same breakage reported across the ecosystem: livekit/agents#4907, crewAIInc/crewAI#4798, agno-agi/agno#7015.
Changes
supports_assistant_prefill: falsefor all*sonnet-4-6*entries in both cost maps (theopus-4-6/4-7/4-8entries were alreadyfalse— sonnet-4-6 was missed). The capability pin intest_claude_sonnet_4_6_config.pyis updated accordingly.router_utils/fallback_event_handlers.py: newbuild_mid_stream_continuation_messages— when the registry explicitly marks the primary model OR any configured fallback target for the group as not supporting prefill, the partial response rides a trailing user message (the continuation pattern Anthropic's migration guide documents: "Your previous response was interrupted and ended with [previous_response]. Continue from where you left off."). All other models (capability true, absent, or unknown) keep the existing prefill-resume behavior byte-identical.router.py: both injection sites (sync + async) now share the helper and pass the active fallback config.Pre-Submission checklist
tests/test_litellm/router_utils/test_fallback_event_handlers.py— 10 cases: prefill-rejecting primaries -> user continuation; prefill-rejecting fallback TARGET -> user continuation; prefill-supporting / capability-absent / unknown / no-model-group / unrelated-fallback-group -> legacy prefill)make test-unit(new file 10/10;test_claude_sonnet_4_6_config.py2/2; existing mid-stream fallback tests intests/test_litellm/test_router.pyall pass —gpt-4resume behavior unchanged)@greptileai(both findings addressed in cd53134)Screenshots / Proof of Fix