fix(anthropic): strip thinking_blocks for non-Anthropic backends - #32684
fix(anthropic): strip thinking_blocks for non-Anthropic backends#32684samagana wants to merge 1 commit into
Conversation
Greptile SummaryThis PR fixes a regression in
Confidence Score: 5/5Safe to merge — the change is narrowly scoped to the thinking_blocks gating logic, follows the exact same pattern used for cache_control and thinking parameter translation throughout the file, and is covered by tests that exercise both the strip and preserve paths. The fix aligns with the established helper pattern (is_anthropic_claude_model + is_bedrock_arn_model) used everywhere else in this file for Anthropic-specific field handling. The model is None fallback preserves prior behavior, the Bedrock ARN detection is tightly scoped to :bedrock: in the service segment, and the test suite covers the representative cases including mixed redacted/unredacted blocks. No files require special attention.
|
| Filename | Overview |
|---|---|
| litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py | Adds preserve_thinking_blocks gate using is_anthropic_claude_model and is_bedrock_arn_model, matching the established pattern already used for cache_control and thinking parameter translation in this file. Non-Anthropic backends get reasoning_content instead; Bedrock ARNs and Claude backends keep thinking_blocks unchanged. |
| tests/test_litellm/llms/anthropic/experimental_pass_through/adapters/test_anthropic_experimental_pass_through_adapters_transformation.py | Adds two parametrized tests covering: stripping of thinking_blocks and conversion to reasoning_content for non-Anthropic models; preservation of thinking_blocks for Anthropic/Claude/Bedrock ARN models. Also verifies redacted blocks are correctly dropped from the reasoning_content concatenation. |
Reviews (3): Last reviewed commit: "fix(anthropic): strip thinking_blocks fo..." | Re-trigger Greptile
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
The Anthropic /v1/messages -> OpenAI chat-completions pass-through adapter
(translate_anthropic_messages_to_openai) attaches the Anthropic-specific
thinking_blocks field to assistant messages unconditionally. Non-Anthropic
OpenAI-compatible backends reject it: on multi-turn conversations, models
like GLM behind an OpenAI-compatible endpoint fail with
400 invalid_request_error: Extra inputs are not permitted,
field: 'messages[1].thinking_blocks'
This breaks any multi-turn conversation once an earlier assistant turn
carried reasoning.
Verified directly against an OpenAI-compatible GLM endpoint (bypassing
litellm):
- assistant turn with thinking_blocks -> 400 (field rejected)
- assistant turn with reasoning_content -> 200 OK (the model consumes it and
reasons over the prior turn)
So the fix is to convert, not just drop: for non-Anthropic backends, strip the
raw thinking_blocks and set the OpenAI-style reasoning_content string
(concatenating the unredacted thinking blocks; redacted blocks carry no
readable text and are dropped).
Gate the thinking_blocks attachment on is_anthropic_claude_model or
is_bedrock_arn_model, the same pair of checks already used together elsewhere
in this file (e.g. for cache_control). Anthropic Claude backends (anthropic/*,
bedrock *anthropic*, vertex *claude*, and Bedrock ARNs such as Application
Inference Profiles that point at Claude) keep thinking_blocks and their signed
signatures unchanged. Everyone else gets reasoning_content instead. When the
target model is unknown (None) the prior behaviour is preserved (blocks
kept), so no existing caller changes.
This is the complete form of the half-fixes in BerriAI#27947 and BerriAI#28258, both of
which only add reasoning_content and leave thinking_blocks attached, so they
do not resolve the 400. Closes BerriAI#27946.
ae57ebb to
c2fd9b3
Compare
Relevant issues
Fixes #27946
Linear ticket
Pre-Submission checklist
@greptileaito re-request a review after pushing changes)Screenshots / Proof of Fix
The Anthropic
/v1/messages-> OpenAI chat-completions adapter (translate_anthropic_messages_to_openai) attaches the Anthropic-specificthinking_blocksfield to assistant messages unconditionally. Non-Anthropic OpenAI-compatible backends reject that field outright, which breaks any multi-turn conversation once an earlier assistant turn carried reasoning.This was verified directly against an OpenAI-compatible GLM endpoint, bypassing litellm entirely, to confirm the backend's actual contract rather than litellm's behavior at some commit:
thinking_blocksgets400 invalid_request_error: Extra inputs are not permitted, field: 'messages[1].thinking_blocks'reasoning_contentinstead gets200 OK, and the model consumes it and reasons over the prior turn correctlyThat contract is what motivates the fix: convert, rather than merely drop,
thinking_blocksfor non-Anthropic backends.I don't have a from-litellm before/after repro to attach a commit hash to, since the failing request never reaches litellm's HTTP layer; it fails on the backend's own schema validation regardless of which litellm commit constructed it. The regression tests added in this PR exercise the actual code path (
translate_anthropic_messages_to_openai) that produces the request body, asserting the pre-fix behavior (rawthinking_blocksattached) would fail and the post-fix behavior (convertedreasoning_content) passes for a representative set of non-Anthropic and Anthropic model identifiers.Type
🐛 Bug Fix
✅ Test
Changes
Gate the
thinking_blocksattachment onis_anthropic_claude_modeloris_bedrock_arn_model, the same pair of checks already used together elsewhere in this file (for example thecache_controlpath). Anthropic Claude backends keepthinking_blocksand their signed signatures unchanged; that coversanthropic/*, Bedrock*anthropic*, Vertex*claude*, and Bedrock ARNs such as Application Inference Profiles that point at Claude but contain neither "anthropic" nor "claude" in the identifier. Every other backend gets the raw blocks converted to an OpenAI-stylereasoning_contentstring instead, concatenating the unredacted thinking blocks (redacted blocks carry no readable text and are dropped). When the target model is unknown (None), the prior behavior is preserved so no existing caller changes.This is the complete form of the half-fixes in #27947 and #28258, both of which only add
reasoning_contentand leavethinking_blocksattached, so they do not resolve the 400.