feat(anthropic): per-deployment opt-out for legacy thinking rewrite - #77
Merged
Merged
Conversation
…e rewrite
Add `disable_adaptive_thinking_rewrite` to `GenericLiteLLMParams`. When
set on a model_list deployment, the /v1/messages passthrough preserves
the caller's legacy `thinking={"type":"enabled","budget_tokens":N}`
verbatim instead of rewriting it to the adaptive form
(`{"type":"adaptive"}` + `output_config.effort`).
Motivation: on Claude Sonnet/Opus 4.6, the legacy thinking shape is
deprecated upstream but still functional. LiteLLM rewrites it
proactively, which discretizes `budget_tokens` into a 4-bucket effort
band — callers that need byte-exact passthrough (precise cache pricing,
A/B parity with non-LiteLLM clients) have no way to opt out today.
The flag is duck-typed: the gate reads from either a dict-shaped
`litellm_params` or a `GenericLiteLLMParams` pydantic instance,
mirroring the `additional_drop_params` pattern. It only short-circuits
the rewrite path — it does NOT bypass upstream Anthropic's 400 rejection
of the legacy shape on Opus 4.7+ (those models reject it at the
provider, not in LiteLLM).
Usage:
model_list:
- model_name: claude-sonnet-4-6-raw
litellm_params:
model: anthropic/claude-sonnet-4-6
api_key: os.environ/ANTHROPIC_API_KEY
disable_adaptive_thinking_rewrite: true
Tests cover: default rewrite preserved, flag-via-dict opt-out, flag-via-
pydantic-model opt-out, no-op on non-adaptive models, and default-False
on a fresh pydantic instance.
Tier: D (universal mechanism + opinionated default of False preserves
existing behavior). To be filed upstream as a passthrough hook.
Merged
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tier classification
litellm_extras/only)litellm/corelitellm/coreIf Tier C or D, did you try upstream first?
<link>False) preserves existing behavior, so this carries zero risk on the ship branch; upstreaming the mechanism in parallel lets us release internally without blocking on BerriAI review.Relevant issues
None tracked externally. Internal motivation: callers hitting
/v1/messageswithclaude-sonnet-4-6need byte-exact passthrough of the deprecated legacythinking={type:"enabled",budget_tokens:N}shape (e.g. for precise cache pricing, A/B parity with non-LiteLLM clients). Today LiteLLM rewrites it unconditionally into the adaptive form ({type:"adaptive"}+output_config.effort), discretizingbudget_tokensinto a 4-bucket effort band with no opt-out.Pre-Submission checklist
tests/test_litellm/llms/anthropic/experimental_pass_through/messages/test_legacy_thinking_passthrough.py(5 cases — default rewrite locked, flag-via-dict opt-out, flag-via-pydantic opt-out, no-op on non-adaptive models, default-False on pydantic instance)python3 -m pytest tests/test_litellm/llms/anthropic/experimental_pass_through/messages/ -q→ 41 passed_translate_legacy_thinking_for_adaptive_modelon a newlitellm_paramsflagType
🆕 New Feature
Changes
What
Add
disable_adaptive_thinking_rewrite: Optional[bool] = FalsetoGenericLiteLLMParams(and mirror it inLiteLLMParamsTypedDict). When the flag is True on a model_list deployment, the/v1/messagespassthrough preserves the caller's legacy thinking shape verbatim instead of rewriting it.Where
litellm/types/router.pydisable_adaptive_thinking_rewriteon the pydantic model + TypedDict, alongsideanthropic_beta_overrides(same plumbing pattern)litellm/llms/anthropic/experimental_pass_through/messages/transformation.pylitellm_paramsarg to_translate_legacy_thinking_for_adaptive_model, duck-typed read of the flag from either dict or pydantic model, short-circuit before the existing_is_adaptive_thinking_modelgate. Threadlitellm_paramsthrough the single call site intransform_anthropic_messages_request.tests/test_litellm/llms/anthropic/experimental_pass_through/messages/test_legacy_thinking_passthrough.pyBlast radius
Limited to the
/v1/messagespassthrough. The same translation lives inlitellm/llms/anthropic/chat/transformation.py(the/v1/chat/completionspath) and in the bedrock/databricks adaptive-thinking handlers — those are intentionally out of scope for this PR. If callers need the same opt-out on chat completions, the pattern is identical: gate the_translate_*calls on the same flag. Will be a follow-up.Limitations
Usage
Request body sent to upstream Anthropic preserves
thinking={"type":"enabled","budget_tokens":2048}byte-for-byte instead of being rewritten tothinking={"type":"adaptive"}+output_config={"effort":"low"}.Test plan
python3 -m pytest tests/test_litellm/llms/anthropic/experimental_pass_through/messages/test_legacy_thinking_passthrough.py -v— 5/5 passpython3 -m pytest tests/test_litellm/llms/anthropic/experimental_pass_through/messages/ -q— 41 passed, 0 new failuresdisable_adaptive_thinking_rewrite: true, curl/v1/messageswith the legacy thinking shape, confirm the upstream request body matches via httpx debug logging