/model: probe effort combo on switch, stop passing reasoning_effort to litellm - #62
Merged
Conversation
…o litellm
Switching to claude-opus-4-7 with /effort set would 400 with
"thinking.type.enabled is not supported" because litellm 1.83.0's
Anthropic adapter substring-matches "4.6" to decide which thinking API
shape to send, and doesn't know about 4.7's adaptive + output_config.effort
contract. Rather than maintain a per-model capability table that rots
every time a new Claude family ships, this change trusts the API itself:
- llm_params.py: for anthropic/*, bypass litellm's reasoning_effort -> thinking
mapping and pass thinking={type: adaptive} plus output_config={effort: ...}
as top-level kwargs directly. litellm forwards unknown top-level params
into Anthropic request bodies (extra_body does NOT work here — Anthropic
rejects it as "Extra inputs are not permitted"). One localized monkey-patch
widens litellm 1.83's hardcoded _is_opus_4_6_model check so effort=max
isn't rejected synchronously on 4.7 — removable once litellm ships PR
#25867 upstream.
- effort_probe.py: new probe that fires a 1-token ping on /model switch
with the same params we'd use for real, walking a cascade
max -> xhigh -> high -> medium -> low until the provider stops rejecting.
Three outcomes: success (cache the level), thinking-unsupported (cache
None, strip on future calls), inconclusive (switch anyway with warning).
Persistent non-thinking 4xx (auth, model-not-found) bubbles up so
/model rejects the switch and keeps the current model.
- session.py: per-model effective_effort cache + effective_effort_for()
helper. Populated by the probe, read by the real LLM call so resolved
levels don't re-probe on every message. /effort change invalidates.
- agent_loop.py: safety net — if a real call 400s with thinking/effort
config errors mid-conversation (e.g. after /effort change without
re-probe), heal the cache and retry once before propagating.
- main.py: default reasoning_effort = "max" (was "high"); /model runs
the probe and prints (effort: X, Nms); /effort accepts xhigh and max
and shows per-model probed ceilings; SUGGESTED_MODELS includes Opus 4.7.
Live-tested against Opus 4.7, Haiku 4.5, DeepSeek-R1, Qwen3.5-9B,
Llama-3.1-8B, MiMo-V2-Flash, Gemma-4-31B, Arch-Router-1.5B, Kimi-K2.5,
and a non-existent id. All outcomes matched expectations.
main.py was accumulating model-switch specifics (suggested list, id format check, HF routing info printer, probe-and-switch flow, commit helper). Moving them to a dedicated module keeps the REPL dispatcher focused on input parsing and makes the switcher independently testable. Net: main.py down ~160 lines, /model handler is now a 10-line delegation. No behavior change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
claude-opus-4-7with/effortset was 400ing because litellm 1.83.0 doesn't know 4.7's thinking API shape (thinking.type.adaptive+output_config.effort). Rather than maintain a per-model capability table that rots on every Claude release, this PR trusts the API itself: send Anthropic-native thinking params directly, and validate model+effort with a 1-token ping on/modelswitch. If the provider rejects a level, a cascade walksmax → xhigh → high → medium → lowuntil something sticks, and the result is cached per-model for the session./effortgainsxhighandmax; default preference bumped fromhightomax(user asked for "best effort per model, usually the highest"). Non-thinking models get thinking stripped automatically. New models from any family should Just Work — no table updates required.agent_loopheals & retries once if a real call hits a thinking/effort 400 mid-conversation (e.g. after/effortchange).Key design choices
thinking+output_configas top-level kwargs; litellm forwards unknown top-level params into the Anthropic body (confirmed by live probe).extra_bodydoes not work here — Anthropic rejects it as "Extra inputs are not permitted".llm_params.pywidens litellm 1.83's hardcoded_is_opus_4_6_modelsubstring check soeffort=maxisn't pre-flight-rejected for 4.7. Self-documenting, self-removes once the pin catches up to litellm PR #25867 (merged on main, not yet in a PyPI stable).Live test
Ran the probe against a mix of obscure HF models + Anthropic direct:
anthropic/claude-opus-4-7anthropic/claude-haiku-4-5deepseek-ai/DeepSeek-R1Qwen/Qwen3.5-9B,meta-llama/Llama-3.1-8B-InstructXiaomiMiMo/MiMo-V2-Flash,google/gemma-4-31B-itkatanemo/Arch-Router-1.5B(1.5B!),moonshotai/Kimi-K2.5anthropic/does-not-existTest plan
uv run python -m agent.main, then/model anthropic/claude-opus-4-7— expect effort:max, ~1–2s/model anthropic/claude-haiku-4-5— expect effort:off with "doesn't support reasoning" note/model deepseek-ai/DeepSeek-R1— expect effort:high with "max not supported" note/model anthropic/does-not-exist— expect red "Switch failed" and current model preserved/effort xhighthen/effort— preference shows, per-model cache clearsthinking.type.enabled400Known limitation
xhighon Opus 4.7 is unreachable via the probe until litellm's internal valid-effort list is updated (it still hardcodes{high, medium, low, max}and rejectsxhighsynchronously). Cascade walks tohigh. Unlocks automatically once the litellm pin moves forward; our monkey-patch becomes a no-op at that point and can be deleted.