fix(anthropic): resolve /v1/messages effort tiers through the capability owner - #38492
Conversation
|
@greptileai can you reply here with a score and why that score? |
|
bugbot run |
|
Confidence score: 4/5 This is a well-scoped and credible fix. It makes I’m not giving 5/5 because the PR documents intentional behavior changes for some Azure GPT-5 entries, while |
Greptile SummaryThis PR adds exact reasoning-effort declarations to Kimi K3 metadata and makes Anthropic Messages normalization use the shared capability resolver
Confidence Score: 4/5The PR needs a fallback fix before merging because exact declarations can still produce an effort the deployment does not accept Empty declarations return a chain floor, and nonempty declarations disjoint from a degradation chain return medium without verifying that either fallback is accepted Files Needing Attention: litellm/llms/anthropic/experimental_pass_through/utils.py, litellm/router_utils/reasoning_effort_capability.py
|
| Filename | Overview |
|---|---|
| litellm/llms/anthropic/experimental_pass_through/utils.py | Routes degradation through shared capability metadata, but can still emit a tier outside an exact declared set |
| litellm/router_utils/reasoning_effort_capability.py | Adds exact-list precedence and bare-twin resolution with strong tests, alongside overly extensive explanatory docstrings |
| litellm/types/utils.py | Extends model information typing so exact reasoning-effort declarations survive hydration |
| litellm/utils.py | Copies supported_reasoning_efforts from catalog metadata into hydrated model information |
| model_prices_and_context_window.json | Declares low, high, and max support across the current Kimi K3 catalog entries |
| ci_cd/generate_model_prices_schema.py | Adds the exact effort-list field and allowed values to generated model metadata schemas |
| tests/test_litellm/llms/anthropic/experimental_pass_through/test_reasoning_effort_fields.py | Covers mapped degradation and advertised-level forwarding but omits empty or chain-disjoint declarations |
| tests/test_litellm/router_utils/test_reasoning_effort_capability.py | Thoroughly tests declaration precedence, malformed input handling, intersections, hydration, and Kimi aliases |
Comments Outside Diff (1)
-
litellm/router_utils/reasoning_effort_capability.py, line 1331-1363 (link)Docstrings duplicate changing details
These expanded docstrings duplicate catalog counts and cross-file behavior, increasing maintenance cost and leaving misleading guidance when related implementations change
Context Used: CLAUDE.md (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Reviews (1): Last reviewed commit: "fix(anthropic): resolve /v1/messages eff..." | Re-trigger Greptile
| if not supported: | ||
| return chain[-1] | ||
| return next((level for level in chain if level in supported), _UNCONDITIONALLY_ACCEPTED_EFFORT) |
There was a problem hiding this comment.
Fallback violates accepted set
When an exact accepted set is empty or disjoint from the requested chain, this fallback forwards an unsupported tier, causing upstream rejection
Knowledge Base Used: Provider adapters and capabilities
There was a problem hiding this comment.
Fixed in 8f814e2: the fallback now reads an accepted level off the resolved set and never picks none. Empty sets keep the prior floor, pinned by a test.
a80c01d to
8f814e2
Compare
|
@greptileai can you reply here with a score and why that score? |
|
bugbot run |
|
Confidence score: 5/5. The fix is correctly scoped and addresses the root inconsistency: The tests provide strong coverage for declared lists, malformed/empty declarations, provider-prefixed hydration, Kimi K3 spellings, mixed-group intersection, gpt-5.5-pro’s |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 8f814e2. Configure here.
8f814e2 to
26cbaf1
Compare
|
Rebased onto #38481's new tip, which now carries the narrow short-circuit. This commit replaces it; re-ran the whole before/after against that base. |
|
@greptileai can you reply here with a score and why that score? |
|
bugbot run |
|
Score: 4/5. The fix is focused and technically sound: I’m not giving 5/5 because the PR explicitly leaves two behavioral gaps: |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 26cbaf1. Configure here.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
0dc08e6 to
e377a04
Compare
…ity owner The bridge normalizer read three supports_*_reasoning_effort booleans of its own, so it answered "which levels does this deployment take" independently of the resolver behind /model_group/info. The two disagreed: a proxy advertising kimi-k3 max forwarded high. Degrade against resolve_supported_reasoning_efforts instead, with the chains as a declared table. When no step of a chain is accepted, the fallback is read off that same resolved set rather than assumed, since an entry naming its levels outright can exclude the tiers the per-level flags treat as unconditional. none is never chosen as that fallback, being an off switch rather than a tier, and a deployment accepting no tier at all keeps the floor every deployment degraded to before.
26cbaf1 to
572ee56
Compare
|
@greptileai can you reply here with score and why |
|
bugbot run |
|
Score: 4/5. The change correctly removes the duplicated per-level ladder and resolves through |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 572ee56. Configure here.
…t boundary The existing coverage stopped at normalize_reasoning_effort_value, so nothing failed if the handler dropped or overwrote the normalized tier on its way into completion_kwargs. Drive _prepare_completion_kwargs instead and assert on the kwargs handed to acompletion, in both the string and the dict effort shapes, including the provider-prefixed model name the handler is actually called with. Against the pre-fix normalizer the fallback case fails, and against the baseline before a map entry could declare its levels 7 of the 12 fail, so the boundary is pinned rather than restated.
|
Added |
|
Confidence: 5/5. The new adapter-level coverage verifies the value at the boundary that matters: |
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 6930b6b. Configure here.
TLDR
Problem this solves:
gpt-5.5-proasking forminimalgetslow, which it refusesHow it solves it:
/model_group/info, so both agreeUser Flow
Before: a developer on a model that refuses the bottom effort tier silently gets a request the provider rejects
supported_reasoning_effortsof medium, high and xhigh, with nolow"thinking": {"type": "adaptive"}and"output_config": {"effort": "minimal"}lowto the provider, a level the same proxy just said the model does not takemaxonly, hits the same thing: asking forminimalsendslow, which is not in the set they declaredAfter: every level the gateway forwards is one the model map says the model accepts
"output_config": {"effort": "minimal"}medium, the nearest level the model actually takesmaxonly now getsmaxfor that same ask, staying inside the set they declaredRelevant issues
litellm_internal_stagingand is no longer stackedminimalpolarity were all still wrong/model_group/infoalready makesLinear ticket
Pre-Submission checklist
Screenshots / Proof of Fix
Shared setup, identical for both runs. A local proxy holding a fireworks kimi-k3 deployment (
kimi-declared, which the map gives low, high and max) and anopenai/gpt-5.5-prodeployment (gpt55pro, whose entry setssupports_low_reasoning_effortfalse), both pointed at a local stub that answers on/v1/chat/completions,/v1/responsesand/v1/messagesand records the body it received.LITELLM_LOCAL_MODEL_COST_MAP=Trueso the proxy reads the map on this branch.Substitution declared: no provider on this account sells Kimi K3, and gpt-5.5-pro is not on this key, so the upstream is a local stub. The stub only ever receives what the gateway decided to send, which is the whole question here.
/model_group/infois real proxy output either way.Before (44d8436, the staging tip this branch now sits on)
Case 1: /v1/messages
low, and/model_group/inforeports['medium', 'high', 'xhigh']for that group, so the gateway sent a level it had just called unsupportedmaxis already correct here, fixed by feat(model_prices): let a map entry declare its exact reasoning_effort levels #38481 and carried in stagingCase 2: an entry declaring an exact set disjoint from a chain
max, then ask for the two tiers no chain step matches:Case 3: /v1/chat/completions and /v1/responses
minimalask on gpt-5.5-pro, on both other routes:After (6930b6b)
Case 1: /v1/messages
medium, the nearest level it accepts, instead of thelowit refusesmaxstays fixed, andxhighandminimalstill degrade because kimi-k3 declares neitherCase 2: an entry declaring an exact set disjoint from a chain
Case 3: /v1/chat/completions and /v1/responses
Type
🐛 Bug Fix
Caveats (if any)
Medium
minimalstops degrading on 14 map entries, all azure gpt-5.xminimalone, andminimalis opt-out/v1/chat/completionsalready doesLow
thinkingfor them and dropsoutput_config.effortentirely/v1/responsesstill forwards a level the entry refuses/v1/messagesgates, so gpt-5.5-pro takesminimalthere{"supports_max_reasoning_effort": True}never says the model reasons, so the resolver cannot answer for ittest_reasoning_effort_fields.pypins whatnormalize_reasoning_effort_valuedecidesadapters/test_handler_reasoning_effort_normalization.pypins that the same value is the one the adapter puts on the outgoing request, for the plain string and the{"effort": ..., "summary": ...}shape alikeFinal Attestation
Note
Medium Risk
Changes request-shaping for reasoning effort on the Anthropic messages pass-through path; behavior shifts for some Azure GPT-5.x entries and any model whose declared levels disagree with the old flag ladder, though intent is to match advertised capabilities.
Overview
/v1/messagesreasoning effort normalization no longer walks per-levelsupports_*flags or a separate declaration helper.normalize_reasoning_effort_valuenow degradesmax,xhigh, andminimalusing the sameresolve_supported_reasoning_effortspath as/model_group/info, picking the first level in the degradation chain (then other accepted tiers) that the deployment actually supports.That fixes cases where the proxy advertised one effort set but forwarded another—e.g.
minimal→lowon models that rejectlow(like gpt-5.5-pro, which should getmedium), andmax-only declared entries gettinglow/highoutside the declared set.noneis excluded from fallback tiers so degradation cannot silently turn thinking off.Tests drop mocked flag dicts in favor of the bundled model map and synthetic declared entries; a new handler test file asserts the normalized tier is what leaves the adapter (string and dict
reasoning_effortshapes).Reviewed by Cursor Bugbot for commit 6930b6b. Bugbot is set up for automated code reviews on this repo. Configure here.