Skip to content

fix(anthropic): resolve /v1/messages effort tiers through the capability owner - #38492

Merged
tin-berri merged 2 commits into
litellm_internal_stagingfrom
litellm_messages_effort_gap
Aug 28, 2026
Merged

fix(anthropic): resolve /v1/messages effort tiers through the capability owner#38492
tin-berri merged 2 commits into
litellm_internal_stagingfrom
litellm_messages_effort_gap

Conversation

@tin-berri

@tin-berri tin-berri commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • The degradation ladder still reads per-level flags of its own
  • So it can send a level the model map says is rejected
  • gpt-5.5-pro asking for minimal gets low, which it refuses

How it solves it:

  • Degrade against the shared capability resolver instead
  • Same owner that answers /model_group/info, so both agree
  • Chains become a declared table, and the ladder is deleted

User Flow

Before: a developer on a model that refuses the bottom effort tier silently gets a request the provider rejects

  1. Their admin registers gpt-5.5-pro and sees at https://litellm-domain/model_group/info that the group reports supported_reasoning_efforts of medium, high and xhigh, with no low
  2. They send POST https://litellm-domain/v1/messages with "thinking": {"type": "adaptive"} and "output_config": {"effort": "minimal"}
  3. The gateway forwards low to the provider, a level the same proxy just said the model does not take
  4. The same ask on POST https://litellm-domain/v1/chat/completions comes back 400 rather than sending it, so the two routes disagree about the same model
  5. An admin who instead declares an exact level set on a model, say max only, hits the same thing: asking for minimal sends low, which is not in the set they declared

After: every level the gateway forwards is one the model map says the model accepts

  1. Their admin registers gpt-5.5-pro and sees the same three levels at https://litellm-domain/model_group/info
  2. They send the same POST https://litellm-domain/v1/messages with "output_config": {"effort": "minimal"}
  3. The gateway forwards medium, the nearest level the model actually takes
  4. The admin who declared max only now gets max for that same ask, staying inside the set they declared
  5. Nothing the gateway sends is a level the proxy reports as unsupported

Relevant issues

Linear ticket

Pre-Submission checklist

  • I have added meaningful tests
  • The handful of test files covering my change pass locally
  • My PR passes all required CI/CD checks
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5

Screenshots / Proof of Fix

Shared setup, identical for both runs. A local proxy holding a fireworks kimi-k3 deployment (kimi-declared, which the map gives low, high and max) and an openai/gpt-5.5-pro deployment (gpt55pro, whose entry sets supports_low_reasoning_effort false), both pointed at a local stub that answers on /v1/chat/completions, /v1/responses and /v1/messages and records the body it received. LITELLM_LOCAL_MODEL_COST_MAP=True so the proxy reads the map on this branch.

LITELLM_LOCAL_MODEL_COST_MAP=True litellm --config rig/config.yaml --port 4492

Substitution declared: no provider on this account sells Kimi K3, and gpt-5.5-pro is not on this key, so the upstream is a local stub. The stub only ever receives what the gateway decided to send, which is the whole question here. /model_group/info is real proxy output either way.

Before (44d8436, the staging tip this branch now sits on)

Case 1: /v1/messages

  1. Ask each group for a level, and read back what the stub received:
for spec in "kimi-declared max" "kimi-declared xhigh" "kimi-declared minimal" "gpt55pro minimal" "gpt55pro max"; do
  set -- $spec
  curl -s -X POST http://127.0.0.1:4492/v1/messages \
    -H "Authorization: Bearer $KEY" -H 'content-type: application/json' \
    -d "{\"model\":\"$1\",\"max_tokens\":64,\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}],\"thinking\":{\"type\":\"adaptive\"},\"output_config\":{\"effort\":\"$2\"}}"
done
  1. Output:
kimi-declared  asked=max      http=200  upstream got reasoning_effort='max'
kimi-declared  asked=xhigh    http=200  upstream got reasoning_effort='high'
kimi-declared  asked=minimal  http=200  upstream got reasoning_effort='low'
gpt55pro       asked=minimal  http=200  upstream got reasoning_effort='low'
gpt55pro       asked=max      http=200  upstream got reasoning_effort='xhigh'
  1. gpt-5.5-pro receives low, and /model_group/info reports ['medium', 'high', 'xhigh'] for that group, so the gateway sent a level it had just called unsupported
  2. kimi-k3 max is already correct here, fixed by feat(model_prices): let a map entry declare its exact reasoning_effort levels #38481 and carried in staging

Case 2: an entry declaring an exact set disjoint from a chain

  1. Register a model declaring only max, then ask for the two tiers no chain step matches:
curl -s -X POST http://127.0.0.1:4492/v1/messages \
  -H "Authorization: Bearer $KEY" -H 'content-type: application/json' \
  -d '{"model":"max-only","max_tokens":64,"messages":[{"role":"user","content":"hi"}],"thinking":{"type":"adaptive"},"output_config":{"effort":"minimal"}}'
  1. Output:
declared=['max']  asked=minimal  upstream got reasoning_effort='low'    (not in the declared set)
declared=['max']  asked=xhigh    upstream got reasoning_effort='high'   (not in the declared set)
  1. A declaration is honored whole, but a chain it matches nothing in still stops on that chain's terminal, and the terminal is outside the set the admin declared

Case 3: /v1/chat/completions and /v1/responses

  1. The same minimal ask on gpt-5.5-pro, on both other routes:
curl -s -X POST http://127.0.0.1:4492/v1/chat/completions \
  -H "Authorization: Bearer $KEY" -H 'content-type: application/json' \
  -d '{"model":"gpt55pro","messages":[{"role":"user","content":"hi"}],"reasoning_effort":"minimal"}'
  1. Output:
/v1/chat/completions  http=400  (never reaches upstream)
/v1/responses         http=200  upstream got reasoning_effort='minimal'
  1. Three routes, three different answers for one ask

After (6930b6b)

Case 1: /v1/messages

  1. Same loop, same commands
  2. Output:
kimi-declared  asked=max      http=200  upstream got reasoning_effort='max'
kimi-declared  asked=xhigh    http=200  upstream got reasoning_effort='high'
kimi-declared  asked=minimal  http=200  upstream got reasoning_effort='low'
gpt55pro       asked=minimal  http=200  upstream got reasoning_effort='medium'
gpt55pro       asked=max      http=200  upstream got reasoning_effort='xhigh'
  1. gpt-5.5-pro now receives medium, the nearest level it accepts, instead of the low it refuses
  2. Every kimi row is identical to the Before run: max stays fixed, and xhigh and minimal still degrade because kimi-k3 declares neither

Case 2: an entry declaring an exact set disjoint from a chain

  1. Same commands
  2. Output:
declared=['max']  asked=minimal  upstream got reasoning_effort='max'
declared=['max']  asked=xhigh    upstream got reasoning_effort='max'
  1. The fallback is now read off the resolved set, so a declaration disjoint from a chain still lands inside that declaration

Case 3: /v1/chat/completions and /v1/responses

  1. Same commands
  2. Output:
/v1/chat/completions  http=400  (never reaches upstream)
/v1/responses         http=200  upstream got reasoning_effort='minimal'
  1. Both identical to Before. Neither route runs the code this PR changes

Type

🐛 Bug Fix

Caveats (if any)

Medium

  • minimal stops degrading on 14 map entries, all azure gpt-5.x
    • They carry an effort flag but no explicit minimal one, and minimal is opt-out
    • So the resolver reads them as accepting it, which is what /v1/chat/completions already does
    • Measured, not estimated: 167 entries carry a flag, 153 reach this route, 77 would change, and 63 of those are Claude models whose effort this route drops before the wire, leaving 14

Low

  • A deployment accepting no effort tier at all keeps the old floor
    • An empty declaration, or a non-reasoning entry, has no correct level to send
    • Left exactly as it behaved before, and pinned by a test
    • Dropping the parameter outright is the real answer, and belongs with the callers that build the request
  • Claude models reached through openrouter or azure_ai never see this code
  • /v1/responses still forwards a level the entry refuses
  • The rewritten unit tests drop hand-built flag dicts
    • A bare {"supports_max_reasoning_effort": True} never says the model reasons, so the resolver cannot answer for it
    • Each case now names a real map entry, or a synthetic one built through the fixture
  • Coverage sits on both sides of the helper
    • test_reasoning_effort_fields.py pins what normalize_reasoning_effort_value decides
    • adapters/test_handler_reasoning_effort_normalization.py pins that the same value is the one the adapter puts on the outgoing request, for the plain string and the {"effort": ..., "summary": ...} shape alike

Final Attestation

  • The tests check the right things, including the edge cases

Note

Medium Risk
Changes request-shaping for reasoning effort on the Anthropic messages pass-through path; behavior shifts for some Azure GPT-5.x entries and any model whose declared levels disagree with the old flag ladder, though intent is to match advertised capabilities.

Overview
/v1/messages reasoning effort normalization no longer walks per-level supports_* flags or a separate declaration helper. normalize_reasoning_effort_value now degrades max, xhigh, and minimal using the same resolve_supported_reasoning_efforts path as /model_group/info, picking the first level in the degradation chain (then other accepted tiers) that the deployment actually supports.

That fixes cases where the proxy advertised one effort set but forwarded another—e.g. minimallow on models that reject low (like gpt-5.5-pro, which should get medium), and max-only declared entries getting low/high outside the declared set. none is excluded from fallback tiers so degradation cannot silently turn thinking off.

Tests drop mocked flag dicts in favor of the bundled model map and synthetic declared entries; a new handler test file asserts the normalized tier is what leaves the adapter (string and dict reasoning_effort shapes).

Reviewed by Cursor Bugbot for commit 6930b6b. Bugbot is set up for automated code reviews on this repo. Configure here.

@tin-berri
tin-berri requested a review from mateo-berri as a code owner August 27, 2026 08:47
@tin-berri

Copy link
Copy Markdown
Contributor Author

@greptileai can you reply here with a score and why that score?

@tin-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

@greptile-apps

greptile-apps Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Confidence score: 4/5

This is a well-scoped and credible fix. It makes supported_reasoning_efforts authoritative in the shared capability resolver, hydrates that field through ModelInfo, and has /v1/messages use the same resolver as /model_group/info. The explicit degradation table preserves existing fallback behavior while fixing Kimi K3 max and avoiding unsupported low for gpt-5.5-pro. The tests cover provider/model spellings, mixed-group intersection, empty/malformed declarations, hydration, and the key regression cases. The required Buildkite check is also passing.

I’m not giving 5/5 because the PR documents intentional behavior changes for some Azure GPT-5 entries, while /v1/responses and some Claude provider paths still have separate effort handling. Those are clearly identified as out of scope rather than blockers, so I have high confidence in this targeted fix but not in complete cross-route/provider parity.

@greptile-apps

greptile-apps Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds exact reasoning-effort declarations to Kimi K3 metadata and makes Anthropic Messages normalization use the shared capability resolver

  • Adds supported_reasoning_efforts to the model metadata type, schema generator, committed schema, primary catalog, and packaged backup
  • Reworks max, xhigh, and minimal degradation through the shared resolver
  • Expands resolver and Anthropic pass-through regression coverage

Confidence Score: 4/5

The PR needs a fallback fix before merging because exact declarations can still produce an effort the deployment does not accept

Empty declarations return a chain floor, and nonempty declarations disjoint from a degradation chain return medium without verifying that either fallback is accepted

Files Needing Attention: litellm/llms/anthropic/experimental_pass_through/utils.py, litellm/router_utils/reasoning_effort_capability.py

Important Files Changed

Filename Overview
litellm/llms/anthropic/experimental_pass_through/utils.py Routes degradation through shared capability metadata, but can still emit a tier outside an exact declared set
litellm/router_utils/reasoning_effort_capability.py Adds exact-list precedence and bare-twin resolution with strong tests, alongside overly extensive explanatory docstrings
litellm/types/utils.py Extends model information typing so exact reasoning-effort declarations survive hydration
litellm/utils.py Copies supported_reasoning_efforts from catalog metadata into hydrated model information
model_prices_and_context_window.json Declares low, high, and max support across the current Kimi K3 catalog entries
ci_cd/generate_model_prices_schema.py Adds the exact effort-list field and allowed values to generated model metadata schemas
tests/test_litellm/llms/anthropic/experimental_pass_through/test_reasoning_effort_fields.py Covers mapped degradation and advertised-level forwarding but omits empty or chain-disjoint declarations
tests/test_litellm/router_utils/test_reasoning_effort_capability.py Thoroughly tests declaration precedence, malformed input handling, intersections, hydration, and Kimi aliases

Comments Outside Diff (1)

  1. litellm/router_utils/reasoning_effort_capability.py, line 1331-1363 (link)

    P2 Docstrings duplicate changing details

    These expanded docstrings duplicate catalog counts and cross-file behavior, increasing maintenance cost and leaving misleading guidance when related implementations change

    Context Used: CLAUDE.md (source)

    Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Reviews (1): Last reviewed commit: "fix(anthropic): resolve /v1/messages eff..." | Re-trigger Greptile

Comment on lines +67 to +69
if not supported:
return chain[-1]
return next((level for level in chain if level in supported), _UNCONDITIONALLY_ACCEPTED_EFFORT)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Fallback violates accepted set

When an exact accepted set is empty or disjoint from the requested chain, this fallback forwards an unsupported tier, causing upstream rejection

Knowledge Base Used: Provider adapters and capabilities

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 8f814e2: the fallback now reads an accepted level off the resolved set and never picks none. Empty sets keep the prior floor, pinned by a test.

Comment thread litellm/llms/anthropic/experimental_pass_through/utils.py Outdated
@tin-berri
tin-berri force-pushed the litellm_messages_effort_gap branch from a80c01d to 8f814e2 Compare August 27, 2026 09:01
@tin-berri

Copy link
Copy Markdown
Contributor Author

Good catch on the fallback. A declared set can exclude medium, so 8f814e2 reads the fallback off the resolved set instead of assuming, and never picks none. Empty sets keep the old floor, pinned by a test. The docstring finding is on #38481's file, not this diff.

@tin-berri

Copy link
Copy Markdown
Contributor Author

@greptileai can you reply here with a score and why that score?

@tin-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

@greptile-apps

greptile-apps Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Confidence score: 5/5.

The fix is correctly scoped and addresses the root inconsistency: /v1/messages now uses the shared resolve_supported_reasoning_efforts() capability owner instead of independently interpreting per-level flags. The declared supported_reasoning_efforts list is propagated through the schema, model map, ModelInfo, and bare-entry fallback, so Kimi K3 can preserve max while unsupported values still degrade. The fallback also avoids selecting none and handles disjoint/empty declarations deterministically.

The tests provide strong coverage for declared lists, malformed/empty declarations, provider-prefixed hydration, Kimi K3 spellings, mixed-group intersection, gpt-5.5-pro’s minimal behavior, and existing degradation paths. CI is passing, and the documented caveats are appropriately identified as out of scope.

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 8f814e2. Configure here.

@tin-berri
tin-berri force-pushed the litellm_messages_effort_gap branch from 8f814e2 to 26cbaf1 Compare August 27, 2026 09:29
@tin-berri

Copy link
Copy Markdown
Contributor Author

Rebased onto #38481's new tip, which now carries the narrow short-circuit. This commit replaces it; re-ran the whole before/after against that base.

@tin-berri

Copy link
Copy Markdown
Contributor Author

@greptileai can you reply here with a score and why that score?

@tin-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

@greptile-apps

greptile-apps Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Score: 4/5.

The fix is focused and technically sound: /v1/messages now uses the shared resolve_supported_reasoning_efforts() resolver, so declared effort sets and per-level capability flags have one source of truth. The explicit degradation table is easier to audit, excludes none as a fallback, and correctly handles the reported gpt-5.5-pro, Kimi K3, disjoint declaration, and empty declaration cases. The rewritten tests cover those regressions and assert the key invariant that forwarded tiers are advertised as supported. The reported Buildkite check is also passing.

I’m not giving 5/5 because the PR explicitly leaves two behavioral gaps: /v1/responses can still forward an unsupported effort, and deployments with no accepted tier retain the historical fallback rather than dropping the effort parameter. Both are documented and scoped for follow-up, so they do not block this targeted fix.

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 26cbaf1. Configure here.

@codecov

codecov Bot commented Aug 27, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@tin-berri

Copy link
Copy Markdown
Contributor Author

Filed the two documented gaps as #38529 and #38530 so they are tracked rather than left in the caveats.

@tin-berri
tin-berri force-pushed the litellm_kimi_k3_reasoning_efforts branch 4 times, most recently from 0dc08e6 to e377a04 Compare August 27, 2026 19:33
Base automatically changed from litellm_kimi_k3_reasoning_efforts to litellm_internal_staging August 27, 2026 22:38
…ity owner

The bridge normalizer read three supports_*_reasoning_effort booleans of its own, so it
answered "which levels does this deployment take" independently of the resolver behind
/model_group/info. The two disagreed: a proxy advertising kimi-k3 max forwarded high.

Degrade against resolve_supported_reasoning_efforts instead, with the chains as a declared
table. When no step of a chain is accepted, the fallback is read off that same resolved set
rather than assumed, since an entry naming its levels outright can exclude the tiers the
per-level flags treat as unconditional. none is never chosen as that fallback, being an off
switch rather than a tier, and a deployment accepting no tier at all keeps the floor every
deployment degraded to before.
@tin-berri
tin-berri force-pushed the litellm_messages_effort_gap branch from 26cbaf1 to 572ee56 Compare August 27, 2026 23:08
@tin-berri

Copy link
Copy Markdown
Contributor Author

@greptileai can you reply here with score and why

@tin-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

@greptile-apps

greptile-apps Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Score: 4/5.

The change correctly removes the duplicated per-level ladder and resolves through resolve_supported_reasoning_efforts, keeping /v1/messages aligned with the capability set advertised by /model_group/info. The tests cover the key regressions: gpt-5.5-pro’s medium fallback, Kimi’s declared levels, disjoint and empty declarations, excluding none, unknown models, and preservation of historical floors. I’m holding at 4 rather than 5 because coverage is primarily at the normalization-helper level; an explicit adapter/request integration assertion that the normalized value is the value sent by /v1/messages would make the provider boundary fully regression-proof. The documented /v1/responses gap is appropriately scoped out.

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 572ee56. Configure here.

…t boundary

The existing coverage stopped at normalize_reasoning_effort_value, so nothing failed if the
handler dropped or overwrote the normalized tier on its way into completion_kwargs. Drive
_prepare_completion_kwargs instead and assert on the kwargs handed to acompletion, in both the
string and the dict effort shapes, including the provider-prefixed model name the handler is
actually called with.

Against the pre-fix normalizer the fallback case fails, and against the baseline before a map
entry could declare its levels 7 of the 12 fail, so the boundary is pinned rather than restated.
@tin-berri

Copy link
Copy Markdown
Contributor Author

Added adapters/test_handler_reasoning_effort_normalization.py, asserting the normalized tier is the one /v1/messages actually sends. @greptileai please reply here with your confidence score and why

@greptile-apps

greptile-apps Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Confidence: 5/5.

The new adapter-level coverage verifies the value at the boundary that matters: _prepare_completion_kwargs() produces the same reasoning_effort that /v1/messages passes to litellm.acompletion. It covers both the plain-string and {"effort": ..., "summary": ...} forms, preserves sibling fields, checks provider/model spellings, confirms declared tiers survive normalization, and exercises the gpt-5.5-pro fallback from minimal to medium. Combined with the resolver-level tests and the passing CI status, this gives high confidence the normalized tier is actually the one sent upstream.

@codspeed-hq

codspeed-hq Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing litellm_messages_effort_gap (6930b6b) with litellm_internal_staging (67c7b97)1

Open in CodSpeed

Footnotes

  1. No successful run was found on litellm_internal_staging (d392e7f) during the generation of this report, so 67c7b97 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

@tin-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 6930b6b. Configure here.

@tin-berri
tin-berri enabled auto-merge (squash) August 28, 2026 02:08
@tin-berri
tin-berri merged commit 49e6081 into litellm_internal_staging Aug 28, 2026
81 checks passed
@tin-berri
tin-berri deleted the litellm_messages_effort_gap branch August 28, 2026 02:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants