Repository navigation
fix(bedrock): drop unsupported sampling params on converse reasoning models - #39834
Conversation
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
@greptileai please re-review following the update to remove the hardcoded check and sync the packaged model backup |
|
@greptileai please re-review following the fix for the basedpyright loop assignment error and added coverage |
mateo-berri
left a comment
There was a problem hiding this comment.
This drops temperature from supported params for Bedrock Anthropic too, so temperature=1 on us.anthropic.claude-opus-4-7 now 400s where it returned 200 before
08e000a to
01e3da2
Compare
01e3da2 to
bf802ef
Compare
|
Rebased onto main and verified the affected tests locally. @greptileai please review the current tip for remaining regressions bugbot run |
|
@veria-ai please review the current tip after rebasing onto main and resolving the overlapping changes from upstream |
|
bugbot run |
|
@veria-ai |
|
Can we get this merged? We are also experiencing issues e.g. with the |
|
@yuneng-berri any update? |
bf802ef to
65eed87
Compare
|
@greptileai @veria-ai Please review the current tip after this rebase and live proxy QA for regressions. bugbot run |
|
@veria-ai Please review the current tip after the rebase and live Bedrock proxy QA, including Anthropic temperature handling |
|
bugbot run Please review the current tip after the rebase and live Bedrock QA, including the Anthropic temperature regression |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 65eed87. Configure here.
|
@mateo-berri The Anthropic temperature regression is fixed and verified through the live proxy. Current CI, coverage, Greptile, Veria, and Bugbot pass. Please re-review |
mateo-berri
left a comment
There was a problem hiding this comment.
LGTM. Thanks for the contribution!
… repo rule and BerriAI#39834 do The GPT-6 Sol, Luna and Astra Converse rows now set supports_sampling_params false in both cost map files, like the GPT-5.6 rows. Tools and tool choice already come from the rows BerriAI#42746 and BerriAI#42798 added, so the name-based checks in converse_transformation.py are removed
TLDR
Problem this solves:
temperatureandtop_pon GPT-5.6 reasoning modelsdrop_params: truedid not drop them, so every such request 400'dtemperature=1How it solves it:
supports_sampling_params: falsetemperatureandtop_pfor those models whendrop_paramsis ondrop_params, LiteLLM returns its own 400 naming the param and the fixUser Flow
Before: a developer whose gateway routes a GPT-5.6 model to Bedrock Converse with
drop_params: truegets a 400 on every request that carries a temperaturebedrock/converse/global.openai.gpt-5.6-lunaasbedrock-reasoningwithdrop_params: true, andbedrock/converse/us.anthropic.claude-opus-4-7asbedrock-anthropic, then starts the proxy"model": "bedrock-reasoning","temperature": 1, and a user messageBedrockException - This model doesn't support the temperature field. Remove temperature and try again."model": "bedrock-anthropic"returns HTTP 200 withpongAfter: the same request returns 200 because the gateway drops the temperature before calling Bedrock, and Claude keeps accepting
temperature=1"model": "bedrock-reasoning","temperature": 1, and a user messagepongand token usagepong"model": "bedrock-anthropic"still returns HTTP 200 withpongRelevant issues
Fixes #39829
Affected release
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Both legs hit real AWS Bedrock in us-east-1 with the same credentials and this config saved as
bedrock-qa.yaml. The master key is exported asLITELLM_MASTER_KEYand every proxy runs with 2 uvicorn workersBefore ran on port 48951, After on port 58714
Before (238f434)
GPT-5.6 with drop_params, temperature=1: POST /v1/chat/completions
Send the request
Observe the response
GPT-5.6 with drop_params, temperature=1: POST /v1/responses
Send the request
Observe the response
GPT-5.6 with drop_params, temperature=1: POST /v1/messages
Send the request
Observe the response
GPT-5.6 with drop_params, temperature=1, streaming: POST /v1/chat/completions
Send the request
Observe the response
Claude Opus 4.7, temperature=1: POST /v1/chat/completions
Send the request
Observe the response
Claude Opus 4.7, temperature=1: POST /v1/responses
Send the request
Observe the response
Claude Opus 4.7, temperature=1: POST /v1/messages
Send the request
Observe the response
Claude Opus 4.7, temperature=0.5: POST /v1/chat/completions
Send the request
Observe the response
GPT-5.6 without drop_params, temperature=1: POST /v1/chat/completions
Send the request
Observe the response
GPT-5.6 without drop_params, top_p=0.9: POST /v1/chat/completions
Send the request
Observe the response
GPT-5.6 without drop_params, no sampling params: POST /v1/chat/completions
Send the request
Observe the response
GPT-6 with drop_params, temperature=1: POST /v1/chat/completions
Send the request
Observe the response
After (65eed87)
GPT-5.6 with drop_params, temperature=1: POST /v1/chat/completions
Send the request
Observe the response
GPT-5.6 with drop_params, temperature=1: POST /v1/responses
Send the request
Observe the response
GPT-5.6 with drop_params, temperature=1: POST /v1/messages
Send the request
Observe the response
GPT-5.6 with drop_params, temperature=1, streaming: POST /v1/chat/completions
Send the request
Observe the response
Claude Opus 4.7, temperature=1: POST /v1/chat/completions
Send the request
Observe the response
Claude Opus 4.7, temperature=1: POST /v1/responses
Send the request
Observe the response
Claude Opus 4.7, temperature=1: POST /v1/messages
Send the request
Observe the response
Claude Opus 4.7, temperature=0.5: POST /v1/chat/completions
Send the request
Observe the response
GPT-5.6 without drop_params, temperature=1: POST /v1/chat/completions
Send the request
Observe the response
GPT-5.6 without drop_params, top_p=0.9: POST /v1/chat/completions
Send the request
Observe the response
GPT-5.6 without drop_params, no sampling params: POST /v1/chat/completions
Send the request
Observe the response
GPT-6 with drop_params, temperature=1: POST /v1/chat/completions
Send the request
Observe the response
Observations from the run:
temperature=1returns 200 on all three endpoints both legstemperature=0.5400s identically on both legs, pre-existingdrop_params, the 400 now comes from LiteLLM, not Bedrockmaingives identical results on all twelve casesType
🐛 Bug Fix
Caveats (if any)
Low
us.openai.gpt-6-astraandglobal.openai.gpt-6-astraare not flagged, sodrop_paramsstill does not save them (follow-up cost-map PR)/model_group/infoandlitellm.get_supported_openai_paramsno longer listtemperatureandtop_pfor Bedrock GPT-5.6 groups, which is the intended contractdrop_params, the 400 is LiteLLM's own with a fix hint instead of Bedrock's; the request never succeeded either wayallowed_openai_params: ["temperature"]withoutdrop_paramsstill 400s, now from LiteLLM instead of Bedrocktest_bedrock_completion.py,test_bedrock_gpt_oss.py,test_bedrock_anthropic_regression.py,test_bedrock_moonshot.py,test_optional_params.py -k bedrock) ran locally at the tip: all pass excepttest_bedrock_moonshot.py::TestBedrockMoonshotInvoke::test_json_response_format_stream, which fails identically at the merge baseFinal Attestation
The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR
65eed87 passes /live-pr-risk