Repository navigation
fix(bedrock): honor disabled reasoning efforts on native chat route - #44856
Aasif-Simpplr wants to merge 1 commit into
Conversation
…t Completions GPT 5.6 and newer default to bedrock-runtime's native Chat Completions route since BerriAI#44307, but that route never read the model map's supports_<effort>_reasoning_effort flags that BerriAI#44183 wired into Converse and Responses. reasoning_effort="minimal" was forwarded verbatim and Bedrock answered 400, even with drop_params set Refuse such an effort with UnsupportedParamsError, or drop it under drop_params, the same way Converse does. Also correct supports_none_reasoning_effort on the Bedrock GPT-6 Sol and Luna rows, which copied Astra's false: both accept "none" on Converse, Chat Completions and Responses, so the flag would otherwise refuse a working request
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
VANDRANKI
left a comment
There was a problem hiding this comment.
Community review. This does not clear the merge gate. I checked this by running AmazonBedrockRuntimeChatCompletionsConfig().map_openai_params on the PR head and on main, using the local cost map (LITELLM_LOCAL_MODEL_COST_MAP=True). I did not run the repo's pytest suite and I did not make a real call to AWS.
What I ran, each with max_tokens=64:
- On main,
global.openai.gpt-5.6-solwithreasoning_effort="minimal"passes through untouched, with or withoutdrop_params. That matches the PR's description of the bug. - On the PR head, the same call raises
UnsupportedParamsError(400, message namesreasoning_effort=minimal) withoutdrop_params, and withdrop_params=Truereturns{'max_completion_tokens': 64}with the effort removed. bedrock/us-east-1/us.openai.gpt-5.6-luna(region-prefixed name) behaves the same, so the lookup throughbedrock_reasoning_effort_disabledworks with the region path.us.openai.gpt-6.1-solandus.openai.gpt-6-astrarefusenone.us.openai.gpt-6-sol,openai.gpt-6-luna,global.openai.gpt-5.6-solandus.openai.gpt-5.4acceptnone. The flippedsupports_none_reasoning_effortrows are what makes the gpt-6-sol/luna cases pass.us.openai.gpt-oss-120b-1:0is untouched, since it is not covered bybedrock_model_is_openai_gpt.
The change reads correct to me. disabled_reasoning_effort only fires when the map says false explicitly, so models with no flag keep today's behavior. It is also merged into the same _without_params call as the other two refusal sets, so the max_completion_tokens mapping is preserved, which I saw in the output.
Two small observations, neither blocking:
- The effort is matched case-sensitively through
f"supports_{effort}_reasoning_effort".reasoning_effort="MINIMAL"on gpt-5.6-sol still goes through unchanged on the PR head. I do not know whether AWS lowercases it, so I am not claiming that is a bug. - The map edit only touches the
bedrock_conversegpt-6-sol/luna rows. Thebedrock_mantle/openai.gpt-6-soland-lunarows still saynonefalse in this checkout. The PR text says #42854 covers those, so this is fine if that one lands, but it means the two rows disagree until it does.
I did not verify the claim that AWS returns 400 for minimal on 5.4+ since I have no AWS access. I am taking the description's word for it.
|
Thanks for running it on both main and the PR head, including the region-prefixed name. On the two notes:
|
TLDR
Problem this solves:
reasoning_effort: minimalreaches Bedrock and gets a 400, even withdrop_paramsnoneunsupportednonefor them, though Bedrock accepts itHow it solves it:
supports_<effort>_reasoning_effortlike Conversedrop_params, otherwise refused before the callnoneis supportedThe row fix is part of the same change, not a second one: once native Chat Completions reads these flags, the wrong
nonerows would refuse a working GPT-6 Sol request on the default route (an existing test catches it)User Flow
Before: a developer asking for
minimalreasoning on a Bedrock GPT-5.6 deployment gets a 400 on every call, even withdrop_paramsonbedrock/us.openai.gpt-5.6-lunawithdrop_params: true"reasoning_effort": "minimal"Unsupported value: 'reasoning_effort' does not support 'minimal' with this model"reasoning_effort": "none"tobedrock/converse/us.openai.gpt-6-soland gets a 400 from LiteLLM sayingnoneis unsupportedAfter: the same requests succeed with
drop_paramson, and fail fast with a clear message when it is offbedrock/us.openai.gpt-5.6-lunawithdrop_params: true"reasoning_effort": "minimal"drop_paramsoff they get a 400 from LiteLLM namingminimal, without a round trip to Bedrocknonerequest tobedrock/converse/us.openai.gpt-6-solcomes back 200Relevant issues
Follow-up to #44183, which wired the same model map flags into Converse and native Responses but landed after #44307 moved GPT-5.6 and newer onto native Chat Completions
#42854 makes the same
supports_none_reasoning_effortchange on these GPT-6 Sol and Luna rows (plus the Mantle ones) for/v1/messages. The row edits here are identical, so whichever lands second merges cleanlyPre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Live proxy on localhost:4000 (
python litellm/proxy/proxy_cli.py --config proof-config.yaml --port 4000 --detailed_debug), real Bedrock calls in us-east-1 withAWS_BEARER_TOKEN_BEDROCKset andLITELLM_LOCAL_MODEL_COST_MAP=trueso each side uses its own model map. Config:Every case sends this request, changing only
modelandreasoning_effort:The "sent to Bedrock" lines come from the
POST Request Sent from LiteLLMentries in the--detailed_debuglogBefore (10444df)
minimal, drop_params off
"model":"gpt-5.6-luna","reasoning_effort":"minimal"POST https://bedrock-runtime.us-east-1.amazonaws.com/openai/v1/chat/completionswith'reasoning_effort': 'minimal'litellm.BadRequestError: BedrockException - {"error":{"message":"Unsupported value: 'reasoning_effort' does not support 'minimal' with this model. Supported values are: 'none', 'low', 'medium', 'high', and 'xhigh'.","type":"invalid_request_error"...minimal, drop_params on
"model":"gpt-5.6-luna-drop","reasoning_effort":"minimal"'reasoning_effort': 'minimal'Unsupported valueerror, sodrop_paramsdid not helplow, drop_params off and on
"model":"gpt-5.6-luna"and"model":"gpt-5.6-luna-drop", both with"reasoning_effort":"low"Hi!none on GPT-6 Sol, default route
"model":"gpt-6-sol","reasoning_effort":"none"/openai/v1/chat/completionswith'reasoning_effort': 'none'Hi!none on GPT-6 Sol, Converse
"model":"gpt-6-sol-converse","reasoning_effort":"none"litellm.UnsupportedParamsError: converse/us.openai.gpt-6-sol does not support reasoning_effort=none. To drop unsupported params, set litellm.drop_params = True.After (e0b6eb0)
minimal, drop_params off
"model":"gpt-5.6-luna","reasoning_effort":"minimal"litellm.UnsupportedParamsError: us.openai.gpt-5.6-luna does not support reasoning_effort=minimal. To drop unsupported params, set litellm.drop_params = True.minimal, drop_params on
"model":"gpt-5.6-luna-drop","reasoning_effort":"minimal"POST https://bedrock-runtime.us-east-1.amazonaws.com/openai/v1/chat/completionswith{'model': 'us.openai.gpt-5.6-luna', 'messages': [...], 'stream': False}, noreasoning_effortHi!low, drop_params off and on
"model":"gpt-5.6-luna"and"model":"gpt-5.6-luna-drop", both with"reasoning_effort":"low"'reasoning_effort': 'low'both timesnone on GPT-6 Sol, default route
"model":"gpt-6-sol","reasoning_effort":"none"/openai/v1/chat/completionswith'reasoning_effort': 'none'Hi!none on GPT-6 Sol, Converse
"model":"gpt-6-sol-converse","reasoning_effort":"none"POST .../model/us.openai.gpt-6-sol/conversewith"additionalModelRequestFields": {"reasoning": {"effort": "none"}}Hi!/v1/responses, affected by the row change
Same proxy,
curl -s localhost:4000/v1/responses -d '{"model":"gpt-6-sol","reasoning":{"effort":"none"},"input":"Say hi"}', changing onlymodelandeffort. Both models take the native Responses route (POST https://bedrock-runtime.us-east-1.amazonaws.com/openai/v1/responses)Before (10444df):
gpt-6-sol,none: HTTP 400 from LiteLLM, no Bedrock call:us.openai.gpt-6-sol does not support reasoning.effort=nonegpt-6-sol,minimal: HTTP 400 from LiteLLM:does not support reasoning.effort=minimalgpt-5.6-luna,none: HTTP 200, outputHi!gpt-5.6-luna,minimal: HTTP 400 from LiteLLM:does not support reasoning.effort=minimalAfter (e0b6eb0):
gpt-6-sol,none: sent to Bedrock/openai/v1/responses, HTTP 200, outputHi!gpt-6-sol,minimal: unchanged, HTTP 400 from LiteLLMgpt-5.6-luna,none: unchanged, HTTP 200gpt-5.6-luna,minimal: unchanged, HTTP 400 from LiteLLMFor the row change, raw Bedrock with no LiteLLM in between, us-east-1 and us-west-2,
reasoning_effort: none: GPT-6 Sol and Luna (us.andglobal.) return 200 on both/openai/v1/chat/completionsand Converse, while GPT-6 Astra and GPT-6.1 Sol return 400'none' is not supported with this model, which matches the rows that keepfalseType
🐛 Bug Fix
Caveats (if any)
Low
drop_paramson, the call runs at the model's default effort, not something close tominimalnonefix reaches proxies once the map refreshes/v1/responses: GPT-6 Sol and Lunanonenow goes to Bedrock instead of being refused, shown live abovenoneon the default native route used to reach Bedrock and get its 400; LiteLLM now refuses them before the call, or dropsnoneunderdrop_params, matching the raw Bedrock results abovechat_completions/us.openai.gpt-5.6-luna), the same way Converse saysconverse/.../v1/messages, which this change does not touchFinal Attestation