fix(bedrock): return JSON for Claude Opus 5.5 response_format when tool_choice can't be forced by kimnamu · Pull Request #42718 · BerriAI/litellm

kimnamu · 2026-09-23T08:53:27Z

Thanks for the Fable 5.1 backstop in #39184 and the Opus 5.5 cost map entries in #42588. This picks up the unforced path #42644 left as "its own feature"

Size: 28 source lines, 94 test lines

Diff	Files	Lines
Source	3	+28 −4
Tests	2	+94 −4

TLDR

Problem this solves:

Bedrock Opus 5.5 answers response_format in plain text
The unforced json_tool_call tool gets skipped

How it solves it:

Directive tool description when the tool can't be forced
Other requests unchanged

Case	Before (`59db39626f`)	After (`79eef9aa8d`)
Opus 5.5, `/v1/chat/completions`, Converse	0/5 JSON (`Paris.`)	5/5 JSON
Opus 5.5, `/v1/chat/completions`, Invoke	0/5 JSON	5/5 JSON
Opus 5.5, `/v1/responses`	0/5 JSON	5/5 JSON
Opus 5.5, `/v1/messages`, `bedrock/converse/` deployment	0/5 JSON	5/5 JSON
Opus 5.5, `"stream": true`, 3 runs	0/3 JSON (`Paris.`)	3/3 JSON
Opus 5 without thinking (forced tool)	5/5 JSON	unchanged, same request body
Opus 5 or Claude 3.5 Sonnet plus a caller tool, `tool_choice` auto, required or named, both routes	no directive	unchanged, same request body
Caller tool `get_weather`, `tool_choice` auto, Opus 5 Converse and Invoke, Opus 5.5	calls `get_weather` 3/3 each	unchanged, 3/3 each
`/v1/messages` on the default Invoke route (schema inlined, no tool)	JSON in the text, bare or fenced, 3/3	unchanged, 3/3
Sonnet 4.6, Haiku 4.5 (native `outputConfig`)	native path	unchanged, same request body

User Flow

Before: Opus 5.5 answers JSON requests in prose

POST http://localhost:4000/v1/chat/completions, model opus-5-5, json_schema response_format
200, content "Paris.", json.loads fails

After: the same request returns JSON

Same POST
200, content {"name": "Paris", "country": "France"}

Details

Route	Base behaviour	This PR
Converse	`converse_transformation.py:1242-1249` skips the forced choice when thinking is on or the model rejects it, and the description falls back to the tool name (`factory.py:5132-5135`)	directive description in that case
Invoke	the forced choice is dropped for Opus 5.5 (`anthropic_claude3_transformation.py:118-126`) and when thinking is on, and the tool has no description	same condition, same description
Both	Bedrock rejects `toolSpec.strict` and `outputConfig.textFormat` on Opus 5.5	no strict mode or native format

References: https://www.anthropic.com/news/claude-opus-5-5 , https://platform.claude.com/docs/en/models/opus-5-5/migration-guide , https://platform.claude.com/docs/en/about-claude/models/overview

SDK, 5 runs unless noted	Before (`59db39626f`)	After (`79eef9aa8d`)
Fable 5.1	0/5 JSON	5/5 JSON
Opus 5, `reasoning_effort="low"`, Converse	0/5 JSON	5/5 JSON
Opus 5, `reasoning_effort="low"`, Invoke	0/5 JSON	5/5 JSON
gpt-oss-120b, `reasoning_effort="low"`, Converse, 3 runs	0/3 JSON	3/3 JSON
Nova 2 Lite, `reasoning_effort="low"`, Converse, 3 runs	3/3 JSON	3/3 JSON

Offline body capture (7 Claude models on Converse and Invoke, 6 others on Converse, 5 variants each): 28 of 100 rows change, all thinking-on or forced-tool-rejected; tip bodies identical to 036cfb951b

# the two transformation files reverted to 59db39626f, tests and constant kept
$ pytest -p no:xdist -q tests/test_litellm/llms/bedrock/chat/test_converse_transformation.py \
    tests/unit/llms/bedrock/chat/invoke_transformations/test_bedrock_chat_invoke_transformations_anthropic_claude3_transformation.py \
    -k "avoids_native_and_forced_tool_choice_on_fable_5_1_converse or asks_for_the_tool_only_when_it_cannot_be_forced or fable_5_1_response_format_avoids_forced_tool_choice"
E       KeyError: 'description'
E       AssertionError: assert 'A city' == 'Always respo... text. A city'
E       AssertionError: assert None == 'Always respond by calling this tool. Its input is your final answer; do not reply in plain text.'
10 failed, 4 passed, 376 deselected
# at 79eef9aa8d
14 passed, 376 deselected

The 4 cases that also pass on main are the forced-tool and caller-tool rows. They fail if the directive leaks into requests where the tool can be forced

Check	`59db39626f`	`79eef9aa8d`
`tests/test_litellm/llms/bedrock`	1401 passed	1406 passed
`tests/unit/llms/bedrock`	533 passed	538 passed
`tests/test_litellm/llms/anthropic/chat`	503 passed	503 passed
ruff format/check on changed `litellm/` files (CI scope), ruff strict, type discipline, test quality gates		OK

Relevant issues

Fixes #42717

Affected release

Linear ticket

Pre-Submission checklist

Screenshots / Proof of Fix

Live Bedrock calls through a local proxy

# LITELLM_LOCAL_MODEL_COST_MAP=True python litellm/proxy/proxy_cli.py --config config.yaml --port 4000
model_list:
  - model_name: opus-5-5
    litellm_params: {model: bedrock/us.anthropic.claude-opus-5-5, aws_region_name: us-east-1}
  - model_name: opus-5-5-invoke
    litellm_params: {model: bedrock/invoke/us.anthropic.claude-opus-5-5, aws_region_name: us-east-1}
  - model_name: opus-5-5-converse
    litellm_params: {model: bedrock/converse/us.anthropic.claude-opus-5-5, aws_region_name: us-east-1}
  - model_name: opus-5
    litellm_params: {model: bedrock/us.anthropic.claude-opus-5, aws_region_name: us-east-1}
  - model_name: opus-5-invoke
    litellm_params: {model: bedrock/invoke/us.anthropic.claude-opus-5, aws_region_name: us-east-1}
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

S='{"type":"object","properties":{"name":{"type":"string"},"country":{"type":"string"}},"required":["name","country"]}'
S2='{"type":"object","properties":{"name":{"type":"string"},"country":{"type":"string"}},"required":["name","country"],"additionalProperties":false}'
P='"Name one city in France. Be brief."'
W='[{"type":"function","function":{"name":"get_weather","description":"Get the current weather for a city","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}]'
chat() { curl -s http://localhost:4000/v1/chat/completions -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
  -d "{\"model\":\"$1\",\"max_tokens\":4096,\"messages\":[{\"role\":\"user\",\"content\":$P}],\"response_format\":{\"type\":\"json_schema\",\"json_schema\":{\"name\":\"City\",\"schema\":$S}}}" \
  | jq -c '.choices[0].message.content // .error.message'; }
resp() { curl -s http://localhost:4000/v1/responses -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
  -d "{\"model\":\"$1\",\"max_output_tokens\":4096,\"input\":$P,\"text\":{\"format\":{\"type\":\"json_schema\",\"name\":\"City\",\"schema\":$S}}}" \
  | jq -c '[.output[]? | select(.type=="message") | .content[0].text][0] // .error.message'; }
msgs() { curl -s http://localhost:4000/v1/messages -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -H "anthropic-version: 2023-06-01" \
  -d "{\"model\":\"$1\",\"max_tokens\":4096,\"messages\":[{\"role\":\"user\",\"content\":$P}],\"output_config\":{\"format\":{\"type\":\"json_schema\",\"schema\":$S2}}}" \
  | jq -c '[.content[]? | .type + ":" + ((.text // (.input|tostring)) // "")]'; }
stream() { curl -sN http://localhost:4000/v1/chat/completions -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
  -d "{\"model\":\"$1\",\"stream\":true,\"max_tokens\":4096,\"messages\":[{\"role\":\"user\",\"content\":$P}],\"response_format\":{\"type\":\"json_schema\",\"json_schema\":{\"name\":\"City\",\"schema\":$S}}}" \
  | sed -n 's/^data: //p' | grep -v '^\[DONE\]' | jq -rj '.choices[0].delta.content // empty'; echo; }
weather() { curl -s http://localhost:4000/v1/chat/completions -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
  -d "{\"model\":\"$1\",\"max_tokens\":4096,\"messages\":[{\"role\":\"user\",\"content\":\"What is the weather in Paris right now?\"}],\"tools\":$W,\"tool_choice\":\"auto\",\"response_format\":{\"type\":\"json_schema\",\"json_schema\":{\"name\":\"City\",\"schema\":$S}}}" \
  | jq -c '[.choices[0].message.tool_calls[]?.function.name] // .error.message'; }

Before (`59db396`)

Converse

Run

$ for i in 1 2 3 4 5; do chat opus-5-5; done
"Paris"
"Paris."
"Paris."
"Paris"
"Paris."

Invoke

Run

$ for i in 1 2 3 4 5; do chat opus-5-5-invoke; done
"Paris."
"Paris"
"Paris"
"Paris"
"Paris."

Responses

Run

$ for i in 1 2 3 4 5; do resp opus-5-5; done
"Paris."
"Paris."
"Paris."
"Paris."
"Paris."

Messages

Run

$ for i in 1 2 3 4 5; do msgs opus-5-5-converse; done
["thinking:null","text:Paris"]
["thinking:null","text:Paris."]
["thinking:null","text:Paris."]
["thinking:null","text:Paris"]
["thinking:null","text:Paris."]

Streaming

Run

$ for i in 1 2 3; do stream opus-5-5; done
Paris.
Paris
Paris.

Control

Run

$ for i in 1 2 3 4 5; do chat opus-5; done
"{\"name\": \"Paris\", \"country\": \"France\"}"
"{\"name\": \"Paris\", \"country\": \"France\"}"
"{\"name\": \"Paris\", \"country\": \"France\"}"
"{\"name\": \"Paris\", \"country\": \"France\"}"
"{\"name\": \"Paris\", \"country\": \"France\"}"

Caller tool

Run

$ for i in 1 2 3; do weather opus-5-invoke; done
["get_weather"]
["get_weather"]
["get_weather"]

Run

$ for i in 1 2 3; do weather opus-5; done
["get_weather"]
["get_weather"]
["get_weather"]

Run

$ for i in 1 2 3; do weather opus-5-5; done
["get_weather"]
["get_weather"]
["get_weather"]

After (`79eef9a`)

Converse

Run

$ for i in 1 2 3 4 5; do chat opus-5-5; done
"{\"name\": \"Paris\", \"country\": \"France\"}"
"{\"name\": \"Paris\", \"country\": \"France\"}"
"{\"name\": \"Paris\", \"country\": \"France\"}"
"{\"name\": \"Paris\", \"country\": \"France\"}"
"{\"name\": \"Paris\", \"country\": \"France\"}"

Invoke

Run

$ for i in 1 2 3 4 5; do chat opus-5-5-invoke; done
"{\"name\": \"Paris\", \"country\": \"France\"}"
"{\"name\": \"Paris\", \"country\": \"France\"}"
"{\"name\": \"Paris\", \"country\": \"France\"}"
"{\"name\": \"Paris\", \"country\": \"France\"}"
"{\"name\": \"Paris\", \"country\": \"France\"}"

Responses

Run

$ for i in 1 2 3 4 5; do resp opus-5-5; done
"{\"name\": \"Paris\", \"country\": \"France\"}"
"{\"name\": \"Paris\", \"country\": \"France\"}"
"{\"name\": \"Paris\", \"country\": \"France\"}"
"{\"name\": \"Paris\", \"country\": \"France\"}"
"{\"name\": \"Paris\", \"country\": \"France\"}"

Messages

Run

$ for i in 1 2 3 4 5; do msgs opus-5-5-converse; done
["text:{\"name\": \"Paris\", \"country\": \"France\"}"]
["text:{\"name\": \"Paris\", \"country\": \"France\"}"]
["text:{\"name\": \"Paris\", \"country\": \"France\"}"]
["text:{\"name\": \"Paris\", \"country\": \"France\"}"]
["text:{\"name\": \"Paris\", \"country\": \"France\"}"]

Streaming

Run

$ for i in 1 2 3; do stream opus-5-5; done
{"name": "Paris", "country": "France"}
{"name": "Paris", "country": "France"}
{"name": "Paris", "country": "France"}

Control

Run

$ for i in 1 2 3 4 5; do chat opus-5; done
"{\"name\": \"Paris\", \"country\": \"France\"}"
"{\"name\": \"Paris\", \"country\": \"France\"}"
"{\"name\": \"Paris\", \"country\": \"France\"}"
"{\"name\": \"Paris\", \"country\": \"France\"}"
"{\"name\": \"Paris\", \"country\": \"France\"}"

Caller tool

Run

$ for i in 1 2 3; do weather opus-5-invoke; done
["get_weather"]
["get_weather"]
["get_weather"]

Run

$ for i in 1 2 3; do weather opus-5; done
["get_weather"]
["get_weather"]
["get_weather"]

Run

$ for i in 1 2 3; do weather opus-5-5; done
["get_weather"]
["get_weather"]
["get_weather"]

Type

🐛 Bug Fix

Caveats (if any)

Medium

Pre-existing: Invoke passes a caller's forced tool_choice to Opus 5.5 with response_format (Bedrock 400)

Low

Unforced tool use is best effort
Also changes thinking-on non-Claude Converse models (tested live)
Live runs used us. profiles only

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

…ol_choice can't be forced Opus 5.5 and Fable 5.1 reject a forced tool_choice on Bedrock, and litellm also skips the forced tool_choice whenever thinking is on, so the synthetic json_tool_call tool goes out unforced on those requests. With the default description the model often skips the tool and answers in plain text. When thinking is on or the model rejects forced tool use, give the tool a directive description, with the same condition in the Converse and Invoke routes. Other requests are unchanged

greptile-apps · 2026-09-23T08:57:58Z

Confidence Score: 4/5

The behavior appears sound, but the explicit repository no-mutation requirement must be satisfied before merging

Findings

Caller-Owned Tools Are Mutated ▶

Summary

This PR adds a shared directive for synthetic structured-output tools when Bedrock cannot force their selection

Converse preserves any schema description while prepending the directive
Invoke adds the directive after removing or omitting an unsupported forced tool choice
Regression tests cover Opus 5.5, Fable 5.1, thinking-enabled requests, and caller tools

_{Reviews (1) · Last reviewed commit: "fix(bedrock): return JSON for Claude Opu..."}

codspeed · 2026-09-23T09:00:28Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing kimnamu:bedrock_opus_5_5_response_format (0a9d0a1) with main (254c2f6)}

codecov · 2026-09-23T09:04:17Z

Codecov Report

❌ Patch coverage is 83.33333% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...tellm/llms/bedrock/chat/converse_transformation.py	33.33%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

…ting into it Greptile flagged the Invoke directive for mutating tool dicts in place. Anthropic-format tools reach map_openai_params as the caller's own dicts, so a caller tool that reused the reserved json_tool_call name would have carried the directive back into its later requests, and AGENTS.md rules out in-place mutation. Build a new dict for the json_tool_call tool and store the rebuilt list on optional_params, the way _add_tools_to_optional_params already does. Request bodies are unchanged.

kimnamu · 2026-09-24T07:33:44Z

Thanks for the quick review pass on this PR; the no-mutation point was worth acting on. Pushed 6bd579c so the Invoke path rebuilds the json_tool_call tool instead of writing into it. Request bodies are unchanged.

Review item	Verdict	Commit
P2 "Caller-owned tools are mutated" (`anthropic_claude3_transformation.py:133`)	Applied: build a new dict for the `json_tool_call` tool and store the rebuilt list on `optional_params`, like `_add_tools_to_optional_params` does. Guarded on `"tools" in optional_params` so `response_format: {"type": "text"}` still adds no `tools` key	`6bd579c`

Why the commit: at 036cfb951b the caller's non_default_params were already left intact and the synthetic tool is created per request, but Anthropic-format tools reach map_openai_params as the caller's own dicts, so a caller tool that reused the reserved name was rewritten. AGENTS.md rules out in-place mutation either way. No new test: the request body is identical for every input the existing tests cover, so a test could only pin the duplicate-name case shown below.

Checks

Caller-owned dict, before and after (Opus 5.5, Invoke, response_format plus an Anthropic-format caller tool that reuses the reserved name):

# 036cfb951b
caller non_default_params unchanged=True
json_tool_call same object across two calls: False
caller tool named json_tool_call: description after call = 'Always respond by calling this tool. Its input is your final answer; do not reply in plain text.'
# 6bd579c84e
caller non_default_params unchanged=True
json_tool_call same object across two calls: False
caller tool named json_tool_call: description after call = 'mine'

$ pytest -p no:xdist -q tests/unit/llms/bedrock/chat/invoke_transformations/test_bedrock_chat_invoke_transformations_anthropic_claude3_transformation.py tests/test_litellm/llms/bedrock/chat/test_converse_transformation.py
390 passed
$ pytest -q tests/test_litellm/llms/bedrock tests/unit/llms/bedrock
1934 passed, 1 skipped, 10 failed   # the 10 fail identically at 036cfb951b: ModuleNotFoundError for proxy extras not installed here
$ pytest -q tests/test_litellm/llms/anthropic/chat
502 passed, 1 failed                # same at 036cfb951b: litellm.proxy.proxy_server not importable here
$ ruff format --check; ruff check   # changed file
OK
$ python scripts/type_discipline_gate.py --base origin/main
OK: every LIT rule is within its codebase ceiling (base origin/main)
$ python scripts/ruff_strict_gate.py --base origin/main
OK: every strict rule is within its codebase ceiling (base origin/main)
$ python scripts/test_quality_gate.py --base origin/main
OK: every TQ rule is within its test-suite ceiling (base origin/main)

basedpyright on the changed file: 128 errors at 036cfb951b, 127 at 6bd579c84e, all pre-existing.

…onse_format # Conflicts: # tests/test_litellm/llms/bedrock/chat/test_converse_transformation.py

…onse_format

kimnamu requested a review from a team September 23, 2026 08:53

greptile-apps Bot reviewed Sep 23, 2026

View reviewed changes

Comment thread litellm/llms/bedrock/chat/invoke_transformations/anthropic_claude3_transformation.py Outdated

kimnamu mentioned this pull request Sep 23, 2026

fix(bedrock): drop temperature and top_p for OpenAI GPT-6 models on Converse #42726

Closed

6 tasks done

kimnamu added 4 commits September 24, 2026 18:26

ci: retrigger

79eef9a

Merge remote-tracking branch 'origin/main' into bedrock_opus_5_5_resp…

0bc63d9

…onse_format # Conflicts: # tests/test_litellm/llms/bedrock/chat/test_converse_transformation.py

Merge remote-tracking branch 'origin/main' into bedrock_opus_5_5_resp…

2b429de

…onse_format

ci: retrigger

0a9d0a1

Navigation Menu

Repository navigation

fix(bedrock): return JSON for Claude Opus 5.5 response_format when tool_choice can't be forced - #42718

fix(bedrock): return JSON for Claude Opus 5.5 response_format when tool_choice can't be forced#42718
kimnamu wants to merge 6 commits into
BerriAI:mainfrom
kimnamu:bedrock_opus_5_5_response_format

kimnamu commented Sep 23, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Sep 23, 2026 •

edited

Loading

Uh oh!

Uh oh!

codspeed Bot commented Sep 23, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Sep 23, 2026 •

edited

Loading

Uh oh!

kimnamu commented Sep 24, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

kimnamu commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Relevant issues

Affected release

Linear ticket

Pre-Submission checklist

Screenshots / Proof of Fix

Before (59db396)

Converse

Invoke

Responses

Messages

Streaming

Control

Caller tool

After (79eef9a)

Converse

Invoke

Responses

Messages

Streaming

Control

Caller tool

Type

Caveats (if any)

Medium

Low

Final Attestation

Uh oh!

greptile-apps Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 4/5

Findings

Summary

Uh oh!

Uh oh!

codspeed Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

codecov Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

kimnamu commented Sep 24, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

kimnamu commented Sep 23, 2026 •

edited

Loading

Before (`59db396`)

After (`79eef9a`)

greptile-apps Bot commented Sep 23, 2026 •

edited

Loading

codspeed Bot commented Sep 23, 2026 •

edited

Loading

codecov Bot commented Sep 23, 2026 •

edited

Loading