Skip to content

fix(bedrock): drop toolSpec.strict for Claude Sonnet 5 and Fable 5 - #32455

Open
seanturner83 wants to merge 2 commits into
BerriAI:litellm_internal_stagingfrom
seanturner83:fix/bedrock-converse-drop-strict-sonnet5-fable5
Open

fix(bedrock): drop toolSpec.strict for Claude Sonnet 5 and Fable 5#32455
seanturner83 wants to merge 2 commits into
BerriAI:litellm_internal_stagingfrom
seanturner83:fix/bedrock-converse-drop-strict-sonnet5-fable5

Conversation

@seanturner83

Copy link
Copy Markdown

Relevant issues

Follow-up to #31923 (Opus 4.7/4.8) and #31943 (Sonnet 4) — same root cause, two more models.

Pre-Submission checklist

  • I have added meaningful tests
  • My PR passes all CI/CD checks (lint, format, unit tests)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem

What this fixes

Bedrock Converse routes Claude Sonnet 5 and Fable 5 through the same Anthropic-compatible validator as Opus 4.7/4.8 and Sonnet 4, which rejects toolSpec.strict even though Anthropic's native API accepts strict as a top-level tool field:

tools.0.custom.strict: Extra inputs are not permitted

Both models were missing the bedrock_converse_supports_strict_tools: false opt-out flag, so the default strict-forwarding path 400s any tool-calling request against bedrock/*.claude-sonnet-5 or bedrock/*.claude-fable-5 (SDKs like openai-agents stamp strict on every tool, so it fires universally at the first call).

The change

  • Add bedrock_converse_supports_strict_tools: false to all bedrock_converse Sonnet 5 (6) and Fable 5 (4) inference-profile entries in model_prices_and_context_window.json and its backup.
  • Extend the existing strict-tools regression tests (test_bedrock_converse_strict_tools_opus_47_48.py) to cover Sonnet 5 / Fable 5 in the dropped-list, the helper check, and the cost-map-flag check.

No logic change — this reuses the exact mechanism from #31923/#31943; the models just needed the flag.

Screenshots / Proof of Fix

Live Bedrock Converse calls (us-east-1, real $), captured at commit bbb8af5:

# us.anthropic.claude-sonnet-5
  BEFORE (strict forwarded, as litellm does today): 400 ValidationException — tools.0.custom.strict: Extra inputs are not permitted
  AFTER  (strict stripped, what this flag does):     200 OK (stopReason=end_turn)

# us.anthropic.claude-fable-5
  BEFORE (strict forwarded, as litellm does today): 400 ValidationException — tools.0.custom.strict: Extra inputs are not permitted
  AFTER  (strict stripped, what this flag does):     200 OK (stopReason=end_turn)

Confirmed for both us. and global. inference-profile variants of each model. Unit tests: pytest tests/test_litellm/litellm_core_utils/prompt_templates/test_bedrock_converse_strict_tools_opus_47_48.py → 44 passed.

Type

🐛 Bug Fix

Co-authored with Claude Opus 4.8.

Bedrock Converse routes Claude Sonnet 5 and Fable 5 through the same
Anthropic-compatible validator as Opus 4.7/4.8 and Sonnet 4, which rejects
`toolSpec.strict` even though Anthropic's native API accepts `strict` as a
top-level tool field:

    tools.0.custom.strict: Extra inputs are not permitted

Both were missing the `bedrock_converse_supports_strict_tools: false`
opt-out flag, so the blanket strict-forwarding path 400s any tool-calling
request against `bedrock/*.claude-sonnet-5` / `bedrock/*.claude-fable-5`.

Verified against live Bedrock Converse (us-east-1): with `strict` present
→ 400 (above error); with `strict` stripped → 200. Confirmed for both the
`us.` and `global.` inference-profile variants of each model.

Add the flag to all Bedrock (`bedrock_converse`) Sonnet 5 and Fable 5
entries in model_prices_and_context_window.json and its backup, and extend
the strict-tools regression tests to cover them.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds bedrock_converse_supports_strict_tools: false to all Bedrock inference-profile entries for Claude Sonnet 5 (6 entries) and Claude Fable 5 (4 entries) in both the primary and backup model-pricing JSON files, preventing toolSpec.strict from being forwarded to Bedrock Converse where it causes a ValidationException. Tests are extended to cover the new models in all three test functions.

Confidence Score: 5/5

Safe to merge — the change is additive JSON data only, touching no runtime logic, and every new entry is covered by tests.

The fix follows an established, tested pattern from two prior PRs. Both JSON files are kept in sync, all 10 new cost-map keys have direct test coverage across three test functions, and the live-call evidence in the PR description confirms the before/after behaviour on real Bedrock endpoints.

No files require special attention.

Important Files Changed

Filename Overview
model_prices_and_context_window.json Adds bedrock_converse_supports_strict_tools: false to 10 model entries (6 Sonnet 5, 4 Fable 5); consistent with existing flag placement for Opus 4.7/4.8 and Sonnet 4.
litellm/model_prices_and_context_window_backup.json Identical flag additions as the primary JSON; the two files are kept in sync.
tests/test_litellm/litellm_core_utils/prompt_templates/test_bedrock_converse_strict_tools_opus_47_48.py Extends parametrized tests for all three test functions to cover Sonnet 5 and Fable 5; function docstring updated; module-level docstring still only mentions Opus 4.7/4.8 and Sonnet 4.

Reviews (2): Last reviewed commit: "test(bedrock): cover all Sonnet 5 / Fabl..." | Re-trigger Greptile

@codecov

codecov Bot commented Jul 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@codspeed-hq

codspeed-hq Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will improve performance by 13.29%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 29 untouched benchmarks

Performance Changes

Benchmark BASE HEAD Efficiency
test_completion_simple_message 4.6 ms 4.1 ms +13.29%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing seanturner83:fix/bedrock-converse-drop-strict-sonnet5-fable5 (70ee6ff) with litellm_internal_staging (cd6e8cd)

Open in CodSpeed

…ocstring

Address Greptile review on the strict-tools regression test:
- Add the 4 regional variants that were in the cost-map flag test but missing
  from the end-to-end _bedrock_tools_pt strip test (au./jp. Sonnet 5,
  global./eu. Fable 5) — that test is the only one exercising the full strip
  pipeline, so those profiles had partial coverage.
- Add the bare anthropic.claude-fable-5 key to the helper assertion.
- Update the stale docstring to include Sonnet 5 and Fable 5.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@seanturner83

Copy link
Copy Markdown
Author

Thanks @greptileai — addressed both gaps in 70ee6ff:

  • Added the 4 missing regional variants to test_bedrock_tools_pt_strict_dropped_for_strict_unsupported_models (the full-pipeline test): au./jp. Sonnet 5 and global./eu. Fable 5.
  • Added the bare anthropic.claude-fable-5 key to test_bedrock_converse_supports_strict_tools_helper.
  • Fixed the stale docstring to include Sonnet 5 and Fable 5.

All 10 cost-map keys now have end-to-end _bedrock_tools_pt coverage. Tests: 48 passed.

@greptileai

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant