feat(bedrock): route OpenAI models to bedrock-runtime's native Chat Completions API by 6matt · Pull Request #43264 · BerriAI/litellm

6matt · 2026-09-26T01:25:41Z

TLDR

Problem this solves:

Bedrock OpenAI models (gpt-5.4–6) were forced through Converse for chat
Converse 400s on unsupported sampling params instead of dropping them
It also loses reasoning tokens and breaks implicit prompt caching

How it solves it:

Route these models to bedrock-runtime's native /openai/v1/chat/completions
Data-driven from the model's supported_endpoints; no model-name matching
Function-tools + reasoning (which the chat surface rejects on gpt-5.6/6) bridge to native /v1/responses, so no Converse regression
bedrock/converse/ stays an escape hatch to Converse

User Flow

Before: a developer sends a chat request with a temperature to a Bedrock GPT-6 model and gets a hard 400 instead of an answer.

They send POST https://litellm/v1/chat/completions with model: bedrock/global.openai.gpt-6-luna, temperature: 0.3, reasoning_effort: low
The gateway translates it to Bedrock Converse and calls .../model/global.openai.gpt-6-luna/converse
Bedrock returns 400 This model doesn't support the temperature field. Remove temperature and try again.

After: the same request succeeds; the unsupported sampling param is dropped and the native surface answers.

The developer sends the same POST with the same body
The gateway calls .../openai/v1/chat/completions, having dropped temperature (reasoning model) and kept reasoning_effort
A normal chat completion comes back with content and real usage (incl. reasoning tokens), logged at non-zero spend
A request that also carries function tools is instead sent to .../openai/v1/responses and comes back with normal tool_calls, unchanged from the caller's perspective

Relevant issues

Related [Bug]: Bedrock OpenAI GPT-5.6 (us.openai.*) — Responses routing hang + missing chat/stream transforms #37132 (the "missing chat/stream transforms" it reports on the Invoke/bridge path)
Related [Bug]: bedrock/converse hoists mid-conversation role:"system" messages and collapses implicit prompt caching (#32730 fix only covers Invoke); bedrock/invoke is broken for the OpenAI models #41043 (Converse hoists system messages and collapses implicit caching for these OpenAI models)

Type

🆕 New Feature

Screenshots / Proof of Fix

Shared setup: AWS profile with Bedrock access in us-east-1; LITELLM_LOCAL_MODEL_COST_MAP=True to load the price-map entries in this PR. model="bedrock/global.openai.gpt-6-luna". drop_params=True. Same request bodies on both sides.

Before (`a11a93f`)

chat non-stream (temperature + reasoning)

litellm.completion(..., temperature=0.3, reasoning_effort="low", max_completion_tokens=400)
Request hits .../model/global.openai.gpt-6-luna/converse
Raises litellm.BadRequestError: BedrockException - {"message":"This model doesn't support the temperature field. Remove temperature and try again."}

function tools + reasoning

litellm.completion(..., tools=[get_weather], reasoning_effort="low", max_completion_tokens=600)
Request hits .../model/global.openai.gpt-6-luna/converse
Returns a get_weather tool call (Converse served this)

After (`5678141`)

chat non-stream (temperature + reasoning)

Same call as Before
Request hits .../openai/v1/chat/completions (temperature dropped, reasoning_effort kept)
Returns content: "ok"

chat stream

Same call with stream=True
Request hits .../openai/v1/chat/completions
Streamed deltas assemble to "stream ok"

function tools + reasoning

Same call as Before
Request is bridged to .../openai/v1/responses
Returns finish_reason: tool_calls, object: chat.completion, get_weather({"city":"Paris"}) — unchanged shape for the caller, no Converse

Caveats (if any)

Low

Scope: gpt-5.4/5.5/5.6/6 cross-Region inference profiles only; bare on-demand ids excluded (Bedrock rejects them)
Function-tools + active reasoning on gpt-5.6/6 is served via the native /v1/responses bridge (chat surface rejects it); gpt-5.4/5.5 serve it on chat directly
Version boundary (gpt-5.6+) is measured against bedrock-runtime, not advertised in metadata; a future model shifting it would need the check updated

QA runbook

tests/unit/llms/bedrock/chat/test_bedrock_openai_native_chat.py — a model advertising /v1/chat/completions routes to the native config/URL and drops reasoning-model sampling params; function-tools + active reasoning on gpt-5.6/6 bridges to /v1/responses while gpt-5.4/5.5, reasoning=none, and no-tools requests stay on chat; the converse/ prefix stays an escape hatch
- Register a model with supported_endpoints: ["/v1/chat/completions", "/v1/responses"]
- Send /v1/chat/completions with temperature=0.5 + reasoning_effort=low; expect temperature dropped, request to .../openai/v1/chat/completions
- Send the same with function tools + reasoning_effort=low; expect the request to .../openai/v1/responses and normal tool_calls back
- Send bedrock/converse/<model>; expect it stays on Converse
- Sanity check: this test makes sense to add and is not hand-wavey or flaky

…ompletions API bedrock-runtime serves the OpenAI models (gpt-5.4/5.5/5.6/6) on an OpenAI-compatible surface at /openai/v1/chat/completions, alongside Converse. Previously the bedrock provider translated every chat request into Converse, which is lossy for these models: it 400s on unsupported sampling params instead of dropping them, loses reasoning tokens, and hoists mid-conversation system messages (breaking implicit prompt caching). Add BedrockOpenAIChatConfig, the Chat Completions sibling of BedrockOpenAIResponsesConfig. It inherits OpenAIGPT5Config (the same reasoning param mapper the openai provider uses) and overrides only the endpoint URL, Bearer/SigV4 auth, and AWS-param stripping. Routing is data-driven from the model's supported_endpoints, via a shared bedrock_uses_native_openai_chat predicate used by the config selector, the completion dispatcher, and get_optional_params. An explicit bedrock/converse/<model> prefix stays an escape hatch to the Converse translation. Enables the surface for the 16 gpt-5.4/5.5/5.6/6 cross-Region inference-profile entries that advertise /v1/chat/completions.

…sponses API bedrock-runtime's native chat surface rejects function tools while reasoning is active on gpt-5.6/6 ("use /v1/responses or set reasoning_effort to 'none'"), which Converse previously served. Rather than regress that path, extend the existing responses_api_bridge_check (already used for openai/azure with the same constraint) to fire for bedrock: a chat request with function tools + active reasoning on a model that supports the native Responses surface is bridged to /v1/responses, which handles tools + reasoning and returns proper tool_calls. The version boundary is measured against bedrock-runtime (gpt-5.6+ reject; 5.4/5.5 serve tools with reasoning natively) via bedrock_chat_rejects_function_tools_while_reasoning. Custom tools, reasoning_effort "none", and non-tool requests stay on native chat completions.

greptile-apps · 2026-09-26T01:30:17Z

Confidence Score: 5/5

[High risk] Routes OpenAI models through a new Bedrock chat endpoint.

The PR appears safe to merge based on this re-review

Summary

Routes eligible Bedrock OpenAI chat models to the native Chat Completions endpoint, with a Responses bridge for function tools used with reasoning. The latest changes remove a redundant comment and strengthen the signed-body test

_{Reviews (6) · Last reviewed commit: "test(bedrock): assert exact model field ..."}

…ANN204)

veria-ai · 2026-09-26T01:34:46Z

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 2 · PR risk: 0/10

…; trim comment Address review: move the gpt-5.6/6 "native chat rejects function tools while reasoning" boundary out of a hardcoded version check and into a bedrock_chat_rejects_function_tools_while_reasoning flag on the price-map entries, read like the other data-driven Bedrock capabilities, so onboarding a model stays a JSON change. Add a wire-level test asserting the SigV4-signed streaming body carries stream:true, and drop a narrating dispatch comment.

6matt · 2026-09-26T01:38:03Z

@greptileai Pushed 25d0ab3 addressing the review: moved the tools+reasoning boundary to a price-map flag, verified the streaming-body concern is a false positive (added a wire-level test), and trimmed the narrating comment. Please re-review.

codecov · 2026-09-26T01:39:34Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

…ta on native chat validate_environment passed caller headers through untouched, so a caller could set X-Amzn-Bedrock-Request-Metadata and have it signed and recorded in AWS billing / CloudTrail, forging identity attribution. Reuse the existing ownership merge (bedrock_request_metadata_headers + merge_bedrock_invoke_headers, as the Invoke OpenAI path does): the proxy-owned header is dropped from caller input (case-insensitively) and re-supplied only from trusted litellm_params.

…ster price-map key - Add coverage for the get_optional_params branch, the _complete_bedrock native dispatch, and predicate edge cases (None model, non-converse route). - Drop untyped **kwargs from BedrockOpenAIChatConfig.__init__ (BaseAWSLLM takes no kwargs; every call site is arg-less) to satisfy the typed-parameter guideline. - Register bedrock_chat_rejects_function_tools_while_reasoning in the schema generator's boolean key table and regenerate the price-map schema.

6matt · 2026-09-26T01:47:16Z

@greptileai Pushed through 55f819c: fixed the request-attribution forgery (proxy now owns the X-Amzn-Bedrock-Request-Metadata header via the existing ownership merge), typed __init__, added patch coverage for the dispatch/param-mapping/predicate branches, and registered the new price-map boolean key in the schema generator. Please re-review.

… native chat The model rides in the request body on the native /openai/v1/chat/completions surface (not the URL like Converse/Invoke), and the generic handler merges extra_body over the body before SigV4 signing. bedrock isn't in the extra_body model-stripping path, so a key limited to one model could send extra_body={"model": "<other>"} and invoke another model on the proxy's AWS credentials. Pin the authorized model back in sign_request (post-merge, pre-sign), covering both the SigV4 and Bearer paths.

6matt · 2026-09-26T01:51:55Z

@greptileai Pushed 582fe71 fixing the extra_body model-authorization bypass on the native chat surface (sign_request now pins the authorized model post-merge/pre-sign). Please re-review.

greptile-apps · 2026-09-26T01:54:46Z

Want your agent to iterate on Greptile's feedback? Try greploops.

…-openai-chat

codspeed · 2026-09-26T02:05:33Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing 6matt:feat/bedrock-native-openai-chat (ad1f2a4) with main (d08746f)}

…tation - Add tests for custom_llm_provider, get_error_class, the non-reasoning param fallbacks, and the SigV4 sign_request path (brings the adapter to 100% patch coverage). - Move the model-pin suppression inline on the mutated line as `# rebind-ok` (LIT011) and drop the misplaced block comment.

6matt · 2026-09-26T02:06:19Z

@greptileai Pushed 4466a96. Also merged the latest main (cost-map guard was failing on the merge because the schema was stale against the merged cost map — regenerated), brought the adapter to 100% patch coverage, and added the # rebind-ok justification. Please re-review.

…t comment

6matt · 2026-09-26T02:12:54Z

@greptileai Pushed c7d2829: dropped the redundant model-pin comment and tightened the signed-body assertion to check the exact "model" field. Please re-review.

…-openai-chat

The cost-map-guard runs under pull_request_target with the BASE branch's schema generator (PR code is never executed, only the cost map is read as data), so a price-map key a PR introduces can never be classified by that generator -- the guard fails every run with "Unclassified keys". Move the gpt-5.6+ boundary back into bedrock_chat_rejects_function_tools_while_reasoning as a version check on the normalized OpenAI model name, matching the openai/azure arms already in the same bridge, and drop the flag from the 12 cost-map entries, the schema generator, and the schema. No new cost-map key, so the guard passes.

krrish-berri-2 · 2026-09-27T04:09:45Z

seems like a dup of #40775

6matt · 2026-09-27T17:32:33Z

@krrish-berri-2 mostly, but #40775 doesn't include GPT 6 and it routes function-tools + reasoning to converse rather than responses. I can close this one and create a followup after the #40775 lands

6matt added 2 commits September 25, 2026 21:12

6matt requested a review from a team September 26, 2026 01:25

6matt requested review from kerry-berri, mateo-berri and ryan-crabbe-berri as code owners September 26, 2026 01:25

6matt mentioned this pull request Sep 26, 2026

feat(bedrock): route OpenAI models to bedrock-runtime's native Chat Completions API 6matt/litellm#1

Closed

5 tasks

style(bedrock): add __init__ return annotation for ruff strict gate (…

4b67e19

…ANN204)

greptile-apps Bot reviewed Sep 26, 2026

View reviewed changes

Comment thread litellm/llms/bedrock/chat/openai_native/transformation.py

Comment thread litellm/llms/bedrock/common_utils.py

Comment thread litellm/main.py

Comment thread litellm/main.py Outdated

Comment thread tests/unit/llms/bedrock/chat/test_bedrock_openai_native_chat.py

veria-ai Bot reviewed Sep 26, 2026

View reviewed changes

Comment thread litellm/llms/bedrock/chat/openai_native/transformation.py Outdated

greptile-apps Bot reviewed Sep 26, 2026

View reviewed changes

Comment thread litellm/llms/bedrock/chat/openai_native/transformation.py Outdated

veria-ai Bot reviewed Sep 26, 2026

View reviewed changes

Comment thread litellm/main.py

greptile-apps Bot reviewed Sep 26, 2026

View reviewed changes

Comment thread litellm/llms/bedrock/chat/openai_native/transformation.py Outdated

Merge remote-tracking branch 'upstream/main' into feat/bedrock-native…

33c4c47

…-openai-chat

greptile-apps Bot reviewed Sep 26, 2026

View reviewed changes

Comment thread litellm/llms/bedrock/chat/openai_native/transformation.py Outdated

Comment thread tests/unit/llms/bedrock/chat/test_bedrock_openai_native_chat.py Outdated

style(bedrock): keep the model-pin line single-line for ruff format

cab8ae0

6matt added 3 commits September 25, 2026 22:09

style(bedrock): ruff-format the native-chat test file

bdcc4ae

test(bedrock): assert SigV4-signed body carries the pinned model

0d9ceab

test(bedrock): assert exact model field in signed body; drop redundan…

c7d2829

…t comment

6matt added 3 commits September 25, 2026 22:19

Merge remote-tracking branch 'upstream/main' into feat/bedrock-native…

b6959df

…-openai-chat

fix(cost-map): dedupe perceptron-mk1.5, keep newer capability flags

ad1f2a4

6matt closed this Sep 27, 2026

devin-ai-integration Bot mentioned this pull request Sep 28, 2026

feat(bedrock): pass chat through to each model's native openai endpoint on runtime and mantle #43581

Closed

4 of 6 tasks

6matt mentioned this pull request Oct 9, 2026

feat(bedrock): bridge gpt-5.6+ function tools with reasoning to native /responses instead of Converse #45609

Open

6 of 7 tasks

Conversation

6matt commented Sep 26, 2026

TLDR

User Flow

Relevant issues

Type

Screenshots / Proof of Fix

Before (a11a93f)

chat non-stream (temperature + reasoning)

function tools + reasoning

After (5678141)

chat non-stream (temperature + reasoning)

chat stream

function tools + reasoning

Caveats (if any)

Low

QA runbook

Uh oh!

greptile-apps Bot commented Sep 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

veria-ai Bot commented Sep 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR overview

Security review

Uh oh!

6matt commented Sep 26, 2026

Uh oh!

codecov Bot commented Sep 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

6matt commented Sep 26, 2026

Uh oh!

6matt commented Sep 26, 2026

Uh oh!

Uh oh!

greptile-apps Bot commented Sep 26, 2026

Uh oh!

codspeed Bot commented Sep 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

6matt commented Sep 26, 2026

Uh oh!

Uh oh!

Uh oh!

6matt commented Sep 26, 2026

Uh oh!

krrish-berri-2 commented Sep 27, 2026

Uh oh!

6matt commented Sep 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Before (`a11a93f`)

After (`5678141`)

greptile-apps Bot commented Sep 26, 2026 •

edited

Loading

veria-ai Bot commented Sep 26, 2026 •

edited

Loading

codecov Bot commented Sep 26, 2026 •

edited

Loading

codspeed Bot commented Sep 26, 2026 •

edited

Loading