feat(bedrock): serve the OpenAI models on bedrock-runtime's native Responses API

[leonardofreitass]

TLDR

Problem this solves:

• bedrock has no Responses config, so /v1/responses rides the Converse bridge
• A Codex session's tool history fails outright on that path
• Codex history item types are rejected by both Bedrock endpoints

How it solves it:

• Add a Responses config for bedrock-runtime's /openai/v1/responses surface
• Opt in per model from the price map, so unsignalled models keep the bridge
• Share the Codex history normalization with bedrock_mantle

User Flow

Before: a developer running Codex CLI through a LiteLLM gateway on a Bedrock GPT-5.6 model cannot get past the first tool call.

1. They start Codex against POST https://litellm-domain/v1/responses with a bedrock/global.openai.gpt-5.6-sol deployment
2. Turn one works; the model calls a tool
3. On turn two, with that tool call in history, the request fails: BedrockException - The toolConfig field must be defined when using toolUse and toolResult content blocks
4. The session is stuck — every subsequent turn carries the same history and fails the same way

After: the same session continues normally.

1. They start Codex against the same POST https://litellm-domain/v1/responses with the same deployment
2. Turn one works
3. Turn two succeeds, and the assistant answers using prior-turn context
4. The session proceeds across many turns

Relevant issues

Related: #29818, #36182

Linear ticket

Pre-Submission checklist

• I have added meaningful tests
• The handful of test files covering my change pass locally: uv run pytest tests/test_litellm/llms/bedrock/responses/ tests/test_litellm/llms/base_llm/responses/ tests/test_litellm/llms/bedrock_mantle/ -v (252 pass; 42 new, both new files at 100% patch coverage). The wider Responses suites pass too: tests/test_litellm/responses/ tests/test_litellm/llms/openai/responses/ (1007 pass)
• My PR passes all required CI/CD checks — verified locally against upstream/main: format-check-changed, ruff (litellm + tests config), ruff-strict ratchet, type-discipline ratchet, test-quality ratchet, basedpyright (139808, no higher than base), circular-imports, import-safety
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 — 3/5 on an earlier commit, pending a re-review of this one. Both findings it raised are addressed: provider policy moved onto the adapter via for_model(), and the authorization dependency that prompted the second finding is no longer this PR's (see Caveats → Low).

Screenshots / Proof of Fix

End-to-end against real Amazon Bedrock. No mocks. Multi-turn, because the item types at issue are history items — a first-turn request succeeds on both sides and hides the problem entirely.

Shared setup — bedrock/global.openai.gpt-5.6-sol in us-east-1, called via litellm.aresponses, with a Codex-shaped history:

input = [
    {"role": "user", "content": "Remember the project codename I gave you."},
    {"type": "agent_message", "role": "assistant",
     "content": [{"type": "output_text", "text": "Noted. The project codename is BLUEBIRD."}]},
    {"type": "local_shell_call", "call_id": "call_1", "action": {"command": ["ls"]}},
    {"type": "function_call_output", "call_id": "call_1", "output": "README.md"},
    {"role": "user", "content": "What is the project codename? Reply with only the codename."},
]

Before (cd63c7e)

1. Run the request above against bedrock/global.openai.gpt-5.6-sol
2. Observed: BadRequestError: BedrockException - {"message":"The toolConfig field must be defined when using toolUse and toolResult content blocks."}
3. The request never reaches the model — the bridge turned the Codex tool history into Converse toolUse/toolResult blocks with no toolConfig

After (b6f84cb)

1. Run the identical request
2. Observed: HTTP 200, output item types ['message'], usage in=84 out=7
3. Observed answer: 'BLUEBIRD' — the prior turn survived, normalized into a supported item type

Measured on b6f84cb4e0, the commit before the rebase onto main. The rebase changed only bedrock_mantle, not the bedrock-runtime path this exercises, so the result stands for the current commit; I have cited the ref actually run rather than the tip.

Prompt caching, verified end-to-end

Asked about in this comment. Explicit prompt caching works over this adapter unchanged. Two litellm.aresponses calls against bedrock/global.openai.gpt-5.6-luna, same prompt_cache_key, identical ~1.2K-token prefix carrying prompt_cache_breakpoint:

call 1:  input_tokens=1217  cached_tokens=0     cache_write_tokens=1205
call 2:  input_tokens=1217  cached_tokens=1205  cache_write_tokens=0

prompt_cache_key, prompt_cache_options and prompt_cache_breakpoint reach the wire unmodified — the normalizer returns non-Codex items by identity — and usage comes back on usage.input_tokens_details. Same result on gpt-5.6-sol. No code change was needed for this.

Error responses keep the Bedrock request id

get_error_class routes through BedrockError rather than the OpenAI base, which builds a blank response. Verified against a real 400 from bedrock-runtime: x-amzn-RequestId: 3e832927-48fb-4231-abd2-03fe9ac92d18 survives to the caller, where it was previously dropped.

Type

🆕 New Feature

Caveats (if any)

Medium

• Chat Completions on bedrock-runtime is not addressed; this covers /v1/responses only
• The two Bedrock endpoints do not share one input validator, so each provider opts in explicitly rather than inheriting. Verified against bedrock-runtime with global.openai.gpt-5.6-sol: additional_tools is accepted there while agent_message, context_compaction and local_shell_call are rejected with 400 Invalid 'input': value did not match any expected variant. On bedrock-mantle, additional_tools is rejected.

Low

• Rebased from litellm_internal_staging onto main and retargeted, since main is now the repository's default branch. bedrock_mantle was the one conflict: fix(responses): hoist Codex additional_tools input items into the chat bridge tools #40989 replaced its private _hoist_codex_additional_tools with the shared hoist_additional_tools, which is the same method this PR was refactoring. Resolved by keeping that shared hoist and layering this PR's shared history normalizer on top, so the file now uses both shared helpers and carries neither copy of its own.
• An earlier revision of this PR carried a Severe caveat asking that it merge after fix(responses): lift additional_tools input items into tools on the chat bridge #38388, which extended extract_request_tool_names to descend into additional_tools. That is no longer this PR's dependency. fix(responses): lift additional_tools input items into tools on the chat bridge #38388 was closed in favour of fix(responses): hoist Codex additional_tools input items into the chat bridge tools #40989, which landed the hoist but not the extractor change, so the extractor still reads only top-level tools while nested tools are hoisted into the live tool list — meaning a key restricted by metadata.allowed_tools does not see them. That is the state of main on the bridge route for every bridged provider, independent of this PR, which touches no file in that path. Probed on this branch: nested tools give [] where an equivalent top-level tool gives its name. Worth a separate issue; happy to open one, or to port the extractor half of fix(responses): lift additional_tools input items into tools on the chat bridge #38388 if a maintainer prefers it here.
• supported_endpoints is the only signal that selects this surface. An earlier note here referenced test_bedrock_gpt_5_6_advertises_only_converse_supported_features, which asserted the field was absent; 3023497 removed it from the tree while this PR was open. The opt-in is covered directly by this PR's own suite instead, which reads the shipped map.
• cache_write_tokens reaches the caller through extra="allow" on InputTokensDetails rather than a declared field, so it passes through but is not part of a typed contract. Separately, these price-map entries do not set supports_prompt_cache_breakpoint, so the cache_control → breakpoint translation in AnthropicCacheControlHook does not apply to them; passing prompt_cache_breakpoint directly does work, as measured above. Both are pre-existing and out of scope here — happy to set the flag if a maintainer wants it in this PR.
• Follow-up: fix(responses): hoist Codex additional_tools input items into the chat bridge tools #40989 created litellm/responses/additional_tools.py as the shared home for Codex tool hoisting, while this PR's history normalizer sits in litellm/llms/base_llm/responses/codex_compat.py. Those two belong together; consolidating them is a small follow-up either way round.
• bedrock_mantle keeps its exact warning wording. The shared normalizer is a pure transform returning the rewritten types, and each caller logs in its own words, so all bedrock_mantle tests pass unchanged.

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR adds a native OpenAI Responses adapter for Bedrock Runtime, selects it through model capability metadata, and shares Codex history normalization with Bedrock Mantle.

• Registers and exports the new Bedrock Responses configuration.
• Adds bearer-token and SigV4 authentication plus native endpoint construction.
• Marks GPT-5.6 Bedrock profiles as supporting /v1/responses.
• Adds coverage for adapter selection, authentication, URLs, and history normalization.

Confidence Score: 3/5

This PR is not safe to merge until the stated authorization dependency is present; the provider-specific selection logic should also be moved under the Bedrock adapter.

The native route currently permits nested tool declarations to reach Bedrock without being included in the existing allowed-tool evaluation, while the shared utility layer also takes on Bedrock-specific capability policy.

Files Needing Attention: litellm/utils.py and litellm/llms/bedrock/responses/transformation.py

Security Review

The new route must not merge until its stated authorization dependency is present; otherwise restricted requests can reach tools that were not included in policy evaluation.

Important Files Changed

Filename	Overview
litellm/llms/bedrock/responses/transformation.py	Introduces the native Bedrock Runtime Responses adapter, endpoint construction, authentication, and Codex history normalization.
litellm/llms/base_llm/responses/codex_compat.py	Extracts Codex history-item normalization into a shared pure transformation with focused tests.
litellm/llms/bedrock_mantle/responses/transformation.py	Reuses the shared normalizer while preserving Mantle-specific tool hoisting and warning behavior.
litellm/utils.py	Selects the native adapter through capability metadata, but exposes the acknowledged authorization dependency and places provider-specific policy in shared code.
litellm/llms/bedrock/common_utils.py	Adds metadata-driven detection of Bedrock models supporting the native Responses endpoint.
model_prices_and_context_window.json	Opts six regional GPT-5.6 profiles into the native Responses surface.

_{Reviews (1): Last reviewed commit: "feat(bedrock): serve the OpenAI models o..." | Re-trigger Greptile}

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing leonardofreitass:feat/bedrock-openai-responses-endpoint (48cf730) with main (c8114ba)}

[hil00137]

@leonardofreitass Thanks for adding the native Bedrock Runtime
Responses adapter. We are evaluating it for a Hindsight retain
workload and would like to confirm the end-to-end prompt-caching
behavior.

• Provider: bedrock
• API: LiteLLM Responses API (aresponses())
• Model: global.openai.gpt-5.6-luna
• Authentication: AWS IAM/SigV4
• Endpoint: bedrock-runtime.{region}.amazonaws.com/openai/v1/responses

Our main question is whether GPT-5.6 Luna prompt caching works end-to-end through this adapter.

According to the Amazon Bedrock documentation, GPT-5.6 Luna supports explicit prompt caching with:

• A minimum cacheable prefix of 1,024 tokens per breakpoint
• Up to four breakpoints per request
• prompt_cache_breakpoint on input_text, input_image, and input_file blocks
• A default 30-minute cache TTL
• A stable prompt_cache_key to route requests to the same cache

Documentation:
https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html

For a stable prefix of at least 1,024 tokens, could you confirm whether the following request structure is supported and transmitted unchanged to the Bedrock Runtime Responses endpoint?

{
  "model": "global.openai.gpt-5.6-luna",
  "prompt_cache_key": "hindsight:retain:stable-prefix-v1",
  "prompt_cache_options": {
    "mode": "explicit",
    "ttl": "30m"
  },
  "input": [
    {
      "type": "message",
      "role": "developer",
      "content": [
        {
          "type": "input_text",
          "text": "<stable prefix of at least 1,024 tokens>",
          "prompt_cache_breakpoint": {
            "mode": "explicit"
          }
        }
      ]
    },
    {
      "type": "message",
      "role": "user",
      "content": [
        {
          "type": "input_text",
          "text": "<dynamic retain content>"
        }
      ]
    }
  ]
}

In particular, for global.openai.gpt-5.6-luna, is the following behavior supported and tested?

1. The prompt_cache_key is accepted and can be reused across requests with the same stable prefix.
2. The prompt_cache_options.mode="explicit" and ttl="30m" fields are accepted by the native Bedrock Runtime Responses path.
3. The prompt_cache_breakpoint field survives LiteLLM input validation and request transformation.
4. The breakpoint is sent unchanged on the wire to: bedrock-runtime.{region}.amazonaws.com/openai/v1/responses
5. The first request exposes cache-write usage when the stable prefix meets the 1,024-token requirement.
6. A subsequent request using the same prompt_cache_key and identical stable prefix exposes cache-read usage.
7. The cache usage is returned through LiteLLM in a usable form, such as: usage.input_tokens_details.cached_tokens and usage.input_tokens_details.cache_write_tokens.
8. The behavior applies to global.openai.gpt-5.6-luna, not only to gpt-5.6-sol.
9. The behavior works with AWS IAM/SigV4 authentication without requiring a different authentication flow.

If explicit prompt-cache breakpoint handling, prompt_cache_key propagation, or cache usage propagation is outside the scope of this PR, could you clarify whether these are planned for a follow-up change?

[leonardofreitass]

@hil00137 Thanks for the detailed question. I ran your exact request structure against real Bedrock Runtime rather than answering from the code, so the numbers below are from billed calls.

• Provider: bedrock
• API: LiteLLM Responses API (aresponses())
• Model: global.openai.gpt-5.6-luna
• Authentication: AWS IAM/SigV4
• Region: us-east-1
• Commit: b6f84cb4e0

All nine behaviors are supported. Nothing in this PR needs to change for prompt caching to work.

1. prompt_cache_key is accepted and reused. Two calls with the same key and the same stable prefix hit the same cache.
2. prompt_cache_options.mode="explicit" and ttl="30m" are accepted. Both appear in the supported-params list for this provider and reach the request top level unmodified.
3. prompt_cache_breakpoint survives validation and transformation. The Codex history normalizer this PR adds only rewrites the three item types Bedrock rejects; every other item is returned by object identity, so a message item carrying a breakpoint is the same object on the way out.
4. The breakpoint is sent unchanged to bedrock-runtime.us-east-1.amazonaws.com/openai/v1/responses. I inspected the signed request body, and the prompt_cache_breakpoint key is intact on the input_text block.
5. The first request exposes cache-write usage with a 1,205-token prefix.
6. A second request with the same prompt_cache_key and identical prefix exposes cache-read usage.
7. Usage is returned on usage.input_tokens_details, as cached_tokens and cache_write_tokens. See the caveat below on cache_write_tokens.
8. It applies to global.openai.gpt-5.6-luna. I ran the same test on global.openai.gpt-5.6-sol and got the same result, so this is family-wide rather than model-specific.
9. It works with AWS IAM/SigV4. Every call below was SigV4-signed through this adapter's sign_request. No separate authentication flow is involved.

Measured usage for 5, 6 and 7:

call 1:  input_tokens=1217  cached_tokens=0     cache_write_tokens=1205
call 2:  input_tokens=1217  cached_tokens=1205  cache_write_tokens=0

I also ran three controls, since a cache hit on its own does not prove the fields did the work:

• Same prefix with no prompt_cache_breakpoint and no prompt_cache_key: cached_tokens=1554. Bedrock caches this prefix implicitly, so a hit alone is not evidence that your fields were honored.
• Same prefix under a different prompt_cache_key: cached_tokens=0, cache_write_tokens=1205. The key does partition the cache, which is what the previous control could not show.
• Same setup on global.openai.gpt-5.6-sol: cached_tokens=1545.

Two caveats, both pre-existing and outside this PR:

• cache_write_tokens reaches you through extra="allow" on InputTokensDetails rather than a declared field. It is present and correct, but it is not part of a typed contract, so a strict client should read it defensively.
• The price-map entries for these models do not set supports_prompt_cache_breakpoint. That flag gates AnthropicCacheControlHook, which translates cache_control into a breakpoint. Passing prompt_cache_breakpoint directly, as in your JSON, works today. Passing cache_control and expecting it to be translated does not. I am happy to set the flag in this PR if a maintainer wants it here.

To your closing question: none of this is deferred to a follow-up. It works on this branch as written.

[hil00137]

@leonardofreitass
Thanks for verifying this end-to-end with
global.openai.gpt-5.6-luna and IAM/SigV4.

The cache write/read results are exactly what we needed. We appreciate
your help and hope the PR gets merged soon.

[pccampos009]

Thank you for this PR — it is the missing piece for Bedrock GPT-5.6 on bedrock-runtime.

We hit the AWS validation error in production-shaped traffic:

Function tools with reasoning_effort are not supported for global.openai.gpt-5.6-* in /v1/chat/completions. To use function tools, use /v1/responses or set reasoning_effort to none.

On current LiteLLM, even POST /v1/responses for bedrock/openai/global.openai.gpt-5.6-luna still rides the chat-completions emulation, so the same 400 comes back. The only working path today is bedrock_mantle (In-Region). This adapter is what lets us move those aliases onto runtime global.* / us.* with tools and reasoning together.

Could maintainers please review and approve this as soon as you can? The #38388 merge-order caveat is gone (#40989 landed), and the prompt-cache check on global.openai.gpt-5.6-luna already looks solid.

One follow-up that would unlock most OpenAI-compatible proxies (we only call /v1/chat/completions): please also bridge chat completions → this native Responses config, the same way bedrock_mantle already does via mode: responses. As written, chat completions on bedrock-runtime stays out of scope, so gateways that cannot switch clients would still need Mantle after this merges.

Happy to test a chat-completions follow-up against real Bedrock if that helps.

[mateo-berri]

Superseded by #42767, the internal copy of this work with the author's commits preserved, merged into main as fecc8c8. Thanks for the contribution!