fix(bedrock): split the <reasoning> tag for gpt-oss only on native Chat Completions by devin-ai-integration[bot] · Pull Request #44432 · BerriAI/litellm

devin-ai-integration · 2026-10-03T20:54:04Z

TLDR

Problem this solves:

Bedrock's native Chat Completions route split a leading <reasoning> tag out of every model's answer
Only gpt-oss writes its reasoning inline; GPT 5.6 and Grok send it as reasoning_content
A GPT 5.6 or Grok answer starting with a literal <reasoning> tag lost that text, streaming or not
Greptile's P1 on feat(bedrock): serve gpt-5.6+ chat completions natively by default, with chat_completions/ opt-in for gpt-oss and grok #44307 flagged this after the merge

How it solves it:

A cost-map flag, supports_bedrock_runtime_chat_completions_inline_reasoning, gates the split in both the non-streaming and the streaming path
The ten Bedrock gpt-oss rows carry the flag (the 20b and 120b ids in the commercial and GovCloud regions, plus the two safeguard ids), so no model id is hardcoded
Non-streaming reads the request's model id, streaming reads the id AWS echoes in every chunk
Regression tests cover GPT 5.6, Grok, and the gpt-oss control, streaming and non-streaming

User Flow

Before: a developer whose prompt asks GPT 5.6 on Bedrock for tagged output gets the opening <reasoning> block cut out of the answer

The proxy admin adds bedrock/global.openai.gpt-5.6-sol to the config and starts the proxy
The developer sends POST https://litellm-domain/v1/chat/completions asking the model to reply with exactly <reasoning>not thinking</reasoning> pong, with or without "stream": true
They get 200 with "content": "pong" and "reasoning_content": "not thinking", so their parser never sees the tag they asked for; Grok through bedrock/chat_completions/us.xai.grok-4.6 answers the same way

After: the same answer comes back as the model wrote it

The proxy admin adds bedrock/global.openai.gpt-5.6-sol to the config and starts the proxy
The developer sends the same POST https://litellm-domain/v1/chat/completions, with or without "stream": true
They get 200 with "content": "<reasoning>not thinking</reasoning> pong" and no reasoning_content, Grok included, while a gpt-oss deployment still gets its inline reasoning moved into reasoning_content

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Shared setup: one proxy per commit, each started from a checkout of that commit with the same config and two uvicorn workers, and the same 18 calls sent to each: the three unified endpoints, non-streaming and streaming, for GPT 5.6, Grok 4.6, and the gpt-oss control. Real Bedrock calls in us-west-2; the proxy logs show every GPT 5.6 Responses API call going to https://bedrock-runtime.us-west-2.amazonaws.com/openai/v1/responses and every other call to https://bedrock-runtime.us-west-2.amazonaws.com/openai/v1/chat/completions

model_list:
  - model_name: gpt-5.6-sol
    litellm_params:
      model: bedrock/global.openai.gpt-5.6-sol
      aws_profile_name: litellm-bedrock-888
      aws_region_name: us-west-2
  - model_name: grok-4.6
    litellm_params:
      model: bedrock/chat_completions/us.xai.grok-4.6
      aws_profile_name: litellm-bedrock-888
      aws_region_name: us-west-2
  - model_name: gpt-oss-20b
    litellm_params:
      model: bedrock/chat_completions/openai.gpt-oss-20b-1:0
      aws_profile_name: litellm-bedrock-888
      aws_region_name: us-west-2
general_settings:
  master_key: sk-reasoning-proof

LITELLM_LOCAL_MODEL_COST_MAP=true python litellm/proxy/proxy_cli.py --config reasoning_cfg.yaml --port <port> --num_workers 2 --detailed_debug

Both proxies read the cost map of their own checkout through LITELLM_LOCAL_MODEL_COST_MAP=true. The After checkout's map carries the new flag, which the hosted map only gets once this merges. Each proxy logged two Started server process lines, one per worker

The GPT 5.6 and Grok cases send the prompt Reply with exactly this text and nothing else: <reasoning>not thinking</reasoning> pong, so the expected answer starts with a literal tag. The gpt-oss control sends Reply with the single word: pong, since gpt-oss writes its own reasoning inside the tag. Each cell prints the answer text next to that API's reasoning field (reasoning_content, the Responses reasoning item, or the Messages thinking block), with streaming deltas joined

#!/bin/zsh
# usage: zsh reasoning_proof.sh <port>
PORT=$1
AUTH=(-H "Authorization: Bearer sk-reasoning-proof" -H "Content-Type: application/json")
TAGGED='Reply with exactly this text and nothing else: <reasoning>not thinking</reasoning> pong'
PLAIN='Reply with the single word: pong'
OUT=reasoning_proof_$PORT.out

call () {
  local route=$1 body=$2 full=$3 sse=$4
  local code=$(curl -s -N -o $OUT -w '%{http_code}' http://127.0.0.1:$PORT$route $AUTH -d "$body")
  if grep -q '^data: ' $OUT; then
    grep '^data: ' $OUT | sed 's/^data: //' | grep -v '^\[DONE\]' | jq -sc "$sse"
  else
    jq -c "$full" $OUT
  fi
  echo "HTTP $code"
}

for route in /v1/chat/completions /v1/responses /v1/messages; do
  for stream in false true; do
    for model in gpt-5.6-sol grok-4.6 gpt-oss-20b; do
      ask=$TAGGED; [[ $model == gpt-oss-20b ]] && ask=$PLAIN
      echo "### $route $model stream=$stream"
      case $route in
        /v1/chat/completions) call $route "{\"model\":\"$model\",\"messages\":[{\"role\":\"user\",\"content\":\"$ask\"}],\"max_tokens\":1500,\"stream\":$stream}" \
          '{content: .choices[0].message.content, reasoning_content: .choices[0].message.reasoning_content}' \
          '{content: (map(.choices[0].delta.content // "") | join("")), reasoning_content: (map(.choices[0].delta.reasoning_content // "") | join("") | if . == "" then null else . end)}' ;;
        /v1/responses) call $route "{\"model\":\"$model\",\"input\":\"$ask\",\"max_output_tokens\":1500,\"stream\":$stream}" \
          '{output_text: ([.output[] | select(.type=="message") | .content[].text] | join("")), reasoning: ([.output[] | select(.type=="reasoning") | ((.summary // []) + (.content // []))[].text] | join("") | if . == "" then null else . end)}' \
          '{output_text: ([.[] | select(.type=="response.output_text.delta").delta] | join("")), reasoning: ([.[] | select(.type=="response.reasoning_summary_text.delta" or .type=="response.reasoning_text.delta").delta] | join("") | if . == "" then null else . end)}' ;;
        /v1/messages) call $route "{\"model\":\"$model\",\"messages\":[{\"role\":\"user\",\"content\":\"$ask\"}],\"max_tokens\":1500,\"stream\":$stream}" \
          '{text: ([.content[] | select(.type=="text").text] | join("")), thinking: ([.content[] | select(.type=="thinking").thinking] | join("") | if . == "" then null else . end)}' \
          '{text: ([.[] | select(.delta.type=="text_delta").delta.text] | join("")), thinking: ([.[] | select(.delta.type=="thinking_delta").delta.thinking] | join("") | if . == "" then null else . end)}' ;;
      esac
    done
  done
done

Before (`dd31692`)

zsh reasoning_proof.sh 27523

### /v1/chat/completions gpt-5.6-sol stream=false
{"content":"pong","reasoning_content":"not thinking"}
HTTP 200
### /v1/chat/completions grok-4.6 stream=false
{"content":"pong","reasoning_content":"not thinking"}
HTTP 200
### /v1/chat/completions gpt-oss-20b stream=false
{"content":"pong","reasoning_content":"We are asked: \"Reply with the single word: pong\". So just output \"pong\"."}
HTTP 200
### /v1/chat/completions gpt-5.6-sol stream=true
{"content":"pong","reasoning_content":"not thinking"}
HTTP 200
### /v1/chat/completions grok-4.6 stream=true
{"content":"pong","reasoning_content":"not thinking"}
HTTP 200
### /v1/chat/completions gpt-oss-20b stream=true
{"content":"pong","reasoning_content":"The user says: \"Reply with the single word: pong\". So respond with \"pong\" as single word. No other formatting."}
HTTP 200
### /v1/responses gpt-5.6-sol stream=false
{"output_text":"<reasoning>not thinking</reasoning> pong","reasoning":null}
HTTP 200
### /v1/responses grok-4.6 stream=false
{"output_text":"pong","reasoning":"not thinking"}
HTTP 200
### /v1/responses gpt-oss-20b stream=false
{"output_text":"pong","reasoning":"User says: \"Reply with the single word: pong\". So just \"pong\"."}
HTTP 200
### /v1/responses gpt-5.6-sol stream=true
{"output_text":"<reasoning>not thinking</reasoning> pong","reasoning":null}
HTTP 200
### /v1/responses grok-4.6 stream=true
{"output_text":"pong","reasoning":"not thinking"}
HTTP 200
### /v1/responses gpt-oss-20b stream=true
{"output_text":"<reasoning> the assistant to just reply with \"pong\". That's what to do.</reasoning>pong","reasoning":"User asks: \"Reply with the single word: pong\"\n\nThey want"}
HTTP 200
### /v1/messages gpt-5.6-sol stream=false
{"text":"pong","thinking":"not thinking"}
HTTP 200
### /v1/messages grok-4.6 stream=false
{"text":"pong","thinking":"not thinking"}
HTTP 200
### /v1/messages gpt-oss-20b stream=false
{"text":"pong","thinking":"This is an instruction: user wants: Reply with the single word: pong. So answer should simply be \"pong\".\n\nNo additional commentary. Use the word exactly. Probably just the word."}
HTTP 200
### /v1/messages gpt-5.6-sol stream=true
{"text":"pong","thinking":"not thinking"}
HTTP 200
### /v1/messages grok-4.6 stream=true
{"text":"pong","thinking":"not thinking"}
HTTP 200
### /v1/messages gpt-oss-20b stream=true
{"text":"pong","thinking":"We need to reply with single word: pong. Simple."}
HTTP 200

GPT 5.6 and Grok lose the literal tag on /v1/chat/completions and /v1/messages in both modes, and Grok loses it on /v1/responses too: the text comes back as pong with not thinking moved into the reasoning field

After (`fd23b9f`)

zsh reasoning_proof.sh 24178

### /v1/chat/completions gpt-5.6-sol stream=false
{"content":"<reasoning>not thinking</reasoning> pong","reasoning_content":null}
HTTP 200
### /v1/chat/completions grok-4.6 stream=false
{"content":"<reasoning>not thinking</reasoning> pong","reasoning_content":null}
HTTP 200
### /v1/chat/completions gpt-oss-20b stream=false
{"content":"pong","reasoning_content":"The user says: \"Reply with the single word: pong\". So the reply should be just \"pong\". No additional text."}
HTTP 200
### /v1/chat/completions gpt-5.6-sol stream=true
{"content":"<reasoning>not thinking</reasoning> pong","reasoning_content":null}
HTTP 200
### /v1/chat/completions grok-4.6 stream=true
{"content":"<reasoning>not thinking</reasoning> pong","reasoning_content":null}
HTTP 200
### /v1/chat/completions gpt-oss-20b stream=true
{"content":"pong","reasoning_content":"The user wants a reply with the single word \"pong\". They said: \"Reply with the single word: pong\". So just write \"pong\". No other text."}
HTTP 200
### /v1/responses gpt-5.6-sol stream=false
{"output_text":"<reasoning>not thinking</reasoning> pong","reasoning":null}
HTTP 200
### /v1/responses grok-4.6 stream=false
{"output_text":"<reasoning>not thinking</reasoning> pong","reasoning":null}
HTTP 200
### /v1/responses gpt-oss-20b stream=false
{"output_text":"pong","reasoning":"The user says: \"Reply with the single word: pong\". So I must reply with \"pong\". Probably just output the word pong. Ensure no extra text."}
HTTP 200
### /v1/responses gpt-5.6-sol stream=true
{"output_text":"<reasoning>not thinking</reasoning> pong","reasoning":null}
HTTP 200
### /v1/responses grok-4.6 stream=true
{"output_text":"<reasoning>not thinking</reasoning> pong","reasoning":null}
HTTP 200
### /v1/responses gpt-oss-20b stream=true
{"output_text":"pong","reasoning":"The user asked: \"Reply with the single word: pong\". So output \"pong\". It's simple."}
HTTP 200
### /v1/messages gpt-5.6-sol stream=false
{"text":"<reasoning>not thinking</reasoning> pong","thinking":null}
HTTP 200
### /v1/messages grok-4.6 stream=false
{"text":"<reasoning>not thinking</reasoning> pong","thinking":null}
HTTP 200
### /v1/messages gpt-oss-20b stream=false
{"text":"pong","thinking":"The user: \"Reply with the single word: pong\" So I just output \"pong\"."}
HTTP 200
### /v1/messages gpt-5.6-sol stream=true
{"text":"<reasoning>not thinking</reasoning> pong","thinking":null}
HTTP 200
### /v1/messages grok-4.6 stream=true
{"text":"<reasoning>not thinking</reasoning> pong","thinking":null}
HTTP 200
### /v1/messages gpt-oss-20b stream=true
{"text":"pong","thinking":"User says: \"Reply with the single word: pong\". So answer \"pong\"."}
HTTP 200

GPT 5.6 and Grok keep <reasoning>not thinking</reasoning> pong as sent on all three endpoints in both modes, with no reasoning field, while gpt-oss still gets its reasoning split out everywhere

Observations from the run:

gpt-oss's second streamed reasoning block lands in content, both legs
- AWS sometimes streams two or three tag pairs per answer
- The PR leaves it alone, the splitter is unchanged
GPT 5.6 on /v1/responses rides AWS's native Responses route
- Never split on either leg, the PR leaves it alone
Cross-region us. and application inference profile ARN gpt-oss ids never reach the split
- AWS answers 400 for both on the native route, on both legs

Type

🐛 Bug Fix

Caveats (if any)

Medium

A proxy whose cost map lacks the new flag keeps gpt-oss's <reasoning> block inline on the native Chat Completions route, where main splits it today
- The hosted map is read from main, so a default setup picks the flag up the moment this merges; a mirror behind LITELLM_MODEL_COST_MAP_URL keeps the old behavior until it carries the flagged rows, and refreshing the mirror is the fix
- The two sibling flags on this route already work this way; the proof runs the After proxy with LITELLM_LOCAL_MODEL_COST_MAP=true for the same reason

Low

The streaming gate reads the model id AWS echoes in each chunk, which is the id litellm put in the request
- A chunk without a model field streams unsplit; AWS's Chat Completions contract echoes it in every chunk, as the proof's gpt-oss streaming cell shows
- Threading the request model through the config instead was rejected, since provider configs hold no per-request state
gpt-oss streaming leaves a second or third <reasoning> block in content when AWS streams more than one pair, on both legs
- The splitter is unchanged here, and a follow-up tracks the multi-block case

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR
fd23b9f passes /live-pr-risk

Note

Medium Risk
Changes response shaping for Bedrock Chat Completions (content vs reasoning_content) for all non–gpt-oss models; behavior depends on the hosted cost map until the new flag ships.

Overview
Bedrock native Chat Completions used to strip a leading <reasoning>...</reasoning> block from every model’s answer into reasoning_content. That was wrong for GPT 5.6 and Grok, which can emit that text literally in content while reasoning lives elsewhere.

The PR gates that split behind a new price-map flag, supports_bedrock_runtime_chat_completions_inline_reasoning, exposed via bedrock_runtime_chat_completions_serves_reasoning_inline(). Only flagged gpt-oss rows (commercial, GovCloud, safeguard variants) get the split on non-streaming (transform_response) and streaming (BedrockRuntimeChatCompletionsStreamingHandler, using the model id on each chunk). Everyone else’s text is returned unchanged.

Schema and both cost-map JSON files are updated; unit tests cover GPT 5.6, Grok, gpt-oss, and cost-map-driven behavior.

^{Reviewed by Cursor Bugbot for commit fd23b9f. Bugbot is set up for automated code reviews on this repo. Configure here.}

…at Completions The native Chat Completions route moved a leading <reasoning>...</reasoning> block into reasoning_content for every model, while only gpt-oss writes its reasoning inline. A GPT 5.6 or Grok answer that starts with a literal <reasoning> tag lost that text, streaming and non-streaming alike. Both paths now split only when the model id is gpt-oss.

devin-ai-integration · 2026-10-03T20:54:06Z

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

Disable automatic comment, CI, and merge conflict monitoring

codecov · 2026-10-03T20:58:55Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

greptile-apps · 2026-10-03T20:59:10Z

Confidence Score: 5/5

[Medium risk] Bedrock chat completions reasoning tag handling for specific models.

The PR appears safe to merge; no new actionable issue was established.

Summary

The PR limits Bedrock native Chat Completions’ inline <reasoning> extraction to models marked in the cost map.

Adds the capability flag to gpt-oss metadata and its schema.
Gates both complete-response and streaming extraction, with regression tests for tagged GPT 5.6 and Grok answers.

_{Reviews (2) · Last reviewed commit: "fix(bedrock): drive the inline reasoning..."}

codspeed · 2026-10-03T21:08:31Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_bedrock_reasoning_split_gpt_oss_only (fd23b9f) with main (0ea166c)}

mateo-berri · 2026-10-03T22:16:33Z

@greptileai

mateo-berri · 2026-10-03T22:48:24Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit fd23b9f. Configure here.}

mateo-berri

LGTM

devin-ai-integration Bot requested a review from a team October 3, 2026 20:54

devin-ai-integration Bot assigned mateo-berri Oct 3, 2026

greptile-apps Bot reviewed Oct 3, 2026

View reviewed changes

Comment thread litellm/llms/bedrock/common_utils.py Outdated

fix(bedrock): drive the inline reasoning split from a cost-map flag

fd23b9f

mateo-berri requested review from kerry-berri, mateo-berri and ryan-crabbe-berri as code owners October 3, 2026 22:16

cursor Bot reviewed Oct 3, 2026

View reviewed changes

devin-ai-integration Bot mentioned this pull request Oct 8, 2026

fix(model_prices): add Bedrock flex prices, Anthropic web search flags, Gemini shutdown dates, OpenRouter alias drift, Mistral Large 4 context #44910

Merged

mateo-berri approved these changes Oct 8, 2026

View reviewed changes

mateo-berri merged commit 6a2d3ed into main Oct 8, 2026
105 of 106 checks passed

mateo-berri deleted the litellm_bedrock_reasoning_split_gpt_oss_only branch October 8, 2026 23:32

Navigation Menu

Repository navigation

fix(bedrock): split the <reasoning> tag for gpt-oss only on native Chat Completions - #44432

fix(bedrock): split the <reasoning> tag for gpt-oss only on native Chat Completions#44432
mateo-berri merged 2 commits into
mainfrom
litellm_bedrock_reasoning_split_gpt_oss_only

devin-ai-integration Bot commented Oct 3, 2026 •

edited by mateo-berri

Loading

Uh oh!

devin-ai-integration Bot commented Oct 3, 2026 •

edited by mateo-berri

Loading

Uh oh!

codecov Bot commented Oct 3, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Oct 3, 2026 •

edited

Loading

Uh oh!

Uh oh!

codspeed Bot commented Oct 3, 2026 •

edited

Loading

Uh oh!

mateo-berri commented Oct 3, 2026

Uh oh!

mateo-berri commented Oct 3, 2026

Uh oh!

cursor Bot left a comment

Uh oh!

mateo-berri left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

devin-ai-integration Bot commented Oct 3, 2026 • edited by mateo-berri Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Pre-Submission checklist

Delays in PR merge?

Screenshots / Proof of Fix

Before (dd31692)

After (fd23b9f)

Type

Caveats (if any)

Medium

Low

Final Attestation

Uh oh!

devin-ai-integration Bot commented Oct 3, 2026 • edited by mateo-berri Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

greptile-apps Bot commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

codspeed Bot commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

mateo-berri commented Oct 3, 2026

Uh oh!

mateo-berri commented Oct 3, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

mateo-berri left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

devin-ai-integration Bot commented Oct 3, 2026 •

edited by mateo-berri

Loading

Before (`dd31692`)

After (`fd23b9f`)

devin-ai-integration Bot commented Oct 3, 2026 •

edited by mateo-berri

Loading

codecov Bot commented Oct 3, 2026 •

edited

Loading

greptile-apps Bot commented Oct 3, 2026 •

edited

Loading

codspeed Bot commented Oct 3, 2026 •

edited

Loading