fix(anthropic): split mixed reasoning stream chunks by Napuh · Pull Request #34701 · BerriAI/litellm

Napuh · 2026-07-26T10:59:01Z

TLDR

Problem this solves:

Mixed chunks emitted thinking deltas inside text blocks
Strict Anthropic clients like Claude Code rejected this as an invalid SSE stream

How it solves it:

Splits reasoning and text into separate upstream chunks
Keeps finish reason and usage on the final chunk
Covers sync and async streams with a finishing mixed chunk

Relevant issues

Fixes #33224

Pre-Submission checklist

I have added meaningful tests
My PR passes all CI/CD checks (e.g., lint, format, unit tests)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

This is a deterministic reproduction using an OpenAI-compatible SSE backend that emits:

reasoning_content: "The"
reasoning_content: " assistant." and content: " Hello" in one chunk
content: " world"
finish_reason: "stop"

It exercises the real LiteLLM proxy and /v1/messages adapter without loading a model

Before, commit 24123269cc:

curl --no-buffer http://127.0.0.1:4100/v1/messages \
  -H 'x-api-key: sk-mixed-reasoning-demo' \
  -H 'anthropic-version: 2023-06-01' \
  -H 'content-type: application/json' \
  -d '{"model":"mixed-reasoning-demo","max_tokens":128,"stream":true,"thinking":{"type":"enabled","budget_tokens":64},"messages":[{"role":"user","content":"Say hello"}]}'

event: content_block_start
data: {"type":"content_block_start","index":2,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":2,"delta":{"type":"thinking_delta","thinking":" assistant."}}

event: content_block_delta
data: {"type":"content_block_delta","index":2,"delta":{"type":"text_delta","text":" world"}}

The stream is invalid because a thinking_delta is emitted in a text block. The mixed chunk's " Hello" text is also lost

After, commit 2a8e29cc0b:

curl --no-buffer http://127.0.0.1:4101/v1/messages \
  -H 'x-api-key: sk-mixed-reasoning-demo' \
  -H 'anthropic-version: 2023-06-01' \
  -H 'content-type: application/json' \
  -d '{"model":"mixed-reasoning-demo","max_tokens":128,"stream":true,"thinking":{"type":"enabled","budget_tokens":64},"messages":[{"role":"user","content":"Say hello"}]}'

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"thinking_delta","thinking":" assistant."}}

event: content_block_stop
data: {"type":"content_block_stop","index":1}

event: content_block_start
data: {"type":"content_block_start","index":2,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":2,"delta":{"type":"text_delta","text":" Hello"}}

event: content_block_delta
data: {"type":"content_block_delta","index":2,"delta":{"type":"text_delta","text":" world"}}

The mixed chunk is split into a valid thinking delta followed by a valid text delta, preserving both " Hello" and " world"

Type

Bug Fix

Changes

Split OpenAI-compatible chunks containing both reasoning_content and content
Preserve chunk ordering as reasoning, text, then finish
Add regression coverage for sync and async iterator paths
Cover the mixed chunk plus finish_reason case

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Note

Medium Risk
Touches core streaming chunk normalization for the Anthropic pass-through adapter; behavior change is scoped to mixed reasoning/text (and usage on split chunks) with new regression tests.

Overview
Fixes invalid /v1/messages SSE when upstream OpenAI-style streams put reasoning_content and content in the same chunk, which made strict clients (e.g. Claude Code) see thinking_delta inside a text block and could drop part of the answer.

_CombinedChunkSplitter now detects those mixed deltas and emits two chunks in order: reasoning-only (no text, no finish/usage on that half), then text-only. That runs after the existing content+finish_reason split on both sync and async paths. Usage is stripped from intermediate split chunks so it stays on the final finish chunk.

Regression tests cover sync and async streams, including a mixed chunk that also carries finish_reason: stop.

^{Reviewed by Cursor Bugbot for commit 2b94847. Bugbot is set up for automated code reviews on this repo. Configure here.}

greptile-apps · 2026-07-26T11:00:56Z

Greptile Summary

Fixes Anthropic-compatible streaming for chunks containing both reasoning and text.

Splits mixed deltas into ordered reasoning and text chunks for synchronous and asynchronous streams.
Keeps finish reasons and usage metadata on the final chunk.
Adds regression coverage for mixed finishing chunks and Anthropic content-block type invariants.

Confidence Score: 5/5

The PR appears safe to merge, with the mixed reasoning and text stream path covered in both synchronous and asynchronous modes.

The splitter emits reasoning before text, retains termination and usage metadata on the final chunk, and the added tests verify that resulting deltas match their Anthropic content-block types without dropping either payload.

Important Files Changed

Filename	Overview
litellm/llms/anthropic/experimental_pass_through/adapters/streaming_iterator.py	Adds mixed reasoning/text chunk splitting while preserving finish and usage metadata on the terminal chunk; no actionable defect found.
tests/test_litellm/llms/anthropic/experimental_pass_through/adapters/test_streaming_iterator_first_delta.py	Adds synchronous and asynchronous regression tests validating payload preservation, event ordering, and content-block type compatibility.

_{Reviews (1): Last reviewed commit: "fix(anthropic): split mixed reasoning st..." | Re-trigger Greptile}

codecov · 2026-07-26T11:05:41Z

Codecov Report

❌ Patch coverage is 90.00000% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...mental_pass_through/adapters/streaming_iterator.py	90.00%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!

codspeed-hq · 2026-07-26T11:40:25Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing Napuh:litellm_anthropic_mixed_reasoning_chunks (2b94847) with litellm_internal_staging (2412326)}

Napuh · 2026-07-27T06:30:40Z

@yuneng-berri @tin-berri @mateo-berri @ryan-crabbe-berri @yucheng-berri please check :)

yucheng-berri · 2026-07-29T05:11:03Z

bugbot run

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 2b94847. Configure here.}

cursor · 2026-07-29T05:13:05Z

+
+        text_chunk = copy.deepcopy(chunk)
+        text_chunk.choices[0].delta.reasoning_content = None
+        return [reasoning_chunk, text_chunk]


Text split keeps thinking_blocks

Medium Severity

When _split_mixed_reasoning_and_text builds the text half of a mixed chunk, it clears reasoning_content but leaves thinking_blocks on the delta. The Anthropic translate path still prefers thinking over text when blocks are present, so the intended text can be dropped and a thinking_delta can be emitted after a text content_block_start, recreating the invalid-stream failure mode this PR targets.

^{Reviewed by Cursor Bugbot for commit 2b94847. Configure here.}

yucheng-berri · 2026-07-29T05:36:45Z

Just to clarify, was this ever reproduced against a live NVIDIA NIM/Nemotron backend, or only with the scripted OpenAI-compatible SSE emitter described in the PR?

yucheng-berri · 2026-07-29T05:37:28Z

Could you also attach evidence that the fix was tested end to end against a live model using valid credentials?

Napuh · 2026-07-29T07:17:12Z

The scripted SSE emitter in the PR was just a quick deterministic test to validate the fix. I tested the fix vs a live vllm endpoint

I ran two identical LiteLLM proxies (v1.92.0 and this PR branch) both pointing at a live GLM-5.2 vLLM backend over hosted_vllm with streaming and thinking enabled. It is a vLLM OpenAI-compatible reasoning model rather than NIM/Nemotron, but the defect is in the OpenAI to Anthropic conversion, so any OpenAI-compatible reasoning backend that packs a trailing reasoning token and a leading answer token into one chunk hits it.

Before the fix, index 2 opens as text and gets a thinking_delta (the answer token is also mislabeled as reasoning):

content_block_start index 2 {"type":"text","text":""}
content_block_delta index 2 {"type":"thinking_delta","thinking":" 81."}

After the fix, the mixed chunk is split, so reasoning stays in the thinking block and the answer lands in text:

content_block_start index 2 {"type":"text","text":""}
content_block_delta index 2 {"type":"text_delta","text":"81"}

The crash is intermittent because it only fires when the parser packs both tokens together. Over 10 identical live runs it was invalid 6/10 times before the fix and 0/10 after.

CURL command used:

curl -sS -N http://localhost:4000/v1/messages \
  -H 'x-api-key: sk-repro-1234' \
  -H 'anthropic-version: 2023-06-01' \
  -H 'content-type: application/json' \
  -d '{"model":"glm","max_tokens":256,"stream":true,"thinking":{"type":"enabled","budget_tokens":128},"messages":[{"role":"user","content":"What is 27 times 3? Reply with just the number."}]}'

Config used:

model_list:
  - model_name: glm
    litellm_params:
      model: hosted_vllm/GLM-5.2
      api_base: http://<remote_server>:8000/v1
      api_key: "dummy"

litellm_settings:
  drop_params: true

general_settings:
  master_key: sk-repro-1234

SSE stream BEFORE this PR:

event: content_block_start
data: {"type": "content_block_start", "index": 0, "content_block": {"type": "text", "text": ""}}

event: content_block_stop
data: {"type": "content_block_stop", "index": 0}

event: content_block_start
data: {"type": "content_block_start", "index": 1, "content_block": {"type": "thinking", "thinking": "", "signature": ""}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": "1"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": ".  Identify"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": " the core"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": " request"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": ": The user wants"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": " to know"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": " the result of"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "del", "thinking": " 27 multiplied"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": " by 3."}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": "\n2.  Perform"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": " the calculation:"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "del", "thinking": " 27 * 3"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "del", "thinking": " = ("}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": "20 *"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": " 3) +"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": " (7 *"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": " 3) ="}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": " 60 + 21"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "del", "thinking": " = 81."}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "del", "thinking": "\n3.  Format"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": " the output: The"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": " user specifically requested \""}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": "Reply"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "del", "thinking": " with just the number"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "del", "thinking": ".\"\n4."}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": "  Final Output:"}}

event: content_block_stop
data: {"type": "content_block_stop", "index": 1}

event: content_block_start
data: {"type": "content_block_start", "index": 2, "content_block": {"type": "text", "text": ""}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 2, "delta": {"type": "thinking_delta", "thinking": " 81."}}

event: content_block_stop
data: {"type": "content_block_stop", "index": 2}

event: message_delta
data: {"type": "message_delta", "delta": {"stop_reason": "end_turn"}, "usage": {"input_tokens": 26, "output_tokens": 89}}

event: message_stop
data: {"type": "message_stop"}

SSE stream AFTER this PR:

event: content_block_start
data: {"type": "content_block_start", "index": 0, "content_block": {"type": "text", "text": ""}}

event: content_block_stop
data: {"type": "content_block_stop", "index": 0}

event: content_block_start
data: {"type": "content_block_start", "index": 1, "content_block": {"type": "thinking", "thinking": "", "signature": ""}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": "1"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": "."}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": "  Identify the core"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": " request"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": ": Calculate"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": " 27 times"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": " 3."}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": "\n2.  Perform"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": " the calculation"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": ": 27 *"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": " 3 = ("}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": "20 *"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": " 3) +"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": " (7 *"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": " 3) ="}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": " 60 +"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": " 21 = 81"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": ".\n3."}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": "  Identify"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": " the constraint: \""}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": "Reply with just the"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": " number.\"\n4."}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": "  Formulate the"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": " response: \"81"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "thinking_delta", "thinking": "\"."}}

event: content_block_stop
data: {"type": "content_block_stop", "index": 1}

event: content_block_start
data: {"type": "content_block_start", "index": 2, "content_block": {"type": "text", "text": ""}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 2, "delta": {"type": "text_delta", "text": "81"}}

event: content_block_stop
data: {"type": "content_block_stop", "index": 2}

event: message_delta
data: {"type": "message_delta", "delta": {"stop_reason": "end_turn"}, "usage": {"input_tokens": 26, "output_tokens": 79}}

event: message_stop
data: {"type": "message_stop"}

Napuh · 2026-07-30T10:27:06Z

@yucheng-berri

yucheng-berri · 2026-07-31T03:11:09Z

Merged in #35289

yucheng-berri · 2026-07-31T03:11:50Z

Adopted and merged via #35289 with your commits preserved under your authorship, extended to also split tool_calls and thinking_blocks payloads. Thank you for the fix and the live vLLM verification @Napuh; closing this in favor of the merged PR

fix(anthropic): split mixed reasoning stream chunks

2a8e29c

style: use builtin generic annotation

2b94847

cursor Bot reviewed Jul 29, 2026

View reviewed changes

yucheng-berri mentioned this pull request Jul 30, 2026

fix(anthropic): split mixed stream chunks by payload kind #35289

Merged

5 tasks

yucheng-berri closed this Jul 31, 2026

Uh oh!

Uh oh!

Conversation

Napuh commented Jul 26, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

Relevant issues

Pre-Submission checklist

Screenshots / Proof of Fix

Type

Changes

Final Attestation

Uh oh!

greptile-apps Bot commented Jul 26, 2026

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

codecov Bot commented Jul 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

codspeed-hq Bot commented Jul 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

Napuh commented Jul 27, 2026

Uh oh!

yucheng-berri commented Jul 29, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Jul 29, 2026

Choose a reason for hiding this comment

Text split keeps thinking_blocks

Uh oh!

yucheng-berri commented Jul 29, 2026

Uh oh!

yucheng-berri commented Jul 29, 2026

Uh oh!

Napuh commented Jul 29, 2026

Uh oh!

Napuh commented Jul 30, 2026

Uh oh!

yucheng-berri commented Jul 31, 2026

Uh oh!

yucheng-berri commented Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Napuh commented Jul 26, 2026 •

edited by cursor Bot

Loading

codecov Bot commented Jul 26, 2026 •

edited

Loading

codspeed-hq Bot commented Jul 26, 2026 •

edited

Loading