Fix Anthropic adapter streaming: reasoning_content thinking blocks + first-delta drop - #4
Merged
Merged
Conversation
…first-delta drop Two fixes for the OpenAI->Anthropic /v1/messages streaming adapter, found driving Claude Code at Kimi K2.7-Code via OpenRouter: 1. transformation.py: reasoning_content deltas (OpenRouter-style reasoning, no thinking_blocks) never matched a block type, so the stream opened a content_block_start of type "text" and then poured thinking_delta events into it - a malformed stream. Claude Code rendered the model's entire reasoning as the visible answer. Cherry-picked from upstream main: a reasoning_content branch that opens a proper thinking block. 2. streaming_iterator.py: on a block-type transition, the trigger chunk's delta was dropped unless it was an input_json_delta. That ate the first reasoning token of every thinking block and the first answer token after it (one-chunk answers vanished entirely). Re-queue text_delta and thinking_delta trigger payloads too; thinking_delta only when the block start doesn't already embed the content (it does for Anthropic-native thinking_blocks providers, which would otherwise duplicate).
jwbron
added a commit
to jwbron/egg
that referenced
this pull request
Jun 13, 2026
) * egg-litellm: port reasoning streaming fixes from jwbron/litellm#4 Add patches 4 and 5 to config/litellm/patch_litellm_cache.py: - Patch 4: OpenRouter-style delta.reasoning_content now opens a proper Anthropic thinking content block instead of falling through to a text block. Without this, clients like Claude Code rendered the model's entire reasoning as visible assistant text and fed it back as assistant content on later turns. - Patch 5: AnthropicStreamWrapper now re-queues the trigger chunk's first text_delta / thinking_delta on block-type transitions, not just input_json_delta tool transitions. This stops the first reasoning token and the first post-reasoning answer token from being silently dropped. Applied to both sync __next__ and async __anext__. Native thinking_blocks providers are guarded against duplication. Update Dockerfile comment and patch labels/docstring to reflect five patches. Validated against a clean litellm==1.86.2 install: all patches apply, idempotent, and a mocked reasoning stream reassembles to thinking: 'We think.' / text: 'The answer.' with first tokens intact. Fixes #3190. * egg-litellm: add patch-script tests + anchor Patch 4 needle Address PR #3199 review (egg-reviewer, approve-with-suggestions): - Anchor the Patch 4 needle on the preceding text-block elif (choice.delta.content ...) so it uniquely targets _translate_streaming_openai_chunk_to_anthropic_content_block and cannot silently retarget the sibling function on an upstream reorder. - Refactor the patch specs into a module-level PATCHES list so tests can apply the exact needles to fixtures with no fixture/needle drift. - Add tests/config/test_patch_litellm_cache.py: marker application, idempotency, fail-loud on drift/missing file, and a regression that locks in the Patch 4 needle-uniqueness anchor. --------- Co-authored-by: egg-reviewer[bot] <261018737+egg-reviewer[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Driving Claude Code at Kimi K2.7-Code through the proxy (
cllm -m kimi-k2.7-code), the model's entire reasoning rendered in the console as the visible assistant text, and answers were missing or missing their first token ('m here.instead ofI'm here.).Root causes (two bugs in the OpenAI→Anthropic
/v1/messagesstreaming adapter)1.
reasoning_contentnever opens a thinking block (transformation.py)_translate_streaming_openai_chunk_to_anthropic_content_blockonly recognized Anthropic-nativedelta.thinking_blocks. OpenRouter reasoning models streamdelta.reasoning→ mapped todelta.reasoning_content, which fell through to the default("text", TextBlock). Meanwhile the delta translator does mapreasoning_content→thinking_delta. Net result: acontent_block_startof typetextreceivingthinking_deltaevents — a malformed stream that Claude Code records as plain text, so the reasoning displays as the answer, gets fed back as assistant content on every subsequent turn, and progressively degrades the session.Fix cherry-picked from upstream
main: areasoning_contentbranch that returns athinkingblock start.2. Block-type transitions drop the trigger chunk's delta (
streaming_iterator.py)On a transition the iterator queues
content_block_stop→content_block_startand re-queues the trigger chunk's delta only forinput_json_delta. The first reasoning token of every thinking block and the first answer token after thinking were silently dropped; a one-chunk answer (391) vanished entirely. This one is still broken on upstreammain.Fix: also re-queue
text_deltaandthinking_deltatrigger payloads —thinking_deltaonly when the new block start doesn't already embed the thinking content (it does for Anthropic-nativethinking_blocksproviders, which would otherwise duplicate it).Validation
test_streaming_iterator_reasoning_blocks.py(sync + async reasoning→text streams reassemble to exactlythinking: 'We think.'/text: 'The answer.'; nativethinking_blocksnot duplicated).tests/test_litellm/llms/anthropic/experimental_pass_through/adapters/pass.Once merged, bump
LITELLM_FORK_COMMITinclm-setupto the new merge commit.