fix(anthropic): keep the replayed prefix byte-stable for preserved thinking on chat completions

[devin-ai-integration]

TLDR

Problem this solves:

• A mid-conversation system message was hoisted into the top-level system prompt
• On the next turn the replayed prefix differed, so Claude dropped or rejected preserved thinking blocks
• Bedrock Converse hoisted the same way, with the same result
• Nothing pinned the replayed prefix for the other features that rewrite messages

How it solves it:

• Carries fix(anthropic): keep mid-conversation system out of the chat completions system prompt #38053 (@oneKn8) and fix(bedrock): keep mid-conversation system messages in place on converse path #41057 (@ege-arhan) with their authorship kept
• Places each system run by its neighbours only, so appended turns never move it
• Flagged models keep role: system in place; Converse and older models get a user turn with an operator note
• Converse imports the operator note from the Anthropic module instead of carrying a copy
• One cross-turn prefix test per audited feature, failing at the merge base where the hoist lived

User Flow

Before: a developer whose app runs multi-turn conversations with preserved thinking on Claude Fable 5.1 through the proxy loses the model's reasoning, or the whole request, as soon as the app adds a mid-conversation system reminder

1. They send POST https://litellm-domain/v1/chat/completions with "model": "claude-fable-5-1", the anthropic-beta: thinking-binding-controls-2026-08-01 header, "thinking": {"type": "adaptive", "block_binding": {"prefix_mismatch_behavior": "error"}}, a leading system message, and a user question
2. The reply comes back 200 with an assistant message carrying a signed thinking_blocks entry and the answer
3. They send the same POST again with the history replayed verbatim, the assistant reply included, a new user message, and {"role": "system", "content": "The user switched to read-only mode."} appended after it
4. The request fails with 400 messages.1.content.0: Invalid signature in thinking block. The block is bound to a different conversation ... The system prompt differs from the one this block was created with
5. With "prefix_mismatch_behavior": "drop_block" instead, the same turn returns 200 but the model answers without its earlier reasoning, and Anthropic's response (visible in the proxy's debug log) lists {"type": "thinking_dropped", "path": "messages.1.content.0"} under input_transformations
6. The same two outcomes happen with "model": "bedrock/invoke/us.anthropic.claude-fable-5-1", and with "model": "bedrock/converse/us.anthropic.claude-fable-5-1" the error setting returns a 400

After: the same second turn succeeds and the model keeps its reasoning

1. They send POST https://litellm-domain/v1/chat/completions with "model": "claude-fable-5-1", the anthropic-beta: thinking-binding-controls-2026-08-01 header, "thinking": {"type": "adaptive", "block_binding": {"prefix_mismatch_behavior": "error"}}, a leading system message, and a user question
2. The reply comes back 200 with an assistant message carrying a signed thinking_blocks entry and the answer
3. They send the same POST again with the history replayed verbatim, the assistant reply included, a new user message, and {"role": "system", "content": "The user switched to read-only mode."} appended after it
4. The request returns 200 with the answer, and Anthropic's response lists "input_transformations": [], so the signed block was accepted
5. With "prefix_mismatch_behavior": "drop_block" the same turn returns 200 with "input_transformations": [] as well
6. The same second turn returns 200 with "model": "bedrock/invoke/us.anthropic.claude-fable-5-1" and with "model": "bedrock/converse/us.anthropic.claude-fable-5-1"

Audit

Anthropic's block binding compares the top-level system, the tools list, and every message ahead of a signed thinking block with what the block was created under. cache_control markers are outside the comparison. Every proxy feature that rewrites messages between turns was checked live against Claude Fable 5.1

Feature	Class	Verdict	What this PR does
Mid-conversation system on /v1/chat/completions through the Anthropic transformation (anthropic, bedrock invoke; vertex and azure_ai share it)	Prefix rewrite	Reproduced live on anthropic and bedrock invoke: thinking_dropped under drop_block, 400 under error; vertex hit its quota, azure_ai was not run	Keeps the run inside messages, placed by its neighbours only
Mid-conversation system on Bedrock Converse	Prefix rewrite	Reproduced: 400 under error	Converts the run to a user turn in place with the operator note
Mid-conversation system on /v1/messages	Passthrough	Clean, input_transformations: []	Nothing
Prompt management, same variables every turn	Stable rewrite	Clean	Test pins the rendered prefix across turns
Prompt management, changed variable or version	Prefix-breaking by design	thinking_dropped	Append-only guidance in the docs PR, BerriAI/litellm-docs#1644
Presidio PII masking	Stable rewrite	Clean	Test pins the masked prefix across turns
modify_params dummy tool result for an orphaned tool call	Stable rewrite	Clean	Test pins the prefix with the inserted result across turns
modify_params trailing whitespace trim on assistant prefill	Not reachable	Thinking rejects assistant prefill	Recorded only
MCP tool injection	Prefix-breaking by design	tools is rebuilt from the server per request	Append-only guidance in the docs PR, BerriAI/litellm-docs#1644
cache_control markers	Excluded	Outside the comparison	Nothing

Design decisions

1. One PR carries both contributor PRs and the ticket's deliverables. fix(anthropic): keep mid-conversation system out of the chat completions system prompt #38053 and fix(bedrock): keep mid-conversation system messages in place on converse path #41057 fix the same hoist on two code paths and share the operator note, so landing them together with one placement policy avoids two half-fixes; their commits are cherry-picked with authorship kept
2. Placement depends only on a run's neighbours. A run right after a user turn stays there as role: system; a run right after an assistant turn slides behind the user turn that immediately follows it, and becomes a user turn in place when the array ends or an assistant turn follows. The alternative, sliding to the nearest later user turn wherever it is, lets a turn appended on request N+1 move a run placed on request N, which is exactly the prefix change the binding rejects
3. Flagged models keep the system role rather than being demoted to user text (the note raised on fix(anthropic): keep mid-conversation system out of the chat completions system prompt #38053). Anthropic accepts role: system inside messages on these models in the placed shape, and demoting would rewrite the same prefix while dropping the instruction's authority
4. Converse and unflagged models get a user turn in place, prefixed with the operator note. Converse rejects role: system inside messages and older Claude models reject it anywhere but the top; hoisting breaks the prefix, dropping the message loses the instruction, and a user turn without the note would read as if the user wrote it. On Converse this applies to every model, the non-Claude ones (Nova, Llama, Mistral, DeepSeek, Writer) included, because the surface has no system role inside messages for any of them and the leading-run hoist plus in-place conversion is one behavior per surface. The alternative, keeping the old hoist for non-Claude models, was rejected: the hoist rewrites the top-level system between turns, which also invalidates the cachePoint prefix those models cache through, and it loses the reminder's position in the conversation; a per-model split would have made one Converse deployment's reminders read differently from the next
5. Runs that land on the same slot merge into one system message, because Anthropic rejects consecutive system messages
6. cache_control is left alone: it is outside the comparison and stripping it would cost prompt caching for nothing
7. The Converse module imports CONVERTED_SYSTEM_NOTE from the shared placement module. The two texts were byte-identical copies, and a drifting copy would make the same conversation read differently per provider
8. The Presidio test talks to an in-process fake analyzer and anonymizer handed in through the guardrail's api_base settings, so the real HTTP path runs without patching class attributes
9. Thinking blocks are never dropped or rewritten to make a prefix match; the fix keeps the prefix stable instead
10. The placement module lives in litellm/litellm_core_utils/prompt_templates/ next to anthropic_messages_pt, not under litellm/llms/anthropic/. The factory needs its wire helper, and a core-utils module importing a provider package loads litellm/llms/anthropic/__init__, whose common_utils reads a factory constant before the factory has defined it; the cycle stayed latent only because litellm/__init__ happens to load the Anthropic package first, and importing the factory first raises ImportError: cannot import name 'THOUGHT_SIGNATURE_SEPARATOR' from partially initialized module. A lazy import would hide the same inversion, so the module moved and every consumer (the factory, the Anthropic chat transformation, Bedrock Converse) now imports core utils
11. A user turn that puts nothing on the wire (content: [] or content: null) cannot anchor a system run. anthropic_messages_pt emits no block for it, so a run anchored there landed first in messages or behind an assistant turn (Greptile's empty-anchor finding, reproduced); such a run now converts in place, which is what it would have done had the empty turn not been there. Empty strings and empty text blocks still anchor because the factory fills them with a placeholder
12. A user turn whose list content holds only parts the converter does not emit (input_audio, unknown types) counts as empty too. _renders mirrors the converter's part types (text, image_url, document, file) instead of testing for a non-empty list, and one test runs every part type through transform_request twice, with and without a reminder behind the turn, asserting the reminder stays system exactly when the bare turn reaches the wire, so a converter that learns a new part type fails the test until the policy follows (Greptile's second empty-anchor finding, reproduced)
13. The Converse helpers are typed over the message TypedDicts (Sequence[AllMessageValues], ChatCompletionSystemMessage -> ChatCompletionUserMessage | None) and build the converted turn in one shot. A mid-conversation system entry that carries no text converts to None and is dropped, which replaces the earlier len(content) > 1 check and is pinned by test_mid_conversation_system_entry_without_text_is_dropped. The role and shape checks are plain predicates because the type-discipline gate bans TypeGuard; the two casts sit right behind the check that justifies them (Greptile's P2 on coarse types)
14. The Converse helpers read every message field through the shared message_field and parts_of accessors instead of isinstance(message, dict) checks. The SDK path replays the previous turn as a litellm.Message object, which those checks silently skipped, so a mid-conversation system stayed between a tool call and its result or reached Converse as role: system; the local role predicate is gone and Converse shares is_system_message with the placement policy (Bugbot's High finding, reproduced with a pydantic assistant turn on both the reorder and the conversion paths)
15. An assistant turn that puts nothing on the wire cannot keep a system run behind its anchor either. anthropic_messages_pt emits no turn for content: null, content: [], or a list holding only an unsigned thinking part, so a run anchored before such a turn landed directly before the next user turn, which Anthropic rejects; _anchor_block now asks _assistant_renders about the turn after the anchor and converts the run in place when it vanishes. Empty and whitespace-only text (a string or a text part) still counts as rendering because the converter pads it with its placeholder, and a test runs each padded shape through the policy to pin that (Bugbot's Medium finding, reproduced)
16. A thinking block the Responses bridge tagged with encrypted reasoning never reaches Anthropic either: anthropic_messages_pt drops it whether the tag sits in a redacted_thinking block's data or a thinking block's signature, so an assistant turn made only of such blocks vanishes from the wire and the run anchored before it has to convert. The predicate the factory used for that moved into common_utils as is_unsignable_thinking_block, next to is_encrypted_reasoning_block, so the factory and the placement policy consult one rule instead of two copies that can drift (Greptile's P1 at the merged tip, reproduced with both encrypted shapes at the policy level and through transform_request)
17. Separate thinking_blocks reach the wire only when the content list carries no inline thinking or redacted_thinking part: anthropic_messages_pt skips the separate blocks as soon as such a part is present, whether or not the part itself survives, so an unsigned inline part next to a signed separate block leaves nothing on the wire. _separate_thinking_blocks_render mirrors that rule (Greptile's second finding at bf3ba0d, reproduced with an unsigned inline part and an inline redacted part at both levels; a text part beside a signed block still renders)

Review feedback carried over

• Greptile on fix(bedrock): keep mid-conversation system messages in place on converse path #41057, consecutive tool results before a system run: covered by test_mid_conversation_system_after_multiple_tool_results from that PR
• Greptile P2 on fix(bedrock): keep mid-conversation system messages in place on converse path #41057, comments in the new Converse helpers: the helpers are comment-free here
• veria-ai on fix(anthropic): keep mid-conversation system out of the chat completions system prompt #38053, demoting system to user text: answered by decisions 3 and 4

Review feedback on this PR

• Greptile P1, circular import through the prompt factory: fixed by decision 10
• Greptile P1, empty user turn as a system anchor: fixed by decision 11, with regression tests at the policy level and through transform_request
• Greptile P1, user turn of unsupported parts as a system anchor: fixed by decision 12
• Greptile P2, coarse types in the Converse helpers: fixed by decision 13
• Greptile P2, Presidio test opening a loopback socket: rebutted in the thread, the guardrail builds its own aiohttp session and only takes URLs (decision 8)
• Bugbot High, Converse helpers ignore pydantic messages: fixed by decision 14, with a regression test per helper
• Bugbot Medium, an empty assistant turn breaks system placement: fixed by decision 15, with policy and transform_request tests over every empty and padded shape
• Greptile P1, encrypted reasoning block counted as a rendering assistant turn: fixed by decision 16, with regression tests for both encrypted shapes at both levels
• Greptile, inline thinking part hides the separate thinking_blocks: fixed by decision 17, with regression tests at both levels

Relevant issues

Supersedes #38053 and #41057, whose commits are here with their authorship kept

Affected release

Linear ticket

Resolves LIT-8372

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live runs against real Claude Fable 5.1 deployments through a local proxy (--num_workers 2 --detailed_debug), each flow run once at the merge base of the day (da82ea8, port 31274) and once at this PR's tip (e05e9dc, port 39422). Deployments: fable = anthropic/claude-fable-5-1, fable-bedrock-invoke = bedrock/invoke/us.anthropic.claude-fable-5-1 (us-west-2), fable-bedrock = bedrock/converse/us.anthropic.claude-fable-5-1 (us-west-2), fable-vertex = vertex_ai/claude-fable-5-1. Every flow is two turns: turn 1 asks a puzzle with a top-level system message and thinking on, turn 2 replays turn 1's assistant message verbatim (thinking blocks and signatures included), adds a user turn, then a role: system reminder, which is what a coding agent's system-reminder looks like on the wire

T1 = {"model": M, "max_tokens": 4096, "reasoning_effort": "max",
  "extra_headers": {"anthropic-beta": "thinking-binding-controls-2026-08-01"},
  "thinking": {"type": "adaptive", "display": "summarized", "block_binding": {"prefix_mismatch_behavior": MODE}},
  "messages": [{"role": "system", "content": SYSTEM}, {"role": "user", "content": PUZZLE}]}
assistant_replay = {"role": "assistant", "content": choices[0].message.content, "thinking_blocks": choices[0].message.thinking_blocks}
T2 = T1 with messages += [assistant_replay, {"role": "user", "content": U2}, {"role": "system", "content": S2}]

# turn 1: curl -s -o t1_resp.json -w '%{http_code}' $BASE/v1/chat/completions -H "Authorization: Bearer $MK" -H 'content-type: application/json' -d @t1.json
# turn 2: curl -s -o t2_resp.json -w '%{http_code}' $BASE/v1/chat/completions -H "Authorization: Bearer $MK" -H 'content-type: application/json' -d @t2.json

Before, da82ea8: the reminder is hoisted into system, so the replayed block no longer matches its prefix

F1 error fable:                turn 1 HTTP 200 (1 thinking block); turn 2 HTTP 400 "messages.1.content.0: Invalid `signature` in `thinking` block. The block is bound to a different conversation. ... The `system` prompt differs from the one this block was created with."
F1 error fable-bedrock-invoke: turn 1 HTTP 200 (1); turn 2 HTTP 400 BedrockException "messages.1.content.0: Invalid `signature` in `thinking` block. ... The `system` prompt differs from the one this block was created with."
F1 error fable-bedrock:        turn 1 HTTP 200 (1); turn 2 HTTP 400 BedrockException "The model returned the following errors: messages.1.content.0: Invalid `signature` in `thinking` block. ... The `system` prompt differs from the one this block was created with."
F1-control fable (turn 2 without S2): turn 1 HTTP 200; turn 2 HTTP 200
F1-drop fable (prefix_mismatch_behavior: drop_block): turn 1 HTTP 200; turn 2 HTTP 200, proxy debug log shows input_transformations thinking_dropped at messages.1.content.0
F1-drop fable-bedrock-invoke:                          turn 1 HTTP 200; turn 2 HTTP 200, debug log thinking_dropped at messages.1.content.0

After, e05e9dc: the reminder stays in messages, the prefix is unchanged, turn 2 is 200 everywhere

turn 1 (fable, error): HTTP 200
  content: Pick from the box labeled "apples and oranges" (it must hold a single fruit type): if you draw an apple, that box is "apples," ...
  thinking_blocks: 1
turn 2 (fable, error): HTTP 200
  content: Since every sticker is a fib, the box that says "both" really has just one kind of fruit inside, so peek at one piece from it, ...
  thinking_blocks: 0

turn 1 (fable-bedrock-invoke, error): HTTP 200
  thinking_blocks: 1
turn 2 (fable-bedrock-invoke, error): HTTP 200
  content: The "both fruits" box is fibbing, so it really has just one kind of fruit inside, ...
  thinking_blocks: 0

turn 1 (fable-bedrock, error): HTTP 200
  thinking_blocks: 1
turn 2 (fable-bedrock, error): HTTP 200
  content: Open the box that says "both kinds," because that sign is fibbing so it only has one kind of fruit inside, ...
  thinking_blocks: 1

turn 1 (fable, drop_block): HTTP 200,  turn 2 (fable, drop_block): HTTP 200,  debug log input_transformations: []   (nothing dropped: the prefix matched)
turn 1 (fable-bedrock-invoke, drop_block): HTTP 200,  turn 2: HTTP 200,  debug log input_transformations: []

On the flagged models the proxy sends S2 as role: system inside messages; on Converse it goes out as a user turn led by the operator note. Under drop_block the debug log's empty input_transformations is the direct provider-side confirmation that no block was dropped, where the before leg logged thinking_dropped

Both legs

Flow	Deployment	Route	Before (da82ea8)	After (e05e9dc)
F1 error	fable	/v1/chat/completions	200 / 400 system differs	200 / 200
F1 error	fable-bedrock-invoke	/v1/chat/completions	200 / 400 system differs	200 / 200
F1 error	fable-bedrock	/v1/chat/completions	200 / 400 system differs	200 / 200
F1 error	fable-vertex	/v1/chat/completions	429 quota, not exercised	429 quota, not exercised
F1-control, no S2	fable	/v1/chat/completions	200 / 200	200 / 200
F1-drop	fable	/v1/chat/completions	200 / 200, block dropped	200 / 200, nothing dropped
F1-drop	fable-bedrock-invoke	/v1/chat/completions	200 / 200, block dropped	200 / 200, nothing dropped
F1-pyd, message replayed verbatim with tool call	fable-bedrock	/v1/chat/completions	not run	200 / 200
F2	fable	/v1/responses	200 / 200, no encrypted reasoning item	200 / 200, no encrypted reasoning item
F3	fable	/v1/messages	200 / 200	200 / 200, input_transformations []
F3	fable-bedrock	/v1/messages	200 / 200, S2 hoisted into system	200 / 200, S2 as operator-note user turn
F3	fable-bedrock-invoke	/v1/messages	200 / 200	200 / 200, input_transformations []
F4 prompt management, same persona	fable	/v1/chat/completions	200	200
F4 prompt management, persona changed	fable	/v1/chat/completions	400 system differs	400 system differs (the rendered system prompt really changed)
F4 Presidio masking, turns 2 and 3	fable	/v1/chat/completions	200, 200	200, 200
F4 modify_params, orphaned tool call, turns 2 and 3	fable	/v1/chat/completions	200, 200	200, 200
F5 MCP tool injection	none	none	not exercised, needs mcp_servers in the config	not exercised

A separate live A/B over 177 scenarios (mid, trailing, wedged, consecutive, tool-result, pydantic SDK replay, empty assistant, signed and unsigned thinking, streaming, Presidio, leading multi-system) at the merge base 993a5b9 and at e05e9dc on anthropic, azure_ai, bedrock invoke, Converse (Claude, Nova, Llama), Claude Platform on Bedrock and openai found no scenario going from 200 to non-200. Two went from 400 to 200: a conversation that ends with a system reminder used to have that reminder hoisted away, leaving the array ending on the assistant turn, which Claude Fable 5.1 rejects on anthropic and Azure Foundry ("This model does not support assistant message prefill"); the tip converts the trailing run in place and the call succeeds

The /v1/responses and /v1/messages rows are the same on both legs because neither surface replays a signed block to the provider today: /v1/responses on Fable 5.1 returns no reasoning item with encrypted_content, and the native /v1/messages handler strips Fable 5.1's empty-text signed thinking block before sending (the same block replays fine sent straight to Anthropic: 200, input_transformations: []), while the Converse bridge sends the assistant turn as text only. Both are pre-existing and tracked separately (see Caveats). The full runs, request and response bodies, and proxy logs for both legs are in the QA directories named on the ticket

Type

🐛 Bug Fix
🧹 Refactoring
✅ Test

Caveats (if any)

Low

• Vertex AI was not exercised live in the audit (per-minute quota 429s for 40 minutes); it shares the Anthropic transformation and has the same unit test
• Anthropic's input_transformations is not surfaced in the OpenAI-format response, so under drop_block only the proxy debug log shows a dropped block
• The before leg of the proof ran at da82ea8, the merge base at the time; the branch has since merged main and the merge base is 993a5b9. The main-side commits touching this PR's files in between (f0e671f, d17c0d7, 1175559, 5db8543, 30004f5) are the sampling-params refactor from fix(bedrock): drop unsupported sampling params on converse reasoning models #39834, type and lint adaptations, and the MCP tool-result bridge fix, none of them about system placement, so the before rows still describe the merge base for this symptom
• The /v1/messages rows in the proof are not prefix-stability evidence. The native handler (Anthropic and Bedrock Invoke) strips Fable 5.1's empty-text signed thinking block before sending (strip_empty_content_blocks_from_anthropic_messages, whose docstring predates a model that signs an empty block), even though Anthropic accepts that exact block sent directly (200, input_transformations: []), and the Converse bridge sends the replayed assistant turn as text only. Both pre-exist this PR and are tracked as LIT-8616 and LIT-8615
• anthropic_messages_pt now emits a wire role: system entry for any mid-conversation system message a caller hands it, whatever the model. Every production caller either applies the placement policy first (the Anthropic chat transformation) or feeds it system-free runs (the guardrail translation handler), and the legacy text-completion caller stringifies the result, so no path reaches a model with an unplaced system entry today
• translate_system_message still strips x-anthropic-billing-header: blocks, but now only from the leading run; a mid-conversation billing-header system message would be placed rather than dropped. The only injector today is Claude Code on the top-level system of /v1/messages, so the shape does not occur
• The Converse _text_part helper still requires a dict part while the shared policy reads parts through message_field, so a pydantic content part inside a mid-conversation system message is dropped on Converse where the Anthropic path converts it. Decision 14 covered message-level objects only; part-level objects inside a system message are not a shape any client sends
• Claude Platform on Bedrock (bedrock/claude_platform/claude-fable-5-1) takes the unflagged conversion for a flagged model: the prefix is stripped before the handler runs and the config's provider is bedrock, so supports_mid_conversation_system("claude-fable-5-1", "bedrock") is unmapped and falls back to False. The prefix stays stable and all 15 live scenarios on that route are 200 on both sides, but the reminder reaches the model as user text with the operator note instead of role: system; tracked as LIT-8624
• The test file test_anthropic_chat_transformation.py shows 353 deletions at bf3ba0d that are a parametrize and signature reflow: every removed def test_ has a re-added twin and 23 tests are new, so no coverage was lost, but the reflow inflates the diff
• The branch carries main up to cbba146, so CI at e05e9dc is the run to read (the merge also adapts the PR's code to the LIT013 and LIT014 lint rules that landed on main in the meantime). At the earlier tips the reds were fleet-wide: every one of this PR's 67 failing tests on PR pipeline 90120 is in the 79 failing on main pipeline 90098 (deprecated models removed by chore(cost-map): remove models past their deprecation date #42435), and main pipeline 90114 is red on the same nine jobs; using_litellm_on_windows times out in uv sync on main too (3 of its last 6 runs). The one red outside that set on PR pipeline 90128, tests/local_testing/test_streaming.py::test_openai_stream_options_call_text_completion ('MockValSer' object is not an instance of 'SchemaSerializer'), fails the same way run alone at the merge base da82ea8 and at main's b395bfe, and passed on pipeline 90120 with the same text-completion code, so it is a pre-existing order-dependent failure rather than this PR's

QA runbook

Prerequisites: ANTHROPIC_API_KEY, AWS credentials with Bedrock access in the region the test names, and a proxy with STORE_MODEL_IN_DB=True. Each test registers its own deployment and deletes it afterwards

• tests/e2e/llm_translation/test_chat_mid_conversation_system_e2e.py::TestAnthropicChatMidConversationSystem::test_flagged_model_keeps_prompt_cache_across_system_reminder - on a model that accepts role: system inside messages, a reminder added on turn two leaves the cached turn-one prefix untouched
  • POST /model/new with the master key: {"model_name": "e2e-chat-midsys-1", "litellm_params": {"model": "anthropic/claude-opus-4-8", "api_key": "os.environ/ANTHROPIC_API_KEY"}}
  • Prime the cache: POST /v1/chat/completions with a system message of about 300 short sentences carrying "cache_control": {"type": "ephemeral"}, a cache-marked user turn of about 100 sentences, and "cache": {"no-cache": true}; repeat the exact same body until usage.cache_read_input_tokens covers the whole prefix on three sends in a row
  • Send turn two: the same system and user turns, {"role": "assistant", "content": "Understood."}, {"role": "system", "content": "<system-reminder>Answer with exactly one word.</system-reminder>"}, then a new cache-marked user turn
  • Expect 200, choices[0].message.role of assistant, non-empty text, and usage.cache_read_input_tokens at least the turn-one prefix size, which proves the reminder was not hoisted into system
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
• tests/e2e/llm_translation/test_chat_mid_conversation_system_e2e.py::TestAnthropicChatMidConversationSystem::test_unflagged_model_converts_system_reminder_and_succeeds - on an older model the reminder becomes a user turn in place and the cached prefix still reads back
  • Same three steps with "model": "anthropic/claude-haiku-4-5-20251001"
  • Expect 200, an assistant role, non-empty text, and usage.cache_read_input_tokens at least the turn-one prefix size
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
• tests/e2e/llm_translation/test_chat_mid_conversation_system_e2e.py::TestBedrockInvokeChatMidConversationSystem::test_flagged_model_keeps_prompt_cache_across_system_reminder - the same guarantee on Bedrock Invoke
  • POST /model/new with {"model_name": "e2e-chat-midsys-2", "litellm_params": {"model": "bedrock/invoke/us.anthropic.claude-sonnet-5", "aws_region_name": "<region in the test>"}}, then the same priming and turn-two steps
  • Expect 200, an assistant role, non-empty text, and usage.cache_read_input_tokens at least the turn-one prefix size
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
• tests/e2e/llm_translation/test_chat_mid_conversation_system_e2e.py::TestBedrockInvokeChatMidConversationSystem::test_unflagged_model_converts_system_reminder_and_succeeds - the same conversion on Bedrock Invoke with an older model
  • Same steps with "model": "bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0"
  • Expect 200, an assistant role, non-empty text, and usage.cache_read_input_tokens at least the turn-one prefix size
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Medium Risk
Changes core message translation for Anthropic, Vertex, Azure, and Bedrock paths; incorrect placement could cause 400s or cache/thinking regressions, but behavior is heavily tested.

Overview
Fixes mid-conversation role: system handling on OpenAI-format chat completions routed to Anthropic-shaped APIs (#36559). Previously every system message was hoisted into the top-level system field, which mutates the cached prefix and breaks prompt-cache reads or preserved thinking binding on the next turn.

Only the leading run of system messages is still translated into the system parameter. Later reminders go through a new place_mid_conversation_system policy: models flagged supports_mid_conversation_system keep valid role: system entries in messages (with placement rules mirrored from Anthropic); others get an in-place user turn prefixed with a shared operator note. anthropic_messages_pt can now emit wire system messages; Bedrock Converse hoists the leading run only, converts the rest, and reorders system runs so they do not split tool calls from results.

is_unsignable_thinking_block is centralized in common_utils so placement logic and the factory agree on which thinking blocks never render. Types add AllAnthropicPassThroughMessageValues for in-message system roles. Coverage includes unit tests, provider transform tests, and live e2e cache assertions for Anthropic and Bedrock Invoke.

^{Reviewed by Cursor Bugbot for commit e05e9dc. Bugbot is set up for automated code reviews on this repo. Configure here.}

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[greptile-apps]

Confidence Score: 5/5

[Medium risk] Fixes how system messages are placed in Anthropic chat requests.

The PR appears safe to merge; no new actionable defect or outstanding blocking finding remains.

Summary

This PR preserves Anthropic conversation prefixes when handling mid-conversation system messages, preventing prompt-cache and preserved-thinking invalidation.

• Hoists only the leading system-message run and places later reminders according to neighboring rendered turns.
• Keeps in-message system roles for supported models and converts reminders to operator-marked user turns for older models and Bedrock Converse.
• Aligns placement decisions with the converter’s handling of empty, unsupported, encrypted, inline, and separate thinking content.
• Adds broad unit and end-to-end coverage across Anthropic, Bedrock, Vertex AI, Azure AI, prompt management, and Presidio flows.

_{Reviews (7) · Last reviewed commit: "fix(anthropic): let an inline thinking p..."}

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_anthropic_thinking_prefix_stable (e05e9dc) with main (3fa02ef)}

[codecov]

Codecov Report

❌ Patch coverage is 98.97260% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...itellm_core_utils/prompt_templates/common_utils.py	90.00%	1 Missing ⚠️
..._utils/prompt_templates/mid_conversation_system.py	99.44%	1 Missing ⚠️
...tellm/llms/bedrock/chat/converse_transformation.py	98.86%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[ege-arhan]

Thanks for carrying the Bedrock side over with attribution. Checked the commit list here: it includes the final green state of #41057, role narrowing in the hoisted loop and the gate-suppression placement included, so nothing from our branch is missing. I'll keep #41057 open until this merges and close it after; say the word if you'd rather it close now.

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit e05e9dc. Configure here.}

[mateo-berri]

LGTM