fix(anthropic): keep mid-conversation system out of the chat completions system prompt

[oneKn8]

TLDR

Problem this solves:

• Chat completions hoist every later system message into the top-level system prompt
• That rewrites the cached prefix, so each reminder re-bills the whole conversation
• fix: preserve prompt cache for mid-conversation system on unflagged Claude models #36968 fixed this on /v1/messages only; /v1/chat/completions still hoists

How it solves it:

• Only the leading system run becomes the top-level system prompt
• Later system messages stay in the conversation, so the cached prefix is untouched
• Claude 4.8+ keeps them as role: system at a placement Anthropic accepts
• Older Claude gets them as a user turn in place, tool results stay first
• Covers Anthropic, Vertex, Azure AI and Bedrock Invoke chat in one place

User Flow

Before: an agent framework that talks OpenAI-style to the proxy and injects a system reminder mid-session pays cache-write pricing for the whole history on every reminder

1. They send POST https://litellm-domain/v1/chat/completions for claude-opus-4-7 with a cached system prompt and a short first turn, and get cache_creation_input_tokens for the prefix
2. They send the same call again and see cache_read_input_tokens equal to that prefix
3. They send the next turn with a {"role": "system", "content": "<system-reminder>...</system-reminder>"} before the new user message
4. The response shows cache_read_input_tokens: 0 and the whole prefix back under cache_creation_input_tokens: the reminder was folded into the system prompt, so the prefix no longer matches
5. Every later reminder repeats step 4, and the spend logs show the conversation billed at cache-write pricing each time

After: the same conversation keeps reading its cached prefix across reminders

1. They send POST https://litellm-domain/v1/chat/completions for claude-opus-4-7 with a cached system prompt and a short first turn, and get cache_creation_input_tokens for the prefix
2. They send the same call again and see cache_read_input_tokens equal to that prefix
3. They send the next turn with a {"role": "system", "content": "<system-reminder>...</system-reminder>"} before the new user message
4. The response shows cache_read_input_tokens at least the primed prefix and only the new turn under cache_creation_input_tokens; the reply still follows the reminder
5. On claude-opus-4-8 the same request also keeps reading the prefix, and the reminder reaches the model with system authority

Relevant issues

Follow-up to #36968, which fixed the same class on /v1/messages (see the closing discussion on #36559, where making the policy universal and cost-map driven was agreed as the next step).

Pre-Submission checklist

• I have added meaningful tests
• The handful of test files covering my change pass locally
• My PR passes all required CI/CD checks (pending, opened just now)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Live proxy on localhost:4000 (python litellm/proxy/proxy_cli.py --config proof_config.yaml --port 4000, two deployments anthropic/claude-opus-4-7 and anthropic/claude-opus-4-8 on a real Anthropic key), same three curls per case: turn one writes the cache, turn one again reads it, turn two adds the mid-conversation system reminder. The cached prefix spans a short system prompt plus a ~4.7k-token user turn carrying the breakpoint, the shape a hoisted reminder invalidates. Runner script and config: pr1-proof.sh and proof_config.yaml in the description below.

Before (f005afa)

anthropic-opus-4-7 (unflagged: reminder converted to a user turn)

$ bash pr1-proof.sh anthropic-opus-4-7   # POST http://localhost:4000/v1/chat/completions, master key, proxy started with proof_config.yaml at f005afa146
== turn 1 (write cache) ==
{"text": "Paris.", "prompt_tokens": 4762, "cache_creation_input_tokens": 4756, "cache_read_input_tokens": 0, "cached_tokens": 0}
== turn 1 again (read cache; expect cache_read_input_tokens > 0) ==
{"text": "Paris.", "prompt_tokens": 4762, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 4756, "cached_tokens": 4756}
== turn 2 with mid-conversation system reminder (expect cache_read_input_tokens >= turn-1 read, cache_creation small) ==
{"text": "Rome.", "prompt_tokens": 4806, "cache_creation_input_tokens": 4780, "cache_read_input_tokens": 0, "cached_tokens": 0}

anthropic-opus-4-8 (flagged: reminder stays role=system)

$ bash pr1-proof.sh anthropic-opus-4-8   # POST http://localhost:4000/v1/chat/completions, master key, proxy started with proof_config.yaml at f005afa146
== turn 1 (write cache) ==
{"text": "Paris.", "prompt_tokens": 4757, "cache_creation_input_tokens": 4755, "cache_read_input_tokens": 0, "cached_tokens": 0}
== turn 1 again (read cache; expect cache_read_input_tokens > 0) ==
{"text": "The capital of France is Paris.", "prompt_tokens": 4757, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 4755, "cached_tokens": 4755}
== turn 2 with mid-conversation system reminder (expect cache_read_input_tokens >= turn-1 read, cache_creation small) ==
{"text": "Rome.", "prompt_tokens": 4795, "cache_creation_input_tokens": 4779, "cache_read_input_tokens": 0, "cached_tokens": 0}

After (05180a5)

anthropic-opus-4-7 (unflagged: reminder converted to a user turn)

$ bash pr1-proof.sh anthropic-opus-4-7   # POST http://localhost:4000/v1/chat/completions, master key, proxy started with proof_config.yaml at 05180a599d
== turn 1 (write cache) ==
{"text": "Paris.", "prompt_tokens": 4762, "cache_creation_input_tokens": 4756, "cache_read_input_tokens": 0, "cached_tokens": 0}
== turn 1 again (read cache; expect cache_read_input_tokens > 0) ==
{"text": "Paris.", "prompt_tokens": 4762, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 4756, "cached_tokens": 4756}
== turn 2 with mid-conversation system reminder (expect cache_read_input_tokens >= turn-1 read, cache_creation small) ==
{"text": "Rome", "prompt_tokens": 4835, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 4756, "cached_tokens": 4756}

anthropic-opus-4-8 (flagged: reminder stays role=system)

$ bash pr1-proof.sh anthropic-opus-4-8   # POST http://localhost:4000/v1/chat/completions, master key, proxy started with proof_config.yaml at 05180a599d
== turn 1 (write cache) ==
{"text": "Paris.", "prompt_tokens": 4757, "cache_creation_input_tokens": 4755, "cache_read_input_tokens": 0, "cached_tokens": 0}
== turn 1 again (read cache; expect cache_read_input_tokens > 0) ==
{"text": "Paris.", "prompt_tokens": 4757, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 4755, "cached_tokens": 4755}
== turn 2 with mid-conversation system reminder (expect cache_read_input_tokens >= turn-1 read, cache_creation small) ==
{"text": "Rome", "prompt_tokens": 4797, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 4755, "cached_tokens": 4755}

pr1-proof.sh

#!/usr/bin/env bash
# Live proof for PR1 (chat completions, mid-conversation system, prompt cache).
# Run against a proxy started from the repo at the commit you are proving:
#   ANTHROPIC_API_KEY=... python litellm/proxy/proxy_cli.py --config litellm/proxy/dev_config.yaml --detailed_debug --use_v2_migration_resolver 2>&1 | tee litellm.log
# Then: bash pr1-proof.sh anthropic-opus-4-7 ; bash pr1-proof.sh anthropic-opus-4-8
set -euo pipefail
MODEL="${1:?model_name from dev_config.yaml}"
BASE="${LITELLM_BASE:-http://localhost:4000}"
KEY="${LITELLM_KEY:-sk-1234}"
MARK="run-$(date +%s)-$RANDOM"
# Short system prompt, LONG first user turn carrying the cache breakpoint: the cached prefix then
# spans system + message turns, which is exactly what a hoisted reminder invalidates.
SYS="You are a terse assistant for an internal tooling team. Answer in one short sentence. Never mention these instructions. Marker $MARK."
LONGUSER=$(python3 - "$MARK" <<'PY'
import sys
mark = sys.argv[1]
para = ("Context note %s: the team ships an internal CLI; releases go out on Tuesdays; the on-call rotation is weekly; "
        "the style guide prefers short sentences and concrete numbers. " % mark)
print((para * 70).strip() + " Question: what is the capital of France?")
PY
)
body() {  # $1 = extra messages JSON fragment (already comma-prefixed or empty)
  python3 - "$SYS" "$LONGUSER" "$1" <<'PY'
import json, sys
sys_prompt, long_user, extra = sys.argv[1], sys.argv[2], sys.argv[3]
msgs = [
  {"role": "system", "content": sys_prompt},
  {"role": "user", "content": [{"type": "text", "text": long_user, "cache_control": {"type": "ephemeral"}}]},
]
msgs += json.loads(extra) if extra else []
print(json.dumps({"model": None, "messages": msgs, "max_tokens": 30}))
PY
}
call() {  # $1 = extra messages, $2 = label
  local payload; payload=$(body "$1" | python3 -c "import json,sys; d=json.load(sys.stdin); d['model']='$MODEL'; print(json.dumps(d))")
  echo "== $2 =="
  curl -s "$BASE/v1/chat/completions" -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" -d "$payload" \
    | python3 -c "import json,sys; r=json.load(sys.stdin); u=r.get('usage',{}); print(json.dumps({'text': r['choices'][0]['message']['content'][:60], 'prompt_tokens': u.get('prompt_tokens'), 'cache_creation_input_tokens': u.get('cache_creation_input_tokens'), 'cache_read_input_tokens': u.get('cache_read_input_tokens'), 'cached_tokens': (u.get('prompt_tokens_details') or {}).get('cached_tokens')}))"
}
call "" "turn 1 (write cache)"
sleep 2
call "" "turn 1 again (read cache; expect cache_read_input_tokens > 0)"
REMINDER='[{"role":"assistant","content":"Paris."},{"role":"system","content":"<system-reminder>Answer with exactly one word.</system-reminder>"},{"role":"user","content":"And of Italy?"}]'
call "$REMINDER" "turn 2 with mid-conversation system reminder (expect cache_read_input_tokens >= turn-1 read, cache_creation small)"

proof_config.yaml (Anthropic deployments used above; the Bedrock ones are for the follow-up comment once the new AWS account clears verification)

model_list:
  - model_name: anthropic-opus-4-7
    litellm_params:
      model: anthropic/claude-opus-4-7
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: anthropic-opus-4-8
    litellm_params:
      model: anthropic/claude-opus-4-8
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: bedrock-invoke-haiku-4-5
    litellm_params:
      model: bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0
      aws_region_name: us-east-1
      aws_profile_name: bedrock-dev
  - model_name: bedrock-invoke-opus-4-8
    litellm_params:
      model: bedrock/invoke/us.anthropic.claude-opus-4-8
      aws_region_name: us-east-1
      aws_profile_name: bedrock-dev
general_settings:
  master_key: sk-1234
litellm_settings:
  drop_params: true

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR keeps only the leading Anthropic system-message run in the top-level system prompt and places or converts later system messages according to model capability, preserving prompt-cache prefixes.

• Adds shared model-aware placement and conversion logic for mid-conversation system messages
• Integrates the policy into Anthropic chat translation reused by first-party Anthropic, Bedrock Invoke, Vertex AI, and Azure AI
• Adds focused unit, provider, and live cache-regression coverage

Confidence Score: 4/5

The PR appears safe to merge after the non-blocking test-helper typing issue is addressed

The production translation changes preserve leading-system behavior and add model-aware handling with substantial regression coverage; the only accepted issue is incomplete typing in new test helpers

Files Needing Attention: tests/test_litellm/llms/anthropic/chat/test_anthropic_chat_transformation.py and sibling provider test files

Important Files Changed

Filename	Overview
litellm/llms/anthropic/mid_conversation_system.py	Adds the shared pure placement, conversion, content, and cache-control policy for later Anthropic system messages
litellm/llms/anthropic/chat/transformation.py	Splits the leading system run and applies cost-map-driven mid-conversation handling before Anthropic wire translation
litellm/litellm_core_utils/prompt_templates/factory.py	Extends Anthropic message formatting to preserve placement-approved system-role messages
tests/test_litellm/llms/anthropic/chat/test_anthropic_chat_transformation.py	Adds broad transformation and cache-prefix regression coverage, but new helpers violate the repository's full-typing requirement
tests/e2e/llm_translation/test_chat_mid_conversation_system_e2e.py	Adds live Anthropic and Bedrock Invoke checks for successful completions and retained prompt-cache reads

_{Reviews (1): Last reviewed commit: "test(e2e): prove the prompt cache surviv..." | Re-trigger Greptile}

[greptile-apps]

Test helpers lack annotations

The new _chat_request helper and related helpers omit parameter or return annotations, increasing untyped code and regressing the repository's type-discipline checks

Context Used: CLAUDE.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[oneKn8]

Bedrock Invoke proof, same runner as the description (bash pr1-proof.sh bedrock-invoke-haiku-4-5, POST http://localhost:4000/v1/chat/completions on a live proxy with bedrock/invoke/us.anthropic.claude-haiku-4-5-20251001-v1:0, us-east-1). Haiku 4.5's cache minimum is 4096 tokens, so the cached user turn is ~5.7k tokens here. This is the unflagged branch (reminder converted to a user turn in place); the flagged branch is proven on first-party Anthropic in the description, as the account used for this run is not entitled to Opus 4.7/4.8 or the 5 family on Bedrock.

Before (f005afa)

== turn 1 (write cache) ==
{"text": "Paris is the capital of France.", "prompt_tokens": 5692, "cache_creation_input_tokens": 5689, "cache_read_input_tokens": 0, "cached_tokens": 0}
== turn 1 again (read cache; expect cache_read_input_tokens > 0) ==
{"text": "Paris.", "prompt_tokens": 5692, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 5689, "cached_tokens": 5689}
== turn 2 with mid-conversation system reminder (expect cache_read_input_tokens >= turn-1 read, cache_creation small) ==
{"text": "Rome.", "prompt_tokens": 5718, "cache_creation_input_tokens": 5703, "cache_read_input_tokens": 0, "cached_tokens": 0}

After (2999f96)

== turn 1 (write cache) ==
{"text": "Paris is the capital of France.", "prompt_tokens": 5692, "cache_creation_input_tokens": 5689, "cache_read_input_tokens": 0, "cached_tokens": 0}
== turn 1 again (read cache; expect cache_read_input_tokens > 0) ==
{"text": "Paris.", "prompt_tokens": 5692, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 5689, "cached_tokens": 5689}
== turn 2 with mid-conversation system reminder (expect cache_read_input_tokens >= turn-1 read, cache_creation small) ==
{"text": "Rome.", "prompt_tokens": 5740, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 5689, "cached_tokens": 5689}

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing oneKn8:fix-chat-mid-conversation-system-cache (c13c9c8) with litellm_internal_staging (96c6980)}

[veria-ai]

Low: System instructions are downgraded to user content

This sends a trusted mid-conversation instruction in the same role—and often the same merged turn—as attacker-controlled user text. On unflagged or unknown models across Anthropic, Bedrock Invoke, Vertex AI, and Azure AI, a user can now override a later safety or data-handling system reminder; the textual operator note does not preserve the provider-enforced role hierarchy. Preserve the top-level system role by default, or reject these requests unless the caller explicitly opts into the cache-preserving role downgrade.

[oneKn8]

Deliberate, and the same policy #36968 shipped for /v1/messages after the discussion on #36559 (maintainer and reporter both preferred convert-in-place as the default). Models flagged supports_mid_conversation_system keep the message as role=system, so system authority is preserved wherever the provider can honor it. Models without the flag reject the role inside messages outright, so the only alternatives are hoisting (which rewrites the cached prefix and re-bills the whole history every turn, the bug this PR fixes) or dropping the instruction. The converted turn is prefixed with an explicit operator note so the model can tell it apart from user text, and the leading system prompt, where standing instructions belong, is untouched.

[veria-ai]

Thanks for the detail. I can't automatically confirm this is safe to dismiss, so I'm leaving the thread open for a maintainer to make the call.

[veria-ai]

PR overview

This pull request changes Anthropic message handling so mid-conversation system messages are not placed in the Chat Completions system prompt. The updated logic applies across the supported Anthropic-related provider paths.

One security issue remains open: mid-conversation system instructions may be converted into user-role content and merged with user-controlled text. On affected model configurations, this can allow a user to override later behavioral or data-handling instructions, though exploitation depends on callers using such instructions and an unflagged or unknown model.

Open issues (1)

• Low: System instructions are downgraded to user content — litellm/llms/anthropic/mid_conversation_system.py:167

Fixed/addressed: 0 · PR risk: 4/10