fix(prompt-caching): default injected /v1/messages breakpoints to implicit lookup

[ryan-crabbe-berri]

TLDR

Problem this solves:

• /v1/messages on GPT-5.6 still sends prompt_cache_options: {"mode": "explicit"} for configured breakpoints
• Explicit mode turns off OpenAI's automatic prefix lookup, so a moving index: -1 checkpoint rewrites instead of reading
• fix(responses): preserve prompt cache reuse in chat bridge #42281 fixed that same default on /v1/chat/completions and /v1/responses only

How it solves it:

• The /v1/messages injection path now defaults to {"mode": "implicit"}, same as the chat path
• A prompt_cache_options from the caller or litellm_params still wins
• One regression test pins both paths to the same default

Intentional product change: configured breakpoints arriving on /v1/messages now also let OpenAI keep its automatic checkpoint and read earlier prefixes. Deployments that want explicit-only writes set prompt_cache_options: {mode: explicit} in litellm_params, the same opt-in the chat path has had since #42281

User Flow

Before: a developer using the Anthropic SDK or Claude Code against a GPT-5.6 deployment with configured breakpoints sees cache reads stall while writes keep growing

1. The proxy admin configures cache_control_injection_points (role: system and index: -1) on an openai/gpt-5.6 deployment
2. The developer sends POST https://litellm-domain/v1/messages with a system prompt, tools and a user turn; it returns cache_creation_input_tokens: 2609
3. Turn 2 adds a tool_use and tool_result and returns cache_read_input_tokens: 2144, cache_creation_input_tokens: 1415
4. Turn 3 adds one more exchange and returns cache_read_input_tokens: 2144 again with cache_creation_input_tokens: 1883, so what turn 2 wrote is never read back

After: the same conversation reads back everything the previous turn wrote

1. The proxy admin configures the same cache_control_injection_points on the same deployment
2. The developer sends the same first POST https://litellm-domain/v1/messages; it returns cache_creation_input_tokens: 2609
3. Turn 2 returns cache_read_input_tokens: 2609, cache_creation_input_tokens: 950
4. Turn 3 returns cache_read_input_tokens: 3559, cache_creation_input_tokens: 468

Relevant issues

Follow-up to #42281

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Both runs hit the real OpenAI API through a DB-less proxy started from the given commit with python litellm/proxy/proxy_cli.py --config cache_mode_config.yaml --port <port> --detailed_debug. The salt in the system prompt keeps the two runs on separate OpenAI cache prefixes

model_list:
  - model_name: gpt-5.6
    litellm_params:
      model: openai/gpt-5.6
      api_key: os.environ/OPENAI_API_KEY
      cache_control_injection_points:
        - location: message
          role: system
        - location: message
          index: -1

general_settings:
  master_key: sk-c4ch3m0d3-9f2b7e1a4d6c8b0f3e5a7c9d1b2f4e6a

#!/usr/bin/env bash
set -euo pipefail
PORT=$1; SALT=$2
filler() { jq -rn --arg p "$1" --argjson n "$2" '[range($n)] | map("\($p)\(.)") | join(" ")'; }
SYS="salt=$SALT $(filler sysword 700)"
TOOLS='[{"name":"get_weather","description":"Get weather for a city","input_schema":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}]'

turn() {
  curl -sS "http://localhost:$PORT/v1/messages" \
    -H "Authorization: Bearer sk-c4ch3m0d3-9f2b7e1a4d6c8b0f3e5a7c9d1b2f4e6a" -H "Content-Type: application/json" \
    -d "$(jq -cn --arg sys "$SYS" --argjson tools "$TOOLS" --argjson msgs "$1" \
      '{model:"gpt-5.6", max_tokens:16, system:$sys, tools:$tools, metadata:{user_id:"cache-mode-demo"}, messages:$msgs}')" \
    | jq -c '{cache_read: .usage.cache_read_input_tokens, cache_write: .usage.cache_creation_input_tokens, input: .usage.input_tokens, err: .error.message}'
}

U1=$(filler userone 150)
T1=$(filler weatherfact 150)
U2=$(filler usertwo 150)
U3=$(filler userthree 150)

M1=$(jq -cn --arg u1 "$U1" '[{"role":"user","content":($u1 + " What is the weather in Paris? Use the tool.")}]')
M2=$(jq -cn --arg u1 "$U1" --arg t1 "$T1" --arg u2 "$U2" '[
  {"role":"user","content":($u1 + " What is the weather in Paris? Use the tool.")},
  {"role":"assistant","content":[{"type":"tool_use","id":"toolu_1","name":"get_weather","input":{"city":"Paris"}}]},
  {"role":"user","content":[{"type":"tool_result","tool_use_id":"toolu_1","content":($t1 + " Paris: sunny, 21C")},{"type":"text","text":($u2 + " Summarize that in 3 words.")}]}
]')
M3=$(jq -cn --argjson m2 "$M2" --arg u3 "$U3" '$m2 + [{"role":"assistant","content":"Sunny and warm."},{"role":"user","content":($u3 + " Now say ok.")}]')

echo "turn 1"; turn "$M1"
echo "turn 2"; turn "$M2"
echo "turn 3"; turn "$M3"

Before (740d043)

1. bash messages_turns.sh 4281 before-r1

   turn 1
   {"cache_read":null,"cache_write":2609,"input":3,"err":null}
   turn 2
   {"cache_read":2144,"cache_write":1415,"input":3,"err":null}
   turn 3
   {"cache_read":2144,"cache_write":1883,"input":3,"err":null}
   

2. Every one of the three upstream litellm.aresponses(...) calls in the proxy log carries prompt_cache_options={'mode': 'explicit'}. Turn 3 reads the same 2144 tokens turn 2 read and rewrites 1883, so the moving checkpoint never pays off

After (98c4d35)

1. bash messages_turns.sh 4282 after-r1

   turn 1
   {"cache_read":null,"cache_write":2609,"input":3,"err":null}
   turn 2
   {"cache_read":2609,"cache_write":950,"input":3,"err":null}
   turn 3
   {"cache_read":3559,"cache_write":468,"input":3,"err":null}
   

2. Every one of the three upstream litellm.aresponses(...) calls in the proxy log carries prompt_cache_options={'mode': 'implicit'}. Turn 2 reads all 2609 tokens turn 1 wrote and turn 3 reads all 3559 tokens turn 2 left cached

Type

🐛 Bug Fix

Caveats (if any)

Low

• Explicit-only writes on /v1/messages are now an opt-in via prompt_cache_options: {mode: explicit} in litellm_params
• Putting the old explicit default back fails test_v1_messages_and_chat_paths_default_to_the_same_implicit_mode and test_v1_messages_string_system_gets_block_breakpoint, so the two paths cannot drift apart again silently

[mateo-berri]

LGTM. Let's gooo :)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Confidence Score: 5/5

[Medium risk] Changes prompt caching mode from explicit to implicit.

The PR appears safe to merge, with no actionable issues found

What we checked:

• Explicit settings remain unchanged: The router passes deployment settings into the request, and setdefault leaves an existing prompt_cache_options value unchanged
• Cache options reach Responses: The adapter leaves this option for the handler to copy from the request into the call to litellm.aresponses

Summary

Changes injected OpenAI breakpoints on /v1/messages to default to prompt_cache_options: {"mode": "implicit"}, matching the chat path

• Keeps caller and deployment options unchanged
• Adds a regression test comparing both defaults and extends coverage for supplied modes

ryan-crabbe-berri explicitly acknowledged the default change as intentional. Deployments needing explicit-only behavior can set prompt_cache_options: {"mode": "explicit"}

_{Reviews (1) · Last reviewed commit: "fix(prompt-caching): default injected /v..." · Reviewed by Greptile}

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_messages_cache_mode_implicit (98c4d35) with main (2c1847f)¹}

Footnotes

1. No successful run was found on main (31b90eb) during the generation of this report, so 2c1847f was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩