feat(prompt-caching): map cache_control_injection_points to OpenAI prompt_cache_breakpoint on GPT-5.6+ targets

[mateo-berri]

TLDR

Problem this solves:

• cache_control_injection_points only emits Anthropic cache_control, which OpenAI strips
• OpenAI GPT-5.6 explicit caching (prompt_cache_breakpoint) is unreachable from config
• /v1/messages and /v1/responses clients cannot place an OpenAI breakpoint

How it solves it:

• OpenAI GPT-5.6+ targets get prompt_cache_breakpoint on the configured block
• The request gets prompt_cache_options: {"mode": "explicit"}, a caller's own value wins
• Covers chat completions, /v1/responses and both /v1/messages bridges
• Non-OpenAI targets and older OpenAI models keep today's behavior byte for byte

Design notes: why, and what changed per file

Why

OpenAI's GPT-5.6+ models support explicit prompt caching through a block-level prompt_cache_breakpoint marker and a request-level prompt_cache_options object. LiteLLM's cache_control_injection_points only ever emitted Anthropic's cache_control, which the OpenAI transformation strips before sending, so a proxy that routes the same deployment config to Anthropic and OpenAI models could not place breakpoints on the OpenAI side. /v1/messages clients (Claude Code, the Anthropic SDK) also had no way to get a breakpoint onto an OpenAI request, because the Responses bridge folded the system prompt into top-level instructions, which cannot carry one.

What changed

litellm/integrations/anthropic_cache_control_hook.py

• New dialect check, decided once per request: the model must be eligible, the provider must be openai, and the request must really target OpenAI. Eligibility comes from the model map when the entry carries the flag: supports_prompt_cache_breakpoint: true is set on the OpenAI gpt-5.6, gpt-5.6-sol, gpt-5.6-terra and gpt-5.6-luna entries, exposed through a new litellm.utils.supports_prompt_cache_breakpoint(model, custom_llm_provider=None) built on the same _supports_factory as supports_prompt_caching, and an explicit false is honored. A model the map does not know, or a listed entry without the flag, falls back to the gpt-<major>[.<minor>] version rule (5.6 or newer, so gpt-5.6 on a published map that does not carry the flag yet, an unlisted gpt-5.7 or gpt-6 all qualify while gpt-4.1 or o3 do not). The first QA round caught this: a proxy on the default remote cost map never fired because the published map lags this PR, and the unit tests only passed because the repo .env pins LITELLM_LOCAL_MODEL_COST_MAP=True. The provider is the custom_llm_provider the caller gave when there is one, else get_llm_provider(model); a routing failure falls back to today's behavior and litellm_proxy/<model> never gets the dialect. The target check resolves the request api_base (or its base_url alias, which completion(), acompletion() and litellm.responses() accept), then litellm.api_base, then OPENAI_BASE_URL / OPENAI_API_BASE: unset, api.openai.com and *.api.openai.com hosts speak the dialect; any other host keeps today's Anthropic-style behavior unless the request carries a top-level prompt_cache_options, which is the opt-in for an OpenAI-compatible target that understands the dialect. maybe_seed_default_injection_points (which sees api_base and the provider, and now takes api_base from both completion() and acompletion()) stamps the finished decision onto the configured points as _litellm_openai_dialect, in the same spirit as the existing _litellm_judged stamp, so the chat-path hook entry point (which never sees api_base) honors it on the sync path too; unstamped points are judged from the request params, and /v1/messages passes its api_base to maybe_inject_cache_control. The stamp is only added for eligible models so every other request keeps its points untouched, and provider resolution is skipped entirely for ineligible models, so unroutable custom model names never trigger the get_llm_provider lookup.
• In the OpenAI dialect the marker goes on the content block only. String content is wrapped into [{"type": "text", "text": ...}], a string /v1/messages system becomes one text block, and list content gets the marker on the last block OpenAI can carry it on (text, image, image_url, file, input_audio, and the Responses input parts input_text, input_image, input_file), walking back past tool_result and other ineligible blocks. Assistant messages are never marked, and a turn with no eligible block (for example a tool_result-only user turn) is left alone and does not consume one of the four slots. The injection point's control field is ignored, since explicit is the only mode OpenAI accepts.
• After at least one marker was actually written the hook sets prompt_cache_options = {"mode": "explicit"} on the request (non_default_params on the chat path, the handler kwargs on /v1/messages) unless the caller already passed one, so a deployment-level prompt_cache_options: {"mode": "explicit", "ttl": "30m"} in litellm_params wins.
• Breakpoint counting for the four-breakpoint cap and the stand-down check now counts both marker kinds, so a client that already sends prompt_cache_breakpoint makes configured points stand down exactly like a client that sends cache_control.
• Fixes a pre-existing double count in apply_to_anthropic_messages_request: client breakpoints already present in the messages were subtracted from the cap once for the system point budget and again inside the message pass, which silently lost one injectable slot per client-marked message. The message pass now receives the cap minus the system markers only.
• tool_config points no longer reserve one of the four slots in the OpenAI dialect, on the chat path and on /v1/messages alike: OpenAI has no tool-config cache block, so four message points plus a tool_config point now mark all four messages there. Bedrock targets keep the reservation.

litellm/responses/utils.py (/v1/responses)

• The prompt-management merge that feeds cache_control_injection_points on the Responses path now shapes chat-style {"type": "text"} parts on non-assistant input items back into input_text (the hook wraps a string item into a text block), so a marked system, developer or user input item is a valid Responses input item. prompt_cache_options travels through ResponsesAPIOptionalRequestParams and reaches the outbound body on both litellm.responses and litellm.aresponses. The reshape returns copies, so the hook's output and any client message objects a prompt-management hook hands back unchanged are never mutated.

litellm/llms/anthropic/experimental_pass_through/responses_adapters/transformation.py (/v1/messages to Responses API)

• A system list carrying any breakpoint is no longer folded into instructions; it becomes a leading developer message with one input_text part per text block, markers preserved. System prompts without a breakpoint still become instructions.
• prompt_cache_breakpoint is carried from user text to input_text, from user image to input_image, and on mid-turn system blocks. Assistant and tool_result blocks drop it, since OpenAI does not accept breakpoints there.
• prompt_cache_options rides to litellm.aresponses through the existing extra-kwargs forwarding.

litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py (/v1/messages to chat completions)

• Copies prompt_cache_breakpoint onto the OpenAI text and image blocks at the system and user sites. prompt_cache_options reaches acompletion through the existing kwargs forwarding and then extra_body.

litellm/types/llms/openai.py, litellm/types/llms/anthropic.py, litellm/types/integrations/anthropic_cache_control_hook.py, litellm/types/utils.py, litellm/utils.py, litellm/litellm_core_utils/prompt_templates/common_utils.py, model_prices_and_context_window.json (and its backup)

• PromptCacheBreakpoint and PromptCacheOptions TypedDicts, prompt_cache_options on ResponsesAPIOptionalRequestParams so the Responses param filter keeps it, prompt_cache_breakpoint on the Anthropic text, image and system block types, the _litellm_openai_dialect stamp on the injection point types, a supports_prompt_cache_breakpoint capability flag on ProviderSpecificModelInfo / ModelInfo (populated by get_model_info like supports_prompt_caching, schema regenerated) set on the four OpenAI GPT-5.6 entries, and a small pure with_prompt_cache_breakpoint helper shared by the hook and the adapters (it returns a new block carrying the marker and callers use the return value; the chat bridge helper does the same). ChatCompletionTextObject deliberately does not grow the field: pydantic builds a schema for it at import time and warns about every ReadOnly item it sees, while the repo's type-discipline gate requires ReadOnly on TypedDict fields, so the hook sets the key through the helper instead.

Chat completions transformation: no code change. prompt_cache_options is deliberately not added to the supported chat params; get_optional_params already moves unknown params for the openai provider into extra_body, which is the only way it can reach the installed OpenAI SDK (its chat.completions.create has no such kwarg). The existing cache_control strip leaves prompt_cache_breakpoint alone. Tests pin both facts.

Tests: new unit coverage (no network) in the existing modules for the hook (dialect table, provider stamp with custom_llm_provider set to openai, an OpenAI-compatible provider and unset, block placement and walk-back, assistant and tool_result-only turns skipped without consuming a slot, tool role text marked on the chat path, control ignored, caller prompt_cache_options preserved, stand-down and cap counting across both marker kinds, the double-count fix, non-OpenAI and pre-5.6 targets unchanged with point identity preserved), the Responses bridge (developer message, marker carry, assistant and tool_result drop, kwargs forwarding), the OpenAI Responses transformation, the chat transformation, the chat bridge, the /v1/responses request bodies (litellm.responses and litellm.aresponses on a GPT-5.6 target, below 5.6, behind a custom api_base, on a litellm_proxy/ target, with the prompt_cache_options opt-in and on a regional *.api.openai.com host) and the Responses input shaping, the target gate on the chat path and /v1/messages (custom api_base, litellm.api_base and env fallbacks, request api_base over env, litellm_proxy/, the opt-in, the stamped decision and its authority over request params, and completion() with a custom api_base against a mocked OpenAI client), the tool_config slot in both dialects, and the model-map flag with its version fallback plus the model map schema check, the published-map fallback (model-map tests pinned to the bundled map, the hook run against an unflagged gpt-5.6 entry, and the touched suites run with the remote map), base_url on completion(), acompletion() and litellm.responses(), and the copy-on-reshape behavior of the Responses merge.

User Flow

Before: on an OpenAI GPT-5.6 deployment the configured injection point changes nothing. OpenAI's automatic breakpoint decides what gets cached, every turn pays a write for its own tail, and no proxy setting switches the deployment to explicit mode

1. The proxy admin adds a gpt-5.6 deployment (openai/gpt-5.6) with cache_control_injection_points: [{"location": "message", "role": "system"}] to config.yaml and starts the proxy
2. A developer sends POST https://litellm-domain/v1/messages (Anthropic SDK) with a 6k-token system prompt and user question A, which carries about 1500 tokens of session notes of its own: 200, and usage.cache_creation_input_tokens covers the whole prompt including the user turn (7495 in the QA run) while cache_read_input_tokens is 0
3. They send the same system prompt with question B and fresh notes: 200, cache_read_input_tokens: 6002 and cache_creation_input_tokens: 1479, so the system prompt was read and the whole new user turn was written again at OpenAI's automatic breakpoint
4. They send B again with fresh notes: 200, cache_read_input_tokens: 6002 and another 1514 tokens written. Nothing about the configured system breakpoint shows up, and no config setting puts the deployment in explicit mode or sets a ttl
5. The same three calls on POST https://litellm-domain/v1/chat/completions (OpenAI SDK, a system plus a user message): usage.prompt_tokens_details follows the same automatic pattern, 7537 written, then 1494 and 1478 on top of 6001 read
6. The same three calls on POST https://litellm-domain/v1/responses (OpenAI SDK, a system input item plus the user item): same automatic pattern (7500, then 1513 and 1524), and a developer who puts their own prompt_cache_breakpoint on the system item and prompt_cache_options: {"mode": "explicit"} on the request sees both silently dropped: 7497 written, then 1517 and 1505

After: the same injection point becomes an OpenAI prompt_cache_breakpoint and the request runs in explicit mode, so the cache boundary sits exactly at the configured system prompt on all three endpoints and the follow-up turns write nothing

1. The proxy admin adds a gpt-5.6 deployment (openai/gpt-5.6) with cache_control_injection_points: [{"location": "message", "role": "system"}] to config.yaml and starts the proxy
2. A developer sends POST https://litellm-domain/v1/messages (Anthropic SDK) with a 6k-token system prompt and user question A, which carries about 1500 tokens of session notes of its own: 200, and usage.cache_creation_input_tokens now covers only the system prompt (6001) while cache_read_input_tokens is 0 and the user turn is plain input_tokens
3. They send the same system prompt with question B and fresh notes: 200, cache_read_input_tokens: 6001 and no cache_creation_input_tokens at all: nothing past the configured breakpoint is written
4. They send B again with fresh notes: 200, cache_read_input_tokens: 6001 again. The admin can add prompt_cache_options: {"mode": "explicit", "ttl": "30m"} to the deployment's litellm_params and it is honored
5. The same three calls on POST https://litellm-domain/v1/chat/completions: usage.prompt_tokens_details.cache_write_tokens is 6001, then 0 and 0, with cached_tokens: 6001 on calls 2 and 3
6. The same three calls on POST https://litellm-domain/v1/responses with a system input item: cache_write_tokens 6002, then 0 and 0, and the developer's own prompt_cache_breakpoint plus prompt_cache_options on a deployment with no injection point is forwarded instead of dropped (6001 written, then 0 and 0)

Relevant issues

Fixes #37509
Related: #32656

Linear ticket

Resolves LIT-5876

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Both legs drive the same client scripts, unmodified, against a proxy booted from the named commit with this config (__MASTER_KEY__ differs per leg). gpt-5.6 carries the injection point from the ticket, gpt-5.6-plain has none and is used for the client-sent-marker case.

model_list:
  - model_name: gpt-5.6
    litellm_params:
      model: openai/gpt-5.6
      api_key: os.environ/OPENAI_API_KEY
      cache_control_injection_points:
        - location: message
          role: system
  - model_name: gpt-5.6-plain
    litellm_params:
      model: openai/gpt-5.6
      api_key: os.environ/OPENAI_API_KEY
general_settings:
  master_key: __MASTER_KEY__

Every case sends the same ~6k-token reference text as the system prompt (4598 words, plus a per-run Session salt: <hex> line so each run starts from a cold cache), then three calls 6 s apart: question A, question B, question B again. Each user message carries its own ~1500-token block of scratch notes seeded from the salt and the call index, the way a real session's turns differ after the shared prefix. That tail is what tells the two modes apart in the usage numbers: OpenAI's implicit caching caches the whole prompt, so every call writes its new tail on top of the reads, while an explicit breakpoint on the system message caches up to the marker and nothing after it, so call 1 writes ~6000 and calls 2 and 3 write 0. The helper shared by the scripts:

common.py

SYSTEM_PROMPT_PATH = "system_prompt.txt"
QUESTION_A = "In one sentence, what is the main topic of the reference text above?"
QUESTION_B = "In one sentence, name one specific detail from the reference text above."

def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_argument("--port", required=True, type=int)
    parser.add_argument("--key", required=True)
    parser.add_argument("--model", default="gpt-5.6")
    parser.add_argument("--salt", required=True)
    parser.add_argument("--pause", type=float, default=6.0)
    return parser.parse_args()

def system_prompt(salt):
    base = pathlib.Path(SYSTEM_PROMPT_PATH).read_text()
    return f"{base}\n\nSession salt: {salt}\n"

def scratch_notes(salt, index, words=1100):
    vocabulary = pathlib.Path(SYSTEM_PROMPT_PATH).read_text().split()
    rng = random.Random(f"{salt}-{index}")
    return " ".join(rng.choice(vocabulary) for _ in range(words))

def user_message(question, salt, index):
    return f"{question}\n\nScratch notes from my session, ignore them when answering:\n{scratch_notes(salt, index)}"

def run_sequence(label, call, pause, salt=None):
    for index, question in enumerate([QUESTION_A, QUESTION_B, QUESTION_B], start=1):
        usage = call(user_message(question, salt, index) if salt else question)
        tag = "A" if index == 1 else "B"
        print(f"[{label}] call {index} ({tag}): usage={json.dumps(usage, sort_keys=True)}", flush=True)
        if index < 3:
            time.sleep(pause)

case_messages.py (Anthropic SDK, the ticket's client)

client = anthropic.Anthropic(base_url=f"http://127.0.0.1:{args.port}", api_key=args.key)
SYSTEM = system_prompt(args.salt)

def call(question):
    response = client.messages.create(
        model=args.model,
        max_tokens=512,
        system=SYSTEM,
        messages=[{"role": "user", "content": question}],
    )
    return response.usage.model_dump(exclude_none=True)

run_sequence("v1/messages anthropic-sdk", call, args.pause, salt=args.salt)

case_chat.py (OpenAI SDK)

client = openai.OpenAI(base_url=f"http://127.0.0.1:{args.port}/v1", api_key=args.key)
SYSTEM = system_prompt(args.salt)

def call(question):
    response = client.chat.completions.create(
        model=args.model,
        messages=[{"role": "system", "content": SYSTEM}, {"role": "user", "content": question}],
        max_completion_tokens=512,
        reasoning_effort="low",
    )
    return response.usage.model_dump(exclude_none=True)

run_sequence("v1/chat/completions openai-sdk", call, args.pause, salt=args.salt)

case_responses.py (OpenAI SDK)

client = openai.OpenAI(base_url=f"http://127.0.0.1:{args.port}/v1", api_key=args.key)
SYSTEM = system_prompt(args.salt)

def call(question):
    response = client.responses.create(
        model=args.model,
        input=[{"role": "system", "content": SYSTEM}, {"role": "user", "content": question}],
        max_output_tokens=512,
        reasoning={"effort": "low"},
    )
    usage = response.usage.model_dump(exclude_none=True)
    usage["status"] = response.status
    return usage

run_sequence("v1/responses openai-sdk", call, args.pause, salt=args.salt)

case_responses_client_explicit.py (OpenAI SDK, client places its own marker and options, run against gpt-5.6-plain)

client = openai.OpenAI(base_url=f"http://127.0.0.1:{args.port}/v1", api_key=args.key)
SYSTEM = system_prompt(args.salt)

def call(question):
    response = client.responses.create(
        model=args.model,
        input=[
            {
                "role": "system",
                "content": [
                    {"type": "input_text", "text": SYSTEM, "prompt_cache_breakpoint": {"mode": "explicit"}}
                ],
            },
            {"role": "user", "content": question},
        ],
        max_output_tokens=512,
        reasoning={"effort": "low"},
        extra_body={"prompt_cache_options": {"mode": "explicit"}},
    )
    usage = response.usage.model_dump(exclude_none=True)
    usage["status"] = response.status
    return usage

run_sequence("v1/responses client-sent breakpoint+options openai-sdk", call, args.pause, salt=args.salt)

Before (6fcdea0)

/v1/messages (Anthropic SDK)

1. python case_messages.py --port 46429 --key sk-qa-lit5876-before --salt 668d76e2
2. Output: call 1 writes the whole 7495-token prompt, calls 2 and 3 read the 6002-token shared prefix and still write the ~1500-token tail every time; OpenAI's implicit caching is doing the work and the injection point changed nothing

   [v1/messages anthropic-sdk] call 1 (A): usage={"cache_creation_input_tokens": 7495, "input_tokens": 3, "output_tokens": 61}
   [v1/messages anthropic-sdk] call 2 (B): usage={"cache_creation_input_tokens": 1479, "cache_read_input_tokens": 6002, "input_tokens": 3, "output_tokens": 66}
   [v1/messages anthropic-sdk] call 3 (B): usage={"cache_creation_input_tokens": 1514, "cache_read_input_tokens": 6002, "input_tokens": 3, "output_tokens": 32}
   

/v1/chat/completions (OpenAI SDK)

1. python case_chat.py --port 46429 --key sk-qa-lit5876-before --salt 783c4435
2. Output: same implicit pattern, cache_write_tokens 7537 then 1494 then 1478

   [v1/chat/completions openai-sdk] call 1 (A): usage={"completion_tokens": 23, ..., "prompt_tokens": 7540, "prompt_tokens_details": {"audio_tokens": 0, "cache_creation_tokens": 7537, "cache_write_tokens": 7537, "cached_tokens": 0}, "total_tokens": 7563}
   [v1/chat/completions openai-sdk] call 2 (B): usage={"completion_tokens": 31, ..., "prompt_tokens": 7498, "prompt_tokens_details": {"audio_tokens": 0, "cache_creation_tokens": 1494, "cache_write_tokens": 1494, "cached_tokens": 6001}, "total_tokens": 7529}
   [v1/chat/completions openai-sdk] call 3 (B): usage={"completion_tokens": 30, ..., "prompt_tokens": 7482, "prompt_tokens_details": {"audio_tokens": 0, "cache_creation_tokens": 1478, "cache_write_tokens": 1478, "cached_tokens": 6001}, "total_tokens": 7512}
   

/v1/responses (OpenAI SDK)

1. python case_responses.py --port 46429 --key sk-qa-lit5876-before --salt a169348b
2. Output: same implicit pattern on the Responses API, cache_write_tokens 7500 then 1513 then 1524

   [v1/responses openai-sdk] call 1 (A): usage={"input_tokens": 7503, "input_tokens_details": {"cache_write_tokens": 7500, "cached_tokens": 0}, "output_tokens": 22, "output_tokens_details": {"reasoning_tokens": 0}, "status": "completed", "total_tokens": 7525}
   [v1/responses openai-sdk] call 2 (B): usage={"input_tokens": 7516, "input_tokens_details": {"cache_write_tokens": 1513, "cached_tokens": 6000}, "output_tokens": 32, "output_tokens_details": {"reasoning_tokens": 0}, "status": "completed", "total_tokens": 7548}
   [v1/responses openai-sdk] call 3 (B): usage={"input_tokens": 7527, "input_tokens_details": {"cache_write_tokens": 1524, "cached_tokens": 6000}, "output_tokens": 32, "output_tokens_details": {"reasoning_tokens": 0}, "status": "completed", "total_tokens": 7559}
   

/v1/responses with a client-sent prompt_cache_breakpoint and prompt_cache_options (gpt-5.6-plain)

1. python case_responses_client_explicit.py --model gpt-5.6-plain --port 46429 --key sk-qa-lit5876-before --salt 443c99b2
2. Output: the client asked for explicit mode on its system block and still got the implicit pattern (1517 and 1505-token writes on calls 2 and 3), so its prompt_cache_options never reached OpenAI

   [v1/responses client-sent breakpoint+options openai-sdk] call 1 (A): usage={"input_tokens": 7500, "input_tokens_details": {"cache_write_tokens": 7497, "cached_tokens": 0}, "output_tokens": 22, "output_tokens_details": {"reasoning_tokens": 0}, "status": "completed", "total_tokens": 7522}
   [v1/responses client-sent breakpoint+options openai-sdk] call 2 (B): usage={"input_tokens": 7522, "input_tokens_details": {"cache_write_tokens": 1517, "cached_tokens": 6002}, "output_tokens": 32, "output_tokens_details": {"reasoning_tokens": 0}, "status": "completed", "total_tokens": 7554}
   [v1/responses client-sent breakpoint+options openai-sdk] call 3 (B): usage={"input_tokens": 7510, "input_tokens_details": {"cache_write_tokens": 1505, "cached_tokens": 6002}, "output_tokens": 32, "output_tokens_details": {"reasoning_tokens": 0}, "status": "completed", "total_tokens": 7542}
   

Claude Code TUI on the gpt-5.6 deployment

1. tmux new-session -d -s lit5876qa_before -x 180 -y 45 "env ANTHROPIC_BASE_URL=http://127.0.0.1:46429 ANTHROPIC_AUTH_TOKEN=sk-qa-lit5876-before ANTHROPIC_MODEL=gpt-5.6 ANTHROPIC_SMALL_FAST_MODEL=gpt-5.6 CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 claude", accept the trust prompt, then tmux send-keys "Reply with exactly the word pong and nothing else" Enter
2. Pane after the turn (Claude Code v2.1.237 sends its own cache_control markers, the session works and answers)

   │   gpt-5.6 with xhigh effort · API Usage Billing   │
   ❯ Reply with exactly the word pong and nothing else
   ⏺ pong
   ✻ Sautéed for 3s
   

After (e16cf5e)

/v1/messages (Anthropic SDK)

1. python case_messages.py --port 29616 --key sk-qa-lit5876-after --salt 0841ef16
2. Output: call 1 writes exactly the 6001-token system prompt, calls 2 and 3 read it back and write nothing; the ~1500-token tail now shows up as uncached input_tokens instead of being rewritten on every call

   [v1/messages anthropic-sdk] call 1 (A): usage={"cache_creation_input_tokens": 6001, "input_tokens": 1516, "output_tokens": 22}
   [v1/messages anthropic-sdk] call 2 (B): usage={"cache_read_input_tokens": 6001, "input_tokens": 1488, "output_tokens": 31}
   [v1/messages anthropic-sdk] call 3 (B): usage={"cache_read_input_tokens": 6001, "input_tokens": 1519, "output_tokens": 32}
   

/v1/chat/completions (OpenAI SDK)

1. python case_chat.py --port 29616 --key sk-qa-lit5876-after --salt 9775ffc7
2. Output: explicit pattern, cache_write_tokens 6001 then 0 then 0 with cached_tokens 6001 on the follow-ups

   [v1/chat/completions openai-sdk] call 1 (A): usage={"completion_tokens": 21, ..., "prompt_tokens": 7521, "prompt_tokens_details": {"audio_tokens": 0, "cache_creation_tokens": 6001, "cache_write_tokens": 6001, "cached_tokens": 0}, "total_tokens": 7542}
   [v1/chat/completions openai-sdk] call 2 (B): usage={"completion_tokens": 31, ..., "prompt_tokens": 7493, "prompt_tokens_details": {"audio_tokens": 0, "cache_creation_tokens": 0, "cache_write_tokens": 0, "cached_tokens": 6001}, "total_tokens": 7524}
   [v1/chat/completions openai-sdk] call 3 (B): usage={"completion_tokens": 28, ..., "prompt_tokens": 7508, "prompt_tokens_details": {"audio_tokens": 0, "cache_creation_tokens": 0, "cache_write_tokens": 0, "cached_tokens": 6001}, "total_tokens": 7536}
   

/v1/responses (OpenAI SDK)

1. python case_responses.py --port 29616 --key sk-qa-lit5876-after --salt 9aaecf74
2. Output: explicit pattern on the Responses API, cache_write_tokens 6002 then 0 then 0

   [v1/responses openai-sdk] call 1 (A): usage={"input_tokens": 7546, "input_tokens_details": {"cache_write_tokens": 6002, "cached_tokens": 0}, "output_tokens": 22, "output_tokens_details": {"reasoning_tokens": 0}, "status": "completed", "total_tokens": 7568}
   [v1/responses openai-sdk] call 2 (B): usage={"input_tokens": 7496, "input_tokens_details": {"cache_write_tokens": 0, "cached_tokens": 6002}, "output_tokens": 33, "output_tokens_details": {"reasoning_tokens": 0}, "status": "completed", "total_tokens": 7529}
   [v1/responses openai-sdk] call 3 (B): usage={"input_tokens": 7495, "input_tokens_details": {"cache_write_tokens": 0, "cached_tokens": 6002}, "output_tokens": 32, "output_tokens_details": {"reasoning_tokens": 0}, "status": "completed", "total_tokens": 7527}
   

/v1/responses with a client-sent prompt_cache_breakpoint and prompt_cache_options (gpt-5.6-plain)

1. python case_responses_client_explicit.py --model gpt-5.6-plain --port 29616 --key sk-qa-lit5876-after --salt 543756c9
2. Output: the client's own marker and prompt_cache_options now reach OpenAI on a deployment with no injection point, so it gets the explicit pattern it asked for

   [v1/responses client-sent breakpoint+options openai-sdk] call 1 (A): usage={"input_tokens": 7504, "input_tokens_details": {"cache_write_tokens": 6001, "cached_tokens": 0}, "output_tokens": 29, "output_tokens_details": {"reasoning_tokens": 0}, "status": "completed", "total_tokens": 7533}
   [v1/responses client-sent breakpoint+options openai-sdk] call 2 (B): usage={"input_tokens": 7479, "input_tokens_details": {"cache_write_tokens": 0, "cached_tokens": 6001}, "output_tokens": 31, "output_tokens_details": {"reasoning_tokens": 0}, "status": "completed", "total_tokens": 7510}
   [v1/responses client-sent breakpoint+options openai-sdk] call 3 (B): usage={"input_tokens": 7518, "input_tokens_details": {"cache_write_tokens": 0, "cached_tokens": 6001}, "output_tokens": 32, "output_tokens_details": {"reasoning_tokens": 0}, "status": "completed", "total_tokens": 7550}
   

Claude Code TUI on the gpt-5.6 deployment

1. tmux new-session -d -s lit5876qa_after -x 180 -y 45 "env ANTHROPIC_BASE_URL=http://127.0.0.1:29616 ANTHROPIC_AUTH_TOKEN=sk-qa-lit5876-after ANTHROPIC_MODEL=gpt-5.6 ANTHROPIC_SMALL_FAST_MODEL=gpt-5.6 CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 claude", then tmux send-keys "Reply with exactly the word pong and nothing else" Enter
2. Pane after the turn: Claude Code keeps sending its own cache_control markers, the configured injection point stands down as before, and the session works the same way

   │   gpt-5.6 with xhigh effort · API Usage Billing   │
   ❯ Reply with exactly the word pong and nothing else
   ⏺ pong
   ✻ Crunched for 4s
   

Both proxies ran on the default remote cost map (no LITELLM_LOCAL_MODEL_COST_MAP in either environment), which is what caught the first round: the published map has no supports_prompt_cache_breakpoint on gpt-5.6 yet, so the version-rule fallback in e16cf5e is what makes the After leg fire.

Closing observations from the run:

• Remote cost map lacks the new flag; version rule covers it
• Messages input_tokens now count the uncached tail; PR causes, correct
• Client prompt_cache_options on /v1/responses was dropped at base; PR fixes
• Claude Code's own cache_control still stands the point down; PR leaves alone
• Responses instructions string stays on implicit caching; PR leaves alone
• Spend logs record null cache_creation_input_tokens for OpenAI writes; PR leaves alone

Type

🆕 New Feature

Caveats (if any)

• Only provider openai is covered; azure keeps the Anthropic-style behavior for now. A GPT-5.6+ deployment pointed at a custom api_base (request api_base or base_url, litellm.api_base, or OPENAI_BASE_URL / OPENAI_API_BASE) keeps the previous behavior unless the deployment sets prompt_cache_options, which opts it into the dialect. litellm_proxy/ targets never get the dialect: the downstream LiteLLM that fronts OpenAI decides, and once it runs this change it produces the markers itself. Fine-tune ids (ft:gpt-5.6:...) are not in the model map and do not match the version rule, so they keep today's behavior.
• Assistant and tool_result blocks cannot carry a breakpoint on OpenAI, so the hook never marks an assistant message and skips turns with no eligible block; an injection point that only resolves to such turns is a no-op on OpenAI targets and prompt_cache_options is left unset. tool_config points are likewise a no-op there.
• Configured injection points still stand down when the client sends its own markers of either kind. Claude Code sends cache_control, so its requests through /v1/messages are not re-marked.
• On the chat path prompt_cache_options is an extra_body passthrough. A client prompt_cache_options nested inside its own extra_body is invisible to the hook and gets overwritten by the hook's explicit default during the extra_body merge; pass it top-level (or in the deployment's litellm_params) to control mode and ttl.
• On /v1/responses a top-level instructions string cannot carry a breakpoint, so an injection point only marks system, developer and user input items; a caller who wants the system prompt behind an explicit breakpoint sends it as an input item rather than as instructions.
• GPT-5.6 already serves prefix cache reads implicitly without any breakpoint, so this change is about control (explicit mode, placement, ttl through a deployment-level prompt_cache_options) rather than first-time cache hits.

Live PR risk (base 6fcdea0 vs head e16cf5e)

A three-proxy rig ran the same client flows against a base upstream and a head upstream, both fronting a downstream LiteLLM at base that fronts OpenAI, with drop_params off everywhere.

• Breaking: nothing found. Every flow returned 200 on both sides: litellm_proxy/gpt-5.6 through the downstream, a bare gpt-5.6 with api_base at the downstream, Gemini and xAI through that same chain, Claude on /v1/messages with five marked blocks, Bedrock tool calls, a gpt-5.6 deployment with a tool_config point, streaming chat, /model/info and /health/readiness
• Backward incompatible: a gpt-5.6 deployment that sets prompt_cache_options in litellm_params and points api_base at another gateway went from no caching at all at base (explicit options with no marker: 0 written and 0 read on all three calls) to the explicit pattern at head (6001 written, then 0 written and 6001 read through both hops). That is the opt-in the Limitations describe and the only flow outside the marked deployments whose numbers change. On /v1/messages the uncached tail now shows as input_tokens instead of being counted as a write
• Regression risk: the proxy-behind-proxy flows keep the implicit pattern at head exactly as at base (litellm_proxy/ and custom api_base never get the dialect), a client that sends its own cache_control on gpt-5.6 still stands the injection point down, a Responses request with the system prompt in instructions stays on implicit caching with no options forced, /model/info is identical between sides apart from ids, and the spend row is recorded on both
• Dependency graph: hook (anthropic_cache_control_hook.py) to the seed calls in completion(), acompletion() and litellm.responses(), the Responses merge in responses/utils.py, the OpenAI chat extra_body passthrough and the Anthropic messages adapter. The e16cf5e additions got their own walk: the version rule only runs when the map entry lacks the flag, base_url feeds the same gate api_base did, and the copy-on-reshape touches only the Responses merge output
• Not verified: Azure deployments (kept on the Anthropic-style behavior on purpose), fine-tune ids, a downstream LiteLLM that itself runs this change so both hops could mark, the Admin UI pages, and OpenAI-compatible providers other than xAI and Gemini through the chain

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

^{Cursor Bugbot is generating a summary for commit 5f6d22e. Configure here.}

• e16cf5e passes /live-pr-risk

[mateo-berri]

bugbot run

[greptile-apps]

Greptile Summary

The PR maps configured cache-control injection points to OpenAI explicit prompt-cache breakpoints for eligible GPT targets across chat completions, Responses, and Anthropic Messages bridges

• Adds model capability metadata and request-level prompt cache options
• Preserves markers while translating message and Responses content shapes
• Updates breakpoint counting, target detection, and provider bridge coverage

Confidence Score: 5/5

The PR appears safe to merge

No blocking failure remains

Important Files Changed

Filename	Overview
litellm/integrations/anthropic_cache_control_hook.py	Selects the OpenAI caching dialect, injects explicit markers, and coordinates breakpoint limits and request options
litellm/llms/anthropic/experimental_pass_through/responses_adapters/transformation.py	Preserves supported prompt-cache markers while converting Anthropic Messages payloads to Responses input
litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py	Carries supported prompt-cache markers through the Anthropic-to-chat-completions bridge
litellm/responses/utils.py	Retains prompt-cache options and reshapes marked chat-style content into valid Responses input parts
litellm/utils.py	Exposes prompt-cache-breakpoint capability through the shared model-support utility
model_prices_and_context_window.json	Declares explicit prompt-cache-breakpoint support for the targeted OpenAI GPT models

_{Reviews (3): Last reviewed commit: "Fall back to the GPT version rule when t..." | Re-trigger Greptile}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: OpenAI dialect reserves unused tool_config slot
  • Gated the tool_config reservation on not openai_dialect in both get_chat_completion_prompt and apply_to_anthropic_messages_request so OpenAI targets keep the full 4-block budget, and added a regression test asserting all 4 message markers are emitted alongside a tool_config point.

Or push these changes by commenting:

@cursor push bab8efa15b

Preview (bab8efa15b)

diff --git a/litellm/integrations/anthropic_cache_control_hook.py b/litellm/integrations/anthropic_cache_control_hook.py
--- a/litellm/integrations/anthropic_cache_control_hook.py
+++ b/litellm/integrations/anthropic_cache_control_hook.py
@@ -110,14 +110,16 @@
             else:
                 remaining_points.append(point)
 
+        openai_dialect: Final = AnthropicCacheControlHook._targets_openai_prompt_cache_breakpoint(
+            model, injection_points[0].get("_litellm_provider")
+        )
         # Non-message points (currently Bedrock tool_config) are handled in the
         # provider transform, where each tool_config point appends at most one
         # cachePoint to the tools. That block also counts toward Anthropic's
-        # limit, so reserve a slot for it here to leave room.
-        reserved_blocks: Final = 1 if any(p.get("location") == "tool_config" for p in remaining_points) else 0
-
-        openai_dialect: Final = AnthropicCacheControlHook._targets_openai_prompt_cache_breakpoint(
-            model, injection_points[0].get("_litellm_provider")
+        # limit, so reserve a slot for it here to leave room. OpenAI targets
+        # don't consume a tool_config breakpoint (no-op there), so no reserve.
+        reserved_blocks: Final = (
+            1 if not openai_dialect and any(p.get("location") == "tool_config" for p in remaining_points) else 0
         )
         breakpoints_before: Final = AnthropicCacheControlHook._count_request_cache_breakpoints(processed_messages)
         processed_messages = self._apply_message_injections(
@@ -363,7 +365,9 @@
             else:
                 remaining_points.append(point)
 
-        reserved_blocks: Final = 1 if any(p.get("location") == "tool_config" for p in remaining_points) else 0
+        reserved_blocks: Final = (
+            1 if not openai_dialect and any(p.get("location") == "tool_config" for p in remaining_points) else 0
+        )
         max_blocks: Final = MAX_CACHE_CONTROL_BLOCKS - reserved_blocks
 
         message_blocks: Final = AnthropicCacheControlHook._count_request_cache_breakpoints(processed_messages)

diff --git a/tests/test_litellm/integrations/test_anthropic_cache_control_hook.py b/tests/test_litellm/integrations/test_anthropic_cache_control_hook.py
--- a/tests/test_litellm/integrations/test_anthropic_cache_control_hook.py
+++ b/tests/test_litellm/integrations/test_anthropic_cache_control_hook.py
@@ -1281,6 +1281,48 @@
     ]
 
 
+def test_cache_control_hook_openai_dialect_does_not_reserve_tool_config_slot():
+    """OpenAI targets no-op on tool_config, so the slot must not be reserved.
+
+    A shared deployment config that lists 4 message points plus a tool_config
+    point (for the Bedrock target) must still emit 4 prompt_cache_breakpoint
+    markers on the OpenAI target: the tool_config point is unused there.
+    """
+    hook = AnthropicCacheControlHook()
+
+    messages: List[AllMessageValues] = [{"role": "user", "content": f"turn {i}"} for i in range(4)]
+
+    _, processed, non_default_params = hook.get_chat_completion_prompt(
+        model="openai/gpt-5.6",
+        messages=messages,
+        non_default_params={
+            "cache_control_injection_points": [
+                {"location": "message", "index": 0},
+                {"location": "message", "index": 1},
+                {"location": "message", "index": 2},
+                {"location": "message", "index": 3},
+                {"location": "tool_config"},
+            ]
+        },
+        prompt_id=None,
+        prompt_variables=None,
+        dynamic_callback_params={},
+    )
+
+    marker_count = sum(
+        1
+        for msg in processed
+        if isinstance(msg.get("content"), list)
+        for block in msg["content"]
+        if isinstance(block, dict) and block.get("prompt_cache_breakpoint") is not None
+    )
+    assert marker_count == 4
+    assert non_default_params["prompt_cache_options"] == {"mode": "explicit"}
+    assert non_default_params["cache_control_injection_points"] == [
+        {"location": "tool_config", "_litellm_judged": True}
+    ]
+
+
 @pytest.mark.asyncio
 async def test_cache_control_hook_bedrock_payload_caps_with_tool_config_point():
     """End-to-end: message + tool_config injection must not exceed 4 cachePoints."""

_{You can send follow-ups to the cloud agent here.}

[codecov]

Codecov Report

❌ Patch coverage is 97.29730% with 5 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...tellm/integrations/anthropic_cache_control_hook.py	97.56%	3 Missing ⚠️
litellm/responses/utils.py	90.00%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_lit5876_openai_prompt_cache_breakpoint (e16cf5e) with litellm_internal_staging (6d47468)}

[mateo-berri]

bugbot run

[mateo-berri]

@greptileai

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: base_url skipped by dialect gate
  • Passed the named base_url argument as a fallback for api_base when seeding cache injection points in both completion and acompletion, so custom hosts specified via the OpenAI-style base_url alias are now recognized by the dialect gate.

Or push these changes by commenting:

@cursor push c11c442cb1

Preview (c11c442cb1)

diff --git a/litellm/main.py b/litellm/main.py
--- a/litellm/main.py
+++ b/litellm/main.py
@@ -507,7 +507,7 @@
         custom_llm_provider=cast(str | None, custom_llm_provider),  # cast-ok: read from untyped kwargs
         tools=tools,
         enable_prompt_caching=cast(bool | None, kwargs.get("enable_prompt_caching")),  # cast-ok: untyped kwargs
-        api_base=kwargs.get("api_base"),
+        api_base=kwargs.get("api_base") or base_url,
     )
 
     if isinstance(litellm_logging_obj, LiteLLMLoggingObj) and (
@@ -5172,7 +5172,7 @@
         custom_llm_provider=cast(str | None, kwargs.get("custom_llm_provider")),  # cast-ok: untyped kwargs
         tools=tools,
         enable_prompt_caching=cast(bool | None, kwargs.get("enable_prompt_caching")),  # cast-ok: untyped kwargs
-        api_base=kwargs.get("api_base"),
+        api_base=kwargs.get("api_base") or base_url,
     )
 
     if isinstance(litellm_logging_obj, LiteLLMLoggingObj) and (

_{You can send follow-ups to the cloud agent here.}

[mateo-berri]

bugbot run

[mateo-berri]

@greptileai

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit e16cf5e. Configure here.}