feat(anthropic): add enable_anthropic_prompt_caching for automatic cache_control injection

[tin-berri]

Relevant issues

• The issue: Anthropic only caches a prompt when the request carries explicit cache_control breakpoints, unlike OpenAI where prompt caching is automatic. Clients such as Claude Code and Claude Desktop never set them, so Anthropic traffic through the proxy silently pays full price on every repeated prefix. The only way to get caching was for an admin to hand-write cache_control_injection_points into each model's litellm_params, a recipe that is easy to miss and that end users cannot apply themselves
• The fix: an opt-in enable_anthropic_prompt_caching flag, settable from a config file or an environment variable (and from the Admin UI in feat(ui): configure Anthropic automatic prompt caching from the Admin UI #33581). When it is on, litellm injects a default pair of breakpoints covering the system prompt and the trailing turn for Anthropic and Bedrock Claude models, and stands down entirely when the request already carries its own cache_control. Default is off, so no existing deployment changes behavior

Linear ticket

Resolves LIT-4478

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live proxy on localhost:4000 against the real Anthropic API, one Anthropic model and a system prompt of roughly 7.9k tokens, sent twice so the second call can hit the cache

model_list:
  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY

general_settings:
  master_key: sk-1234

litellm_settings:
  enable_anthropic_prompt_caching: false   # flipped to true for the "after" run

Before (enable_anthropic_prompt_caching: false, today's default behavior)

Every call re-reads the same 7940 token prefix at full price and nothing is ever cached, on either surface

curl -s http://localhost:4000/v1/chat/completions -H "Authorization: Bearer sk-1234" \
  -H "Content-Type: application/json" -d @cc.json | jq -c '.usage | {prompt_tokens, cache_creation:.cache_creation_input_tokens, cache_read:.cache_read_input_tokens}'
{"prompt_tokens":7940,"cache_creation":0,"cache_read":0}   # call 1
{"prompt_tokens":7940,"cache_creation":0,"cache_read":0}   # call 2

curl -s http://localhost:4000/v1/messages -H "x-api-key: sk-1234" -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" -d @msg.json | jq -c '.usage'
{"input_tokens":7940,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,...}   # call 1
{"input_tokens":7940,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,...}   # call 2

After (enable_anthropic_prompt_caching: true)

/v1/chat/completions writes the cache on the first call and reads it back on the second

curl -s http://localhost:4000/v1/chat/completions -H "Authorization: Bearer sk-1234" \
  -H "Content-Type: application/json" -d @cc.json | jq -c '.usage | {prompt_tokens, cache_creation:.cache_creation_input_tokens, cache_read:.cache_read_input_tokens, cached:.prompt_tokens_details.cached_tokens}'
{"prompt_tokens":7940,"cache_creation":7938,"cache_read":0,"cached":0}      # call 1, cache written
{"prompt_tokens":7940,"cache_creation":0,"cache_read":7938,"cached":7938}   # call 2, cache hit

/v1/messages (the surface Claude Code and Claude Desktop use) against a cold prefix, same result

curl -s http://localhost:4000/v1/messages -H "x-api-key: sk-1234" -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" -d @msg2.json | jq -c '.usage | {input_tokens, cache_creation_input_tokens, cache_read_input_tokens}'
{"input_tokens":2,"cache_creation_input_tokens":7677,"cache_read_input_tokens":0}   # call 1, cache written
{"input_tokens":2,"cache_creation_input_tokens":0,"cache_read_input_tokens":7677}   # call 2, cache hit

When the client sends its own cache_control, the flag stands down and the client's breakpoint drives the cache, so nothing is double-injected

curl -s http://localhost:4000/v1/messages -H "x-api-key: sk-1234" -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" -d @msg_client_cc.json | jq -c '.usage | {input_tokens, cache_creation_input_tokens, cache_read_input_tokens}'
{"input_tokens":12,"cache_creation_input_tokens":0,"cache_read_input_tokens":7667}

Spend, straight from the spend logs

The same rows show why this matters, and confirm the cache tokens reach the spend log for both surfaces, which is what the Logs UI request drawer already renders

call_type          | prompt_tokens | cache_creation | cache_read | spend
anthropic_messages | 7679          | 0              | 7667       | 0.001597
anthropic_messages | 7679          | 0              | 7677       | 0.001579
anthropic_messages | 7679          | 7677           | 0          | 0.019237
acompletion        | 7940          | 0              | 7938       | 0.001632
acompletion        | 7940          | 7938           | 0          | 0.019889
anthropic_messages | 7940          | 0              | 0          | 0.015920   <- flag off, no caching

An uncached call costs 0.015920, a cache write costs 0.019889 (the documented 1.25x premium for the 5 minute cache) and every subsequent cache read costs 0.001632, roughly 90% less. The flag pays for itself after a single repeat

Environment variables

The flag can also be turned on without a config file, for deployments that template the environment instead. Same config as above with litellm_settings omitted entirely, so the environment is the only thing that can enable caching

Anthropic was intermittently returning overloaded_error while this was captured and was not serving cache reads for any request, including ones sent straight to api.anthropic.com with client-supplied cache_control and litellm entirely out of the loop, so these runs show the write side only. The read side is covered by the runs above

# env var unset, nothing else changed
curl -s http://localhost:4000/v1/messages -H "x-api-key: sk-1234" -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" -d @msg.json | jq -c '.usage | {input_tokens, cache_creation_input_tokens, cache_read_input_tokens}'
{"input_tokens":13669,"cache_creation_input_tokens":0,"cache_read_input_tokens":0}   # no caching, full price

# LITELLM_ENABLE_ANTHROPIC_PROMPT_CACHING=true, same config file, nothing else changed
{"input_tokens":2,"cache_creation_input_tokens":13667,"cache_read_input_tokens":0}   # cache written

litellm_settings still wins when both are set, since the config is applied after import

# LITELLM_ENABLE_ANTHROPIC_PROMPT_CACHING=false, with enable_anthropic_prompt_caching: true in the config
{"input_tokens":2,"cache_creation_input_tokens":13667}   # config wins, caching stays on

Type

🆕 New Feature

Changes

Anthropic only caches a prompt when the request carries explicit cache_control breakpoints, unlike OpenAI where prompt caching is automatic and needs no configuration. litellm can already inject those breakpoints server-side, but only when an admin hand-writes cache_control_injection_points into a model's litellm_params or into router_settings.default_litellm_params. Clients such as Claude Code and Claude Desktop never set cache_control themselves and the admin recipe is easy to miss, so Anthropic traffic through the proxy silently pays full price on every repeated prefix

This adds an opt-in litellm_settings flag, enable_anthropic_prompt_caching. When it is on, and the request has no injection points configured and no client-supplied cache_control, litellm synthesizes a default pair of breakpoints covering the system prompt and the trailing turn, so the stable prefix stays cached while the breakpoint advances with the conversation. Both surfaces are wired to one helper: /chat/completions seeds the points just before the existing prompt-management gate, and /v1/messages resolves them inside maybe_inject_cache_control. The existing AnthropicCacheControlHook then applies them unchanged, so the four-block cap and its refusal to overwrite client breakpoints both still hold

The default is off, so no existing deployment changes behavior. Injection is gated to the providers that actually consume cache_control markers, meaning anthropic and bedrock, and to models the cost map flags as supporting prompt caching. supports_prompt_caching on its own is not a sufficient gate: OpenAI, Azure and Gemini report it too, but they do not treat cache_control the way Anthropic does. A real openai.com host strips the marker, so injecting there is a harmless no-op, while Vertex and Google AI Studio Gemini consume it and convert it into Gemini context caching, which is a different mechanism with its own semantics and cost profile. Gating on that field alone would therefore silently switch on another provider's caching behind a flag named for Anthropic, which is why the provider check sits alongside it. Claude on vertex_ai is deliberately left out for now, because Gemini on Vertex reports the same capability while using a different caching mechanism, and separating them needs a model-level check worth doing on its own

The default ttl is Anthropic's 5 minute ephemeral cache, which is both their API default and what Claude Code uses, with an optional anthropic_prompt_caching_ttl of "5m" or "1h" for long agentic sessions. 1h doubles the write premium instead of the 1.25x above, so it is opt-in rather than the default. ttl is also added to ChatCompletionCachedContent, a field the bedrock and anthropic transforms already read at runtime but the type never declared

Both settings can also be set from the environment, with LITELLM_ENABLE_ANTHROPIC_PROMPT_CACHING and LITELLM_ANTHROPIC_PROMPT_CACHING_TTL, so a deployment that templates env vars rather than a config file can turn this on without one. litellm_settings wins over the environment when both are present, because the config is applied after import. An unsupported ttl value falls back to the provider default instead of reaching the provider verbatim

A note on cache sharing

Worth stating explicitly, since this flag makes it apply to everyone rather than only to callers who opted in themselves.

Provider prompt caching is not per end user. The provider caches a prefix against the upstream credentials that sent it, and any request repeating that prefix exactly reuses it. That sharing is the point, and it is what makes this worth turning on: a long system prompt cached by one user is reused by everyone else on the same credentials. The consequence is that a caller who reproduces a prefix exactly can tell that someone else on those credentials sent it recently, because the response reports cache_read_input_tokens and a cache hit is also faster.

None of that is new behavior. It is how Anthropic prompt caching works, it applies today to anyone who sets cache_control themselves or configures cache_control_injection_points, and hiding the usage fields would not change it because the latency difference carries the same signal. What this flag changes is the population: with it off, a client that never sends cache_control has nothing cached, so nothing about it is observable; with it on, every request through the gateway is cached and therefore observable. The exposure is bounded by the fact that a prefix must be reproduced exactly and providers will not cache a prefix below a minimum size (model-dependent for Anthropic, currently 1k to 4k tokens), so it reveals whether a prompt the caller already holds was recently sent, rather than revealing its contents.

The isolation boundary is the provider account, which is not something the gateway can synthesize; separate credentials per tenant is the only real answer. So this stays an operator decision, and the flag is off by default. The setting description and the env var docs both state it, and the auto-inject checkpoints doc now carries a section on it, since the behavior predates this PR and was not documented anywhere.

QA runbook

1. Point a config at any Anthropic or Bedrock Claude model and leave enable_anthropic_prompt_caching unset. Send the same large prompt (it must clear the provider's minimum cacheable prefix, up to 4k tokens on current Anthropic models, or nothing will cache) twice to /v1/chat/completions and to /v1/messages, and confirm cache_creation_input_tokens and cache_read_input_tokens stay at 0
2. Set enable_anthropic_prompt_caching: true under litellm_settings, restart, and repeat. The first call should report cache_creation_input_tokens greater than 0 and the second should report cache_read_input_tokens greater than 0
3. Send a request that already carries its own cache_control and confirm the breakpoint count does not grow and the client's own TTL is preserved
4. Send traffic to an OpenAI or Gemini model with the flag on and confirm no cache_control appears in the outgoing request
5. Set anthropic_prompt_caching_ttl: "1h" and confirm cache_creation.ephemeral_1h_input_tokens is what moves
6. Open http://localhost:4000/ui/?page=logs, click one of the requests from step 2, and confirm the Metrics card shows Cache Read Tokens and Cache Creation Tokens
7. Remove the flag from the config, restart with LITELLM_ENABLE_ANTHROPIC_PROMPT_CACHING=true in the environment, and confirm caching engages exactly as in step 2

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Medium Risk
Changes outbound request shaping and billing-related caching behavior for Anthropic/Bedrock when enabled; mitigated by default-off, provider/model gating, and stand-down when clients supply cache_control.

Overview
Adds an opt-in enable_anthropic_prompt_caching setting (config, env LITELLM_ENABLE_ANTHROPIC_PROMPT_CACHING, default off) plus optional anthropic_prompt_caching_ttl (5m / 1h) so LiteLLM can inject Anthropic cache_control breakpoints without per-model cache_control_injection_points.

When enabled, AnthropicCacheControlHook synthesizes two default breakpoints (system + last message) for anthropic and bedrock models that support prompt caching, and stands down if the client already set cache_control on messages, system, or tools. /chat/completions seeds cache_control_injection_points before prompt-management hooks; /v1/messages resolves defaults inside maybe_inject_cache_control. ChatCompletionCachedContent gains an optional ttl field; tests cover gating, tool stand-down, and env parsing.

^{Reviewed by Cursor Bugbot for commit 53c285a. Bugbot is set up for automated code reviews on this repo. Configure here.}

[codecov]

Codecov Report

❌ Patch coverage is 92.18750% with 5 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...tellm/integrations/anthropic_cache_control_hook.py	90.00%	5 Missing ⚠️

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR adds enable_anthropic_prompt_caching, an opt-in flag (default off) that automatically injects cache_control breakpoints on the system prompt and trailing message for Anthropic and Bedrock Claude requests, so callers like Claude Code and Claude Desktop get prompt caching without hand-crafting per-model config. It is gated by provider (anthropic/bedrock only), model capability (supports_prompt_caching), and stands down if the request already carries its own cache_control — so the existing four-block cap and client-supplied breakpoints are never overridden.

• New module-level flags in __init__.py (enable_anthropic_prompt_caching, anthropic_prompt_caching_ttl) with both litellm_settings and env-var precedence; config always wins over env.
• New static methods on AnthropicCacheControlHook: get_default_injection_points, _request_has_cache_control, maybe_seed_default_injection_points for the /chat/completions path, and extended maybe_inject_cache_control for the /v1/messages path.
• ChatCompletionCachedContent gains the previously-undeclared ttl: NotRequired[Literal["5m", "1h"]] field, now read and handled by the existing Bedrock transforms and passed verbatim on the Anthropic direct path.

Confidence Score: 5/5

Safe to merge. The flag is off by default, so no existing deployment changes behavior. All injection logic is gated behind multiple explicit conditions and the existing four-block cap remains enforced by the upstream hook.

The implementation is narrow and well-scoped: provider and model-capability checks prevent injection outside the intended target, the stand-down path when client-supplied cache_control is present is exercised by tests, and the double-seeding path through acompletion → completion is a no-op due to the guard. No pre-existing tests were modified, and all new tests are pure-mock or subprocess-only with no network calls.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/init.py	Adds two new module-level attributes: enable_anthropic_prompt_caching (bool, read from env) and anthropic_prompt_caching_ttl (Literal["5m","1h"]
litellm/integrations/anthropic_cache_control_hook.py	Adds four new static methods for default injection point management. Logic is sound: provider gating, supports_prompt_caching check, client cache_control detection (messages + system list + tools), and a no-op guard when explicit points are already configured all work correctly.
litellm/main.py	Seeds cache_control_injection_points early in both acompletion and completion before the prompt-management gate. The seeding in acompletion flows through **kwargs into completion; the second seeding call in completion is a no-op because the guard checks non_default_params.get("cache_control_injection_points").
litellm/llms/anthropic/experimental_pass_through/messages/handler.py	Passes model, custom_llm_provider, and tools to the updated maybe_inject_cache_control signature — minimal, targeted change that enables auto-injection on the /v1/messages path.
litellm/types/llms/openai.py	Adds ttl: NotRequired[Literal["5m", "1h"]] to ChatCompletionCachedContent. The existing Bedrock transforms already read this field at runtime; this PR just adds the missing type annotation.
tests/test_litellm/integrations/test_anthropic_cache_control_hook.py	Adds comprehensive mock-only tests. TestAnthropicPromptCachingEnvVars uses subprocess to re-import litellm with different env vars — no real network calls made. Coverage is good: default-off, bedrock, non-Anthropic gate, tool-caching stand-down, TTL override, and both API paths.
ui/litellm-dashboard/src/lib/http/schema.d.ts	Auto-generated OpenAPI schema file updated to reflect the new ttl field on ChatCompletionCachedContent.

_{Reviews (3): Last reviewed commit: "fix(anthropic): stand down when the clie..." | Re-trigger Greptile}

[veria-ai]

Medium: Cross-tenant prompt membership oracle

This breakpoint caches the complete request-controlled conversation in the upstream workspace/account cache. Because LiteLLM returns cache_read_input_tokens and cache_creation_input_tokens, an authenticated user sharing those provider credentials can submit an exact candidate prefix and determine whether another tenant recently submitted it. Limit automatic caching to operator-controlled static prefixes, or only enable the trailing-turn breakpoint when upstream credentials and caches are tenant-scoped.

[tin-berri]

Good catch, and the exposure change is real, so I have documented it at the point of decision. One correction on the suggested mitigation though

Confirmed the flag broadens exposure. With it off, a client that never sends cache_control has nothing cached, so there is nothing to probe. With it on, the same request caches, which is what makes the prefix probeable:

flag off -> cache_creation_input_tokens 0
flag on  -> cache_creation_input_tokens 13667

The suggested mitigation does not close it here. Limiting injection to operator-controlled static prefixes assumes the system prompt is operator-controlled, but system is a client-supplied request parameter on /v1/messages (handler.py:204), and Claude Code supplies its own. So a system-only breakpoint would still cache client-controlled content on the shared account while removing most of the feature value

The underlying property is Anthropic prompt caching itself: a prefix is cached against the upstream credentials that sent it, not per end user. Any caller can already self-serve a cache_control breakpoint in their own request body, so the gateway cannot make the cache tenant-scoped; only issuing per-tenant upstream credentials does that

Given that, this stays an operator decision, which is why the flag is opt-in and off by default. What was missing was surfacing the tradeoff where the decision is made, so the setting description and the env var docs now state that the provider caches against the upstream credentials rather than per end user, and to leave it off if callers sharing credentials must not learn whether another caller recently sent a given prompt

[veria-ai]

PR overview

This PR adds an Anthropic prompt-caching integration that can automatically inject cache_control breakpoints into messages when enable_anthropic_prompt_caching is enabled. The changed hook logic decides where to place cache controls on request conversations for Anthropic-compatible calls.

There is one open security concern around automatic caching of request-controlled conversation content when multiple tenants share the same upstream Anthropic credentials or cache namespace. In that configuration, an authenticated user could use returned cache token counters to test whether another tenant recently submitted an exact prompt prefix. No issues have been addressed yet, so the PR still needs a scoping or restriction change before this behavior is safe for shared-provider deployments.

Open issues (1)

• Medium: Cross-tenant prompt membership oracle — litellm/integrations/anthropic_cache_control_hook.py:372

Fixed/addressed: 0 · PR risk: 6/10

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_lit4478_anthropic_auto_cache (7bcb3a2) with litellm_internal_staging (561b679)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (0e88b57) during the generation of this report, so 561b679 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[tin-berri]

@greptileai rereview

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

[tin-berri]

bugbot run

[tin-berri]

bugbot run

[tin-berri]

@greptileai rereview

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 53c285a. Configure here.}

[ryan-crabbe-berri]

lgtm