fix(anthropic): keep cache_control for Gemini targets on /v1/messages and normalize Anthropic ttl units

[devin-ai-integration]

Supersedes #32366 at its head 2303379, moved from the fork branch to an internal litellm_ branch so CircleCI runs the llm_translation and local_testing suites; the contributor's commits and authorship are carried over unchanged.

TLDR

Problem this solves:

• Claude Code on a Gemini model through /v1/messages never gets prompt caching
• The /v1/messages bridge dropped cache_control for every non-Claude target
• Anthropic clients send the ttl as 1h or 5m; Gemini only accepts 3600s
• Gemini 2.5 Pro returns HTTP 400 for cached blocks under 2048 tokens

How it solves it:

• /v1/messages keeps cache_control when the target model is a Gemini model
• Anthropic ttl units are converted to Gemini's <seconds>s form
• The cost map carries 2.5 Pro's 2048 minimum, so shorter prompts skip the cache
• A ttl whose expiry Google cannot store (past the year 9999) falls back to Google's default hour instead of a 400

User Flow

Before: a developer running Claude Code against a Gemini model through the gateway pays for the whole system prompt on every turn, and a short cached block on Gemini 2.5 Pro fails outright

1. They point Claude Code at the gateway with ANTHROPIC_BASE_URL=https://litellm-domain and ANTHROPIC_MODEL=gemini-25-flash, then ask a question
2. Claude Code sends POST https://litellm-domain/v1/messages with its system prompt marked "cache_control": {"type": "ephemeral"}
3. The reply's usage counts every prompt token as input_tokens and carries no cache_read_input_tokens, and the next turn re-sends the whole system prompt at full price
4. They run /cost and see 0 cache read, 0 cache write and "no prompt caching reported by the API"
5. Their own script sends the same kind of cached block with "ttl": "5m" to POST https://litellm-domain/v1/messages and gets the same uncached usage back; GET https://generativelanguage.googleapis.com/v1beta/cachedContents on their Gemini key lists no cache for it
6. A teammate's app sends a 1507-token system block marked cache_control to Gemini 2.5 Pro through POST https://litellm-domain/v1/chat/completions or POST https://litellm-domain/v1/responses
7. The gateway answers HTTP 400 Cached content is too small. total_token_count=1507, min_total_token_count=2048

After: the same turns read the system prompt from Gemini's cache, the ttl lands as sent, and the short block on 2.5 Pro goes through as a plain request

1. They point Claude Code at the gateway with ANTHROPIC_BASE_URL=https://litellm-domain and ANTHROPIC_MODEL=gemini-25-flash, then ask a question
2. Claude Code sends POST https://litellm-domain/v1/messages with its system prompt marked "cache_control": {"type": "ephemeral"}
3. The reply's usage now carries the system prompt as cache_read_input_tokens and only the new text as input_tokens
4. They run /cost and see 28.4k cache read and "46% of input tokens from cache", at a lower dollar figure than before
5. Their own script's "ttl": "5m" request comes back with cache_read_input_tokens, and GET https://generativelanguage.googleapis.com/v1beta/cachedContents lists a cache whose expireTime is five minutes after its createTime
6. A teammate's app sends the same 1507-token block to Gemini 2.5 Pro through POST https://litellm-domain/v1/chat/completions or POST https://litellm-domain/v1/responses
7. The gateway answers HTTP 200 with a normal reply and no cache fields in usage

Relevant issues

Ref #29810, Ref #29166

Affected release

Linear ticket

Resolves LIT-5373

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Both legs ran the same way and differ only in the commit the proxy was booted from. Each proxy ran python litellm/proxy/proxy_cli.py --config config.yaml --num_workers 2 --detailed_debug (2 uvicorn workers, no Redis, no database) with LITELLM_LOCAL_MODEL_COST_MAP=True so each side reads the cost map at its own commit. Before is the merge base 5611607 on port 44911, After is the tip 5ad1847 on port 43175. Every request went to Google's real Gemini API through the gemini/ provider and cost real money

model_list:
  - model_name: gemini-flash
    litellm_params:
      model: gemini/gemini-3.8-flash
      api_key: os.environ/GEMINI_API_KEY
  - model_name: gemini-25-flash
    litellm_params:
      model: gemini/gemini-2.5-flash
      api_key: os.environ/GEMINI_API_KEY
  - model_name: gemini-2.5-pro
    litellm_params:
      model: gemini/gemini-2.5-pro
      api_key: os.environ/GEMINI_API_KEY

general_settings:
  master_key: sk-lit5373-qa

Payloads. The /v1/messages ones carry a 27,823-character generated handbook as the system block (about 5.2k tokens, above every Gemini minimum); messages-ttl-5m.json is the same file with "ttl": "5m". The 2.5 Pro ones carry an 8k-character slice of it (1507 tokens, under 2.5 Pro's 2048 minimum)

// messages-ttl-1h.json
{"model": "gemini-flash", "max_tokens": 256,
 "system": [{"type": "text", "text": "<27,823-char handbook text>", "cache_control": {"type": "ephemeral", "ttl": "1h"}}],
 "messages": [{"role": "user", "content": "In one sentence, what is this document about?"}]}

// chat-25pro.json
{"model": "gemini-2.5-pro", "max_tokens": 2048,
 "messages": [{"role": "system", "content": [{"type": "text", "text": "<8,037-char handbook slice>", "cache_control": {"type": "ephemeral"}}]},
              {"role": "user", "content": "In one sentence, what is this document about?"}]}

// responses-25pro.json
{"model": "gemini-2.5-pro", "max_output_tokens": 2048,
 "input": [{"role": "system", "content": [{"type": "input_text", "text": "<8,042-char handbook slice>", "cache_control": {"type": "ephemeral"}}]},
           {"role": "user", "content": "In one sentence, what is this document about?"}]}

Claude Code v2.1.278 was launched interactively in tmux against each proxy with env -u ANTHROPIC_API_KEY ANTHROPIC_BASE_URL=http://localhost:<port> ANTHROPIC_AUTH_TOKEN=sk-lit5373-qa ANTHROPIC_MODEL=gemini-25-flash ANTHROPIC_SMALL_FAST_MODEL=gemini-25-flash claude, given the same two prompts (Reply with one word: hello, then world), then /cost

Before (5611607)

/v1/messages

1. for i in 1 2; do curl -s http://localhost:44911/v1/messages -H "Authorization: Bearer sk-lit5373-qa" -H "Content-Type: application/json" -d @messages-ttl-1h.json | jq -c .usage; done
2. {"input_tokens":5213,"output_tokens":167} then {"input_tokens":5213,"output_tokens":130}: the whole system block is billed as input both times, no cache fields
3. Same loop with @messages-ttl-5m.json
4. {"input_tokens":5213,"output_tokens":252} twice, no cache fields

/v1/chat/completions

1. curl -s -w "\n%{http_code}\n" http://localhost:44911/v1/chat/completions -H "Authorization: Bearer sk-lit5373-qa" -H "Content-Type: application/json" -d @chat-25pro.json
2. HTTP 400: {"error": {"message": "litellm.BadRequestError: GeminiException BadRequestError - {\n \"error\": {\n \"code\": 400,\n \"message\": \"Cached content is too small. total_token_count=1507, min_total_token_count=2048\",\n \"status\": \"INVALID_ARGUMENT\"\n }\n}\n. Received Model Group=gemini-2.5-pro\nAvailable Model Group Fallbacks=None", "type": "invalid_request_error", "param": null, "code": "400"}}

/v1/responses

1. curl -s -w "\n%{http_code}\n" http://localhost:44911/v1/responses -H "Authorization: Bearer sk-lit5373-qa" -H "Content-Type: application/json" -d @responses-25pro.json
2. HTTP 400 with the same body: "Cached content is too small. total_token_count=1507, min_total_token_count=2048"

Claude Code

1. Launch Claude Code against port 44911 as above, send Reply with one word: hello, then world
2. Both turns answer, and the pane shows nothing about caching (captured during the earlier run against this same merge-base proxy, which has not been restarted since)

3. /cost reads gemini-25-flash: 67.2k input, 257 output, 0 cache read, 0 cache write ($0.3426) and 0% of input tokens from cache · no prompt caching reported by the API

After (5ad1847)

/v1/messages

1. for i in 1 2; do curl -s http://localhost:43175/v1/messages -H "Authorization: Bearer sk-lit5373-qa" -H "Content-Type: application/json" -d @messages-ttl-1h.json | jq -c .usage; done
2. {"input_tokens":11,"output_tokens":252,"cache_read_input_tokens":5204} twice: the system block is read from Gemini's cache, only the question is billed as input
3. Same loop with @messages-ttl-5m.json
4. {"input_tokens":11,"output_tokens":252,"cache_read_input_tokens":5204} then {"input_tokens":11,"output_tokens":227,"cache_read_input_tokens":5204}
5. curl -s https://generativelanguage.googleapis.com/v1beta/cachedContents -H "x-goog-api-key: $GEMINI_API_KEY" | jq -c '.cachedContents[] | {name, model, createTime, expireTime, tokens: .usageMetadata.totalTokenCount}'
6. The 1h request's cache expires an hour after creation and the 5m one five minutes after, so the Anthropic units reached Google as 3600s and 300s:
   {"name":"cachedContents/f0rkj0ax8ud3x83y1ze6pbi2gyuuygtgspdmah81","model":"models/gemini-3.8-flash","createTime":"2026-09-19T08:01:58.179908Z","expireTime":"2026-09-19T09:01:57.954397289Z","tokens":5204}
{"name":"cachedContents/l0w797dy9t2zwojatn3a42bwp4bfreek2q22hymh","model":"models/gemini-3.8-flash","createTime":"2026-09-19T08:02:04.463005Z","expireTime":"2026-09-19T08:07:04.174079086Z","tokens":5204}
{"name":"cachedContents/11gmbsj5b0oqdiohb9dmlxwus1zcw7ikm4bqnrhd","model":"models/gemini-2.5-flash","createTime":"2026-09-19T08:01:53.398979Z","expireTime":"2026-09-19T09:01:52.997352359Z","tokens":14216} (Claude Code's own cache from the case below; it sends no ttl, so Google's default hour applies)
7. Twelve more /v1/messages requests carried a ttl the proxy cannot forward (5, ["1h"], "", 5,000 x characters, 0h, -5m, 0.0000000001s, a 400-digit hour count, 87660001h, 315576000000s, 251700000000s) or one at the edge (1.5h, 999999h, 61320000h): every one answered HTTP 200 with cache_read_input_tokens and the unforwardable ones got Google's default hour, while the edge ones landed as sent (5400s, a cache expiring in 2140, a cache expiring in 9022; the long-lived ones were deleted afterwards)

/v1/chat/completions

1. curl -s -w "\n%{http_code}\n" http://localhost:43175/v1/chat/completions -H "Authorization: Bearer sk-lit5373-qa" -H "Content-Type: application/json" -d @chat-25pro.json
2. HTTP 200, "finish_reason": "stop", content This operations handbook explains how to configure various gateway features like billing, routing, and caching, detailing how each one evaluates requests, handles failures, and reports metrics.
3. usage: {"completion_tokens": 799, "prompt_tokens": 1516, "total_tokens": 2315, "completion_tokens_details": {"reasoning_tokens": 767, "text_tokens": 32}, "prompt_tokens_details": {"text_tokens": 1516}}, no cache fields because the block is under 2.5 Pro's minimum and was sent uncached

/v1/responses

1. curl -s -w "\n%{http_code}\n" http://localhost:43175/v1/responses -H "Authorization: Bearer sk-lit5373-qa" -H "Content-Type: application/json" -d @responses-25pro.json
2. HTTP 200, "status": "completed", output text This operations handbook explains how various gateway features like billing, routing, and caching are configured, how they process requests, and how they report errors and metrics.
3. usage: {"input_tokens": 1516, "input_tokens_details": {"cached_tokens": 0, "text_tokens": 1516}, "output_tokens": 1318, "output_tokens_details": {"reasoning_tokens": 1287, "text_tokens": 31}, "total_tokens": 2834}

Claude Code

1. Launch Claude Code against port 43175 as above, send Reply with one word: hello, then world
2. Both turns answer the same way

3. /cost reads gemini-25-flash: 34.5k input, 223 output, 28.4k cache read, 0 cache write ($0.1923) and Prompt cache (main): 2 requests · 46% of input tokens from cache

Observations from the run, none caused by this PR:

• First cached call reports cache_read_input_tokens, never cache_creation_input_tokens; pre-existing
• Gemini 3.x Flash models keep main's 4096-token cache floor; left alone
• Claude Code's bare system prompt is about 1.5k tokens, under that floor
• The below-minimum debug log names 1024, not the model's floor; pre-existing

Type

🐛 Bug Fix

Caveats (if any)

Low

• The only upper bound on the ttl is Google's own, so 48h reaches Google as 172800s
  • Google rejects any ttl whose expiry passes 9999-12-31 (about 251 billion seconds today, computed from the clock at request time), so such a ttl is dropped and the cache gets Google's default hour; Google bills cache storage per hour, but anything under that bound is what the caller asked for, and the <seconds>s form was never capped before either
• The 2048 minimum is set on Gemini 2.5 Pro only
  • Google's docs list higher Flash minimums, but the live API caches 2.5 Flash and 3.8 Flash at about 1132 tokens, so those keep the 1024 default
• main's 4096 prompt_cache_min_tokens on Gemini 3.x models makes the proxy skip 1024-4095-token blocks Google would cache
  • Left alone: lowering it needs a per-model probe against the live API
• The Gemini check on /v1/messages is by model name
  • An alias containing gemini that routes elsewhere carries cache_control through, the same way claude aliases already do
• The Responses API half of the original PR is already on main via fix(responses): preserve cache_control in Responses API -> Chat Completion transformation #27727, so this PR carries only the Gemini pieces
• The original object and Pydantic normalization helper was dropped as unneeded on main
• The first cached call reports a cache read rather than a cache creation, and the below-minimum debug line names the 1024 default instead of the model's floor; both pre-existing and untouched
• ci/circleci: litellm_utils_testing is red at the tip on test_models_by_provider (transcribe missing from models_by_provider), a test and module this PR does not touch
  • main's own pipelines 89818 and 89834, both before this branch's merge base, fail the same assertion; fix(proxy): register transcribe as a known provider for model grants #41926 carries the fix, and the job is not a required check on main
• ci/circleci: proxy_e2e_anthropic_messages_tests is red at the tip on test_bedrock_invoke_messages_with_all_beta_headers for both Bedrock params, Bedrock answering invalid beta flag
  • main pipeline 89834, before the merge base, fails the same two params; test(e2e): stop sending model-scoped anthropic betas at models without them #41927 carries the fix, and the job is not a required check on main. Both model names contain claude, so the /v1/messages gate this PR changes answers the same as before for them
• The bare gemini-2.5-pro cost map key carries deprecation_date 2026-10-20, so its new 2048 minimum matters only until then; gemini/gemini-2.5-pro has no deprecation date
• docs.litellm.ai documents only the <seconds>s ttl form for Gemini context caching (docs/providers/gemini.md, "TTL Format Requirements"), so the m and h forms need a litellm-docs follow-up
• Cache-aware targets whose names carry none of claude, a Bedrock ARN, or gemini (Bedrock Nova model ids, which Converse caches through cachePoint) still get cache_control stripped on /v1/messages; pre-existing and untouched

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR
• 5ad1847 passes /live-pr-risk

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_gemini_cache_control_messages (5ad1847) with main (5f1268c)}

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge; no outstanding correctness, security, or repository-rule issue remains.

Findings

1.  Hardcoded Model Capability ▶

Summary

This PR preserves Anthropic cache_control metadata when /v1/messages targets Gemini, normalizes supported Anthropic TTL units into Gemini’s seconds format, and configures Gemini 2.5 Pro’s 2,048-token prompt-cache minimum.

• Routes Gemini cache metadata through the Anthropic messages adapter.
• Converts positive s, m, and h TTL values while rejecting malformed or unrepresentable expirations.
• Prevents undersized Gemini 2.5 Pro cache creation attempts.
• Adds focused unit coverage for routing, normalization boundaries, and minimum-token behavior.

_{Reviews (3) · Last reviewed commit: "fix(gemini): drop ttl values whose expir..."}

[greptile-apps]

Hardcoded Model Capability

Gemini support is inferred from the model name, violating the requirement to store model-specific flags in metadata before merging.

Rule Used: What: Do not hardcode model-specific flags in the codebase. Instead, put them in model_prices_and_context_window.json and then read them in via get_model_info Why: Prevents need for users to upgrade litellm each time a new model supports this featu... (source)

[mateo-berri]

Provider capability, not a model flag: the Gemini transformation consumes cache_control, the cost map has no such field, and the claude check matches by name

[greptile-apps]

You're right — my earlier finding does not apply here. target_consumes_cache_control() determines whether the Anthropic bridge should preserve a target-specific request field; it is not a model capability flag used for pricing, limits, or cost-map behavior. The existing Claude and Bedrock checks already use model identity for this routing decision, so adding Gemini consistently extends that target detection. I’m withdrawing this finding.

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 5ad1847. Configure here.}

[mateo-berri]

LGTM