fix(anthropic): keep cache_control for Gemini targets on /v1/messages and normalize Anthropic ttl units

[edelauna]

TLDR

Problem this solves:

• Claude Code on a Gemini model through /v1/messages never gets prompt caching
• The /v1/messages bridge dropped cache_control for every non-Claude target
• Anthropic clients send the ttl as 1h or 5m; Gemini only accepts 3600s
• Gemini 2.5 Pro returns HTTP 400 for cached blocks under 2048 tokens

How it solves it:

• /v1/messages keeps cache_control when the target model is a Gemini model
• Anthropic ttl units are converted to Gemini's <seconds>s form
• The cost map carries 2.5 Pro's 2048 minimum, so shorter prompts skip the cache

User Flow

Before: a developer running Claude Code against a Gemini model through the gateway pays for the whole system prompt on every turn, and a short cached block on Gemini 2.5 Pro fails outright

1. They point Claude Code at the gateway with ANTHROPIC_BASE_URL=https://litellm-domain and ANTHROPIC_MODEL=gemini-25-flash, then ask a question
2. Claude Code sends POST https://litellm-domain/v1/messages with its system prompt marked "cache_control": {"type": "ephemeral"}
3. The reply's usage counts every prompt token as input_tokens and carries no cache_read_input_tokens, and the next turn re-sends the whole system prompt at full price
4. They run /cost and see 0 cache read, 0 cache write and "no prompt caching reported by the API"
5. Their own script sends the same kind of cached block with "ttl": "5m" to POST https://litellm-domain/v1/messages and gets the same uncached usage back; GET https://generativelanguage.googleapis.com/v1beta/cachedContents on their Gemini key lists no cache for it
6. A teammate's app sends a 1507-token system block marked cache_control to Gemini 2.5 Pro through POST https://litellm-domain/v1/chat/completions or POST https://litellm-domain/v1/responses
7. The gateway answers HTTP 400 Cached content is too small. total_token_count=1507, min_total_token_count=2048

After: the same turns read the system prompt from Gemini's cache, the ttl lands as sent, and the short block on 2.5 Pro goes through as a plain request

1. They point Claude Code at the gateway with ANTHROPIC_BASE_URL=https://litellm-domain and ANTHROPIC_MODEL=gemini-25-flash, then ask a question
2. Claude Code sends POST https://litellm-domain/v1/messages with its system prompt marked "cache_control": {"type": "ephemeral"}
3. The reply's usage now carries the system prompt as cache_read_input_tokens and only the new text as input_tokens
4. They run /cost and see 31.7k cache read and "48% of input tokens from cache", at a lower dollar figure than before
5. Their own script's "ttl": "5m" request comes back with cache_read_input_tokens, and GET https://generativelanguage.googleapis.com/v1beta/cachedContents lists a cache whose expireTime is five minutes after its createTime
6. A teammate's app sends the same 1507-token block to Gemini 2.5 Pro through POST https://litellm-domain/v1/chat/completions or POST https://litellm-domain/v1/responses
7. The gateway answers HTTP 200 with a normal reply and no cache fields in usage

Relevant issues

Ref #29810, Ref #29166

Affected release

Linear ticket

Resolves LIT-5373

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Both legs ran the same way and differ only in the commit the proxy was booted from. Each proxy ran python litellm/proxy/proxy_cli.py --config config.yaml --num_workers 2 --detailed_debug (2 uvicorn workers, no Redis, no database) with LITELLM_LOCAL_MODEL_COST_MAP=True so each side reads the cost map at its own commit. Before is the merge base 5611607 on port 44911, After is the tip 2303379 on port 43175. Every request went to Google's real Gemini API through the gemini/ provider and cost real money

model_list:
  - model_name: gemini-flash
    litellm_params:
      model: gemini/gemini-3.8-flash
      api_key: os.environ/GEMINI_API_KEY
  - model_name: gemini-25-flash
    litellm_params:
      model: gemini/gemini-2.5-flash
      api_key: os.environ/GEMINI_API_KEY
  - model_name: gemini-2.5-pro
    litellm_params:
      model: gemini/gemini-2.5-pro
      api_key: os.environ/GEMINI_API_KEY

general_settings:
  master_key: sk-lit5373-qa

Payloads. The /v1/messages ones carry a 27,823-character generated handbook as the system block (about 5.2k tokens, above every Gemini minimum); messages-ttl-5m.json is the same file with "ttl": "5m". The 2.5 Pro ones carry an 8k-character slice of it (1507 tokens, under 2.5 Pro's 2048 minimum)

// messages-ttl-1h.json
{"model": "gemini-flash", "max_tokens": 256,
 "system": [{"type": "text", "text": "<27,823-char handbook text>", "cache_control": {"type": "ephemeral", "ttl": "1h"}}],
 "messages": [{"role": "user", "content": "In one sentence, what is this document about?"}]}

// chat-25pro.json
{"model": "gemini-2.5-pro", "max_tokens": 2048,
 "messages": [{"role": "system", "content": [{"type": "text", "text": "<8,037-char handbook slice>", "cache_control": {"type": "ephemeral"}}]},
              {"role": "user", "content": "In one sentence, what is this document about?"}]}

// responses-25pro.json
{"model": "gemini-2.5-pro", "max_output_tokens": 2048,
 "input": [{"role": "system", "content": [{"type": "input_text", "text": "<8,042-char handbook slice>", "cache_control": {"type": "ephemeral"}}]},
           {"role": "user", "content": "In one sentence, what is this document about?"}]}

Claude Code v2.1.278 was launched interactively in tmux against each proxy with env -u ANTHROPIC_API_KEY ANTHROPIC_BASE_URL=http://localhost:<port> ANTHROPIC_AUTH_TOKEN=sk-lit5373-qa ANTHROPIC_MODEL=gemini-25-flash ANTHROPIC_SMALL_FAST_MODEL=gemini-25-flash claude, given the same two prompts (Reply with one word: hello, then world), then /cost

Before (5611607)

/v1/messages

1. for i in 1 2; do curl -s http://localhost:44911/v1/messages -H "Authorization: Bearer sk-lit5373-qa" -H "Content-Type: application/json" -d @messages-ttl-1h.json | jq -c .usage; done
2. {"input_tokens":5213,"output_tokens":118} then {"input_tokens":5213,"output_tokens":252}: the whole system block is billed as input both times, no cache fields
3. Same loop with @messages-ttl-5m.json
4. {"input_tokens":5213,"output_tokens":252} twice, no cache fields

/v1/chat/completions

1. curl -s -w "\n%{http_code}\n" http://localhost:44911/v1/chat/completions -H "Authorization: Bearer sk-lit5373-qa" -H "Content-Type: application/json" -d @chat-25pro.json
2. HTTP 400: {"error": {"message": "litellm.BadRequestError: GeminiException BadRequestError - {\n \"error\": {\n \"code\": 400,\n \"message\": \"Cached content is too small. total_token_count=1507, min_total_token_count=2048\",\n \"status\": \"INVALID_ARGUMENT\"\n }\n}\n. Received Model Group=gemini-2.5-pro\nAvailable Model Group Fallbacks=None", "type": "invalid_request_error", "param": null, "code": "400"}}

/v1/responses

1. curl -s -w "\n%{http_code}\n" http://localhost:44911/v1/responses -H "Authorization: Bearer sk-lit5373-qa" -H "Content-Type: application/json" -d @responses-25pro.json
2. HTTP 400 with the same body: "Cached content is too small. total_token_count=1507, min_total_token_count=2048"

Claude Code

1. Launch Claude Code against port 44911 as above, send Reply with one word: hello, then world
2. Both turns answer, and the pane shows nothing about caching

3. /cost reads gemini-25-flash: 67.2k input, 257 output, 0 cache read, 0 cache write ($0.3426) and 0% of input tokens from cache · no prompt caching reported by the API

After (2303379)

/v1/messages

1. for i in 1 2; do curl -s http://localhost:43175/v1/messages -H "Authorization: Bearer sk-lit5373-qa" -H "Content-Type: application/json" -d @messages-ttl-1h.json | jq -c .usage; done
2. {"input_tokens":11,"output_tokens":112,"cache_read_input_tokens":5204} then {"input_tokens":11,"output_tokens":252,"cache_read_input_tokens":5204}: the system block is read from Gemini's cache, only the question is billed as input
3. Same loop with @messages-ttl-5m.json
4. {"input_tokens":11,"output_tokens":156,"cache_read_input_tokens":5204} then {"input_tokens":11,"output_tokens":252,"cache_read_input_tokens":5204}
5. curl -s https://generativelanguage.googleapis.com/v1beta/cachedContents -H "x-goog-api-key: $GEMINI_API_KEY" | jq -c '.cachedContents[] | {name, model, createTime, expireTime, tokens: .usageMetadata.totalTokenCount}'
6. The 1h request's cache expires an hour after creation and the 5m one five minutes after, so the Anthropic units reached Google as 3600s and 300s (the 5m row is the one created by a repeat of that request at 07:30, after the first 5m cache had expired):
   {"name":"cachedContents/94pjl84ov0n65s1w59f5f8lt05qiwh1ba3fis4x0","model":"models/gemini-3.8-flash","createTime":"2026-09-19T07:22:22.489285Z","expireTime":"2026-09-19T08:22:22.235822630Z","tokens":5204}
{"name":"cachedContents/5g1w1gdpd0yw6m1xffqio7mljxzpmto8daxnmbly","model":"models/gemini-3.8-flash","createTime":"2026-09-19T07:30:03.221259Z","expireTime":"2026-09-19T07:35:03.001718991Z","tokens":5204}
{"name":"cachedContents/go1dh5afd0v1artc4tgw8j7p28iunn0c7mjw3n50","model":"models/gemini-2.5-flash","createTime":"2026-09-19T07:23:48.496788Z","expireTime":"2026-09-19T08:23:47.974610795Z","tokens":15837} (Claude Code's own cache from the case below; it sends no ttl, so Google's default hour applies)

/v1/chat/completions

1. curl -s -w "\n%{http_code}\n" http://localhost:43175/v1/chat/completions -H "Authorization: Bearer sk-lit5373-qa" -H "Content-Type: application/json" -d @chat-25pro.json
2. HTTP 200, "finish_reason": "stop", content This operations handbook describes how various gateway features like billing, routing, and caching are configured, how they evaluate requests, and their consistent approach to error handling and metrics.
3. usage: {"completion_tokens": 966, "prompt_tokens": 1516, "total_tokens": 2482, "completion_tokens_details": {"reasoning_tokens": 933, "text_tokens": 33}, "prompt_tokens_details": {"text_tokens": 1516}}, no cache fields because the block is under 2.5 Pro's minimum and was sent uncached

/v1/responses

1. curl -s -w "\n%{http_code}\n" http://localhost:43175/v1/responses -H "Authorization: Bearer sk-lit5373-qa" -H "Content-Type: application/json" -d @responses-25pro.json
2. HTTP 200, "status": "completed", output text This operations handbook describes how various gateway features are configured and how they consistently process requests, handle errors, and export metrics.
3. usage: {"input_tokens": 1516, "input_tokens_details": {"cached_tokens": 0, "text_tokens": 1516}, "output_tokens": 1362, "output_tokens_details": {"reasoning_tokens": 1338, "text_tokens": 24}, "total_tokens": 2878}

Claude Code

1. Launch Claude Code against port 43175 as above, send Reply with one word: hello, then world
2. Both turns answer the same way

3. /cost reads gemini-25-flash: 35.6k input, 280 output, 31.7k cache read, 0 cache write ($0.2008) and Prompt cache (main): 2 requests · 48% of input tokens from cache

Observations from the run, none caused by this PR:

• First cached call reports cache_read_input_tokens, never cache_creation_input_tokens; pre-existing
• Gemini 3.x Flash models keep main's 4096-token cache floor; left alone
• Claude Code's bare system prompt is about 1.5k tokens, under that floor
• The below-minimum debug log names 1024, not the model's floor; pre-existing

Type

🐛 Bug Fix

Caveats (if any)

Low

• The ttl has no upper cap, so 48h reaches Google as 172800s
  • Google bills cache storage per hour, but the caller asked for it, and the <seconds>s form was never capped before either
• The 2048 minimum is set on Gemini 2.5 Pro only
  • Google's docs list higher Flash minimums, but the live API caches 2.5 Flash and 3.8 Flash at about 1132 tokens, so those keep the 1024 default
• main's 4096 prompt_cache_min_tokens on Gemini 3.x models makes the proxy skip 1024-4095-token blocks Google would cache
  • Left alone: lowering it needs a per-model probe against the live API
• The Gemini check on /v1/messages is by model name
  • An alias containing gemini that routes elsewhere carries cache_control through, the same way claude aliases already do
• The Responses API half of the original PR is already on main via fix(responses): preserve cache_control in Responses API -> Chat Completion transformation #27727, so this PR carries only the Gemini pieces
• The original object and Pydantic normalization helper was dropped as unneeded on main
• The first cached call reports a cache read rather than a cache creation, and the below-minimum debug line names the 1024 default instead of the model's floor; both pre-existing and untouched

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[CLAassistant]

All committers have signed the CLA.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing edelauna:feat/gemini-cache-control-pass-through (2303379) with main (8afbc51)}

[greptile-apps]

Greptile Summary

This PR fixes cache_control being dropped when Responses API input items or content blocks are Pydantic models or custom objects rather than plain dicts, and extends Gemini/Vertex context caching to support Anthropic-style TTL units (1h, 5m) and per-model minimum token thresholds.

• Introduces _normalize_responses_api_object_to_dict as a shared normalizer and threads getattr/hasattr fallbacks through is_cached_message, extract_ttl_from_cached_messages, and the content-transformation pipeline so object-typed inputs are handled everywhere.
• Adds _normalize_ttl_to_seconds to convert Anthropic-style h/m/s units into Gemini's <N>s format (capped at 24 h), and get_gemini_context_caching_min_tokens to look up the per-model floor from model_prices_and_context_window.json (with string-matching fallbacks for models not yet in the file, which conflicts with the project rule requiring all model-specific flags to live in the JSON).
• Adds cache_creation_min_tokens to all known Gemini model entries across every provider prefix in model_prices_and_context_window.json.

Confidence Score: 4/5

Safe to merge with one fix: the hardcoded model-name string matching in get_gemini_context_caching_min_tokens should be replaced with a single conservative default.

The object/dict dual-path fixes are well-tested and the logic is correct. The one concern is get_gemini_context_caching_min_tokens: when get_model_info returns a dict that lacks cache_creation_min_tokens (possible for any model variant not yet in the JSON), the function guesses by matching substrings like gemini-2.5 or gemini-3. A wrong guess causes either a live Gemini 400 (floor too low) or silently skipped caching (floor too high), and future model names may not match the expected substrings at all. The rest of the PR is correct and well-covered by mocked unit tests.

litellm/llms/vertex_ai/context_caching/transformation.py — the fallback string-matching branches in get_gemini_context_caching_min_tokens

Important Files Changed

Filename	Overview
litellm/llms/vertex_ai/context_caching/transformation.py	Adds TTL normalization (h/m/s → seconds), message-level cache_control extraction, and a per-model min-token lookup; the fallback in get_gemini_context_caching_min_tokens uses hardcoded model-name string matching that conflicts with the project rule requiring model-specific flags to live only in model_prices_and_context_window.json
litellm/responses/litellm_completion_transformation/transformation.py	Adds _normalize_responses_api_object_to_dict helper and wires it through content/input-item paths so that cache_control is preserved from Pydantic models and custom objects; cache_control=None is now correctly excluded from downstream dicts
litellm/llms/vertex_ai/context_caching/vertex_ai_context_caching.py	Replaces the single MINIMUM_PROMPT_CACHE_TOKEN_COUNT constant with per-model dynamic lookup via get_gemini_context_caching_min_tokens; both sync and async paths updated consistently
litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py	Extends _copy_cache_control_to_target to accept Gemini models and fixes the non-dict fallback branch to use setattr instead of a bogus cast+dict-access
litellm/utils.py	Updates is_cached_message to handle non-dict message/content objects via getattr fallbacks, and adds min_token_count override parameter to is_prompt_caching_valid_prompt
litellm/types/llms/openai.py	Adds optional cache_control field to GenericChatCompletionMessage TypedDict so the type system recognises the newly propagated field
model_prices_and_context_window.json	Adds cache_creation_min_tokens to all known Gemini model entries across providers; values are 2048 for Gemini 2.5 variants and 2048/4096 for Gemini 3 family
tests/test_litellm/llms/vertex_ai/context_caching/test_context_caching_ttl.py	New unit-test file covering TTL normalization, per-model min-token lookup, object-based cache_control extraction, and end-to-end transformation; all tests are mocked with no real network calls
tests/test_litellm/llms/vertex_ai/context_caching/test_vertex_ai_context_caching.py	Existing context-caching tests; patches is_prompt_caching_valid_prompt globally to bypass token counting, keeping tests isolated from the new dynamic min-token logic

_{Reviews (5): Last reviewed commit: "fix(responses): guard against None cache..." | Re-trigger Greptile}

[edelauna]

@greptile-apps review

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 2 · PR risk: 0/10

[edelauna]

@greptile-apps review

[shivamrawat1]

@greptile review again

[greptile-apps]

model_dump() / dict() include None-valued fields, so when a Pydantic content block has cache_control: Optional[...] = None, the serialized dict contains "cache_control": None. The "cache_control" in item guard then evaluates to True and None is forwarded as the cache_control value on both the image and text paths, which can produce a provider-side validation error (e.g., Anthropic rejects a null cache_control). The plain-object fallback path in _normalize_responses_api_object_to_dict already guards against None; these two call-sites need the same protection.

Suggested change

	elif item.get("type") == "input_image":
	image_block = {
	**LiteLLMCompletionResponsesConfig._transform_input_image_item_to_image_item(item),
	**({"cache_control": item["cache_control"]} if item.get("cache_control") is not None else {}),
	}
	content_list.append(image_block)
	else:
	# Skip text blocks with None text to avoid downstream errors
	text_value = item.get("text")
	if text_value is None:
	continue
	content_block: dict[str, Any] = {
	"type": LiteLLMCompletionResponsesConfig._get_chat_completion_request_content_type(
	item.get("type") or "text"
	),
	"text": text_value,
	**({"cache_control": item["cache_control"]} if item.get("cache_control") is not None else {}),
	}

[mateo-berri]

Superseded by #41938, the same commits on an internal branch so CircleCI can run the llm_translation and local_testing suites. Thanks for the contribution!