feat(vertex_ai): bill explicit context cache storage per token-hour

[roasted-penguin]

TLDR

Problem this solves:

• Explicit Gemini context cache storage is billed hourly by Google
• litellm never prices it, so it reaches no spend aggregation
• Spend for cache-heavy Vertex AI traffic reads lower than the bill

How it solves it:

• New pricing key cache_storage_cost_per_token_per_hour
• On cache creation, record tokens x lifetime on the request
• Price it as additional_costs["cache_storage_cost"] on that request
• Reused caches and models without the key add nothing

User Flow

Before: a developer whose prompts trigger litellm's automatic context caching on Vertex AI sees spend that misses the cache storage line of the Google bill

1. They send POST https://litellm-domain/v1/chat/completions to vertex_ai/gemini-2.5-pro with a large system message carrying cache_control: {"type": "ephemeral", "ttl": "3600s"}
2. The gateway creates a cache for the system message and the call returns 200
3. They open https://litellm-domain/ui/?page=logs and the request's spend covers only token charges, with nothing for the hour of cache storage Google bills

After: the request that created the cache also carries its storage charge

1. They send the same POST https://litellm-domain/v1/chat/completions with the same cache_control system message
2. The gateway creates the cache and the call returns 200
3. https://litellm-domain/ui/?page=logs shows the request's spend including cache tokens x 1 hour x the model's storage rate, listed under cache_storage_cost in the cost breakdown
4. A follow-up request that reuses the same cache shows no storage charge

Relevant issues

Related: #41861, #41874, #11049

#41874 bills the input tokens used to create the cache and says hourly storage is not modeled. This PR covers only storage and stays out of the lines #41874 changes as much as it can (it adds one call after the create response in each of the sync and async paths). #11049 proposed the same key name and was closed without landing

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Why this is billable and can be priced at creation

The Vertex AI context caching overview says "For explicit caching, there are also storage costs based on how long caches are stored. There are no storage costs for implicit caching" (https://cloud.google.com/vertex-ai/generative-ai/docs/context-cache/context-cache-overview). On a Cloud Billing export this shows up as a separate "... Text Input Caching Storage" style SKU metered in token-hours, apart from the input token SKUs

The Vertex AI pricing page has a "Context Cache Storage price for Explicit Caching" table priced per token per hour, with the same rate below and above 200K input tokens (https://cloud.google.com/vertex-ai/generative-ai/pricing, read 2026-10-07). Rates added to the cost map from that table:

Pricing page row	Cost map entries	USD per token per hour
Gemini 3.1 Pro, Gemini 3 Pro, Gemini 2.5 Pro	gemini-2.5-pro, gemini-3.1-pro-preview, vertex_ai/gemini-3.1-pro-preview, vertex_ai/gemini-3-pro-preview	0.0000045
Gemini 3.8 Flash Cyber, 3.8 / 3.7 / 3.6 / 3.5 Flash, Gemini 3 Flash	the matching bare and vertex_ai/ entries, plus gemini-3-flash-preview	0.000001
Gemini 3.5 Flash Lite, Gemini 3.1 Flash Lite	the matching bare and vertex_ai/ entries, plus the 3.1 Flash Lite preview ids	0.000001
Gemini 2.5 Flash, Gemini 2.5 Flash Lite	gemini-2.5-flash, gemini-2.5-flash-lite	0.000001

Entries the table does not name (dated preview snapshots, -customtools, TTS, computer use, image and Gemma models, Gemini 2.0) and all gemini/ (Google AI Studio) entries are left without the key, so they keep today's behavior. The AI Studio rates live on https://ai.google.dev/gemini-api/docs/pricing and can follow in a separate PR

The cost is fully known when litellm creates the cache. The cachedContents create response carries usageMetadata.totalTokenCount, createTime and expireTime, so the charge is totalTokenCount x (expireTime - createTime) in hours x rate. If the response has no times, litellm falls back to the ttl it sent, then to the 60 minute default ("The default expiration time of a context cache is 60 minutes after it's created", https://cloud.google.com/vertex-ai/generative-ai/docs/context-cache/context-cache-create)

How it is wired

The vertex context caching code computes the token-hours with a small pure function (litellm/llms/vertex_ai/context_caching/storage_cost.py) and records them on the request's logging object, only when it actually creates a cache. completion_cost then multiplies them by the rate, taking the deployment's own storage rate first and otherwise the cost map entry of the underlying model (not the deployment id), and adds the result as additional_costs["cache_storage_cost"]. That puts the charge in the same request's response_cost, cost_breakdown and SpendLogs row, so key, team, user and tag attribution come for free

I picked additional_costs over the RAG-style model_call_details["additional_response_cost"]. The latter is added to the total after completion_cost returns, so it never appears in cost_breakdown, skips the discount and margin config, and is invisible to callers that call completion_cost directly. _get_additional_costs itself is left alone because it only sees provider config and token counts, not the logging object

cache_storage_cost_per_token_per_hour is added to the cost map sample_spec, the generated JSON schema (regenerated with the unchanged generator, since cost-map-guard renders it with the base branch's generator), and ModelInfoBase / get_model_info. A deployment overrides it under model_info, next to cost_per_ptu_per_hour, not under litellm_params: it is a separate per-hour charge rather than a token price, so it never turns on custom pricing (a deployment that sets only this key keeps the model's token rates), and it is kept off the shared {provider}/{model} cost-map key so one deployment's rate never reaches its siblings. schema.d.ts gains the optional field on the router ModelInfo

Backward compatibility: a model without the key, a request that reuses an existing cache, a request that passes its own cached_content, and every non-Gemini request all add 0 and leave totals and cost_breakdown unchanged

Screenshots / Proof of Fix

Live run against a real Vertex AI project through a local proxy (litellm --config config.yaml, Postgres for SpendLogs). Project, credentials and resource names are omitted.

config.yaml:

model_list:
  - model_name: gemini-2.5-flash
    litellm_params:
      model: vertex_ai/gemini-2.5-flash
      vertex_project: os.environ/VERTEX_PROJECT
      vertex_location: us-central1
      vertex_credentials: os.environ/VERTEX_CREDENTIALS
general_settings:
  master_key: sk-local-e2e
  database_url: os.environ/DATABASE_URL

Request (system prompt of ~6.5k tokens marked for caching with a 600s TTL; a fresh marker per run so each run creates a new cache):

curl -sS -D - http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-local-e2e" -H "Content-Type: application/json" \
  -d '{"model": "gemini-2.5-flash",
       "messages": [
         {"role": "system", "content": [{"type": "text", "text": "<~6.5k tokens of notes>",
          "cache_control": {"type": "ephemeral", "ttl": "600s"}}]},
         {"role": "user", "content": "In one sentence, what do the notes describe?"}],
       "metadata": {"tags": ["TEST"]}}'

Before (merge base d364d5e7ca): the cache is created (cache_read_input_tokens: 6496), but nothing accounts for storing it

x-litellm-response-cost: 0.00047508
SpendLogs.spend = 0.00047508
cost_breakdown = {"input_cost": 0.00019758, "output_cost": 0.0002775, "cache_read_cost": 0.00019488, ...}   # no storage

After (PR tip c231b7f530), with LITELLM_LOCAL_MODEL_COST_MAP=True so the proxy reads the cost map from this branch:

Call 1 creates the cache. The create response reports usageMetadata.totalTokenCount = 6498 and expireTime - createTime ≈ 600s, so storage = 6498 × (600 / 3600) h × $1e-6 = $0.0010830

x-litellm-response-cost: 0.0014531069685000003
SpendLogs.spend = 0.0014531069685   (prompt 6507, completion 69, tags ["TEST"])
cost_breakdown = {"input_cost": 0.00019764, "output_cost": 0.0001725, "cache_read_cost": 0.00019494,
                  "additional_costs": {"cache_storage_cost": 0.0010829669685000001},
                  "total_cost": 0.0014531069685000003, ...}

0.00019764 + 0.0001725 + 0.00108297 = 0.00145311

Call 2 sends the identical request, reuses the existing cache, and adds no storage

x-litellm-response-cost: 0.00052764
SpendLogs.spend = 0.00052764
cost_breakdown = {"input_cost": 0.00019764, "output_cost": 0.00033, "cache_read_cost": 0.00019494, ...}   # no additional_costs

Unit coverage, with only the HTTP client mocked:

1. uv run pytest tests/unit/llms/vertex_ai/context_caching/ tests/unit/test_cost_calculator.py tests/unit/test_utils.py tests/unit/test_model_prices_schema.py
2. 966 passed
3. make lint passes

New tests:

• tests/unit/llms/vertex_ai/context_caching/test_storage_cost.py: lifetime from the response, requested ttl fallback, 60 minute default, unparseable ttl, malformed or token-less responses give 0
• tests/unit/llms/vertex_ai/context_caching/test_vertex_ai_context_caching.py: creating a cache records tokens x hours on a real logging object, sync and async; a second identical request that reuses the cache records nothing, sync and async
• tests/unit/test_cost_calculator.py: storage is added to the creating request's cost and breakdown at the cost map rate; a model without the key costs exactly the same as before; a deployment's custom rate wins over the cost map; a deployment with custom token prices but no storage rate bills storage at the underlying model's rate; a storage-only model_info override keeps the underlying token prices and never reaches the shared backend key (Router based); storage joins provider additional costs such as the Azure Model Router fee instead of replacing them
• tests/unit/test_router_model_cost_isolation.py: the shared-key denylist guard also accepts the deployment-scoped storage rate
• tests/unit/test_utils.py: get_model_info surfaces the key for every cost map entry that has it

Rates in tests are read from the cost map, never pinned

CI notes for 0b07cb5788 (101 pass, 2 fail, both in the Rust workflow; this PR changes no Rust):

• rust-wheel fails tests/test_litellm_rust/test_traces.py::test_schema_setup_uses_configured_retention, the same failure main shows right now (run 37735511563 at 91f36f1774)
• rust-test ran out of runner disk while compiling ("No space left on device" building litellm-secrets); on main the same job is currently cancelled. A re-run needs repo admin rights

The lint gates (basedpyright, type discipline, ruff strict, test quality) pass at their merge-base counts.

Type

🆕 New Feature

Caveats (if any)

Medium

• Full TTL is billed upfront at creation
  • litellm never deletes caches, so this matches Google unless someone else does
  • Deletes or TTL updates made outside litellm are not reflected
• Cache creates through pass-through cachedContents routes are not covered, follow-up
• If the request fails after the cache was created, the storage charge rides on the failed attempt and is not billed

Low

• gemini/ (AI Studio) entries have no storage rate yet
• Vertex regional endpoint uplift is not applied to storage; the pricing page lists one storage rate
• Until this lands on main, proxies that fetch the remote cost map (the default) see no storage rate and add 0; after merge the remote map carries the key

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[CLAassistant]

All committers have signed the CLA.

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing roasted-penguin:feat/vertex-context-cache-storage-cost (0b07cb5) with main (91f36f1)¹}

Footnotes

1. No successful run was found on main (9df3d4c) during the generation of this report, so 91f36f1 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[roasted-penguin]

@greptileai

[greptile-apps]

Confidence Score: 5/5

[Critical risk] Adds billing calculation for cached content storage costs.

The PR appears safe to merge, with no new actionable issues found

What we checked:

• Storage rates stay deployment-specific: The router stores full prices under each deployment’s ID and removes the storage field before updating shared model entries
• Pricing tests restore shared state: An automatic fixture restores the registered-price records and live routers after each test and clears cached model information

Summary

This PR adds hourly storage charges to requests that create explicit Vertex AI context caches

• Cache creation records token-hours on the request’s logging object
• completion_cost applies the deployment’s storage rate first, then the underlying model’s rate
• The charge joins additional_costs, the total, and cost_breakdown
• Tests cover creation, reuse, TTL fallbacks, custom prices, and deployment isolation
• The earlier fixes remain in place. roasted-penguin intentionally kept plain annotations inside the cost-calculation loop because the type checker rejects Final there; the previous thread accepted that explanation
• roasted-penguin documents that storage created before a failed completion remains unbilled, and that external cache deletion or TTL changes are not reflected

_{Reviews (5) · Last reviewed commit: "test(vertex_ai): cover cache storage joi..." · Reviewed by Greptile}

[roasted-penguin]

@greptileai

[roasted-penguin]

@greptileai