Repository navigation
fix(pricing): correct cached-token and limit fields on the realtime cost-map entries - #40018
marty-sullivan wants to merge 3 commits into
Conversation
Greptile SummaryThis PR corrects pricing, token limits, and capability metadata for Gemini Live and Azure realtime models, with focused regression coverage.
Confidence Score: 5/5The PR appears safe to merge because the pricing and capability corrections have focused regression coverage and no blocking issue remains. The cached-token regression test is present, and the test docstrings were reduced to concise descriptions. Both previous threads were manually resolved without explanatory replies.
|
| Filename | Overview |
|---|---|
| model_prices_and_context_window.json | Corrects the primary model cost map’s cached-token pricing, limits, and capability metadata. |
| litellm/model_prices_and_context_window_backup.json | Keeps the backup cost map synchronized with the primary map. |
| tests/test_litellm/test_cost_calculator.py | Adds focused regression tests for cached-token billing, capability discovery, limits, and realtime audio cache rates. |
Reviews (5): Last reviewed commit: "test(cost): trim the docstrings on the r..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
|
@greptileai added the cached-token regression test you flagged. It fails against the unfixed cost map, and the file now passes at 139. |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
Thanks for the contribution! The Azure half checks out, but I think the gemini rate is wrong
azure/gpt-realtime-2
- I pulled Azure's retail meters myself ->
gpt-realtime-2 Audio cd inp Glis 0.4, so4e-07is right - Nothing in the cost calculator reads either audio cache key today, so this half is consistency-only, as you say
gemini-live-2.5-flash-native-audio
- The bug is real. I ran the base map and 100k cached tokens bill $0.00, so thanks for catching it
- But
7.5e-08is 15% of this model's5e-07input rate, and I don't think anything published backs it - Google's pricing page carries no cached-input row for the Live native-audio models, so the siblings aren't really a source
- Every current Gemini 2.5 entry in our map prices cache reads at 10% of input:
3e-07 -> 3e-08,1e-07 -> 1e-08,1.25e-06 -> 1.25e-07 - That tracks Google's published 90% discount, e.g. 2.5 Flash is $0.30 input against $0.03 cached
7.5e-08came in with the sibling entry in #16183, back when its input rate was3e-07, so it was 25% thengemini-2.5-flashwas already3e-08by that point, and d565860 later corrected the input to5e-07without rescaling the cached rate- So I think this wants
5e-08, and the two preview siblings want the same correction here rather than being cited as the source
On the test
- The
model_info[...]equality asserts are shape checks, and thecache_read_input_token_costone raises before the billing assert ever runs - Dropping them leaves the cost assertion, which is the part with teeth, and stops the constant being pinned in two places
Happy to merge once the rate is settled and it's green
marty-sullivan
changed the title
|
@mateo-berri agreed, 5e-08. Google's flat 90% discount confirms it, so both preview siblings are corrected too, and the shape asserts are gone. |
|
@greptileai rate corrected to 5e-08 per review, preview siblings fixed alongside, shape asserts dropped. Please re-review. |
5c150d1 to
c382c37
Compare
azure/gpt-realtime-2 was the only member of the gpt-realtime-2 family priced on one side of its cached-audio meter. Azure publishes that meter as "gpt-realtime-2 Audio cd inp Gl 1M Tokens" at 0.4 per 1M and charges the same rate for the write that populates the cache and the read that hits it, so cache_creation_input_audio_token_cost lands at 4e-07, matching azure/gpt-realtime-2.1, azure/gpt-realtime-2.1-mini and the openai gpt-realtime-2 entry. No cost path reads that field yet, so this corrects what get_model_info reports rather than what anything bills. The gemini Live entries go the other way. Google's Vertex context-caching page publishes separate supported-model lists for implicit and explicit caching, and no Live or native-audio model is in either one. Its pricing page prints N/A in both cached-input columns for every Gemini 2.5 Flash Live API row, where plain 2.5 Flash and 2.5 Flash-Lite both carry real cached prices, and the Vertex model card for the family marks context caching not supported outright. Vertex never reports cachedContentTokenCount on a Live session either, including for a byte-identical 7,021-token prefix replayed across sessions minutes apart, which is well past the 2,048-token minimum the same page sets for the Gemini 2 family. So the 7.5e-08 on the two preview siblings priced something the provider does not sell, and supports_prompt_caching on all three claimed a capability the model does not have. The rate comes out. The flag is set to false rather than removed, because get_model_info maps an absent key to None, and None is how this map spells "nobody checked" across the 2,788 entries that omit it, where false records the vendor's documented no. Both readers of the flag gate on `is True`, so nothing bills or behaves differently either way. Only the cached fields change on the two 09-2025 preview entries. Their source field points at the Gemini API pricing page rather than the Vertex one, so they describe a different surface with its own published limits, and their context windows are left alone rather than assumed to match the Vertex model card that drives the GA entry. Tests cover all three halves: the family invariant that a cached audio read implies an equal cached audio write, a cached count on a Live entry leaving the bill at the fresh-input total instead of adding the old 7.5e-08, and supports_prompt_caching answering false for all three entries while still answering true for 2.5 Flash, so the false cannot be a swallowed lookup error.
…bilities Google's model card for model ID gemini-live-2.5-flash-native-audio gives a 128K context window and 64K maximum output tokens, and marks structured output, context caching and URL context as not supported. Its modality list is text in and out, image in, audio in and out, and video in, with no document input of any kind. The entry advertised a 1M context window, an off-by-one 65535 output cap, and three capability flags the vendor marks unsupported. Context caching is the fourth and is handled in the cached-fields change alongside its two preview siblings. Both the bare id and vertex_ai/gemini-live-2.5-flash-native-audio resolve to this single entry, so the test drives the corrected values through both.
c382c37 to
9058a84
Compare
marty-sullivan
changed the title
|
@greptileai reworked and rebased: the Live cached rate now comes out entirely, plus the native-audio entry's limits and capability flags |
|
@mateo-berri digging further flipped this: Google lists Live under neither caching mode and prices N/A, so the rate comes out rather than moving to 5e-08 |
Each of the four tests this PR added carried several lines of rationale for why the map was wrong. Cut them to one line saying what the test pins, and keep the two facts that explain a number rather than the change: the cached tokens dropping out of the bill, and why the 2.5 Flash control has to be there, both now assertion messages where they fire
|
@greptileai trimmed the four docstrings to one line each, moving the two load-bearing facts into assertion messages instead |
|
That addresses my concern. Keeping each docstring as a concise purpose statement and moving the load-bearing rationale into assertion messages makes the tests clearer while preserving useful failure diagnostics. No further changes needed. |
|
Thanks, you were right twice: 7.5e-08 had nothing behind it, and the $0.00 was worth catching Chasing the published number only confirmed what you noticed. Vertex's table prints N/A in both cached columns on every Gemini 2.5 Flash Live API row, the same row that sources every other rate here. Two rows below, non-Live Flash reads $0.30 against $0.03, so your 90% rule holds wherever Google publishes a cached price. Live is not one, and the model card is blunter: "Context caching Not supported" So 5e-08 fails the same test 7.5e-08 did. The map mostly agrees already: 10 of its 12 Gemini realtime entries carry no cached rate, and the 2 that do are the preview siblings you flagged as not a source supports_prompt_caching false records the vendor's no, since an absent key reads back as None I also dropped the gemini-live-2.5-flash entry: the card names only the native-audio Model ID, the bare id is undocumented now, and we are retiring it on our side anyway |
|
@greptileai please re-score. Both P2s are addressed: the cached-token regression test landed and the docstrings were trimmed |
3 participants
TLDR
Problem this solves:
azure/gpt-realtime-2prices one side of its cached-audio meterHow it solves it:
supports_prompt_caching: falseThis revision reverses the
5e-08cached rate the last review asked for, on that review's own reasoning.7.5e-08was refused because nothing published backs it, and nothing published backs5e-08either: the flat 90% discount is read off Gemini rows that sell caching, and this family does not sell it. Google's context-caching page names no Live model in either its implicit or explicit supported-model list, its pricing page printsN/Ain both cached-input columns for every "Gemini 2.5 Flash Live API" row where plain 2.5 Flash carries $0.03, and the model card says "Context caching Not supported" outright. So the rate comes out instead of being rescaled, and thesupports_prompt_caching: truethat made an absent rate look like a pricing gap becomesfalseOn "not on the linked pricing pages": that page is the
sourceupstream's own mergedgemini-live-2.5-flash-native-audioentry already names, and it renders its tables in JavaScript, so anything that does not execute JS reports no prices at all. The shared setup below has thecurl -sLthat returns them, and the vendor reads there are the whole basis of this PR. Every value comes from a page a reviewer can open, with no rate inference anywhere. Theazure/gpt-realtime-2half is unchanged from the revision that was already acceptedUser Flow
Before: a developer building on the GA Gemini Live model plans against the gateway's own catalog, and the numbers in it that matter are all wrong
gemini-live-2.5-flash-native-audioentry reportsmax_input_tokens: 1048576,supports_response_schema: true,supports_url_context: true,supports_pdf_input: trueandsupports_prompt_caching: truesupports_prompt_caching: true, they restructure their prompts around a shared prefix to earn the cached discount, replay a 7,021-token prefix across sessions minutes apart, and get no cached tokens reported back on any of themgemini-live-2.5-flash-preview-native-audio-09-2025: POST https://litellm-domain/spend/calculate with 100,000 cached and 1,000 fresh input tokens returns{"cost":0.008}, charging the cached tokens at $0.075 per 1MAfter: the same catalog reports what Google documents, so the developer plans against the real limits and never chases a cache that does not exist
max_input_tokens: 131072,supports_response_schema: false,supports_url_context: false,supports_pdf_input: falseandsupports_prompt_caching: false{"cost":0.0005}, the fresh tokens alone at $0.50 per 1MRelevant issues
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Tests are three functions in
tests/test_litellm/test_cost_calculator.py, nine cases in total. They check that no cached rate is present on any of the three Live entries and that a cached-token bill lands at the fresh-input total, thatsupports_prompt_cachinganswers false on all three withgemini-2.5-flashas a control so a swallowed lookup error cannot read as a pass, that the corrected limits and capability flags come back through both the bare and thevertex_ai/-prefixed name, and that the gpt-realtime-2 family agrees a cached audio read implies an equal cached audio writeReverting the two map files to the merge base fails 8 of those 9 cases. The ninth asserts the GA entry carries no cached rate, which it already does not, because that is the state the two preview siblings are being brought to
tests/test_litellm/test_cost_calculator.pyis 177 passed. Across the five files this change can reach (that file plustest_gemini_tts_native_audio_pricing.py,test_model_prices_schema.py,test_gpt_realtime_mode.pyandtest_utils.py) it is 618 passed, and 8 failed with 610 passed once the maps go back to the base. The three lint and type budget gates all report within ceiling and no budget file is touchedazure/gpt-realtime-2is absent from the user flow above because nothing reads either audio cache key, so no caller observes a difference. It changes what the map declares, which is why its test is a family invariant rather than a billing assertionDelays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Shared setup. A proxy on the local cost map, with the two preview entries and
azure/gpt-realtime-2alongside the GA model, so one/v1/model/inforead covers every entry this PR touches:The vendor reads below are the sources for every value in this PR. They are identical on both runs, so they sit here rather than being repeated under Before and After
Google's Vertex pricing page 301s to
cloud.google.com/gemini-enterprise-agent-platform/generative-ai/pricingand renders its tables in JavaScript. Fetching it with anything that does not execute JS returns navigation chrome and no prices, which is worth knowing before checking the cached columns.curl -sLreturns them as ordinary<table>markup:Every Live row is
N/Ain both cached-input columns, and the two Gemini 2.5 rows either side of it carry real cached prices, so theN/Ais a statement about this product rather than a page that omits cached pricing. That is the same page the merged entry already cites:jq '."gemini-live-2.5-flash-native-audio".source' model_prices_and_context_window.jsonGoogle's context-caching page carries the supported-model lists, and no Live or native-audio model is in either one. It also sets the minimum prefix a cache needs, which the replay in the user flow is well past:
Google's model card for model ID
gemini-live-2.5-flash-native-audiogives the limits and the three capability answers, and its footer reads "Last updated 2026-09-03 UTC":The same card lists MIME types for image, video and audio and no document type anywhere on the page, which is where
supports_pdf_input: falsecomes from:Azure publishes the cached-audio meter for the gpt-realtime-2 deployment:
Before (168a005)
Context caching on the three Gemini Live entries
{"model":"vertex_ai/gemini-live-2.5-flash-native-audio","cache_read_input_token_cost":null,"supports_prompt_caching":true} {"model":"vertex_ai/gemini-live-2.5-flash-preview-native-audio-09-2025","cache_read_input_token_cost":7.5E-8,"supports_prompt_caching":true} {"model":"gemini/gemini-live-2.5-flash-preview-native-audio-09-2025","cache_read_input_token_cost":7.5E-8,"supports_prompt_caching":true}All three advertise prompt caching. Two of them price a cached read at
7.5e-08, a rate that appears on no Google page, and the GA entry advertises caching with no cached price at allThe GA entry charges $0.0005, the fresh tokens alone, while claiming to support caching. The two preview entries charge $0.008, which is $0.0075 of invented cached-token cost
Limits and capabilities on gemini-live-2.5-flash-native-audio
{ "max_input_tokens": 1048576, "max_output_tokens": 65535, "max_tokens": 65535, "supports_pdf_input": true, "supports_response_schema": true, "supports_url_context": true }max_input_tokensis 1,048,576 against the card's "Context window 128K", eight times too large, and it is the field a caller uses to decide whether a request fits.max_output_tokensis 65535 against "Maximum output tokens 64K", which is 65536. All three capability flags say true where the card says not supported or lists no MIME typeCached audio on azure/gpt-realtime-2
{"azure/gpt-realtime-2":[4E-7,null],"azure/gpt-realtime-2.1":[4E-7,4E-7],"azure/gpt-realtime-2.1-mini":[3E-7,3E-7],"gpt-realtime-2":[4E-7,4E-7],"gpt-realtime-2.1":[4E-7,4E-7],"gpt-realtime-2.1-mini":[3E-7,3E-7]}azure/gpt-realtime-2is the only member of the family with a cached audio read and no matching write, for a deployment whose meter publishes0.4for exactly one cached-audio rate{"cache_read_input_audio_token_cost":null,"cache_creation_input_audio_token_cost":null}Both read null because
get_model_infocarries neither audio cache key, which is why this half changes what the map declares and no billAfter (9058a84)
Context caching on the three Gemini Live entries
{"model":"vertex_ai/gemini-live-2.5-flash-native-audio","cache_read_input_token_cost":null,"supports_prompt_caching":false} {"model":"vertex_ai/gemini-live-2.5-flash-preview-native-audio-09-2025","cache_read_input_token_cost":null,"supports_prompt_caching":false} {"model":"gemini/gemini-live-2.5-flash-preview-native-audio-09-2025","cache_read_input_token_cost":null,"supports_prompt_caching":false}No entry prices a cached read and none claims caching, which is what all three vendor pages say
All three now charge the same $0.0005, and the flag says why: there is no cache to discount, so the fresh-input total is the whole bill
Limits and capabilities on gemini-live-2.5-flash-native-audio
{ "max_input_tokens": 131072, "max_output_tokens": 65536, "max_tokens": 65536, "supports_pdf_input": false, "supports_response_schema": false, "supports_url_context": false }131072 is the card's 128K, 65536 is its 64K, and the three flags match "Structured output Not supported", "URL context Not supported" and a MIME list with no document type
Cached audio on azure/gpt-realtime-2
{"azure/gpt-realtime-2":[4E-7,4E-7],"azure/gpt-realtime-2.1":[4E-7,4E-7],"azure/gpt-realtime-2.1-mini":[3E-7,3E-7],"gpt-realtime-2":[4E-7,4E-7],"gpt-realtime-2.1":[4E-7,4E-7],"gpt-realtime-2.1-mini":[3E-7,3E-7]}Every family member now reports its cached audio write at the same rate as its read, and
azure/gpt-realtime-2lands on the0.4its own meter publishes{"cache_read_input_audio_token_cost":null,"cache_creation_input_audio_token_cost":null}Unchanged, as expected. No bill moves on this half
Type
🐛 Bug Fix
Caveats (if any)
Medium
5e-08cached rate, which a review asked forcache_read_input_token_costfrom two entries that ship it todayfalsein the map forsupports_pdf_inputandsupports_url_contexttrueand none carriedfalse, so these are the first vendor-documented refusals for either field.supports_response_schemaalready has 81, andsupports_prompt_caching40Low
*-preview-native-audio-09-2025entries keepmax_input_tokens: 1048576sourceis the Gemini API pricing page, a different surface with its own published limits, so only their cached fields change here rather than being assumed to match the Vertex cardcache_creation_input_audio_token_coststill surfaces nowhereget_model_infocarries neither audio cache key, so the Azure half corrects what the map declares and changes no billsupports_prompt_cachingis set tofalserather than deletedNone, which is how the map spells "nobody checked" across the 2,788 entries that omit it, wherefalserecords the vendor's documented no. Both places that act on the flag gate onis True, so nothing behaves differently either wayFinal Attestation