fix(cost): bill cached realtime audio tokens at the audio cache-read rate

[devin-ai-integration]

TLDR

Problem this solves:

• Realtime requests with cached audio input are overbilled, about 2x on the ticket's turn
• OpenAI's cached_tokens_details text/audio split was dropped at every layer
• Cached audio tokens were then billed at the full $32/M audio rate

How it solves it:

• Carry cached_tokens_details through realtime usage into prompt_tokens_details
• Bill cached audio at cache_read_input_audio_token_cost, fallback to cache_read_input_token_cost
• Split the cost breakdown's cache read cost the same way, so cached audio is priced at the audio cache-read rate there too
• Carry cache_read_input_audio_token_cost through get_model_info, so proxy and router lookups bill cached audio at that rate (the mini realtime family was falling back to the text cache-read rate)
• Subtract cached text/audio/image from the full-rate buckets
• Sum the nested split when combining a session's response.done events
• Add cache_read_input_audio_token_cost to gpt-realtime, gpt-realtime-1.5, gpt-realtime-2025-08-28
• Fill the realtime entries that still lacked a cache-read rate: gpt-realtime-mini gains cache_read_input_token_cost $0.06/M (it billed every cached token at $0 before), azure/gpt-realtime-mini and azure/gpt-realtime-mini-2025-10-06 gain cache_read_input_audio_token_cost $0.30/M, and azure/gpt-realtime-2025-08-28 and azure/gpt-realtime-1.5-2026-02-23 gain cache_read_input_audio_token_cost $0.40/M with cache_read_input_token_cost lowered from $4/M (the full input rate) to $0.40/M. The rates are OpenAI's pricing page for the mini family and Azure's retail prices API for the Azure entries (Global meters gpt rt txt 0828 cchd Inp glbl, gpt rt aud 0828 cchd Inp glbl, gpt rt 1.5 txt cd inp Gl, gpt rt 1.5 aud cd inp Gl, gpt rt txt mini cchd Inp glbl, gpt rt aud mini cchd Inp glbl), and each Azure entry now matches the OpenAI entry for the same model

User Flow

Before: a voice app on gpt-realtime-2 sees each turn's spend come out about double what OpenAI charges

1. They open a websocket to wss://litellm-domain/v1/realtime?model=gpt-realtime-2 and hold a multi-turn audio conversation
2. A turn's response.done event reports input_tokens: 283, text_tokens: 116, audio_tokens: 167, cached_tokens: 192, with cached_tokens_details: {text_tokens: 64, audio_tokens: 128}
3. They open https://litellm-domain/ui/?page=logs and the turn's input spend reads $0.0029888, against OpenAI's $0.0015328

After: the same turn is billed at what OpenAI charges

1. They open a websocket to wss://litellm-domain/v1/realtime?model=gpt-realtime-2 and hold the same conversation
2. The same response.done event arrives with the same usage
3. https://litellm-domain/ui/?page=logs shows the turn's input spend at $0.0015328: 52 text at $4/M, 39 audio at $32/M, and all 192 cached tokens at $0.40/M

Relevant issues

Linear ticket

Resolves LIT-4994

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Both legs run the same 4-turn websocket session on three models: gpt-realtime-2 (the customer's model, where the text and audio cache-read rates coincide), gpt-realtime-2.1-mini (where they differ), and gpt-realtime-mini (whose cost map entry had no cache-read rate before this PR), against a live proxy with 2 uvicorn workers, its own Postgres database, and the checkout's own cost map, real OpenAI calls. The before proxy is the merge base c134fb7 on port 32566 and the after proxy is this PR's tip a94c060 on port 42155, each booted with

LITELLM_LOCAL_MODEL_COST_MAP=True LITELLM_MASTER_KEY=$KEY PROXY_BASE_URL=http://localhost:$PORT python litellm/proxy/proxy_cli.py --config config.yaml --port $PORT --num_workers 2 --use_v2_migration_resolver

config.yaml:

model_list:
  - model_name: gpt-realtime-2
    litellm_params:
      model: openai/gpt-realtime-2
      api_key: os.environ/OPENAI_API_KEY
  - model_name: gpt-5-mini
    litellm_params:
      model: openai/gpt-5-mini
      api_key: os.environ/OPENAI_API_KEY
  - model_name: gpt-audio-mini
    litellm_params:
      model: openai/gpt-audio-mini
      api_key: os.environ/OPENAI_API_KEY
  - model_name: claude-haiku-4-5
    litellm_params:
      model: anthropic/claude-haiku-4-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: gpt-realtime-2.1-mini
    litellm_params:
      model: openai/gpt-realtime-2.1-mini
      api_key: os.environ/OPENAI_API_KEY
  - model_name: gpt-realtime-mini
    litellm_params:
      model: openai/gpt-realtime-mini
      api_key: os.environ/OPENAI_API_KEY

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

The client opens ws://localhost:$PORT/v1/realtime?model=$MODEL with Authorization: Bearer $KEY, sends session.update (text output, pcm16 input, no turn detection), and for each of 4 turns appends about 25s of pcm16 speech (generated once with POST /v1/audio/speech, response_format: pcm), commits, and calls response.create, printing each response.done usage. Once the conversation crosses OpenAI's 1024-token cache threshold, cached_tokens_details reports a text/audio split. The script then adds up the session, prints what it costs at the model's cost map rates ((text - cached text) x input rate + (audio - cached audio) x audio rate + cached text x cache-read rate + cached audio x audio cache-read rate + output text x output rate), and reads the spend row back through GET /spend/logs

for MODEL in gpt-realtime-2 gpt-realtime-2.1-mini gpt-realtime-mini; do python realtime_session.py $PORT $KEY speech.pcm 4 $MODEL; done

realtime_session.py

import asyncio, base64, json, sys, time, urllib.request
from datetime import datetime, timezone

port, key, audio_path, turns, model = sys.argv[1], sys.argv[2], sys.argv[3], int(sys.argv[4]), sys.argv[5]
pcm = open(audio_path, "rb").read()
RATES = {
    "gpt-realtime-2": {"text": 4e-06, "audio": 3.2e-05, "cache_text": 4e-07, "cache_audio": 4e-07, "out_text": 2.4e-05},
    "gpt-realtime-2.1-mini": {"text": 6e-07, "audio": 1e-05, "cache_text": 6e-08, "cache_audio": 3e-07, "out_text": 2.4e-06},
    "gpt-realtime-mini": {"text": 6e-07, "audio": 1e-05, "cache_text": 6e-08, "cache_audio": 3e-07, "out_text": 2.4e-06},
}


async def run():
    import websockets

    url = f"ws://localhost:{port}/v1/realtime?model={model}"
    totals = {"text": 0, "audio": 0, "cached": 0, "cached_text": 0, "cached_audio": 0, "out_text": 0, "out_audio": 0}
    async with websockets.connect(url, additional_headers={"Authorization": f"Bearer {key}"}, max_size=None) as ws:
        await ws.send(json.dumps({"type": "session.update", "session": {"type": "realtime", "output_modalities": ["text"], "audio": {"input": {"format": {"type": "audio/pcm", "rate": 24000}, "turn_detection": None}}}}))
        for turn in range(1, turns + 1):
            for i in range(0, len(pcm), 96000):
                await ws.send(json.dumps({"type": "input_audio_buffer.append", "audio": base64.b64encode(pcm[i:i + 96000]).decode()}))
            await ws.send(json.dumps({"type": "input_audio_buffer.commit"}))
            await ws.send(json.dumps({"type": "response.create"}))
            while True:
                ev = json.loads(await ws.recv())
                if ev["type"] == "error":
                    print("error:", json.dumps(ev)[:400]); return
                if ev["type"] == "response.done":
                    u = ev["response"]["usage"]
                    d, o = u["input_token_details"], u["output_token_details"]
                    ct = d.get("cached_tokens_details") or {}
                    print(f"turn {turn}: input={u['input_tokens']} text={d['text_tokens']} audio={d['audio_tokens']} cached={d['cached_tokens']} cached_tokens_details={ct} output={u['output_tokens']}")
                    totals["text"] += d["text_tokens"]; totals["audio"] += d["audio_tokens"]; totals["cached"] += d["cached_tokens"]
                    totals["cached_text"] += ct.get("text_tokens", 0); totals["cached_audio"] += ct.get("audio_tokens", 0)
                    totals["out_text"] += o.get("text_tokens", 0); totals["out_audio"] += o.get("audio_tokens", 0)
                    break
    print(f"session totals: text={totals['text']} audio={totals['audio']} cached={totals['cached']} (cached text={totals['cached_text']}, cached audio={totals['cached_audio']}) output text={totals['out_text']} audio={totals['out_audio']}")
    return totals


started = datetime.now(timezone.utc)
totals = asyncio.run(run())
if totals is None:
    sys.exit(1)
r = RATES[model]
expected = ((totals["text"] - totals["cached_text"]) * r["text"] + (totals["audio"] - totals["cached_audio"]) * r["audio"] + totals["cached_text"] * r["cache_text"] + totals["cached_audio"] * r["cache_audio"] + totals["out_text"] * r["out_text"])
print(f"expected cost at the cost map rates for {model} (text ${r['text']*1e6:g}/M, audio ${r['audio']*1e6:g}/M, cached text ${r['cache_text']*1e6:g}/M, cached audio ${r['cache_audio']*1e6:g}/M, output text ${r['out_text']*1e6:g}/M): {round(expected, 7)}")
deadline = time.time() + 90
while time.time() < deadline:
    req = urllib.request.Request(f"http://localhost:{port}/spend/logs", headers={"Authorization": f"Bearer {key}"})
    rows = [r_ for r_ in json.load(urllib.request.urlopen(req)) if datetime.fromisoformat(r_["startTime"].replace("Z", "+00:00")) >= started and model in r_["model"]]
    if rows:
        for r_ in rows:
            print(f"GET /spend/logs (rows since this session started) -> model={r_['model']} spend={r_['spend']} prompt_tokens={r_['prompt_tokens']} completion_tokens={r_['completion_tokens']} request_id={r_['request_id']}")
        break
    time.sleep(3)
else:
    print("GET /spend/logs -> no rows for this session after 90s")

To see the same rows in the Admin UI, open http://localhost:$PORT/ui/logs, log in as admin with the master key, and click the gpt-realtime-2, gpt-realtime-2.1-mini, or gpt-realtime-mini row; the detail pane shows Cost and Prompt Cache Read Tokens

gpt-realtime-2, before (c134fb7)

turn 1: input=443 text=126 audio=317 cached=0 cached_tokens_details={'text_tokens': 0, 'audio_tokens': 0, 'image_tokens': 0} output=312
turn 2: input=1009 text=375 audio=634 cached=0 cached_tokens_details={'text_tokens': 0, 'audio_tokens': 0, 'image_tokens': 0} output=252
turn 3: input=1518 text=567 audio=951 cached=1024 cached_tokens_details={'text_tokens': 384, 'audio_tokens': 640, 'image_tokens': 0} output=254
turn 4: input=2030 text=762 audio=1268 cached=1536 cached_tokens_details={'text_tokens': 576, 'audio_tokens': 960, 'image_tokens': 0} output=208
session totals: text=1830 audio=3170 cached=2560 (cached text=960, cached audio=1600) output text=1026 audio=0
expected cost at the cost map rates for gpt-realtime-2 (text $4/M, audio $32/M, cached text $0.4/M, cached audio $0.4/M, output text $24/M): 0.079368
GET /spend/logs (rows since this session started) -> model=openai/gpt-realtime-2 spend=0.103728 prompt_tokens=5000 completion_tokens=1026 request_id=b070841a-9ccb-49d8-bdab-ec97fe54a9bc

Logged spend is $0.103728 against $0.079368 at the cost map rates, $0.02436 (31%) over. With the cached text/audio split dropped, all 2560 cached tokens are subtracted from the 1830 text tokens first and the 730 remainder from audio, so 2440 audio tokens bill at $32/M where only 1570 were uncached

Logs page and the row's detail pane:

gpt-realtime-2, after (a94c060)

turn 1: input=443 text=126 audio=317 cached=0 cached_tokens_details={'text_tokens': 0, 'audio_tokens': 0, 'image_tokens': 0} output=399
turn 2: input=1064 text=430 audio=634 cached=448 cached_tokens_details={'text_tokens': 128, 'audio_tokens': 320, 'image_tokens': 0} output=255
turn 3: input=1577 text=626 audio=951 cached=1088 cached_tokens_details={'text_tokens': 448, 'audio_tokens': 640, 'image_tokens': 0} output=236
turn 4: input=2107 text=839 audio=1268 cached=1600 cached_tokens_details={'text_tokens': 640, 'audio_tokens': 960, 'image_tokens': 0} output=212
session totals: text=2021 audio=3170 cached=3136 (cached text=1216, cached audio=1920) output text=1102 audio=0
expected cost at the cost map rates for gpt-realtime-2 (text $4/M, audio $32/M, cached text $0.4/M, cached audio $0.4/M, output text $24/M): 0.0709224
GET /spend/logs (rows since this session started) -> model=openai/gpt-realtime-2 spend=0.0709224 prompt_tokens=5191 completion_tokens=1102 request_id=dea5e569-65a4-4279-993e-87896cd906d9

Logged spend equals the cost map rate cost to the last digit: 805 non-cached text at $4/M, 1250 non-cached audio at $32/M, 3136 cached at $0.40/M, 1102 output text at $24/M

gpt-realtime-2.1-mini, before (c134fb7)

turn 1: input=443 text=126 audio=317 cached=0 cached_tokens_details={'text_tokens': 0, 'audio_tokens': 0, 'image_tokens': 0} output=218
turn 2: input=907 text=273 audio=634 cached=448 cached_tokens_details={'text_tokens': 128, 'audio_tokens': 320, 'image_tokens': 0} output=210
turn 3: input=1390 text=439 audio=951 cached=896 cached_tokens_details={'text_tokens': 256, 'audio_tokens': 640, 'image_tokens': 0} output=179
turn 4: input=1854 text=586 audio=1268 cached=1408 cached_tokens_details={'text_tokens': 448, 'audio_tokens': 960, 'image_tokens': 0} output=222
session totals: text=1424 audio=3170 cached=2752 (cached text=832, cached audio=1920) output text=829 audio=0
expected cost at the cost map rates for gpt-realtime-2.1-mini (text $0.6/M, audio $10/M, cached text $0.06/M, cached audio $0.3/M, output text $2.4/M): 0.0154707
GET /spend/logs (rows since this session started) -> model=openai/gpt-realtime-2.1-mini spend=0.02057472 prompt_tokens=4594 completion_tokens=829 request_id=c9fe4048-bce5-4c4b-82c4-b616042d8504

Logged spend is $0.02057472 against $0.0154707 at the cost map rates, 33% over: the 2752 cached tokens are subtracted from the 1424 text tokens first and the 1328 remainder from audio, so 1842 audio tokens bill at $10/M where only 1250 were uncached, and every cached token bills at the $0.06/M text cache-read rate

gpt-realtime-2.1-mini, after (a94c060)

turn 1: input=443 text=126 audio=317 cached=0 cached_tokens_details={'text_tokens': 0, 'audio_tokens': 0, 'image_tokens': 0} output=448
turn 2: input=1103 text=469 audio=634 cached=0 cached_tokens_details={'text_tokens': 0, 'audio_tokens': 0, 'image_tokens': 0} output=255
turn 3: input=1634 text=683 audio=951 cached=1088 cached_tokens_details={'text_tokens': 448, 'audio_tokens': 640, 'image_tokens': 0} output=173
turn 4: input=2076 text=808 audio=1268 cached=1664 cached_tokens_details={'text_tokens': 704, 'audio_tokens': 960, 'image_tokens': 0} output=236
session totals: text=2086 audio=3170 cached=2752 (cached text=1152, cached audio=1600) output text=1112 audio=0
expected cost at the cost map rates for gpt-realtime-2.1-mini (text $0.6/M, audio $10/M, cached text $0.06/M, cached audio $0.3/M, output text $2.4/M): 0.0194783
GET /spend/logs (rows since this session started) -> model=openai/gpt-realtime-2.1-mini spend=0.01947832 prompt_tokens=5256 completion_tokens=1112 request_id=40f940a0-864e-42a3-ae71-a5a72ad555a0

Logged spend $0.01947832 is the cost map rate cost (the expected line rounds it to 7 places): 934 non-cached text at $0.60/M, 1570 non-cached audio at $10/M, 1152 cached text at $0.06/M, 1600 cached audio at $0.30/M, 1112 output text at $2.40/M. Without the get_model_info change the same session bills cached audio at the $0.06/M text cache-read rate

gpt-realtime-mini, before (c134fb7)

turn 1: input=436 text=119 audio=317 cached=0 cached_tokens_details={'text_tokens': 0, 'audio_tokens': 0, 'image_tokens': 0} output=164
turn 2: input=927 text=293 audio=634 cached=576 cached_tokens_details={'text_tokens': 256, 'audio_tokens': 320, 'image_tokens': 0} output=140
turn 3: input=1394 text=443 audio=951 cached=0 cached_tokens_details={'text_tokens': 0, 'audio_tokens': 0, 'image_tokens': 0} output=131
turn 4: input=1852 text=584 audio=1268 cached=1536 cached_tokens_details={'text_tokens': 576, 'audio_tokens': 960, 'image_tokens': 0} output=177
session totals: text=1439 audio=3170 cached=2112 (cached text=832, cached audio=1280) output text=612 audio=0
expected cost at the cost map rates for gpt-realtime-mini (text $0.6/M, audio $10/M, cached text $0.06/M, cached audio $0.3/M, output text $2.4/M): 0.0211669
GET /spend/logs (rows since this session started) -> model=openai/gpt-realtime-mini spend=0.0264388 prompt_tokens=4609 completion_tokens=612 request_id=656aef21-a4b0-46bb-afeb-da120b66330c

Logged spend is $0.0264388 against $0.0211669 at the cost map rates, 25% over. The old cost map entry had no cache-read rate at all, so all 2112 cached tokens bill at $0, while the split-less subtraction still charges 2497 audio tokens at $10/M where only 1890 were uncached: 2497 x $10/M + 612 output x $2.40/M = $0.0264388 exactly

gpt-realtime-mini, after (a94c060)

turn 1: input=436 text=119 audio=317 cached=64 cached_tokens_details={'text_tokens': 64, 'audio_tokens': 0, 'image_tokens': 0} output=138
turn 2: input=901 text=267 audio=634 cached=576 cached_tokens_details={'text_tokens': 256, 'audio_tokens': 320, 'image_tokens': 0} output=108
turn 3: input=1336 text=385 audio=951 cached=1024 cached_tokens_details={'text_tokens': 384, 'audio_tokens': 640, 'image_tokens': 0} output=109
turn 4: input=1772 text=504 audio=1268 cached=1408 cached_tokens_details={'text_tokens': 448, 'audio_tokens': 960, 'image_tokens': 0} output=148
session totals: text=1275 audio=3170 cached=3072 (cached text=1152, cached audio=1920) output text=503 audio=0
expected cost at the cost map rates for gpt-realtime-mini (text $0.6/M, audio $10/M, cached text $0.06/M, cached audio $0.3/M, output text $2.4/M): 0.0144261
GET /spend/logs (rows since this session started) -> model=openai/gpt-realtime-mini spend=0.01442612 prompt_tokens=4445 completion_tokens=503 request_id=9bd46713-5be2-4da4-a31d-44b7918e29ab

Logged spend $0.01442612 is the cost map rate cost with the entry's new $0.06/M cache-read rate: 123 non-cached text at $0.60/M, 1250 non-cached audio at $10/M, 1152 cached text at $0.06/M, 1920 cached audio at $0.30/M, 503 output text at $2.40/M

GET /model/info on both proxies (same config, same request) differs only in the touched keys: gpt-realtime-mini reports cache_read_input_token_cost 6e-08 and cache_read_input_audio_token_cost 3e-07 on the after proxy where the before proxy reported neither, and gpt-realtime-2 and gpt-realtime-2.1-mini gain cache_read_input_audio_token_cost (4e-07 and 3e-07); input, audio, and output rates are identical on every model

Observed alongside the fix:

• OpenAI cache hits vary per run; leg totals differ
• Some turns report cached=0 mid-session; provider behavior
• gpt-realtime-mini after turn 1 reused an earlier session's cache
• UI shows cached total only, no text/audio split; unchanged

Type

🐛 Bug Fix

Caveats (if any)

Medium

• When a provider's nested cached_tokens_details add up to more than cached_tokens, parse_prompt_tokens_details caps them by clipping audio first, then text, then image. Nothing in the OpenAI usage contract picks that order and the PR does not argue for it; the alternatives are scaling every modality down proportionally, trusting the nested counts over the total, or dropping the split and billing all cached tokens at the text cache-read rate when the two disagree. No response in the QA runs hit this branch, so it only changes billing on a self-inconsistent usage payload; the order is an open decision for the author to confirm or change

Low

• Cached image tokens bill at the generic cache-read rate; no image-specific cache key exists
• Non-OpenAI realtime providers do not emit cached_tokens_details, so their billing is unchanged
• A session where only some response.done events carry cached_tokens_details keeps the split from the events that did and bills the rest of the cached total at the text cache-read rate, which is the pre-fix billing for those tokens; OpenAI sent the split on every response.done in the QA legs (12 of 12) and test_realtime_combine_keeps_cached_split_when_only_one_usage_has_details pins both orders
• 76ae35d moves CachedTokensDetails into litellm.types.llms.base to clear the two CodeQL cyclic-import alerts, adds the test above, and commits the regenerated schema.d.ts (two soft_budget docstring lines that drifted on main); it cannot change behavior, so the QA legs and the /live-pr-risk pass at a94c060 carry over
• 6f7882d keeps the BaseLiteLLMOpenAIResponseObject import line in litellm/types/llms/openai.py byte-identical to main and pulls CachedTokensDetails in on its own relative import. CodeQL resolves from openai import Omit in that file to the module itself, so every importer of a name whose definition line is in the diff gets a py/unsafe-cyclic-import alert; the 76ae35d edit to that line produced two such alerts at unchanged files (litellm/types/google_genai/main.py:4 and litellm/proxy/hooks/parallel_request_limiter_v3.py:63), and main itself reports none. The commit cannot change behavior either
• get_model_info() and GET /model/info now carry cache_read_input_audio_token_cost for every model, null where the cost map has no such rate
• Every /v1/responses usage now carries input_tokens_details.cached_tokens_details, null when the provider reports no split; additive, the existing fields are unchanged
• azure/gpt-realtime-2025-08-28 and azure/gpt-realtime-1.5-2026-02-23 cached text drops from $4/M to $0.40/M, so logged spend for cached prompts on those Azure deployments goes down; the old value equalled the full input rate and Azure's retail meter lists $0.40/M
• The Azure realtime entries were not driven live (no Azure realtime deployment was available to this run); their fills are covered by the parametrized cost test and the retail meters above, and the live legs cover the OpenAI entries only
• CircleCI ocr_testing is red at this tip because the job collects zero tests: .circleci/config.yml still names tests/test_litellm/ocr/test_rust_bridge.py, which refactor(ocr): complete native lifecycle and preserve Azure auth #40734 deleted and that commit is in this PR's merge base. The base branch's own scheduled pipeline fails the same way and a follow-up tracks the config fix; nothing in this PR touches OCR or the CI config. CircleCI upload-coverage runs only after every test job passes, so it stays not run behind that failure
• GitHub Actions proxy-behavior (Postgres Tests) is red at this tip on tests/proxy_behavior/auth/test_auth_object_prefetch.py::test_join_binds_the_membership_to_the_requested_team, which awaits a MagicMock. main's own Postgres Tests runs at the merge base fail the same way (34737619378 and 34737299659); nothing in this PR touches auth or that test, and the check is not in main's required set

Final Attestation

• 5737cab passes /live-pr-risk

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Medium Risk
Changes billing and spend logging for realtime and any usage with cached_tokens_details; incorrect caps when nested details exceed cached_tokens could mis-price edge-case payloads.

Overview
Fixes overbilling on OpenAI/Azure realtime when prompt cache hits include audio: the provider’s cached_tokens_details split is now kept end-to-end and used in cost math instead of treating all cached tokens as text.

Adds CachedTokensDetails on prompt_tokens_details / input_tokens_details and threads it through Responses/realtime usage transforms, usage merging (_combine_prompt_tokens_details), and logging. parse_prompt_tokens_details derives capped cached text/audio/image counts and bills non-cached text/audio/image at full input rates. Cache-read cost splits: non-audio cached tokens at cache_read_input_token_cost, cached audio at cache_read_input_audio_token_cost (fallback to text cache-read). BilledTokenRates, get_model_info, and breakdown helpers expose the audio cache-read rate.

Updates model cost maps for gpt-realtime* / azure/gpt-realtime* (new or corrected cache_read_input_* keys, including fixing Azure entries that used the full input rate for cache read).

^{Reviewed by Cursor Bugbot for commit 6f7882d. Bugbot is set up for automated code reviews on this repo. Configure here.}

Link to Devin session: https://app.devin.ai/sessions/02c812fd9fa940c4baa2543d662d6754
Open in Devin Desktop: https://app.devin.ai/desktop/session/02c812fd9fa940c4baa2543d662d6754?variant=devin
Requested by: @shivamrawat1

• a94c060 passes /live-pr-risk

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_fix_realtime_cached_audio_cost (6f7882d) with main (30f33a9)}

[greptile-apps]

Greptile Summary

This PR preserves cached-token modality details throughout realtime usage processing and applies the appropriate text and audio cache-read rates during cost calculation.

• Aggregates nested cached-token details across realtime events.
• Separates cached audio from full-rate audio billing and cost breakdowns.
• Exposes audio cache-read rates through model metadata.
• Updates OpenAI and Azure realtime pricing entries and adds regression coverage.
• The only change since the previous review adjusts an internal import without changing runtime behavior.

Confidence Score: 5/5

The PR appears safe to merge, with no outstanding previous findings or actionable regressions introduced since the previous review.

All previous threads are resolved, including the fully addressed cached-token capping, comment, and post-construction mutation findings; the parameter-mutation finding was correctly withdrawn. The only subsequent change switches one package import from absolute to relative syntax, which resolves to the same module and introduces no loading or cycle issue.

Important Files Changed

Filename	Overview
litellm/cost_calculator.py	Aggregates cached-token modality details when combining realtime usage.
litellm/litellm_core_utils/llm_cost_calc/utils.py	Separates cached audio from uncached audio and bills it at the configured audio cache-read rate.
litellm/responses/litellm_completion_transformation/transformation.py	Preserves nested cached-token details while transforming usage into Responses API structures.
litellm/types/llms/openai.py	Defines realtime cached-token detail typing; the post-review import adjustment is behaviorally equivalent.
model_prices_and_context_window.json	Adds and corrects cache-read pricing for affected OpenAI and Azure realtime models.

_{Reviews (15): Last reviewed commit: "fix(types): import CachedTokensDetails o..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[devin-ai-integration]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

@greptileai

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Bugbot couldn't run

Something went wrong. Try again by commenting "Cursor review" or "bugbot run", or contact support (requestId: serverGenReqId_569613cf-ab77-4823-8948-1ce0bef81c39).

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 6f7882d. Configure here.}

[mateo-berri]

LGTM