[Bug]: cost_breakdown.cache_read_cost / cache_creation_cost always null for OpenAI Responses API (only read from Anthropic-style top-level usage keys)

[sarveshb-p72]

Check for existing issues

• I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

What happened

On the OpenAI Responses API (/v1/responses) path, the total response_cost is computed correctly (cache-read and cache-write dollars are folded into prompt_tokens_cost), but the itemized StandardLoggingPayload.response_cost_breakdown fields cache_read_cost and cache_creation_cost are always None/null.

Any downstream consumer of the standard logging payload (S3, a data warehouse, cost dashboards) therefore cannot attribute spend to cache-read vs cache-write vs fresh input for OpenAI models — all prompt-side dollars appear only under input_cost. This is an attribution/observability gap; the grand total is unaffected.

Root cause

In litellm/cost_calculator.py, the breakdown-itemization block (v1.89.1, around lines 1605-1631) only derives the cache token counts from Anthropic-style top-level usage attributes:

_cr = getattr(cost_per_token_usage_object, "cache_read_input_tokens", None) \
    or (cost_per_token_usage_object.model_extra or {}).get("cache_read_input_tokens")
_cc = getattr(cost_per_token_usage_object, "cache_creation_input_tokens", None) \
    or (cost_per_token_usage_object.model_extra or {}).get("cache_creation_input_tokens")
if (_cr or _cc) and model:
    _mi = litellm.get_model_info(model=model, custom_llm_provider=custom_llm_provider)
    _cr_rate = _mi.get("cache_read_input_token_cost")
    if _cr and _cr_rate is not None:
        _cache_read_cost = float(_cr) * float(_cr_rate)
    _cc_rate = _mi.get("cache_creation_input_token_cost")
    if _cc and _cc_rate is not None:
        _cache_creation_cost = float(_cc) * float(_cc_rate)

OpenAI's Responses API does NOT set cache_read_input_tokens / cache_creation_input_tokens. It reports cache tokens under usage.input_tokens_details:

• cache-read -> prompt_tokens_details.cached_tokens
• cache-write -> prompt_tokens_details.cache_write_tokens

So both _cr and _cc resolve to None, the pricing branch is skipped, and _store_cost_breakdown_in_logging_obj(...) is called with cache_read_cost=None, cache_creation_cost=None -> the fields serialize as null in the StandardLoggingPayload.

Note this is consistent with how the total is computed: _parse_prompt_tokens_details DOES read cached_tokens (and, with the cache-write mapping, cache_creation_tokens) from prompt_tokens_details, so the dollars are correctly in the total — they're just never surfaced in the itemized breakdown for OpenAI.

Expected behavior

For OpenAI (and any provider that reports cache tokens via prompt_tokens_details), the breakdown should itemize cache_read_cost and cache_creation_cost from those counts, using the same cache_read_input_token_cost / cache_creation_input_token_cost rates.

Suggested fix

Extend the itemization block to fall back to prompt_tokens_details when the Anthropic-style top-level keys are absent:

if _cr is None:
    _cr = getattr(prompt_tokens_details, "cached_tokens", None)
if _cc is None:
    _cc = getattr(prompt_tokens_details, "cache_write_tokens", None) \
       or getattr(prompt_tokens_details, "cache_creation_tokens", None)

Semantics should match the existing Anthropic path: input_cost stays the FULL prompt-side cost, and cache_read_cost / cache_creation_cost are additive breakouts that overlap it (not a disjoint partition) — so the total is unchanged.

Environment

• litellm v1.89.1
• OpenAI Responses API (/v1/responses); models emitting input_tokens_details.{cached_tokens, cache_write_tokens} (e.g. GPT-5.x family)

Steps to Reproduce

1. Configure an OpenAI model with cache pricing in model_info:

   • model_name: gpt-cache-test
     litellm_params:
     model: openai/gpt-5.1 # any OpenAI model that emits cache tokens
     model_info:
     input_cost_per_token: 0.000005
     output_cost_per_token: 0.00003
     cache_read_input_token_cost: 0.0000005
     cache_creation_input_token_cost: 0.00000625

2. Send a Responses API request large enough to trigger a cache write, then an identical one to trigger a cache read. The provider usage will contain:

   "input_tokens_details": { "cache_write_tokens": 4012, "cached_tokens": 0 } // write
   "input_tokens_details": { "cache_write_tokens": 0, "cached_tokens": 4012 } // read

3. Inspect the StandardLoggingPayload / response_cost_breakdown:

   • Actual: cache_read_cost = null, cache_creation_cost = null; all prompt-side spend in input_cost.
   • Expected: cache_read_cost = 4012 * 0.0000005, cache_creation_cost = 4012 * 0.00000625, non-null.
   • total_cost is correct in both cases.

Relevant log output

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on ?

1.89.1

Twitter / LinkedIn details

No response

[sarveshb-p72]

Tests 1 -

`

from future import annotations

import json
import time
import uuid

import httpx
import pytest
from base_integration_test import (
BaseLiteLLMIntegrationTest,
get_mock_s3_base_url,
use_mock_models,
)

MODEL = "openai-fake-cache-write"

Base-tier rates; must match the model_info in local-litellm/litellm-config.yaml.

INPUT_RATE = 5.0e-6
CACHE_READ_RATE = 5.0e-7
CACHE_WRITE_RATE = 6.25e-6
OUTPUT_RATE = 3.0e-5

FRESH = 1000
CACHED = 4000
CACHE_WRITE = 5000
OUTPUT = 200

OpenAI input_tokens is inclusive of cached + cache-write tokens.

INPUT_TOKENS = FRESH + CACHED + CACHE_WRITE

def _marker(nonce: str) -> str:
payload = {
"input_tokens": INPUT_TOKENS,
"cached_tokens": CACHED,
"cache_write_tokens": CACHE_WRITE,
"output_tokens": OUTPUT,
}
return f"{nonce} MOCK_USAGE{json.dumps(payload)}"

@pytest.mark.skipif(
not use_mock_models(),
reason="Uses the openai-fake-cache-write mock model + mock S3; requires the local stack.",
)
class TestCacheCostItemization(BaseLiteLLMIntegrationTest):
def _served(self) -> bool:
return MODEL in self._fetch_served_model_names()

def _clear_s3_callbacks(self) -> None:
    resp = httpx.delete(f"{get_mock_s3_base_url()}/mock-s3/callbacks")
    assert resp.status_code == 200, f"Failed to clear callbacks: {resp.text}"

def _get_s3_callbacks(self) -> list[dict]:
    resp = httpx.get(f"{get_mock_s3_base_url()}/mock-s3/callbacks", timeout=30)
    assert resp.status_code == 200, f"Failed to get callbacks: {resp.text}"
    return resp.json().get("callbacks", [])

def _find_payload_for(self, nonce: str) -> dict | None:
    """Locate the success StandardLoggingPayload whose request carried our nonce.
    The unique nonce in the request input is the reliable key; the payload's
    ``model`` carries the provider prefix (e.g. ``openai/openai-fake-cache-write``),
    so match the model by suffix rather than exact equality.
    """
    for record in self._get_s3_callbacks():
        content = record.get("content", {})
        model = content.get("model") or ""
        if not model.endswith(MODEL):
            continue
        if nonce in json.dumps(content.get("messages", "")):
            return content
    return None

def _wait_for_payload(self, nonce: str, timeout_secs: int = 60) -> dict:
    deadline = time.time() + timeout_secs
    last = None
    while time.time() < deadline:
        last = self._find_payload_for(nonce)
        if last is not None:
            return last
        time.sleep(2)
    raise AssertionError(
        f"No S3 StandardLoggingPayload captured for nonce {nonce} within {timeout_secs}s",
    )

def test_responses_cache_costs_itemized_in_s3_payload(self):
    if not self._served():
        pytest.skip(f"{MODEL} not served by this proxy")

    nonce = uuid.uuid4().hex
    self._clear_s3_callbacks()

    raw = self.openai_client.responses.with_raw_response.create(
        model=MODEL,
        input=_marker(nonce),
        max_output_tokens=OUTPUT,
        timeout=self.get_timeout_secs(MODEL),
    )
    assert raw.status_code == 200, f"Request failed: {raw.status_code} - {raw.text}"

    payload = self._wait_for_payload(nonce)
    breakdown = payload.get("cost_breakdown")
    assert breakdown is not None, (
        f"cost_breakdown missing from StandardLoggingPayload: keys={list(payload)}"
    )

    expected_cache_read = CACHED * CACHE_READ_RATE
    expected_cache_creation = CACHE_WRITE * CACHE_WRITE_RATE

    # The GEN-8478 fix: these break-out fields are non-null on /responses.
    assert breakdown.get("cache_read_cost") == pytest.approx(
        expected_cache_read,
        rel=1e-4,
    ), (
        f"cache_read_cost not itemized in S3 payload: got "
        f"{breakdown.get('cache_read_cost')!r}, expected {expected_cache_read} "
        f"(pre-patch: null). breakdown={breakdown}"
    )
    assert breakdown.get("cache_creation_cost") == pytest.approx(
        expected_cache_creation,
        rel=1e-4,
    ), (
        f"cache_creation_cost not itemized in S3 payload: got "
        f"{breakdown.get('cache_creation_cost')!r}, expected {expected_cache_creation} "
        f"(pre-patch: null). breakdown={breakdown}"
    )

    # input_cost stays the full folded prompt-side cost (annotation convention,
    # consistent with the Anthropic path and downstream silver).
    expected_input_cost_full = (
        FRESH * INPUT_RATE
        + CACHED * CACHE_READ_RATE
        + CACHE_WRITE * CACHE_WRITE_RATE
    )
    assert breakdown.get("input_cost") == pytest.approx(
        expected_input_cost_full,
        rel=1e-4,
    ), (
        f"input_cost should remain the full prompt-side cost: got "
        f"{breakdown.get('input_cost')!r}, expected {expected_input_cost_full}"
    )

`

Tests 2 -
`
import time

import litellm
import pytest
from litellm.cost_calculator import completion_cost
from litellm.litellm_core_utils.litellm_logging import Logging
from litellm.types.utils import (
ModelResponse,
PromptTokensDetailsWrapper,
Usage,
)

MODEL = "openai/gpt-5.6-cacheitemize-test"

Standard sol-style rates: input $5/M, output $30/M, cache-read $0.5/M, cache-write $6.25/M

INPUT_RATE = 5e-6
OUTPUT_RATE = 30e-6
CACHE_READ_RATE = 5e-7
CACHE_WRITE_RATE = 6.25e-6

@pytest.fixture(autouse=True)
def _register_model():
litellm.register_model(
{
MODEL: {
"litellm_provider": "openai",
"mode": "chat",
"input_cost_per_token": INPUT_RATE,
"output_cost_per_token": OUTPUT_RATE,
"cache_read_input_token_cost": CACHE_READ_RATE,
"cache_creation_input_token_cost": CACHE_WRITE_RATE,
},
},
)
yield

def _new_logging_obj() -> Logging:
"""A minimal live Logging object so completion_cost can attach cost_breakdown."""
return Logging(
model=MODEL,
messages=[{"role": "user", "content": "hi"}],
stream=False,
call_type="completion",
start_time=time.time(),
litellm_call_id="gen-8478-itemize-test",
function_id="gen-8478",
)

def _openai_usage(cached: int, cache_write: int, fresh: int, out: int) -> Usage:
"""OpenAI Responses-API style usage.
cache-read under prompt_tokens_details.cached_tokens
cache-write under prompt_tokens_details.cache_write_tokens (mapped to
cache_creation_tokens by the responses transform in the deployed image)
"""
ptd = PromptTokensDetailsWrapper(cached_tokens=cached)
# Deployed image maps cache_write_tokens -> cache_creation_tokens on the wrapper
# (GEN-8280). Set the mapped field so this test is independent of that transform.
setattr(ptd, "cache_write_tokens", cache_write)
setattr(ptd, "cache_creation_tokens", cache_write)
return Usage(
prompt_tokens=cached + cache_write + fresh,
completion_tokens=out,
total_tokens=cached + cache_write + fresh + out,
prompt_tokens_details=ptd,
)

def _response_with_usage(usage: Usage) -> ModelResponse:
resp = ModelResponse()
resp.model = MODEL
resp.usage = usage # type: ignore[attr-defined]
return resp

def test_openai_cache_costs_itemized_in_breakdown():
"""GREEN after patch: cost_breakdown itemizes OpenAI cache-read and cache-write dollars.
RED before patch: the itemization block reads only usage.cache_read_input_tokens /
cache_creation_input_tokens (Anthropic naming), which OpenAI doesn't set, so both
cache fields are absent from cost_breakdown (NULL downstream).
"""
cached = 16000
cache_write = 26022
fresh = 100
out = 50
usage = _openai_usage(cached, cache_write, fresh, out)
logging_obj = _new_logging_obj()

total = completion_cost(
    completion_response=_response_with_usage(usage),
    model=MODEL,
    custom_llm_provider="openai",
    litellm_logging_obj=logging_obj,
)

breakdown = logging_obj.cost_breakdown
assert breakdown is not None, "cost_breakdown was never populated"

expected_cache_read = cached * CACHE_READ_RATE
expected_cache_creation = cache_write * CACHE_WRITE_RATE

assert breakdown.get("cache_read_cost") == pytest.approx(
    expected_cache_read,
    rel=1e-6,
), (
    f"cache_read_cost not itemized: got {breakdown.get('cache_read_cost')!r}, "
    f"expected {expected_cache_read} (pre-patch: None -> NULL downstream)"
)
assert breakdown.get("cache_creation_cost") == pytest.approx(
    expected_cache_creation,
    rel=1e-6,
), (
    f"cache_creation_cost not itemized: got {breakdown.get('cache_creation_cost')!r}, "
    f"expected {expected_cache_creation} (pre-patch: None -> NULL downstream)"
)
# Total is unaffected by itemization.
assert total == pytest.approx(breakdown["total_cost"], rel=1e-6)

def test_openai_itemization_matches_anthropic_semantics():
"""OpenAI itemization must follow the SAME convention as the working Anthropic path:
input_cost remains the full folded prompt-side cost (it includes cache dollars),
and cache_read_cost / cache_creation_cost are additive break-out annotations on
top of it. This is the convention downstream Databricks silver already assumes; the fix
must not turn input_cost into a fresh-only partition (that would double-shift dollars
vs every existing Anthropic row).
RED before patch: the two cache break-out fields are absent for OpenAI (only present on
the Anthropic top-level path).
"""
cached = 16000
cache_write = 26022
fresh = 100
out = 50
usage = _openai_usage(cached, cache_write, fresh, out)
logging_obj = _new_logging_obj()

completion_cost(
    completion_response=_response_with_usage(usage),
    model=MODEL,
    custom_llm_provider="openai",
    litellm_logging_obj=logging_obj,
)
b = logging_obj.cost_breakdown
assert b is not None

# input_cost stays the full prompt-side spend (fresh + cache-read + cache-write folded),
# exactly as the pre-existing behaviour and the Anthropic path both produce.
expected_input_cost_full = (
    fresh * INPUT_RATE + cached * CACHE_READ_RATE + cache_write * CACHE_WRITE_RATE
)
assert b.get("input_cost") == pytest.approx(expected_input_cost_full, rel=1e-6), (
    "input_cost must remain the full prompt-side cost (cache dollars folded in), "
    "matching the Anthropic path and silver's convention"
)
# The break-out annotations overlap input_cost (they are NOT disjoint from it).
assert b.get("cache_read_cost") == pytest.approx(cached * CACHE_READ_RATE, rel=1e-6)
assert b.get("cache_creation_cost") == pytest.approx(
    cache_write * CACHE_WRITE_RATE,
    rel=1e-6,
)
# Total is unchanged by itemization.
assert (b["input_cost"] + b.get("output_cost", 0.0)) == pytest.approx(
    b["total_cost"],
    rel=1e-6,
), "total_cost must equal input_cost + output_cost (cache fields are annotations)"

def test_anthropic_style_itemization_still_works():
"""Regression guard: Anthropic-style usage (top-level cache_*_input_tokens) still itemizes."""
cache_read_tokens = 2000
cache_creation_tokens = 3000
fresh = 100
out = 50
usage = Usage(
prompt_tokens=cache_read_tokens + cache_creation_tokens + fresh,
completion_tokens=out,
total_tokens=cache_read_tokens + cache_creation_tokens + fresh + out,
prompt_tokens_details=PromptTokensDetailsWrapper(
cached_tokens=cache_read_tokens,
),
)
# Anthropic reports these as top-level usage attributes.
setattr(usage, "cache_read_input_tokens", cache_read_tokens)
setattr(usage, "cache_creation_input_tokens", cache_creation_tokens)
logging_obj = _new_logging_obj()

completion_cost(
    completion_response=_response_with_usage(usage),
    model=MODEL,
    custom_llm_provider="openai",
    litellm_logging_obj=logging_obj,
)
b = logging_obj.cost_breakdown
assert b is not None
assert b.get("cache_read_cost") == pytest.approx(
    cache_read_tokens * CACHE_READ_RATE,
    rel=1e-6,
)
assert b.get("cache_creation_cost") == pytest.approx(
    cache_creation_tokens * CACHE_WRITE_RATE,
    rel=1e-6,
)

`

[rs-03]

This looks like it was already resolved on current mainline. That inline itemization block no longer exists; response_cost_breakdown now derives cache costs through get_token_type_cost_breakdown in litellm/litellm_core_utils/llm_cost_calc/utils.py, which reads the cache-read count from prompt_tokens_details.cached_tokens and the cache-write count from prompt_tokens_details.cache_write_tokens (with a fallback to the top-level Anthropic-style counters), so the OpenAI Responses path is covered by the same source the total already uses

I checked against the current source with a registered model carrying cache_read_input_token_cost=5e-7 and cache_creation_input_token_cost=6.25e-6: a usage object with prompt_tokens_details.cached_tokens=4012 gives cache_read_cost=0.002006, and cache_write_tokens=4012 gives cache_creation_cost=0.025075, both matching count*rate and reconciled with the total. Could you confirm whether you still see null on the latest daily build rather than v1.89.1?

[IgorGanapolsky]

Reliability note

Production automation failures usually reduce to a contract issue before a model/prompt issue:

1. Success criteria — durable artifact id (message/booking/row), not only node green
2. Side-effect before complete — never blind-retry without an idempotency key on the business object
3. Tool vs path split — external action can succeed while the node/agent path still errors (silent-green / false-red)

If you have one failing execution + expected vs actual, reply with stack/version and I will send a free root-cause checklist.

[mateo-berri]

Fixed by #31686, first stable release v1.92.0. Requests with zero cache activity still serialize null cache fields by design. Verified at current HEAD, so closing as completed