fix(responses_adapter): read cache_read_tokens from input_tokens_details on Responses usage (#28354) - #28380
Conversation
…ils on Responses usage The Anthropic→Responses streaming adapter was reading usage.cache_read_input_tokens (an Anthropic-only field) and so always emitted cache_read_input_tokens=0 to Anthropic-format clients, even when OpenAI's underlying response correctly reported a non-zero cached_tokens. This broke observability for Anthropic-compatible clients (Claude Code, Claude Agent SDK, dashboards, Sentry, Langfuse, billing) routing OpenAI Responses traffic through LiteLLM. Read cached prompt tokens from usage.input_tokens_details.cached_tokens (parallel to the existing Chat Completions path's use of prompt_tokens_details.cached_tokens), fall back to the Anthropic-named field if the upstream provider already speaks Anthropic usage, and subtract cached from input_tokens to preserve Anthropic's 'uncached' semantics for the input_tokens count. Also drops the dead first assignment that stored input_tokens_details / output_tokens_details into the cache token variables before immediately overwriting them. Fixes BerriAI#28354
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Greptile SummaryThis PR fixes a bug where
Confidence Score: 4/5Safe to merge only after the subtraction-on-fallback bug is resolved; the OpenAI Responses path fix itself is correct. The OpenAI Responses path fix (reading
|
| Filename | Overview |
|---|---|
| litellm/llms/anthropic/experimental_pass_through/responses_adapters/streaming_iterator.py | Reads cache_read_tokens from usage.input_tokens_details.cached_tokens (OpenAI Responses path) with fallback to cache_read_input_tokens (Anthropic path), then subtracts cached tokens from input_tokens; the subtraction is applied unconditionally to both paths, which double-subtracts on Anthropic-native upstreams where input_tokens is already the uncached count (flagged in a prior review thread). |
Reviews (2): Last reviewed commit: "fix(responses_adapter): read cache_read_..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| # Fall back to Anthropic-style direct fields if the upstream | ||
| # provider already speaks Anthropic usage. | ||
| if not cache_read_tokens: | ||
| cache_read_tokens = int( | ||
| getattr(usage, "cache_read_input_tokens", 0) or 0 | ||
| ) | ||
| cache_creation_tokens = int( | ||
| getattr(usage, "cache_creation_input_tokens", 0) or 0 | ||
| ) | ||
| cache_read_tokens = int( | ||
| getattr(usage, "cache_read_input_tokens", 0) or 0 | ||
| ) | ||
| # Anthropic semantics: ``input_tokens`` is the *uncached* prompt | ||
| # count. OpenAI reports total prompt tokens (cached + uncached), | ||
| # so subtract to keep client-side accounting consistent. | ||
| if cache_read_tokens and input_tokens >= cache_read_tokens: | ||
| input_tokens = input_tokens - cache_read_tokens |
There was a problem hiding this comment.
input_tokens for Anthropic-native upstreams
When input_tokens_details is absent (Anthropic providers don't include it) and cache_read_input_tokens > 0, the fallback correctly captures cached tokens — but the unconditional subtraction that follows then reduces an input_tokens value that is already the uncached count (Anthropic semantics). For a request with 3 000 uncached tokens and 500 cached, the emitted input_tokens becomes 2 500 instead of 3 000.
The subtraction is only valid for the OpenAI Responses path (where input_tokens is the total prompt count). It should be skipped whenever the value came from the cache_read_input_tokens fallback.
| if usage is not None: | ||
| input_tokens = getattr(usage, "input_tokens", 0) or 0 | ||
| output_tokens = getattr(usage, "output_tokens", 0) or 0 | ||
| cache_creation_tokens = getattr(usage, "input_tokens_details", None) # type: ignore[assignment] | ||
| cache_read_tokens = getattr(usage, "output_tokens_details", None) # type: ignore[assignment] | ||
| # Prefer direct cache fields if present | ||
| input_tokens = int(getattr(usage, "input_tokens", 0) or 0) | ||
| output_tokens = int(getattr(usage, "output_tokens", 0) or 0) | ||
| # OpenAI Responses API exposes cached prompt tokens at | ||
| # ``usage.input_tokens_details.cached_tokens`` (parallel to | ||
| # ``usage.prompt_tokens_details.cached_tokens`` on Chat Completions). | ||
| itd = getattr(usage, "input_tokens_details", None) | ||
| if itd is not None: | ||
| if isinstance(itd, dict): | ||
| cache_read_tokens = int(itd.get("cached_tokens", 0) or 0) | ||
| else: | ||
| cache_read_tokens = int( | ||
| getattr(itd, "cached_tokens", 0) or 0 | ||
| ) | ||
| # Fall back to Anthropic-style direct fields if the upstream | ||
| # provider already speaks Anthropic usage. | ||
| if not cache_read_tokens: | ||
| cache_read_tokens = int( | ||
| getattr(usage, "cache_read_input_tokens", 0) or 0 | ||
| ) | ||
| cache_creation_tokens = int( | ||
| getattr(usage, "cache_creation_input_tokens", 0) or 0 | ||
| ) | ||
| cache_read_tokens = int( | ||
| getattr(usage, "cache_read_input_tokens", 0) or 0 | ||
| ) | ||
| # Anthropic semantics: ``input_tokens`` is the *uncached* prompt | ||
| # count. OpenAI reports total prompt tokens (cached + uncached), | ||
| # so subtract to keep client-side accounting consistent. | ||
| if cache_read_tokens and input_tokens >= cache_read_tokens: | ||
| input_tokens = input_tokens - cache_read_tokens |
There was a problem hiding this comment.
_process_event
The existing test file for this module only exercises transformation.py; streaming_iterator.py has no test coverage for the usage-processing path. The PR's test-plan checkboxes are all unchecked, so there is no automated verification that the OpenAI-path fix, the Anthropic fallback, or the subtraction guard behaves correctly. A small parametrised unit test over mock usage objects with input_tokens_details, cache_read_input_tokens, or both absent would prevent regressions here.
Rule Used: What: Ensure that any PR claiming to fix an issue ... (source)
|
🤖 litellm-agent: This PR is currently BLOCKED from merge. Score: 2/5 ❌ Why blocked:
Details: Score docked for: 1 PR-related CI failure (Greptile gate: score 3/5 below required 4/5 — request a Greptile review ( Fix the issues above and push an update — the bot will re-review automatically.
|
|
@greptileai please review — flagged as missing Confidence Score by litellm-agent. |
|
Closing in favor of #32445, which addresses the same issue (#28354) more completely — it patches both |
1 participant
Summary
Fixes #28354.
AnthropicResponsesStreamWrapper._process_eventreadsusage.cache_read_input_tokens— an Anthropic-only field name — from the OpenAI Responsesusageobject onresponse.completed, so the value is always0and clients (Claude Code / Agent SDK, Langfuse, Sentry, billing dashboards) routed through/v1/messages→ Responses seecache_read_input_tokens = 0even when OpenAI is correctly reporting thousands of cached prompt tokens.The OpenAI Responses API exposes cached tokens at
usage.input_tokens_details.cached_tokens— the same pattern the parallel Chat Completions adapter already handles viaprompt_tokens_details.cached_tokens(adapters/streaming_iterator.py:280-291).Change
In
responses_adapters/streaming_iterator.py::AnthropicResponsesStreamWrapper._process_event:cache_read_tokensfromusage.input_tokens_details.cached_tokens(object or dict shape).usage.cache_read_input_tokensif the upstream provider already speaks Anthropic usage.cache_read_tokensfrominput_tokensto preserve Anthropic's "uncached prompt" semantics, matching the Chat Completions path.input_tokens_details/output_tokens_detailsinto the cache-token variables before immediately overwriting them.cache_creation_input_tokenskeeps reading the Anthropic-named field only — OpenAI Responses has no direct equivalent, so emitting 0 there is correct.Test plan
tests/.../responses_adapters/still pass./v1/messagesagainst a configuredmode: "responses"model with a >1024-token prompt repeated; confirm the streamedmessage_deltaevent emits a non-zerocache_read_input_tokensonce OpenAI starts reportingcached_tokenson subsequent calls. Reproduction script in [Bug]: Anthropic→Responses streaming adapter never extracts cache_read_input_tokens (always 0) #28354.AI-assisted, human reviewed.