Skip to content

fix(responses_adapter): read cache_read_tokens from input_tokens_details on Responses usage (#28354) #28380

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Anai-Guo wants to merge 2 commits into from
Closed

fix(responses_adapter): read cache_read_tokens from input_tokens_details on Responses usage (#28354) #28380

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b31aceb
fix(responses_adapter): read cache_read_tokens from input_tokens_deta…
Anai-Guo May 20, 2026
1cdb2a3
chore: empty commit to re-trigger source-branch check after base reta…
Anai-Guo May 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 24 additions & 8 deletions litellm/llms/anthropic/experimental_pass_through/responses_adapters/streaming_iterator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -257,17 +257,33 @@ def _process_event(self, event: Any) -> None: # noqa: PLR0915
stop_reason = "max_tokens"
usage = getattr(response_obj, "usage", None)
if usage is not None:
input_tokens = getattr(usage, "input_tokens", 0) or 0
output_tokens = getattr(usage, "output_tokens", 0) or 0
cache_creation_tokens = getattr(usage, "input_tokens_details", None) # type: ignore[assignment]
cache_read_tokens = getattr(usage, "output_tokens_details", None) # type: ignore[assignment]
# Prefer direct cache fields if present
input_tokens = int(getattr(usage, "input_tokens", 0) or 0)
output_tokens = int(getattr(usage, "output_tokens", 0) or 0)
# OpenAI Responses API exposes cached prompt tokens at
# ``usage.input_tokens_details.cached_tokens`` (parallel to
# ``usage.prompt_tokens_details.cached_tokens`` on Chat Completions).
itd = getattr(usage, "input_tokens_details", None)
if itd is not None:
if isinstance(itd, dict):
cache_read_tokens = int(itd.get("cached_tokens", 0) or 0)
else:
cache_read_tokens = int(
getattr(itd, "cached_tokens", 0) or 0
)
# Fall back to Anthropic-style direct fields if the upstream
# provider already speaks Anthropic usage.
if not cache_read_tokens:
cache_read_tokens = int(
getattr(usage, "cache_read_input_tokens", 0) or 0
)
cache_creation_tokens = int(
getattr(usage, "cache_creation_input_tokens", 0) or 0
)
cache_read_tokens = int(
getattr(usage, "cache_read_input_tokens", 0) or 0
)
# Anthropic semantics: ``input_tokens`` is the *uncached* prompt
# count. OpenAI reports total prompt tokens (cached + uncached),
# so subtract to keep client-side accounting consistent.
if cache_read_tokens and input_tokens >= cache_read_tokens:
input_tokens = input_tokens - cache_read_tokens
Comment on lines +273 to +286

@greptile-apps greptile-apps Bot May 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Fallback + subtraction produces wrong input_tokens for Anthropic-native upstreams

When input_tokens_details is absent (Anthropic providers don't include it) and cache_read_input_tokens > 0, the fallback correctly captures cached tokens — but the unconditional subtraction that follows then reduces an input_tokens value that is already the uncached count (Anthropic semantics). For a request with 3 000 uncached tokens and 500 cached, the emitted input_tokens becomes 2 500 instead of 3 000.

The subtraction is only valid for the OpenAI Responses path (where input_tokens is the total prompt count). It should be skipped whenever the value came from the cache_read_input_tokens fallback.

Comment on lines 259 to +286

@greptile-apps greptile-apps Bot May 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 No unit tests cover the new cache-token logic in _process_event

The existing test file for this module only exercises transformation.py; streaming_iterator.py has no test coverage for the usage-processing path. The PR's test-plan checkboxes are all unchecked, so there is no automated verification that the OpenAI-path fix, the Anthropic fallback, or the subtraction guard behaves correctly. A small parametrised unit test over mock usage objects with input_tokens_details, cache_read_input_tokens, or both absent would prevent regressions here.

Rule Used: What: Ensure that any PR claiming to fix an issue ... (source)


# Check if tool_use was in the output to override stop_reason
if response_obj is not None:
Expand Down
Loading