fix(anthropic): map responses cached tokens to cache reads by silencedoctor · Pull Request #32445 · BerriAI/litellm

silencedoctor · 2026-07-08T07:36:47Z

Relevant issues

Fixes #28354

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
My PR passes local lint, format, and targeted unit tests listed below
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

This is a deterministic adapter translation bug. The public issue #28354 already includes a live-provider reproduction showing OpenAI Responses returns non-zero usage.input_tokens_details.cached_tokens while LiteLLM's Anthropic /v1/messages response reports zero cache reads. This PR's proof isolates the affected LiteLLM translation entrypoints with sanitized Responses usage data. It contains no real provider request, API key, hostname, customer payload, or organization-specific identifier.

Before fix, captured from upstream/litellm_oss_staging at 710f6b2dcd:

commit=710f6b2dcd (upstream/litellm_oss_staging before fix)
input_usage={'input_tokens': 1000, 'output_tokens': 75, 'input_tokens_details': {'cached_tokens': 800}}
non_streaming={'input_tokens': 1000, 'output_tokens': 75}
streaming={'input_tokens': 1000, 'output_tokens': 75}

After fix, captured from this PR at 82d34743da:

commit=82d34743da (after fix)
input_usage={'input_tokens': 1000, 'output_tokens': 75, 'input_tokens_details': {'cached_tokens': 800}}
non_streaming={'input_tokens': 200, 'output_tokens': 75, 'cache_creation_input_tokens': 0, 'cache_read_input_tokens': 800}
streaming={'input_tokens': 200, 'output_tokens': 75, 'cache_read_input_tokens': 800}
fallback_input_usage={'input_tokens': 200, 'output_tokens': 75, 'cache_read_input_tokens': 800, 'cache_creation_input_tokens': 10}
anthropic_fallback={'input_tokens': 200, 'output_tokens': 75, 'cache_creation_input_tokens': 10, 'cache_read_input_tokens': 800}

Local validation:

$ uv run pytest tests/test_litellm/llms/anthropic/experimental_pass_through/responses_adapters/test_responses_adapters_transformation.py tests/test_litellm/llms/anthropic/experimental_pass_through/responses_adapters/test_responses_adapters_streaming_iterator.py -q
........................................................................ [ 79%]
...................                                                      [100%]
91 passed in 5.18s

$ uv run ruff check litellm/llms/anthropic/experimental_pass_through/responses_adapters/transformation.py litellm/llms/anthropic/experimental_pass_through/responses_adapters/streaming_iterator.py tests/test_litellm/llms/anthropic/experimental_pass_through/responses_adapters/test_responses_adapters_transformation.py tests/test_litellm/llms/anthropic/experimental_pass_through/responses_adapters/test_responses_adapters_streaming_iterator.py
All checks passed!

$ uv run ruff format --check litellm/llms/anthropic/experimental_pass_through/responses_adapters/transformation.py litellm/llms/anthropic/experimental_pass_through/responses_adapters/streaming_iterator.py
2 files already formatted

Type

🐛 Bug Fix
✅ Test

Changes

OpenAI Responses usage reports cached prompt tokens under usage.input_tokens_details.cached_tokens, and its input_tokens includes those cached tokens. Anthropic Messages usage expects cache reads under cache_read_input_tokens, while input_tokens should be the uncached input count.

This PR adds one shared Responses usage -> Anthropic usage mapper and uses it in both affected Responses adapter return paths:

non-streaming translate_response
streaming response.completed -> message_delta

It also preserves Anthropic-style fallback semantics: when usage already provides cache_read_input_tokens directly and no Responses input_tokens_details.cached_tokens is present, the mapper does not subtract cache reads from input_tokens again.

Regression coverage added for:

OpenAI Responses object-shaped input_tokens_details.cached_tokens
dict-shaped input_tokens_details.cached_tokens
Anthropic-style fallback usage without double subtraction
both non-streaming and streaming adapter paths

codecov · 2026-07-08T07:42:49Z

Codecov Report

❌ Patch coverage is 91.95804% with 23 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...tellm/llms/base_llm/guardrail_translation/utils.py	86.36%	9 Missing ⚠️
...ms/anthropic/chat/guardrail_translation/handler.py	92.59%	6 Missing ⚠️
...drail_hooks/unified_guardrail/unified_guardrail.py	95.45%	3 Missing ⚠️
litellm/proxy/proxy_server.py	75.00%	2 Missing ⚠️
..._pass_through/responses_adapters/transformation.py	96.42%	1 Missing ⚠️
...base_llm/guardrail_translation/base_translation.py	50.00%	1 Missing ⚠️
.../openai/responses/guardrail_translation/handler.py	75.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

chore(ci): promote internal staging to main

_await_model_servable used poll_timeout (120s), the spend/log read-back budget. A stuck model reload therefore stalled every suite that creates a deployment for two minutes before failing Give create_model a fixed harness middle ground: model_servable_timeout=40s, polled every 2s, with each /v1/models call capped at 5s and clamped to the remaining deadline so one slow GET cannot overrun the wait. Happy path still returns on the first listing. Not derived from proxy general_settings or env Transport.get accepts an optional per-call timeout for that clamp. Unit tests cover the deadline arithmetic and clamp without a live proxy (cherry picked from commit c082a0e)

create_model returned after the first /v1/models hit that listed the model, so chat could still land on a cold gateway worker (numWorkers>1 / peer pod) and 400 Invalid model name. Require continuous listing for the product default add_deployment interval (30s) after first sight so every worker has synced from the DB; first listing still bounded at 40s (cherry picked from commit 7d1ee2f)

Keep the create_model DB-sync wait in the harness; the pure-function unit file is not needed for this PR (cherry picked from commit 8920465)

When less than one full poll interval remained in the first-listing budget, the pre-sleep check returned NotServable without another /v1/models call. Sleep only min(interval, time left) so a model that becomes listable in the last seconds of the timeout still gets a clamped final poll (cherry picked from commit 8439195)

A poll may start with remaining budget and still return after started+timeout if the transport overruns its clamp. Recheck the first-listing deadline after the response so a late listing does not open the continuous DB-sync phase (cherry picked from commit 7ff2bcb)

…l_servable_timeout test(e2e): bound the post-/model/new servable wait at 40s

CLAassistant · 2026-07-29T11:12:04Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
8 out of 9 committers have signed the CLA.

✅ yuneng-berri
✅ mubashir1osmani
✅ mateo-berri
✅ ryan-crabbe-berri
✅ shivamrawat1
✅ tin-berri
✅ yucheng-berri
✅ silencedoctor
❌ devin-ai-integration[bot]
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

shivamrawat1 · 2026-08-07T21:43:05Z

Thanks for the PR and the synthetic repro on #28354. This was superseded by #34957, which routes both adapter paths through a shared usage translation covering cache reads and writes; it will be in 1.97.0-rc.1. Closing this one

silencedoctor mentioned this pull request Jul 8, 2026

[Bug]: Anthropic→Responses streaming adapter never extracts cache_read_input_tokens (always 0) #28354

Closed

silencedoctor force-pushed the fix/anthropic-responses-cache-usage branch from 76223b8 to 82d3474 Compare July 8, 2026 07:51

yuneng-berri added 3 commits July 21, 2026 19:03

Merge pull request BerriAI#34200 from BerriAI/litellm_internal_staging

f2479cc

chore(ci): promote internal staging to main

Merge pull request BerriAI#34450 from BerriAI/litellm_internal_staging

7495b1f

chore(ci): promote internal staging to main

Merge pull request BerriAI#34519 from BerriAI/litellm_internal_staging

0cd588a

chore(ci): promote internal staging to main

Anai-Guo mentioned this pull request Jul 27, 2026

fix(responses_adapter): read cache_read_tokens from input_tokens_details on Responses usage (#28354) #28380

Closed

2 tasks

yuneng-berri and others added 8 commits July 28, 2026 16:05

Merge pull request BerriAI#34864 from BerriAI/litellm_internal_staging

9ead580

chore(ci): promote internal staging to main

test(e2e): drop proxy_client model-servable unit tests

5953a66

Keep the create_model DB-sync wait in the harness; the pure-function unit file is not needed for this PR (cherry picked from commit 8920465)

Merge pull request BerriAI#35020 from BerriAI/litellm_hotfix_e2e_mode…

2cd62cf

…l_servable_timeout test(e2e): bound the post-/model/new servable wait at 40s

fix(anthropic): map responses cached tokens to cache reads

5e80efc

silencedoctor force-pushed the fix/anthropic-responses-cache-usage branch from 82d3474 to 5e80efc Compare July 29, 2026 11:11

silencedoctor changed the base branch from litellm_oss_staging to litellm_internal_staging July 29, 2026 12:24

shivamrawat1 closed this Aug 7, 2026

github-actions Bot mentioned this pull request Aug 7, 2026

🦞 OpenClaw 生态日报 2026-08-08 junlinzhao327-oss/big_model_radar#290

Open

Navigation Menu

Uh oh!

Uh oh!

fix(anthropic): map responses cached tokens to cache reads - #32445

fix(anthropic): map responses cached tokens to cache reads#32445
silencedoctor wants to merge 11 commits into
BerriAI:litellm_internal_stagingfrom
silencedoctor:fix/anthropic-responses-cache-usage

silencedoctor commented Jul 8, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jul 8, 2026 •

edited

Loading

Uh oh!

CLAassistant commented Jul 29, 2026 •

edited

Loading

Uh oh!

shivamrawat1 commented Aug 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Uh oh!

Conversation

silencedoctor commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Relevant issues

Linear ticket

Pre-Submission checklist

Delays in PR merge?

Screenshots / Proof of Fix

Type

Changes

Uh oh!

codecov Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

CLAassistant commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shivamrawat1 commented Aug 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

silencedoctor commented Jul 8, 2026 •

edited

Loading

codecov Bot commented Jul 8, 2026 •

edited

Loading

CLAassistant commented Jul 29, 2026 •

edited

Loading