fix(responses_adapters): map OpenAI Responses cache tokens on /v1/messages

[devin-ai-integration]

TLDR

Problem this solves:

• /v1/messages to an OpenAI model dropped cache-read tokens
• cache reads were billed at the full input rate

How it solves it:

• read cache split from the Responses input_tokens_details
• keep Anthropic-native names as first choice

Relevant issues

Fixes #35127

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Live proxy backed by openai/gpt-4o on /v1/messages, with a ~6000 token stable prefix so OpenAI caches it. A control call to /v1/responses confirms OpenAI actually cached the prefix (cached_tokens: 5888); the same warm prefix is then sent through /v1/messages

Config

model_list:
  - model_name: gpt-4o-messages
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

Control, /v1/responses (proves OpenAI cached the prefix)

$ curl -sS -X POST http://127.0.0.1:4000/v1/responses ... | jq .usage.input_tokens_details
{ "cached_tokens": 5888, "cache_write_tokens": 0 }

Before the fix, code at cad32fd9bc (source files reverted on the running proxy)

# streaming /v1/messages, warm cache
data: {"type": "message_delta", "delta": {...}, "usage": {"input_tokens": 6012, "output_tokens": 2}}
# non-streaming /v1/messages, warm cache
usage: {"input_tokens": 6012, "output_tokens": 2}

After the fix, code at df2ec624ac

# streaming /v1/messages, warm cache
data: {"type": "message_delta", "delta": {...}, "usage": {"input_tokens": 6012, "output_tokens": 2, "cache_read_input_tokens": 5888}}
# non-streaming /v1/messages, warm cache
usage: {"input_tokens": 6012, "output_tokens": 3, "cache_read_input_tokens": 5888}

Type

🐛 Bug Fix

Changes

When /v1/messages routes to an OpenAI (or Azure) model, the request goes through the Responses-API bridge and the result is translated back to Anthropic format. Both the streaming adapter (AnthropicResponsesStreamWrapper._process_event) and the non-streaming adapter (LiteLLMAnthropicToResponsesAPIAdapter.translate_response) read cache counts only from the Anthropic-native usage keys cache_read_input_tokens / cache_creation_input_tokens. An OpenAI Responses usage object never has those names, so cache reads were always reported as 0

OpenAI reports the split under input_tokens_details, as cached_tokens for reads and cache_write_tokens for writes. cache_write_tokens is a pydantic extra, so it survives model_dump() but is missed by a fixed getattr list. The plain /responses path already reads these correctly in litellm/responses/utils.py; this brings the Messages bridge in line

The extraction now lives in one helper, _extract_cache_tokens, shared by both adapters. It prefers the Anthropic-native names when present (so Bedrock/Vertex on /v1/messages is unaffected) and otherwise falls back to input_tokens_details, handling both a pydantic model and a plain dict. input_tokens is left inclusive of cached tokens, matching what the /responses path reports

Pseudocode:

def _extract_cache_tokens(usage):
    read = int(getattr(usage, "cache_read_input_tokens", 0) or 0)
    write = int(getattr(usage, "cache_creation_input_tokens", 0) or 0)
    if read and write:
        return read, write
    details = getattr(usage, "input_tokens_details", None)
    d = details if isinstance(details, dict) else _model_dump(details)
    read = read or int(d.get("cached_tokens") or 0)
    write = write or int(d.get("cache_write_tokens") or d.get("cache_creation_tokens") or 0)
    return read, write

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/b31da05dd8194af69c5601b6aa979e5f

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
2 out of 3 committers have signed the CLA.

✅ yuneng-berri
✅ mubashir1osmani
❌ devin-ai-integration[bot]
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Avoid unconditional stabilization delay

Every create_model call now waits through a fixed 30-second observation window after the model first appears, including single-worker deployments where the first successful listing already establishes readiness. Tests that create several models therefore accumulate minutes of unnecessary setup time and can exhaust CI job budgets; apply this stabilization period only to multi-worker topologies.

Context Used: CLAUDE.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[greptile-apps]

Greptile Summary

Maps OpenAI Responses cache token details into Anthropic Messages usage

• Adds shared cache-read and cache-creation extraction for streaming and non-streaming adapters
• Adds regression coverage for model and dictionary usage-detail shapes
• Expands E2E synchronization helpers for model and MCP registration propagation

Confidence Score: 4/5

The PR appears safe to merge, though the unconditional model stabilization delay should be narrowed to multi-worker E2E deployments

Cache-token mapping is consistently applied across streaming and non-streaming adapters with focused regression coverage; the remaining concern is non-blocking E2E runtime inflation from the fixed 30-second wait

Files Needing Attention: tests/e2e/proxy_client.py

Important Files Changed

Filename	Overview
litellm/llms/anthropic/experimental_pass_through/responses_adapters/streaming_iterator.py	Adds cache-token extraction and uses it when producing streaming Anthropic usage
litellm/llms/anthropic/experimental_pass_through/responses_adapters/transformation.py	Adds cache-read and cache-creation fields to non-streaming Anthropic usage
tests/e2e/proxy_client.py	Adds stronger model propagation polling, but its unconditional 30-second stabilization window substantially slows all model creation
tests/e2e/mcp/mcp_client.py	Adds targeted retries for recognized multi-worker MCP registry misses
tests/e2e/transport.py	Supports per-request GET timeouts used by deadline-aware polling

_{Reviews (1): Last reviewed commit: "fix(responses_adapters): map OpenAI Resp..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_fix_35127_responses_cache_tokens (df2ec62) with litellm_internal_staging (bf5334b)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (440b1bc) during the generation of this report, so bf5334b was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[mateo-berri]

Superseded by #34957, which shipped the same mapping via shared transforms with native-name precedence. Verified live on staging today, so closing this PR