fix(streaming): let a later usage event zero out stale cache counts (#40736)

[muhammadwaqar12]

Superseded by #42330, the same commits on a litellm_ branch so the CircleCI suites can run; review and merge happen there

TLDR

Problem this solves:

• A streamed cache write count survives after a later usage event reports it as 0
• The merged usage then counts a 58k write and a 58k read together
• Uncached input derived from it goes negative and the write is billed twice

How it solves it:

• A usage event that reports any prompt-side count now sets both cache fields
• Explicit zeros in that event replace the earlier values
• An output-only event still leaves the earlier cache counts alone

User Flow

Before: a developer whose app streams a cached Claude prompt through the proxy gets a usage block that still counts a cache write the provider took back, so the app's derived uncached input goes negative

1. They send POST https://litellm-domain/v1/chat/completions with "stream": true, "stream_options": {"include_usage": true}, and a system block of about 58k tokens marked "cache_control": {"type": "ephemeral"}, routed to Claude Opus on Bedrock
2. Their app reads the SSE stream to its end, through the final usage chunk and data: [DONE]
3. That usage chunk carries prompt_tokens: 58354, cache_read_input_tokens: 58352, and cache_creation_input_tokens: 58352, even though the provider's final count for this request said the cache was read, not written (the cache_creation_token_details in the same block already reads 0 and 0)
4. Their app computes uncached input as prompt_tokens - cache_read_input_tokens - cache_creation_input_tokens, stores -58350, and their usage dashboard shows negative input tokens for that user

After: the same stream's usage block keeps the provider's final counts, so the derived uncached input is 2

1. They send POST https://litellm-domain/v1/chat/completions with "stream": true, "stream_options": {"include_usage": true}, and a system block of about 58k tokens marked "cache_control": {"type": "ephemeral"}, routed to Claude Opus on Bedrock
2. Their app reads the SSE stream to its end, through the final usage chunk and data: [DONE]
3. That usage chunk carries prompt_tokens: 58354, cache_read_input_tokens: 58352, and cache_creation_input_tokens: 0, matching the provider's final count
4. Their app computes 58354 - 58352 - 0 = 2 and the dashboard shows 2 uncached input tokens for that user

Relevant issues

Fixes #40736

Affected release

Linear ticket

Resolves LIT-7585

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Both legs ran a LiteLLM proxy booted with --num_workers 2 (two server processes behind one random port) in front of Claude Opus 5 on Bedrock through the Invoke route in us-east-1, real spend. Each of the three unified endpoints got the same streamed request twice with a fresh 800-clause system block marked cache_control (first request: cache write expected, second request: cache read expected). What is shown is the usage the client receives, plus, for chat, the uncached input the reporter's app derives as prompt_tokens - cache_read_input_tokens - cache_creation_input_tokens

Config, lit7585_config.yaml:

model_list:
  - model_name: bedrock-opus
    litellm_params:
      model: bedrock/invoke/us.anthropic.claude-opus-5
      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
      aws_region_name: us-east-1

general_settings:
  master_key: sk-lit7585-local

Boot: python litellm/proxy/proxy_cli.py --config lit7585_config.yaml --port $PORT --num_workers 2

Filler, unique per run so the first request is always a cache write: FILLER=$(python3 -c "print(' '.join(f'clause {i} of the reference manual for run $RUN_ID states that widget {i % 97} must be inspected before shipment' for i in range(800)))")

Request bodies (body.json below is one of these, sent twice unchanged):

{"model": "bedrock-opus", "stream": true, "stream_options": {"include_usage": true}, "max_tokens": 40,
 "messages": [{"role": "system", "content": [{"type": "text", "text": "$FILLER", "cache_control": {"type": "ephemeral"}}]},
              {"role": "user", "content": "Reply with the single word OK."}]}

{"model": "bedrock-opus", "stream": true, "max_output_tokens": 40,
 "input": [{"role": "system", "content": [{"type": "input_text", "text": "$FILLER", "cache_control": {"type": "ephemeral"}}]},
           {"role": "user", "content": "Reply with the single word OK."}]}

{"model": "bedrock-opus", "stream": true, "max_tokens": 40,
 "system": [{"type": "text", "text": "$FILLER", "cache_control": {"type": "ephemeral"}}],
 "messages": [{"role": "user", "content": "Reply with the single word OK."}]}

The output shown for chat is the final usage SSE chunk, for responses the response.completed event's usage, and for messages the usage of the message_start and message_delta events

The reporter's symptom needs one stream whose message_start reports a cache write and whose message_delta then reports a cache read with cache_creation_input_tokens: 0. That sequence never occurred live during this QA: 20 Bedrock Invoke streams (12 plain, 8 started 400 ms apart on one identical cached prompt) and 8 Anthropic-direct streams sent straight to the providers all reported the same cache counts in both events, and so did the 18 proxy streams of the two legs. So the legs below show that the plain write and read shapes read the same before and after on every endpoint, and the reporter's exact event sequence is covered by the regression tests named under Caveats

Before (3597d61)

/v1/chat/completions

1. Cache write: curl -sS -N http://localhost:59448/v1/chat/completions -H 'Authorization: Bearer sk-lit7585-local' -H 'Content-Type: application/json' -d @body.json

   {"completion_tokens": 1, "prompt_tokens": 35217, "total_tokens": 35218, "completion_tokens_details": {"reasoning_tokens": 0, "text_tokens": 1}, "prompt_tokens_details": {"cached_tokens": 0, "text_tokens": 14, "cache_write_tokens": 35203, "cache_creation_tokens": 35203, "cache_creation_token_details": {"ephemeral_5m_input_tokens": 35203, "ephemeral_1h_input_tokens": 0}}, "cost": 0.24212512500000002, "cache_creation_input_tokens": 35203, "cache_read_input_tokens": 0}

   derived uncached input = 35217 - 0 - 35203 = 14
2. Cache read, same command and body:

   {"completion_tokens": 1, "prompt_tokens": 35217, "total_tokens": 35218, "completion_tokens_details": {"reasoning_tokens": 0, "text_tokens": 1}, "prompt_tokens_details": {"cached_tokens": 35203, "text_tokens": 14, "cache_write_tokens": 0, "cache_creation_tokens": 0, "cache_creation_token_details": {"ephemeral_5m_input_tokens": 0, "ephemeral_1h_input_tokens": 0}}, "cost": 0.01946615, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 35203}

   derived uncached input = 35217 - 35203 - 0 = 14

/v1/responses

1. Cache write: curl -sS -N http://localhost:59448/v1/responses -H 'Authorization: Bearer sk-lit7585-local' -H 'Content-Type: application/json' -d @body.json

   {"input_tokens": 35217, "input_tokens_details": {"cached_tokens": 0, "text_tokens": 14, "cache_write_tokens": 35203}, "output_tokens": 4, "output_tokens_details": {"reasoning_tokens": 0, "text_tokens": 4}, "total_tokens": 35221, "cost": 0.242207625}

2. Cache read, same command and body:

   {"input_tokens": 35217, "input_tokens_details": {"cached_tokens": 35203, "text_tokens": 14, "cache_write_tokens": 0}, "output_tokens": 4, "output_tokens_details": {"reasoning_tokens": 0, "text_tokens": 4}, "total_tokens": 35221, "cost": 0.01954865}

/v1/messages

1. Cache write: curl -sS -N http://localhost:59448/v1/messages -H 'Authorization: Bearer sk-lit7585-local' -H 'Content-Type: application/json' -d @body.json

   message_start usage: {"input_tokens": 14, "cache_creation_input_tokens": 35203, "cache_read_input_tokens": 0, "cache_creation": {"ephemeral_5m_input_tokens": 35203, "ephemeral_1h_input_tokens": 0}, "output_tokens": 1, "service_tier": "standard"}
   message_delta usage: {"input_tokens": 14, "cache_creation_input_tokens": 35203, "cache_read_input_tokens": 0, "output_tokens": 4, "output_tokens_details": {"thinking_tokens": 0}, "cache_creation": {"ephemeral_5m_input_tokens": 35203, "ephemeral_1h_input_tokens": 0}}
   

2. Cache read, same command and body:

   message_start usage: {"input_tokens": 14, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 35203, "cache_creation": {"ephemeral_5m_input_tokens": 0, "ephemeral_1h_input_tokens": 0}, "output_tokens": 1, "service_tier": "standard"}
   message_delta usage: {"input_tokens": 14, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 35203, "output_tokens": 4, "output_tokens_details": {"thinking_tokens": 0}, "cache_creation": {"ephemeral_5m_input_tokens": 0, "ephemeral_1h_input_tokens": 0}}
   

After (9a4b19a)

/v1/chat/completions

1. Cache write: curl -sS -N http://localhost:26333/v1/chat/completions -H 'Authorization: Bearer sk-lit7585-local' -H 'Content-Type: application/json' -d @body.json

   {"completion_tokens": 1, "prompt_tokens": 35217, "total_tokens": 35218, "completion_tokens_details": {"reasoning_tokens": 0, "text_tokens": 1}, "prompt_tokens_details": {"cached_tokens": 0, "text_tokens": 14, "cache_write_tokens": 35203, "cache_creation_tokens": 35203, "cache_creation_token_details": {"ephemeral_5m_input_tokens": 35203, "ephemeral_1h_input_tokens": 0}}, "cost": 0.24212512500000002, "cache_creation_input_tokens": 35203, "cache_read_input_tokens": 0}

   derived uncached input = 35217 - 0 - 35203 = 14
2. Cache read, same command and body:

   {"completion_tokens": 1, "prompt_tokens": 35217, "total_tokens": 35218, "completion_tokens_details": {"reasoning_tokens": 0, "text_tokens": 1}, "prompt_tokens_details": {"cached_tokens": 35203, "text_tokens": 14, "cache_write_tokens": 0, "cache_creation_tokens": 0, "cache_creation_token_details": {"ephemeral_5m_input_tokens": 0, "ephemeral_1h_input_tokens": 0}}, "cost": 0.01946615, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 35203}

   derived uncached input = 35217 - 35203 - 0 = 14

/v1/responses

1. Cache write: curl -sS -N http://localhost:26333/v1/responses -H 'Authorization: Bearer sk-lit7585-local' -H 'Content-Type: application/json' -d @body.json

   {"input_tokens": 35217, "input_tokens_details": {"cached_tokens": 0, "text_tokens": 14, "cache_write_tokens": 35203}, "output_tokens": 1, "output_tokens_details": {"reasoning_tokens": 0, "text_tokens": 1}, "total_tokens": 35218, "cost": 0.24212512500000002}

2. Cache read, same command and body:

   {"input_tokens": 35217, "input_tokens_details": {"cached_tokens": 35203, "text_tokens": 14, "cache_write_tokens": 0}, "output_tokens": 1, "output_tokens_details": {"reasoning_tokens": 0, "text_tokens": 1}, "total_tokens": 35218, "cost": 0.01946615}

/v1/messages

1. Cache write: curl -sS -N http://localhost:26333/v1/messages -H 'Authorization: Bearer sk-lit7585-local' -H 'Content-Type: application/json' -d @body.json

   message_start usage: {"input_tokens": 14, "cache_creation_input_tokens": 35203, "cache_read_input_tokens": 0, "cache_creation": {"ephemeral_5m_input_tokens": 35203, "ephemeral_1h_input_tokens": 0}, "output_tokens": 1, "service_tier": "standard"}
   message_delta usage: {"input_tokens": 14, "cache_creation_input_tokens": 35203, "cache_read_input_tokens": 0, "output_tokens": 1, "cache_creation": {"ephemeral_5m_input_tokens": 35203, "ephemeral_1h_input_tokens": 0}}
   

2. Cache read, same command and body:

   message_start usage: {"input_tokens": 14, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 35203, "cache_creation": {"ephemeral_5m_input_tokens": 0, "ephemeral_1h_input_tokens": 0}, "output_tokens": 4, "service_tier": "standard"}
   message_delta usage: {"input_tokens": 14, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 35203, "output_tokens": 4, "cache_creation": {"ephemeral_5m_input_tokens": 0, "ephemeral_1h_input_tokens": 0}}
   

Observed alongside, none caused or changed by this PR:

• Chat and responses input counts include cached tokens; messages input_tokens excludes them
• Responses usage carries cache_write_tokens only, no cache_creation_input_tokens key
• Messages output_tokens in message_start already equals the final count
• Neither event pair in the messages stream ever disagreed on cache counts

Type

Bug Fix
Test

Caveats (if any)

Medium

• Live QA never reached the trigger, so before and after read the same on every endpoint
  • The write-then-read flip inside one stream never occurred in 46 streams across Bedrock and Anthropic direct
  • The reporter's exact event sequence is pinned by tests/test_litellm/litellm_core_utils/test_streaming_chunk_builder_utils.py::test_anthropic_stream_usage_takes_cache_counts_from_last_event_that_reports_them, three cases through the real Anthropic stream parser and the merger, all failing on the merge base and passing here

Low

• The Anthropic stream parser has been cumulative since fix(proxy): estimate auto-router baseline costs from durable cache history #41177, so a message_delta without cache fields inherits the message_start counts; the delta_reports_input_and_output_only case pins that, so the earlier worry that such an event would clear both counts does not apply at this head
• The CircleCI suites under tests/local_testing and tests/llm_translation do not run on a fork head, so tests/local_testing/test_anthropic_prompt_caching.py was run locally at this tip against the real Anthropic API: 7 passed (the streaming case included), 2 skipped, 4 failed
  • The 4 failures fail identically at the merge base: test_anthropic_vertex_ai_prompt_caching names a Vertex model that is no longer in the cost map, and test_litellm_anthropic_prompt_caching_system and test_litellm_anthropic_prompt_caching_tools assert a mocked post call without the logging_obj kwarg the handler now passes; none of them reach the usage merger

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[CLAassistant]

All committers have signed the CLA.

[greptile-apps]

Greptile Summary

The PR corrects streamed usage aggregation so a later prompt-side usage event can replace stale cache counters with explicit zeros while output-only events preserve prior cache usage.

• Adds a provider-neutral prompt-side usage predicate to the shared chunk merger.
• Updates both cache creation and cache read aggregation to honor explicit zero values on prompt-side events.
• Adds an Anthropic parser-to-chunk-builder regression test covering cache-count replacement and output-only deltas.

Confidence Score: 5/5

The PR appears safe to merge; the earlier provider-boundary and comment concerns are removed, and the parser-backed regression test exercises the intended behavior.

No actionable new defects remain. The prior correctness thread was manually resolved, while the provider-specific shared-merger logic and excessive comments identified in the other previous threads are absent from the current code.

Important Files Changed

Filename	Overview
litellm/litellm_core_utils/streaming_chunk_builder_utils.py	Updates generic streaming usage aggregation to replace cache counters when a later chunk reports prompt-side usage.
tests/test_litellm/litellm_core_utils/test_streaming_chunk_builder_utils.py	Adds end-to-end unit coverage through the Anthropic stream parser for authoritative cache counts and output-only usage events.

_{Reviews (2): Last reviewed commit: "fix(streaming): let a later usage event ..." | Re-trigger Greptile}

[greptile-apps]

Provider logic in shared merger This adds Anthropic-specific event handling outside llms/, violating the repository requirement and must be moved before merge

Rule Used: What: Avoid writing provider-specific code outside of the llms/ directory. Why: This practice ensures better maintainability and reduces complexity over time. Good: ```python # Handle provider-specific logic within llms/vertex_ai/transformation.py ... (source)

[greptile-apps]

Comments exceed repository guidance These narratives repeat implementation history; repository rules require only necessary, concise source comments and must be satisfied before merging

Context Used: CLAUDE.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing muhammadwaqar12:fix/40736-cache-write-zero-update (9a4b19a) with main (f5f53a4)}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[muhammadwaqar12]

@greptileai the new test runs raw Anthropic SSE events through the real parser, and the comments and provider logic are gone. Please re-review

[devin-ai-integration]

Thanks for the fix. It landed on main through #42330 with your commit and authorship preserved, so closing this one.