fix(cost): match streamed Messages usage cost to the recorded spend

[devin-ai-integration]

TLDR

Problem this solves:

• Streamed usage.cost disagreed with the logged spend
• Cached input tokens were billed as $0 input
• 1h cache writes were billed at the 5m rate
• Custom deployment pricing was ignored, streaming sticker price

How it solves it:

• Rebuild streamed usage with the non-streaming Anthropic transformation
• Price the chunk through the call's own cost calculator
• Fall back to model-name pricing when unavailable

User Flow

Before: a developer streaming Anthropic Messages through the gateway gets a per-request cost in the stream that does not match the spend the gateway records, so their in-app cost meter and their invoice disagree

1. Their proxy admin configures a Bedrock Sonnet deployment with negotiated per-token prices, plus include_cost_in_streaming_usage: True, and restarts the proxy
2. They send POST https://litellm-domain/v1/messages with "stream": true and a large cached system prompt
3. The final message_delta frame reports "input_tokens": 12, "cache_read_input_tokens": 3467 alongside "cost": 0.0011001
4. They look the same request up with GET https://litellm-domain/spend/logs?request_id=<the response's x-litellm-call-id> and the row reads "spend": 0.0003787, so the stream claimed 2.9x what they are billed
5. They retry with a 1 hour cache TTL, the frame claims 0.013155 while the spend row reads 0.007016, off by 1.875x
6. They try the same prompt on POST https://litellm-domain/anthropic/v1/messages, and the frame reads 0.0011001 against a recorded 0.0011361, short by exactly the 12 uncached input tokens

After: the same requests report the same cost in the stream as the gateway records, so the in-app meter reconciles

1. Their proxy admin configures the same deployment and restarts the proxy
2. They send the same POST https://litellm-domain/v1/messages with "stream": true and a large cached system prompt
3. The final message_delta frame carries a cost equal to the spend on the matching row from GET https://litellm-domain/spend/logs
4. They retry with a 1 hour cache TTL and the two numbers agree again, now at the configured 1h write rate
5. The same holds on POST https://litellm-domain/anthropic/v1/messages, where the uncached input tokens are no longer dropped

Relevant issues

Linear ticket

Resolves LIT-4902

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Both sides run a real proxy against real providers with real spend. The recorded number is read back through the gateway's own spend API rather than a callback file, so every figure below is one an operator can pull for themselves: the streaming response returns x-litellm-call-id, and that value is the spend row's request_id, so GET /spend/logs?request_id=<id> pins each comparison to a single request.

The deployment carries negotiated per-token prices well under sticker, and a 1h cache write rate distinct from the 5m one:

model_list:
  - model_name: qa-sonnet
    litellm_params:
      model: bedrock/us.anthropic.claude-sonnet-4-6
      aws_region_name: us-east-1
      input_cost_per_token: 0.000001
      output_cost_per_token: 0.000005
      cache_read_input_token_cost: 0.0000001
      cache_creation_input_token_cost: 0.00000125
      cache_creation_input_token_cost_above_1hr: 0.000002

general_settings:
  master_key: sk-1234

litellm_settings:
  include_cost_in_streaming_usage: True

Sticker for the same model is 0.000003 / 0.000015 / 0.0000003 / 0.00000375 / 0.000006. The after run names the deployment qa-sonnet, which has no entry in the cost map at all, so a sticker fallback could not quietly supply the number and any correct discounted cost has to come from the deployment's own pricing.

Before (cc812cd, the merge base)

Cache read on POST /v1/messages, x-litellm-call-id: 5c9902e3-8c5a-47cc-91a7-c0694e67d872:

data: {"type": "message_delta", "delta": {"stop_reason": "end_turn", "stop_sequence": null, "stop_details": null}, "usage": {"input_tokens": 12, "cache_creation_input_tokens": 0, "cache_read_input_tokens": 3467, "output_tokens": 4, "cache_creation": {"ephemeral_5m_input_tokens": 0, "ephemeral_1h_input_tokens": 0}, "cost": 0.0011001}}

GET /spend/logs?request_id=5c9902e3-8c5a-47cc-91a7-c0694e67d872
{"request_id": "5c9902e3-...", "call_type": "anthropic_messages", "spend": 0.0003787, "prompt_tokens": 3479, "completion_tokens": 4}

The recorded 0.0003787 is 12*1e-6 + 3467*1e-7 + 4*5e-6, the configured prices. The streamed 0.0011001 is 3467*3e-7 + 4*1.5e-5, sticker rates with the 12 non-cached input tokens charged at zero. The stream claims 2.9x the spend.

1h cache write on POST /v1/messages, x-litellm-call-id: bef853eb-de56-4418-a69a-85eb64168858:

data: {"type": "message_delta", "delta": {"stop_reason": "end_turn", "stop_sequence": null, "stop_details": null}, "usage": {"input_tokens": 12, "cache_creation_input_tokens": 3492, "cache_read_input_tokens": 0, "output_tokens": 4, "cache_creation": {"ephemeral_5m_input_tokens": 0, "ephemeral_1h_input_tokens": 3492}, "cost": 0.013155}}

Recorded spend 0.007016, which is 12*1e-6 + 3492*2e-6 + 4*5e-6 at the 1h rate. The streamed 0.013155 is 3492*3.75e-6 + 4*1.5e-5, the 5m sticker rate applied to a genuine 1h write, 1.875x the spend.

Pass-through on POST /anthropic/v1/messages, x-litellm-call-id: 9163406d-b81f-47aa-b542-773e0edebabf, streamed 0.0011001 against a recorded 0.0011361. Pass-through has no deployment, so sticker is correct here and only the dropped input tokens show: the gap is exactly 12*3e-6 = 0.000036.

After (655d107, the branch tip)

Six legs, every one showing the streamed usage.cost equal to the recorded spend.

Leg	Route	Streamed	Recorded	At sticker
5m write	/v1/messages	0.0034045	0.0034045	0.0102135
cache read	/v1/messages	0.000436	0.000436	0.001308
1h write	/v1/messages	0.005314	0.005314	0.015942
chat completions	/v1/chat/completions	0.0007515	0.0007515	0.0022545
anthropic passthru	/anthropic/v1/messages	0.01932225	0.01932225	same
openai passthru	/openai/v1/chat/completions	1.845e-05	1.845e-05	same

curl -sS -D legB.headers -X POST http://127.0.0.1:52943/v1/messages \
  -H 'Content-Type: application/json' -H 'Authorization: Bearer sk-1234' \
  -H 'anthropic-version: 2023-06-01' \
  -H 'anthropic-beta: extended-cache-ttl-2025-04-11' \
  --data @legB.json

The 1h write leg is the sharpest of the six, x-litellm-call-id: 2bc4d9ae-fe45-4684-b8f0-338116feddd7:

data: {"type": "message_delta", "delta": {"stop_reason": "end_turn", "stop_sequence": null, "stop_details": null}, "usage": {"input_tokens": 17, "cache_creation_input_tokens": 2576, "cache_read_input_tokens": 0, "output_tokens": 29, "cache_creation": {"ephemeral_5m_input_tokens": 0, "ephemeral_1h_input_tokens": 2576}, "cost": 0.005313999999999999}}

GET /spend/logs?request_id=2bc4d9ae-fe45-4684-b8f0-338116feddd7
{"spend": 0.005313999999999999, "metadata": {"cost_breakdown": {"input_cost": 0.005168999999999999, "total_cost": 0.005313999999999999, "output_cost": 0.000145, "cache_creation_cost": 0.0051519999999999995}}}

17*1e-6 + 2576*2e-6 + 29*5e-6 = 0.005314 on both sides. Had those 1h tokens been billed at the configured 5m write rate the answer would be 0.003382, and at sticker 0.015942, so this leg pins the tiered cache-write half of the fix on its own. The cache-read leg pins the other half: 17*1e-6 + 2590*1e-7 + 32*5e-6 = 0.000436, with the 17 non-cached input tokens counted rather than dropped.

The OpenAI pass-through leg is there because it is the only route that reaches the chat.completion.chunk branch. On a routed /v1/chat/completions the chunk arrives as a ModelResponseStream object, which _process_chunk_with_cost_injection skips, and the cost is already correct before this PR. /openai/* delivers those frames as raw SSE through PassThroughStreamingHandler, and it prices to 1.845e-05 on both sides, matching 19*1.5e-7 + 26*6e-7.

On the cost_breakdown restore

Pricing a frame through the request's own logging object also stamps cost_breakdown onto that object, and the pass-through handlers compute their final cost with a bare completion_cost. _logging_obj_cost_or_none therefore snapshots that field, and the cost-failure debug key, and puts both back once the cost is read.

That guard is defensive rather than a fix for something observable. Anthropic emits exactly one usage-bearing message_delta, at the end of the stream, and only message_delta is priced, so the single frame that gets priced already carries the full cumulative usage and a frame-derived breakdown is numerically identical to the correct one. Checked directly: the pass-through spend row carries a present and correct cost_breakdown both at 101ef7e167 and at 655d10775c, with no failure-debug key on either. The restore would start to matter for a provider that emits several usage-bearing frames mid-stream, or for a pricing failure that stamps the debug key.

Type

🐛 Bug Fix

Caveats (if any)

Moderately serious. The cost still comes out wrong when no logging object reaches the injection point. _streamed_usage_cost prefers litellm_logging_obj._response_cost_calculator, which knows the deployment and therefore its negotiated prices, and falls back to _completion_cost_or_none, which prices by model name off the public cost map. On the fallback the streamed number is sticker again and a discounted deployment will still disagree with its spend row. Every route exercised above takes the preferred path; the fallback is there so a missing logging object degrades to the old behavior instead of dropping cost entirely.

Moderately serious. The arithmetic assumes Anthropic reports input_tokens exclusive of cache tokens, so the rebuilt usage adds cache reads and cache writes on top of it. That matches what Anthropic documents and what the frames above show, and it is the same assumption the non-streaming path already makes, but a provider-side change to inclusive counting would double count the cached tokens rather than fail loudly.

Minor. The cost_breakdown restore is defensive and changes nothing observable on any route tested here, for the reason set out above: the spend row's breakdown is present and correct both before and after. It is not covered by the e2e proof because there is no user-visible difference to capture. It earns its place by keeping a mid-stream pricing call from leaving a partial breakdown behind on a provider that emits more than one usage-bearing frame.

Minor. The proof drives the routes with curl rather than an SDK. The thing under test is a field in the raw SSE frame, so curl reads exactly what any client's stream parser would hand to application code, and no SDK sits between the assertion and the bytes. An SDK run would add a layer without adding evidence.

Minor. The /v1/chat/completions leg runs against Anthropic direct rather than Bedrock. Bedrock on that route is broken on litellm_internal_staging independently of this PR, unrelated to cost: converse_handler.py reads credentials.access_key before the bearer-token branch and raises AttributeError on a bearer-auth deployment. Filed as LIT-5928, not touched here. The Bedrock legs above go through /v1/messages and /anthropic/v1/messages, which are unaffected.

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/177117598f214818bc610155fb711c6a
Requested by: @mateo-berri

---

Note

Medium Risk
Touches proxy streaming cost injection and live logging-object pricing, which can affect billed usage.cost on the wire. Snapshot/restore of cost_breakdown is defensive, but a pricing-path bug could still misreport stream costs.

Overview
Makes streamed usage.cost match the spend the gateway records when include_cost_in_streaming_usage is on.

Anthropic message_delta frames now rebuild usage via AnthropicConfig.calculate_usage instead of treating input_tokens as the full prompt. That bills uncached input plus cache reads, and keeps the 5m/1h cache-creation split so 1h writes are not priced at the cheaper 5m rate.

Chunks are priced through the call’s _response_cost_calculator so custom deployment rates apply, with a fallback to model-name sticker pricing. Side effects on cost_breakdown and the cost-failure debug key are snapshotted and restored so a mid-stream price does not leak into the spend log. The logging object is threaded through both the chat stream path and pass-through SSE.

^{Reviewed by Cursor Bugbot for commit 655d107. Bugbot is set up for automated code reviews on this repo. Configure here.}

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[greptile-apps]

Greptile Summary

Updates streamed usage-cost calculation to align with recorded spend:

• Normalizes Anthropic streamed usage through the existing Anthropic transformation, including cache-read and cache-creation token details.
• Uses the request logging object’s cost calculator so deployment-specific pricing, discounts, and margins are reflected in streamed costs.
• Preserves logging state while calculating streamed cost and forwards the logging object through proxy and pass-through streaming paths.
• Adds regression coverage for cached tokens, one-hour cache writes, custom pricing, fallback pricing, and logging-state preservation.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
litellm/llms/anthropic/chat/transformation.py	Broadens Anthropic usage transformation input typing to accept mappings without changing its runtime normalization behavior.
litellm/proxy/common_request_processing.py	Rebuilds streamed Anthropic usage and prices it through request-specific logging configuration while restoring mutable logging state.
litellm/proxy/pass_through_endpoints/streaming_handler.py	Passes the active logging object into cost injection for pass-through streaming responses.
tests/test_litellm/proxy/test_common_request_processing.py	Adds focused regression tests covering cached usage normalization, custom deployment pricing, fallback behavior, and state restoration.

_{Reviews (4): Last reviewed commit: "fix(cost): keep mid-stream pricing from ..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 94.87179% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/common_request_processing.py	94.87%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_fix_messages_stream_cost_cache_tokens (655d107) with litellm_internal_staging (4e02e7e)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (35416c7) during the generation of this report, so 4e02e7e was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 655d107. Configure here.}

[mateo-berri]

LGTM