feat(proxy): per-component response cost headers - #36965
Merged
mateo-berri merged 4 commits intoAug 15, 2026
Merged
Conversation
- Extract input_cost, output_cost, cache_read_cost, cache_creation_cost, reasoning_cost, and tool_usage_cost from logging object cost breakdown - Populate x-litellm-response-cost-* component headers in ProxyBaseLLMRequestProcessing.get_custom_headers - Ensure headers are omitted when cost breakdown is absent or values are None - Add comprehensive test suite covering component headers, math invariants, caching, reasoning, and discounts/margins
|
|
Contributor
Greptile SummaryThe PR adds per-component cost headers to non-streaming proxy responses while preserving existing total, discount, and margin headers
Confidence Score: 5/5The PR appears safe to merge No blocking failure remains
|
| Filename | Overview |
|---|---|
| litellm/proxy/common_request_processing.py | Adds component cost extraction and response headers, with previous direct callers migrated safely to named-field access |
| tests/test_litellm/proxy/test_common_request_processing.py | Adds component-header coverage and updates all helper callers to the expanded named return contract |
Reviews (4): Last reviewed commit: "fix(proxy): emit uncached input cost so ..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Contributor
Contributor
|
bugbot run |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Contributor
Contributor
|
bugbot run |
Contributor
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
1 issue from previous review remains unresolved.
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 7562445. Configure here.
Contributor
Contributor
|
bugbot run |
Contributor
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 05beb7a. Configure here.
mateo-berri
merged commit Aug 15, 2026
cba2bea
into
BerriAI:litellm_internal_staging
75 checks passed
This was referenced Aug 15, 2026
fix(router): honor key-level tag filtering in pre-routing and pin auto-router e2e regressions
#37366
Merged
Duxl-Ai
pushed a commit
to Duxl-Ai/litellm
that referenced
this pull request
Aug 20, 2026
Seven live e2e tests covering cost-tracking regressions that currently ship unnoticed: cache-write tokens billed at the cache-creation rate (BerriAI#34046), per-component cost_breakdown on the spend row (BerriAI#31686), cache reads billed at the cache-read discount on streamed calls (BerriAI#34812), cache tokens surviving the anthropic-messages to Responses bridge (BerriAI#34957), priority-tier rates applied to input, output and reasoning (BerriAI#35923, BerriAI#35925), the per-component response cost headers summing to the total (BerriAI#36965), and cost injected into the final usage frame of an /openai passthrough stream (BerriAI#36503). Every test registers its own deployment with a distinct custom rate per component, so a component billed at the wrong rate cannot pass. The shared helpers in cost_rows.py encode the one thing the two surfaces disagree on: the spend row's input_cost is gross of cache while the response's cost-input header is net of it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
Problem this solves:
How it solves it:
x-litellm-response-cost-*component headers next to the totalUser Flow
Before: a platform team fronting all LLM traffic through the LiteLLM proxy wants to show each internal app where its spend goes, but callers only ever see one total, so they rebuild the split by hand from their own pricing tables
x-litellm-response-cost: 7.4e-06and the body'susageblock carries token counts (prompt_tokens,completion_tokens,cached_tokens,reasoning_tokens)usageat allAfter: the same response states the split directly and it sums to the total
x-litellm-response-costtotal and now alsox-litellm-response-cost-input,-output,-cache-read,-cache-creation,-reasoning, and-tool-usageRelevant issues
Closes #36875
Linear ticket
Resolves LIT-5617
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Live-proxy legs with real OpenAI and Anthropic spend, no mocks. Before leg at commit c9917cb (this PR's merge base), after leg re-proven at commit 05beb7a (this PR's behavioral tip). Each leg booted a bootstrapped worktree with
.venv/bin/python litellm/proxy/proxy_cli.py --config qa_config.yaml --port <port>and hit all three unified endpoints: /v1/chat/completions and /v1/responses with gpt-5.4-nano, /v1/messages with claude-haiku-4-5 carrying a ~6k-tokencache_control: ephemeralsystem block sent twice to exercise cache creation then cache read. An earlier eight-scenario after leg at 9079e4c surfaced that the input header still contained the cache costs nested inside it (Cursor Bugbot flagged the same double-count); 05beb7a fixed that by subtracting the cache components from the input header, the same split the Admin UI's cost breakdown viewer shows, and the leg below re-proves the final contractBefore leg (c9917cb, port 41387): component headers absent everywhere
A1 and A2 (identical, 200 OK, prompt_tokens=1177, completion_tokens=4, answer "391"):
B (/v1/responses, 200 OK, input_tokens=12, output_tokens=5):
C1 and C2 (/v1/messages; C1 cache_creation_input_tokens=6242, C2 cache_read_input_tokens=6242):
Across all five before-leg responses, zero occurrences of
-input,-output,-cache-read,-cache-creation,-reasoning, or-tool-usageAfter leg (05beb7a, port 46746): six component headers present and additive
A1 (chat completions, gpt-5.4-nano, prompt_tokens=1417, completion_tokens=4, answer "391"; A2 byte-identical):
B (/v1/responses, input_tokens=12, output_tokens=5):
C1 (/v1/messages, cold cache, cache_creation_input_tokens=6242, input_tokens=12) then C2 (warm, cache_read_input_tokens=6242):
C1n/C2n (same shape, fresh nonce guaranteeing a cold cache; cache_creation then cache_read of 6267 tokens) behaved identically: creation 0.00783375, read 0.0006267, input 1.2e-05 on both
Br (/v1/responses with
"reasoning": {"effort": "high"}, input_tokens=25, output_tokens=175, reasoning_tokens=168):Sum invariant across all eight after-leg responses
Python check over the captured header files (rel_tol 1e-9, absent header = 0). Additive = input + cache-read + cache-creation + output + tool-usage; nested = input + output + tool-usage (the pre-fix contract)
On every cache scenario the input header is exactly the 12 uncached tokens x 1e-06 = 1.2e-05, cache creation = tokens x 1.25e-06, and cache read = tokens x 1e-07, matching Haiku 4.5 pricing to the digit (rel_tol 1e-9)
Component semantics:
_store_cost_breakdown_in_logging_objin cost_calculator.py records the full prompt cost (cache pricing already applied) as input_cost, so emitting it unchanged would double-count the cache once the cache headers exist. The header path therefore subtracts cache read and cache creation from input before emitting, the exact split the Admin UI's cost breakdown viewer already shows, delivering the additive contract issue #36875 asked for: input + cache-read + cache-creation + output + tool-usage = total, reasoning a subset of output. Spend-log storage is untouchedQA surprises:
Type
🆕 New Feature
Caveats (if any)
Final Attestation