fix(spend): read what a request cost from the record instead of pricing it again

[tin-berri]

TLDR

Problem this solves:

• Auto-router savings re-price the request that ran, instead of reading what it was billed
• A priority-tier request is re-priced at standard rates, so the driver reads half the row's own spend
• Neither the tier nor the regional uplift cancels between the two arms; both are per-model

How it solves it:

• The served arm reads cost_breakdown, the decomposition the cost calculator already recorded
• The baseline, which never ran, is still priced, now on the basis the biller used
• CostBreakdown carries that basis, because the tier cannot be recovered after the fact

Relevant issues

• Follow-up to feat(spend): add net auto-router savings to the cost-optimization dashboard #35521, which added the driver; this fixes how its served arm is priced
• The driver was the only downstream consumer in the tree still re-pricing a completed request
• Two smaller things in the same path: a router fetched on every spend write, and messages resolved twice per hook

Linear ticket

Pre-Submission checklist

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Not yet captured against a live proxy, and the gap is stated rather than glossed. What is owed is real auto-routed traffic sent with "service_tier": "priority" against a model that has priority rates, then /user/daily/activity showing autorouter_savings_spend computed against the same number the row's spend column carries.

The arithmetic the change turns on, from the shipped cost map, 20k prompt and 1k completion:

gpt-5.4-mini, service_tier=priority
  billed by litellm                       0.024000
  re-priced by the savings driver today   0.012000   <- half

baseline gpt-5.5 against a served model with no tiered rates
  baseline at standard                    0.130000
  baseline at priority                    0.260000
  baseline at eu residency                0.143000

Type

🐛 Bug Fix

Changes

• The served arm reads input_cost + output_cost off the recorded cost_breakdown rather than re-pricing the request from its tokens
• CostBreakdown gains service_tier and data_residency, the basis the cost was computed on, written at all three sites that record a breakdown
• The baseline arm is priced on that recorded basis, so both arms sit on one price book
• llm_router is passed as a provider, so a spend write that was never auto-routed no longer fetches and discards one
• The complexity router resolves its messages once per hook and threads the list into the classifier

Why the served arm cannot be recomputed. It is the request that ran. Recomputing it means restating every pricing dimension the biller applied, and each one omitted is a silent disagreement with the spend column beside it. cost_breakdown is the established carrier for this: the cost calculator records it, it rides the standard logging payload into the spend log's metadata, and OTEL, the log drawer and the response headers all read it rather than re-deriving. input_cost already carries the cache buckets, so it and output_cost sum to exactly what generic_cost_per_token returns; the separate cache_read_cost entry decomposes that sum rather than adding to it. Tool spend, discount and margin stay out, because the counterfactual cannot be priced with them and charging them to one arm alone would read as the router losing money on every tool call.

Why the basis has to be recorded rather than re-derived. The tier the biller used comes from optional_params, which no log record carries. The served tier that does survive, on the usage object, is a different fact with the opposite precedence, and only Anthropic reports it there while no Anthropic entry in the cost map has a tiered rate, so reading it would move no number on any request. data_residency is not persisted anywhere at all today.

Why it does not cancel. The uplift is a per-model multiplier, so the symmetric case is 1.1*A - 1.1*B, understating by 10% rather than cancelling. Tier ratios are per-model and not uniform, and coverage is asymmetric: 100 of 2986 entries carry a priority rate, 18 carry a regional uplift.

Things a reviewer will ask about

There is no backfill, so the metric's basis changes on the deploy date. Rows written before this carry no basis and price at standard rates, which is exactly what they do today. Only new rows price as billed.

Cross-vendor comparisons are one-sided by construction, and that is the cost map, not this change. Azure bakes region into the model name (azure/us/gpt-5.4 carries the uplift in its rate, with no multiplier key), so an Azure arm can never take a residency uplift while an OpenAI arm always can. Anthropic, Gemini and Bedrock have no tiered rates at all.

gpt-5-nano will produce large negative savings on priority traffic, and that is a cost-map bug this change surfaces rather than causes. Its input_cost_per_token_priority is 50x its base rate with no matching output key, the only such entry in the map. The invoice is affected identically, so the driver is right to mirror it.

The realtime path records a residency but no tier, and that is the accurate basis rather than a missing one. handle_realtime_stream_cost_calculation has no service_tier parameter and never passes one to generic_cost_per_token, so a realtime request is not tier-priced; the two realtime entries in the cost map carry a regional uplift and no tiered rates at all, so a tier there would resolve to the base key and change nothing. The field records the basis the cost was computed on, so None is what that row was billed at, and the baseline is priced the same way. Anyone later adding a tier to that path has to pass it to the breakdown as well as to the pricer, or the recorded basis will understate what was used.

Compression and prompt caching are still tier-blind, deliberately. _input_and_cache_read_cost reads flat rate keys, so those two drivers ignore tiered pricing the same way. That is pre-existing shipped behaviour on drivers this PR does not introduce, and changing their numbers moves the dashboard for every existing customer, so it belongs in its own change.

QA runbook

1. Configure an auto-router with autorouter_savings_baseline_model set to a model that has priority rates, e.g. openai/gpt-5.5
2. Send an auto-routed request with "service_tier": "priority" in the body
3. GET /user/daily/activity and compare autorouter_savings_spend against the spend on the same row; the served arm must now reconcile with what was billed
4. Repeat without service_tier; the number must be unchanged from before this PR

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[tin-berri]

@greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

Score: 4/5

The core fix is architecturally sound and the implementation is clean. Here's my breakdown:

What's strong:

• The root cause diagnosis is correct: re-pricing the served arm means silently restating every billing dimension the cost calculator applied, and any missed dimension produces a permanent disagreement with the spend column. Reading from cost_breakdown eliminates that class of error entirely.
• PricingBasis as a NamedTuple with a sentinel _STANDARD_RATES is clean. The non-string type guard in _pricing_basis is the right place to catch JSON round-trip corruption before it causes a silent AttributeError → $0.00 inside the pricer.
• The lazy callable pattern for llm_router (_get_llm_router instead of _get_llm_router()) is a correct micro-optimization — every spend write reached compute_savings_spend, and the router lookup was wasted on the non-auto-routed majority.
• The _resolve_messages deduplication in the complexity router is a contained, clear fix.
• Tests are well-structured: the negotiated-rate pin test is a solid guard against re-pricing regressions, and the parametrized basis test covers the four non-trivial cases including the JSON corruption guard.

What gives me pause:

1. Asymmetric service_tier on the realtime path. handle_realtime_stream_cost_calculation picks up data_residency but not service_tier. If the realtime API can't be called with a service tier this is correct, but the PR doesn't say so, and the same savings driver reads both fields off the breakdown. A row written by the realtime path will have data_residency but silently miss service_tier.

2. CI not passing. The author's own checklist leaves that box unchecked. Until CI is green it's hard to know whether any of the surrounding paths broke.

3. No live traffic validation yet. The QA runbook is written but the author explicitly notes the proof isn't captured. For a fix whose correctness depends on the numbers reconciling end-to-end, that's the one thing I'd want before merging.

The design decision is right and the implementation follows from it cleanly. The realtime asymmetry and the missing CI green are the two things standing between this and a 5.

[greptile-apps]

Greptile Summary

This PR makes auto-router savings use the served request’s recorded token cost and carries its pricing tier and residency basis into baseline pricing.

• Adds pricing-basis fields to CostBreakdown and propagates them through cost calculation and logging.
• Lazily retrieves the configured router only for applicable auto-router savings calculations.
• Reuses resolved messages across the complexity-router hook.
• Adds regression tests for recorded served costs and tier/residency-adjusted baselines.

Confidence Score: 5/5

The PR appears safe to merge, with no concrete changed-code defect identified.

The recorded cost and pricing basis are propagated through the existing logging metadata path, consumed defensively by the savings calculation, and covered by focused regression tests.

Important Files Changed

Filename	Overview
litellm/cost_calculator.py	Records service tier and data residency alongside token cost components on the relevant completion-cost paths.
litellm/litellm_core_utils/litellm_logging.py	Persists the resolved pricing basis in the standard CostBreakdown payload.
litellm/proxy/db/db_spend_update_writer.py	Passes the router provider lazily and forwards recorded cost-breakdown metadata into savings calculation.
litellm/proxy/spend_tracking/savings.py	Uses recorded input and output costs for the served arm and applies the recorded pricing basis to the counterfactual baseline.
litellm/router_strategy/complexity_router/complexity_router.py	Resolves structured messages once and reuses the result without changing the pre-existing resolution timing.
litellm/types/utils.py	Extends the optional cost-breakdown schema with service-tier and data-residency fields.
tests/test_litellm/proxy/spend_tracking/test_savings.py	Adds focused coverage for recorded served costs, priority pricing, residency uplift, legacy rows, and malformed basis values.

_{Reviews (1): Last reviewed commit: "fix(spend): read what a request cost fro..." | Re-trigger Greptile}

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit c3110bc. Configure here.}

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit c3110bc. Configure here.}

[tin-berri]

Correct as written: the realtime path never tier-prices, and both realtime cost-map entries have zero tiered rate keys, so None is that row's real basis. Documented in the description.

[codecov]

Codecov Report

❌ Patch coverage is 96.77419% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/spend_tracking/savings.py	96.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[mateo-berri]

LGTM. Thanks!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_savings_recorded_cost (c3110bc) with litellm_internal_staging (cb8c734)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (9e3a8df) during the generation of this report, so cb8c734 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩