feat(spend): add net auto-router savings to the cost-optimization dashboard - #35402
feat(spend): add net auto-router savings to the cost-optimization dashboard#35402tin-berri wants to merge 24 commits into
Conversation
|
@greptileai can you review this PR, give me a score, and explain why you gave the score |
Greptile SummaryAdds net auto-router savings to the cost-optimization dashboard.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains; the previously reported pricing, metadata-production, fallback-cleanup, queue-aggregation, and migration issues are addressed in the current code.
|
| Filename | Overview |
|---|---|
| litellm/proxy/spend_tracking/savings.py | Prices both counterfactual and selected-model usage through the shared cost engine while transforming cold cache writes into warm baseline reads. |
| litellm/router.py | Records routing decisions and savings baselines together and clears both when a fallback attempt does not use an auto-router strategy. |
| litellm/router_strategy/savings_baseline.py | Resolves provider-qualified, priceable baseline deployments from each router's candidate model groups. |
| litellm/proxy/db/db_transaction_queue/daily_spend_update_queue.py | Aggregates signed auto-router savings across same-key queued requests, including payloads from older pods that omit the field. |
| litellm/proxy/db/db_spend_update_writer.py | Computes and persists auto-router savings through create and increment paths for all daily spend entities. |
| litellm-proxy-extras/litellm_proxy_extras/migrations/20260731000000_add_autorouter_savings_spend/migration.sql | Adds the required non-null savings column with a zero default to all six daily rollup tables. |
| litellm/proxy/management_endpoints/common_daily_activity.py | Carries the new savings driver through daily aggregation and activity responses. |
| ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/UsageTab.tsx | Displays signed auto-router savings in the dashboard card, chart, and positive-only donut visualization. |
Reviews (5): Last reviewed commit: "style: bring the savings changes under t..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Proof of ConceptAdded comprehensive documentation: PROOF_OF_CONCEPT.md This document verifies:
Ready for live-proxy repro once baseline_model stashing is implemented in auto-router hook. |
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
…hboard Adds auto-router as a third savings driver alongside compression and prompt caching: a summary card, a donut segment, and a series in the savings graph. Savings are the net dollars from routing a request to the selected model instead of a counterfactual baseline. The delta prices prompt tokens at each model's input rate and completion tokens at each model's output rate, then subtracts the cache-write penalty the selected deployment incurs on a cold cache. Cache-read discounts stay attributed to the prompt-caching driver to avoid double-counting. The result is floored at zero so an escalation to a pricier model never reads as negative savings. The baseline model defaults to claude-opus-5 and is operator-configurable per deployment via the auto_router_savings_baseline_model litellm_param. It flows AutoRouter to PreRoutingHookResponse to the metadata bucket to SpendLogsMetadata to the daily spend writer, mirroring the routing_decision path, and is stripped from untrusted caller metadata so it cannot be spoofed. Savings accrue into a new autorouter_savings_spend column on the six LiteLLM_Daily*Spend rollup tables; no LiteLLM_SpendLogs queries are added.
d8a5cde to
e30b7f2
Compare
|
@greptileai re-review after force-push to commit e30b7f2; pricing fixed (completion tokens at output rate, cache-write at selected model rate), baseline producer added, migration included |
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 2 · PR risk: 0/10 |
tin-berri
changed the title
…nd surface them Three defects, one cause: the driver was wired by hand at each stage of the savings pipeline instead of going through the owner of each stage. Pricing re-derived per-token arithmetic instead of calling the cost engine. `prompt_tokens` already includes cache-read and cache-creation tokens, so charging the whole total at the flat input rate and then subtracting a separate cache-write penalty priced those tokens twice. Both arms now price the identical usage through `generic_cost_per_token`, so each token is charged once in its own dimension and tiered rates, ephemeral cache-write tiers and regional uplifts stay consistent with what the request was actually billed. On a cache-heavy opus-to-haiku switch the old formula reported $0.0458 against a true $0.0717. The savings baseline was recorded by a second per-attempt writer sitting beside the routing decision, and the exit path that runs when no pre-routing strategy applies only cleared the decision. A fallback to a plain model group therefore kept the previous attempt's baseline, letting a caller who forces a router failure inflate the recorded savings. Both facts now travel from one response through one recorder, so no exit can clear one and leave the other. Aggregation summed every daily metric except this one, so two requests sharing a rollup key kept only the first value, and since the cross-pod Redis drain runs the same merge the field was dropped on every flush. The driver was also absent from the entire read path: no column in the rollup query, no accumulation, no field on the response model. The dashboard read a key the API never sent, so the card, donut segment and graph series would have rendered $0.00 forever however much routing saved. Wiring the write path without the read path is the failure this had already shipped, so the drivers are now enumerated from the response model itself and each is asserted to be summed, accumulated, carried and totalled. Also collapses the twelve hand-written per-field blocks in the daily upsert into one enumeration feeding both the create and the increment.
|
@greptileai re-review at 4897f6e All three findings are fixed, plus a fourth that no bot flagged and that mattered more: |
A malformed usage_object and a model with no discounted cache-read rate both degrade to zero savings rather than failing the daily spend write; pin both so the fail-open contract cannot regress into a raise.
The rationale for pricing both arms through one cost engine belongs in the commit history, not restated above every function.
…ed it Staying on one model writes the prompt cache once and reads it thereafter. Switching leaves the new model cold, so it pays to write the whole prompt again, and that charge exists only because the router switched. Both arms were priced as if each model wrote the cache, which credited the baseline a cache-creation charge it would never have paid again. On a sonnet to haiku switch mid-conversation that phantom write was larger than the entire real cost of the request: the route lost $0.0104 and was reported as having saved $0.0179, with the sign inverted. The baseline is now priced as the warm cache a single-model deployment would have had, so the cold-cache write counts against the saving. A request that read nothing from cache is a genuine first turn the baseline would have paid to write too, so both arms still write there and cold-start savings stay honest. The result is signed rather than floored at zero. A cache-thrashing route is a real cost and the dashboard has to be able to report it; flooring per request would leave a number that can only ever go up and would hide exactly the routing behaviour an operator needs to see. The donut plots only drivers that saved, since a negative slice has no meaning, while the card and the range total keep the signed truth. usd() now sizes and signs off the magnitude so a small loss reads as -$0.0004 rather than $-0.00.
|
Re-review at db15ef3 The cache-write accounting changed materially since the last pass. Previously both arms were priced as if each model wrote the prompt cache, which credited the baseline a cache-creation charge it would never have paid again; a mid-conversation sonnet to haiku switch that lost $0.0104 was reported as saving $0.0179, sign inverted. The baseline is now priced as the warm cache a single-model deployment would have had, so the cold-cache write counts against the saving, and a request that read nothing from cache is treated as a genuine first turn where both arms write. The result is signed rather than floored at zero, so a cache-thrashing route reduces the total instead of disappearing. The donut plots only positive drivers. |
|
bugbot run |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
The two sides of the comparison arrived spelled differently. The spend log records a normalized model name alongside its provider, while the baseline arrives as the operator wrote it in config, with the provider prefixed, implied or absent, so the raw strings were never comparable. Read as a switch, `anthropic/claude-opus-5` against a served `claude-opus-5` priced one deployment against itself, and because the baseline arm is priced warm while the served arm pays its cold-cache write, a request that never changed model reported a $0.0707 loss. That mis-comparison was harmless until the cache fix made the two arms asymmetric, so the identity check has to land with it. Pricing had the same root cause: a baseline resolved without a provider takes whichever vendor owns the bare name. `azure_ai/deepseek-r1` and `deepseek/deepseek-r1` are the same bare model at different rates, and the difference decides the sign, +$0.039 against -$0.0731 on the same request. Both now resolve through `get_llm_provider` to a canonical (model, provider) before being compared or priced, so one deployment spelled two ways is not a switch and every arm is priced under the vendor that serves it. The per-day chart no longer stacks its drivers. Stacking sums the series into one bar, and a driver that goes negative would be drawn below the axis while the rest of the bar still read as the day's total.
…ates Without an auto-router a deployment has to pick one model, and it has to be one that can carry the hardest request, so the counterfactual is the priciest model that router could have chosen. A fixed flagship default measured savings against a model the operator may never have run. A router choosing only between sonnet and haiku saved nobody the price of opus, so every such deployment would have opened the dashboard to savings it was never going to make, and the figure drifted the moment the routes changed. The baseline is now the priciest candidate the router itself can reach, by output rate with input breaking the tie. Routes name model groups rather than models, so each is resolved through the parent router's deployments before being priced, and the default model counts as a candidate. Resolution is lazy and cached, because the parent router's deployments are still being assembled while the auto-router is constructed. `auto_router_savings_baseline_model` still overrides it for operators who would genuinely have run something else. When nothing can be priced the baseline is None and the driver reports zero, since a missing number beats a fabricated one. Total saved needs no change: it sums the drivers, so the derived baseline flows into it. Compression and prompt caching stay priced at the served model's rates, which answers what each optimization saved on the request that actually ran.
"vs. the router's baseline model" defined the number by a config field most operators never set, and now that the baseline is derived it named nothing at all. The comparison is the priciest model the router can reach, so the card says that.
|
@greptileai can you review this and give me a score and explain why |
|
bugbot run |
…yment declares A deployment can name its vendor in `custom_llm_provider` rather than in the model prefix, which is the normal shape for Azure, Bedrock and Vertex. Pricing the bare name alone resolves it to whichever vendor owns that name, or to nothing: `claude-sonnet-4@20250514` prices at $0 without vertex_ai, and `deepseek-r1` raises without azure_ai. Either way the candidate lost the priciest-candidate contest, so the derived baseline silently became a cheaper model and the driver under-reported. Candidates now resolve through `get_llm_provider` and are carried as `provider/model`, which is also what reaches the spend writer, so the baseline resolves back to the vendor that served it rather than to whoever owns the bare name. A candidate with no per-token price is no longer eligible. Nothing that costs nothing can stand in for what the traffic would otherwise have cost, and as a baseline it would report the whole real spend as a loss. That outcome was already unreachable, but only because the served model is drawn from the same candidate set and fallbacks clear the baseline; the driver should not depend on that chain holding.
|
The three previous issues — Score: 4/5 What's well done:
Remaining concerns (why not 5):
The core logic is sound, backward compatibility is properly handled, and the test coverage is thorough. Closing the live-proxy gap and documenting the |
…equest `auto_router_savings_baseline_model` was missing from `all_litellm_params`, so unlike the four auto-router fields beside it, it was classified as a provider-specific parameter and rode along in extra_body where a downstream provider could read it. The guard that exists for exactly this was a hand-written tuple of eleven params, which can only catch a field being removed from the strip list, never a new one that was never registered. That is the way this actually goes wrong, and it is how this field slipped through. The test now derives the list from the params model itself, so any future router-strategy field is covered the moment it is declared. The baseline is also qualified when it comes from configuration, not only when derived. It travels to the spend writer as a bare string with no provider beside it, so an operator writing `deepseek-r1` meaning Azure would otherwise be priced against whoever owns that name. With every baseline qualified at the source, `compute_savings_spend` no longer takes a `baseline_provider` it could never be given: the parameter existing at all was the implicit contract, and removing it is what makes the invariant explicit.
|
Addressed both at c507283. Veria's leak: On the The three schema copies are pre-existing and I have deliberately left them alone; a fix there belongs in its own PR. Live-proxy proof is still genuinely outstanding and I am not claiming otherwise. The read path, dashboard, signed handling and baseline derivation are demonstrated end to end on a local proxy against a real Postgres, but no request has gone through the auto-router because the available Anthropic key is out of credit and there is no embedding key for the semantic router. The pricing math is unit-tested only. bugbot run @greptileai can you reply here with a score and why that score? |
|
3.5 / 5 The two specific issues I raised are addressed correctly at c507283. Here is my full accounting of what I find well-done and what still gives me pause. What is solid
Baseline qualification at source — the auto-router qualifies the baseline via
Signed savings — the deliberate choice not to floor at zero is the right call for an honest dashboard. The test Rollup merge and upsert refactor — the prior omission of Test coverage — the unit-test suite for What still concerns me1. @cached_property
def _derived_savings_baseline_model(self) -> str | None:
return self._most_expensive_candidate()
2.
3. No live-proxy proof. The author is explicit that no real auto-routed request has gone through the system. The read path, dashboard rendering, and schema migration are all unit-tested and locally verified against Postgres, but the end-to-end spend write → daily rollup → API response → UI render chain has not been exercised with real routing traffic. This is acknowledged, not hidden, and the author says it is outstanding before requesting maintainer review — but it does mean the integration path is unvalidated. 4. Three schema copies pre-exist; the migration only covers one. The author states this is deliberate and scopes the fix to a separate PR. That is a reasonable call but it means the rollup column is absent in two of the three schema copies until that follow-up lands, which will cause SummaryThe two issues from the prior review are resolved correctly and the new design is structurally better than what it replaced. The four items above are all real gaps, but items 3 and 4 are acknowledged pre-conditions the author is not claiming to have closed. Items 1 and 2 are correctness/observability concerns worth addressing before this merges. The score reflects solid core work with two remaining gaps in the correctness and observability of the live path. |
The parent router adds and removes deployments while it runs (`model_list` is appended to and popped from during deployment updates and health checks), so a baseline pinned on first access keeps naming a model the router no longer has, and a pricier deployment added later can never become the baseline. Nothing recomputes it, so the value stays wrong for the life of the instance. The cache was never worth having: resolving the baseline over a four-candidate router takes about 60 microseconds against a network call three orders of magnitude larger. Removing it also removes the question the previous round was spent answering, since there is no longer a cached value whose "not computed yet" state has to be told apart from "computed to nothing". A malformed `usage_object` now logs at warning rather than debug. It zeroes the auto-router driver for every affected row, and a shape change in `Usage` would otherwise surface only as a dashboard that quietly reads $0.00.
The card header was a wrapping flex row, and the subtitle is longer on
Cumulative ("Running total saved") than on Per day ("Saved per day"). The extra
width pushed the legend and the accumulation toggle onto a second row, so both
jumped whenever the tab changed.
`CardHeader` is now the row itself, the same structure `SummaryCard` in this file
already uses, with the controls held in a shrink-0 box so they keep their place
whatever the subtitle says. The other two headers in the file are title-only and
correctly stay plain.
|
Pushed e9b4f95. Two things since the last review. The A malformed On the schema copies: I believe that one is mistaken. All three carry the column and are byte-identical (md5 b01b030ff2812b1e25e9f4bc9c6949b6 for schema.prisma, litellm/proxy/schema.prisma and litellm-proxy-extras/litellm_proxy_extras/schema.prisma), and the "Verify schema.prisma copies match root" CI check passes. The single migration in litellm-proxy-extras is the only one that exists because that is the package that ships migrations; the other two copies are not independently migrated. Happy to be corrected if you are seeing something I am not. Live-proxy proof is still outstanding and I am not claiming otherwise. bugbot run @greptileai can you reply here with a score and why that score? |
|
Score: 4/5 The round of revisions is solid. Here's the breakdown: What landed well:
Why not 5/5: Live-proxy proof is explicitly still outstanding. You've said so yourself, and it's the right call not to claim otherwise. Unit tests can verify the arithmetic and the wiring, but they cannot catch a misconfigured deployment where the baseline never reaches the spend writer, or a subtle provider-resolution difference between the router's |
Title, legend, toggle and subtitle all shared a row. The subtitle is longer on Cumulative than on Per day, so it wrapped on one tab and not the other, growing the header by a line and shifting the legend, the toggle and the chart with it. The title and the controls now hold a fixed row and the subtitle sits on its own line beneath, so nothing above the chart depends on how long that text is. The test that guards this selected the subtitle by class and matched a summary card's hint instead, which made it pass with the subtitle moved back into the row. It now finds the element by its text.
The header was hand-rolled rows, so the title, the subtitle, the legend and the tab control all competed for one line. The subtitle is longer on Cumulative than on Per day, so it wrapped on one tab and not the other and the chart moved with it; pinning the controls against shrinking then pushed them past the card edge once the viewport narrowed. CardHeader already solves this. It is a grid that switches to `grid-cols-[1fr_auto]` when a card-action slot is present, sizing the controls to their content and giving the rest to the title, with the description on its own row. Using CardTitle, CardDescription and CardAction removes the bespoke layout rather than tuning it, and the controls wrap inside their own column instead of overflowing. The guard test now anchors on those slots. Its previous selector matched a summary card's hint rather than this subtitle, so it passed with the subtitle moved back into the controls.
|
QA pass on The plumbing is sound end to end. The baseline reaches the spend log metadata, survives serialization, and lands as a non-zero The cold-cache write is still charged on the wrong branch
Numbers below are
The first row is the problem: a cold switch reports more saving than the identical traffic with caching turned off entirely, because the counterfactual is credited a 20k-token cache-write premium it would never have paid, while the model that actually paid it is charged nothing extra. Rows two and three are the same request with one token moved between buckets, and the sign flips on a $0.115 swing The condition is doing real work in the steady-state case, so the fix is not to delete it; the read-nothing case is the one that needs the opposite treatment. Charge the write when Track the previously serving model per conversation and charge the write only on an actual change. Correct, and more state than this is probably worth right now Drop the cache-switch modelling, price identical usage on both arms, and say so on the card. Under-claims, but never reports a number that is wrong in the flattering direction Relatedly, the tooltip on the card no longer mentions the cold-cache write or the possibility of a negative total, while the code still produces both; the bar chart comment two hundred lines down explicitly reasons about negative drivers. Whichever way the semantics land, the popover should describe what the number actually is Smaller findingsA candidate that does not resolve in the cost map drops out of the baseline pool with only a debug log.
The complexity and adaptive routers never populate Deriving the baseline from the router's candidates is a better counterfactual than a hardcoded flagship, and I agree with the change; note it does mean the number answers "versus the priciest model this router could have picked", not "versus opus-5". On a haiku plus sonnet pool the savings are measured against sonnet. Operators who want the flagship comparison need Verification was numeric against the branch: direct |
…paid it The warm baseline was gated on the request having read from cache, so it never applied to the case it exists for. A switch to a cold model reads nothing precisely because that model's cache is empty, so the gate skipped it and priced the baseline as if it too had written the whole prompt. Staying on one model would have had that prompt cached already and paid only the read rate, so the counterfactual side was inflated by a write it would never repeat. On a 20k-token prompt switching opus-5 to haiku that reported +$0.1200 saved against +$0.1000 for the same traffic with caching off entirely, so paying to re-warm a cold model looked better than not caching at all. It now reports +$0.0050, which is worse, as it should be. The gate also made the write bucket a proxy for "this was a switch", which put a cliff between a request that read nothing and one that read a single token: the same prompt moved between +$0.1200 and +$0.0050 depending on one token. Pricing the baseline warm whenever there is anything to write removes the cliff; the two now differ by a rounding error. A first turn of a genuinely new conversation is charged the same way, which understates its saving slightly, since nothing was cached anywhere and the baseline would have paid to write too. A single rollup row cannot tell that apart from a switch, and understating is the safe direction for this number. The test that asserted both arms write on a cold start was asserting the bug.
|
Re-QA'd on Numbers below are from the branch, scriptimport os
os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] = "True"
import litellm
from litellm.proxy.spend_tracking.savings import compute_autorouter_savings
from litellm.types.utils import PromptTokensDetailsWrapper, Usage
PROMPT, OUT = 20_000, 1_000
opus = litellm.get_model_info("claude-opus-5", "anthropic")
haiku = litellm.get_model_info("claude-haiku-4-5", "anthropic")
def usage(read: int, write: int) -> Usage:
return Usage(
prompt_tokens=PROMPT,
completion_tokens=OUT,
total_tokens=PROMPT + OUT,
prompt_tokens_details=PromptTokensDetailsWrapper(
cached_tokens=read, cache_creation_tokens=write, text_tokens=max(PROMPT - read - write, 0)
),
)
def savings(read: int, write: int) -> float:
return compute_autorouter_savings("anthropic/claude-opus-5", "claude-haiku-4-5", "anthropic", usage(read, write))
print("opus-5 -> haiku-4-5, 20k prompt / 1k completion held constant")
for label, read, write in (
("cold switch read 0 write 20000", 0, 20_000),
("one token read read 1 write 19999", 1, 19_999),
("steady state read 19900 write 100", 19_900, 100),
("warm read only read 20000 write 0", 20_000, 0),
("caching off read 0 write 0", 0, 0),
):
print(f" {label} {savings(read, write):+.4f}")
true_first_turn = (PROMPT * opus["cache_creation_input_token_cost"] + OUT * opus["output_cost_per_token"]) - (
PROMPT * haiku["cache_creation_input_token_cost"] + OUT * haiku["output_cost_per_token"]
)
print(f"\ngenuine first turn: true {true_first_turn:+.4f}, reported {savings(0, 20_000):+.4f}")
print("\none conversation, first turn cold then N-1 warm turns")
for turns in (1, 2, 5, 10):
reported = savings(0, 20_000) + (turns - 1) * savings(19_900, 100)
truth = true_first_turn + (turns - 1) * savings(19_900, 100)
print(f" turns={turns:2d} reported {reported:+.4f} true {truth:+.4f} {reported / truth:.0%} of it")
print("\ncold switch by completion length, 20k cached prompt")
for out_tokens in (200, 500, 750, 1_000, 4_000):
baseline = PROMPT * opus["cache_read_input_token_cost"] + out_tokens * opus["output_cost_per_token"]
paid = PROMPT * haiku["cache_creation_input_token_cost"] + out_tokens * haiku["output_cost_per_token"]
print(f" completion {out_tokens:5d} tokens {baseline - paid:+.4f}")On the tradeoff you namedAgreed with the direction, and agreed that recording the previous model belongs in its own PR. The one thing I'd push back on is the word "slightly". The first turn of a new conversation with The sharper edge is that the understated value can be negative. The write premium is fixed by prompt size while the saving grows with completion length, so on a 20k cached prompt the reported number crosses zero at roughly 750 completion tokens; anything shorter reads as a loss. Traffic that is mostly single-turn with a large cached system prompt and a short answer, which is a pretty ordinary agent shape, can therefore make the card show the auto-router losing money on requests that each genuinely saved about $0.12 None of that makes the current behavior wrong to ship; it makes the caveat load-bearing. Two things would carry it: put it in the PR description as a known limitation rather than only in the commit body, and put it on the card. The popover currently reads "What this traffic would have cost had every request gone to the most expensive model the auto-router can route to, minus what it actually cost", which does not prepare anyone for a negative total or for a systematic undercount, and the earlier copy that did mention the cold-cache write was dropped in Still open from the previous passThe Azure one is the one I'd actually fix before merge.
The complexity and adaptive routers still never populate Test run
The two new tests are the right ones; |
…ardest tier The savings driver only ever worked for the semantic auto-router. It was the one strategy router that declared a baseline model, so every complexity, quality and adaptive router fell through to no baseline, compute_autorouter_savings short-circuited, and autorouter_savings_spend was structurally zero. A deployment routing exclusively through complexity routers saw $0.00 against real spend, which reads as "routing saved nothing" rather than "nothing was measured". A complexity router's tier ladder already names the model an operator would have had to run to serve the hardest request, so the counterfactual is the priciest model in the REASONING tier, falling back to the highest-severity tier configured when REASONING is absent. Deliberately not the priciest model the router can reach: a pricey model sitting in a low tier is a choice the router made, not a ceiling it was bounded by, and crediting savings against it would overstate them. The pricing and resolution both routers need is now one module rather than two copies. Deployments resolve through model_info.base_model before litellm_params .model, because on Azure the latter is a deployment name that is absent from the cost map; without that hop an Azure candidate never prices, and if it was the priciest the baseline silently drops to the second priciest and understates every saving. resolve_baseline can no longer raise. It is read on the routing path while decorating a request that is about to be served, and a dashboard counterfactual must not be able to take a live request down; an unresolvable baseline zeroes the driver instead.
…ent name `litellm_params.model` is not always a model. On Azure it is the deployment name, which is absent from the cost map, so pricing it directly returned nothing and the candidate dropped out of the pool with only a debug line. If that candidate was the priciest, the baseline quietly became the second priciest and every saving was understated; if the pool was all Azure, nothing priced, the driver was disabled and the card read $0.00 with nothing at default log level saying why. Wildcard and aliased deployments drop the same way. `model_info.base_model` is what names the real model, and router.py already resolves pricing through the same base_model, base_model, model chain in `_get_model_from_deployment` and its cooldown counterpart. Candidate resolution now follows it.
`litellm/proxy/_experimental/out` is release build output, not source. Rebuilding it locally to look at the dashboard left the files dirty in the worktree, and a `git add -A` on the previous commit swept 145 of them in. The path is restored to exactly what litellm_internal_staging carries, so this branch no longer touches it.
litellm/proxy/_experimental/out is release build output, not source, and this branch has no business touching it.
…he split `_baseline_usage` rebuilt `PromptTokensDetailsWrapper` from three fields, so everything else the request was priced on came back `None`. Audio, image and video counts, the character and image counts Vertex prices multimodal embeddings by, and the audio and video durations all vanished from the baseline arm, which then priced a text-only request that never ran. On a 20k prompt carrying 4k image tokens the baseline came out 36% light, and since it is the arm being subtracted from, every one of those requests reported less saving than it earned. Dump the details and override only the cache buckets, so a field added to the wrapper later is carried without anyone remembering to add it here. The 5m/1h creation breakdown is dropped along with the creation count: `generic_cost_per_token` charges a cache-write whenever that breakdown is present, even against a zeroed count, which would have put the phantom write back on the baseline for every long-cache request. The exclude set is a frozenset fed straight to `model_dump`, so the override is a call signature rather than a dict literal the reader has to trust.
The popover explained the counterfactual and stopped there, so nothing prepared a reader for a total below zero or for a first turn reading at a fraction of what it saved. The copy that did mention the cold-cache write was dropped in f66687b; this puts it back and adds the two consequences that follow from it.
…itellm_lit5046_autorouter_savings_fixes # Conflicts: # tests/test_litellm/router_strategy/test_complexity_router.py
… budgets `Optional[PreRoutingHookResponse]` becomes the union form the codebase is moving to, the recorded metadata is built in one shot rather than seeded empty and mutated, and the spend log's usage object is typed as a Mapping of object rather than a bare mutable dict, which also drops the Any the strict gate bans. The local run compared against a stale base and read clean, so these only surfaced once CI diffed against the real merge base.
|
@greptileai can you review, give a score and tell me why |
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 5410306. Configure here.
1 participant
TLDR
Problem this solves:
How it solves it:
Relevant issues
Resolves LIT-5046
Linear ticket
Resolves LIT-5046
Pre-Submission checklist
Changes
Auto-router savings join compression and prompt caching as a third optimization driver on the cost-optimization dashboard, with a summary card, a donut segment and a series in the savings graph across the cumulative and per-day views.
What it is measured against. Without an auto-router a deployment has to pick one model, and it has to be one that can carry the hardest request, so the counterfactual is the priciest model that router could have chosen. That baseline is derived from the router's own candidates rather than fixed, because a hard-coded flagship credits savings against a model the operator may never have run: a router choosing only between sonnet and haiku saved nobody the price of opus. Routes name model groups rather than models, so each is resolved through the parent router's deployments and qualified with its provider before being priced; a bare name can otherwise resolve to a different vendor's rates, or to nothing at all. A deployment is priced by its
base_modelwhere it has one, which is how Azure deployments are priced everywhere else in the codebase; readinglitellm_params.modelalone dropsazure/my-deployment-nameout of the candidate pool and silently measures against the second priciest model instead.Each router supplies its own candidates. A semantic auto-router offers every model group its routes can reach. A complexity router offers its hardest configured tier rather than everything it can reach, because the tier ladder already names the model an operator would have had to run to serve the hardest request; a cheap tier is a choice the router made, not a ceiling it was bounded by.
auto_router_savings_baseline_modeloverrides the derived value for operators who would genuinely have run something else, and is qualified the same way. When nothing can be priced the baseline is absent and the driver reports zero rather than inventing one.How the dollars are computed. Both arms price the request's real usage through
generic_cost_per_token, litellm's own cost engine, rather than re-deriving per-token arithmetic, so tiered rates, ephemeral cache-write tiers and regional uplifts stay consistent with what the request was actually billed.Cache is where this gets decided.
prompt_tokensalready includes cache-read and cache-creation tokens, so charging them again at the input rate prices the same tokens twice. More importantly, a deployment that stays on one model writes the prompt cache once and reads it thereafter; only switching leaves a model cold and forces it to write the whole prompt again. The baseline is therefore priced as the warm cache a single-model deployment would have had, so the cold-cache write counts against the saving instead of being credited to both sides. That holds whether or not the request also read anything, because a switch to a cold model reads nothing precisely because its cache is empty; gating the warm baseline on a read would charge the baseline a write it never repeats, and a cold switch would then report a larger saving than the same traffic with caching turned off.Both arms are priced on the request's whole usage, not just its cache split. Audio, image and video counts, and the character counts and durations Vertex prices multimodal embeddings by, all travel into the baseline; pricing it as a text-only request would have made multimodal traffic report less saving than it earned.
Known limitation: a first turn is undercounted, and can read as a loss. A cold cache reads the same in the spend log whether it is cold because the router switched or because the conversation is new, so a genuine first turn is charged as a switch. The gap is not a rounding error. On a 20k prompt with a 1k completion, an opus-5 to haiku-4-5 first turn reports +$0.005 against a true +$0.12, and a conversation only converges as it lengthens: 22% of its real value at two turns, 50% at five, 69% at ten. Because the write premium is fixed by prompt size while the saving grows with completion length, the understated number can cross zero; on a 20k cached prompt anything under roughly 750 completion tokens reads as a loss. Traffic that is mostly single-turn with a large cached system prompt and a short answer, an ordinary agent shape, can therefore show the auto-router losing money on requests that each genuinely saved around $0.12.
The error never runs in the flattering direction, which for a savings claim is the half that matters, and the card says so in its popover. Telling a first turn from a switch needs the previously served model, which is not recorded today; that is LIT-5087, which follows this PR
Why it is signed. Whether a switch pays off is a race between the rate gap and the cache-write cost, and a narrow gap loses: a mid-conversation sonnet to haiku switch costs $0.0104 on the shape used in the tests. Flooring that at zero would leave a number that can only move in one direction and would hide exactly the routing behaviour an operator needs to see, so a cache-thrashing route reduces the total. The donut plots only drivers that saved, since a negative slice has no meaning, while the card and the range total keep the sign.
Where it is stored and read. Savings accrue into a new
autorouter_savings_spendcolumn on the six LiteLLM_Daily*Spend rollup tables, declaredNotRequiredbecause rows queued by a pod on the previous release carry no such key. The field is summed by the rollup merge that the cross-pod Redis drain also runs, and is carried through the aggregation query, the per-row accumulation and the response model, so the dashboard reads a value the API actually sends. Regression tests enumerate the savings drivers from the response model itself and assert each one is summed, accumulated, carried and totalled, so a driver added later cannot be half-wired.The baseline is recorded alongside the routing decision by a single per-attempt writer, so a fallback that re-enters the hook with no strategy clears both facts together. It is stripped from untrusted caller metadata and registered in
all_litellm_params, so a caller cannot supply one and a provider never receives it.Screenshots / Proof of Fix
Partial, and the gap is stated rather than glossed.
Demonstrated on a live proxy on port 4300 against a real Postgres, at commit
f66687b96d, with the daily rollup rows written bycompute_savings_spenditself:The negative day reduces the total rather than being floored away. On the base commit the response model carried no such field at all, so the dashboard would have rendered $0.00 permanently however much routing saved.
Not demonstrated: no request has been routed through the auto-router against a real provider, because the available Anthropic key is out of credit and there is no embedding key for the semantic router. The read path, the schema migration, the signed handling, the dashboard render and the baseline derivation are exercised end to end; the pricing arithmetic is covered by unit tests only. That is the remaining gap before this is ready for a maintainer
Type
New Feature
Final Attestation
Note
Cursor Bugbot is generating a summary for commit 5410306. Configure here.