feat(ui): rebuild the auto-router benchmarks tab around savings and cache behaviour - #35488
Conversation
Greptile SummaryThe PR adds a new Auto-Router Benchmarks dashboard tab that aggregates router savings, session statistics, cache hit buckets, and cache-warming estimates.
Confidence Score: 4/5The aggregate savings and mixed-TTL warming calculations need correction before merging because they can show operators materially misleading benchmark values The all-router fold combines spend from routers without baselines into a labeled measured comparison, and it can pair a five-minute break-even percentage with a one-hour TTL when routers use different cache regimes Files Needing Attention: ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/autoRouterBenchmarks.ts and AutoRouterBenchmarksTab.tsx
|
| Filename | Overview |
|---|---|
| ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/autoRouterBenchmarks.ts | Adds the central aggregation fold, but mixed measured/unmeasured baselines and mixed TTLs produce misleading combined metrics |
| ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/AutoRouterBenchmarksTab.tsx | Adds the benchmark dashboard UI; it exposes the incorrect mixed-TTL aggregate as a single break-even/TTL pair |
| ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/useAutoRouterBenchmarks.ts | Adds a stable thirty-day React Query request window for the benchmark endpoint |
| ui/litellm-dashboard/src/components/networking.tsx | Adds an API-client wrapper for the auto-router benchmark endpoint |
| ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/CostOptimizationView.tsx | Registers the new Auto-Router Benchmarks tab in the cost-optimization view |
Reviews (1): Last reviewed commit: "feat(ui): rebuild the auto-router benchm..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| * against the same one; otherwise naming one of them would misattribute the rest. | ||
| */ | ||
| const combineBaselineLabel = (groups: readonly AutoRouterGroupBenchmark[]): string | null => { | ||
| const baselines = new Set(groups.map((g) => g.baseline_model).filter((m): m is string => Boolean(m))); |
There was a problem hiding this comment.
When the all-router selection includes both measured and unmeasured routers, combineBaselineLabel drops the null baselines while toView still includes those routers' spend, causing the displayed percentage to understate measured savings and attribute the comparison to a baseline that does not cover all included traffic
Knowledge Base Used: Admin dashboard (ui/litellm-dashboard)
| return_hit_rate_pct: rate(returnHits, returnTurns), | ||
| stale_miss_share_pct: rate(staleMisses, returnTurns - returnHits), | ||
| warming_savable_miss_pct: rate(savableMisses, turns - hits), | ||
| warming_break_even_pct: Math.max(...caches.map((c) => c.warming_break_even_pct)), |
There was a problem hiding this comment.
When the selection contains both five-minute and one-hour TTL routers, the independent maxima pair the five-minute 9% break-even with the one-hour TTL, causing the warming card to display a break-even regime that does not exist
Knowledge Base Used: Admin dashboard (ui/litellm-dashboard)
19fb0d1 to
ebc9d1f
Compare
ebc9d1f to
f1d4b84
Compare
f1d4b84 to
5b7f20a
Compare
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
56ee9b4 to
756504b
Compare
…s earned over Renders the per-session rollup as a tab on the cost-optimization page. Savings is the single hero figure on the view, paired in one card with the session count and turn count that number was earned over, because a dollar figure with no denominator beside it is not something an operator can size. That pairing makes two derived numbers worth showing, both computed from the response the tab already fetches: savings per session, and how many auto-routers are in scope. Neither needs a backend change, and both move with the router selector. Beneath it the three session-shape metrics, then the prompt-caching section: hit rate over a stacked share-of-turns bar with a three-bucket breakdown, and a warming estimate showing rescued writes and replay cost separately against the break-even for the TTL in use. Every rate is recomputed from summed counts whenever more than one router is in view. Averaging the routers' own rates would weight a router with three turns the same as one with three thousand, which lands the blended figure nowhere near either. Savings reads "Not measured" rather than $0.00 when no baseline is configured, since autorouter_savings_baseline_model is what the driver needs and a confident zero against real spend is worse than saying nothing. Built on the base shadcn primitives (Card, CardHeader, CardAction, Separator, Table) so padding, header layout and table density come from the design system rather than being restated per block. The tab test renders a real component tree and stubs only the network, so it is named for the integration tier; the fold it exercises is unit-tested separately against its own module.
756504b to
b457caa
Compare
|
Superseded by a replacement PR. GitHub would not let me retarget the base off the closed #35402 branch ("Cannot change the base branch because the pull request is part of a stack"), so this is reopened on top of the rebased rollup instead. Same code, rebased onto litellm_internal_staging. |
1 participant
TLDR
Problem this solves:
$0.00for any router that declares no counterfactual baseline, which reads as "the router saved nothing" rather than "nothing was measured"How it solves it:
Relevant issues
savings_baseline_modelis currently declared only on the semantic auto-router; complexity, quality and adaptive routers never set one, so their savings are structurally zero until that is fixed on feat(spend): add net auto-router savings to the cost-optimization dashboard #35402's branchLinear ticket
Resolves LIT-4712
Pre-Submission checklist
Screenshots / Proof of Fix
Rendered against a live proxy holding 30 days of real auto-routed traffic (99 sessions, 3,145 turns across three routers). Savings reads "Not measured" because all three are complexity routers, which declare no baseline; that is the honest state rather than a zero.
The three buckets sum to the headline turn count: 2,560 + 146 + 439 = 3,145.
To reproduce:
python litellm/proxy/proxy_cli.py --config <a config with at least one auto-router> --port 4715curl -X POST "http://localhost:4715/auto_router/benchmarks/backfill?start_date=<30d ago>&end_date=<today>" -H "Authorization: Bearer $LITELLM_MASTER_KEY"to populate the rollup from existing trafficcd ui/litellm-dashboard && NEXT_PUBLIC_BASE_URL=http://localhost:4715 npm run devhttp://localhost:3000/cost-optimization/and click the Auto-Router Benchmarks tabType
🆕 New Feature
Changes
autoRouterBenchmarks.tsholds the API types andtoView, the pure fold that combines however many routers are in view into the one shape the tab renders. Every rate is recomputed there from summed counts rather than averaged across routers, because averaging weights a router with three turns the same as one with three thousand; the tests pin that a 1000-turn router at 90% and a 10-turn router at 0% read as 89.1%, not 45%.The baseline label is deliberately three-valued. One shared baseline names it, several name none of them ("each router's own baseline"), and no baseline at all returns null, which is what drives the "Not measured" tile.
Marks follow the house chart specs: a 2px surface gap separates the stacked segments rather than a stroke, the fills are three steps of one neutral ramp, and every bucket carries its own swatch and label in the table below so identity never rests on colour alone. Values and labels wear text tokens, never the mark colour. The bucket table doubles as the table view for the bar.
useAutoRouterBenchmarkscomputes its window once per mount so the query key stays stable and the request is not refired on every parent render.QA runbook
__all__$0.00+N%badge in the destructive toneFinal Attestation