feat(ui): lead the benchmarks tab with savings and the sessions it was earned over by tin-berri · Pull Request #35741 · BerriAI/litellm

tin-berri · 2026-08-04T04:31:48Z

TLDR

Problem this solves:

The benchmarks tab was a flat row of four equal stat tiles followed by a wide per-router table, so the two things an operator opens it for (is the router saving money, is the cache working) had no more weight than anything else
The cache numbers were spread across nine table columns of percentages with no sense of how many turns sat behind each one, and the warming estimate was a single net figure with its two sides buried in a footnote
Savings rendered $0.00 for any router that declares no counterfactual baseline, which reads as "the router saved nothing" rather than "nothing was measured"

How it solves it:

Estimated savings leads as the leftmost tile, with its delta badge and the routed-versus-baseline comparison that produced it; the three session-shape tiles follow as context
Two cards replace the table: hit rate over a stacked share-of-turns bar with a three-bucket breakdown, and a warming estimate that shows rescued writes and replay cost separately against a break-even marker
A router with no baseline reads "Not measured" instead of a confident zero

Relevant issues

Consumes the rollup endpoint added in feat(spend): fold auto-router benchmarks into a per-session rollup #35740, which is where the three cache buckets became mutually exclusive and exhaustive; the bucket table here sums to the headline turn count because of that change
Stacked on feat(spend): fold auto-router benchmarks into a per-session rollup #35740, which sits directly on litellm_internal_staging; replaces feat(ui): rebuild the auto-router benchmarks tab around savings and cache behaviour #35488, which could not be retargeted because GitHub blocks base changes on a stacked PR
The "Not measured" state shows when no litellm.autorouter_savings_baseline_model is configured, which is what feat(ui): show auto-router savings on the cost-optimization dashboard #35522 made the driver depend on; a confident $0.00 against real spend would be worse than saying nothing

Linear ticket

Resolves LIT-4712

Pre-Submission checklist

I have added meaningful tests
My PR passes all CI/CD checks (e.g., lint, format, unit tests)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Rendered against a live proxy holding 30 days of real auto-routed traffic (99 sessions, 3,145 turns across three routers). Savings reads "Not measured" because all three are complexity routers, which declare no baseline; that is the honest state rather than a zero.

The three buckets sum to the headline turn count: 2,560 + 146 + 439 = 3,145.

To reproduce:

python litellm/proxy/proxy_cli.py --config <a config with at least one auto-router> --port 4715
curl -X POST "http://localhost:4715/auto_router/benchmarks/backfill?start_date=<30d ago>&end_date=<today>" -H "Authorization: Bearer $LITELLM_MASTER_KEY" to populate the rollup from existing traffic
cd ui/litellm-dashboard && NEXT_PUBLIC_BASE_URL=http://localhost:4715 npm run dev
Open http://localhost:3000/cost-optimization/ and click the Auto-Router Benchmarks tab

Type

🆕 New Feature

Changes

autoRouterBenchmarks.ts holds the API types and toView, the pure fold that combines however many routers are in view into the one shape the tab renders. Every rate is recomputed there from summed counts rather than averaged across routers, because averaging weights a router with three turns the same as one with three thousand; the tests pin that a 1000-turn router at 90% and a 10-turn router at 0% read as 89.1%, not 45%.

The baseline label is deliberately three-valued. One shared baseline names it, several name none of them ("each router's own baseline"), and no baseline at all returns null, which is what drives the "Not measured" tile.

Marks follow the house chart specs: a 2px surface gap separates the stacked segments rather than a stroke, the fills are three steps of one neutral ramp, and every bucket carries its own swatch and label in the table below so identity never rests on colour alone. Values and labels wear text tokens, never the mark colour. The bucket table doubles as the table view for the bar.

useAutoRouterBenchmarks computes its window once per mount so the query key stays stable and the request is not refired on every parent render.

QA runbook

With at least one auto-router configured and backfilled, confirm the tile order reads estimated savings, avg turns per session, avg session length, avg tokens per session
Confirm the three bucket turn counts sum to the "turns total" figure beside the stacked bar
Switch the selector from "All auto-routers" to a single router; the tiles and both cards should scope to it, and the label should never show __all__
On a deployment whose routers declare no baseline, confirm the savings tile reads "Not measured" rather than $0.00
On a router that costs more than its baseline, confirm the tile shows a negative dollar figure and a +N% badge in the destructive tone
Toggle dark mode and confirm both cards, the stacked bar and the break-even marker stay legible

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

The benchmarks dashboard answered every question by scanning LiteLLM_SpendLogs at read time: four aggregate queries per auto-router, two of them window functions over the response JSONB, re-deriving on every page load which model the previous turn used, how long a tier had been idle, and how big the prefix was last time. Those are sequential facts and the request that produces them already knows all of them, so they are now computed once, when the turn happens. A new LiteLLM_AutoRouterSession row per (session, auto-router) carries both the counters and the state that classifies the next turn. fold_turn is pure, so every rate and dollar formula is unit-testable without a database, and the in-memory queue plus background flusher follow AdaptiveRouterUpdateQueue: atomic increment upserts, so two pods writing one session compose instead of overwriting. A pod that has never seen a session loads its row once and classifies from memory after, which is what keeps a session correct across a restart or a pod move. The counters are declared once, on TurnDelta. COUNTER_FIELDS derives from that declaration and the merge, the flush payload and the read query all build off it, so a metric added there reaches the database and the dashboard without a second edit. A test asserts the read query aggregates every declared counter; it caught two that were being written on every request and read by nothing. The read path is a single aggregate over pre-folded rows covering every auto-router at once, and touches no per-request table at all. Rollup rows expire on the existing spend-log retention cutoff, keyed on last activity so a live conversation is not pruned out from under itself. Two behaviour fixes came with the move. The turn buckets are now exhaustive: a session's opening turn used to land in the headline turn count and in none of the three buckets, so the bucket totals silently disagreed with the headline. And a turn with no ephemeral cache-creation evidence now reads as the five minute tier rather than the one hour tier, which had been the default purely because zero is not less than zero. Savings come from compute_savings_spend, the same primitive the usage tab uses, so the two surfaces cannot report different numbers for the same traffic. The baseline recorded on each row is the one that priced its turns, so the tab names what the numbers were computed against rather than whatever the config says by the time someone opens it.

…s earned over Renders the per-session rollup as a tab on the cost-optimization page. Savings is the single hero figure on the view, paired in one card with the session count and turn count that number was earned over, because a dollar figure with no denominator beside it is not something an operator can size. That pairing makes two derived numbers worth showing, both computed from the response the tab already fetches: savings per session, and how many auto-routers are in scope. Neither needs a backend change, and both move with the router selector. Beneath it the three session-shape metrics, then the prompt-caching section: hit rate over a stacked share-of-turns bar with a three-bucket breakdown, and a warming estimate showing rescued writes and replay cost separately against the break-even for the TTL in use. Every rate is recomputed from summed counts whenever more than one router is in view. Averaging the routers' own rates would weight a router with three turns the same as one with three thousand, which lands the blended figure nowhere near either. Savings reads "Not measured" rather than $0.00 when no baseline is configured, since autorouter_savings_baseline_model is what the driver needs and a confident zero against real spend is worse than saying nothing. Built on the base shadcn primitives (Card, CardHeader, CardAction, Separator, Table) so padding, header layout and table density come from the design system rather than being restated per block. The tab test renders a real component tree and stubs only the network, so it is named for the integration tier; the fold it exercises is unit-tested separately against its own module.

greptile-apps · 2026-08-04T04:34:08Z

Greptile Summary

The PR adds an Auto-Router Benchmarks tab that folds per-router session and cache rollups into savings, session-shape, cache-hit, and warming views.

Adds the authenticated benchmark API wrapper and stable 30-day query hook
Adds aggregation and formatting helpers for multi-router benchmark data
Adds savings, cache-bucket, and warming cards with router selection
Adds unit and integration coverage for aggregation and rendering

Confidence Score: 4/5

The mixed-baseline and mixed-TTL aggregation defects should be fixed before merging because they present incorrect benchmark economics to operators

The aggregate fold can label partially unmeasured spend as measured and can synthesize a 9% one-hour warming threshold that is inconsistent with the backend’s TTL-specific economics

Files Needing Attention: ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/autoRouterBenchmarks.ts and AutoRouterBenchmarksTab.tsx

Important Files Changed

Filename	Overview
ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/autoRouterBenchmarks.ts	Adds the aggregate view fold, but mixed baseline availability and mixed TTL regimes produce misleading combined values
ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/AutoRouterBenchmarksTab.tsx	Adds the complete benchmarks UI; its warming card exposes the invalid aggregate TTL and threshold pair
ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/useAutoRouterBenchmarks.ts	Adds a stable 30-day TanStack Query hook with no accepted blocking defect
ui/litellm-dashboard/src/components/networking.tsx	Adds the benchmark endpoint wrapper through the existing authenticated API client
ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/autoRouterBenchmarks.test.ts	Covers weighted aggregation and homogeneous baseline states but omits the two mixed-state cases implicated by the findings
ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/AutoRouterBenchmarksTab.integration.test.tsx	Provides substantial rendering coverage but does not exercise mixed baseline availability or mixed TTL warming output
ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/CostOptimizationView.tsx	Registers the new benchmarks tab in the cost-optimization view

_{Reviews (1): Last reviewed commit: "feat(ui): lead the benchmarks tab with s..." | Re-trigger Greptile}

greptile-apps · 2026-08-04T04:34:12Z

+  const baselines = new Set(groups.map((g) => g.baseline_model).filter((m): m is string => Boolean(m)));
+  if (baselines.size === 0) return null;
+  if (baselines.size === 1) return [...baselines][0];
+  return "each router's own baseline";
+};


Mixed baselines appear fully measured

When the all-routers selection contains both measured and unmeasured routers, combineBaselineLabel discards the null baselines while toView still includes those routers' spend. This names the remaining baseline as though it covered the entire selection and dilutes the displayed savings percentage with spend that has no counterfactual.

Knowledge Base Used: Admin dashboard (ui/litellm-dashboard)

greptile-apps · 2026-08-04T04:34:12Z

+    return_hit_rate_pct: rate(returnHits, returnTurns),
+    stale_miss_share_pct: rate(staleMisses, returnTurns - returnHits),
+    warming_savable_miss_pct: rate(savableMisses, turns - hits),
+    warming_break_even_pct: Math.max(...caches.map((c) => c.warming_break_even_pct)),


Mixed TTLs create invalid threshold

When five-minute and one-hour cache regimes are combined, the fold independently selects the maximum TTL and maximum break-even percentage, producing the nonexistent pair 1h and 9%. The warming card then displays that pair and compares the aggregate savable-miss rate against it, giving operators an incorrect break-even assessment.

Knowledge Base Used: Admin dashboard (ui/litellm-dashboard)

tin-berri added 2 commits August 3, 2026 21:30

tin-berri requested a review from ryan-crabbe-berri as a code owner August 4, 2026 04:31

greptile-apps Bot reviewed Aug 4, 2026

View reviewed changes

tin-berri force-pushed the litellm_lit4712_autorouter_session_rollup branch 2 times, most recently from 4022c43 to 71f2e85 Compare August 4, 2026 20:51

tin-berri closed this Aug 5, 2026

tin-berri mentioned this pull request Aug 5, 2026

feat(ui): add the auto-router usage tab to cost optimization #35995

Merged

5 tasks

Navigation Menu

Uh oh!

Uh oh!

feat(ui): lead the benchmarks tab with savings and the sessions it was earned over - #35741

feat(ui): lead the benchmarks tab with savings and the sessions it was earned over#35741
tin-berri wants to merge 2 commits into
litellm_lit4712_autorouter_session_rollupfrom
litellm_lit4712_autorouter_benchmarks_ui

tin-berri commented Aug 4, 2026

Uh oh!

greptile-apps Bot commented Aug 4, 2026

Important Files Changed

Uh oh!

greptile-apps Bot Aug 4, 2026

Uh oh!

greptile-apps Bot Aug 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

tin-berri commented Aug 4, 2026

TLDR

Relevant issues

Linear ticket

Pre-Submission checklist

Screenshots / Proof of Fix

Type

Changes

QA runbook

Final Attestation

Uh oh!

greptile-apps Bot commented Aug 4, 2026

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Uh oh!

greptile-apps Bot Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant