feat(proxy): auto-router session benchmarks endpoint and UI - #35190
feat(proxy): auto-router session benchmarks endpoint and UI#35190tin-berri wants to merge 1 commit into
Conversation
Adds GET /auto_router/benchmarks, an admin-only read over spend logs that reports, per configured auto-router, the four session metrics customers ask for (turns per session, average session length, tokens per session) plus a routed-vs-baseline dollar savings estimate. Session-scoped metrics can't come from the daily rollups because session_id lives only on LiteLLM_SpendLogs, so this reads spend logs directly over a window clamped to 30 days, matching the tool-spend endpoint. Spend is filtered by the auto-router alias, which keeps the turn count honest: the LLM classifier's own judge calls share the session but carry the judge model's group, not the alias. The savings figure compares the routed model mix against sending every request to a single baseline model, defaulting to the priciest model the router actually routed to in the window. A per-router benchmark_baseline_model param pins it to a fixed flagship instead. The estimate uses list prices on the tokens actually spent and does not model the caching a single-model baseline would have had; the UI says so. Surfaced as stat tiles on the Cost Optimization autorouter tab.
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| COALESCE(SUM(spend), 0.0) AS actual_spend | ||
| FROM "LiteLLM_SpendLogs" | ||
| WHERE model_group = $1 | ||
| AND session_id IS NOT NULL |
There was a problem hiding this comment.
When an auto-router request lacks an explicit trace or session identifier, spend logging assigns that request a fresh UUID and this query accepts it as a session. This reports each request as a separate one-turn, near-zero-length session, understating turns and tokens per actual session while overstating the session count.
Knowledge Base Used: Cost Tracking and Budget Enforcement
| return tuple( | ||
| ( | ||
| str(entry["model_name"]), | ||
| kind, | ||
| _configured_baseline(entry.get("litellm_params")), | ||
| ) | ||
| for entry in (router.model_list or []) | ||
| if (model := _entry_model(entry)) is not None and (kind := classify_strategy_router_model(model)) is not None | ||
| ) |
There was a problem hiding this comment.
When multiple auto-router deployments share a model_name, this enumeration emits each deployment independently even though every subsequent query filters by the same public model group. The response and dashboard therefore repeat identical traffic in multiple cards, potentially labeling it with different router kinds or baselines.
Knowledge Base Used: Cost Tracking and Budget Enforcement
| model=model, | ||
| input_cost_per_token=float(info.get("input_cost_per_token") or 0.0), | ||
| output_cost_per_token=float(info.get("output_cost_per_token") or 0.0), | ||
| ) | ||
| verbose_proxy_logger.warning( | ||
| "auto_router_benchmarks: baseline model %s is not priced; savings will read zero", model | ||
| ) |
There was a problem hiding this comment.
When benchmark_baseline_model names an unmapped custom, aliased, or newly released model, failed pricing lookup becomes zero rates and the comparison is still returned. This produces a negative dollar saving with savings_pct forced to zero, so the dashboard presents missing price data as a meaningful 0% vs <model> benchmark.
Knowledge Base Used: Cost Tracking and Budget Enforcement
| floor = (end - timedelta(days=BENCHMARKS_MAX_WINDOW_DAYS)).replace(hour=0, minute=0, second=0, microsecond=0) | ||
| clamped_start = max(start, floor) |
There was a problem hiding this comment.
When start_date is later than end_date, the clamped start remains after the end and the SQL receives an impossible range. The endpoint returns an empty benchmark response instead of rejecting the invalid input, making a malformed request indistinguishable from a window with no routed sessions.
Knowledge Base Used: Cost Tracking and Budget Enforcement
| if (error) { | ||
| return null; | ||
| } |
There was a problem hiding this comment.
When the endpoint returns an authorization, validation, or database error, the hook records the failure and this component renders nothing. Users receive the same blank panel as an empty successful response, with no diagnostic or retry path to distinguish unavailable benchmarks from no routed traffic.
Knowledge Base Used: Admin dashboard (ui/litellm-dashboard)
Greptile SummaryAdds an admin-only auto-router benchmark API and dashboard panel.
Confidence Score: 2/5This PR is not safe to merge until session qualification, duplicate router aliases, invalid date ranges, and unavailable baseline pricing are handled without publishing incorrect benchmark results. The endpoint can classify individual requests as sessions, duplicate identical traffic under repeated public aliases, silently accept reversed date ranges, and present missing baseline prices as meaningful savings; the dashboard also hides endpoint failures. Files Needing Attention: litellm/proxy/spend_tracking/auto_router_benchmarks.py and ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/AutoRouterBenchmarksPanel.tsx
|
| Filename | Overview |
|---|---|
| litellm/proxy/spend_tracking/auto_router_benchmarks.py | Implements benchmark aggregation, but synthetic session IDs, duplicate aliases, invalid ranges, and unpriceable configured baselines produce incorrect or misleading results. |
| litellm/proxy/proxy_server.py | Adds the authenticated, admin-view-gated endpoint and validates router and database availability. |
| ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/AutoRouterBenchmarksPanel.tsx | Renders the new metrics but hides endpoint failures and presents unpriceable baselines as zero-percent comparisons. |
| ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/useAutoRouterBenchmarks.ts | Fetches a rolling 30-day benchmark window with guarded asynchronous state updates. |
| tests/test_litellm/proxy/spend_tracking/test_auto_router_benchmarks.py | Covers core calculations but codifies zero pricing for unknown configured baselines and omits generated session IDs, duplicate aliases, and reversed ranges. |
Reviews (1): Last reviewed commit: "feat(proxy): add auto-router session ben..." | Re-trigger Greptile
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
| AND session_id IS NOT NULL | ||
| AND "startTime" >= ($2::timestamptz AT TIME ZONE 'UTC') | ||
| AND "startTime" < (($3::timestamptz + INTERVAL '1 day') AT TIME ZONE 'UTC') | ||
| GROUP BY session_id |
There was a problem hiding this comment.
Low: Unbounded session aggregation
An authenticated user can generate high-cardinality session data by making auto-router requests without reusing a session ID; spend logging assigns each such request a unique UUID. This query returns one row per ID and _fetch_sessions materializes the entire result, so loading the dashboard over a busy 30-day window can exhaust database or worker resources. Aggregate the per-session rows again in SQL to return only the final count, sums, and average session duration needed by the response.
PR overviewThis PR adds an auto-router session benchmarks endpoint and corresponding proxy UI for viewing session-level performance and spend metrics. One resource-exhaustion concern remains open in the session aggregation path. An authenticated user can create many unique sessions, causing the benchmarks query and dashboard request to materialize an unbounded result set over the selected window, potentially exhausting database or worker resources. No issues have yet been addressed. Open issues (1)
Fixed/addressed: 0 · PR risk: 5/10 |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
1 participant
TLDR
Problem this solves:
How it solves it:
/auto_router/benchmarksendpoint over spend logsRelevant issues
Customers evaluating the auto-router keep asking for evidence that it saves money before they roll it out. The Logs and Session views already show which tier a request landed on (#34434), but there was nothing that rolls that up into "your routed sessions cost X, the same traffic on your flagship model would have cost Y". This adds that view.
Linear ticket
Resolves LIT-4712
Pre-Submission checklist
@greptileaito re-request a review after pushing changes)Screenshots / Proof of Fix
Captured against a local proxy on port 4712 running this branch, reading a Postgres with real auto-router traffic (611 sessions of Claude Code usage across haiku, sonnet, and opus tiers). Commit
703fde9f1e.Admin call over a 30-day window returns one entry per configured auto-router, each with the four session metrics and the savings estimate:
The window is clamped to 30 days; asking for a year still returns a 30-day window and says so:
Non-admin / unauthenticated is rejected:
UI: go to http://localhost:4000/ui/?page=cost-optimization, open the Autorouter tab. The benchmarks panel renders above the creation form, one card per auto-router, each with four stat tiles (turns per session, avg session length, tokens per session, estimated savings) and a caption naming the baseline it compared against.
Type
🆕 New Feature
Changes
The one primitive that was missing is a session-level rollup of auto-router traffic. The four metrics the customers ask for are all session-scoped, and
session_idlives only onLiteLLM_SpendLogs, never on the daily rollups, because a session does not close on a day boundary and so can't be pre-aggregated the way user or team spend is. This reads spend logs directly over a window clamped to 30 days, the same discipline the tool-spend endpoint uses (#34582). A durable per-session rollup table is the natural follow-up if the 30-day cap ever becomes a product limit.GET /auto_router/benchmarksenumerates every configured auto-router by its publicmodel_group, then runs one pair of aggregate queries per group: sessions grouped bysession_id, and spend grouped by resolved model. Filtering spend logs by the auto-router alias is what keeps the turn count honest: the LLM classifier's own judge calls land in the same session but carry the judge model's group, not the alias, so grouping by the alias yields one row per routed turn with no classifier noise.The savings figure compares the routed model mix against sending every request to a single baseline model. The baseline defaults to the priciest model the router actually routed to in the window, which is uniform across all four router types and is always a model the router really used. A per-router
benchmark_baseline_modelparam pins it to a fixed flagship instead. The estimate uses list prices on the tokens actually spent; it does not model the caching a single-model baseline would have had, and the UI says so.Admin-only, matching
/adaptive_router/state. The endpoint carries no new database migration; the new param rides existing router config.QA runbook
Point a proxy at a database with auto-router traffic that carries
session_id(Claude Code sets this automatically). CallGET /auto_router/benchmarks?start_date=<iso>&end_date=<iso>with the master key and confirm one entry per configured auto-router with non-zero metrics. Confirm a non-admin key gets 403 and no key gets 401. Confirm a window wider than 30 days comes back clamped withstart_datereflecting the served window. In the UI, open the Autorouter tab under Cost Optimization and confirm the benchmarks panel renders above the creation form. To check the fixed-baseline path, setbenchmark_baseline_model: claude-opus-5on an auto-router deployment and confirmbaseline_modelin the response reflects it.Final Attestation