Skip to content

feat: add model leaderboard page - #43649

Merged
ishaan-berri merged 54 commits into
mainfrom
litellm_model_leaderboard
Sep 29, 2026
Merged

ishaan-berri merged 54 commits into
mainfrom
litellm_model_leaderboard

Conversation

@ishaan-berri

@ishaan-berri ishaan-berri commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

[TLDR](feat: add model leaderboard page)

Xnapper-2026-09-28-18 24 37

Problem this solves:

  • Admins can't see which models the gateway actually uses
  • No view of model usage trends over the past year

How it solves it:

  • New Model Leaderboard page with weekly stacked bars, 365 days back
  • Ranking table with provider logos, share, and change over the period
  • Treemap of top models by task, grouped into General, Agent, Code, Data

User Flow

Before: a proxy admin has no page that shows which models the gateway is using

  1. They open http://localhost:4000/ui/ and look through Observability in the sidebar
  2. There is no model leaderboard, only Usage and Logs
  3. They have to add up spend per model by hand

After: the same admin sees model usage ranked and charted in one place

  1. They open http://localhost:4000/ui/ and click Model Leaderboard (marked Beta) under Observability
  2. A stacked bar chart shows weekly tokens per model, with Requests, Spend, Tokens and Linear, Log toggles
  3. A leaderboard right below lists each model with its provider logo, share, and change between the first and second half of the period
  4. A treemap shows each task's share, labelled with its leading model
  5. Only proxy admins get data. Other roles get a 403 from GET /model-insights

Relevant issues

Linear ticket

Pre-Submission checklist

  • I have added meaningful tests
  • The handful of test files covering my change pass locally
  • My PR passes all required CI/CD checks
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Before (98c710c)

  1. Open http://localhost:4000/ui/ and expand Observability: no Model Leaderboard entry
  2. GET /model-insights returns 404

After (ebcedb6)

  1. Start the proxy with a local Postgres and --use_v2_migration_resolver, then open http://localhost:3000/model-insights
  2. Send a real request: curl http://localhost:4000/v1/chat/completions -H "Authorization: Bearer sk-1234" -d '{"model":"claude-sonnet-4-5","messages":[{"role":"user","content":"Reply with the word ok"}],"max_tokens":10,"metadata":{"tags":["task:debugging"]}}'
  3. Query the rolled up row: one debugging request for claude-sonnet-4-5 appears in LiteLLM_DailyModelUsage, with 12 prompt and 4 completion tokens

Type

🆕 New Feature

Caveats (if any)

Medium

  • Tasks come only from task:<name> request tags
  • Untagged requests show as uncategorized
  • Task names are a fixed list of 18
  • Treemap has one tile per task, no per-model tiles yet

Low

  • Dates are UTC, so "today" can differ from the admin's local day
  • Data is rolled up from new traffic only, there is no backfill

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

@CLAassistant

CLAassistant commented Sep 29, 2026 •

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
0 out of 2 committers have signed the CLA.

❌ github-actions[bot]
❌ ishaan-berri
You have signed the CLA already but the status is still pending? Let us recheck it.

@greptile-apps

greptile-apps Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 4/5

[Critical risk] Adds database schema and migration for model usage tracking.

The PR should not merge until the task treemap stops presenting stale shares as the selected metric.

Findings

  1. P1 Task shares use stale data ▶
  2. P2 Task responses grow with deployments ▶

Summary

The PR adds an admin Model Leaderboard backed by a daily usage rollup, with weekly model rankings and a separately fetched task treemap.

  • The latest changes move task summaries to a dedicated endpoint and reduce the task response to per-task results.
  • The task treemap can show results for a previously selected metric under the current metric’s label.

Reviews (7) · Last reviewed commit: "test: task breakdown is not refetched on..."

Comment thread litellm/proxy/management_endpoints/model_insights_endpoints.py Outdated
Comment thread litellm/proxy/management_endpoints/model_insights_endpoints.py
Comment thread litellm/proxy/db/model_usage_rollup.py Outdated
@codspeed

codspeed Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing litellm_model_leaderboard (e795e50) with main (39d14bd)

Open in CodSpeed

@codecov

codecov Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.29630% with 7 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...y/management_endpoints/model_insights_endpoints.py 95.65% 4 Missing ⚠️
litellm/proxy/db/model_usage_rollup.py 93.02% 3 Missing ⚠️

📢 Thoughts on this report? Let us know!

@ishaan-berri

Copy link
Copy Markdown
Contributor Author

@greptile review

@ishaan-berri

Copy link
Copy Markdown
Contributor Author

@greptile review

Comment thread litellm/proxy/management_endpoints/model_insights_endpoints.py Outdated
@ishaan-berri

Copy link
Copy Markdown
Contributor Author

@greptile review

Comment on lines +159 to +164
await table.group_by(
by=["task_type", "model_group", "model", "custom_llm_provider"],
sum=_SUM_FIELDS,
where=date_window,
order={"_sum": {"completion_tokens": "desc"}},
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Task responses grow with deployments

Removing the 500-row cap makes each /model-insights response include every task/deployment/provider combination in the selected period. On gateways with many deployments, the default 365-day view can return a large amount of data, increasing response and browser processing costs. Switching the chart metric fetches the same task breakdown again even though it does not depend on that metric.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

@ishaan-berri

Copy link
Copy Markdown
Contributor Author

@greptile review

@ishaan-berri
ishaan-berri merged commit 118ce3c into main Sep 29, 2026
112 of 117 checks passed
@ishaan-berri
ishaan-berri deleted the litellm_model_leaderboard branch September 29, 2026 02:40

This branch is waiting to be deployed

1 waiting deployment
e2e-changed — e795e504 Waiting Sep 29, 2026 by ishaan-berri via oauth #1725
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants