feat(usage): server-side per-model top keys for tag/org/customer/agent and aggregated User Usage

[devin-ai-integration]

TLDR

Problem this solves:

• Tag, org, customer and agent usage ranked per-model keys from the globally capped key set
• User Usage drained /user/daily/activity?page=N and had no model top keys or search
• A key that only leads on one model vanished when it was outside the global top N

How it solves it:

• Adds /{tag,organization,customer,agent}/daily/activity/aggregated/model_top_keys mirroring the team route
• Each route keeps its entity's existing auth scope and returns at most USAGE_MODEL_TOP_API_KEYS_LIMIT keys
• Dashboard dispatches per-model top keys through one entity map instead of a team-only branch
• User Usage joins the aggregated, key search, export and model top keys maps with one request
• Nonempty /user/daily/activity/export no longer 500s (the stacked branches disagreed on the row field name)
• /user/daily/activity/aggregated folds the per-user rollup and user emails into breakdown.entities, so the admin Spend Per User chart is populated

Stacked on #43137 (user/team model top keys), #43138 (user export) and #43141 (entity aggregated/search/export routes). The base branch litellm_lit_8647_stack_base is main plus those three merged, so the diff shown here is only this PR's work. Retarget to main once the three parents land

User Flow

Before: an admin with USAGE_TOP_API_KEYS_LIMIT=2 looks at tag-b's gpt-4o spend and sees keys that never used tag-b

1. Open http://localhost:3000/usage, choose Tag Usage, then the Model Activity tab
2. Expand gpt-4o: Top Virtual Keys shows key-1 ($100) and key-2 ($90), the two global leaders, and never key-3 or key-4
3. GET http://localhost:4000/tag/daily/activity/aggregated/model_top_keys?model=gpt-4o&tags=tag-b returns 404 Not Found
4. Switch to User Usage: the browser fetches GET /user/daily/activity?page_size=1000&page=1 and keeps paging

After: the same admin sees every key ranked by its spend on that model, scoped to the selected tag

1. Open http://localhost:3000/usage, choose Tag Usage, then the Model Activity tab
2. Expand gpt-4o: the browser calls GET /tag/daily/activity/aggregated/model_top_keys?model=gpt-4o and lists key-1, key-2, key-3, key-4 by spend
3. GET http://localhost:4000/tag/daily/activity/aggregated/model_top_keys?model=gpt-4o&tags=tag-b returns 200 with key-3 ($5) and key-4 ($4)
4. Switch to User Usage: one GET /user/daily/activity/aggregated request, Key Activity still shows "Only the 2 highest-spend keys of 4 are loaded" and typing key-4 calls GET /user/daily/activity/aggregated/search

Before (tag view, gpt-4o expanded, capped to the two global leaders):

After (same page, same data, all four keys ranked server side):

Linear ticket

Resolves LIT-8647

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

The stack was re-squashed onto the current parent branch, so the commit ids named below are the original proof commits; the current head is a content-identical squash plus regenerated schema.d.ts, unused-import cleanup in team_endpoints.py, and an unbounded_in_baseline.txt refresh for renamed functions. Greptile 5/5 judged the current head

Setup: proxy from litellm/proxy/dev_config.yaml against local Postgres with USAGE_TOP_API_KEYS_LIMIT=2, dashboard via npm run dev. Seed: four virtual keys (key-1..key-4) owned by user lit8647-user. tag-a: key-1 gpt-4o $100, key-2 gpt-4o $90. tag-b: key-3 gpt-4o $5, key-4 gpt-4o $4, key-3 gpt-4o-mini $50. So the tag-b gpt-4o keys sit outside the global top two

Before (60ec99a)

Tag model top keys route

1. curl -s -o /dev/null -w "%{http_code}\n" "localhost:4000/tag/daily/activity/aggregated/model_top_keys?model=gpt-4o&tags=tag-b&start_date=2026-09-20&end_date=2026-09-26" -H "Authorization: Bearer $LITELLM_MASTER_KEY"
2. 404 (same for /organization, /customer, /agent)

Tag aggregate key cap

1. curl -s "localhost:4000/tag/daily/activity/aggregated?start_date=2026-09-20&end_date=2026-09-26" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c '{keys: [.results[].breakdown.api_keys | keys[]] | unique, api_key_limit: .metadata.api_key_limit, total_api_keys: .metadata.total_api_keys}'
2. Only the two globally highest keys (key-1, key-2) are present, "api_key_limit":2,"total_api_keys":4

Tag UI, gpt-4o expanded

1. http://localhost:3000/usage, Tag Usage, Model Activity, expand gpt-4o
2. Top Virtual Keys by Spend lists key-1 $100.00 and key-2 $90.00 only (screenshot above)

User Usage requests

1. http://localhost:3000/usage, User Usage; network requests to the proxy:
2. /user/daily/activity?start_date=2026-09-18&end_date=2026-09-25&page_size=1000&page=1&timezone=0

After (a29426e)

Tag model top keys route

1. for e in tag organization customer agent; do curl -s "localhost:4000/$e/daily/activity/aggregated/model_top_keys?model=gpt-4o-mini&start_date=2026-09-20&end_date=2026-09-26" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c "{route: \"$e\", limit, keys: [.api_keys[]? | {alias: .key_alias, spend}]}"; done

2. {"route":"tag","limit":5,"keys":[{"alias":"key-3","spend":50}]}
   {"route":"organization","limit":5,"keys":[]}
   {"route":"customer","limit":5,"keys":[]}
   {"route":"agent","limit":5,"keys":[]}
   

3. curl -s "localhost:4000/tag/daily/activity/aggregated/model_top_keys?model=gpt-4o&start_date=2026-09-20&end_date=2026-09-26" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c '{limit, keys: [.api_keys[] | {alias: .key_alias, spend}]}'
4. {"limit":5,"keys":[{"alias":"key-1","spend":100},{"alias":"key-2","spend":90},{"alias":"key-3","spend":5},{"alias":"key-4","spend":4}]}
5. curl -s "localhost:4000/tag/daily/activity/aggregated/model_top_keys?model=gpt-4o&tags=tag-b&start_date=2026-09-20&end_date=2026-09-26" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c '{limit, keys: [.api_keys[] | {alias: .key_alias, spend}]}'
6. {"limit":5,"keys":[{"alias":"key-3","spend":5},{"alias":"key-4","spend":4}]}
7. curl -s -w " HTTP %{http_code}\n" "localhost:4000/tag/daily/activity/aggregated/model_top_keys?model=gpt-4o&start_date=2026-09-27&end_date=2026-09-20" -H "Authorization: Bearer $LITELLM_MASTER_KEY"
8. {"detail":{"error":"end_date must be on or after start_date"}} HTTP 400

Tag aggregate key cap

1. curl -s "localhost:4000/user/daily/activity/aggregated?user_id=lit8647-user&start_date=2026-09-20&end_date=2026-09-26" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -c '.metadata | {page, total_pages, has_more, api_key_limit, total_api_keys}'
2. {"page":1,"total_pages":1,"has_more":false,"api_key_limit":2,"total_api_keys":4} (the aggregate stays capped by design; per-model ranking now comes from the route above)

Tag UI, gpt-4o expanded

1. http://localhost:3000/usage, Tag Usage, Model Activity, expand gpt-4o
2. Browser calls /tag/daily/activity/aggregated/model_top_keys?start_date=2026-09-18&end_date=2026-09-25&timezone=0&model=gpt-4o&group_by=model_group
3. Top Virtual Keys by Spend lists key-1 $100.00, key-2 $90.00, key-3 $5.00, key-4 $4.00 (screenshot above); expanding gpt-4o-mini lists key-3 $50.00

User Usage requests

1. http://localhost:3000/usage, User Usage; network requests to the proxy:
2. /user/daily/activity/aggregated?start_date=2026-09-18&end_date=2026-09-25&timezone=0 and, on expanding gpt-4o, /user/daily/activity/aggregated/model_top_keys?...&model=gpt-4o&group_by=model_group. No /user/daily/activity?page= request
3. Key Activity tab shows "Showing 2 of 2 keys Only the 2 highest-spend keys of 4 are loaded"; typing key-4 issues /user/daily/activity/aggregated/search?...&search=key-4 and shows key-4 $4.00

User export (92c7bba)

1. curl -s -w "\nHTTP %{http_code}\n" -H "Authorization: Bearer $LITELLM_MASTER_KEY" "localhost:4000/user/daily/activity/export?start_date=2026-09-18&end_date=2026-09-25&export_type=daily_with_keys&user_id=lit8647-user"

2. Date,User,User ID,Key Alias,Key ID,Spend ($),Requests,Successful Requests,Failed Requests,Total Tokens,Prompt Tokens,Completion Tokens,Cache Read Input Tokens,Cache Creation Input Tokens
   2026-09-24,lit8647-user,lit8647-user,key-2,5f26...,90.0000,9,9,0,135,90,45,0,0
   2026-09-24,lit8647-user,lit8647-user,key-3,6015...,55.0000,10,10,0,825,550,275,0,0
   2026-09-24,lit8647-user,lit8647-user,key-4,7594...,4.0000,4,4,0,60,40,20,0,0
   2026-09-24,lit8647-user,lit8647-user,key-1,e872...,100.0000,10,10,0,150,100,50,0,0
   
   HTTP 200
   

Reviewer fix: Spend Per User empty for admins (c20e66b, 5a1e4d0, faa1a23)

Yuneng flagged that /user/daily/activity/aggregated never asked for the per-user rollup, so the admin all-users view rendered "No user spend data". c20e66b folds the per-user rollup plus user email/alias into breakdown.entities on that route. 5a1e4d0 makes that rollup key-free for the user route (one row per user per day, GROUP BY date, user_id, no per-key grouping set), so it scales with the user count only, and types the resolver as Mapping[str, _UserEmailMetadata]. Because that rollup no longer carries per-key rows, faa1a23 routes User Usage exports through /user/daily/activity/export even when no key truncation applies, so the daily-with-keys and daily-with-models CSVs keep every key (verified live: the export CSV lists key-1 100, key-2 90, key-3 55, key-4 4). 2add5a5, the current head, covers the key-free rollup end to end in tests. Same page, same data, same URL (http://localhost:3000/usage, User Usage, all users, 2026-09-18 to 2026-09-25), proxy with USAGE_TOP_API_KEYS_LIMIT=2

Before (92c7bba, Spend Per User chart and table empty despite $249 total spend):

After (c20e66b, lit8647-user at $249; the Spend Per User table only reads the per-user metrics, which 5a1e4d0 leaves unchanged):

Before, same request against the old head:

1. curl -sS -H "Authorization: Bearer $LITELLM_MASTER_KEY" "localhost:4001/user/daily/activity/aggregated?start_date=2026-09-18&end_date=2026-09-25&timezone=0" | jq '.results[0].breakdown.entities'
2. {} (HTTP 200, metadata.total_spend 249, api_key_limit 2)

After:

1. curl -s -H "Authorization: Bearer $LITELLM_MASTER_KEY" "localhost:4000/user/daily/activity/aggregated?start_date=2026-09-18&end_date=2026-09-25" | jq '.results[0].breakdown.entities'

2. {
     "lit8647-user": {
       "metrics": {"spend": 249, "api_requests": 33, "successful_requests": 33, "failed_requests": 0, "total_tokens": 1170, ...},
       "metadata": {"user_email": null, "user_alias": "lit8647-user"},
       "api_key_breakdown": {}
     }
   }
   

   (at 5a1e4d0; at c20e66b the same bucket also carried the four keys under api_key_breakdown, which the dashboard never read)

The second point (the aggregated route caps keys at the global top USAGE_TOP_API_KEYS_LIMIT) is intentional and matches Team and Tag usage: the truncation notice, key search, per-model top keys and export reach the keys past the cap

Type

🆕 New Feature

Caveats (if any)

Medium

• Organization, customer and agent routes were verified live only with empty seed data for those entities; scoping for them is covered by unit tests
• Customer route keeps the existing admin-only 401 from the sibling customer routes rather than 403

Low

• ENTITY_FETCH_FNS and ENTITY_AGGREGATED_FETCH_FNS keep their pre-existing any callable types when moved into entityFetchFns.ts

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/cfa33299feca496383a95283e4e365a4
Open in Devin Desktop: https://app.devin.ai/desktop/session/cfa33299feca496383a95283e4e365a4?variant=devin
Requested by: @yassin-berriai

Link to Devin session: https://app.devin.ai/sessions/c0361ef9973f4f409b51af4495e3099f
Open in Devin Desktop: https://app.devin.ai/desktop/session/c0361ef9973f4f409b51af4495e3099f?variant=devin

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_lit_8647_entity_model_top_keys_user_usage (6f8eaee) with litellm_entity_usage_aggregated_routes (b6ac6d5)}

[codecov]

Codecov Report

❌ Patch coverage is 95.31250% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/agent_endpoints/endpoints.py	92.85%	1 Missing ⚠️
...m/proxy/management_endpoints/customer_endpoints.py	92.30%	1 Missing ⚠️
...y/management_endpoints/tag_management_endpoints.py	92.30%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[greptile-apps]

Confidence Score: 5/5

[Medium risk] Adds new usage analytics endpoints for per-model top API keys.

The PR appears safe to merge on the stated CI merge ref, with no outstanding review finding.

Findings

1.  Redundant test docstring ▶

Summary

The PR adds scoped, server-ranked per-model top-key routes for tag, organization, customer, and agent usage, and moves User Usage to aggregated data with server-side search and export. The only change since the previous review removes an obsolete coverage-baseline entry.

_{Reviews (13) · Last reviewed commit: "feat(usage): server-side per-model top k..."}

[devin-ai-integration]

@greptileai

[yuneng-berri]

1. Spend Per User is always empty for admins viewing all users, since /user/daily/activity/aggregated never sets include_entity_breakdown to true
2. nit: user usage used to page through every key. On /aggregated it now caps at the top 100 keys across the whole proxy

[devin-ai-integration]

@greptileai

[devin-ai-integration]

Point 1 fixed in c20e66b: /user/daily/activity/aggregated now sets include_entity_breakdown and resolves user emails, with a red-then-green regression test

[devin-ai-integration]

Point 2 is intentional: same cap Team and Tag usage already have. The truncation note, key search, model top keys and export reach the tail

[devin-ai-integration]

@greptileai

[devin-ai-integration]

@greptileai

[devin-ai-integration]

@greptileai

[devin-ai-integration]

@greptileai

[devin-ai-integration]

@greptileai

[devin-ai-integration]

@greptileai

[yuneng-berri]

the per key widgets on the user usage tab are still limited to the top N keys, can you verify?

[devin-ai-integration]

@yuneng-berri verified, yes: the aggregated route caps keys at USAGE_TOP_API_KEYS_LIMIT like Team/Tag. Search, model top keys and export cover the tail. Want the cap lifted for users?

[devin-ai-integration]

@greptileai please re-review at 0d68934. The stack was re-squashed onto the current parent branch so the earlier 5/5 (f1ae967) is no longer an ancestor of head; the PR-scoped diff is content-identical apart from helper renames that moved into the parent

[devin-ai-integration]

@greptileai please re-review at 91c6199. Only change since 0d68934: regenerated ui/litellm-dashboard/src/lib/http/schema.d.ts (adds the /agent/daily/activity/aggregated/model_top_keys route types), folded into the single commit.

[devin-ai-integration]

@greptileai please re-review at 20a821f. Changes since 91c6199: removed the redundant test docstrings you flagged, dropped two unused imports in team_endpoints.py (repo-wide lint), and refreshed tests/code_coverage_tests/unbounded_in_baseline.txt for renamed functions (same queries, new fingerprints).

[devin-ai-integration]

@greptileai please re-review at 6f8eaee. Only change since 20a821f: dropped one stale line from tests/code_coverage_tests/unbounded_in_baseline.txt (the query was already migrated to find_many_in on main, which the CI merge ref includes).

[devin-ai-integration]

Superseded by the redesigned bounded-usage stack: repository #43398, routes #43408, UI #43409.