fix(usage): rank cache leakage keys server-side by uncached prompt tokens

[devin-ai-integration]

TLDR

Problem this solves:

• Cost Optimization cache leakage by key only sees the top-N spend keys
• A low-spend key leaking the most uncached tokens never shows up
• The savings rate was computed over the truncated key set

How it solves it:

• New GET /user/daily/activity/cache_leakage ranks every visible key by uncached prompt tokens in SQL
• Window totals over the full key set feed the net savings per cached token
• Same user scoping as /user/daily/activity/aggregated, PTU sentinel excluded
• Dashboard key dimension reads the new route; model dimension unchanged

Intentional product change: the "only the N highest-spend keys are loaded" note under the key ranking is gone because the ranking is no longer spend-capped. When the server truncates, the card says "Showing the top N of M keys, ranked by uncached prompt tokens" instead

User Flow

Before: an admin wants to know which virtual key wastes the most uncached prompt tokens, and the worst offender is missing

1. They open http://localhost:3000/cost-optimization/, Prompt Caching tab, and switch the Cache leakage card to By key
2. The card lists only the two highest-spend keys, mid-cached and whale-cached, with 30,000 and 10,000 uncached tokens
3. leaky-low-spend, which burned 900,000 uncached tokens on $5 of spend, is nowhere in the list
4. A note under the table says only the 2 highest-spend keys are loaded and tells them to raise USAGE_TOP_API_KEYS_LIMIT

After: the same card puts the real leaker first

1. They open the same page and switch the card to By key
2. leaky-low-spend is first with 900,000 uncached tokens and 0.0% cache hit rate, then mid-cached with 30,000
3. The card says "Showing the top 2 of 6 keys, ranked by uncached prompt tokens" and the potential savings column is priced against every key in the range
4. A non-admin user hitting GET /user/daily/activity/cache_leakage with someone else's user_id still gets 403

Linear ticket

Resolves LIT-8620

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Setup shared by both legs: two proxies on the same real PostgreSQL, each launched with PYTHONPATH=<its checkout>:<its checkout>/enterprise and USAGE_TOP_API_KEYS_LIMIT=2. The Before proxy (port 4001) runs the merge base, the After proxy (port 4000) runs the PR branch. Three control requests to anthropic-haiku-4-5 through POST /v1/chat/completions returned 200 with real Anthropic usage. The same fixture SQL (sha256 5f2dccee9c7b49442b8ee2b0c55be23af6611a6e0bdfafcd8c1e63e34649375d) seeds three keys for D=2026-09-25 under user lit8620-user: whale-cached ($90 spend, 1,000,000 prompt, 990,000 cached), mid-cached ($40, 300,000 prompt, 270,000 cached), leaky-low-spend ($5, 900,000 prompt, 0 cached). The Admin UI screenshots above are the UI proof for both legs, taken at http://localhost:3000/cost-optimization/ on the same rows (the dashboard's default range also includes the three control keys, hence "of 6 keys"). The After proxy's runtime code is identical between 30a957a, a7196b0 and the current head 7eca00c (rebase onto main plus test formatting only); the new route was re-verified on 7eca00c with the same fixture: leaky-low-spend first with 900,000 uncached tokens, then mid-cached with 30,000, total_api_keys: 3, limit: 2, and the non-admin cross-user call returned 403

Before (1e6c983)

Existing aggregated route, which the card ranks today

1. curl -s "localhost:4001/user/daily/activity/aggregated?start_date=$D&end_date=$D" -H "Authorization: Bearer $MK" piped through a one-liner printing api_key_limit, total_api_keys, and alias/spend/uncached per key
2. Observed, leaky-low-spend (900,000 uncached) is dropped before the card can rank it:

   api_key_limit= 2 total_api_keys= 3
   whale-cached spend= 90.0 uncached= 10000
   mid-cached spend= 40.0 uncached= 30000
   

New cache leakage route

1. curl -s -w " HTTP %{http_code}" "localhost:4001/user/daily/activity/cache_leakage?start_date=$D&end_date=$D" -H "Authorization: Bearer $MK"
2. Observed: {"detail":"Not Found"} HTTP 404

Non-admin scoping

1. Not applicable, the route does not exist

Admin UI

1. Open http://localhost:3001/cost-optimization/ (dashboard pointed at the Before proxy), Prompt Caching tab, Cache leakage card, By key
2. Observed: rows mid-cached 30,000 and whale-cached 10,000, plus the note "Only the 2 highest-spend keys of 6 are loaded, so a lower-spend key that leaks more is not listed here" (Before screenshot in User Flow)

After (a7196b0, re-run on 7eca00c)

Existing aggregated route, which the card ranks today

1. Same command against port 4000
2. Observed, unchanged spend cap behaviour:

   api_key_limit= 2 total_api_keys= 3
   whale-cached spend= 90.0 uncached= 10000
   mid-cached spend= 40.0 uncached= 30000
   

New cache leakage route

1. curl -s -w " HTTP %{http_code}" "localhost:4000/user/daily/activity/cache_leakage?start_date=$D&end_date=$D" -H "Authorization: Bearer $MK"
2. Observed, leaky-low-spend first, whale-cached (10,000 uncached) correctly cut by the limit of 2, totals over all 3 keys (api_key hashes elided):

   {
     "results": [
       {"key_alias": "leaky-low-spend", "prompt_tokens": 900000, "cache_read_input_tokens": 0, "cache_creation_input_tokens": 0, "uncached_prompt_tokens": 900000, "cache_hit_ratio": 0.0, "prompt_caching_savings_spend": 0.0},
       {"key_alias": "mid-cached", "prompt_tokens": 300000, "cache_read_input_tokens": 250000, "cache_creation_input_tokens": 20000, "uncached_prompt_tokens": 30000, "cache_hit_ratio": 0.8333333333333334, "prompt_caching_savings_spend": 20.0}
     ],
     "metadata": {"total_api_keys": 3, "limit": 2, "total_cached_tokens": 1260000, "total_prompt_caching_savings_spend": 80.0}
   }
   HTTP 200

Non-admin scoping

1. curl -s localhost:4000/key/generate -H "Authorization: Bearer $MK" -d '{"user_id":"lit8620-internal"}' for an internal_user key $UK
2. curl -s -w " HTTP %{http_code}" "localhost:4000/user/daily/activity/cache_leakage?start_date=$D&end_date=$D&user_id=some-other-user" -H "Authorization: Bearer $UK"
3. Observed: {"detail":{"error":"Non-admin users can only view their own spend data."}} HTTP 403
4. curl -s -w " HTTP %{http_code}" "localhost:4000/user/daily/activity/cache_leakage?start_date=$D&end_date=$D" -H "Authorization: Bearer $UK"
5. Observed, scoped to the caller's own (empty) rows: {"results":[],"metadata":{"total_api_keys":0,"limit":2,"total_cached_tokens":0,"total_prompt_caching_savings_spend":0.0}} HTTP 200

Admin UI

1. Open http://localhost:3000/cost-optimization/, Prompt Caching tab, Cache leakage card, By key
2. Observed: "Showing the top 2 of 6 keys, ranked by uncached prompt tokens", leaky-low-spend 900,000 / 0.0% / $57.14, mid-cached 30,000 / 83.3% / $1.90 (After screenshot in User Flow)

Type

🐛 Bug Fix

Caveats (if any)

Low

• Key ranking reads one extra request per card load, bounded by limit (default 10, capped at USAGE_TOP_API_KEYS_LIMIT)
• The key table caps at 10 rows; the model dimension still aggregates client-side as before
• Team-level pages do not use the new route yet, only the user-scoped card does
• Repo-wide tsc --noEmit -p . in ui/litellm-dashboard reports 957 pre-existing errors outside this diff; the changed files type-check clean and ESLint reports 0 errors on them
• CI: Analyze (python) (CodeQL) fails with LogInjection.ql: Result set is larger than the limit of 2GiB, an analyzer-wide result-size failure, not an alert on any line: the latest CodeQL run on main (f4308bc) fails on the same query with the same error; the CodeQL check itself passes and there are zero github-advanced-security[bot] review threads on this PR. oauth sits in waiting because its workflow is gated on the e2e-changed environment and its path filters (MCP/OAuth files) do not match this diff; recently merged main PRs show it in the same state
• Taxonomy audit on the current diff (blocks A-L, M-S, T-BB): no blocking findings. D2/Z7 scoping goes through the shared user resolver, PTU sentinel excluded in SQL; E5/X1 nullable sums coalesce to zero and negative uncached counts clamp with GREATEST; A1 no new cache, React Query key includes the scoped user; F3 model dimension deliberately unchanged, team pages out of scope; H3 the one em dash on an added line is the preexisting "—" empty-value placeholder moved during JSX restructuring, no new copy; H7 schema.d.ts regenerated via npm run gen:api; T18 useCacheLeakageKeys.test.ts calls the hook as a plain function with the query hook mocked, judged cosmetic since React hook semantics are not under test

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/565648bb3887474eb6fbd05c68e4cd6f
Open in Devin Desktop: https://app.devin.ai/desktop/session/565648bb3887474eb6fbd05c68e4cd6f?variant=devin
Requested by: @yassin-berriai

Link to Devin session: https://app.devin.ai/sessions/a7f364da79df46aa9a2873f31b955e65
Open in Devin Desktop: https://app.devin.ai/desktop/session/a7f364da79df46aa9a2873f31b955e65?variant=devin

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[codecov]

Codecov Report

❌ Patch coverage is 91.17647% with 6 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...xy/management_endpoints/internal_user_endpoints.py	70.58%	5 Missing ⚠️
...roxy/management_endpoints/common_daily_activity.py	96.87%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[mateo-berri]

@greptileai

[greptile-apps]

Confidence Score: 5/5

[Medium risk] Adds cache leakage ranking endpoint and UI for cost optimization.

The PR appears safe to merge based on the reviewed changes.

Summary

The PR adds a user-scoped cache-leakage endpoint that ranks keys by uncached prompt tokens and calculates savings from totals across all visible keys. The dashboard’s key view consumes that ranking while its model view remains unchanged. Changes since the previous review do not introduce a new issue in this PR.

_{Reviews (9) · Last reviewed commit: "fix(usage): rank cache leakage keys serv..."}

[mateo-berri]

@greptileai

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_cache_leakage_key_ranking (7eca00c) with main (f4308bc)}

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[devin-ai-integration]

@greptileai please review the current head 30a957a (merged main into the branch).

[devin-ai-integration]

@greptileai please review the current head a7196b0

[devin-ai-integration]

@greptileai please review the current head 7eca00c (rebased onto main, no code change versus a7196b0 beyond formatting)

[devin-ai-integration]

Superseded by the redesigned bounded-usage stack: repository #43398, routes #43408, UI #43409.