Repository navigation
fix(proxy): recover session key owners from daily spend for usage attribution - #43642
Conversation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
bugbot run |
…ry usage route Thirty five integration cells under tests/integration/spend cover the daily spend owner fallback on all nine daily activity routes and /usage/ai/chat: the happy path per route, the unanimity rules (two users, blank and null rows, an owner the user table lacks, live and deleted keys with and without their own user, a spend log alias), a non admin reader, an invalid key, a 5 KB key, a locked LiteLLM_DailyUserSpend, 300 keys of one team, repeated reads, a second user landing between reads, a concurrent burst across the unified endpoints, a killed worker, and a proxy restart The traffic cells ignore the GET /v1/models call the proxy's five minute token limit refresh makes to every registered OpenAI compatible deployment, since it lands on a test's provider wire whenever the refresh instant falls inside the test
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 0059275. Configure here.
Hand-ported prerequisite for the BerriAI#43642 backport to stable/1.103.x. Taken from dd636373229b1ab3bfb2ed5b9b17b0ebd68b3ac7 (BerriAI#43288, main), scratch_database only.
…ribution (BerriAI#43642) Backport of BerriAI#43642 to stable/1.103.x. Cherry-picked from f5a1c9f (main).
…03.2) (#328) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/berriai/litellm](https://images.chainguard.dev/directory/image/wolfi-base/overview) ([source](https://github.com/BerriAI/litellm)) | patch | `v1.103.1` → `v1.103.2` | --- ### Release Notes <details> <summary>BerriAI/litellm (ghcr.io/berriai/litellm)</summary> ### [`v1.103.2`](https://github.com/BerriAI/litellm/releases/tag/v1.103.2) [Compare Source](BerriAI/litellm@v1.103.1...v1.103.2) #### Verify Docker Image Signature All LiteLLM Docker images are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](BerriAI/litellm@0112e53). **Verify using the pinned commit hash (recommended):** A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \ ghcr.io/berriai/litellm:v1.103.2 ``` **Verify using the release tag (convenience):** Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/v1.103.2/cosign.pub \ ghcr.io/berriai/litellm:v1.103.2 ``` Expected output: ``` The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key ``` *** #### What's Changed - chore(release): sync stable/1.103.x to v1.103.1 by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​43824](BerriAI/litellm#43824) - fix(proxy): backport [#​40541](BerriAI/litellm#40541), [#​43642](BerriAI/litellm#43642), and [#​43656](BerriAI/litellm#43656) to stable/1.103.x for v1.103.2 by [@​devin-ai-integration](https://github.com/devin-ai-integration)\[bot] in [#​43897](BerriAI/litellm#43897) - fix(anthropic): backport [#​42152](BerriAI/litellm#42152) and [#​42288](BerriAI/litellm#42288) to stable/1.103.x by [@​devin-ai-integration](https://github.com/devin-ai-integration)\[bot] in [#​43662](BerriAI/litellm#43662) - fix(proxy): backport [#​43962](BerriAI/litellm#43962) to stable/1.103.x by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​43984](BerriAI/litellm#43984) **Full Changelog**: <BerriAI/litellm@v1.103.1...v1.103.2> </details> --- ### Configuration 📅 **Schedule**: (in timezone America/New_York) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4xMTUuMTMiLCJ1cGRhdGVkSW5WZXIiOiI0NC4xMTUuMTMiLCJ0YXJnZXRCcmFuY2giOiJtYWluIiwibGFiZWxzIjpbInJlbm92YXRlL2NvbnRhaW5lciIsInR5cGUvcGF0Y2giXX0=--> Reviewed-on: https://git.greyrock.io/todd/home-ops/pulls/328
TLDR
Problem this solves:
How it solves it:
This completes the fix started in #40729, which only recovered the owner from spend logs
Intentional product change: on the Usage page's Top Virtual Keys (Global, Team, and Tag views), a session key with no alias now shows its owner's email in the Key Alias cell and a User column appears, Model Activity key rows name the owner, and the Key Activity search finds these keys by email. This is what makes the keys attributable. Keys with no known owner still show the truncated hash, and no column goes away, but on the Tag view the wider table puts Spend (USD) behind a sideways scroll in a browser window 1500 px wide (it fits at 1728 px), the same as for any key with an owner today
User Flow
Before: an admin on a proxy with spend logs off cannot tell which user a CLI session key belongs to
lite login, signs in with SSO, and sends chat completions through the proxy with the session key"user_id": nulland"user_email": null-After: the same report and the same page name the owner of every session key
lite login, signs in with SSO, and sends chat completions through the proxy with the session keyuser_idanduser_emailLinear ticket
Resolves LIT-7572
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Everything below ran against one Postgres and one Redis, with real SSO logins and real Anthropic calls, and nothing mocked. Setup shared by both sides:
A v1.103.0-rc.1 proxy on :58712 (2 workers, the config below) recorded the spend, since that release still records CLI session spend under a key hash that is not in the keys table
One developer signed in twice with
lite loginthrough Google SSO, which gave them two CLI session keysThey sent 5 real
claude-haiku-4-5chat completions through that proxy, 3 on one key and 2 on the otherPOST http://localhost:58712/user/updateset their email todev.one@example.comTwo more proxies read the same database with the same config, 2 workers each, each serving its own dashboard build: base dab2deb on :21988 and tip 0059275 on :38554
For the timeout case only,
seed_timeout.sqlandseed_timeout_wide.sqlbelow added 8,400,000 daily spend rows for the two keys and the rig droppedLiteLLM_DailyUserSpend_api_key_idx, so the owner lookup had to scan an 8,400,002-row table (10220 MB with its remaining indexes, 6642 MB for the table alone, measured after the drop). The rows were deleted and the index was put back after the caseFor the two-user case only,
seed_probe_keys.sqlbelow added 4 daily spend rows for two made-up keys, one used by two users and one with a row that names no user. The rows were deleted afterwardsRows and filter for the two-user case
probes.jqProxy config, the same file on all three proxies
seed_timeout.sql and seed_timeout_wide.sql from step 6
Every case prints with
owners.jq, which cuts key hashes to 12 characters and user ids to 4:Before (dab2deb)
Aggregated usage API
Ask for the day's usage twice. Both session keys come back with no
user_idand nouser_emailPer-day usage API
Ask for the same day from the per-day route twice. Both session keys come back with no owner
Team and tag usage APIs
Ask the team and tag routes for the same day. Both session keys come back with no owner on all three
Admin UI
Open http://localhost:21988/ui/?page=new_usage and log in as
admin. The Cost and Model Activity tabs look like the Before screenshots under User FlowPick "Team Usage" in the view selector at the top. Top Virtual Keys shows Key ID, Key Alias (
-), and SpendPick "Tag Usage" with the browser 1500 px wide. The keys table has four columns and all of them fit
Pick "Global Usage", open the Key Activity tab, and type
dev.one@example.cominto "Search keys". It reads "Showing 0 of 2 keys"Owner lookup timeout
With the table from setup step 6 in place, ask for the day's usage twice. Both answers are fast and carry no owner (captured 2026-09-29 21:51 UTC, times rounded to the millisecond)
Keys with two users or a blank user
With the rows from setup step 7 in place, ask for the day's usage twice. Neither made-up key gets an owner (captured 2026-09-29 21:24 UTC)
After (0059275)
Aggregated usage API
Ask for the day's usage twice. Both session keys come back with the developer's
user_idanduser_emailPer-day usage API
Ask for the same day from the per-day route twice. Both session keys come back with their owner
Team and tag usage APIs
Ask the team and tag routes for the same day. Both session keys come back with their owner on all three
Admin UI
Open http://localhost:38554/ui/?page=new_usage and log in as
admin. The Cost and Model Activity tabs look like the After screenshots under User FlowPick "Team Usage" in the view selector at the top. Top Virtual Keys shows the developer's email in Key Alias and in a new User column
Pick "Tag Usage" with the browser 1500 px wide. The keys table has five columns, and Spend (USD) shows after a sideways scroll of the table
Pick "Global Usage", open the Key Activity tab, and type
dev.one@example.cominto "Search keys". It reads "Showing 2 of 2 keys"Owner lookup timeout
With the table from setup step 6 in place, ask for the day's usage twice. Both answers are HTTP 200 after 5 to 6 s, and neither carries an owner since the lookup hit its 5 s limit both times (captured 2026-09-29 21:51 UTC, times rounded to the millisecond)
The proxy log names the timeout, once per answer
Keys with two users or a blank user
With the rows from setup step 7 in place, ask for the day's usage twice. The key with two users still gets no owner, and the key with a blank row gets the one user its other row names (captured 2026-09-29 21:24 UTC)
Audit matrix
The three test files this PR adds under
tests/integration/spend/ran throughtests/integration/run.py accountingon a rig shaped like CircleCI's: an owned Postgres and Redis, the scripted upstream fromtests/integration/_support/upstream.py, and a proxy with 2 workers per side, base dab2deb on :52094 and tip 0059275 on :52097, each on its own database. The base ran once and the tip six times. Every tip run collected the same 35 tests and passed all of them with no skip, retry, or sleep, and the base failed exactly the 22 cells the fix changes and passed the 13 it must not touchThe 35 cells, base dab2deb against tip 0059275 six times
test_daily_activity_key_owner.py::test_key_missing_from_the_key_tables_is_reported_with_the_one_user_its_daily_spend_names[user_daily_activity]test_daily_activity_key_owner.py::test_key_missing_from_the_key_tables_is_reported_with_the_one_user_its_daily_spend_names[user_daily_activity_aggregated]test_daily_activity_key_owner.py::test_key_missing_from_the_key_tables_is_reported_with_the_one_user_its_daily_spend_names[team_daily_activity]test_daily_activity_key_owner.py::test_key_missing_from_the_key_tables_is_reported_with_the_one_user_its_daily_spend_names[team_daily_activity_aggregated]test_daily_activity_key_owner.py::test_key_missing_from_the_key_tables_is_reported_with_the_one_user_its_daily_spend_names[tag_daily_activity]test_daily_activity_key_owner.py::test_key_missing_from_the_key_tables_is_reported_with_the_one_user_its_daily_spend_names[organization_daily_activity]test_daily_activity_key_owner.py::test_key_missing_from_the_key_tables_is_reported_with_the_one_user_its_daily_spend_names[customer_daily_activity]test_daily_activity_key_owner.py::test_key_missing_from_the_key_tables_is_reported_with_the_one_user_its_daily_spend_names[end_user_daily_activity]test_daily_activity_key_owner.py::test_key_missing_from_the_key_tables_is_reported_with_the_one_user_its_daily_spend_names[agent_daily_activity]test_daily_activity_key_owner_traffic.py::test_usage_ai_chat_hands_the_model_the_usage_summary_without_any_key_ownertest_daily_activity_key_owner.py::test_key_whose_daily_spend_names_two_users_is_reported_with_no_ownertest_daily_activity_key_owner.py::test_daily_spend_rows_naming_no_user_do_not_hide_the_one_user_the_others_name[blank_user]test_daily_activity_key_owner.py::test_daily_spend_rows_naming_no_user_do_not_hide_the_one_user_the_others_name[null_user]test_daily_activity_key_owner.py::test_key_whose_daily_spend_names_no_user_at_all_is_reported_with_no_ownertest_daily_activity_key_owner.py::test_owner_the_user_table_does_not_hold_is_reported_by_id_with_no_emailtest_daily_activity_key_owner.py::test_live_key_keeps_its_own_user_when_its_daily_spend_names_anothertest_daily_activity_key_owner.py::test_live_key_with_no_user_is_not_given_the_user_its_daily_spend_namestest_daily_activity_key_owner.py::test_deleted_key_keeps_its_own_user_when_its_daily_spend_names_anothertest_daily_activity_key_owner.py::test_deleted_key_with_no_user_keeps_its_alias_and_gains_the_one_user_its_daily_spend_namestest_daily_activity_key_owner.py::test_key_named_only_by_a_spend_log_alias_keeps_that_alias_and_gains_the_one_user_its_daily_spend_namestest_daily_activity_key_owner_traffic.py::test_cli_session_spend_is_reported_with_the_user_and_team_of_the_sessiontest_daily_activity_key_owner_traffic.py::test_key_used_on_every_unified_endpoint_is_reported_with_its_own_alias_and_usertest_daily_activity_key_owner_faults.py::test_owner_lookup_gives_up_while_daily_user_spend_is_locked_and_answers_once_it_is_nottest_daily_activity_key_owner_faults.py::test_user_reading_a_key_shared_with_another_user_is_shown_no_owner_and_nothing_of_the_other_usertest_daily_activity_key_owner_faults.py::test_user_reading_a_key_only_they_spent_with_is_shown_themselves_as_its_ownertest_daily_activity_key_owner_faults.py::test_user_reading_a_key_only_another_user_spent_with_is_shown_nothing_of_ittest_daily_activity_key_owner_faults.py::test_invalid_key_is_refused_without_naming_the_ownertest_daily_activity_key_owner_faults.py::test_five_kilobyte_key_is_reported_with_the_one_user_its_daily_spend_namestest_daily_activity_key_owner_faults.py::test_key_with_no_daily_spend_is_reported_as_no_activitytest_daily_activity_key_owner_faults.py::test_every_key_of_a_team_is_reported_with_its_own_usertest_daily_activity_key_owner_faults.py::test_reading_the_same_activity_twice_gives_the_same_answertest_daily_activity_key_owner_faults.py::test_key_stops_being_reported_with_an_owner_once_a_second_user_spends_with_ittest_daily_activity_key_owner_traffic.py::test_owner_is_reported_on_every_route_while_a_burst_of_requests_waits_on_the_providertest_daily_activity_key_owner_faults.py::test_owner_is_reported_while_a_worker_is_killed_and_after_it_is_replacedtest_daily_activity_key_owner_faults.py::test_owner_is_reported_again_after_the_proxy_restartsSeen along the way, none of it caused by this PR:
key_aliasstays null on both sides with spend logs off (fix(proxy): look up hashed key names with two spend log rows per key #43656 covers it)/ui.txt?login=successfollows the dashboard login on both sidesType
🐛 Bug Fix
Caveats (if any)
Low
misc / Run testsis red on 4test_openapi_compliance.pytestsmain(run), fix is in test(interactions): pin and refresh OpenAPI compliance contract #43532maintoo (pipeline 90621), fix is open in test(ci): refresh retired OpenAI tool-call models #43676test_aaaaaschema_compatibilityneeds feat(mcp): scan and pin upstream tool descriptions #43283 and feat: add model leaderboard page #43649, which landed onmainafter this branch's basetest_async_shadow_does_not_inherit_primary_deployment_tagsis on CircleCI's flaky list (pipelines 90517 and 90562) and passed on the first runintegration-accountingfails on a shutdown flush test from the flaky list (pipeline 90388), which never calls a usage routerun-cilabel trigger is offlocal_testing_part2and 1 inlangfuse_logging_unit_tests, all 12 among the 31 listed for pipeline 90599 aboveintegrationworkflow was canceled too, so nointegration-accountingresult exists at this tiposv-scanis red on 3 Medium advisories against 2 packages inuv.lock, which this PR does not touchSeed behind the lookup timings
Final Attestation
Link to Devin session: https://app.devin.ai/sessions/44c211256d5d4fd3886c33c80dce006e
Open in Devin Desktop: https://app.devin.ai/desktop/session/44c211256d5d4fd3886c33c80dce006e?variant=devin
Requested by: @jesus-berri
Risk report
/live-pr-risk at 0059275 against base dab2deb, both builds live on the same database. Verdict: nothing broke and no regression was observed. Five changes a client or an admin can see are listed under Backward incompatible, each with its decision
Breaking
None observed. Every route below answered HTTP 200 on both builds, and a whole-body
jq -Sdiff of base against tip shows onlyuser_idanduser_emailchangingBackward incompatible
metadata.user_idandmetadata.user_emailgo from null to the owner for session keys, on the user, team, and tag daily activity routes. Observed in the API cases above. A client that groups keys by a null owner sees them move. This is the change LIT-7572 asks forTop Virtual Keys on the Global, Team, and Tag views shows the email in Key Alias and gains a User column, and Model Activity rows name the owner. Observed in the screenshots above. This is the intended product change named in the TLDR
On the Tag view the five-column keys table puts Spend (USD) behind a sideways scroll at 1280 and 1500 px wide. Observed on tip, and on base with one extra key that has an owner (below, Spend cut off at the right edge), so the layout predates this PR and session keys now reach it. Accepted as a Low caveat by @mateo-berri on 2026-09-29, in this description
The Key Activity search by email goes from "Showing 0 of 2 keys" to "Showing 2 of 2 keys". Observed in the Admin UI case. This follows from the change LIT-7572 asks for
Usage routes can take up to 5 s longer when the owner lookup is slow, and then answer with no owner. Observed in the timeout case. Accepted as a Low caveat by @mateo-berri on 2026-09-29, in this description
Regression risk
The AI usage chat runs the same lookup and never reads key owners.
POST /usage/ai/chatgave the same answer on both builds, HTTP 200 in 2.9 s on the base and 2.5 s on the tipThe old usage page calls none of the changed routes and imports none of the changed components, so it cannot differ
A key with two users gets no owner on both builds, shown in the two-user case
Dependency graph
GET /user/daily/activityGET /user/daily/activity/aggregatedGET /team/daily/activityGET /team/daily/activity/aggregatedGET /tag/daily/activityGET /organization/daily/activityGET /customer/daily/activityGET /end_user/daily/activityGET /agent/daily/activityNot verified
The top cases were not run again on a tree merged into today's
main.git merge-treeagainstmainat 2d034bb is clean, and no commit onmainsince the base touches this PR's files (checked 2026-09-29 22:07 UTC)The organization, customer, end user, and agent routes returned the same empty body on both builds, since the rig has no such spend
No outbound recorder was used, since the change adds one database read and no outbound call
0059275 passes /live-pr-risk
Note
Medium Risk
Changes usage API metadata and adds an uncached DB lookup on daily-activity loads (up to ~5s timeout per request); behavior is gated but affects reporting and UI attribution for shared or ambiguous keys.
Overview
Adds a daily-spend fallback so usage APIs and the dashboard can attribute ownerless session keys (e.g. CLI session hashes not in the keys table) when spend logs are unavailable.
Proxy:
get_api_key_metadatanow callsrecover_key_owner_from_daily_spend, which readsLiteLLM_DailyUserSpend(with the same statement timeout as spend-log recovery) for keys that still lackuser_idand are not active in the keys table. An owner is set only when all non-empty daily rows agree on one user; conflicting or all-blank rows leave owner null. Live/deleted keys keep their table-backed user and are not overridden by daily spend.Dashboard: Model Activity “Top Virtual Keys” and related aggregations expose owner email or user id on each key row (with UI to avoid duplicating the alias label).
Tests: New integration coverage across daily-activity routes, auth/timeout/worker fault cases, and live traffic; unit tests for recovery and metadata wiring.
Reviewed by Cursor Bugbot for commit 0059275. Bugbot is set up for automated code reviews on this repo. Configure here.