Repository navigation
fix(proxy): backport #40541, #43642, and #43656 to stable/1.103.x for v1.103.2 - #43897
mateo-berri merged 7 commits into
Conversation
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
|
|
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
… on stable/1.103.x
1 similar comment
|
bugbot run |
| _spend_logs_postgresql_proc: Final = factories.postgresql_proc() | ||
| _spend_logs_postgresql: Final = factories.postgresql("_spend_logs_postgresql_proc") |
There was a problem hiding this comment.
tests/test_litellm/ to use mocks rather than make real network calls. This requirement must be satisfied before merging; move these tests to an allowed integration-test location or remove their database connection.
Rule Used: What: prevent any tests from being added here that make real network calls - only mock tests can be added to this folder. Exception: tests/e2e folder, this folder can contain network calls. Why: ensure it runs correctly on github ci/cd + for all d... (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
There was a problem hiding this comment.
pytest-postgresql starts a throwaway local PostgreSQL process for the test session. Nothing leaves the machine, so this is not a real network call
There was a problem hiding this comment.
You're right that pytest-postgresql is local and isolated; however, repository policy still says tests/test_litellm can only contain mocked tests.
There was a problem hiding this comment.
Same fixtures merged on main in #43656, where Greptile withdrew this finding on 2026-09-29. Recording it as a false positive, no code change
1 similar comment
|
bugbot run |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit fe87252. Configure here.
![renovate[bot]](https://github.githubassets.com/images/gravatars/gravatar-user-420.png){kind=avatar}
…03.2) (#328) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/berriai/litellm](https://images.chainguard.dev/directory/image/wolfi-base/overview) ([source](https://github.com/BerriAI/litellm)) | patch | `v1.103.1` → `v1.103.2` | --- ### Release Notes <details> <summary>BerriAI/litellm (ghcr.io/berriai/litellm)</summary> ### [`v1.103.2`](https://github.com/BerriAI/litellm/releases/tag/v1.103.2) [Compare Source](BerriAI/litellm@v1.103.1...v1.103.2) #### Verify Docker Image Signature All LiteLLM Docker images are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](BerriAI/litellm@0112e53). **Verify using the pinned commit hash (recommended):** A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \ ghcr.io/berriai/litellm:v1.103.2 ``` **Verify using the release tag (convenience):** Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/v1.103.2/cosign.pub \ ghcr.io/berriai/litellm:v1.103.2 ``` Expected output: ``` The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key ``` *** #### What's Changed - chore(release): sync stable/1.103.x to v1.103.1 by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​43824](BerriAI/litellm#43824) - fix(proxy): backport [#​40541](BerriAI/litellm#40541), [#​43642](BerriAI/litellm#43642), and [#​43656](BerriAI/litellm#43656) to stable/1.103.x for v1.103.2 by [@​devin-ai-integration](https://github.com/devin-ai-integration)\[bot] in [#​43897](BerriAI/litellm#43897) - fix(anthropic): backport [#​42152](BerriAI/litellm#42152) and [#​42288](BerriAI/litellm#42288) to stable/1.103.x by [@​devin-ai-integration](https://github.com/devin-ai-integration)\[bot] in [#​43662](BerriAI/litellm#43662) - fix(proxy): backport [#​43962](BerriAI/litellm#43962) to stable/1.103.x by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​43984](BerriAI/litellm#43984) **Full Changelog**: <BerriAI/litellm@v1.103.1...v1.103.2> </details> --- ### Configuration 📅 **Schedule**: (in timezone America/New_York) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4xMTUuMTMiLCJ1cGRhdGVkSW5WZXIiOiI0NC4xMTUuMTMiLCJ0YXJnZXRCcmFuY2giOiJtYWluIiwibGFiZWxzIjpbInJlbm92YXRlL2NvbnRhaW5lciIsInR5cGUvcGF0Y2giXX0=--> Reviewed-on: https://git.greyrock.io/todd/home-ops/pulls/328
3 participants
TLDR
Problem this solves:
/user/daily/activity/aggregateddaily into their BIkey_aliasanduser_emailHow it solves it:
stable/1.103.x, one commit per PR, original authors keptv1.103.2User Flow
Before: a BI job pulling daily usage from a v1.103.0 gateway gets hashed CLI keys with no owner, so its per-user cost report is dirty
litellm-proxy login, gets a session token, and sends chat requests through the gateway all dayresults[].breakdown.api_keysunder a 64-character hash, and on a day where that hash has more spend logs than a 5-second lookup can scan, its"key_alias"and"user_email"come backnullwhile the gateway log sayscanceling statement due to statement timeoutAfter: the same pull names the user behind every CLI session key, old hashes included
litellm-proxy login, gets a session token, and sends chat requests through the gateway all daycli-session-<user_id>withkey_aliasanduser_emailfilled, and hashes written before the upgrade come back withuser_emailfrom the user their spend rows name andkey_aliasfrom their spend logs, however many spend logs the hash has that dayBackport checklist
bump: version 1.103.2as 460d51f, its own committests/integration/contracts.json, test files onlychore(release): bump litellm-enterprise 0.1.69 -> 0.1.69.post1 for stable/1.103.xas fe87252, its own commit, so the pip wheel carries fix(spend): attribute CLI session spend to the per-user cli-session alias instead of the hashed session token #40541's twoenterprise/changes (see Caveats)Notes for the reviewer:
The line still enforces the integration manifest (
tests/integration/contracts.json) that main dropped in test(integration): add MCP gateway coverage wave 1 with a dedicated mcp shard and proxy coverage artifact #42711, so without nodes for the four spend integration files fix(proxy): recover session key owners from daily spend for usage attribution #43642 and fix(proxy): look up hashed key names with two spend log rows per key #43656 add,tests/integration/run.pyrefuses the whole accounting group; 1bbc9ee registers their 72 nodes underquota_management.spend_tracking.*ids with matchingcoversmarkersThe customer runs v1.103.0, so the target is
stable/1.103.x, and v1.103.1 is the latest release on the line, so the next patch is v1.103.2fix(proxy): recover session key owners from daily spend for usage attribution #43642's integration tests import a
scratch_databasetest helper that test(integration): run the Langfuse DB-callback test on its own scratch database #43288 added to main after the line branched, so 74952e9 hand-ports that helper alone (test-only) as its own commit with the same provenance noteNo migration rides along: the
(api_key, "startTime")spend-logs index that fix(proxy): look up hashed key names with two spend log rows per key #43656 relies on is already on the line (20260823000000_add_spend_logs_api_key_starttime_index)Conflicts in the picked test files were resolved by dropping main-only context (tests and helpers that other PRs added after the line branched), so the line only gains what each PR added
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Screenshots / Proof of Fix
Last updated: fe87252. The integration accounting run below is at fe87252; QA and /live-pr-risk ran on 1bbc9ee, the last commit with a product diff (fe87252 only bumps the
litellm-enterpriseversion and pin)Two live proxies booted from this worktree, no mocks, real
gpt-5.6calls through OpenAI. Before on :4131 from the line's tip a32e70a (a detached worktree at that commit, put first on the import path), After on :4577 from this PR's tip 1bbc9ee. Each runs--num_workers 2, both share one Postgres 18 database (no Redis), onegpt-5.6deployment,PROXY_BATCH_WRITE_AT=5. The developercli-user-1was created withPOST /user/new, and the CLI session token was minted withExperimentalUIJWTToken.get_cli_jwt_auth_token(user_info), the same functionGET /sso/cli/poll/{key_id}handslitellm-proxy login(SSO is not wired locally). To match the customer's volume, the day's spend logs under that developer's hash were seeded to 800,000 rows, every one a clone of a real chat completion's spend row with its full 3.3 KB metadata as the proxy writes it (so it is TOASTed out of line like production rows); the line's whole-window scan over them took 46 s and 111 s in two timed passes right after seeding, far past the hardcoded 5 s statement timeout, which is the customer's condition. Both proxies were restarted after seeding so no in-memory key metadata cache carries over, both worktrees read the same.env(the enterprise license is what makes/organization/daily/activityanswer 200), and both legs ran the same commands in the same order. The Before leg was re-run after the base worktree got that.env, which is why its timestamp is later than the After leg'sIntegration accounting group, merge base vs head
Merge base a32e70a and head fe87252, each booted as its own proxy on its own database (
litellm_slp427_audit_baseon :42526 with Redis :35707 and the scripted upstream on :28361,litellm_slp427_audit_headon :39003 with Redis :50663 and the upstream on :45689, PostgreSQL 18 with every migration applied at boot), ran the same 85 cells oftests/integration/run.py accounting(thepricingandspendgroups oftests/integration/contracts.json) with--seed 4106601 --order-seed 0. The rig is the CircleCI shape from.circleci/scripts/run_integration.sh: the proxy ispython -m integration._support.proxy --config tests/integration/proxy_config.yaml --num_workers 1 --use_prisma_db_push --enforce_prisma_migration_checkin production mode withSTORE_MODEL_IN_DB=True,LITELLM_LOCAL_MODEL_COST_MAP=Trueandrouter_settings.num_retriesset to 0 overPOST /config/updatebefore the run, the base tree first on the import path for its leg, nothing mocked inside the proxy beyond the shim's route entitlement that CI grants too. Base (started 2026-09-30T22:46:41Z): 57 failed, 28 passed, 71 warnings in 550.74 s, 28 green and 57 red of 85 collected. Head (started 2026-09-30T22:56:33Z): 85 passed, 71 warnings in 702.08 s, 85 of 85 collected, no skip and no retry (-p no:pytest-retry -p no:rerunfailures). The 57 cells red on base are the behavior the three picks change and nothing else: 22 assert the owner that #43642 recovers from the daily spend rows, 30 are the alias-probe cells that assert that same owner next to the alias (everykey_metadata()expectation intest_daily_activity_key_alias_probes.pycarries it, so those cells are red on base with the alias itself already recovered where the cell expects it), 2 are the two alias window edges #43656 changes deliberately (its two Low caveats, called deliberate by mateo-berri in #43656 (comment) and #43656 (comment)), 1 is the CLI session alias #40541 writes, and 2 are the team-next-to-owner cells whose base outcome the inventory left open. Two earlier runs were retired, not counted, and neither touched the product: a base leg that booted the proxy throughproxy_cli.pyrather than CI's shim, soPOST /key/regenerateanswered as unlicensed (that cell is green on the base leg above) and/organization/daily/activityanswered 403 before reaching the owner assertion, every other cell with the outcome above; and a head run started next to that base leg on a loaded box, where one cell's owned proxy missed the 70 s readiness deadline oftests/integration/_support/process.pyand the other 84 cells passed, after which the head run above ran alone on a fresh database. A second head run with the same seeds (started 2026-09-30T23:11:45Z) was stopped at 52% with 45 cells passed and none failed when the full /audit was waived for this PR: it backports three PRs that already passed their review gates on main, so the runs above are recorded as the integration proof they are, not as an audit verdict. Cherry-pick fidelity:litellm/proxy/spend_tracking/key_metadata_recovery.pyat this head differs from main's merge of #43656 only by two main-only refactors this line never took (thefind_many_inchunking helper from #42629 and a predicate pulled into_user_id_needing_details), and the enterprise wheel built from this head islitellm_enterprise-0.1.69.post1-py3-none-any.whlcarrying_get_key_aliasincheck_batch_cost.pyandget_logged_api_keyinmanaged_files.pyMatrix, one row per cell
tests/integration/pricing/test_configured_prices.py::test_custom_price_is_reported_and_chargedtests/integration/pricing/test_configured_prices.py::test_default_prices_survive_nullable_sibling_and_reloadtests/integration/pricing/test_configured_prices.py::test_loaded_router_preserves_cached_defaults_during_real_requeststests/integration/pricing/test_off_peak_pricing.py::test_closed_off_peak_window_bills_standard_ratestests/integration/pricing/test_off_peak_pricing.py::test_open_off_peak_window_bills_off_peak_ratestests/integration/pricing/test_price_precedence.py::test_generated_zero_null_and_omitted_prices_follow_independent_arithmetictests/integration/pricing/test_price_precedence.py::test_same_upstream_aliases_keep_distinct_prices_after_reloadtests/integration/spend/test_cache_and_quota.py::test_different_system_messages_do_not_share_a_cached_responsetests/integration/spend/test_cache_and_quota.py::test_generated_cache_sequences_preserve_content_usage_and_zero_hit_costtests/integration/spend/test_cache_and_quota.py::test_key_budget_at_boundary_blocks_provider_then_explicit_reset_restorestests/integration/spend/test_cache_and_quota.py::test_repeated_hits_keep_response_identity_and_create_distinct_zero_cost_rowstests/integration/spend/test_daily_activity_key_alias_probes.py::test_alias_found_once_is_served_from_the_cache_for_the_same_window_onlytests/integration/spend/test_daily_activity_key_alias_probes.py::test_alias_logged_after_a_cached_miss_shows_once_the_miss_expirestests/integration/spend/test_daily_activity_key_alias_probes.py::test_alias_lookup_gives_up_while_spend_logs_are_locked_and_answers_once_they_are_nottests/integration/spend/test_daily_activity_key_alias_probes.py::test_alias_named_only_by_a_spend_log_is_reported_on_every_daily_activity_route[agent_daily_activity]tests/integration/spend/test_daily_activity_key_alias_probes.py::test_alias_named_only_by_a_spend_log_is_reported_on_every_daily_activity_route[customer_daily_activity]tests/integration/spend/test_daily_activity_key_alias_probes.py::test_alias_named_only_by_a_spend_log_is_reported_on_every_daily_activity_route[end_user_daily_activity]tests/integration/spend/test_daily_activity_key_alias_probes.py::test_alias_named_only_by_a_spend_log_is_reported_on_every_daily_activity_route[organization_daily_activity]tests/integration/spend/test_daily_activity_key_alias_probes.py::test_alias_named_only_by_a_spend_log_is_reported_on_every_daily_activity_route[tag_daily_activity]tests/integration/spend/test_daily_activity_key_alias_probes.py::test_alias_named_only_by_a_spend_log_is_reported_on_every_daily_activity_route[team_daily_activity]tests/integration/spend/test_daily_activity_key_alias_probes.py::test_alias_named_only_by_a_spend_log_is_reported_on_every_daily_activity_route[team_daily_activity_aggregated]tests/integration/spend/test_daily_activity_key_alias_probes.py::test_alias_named_only_by_a_spend_log_is_reported_on_every_daily_activity_route[user_daily_activity]tests/integration/spend/test_daily_activity_key_alias_probes.py::test_alias_named_only_by_a_spend_log_is_reported_on_every_daily_activity_route[user_daily_activity_aggregated]tests/integration/spend/test_daily_activity_key_alias_probes.py::test_alias_named_only_in_the_middle_of_two_hundred_nameless_rows_is_not_picked_uptests/integration/spend/test_daily_activity_key_alias_probes.py::test_alias_of_an_unexpected_shape_is_reported_as_postgres_renders_it[five_kb_string]tests/integration/spend/test_daily_activity_key_alias_probes.py::test_alias_of_an_unexpected_shape_is_reported_as_postgres_renders_it[json_int]tests/integration/spend/test_daily_activity_key_alias_probes.py::test_alias_of_an_unexpected_shape_is_reported_as_postgres_renders_it[json_list]tests/integration/spend/test_daily_activity_key_alias_probes.py::test_alias_on_an_edge_of_the_window_is_reported_whatever_surrounds_it[100_nameless_named_99_nameless]tests/integration/spend/test_daily_activity_key_alias_probes.py::test_alias_on_an_edge_of_the_window_is_reported_whatever_surrounds_it[99_nameless_named_100_nameless]tests/integration/spend/test_daily_activity_key_alias_probes.py::test_alias_on_an_edge_of_the_window_is_reported_whatever_surrounds_it[both_edges_named_150_nameless_between]tests/integration/spend/test_daily_activity_key_alias_probes.py::test_alias_on_an_edge_of_the_window_is_reported_whatever_surrounds_it[named_between_50_and_50_nameless]tests/integration/spend/test_daily_activity_key_alias_probes.py::test_alias_on_an_edge_of_the_window_is_reported_whatever_surrounds_it[newest_named_150_nameless_older]tests/integration/spend/test_daily_activity_key_alias_probes.py::test_alias_on_an_edge_of_the_window_is_reported_whatever_surrounds_it[oldest_named_150_nameless_newer]tests/integration/spend/test_daily_activity_key_alias_probes.py::test_concurrent_reads_over_every_route_all_name_a_fresh_keytests/integration/spend/test_daily_activity_key_alias_probes.py::test_hashed_jwt_digest_is_named_by_its_spend_logtests/integration/spend/test_daily_activity_key_alias_probes.py::test_key_renamed_and_renamed_back_is_reported_with_the_alias_on_both_edgestests/integration/spend/test_daily_activity_key_alias_probes.py::test_rows_without_a_usable_alias_do_not_hide_the_named_row_after_them[array_then_string_metadata]tests/integration/spend/test_daily_activity_key_alias_probes.py::test_rows_without_a_usable_alias_do_not_hide_the_named_row_after_them[empty_string_alias]tests/integration/spend/test_daily_activity_key_alias_probes.py::test_spend_log_names_the_key_only_from_one_day_before_to_two_days_after_the_read[first_second_after_the_window]tests/integration/spend/test_daily_activity_key_alias_probes.py::test_spend_log_names_the_key_only_from_one_day_before_to_two_days_after_the_read[first_second_of_the_window]tests/integration/spend/test_daily_activity_key_alias_probes.py::test_spend_log_names_the_key_only_from_one_day_before_to_two_days_after_the_read[last_second_of_the_window]tests/integration/spend/test_daily_activity_key_alias_probes.py::test_spend_log_names_the_key_only_from_one_day_before_to_two_days_after_the_read[second_before_the_window]tests/integration/spend/test_daily_activity_key_alias_probes.py::test_team_named_only_by_a_spend_log_is_reported_next_to_the_daily_owner[team_id_column]tests/integration/spend/test_daily_activity_key_alias_probes.py::test_team_named_only_by_a_spend_log_is_reported_next_to_the_daily_owner[team_id_in_metadata]tests/integration/spend/test_daily_activity_key_alias_probes.py::test_two_aliases_on_the_two_edges_leave_the_key_unnamedtests/integration/spend/test_daily_activity_key_alias_probes.py::test_user_named_by_a_spend_log_beats_the_owner_the_daily_rows_name[user_column]tests/integration/spend/test_daily_activity_key_alias_probes.py::test_user_named_by_a_spend_log_beats_the_owner_the_daily_rows_name[user_id_in_metadata]tests/integration/spend/test_daily_activity_key_owner.py::test_daily_spend_rows_naming_no_user_do_not_hide_the_one_user_the_others_name[blank_user]tests/integration/spend/test_daily_activity_key_owner.py::test_daily_spend_rows_naming_no_user_do_not_hide_the_one_user_the_others_name[null_user]tests/integration/spend/test_daily_activity_key_owner.py::test_deleted_key_keeps_its_own_user_when_its_daily_spend_names_anothertests/integration/spend/test_daily_activity_key_owner.py::test_deleted_key_with_no_user_keeps_its_alias_and_gains_the_one_user_its_daily_spend_namestests/integration/spend/test_daily_activity_key_owner.py::test_key_missing_from_the_key_tables_is_reported_with_the_one_user_its_daily_spend_names[agent_daily_activity]tests/integration/spend/test_daily_activity_key_owner.py::test_key_missing_from_the_key_tables_is_reported_with_the_one_user_its_daily_spend_names[customer_daily_activity]tests/integration/spend/test_daily_activity_key_owner.py::test_key_missing_from_the_key_tables_is_reported_with_the_one_user_its_daily_spend_names[end_user_daily_activity]tests/integration/spend/test_daily_activity_key_owner.py::test_key_missing_from_the_key_tables_is_reported_with_the_one_user_its_daily_spend_names[organization_daily_activity]tests/integration/spend/test_daily_activity_key_owner.py::test_key_missing_from_the_key_tables_is_reported_with_the_one_user_its_daily_spend_names[tag_daily_activity]tests/integration/spend/test_daily_activity_key_owner.py::test_key_missing_from_the_key_tables_is_reported_with_the_one_user_its_daily_spend_names[team_daily_activity]tests/integration/spend/test_daily_activity_key_owner.py::test_key_missing_from_the_key_tables_is_reported_with_the_one_user_its_daily_spend_names[team_daily_activity_aggregated]tests/integration/spend/test_daily_activity_key_owner.py::test_key_missing_from_the_key_tables_is_reported_with_the_one_user_its_daily_spend_names[user_daily_activity]tests/integration/spend/test_daily_activity_key_owner.py::test_key_missing_from_the_key_tables_is_reported_with_the_one_user_its_daily_spend_names[user_daily_activity_aggregated]tests/integration/spend/test_daily_activity_key_owner.py::test_key_named_only_by_a_spend_log_alias_keeps_that_alias_and_gains_the_one_user_its_daily_spend_namestests/integration/spend/test_daily_activity_key_owner.py::test_key_whose_daily_spend_names_no_user_at_all_is_reported_with_no_ownertests/integration/spend/test_daily_activity_key_owner.py::test_key_whose_daily_spend_names_two_users_is_reported_with_no_ownertests/integration/spend/test_daily_activity_key_owner.py::test_live_key_keeps_its_own_user_when_its_daily_spend_names_anothertests/integration/spend/test_daily_activity_key_owner.py::test_live_key_with_no_user_is_not_given_the_user_its_daily_spend_namestests/integration/spend/test_daily_activity_key_owner.py::test_owner_the_user_table_does_not_hold_is_reported_by_id_with_no_emailtests/integration/spend/test_daily_activity_key_owner_faults.py::test_every_key_of_a_team_is_reported_with_its_own_usertests/integration/spend/test_daily_activity_key_owner_faults.py::test_five_kilobyte_key_is_reported_with_the_one_user_its_daily_spend_namestests/integration/spend/test_daily_activity_key_owner_faults.py::test_invalid_key_is_refused_without_naming_the_ownertests/integration/spend/test_daily_activity_key_owner_faults.py::test_key_stops_being_reported_with_an_owner_once_a_second_user_spends_with_ittests/integration/spend/test_daily_activity_key_owner_faults.py::test_key_with_no_daily_spend_is_reported_as_no_activitytests/integration/spend/test_daily_activity_key_owner_faults.py::test_owner_is_reported_again_after_the_proxy_restartstests/integration/spend/test_daily_activity_key_owner_faults.py::test_owner_is_reported_while_a_worker_is_killed_and_after_it_is_replacedtests/integration/spend/test_daily_activity_key_owner_faults.py::test_owner_lookup_gives_up_while_daily_user_spend_is_locked_and_answers_once_it_is_nottests/integration/spend/test_daily_activity_key_owner_faults.py::test_reading_the_same_activity_twice_gives_the_same_answertests/integration/spend/test_daily_activity_key_owner_faults.py::test_user_reading_a_key_only_another_user_spent_with_is_shown_nothing_of_ittests/integration/spend/test_daily_activity_key_owner_faults.py::test_user_reading_a_key_only_they_spent_with_is_shown_themselves_as_its_ownertests/integration/spend/test_daily_activity_key_owner_faults.py::test_user_reading_a_key_shared_with_another_user_is_shown_no_owner_and_nothing_of_the_other_usertests/integration/spend/test_daily_activity_key_owner_traffic.py::test_cli_session_spend_is_reported_with_the_user_and_team_of_the_sessiontests/integration/spend/test_daily_activity_key_owner_traffic.py::test_key_purged_from_the_key_tables_is_reported_with_the_alias_its_spend_logs_nametests/integration/spend/test_daily_activity_key_owner_traffic.py::test_key_used_on_every_unified_endpoint_is_reported_with_its_own_alias_and_usertests/integration/spend/test_daily_activity_key_owner_traffic.py::test_owner_is_reported_on_every_route_while_a_burst_of_requests_waits_on_the_providertests/integration/spend/test_daily_activity_key_owner_traffic.py::test_usage_ai_chat_hands_the_model_the_usage_summary_without_any_key_ownertests/integration/spend/test_filtered_ledger.py::test_rotated_keys_users_and_model_groups_preserve_success_failure_cache_ledgertests/integration/spend/test_team_member_spend_flush.py::test_fractional_member_spend_lands_after_a_whole_number_flush_on_the_same_connectionAfter (1bbc9ee, :4577)
Observations
key_existsfalse for session keys both legs; left aloneType
🐛 Bug Fix
Caveats (if any)
Medium, fixed by fe87252
enterprise/files (check_batch_cost.pywrites the batch's spend under the session alias,managed_files.pydoes the same for managed-file spend). The Docker image is fine either way, since the Dockerfile installslitellm-enterprisefrom the workspace source (uv sync --frozen ... --no-editableafterCOPY . .). Apip install litellm[proxy]==1.103.2resolvedlitellm-enterprise==0.1.69from PyPI instead, a wheel published from main on 2026-09-20, before fix(spend): attribute CLI session spend to the per-user cli-session alias instead of the hashed session token #40541 landed there on 2026-09-24, so a pip-installed proxy would still write enterprise batch and managed-file spend under the hash and only name it through the recovery. fe87252 bumps the line tolitellm-enterprise 0.1.69.post1(bothenterprise/pyproject.tomlversion lines, the pin inpyproject.toml, theuv.lockentry;uv lock --checkpasses) the wayc4b58deaacfdid for stable/1.88.x, so the release publishes a wheel built from this branch and the pip install pins it. Main's own bump0.1.71is not reused because it names main's tree, not the line'sLow
rust-wheel(4 sync/async ordering failures intests/test_litellm_rust/messages/test_callbacks.py) is red on the line's own latest run at the merge base a32e70a (run 36755425882) and on the line's last merge chore(release): sync stable/1.103.x to v1.103.1 #43824 (run 36652970017); this branch's runs (36783173067 at fe87252, fired by thepyproject.tomlanduv.lockpush paths) fail the same four, and the picks touch no Rust path. Left alone: fixing the line's Rust build here would widen a three-PR backport into unrelated build tooling, and the check is not required onstable/**codecov/patchon backport lines is a known coverage-upload gap, not actionable here. Left alone: the shards that would fix it live in the CircleCI config, which this line does not run for PRsReview bot verdicts at fe87252
tests/test_litellm/proxy/spend_tracking/test_key_metadata_recovery.py:610("Real database in mock-only tests") is a false positive:pytest-postgresqlstarts a throwaway local PostgreSQL process, nothing leaves the machine, and the same fixtures at the same line are already on main through fix(proxy): look up hashed key names with two spend log rows per key #43656, where Greptile raised the identical finding on 2026-09-29 and withdrew it on that rebuttal. Rebutted in-thread here, no code change; Greptile's re-run after that reply finished at 22:21Z with no counter and moved the score to 5/5, the thread stays open as its recordFinal Attestation
Risk report
/live-pr-risk at 1bbc9ee against the line's tip a32e70a, the same two-worker proxies and shared Postgres as the proof above, no mocks. The picks are byte-for-byte main's product code at 61a73c5 except for three conflict resolutions that keep the line's own helpers:
_details_for_user_idscalls Prismafind_many(where={"user_id": {"in": [...]}})because the chunkedfind_many_inhelper (#42629) never landed on the line,attach_user_detailskeeps the inline comprehension main later split into_user_id_needing_details, and the pagination incommon_daily_activity.pykeepsgetattr(prisma_client.db, table_name)where main names aTableActionslocal. All three are the same behavior in different spellingBreaking
None observed. Every route below answered HTTP 200 on both builds and the aggregated body differs only in the recovered
key_alias,user_id, anduser_emailvalues and in the alias the new session rows sit underBackward incompatible
The same five changes main shipped with #40541, #43642, and #43656, now on the line
cli-session-<user_id>inapi_keyandmetadata.user_api_keyinstead of the session token's sha256, soGET /spend/logsand the daily activity routes list new sessions under the alias. Observed above on the After leg for all three unified endpoints. This is the change fix(spend): attribute CLI session spend to the per-user cli-session alias instead of the hashed session token #40541 made on main, approved by @tin-berri therekey_alias,user_id, anduser_emailfilled from their spend logs and daily spend rows. Observed above on the old hash. This is what fix(proxy): recover session key owners from daily spend for usage attribution #43642 and fix(proxy): look up hashed key names with two spend log rows per key #43656 made on mainuser_api_key_hashkey session requests by the alias, so CLI logins of one user share one bucket. Not driven live here (no limits on the session token, no callbacks configured); main's fix(spend): attribute CLI session spend to the per-user cli-session alias instead of the hashed session token #40541 unit tests cover the alias lookup and its Medium caveat names itkey_alias,user_email, andteam_id; the rows and spend stay intact. Read from the line's tip code path, which only recovers sha256 digestsRegression risk
stable/1.103.xis still at a32e70a, this branch's base, and the line's last merge chore(release): sync stable/1.103.x to v1.103.1 #43824 shares onlypyproject.tomlanduv.lock(version lines) with the picks, sogit merge-treeof the line and this tip is clean and the merged tree is this tip(api_key, "startTime")index the probes rely on is already on the line (20260823000000_add_spend_logs_api_key_starttime_index), and the QA database has it, so a database that skipped that migration was not observedDependency graph
The picked symbols have the same caller files on the line as on main (
get_logged_api_keyin 11 files, the recovery helpers only incommon_daily_activity.py,fill_missing_api_key_aliasesin the CloudZero and FOCUS exports), checked with a grep of both treesPOST /v1/chat/completions,/v1/messages,/v1/responseswith a session tokenPOST /openai/v1/chat/completionspass-through with a session tokenGET /spend/logs?request_id=GET /user/daily/activity/aggregatedas admin and as the developerGET /user/daily/activity,/team/daily/activity{,/aggregated},/tag,/organization,/customer,/end_user,/agentactivity_metrics.tsx)activity_metrics.test.tsx, 60 passed)Not verified
The Prometheus label and third-party loggers for session requests (no callbacks in the QA config)
The enterprise callers (
managed_files.py,check_batch_cost.py) and the batch, skill, and websearch spend pathsA database without the
(api_key, "startTime")indexThe Admin UI Usage page against these proxies; the dashboard column change is covered by its Vitest suite
1bbc9ee passes /live-pr-risk