Skip to content

fix(proxy): attribute org spend via team when key has no org_id - #30799

Closed
mateo-berri wants to merge 3 commits into
litellm_internal_stagingfrom
litellm_org-spend-team-fallback-49f5
Closed

fix(proxy): attribute org spend via team when key has no org_id#30799
mateo-berri wants to merge 3 commits into
litellm_internal_stagingfrom
litellm_org-spend-team-fallback-49f5

Conversation

@mateo-berri

@mateo-berri mateo-berri commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Relevant issues

Surfaced by the new every-12h e2e suite in #30790 (tests/e2e_tests/budgets/test_budget_enforcement_e2e.py::test_organization_budget_blocks), which is the "org level budget gap" flagged in #eng.

Linear ticket

Pre-Submission checklist

  • I have added meaningful tests
  • My PR passes all unit tests on make test-unit
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review (5/5)

Changes

Org budget enforcement and org spend tracking disagreed on how they resolve a request's organization. Enforcement (auth_checks._organization_max_budget_check and budget_reservation._get_org_budget_counter) falls back to the team's organization_id when the key itself has no org_id, but spend tracking did not: in proxy_track_cost_callback._PROXY_track_cost_callback the org came straight from metadata["user_api_key_org_id"] (i.e. valid_token.org_id), which is null for a key created under a team that belongs to an org. Keys generated with only a team_id do not inherit the team's org onto the key row, so that value is null in the common case.

The result is that for a key whose org comes from its team, _update_org_db, the daily org transaction, and the unreserved spend:org:{org_id} counter were all skipped (org_id None). The org's LiteLLM_OrganizationTable.spend column, which is the persisted floor that read-time enforcement reads back through get_current_spend(..., fallback_spend=org_table.spend), therefore stayed at zero. Org budgets then fail to enforce whenever the in-memory reservation counter isn't the source of truth: across workers that don't share it (the e2e proxy runs with --workers), after the counter's TTL, or when reservation is disabled / the model is unpriced.

The fix resolves the spend-tracking org the same way enforcement already does: when the request has no org_id but does have a team_id, look up the (cached) team and use its organization_id. This is confined to the spend-tracking path and deliberately does not mutate valid_token.org_id, so org-scoped access control (resource ownership, MCP, rate limiting) is untouched.

A team-lookup failure during this resolution is swallowed so cost tracking keeps working, but it now logs at warning (not debug) with the attribution impact; at debug level that gap would be invisible in production and an unenforced org budget would have no signal to investigate. Follow-up commits also add regression coverage for the no-DB-client and lookup-failure branches so the patch is fully exercised.

Where the gap is documented

The PR #30790 budget matrices describe the org row as enforcing off real-time reservation counters; they do not call out that the persisted org floor is never written for team-inherited orgs. The closest existing pointer is the unit suite tests/test_litellm/proxy/auth/test_organization_budget_enforcement.py, which only passes because it injects org spend directly and mocks get_org_object, so it never exercises the team->org spend write path that the live e2e test does. There is no public docs page describing this gap; it lives in those test matrices plus the live suite.

Screenshots / Proof of Fix

Unit + regression (the two new behavioral tests fail before the fix with org_id == None, pass after):

$ uv run pytest tests/test_litellm/proxy/hooks/test_proxy_track_cost_callback.py \
                tests/test_litellm/proxy/auth/test_organization_budget_enforcement.py -q
35 passed

Mateo: to reproduce the original gap on a live proxy without waiting 12h, run the proxy with --workers 2 (no shared redis) and the budget suite's org case against a real model; pre-fix the org never blocks because organization.spend stays 0, post-fix it blocks once accumulated org spend crosses max_budget. I did not run the full live suite here since it needs the multi-worker proxy + real provider keys; happy to attach run logs if you want them before the deploy.

Type

🐛 Bug Fix

Slack Thread

Open in Web Open in Cursor 

Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
@mateo-berri
mateo-berri marked this pull request as ready for review June 19, 2026 01:44
@codecov

codecov Bot commented Jun 19, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@greptile-apps

greptile-apps Bot commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes a spend-attribution gap where keys created under a team (with no direct org_id) never wrote spend to the org's DB row, causing org budgets to fail enforcement across multi-worker deployments or after reservation-counter TTL expiry.

  • Adds _resolve_spend_tracking_org_id to fall back to the team's organization_id when the key's own org_id is None, mirroring the existing fallback already in auth_checks._organization_max_budget_check.
  • The fix is confined to the spend-tracking path and deliberately avoids mutating valid_token.org_id, leaving auth, rate limiting, and resource-ownership checks unaffected.
  • Six new unit tests cover all branches of the resolution helper, and one end-to-end behavioral test verifies the resolved org_id propagates through to update_database and increment_spend_counters.

Confidence Score: 5/5

Safe to merge — the change is narrowly scoped to the async spend-tracking path, uses the existing cached get_team_object helper, does not touch auth or access-control logic, and is guarded by a full exception handler that logs at warning level and falls back gracefully.

The fix is a straightforward parity change: spend tracking now resolves org the same way enforcement already does. The lazy team lookup is cache-backed, the error path is safe, existing tests are untouched, and six new tests including an end-to-end behavioral assertion cover all branches.

No files require special attention.

Important Files Changed

Filename Overview
litellm/proxy/hooks/proxy_track_cost_callback.py Adds _resolve_spend_tracking_org_id to fall back to the team's organization_id when the key carries no org_id, matching the existing enforcement-time fallback in _organization_max_budget_check.
tests/test_litellm/proxy/hooks/test_proxy_track_cost_callback.py Adds six new unit tests covering all branches of _resolve_spend_tracking_org_id and one end-to-end behavioral test verifying org_id propagates to update_database and increment_spend_counters. No existing tests modified.

Reviews (3): Last reviewed commit: "fix(proxy): log team-lookup failures in ..." | Re-trigger Greptile

Comment thread litellm/proxy/hooks/proxy_track_cost_callback.py Outdated
@CLAassistant

CLAassistant commented Jun 19, 2026

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ mateo-berri
❌ cursoragent
You have signed the CLA already but the status is still pending? Let us recheck it.

…rage

Add cases for the no-db-client early return and the swallowed team-lookup
failure in _resolve_spend_tracking_org_id so every new line in the patch is
exercised, satisfying codecov's 100% patch coverage gate.
@mateo-berri

Copy link
Copy Markdown
Contributor Author

@greptileai

…ng level

A team lookup failure during spend tracking silently dropped org attribution.
At debug level this gap is invisible in production, so an unenforced org budget
has no signal to investigate. Log at warning with the impact, and assert the
warning in the regression test so it can't quietly fall back to debug.
@mateo-berri

Copy link
Copy Markdown
Contributor Author

@greptileai

@mateo-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 51e7fd4. Configure here.

@mateo-berri

Copy link
Copy Markdown
Contributor Author

I think this might be a no-op. Closing for now pending more investigation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants