Skip to content

test(e2e): a member's team budget cuts off only that member's key - #33718

Merged
ryan-crabbe-berri merged 2 commits into
litellm_internal_stagingfrom
litellm_e2e_member_budget_isolation
Jul 18, 2026
Merged

test(e2e): a member's team budget cuts off only that member's key#33718
ryan-crabbe-berri merged 2 commits into
litellm_internal_stagingfrom
litellm_e2e_member_budget_isolation

Conversation

@ryan-crabbe-berri

@ryan-crabbe-berri ryan-crabbe-berri commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • My PR passes all CI/CD checks (e.g., lint, format, unit tests)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Captured at f83b515, before the branch was rebased onto litellm_internal_staging once #33632 squash-merged; the same assertions now sit at e9ab684 and the member case re-ran green after each revision against a live proxy (docker compose stack in tests/e2e) with real paid llama-3.3-70b-versatile calls through the claude-haiku-4-5 model group. A team with a roomy max_budget (100.0), members A and B (both users with roomy 100.0 budgets), A capped with max_budget_in_team 3e-06, B uncapped, one team-member key each (team_id + user_id):

--- member A key, call 1 ---
... "usage":{"completion_tokens":16,"prompt_tokens":39,"total_tokens":55, ...
HTTP 200
--- member A key, call 2 ---
{"error":{"message":"Budget has been exceeded! Current cost: 3.565e-05, Max budget: 3e-06","type":"budget_exceeded","param":null,"code":"429"}}
HTTP 429
--- member A key, call 3 ---
{"error":{"message":"Budget has been exceeded! Current cost: 3.565e-05, Max budget: 3e-06","type":"budget_exceeded","param":null,"code":"429"}}
HTTP 429
--- member B key, first call ---
... "usage":{"completion_tokens":16,"prompt_tokens":39,"total_tokens":55, ...
HTTP 200

With team and user budgets at 100.0 and the key uncapped, only A's member cap can block, and B keeps serving after A is cut off. All 6 enforcement cases pass against the same live proxy (6 passed in 14.82s) and basedpyright tests/e2e reports 0 errors

Type

✅ Test

Changes

The team-member budget case in tests/e2e/quota_management/budgets/test_budget_enforcement_e2e.py (covering quota_management.budget.team_member.blocks_over_limit) already proved the enforcement half: member A with a tiny max_budget_in_team under a roomy team gets blocked. What nothing asserted is the isolation half of the same customer story: when A burns through their member cap, their teammates must keep working. TeamMemberBudgetCase now adds an uncapped member B with their own team-member key; after A's key is refused, one call on B's key must return a successful completion, proving A's cap did not leak onto the team or its other members

The refusal shape is also pinned: A's block must be HTTP 429 budget_exceeded. Attribution to the member cap is structural rather than message-based: with the team and both user budgets at 100.0 and the key uncapped, no other cap can block. An earlier revision also matched the cap value in the error message, but that couples the test to the proxy's float formatting, so it was dropped

Targets litellm_internal_staging directly; the _assert_budget_blocks return value it builds on landed with #33632. The diff here is only the team-member case

QA runbook

  • tests/e2e/quota_management/budgets/test_budget_enforcement_e2e.py::test_budget_enforcement[TeamMemberBudgetCase] - a member's per-team budget cuts off that member's key while an uncapped teammate's key on the same team keeps serving
    • POST /team/new with {"team_alias":"qa-member-budget","max_budget":100.0} and save the team_id
    • POST /user/new with {"max_budget":100.0} twice and save user_a and user_b
    • POST /team/member_add with {"team_id":"<team_id>","member":{"role":"user","user_id":"<user_a>"},"max_budget_in_team":0.000003}, then again for user_b without max_budget_in_team
    • POST /key/generate with {"team_id":"<team_id>","user_id":"<user_a>"}, then the same for user_b
    • Send short chat completions with A's key a couple of seconds apart until one is refused and confirm a 429 whose body has type budget_exceeded
    • Send 1 chat completion with B's key and confirm 200
    • POST /key/delete, /user/delete, /team/delete to clean up

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

@greptile-apps

greptile-apps Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds isolation coverage to TeamMemberBudgetCase in the e2e budget enforcement suite. Previously the test only confirmed that a member with a tiny max_budget_in_team gets blocked; it never verified that the block stays isolated to that member and doesn't bleed onto the team or other members.

  • Adds an uncapped teammate B (user + key on the same team) alongside the already-present capped member A, and registers both for LIFO teardown in the correct order (keys before users, users before team).
  • Overrides run() to assert A's block is a 429, then makes a single call with B's key and requires it to succeed, structurally proving the member cap is isolated.
  • Attribution is structural rather than message-based: with the team and both user budgets at 100.0 and B's key uncapped, no cap other than A's max_budget_in_team can produce the block.

Confidence Score: 5/5

Test-only change that adds a well-scoped isolation assertion; no production code is touched and teardown ordering is correct.

The diff is confined to a single e2e test file and only adds a teammate B fixture plus a run() override. Teardown registration follows the same LIFO pattern used by every other case in the file (keys appended after users, so deleted first in reverse). Lambda closures over teammate_id are safe because the variable is never reassigned after the lambda is created. The _teammate_key attribute is set in init() before run() can reference it, consistent with the framework contract. No production paths are affected.

No files require special attention.

Important Files Changed

Filename Overview
tests/e2e/quota_management/budgets/test_budget_enforcement_e2e.py Extends TeamMemberBudgetCase with an uncapped teammate B and a run() method that verifies member A's block doesn't bleed onto B; teardown ordering, closure semantics, and structural attribution are all correct.

Reviews (3): Last reviewed commit: "test(e2e): drop the float-formatted cap ..." | Re-trigger Greptile

Comment thread tests/e2e/quota_management/budgets/test_budget_enforcement_e2e.py Outdated
@codecov

codecov Bot commented Jul 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Base automatically changed from litellm_e2e_bare_key_budget_429 to litellm_internal_staging July 17, 2026 18:29
@ryan-crabbe-berri

Copy link
Copy Markdown
Contributor Author

@greptileai re review

@ryan-crabbe-berri
ryan-crabbe-berri force-pushed the litellm_e2e_member_budget_isolation branch from 300c844 to e9ab684 Compare July 17, 2026 19:19
@ryan-crabbe-berri

Copy link
Copy Markdown
Contributor Author

@greptileai re review

@codspeed-hq

codspeed-hq Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing litellm_e2e_member_budget_isolation (e9ab684) with litellm_internal_staging (7015bd2)

Open in CodSpeed

@ryan-crabbe-berri
ryan-crabbe-berri merged commit 0e03795 into litellm_internal_staging Jul 18, 2026
78 of 80 checks passed
@ryan-crabbe-berri
ryan-crabbe-berri deleted the litellm_e2e_member_budget_isolation branch July 18, 2026 01:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants