fix(proxy): stop fail-closed 503 for team users without a membership row

[devin-ai-integration]

TLDR

Problem this solves:

• With fail_closed_budget_enforcement: true, team users without a membership row get 503 forever
• Happens whenever the team has a default member budget, even with Redis and the DB healthy

How it solves it:

• A confirmed missing membership row now counts as verified zero member spend
• Real read failures (and existing rows with unverifiable counters) still fail closed

User Flow

Before: a user on a team with a default member budget, who was never added as a member, cannot make any call

1. The admin runs the proxy with fail_closed_budget_enforcement: true and Redis
2. The admin creates a team with team_member_budget via POST http://localhost:4000/team/new and a key for the user on that team
3. The user sends POST http://localhost:4000/chat/completions with that key
4. Every request returns 503 "Budget enforcement unavailable: current spend could not be verified against Redis or the database..."

After: the same user gets a normal completion

1. The admin runs the proxy with fail_closed_budget_enforcement: true and Redis
2. The admin creates a team with team_member_budget via POST http://localhost:4000/team/new and a key for the user on that team
3. The user sends POST http://localhost:4000/chat/completions with that key
4. The request returns 200 with a real completion, and a member with a row who is over budget still gets 422 budget_exceeded

Linear ticket

Resolves LIT-8984

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live proxy with real Postgres and Redis, config:

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
  fail_closed_budget_enforcement: true
litellm_settings:
  enable_redis_auth_cache: true
  cache: true
  cache_params: {type: redis, host: 127.0.0.1, port: 6379}
model_list:
  - model_name: gpt-6.1-sol
    litellm_params: {model: openai/gpt-6.1-sol, api_key: os.environ/OPENAI_API_KEY}

Setup for each run: POST /team/new with team_member_budget: 100, POST /user/new with max_budget: 100, POST /key/generate with that team_id and user_id, no /team/member_add. A SQL check on the membership table returns no row for the pair

Request body:

{"model":"gpt-6.1-sol","messages":[{"role":"user","content":"Verify missing team membership budget behavior"}],"stream":false,"max_tokens":16,"cache":{"no-cache":true}}

Before (0fe4028)

User without a membership row

1. curl -s -w '\nHTTP_STATUS=%{http_code}' http://127.0.0.1:4000/chat/completions -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d "$BODY"
2. Output:

{"error":{"message":"{'error': 'Budget enforcement unavailable: current spend could not be verified against Redis or the database, and fail_closed_budget_enforcement is enabled, so the request was rejected to avoid exceeding the configured budget. Retry shortly.'}","type":"auth_error","param":"None","code":"503"}}
HTTP_STATUS=503

After (95585ba, the tip only adds test and CI files on top)

User without a membership row

1. Same curl with a fresh team, user and key set up the same way
2. Output:

HTTP_STATUS=200
{"id":"chatcmpl-ETWl5bbfq8q7wEkfYpdI3X3liOY9E","model":"gpt-6.1-sol","object":"chat.completion","choices":[{"finish_reason":"length","index":0,...}],"usage":{"completion_tokens":16,"prompt_tokens":12,"total_tokens":28,...}}

3. The membership table still has no row for the pair after the request

Member with a row over budget is still blocked

1. Add the user with a membership row, send one request (200), then lower the member budget to 0.000001
2. Same curl returns:

{"error":{"message":"Budget has been exceeded! TeamMember=<user>:<team> Current cost: 0.000184, Max budget: 1e-06","type":"budget_exceeded","param":null,"code":"422"}}

Type

🐛 Bug Fix
✅ Test

Caveats (if any)

Medium

• A user with no membership row is only capped by Redis counters
  • The spend writer does not create a row for users off the roster, so the DB holds no member spend for them
  • If Redis is down, their member budget reads as 0 spent instead of 503
• The new e2e test needs a fail-closed gateway, so it runs in its own opt-in workflow
  • E2E_FAIL_CLOSED_BUDGET_STACK gates it, and the shared e2e gateway stays fail-open

QA runbook

• tests/e2e/quota_management/budgets/test_team_member_budget_e2e.py::TestFailClosedTeamMemberBudgetWithoutMembership::test_missing_membership_counts_as_verified_zero_spend - a team user with a default member budget and no membership row is admitted under fail-closed budgets
  • Start the proxy with tests/e2e/gateway/fail_closed_team_member_budget_ci_config.yml (Postgres, Redis on 127.0.0.1:6379, Anthropic key for claude-haiku-4-5)
  • POST /team/new with team_member_budget: 100, POST /user/new, POST /key/generate with that team_id and user_id, and skip /team/member_add
  • POST /chat/completions with the key and model claude-haiku-4-5, expect 200 (503 "Budget enforcement unavailable" before this PR)
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/06700bf6371c45d68fe0dba6c6d20084
Open in Devin Desktop: https://app.devin.ai/desktop/session/06700bf6371c45d68fe0dba6c6d20084?variant=devin

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Confidence Score: 3/5

[Critical risk] Changes how team member budget enforcement handles missing database rows.

The PR is not safe to merge while fail-closed member-budget checks can admit requests on an unverifiable zero-spend fallback.

Findings

1.   Unverified spend bypasses member budget ▶
2.  Helper changes skip regression test ▶

Summary

The PR treats an absent team-membership row as verified zero fallback spend and adds unit tests plus a dedicated fail-closed E2E workflow.

• The new fallback permits budget checks to proceed when neither Redis nor the database can verify previously accrued member spend.
• The workflow’s path filter misses a helper required by its regression test.

_{Reviews (1) · Last reviewed commit: "test(proxy): isolate fail-closed team-me..."}

[greptile-apps]

Unverified spend bypasses member budget
With fail-closed enforcement enabled, a user without a membership row can accumulate spend in the Redis team-member counter. If Redis becomes unavailable, the database has no membership spend to recover, but this flag treats the zero fallback as verified. Requests can then continue past the member budget instead of receiving a 503. That violates the repository directive to prevent security incidents in the authentication layer.

How this was verified: The spend writer does not persist spend for a user absent from both the roster and membership table, while this flag suppresses the unverifiable-spend rejection.

Rule Used: What: Fail any PR which may contains a security incident on litellm's authentication layer Why: Do not cause security incidents Bad: ```python # Check cache first cache_key = ( f"oidc_userinfo_{token[:20]}" # Use fi... (source)

[greptile-apps]

Helper changes skip regression test
The new test uses budget_client.py to create a team with a member budget, but that file is missing from this workflow’s path filter. A PR changing only the helper will not run this regression test, so a broken team-creation request could go unnoticed by this lane.

Suggested change

	paths:
	- litellm/proxy/auth/auth_checks.py
	- tests/e2e/conftest.py
	- tests/e2e/pytest.ini
	- tests/e2e/gateway/fail_closed_team_member_budget_ci_config.yml
	- tests/e2e/quota_management/budgets/test_team_member_budget_e2e.py
	- .github/workflows/test-e2e-fail-closed-team-member-budget.yml
	paths:
	- litellm/proxy/auth/auth_checks.py
	- tests/e2e/conftest.py
	- tests/e2e/pytest.ini
	- tests/e2e/gateway/fail_closed_team_member_budget_ci_config.yml
	- tests/e2e/quota_management/budgets/budget_client.py
	- tests/e2e/quota_management/budgets/test_team_member_budget_e2e.py
	- .github/workflows/test-e2e-fail-closed-team-member-budget.yml

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_fail_closed_team_member_no_membership_503 (ab004e5) with main (f4a7c04)}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!