fix(auth): load team membership once per request and skip prisma on an L1 hit

[devin-ai-integration]

TLDR

Problem this solves:

• Team-scoped keys look up team membership up to three times per request
• A cold burst for one member fans out into one DB read per request
• Routes that never read membership still pay for the lookup

How it solves it:

• Load team membership once per request and reuse it downstream
• Overlapping lookups for the same member share one in-flight read
• Skip the lookup on routes with no model and no budget check
• Cache writes stay synchronous, so invalidation cannot race a delayed write
• Invalidation also drops the in-flight load for that member, so a read that started before an admin update cannot write the old row back
• Invalidation waits for that dropped load to finish before it evicts the key, so a write that was already in flight lands first and gets evicted with everything else, and no stale task is left around to delete a fresh reload later

Replaces #41087, which carried unrelated diff from an older base. Same change, rebased onto main

User Flow

Before: a team whose key belongs to a user with no membership row sends a cold burst, and the operator sees one membership lookup per request land on their database

1. They send 50 concurrent POST http://localhost:20087/v1/chat/completions with a team-scoped key and "model": "gpt-5.4-nano"
2. All 50 come back HTTP 200 with a normal assistant message
3. The operator's Postgres query log shows 14 to 18 separate membership lookups for that one user during the burst

After: the same burst gets the same answers and the database sees a handful of membership lookups, not one per request

1. They send the same 50 concurrent POST http://localhost:20088/v1/chat/completions with the same key and model
2. All 50 come back HTTP 200 with a normal assistant message
3. The operator's Postgres query log shows 2 membership lookups for the burst (one per worker); the rest were served from memory or shared one in-flight read

Relevant issues

Affected release

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Shared setup: two proxies from two worktrees, each booted with --num_workers 2, each with its own Postgres database, its own Redis, and the same fixture. Before is the merge base on main, After is this PR's tip. Postgres has log_statement = 'all' so membership reads can be counted per burst. Every side restarts cold (proxy restarted, Redis flushed) before each burst, and each burst runs twice with the side order swapped so cold-start order does not favor one side. The two-process checks start a second --num_workers 2 proxy on the same database and Redis

model_list:
  - model_name: gpt-5.4-nano
    litellm_params:
      model: openai/gpt-5.4-nano
      api_key: os.environ/OPENAI_API_KEY
  - model_name: mock-model
    litellm_params:
      model: openai/mock-model
      api_key: not-used
      mock_response: "ok"
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
  coordination_redis:
    host: 127.0.0.1
    port: 16087   # 16088 for After
litellm_settings:
  enable_redis_auth_cache: true

python litellm/proxy/proxy_cli.py --config config.yaml --port $PORT --num_workers 2 --use_v2_migration_resolver

# fixture, run once per side against the master key
curl -X POST $B/team/new -d '{"team_id":"team-perf","models":["gpt-5.4-nano","mock-model"]}'
curl -X POST $B/team/member_add -d '{"team_id":"team-perf","member":[{"user_id":"member-perf","role":"user"},{"user_id":"member-restricted","role":"user"},{"user_id":"member-budget","role":"user"}],"max_budget_in_team":50}'
curl -X POST $B/team/member_update -d '{"team_id":"team-perf","user_id":"member-restricted","allowed_models":["mock-model"]}'
curl -X POST $B/team/member_update -d '{"team_id":"team-perf","user_id":"member-budget","max_budget_in_team":0.000001}'
# keys: member-perf (has a membership row), nonmember-perf (team key, no membership row), member-restricted, member-budget, solo (no team), team key with no user_id
curl -X POST $B/key/generate -d '{"team_id":"team-perf","user_id":"nonmember-perf"}'

# cold burst: restart both proxies, FLUSHALL both Redis, then 50 concurrent real calls
seq 1 50 | xargs -P 50 -I{} curl -s -o /dev/null -w "%{http_code} %{time_starttransfer}\n" \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -X POST http://127.0.0.1:$PORT/v1/chat/completions \
  -d '{"model":"gpt-5.4-nano","messages":[{"role":"user","content":"Say hi"}],"max_tokens":16}'
# then count the membership reads logged between two marker queries
grep -c 'SELECT "public"."LiteLLM_TeamMembership"' /var/log/postgresql/postgresql-14-main.log   # scoped to the burst window

Before (6e9e084)

Cold burst, team key whose user has no membership row

1. 50 concurrent POST /v1/chat/completions with gpt-5.4-nano, base first: codes={'200': 50}, ttfb_ms p50=1813.9 p95=2059.6
2. Membership reads in that window: SELECT TeamMembership=14
3. Same burst, base second: codes={'200': 50}, ttfb_ms p50=1772.5 p95=2043.2
4. Membership reads in that window: SELECT TeamMembership=18

Cold burst, team key whose user has a membership row

1. 50 concurrent POST /v1/chat/completions with gpt-5.4-nano, base first: codes={'200': 50}, ttfb_ms p50=1821.8 p95=2191.4
2. Same burst, base second: codes={'200': 50}, ttfb_ms p50=1917.4 p95=2243.8
3. Membership reads in those windows: 45 and 45, from the team member spend counter reseed that runs on both sides when Redis is empty; the auth prefetch already primes memory on main, so the auth path itself does not hit the DB here

Authorization, budgets, and invalidation

1. Member key, mock-model then gpt-5.4-nano: 200 ok, 200 Hi!
2. Member restricted to mock-model: mock-model 200, gpt-5.4-nano 403 Team member not allowed to access model, repeat 403
3. Member with max_budget_in_team 1e-6, already over budget from an earlier run, two real calls: 429 Budget has been exceeded! TeamMember=member-budget:team-perf Current cost: 1.16e-05, Max budget: 1e-06, 429
4. Key with no team: 200. Team key with no user_id: 200
5. POST /team/member_update allowing gpt-5.4-nano for the restricted member: next call 200. Revoke it again: next call 403, mock-model still 200
6. Nonmember key on gpt-5.4-nano before any row exists: 200. POST /team/member_add then /team/member_update with allowed_models=["mock-model"]: next call 403. POST /team/member_delete: next call still 403 (cached row lives until its TTL on both sides)

Two proxy processes on one database and one Redis

1. Second --num_workers 2 proxy started on :20187 from the same worktree, same database and Redis as :20087
2. Restricted member on gpt-5.4-nano through each process: 403, 403
3. POST /team/member_update on :20087 allowing gpt-5.4-nano, then the same call through :20187: 200, and through :20087: 200
4. POST /team/member_update on :20187 revoking it, then through :20087: 403, through :20187: 403, mock-model through :20087: 200
5. A never-seen team user, first call through :20087 then :20187: 200, 200

Redis stopped mid-run

1. redis-cli -p 16087 shutdown nosave, then member mock-model: 200, restricted gpt-5.4-nano: 403, restricted mock-model: 200
2. POST /team/member_update allow then revoke while Redis is down: 200 then 403
3. A never-seen team user, two calls while Redis is down: 200, 200
4. Redis restarted, member mock-model: 200

After (db8dfe9)

Cold burst, team key whose user has no membership row

1. 50 concurrent POST /v1/chat/completions with gpt-5.4-nano, head second: codes={'200': 50}, ttfb_ms p50=1778.2 p95=1946.7
2. Membership reads in that window: SELECT TeamMembership=2
3. Same burst, head first: codes={'200': 50}, ttfb_ms p50=1794.8 p95=2004.7
4. Membership reads in that window: SELECT TeamMembership=2

Cold burst, team key whose user has a membership row

1. 50 concurrent POST /v1/chat/completions with gpt-5.4-nano, head second: codes={'200': 50}, ttfb_ms p50=1511.9 p95=2008.7
2. Same burst, head first: codes={'200': 50}, ttfb_ms p50=1669.9 p95=1963.8
3. Membership reads in those windows: 22 and 44, the same spend counter reseed as Before (how many requests reach the reseed before the counter lands in Redis varies run to run); no auth path reads on either side

Authorization, budgets, and invalidation

1. Member key, mock-model then gpt-5.4-nano: 200 ok, 200 Hi!
2. Member restricted to mock-model: mock-model 200, gpt-5.4-nano 403 Team member not allowed to access model, repeat 403
3. Member with max_budget_in_team 1e-6, already over budget from an earlier run, two real calls: 429 Budget has been exceeded! TeamMember=member-budget:team-perf Current cost: 1.16e-05, Max budget: 1e-06, 429
4. Key with no team: 200. Team key with no user_id: 200
5. POST /team/member_update allowing gpt-5.4-nano for the restricted member: next call 200. Revoke it again: next call 403, mock-model still 200
6. Nonmember key on gpt-5.4-nano before any row exists: 200. POST /team/member_add then /team/member_update with allowed_models=["mock-model"]: next call 403. POST /team/member_delete: next call still 403 (same TTL behavior as Before)

Two proxy processes on one database and one Redis

1. Second --num_workers 2 proxy started on :20188 from the same worktree, same database and Redis as :20088
2. Restricted member on gpt-5.4-nano through each process: 403, 403
3. POST /team/member_update on :20088 allowing gpt-5.4-nano, then the same call through :20188: 200, and through :20088: 200
4. POST /team/member_update on :20188 revoking it, then through :20088: 403, through :20188: 403, mock-model through :20088: 200
5. A never-seen team user, first call through :20088 then :20188: 200, 200

Redis stopped mid-run

1. redis-cli -p 16088 shutdown nosave, then member mock-model: 200, restricted gpt-5.4-nano: 403, restricted mock-model: 200
2. POST /team/member_update allow then revoke while Redis is down: 200 then 403 (same as Before)
3. A never-seen team user, two calls while Redis is down: 200, 200
4. Redis restarted, member mock-model: 200

Observed on both sides, unchanged by this PR:

• An admin update reaches the worker that served it; a sibling worker keeps its in-memory row until TTL or the Redis-backed refresh, so which status the next call gets depends on which worker answers (earlier runs of the same script got 403 on the cross-process allow, and 403 on the Redis-down allow, on both sides)
• Deleted membership row keeps its cached allow list until TTL

Local gates on db8dfe9: make check PASS, uv run pytest tests/test_litellm/proxy/auth/test_auth_checks.py tests/test_litellm/proxy/auth/test_auth_object_prefetch.py tests/test_litellm/proxy/auth/test_model_access_group_budgets.py tests/test_litellm/proxy/auth/test_user_api_key_auth.py tests/test_litellm/proxy/common_utils/test_user_api_key_cache.py 574 passed. Two regression tests pin the invalidation race: test_get_team_membership_invalidation_waits_for_in_flight_load_then_evicts_it (invalidation lands during the DB read; asserts the key is empty afterwards, the next load fetches budget-new, and the load after that is served from cache with no third DB read) and test_get_team_membership_invalidation_during_cache_write_evicts_stale_entry (invalidation lands while async_set_cache is suspended). Removing the asyncio.wait on the dropped task fails both with budget-old left in the cache

Type

🐛 Bug Fix

Caveats (if any)

Medium

• Latency win is not measurable against current main on this rig
  • main already prefetches membership into memory for users with a row, so both sides answer from memory there
  • Cold bursts above land within run-to-run provider noise in both orders
  • The measured gain is DB reads: 14 to 18 down to 2 per 50-request cold burst for a user with no row

Low

• A failed membership lookup still reads as "no membership row", same as main; the failure is logged and not cached, so the next request retries
• A membership row deleted with /team/member_delete stays cached until TTL on both Before and After; pre-existing, not changed here
• With two uvicorn workers, a membership update invalidates the worker that served the admin call; a sibling worker keeps its in-memory row until TTL or the Redis-backed refresh, and while Redis is down it keeps it until TTL. Identical on main, not changed here
• Bugbot's run on ff02390 flagged that an in-flight load could survive invalidate_team_member_spend_state and write the pre-update row back; 4ac168b drops the in-flight entry on invalidation
• Greptile's runs on 4ac168b and ffeea30 flagged the two follow-on gaps: the write itself can suspend past the invalidation, and a stale task that cleans up after itself can delete a fresh reload. db8dfe9 drops both ownership checks in the loader and instead has invalidate_team_member_spend_state await the dropped task before it evicts, so the eviction always runs after the stale write and the loader never deletes anything. The admin call pays at most one membership read of extra latency, bounded by Prisma's 30s httpx timeout and the Redis socket_timeout, the same limits every request already has. Greptile's run on db8dfe9 raised that wait as unbounded; stood down in-thread on that basis and its re-review reports 5/5 with no outstanding findings
• Bugbot's run on db8dfe9 found no new issues
• Veria did not run on this PR; it reviewed the predecessor fix(auth): load team membership once per request and skip prisma on an L1 hit #41087 at f9d0d66, flagged only the background Redis write, and that code is gone from this branch
• CodeQL passes at db8dfe9; its one open "Cyclic import" thread points at the lazy from litellm.proxy.proxy_server import inside invalidate_team_member_spend_state, which exists on main unchanged and only moved down by the new in-flight wait above it. No import line is added or removed by this diff
• CLA: the CLA assistant lists yassin-berriai as unsigned; that is a click on the author's side, not a code change

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

• db8dfe9 passes /live-pr-risk (two-worker live run above recorded at this commit)

---

Note

Medium Risk
Touches proxy auth, per-member model access, and budget enforcement with new async caching/invalidation behavior; regressions are mitigated by extensive tests but correctness under concurrent invalidation is subtle.

Overview
Loads team membership once per auth request and passes the result through model-access, access-group stamping, and team-member budget checks instead of calling get_team_membership up to three times.

Reworks get_team_membership with an L1-only cache probe first, in-flight coalescing (parallel callers share one asyncio task via asyncio.shield), and split helpers for Redis/DB load and negative-cache handling. invalidate_team_member_spend_state now drops the in-flight entry, awaits that task before evicting cache keys so stale loads cannot repopulate membership after an admin update.

Skips the membership fetch when the route has no model and all budget checks are skipped (e.g. management routes like /key/info). Tests cover coalescing, invalidation races, single-load-per-request, lazy skip, and DB error retry behavior.

^{Reviewed by Cursor Bugbot for commit db8dfe9. Bugbot is set up for automated code reviews on this repo. Configure here.}

Link to Devin session: https://app.devin.ai/sessions/c394261a0cdf4a3fb5d4deab8ad5a2b5
Open in Devin Desktop: https://app.devin.ai/desktop/session/c394261a0cdf4a3fb5d4deab8ad5a2b5?variant=devin
Requested by: @yassin-berriai

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_team_membership_once_main (db8dfe9) with main (4123b4b)}

[greptile-apps]

Greptile Summary

This PR reduces repeated team-membership lookups during proxy authorization. It reuses membership data within each request, coalesces concurrent cache misses, preserves fail-open lookup behavior, skips unused lookups, and coordinates invalidation with in-flight cache writes. Regression tests cover coalescing, cancellation, errors, lazy loading, and invalidation races

Confidence Score: 5/5

The PR appears safe to merge, with no outstanding findings or newly introduced issues

All previous findings are resolved, including the cache invalidation race and stale-writer cleanup concerns, and no changes were made after the previous review

Important Files Changed

Filename	Overview
litellm/proxy/auth/auth_checks.py	Adds request-scoped membership reuse, L1 cache handling, concurrent-load coalescing, and invalidation coordination without a new unresolved issue
tests/proxy_behavior/auth/test_auth_object_prefetch.py	Extends prefetch coverage to confirm organization data remains available in the frozen cache
tests/test_litellm/proxy/auth/test_auth_checks.py	Adds focused regression coverage for membership coalescing, invalidation races, cancellation, error retries, and lazy loading
tests/test_litellm/proxy/auth/test_model_access_group_budgets.py	Updates budget test fixtures to represent missing memberships through the production negative-cache sentinel

_{Reviews (7): Last reviewed commit: "fix(auth): wait for the in-flight member..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 98.55072% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/auth/auth_checks.py	98.55%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ shivijain2323
❌ yassin-berriai
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[devin-ai-integration]

@greptileai

[devin-ai-integration]

@greptileai

[mateo-berri]

bugbot run

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[devin-ai-integration]

@greptileai

[mateo-berri]

bugbot run

[devin-ai-integration]

@greptileai

[devin-ai-integration]

@greptileai

[devin-ai-integration]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit db8dfe9. Configure here.}