Skip to content

fix(proxy): make /team/member_delete's four cleanups atomic - #37959

Merged
yassin-berriai merged 1 commit into
litellm_internal_stagingfrom
litellm_lit5541_team_member_delete_atomic
Aug 22, 2026
Merged

fix(proxy): make /team/member_delete's four cleanups atomic#37959
yassin-berriai merged 1 commit into
litellm_internal_stagingfrom
litellm_lit5541_team_member_delete_atomic

Conversation

@yassin-berriai

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • /team/member_delete writes the team roster, the user's teams list, the team membership row, and the verification token as four separate statements
  • A failure between any two writes leaves the removal half applied: the roster can show a member gone while their key and membership row still work

How it solves it:

  • Wraps all four writes in one prisma transaction so they commit or roll back together, following the same tx.<table> pattern /team/member_add and /team/member_update already use

User Flow

Before: an admin removing a member from a team can leave that member with full API access if any of the four database writes /team/member_delete performs fails partway through

  1. Admin sends POST https://litellm-domain/team/member_delete with {"team_id": "team-1", "user_id": "user-1"}
  2. A transient database failure partway through the cleanup returns a 500, but the team roster and the user's team list were already updated to drop the member before the failure hit
  3. Admin opens https://litellm-domain/ui/?page=teams and sees user-1 no longer listed on the team
  4. user-1's API key for that team is still active in the database, so they can keep sending requests to /v1/chat/completions and get billed against the team's budget as if nothing happened

After: the same transient failure now leaves the member fully in place instead of half removed

  1. Admin sends the same POST https://litellm-domain/team/member_delete
  2. The same transient failure still returns a 500, but the roster, the user's team list, the membership row, and the key are all left untouched
  3. Admin opens https://litellm-domain/ui/?page=teams and still sees user-1 listed, matching the fact that nothing was actually removed
  4. user-1's key still works, same as before the failed call. Once the admin retries after the transient issue clears, the request returns 200 and the roster, team list, membership row, and key are all removed together, so user-1 can no longer authenticate with that key

Relevant issues

Linear ticket

Resolves LIT-5541

Pre-Submission checklist

  • I have added meaningful tests
  • The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
  • My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Setup shared by both runs: a fresh Postgres 16 container, schema pushed with prisma db push, proxy started with DISABLE_SCHEMA_UPDATE=true and no other flags, master key sk-lit5541-test. A team is seeded with one extra member who has a per-member budget (so a membership row exists) and an active key:

curl -X POST http://localhost:24541/team/new -H "Authorization: Bearer sk-lit5541-test" \
  -d '{"team_id": "lit5541-team", "team_alias": "lit5541"}'
curl -X POST http://localhost:24541/team/member_add -H "Authorization: Bearer sk-lit5541-test" \
  -d '{"team_id": "lit5541-team", "member": {"role": "user", "user_id": "lit5541-user"}}'
curl -X POST http://localhost:24541/team/member_update -H "Authorization: Bearer sk-lit5541-test" \
  -d '{"team_id": "lit5541-team", "user_id": "lit5541-user", "max_budget_in_team": 100}'
curl -X POST http://localhost:24541/key/generate -H "Authorization: Bearer sk-lit5541-test" \
  -d '{"team_id": "lit5541-team", "user_id": "lit5541-user"}'

A failure between writes is forced with a Postgres trigger that raises on the third write:

CREATE OR REPLACE FUNCTION lit5541_block_membership_delete() RETURNS trigger AS $$
BEGIN
  RAISE EXCEPTION 'lit5541 injected failure: membership delete blocked';
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER lit5541_block_delete
BEFORE DELETE ON "LiteLLM_TeamMembership"
FOR EACH ROW EXECUTE FUNCTION lit5541_block_membership_delete();

Before (490c9f9)

  1. With the trigger installed, call the endpoint:
    curl -X POST http://localhost:24541/team/member_delete -H "Authorization: Bearer sk-lit5541-test" \
      -d '{"team_id": "lit5541-team-before", "user_id": "lit5541-user-a"}'
    
    Response: {"error":{"message":"Internal server error","type":"internal_server_error"}}, HTTP 500
  2. Query the four rows the request touches:
    SELECT members_with_roles FROM "LiteLLM_TeamTable" WHERE team_id='lit5541-team-before';
    SELECT teams FROM "LiteLLM_UserTable" WHERE user_id='lit5541-user-a';
    SELECT team_id, user_id FROM "LiteLLM_TeamMembership" WHERE team_id='lit5541-team-before';
    SELECT user_id, team_id FROM "LiteLLM_VerificationToken" WHERE team_id='lit5541-team-before';
    
    Result: the roster no longer contains lit5541-user-a and LiteLLM_UserTable.teams is now {}, both committed, while the membership row and the verification token are both still present, since the delete on them is what the trigger blocked. The member reads as removed while their key and membership row still work

After (9c5f874)

  1. Same seed, same trigger, same call against the team_id lit5541-team-after / lit5541-user-b:
    curl -X POST http://localhost:24541/team/member_delete -H "Authorization: Bearer sk-lit5541-test" \
      -d '{"team_id": "lit5541-team-after", "user_id": "lit5541-user-b"}'
    
    Response: {"error":{"message":"Internal server error","type":"internal_server_error"}}, HTTP 500
  2. Same four queries against lit5541-team-after / lit5541-user-b: the roster still lists lit5541-user-b, LiteLLM_UserTable.teams still contains the team, the membership row still exists, and the verification token still exists. Nothing committed
  3. Drop the trigger and retry the same request:
    DROP TRIGGER lit5541_block_delete ON "LiteLLM_TeamMembership";
    curl -X POST http://localhost:24541/team/member_delete -H "Authorization: Bearer sk-lit5541-test" \
      -d '{"team_id": "lit5541-team-after", "user_id": "lit5541-user-b"}'
    
    Response: 200, with members_with_roles down to just the team admin
  4. Re-run the same four queries: the roster no longer lists lit5541-user-b, LiteLLM_UserTable.teams is {}, the membership row is gone, the verification token is gone, and LiteLLM_DeletedVerificationToken now carries its audit record. All four writes landed together

Type

🐛 Bug Fix

Caveats (if any)

  • Locking concurrent /team/member_delete calls against each other is a separate change, tracked in LIT-5544

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

The team roster update, the user.teams update, the team membership
delete, and the team-scoped verification token delete ran as four
sequential writes with no transaction around them, so a failure
between any two left the removal half applied. Thread a single
prisma transaction through all four writes, following the same
tx.<table> pattern /team/member_add and /team/member_update already
use, so either all four land or none do.
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@yassin-berriai

Copy link
Copy Markdown
Contributor Author

@greptileai

@yassin-berriai

Copy link
Copy Markdown
Contributor Author

@greptileai please review commit 9c5f874

@greptile-apps

greptile-apps Bot commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR makes /team/member_delete atomically update the team roster, user team list, membership rows, verification-token audit records, and live tokens.

  • Adds transaction-aware deleted-token audit persistence.
  • Moves all member-removal writes onto one Prisma transaction.
  • Adds and updates mocked tests to verify transaction rollback behavior and existing cleanup paths.

Confidence Score: 5/5

The PR appears safe to merge because the affected cleanup writes and deleted-token audit insertion consistently use the same Prisma transaction.

The transaction interface matches established management-endpoint patterns, failures roll back the grouped database mutations, and no new reachable correctness or security failure remains.

Important Files Changed

Filename Overview
litellm/proxy/management_endpoints/team_endpoints.py Moves the member-removal writes into one transaction and emits the membership metric only after a successful commit; no changed-code defect was established.
litellm/proxy/management_endpoints/key_management_endpoints.py Adds an optional transaction client for deleted verification-token audit inserts while preserving existing non-transactional callers.
tests/test_litellm/proxy/management_endpoints/test_team_endpoints.py Wires transaction table mocks into existing deletion tests and adds coverage for failure between transactional writes.

Reviews (1): Last reviewed commit: "fix(proxy): make /team/member_delete's f..." | Re-trigger Greptile

@codecov

codecov Bot commented Aug 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@codspeed-hq

codspeed-hq Bot commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing litellm_lit5541_team_member_delete_atomic (9c5f874) with litellm_internal_staging (490c9f9)

Open in CodSpeed

@yassin-berriai
yassin-berriai merged commit 7ed91df into litellm_internal_staging Aug 22, 2026
71 checks passed
@yassin-berriai
yassin-berriai deleted the litellm_lit5541_team_member_delete_atomic branch August 22, 2026 21:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants