fix(access groups): sync assigned_team_ids from the team write paths

[yassin-berriai]

TLDR

Problem this solves:

• Clearing a team's access groups left it under Attached Teams
• The stale entry kept granting that group's models to the team's keys
• Only the access-group side ever wrote the team assignment

How it solves it:

• Team create, update and delete now mirror the change back
• The mirror reads the committed team row, so retries heal
• Set-based statements under a per-team lock, so concurrent writers converge

User Flow

Before: an admin who removes an access group from a team on the Teams page sees the team still attached to that group, and keys on the team keep the group's models

1. They open https://litellm-domain/ui/?page=access-groups, open the "premium" group, and add the "accounting" team under Attached Teams
2. They open https://litellm-domain/ui/?page=teams, edit "accounting", clear its Access Groups field and save
3. They go back to the "premium" group's page and Attached Teams still shows "accounting"
4. The same disagreement is visible over the API: GET https://litellm-domain/v1/access_group/{id} returns assigned_team_ids: ["<accounting id>"] while GET https://litellm-domain/team/info?team_id= returns access_group_ids: []
5. Deleting "accounting" outright does not clear it either, so the "premium" group lists a team that no longer exists
6. A key issued on "accounting" that carries the "premium" group can still call the models "premium" grants, even though the admin revoked the group from the team

After: the removal takes effect on both pages, and the surviving grant goes with it

1. They open https://litellm-domain/ui/?page=access-groups, open the "premium" group, and add the "accounting" team under Attached Teams
2. They open https://litellm-domain/ui/?page=teams, edit "accounting", clear its Access Groups field and save
3. They go back to the "premium" group's page and Attached Teams is now empty
4. GET https://litellm-domain/v1/access_group/{id} returns assigned_team_ids: [] and GET https://litellm-domain/team/info?team_id= returns access_group_ids: [], so the two agree
5. Re-adding "premium" to the team from the Teams page puts "accounting" back under Attached Teams, so the sync runs both ways, and deleting the team clears it from every group
6. A key issued on "accounting" that carries the "premium" group no longer receives the models "premium" grants once the admin has revoked the group from the team

Relevant issues

Linear ticket

Resolves LIT-4916

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live proxy on port 4916 against a dedicated Postgres 16, same script every leg, master key sk-1234. Both legs return 200 with real objects at every step, so the two runs differ only where the fix lands

Reproduce with, in order:

curl -s -X POST localhost:4916/team/new -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"team_alias":"accounting"}'
curl -s -X POST localhost:4916/v1/access_group -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"access_group_name":"premium","access_model_names":["fake-model"],"assigned_team_ids":["<TEAM_ID>"]}'
curl -s -X POST localhost:4916/team/update -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"team_id":"<TEAM_ID>","access_group_ids":[]}'
curl -s localhost:4916/v1/access_group/<AG_ID> -H 'Authorization: Bearer sk-1234' | jq -c .assigned_team_ids
curl -s "localhost:4916/team/info?team_id=<TEAM_ID>" -H 'Authorization: Bearer sk-1234' | jq -c .team_info.access_group_ids
curl -s -X POST localhost:4916/team/update -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"team_id":"<TEAM_ID>","access_group_ids":["<AG_ID>"]}'
curl -s -X POST localhost:4916/team/delete -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"team_ids":["<TEAM_ID>"]}'

BEFORE, captured at 262ed53, current litellm_internal_staging. grep -c sync_team_access_group_membership litellm/proxy/management_endpoints/team_endpoints.py returns 0 on that tree, so the leg really ran without the fix

### 3. both sides agree after attaching
access group assigned_team_ids: ["97b919b2-dc08-4e35-ad13-707649e02407"]
team access_group_ids:          ["a777ceca-3bd4-4d27-9917-c467436622c0"]

### 4. admin clears Access Groups on the team (Teams page -> Save)
{"team_id":"97b919b2-dc08-4e35-ad13-707649e02407","access_group_ids":null}

### 5. the two views after clearing
access group assigned_team_ids (Attached Teams): ["97b919b2-dc08-4e35-ad13-707649e02407"]
team access_group_ids (enforcement):            []

### 6. re-attach from the team side, both views again
access group assigned_team_ids (Attached Teams): ["97b919b2-dc08-4e35-ad13-707649e02407"]
team access_group_ids (enforcement):            ["a777ceca-3bd4-4d27-9917-c467436622c0"]

### 7. delete the team, then read the access group
access group assigned_team_ids (Attached Teams): ["97b919b2-dc08-4e35-ad13-707649e02407"]

On that same tree the concurrency script leaves the group with no attached teams at all, since nothing on the team side writes the mirror:

Attached Teams: []
team A access_group_ids: ["81ffe4e5-da00-462c-bf64-698f3c335b2d"]
team B access_group_ids: ["81ffe4e5-da00-462c-bf64-698f3c335b2d"]

AFTER, captured at 9bb0ed2, this PR's head

### 3. both sides agree after attaching
access group assigned_team_ids: ["81f2cf30-3693-49e5-ba4f-d16b0f4b187d"]
team access_group_ids:          ["f4b39ce3-4680-480e-bae2-a5c79cc34c40"]

### 4. admin clears Access Groups on the team (Teams page -> Save)
{"team_id":"81f2cf30-3693-49e5-ba4f-d16b0f4b187d","access_group_ids":null}

### 5. the two views after clearing
access group assigned_team_ids (Attached Teams): []
team access_group_ids (enforcement):            []

### 6. re-attach from the team side, both views again
access group assigned_team_ids (Attached Teams): ["81f2cf30-3693-49e5-ba4f-d16b0f4b187d"]
team access_group_ids (enforcement):            ["f4b39ce3-4680-480e-bae2-a5c79cc34c40"]

### 7. delete the team, then read the access group
access group assigned_team_ids (Attached Teams): []

CONCURRENCY, the review finding that changed this diff. Two teams attach the same access group at the same instant, so the group must end up listing both

curl -s -X POST localhost:4916/team/update -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"team_id":"<TEAM_A>","access_group_ids":["<AG_ID>"]}' &
curl -s -X POST localhost:4916/team/update -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"team_id":"<TEAM_B>","access_group_ids":["<AG_ID>"]}' &
wait
curl -s localhost:4916/v1/access_group/<AG_ID> -H 'Authorization: Bearer sk-1234' | jq -c .assigned_team_ids

At a6d02b3, this PR's first revision, which mirrored with a read-modify-write of the whole array. One team is lost on all three runs, while both teams' own rows say they are attached, so the group under-grants silently

=== RMW run 1 ===
Attached Teams: ["b41af4aa-989e-4d9a-bac6-3bddacf0cc75"]
team A access_group_ids: ["a8bc5879-c378-4941-b4db-8ada2411c52a"]
team B access_group_ids: ["a8bc5879-c378-4941-b4db-8ada2411c52a"]
=== RMW run 2 ===
Attached Teams: ["c541e715-aaf1-41bc-b2e8-32330f5f3a75"]
=== RMW run 3 ===
Attached Teams: ["2365651a-21f7-4e6d-9d94-15caf46a261f"]

At 9bb0ed2, this PR's head. Both teams survive on all three runs

=== run 1 ===
Attached Teams: ["70197048-81d7-4146-b8e3-87f78cda3522","b755f46b-bb79-4796-be68-565e5a00f7ae"]
=== run 2 ===
Attached Teams: ["55dfb75b-9379-42b3-a8ce-555dc5dcf5fe","615f8802-5080-488e-a47f-67040ab5cd82"]
=== run 3 ===
Attached Teams: ["a3fd4c7b-a210-4cda-aac2-319f0013098f","bd45acf7-2628-429c-bb89-c397f3f0d405"]

CREATE PATH, re-run at 5399f0d after the create insert and its mirror moved into one transaction, and after merging current staging. A team created with an access group already set shows up under Attached Teams, clearing it on update removes it, and deleting the other team removes that one too

curl -s -X POST localhost:4916/team/new -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"team_alias":"created-with-group","access_group_ids":["<AG_ID>"]}'
curl -s localhost:4916/v1/access_group/<AG_ID> -H 'Authorization: Bearer sk-1234' | jq -c .assigned_team_ids
curl -s -X POST localhost:4916/team/update -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"team_id":"<NEW_TEAM>","access_group_ids":[]}'
curl -s -X POST localhost:4916/team/delete -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"team_ids":["<TEAM>"]}'

--- team/new with the group set at create time ---
assigned_team_ids: ['ef8f5d12-6beb-4889-ab5d-cbeef0a1d286', 'f1a7b186-186d-41c3-a927-7c8ea629b3c0']
--- team/update clearing the groups ---
assigned_team_ids: ['ef8f5d12-6beb-4889-ab5d-cbeef0a1d286']
--- team/delete of the first team ---
assigned_team_ids: []

Type

🐛 Bug Fix

Caveats (if any)

• assigned_key_ids has the same hole via /key/update
• A cache drop that fails leaves that entry stale until a later write retries it
• A team-side and access-group-side edit can still interleave across the two
• A mirror failure after team delete waits for that id to be reused
• The detach scans the access group table, which is admin-sized
• Deriving attached teams instead would hide a live grant

Review notes

Four things worth knowing before reading the diff.

First, assigned_team_ids is not a display-only mirror. get_authorized_resources_from_key_access_groups in litellm/proxy/auth/auth_checks.py reads it directly to decide whether an access group carried on a key is authorized for that key's team, and there is a test pinning that behavior with the team's own access_group_ids left empty on purpose. So the stale row was granting models, MCP servers and agents that the admin had already revoked, which makes this a correctness fix on an authorization input rather than a UI cleanup, and it is why the sync invalidates the group's cache entry instead of only writing the row.

Second, the mirror takes no desired-state argument at all. It locks the team with pg_advisory_xact_lock and reads the team row as committed, inside the same transaction as its writes. Reconciling against the committed row rather than a before/after delta is what lets a retry heal a half-applied sync, because the team row commits first and a retry's delta is empty. Holding the lock across the read is what makes two writers on the same team converge, rather than the slower one replaying its own stale snapshot over the newer one. Worth noting that both properties made the diff smaller. The signature lost previous_access_group_ids and then access_group_ids too, so all three call sites now pass just a team id, and team deletion needs no special case: a team with no row reconciles to the empty set and detaches everywhere.

On why an advisory lock rather than folding the team write into the mirror transaction or taking SELECT ... FOR UPDATE on the team row, which is the more obvious reading of the review comment. The access-group endpoints already run a transaction that locks an access group and then a team. A mirror that locks the team and then access groups is the reverse order, and the two deadlock. I checked rather than reasoned about it, and Postgres aborts one side:

team_side: RawQueryError: ERROR: deadlock detected
DETAIL: Process 81 waits for ShareLock on transaction 998; blocked by process 82.
ag_side: committed

The advisory lock is never taken by the access-group endpoints, so it cannot join their lock order to form a cycle, and the same probe reports no deadlock with it in place. It serializes per team, which is the property actually needed.

Third, the mirror is set-based statements inside one transaction, not a per-group read-modify-write loop. Greptile flagged the loop as a lost-update risk and it was right, so the concurrency leg above reproduces it three times out of three against the earlier revision and shows it fixed. array_append and array_remove mutate the array inside the statement, so a concurrent write for another team cannot clobber it, the pair commits together, and the statement count no longer scales with how many group ids a caller sends. COALESCE is load-bearing here because the column is nullable in Postgres, and without it both guards evaluate to NULL and the row is silently skipped.

Fourth, the sync helper deliberately lives in management_helpers rather than in access_group_endpoints. That module is a lazily registered feature router, and its routes reach the schema through the committed snapshot that inject_lazy_stubs injects for features it believes are unloaded. That check is module_path in sys.modules, so importing the module eagerly from team_endpoints marks the feature loaded while its router was never included, and every one of its paths and schemas disappears. This is a live /openapi.json defect, not only a CI gate: a running proxy would serve a spec missing those routes until some request happened to warm the router, so Swagger and any generated client lose them. Counting access_group paths off app.openapi() in one interpreter shows it directly, 10 with the helper extracted and 6 with the eager import restored. The first revision of this PR had the eager import, which is what turned the schema check red.

On the one review point I did not act on. Team deletion commits before the mirror runs, and a retry of /team/delete is rejected at validation with a 404 before it reaches the mirror, so a mirror that failed after a successful delete leaves the team id behind on its groups. That is accurate, and I checked the 404 against a live proxy rather than reading for it.

I left it alone because the remedies are worse and the residual is narrower than it looks. Detaching before the delete means a delete that fails partway leaves a live team stripped of its access groups, which is an outage for a team still in use, traded against a stale row for a team that no longer exists. Putting the delete in the mirror's transaction means replacing delete_data with raw SQL inside a function that also deletes that team's keys, models and memberships. Neither is a good trade for this.

The residual also only bites if the team id is reused, since a stale id grants nothing while no team holds it, and team ids are caller-suppliable so reuse is possible. But reuse is exactly the moment POST /team/new runs the reconcile, which clears it. Seeding a dangling id directly in the database and then recreating the team with that id:

--- seeded a DANGLING id: the team does not exist ---
{"team_id":"reused-team-id-probe","access_group_ids":[]}
--- team recreated with the SAME id; does it inherit the grant? ---
{"assigned_team_ids":[]}

So the stale row is cleaned before anything can be issued against it. Worth saying plainly that this path is strictly better than before this PR either way: team deletion previously never cleaned the mirror at all, so what is left here is the old behavior surviving in a failure case rather than anything new.

The create path had a milder version of the same shape, and that one is now fixed rather than argued down. POST /team/new used to commit the team row and then mirror it, so a mirror that failed left a team whose groups never learned about it while the retry was rejected as a duplicate team_id. The insert and the reconcile now share one transaction, so the team row goes away with the failed mirror and the retry is a clean create. tests/proxy_admin_ui_tests/test_access_group_team_sync.py pins that against a real Postgres: raise inside the transaction after the reconcile and both the groups and the team table come back untouched.

Cache invalidation is now driven by desired state rather than by which rows the current attempt changed. The mirror reads the groups the team should be on plus the groups that currently list it, before its writes, and the caller drops all of them. That is what makes a failed drop retryable: the retry's UPDATEs match nothing because the database is already reconciled, and it still finds the same set to invalidate. Every drop is attempted even when one raises, so one unreachable cache cannot leave the rest of the groups serving a revoked grant, and the error is re-raised afterward so the caller sees the failure.

On the merge with current staging, #36819 reworked team delete to evict the team cache and sweep dangling references after the row delete. The mirror now runs last on that path, after both, so it still reconciles against a team that is gone and nothing fallible sits between the delete and the eviction that comment asks to keep adjacent.

On completeness of the writer set, team.access_group_ids is only ever written by POST /team/new, POST /team/update, and team delete. PATCH /team/{team_id} is not a fourth writer: it builds an UpdateTeamRequest and calls update_team. Every one of those three now calls the same helper, so a reviewer can check completeness by grepping for writes to that column rather than trusting the list.

Two notes on the tests. The three unrelated-looking edits in test_team_endpoints.py are a harness artifact: those tests build the team row as MagicMock(spec=LiteLLM_TeamTable), and pydantic v2 model fields are not class attributes, so a spec'd mock does not carry them and reading access_group_ids off it raises AttributeError. Each edit is a single access_group_ids = None on the mock. Separately, the raw SQL needs a real database to mean anything, since a mocked prisma has to reimplement the array semantics in Python and then passes whatever the SQL says. I confirmed that by mutating the detach guard to AND TRUE and watching the unit test stay green. tests/proxy_admin_ui_tests/test_access_group_team_sync.py runs the statements against a real Postgres and dropping either NOT (... = ANY(...)) guard fails it 4/4.

Be aware that this file is a local harness rather than a CI gate today. tests/proxy_admin_ui_tests/ is only referenced by .circleci/config.yml, and CircleCI is no longer connected to this repository, so no job on a pull request runs that directory. No PR workflow provides a Postgres service at all. Run it by hand with a database up:

DATABASE_URL=postgresql://... python -m prisma db push --schema litellm/proxy/schema.prisma --skip-generate
cd tests/proxy_admin_ui_tests && DATABASE_URL=postgresql://... python -m pytest test_access_group_team_sync.py -q

Giving that directory a home in GitHub Actions is worth doing, and it would light up five other files that have not run since the migration, but it is CI infrastructure work that does not belong in a bug fix.

The proxy-endpoints red on this branch was a shared test-isolation bug, not this diff. tests/test_litellm/conftest.py has an autouse fixture that requests monkeypatch, so its undo runs after every other finalizer, and a test that monkeypatches litellm.proxy.proxy_server.prisma_client while a module-level patch() of the same global is live gets that mock reinstalled permanently for the rest of the xdist worker, so later tests in the shard await a MagicMock. That fix has since landed on staging in #36872, so this branch carries staging's version of it and no isolation change of its own

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/2113173845894b95ab154df03e2d7637
Requested by: @yassin-berriai

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

The PR adds team-to-access-group reverse-membership reconciliation using transactional, set-based PostgreSQL updates and cache invalidation.

• Wires reconciliation into team creation, update, and deletion.
• Serializes same-team reconciliation with a transaction-scoped advisory lock.
• Adds focused unit and real-PostgreSQL coverage for membership and concurrency behavior.

Confidence Score: 4/5

The PR is not yet safe to merge because a synchronization failure during team creation can leave a committed, incorrectly mirrored team that the creation request cannot retry.

Team creation persists before reconciliation, while duplicate-team validation prevents a retry with the same explicit id from reaching that reconciliation, leaving reverse membership and resulting resource authorization stale until a separate update occurs.

Files Needing Attention: litellm/proxy/management_endpoints/team_endpoints.py

Important Files Changed

Filename	Overview
litellm/proxy/management_helpers/access_group_team_sync.py	Adds atomic, idempotent reverse-membership reconciliation with same-team serialization and post-commit cache invalidation.
litellm/proxy/management_endpoints/team_endpoints.py	Wires reconciliation into all team write paths, but creation can commit before a synchronization failure and reject the corresponding retry.
litellm/proxy/management_endpoints/access_group_endpoints.py	Reuses the extracted access-group cache invalidation helper without changing endpoint behavior.
tests/proxy_admin_ui_tests/test_access_group_team_sync.py	Adds real-PostgreSQL coverage for SQL semantics, idempotency, deletion, null arrays, and same-team serialization.
tests/test_litellm/proxy/management_endpoints/test_team_endpoints.py	Adds wiring and authorization-consequence coverage while adapting existing mocks for the newly read field.

_{Reviews (5): Last reviewed commit: "fix(access groups): sync assigned_team_i..." | Re-trigger Greptile}

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 3 · PR risk: 0/10

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_lit4916_access_group_team_sync (c0a58eb) with litellm_internal_staging (7a519e2)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (c278455) during the generation of this report, so 7a519e2 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[yassin-berriai]

@greptileai please re-review at d82571c. The mirror is now two set-based statements in one transaction, replacing the read-modify-write loop both P1s flagged.

[yassin-berriai]

@greptileai re-review at 2030899. Same-team race closed with a per-team advisory lock; the mirror now reads the committed team row.

[yassin-berriai]

@greptileai re-review at 9bb0ed2. Rebased onto current staging; same-team race closed with a per-team advisory lock over a committed-row read.

[yassin-berriai]

Delete-then-mirror point is accurate but left as-is: both remedies are worse trades. Reasoning and the self-heal evidence are under Review notes.

[yassin-berriai]

@greptileai re-review 9bb0ed2. The delete-path finding is addressed under Review notes with live evidence that the stale row self-heals on team id reuse.

[greptile-apps]

Creation failure cannot reconcile

When access-group synchronization fails after new_team commits the team row, retrying the request with the same explicit team_id stops at duplicate-team validation before synchronization runs. The team remains created with stale reverse membership, causing its keys to miss the access-group resources until a separate team update reconciles it.

Knowledge Base Used:

• Management Endpoints
• Proxy Authentication and Authorization

[devin-ai-integration]

Verified all six legs through the Admin UI at 5399f0d, including the core clear-groups case. Merged current staging; both review findings addressed.