fix(scim): propagate team roster write failures on group and user writes

[devin-ai-integration]

TLDR

Problem this solves:

• SCIM team roster writes were swallowed, returning 200
• IdP saw green pushes, so it never retried
• Aborting on the first bad write strands the rest

How it solves it:

• Every roster add and remove in a push is attempted
• The writes that did not land are reported together, by name
• The user row is not saved when a roster write fails
• A unanimous 404 stays 404, a mixed batch reports 500

User Flow

Before: an admin pushing an Okta user whose group list still carries two stale groups gets a green push, and the user ends up claiming teams nobody put them in

1. The admin sends PUT https://litellm-domain/scim/v2/Users/okta-user with "groups" naming missing-team-a, engineering, and missing-team-b, where the first and last no longer exist
2. The response is 200 with a SCIM user body, so the IdP records the push as successful and never retries it
3. The admin opens https://litellm-domain/user/info?user_id=okta-user and the user claims ["missing-team-a", "engineering", "missing-team-b"]
4. The admin opens https://litellm-domain/team/info?team_id=engineering and the user is on that roster, but nothing anywhere backs the other two, so the user's own record disagrees with every team roster and the admin gets no signal that anything went wrong

After: the same push fails, names both stale groups, and still lands the group that was good

1. The admin sends the same PUT https://litellm-domain/scim/v2/Users/okta-user
2. The response is 404 whose message names both writes that failed, missing-team-a and missing-team-b, so the IdP records the push as failed and retries it
3. The admin opens https://litellm-domain/user/info?user_id=okta-user and the user claims ["engineering"] only, with no membership that no roster backs
4. The admin opens https://litellm-domain/team/info?team_id=engineering and the user is on that roster, because the good write still ran after an earlier one failed, so the retry only has to land the two groups the error named

Relevant issues

Linear ticket

Resolves LIT-5381

Pre-Submission checklist

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Both legs are one uninterrupted run against a live proxy on a pinned port 4700, backed by its own Postgres, swapping only scim_v2.py between them. Each leg prints the blob hash of the file it is serving next to the hash git holds for that commit, so the two halves are attributable:

$ git hash-object litellm/proxy/management_endpoints/scim/scim_v2.py     # before leg
e50e9bf05373139d1c4e4f105cd06ee3416e1ba1
$ git rev-parse cde134488c:litellm/proxy/management_endpoints/scim/scim_v2.py
e50e9bf05373139d1c4e4f105cd06ee3416e1ba1

$ git hash-object litellm/proxy/management_endpoints/scim/scim_v2.py     # after leg
bc935d09563ad0a3524ab49f22f457859b2e4a69
$ git rev-parse c3daf63a2e:litellm/proxy/management_endpoints/scim/scim_v2.py
bc935d09563ad0a3524ab49f22f457859b2e4a69

The push under test puts a group that exists between two that do not, so a roster write can only fail after another one has already landed. That is what separates "the whole push was rejected" from "the push kept going and told you exactly which writes it lost".

Before (cde1344, merge base)

1. Created the team and the user

$ curl -s -X POST http://localhost:4700/team/new -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"team_id":"scim-v3-mb-team","team_alias":"scim-v3-mb-team"}' -o /dev/null -w '%{http_code}\n'
200
$ curl -s -X POST http://localhost:4700/user/new -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"user_id":"scim-v3-mb-user","user_email":"scim-v3-mb-user@example.com"}' -o /dev/null -w '%{http_code}\n'
200

2. Sent the SCIM push and got a success, whose groups lists only the one team the roster actually received

$ curl -s -X PUT http://localhost:4700/scim/v2/Users/scim-v3-mb-user -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' \
    -d '{"schemas":["urn:ietf:params:scim:schemas:core:2.0:User"],"userName":"scim-v3-mb-user@example.com","name":{"familyName":"U","givenName":"A"},"emails":[{"value":"scim-v3-mb-user@example.com"}],"groups":[{"value":"missing-team-a"},{"value":"scim-v3-mb-team"},{"value":"missing-team-b"}]}' -w '\n%{http_code}\n'
{"schemas":["urn:ietf:params:scim:schemas:core:2.0:User"],"id":"scim-v3-mb-user","externalId":null,"meta":{"resourceType":"User","created":"2026-08-20T22:39:08.101000+00:00","lastModified":"2026-08-20T22:39:08.227000+00:00"},"userName":"scim-v3-mb-user@example.com","name":{"familyName":"U","givenName":"A","formatted":null,"middleName":null,"honorificPrefix":null,"honorificSuffix":null},"displayName":"scim-v3-mb-user@example.com","active":true,"emails":[{"value":"scim-v3-mb-user@example.com","type":null,"primary":true}],"groups":[{"value":"scim-v3-mb-team","display":"scim-v3-mb-team","type":"direct"}]}
200

3. Read the user back, and it claims all three

$ curl -s "http://localhost:4700/user/info?user_id=scim-v3-mb-user" -H 'Authorization: Bearer sk-1234' | jq -c '.user_info.teams'
["missing-team-a","scim-v3-mb-team","missing-team-b"]

4. Read the real team's roster back, which holds the user

$ curl -s "http://localhost:4700/team/info?team_id=scim-v3-mb-team" -H 'Authorization: Bearer sk-1234' | jq -c '[.team_info.members_with_roles[].user_id]'
["default_user_id","scim-v3-mb-user"]

so two of the three memberships exist only on the user record, and the IdP was told the whole push succeeded

After (c3daf63, PR tip)

1. Created the team and the user

$ curl -s -X POST http://localhost:4700/team/new -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"team_id":"scim-v3-head-team","team_alias":"scim-v3-head-team"}' -o /dev/null -w '%{http_code}\n'
200
$ curl -s -X POST http://localhost:4700/user/new -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"user_id":"scim-v3-head-user","user_email":"scim-v3-head-user@example.com"}' -o /dev/null -w '%{http_code}\n'
200

2. Sent the same SCIM push and got a failure naming both writes that did not land, so all three were attempted instead of the first failure ending the push

$ curl -s -X PUT http://localhost:4700/scim/v2/Users/scim-v3-head-user -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' \
    -d '{"schemas":["urn:ietf:params:scim:schemas:core:2.0:User"],"userName":"scim-v3-head-user@example.com","name":{"familyName":"U","givenName":"A"},"emails":[{"value":"scim-v3-head-user@example.com"}],"groups":[{"value":"missing-team-a"},{"value":"scim-v3-head-team"},{"value":"missing-team-b"}]}' -w '\n%{http_code}\n'
{"error":{"message":"SCIM roster sync failed on 2 of 3 team membership writes, leaving the roster partially updated. Retry the push to reconcile it. Failed writes: add scim-v3-head-user to missing-team-b: 404: {'error': \"Team doesn't exist in db. Team=missing-team-b. Create team via `/team/new` call.\"}; add scim-v3-head-user to missing-team-a: 404: {'error': \"Team doesn't exist in db. Team=missing-team-a. Create team via `/team/new` call.\"}","type":"internal_server_error","param":"None","code":"404"}}
404

3. Read the user back, and it claims only what a roster backs

$ curl -s "http://localhost:4700/user/info?user_id=scim-v3-head-user" -H 'Authorization: Bearer sk-1234' | jq -c '.user_info.teams'
["scim-v3-head-team"]

4. Read the real team's roster back, which still holds the user

$ curl -s "http://localhost:4700/team/info?team_id=scim-v3-head-team" -H 'Authorization: Bearer sk-1234' | jq -c '[.team_info.members_with_roles[].user_id]'
["default_user_id","scim-v3-head-user"]

so the good write still landed after an earlier one failed, and the two groups the error named are all the retry has to fix

Type

🐛 Bug Fix

Caveats

• Duplicate add and missing remove stay idempotent no-ops
• A unanimous failure status is preserved, a mixed batch reports 500
• Partial roster state is possible, named in the error, and retryable
• Failed writes are named in set order, so it varies between runs

Review notes

Greptile's P1 was that failing part-way through a reconciliation leaves the roster half written. That is real, and rollback is the wrong answer to it. The compensating delete can fail on its own, and it can strip a membership that pre-dated the push, so a tidy-looking undo can destroy state the IdP never asked to change

The fix is to stop aborting instead. Every add and remove in a push is attempted, and the ones that failed are reported together by name, which is what makes the leftover partial state safe to leave. SCIM reconciliation is idempotent by design: the IdP re-pushes the whole desired member set, so the writes that already landed replay as no-ops and only the named failures have to succeed

Set against what this replaces, a silently lost write is unrecoverable because nobody knows it happened, while a named partial failure converges on the next push. The live run above shows both halves of that: the response names missing-team-a and missing-team-b, and scim-v3-head-team still received its member even though a write ahead of it failed

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/043f92f2e3a041eeb796f63ad87c68c9
Requested by: @yassin-berriai

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

The PR replaces fail-fast SCIM roster reconciliation with collect-then-report behavior, allowing every requested membership write to run before an aggregate error is returned.

• Aggregates failed team additions and removals with their operation names and status codes.
• Prevents user-team arrays from being persisted after roster reconciliation fails.
• Adds coverage for user and group writes, partial failures, idempotent operations, and aggregate status selection.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the previously reported fail-fast reconciliation path now attempts every membership write and reports all failures while preventing inconsistent user-team data from being saved.

Important Files Changed

Filename	Overview
litellm/proxy/management_endpoints/scim/scim_v2.py	The previous fail-fast issue is resolved: all roster writes are attempted, nested failures are preserved, and aggregate errors propagate through the SCIM endpoints before inconsistent user-team data is saved.
tests/test_litellm/proxy/management_endpoints/scim/test_scim_v2_endpoints.py	Tests exercise every failure position, aggregate status handling, endpoint propagation, continued reconciliation after failures, and suppression of inconsistent user-row updates.

_{Reviews (2): Last reviewed commit: "fix(scim): propagate team roster write f..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 92.85714% with 4 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/management_endpoints/scim/scim_v2.py	92.85%	4 Missing ⚠️

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing devin/lit-5381-scim-group-and-user-writes-report-success-to-the-idp-when (832bdc5) with litellm_internal_staging (e07a712)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (bf59b7e) during the generation of this report, so e07a712 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[yassin-berriai]

@greptileai re-review c3daf63 please. The P1 fail-fast reconciliation is replaced by collect-then-report, so every roster write runs and failures are named

[tin-berri]

Auth/provisioning-critical (SCIM roster sync) so reviewed the diff directly. This is a genuine improvement over the abort-on-first-failure approach: every roster write in a push is now attempted before any failures are reported, so one bad write can't strand later good writes unattempted (the exact gap the previous fix left — tested directly via test_patch_team_membership_attempts_every_team_before_reporting and test_handle_group_membership_changes_attempts_every_member_and_names_failures, both confirming every write is called even after an earlier one fails). Status-code aggregation is correct and tested: a unanimous failure status is preserved (so a 404 an IdP shouldn't retry stays 404), while a mixed batch correctly falls back to 500. Idempotent no-ops (already-in-team, already-absent) are still correctly swallowed in _add_user_to_team/_remove_user_from_team rather than counted as failures — same logic, just relocated. Live proxy proof with content-addressed before/after file hashes for attribution. Small structural concern to flag, not a blocker: this touches the exact same functions (_handle_group_membership_changes, patch_team_membership, _handle_team_membership_changes) that #37688 (also open, not yet merged) modified with a different — now-superseded — abort-on-first-failure strategy; whichever of the two merges second will need a rebase to reconcile, worth a heads up to the author. CI green.

[devin-ai-integration]

@tin-berri thanks for the direct review. Correcting my first take on the overlap you flagged: #37688 is a different ticket (LIT-5105, group members silently dropped in larger groups, plus member-id resolution and dedupe), not a competing fix for LIT-5381, so the two are complementary in intent but do collide on _handle_group_membership_changes, patch_team_membership and _handle_team_membership_changes. Whichever merges second needs a real reconciliation rather than a mechanical rebase, and the surviving shape should be this collect-then-report one so LIT-5105's per-member failures are all attempted and named together. Happy to do that rebase on this branch if #37688 lands first.

Re-verified locally on the current head: make lint clean, and tests/test_litellm/proxy/management_endpoints/scim/ plus tests/test_litellm/proxy/auth/test_handle_jwt.py at 348 passed.