feat(ci): scope mutation testing to the pull request diff

[ryan-crabbe-berri]

TLDR

Problem this solves:

• Mutation testing only runs manually, over a whole folder
• A whole-folder run takes hours, so nobody triggers it
• Pull requests get no signal on their own test quality

How it solves it:

• Mutate only the functions the diff changed
• Run only the tests mirroring the changed files
• Report survivors in the job summary, advisory, not blocking

User Flow

Before: a contributor changes one line of production code and merges with no idea whether their tests would notice if that line were wrong

1. They edit a line in litellm/proxy/management_endpoints/common_utils.py and push the branch
2. They open a pull request and watch the checks; lint, types and unit tests all run
3. No check looks at whether any test actually distinguishes their line from a broken version of it
4. The only mutation job in the repo is on the Actions tab behind a manual "Run workflow" button, covering a whole folder over hundreds of minutes, so they skip it
5. They merge, and a test that asserts nothing about their line stays green forever

After: the same pull request gets a per-diff mutation result in under a minute of test time

1. They edit the same line and push the branch
2. They open a pull request and a new "Mutation Test (PR diff)" check starts
3. The check reports which functions it mutated, which tests it ran against them, and how many mutations those tests killed
4. When every mutation is killed, the summary says so and the check stays green
5. When one survives, the summary shows the surviving version of their function with the changed line marked, plus the tests that were supposed to catch it, so they can add the missing case before review

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Shared setup, applied on top of each hash below, standing in for a contributor's one-line change:

--- a/litellm/proxy/management_endpoints/common_utils.py
+++ b/litellm/proxy/management_endpoints/common_utils.py
@@ -135,7 +135,7 @@ def _is_user_team_admin(user_api_key_dict: UserAPIKeyAuth, team_obj: LiteLLM_Tea
     for member in team_obj.members_with_roles:
-        if (member.user_id is not None and member.user_id == user_api_key_dict.user_id) and member.role == "admin":
+        if (member.user_id is not None and member.user_id == user_api_key_dict.user_id) and member.role.lower() == "admin":

Before (282bcda)

1. Look for a mutation job that reacts to the pull request:

$ git show 282bcdad:.github/workflows/mutation-test.yml | sed -n '12,14p'
on:
  workflow_dispatch:

2. Nothing is triggered by pull_request, so pushing the change above produces no mutation result at all

3. Trigger the manual job by hand instead, and it mutates a whole folder on a 350 minute budget rather than the one changed function:

$ git show 282bcdad:pyproject.toml | sed -n '/tool.mutmut/,/^tests_dir/p' | grep -v '^#'
[tool.mutmut]
paths_to_mutate = [
    "litellm/proxy/management_endpoints/",
]
tests_dir = [

$ git show 282bcdad:.github/workflows/mutation-test.yml | grep timeout-minutes
    timeout-minutes: 350

After (65df36c)

Case A: as written, the mirrored tests kill every mutation

1. Scope the run to the diff:

$ python scripts/mutation_diff_scope.py --base origin/litellm_internal_staging --write-pyproject
changed production files: 1
  litellm/proxy/management_endpoints/common_utils.py (1 lines)
test selection: 2
  tests/test_litellm/proxy/management_endpoints/test_common_utils.py
  tests/test_litellm/test_mutation_diff_scope.py
functions to mutate: 1
  litellm.proxy.management_endpoints.common_utils.x__is_user_team_admin__mutmut_*
rewrote [tool.mutmut] in pyproject.toml for this diff

2. Run mutmut against that one function:

$ mutmut run "$(cat mutmut-scope-globs.txt)"
Mutant results
--------------
🎉 litellm.proxy.management_endpoints.common_utils.x__is_user_team_admin__mutmut_1
🎉 litellm.proxy.management_endpoints.common_utils.x__is_user_team_admin__mutmut_10
🎉 litellm.proxy.management_endpoints.common_utils.x__is_user_team_admin__mutmut_2
🎉 litellm.proxy.management_endpoints.common_utils.x__is_user_team_admin__mutmut_3
🎉 litellm.proxy.management_endpoints.common_utils.x__is_user_team_admin__mutmut_4
🎉 litellm.proxy.management_endpoints.common_utils.x__is_user_team_admin__mutmut_5
🎉 litellm.proxy.management_endpoints.common_utils.x__is_user_team_admin__mutmut_6
🎉 litellm.proxy.management_endpoints.common_utils.x__is_user_team_admin__mutmut_7
🎉 litellm.proxy.management_endpoints.common_utils.x__is_user_team_admin__mutmut_8
🎉 litellm.proxy.management_endpoints.common_utils.x__is_user_team_admin__mutmut_9

real	0m34.079s

3. Render the report that the job posts to its summary:

$ python scripts/mutation_report.py && head -9 mutation-report.md
Wrote mutation-report.md (0 survivors, 190 chars)
# Mutation Test Report

## Summary

- Total mutants: **10**
- Killed: **10**
- Survived: **0**
- Mutation score: **100.0%**

Case B: same change, after deleting the existing test_is_user_team_admin_user_not_in_team

1. Scope the run to the diff, unchanged from Case A:

$ python scripts/mutation_diff_scope.py --base origin/litellm_internal_staging --write-pyproject
changed production files: 1
  litellm/proxy/management_endpoints/common_utils.py (1 lines)
test selection: 2
  tests/test_litellm/proxy/management_endpoints/test_common_utils.py
  tests/test_litellm/test_mutation_diff_scope.py
functions to mutate: 1
  litellm.proxy.management_endpoints.common_utils.x__is_user_team_admin__mutmut_*
rewrote [tool.mutmut] in pyproject.toml for this diff

2. Run mutmut again, and one mutation now survives:

$ mutmut run "$(cat mutmut-scope-globs.txt)"
Mutant results
--------------
🎉 litellm.proxy.management_endpoints.common_utils.x__is_user_team_admin__mutmut_1
🎉 litellm.proxy.management_endpoints.common_utils.x__is_user_team_admin__mutmut_10
🙁 litellm.proxy.management_endpoints.common_utils.x__is_user_team_admin__mutmut_2
🎉 litellm.proxy.management_endpoints.common_utils.x__is_user_team_admin__mutmut_3
🎉 litellm.proxy.management_endpoints.common_utils.x__is_user_team_admin__mutmut_4
🎉 litellm.proxy.management_endpoints.common_utils.x__is_user_team_admin__mutmut_5
🎉 litellm.proxy.management_endpoints.common_utils.x__is_user_team_admin__mutmut_6
🎉 litellm.proxy.management_endpoints.common_utils.x__is_user_team_admin__mutmut_7
🎉 litellm.proxy.management_endpoints.common_utils.x__is_user_team_admin__mutmut_8
🎉 litellm.proxy.management_endpoints.common_utils.x__is_user_team_admin__mutmut_9

real	0m34.851s

3. The report scores only the mutants this run executed, and names the survivor:

$ python scripts/mutation_report.py && head -9 mutation-report.md
Wrote mutation-report.md (1 survivor, 48498 chars)
# Mutation Test Report

## Summary

- Total mutants: **10**
- Killed: **9**
- Survived: **1**
- Mutation score: **90.0%**

4. It shows the surviving version of the function with the changed line marked:

$ sed -n "/Mutated function/,/MUTANT END/p" mutation-report.md
Mutated function (the bug is delimited by `# MUTANT START` / `# MUTANT END`):

def _is_user_team_admin(user_api_key_dict: UserAPIKeyAuth, team_obj: LiteLLM_TeamTable) -> bool:
    for member in team_obj.members_with_roles:
        # MUTANT START
        if (member.user_id is not None or member.user_id == user_api_key_dict.user_id) and member.role.lower() == "admin":
        # MUTANT END
            return True

    return False

Case C: a class method, and a run that checks nothing

1. Same flow against a method instead of a module-level function, here a one-line edit inside CoroutineChecker.is_async_callable:

$ python scripts/mutation_diff_scope.py --base HEAD --write-pyproject
changed production files: 1
  litellm/litellm_core_utils/coroutine_checker.py (1 lines)
test selection: 1
  tests/test_litellm/litellm_core_utils/test_coroutine_checker.py
functions to mutate: 1
  litellm.litellm_core_utils.coroutine_checker.xǁCoroutineCheckerǁis_async_callable__mutmut_*
rewrote [tool.mutmut] in pyproject.toml for this diff

$ mutmut run "$(cat mutmut-scope-globs.txt)" && python scripts/mutation_report.py && head -9 mutation-report.md
# Mutation Test Report

## Summary

- Total mutants: **27**
- Killed: **21**
- Survived: **6**
- Mutation score: **77.8%**

2. The survivors resolve to the method, not to the module-level function of the same name, and each one renders with the marked line:

$ grep -n "^## \`litellm" mutation-report.md
## `litellm.litellm_core_utils.coroutine_checker.CoroutineChecker.is_async_callable`

$ sed -n '/#### Mutation 1 of 6/,/MUTANT END/p' mutation-report.md | tail -8
    def is_async_callable(self, callback: Any) -> bool:
        # Fast path: check cache first (most common case)
        try:
            # MUTANT START
            cached: Final = None
            # MUTANT END

3. If the mutation run itself breaks, the report refuses to render a clean sweep and the job goes red:

$ MUTMUT_CMD=true python scripts/mutation_report.py; echo "exit=$?"
error: mutmut reported no checked mutants; the run did not complete
exit=1

$ sed -n '8p' mutation-report.md
**The mutation run produced no results at all. Treat this as a failed run, not as a passing one: nothing was executed to survive.**

Case D: a run that stops partway through

mutmut catches an interrupt and still exits 0, so the job cannot read completeness off its exit code. What the run was asked for is the contract, so the report compares the requested globs against what came back. Executing 3 of the 27 requested mutants leaves exactly the state a killed run leaves:

$ cat mutmut-scope-globs.txt
litellm.litellm_core_utils.coroutine_checker.xǁCoroutineCheckerǁis_async_callable__mutmut_*

$ mutmut run ...is_async_callable__mutmut_1 ...__mutmut_2 ...__mutmut_3 > /dev/null; echo "mutmut exit code: $?"
mutmut exit code: 0

$ python scripts/mutation_report.py; echo "exit=$?"
error: 24 in-scope mutants were never checked; the run did not complete (first: litellm.litellm_core_utils.coroutine_checker.xǁCoroutineCheckerǁis_async_callable__mutmut_4)
exit=1

$ head -12 mutation-report.md | tail -4
- Mutation score: **0.0%**
- Never checked: **24** of the mutants this run asked for

**The run stopped early: 24 in-scope mutants were never checked, so the score above covers less than the diff does. Treat this as a failed run.**

Case E: deletions at a function boundary

Deleted lines are charged to the line above them, so a deletion anywhere inside a function, including its last statement, keeps that function in scope. Deleting a whole function leaves nothing to mutate, and the neighbour that moves up into its place is not charged for it:

$ git diff -U0 base -- litellm/mod.py | tail -2
@@ -4 +3,0 @@ def first(x: int) -> int:
-    return total
$ python scripts/mutation_diff_scope.py --base base
functions to mutate: 1
  litellm.mod.x_first__mutmut_*

$ git diff -U0 base -- litellm/mod.py | tail -7
@@ -1,6 +0,0 @@
-def first(x: int) -> int:
-    total = x + 1
-    total = total * 2
-    return total
-
-
$ python scripts/mutation_diff_scope.py --base base
functions to mutate: 0

Case F: three real open pull requests

Every number so far comes from a one-function diff, so the gate was also pointed at the head of three unrelated open pull requests, base origin/litellm_internal_staging, same machine, one at a time:

Pull request	changed files	functions	mutants	mutmut wall clock	killed	survived
#37740, redis auth	1	5	238	14s	142	96
#37734, partial-stream spend	2	3	164	166s	98	66
#37742, semantic cache	7	10	694	180s	255	439

Nine of those seconds are fixed, measured by asking for a single mutant: building the mutants/ sandbox and running the selected tests once against clean code. The rest divides into 0.02s per mutant where the selection is one self-contained test file and about a second per mutant where it reaches into the proxy import graph, so wall clock follows the tests a diff selects much more closely than it follows the diff's size. This machine runs the gate about three times faster than the CI runner did on the same ten-mutant scope, 34s against 102s

The exercise paid for itself twice. Both #37740 and #37742 first came back selecting an entire test directory, because litellm/_redis.py looks for test__redis*.py while the file is test_redis.py, and a miss fell through to the mirror directory, which for a module at the top of litellm/ is the whole unit suite. On #37742 that dragged in an unrelated order-dependent s3 test that fails on unmutated code, and mutmut refuses to trust a red baseline, so the run aborted before a single mutant ran. Touching an overloaded signature also asked for the same function once per @overload stub. Both are fixed, and the table above is the run after the fix

In CI (both paths, on this pull request's own checks)

1. Hot path: the same one-line change, pushed as a throwaway commit (1cc9746d18) purely to trigger the job, then dropped from the branch. Run 32405681200 is green, and its mutmut-scope.log artifact shows CI picked the same single function the local run did:

changed production files: 1
  litellm/proxy/management_endpoints/common_utils.py (1 lines)
test selection: 2
  tests/test_litellm/proxy/management_endpoints/test_common_utils.py
  tests/test_litellm/test_mutation_diff_scope.py
functions to mutate: 1
  litellm.proxy.management_endpoints.common_utils.x__is_user_team_admin__mutmut_*
rewrote [tool.mutmut] in pyproject.toml for this diff

2. The mutation-report.md artifact from that run reproduces Case A, so the summary a reviewer sees in the job is the one produced locally:

# Mutation Test Report

## Summary

- Total mutants: **10**
- Killed: **10**
- Survived: **0**
- Mutation score: **100.0%**

3. Skip path: on the branch tip, where the diff touches only tooling and tests, run 32405523726 is green in 66 seconds, skips every heavy step and posts "nothing in scope" instead. A pull request that changes no production code pays checkout and nothing else

4. Cost of the hot run, every step, cold caches:

    2s  Set up job
   51s  Run actions/checkout
    1s  Set up Python
    1s  Scope the run to the diff
    4s  Set up uv
    8s  Cache uv dependencies
  154s  Install dependencies
    4s  Cache Prisma binaries
   28s  Generate Prisma client
   17s  Reinstall litellm non-editable (so mutants/ is not shadowed)
    0s  Remove pytest plugins that conflict with mutmut
  102s  Run mutmut on the changed functions
    2s  Generate mutation report
    2s  Upload mutation artifacts
  382s  total

Mutation itself is 102 of those 382 seconds, and everything around it is fixed cost the diff cannot change. The job clears 5 minutes only once the uv and Prisma caches are warm; cold it runs about 6m20s. Scale Case F by the 3x this runner costs against the machine those numbers came from and a real pull request adds roughly 1 to 9 minutes of mutation on top, so the heaviest of the three would sit near the 10 minute step timeout. continue-on-error keeps that off the critical path: a timeout reads as an unfinished run, which is the one thing this gate refuses to call clean

Type

🚄 Infrastructure
✅ Test

Caveats (if any)

• Surviving mutants are advisory; the job fails only when the run did not finish what it was asked to check
• Granularity is the enclosing function, not the single changed line
• --max-functions caps a big diff at 40 functions
• Runtime follows the tests a diff selects, not the diff's size, so a change under the proxy import graph costs far more per mutant than a self-contained one
• Fork pull requests get the summary, not a comment
• Setup (dependencies, Prisma) dominates the runtime, not mutation

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[greptile-apps]

Greptile Summary

The PR adds pull-request mutation testing scoped to changed production functions and their mirrored tests.

• Adds diff parsing, function-level mutant glob generation, and dynamic mutmut configuration.
• Adds reporting for surviving mutants and explicit detection of empty or incomplete runs.
• Adds a non-blocking GitHub Actions workflow and regression tests for scoping and reporting behavior.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
.github/workflows/mutation-test-pr.yml	Adds an advisory PR workflow that scopes mutation execution and propagates incomplete-run report failures.
scripts/mutation_diff_scope.py	Builds mutation scope from the merge-base diff and now handles the previously reported deletion-boundary cases.
scripts/mutation_report.py	Produces class-aware survivor reports and rejects empty or explicitly incomplete mutation runs.
tests/test_litellm/test_mutation_diff_scope.py	Covers function and method scoping, mirrored test selection, and deletion-only boundary behavior.
tests/test_litellm/test_mutation_report.py	Covers mutant-name parsing, class disambiguation, result summaries, and incomplete-run detection.

_{Reviews (5): Last reviewed commit: "fix(ci): drop deletions with no survivin..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[ryan-crabbe-berri]

@greptile re review

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_mutation_test_pr_gate (1cc9746) with litellm_internal_staging (d542c82)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (4873567) during the generation of this report, so d542c82 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[ryan-crabbe-berri]

@greptileai re review

[ryan-crabbe-berri]

@greptile re review

[ryan-crabbe-berri]

Verified: deleting a function's last statement keeps it in scope. Deleted functions have no code to mutate. Added both tests.

[ryan-crabbe-berri]

@greptileai re review