fix(ci): let the mutation workflow find covered lines so it generates mutants

[yuneng-berri]

TLDR

Problem this solves:

• The mutation workflow reports nothing, and has for months
• It burns 26 minutes collecting coverage, then finds 0 mutants
• A run that overruns its budget publishes no report at all

How it solves it:

• Point mutation runs at a coverage config that stores absolute paths
• Give the mutmut step a deadline inside the job's own
• Upload the per-mutant .meta files, which are what a partial run leaves behind
• Stop one test from deleting mutmut's own sentinel variable

User Flow

Before: someone who wants a mutation score for the management endpoints clicks the workflow and gets nothing back

1. They open https://github.com/BerriAI/litellm/actions/workflows/mutation-test.yml and click "Run workflow"
2. The job spends about 26 minutes running the 2854 tests in scope
3. It ends on "Stopping early, because we could not find any test case for any mutant"
4. The run page summary reads "mutmut produced no results, so the report cannot say anything about the suite", and the uploaded report is 221 characters
5. They learn nothing about the suite, and the last four dispatches all ended this way

After: mutants now exist to be tested, so a scope whose tests all pass comes back with a real score and a list of survivors

1. They narrow the scope to a module whose tests pass in one process, then click "Run workflow"
2. The job runs the same tests, then reports mutants generated and starts testing them
3. The run page summary shows how many mutants were killed and how many survived
4. The uploaded mutation-report.md names each surviving mutant with its diff, the function it lives in, and the test files that were supposed to catch it
5. If the run outlasts its budget, the summary and report still cover every mutant it finished rather than being cancelled with nothing

One test in that folder cleared the whole process environment, which took mutmut's MUTANT_UNDER_TEST with it, so the next call through a mutation trampoline raised KeyError. Under -x that ended the run before any mutant was tested. The test now keeps every variable it does not care about, so the click above no longer stops there. See the cases under Proof of Fix.

Relevant issues

Linear ticket

Pre-Submission checklist

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

The end user here is whoever dispatches the workflow, so the proof is a mutation run: the same scope, scored before and after. Reproduced locally against one module so the whole loop fits in seconds, plus the real CI dispatch on each side.

Shared setup, run from the repo root:

UV_PROJECT_ENVIRONMENT=.venv-mutmut uv sync --frozen --group ci --group proxy-dev --extra google --extra proxy --extra semantic-router --extra saml
UV_PROJECT_ENVIRONMENT=.venv-mutmut uv run --no-sync prisma generate --schema litellm/proxy/schema.prisma
rm -rf .venv-mutmut/lib/python3.*/site-packages/pytest_retry*

Scope narrowed to one module in [tool.mutmut] so a full loop takes seconds instead of hours:

paths_to_mutate = ["litellm/proxy/management_endpoints/sso_helper_utils.py"]
tests_dir = ["tests/local_testing/test_ui_sso_helper_utils.py"]

A second harness builds mutmut's mutants/ sandbox by hand, so the whole folder can be run inside it in ten minutes instead of waiting out a CI dispatch. Mutating every line rather than only the covered ones is deliberate: it is a superset, so more functions carry a trampoline, not fewer.

SB=/tmp/mutmut-sandbox
rm -rf "$SB" && mkdir -p "$SB/mutants/tests" "$SB/harness"
cp -R litellm "$SB/mutants/litellm"
cp -R tests/test_litellm "$SB/mutants/tests/test_litellm"
cp pyproject.toml "$SB/mutants/"
uv pip install --python .venv/bin/python --target "$SB/mmpkgs" mutmut==3.5.0
rm -rf "$SB"/mmpkgs/_pytest "$SB"/mmpkgs/pytest "$SB"/mmpkgs/coverage "$SB"/mmpkgs/pluggy "$SB"/mmpkgs/packaging "$SB"/mmpkgs/iniconfig "$SB"/mmpkgs/typing_extensions.py "$SB"/mmpkgs/a1_coverage.pth
printf 'from mutmut.__main__ import ensure_config_loaded\nensure_config_loaded()\n' > "$SB/harness/mmconfig.py"
PYTHONPATH="$SB/mmpkgs" .venv/bin/python - <<'EOF'
from pathlib import Path
from mutmut.file_mutation import mutate_file_contents
out = Path("/tmp/mutmut-sandbox/mutants")
for f in sorted(Path("litellm/proxy/management_endpoints").rglob("*.py")):
    (out / f).write_text(mutate_file_contents(str(f), f.read_text(), None)[0])
EOF

Running the folder in it is what mutmut's stats phase does (mutmut/__main__.py:574): pytest from inside mutants/, with MUTANT_UNDER_TEST=stats.

REPO=$PWD
cd "$SB/mutants" && MUTANT_UNDER_TEST=stats PY_IGNORE_IMPORTMISMATCH=1 \
  PYTHONPATH="$SB/mutants:$SB/mmpkgs:$SB/harness" \
  "$REPO/.venv/bin/python" -m pytest --rootdir=. --tb=native -q \
  -p mmconfig -p no:retry -p no:rerunfailures -p no:xdist -p no:randomly \
  --ignore=tests/test_litellm/proxy/management_endpoints/test_saml_sso.py \
  tests/test_litellm/proxy/management_endpoints/

Before (ae63786cfb^, the merge base)

Local, one module

1. Run mutmut the way the workflow does today:

rm -rf mutants && mkdir -p mutants
UV_PROJECT_ENVIRONMENT=.venv-mutmut PYTHONPATH=$PWD/mutants \
  uv run --no-sync --with mutmut==3.5.0 mutmut run

2. Output:

    done in 17425ms (1 files mutated, 0 ignored, 0 unmodified)
2 passed in 0.06s
Stopping early, because we could not find any test case for any mutant. It seems that the selected tests do not cover any code that we mutated.

Zero mutants tested, exit code 1.

Local, the mutated folder in mutmut's sandbox

1. Run the folder through the sandbox harness above
2. Output:

1 failed, 2901 passed, 58 warnings in 573.94s (0:09:33)
FAILED tests/test_litellm/proxy/management_endpoints/test_ui_sso.py::TestCLIKeyRegenerationFlow::test_google_login_only_threads_user_code_when_enabled

3. Run that one test on its own to see what it actually hit:

  File ".../mutants/litellm/proxy/management_endpoints/ui_sso.py", line 29951, in google_login
    _get_cli_sso_flow_or_raise(login_id=key, cache=cli_sso_session_cache)
  File ".../mutants/litellm/proxy/management_endpoints/ui_sso.py", line 183, in _mutmut_trampoline
    mutant_under_test = os.environ['MUTANT_UNDER_TEST']
KeyError: 'MUTANT_UNDER_TEST'

The test cleared the whole environment, so mutmut's own sentinel went with it. google_login is decorated and mutmut never trampolines a decorated function, which is why the raise lands one frame further in. The bare except Exception: pass around the call then swallowed it, and the assertion read call_args on a mock that was never called, which is the 'NoneType' object has no attribute 'kwargs' everyone saw.

CI, full scope

1. The most recent dispatch before this PR: https://github.com/BerriAI/litellm/actions/runs/32481213728
2. Its log:

2854 passed, 58 warnings in 1569.11s (0:26:09)
Stopping early, because we could not find any test case for any mutant.
...
error: mutmut produced no results, so the report cannot say anything about the suite
Wrote /home/runner/work/litellm/litellm/mutation-report.md (0 survivors, 221 chars)

3. With only the coverage fix in place, mutants exist but the run still ends in the stats phase: https://github.com/BerriAI/litellm/actions/runs/32937470190

1 failed, 1167 passed, 26 warnings in 759.88s (0:12:39)
failed to collect stats. runner returned 1

4. -x names the first failure and stops, so a throwaway branch added --maxfail=200 to let the whole folder run and count every sandbox failure. mutmut appends pytest_add_cli_args after its own -x (mutmut/__main__.py:563) and a later --maxfail wins: https://github.com/BerriAI/litellm/actions/runs/32959717115

1 failed, 2901 passed, 56 warnings in 1697.46s (0:28:17)
FAILED tests/test_litellm/proxy/management_endpoints/test_ui_sso.py::TestCLIKeyRegenerationFlow::test_google_login_only_threads_user_code_when_enabled - AttributeError: 'NoneType' object has no attribute 'kwargs'

5. Same folder, same flags, outside the sandbox on the same runner image, as the control:

2902 passed, 56 warnings in 1597.40s (0:26:37)

One test, and it is the same one both locally and on CI. --maxfail is deliberately not shipped: the same args reach the per-mutant runs, where -x is what keeps them fast.

After (3fdeb79737)

Local, one module

1. Same command, with the workflow's new environment variable:

rm -rf mutants && mkdir -p mutants
UV_PROJECT_ENVIRONMENT=.venv-mutmut PYTHONPATH=$PWD/mutants \
  COVERAGE_RCFILE=$PWD/.github/mutmut-coverage.rc \
  uv run --no-sync --with mutmut==3.5.0 mutmut run

2. Output:

⠸ 8/8  🎉 8 🫥 0  ⏰ 0  🤔 0  🙁 0  🔇 0  🧙 0
40.87 mutations/second

8 mutants generated, all 8 killed. That module's tests hold up; the point is that the score exists at all.

Local, the mutated folder in mutmut's sandbox

1. Same harness, same command
2. Output:

2902 passed, 58 warnings in 580.31s (0:09:40)

Every test in the folder now runs where mutmut needs it to, and the count matches the outside-the-sandbox control above.

CI, full scope

1. Dispatched at this tip: https://github.com/BerriAI/litellm/actions/runs/32984754749
2. Still running as of this edit. Generation alone took 57 minutes last time, so the numbers go in here once it clears the stats phase.

Type

🚄 Infrastructure

Caveats (if any)

Medium

• A whole-folder run has still never finished, so its budget is unmeasured
  • Both known blockers are fixed, but the full run takes hours
  • Generation alone was 57 minutes across 48 files
• Seven other tests in that file still clear the whole environment
  • None of them calls through a trampoline while cleared, so they pass today
  • A helper moving out of a decorated endpoint would break them the same way

Low

• COVERAGE_RCFILE replaces the whole coverage config, not just one key
• That is fine here: mutmut passes source explicitly and writes no data file
• Local runs need pytest_retry removed from site-packages, same as CI does

[greptile-apps]

Greptile Summary

The PR repairs mutation-test coverage discovery and makes long-running mutation jobs retain useful partial output.

• Adds a mutation-specific coverage configuration that stores absolute file paths.
• Gives mutmut a deadline before the overall job timeout and uploads per-mutant metadata.
• Preserves mutmut’s environment sentinel while isolating SSO provider configuration in the affected test.
• Ignores local mutation-testing environments and generated sandboxes.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
.github/workflows/mutation-test.yml	Configures absolute-path coverage collection, a bounded mutmut step, and partial-result artifact uploads without an identified blocking failure.
.github/mutmut-coverage.rc	Overrides relative coverage paths so mutmut’s absolute-path covered-line lookup can discover mutants.
tests/test_litellm/proxy/management_endpoints/test_ui_sso.py	Retains unrelated environment variables, clears provider-selection variables, and strengthens failure and mock-call assertions.
.gitignore	Excludes the local mutation virtual environment and generated mutant sandbox.

_{Reviews (4): Last reviewed commit: "fix(test): keep the mutmut sentinel out ..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[yuneng-berri]

@greptileai the last review scored ab8134c; head is now ac1eb10, which corrects the comment naming where mutmut writes partial results.

[yuneng-berri]

@greptileai

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_/testing-strategy-audit-c39e33 (5bbbbff) with litellm_internal_staging (40423e6)}

[yuneng-berri]

@greptileai