perf(ci): measure unit-shard coverage with the sys.monitoring core

[yuneng-berri]

TLDR

Problem this solves:

• Coverage costs more than the tests on most shards
• Shards run Python 3.12, where sysmon is not the default
• So they pay for the slow tracer on every run

How it solves it:

• Ask for the sys.monitoring core explicitly on the shared base
• One env var, same lines measured, same tests passing
• Cuts the pytest step 47% to 89%, measured on four shards
• Records why this and branch = true cannot both be on

User Flow

Before: a contributor waits on a required check that is spending much of its time in coverage bookkeeping

1. They push a backend change and watch the checks on their pull request
2. The slowest unit shard, proxy-infra / Run tests, takes about 15 minutes
3. The same directory under the legacy no-coverage workflow finished in roughly 5, so a large share of that wall clock is the tracer, not the tests
4. Nothing about the coverage number they get back is different for the wait

After: the same check measures the same lines through the cheapest core coverage.py ships

1. They push the same change
2. The shards run with COVERAGE_CORE=sysmon, so coverage.py drives PEP 669 sys.monitoring instead of the settrace tracer
3. Codecov receives the same line coverage from the same --cov=./litellm invocation
4. This pull request's own run against chore(ci): close the test-census blind spots and move scripts out of workflows/ #37586, which is the same code on the same day without the change, is the measurement; the numbers are in Proof of Fix below

Relevant issues

Linear ticket

Pre-Submission checklist

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Two things need proving: that asking for sysmon is not a no-op at this Python, and that it cannot be combined with branch coverage. Both are shown against the exact versions CI pins, Python 3.12 and coverage 7.14.0. The wall-clock effect is measured by this pull request's own shard run, since a laptop under other load cannot produce a number worth quoting.

Shared setup:

uv venv --python 3.12 .venv312
uv pip install --python .venv312/bin/python coverage==7.14.0
.venv312/bin/python -V

Python 3.12.13

Before (5290150)

coverage does not choose sysmon on its own at 3.12

1. Command:

.venv312/bin/python -c "from coverage import env; print('SYSMON_DEFAULT =', env.SYSMON_DEFAULT)"

2. Output, so without the env var the shards get the slow core:

SYSMON_DEFAULT = False

After (e4ffd65)

Asking for sysmon works, and is silent

1. Command:

COVERAGE_CORE=sysmon .venv312/bin/python -m coverage run --source=. run.py

2. Output, no warning, so the core was accepted:

pos nonpos

Asking for sysmon together with branch coverage is refused

1. Command:

COVERAGE_CORE=sysmon .venv312/bin/python -m coverage run --branch --source=. run.py

2. Output:

coverage/core.py:94: CoverageWarning: Can't use core=sysmon: sys.monitoring can't measure branches in this version, using default core (no-sysmon); see https://coverage.readthedocs.io/en/7.14.0/messages.html#warning-no-sysmon
  warn(f"Can't use core=sysmon: {reason_no_sysmon}, using default core", slug="no-sysmon")
pos nonpos

This is the open question answered: the two are mutually exclusive until the runners move to Python 3.14. Nothing in this change turns branch coverage on.

Wall clock, this run against #37586

#37586 is the same tree on the same day with no coverage-core change, so its shards are the baseline. Comparing whole-job durations would be misleading, since a job also pays for checkout, dependency install and Prisma generation, so the numbers below are pytest's own reported wall time from each job's log, and each row states the test counts on both sides to show the same work was done.

1. Command, per shard and per pull request:

jobid=$(gh pr checks <pr> | grep -F "<shard> / Run tests" | awk -F'\t' '{print $4}' | sed 's#.*/job/##')
gh api "repos/BerriAI/litellm/actions/jobs/$jobid/logs" | grep -E "=+ .*(passed|failed).* =+" | tail -1

2. Output:

budgets
  before  38 passed, 8 warnings in 432.37s (0:07:12)
  after   38 passed, 8 warnings in 47.68s

proxy-utils
  before  200 passed, 6 warnings in 429.17s (0:07:09)
  after   200 passed, 6 warnings in 48.73s

core-utils
  before  1692 passed, 1 skipped, 35 warnings in 209.43s (0:03:29)
  after   1692 passed, 1 skipped, 35 warnings in 88.58s (0:01:28)

All Other Providers
  before  7742 passed, 80 skipped, 162 warnings in 464.28s (0:07:44)
  after   7742 passed, 80 skipped, 163 warnings in 245.20s (0:04:05)

Same counts on both sides of every row, so this is the same work measured twice.

3. Step-level confirmation that the gain is in the tests and not in setup, for the budgets shard:

gh api "repos/BerriAI/litellm/actions/jobs/$jobid" --jq '.steps[] | "\(.name)\t\(.started_at)\t\(.completed_at)"'

before                        after
  150s  Install dependencies    168s  Install dependencies
   27s  Generate Prisma client   11s  Generate Prisma client
  659s  Run tests                 74s  Run tests
  860s  TOTAL                    281s  TOTAL

Setup did not get faster, it got slightly slower. The whole difference is the test step.

The size of the gain tracks how much work each test does rather than how many tests there are: budgets runs 38 tests in what was seven minutes, and it drops about 89%, while All Other Providers runs 7,742 fast tests and drops about 47%. Shards dominated by heavy execution under the tracer are the ones that were paying most for it.

Type

🚄 Infrastructure

Caveats

• Measured 47% to 89% off the pytest step, four shards
• Blocks branch = true on 3.12; that trade is now documented
• Same lines measured, so the coverage number should not move
• Test counts identical on both sides, so it is like for like

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[greptile-apps]

Greptile Summary

The PR configures unit-test coverage to use coverage.py’s sys.monitoring core on Python 3.12, reducing tracing overhead without changing the measured source scope.

• Adds COVERAGE_CORE=sysmon to the shared unit-test workflow.
• Documents its incompatibility with branch coverage on the current Python version.

Confidence Score: 5/5

The PR appears safe to merge with the current Python 3.12 and coverage 7.14.0 unit-shard configuration.

All callers use Python 3.12, both pytest branches collect coverage, and the active coverage configuration does not enable branch measurement or another option that would prevent use of the sysmon core.

Important Files Changed

Filename	Overview
.github/workflows/_test-unit-base.yml	Selects the sysmon coverage core for all unit shards; current Python, coverage version, commands, and coverage configuration are compatible.

_{Reviews (1): Last reviewed commit: "perf(ci): measure unit-shard coverage wi..." | Re-trigger Greptile}

[yuneng-berri]

Measured on CI rather than a laptop: pytest's own wall time on the budgets shard goes 432s to 48s with 38 passed on both sides, proxy-utils 429s to 49s with 200 passed. Setup did not change. Numbers for four shards are in the description.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[tin-berri]

Switches coverage measurement to sys.monitoring (PEP 669) instead of the settrace tracer via COVERAGE_CORE=sysmon — 1-line env var change, well-proven: confirmed sysmon isn't the 3.12 default, confirmed it's silently incompatible with branch coverage (documented, not silently broken), and measured a real 47-89% pytest-step speedup across 4 shards with identical test counts on both sides (isolated via step-level timing showing setup didn't change, only the test step did). Minimal, well-justified infra change. CI green.