Skip to content

ci: use --cov=./litellm so coverage paths resolve unambiguously in Codecov - #27960

Merged
shin-berri merged 4 commits into
litellm_internal_stagingfrom
litellm_/peaceful-jang-c0e43b
May 15, 2026
Merged

ci: use --cov=./litellm so coverage paths resolve unambiguously in Codecov#27960
shin-berri merged 4 commits into
litellm_internal_stagingfrom
litellm_/peaceful-jang-c0e43b

Conversation

@yuneng-berri

Copy link
Copy Markdown
Contributor

Summary

Switch --cov=litellm--cov=./litellm in every CI invocation. --cov=<module-name> makes pytest-cov treat the argument as a Python package and emit XML paths relative to the package root (proxy/proxy_server.py), which strips the litellm/ prefix. Codecov then drops any path whose basename is ambiguous in the repo. --cov=<./path> treats the argument as a directory and emits repo-relative paths (litellm/proxy/proxy_server.py), which Codecov resolves unambiguously.

Why it matters

11 of the top-18 highest-fix-rate files have never appeared in Codecov despite being exercised by tests with coverage instrumentation:

File Fix PRs (past 6 mo) Before After
litellm/proxy/proxy_server.py 133 missing resolved
litellm/router.py 69 missing resolved
litellm/proxy/utils.py 67 missing resolved
litellm/litellm_core_utils/litellm_logging.py 62 missing resolved
litellm/constants.py 61 missing resolved
litellm/proxy/management_endpoints/key_management_endpoints.py 60 missing resolved
litellm/utils.py 59 missing resolved
litellm/main.py 55 missing resolved
litellm/proxy/auth/user_api_key_auth.py 42 missing resolved
litellm/proxy/management_endpoints/team_endpoints.py 37 missing resolved
litellm/proxy/litellm_pre_call_utils.py 35 missing resolved

All have ambiguous basenames in the repo (e.g. proxy_server.py exists 3 times under enterprise/ and litellm/proxy/; router.py exists twice under litellm/ and litellm/types/; utils.py, main.py each have 20+ copies across subpackages).

Files like _types.py (unique basename) were already resolving correctly, so the gap was invisible from looking at any single file.

How the fix was found

  1. Downloaded a recent coverage.xml from a test-unit-proxy-db artifact and confirmed it contained filename="proxy/proxy_server.py" and filename="router.py" — paths stripped of the litellm/ prefix.
  2. First tried codecov.yaml: fixes: ["::litellm/"] to prepend the prefix server-side. It was applied after Codecov's auto-resolution, double-prefixing every previously-working file to litellm/litellm/... while still dropping the ambiguous ones. Reverted.
  3. Tested pytest-cov locally with multiple configs and confirmed that --cov=./litellm (path form, leading ./) makes coverage.py treat the argument as a directory and emit repo-relative paths.

Test plan

  • Local: rerun tests/proxy_unit_tests/test_proxy_server.py with --cov=./litellm; verified coverage.xml emits filename="litellm/proxy/proxy_server.py", litellm/router.py, and litellm/types/router.py as distinct entries.
  • Branch CI: pushed the change to this branch, which auto-triggers Unit Tests: Proxy DB Operations (its push: [main, "litellm_**"] trigger matches). Verified proxy_server.py and router.py appear under their proper litellm/... paths in Codecov for this branch's commit.
  • (After merge) Confirm the next main merge commit shows proxy_server.py + siblings in Codecov.

Scope

  • .github/workflows/_test-unit-base.yml (1)
  • .github/workflows/_test-unit-services-base.yml (2)
  • .github/workflows/test-mcp.yml (1)
  • .circleci/config.yml (14)

pytest-cov runs with --cov=litellm, which makes coverage.xml store paths
relative to the package root (e.g. `proxy/proxy_server.py` instead of
`litellm/proxy/proxy_server.py`). Codecov auto-resolves these only when
the basename is unique in the repo. Files like proxy_server.py, router.py,
utils.py, main.py, and constants.py — which have duplicates under
enterprise/ or other subpackages — get silently dropped during ingest.

The `fixes: ["::litellm/"]` rule prepends `litellm/` to every uploaded
path so they resolve unambiguously. Confirmed against multiple recent
coverage.xml artifacts that no uploader currently emits paths already
prefixed with `litellm/`, so the rule is safe to apply universally.

This restores Codecov visibility for the highest-fix-rate hotspots:
proxy_server.py, router.py, proxy/utils.py, litellm_logging.py,
constants.py, key_management_endpoints.py, utils.py, main.py,
user_api_key_auth.py, team_endpoints.py, and litellm_pre_call_utils.py.
This reverts commit e25a988.

The `fixes: ["::litellm/"]` rule turned out to be applied *after* Codecov's
auto-resolution, not before. Files with unique basenames (which were
auto-resolving correctly to `litellm/<path>`) got an extra `litellm/`
prepended, producing `litellm/litellm/<path>` storage. Files with
ambiguous basenames (the actual target of the fix) continued to be
dropped because the auto-resolution still failed for them.

Net result on the verification run: 1375 files now stored under
unresolvable `litellm/litellm/...` paths, and the 11 originally-missing
hotspots are still missing. Reverting before piling on further changes.
…decov

pytest-cov treats --cov=<module-name> as a Python package and emits XML
paths relative to the package root, stripping the litellm/ prefix
(`proxy/proxy_server.py` instead of `litellm/proxy/proxy_server.py`).
Codecov's auto-prefix heuristic then drops every file whose basename is
ambiguous in the repo — `proxy_server.py` (3 copies under enterprise/),
`router.py` (2 copies), `utils.py` (20+), `main.py` (20+), `constants.py`
(2). The 11 highest-fix-rate hotspots have never appeared in Codecov.

Switching to --cov=./litellm treats the argument as a path, which makes
coverage.xml emit repo-relative paths (`litellm/proxy/proxy_server.py`).
Each path is unambiguous, so Codecov resolves all files correctly.

Verified locally: rerunning a single proxy_unit_tests test with
--cov=./litellm produced `filename="litellm/proxy/proxy_server.py"`,
`filename="litellm/router.py"`, and `filename="litellm/types/router.py"`
as distinct entries — exactly the disambiguation Codecov needs.

Touches every workflow that uploads coverage: the two reusable GHA
workflows (_test-unit-base.yml, _test-unit-services-base.yml),
test-mcp.yml, and all 14 invocations in .circleci/config.yml.
@yuneng-berri
yuneng-berri requested a review from a team May 14, 2026 21:12
@greptile-apps

greptile-apps Bot commented May 14, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes Codecov path resolution for the litellm package by switching --cov=litellm (module form) to --cov=./litellm (path form) in all 18 CI invocations across 4 config files. The module form causes pytest-cov to strip the litellm/ directory prefix from emitted paths (e.g., proxy/proxy_server.py), making Codecov unable to resolve files whose basenames are ambiguous in the repo; the path form produces full repo-relative paths (e.g., litellm/proxy/proxy_server.py) that Codecov can match unambiguously.

  • .circleci/config.yml — 14 mechanical substitutions, one per CI job that uploads coverage; all updated consistently.
  • .github/workflows/_test-unit-base.yml / _test-unit-services-base.yml — 3 substitutions across the shared base workflows (both branches of the if/else in the services base are covered).
  • .github/workflows/test-mcp.yml — 1 substitution in the standalone MCP test step.

Confidence Score: 5/5

Pure CI tooling change with no production code modifications; all 18 occurrences are updated and no other workflow files contain the old form.

Every instance of --cov=litellm has been replaced with --cov=./litellm across all four affected files, with no missed occurrences in any other workflow. The change is mechanical and isolated to coverage reporting configuration; it does not affect test execution, build steps, or any runtime code. The PR description includes local verification that coverage.xml emits the expected repo-relative paths after the change.

No files require special attention.

Important Files Changed

Filename Overview
.circleci/config.yml 14 occurrences of --cov=litellm replaced with --cov=./litellm to emit repo-relative paths in coverage.xml; changes are mechanical and consistent across all jobs.
.github/workflows/_test-unit-base.yml Single --cov=litellm--cov=./litellm replacement; base workflow used by all non-service unit test jobs.
.github/workflows/_test-unit-services-base.yml Two --cov=litellm--cov=./litellm replacements (one per branch of the if/else that controls parallelism); both branches updated correctly.
.github/workflows/test-mcp.yml Single --cov=litellm--cov=./litellm replacement in the MCP test run step.

Reviews (1): Last reviewed commit: "Merge remote-tracking branch 'origin/lit..." | Re-trigger Greptile

@codecov

codecov Bot commented May 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@shin-berri
shin-berri merged commit fd92769 into litellm_internal_staging May 15, 2026
114 of 115 checks passed
@shin-berri
shin-berri deleted the litellm_/peaceful-jang-c0e43b branch May 15, 2026 16:08
fzowl pushed a commit to fzowl/litellm that referenced this pull request Jun 24, 2026
…c0e43b

ci: use --cov=./litellm so coverage paths resolve unambiguously in Codecov
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants