ci: fail the build when a test file or Dockerfile is invoked by no job

[yassin-berriai]

TLDR

Problem this solves:

• A missing CI signal looks exactly like a passing one
• 275 test files are invoked by no job
• The root Dockerfile, our primary image, is built by none

How it solves it:

• Fail the build when a test file or Dockerfile is dark
• Build the root image and exercise its prisma bake
• Wire 8 dark tests/test_litellm/ subdirectories into shards

Relevant issues

Linear ticket

Resolves LIT-5249

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

The end user here is the next contributor who adds a test file or a Dockerfile, so the proof is the guard failing on real regressions and the new image leg actually passing. Captured at e4aa34b701.

The guard is green on this branch

python .github/scripts/assert_ci_coverage.py; echo "exit=$?"

OK: 2355 test files and 10 Dockerfiles are each invoked by at least one job or carry an explicit allowlist entry.
exit=0

It goes red on real regressions, and stays quiet otherwise

Each mutation was applied, measured, and reverted.

M1  new test file in an already-covered directory   exit 0   no false alarm
M2  new test file in a brand-new directory          exit 1   tests/mutation_probe_dir
M3  new file inside an ALLOWLISTED directory        exit 1   tests/litellm
M4  revert one of the shard entries added here      exit 1   tests/test_litellm/rerank_api
M5  new Dockerfile no job builds                    exit 1   probe_component/Dockerfile
M6  allowlist entry with an empty reason            exit 1   parse abort
M7  typo in a top-level allowlist key               exit 1   parse abort

M3 is what makes this a ratchet rather than a snapshot. Allowlist entries name individual files, never directories (the sole exception being tests/e2e), so adding a 34th file to tests/litellm fails instead of inheriting the exemption. The backlog can only shrink.

M6 and M7 keep the allowlist auditable:

ci-coverage-allowlist.yml: every 'dockerfiles' entry needs a non-empty 'paths' list of strings and a non-empty 'reason'
ci-coverage-allowlist.yml: unknown top-level key(s) ['dockerfile']; expected only ['description', 'dockerfiles', 'test_paths']

The new runtime-image leg was run locally before being written

docker build -f Dockerfile -t lit5249-root-image:latest .
LITELLM_IMAGE=lit5249-root-image:latest \
  pytest tests/proxy_migration_tests/test_offline_image_migration.py -v

tests/proxy_migration_tests/test_offline_image_migration.py::test_migration_offline_as_non_root_uid PASSED
tests/proxy_migration_tests/test_offline_image_migration.py::test_runtime_cache_env_not_read_only  PASSED
2 passed in 27.29s

The leg builds the image and then exercises the prisma bake as uid 12345 on a no-egress network; those prove different things, and a build-only leg would only have shown the image compiles.

Findings were cross-checked by hand, not taken from the tool

Every flagged directory was verified with plain grep -rF against .github/workflows/ and .circleci/config.yml, using controls with known non-zero hits so that "zero everywhere" could not pass for a working detector.

tests/test_litellm/rag                         hits=0
tests/test_litellm/realtime_api                hits=0
tests/litellm_core_utils                       hits=0
tests/vector_store_tests                       hits=0
tests/test_litellm/proxy/management_endpoints  hits=1   (control)
tests/proxy_migration_tests                    hits=4   (control)
tests/llm_translation                          hits=6   (control)

Type

🚄 Infrastructure

Changes

assert-shard-coverage in test-unit-proxy-db.yml already asserts that every file under tests/proxy_unit_tests/ is assigned to a shard, but its denominator stops there. Everything outside that one directory can be invisible with nothing noticing, which is how several independent holes accumulated.

.github/scripts/assert_ci_coverage.py widens the denominator to every test_*.py under tests/ (2355 files) and every Dockerfile* in the repo (10). The numerator is what the workflows and .circleci/config.yml actually invoke, which is deliberately narrower than what they mention. paths: and paths-ignore: keys are skipped, # comment lines inside run: blocks are stripped, and a path is only credited when its scalar carries a test runner (pytest, python <path>, circleci tests, helm unittest, playwright test) or sits under a test-path key. Crediting a mention would repeat, one level up, the exact mistake the guard exists to catch: an early version of this script counted basedpyright tests/e2e, a lint step running zero tests, as coverage of 178 files.

Glob matching uses shell semantics where * stops at / and ** does not. Python's fnmatch lets * cross /, which made the token tests/test_*.py match all 2355 files and turned the whole guard green. A coverage guard that passes is precisely the failure mode being fixed here, so that behaviour is worth naming.

That distinction also explains a real gap: tests/test_litellm/test_router was dark because the misc shard lists tests/test_litellm/test_*.py, and a shell glob of that shape matches files and never a directory of the same name.

Fixed here, out of what the guard found:

The root Dockerfile gets a runtime-image leg in image-scan.yml. It was built by no job at all, so a change that broke the build or the prisma bake only surfaced at release. Dockerfile is also added to that workflow's paths filter, since a leg that cannot be triggered by the file it guards is the same defect in a different costume.

Eight tests/test_litellm/ subdirectories join the shards that already enumerate their siblings: rag, realtime_api, rerank_api and test_router to misc; proxy/analytics_endpoints and proxy/memory to endpoints; proxy/enterprise_billing and proxy/types_utils to infra. Ten files, 125 tests, on shards that already run.

Everything else is listed in .github/ci-coverage-allowlist.yml with a written reason, file by file. tests/e2e is exempt as a directory because that suite runs against a deployed proxy from its own in-cluster rig. The remaining 87 files across nine directories are tracked in LIT-5254, and five unbuilt Dockerfiles are exempt with their reasons stated.

On cost, since the alternative was worth pricing rather than dodging: wiring all six unbuilt Dockerfiles into image-scan would add roughly 50 minutes to a workflow that currently costs about 18, at around nine minutes per image build. The root image is the one leg worth that on its own, and image-scan only runs when one of its paths entries changes. The guard job itself is a checkout plus one pinned pip install, well under a minute, and the shard additions are seconds.

The guard posts a check but is not in branch protection, so it does not block merges yet. Making it blocking is a deliberate follow-up once the allowlist has survived contact with a few PRs. continue-on-error was rejected because it would report green while failing, and leaving the script out of CI was rejected because it would ship a guard that is itself dark.

Two limits worth stating. The guard asserts that a test file is invoked by some job, not that the job's result is visible; CircleCI posts no check-runs on PRs in this repo (76 checks on #35832, zero from CircleCI), so a CircleCI-only suite can pass this guard while its outcome is unobservable, which is filed as LIT-5253. This PR demonstrated that live: it opened with migrations-image red on an inherited nodeenv regression, caught because that job runs on GitHub Actions, while the same test file's siblings run only under CircleCI and would have surfaced nothing a reviewer could see. The guard was green throughout, because the suite was invoked the whole time; only its visibility differed. And it does not know how a job exercises an image, so an image built and only ever run as root still passes, which is LIT-5239 and the reason to prefer image-scan over CircleCI for image coverage: its results are visible on the PR.

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Low Risk
Changes are CI guardrails, allowlist documentation, and shard/path wiring; no production runtime or auth logic is modified.

Overview
Adds a CI coverage ratchet so “dark” tests and Dockerfiles cannot slip in unnoticed: a new assert_ci_coverage.py job scans GitHub Actions and CircleCI for real pytest/build invocations (not bare paths: mentions), and fails unless every test_*.py and Dockerfile* is either referenced or documented in .github/ci-coverage-allowlist.yml with a required reason.

Wiring fixes from what the guard surfaced: eight previously unsharded tests/test_litellm/ trees are added to misc / proxy-endpoints / proxy-infra workflows (including test_router, which test_*.py globs do not cover as a directory). image-scan.yml now triggers on the root Dockerfile and adds a runtime-image job that builds it and runs test_offline_image_migration.py as a non-root uid, matching the prisma bake checks used for other images.

^{Reviewed by Cursor Bugbot for commit 0f3e55d. Bugbot is set up for automated code reviews on this repo. Configure here.}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

This PR adds a CI coverage guard that detects test files and Dockerfiles not invoked by any job, while explicitly tracking deferred coverage in an allowlist.

• Adds repository-wide test and Dockerfile invocation analysis.
• Builds the root runtime image and verifies offline migration behavior.
• Assigns previously uncovered test directories to existing unit-test shards.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
.github/scripts/assert_ci_coverage.py	Adds the coverage analyzer, allowlist validation, workflow token extraction, and shell-compatible path matching.
.github/ci-coverage-allowlist.yml	Records test files and Dockerfiles intentionally excluded from current CI coverage with explicit reasons.
.github/workflows/ci-coverage.yml	Introduces a dedicated job that runs the new CI coverage assertion.
.github/workflows/image-scan.yml	Adds a root-image build and offline non-root migration verification leg.
.github/workflows/test-unit-misc.yml	Adds four previously uncovered LiteLLM test directories to the miscellaneous unit-test shard.
.github/workflows/test-unit-proxy-endpoints.yml	Adds analytics and memory endpoint tests to the proxy endpoint shard.
.github/workflows/test-unit-proxy-infra.yml	Adds enterprise billing and proxy type utility tests to the infrastructure shard.

_{Reviews (3): Last reviewed commit: "ci: fail the build when a test file or D..." | Re-trigger Greptile}

[yassin-berriai]

Those version tags aren't prose comments; dependabot reads them to bump SHA pins. All 107 pinned uses lines in this repo carry one, zero don't.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[mateo-berri]

q: why is ci red here?

[yassin-berriai]

@greptileai please re-review at 0f3e55d.

The delta from the previous head e4aa34b is a rebase onto current litellm_internal_staging with no change to the diff; git diff --stat against the new base is the same 7 files and 501 insertions. The rebase picks up the nodeenv toolchain fix so the inherited migrations-image failure clears.

[yassin-berriai]

On "why is ci red here": it was migrations-image, and it was inherited from the base rather than caused by this PR. It is green now at 0f3e55d.

The failure was the nodeenv heal raising EACCES under a non-root uid, prisma_toolchain.py:124 in heal_incomplete_nodeenv_cache, which broke test_migration_offline_as_non_root_uid. That was a regression on staging, fixed by #35986. Two checks pinned it to the base rather than to this diff: the traceback landed on the exact line #35986 changes, and the same failure reproduced on an unrelated UI branch while passing only on the fix branch. One rules the cause in, the other rules this PR out.

#35986 merged, this branch was rebased onto a staging containing it, and migrations-image passed on the new head. Currently 73 green with no failures; the remaining few are coverage uploads still finishing.

Worth noting that the job ran here at all only because editing image-scan.yml is in that workflow's own paths filter. The failing test lives in tests/proxy_migration_tests, which otherwise runs under CircleCI, and CircleCI posts no check-runs to PRs, so that suite's result is normally invisible here. That is tracked separately as its own defect.

Thanks for the call on the action-version annotations. Agreed, and the reason they stay is that dependabot reads the trailing tag to know which version a SHA pin corresponds to when it bumps it, so removing them would degrade pin updates rather than just style. 107 SHA-pinned uses: lines in the repo carry one and zero do not.

[mateo-berri]

bugbot run

[mateo-berri]

@greptileai

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 0f3e55d. Configure here.}

[mateo-berri]

LGTM. Thanks!