test(e2e): add coverage registry and collector

[yuneng-berri]

Relevant issues

Part of the ongoing effort to quantify and track e2e test coverage per component and drive escaped regressions down. This PR lands the first concrete piece: the denominator and the tool that measures against it

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

This is test tooling rather than a proxy behavior change, so there is no endpoint to curl; the demonstration is the collector's own output. Running it against the current suite reports the real baseline coverage and flags markers that do not line up with the registry

$ cd tests/e2e && PYTHONPATH=. python -m coverage_registry.collector

MODULE                             COVERED    P0 COVERED
LLMs                                  0/96          0/58
MCPs                                  0/14           0/4
Management/UI                         0/66          0/19
Reliability & Performance             0/29          0/13
Logging & Guardrails                  1/51          1/20
Other                                 0/26          0/11
--------------------------------------------------------
ALL                                  1/282         1/125

Headline (P0 coverage): 1/125  (0.8%)

3 marker(s) point at ids not in the registry (reconcile: fix the marker or add the cell):
  llm.chat_completions.bedrock_converse.prompt_cache_5m.nonstream.cache_hit
  llm.chat_completions.openai.service_tier.works
  llm.chat_completions.provider.basic.nonstream.works

The tooling tests pass as well:

$ PYTHONPATH=. python -m pytest coverage_registry/test_collector.py -q
5 passed

Type

✅ Test

Changes

This adds tests/e2e/coverage_registry/, the set of e2e behaviors we want covered, one validated row per behavior. A cell is one customer-noticeable behavior a single test can assert pass/fail on, for example llm.chat_completions.bedrock_converse.tool_use.stream.works. There are 282 cells across the six tracking modules (125 of them P0), enumerated from the codebase and grouped module > feature > test per the grammar already documented in tests/e2e/CLAUDE.md

The rows live in per-prefix YAML files and validate against a pydantic discriminated union in schema.py, so a row cannot carry a field from another module and duplicate ids are rejected at load time. collector.py diffs the registry against the @pytest.mark.covers markers on the tests and reports coverage per module. It is static: a collect-only pass reads the markers, so it runs no test and needs no live proxy. It also lists markers that point at ids not in the registry, so a typo or an unenumerated behavior surfaces instead of being silently dropped. The covers marker is now registered suite-wide in the shared conftest so that collect-only pass works under --strict-markers

Please read this as a draft for review rather than a finished set. The cells were enumerated from the code and the tiers are a first proposal; the things worth settling before treating the denominator as final are written up in tests/e2e/coverage_registry/README.md (tier sign-off, a few cells that need a support check or prune, the auth double-coverage boundary, and the deliberately P0-weighted smoke cohorts)

---

Note

Low Risk
Test-only infrastructure under tests/e2e/; no proxy or runtime behavior changes.

Overview
Introduces a checked-in e2e coverage denominator under tests/e2e/coverage_registry/: ~282 validated cells (125 P0) across six dashboard modules, stored in per-prefix YAML and enforced by a Pydantic discriminated union in schema.py (duplicate ids fail at load).

Tests can declare coverage with @pytest.mark.covers("cell.id"), registered in tests/e2e/conftest.py for --strict-markers. collector.py runs a static pytest collect-only pass, diffs markers against the registry, and prints per-module totals with P0 as the headline; it also surfaces orphan markers and collection import failures. registry.py loads YAML; test_collector.py covers the math and a real-registry canary. README notes the cell set and tiers are still a draft for review.

^{Reviewed by Cursor Bugbot for commit 5ecc6e1. Bugbot is set up for automated code reviews on this repo. Configure here.}

[greptile-apps]

Greptile Summary

Adds tests/e2e/coverage_registry/, a static denominator for e2e test coverage: 282 registry cells across six modules (125 P0), validated against a Pydantic discriminated union, with a collector that diffs the registry against @pytest.mark.covers markers via a pytest collect-only pass and reports per-module coverage. The covers marker is also registered suite-wide in conftest.py so --strict-markers passes work.

• Registry (schema.py, registry.py, *.yaml): cells are per-YAML, validated on load, and duplicate ids are rejected; the Pydantic discriminated-union ensures an LLM cell cannot carry guardrail fields.
• Collector (collector.py): purely static — runs no live tests, needs no proxy; reports coverage and orphan markers (markers pointing at ids not in the registry).
• Tooling tests (test_collector.py): unit-tests the coverage math and a registry canary asserting > 250 unique ids.

Confidence Score: 4/5

Safe to merge — all changes are confined to the test tooling directory with no production code affected; the only shared file touched is conftest.py with an additive marker registration.

The new coverage tooling is well-structured and the tests pass. The three findings are all in the tooling itself: stderr leakage from the pytest subprocess can make CLI output noisy, the ROLLUP dict has no exhaustiveness guard against future module additions, and the cell id prefix is not validated against module so a miscategorised cell would silently skew coverage numbers for two modules.

tests/e2e/coverage_registry/collector.py (stderr not suppressed, ROLLUP KeyError risk) and tests/e2e/coverage_registry/registry.py (no id-prefix-vs-module validation).

Important Files Changed

Filename	Overview
tests/e2e/coverage_registry/collector.py	New CLI tool that runs a pytest collect-only pass and diffs the registry against @pytest.mark.covers markers; stdout is suppressed but stderr is not, so collection warnings can pollute terminal output.
tests/e2e/coverage_registry/schema.py	Pydantic discriminated union for registry cells; ROLLUP dict coupling with module literals is not exhaustiveness-verified and could KeyError if a new Cell subtype is added without updating ROLLUP.
tests/e2e/coverage_registry/registry.py	Loads and validates YAML cells via Pydantic; correctly rejects duplicate ids; does not validate that each cell's id prefix matches its module field.
tests/e2e/coverage_registry/test_collector.py	Unit tests for coverage math and registry loader; includes a canary test asserting > 250 cells and unique ids. All test logic is correct.
tests/e2e/conftest.py	Registers the covers marker suite-wide so collect-only passes work under --strict-markers; minimal change, no issues.

Comments Outside Diff (1)

1. tests/e2e/coverage_registry/registry.py, line 678-686 (link)

   Cell id prefix not validated against module

   The schema enforces the module field via a Literal discriminator, but the id field is a plain str with no check that its first segment matches module. A cell with module: llm and id: mcp.something passes schema validation and duplicate detection, but _module_coverage classifies it by c.module ("llm"), so the coverage numbers for both the LLMs and MCPs modules would be silently wrong. Adding a model_validator to _Base (or a post-load check here) that asserts id.startswith(c.module + ".") would catch this class of typo.

_{Reviews (1): Last reviewed commit: "Merge remote-tracking branch 'origin/lit..." | Re-trigger Greptile}

[greptile-apps]

The collect_covered_ids function redirects stdout into a StringIO buffer to suppress pytest's collection output, but stderr is left open. When collection encounters import errors, missing-fixture warnings, or deprecation notices, those messages are written to stderr and appear in the terminal interleaved with the coverage table, making the output noisy and harder to read. Redirecting stderr in the same with block suppresses the noise while still capturing errors via sink.collection_errors.

Suggested change

	with contextlib.redirect_stdout(io.StringIO()):
	pytest.main(
	with contextlib.redirect_stdout(io.StringIO()), contextlib.redirect_stderr(io.StringIO()):
	pytest.main(

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[greptile-apps]

_module_coverage calls ROLLUP[c.module] for every cell. ROLLUP is defined in schema.py alongside the Cell union, but there is no exhaustiveness check ensuring the two stay in sync. If a new module literal is added to Cell (e.g., Literal["agents"]) and ROLLUP is not updated, this line raises KeyError at runtime for every cell of that new type, crashing the collector entirely. A get with a fallback or an explicit assertion at import time would catch the mismatch earlier.

Suggested change

	def _module_coverage(module: str, cells: tuple[Cell, ...], covered: frozenset[str]) -> ModuleCoverage:
	in_module = tuple(c for c in cells if ROLLUP[c.module] == module)
	def _module_coverage(module: str, cells: tuple[Cell, ...], covered: frozenset[str]) -> ModuleCoverage:
	in_module = tuple(c for c in cells if ROLLUP.get(c.module, "") == module)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[mubashir1osmani]

merging this, we will start adding coverage for all p0 tiers and keep moving forward from there

[mubashir1osmani]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 5ecc6e1. Configure here.}