test: add e2e coverage module metrics

[ishaan-berri]

What improves

• Makes e2e coverage measurable by module instead of a loose list of tests.
• Splits LLM coverage into Core LLMs (chat_completions, messages, responses) and Non-Core LLMs.
• Adds strict registry validation so @pytest.mark.covers(...) markers must point to known coverage cells.
• Adds JSON and Prometheus collector output for Grafana coverage-by-module and coverage-trend panels.

Checks

• PYTHONPATH=tests/e2e uv run pytest tests/e2e/coverage_registry/test_collector.py -q
• PYTHONPATH=. uv run python -m coverage_registry.collector --strict
• PYTHONPATH=. uv run python -m coverage_registry.collector --format prometheus --strict
• uv run ruff check --target-version py312 tests/e2e/coverage_registry

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

This PR splits the single "LLMs" dashboard module into "Core LLMs" (chat_completions, messages, responses) and "Non-Core LLMs" (all other LLM endpoints), types the LLM endpoint/route/capability vocabulary as Literal unions so new taxonomy values require an explicit schema change, and adds --format prometheus|json, --strict, and --fail-on-collection-errors flags to the coverage collector CLI.

• schema.py replaces the flat ROLLUP dict with a dashboard_module() function and typed LlmEndpoint/LlmRoute/LlmCapability literals; PREFIX_ROLLUP covers all six non-llm module keys so no KeyError risk exists.
• Four orphan markers across the management and provider-feature test files are corrected to point at real registry cells (wrong management. prefix, stale .persists assertions, misplaced key-permission cell).
• render_prometheus and render_json are added with correct Prometheus exposition formatting and include per-module p0 counts in JSON; the text renderer switches from a P0-focused headline to overall coverage percentage.

Confidence Score: 4/5

Safe to merge; changes are entirely within the tests/e2e tooling layer with no production code impact.

The split and schema tightening are clean and consistent with all existing YAML registry data. The one omission worth noting is that p0_gaps (the list of specific uncovered P0 cell IDs) is computed on every run but silently dropped from both JSON and Prometheus output, leaving downstream alerting jobs without the gap-ID detail.

tests/e2e/coverage_registry/collector.py — _report_dict omits the p0_gaps field that is always populated by compute_coverage.

Important Files Changed

Filename	Overview
tests/e2e/coverage_registry/collector.py	Adds JSON/Prometheus output formats and --strict/--fail-on-collection-errors CLI flags; logic is correct but p0_gaps is computed yet omitted from JSON serialization.
tests/e2e/coverage_registry/schema.py	Introduces typed LlmEndpoint/LlmRoute/LlmCapability literals and replaces flat ROLLUP dict with dashboard_module() function; PREFIX_ROLLUP covers all non-llm module keys, no KeyError risk.
tests/e2e/coverage_registry/test_collector.py	Adds well-targeted tests for Core/Non-Core rollup, text render language, JSON, and Prometheus output; assertions are correct and match actual render logic.
tests/e2e/management/test_management_e2e.py	Fixes four orphan markers: wrong management. prefix corrected to mgmt., two .persists assertions updated to .happy_path to match actual registry cells, and a misplaced key-permission marker moved to the correct other.auth.virtual_key cell.

_{Reviews (1): Last reviewed commit: "Add e2e coverage dashboard metrics" | Re-trigger Greptile}

[greptile-apps]

p0_gaps computed but never serialized

compute_coverage() always populates report.p0_gaps (the sorted list of uncovered P0 cell IDs), but _report_dict() omits it entirely. As a result, the JSON output shipped to Grafana only includes per-module p0_covered/p0_total counts — the actual gap IDs are silently dropped. Any alert or downstream job that needs to enumerate specific P0 cells currently failing to meet coverage would have no way to retrieve them from the collector output.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 30 untouched benchmarks

---

_{Comparing litellm_e2e_llm_module_breakdown (52033ef) with litellm_internal_staging (db24027)}