Split LLM e2e coverage modules

[ishaan-berri]

Summary

• split e2e coverage reporting into Core LLMs and Non-Core LLMs
• type the LLM registry endpoint/route/capability vocabulary so new taxonomy values require schema review
• add strict marker enforcement and reconcile existing orphan covers markers

Testing

• PYTHONPATH=tests/e2e uv run pytest tests/e2e/coverage_registry/test_collector.py -q
• PYTHONPATH=. uv run python -m coverage_registry.collector --strict
• uv run ruff check --target-version py312 tests/e2e/coverage_registry tests/e2e/llm_translation/test_chat_completions_regression_e2e.py tests/e2e/llm_translation/test_provider_features_e2e.py tests/e2e/management/test_management_e2e.py
• git rebase upstream/litellm_internal_staging

Note: python -m coverage_registry.collector --strict --fail-on-collection-errors still fails on existing collection warnings for logging/test_prometheus_cardinality_e2e.py and realtime; marker-only strict mode passes.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

This PR refactors the e2e coverage reporting infrastructure: LLM cells are now split into "Core LLMs" (chat_completions, messages, responses) and "Non-Core LLMs" for dashboarding, and the LlmCell fields (subject_endpoint, route, capability) are now typed with strict Literal aliases so new taxonomy values require an explicit schema change.

• schema.py replaces the flat ROLLUP dict with PREFIX_ROLLUP + a dashboard_module() function that dispatches LLM cells to Core or Non-Core based on CORE_LLM_ENDPOINTS; MODULE_ORDER gains the two new module names.
• collector.py consumes dashboard_module() and adds --strict / --fail-on-collection-errors CLI flags enabling CI to reject orphan markers.
• All nine @pytest.mark.covers orphan markers across test_management_e2e.py, test_provider_features_e2e.py, and test_chat_completions_regression_e2e.py are reconciled against the registry; all target IDs verified present in the YAML files.

Confidence Score: 5/5

Changes are confined to test infrastructure and the coverage registry; no production code is touched.

All YAML registry entries validate against the new typed schema (no removed routes or capabilities appear in the YAML files), every reconciled marker target was verified present in its registry file, the new test correctly asserts the Core/Non-Core split arithmetic, and the CLI flag additions follow the existing main() return-code pattern cleanly.

No files require special attention.

Important Files Changed

Filename	Overview
tests/e2e/coverage_registry/schema.py	Introduces typed Literals (LlmEndpoint, LlmRoute, LlmCapability) for LlmCell fields; splits ROLLUP into PREFIX_ROLLUP plus a dashboard_module() function that routes LLM cells to "Core LLMs" or "Non-Core LLMs".
tests/e2e/coverage_registry/collector.py	Swaps ROLLUP[c.module] for dashboard_module(c); adds --strict and --fail-on-collection-errors CLI flags; formatting only otherwise.
tests/e2e/coverage_registry/test_collector.py	Adds test_llm_cells_roll_up_by_core_endpoint covering the new split; expands _llm helper to accept subject_endpoint; arithmetic in assertions verified correct.
tests/e2e/management/test_management_e2e.py	Reconciles four orphan markers: management.team.new.persists → mgmt, persists → happy_path for user/org, and mgmt.key.generate.member_forbidden → other.auth.virtual_key.route_permission_enforced; all target IDs verified present in their respective YAML files.
tests/e2e/llm_translation/test_provider_features_e2e.py	Fixes two orphan markers: service_tier marker gains the missing streaming segment; bedrock prompt-cache marker changes assertion from cache_hit (orphan) to works (registered).
tests/e2e/llm_translation/test_chat_completions_regression_e2e.py	Replaces single orphan provider marker with three specific route markers (openai, anthropic, vertex); formatting-only changes elsewhere.
tests/e2e/coverage_registry/llm_conversational.yaml	Adds one new P1 registry entry for llm.chat_completions.openai.service_tier.nonstream.works; all fields validate against the new typed schema.
tests/e2e/CLAUDE.md	Updates module listing, naming grammar, and adds --strict / --fail-on-collection-errors usage notes to reflect the new schema and CLI.
tests/e2e/coverage_registry/README.md	Documents the Core/Non-Core split and strict mode usage; consistent with schema.py and CLAUDE.md changes.

_{Reviews (1): Last reviewed commit: "Split LLM e2e coverage modules" | Re-trigger Greptile}

[greptile-apps]

The annotation frozenset[str] is wider than needed — a static checker won't flag a typo like "chat_completion" (missing 's') being added here. Narrowing to frozenset[LlmEndpoint] makes the type consistent with the LlmCell.subject_endpoint field it guards.

Suggested change

	CORE_LLM_ENDPOINTS: frozenset[str] = frozenset(
	CORE_LLM_ENDPOINTS: frozenset[LlmEndpoint] = frozenset(

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 30 untouched benchmarks

---

_{Comparing ishaan-berri:litellm_e2e_llm_module_breakdown (682d220) with litellm_internal_staging (db24027)}

[ishaan-berri]

Closing this fork-based PR. Reopened from the in-repo branch here: #32403

[ishaan-berri]

Closing this fork-based PR. Reopened from the in-repo branch BerriAI/litellm:litellm_e2e_llm_module_breakdown here: #32403