Split LLM e2e coverage modules

tests/e2e/CLAUDE.md

@@ -63,23 +63,26 @@ The harness is fully typed and new code must not add `Any` or widen the basedpyr
 
 The set of tests we want is a registry checked into this repo, one row per behavior; that file is the definition of done and the denominator. Each e2e test declares what it covers with `@pytest.mark.covers("...")`, and a small collector diffs the registry against the tests and ships coverage to the existing Grafana. No Allure, no new dependencies
 
-Coverage is organized as module > feature > test. There are six modules: LLMs, MCPs, Management/UI, Reliability & Performance, Logging & Guardrails, and Other. A feature is either an endpoint (`/chat/completions`) or a behavior (fallbacks, rate limits; config-driven, with no route of its own). A cell reads like `llm.chat_completions.bedrock_converse.tool_use.stream.works`
+Coverage is organized as module > feature > test. Dashboard modules are Core LLMs, Non-Core LLMs, MCPs, Management/UI, Reliability & Performance, Logging & Guardrails, and Other. A feature is either an endpoint (`/chat/completions`) or a behavior (fallbacks, rate limits; config-driven, with no route of its own). A cell reads like `llm.chat_completions.bedrock_converse.tool_use.stream.works`
 
 The metric is coverage: the share of registry rows that have a passing covering test, reported to Grafana per module so a gap surfaces as an uncovered row rather than a silent absence
 
+Tests do not declare a dashboard module directly. They only declare the registry cell id with `@pytest.mark.covers("...")`; the registry row decides the module, tier, endpoint, and dashboard rollup. Run `python -m coverage_registry.collector --strict` when you want CI to reject unknown marker ids. Add `--fail-on-collection-errors` when the job should also fail on pytest collection errors.
+
 ### Naming grammar per module
 
-LLMs - endpoint features (subject = the route), seeded from the Claude Code compat matrix
+LLMs - endpoint features (subject = the route), seeded from the Claude Code compat matrix. `chat_completions`, `messages`, and `responses` are Core LLMs. Other LLM endpoints, including `batches` and `realtime`, roll up as Non-Core LLMs.
 
 ```
 llm.<endpoint>.<route>.<capability>.<streaming>.<assertion>
   endpoint   : chat_completions | messages | responses | embeddings | batches | files
                | rerank | images_generations | audio_speech | audio_transcriptions | moderations
-  route      : openai | azure_openai | anthropic | bedrock_invoke | bedrock_converse | vertex | azure_foundry
+               | realtime
+  route      : openai | azure_openai | anthropic | bedrock_converse | vertex | azure_foundry
+               | cohere | together_ai
                (vocab varies per endpoint; messages is anthropic-format only)
-  capability : basic | tool_use | prompt_cache_5m | prompt_cache_1h | vision | thinking
-               | thinking_tool_use | pdf_input | web_search | structured_output | count_tokens
-               | tool_search | long_context_1m
+  capability : basic | tool_use | prompt_cache_5m | vision | thinking | structured_output
+               | service_tier
   streaming  : stream | nonstream   (omit where n/a)
   assertion  : works | cost_logged
   label (not in id): model = haiku-4.5 | sonnet-4.6 | opus-4.7 | gpt-*

tests/e2e/coverage_registry/GRAFANA_DASHBOARD.md

@@ -0,0 +1,92 @@
+# Grafana Dashboard Brief
+
+This dashboard should answer two questions:
+
+1. How much e2e coverage do we have by module right now?
+2. Is e2e coverage by module improving or regressing over time?
+
+The dashboard should use plain **Coverage** language. Avoid exposing P0/P1/P2 in the
+default view; tiers can be added later as a filter or drilldown.
+
+## Panels
+
+### Coverage by Module
+
+Use a bar chart or table with one row per module:
+
+- Core LLMs
+- Non-Core LLMs
+- MCPs
+- Management/UI
+- Reliability & Performance
+- Logging & Guardrails
+- Other
+
+Each row should show:
+
+- `covered`
+- `total`
+- `coverage_percent`
+
+Formula:
+
+```text
+coverage_percent = covered / total * 100
+```
+
+### Coverage Trend by Module
+
+Use a time-series chart with one line per module.
+
+- X-axis: CI run timestamp, scrape timestamp, or pushed metric timestamp
+- Y-axis: `coverage_percent`
+- Series label: module name
+
+This shows whether coverage is improving across dates.
+
+## Data Contract
+
+Generate coverage data from the registry collector:
+
+```bash
+cd tests/e2e
+PYTHONPATH=. python -m coverage_registry.collector --format prometheus --strict
+```
+
+For artifact-based jobs, JSON is also available:
+
+```bash
+cd tests/e2e
+PYTHONPATH=. python -m coverage_registry.collector --format json --strict
+```
+
+Prometheus metrics:
+
+```text
+litellm_e2e_coverage_cells{module="<module>",state="covered"} <count>
+litellm_e2e_coverage_cells{module="<module>",state="total"} <count>
+litellm_e2e_coverage_percent{module="<module>"} <percent>
+litellm_e2e_coverage_orphan_markers <count>
+litellm_e2e_coverage_collection_errors <count>
+```
+
+Grafana gets the trend by storing these metrics over time. No date needs to be encoded
+inside the metric itself.
+
+## Alerts
+
+Start with two alerts:
+
+- Unknown marker count is greater than zero.
+- Coverage percent for any module drops compared with the previous successful run.
+
+After existing collection warnings are fixed, add:
+
+- Collection error count is greater than zero.
+
+## Non-Goals
+
+- Do not use line coverage for this dashboard. This is behavior coverage, not source-line
+  coverage.
+- Do not let tests invent module names. Tests only declare `@pytest.mark.covers(...)`;
+  the registry decides which module a cell belongs to.

tests/e2e/coverage_registry/README.md

@@ -3,20 +3,25 @@
 This directory is the **denominator** for e2e test coverage: the set of behaviors we
 want covered, one row per behavior, checked into the repo so coverage is a number we
 can track instead of a guess. It implements the plan in the "E2E Coverage Tracking"
-note; the naming grammar lives in `tests/e2e/CLAUDE.md`.
+note; the naming grammar lives in `tests/e2e/CLAUDE.md`. The Grafana dashboard contract
+lives in `GRAFANA_DASHBOARD.md`.
 
 ## The model
 
 A **cell** is one customer-noticeable behavior a single e2e test can assert pass/fail
 on, for example `llm.chat_completions.bedrock_converse.tool_use.stream.works`. Cells are
-grouped `module > feature > test`, six dashboard modules in all. Each cell carries a
-tier (P0/P1/P2), a source, and a `fail_before_fix` flag.
+grouped `module > feature > test`, with LLM cells split into Core LLMs and Non-Core
+LLMs for dashboarding. Each cell carries a tier (P0/P1/P2), a source, and a
+`fail_before_fix` flag.
 
 The rows live in per-prefix YAML files (`llm_*.yaml`, `mgmt.yaml`, `mcp.yaml`,
 `reliability.yaml`, `logging.yaml`, `guardrail.yaml`, `other.yaml`) and validate against
 the discriminated union in `schema.py`, so an LLM row cannot carry a guardrail field and
-vice versa. `logging` and `guardrail` are two id-prefixes that roll up into the single
-"Logging & Guardrails" dashboard module.
+vice versa. `llm` rows with `subject_endpoint` of `chat_completions`, `messages`, or
+`responses` roll up to "Core LLMs"; all other LLM endpoints roll up to "Non-Core
+LLMs". LLM endpoint, route, and capability values are typed in `schema.py`, so new
+taxonomy values require an explicit schema change. `logging` and `guardrail` are two
+id-prefixes that roll up into the single "Logging & Guardrails" dashboard module.
 
 A test declares what it covers with a marker:
 
@@ -36,9 +41,22 @@ proxy. Whether a covered cell currently passes or fails is a separate, live conc
 cd tests/e2e && PYTHONPATH=. python -m coverage_registry.collector
 ```
 
-The headline is P0 coverage. The collector also lists markers that point at ids not in
-the registry, so a typo or an unenumerated behavior surfaces instead of being silently
-dropped.
+Use `--format prometheus` or `--format json` for CI jobs that publish coverage to
+Grafana.
+
+The headline is overall coverage. The collector also lists markers that point at ids
+not in the registry, so a typo or an unenumerated behavior surfaces instead of being
+silently dropped.
+
+Use strict mode in CI once existing draft markers are reconciled:
+
+```
+cd tests/e2e && PYTHONPATH=. python -m coverage_registry.collector --strict
+```
+
+Strict mode exits non-zero on `@pytest.mark.covers(...)` ids that are not checked into
+the registry. Add `--fail-on-collection-errors` when the job should also fail on pytest
+collection errors.
 
 ## Status: this is a draft for review