test(e2e): wire claude_code compat matrix cells into the coverage registry

[mateo-berri]

Relevant issues

Resolves caveat 5 of #32548: the Claude Code compat matrix cells under tests/e2e/claude_code/ carried no @pytest.mark.covers(...) markers, so they never fed the coverage registry even though the registry's LLM grammar was seeded from this matrix

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

This PR is coverage-registry wiring with no proxy behavior change, so the end-user-visible surface is the collector itself: its output is what ships coverage to Grafana. The proof is therefore the collector run, before and after

Before, at base ff4a40f017 (compat cells invisible to the registry):

$ cd tests/e2e && PYTHONPATH=. python -m coverage_registry.collector --strict
MODULE                             COVERED    COVERAGE
Core LLMs                             8/55       14.5%
...
ALL                                 58/316       18.4%

Headline coverage: 58/316  (18.4%)
exit=0

After, at c24abe0fd8 (all 75 compat cells feed the registry):

$ cd tests/e2e && PYTHONPATH=. python -m coverage_registry.collector --strict
MODULE                             COVERED    COVERAGE
Core LLMs                           83/123       67.5%
...
ALL                                133/384       34.6%

Headline coverage: 133/384  (34.6%)
exit=0

Proof that the wiring bites, captured at c24abe0fd8 by mutating one marker id to a nonexistent row and rerunning (the mutation was reverted before commit):

$ sed -i '' 's/llm.messages.vertex.pdf.nonstream.works/llm.messages.vertex.pdf.nonstream.doesnotexist/' claude_code/pdf_input/test_vertex_ai.py
$ PYTHONPATH=. python -m coverage_registry.collector --strict
...
1 marker(s) point at ids not in the registry (reconcile: fix the marker or add the cell):
  llm.messages.vertex.pdf.nonstream.doesnotexist
exit=1

Type

✅ Test

Changes

Adds one @pytest.mark.covers(...) decorator to each of the 75 Claude Code compat matrix cell tests (15 features x 5 providers), mapping every cell onto the registry grammar llm.messages.<route>.<capability>.<streaming>.works. Matrix provider ids map to registry routes as vertex_ai -> vertex and azure -> azure_foundry; the other three match directly. The 10 basic_messaging_* files gain an import pytest since they previously only imported the shared cell helper. No test logic changes

7 markers reuse the existing seeded llm.messages.anthropic.* rows (basic nonstream/stream, tool_use nonstream/stream, vision, prompt_cache_5m, thinking). The remaining 68 cells had no registry row, so llm_conversational.yaml gains 68 new llm.messages.* rows grouped by route, sourced to their covering cell file. Tiers follow the established per-route pattern: basic, tool_use, and vision are P0 on the bedrock_invoke, bedrock_converse, and vertex routes; everything else, including all azure_foundry rows, is P1

Seven capability values had no vocabulary entry, so LlmCapability in coverage_registry/schema.py gains count_tokens, long_context, pdf, prompt_cache_1h, thinking_with_tool_use, tool_search, and web_search, and the grammar vocab lines in tests/e2e/CLAUDE.md are updated to match so that file stays self-describing

QA runbook

• tests/e2e/claude_code/<feature>/test_<provider>.py (75 cells) - every compat matrix cell now declares the registry row it covers, and the collector counts it
  • Run cd tests/e2e && PYTHONPATH=. python -m coverage_registry.collector --strict and expect exit 0 with Core LLMs at 83/123
  • Pick any cell file, e.g. tests/e2e/claude_code/vision/test_vertex_ai.py, and confirm its single test carries @pytest.mark.covers("llm.messages.vertex.vision.nonstream.works") matching a row in tests/e2e/coverage_registry/llm_conversational.yaml
  • Mutate that marker id to a nonexistent row, rerun the collector with --strict, and expect exit 1 naming the orphan marker; revert the mutation
  • Run pytest tests/e2e/claude_code --collect-only -q and expect all 414 nodes to collect cleanly under the strict ini
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[greptile-apps]

Greptile Summary

This PR wires all 75 Claude Code compat matrix test cells (15 features × 5 providers) into the coverage registry by adding @pytest.mark.covers(...) decorators, so the collector can track them. There are no changes to test logic or production code.

• 75 test functions each gain one @pytest.mark.covers("llm.messages.<route>.<capability>.<streaming>.works") line; the 10 basic_messaging_* files also gain import pytest since they had no prior pytest import.
• llm_conversational.yaml gains 68 new registry rows (7 anthropic cells reuse pre-existing rows); schema.py adds 7 new LlmCapability literals; CLAUDE.md updates the grammar vocabulary to match.

Confidence Score: 5/5

Safe to merge — the change is purely additive test metadata with no production code or test logic affected.

All 75 marker IDs in the test decorators match their corresponding rows in llm_conversational.yaml. The 7 reused anthropic rows already existed before this PR; the 68 new rows are correctly typed against the updated LlmCapability schema. The basic_messaging_* files correctly gain import pytest, and no existing assertion logic was touched anywhere in the diff.

No files require special attention.

Important Files Changed

Filename	Overview
tests/e2e/coverage_registry/llm_conversational.yaml	Adds 68 new registry rows across bedrock_invoke, bedrock_converse, vertex, azure_foundry, and anthropic routes; all IDs, capability values, and source paths match their corresponding test file markers.
tests/e2e/coverage_registry/schema.py	Extends LlmCapability with 7 new literals (count_tokens, long_context, pdf, prompt_cache_1h, thinking_with_tool_use, tool_search, web_search) in alphabetical order; no other schema logic changed.
tests/e2e/CLAUDE.md	Grammar vocabulary updated to list the 7 new capability values; kept in sync with schema.py additions.
tests/e2e/claude_code/basic_messaging_non_streaming/test_anthropic.py	Representative of the 10 basic_messaging_* files: adds import pytest and @pytest.mark.covers pointing to the pre-existing llm.messages.anthropic.basic.nonstream.works row.
tests/e2e/claude_code/vision/test_vertex_ai.py	Representative of non-basic_messaging feature cells: adds @pytest.mark.covers mapping to the newly seeded llm.messages.vertex.vision.nonstream.works row; no logic change.

_{Reviews (1): Last reviewed commit: "test(e2e): wire claude_code compat matri..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_compat_matrix_covers_markers (c24abe0) with litellm_internal_staging (ff4a40f)}