fix(e2e): exclude skipped tests from coverage-registry numerator

[yuneng-berri]

TLDR

Problem this solves:

• Skipped e2e tests counted as covered in the coverage number
• Headline read 311/434 while 21 cells had no runnable test
• A cell going red and getting skipped kept its coverage credit

How it solves it:

• Collector splits covers markers by whether pytest would run the test
• Skip state comes from pytest's own skip/skipif evaluator
• Cells left uncovered by a skip are listed and exported

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Up front, so a reviewer does not go looking for it: this change has no proxy or provider surface to curl. coverage_registry/collector.py is a static reporting tool that reads markers out of a collect-only pass, spins up no proxy, and issues no LLM call. What the consumer of this tool actually sees is its stdout and the Grafana series built from it, so the proof below is that output, captured before and after against the same registry (434 rows either way) with no test edits in between

Before, at 83c256f609 (the parent of the fix, both files in coverage_registry/ checked out at that commit):

cd tests/e2e && PYTHONPATH=. python -m coverage_registry.collector

MODULE                             COVERED    COVERAGE
Core LLMs                          117/136       86.0%
Non-Core LLMs                        39/51       76.5%
MCPs                                  7/15       46.7%
Management/UI                        66/70       94.3%
Reliability & Performance             8/26       30.8%
Quota Management                     41/43       95.3%
Logging & Guardrails                 15/56       26.8%
Other                                18/37       48.6%
------------------------------------------------------
ALL                                311/434       71.7%

Headline coverage: 311/434  (71.7%)

After, at b97e29eb5f, same command:

MODULE                             COVERED    COVERAGE
Core LLMs                          103/136       75.7%
Non-Core LLMs                        36/51       70.6%
MCPs                                  5/15       33.3%
Management/UI                        66/70       94.3%
Reliability & Performance             7/26       26.9%
Quota Management                     41/43       95.3%
Logging & Guardrails                 15/56       26.8%
Other                                17/37       45.9%
------------------------------------------------------
ALL                                290/434       66.8%

Headline coverage: 290/434  (66.8%)

21 cell(s) are claimed only by skipped tests, so they count as uncovered (unskip the test or drop the marker):
  llm.batches.hosted_vllm.basic.nonstream.works
  llm.files.hosted_vllm.upload.nonstream.works
  llm.files.openai.list.nonstream.works
  llm.messages.anthropic.long_context_1m.nonstream.works
  llm.messages.azure_foundry.long_context_1m.nonstream.works
  llm.messages.azure_foundry.mid_conversation_system.nonstream.cache_hit
  llm.messages.azure_foundry.tool_search.nonstream.works
  llm.messages.bedrock_converse.long_context_1m.nonstream.works
  llm.messages.bedrock_converse.pdf_input.nonstream.works
  llm.messages.bedrock_converse.thinking.nonstream.works
  llm.messages.bedrock_invoke.long_context_1m.nonstream.works
  llm.messages.bedrock_invoke.mid_conversation_system.nonstream.cache_hit
  llm.messages.bedrock_invoke.tool_search.nonstream.works
  llm.messages.vertex.count_tokens.nonstream.works
  llm.messages.vertex.long_context_1m.nonstream.works
  llm.messages.vertex.mid_conversation_system.nonstream.cache_hit
  llm.messages.vertex.tool_search.nonstream.works
  mcp.call_tool.oauth.succeeds
  mcp.list_tools.oauth.succeeds
  other.config.passthrough.headers_forwarded
  reliability.routing.complexity_llm_classifier.routes_by_llm_tier

Every one of those 21 traces back to a real marker pair in the tree; spot-check any of them the same way:

grep -n -B2 "def test_long_context_1m_bedrock_invoke" tests/e2e/claude_code/long_context_1m/test_bedrock_invoke.py

156-@pytest.mark.skip(reason="stage red: 1M long_context not green on stage Bedrock Invoke deployments yet")
157-@pytest.mark.covers("llm.messages.bedrock_invoke.long_context_1m.nonstream.works")
158:def test_long_context_1m_bedrock_invoke(compat_result):

Two nuances a reviewer should know about that list. mcp.list_tools.oauth.succeeds and mcp.call_tool.oauth.succeeds come from a module-level skipif on E2E_LINEAR_STORAGE_STATE, so they drop out only in an environment without that session captured; where the e2e suite has it, both count as covered again. That is the intended behavior, and it does mean the number is now a property of the environment the collector runs in, so run it where the suite runs. Separately, a pytest.skip() call inside a test body still cannot be seen by a static pass, since it does not exist until the test runs; the module docstring and the registry README both say so rather than leaving it implied

Type

🐛 Bug Fix

Changes

_CoversSink read @pytest.mark.covers(...) off every collected item, and collection never evaluates skips, so a test carrying both a skip and a covers marker handed its cell full credit while asserting nothing. Seventeen files under tests/e2e do exactly that today

A cell now counts as covered only when at least one test pytest would actually run declares it. A cell claimed by both a live test and a skipped one stays covered, since one runnable assertion is enough. Skip state is resolved with pytest's own evaluator rather than a local reimplementation, so skip, skipif (both boolean and string conditions), and module-level pytestmark behave here exactly as they behave in the run itself

Cells knocked out of the numerator by a skip do not just vanish. They print under the headline, ship as skipped_markers in the JSON output, and ship as litellm_e2e_coverage_skipped_markers in the Prometheus output, so an unskip-pending gap stays visible. The Loki formatter keeps its exact line contract, one COVERAGE_TOTAL plus one COVERAGE_MODULE per module; only its numbers change. A marker on a skipped test that points outside the registry is still reported as an orphan, so --strict gains no blind spot

collect_covered_ids becomes collect_markers and returns a frozen CollectedMarkers instead of a widening tuple. Nothing outside the module called it

QA runbook

The collector is harness tooling rather than a product feature, so its tests carry no e2e marker and need no live proxy (tests/e2e/CLAUDE.md calls out coverage_registry/test_collector.py as exactly this exception). The manual steps below are therefore local commands, and they map 1:1 to what the new tests assert

• tests/e2e/coverage_registry/test_collector.py::test_collection_counts_only_markers_on_tests_that_would_run - a real collect-only pass credits a cell only when the test claiming it would run

  • Write a scratch file with five tests: one plain, one under @pytest.mark.skip, one under skipif(True, reason=...), one under skipif(False, reason=...), one under skipif("True"), each with its own @pytest.mark.covers id
  • Add a second scratch file whose module sets pytestmark = pytest.mark.skipif(True, reason=...) and holds one more covering test
  • Run collect_markers(Path("<that dir>")) from tests/e2e with PYTHONPATH=. and expect covered to hold only the plain and skipif(False) ids, with the other four in skipped_only
  • Add two tests claiming the same cell id, one live and one skipped, and expect that id in covered and absent from skipped_only
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/coverage_registry/test_collector.py::test_cell_claimed_only_by_a_skipped_test_is_uncovered - a skipped-only cell is uncovered everywhere the report is consumed, not just in the headline

  • Call compute_coverage with two P0 cells, one in covered and one in skipped_only
  • Expect covered and p0_covered of 1, and the skipped id back in p0_gaps
  • Expect the id in the text render, in skipped_markers in the JSON render, and counted by litellm_e2e_coverage_skipped_markers in the Prometheus render
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/coverage_registry/test_collector.py::test_skipped_marker_outside_the_registry_is_still_an_orphan - skipping a test does not hide a typo'd cell id from --strict

  • Call compute_coverage with one registry cell and an unknown id in skipped_only
  • Expect that id in orphan_markers and skipped_markers empty
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

Whole file, from tests/e2e:

PYTHONPATH=. python -m pytest coverage_registry/ -q

13 passed, 1 warning in 0.26s

The first of those three was checked red before green: with the skip lookup stubbed out to return False, it fails naming exactly the four cells that leak back into covered. make lint-e2e-basedpyright reports 0 errors over tests/e2e

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[greptile-apps]

Greptile Summary

This PR corrects static e2e coverage reporting so skipped tests no longer contribute to the covered numerator.

• Uses pytest’s skip-marker evaluator to separate runnable and skipped-only coverage declarations.
• Exposes skipped-only registry cells in text, JSON, and Prometheus reports while preserving orphan-marker detection.
• Adds tests for direct skips, boolean and string skipif conditions, module-level skips, shared coverage declarations, and orphan markers.
• Documents the updated coverage semantics and the limitation around runtime pytest.skip() calls.

Confidence Score: 5/5

The PR appears safe to merge, with the changed reporting semantics covered by focused tests and no affected repository consumers left incompatible.

The collector now excludes markers from tests pytest would skip, preserves coverage when another runnable test claims the same cell, and consistently reports skipped-only and orphan markers without introducing a reachable build or reporting failure.

Important Files Changed

Filename	Overview
tests/e2e/coverage_registry/collector.py	Correctly separates runnable and skipped-only markers and propagates the resulting data through supported report formats without breaking repository callers.
tests/e2e/coverage_registry/test_collector.py	Adds focused regression coverage for skip evaluation, shared marker ownership, reporting output, and orphan handling.
tests/e2e/coverage_registry/README.md	Accurately documents environment-dependent skip evaluation and the static collector’s runtime-skip limitation.
tests/e2e/CLAUDE.md	Updates test-harness guidance to clarify that skipped tests do not retain coverage credit.

_{Reviews (1): Last reviewed commit: "Merge remote-tracking branch 'origin/lit..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!