test(e2e): typed per-test metadata for the e2e suite

[ryan-crabbe-berri]

Builds on the recorded @step log from #42393, which has merged. This PR is the declared half.

This PR adds typed, declarative metadata to e2e tests, so coverage can be sliced by provider, model, endpoint and capability instead of by folder name. It is additive only. The coverage-registry YAML is untouched, and every existing @pytest.mark.covers("cell.id") keeps working.

The shape

The marker takes one frozen dataclass of enum members as its single argument, not kwargs. That way a typo is a type error rather than a silent miss:

@meta(Subject(
    domain=Domain.SPEND_BUDGETS,
    providers=(Provider.OPENAI,),
    models=(BACKEND,),
    capabilities=(Capability.REASONING,),
))
def test_priority_tier_bills_priority_rates(...): ...

dataclasses.asdict() turns that into <property> pairs with no per-field plumbing.

providers, models and capabilities are all tuples, because one test node often drives several of each. The claude_code matrix runs haiku, sonnet and opus in a single body, and a spend-attribution test calls a Gemini and an Anthropic model on one key.

• Canonical form: all three share one canonicaliser, which dedupes them and sorts by serialised value.
• JUnit encoding: each is written as a repeated property under the singular name, so there is no delimiter to corrupt.
• No pairing: the lists are independent sets. A model is not tied to a provider by position.
• Tuples only: anything but a tuple is refused at collection time. models=("gpt-5.5") is an error naming the file rather than one model per character.
• Constants, not copies: a declared model names the constant the test calls, so the report can't keep naming a model the test stopped driving. A guard in test_e2e_metadata.py fails on any model typed out as a string literal in @meta

The declared fields go after the fixed package / covers / source prefix, which stays byte-identical.

Decisions worth a look

• Every set-valued axis is plural. For capabilities, llm_translation/test_together_ai_e2e.py already hand-rolls a _Needs dataclass that ANDs two capabilities. coverage_registry/schema.py had to smuggle conjunctions in as ad-hoc values like thinking_with_tool_use. A tuple collapses both back.
• class X(str, Enum), not StrEnum, because pyproject.toml floors at Python 3.10.
• The suite stays free of a litellm import. tests/e2e is a black-box HTTP suite that imports litellm in zero files today, and making litellm a collection-time dependency would be a real regression. So Provider is its own enum rather than a re-export of LlmProviders, and TestProviderMirrorsLitellm fails on drift wherever litellm is importable.
• New meta marker, not an overload of covers. A dataclass passed positionally to covers would be silently dropped by dedupe_covers's isinstance(str) filter, so the registry collector would read the test as uncovered.
• route is the endpoint the test checks. A /team/update test gets team_management and a /spend/logs test gets spend_reporting. A budget or rate-limit test whose chat call only triggers the block leaves it unset, since its steps already name the call
• One Subject per test item. subject_properties reads get_closest_marker("meta"), so a parametrized matrix builds its full Subject in one place per param. test(e2e): add conversational matrix across chat, messages and responses #42359's cells_covering() already has that shape.

Pilot usage

tests/e2e/quota_management/ is annotated for real: 29 files, all with domain=spend-budgets. Tests that drive more than one provider or model declare all of them, including the shared-key, budget-fallback, per-model-budget and key-attribution tests. The rest of the suite is a separate follow-up PR, and metadata is optional until that lands.

Verification

Check	Result
test_e2e_metadata.py + test_e2e_junit_report.py (as the Code Quality test_e2e_metadata step runs them) + tests/e2e/test_junit_properties.py	73 passed
Offline harness run (-m "not e2e")	610 passed
Full-tree --collect-only	1345 collected, 2 errors
basedpyright tests/e2e	64 errors, none added
ruff	clean

• tests/code_coverage_tests/test_e2e_junit_report.py runs real pytest with --junitxml through the real tests/e2e/conftest.py, in-process and under -n 2. It asserts that the repeated provider / model / capability properties land in the parsed XML, and that a bare-str models= fails collection.
• Collection errors: both are the same httpx2 import errors as on feat(e2e): record each e2e test's steps, starting with ProxyClient #42393.
• basedpyright: all 64 errors are in untouched mcp/oauth_chat_client.py, the same as on feat(e2e): record each e2e test's steps, starting with ProxyClient #42393.
• Live e2e tests need a deployed proxy and real provider keys, so they were not run here.

This is part of the e2e test-metadata rollout; see the rollout plan. It is consumed by BerriAI/project-releaser#252.

---

Note

Low Risk
Changes are confined to e2e harness metadata, JUnit reporting, and test annotations; production proxy behavior is untouched and existing @pytest.mark.covers behavior is preserved.

Overview
Adds a declared half to e2e test metadata alongside the existing recorded @step log: tests can attach @meta(Subject(...)) with closed enums for domain, route, providers, models, capabilities, and mode. The frozen Subject dataclass validates plural fields (tuple-only, deduped/sorted), serializes via subject_properties() into JUnit <property> entries (repeated singular names for provider/model/capability), and appends after the unchanged package/covers/source prefix in result_properties.

Registers the meta marker in conftest/pytest.ini, documents usage in AGENTS.md, and expands harness tests (test_e2e_metadata.py, test_e2e_junit_report.py) including end-to-end JUnit round-trip and collection failures for bare-string models=(x). Pilot backfill: annotates all tests/e2e/quota_management/** tests with @meta, refactors model strings to named constants where needed, and adds a guard that @meta must not hand-type model string literals.

^{Reviewed by Cursor Bugbot for commit dc70ca0. Bugbot is set up for automated code reviews on this repo. Configure here.}

[greptile-apps]

Confidence Score: 5/5

[Low risk] Adds typed metadata decorators to test files.

The PR appears safe to merge based on this review

Summary

The PR adds typed, optional e2e test metadata to JUnit properties and annotates quota-management tests while retaining existing coverage markers. Since the previous review, changes trim explanatory docstrings

_{Reviews (11) · Last reviewed commit: "docs(e2e): trim excessive docstrings in ..."}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[ryan-crabbe-berri]

@greptileai

[ryan-crabbe-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[ryan-crabbe-berri]

@greptile re review

[ryan-crabbe-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[ryan-crabbe-berri]

@greptile re review

[ryan-crabbe-berri]

@greptile re review

[ryan-crabbe-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit dc70ca0. Configure here.}

[ryan-crabbe-berri]

@greptile re review