feat(e2e): record each e2e test's steps, starting with ProxyClient

[ryan-crabbe-berri]

Record what every e2e test actually did, step by step, from the harness it calls. The recorded steps become the test's user story in the JUnit report, and a failing test's last step is where it died. Nothing is hand-written, so the story can't drift from the code.

This is the steps half of #42044, split out so it can be reviewed on its own. #42044 is re-stacked on top of this branch and now holds only the declared @meta(Subject(...)) half.

The shape

@step goes on harness helpers, never on tests:

class ProxyClient:
    @step("Generate a virtual key with {body}")
    def generate_key(self, body: KeyGenerateBody) -> str: ...

    @step("Add a deployment named {body.model_name} that calls {body.litellm_params.model}")
    def register_model(self, body: ModelNewBody, ...) -> str: ...

test_rpm_limit_blocks_over_limit's report then carries these as repeated <property name="step"> entries:

Generate a virtual key with models: claude-haiku-4-5 and rpm limit: 3
Read the key's settings back from /key/info
Send a /chat/completions request to claude-haiku-4-5 with the prompt "reply with one word d3940a1c4288"

The label is recorded before the wrapped call, so a helper that raises still leaves its own label last.

This PR covers all 45 ProxyClient methods, including responses_stream, which main added after this branch started, and the rate-limit suite's QuotaClient.chat. The other harnesses (the domain clients, lifecycle, idp, the logging readers, migrations, the Claude CLI driver) get steps one area at a time in follow-ups, so each area's wording can be reviewed on its own. Until then their tests show only the ProxyClient calls they make

Label templates

A label can name the helper's own parameters, so the story says what the test asked for. @step("Generate a virtual key with {body}") records "Generate a virtual key with models: claude-haiku-4-5 and rpm limit: 3". Only what a label names reaches the report, a Field(repr=False) field is never shown, and a placeholder the helper doesn't take fails at import. A dotted placeholder reads one field of a request model, as in {body.litellm_params.model}, and a guard test fails if one names a field the model doesn't have. Secret request fields (api_key, AWS and Vertex credentials, credential_values, and the Langfuse and W&B keys a logging callback carries in key metadata) are marked Field(repr=False), so no template can print them. As a backstop, the recorder masks the value of every secret-named environment variable wherever it lands in a label, so a credential in a dict, a prompt or an unmarked field still comes out as ***.

Decisions worth a look

• Only the outermost step records, tracked per thread. Harness layers call each other: ProxyClient.create_model goes through register_model, and a domain client wraps the shared ProxyClient. So every layer carries a label, and the story still reads at the level the test called in at, one beat per action. A helper that fans work out to threads still records its workers' steps.
• Context managers. @step goes above @contextmanager, and the setup and cleanup around yield run inside the step while the with body records normally. Without this, cleanup such as restricted_user's DROP ROLE would append a step after the one a test died on.
• Bare generators are refused. A bare generator function is a TypeError at import, because its body interleaves with the caller's.
• The wrapper is a stack frame. The 5 cleanup warnings raised directly inside decorated ProxyClient.delete_* helpers now use stacklevel=2 + STEP_FRAMES, so they still report at their caller rather than at e2e_metadata.py. A test pins the frame count.
• Timing.
  • The log is emptied in pytest_runtest_setup. An autouse fixture would run after wider-scoped fixtures and inherit a module finalizer's steps.
  • It is attached after setup and after call, so a fixture error keeps the steps recorded before the crash.
  • Teardown doesn't attach, because finalizer steps are cleanup.
  • Each attach replaces the previous story, so --reruns 1 doesn't double it.
  • Consecutive duplicates collapse, so a poll loop is one step.
  • Past 50 steps the oldest are dropped, behind a leading (N earlier steps not recorded) line, so the last step is still where the test died.
• The fixed package / covers / source prefix is byte-identical; steps only ever append after it.
• The harness tests live outside tests/e2e, which holds only tests that drive a live proxy. They are tests/code_coverage_tests/test_e2e_metadata.py and test_e2e_junit_report.py, run by the test_e2e_metadata step of the Code Quality GitHub Actions workflow, so they gate every PR. The unit file covers only the recorder's edge cases (dedupe, the cap, nesting, context managers). Call order, the last step, the per-test reset and the attach are asserted once, in the real-pytest report test

Verification

Check	Result
test_e2e_metadata.py + test_e2e_junit_report.py, run as the Code Quality step runs them	41 passed
All offline harness tests (-m "not e2e", minus the live claude_code CLI tests)	627 passed
Full-tree --collect-only	1507 collected, no errors
basedpyright on every changed harness file	0 errors
ruff check --config ruff-tests.toml	clean
assert_ci_coverage.py	every test file invoked by a job

• test_e2e_junit_report.py runs real pytest with --junitxml through tests/e2e/conftest.py, in-process and under -n 2. It asserts on the parsed XML for the passing, failing, setup-error, rerun and wide-scope-fixture cases.
• Label guard: a test walks every dotted placeholder in ProxyClient and fails if one names a field its request model lacks. A deliberate {body.modle} typo fails it
• Unset-field guard: a third test fails on any label that reads a field with a default, since an unset field prints nothing. The old {body.credential_info} label on create_credential fails it
• Secret guard: a second test walks every request model a ProxyClient label can print, nested ones included, and fails on any secret-named field that isn't repr=False. Unhiding wandb_api_key fails it
• Masking: masking after the 200-character cut, or replacing a shorter secret before a longer one that contains it, each fail a test
• Failed-phase report: a spy plugin reads the failed call report of an mcp_oauth_live test and of a plain one, and both must carry the steps. Attaching the steps after the oauth snapshot again fails it
• Mutations: recording the label after the call, keeping old steps on re-attach, skipping the setup reset, attaching only after call, and putting steps ahead of the fixed prefix each fail the suite. Attaching in teardown as well survives, because pytest has already copied user_properties into the teardown report by then, so it never reaches the XML
• Live e2e tests need a deployed proxy and real provider keys, so they were not run here.

This is safe to merge on its own. The step properties are ignored by the current emitter, and BerriAI/project-releaser#252 regroups them into the results JSON's steps array. It is part of the e2e test-metadata rollout; see the rollout plan.

---

Note

Medium Risk
Changes pytest reporting hooks and published JUnit metadata; incorrect attach timing or masking could mislead triage or leak secrets, though the PR adds broad guards and tests for both.

Overview
Adds runtime step recording for e2e harness calls via a new @step decorator and StepRecorder in e2e_metadata.py. Labels can use argument placeholders, dedupe consecutive repeats, cap at 50 steps, mask env secrets, and only record the outermost call in nested helpers.

ProxyClient (and QuotaClient.chat) are annotated so each test’s story becomes repeated <property name="step"> entries in JUnit XML. conftest.py clears the log at setup and attaches steps after setup/call (not teardown), via attach_step_properties in junit_properties.py. Request models mark sensitive fields repr=False so templates cannot leak credentials.

CI runs new test_e2e_metadata.py (recorder edge cases) and test_e2e_junit_report.py (real pytest + junitxml, including xdist/reruns). tests/e2e/AGENTS.md documents how to write labels and how steps flow to reports.

^{Reviewed by Cursor Bugbot for commit 098c188. Bugbot is set up for automated code reviews on this repo. Configure here.}

[greptile-apps]

Confidence Score: 5/5

[Medium risk] Adds test step recording to the e2e test harness.

The PR appears safe to merge based on the reviewed changes.

Summary

The PR records e2e harness actions as per-test JUnit step properties, adds secret-aware label formatting, and tests report timing and recorder behavior. The changes since the previous review refine a credential label, add a required-field guard, and make failed-phase reports carry their steps.

_{Reviews (5) · Last reviewed commit: "fix(e2e): name the saved credential in i..."}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[ryan-crabbe-berri]

@greptileai

[ryan-crabbe-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[ryan-crabbe-berri]

@greptileai re review

[ryan-crabbe-berri]

@greptileai re review

[ryan-crabbe-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[ryan-crabbe-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[ryan-crabbe-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 098c188. Configure here.}

[ryan-crabbe-berri]

@greptileai re review