refactor(e2e): replace bespoke result reporter with standard JUnit report

[yassin-berriai]

Relevant issues

Linear ticket

Resolves LIT-4552

Pre-Submission checklist

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

This is harness-internal reporting plumbing with no LLM or proxy behavior change, so there is no before/after model call; the proof is that a real pytest run now produces a standard JUnit report carrying the two custom signals per test on every outcome (pass, fail, skip, setup error). Captured at dadf0fb131:

$ pytest logging -o addopts= --junitxml=e2e-report.xml -q
1 failed, 2 passed, 1 skipped, 1 error in 0.01s

$ cat e2e-report.xml
<testsuite name="pytest" errors="1" failures="1" skipped="1" tests="5" ...>
  <testcase classname="logging.test_langfuse_e2e" name="test_pass" time="0.000">
    <properties>
      <property name="package" value="logging"/>
      <property name="covers" value="logging.langfuse.success.logs_spend"/>
    </properties>
  </testcase>
  <testcase classname="logging.test_langfuse_e2e" name="test_fail" time="0.000">
    <properties>
      <property name="package" value="logging"/>
      <property name="covers" value="logging.s3.success.writes_object"/>
    </properties>
    <failure message="assert False">... assert False ...</failure>
  </testcase>
  <testcase classname="logging.test_langfuse_e2e" name="test_skipped" time="0.000">
    <properties>
      <property name="package" value="logging"/>
      <property name="covers" value="logging.datadog.success.logs_spend"/>
    </properties>
    <skipped type="pytest.skip" message="demo">...</skipped>
  </testcase>
  <testcase classname="logging.test_langfuse_e2e" name="test_setup_error" time="0.000">
    <properties>
      <property name="package" value="logging"/>
      <property name="covers" value="logging.otel.success.exports_metric"/>
    </properties>
    <error message='failed on setup with "RuntimeError: setup boom"'>...</error>
  </testcase>
  <testcase classname="logging.test_langfuse_e2e" name="test_harness_only" time="0.000">
    <properties>
      <property name="package" value="logging"/>
      <property name="covers" value=""/>
    </properties>
  </testcase>
</testsuite>

Every <testcase> carries package and covers <property> entries on all four outcomes: pass is a bare <testcase>, fail carries a standard <failure>, skip carries <skipped>, and a fixture setup error carries <error>; the doubled covers marker is deduped to one id, and a test with no marker gets an empty covers

Type

🧹 Refactoring

Changes

tests/e2e/e2e_result_reporter.py hand-rolled a per-test logfmt emitter: it reimplemented outcome mapping, logfmt escaping, and node-id parsing to print one E2E_RESULT line per finished test, which a downstream Loki pipeline scraped from pod stdout. Outcome, duration, and node id are all things a standard pytest reporter already produces, so the only genuinely custom data was the covers marker ids and the normalized package label

This deletes the bespoke module and switches to pytest's built-in JUnit XML report (--junitxml). The two custom signals ride along as user_properties, which JUnit records as <property> entries per testcase. They are attached at collection time in conftest.py::pytest_collection_modifyitems, so they land on every test on every outcome, including skips and setup errors, matching what the old pytest_runtest_makereport hook did. The small package normalization and covers extraction live in junit_properties.py; nothing hand-rolls serialization, outcome mapping, or escaping anymore

JUnit XML is built into pytest, so this adds no dependency and no uv.lock change; pytest-reportlog (JSON-lines) would be the nicer Loki fit but needs a new pinned dist, which uv sync --frozen in CI rejects without a lock regen, so it is a clean follow-up rather than part of this change

The E2E_RESULT logfmt line was consumed only by an external Loki/Grafana pipeline that scraped it from pod stdout; nothing in this repo reads it, and no CI job runs tests/e2e. JUnit XML is not line-based, so shipping it to Loki is a thin converter in the e2e job (walk the XML, print one logfmt line per <testcase> to stdout for the existing scrape); that transform is infra-side and lives outside this repo

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/50af8d97faf642cf836f39016cd70667
Requested by: @yassin-berriai

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

This PR replaces the hand-rolled E2E_RESULT logfmt emitter (e2e_result_reporter.py) with pytest's built-in JUnit XML reporter, attaching the two custom signals — normalized package and covers cell ids — as <property> entries via junit_properties.attach_result_properties in pytest_collection_modifyitems.

• junit_properties.py provides idempotent property attachment (no-ops on a second call), order-preserving covers deduplication via dict.fromkeys, and a package_from_nodeid normalizer that handles both repo-root and suite-cwd invocations.
• test_junit_properties.py covers all four outcome paths (pass, fail, skip, setup error) end-to-end via subprocess, verifying property presence and idempotency. Both previously flagged gaps (skipped outcome and duplicate-property guard) are now exercised by concrete assertions.
• Grafana docs are updated to describe the JUnit artifact and the required infra-side XML-to-logfmt converter; existing LogQL queries remain valid for when that converter ships.

Confidence Score: 5/5

Safe to merge — changes are confined to the e2e test harness, add no new dependencies, and touch no production or proxy code.

The refactor deletes ~150 lines of hand-rolled serialization and replaces them with a 59-line module that delegates entirely to pytest's built-in JUnit reporter. All custom logic (package normalization, covers deduplication, idempotency guard) is unit-tested, and the end-to-end subprocess test verifies all four outcome paths. No production paths are touched.

No files require special attention.

Important Files Changed

Filename	Overview
tests/e2e/junit_properties.py	New module replacing the bespoke logfmt emitter; provides idempotent attach_result_properties, clean package normalization, and order-preserving covers deduplication via dict.fromkeys — all correct.
tests/e2e/conftest.py	Replaced pytest_runtest_makereport with pytest_collection_modifyitems; delegates to attach_result_properties (idempotent). Unused Generator import and e2e_result_reporter imports cleanly removed.
tests/e2e/e2e_result_reporter.py	Deleted; hand-rolled logfmt serialization, outcome mapping, and node-id parsing all superseded by the standard JUnit reporter plus junit_properties.py.
tests/e2e/test_junit_properties.py	Comprehensive test file: unit tests for package_from_nodeid and dedupe_covers, plus subprocess end-to-end checks for pass/fail/skip/setup-error outcomes and idempotency; all four previously flagged gaps are now covered.
tests/e2e/grafana/status_history_panels.md	Updated from logfmt-line artifact to JUnit XML artifact; documents property schema, outcome mapping, and the required infra-side XML-to-logfmt converter; existing LogQL queries left intact for when the converter ships.

_{Reviews (2): Last reviewed commit: "refactor(e2e): replace bespoke result re..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[yassin-berriai]

Addressed both findings in dadf0fb:

• Idempotency: pytest_collection_modifyitems now delegates to junit_properties.attach_result_properties, which no-ops if a package property is already present, so a hook that fires more than once never emits duplicate entries. Covered by a new test that runs the collection hook twice and asserts exactly one package and one covers property per testcase
• Outcome coverage: the end-to-end JUnit test now also exercises a skipped test and a fixture setup error, asserting both carry package + covers and that the report records and respectively, matching the on-every-outcome claim

@greptileai please review the current head dadf0fb

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_e2e_standard_junit_reporter (3b4d749) with litellm_internal_staging (442fdc1)}