fix(e2e): assert on the gen-AI span that served the stream, not the span count

[yassin-berriai]

TLDR

Problem this solves:

• Four otel e2e tests fail whenever the router retries
• A retried call is a success, but the test calls it a failure
• Currently red in the nightly, intermittently

How it solves it:

• Assert on the attempt that served the stream
• Ignore the error span a failed attempt leaves behind
• Keep failing when one call logs two served spans

User Flow

This PR changes only tests/e2e, so no end user's behavior changes. The flow below is the engineer reading the nightly e2e report.

Before: a nightly run goes red on a request the gateway served correctly, so the report says the otel integration is broken when it is not

1. The suite sends POST https://litellm-domain/v1/messages with "stream": true against claude-haiku-4-5
2. The gateway's first upstream attempt fails, it retries, and the second attempt streams the answer back with a 200
3. The suite reads the trace back from the trace store and finds two chat claude-haiku-4-5 spans, one per attempt
4. The run fails with "a streamed call must produce exactly ONE gen-AI span, got 2", naming a request that actually succeeded

After: the same run is green, and still red for the problem these tests were written to catch

1. The suite sends the same POST https://litellm-domain/v1/messages with "stream": true
2. The gateway retries the same way and returns the same 200
3. The suite reads the trace back and looks at the attempt that served the stream, ignoring the failed attempt's error span
4. The run passes, and the time-to-first-token assertions run against the attempt that actually streamed chunks

Relevant issues

Linear ticket

Resolves LIT-5441

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Both legs ran against the same live gateway from a pod built from the nightly's own e2e image, same four tests, same command, back to back.

Before, at base commit bea31871fc (the file straight out of git show HEAD:..., asserted to not contain the fix before running):

$ git show HEAD:tests/e2e/logging/test_otel_trace_e2e.py > base_otel_test.py
$ grep -c 'exactly ONE gen-AI span' base_otel_test.py
4
$ for i in 1 2 3; do pytest logging/test_otel_trace_e2e.py \
    -k "stream_exports_complete_trace or stream_records_real_ttft"; done

E  AssertionError: a streamed call must produce exactly ONE gen-AI span, got 2; spans:
   ['POST /v1/messages', 'auth /v1/messages', 'batch_write_to_db _PROXY_track_cost_callback',
    'chat claude-haiku-4-5', 'chat claude-haiku-4-5', 'postgres get_data', ...]
FAILED logging/test_otel_trace_e2e.py::TestOtelTraceCompleteness::test_messages_stream_records_real_ttft
1 failed, 5 passed, 4 deselected in 48.82s
6 passed, 4 deselected in 36.91s
6 passed, 4 deselected in 47.01s

That is the nightly's failure reproduced, and rounds 2 and 3 passing is the intermittency.

Reading that trace out of the trace store shows the two spans are one request, not a split trace:

op=chat claude-haiku-4-5  start=...176812  dur=150.0ms  status=ERROR  ttft=None
    error.type=AuthenticationError
    error.message=AnthropicException ... {"error":{"type":"authentication_error",
                  "message":"invalid x-api-key"}}
op=chat claude-haiku-4-5  start=...408760  dur=566.3ms  status=None   ttft=0.5199649333953857
    gen_ai.response.id=chatcmpl-be080786-4d5e-4b07-8d97-8af980ec6454

After, same gateway, same command, fifteen rounds: nine at c7f9da4e5e and six more at c881b1426f, which renamed two helpers so the harness type gate would accept the cross-module import and changed nothing else. The staged file is greped first so an after leg cannot silently be running the base code:

$ grep -c one_served_genai_span logging/test_otel_trace_e2e.py
5
$ for i in $(seq 9); do pytest logging/test_otel_trace_e2e.py \
    -k "stream_exports_complete_trace or stream_records_real_ttft"; done
6 passed, 4 deselected in 37.79s
6 passed, 4 deselected in 48.90s
6 passed, 4 deselected in 37.94s
6 passed, 4 deselected in 43.42s
6 passed, 4 deselected in 38.50s
6 passed, 4 deselected in 44.97s
6 passed, 4 deselected in 37.98s
6 passed, 4 deselected in 46.94s
6 passed, 4 deselected in 38.78s

# after the rename, at c881b1426f
6 passed, 4 deselected in 39.12s
6 passed, 4 deselected in 38.74s
6 passed, 4 deselected in 48.59s
6 passed, 4 deselected in 42.79s
6 passed, 4 deselected in 38.22s
6 passed, 4 deselected in 41.47s

90 passes with no failure, against a 17.5% per-call retry rate on this gateway.

Mutation check on the new selection logic, each mutant applied and run separately, every mutation step self-verifying that it changed the source:

M1: stop excluding failed attempts   -> 3 failed, 3 passed   KILLED
M2: weaken the count to at-least-one -> 1 failed, 5 passed   KILLED
M3: wrong-case error status          -> 3 failed, 3 passed   KILLED
restore                              -> 6 passed

Type

🐛 Bug Fix
✅ Test

Caveats (if any)

• The retry is stochastic, so the before leg needs a few rounds
• Retries here come from broken leaked deployments, see LIT-5442
• A plain upstream 429 trips the same assertion, so this fix stands alone

QA runbook

• tests/e2e/logging/test_otel_trace_e2e.py::TestOtelTraceCompleteness::test_messages_stream_records_real_ttft - a streamed /v1/messages call records a real time-to-first-token on the attempt that served it, even when an earlier attempt failed

  • Point a proxy at two deployments of one model name, one with a valid provider key and one with a deliberately invalid key, and set num_retries: 3
  • Send POST /v1/messages with "stream": true, repeating until the response 200s having burned an attempt on the bad deployment
  • Read the trace for that call id back from the trace store and expect two chat <model> spans, one with otel.status_code=ERROR and no TTFT, one with a TTFT between 0 and its own duration
  • Expect the test to pass against that trace, and to fail if the served span's TTFT is missing or exceeds the span duration
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/logging/test_span_selection.py - the span selection itself, over trace payloads, with no proxy involved

  • Run it with no proxy up and expect 6 passed, since it carries no e2e marker
  • Delete the otel.status_code filter and expect 3 failures
  • Change the survivor count from == 1 to >= 1 and expect test_two_served_spans_still_fail to fail
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

The current head updates streaming OTEL e2e assertions to select the non-error gen-AI span that served a retried request rather than requiring one raw attempt span.

• Adds reusable helpers for selecting exactly one served gen-AI span.
• Applies that selection to streaming-flag and time-to-first-token assertions.
• Adds focused coverage for successful, retried, duplicated, failed, and unrelated spans.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
tests/e2e/logging/test_otel_trace_e2e.py	Replaces raw gen-AI span-count assertions with served-attempt selection across streaming trace and TTFT checks; no eligible follow-up defect was established.
tests/e2e/logging/test_span_selection.py	Adds focused fixture-based coverage for the span-selection helper, including retries and duplicate served spans; no eligible follow-up defect was established.

_{Reviews (2): Last reviewed commit: "fix(e2e): assert on the gen-AI span that..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[yassin-berriai]

@greptileai the 5/5 was against c7f9da4, which a force-push replaced. Please re-review the current head c881b14