test(e2e): otel trace completeness on /chat/completions

[yucheng-berri]

Relevant issues

Linear ticket

https://linear.app/litellm-ai/issue/LIT-3787

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

This PR is the first of a 3-PR stack (chat_completions -> messages -> responses, one LIT-3787 scenario each); the two follow-ups are based on this branch.

Verification was run on 2026-07-13 at the stack tip f354f98, which contains all three tests including this PR's chat test. This machine has no docker buildx, so the two proxies under test ran from source at exact git refs (uv run --extra proxy --extra extra_proxy python litellm/proxy/proxy_cli.py); same code paths, no image build. Outputs are pasted verbatim.

1. Pass on current code: bring up the tests/e2e compose stack (which this PR extends with a jaeger service and the arize_phoenix OTEL v2 destination pointed at it) and run the suite:

   $ cd tests/e2e && docker compose -p e2e_otel_pr1 up -d
   $ curl -fs http://localhost:4000/health/liveliness
   "I'm alive!"
   $ LITELLM_PROXY_URL=http://localhost:4000 uv run pytest logging/test_otel_trace_e2e.py -v
   logging/test_otel_trace_e2e.py::TestOtelTraceCompleteness::test_chat_completions_exports_complete_trace PASSED
   logging/test_otel_trace_e2e.py::TestOtelTraceCompleteness::test_messages_exports_complete_trace PASSED
   logging/test_otel_trace_e2e.py::TestOtelTraceCompleteness::test_responses_exports_complete_trace PASSED
   ============================== 3 passed in 19.43s ==============================
   

   The settled chat trace at the Jaeger destination, fetched through its query API by the litellm.call_id span tag:

   - [server] POST /chat/completions
     - [internal] auth /chat/completions
       - [client] postgres get_data
     - [client] chat claude-haiku-4-5
     - [client] batch_write_to_db _PROXY_track_cost_callback
     - [client] postgres get_key_object
   

2. Fail-before-fix: the same tests against a proxy built from 1bd603d, the parent of fix(otel): one v2 logger owns the global provider; scope tenant OTLP creds per exporter #30590 ("one v2 logger owns the global provider"), the commit that fixed server spans exporting through a different provider than the preset's gen-AI spans. DB on a fresh port, PHOENIX_COLLECTOR_HTTP_ENDPOINT at a second throwaway Jaeger:

   $ LITELLM_PROXY_URL=http://localhost:4611 E2E_OTEL_QUERY_URL=http://localhost:16690 \
       uv run pytest logging/test_otel_trace_e2e.py
   E  AssertionError: span(s) ['chat claude-haiku-4-5'] reference a parent that never
      reached the destination (orphaned trace); spans present: ['chat claude-haiku-4-5']
   E  AssertionError: span(s) ['chat claude-haiku-4-5'] reference a parent that never
      reached the destination (orphaned trace); spans present: ['chat claude-haiku-4-5']
   E  AssertionError: span(s) ['chat gpt-5.5'] reference a parent that never
      reached the destination (orphaned trace); spans present: ['chat gpt-5.5']
   ======================== 3 failed in 364.91s (0:06:04) =========================
   

   Exactly the LIT-3787 failure mode: the destination received each gen-AI span alone, dangling. A rubber-stamp test would have passed here.

3. Static gates at the same commit:

   $ make lint-e2e-basedpyright
   0 errors, 0 warnings, 0 notes
   $ cd tests/e2e && uv run python -m coverage_registry.collector --strict
   Logging & Guardrails                  5/51        9.8%
   

   Logging & Guardrails moves 4/51 -> 5/51 (the logging.otel.success.exports_metric P0 cell gains its first covering test).

Before:

After:

Type

✅ Test

Changes

The logging.otel.success.exports_metric registry cell is a P0 row with no covering test: nothing proved that a successful call actually lands at an OTEL destination as one COMPLETE trace (LIT-3787). The regression class is real; #30590 fixed the proxy publishing the global TracerProvider before callback init, which orphaned every gen-AI span on the preset backend.

• docker-compose.yml: adds a throwaway jaeger service (OTLP ingest in-network, query API on host 16686, healthchecked on its admin port so the depends_on condition is accurate) and configures the proxy with LITELLM_OTEL_V2=true + callbacks: ["arize_phoenix"] + PHOENIX_COLLECTOR_HTTP_ENDPOINT=http://jaeger:4318/v1/traces. The arize_phoenix preset is used deliberately: the orphan bug lives on the preset-owned-provider path; a generic callbacks: ["otel"] with OTEL_* envs would send both providers to the same endpoint and could never fail. No cloud credentials involved.
• logging/otel_client.py (new): typed models over the Jaeger query API. Traces are fetched server-side by the litellm.call_id span tag (the request's x-litellm-call-id response header), so read-back is immune to the query page filling with unrelated traffic; the stack's own background jobs alone can push a request trace past a recent-traces page. A failed query hard-fails instead of reading as "the trace never arrived", and the settle predicate waits for every span name the assertion demands, including the prefix-matched db span (poll to the shared deadline, never sleep-once).
• logging/test_otel_trace_e2e.py (new): test_chat_completions_exports_complete_trace, asserting both halves: recorded state (/health/readiness/details reports the OpenTelemetryV2 callback active) and enforced behavior (exactly ONE trace for the call's call id; single root SERVER span POST /chat/completions; auth/postgres/batch_write_to_db _PROXY_track_cost_callback children; gen-AI CLIENT span chat <model> whose CHILD_OF parent chain reaches the root; no span referencing a parent absent from the trace). First call retries on 401 to a deadline (fresh-key propagation; a 401 never reaches the LLM so it exports no gen-AI span).
• e2e_config.py: OTEL_QUERY_URL (env E2E_OTEL_QUERY_URL) and CHEAP_ANTHROPIC_MODEL (env E2E_CHEAP_ANTHROPIC_MODEL); the latter is the identical line test(e2e): cover key rpm/tpm rate limiting, window reset, and pacing headers #32914 adds, so whichever merges second is a trivial dedupe.

Known deferral: the registry row also lists embeddings in exercised_on with no covering test yet; the responses surface is added by the top PR of this stack, and the embeddings surface stays a tracked follow-up under LIT-3787 rather than silently dropped from the row.

Behavior changes

None to the product; all changes are under tests/e2e. The shared compose stack gains a jaeger container and a globally active OTEL v2 callback (extra span export only; no request behavior change for other suites).

QA runbook

• tests/e2e/logging/test_otel_trace_e2e.py::TestOtelTraceCompleteness::test_chat_completions_exports_complete_trace - one successful chat call shows up at the OTEL destination as one connected trace tree
  • GET /health/readiness/details with the master key and confirm OpenTelemetryV2 appears in success_callbacks
  • POST /key/generate with {"models":["claude-haiku-4-5"],"key_alias":"otel-trace-chat-<uniq>"} and save the returned key
  • POST /chat/completions (non-streaming, max_tokens: 16, a unique phrase in the prompt) with the key; if the very first call 401s, retry it for a few seconds (fresh key propagating; a 401 exports no spans); confirm 200 and note the x-litellm-call-id response header
  • Open Jaeger at http://localhost:16686, service litellm, and search by tag litellm.call_id=<that id>; keep refreshing until the trace appears (spans flush in batches; the cost write lands last)
  • Confirm exactly ONE trace holds the call-id; a second trace with the same call-id is the split-trace bug
  • Confirm a single root span POST /chat/completions (kind server) and that no span references a parent missing from the trace
  • Confirm the children: auth /chat/completions, a postgres ... span, and batch_write_to_db _PROXY_track_cost_callback
  • Confirm chat claude-haiku-4-5 (kind client) is in the SAME trace and its parent chain reaches the root
  • POST /key/delete to clean up

Final attestation: "I agree this test is testing what the writer wanted to test."

---

Note

Low Risk
Changes are confined to tests/e2e; compose gains an extra Jaeger service and global OTEL callback export only, with no product runtime behavior changes.

Overview
Adds e2e coverage for registry cell logging.otel.success.exports_metric (LIT-3787): a successful /chat/completions call must export one connected OTEL trace to the destination, not a split tree where the gen-AI span orphans from the server root.

The shared compose stack gains a Jaeger OTLP sink and enables OTEL v2 via the arize_phoenix preset (preset-owned provider path where split-trace bugs show up). otel_client.OtelReader polls Jaeger by litellm.call_id and asserts a single trace with expected auth, postgres, cost, and gen-AI spans parented under POST /chat/completions. test_chat_completions_exports_complete_trace also checks /health/readiness/details lists OpenTelemetryV2 before traffic.

e2e_config adds E2E_OTEL_QUERY_URL and E2E_CHEAP_ANTHROPIC_MODEL; logging conftest wires a session otel_reader fixture. Docs in tests/e2e/CLAUDE.md note OTEL trace read-back under logging/.

^{Reviewed by Cursor Bugbot for commit 6e7af95. Bugbot is set up for automated code reviews on this repo. Configure here.}

[greptile-apps]

Greptile Summary

Adds e2e coverage for registry cell logging.otel.success.exports_metric (LIT-3787): a /chat/completions call must export one connected OTEL trace, not a split tree where the gen-AI span orphans from the server root.

• docker-compose.yml: adds a Jaeger OTLP sink service (healthchecked on its admin port) and enables OTEL v2 via the arize_phoenix preset—the provider path where the split-trace bug actually manifests.
• otel_client.py: new Jaeger read-back client; fetches traces server-side by litellm.call_id tag, polls to a deadline with span-name and prefix settling, and hard-fails on unreachable destination.
• test_otel_trace_e2e.py: test_chat_completions_exports_complete_trace asserts /health/readiness/details reports OpenTelemetryV2 active, then verifies exactly one trace for the call-id with a single server root, no dangling parent references, and the gen-AI span's parent chain reaching the root.

Confidence Score: 5/5

All changes are confined to tests/e2e; the compose stack gains an extra Jaeger container and a globally-active OTEL callback export with no changes to production code paths.

The test logic is correct and well-structured: the dangling-span check catches the orphaned-trace variant of LIT-3787, the len(hits) == 1 check catches the split-trace variant, and the gen-AI span chain-walk catches a connected-but-misparented tree. The Jaeger healthcheck probes the admin port before the proxy is allowed to start, _params serialization correctly passes the JSON-encoded tags string to the Jaeger query API, and no production files are touched.

No files require special attention.

Important Files Changed

Filename	Overview
tests/e2e/docker-compose.yml	Adds Jaeger all-in-one service as OTLP sink, enables OTEL v2 + arize_phoenix callback on the proxy, and adds depends_on: jaeger: condition: service_healthy; healthcheck probes the admin port (14269) which is accurate.
tests/e2e/e2e_config.py	Adds CHEAP_ANTHROPIC_MODEL and OTEL_QUERY_URL env-backed constants; clean additions that follow existing patterns.
tests/e2e/logging/conftest.py	Adds session-scoped otel_reader fixture backed by build_otel_reader(); follows existing conftest patterns exactly.
tests/e2e/logging/otel_client.py	New Jaeger read-back client: typed Pydantic models over the Jaeger query API, poll-to-deadline with settled-names/prefixes logic, and hard-fail on unreachable destination. Logic is correct; _params in e2e_http correctly serializes the _TracesQuery pydantic model including the JSON-encoded tags string.
tests/e2e/logging/test_otel_trace_e2e.py	New test asserting split-trace regression: verifies exactly one trace per call-id, single server root, no dangling spans, and gen-AI span's parent chain reaches the root. The dangling check correctly catches orphaned spans (the LIT-3787 failure mode) while the len(hits) == 1 check catches the split-trace variant.

_{Reviews (3): Last reviewed commit: "Update logging section in CLAUDE.md" | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_e2e_otel_chat_trace (6356734) with litellm_internal_staging (397c846)}

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 6e7af95. Configure here.}

[mateo-berri]

LGTM; thanks!

[mateo-berri]

q: is there a specific reason we have to add this additional message?

[yucheng-berri]

addressed

[mateo-berri]

I appreciate you testing it works on latest changes and fails on the parent of the fix PR commit

[yucheng-berri]

@greptileai

[ishaan-berri]

shouldn't we have a JSON of all the fields we expect in OTEL to be logged ?

[ishaan-berri]

are we testing stream too ?

[yucheng-berri]

are we testing stream too ?

not in this PR, streaming uses a different span lifecycle, I’ll track it as a separate otel e2e test suite with its own fail-before-fix

[yucheng-berri]

shouldn't we have a JSON of all the fields we expect in OTEL to be logged ?

this is intentionally scoped to trace structure, since a full otel json snapshot would be too brittle I think.

we can cover field values such as usage and cost separately.