test(e2e): datadog log delivery for streamed routes, read back from the real datadog api

[yucheng-berri]

Relevant issues

Second of the two sink conversions from Mubashir's review on #33415 (comment): destination fakes cannot be deployed on the cluster, so DataDog tests must use the real provider. #33604 (merged) converted the non-streaming tests; this PR carries the streaming scenarios, rewritten from their original dd-sink form to the same real-DataDog read-back, plus fixes for three read-back defects in the merged #33604 that only running against real DataDog could expose.

Linear ticket

Follow-up of the merged #33415 / #33604 (LIT-3787 e2e program). The red pin references LIT-4447.

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Verified live against real DataDog (us5), full suite of 6:

PASSED test_chat_completions_emits_one_log_event
FAILED test_messages_emits_one_log_event        <- the known LIT-4447 red pin (see below)
PASSED test_responses_emits_one_log_event
PASSED test_chat_completions_stream_emits_one_log_event
PASSED test_messages_stream_emits_one_log_event
PASSED test_responses_stream_emits_one_log_event
=================== 1 failed, 5 passed in 334.08s ===================

• Every green test validated the indexed event end-to-end: source:litellm tag, DataDog ok status, and StandardLoggingPayload fields under attributes — streamed calls additionally pinned stream: true, the aggregated token count equal to the /spend/logs row, and response_cost equal to the spend row (math.isclose rel_tol=1e-9).
• The one red is the intentional LIT-4447 pin: one non-streaming /v1/messages call produced THREE indexed success events with the same request id. Proxy DEBUG logs attribute them: the async DataDog queue flushed once (t+5s) while the sync success-sweep ran twice (t+1s); the sweep re-fires the DataDog logger because _is_sync_litellm_request misclassifies the anthropic call (what fix(logging): classify async anthropic_messages and generate_content as async #33589 fixes). What opens the sweep in this harness config is still being traced — it is NOT the Router callback (that name is filtered as internal). NOTE: the duplicate is race-dependent (an identical earlier run logged once), so this pin flickers red/green until fix(logging): classify async anthropic_messages and generate_content as async #33589 lands.

Static gates: make lint-e2e-basedpyright 0 errors; coverage_registry.collector --strict passes; 6 tests collect.

Type

✅ Test

Changes

1. Read-back fixes for the merged #33604 (datadog_reader.py, e2e_config.py) — live verification found the merged read-back broken three ways, all invisible to the old local sink:

• Marker search found nothing: DataDog consumes the shipped JSON message into the event's attributes and leaves the indexed message empty, so the full-text "marker" query matched zero events and every test failed at the poll deadline. The query is now *:*marker*, which scans all attributes (the marker sits in messages.content).
• Rate limit hard-fail: the Logs Search API budget is 2 requests per 10s org-wide (x-ratelimit-name: logs_public_search_api). Polling at POLL_INTERVAL=5s sat exactly at the limit and the reader hard-failed on the first 429. Searches now pace at DD_SEARCH_INTERVAL (10s default, env-overridable) and a 429 backs off and retries up to 5 times; only non-429 failures remain hard fails.
• Envelope status: DataDog re-derives the indexed event status from the parsed payload's status: "success" attribute and normalizes it to its OK severity — the assertion now expects "ok", not the shipped "info".

2. Streaming scenarios (test_datadog_log_e2e.py, registry) — the original dd-sink version of this PR, rewritten onto the real-DataDog reader:

• One STREAMED call per route (/chat/completions, /v1/messages, /v1/responses); each asserts the response actually streamed (event-stream content type, >0 chunks, no upstream error event), then pins exactly one DataDog event whose payload records stream: true.
• Cost anchor: a stream's headers ship before its cost exists, so the streamed tests anchor response_cost to the /spend/logs row for the call instead of the response header, and additionally require the spend row and the DataDog event to agree on total_tokens. _assert_exactly_one_event takes an explicit cost_anchor (header for non-streaming, spend row for streams) and returns the validated payload.
• Coverage registry: adds logging.datadog.stream.exports_metric exercised on chat_completions, messages, and responses.

Behavior changes

None; test-only changes under tests/e2e. Based on litellm_internal_staging (the e2e staging sync #33613 carried the real-DataDog reader over).

QA runbook

Prereqs for every item: export DD_API_KEY + DD_APP_KEY (Logs Read Data scope, from the secret manager), confirm the proxy env has DD_API_KEY/DD_SITE and NO DD_BASE_URL, and confirm DataDogLogger in success_callbacks via GET /health/readiness/details.

• tests/e2e/logging/test_datadog_log_e2e.py::TestDataDogLogDelivery::test_chat_completions_stream_emits_one_log_event — one successful STREAMED /chat/completions call reaches real DataDog as exactly one event with stream=true, aggregated tokens, and cost equal to the spend row

  • POST /key/generate with {"models":["claude-haiku-4-5"],"key_alias":"dd-stream-chat-<uniq>"} and save the key
  • POST /chat/completions with stream: true, max_tokens: 16, and a unique phrase; confirm an event-stream response with >0 chunks and no error event
  • GET /spend/logs for the key (poll until the row lands) and note spend and total_tokens
  • Search the unique phrase in DataDog Log Explorer (attribute search, e.g. *:*<phrase>*) and wait for ingestion
  • Confirm exactly ONE event; tag source:litellm; status ok
  • Confirm attributes show model_group=claude-haiku-4-5, call_type=acompletion, status=success, stream=true, total_tokens equal to the spend row, response_cost equal to the spend row's spend
  • POST /key/delete to clean up

• tests/e2e/logging/test_datadog_log_e2e.py::TestDataDogLogDelivery::test_messages_stream_emits_one_log_event — one successful STREAMED /v1/messages call reaches real DataDog as exactly one event (the streamed path is clean; only the non-streaming path double-logs per LIT-4447)

  • POST /key/generate with {"models":["claude-haiku-4-5"],"key_alias":"dd-stream-messages-<uniq>"} and save the key
  • POST /v1/messages with stream: true, max_tokens: 16, and a unique phrase; confirm an event-stream response with >0 chunks and no error event
  • GET /spend/logs for the key and note spend and total_tokens
  • Search the unique phrase in DataDog Log Explorer and wait for ingestion
  • Confirm exactly ONE event; tag source:litellm; status ok
  • Confirm attributes show model_group=claude-haiku-4-5, call_type=anthropic_messages, status=success, stream=true, total_tokens equal to the spend row, response_cost equal to the spend row's spend
  • POST /key/delete to clean up

• tests/e2e/logging/test_datadog_log_e2e.py::TestDataDogLogDelivery::test_responses_stream_emits_one_log_event — one successful STREAMED /v1/responses call reaches real DataDog as exactly one event with stream=true, aggregated tokens, and cost equal to the spend row

  • POST /key/generate with {"models":["gpt-5.5"],"key_alias":"dd-stream-responses-<uniq>"} and save the key
  • POST /v1/responses with stream: true and a unique phrase; confirm an event-stream response with >0 chunks and no error event
  • GET /spend/logs for the key and note spend and total_tokens
  • Search the unique phrase in DataDog Log Explorer and wait for ingestion
  • Confirm exactly ONE event; tag source:litellm; status ok
  • Confirm attributes show model_group=gpt-5.5, call_type=aresponses, status=success, stream=true, total_tokens equal to the spend row, response_cost equal to the spend row's spend
  • POST /key/delete to clean up

• Read-back fixes (regression check on the three merged non-streaming tests): run the non-streaming trio and confirm chat_completions and responses pass with events actually FOUND (the pre-fix reader returned zero events for every call), no 429 hard-fails during polling, and the messages test red-pins LIT-4447 when the duplicate race fires (2-3 events, same request id).

Final attestation: "I agree this test is testing what the writer wanted to test."

[greptile-apps]

Greptile Summary

This PR adds three e2e streaming tests for DataDog log delivery (chat completions, messages, and responses) and refactors _assert_exactly_one_event to accept a generic cost_anchor: float instead of a StreamingResponse, enabling both header-based (non-streaming) and spend-row-based (streaming) cost cross-checks without duplicating logic.

• _assert_exactly_one_event now returns _DdMessagePayload and gains an expect_stream flag; all existing non-streaming tests thread through the refactor unchanged, with their pre-call response_cost guard extracted to the call site.
• Each new streaming test asserts: the response is a real event stream with no in-stream error, a positive spend row exists, exactly one DD intake event carries the correct model_group, call_type, stream: true, and response_cost, and the DD event's total_tokens matches the spend row's token count exactly.
• A new logging.datadog.stream.exports_metric P0 coverage row is added to logging.yaml, correctly listing all three exercised surfaces.

Confidence Score: 5/5

Test-only change with no production code modifications; safe to merge.

The change is entirely confined to the e2e test directory. The refactoring of _assert_exactly_one_event preserves every assertion from the original — the response_cost > 0 guard is moved to the call site rather than dropped, and the < 1e-12 tolerance equality is kept. The three new streaming tests cross-check cost and tokens against the spend row with explicit is not None guards before comparisons. No production code is touched.

No files require special attention.

Important Files Changed

Filename	Overview
tests/e2e/coverage_registry/logging.yaml	Adds a single new P0 registry row for the logging.datadog.stream.exports_metric coverage item, correctly referencing all three streaming surfaces (chat_completions, messages, responses).
tests/e2e/logging/test_datadog_log_e2e.py	Refactors _assert_exactly_one_event to accept a cost_anchor: float instead of outcome: StreamingResponse, returns _DdMessagePayload, and adds expect_stream flag; adds three new streaming e2e tests with proper spend-row cross-checks and total_tokens guards. Existing non-streaming test coverage is preserved intact.

_{Reviews (3): Last reviewed commit: "test(e2e): guard the spend-row token cou..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_e2e_datadog_stream_log_delivery (bb130b9) with litellm_internal_staging (224fe67)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (ac29a6a) during the generation of this report, so 224fe67 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[yucheng-berri]

@greptileai

[yucheng-berri]

@greptileai

[yucheng-berri]

@BugBot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 12bb3d6. Configure here.}