test(e2e): datadog log delivery for successful chat, messages, and responses (LIT-4447)

[yucheng-berri]

Relevant issues

Related: LIT-4447 (product bug found while writing these tests: /v1/messages double-logs to DataDog)

Linear ticket

https://linear.app/litellm-ai/issue/LIT-4447 (tracks the one assertion the messages test knowingly relaxes)

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Based directly on litellm_internal_staging (#33304, which this work built on, merged while this PR was being opened). Verification run on 2026-07-15 against the compose stack on the default models; outputs verbatim.

1. Evidence-first probe before assertions were written: one chat call, then the sink's recorded intake dumped. Exactly one event, and every scenario field is present with exact cross-checks available (payload total_tokens 23 equals the response usage; payload response_cost 3.9e-05 equals the x-litellm-response-cost header byte for byte):

   call ok; cost_header=3.9e-05 usage=23
   matching events: 1
   envelope: {'ddsource': 'litellm', 'service': 'litellm-server', 'status': 'info', ...}
   message payload: {"model_group": "claude-haiku-4-5", "total_tokens": 23, "prompt_tokens": 19,
                     "completion_tokens": 4, "response_cost": 3.9e-05, "status": "success",
                     "call_type": "acompletion"}
   

2. The strict exactly-one assertion caught a live product bug on its first run: every successful /v1/messages call ships TWO byte-identical events (four out of four calls; chat and responses ship exactly one). Filed as LIT-4447 with the suspected mechanism (DataDogLogger implements both sync and async success hooks with no cross-hook dedupe; the OTEL v2 emitter guards this exact case with a dedupe LRU):

   E  AssertionError: expected exactly ONE DataDog log event for the call, got 2 -
      more than one event for one call is the duplicate-delivery bug
   chatcmpl-80a3c539-... x2 call_type=anthropic_messages byte-identical=True
   chatcmpl-4c88f679-... x2 call_type=anthropic_messages byte-identical=True
   

   That failing run is the mutation-kill evidence for the exactly-one contract: it fires on a real defect on current code. Per review direction all three tests assert the scenario STRICTLY - exactly one event - so the messages test KNOWINGLY FAILS on current code and is the LIT-4447 regression pin; it goes green when the fix lands. A clean forensic repro (two separate intake POSTs, same litellm_call_id, identical tokens/cost/timestamps, differing synthetic completion ids) is recorded on the ticket.

3. Current-code run on default models; chat and responses pass, messages fails on the live LIT-4447 defect by design:

   $ LITELLM_PROXY_URL=http://localhost:4000 uv run pytest logging/test_datadog_log_e2e.py -v
   test_chat_completions_emits_one_log_event PASSED
   test_messages_emits_one_log_event FAILED   (expected: LIT-4447 regression pin)
   test_responses_emits_one_log_event PASSED
   E  AssertionError: expected exactly ONE DataDog log event for the call, got 2 -
      more than one event for one call is the duplicate-delivery bug (see LIT-4447
      for the currently known /v1/messages instance)
   

4. A git-ref foil was evaluated and rejected with cause: the natural candidate (parent of a302b53, "drain datadog batches safely") predates the DD_BASE_URL testing override entirely, so the sink cannot intercept delivery on that ref and any failure there would be trivial rather than meaningful. The live LIT-4447 kill in item 2 demonstrates the core assertion bites instead.

5. Static gates: make lint-e2e-basedpyright -> 0 errors; coverage_registry.collector --strict passes (Logging & Guardrails 7/52 -> 8/52; responses added to the datadog cell's exercised_on).

Type

✅ Test

Changes

The logging.datadog.success.exports_metric registry cell is a P0 row with no covering test: nothing proved a successful call's log event actually reaches the DataDog intake, exactly once, with the fields dashboards bill and alert on.

• docker-compose.yml: adds a dd-sink service (stdlib-Python recorder for the logs intake; gunzips the compressed batches, replays them on GET /requests) and configures the proxy with callbacks: ["arize_phoenix", "datadog"] plus DD_BASE_URL pointed at the sink (DD_API_KEY/DD_SITE are dummies; the integration's documented testing override does the routing). No cloud credentials involved.
• logging/datadog_sink.py (new): typed reader over the sink; accepts both intake shapes (array and single-object batches), matches events by unique prompt marker, re-reads after a settle interval so a late duplicate cannot hide, and hard-fails if the sink is unreachable.
• logging/test_datadog_log_e2e.py (new): three tests asserting both halves - recorded state (/health/readiness/details reports DataDogLogger active) and enforced behavior (exactly one event from ddsource: litellm whose payload carries the exact model_group, the exact call_type per route, total_tokens > 0, status: success, and a response_cost equal to the same response's x-litellm-response-cost header).
• coverage_registry/logging.yaml: responses added to the datadog cell's exercised_on; each test declares its surface.
• e2e_config.py: DD_SINK_URL (env E2E_DD_SINK_URL).

Behavior changes

None; test-only changes under tests/e2e. The shared compose stack gains the dd-sink container and a globally active datadog callback (log shipping to the local sink only).

QA runbook

• tests/e2e/logging/test_datadog_log_e2e.py::TestDataDogLogDelivery::test_chat_completions_emits_one_log_event - one successful chat call lands at the DataDog intake as exactly one event with model, tokens, and cost

  • GET /health/readiness/details with the master key and confirm DataDogLogger appears in the active callbacks
  • POST /key/generate with {"models":["claude-haiku-4-5"],"key_alias":"dd-chat-<uniq>"} and save the returned key
  • POST /chat/completions (non-streaming, max_tokens: 16, a unique phrase in the prompt); retry the first call on 401 for a few seconds; confirm 200 and note the x-litellm-response-cost header
  • GET http://localhost:9915/requests and wait until a recorded /api/v2/logs batch contains an event whose message carries your phrase (the callback flushes in periodic batches)
  • Confirm exactly ONE event carries the phrase, with ddsource: litellm and status: info
  • Parse the event's message and confirm model_group: claude-haiku-4-5, call_type: acompletion, status: success, total_tokens above zero, and response_cost equal to the header you noted
  • POST /key/delete to clean up

• tests/e2e/logging/test_datadog_log_e2e.py::TestDataDogLogDelivery::test_messages_emits_one_log_event - one successful Anthropic-native call lands at the DataDog intake as exactly one event with model, tokens, and cost (currently fails on the LIT-4447 double-log by design; the runbook describes the target state)

  • GET /health/readiness/details with the master key and confirm DataDogLogger appears in the active callbacks
  • POST /key/generate with {"models":["claude-haiku-4-5"],"key_alias":"dd-messages-<uniq>"} and save the returned key
  • POST /v1/messages (non-streaming, max_tokens: 16, a unique phrase); retry the first call on 401 for a few seconds; confirm 200 and note the x-litellm-response-cost header
  • GET http://localhost:9915/requests and wait for events carrying your phrase
  • Confirm exactly ONE event carries the phrase, with ddsource: litellm and status: info (today you will see TWO - that is LIT-4447, and the test fails on it by design until the fix lands)
  • Parse the message and confirm model_group: claude-haiku-4-5, call_type: anthropic_messages, status: success, total_tokens above zero, and response_cost equal to the header
  • POST /key/delete to clean up

• tests/e2e/logging/test_datadog_log_e2e.py::TestDataDogLogDelivery::test_responses_emits_one_log_event - one successful Responses API call lands at the DataDog intake as exactly one event with model, tokens, and cost

  • GET /health/readiness/details with the master key and confirm DataDogLogger appears in the active callbacks
  • POST /key/generate with {"models":["gpt-5.5"],"key_alias":"dd-responses-<uniq>"} and save the returned key
  • POST /v1/responses (non-streaming, max_output_tokens: 64, a unique phrase); retry the first call on 401 for a few seconds; confirm 200 and note the x-litellm-response-cost header
  • GET http://localhost:9915/requests and wait for the event carrying your phrase
  • Confirm exactly ONE event carries the phrase, with ddsource: litellm and status: info
  • Parse the message and confirm model_group: gpt-5.5, call_type: aresponses, status: success, total_tokens above zero, and response_cost equal to the header
  • POST /key/delete to clean up

Final attestation: "I agree this test is testing what the writer wanted to test."

---

Note

Low Risk
Test-only changes under tests/e2e; compose adds a local dd-sink and routes DataDog logs to it via DD_BASE_URL, with no production runtime behavior changes.

Overview
Adds end-to-end coverage for the P0 registry cell logging.datadog.success.exports_metric by proving successful proxy calls ship exactly one DataDog log event with the expected StandardLogging payload.

The compose stack gains a local dd-sink intake recorder (gzip-aware) and enables the datadog callback with DD_BASE_URL pointed at the sink so tests assert on real intake traffic, not cloud credentials. New datadog_sink read-back helpers poll by unique prompt markers and re-read after a settle window to catch duplicate delivery.

test_datadog_log_e2e.py exercises /chat/completions, /v1/messages, and /v1/responses: readiness shows DataDogLogger active, then the sink must show a single ddsource: litellm event whose cost matches x-litellm-response-cost. The messages case is a strict regression pin for LIT-4447 (known double-log) and is expected to fail until the product fix lands.

first_ok is moved to logging_client and reused by OTEL trace tests. The coverage registry adds responses to the datadog success cell’s exercised_on.

^{Reviewed by Cursor Bugbot for commit e7a59ad. Bugbot is set up for automated code reviews on this repo. Configure here.}

[greptile-apps]

Greptile Summary

This PR adds a live e2e test suite verifying that successful calls to /chat/completions, /v1/messages, and /v1/responses each produce exactly one DataDog log event with correct model, token count, and cost fields. It also promotes the _first_ok retry helper into a shared logging_client.first_ok for reuse across the logging suite.

• docker-compose.yml: introduces a stdlib-Python dd-sink service that records compressed intake POSTs and replays them via GET /requests; routes the proxy's DataDog callback to the sink via DD_BASE_URL with no cloud credentials.
• datadog_sink.py / test_datadog_log_e2e.py: typed read-back client and three test cases; test_messages_emits_one_log_event is a deliberate regression pin for LIT-4447 (double-log on /v1/messages), currently fails by design and goes green when the fix lands.
• logging_client.py / test_otel_trace_e2e.py: refactors the _first_ok 401-retry helper into a public function; all existing OTEL call sites updated.

Confidence Score: 5/5

Test-only changes confined to tests/e2e/; no production code is touched.

Every changed file lives under tests/e2e/. The compose additions use only stdlib Python with no external credentials, the first_ok refactor is a straight move with no logic change, and the three new tests are well-isolated via unique_marker(). The one knowingly-failing test is fully documented as a regression pin and does not affect the other tests.

No files require special attention.

Important Files Changed

Filename	Overview
tests/e2e/logging/test_datadog_log_e2e.py	New e2e tests for DataDog log delivery (chat, messages, responses); the messages test is a deliberate regression pin for LIT-4447 (known double-log) and is documented to fail until that fix lands.
tests/e2e/logging/datadog_sink.py	New typed read-back client for the dd-sink; correctly polls for at least one event then re-reads after one interval to catch late duplicates; _parse_batch handles both array and single-object intake shapes.
tests/e2e/docker-compose.yml	Adds dd-sink service (stdlib Python, no cloud credentials) and wires DD_BASE_URL override; /health correctly returns text/plain and the proxy depends on the sink being healthy before starting.
tests/e2e/logging/logging_client.py	Extracts the _first_ok helper from test_otel_trace_e2e.py into a public first_ok function in logging_client.py for reuse; all existing call sites updated cleanly.
tests/e2e/logging/conftest.py	Adds session-scoped dd_sink fixture; straightforward and consistent with existing otel_reader fixture pattern.
tests/e2e/logging/test_otel_trace_e2e.py	All _first_ok call sites replaced with the promoted first_ok from logging_client; no logic changes, only refactoring.
tests/e2e/e2e_config.py	Adds DD_SINK_URL config constant with a sensible default matching the compose-stack port; consistent with the existing OTEL_QUERY_URL pattern.
tests/e2e/coverage_registry/logging.yaml	Adds responses to the exercised_on list for the datadog cell, consistent with the new test that covers the /v1/responses route.

_{Reviews (4): Last reviewed commit: "test(e2e): assert the scenario strictly;..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_e2e_datadog_log_delivery (8cf6a00) with litellm_internal_staging (906897b)}

[yucheng-berri]

@greptileai

[yucheng-berri]

@greptileai

[yucheng-berri]

@greptileai

[yucheng-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit e7a59ad. Configure here.}

[mateo-berri]

LGTM; thanks!

[mubashir1osmani]

we could hit the actual datadog endpoint, using datadog application key to read logs back. for future references please make sure we use the actual provider as we cannot deploy these on our actual cluster