fix(datadog): split oversized batches on 413 instead of re-queueing forever

[yassin-berriai]

Relevant issues

The Datadog batch logger enters an infinite 413 loop on oversized batches. async_send_batch expects async_send_compressed_data to return a 413 response, but the async httpx handler runs raise_for_status() and converts a 413 into a raised MaskedHTTPStatusError. The if response.status_code == 413 branch is therefore never reached; the exception falls through to the bare except, which re-queues the entire oversized batch unchanged. The flush timer then retries the same batch, gets another 413, and repeats indefinitely, so no logs are ever delivered and the error is logged on every flush.

Linear ticket

LIT-3407. This overlaps with the in-progress PR #29183, which fixes the same loop; this PR is an independent take that also wires DD_BATCH_SIZE and closes the partial-delivery duplicate edge.

Companion PR

Documentation for the new DD_BATCH_SIZE env var lives in BerriAI/litellm-docs#272. The test_env_keys gate validates every os.getenv() against litellm-docs:docs/proxy/config_settings.md, so that PR must merge alongside (or before) this one or the documentation CI here will fail.

Pre-Submission checklist

• I have added meaningful tests
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Changes

An iterative _send_with_413_split helper (a work-list, no recursion, so a large queue cannot blow the stack) recognises a 413 on both the raised (MaskedHTTPStatusError) and the returned-response paths. On a 413 it halves the sub-batch and retries the halves, since Datadog enforces a 5MB uncompressed limit per request; a lone event that still 413s is dropped rather than re-queued so an undeliverable batch can no longer wedge the queue. On a transient (non-413) error the helper returns only the events Datadog did not accept, and async_send_batch re-queues those alone, so events already delivered by an earlier sub-batch are never re-sent.

DD_BATCH_SIZE is now wired through: it is read at init, clamped to [1, DD_MAX_BATCH_SIZE], and falls back to the max with a warning on invalid input, so the value the 413 error message points users at actually takes effect.

The test that previously asserted the buggy requeue-on-413 behaviour is replaced. New regression tests cover: an oversized batch splits and every deliverable event is sent, an undeliverable batch is not re-queued across repeated flushes, a lone oversized event is dropped, the defensive returned-413 path, a transient error after a partial split delivery re-queues only the undelivered events, and DD_BATCH_SIZE resolution including clamping and invalid input. All new tests fail against the pre-fix code and pass after the change; the full tests/test_litellm/integrations/datadog/ suite passes (68 tests).

Screenshots / Proof of Fix

A live curl proof is impractical here because triggering the bug needs a real Datadog intake returning 413 on a >5MB batch. The loop was reproduced deterministically in-process against a clean checkout by stubbing only the inner httpx send to return 413 (so the real AsyncHTTPHandler.post path raises the MaskedHTTPStatusError, exactly as in production) and calling async_send_batch repeatedly.

Before the fix, a 10-event batch stays pinned at length 10 across every flush and logs a 413 traceback each time:

queue length before any flush: 10
after flush #1: queue length = 10
after flush #2: queue length = 10
after flush #3: queue length = 10
branch counters: send_raised=3 send_returned=0

After the fix, the same batch splits down the work-list (send_raised=19 = 2*10-1) and the queue drains to 0 on the first flush, with later flushes as no-ops:

queue length before any flush: 10
after flush #1: queue length = 0
after flush #2: queue length = 0
after flush #3: queue length = 0
branch counters: send_raised=19 send_returned=0

In this reproduction every event is individually oversized, so all are dropped; when only the aggregate batch is too large the split delivers the events that fit, which is covered by test_413_splits_oversized_batch_and_delivers_every_event.

Type

Bug Fix

https://claude.ai/code/session_01R5srSidjGmPtEKuAnRqPXw

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codecov]

Codecov Report

❌ Patch coverage is 97.61905% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/integrations/datadog/datadog.py	97.61%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR fixes the infinite 413-requeue loop in the Datadog batch logger by replacing the broken async_send_batch logic with _send_with_413_split, an iterative work-list that halves oversized sub-batches on 413 (handling both the raised-MaskedHTTPStatusError and returned-response paths), drops lone undeliverable events to unblock the queue, and returns only genuinely undelivered events so previously-accepted events are never re-sent. It also wires the DD_BATCH_SIZE environment variable end-to-end.

• Core fix: _send_with_413_split replaces the old branch that silently fell through to re-queue the entire batch on every 413, ending the infinite loop. The _undelivered helper correctly reconstructs LIFO-stack ordering so re-queued events preserve their original sequence.
• DD_BATCH_SIZE wiring: _resolve_dd_batch_size() reads, clamps, and validates the env-var at init time, falling back to DD_MAX_BATCH_SIZE with a warning on invalid input.
• Tests: the old test asserting buggy behavior is replaced by six targeted regression tests (all mocked, no network calls) that cover split-and-deliver, no-requeue-on-drop, transient-error partial-delivery, returned-413 path, and batch-size clamping.

Confidence Score: 5/5

Safe to merge. The iterative split logic is correct, the LIFO ordering is preserved, and the infinite-requeue bug is genuinely closed. Only a cosmetic mock-mode log condition was found.

The core work-list algorithm correctly handles all 413 surface paths, preserves event ordering in _undelivered, and prevents duplicate delivery after a partial split. The test suite is thorough, all mocked, and the replaced test was correctly asserting the old buggy behavior rather than hiding a real regression.

No files require special attention; the single cosmetic finding is in the mock-mode debug log in datadog.py.

Important Files Changed

Filename	Overview
litellm/integrations/datadog/datadog.py	Replaces the broken 413-requeue loop with _send_with_413_split, an iterative work-list that halves oversized batches, drops lone undeliverable events, and returns only truly undelivered events for re-queue. Also wires DD_BATCH_SIZE env-var through _resolve_dd_batch_size. Logic is correct, though the mock-mode success log fires even when some events were re-queued.
tests/test_litellm/integrations/datadog/test_datadog_logger_batching.py	Old test asserting the buggy forever-requeue-on-413 behavior replaced by a thorough regression suite (split-and-deliver, no-requeue-on-undeliverable, single-event drop, returned-413 path, partial-delivery transient error, DD_BATCH_SIZE clamping). All tests use AsyncMock; no real network calls.

_{Reviews (2): Last reviewed commit: "fix(datadog): split oversized batches on..." | Re-trigger Greptile}

[yassin-berriai]

@greptileai