fix(azure_sentinel): split batches under the 1MB ingestion cap - #39880
yucheng-berri merged 10 commits into
Conversation
…ep undelivered records queued Azure Monitor rejects any Logs Ingestion body over 1MB with a 413. The Sentinel logger posted the whole queue as one body and cleared it in a finally block, so an oversize batch, a transient 5xx, or a failed token call dropped every queued record, and records logged while a send was in flight were cleared with it. Both the standard and the audit queue share the sender. Move Datadog's proactive size split and 413 halving into a shared helper, litellm/integrations/batch_utils.send_batch_with_413_split, and route Sentinel through it with a 1MB size check. A lone record that still 413s is dropped, everything a transient failure leaves undelivered goes back to the front of its queue, and the retry queue is capped at max_queue_size so an unreachable workspace cannot grow memory without bound
Greptile SummaryThis PR makes Azure Sentinel delivery resilient to oversized payloads, transient failures, concurrent flushes, and cancellation while sharing the 413-splitting implementation with Datadog. The latest changes make requeueing the safe shared default while Azure Sentinel explicitly retains its policy of dropping permanent non-413 client rejections.
Confidence Score: 5/5The PR appears safe to merge because current callers explicitly select their intended non-success behavior and no outstanding failure remains All previous findings are resolved, and the latest default change does not alter either production integration because Datadog explicitly requeues while Azure Sentinel explicitly drops permanent non-413 client failures
|
| Filename | Overview |
|---|---|
| litellm/integrations/batch_utils.py | Adds shared batch splitting and cancellation-aware delivery, with requeueing as the default non-success policy |
| litellm/integrations/azure_sentinel/azure_sentinel.py | Adds bounded retry queues, serialized flushing, payload splitting, and an explicit permanent-4xx drop policy |
| litellm/integrations/datadog/datadog.py | Adopts the shared split helper while explicitly preserving Datadog's existing requeue behavior |
| tests/test_litellm/integrations/test_azure_sentinel.py | Covers standard and audit delivery across size limits, failures, retries, cancellation, and concurrency |
| tests/test_litellm/integrations/datadog/test_datadog_logger_batching.py | Covers Datadog cancellation, serialization isolation, and requeue compatibility across HTTP failures |
Reviews (9): Last reviewed commit: "fix(batch_utils): requeue by default and..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
| self._drop_oldest_over_max_queue_size(self.audit_log_queue, "audit logs") | ||
|
|
||
| def _drop_oldest_over_max_queue_size(self, queue: list[_QueuedPayload], log_type: str) -> None: | ||
| overflow: Final = len(queue) - self.max_queue_size |
There was a problem hiding this comment.
Low: Retry queue can exhaust proxy memory
The retry queue is bounded by record count, although standard log records contain user-controlled messages and can vary significantly in size. While Azure ingestion is returning transient errors, an authenticated user can submit large requests until retained payloads exhaust proxy memory; each subsequent threshold flush also repeatedly serializes the retained queue. Add a serialized-byte limit for retained records, or use a substantially smaller integration-specific retry budget, and drop oldest records until both limits are satisfied.
There was a problem hiding this comment.
max_queue_size is the CustomBatchLogger contract every batch logger shares, and operators can lower it. Repeated serialization is gone, retries wait for the timer
There was a problem hiding this comment.
Thanks for the detail. I can't automatically confirm this is safe to dismiss, so I'm leaving the thread open for a maintainer to make the call.
PR overviewThis pull request updates the Azure Sentinel integration to split ingestion batches so each stays below Azure's 1 MB limit. Two queue-management issues remain open in the Azure Sentinel and Datadog integrations. Under failing or rejected ingestion requests, an authenticated user could cause retained log data to grow until the proxy exhausts memory, resulting in a denial of service. No reported issues have yet been addressed. Open issues (2)
Fixed/addressed: 0 · PR risk: 5/10 |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Requeued records made every later event cross the batch_size threshold, so a down ingestion endpoint got one full-queue resend per request. Threshold sends now go through flush_queue, so they take the flush lock instead of racing the timer, and they stand down while records are awaiting retry. A record that cannot be serialized raised out of the size probe and killed the periodic flush task. The probe now runs inside the failure handling, so the batch is split and only the record that cannot be serialized is dropped.
|
@greptileai please review the current head ef94b7d |
|
bugbot run |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Concurrent callbacks all read logs_awaiting_retry before the first send finished, so each one resent the whole queue once that send failed. The flag and the batch_size threshold are now rechecked while holding the flush lock, and each queue sends only itself instead of going through flush_queue, which was retrying the other queue too.
|
@greptileai please review the current head 396f0ee |
|
bugbot run |
|
@greptileai please review the current head add9ef4 |
|
bugbot run |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
… split A batch over the ingestion cap goes out in pieces, so a cancellation partway through requeued pieces the destination had already accepted and sent them a second time on the next flush The split helper now raises a cancellation carrying the records it never delivered, and Azure Sentinel requeues those instead of the whole batch
|
@greptileai please review the current head 577ff5c |
|
bugbot run |
…cept A non-413 4xx from the ingestion endpoint or from the OAuth token call means the request will fail the same way on every retry, so requeueing it held the batch, and every record logged behind it, until the queue cap dropped them. Retryable statuses (5xx, 408, 429) still keep the whole batch, and a shared classifier gives Datadog the same rule The serialization probe now catches any exception, not just TypeError and ValueError, because safe_dumps hands pydantic models to model_dump and can raise anything. It also splits on record count, so a recovery flush sends batch_size records per request instead of serializing the whole requeued queue to measure it Both integrations re-raise a cancelled send as exactly asyncio.CancelledError. Python 3.12's asyncio.wait_for only translates the exact class into TimeoutError, so the BatchSendCancelled subclass escaped the logging worker as an unhandled error The awaiting-retry flag now follows the queue that survived the max_queue_size trim, so a deployment with the cap at zero is not left waiting for a timer flush with nothing queued to retry
|
@greptileai please review the current head 0b0e470 |
|
bugbot run |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Annotate the queue detach and requeue constructions required by the logger's appendable queue contract so the type-discipline budget stays clean
|
@greptileai please review the current head 16e2e80 |
|
bugbot run |
Keep Datadog's existing contract of requeuing every non-413 HTTP failure while Azure Sentinel applies its permanent-client-error policy through the shared splitter
|
@greptileai please review the current head 154ad28 |
|
bugbot run |
| success_status_codes=frozenset({202}), | ||
| integration_name="Datadog", | ||
| drop_error_message=DD_ERRORS.DATADOG_413_ERROR.value, | ||
| non_success_handler=requeue_after_http_error, |
There was a problem hiding this comment.
Low: Permanent failures grow the queue without a bound
This handler requeues records for permanent responses such as 400 and 403. DataDogLogger.flush_queue() bypasses the base class's max_queue_size enforcement, so an authenticated user can continually submit requests while the intake rejects them and grow log_queue until the proxy exhausts memory. Either drop non-retryable responses or enforce the queue limit when records are requeued.
There was a problem hiding this comment.
Live base and head 403 legs both requeued records and recovered them after fault removal; queue bounding predates this PR and remains a follow-up
There was a problem hiding this comment.
Thanks for the detail. I can't automatically confirm this is safe to dismiss, so I'm leaving the thread open for a maintainer to make the call.
The shared splitter's default non-success handler is now requeue_after_http_error, the behavior Datadog had before the extraction, so a caller that omits the argument keeps its records. Azure Sentinel passes undelivered_after_http_error explicitly to drop permanent 4xx rejections Also drops an explicit return None the strict ruff gate flags in the test helper
|
@greptileai please review the current head 4b78699, which makes requeue the shared helper default and has Sentinel opt into dropping explicitly |
|
bugbot run |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 4b78699. Configure here.
d515a28
into
litellm_internal_staging
…current events (#40320) * test(azure_sentinel): pin batch_size as a per-request bound under concurrent events Adds a regression test to the mapped Azure Sentinel test file for the concurrency scenario from LIT-6920: 40 records logged concurrently at batch_size=5 while each ingestion request is still in flight. Asserts no request carries more than batch_size records, every record arrives exactly once in order, and the queue is empty afterwards. Runs for both the standard log queue and the audit log queue. The test fails on the tree before #39880 (whole shared queue serialized per threshold send, then cleared) and passes on current staging. It is independent of the size-split coverage that #39880 added for LIT-5899. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(azure_sentinel): gate the first send on events so later records provably arrive while it is in flight Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2 participants
TLDR
Problem this solves:
How it solves it:
User Flow
Before: a proxy admin who ships request logs to Microsoft Sentinel sees a burst of large prompts vanish from Log Analytics, along with the small requests sent in the same few seconds
LiteLLM_CL | where TimeGenerated > ago(10m): none of the seven requests is thereAzure Sentinel Error sending batch API - Client error '413 Request Entity Too Large', and nothing is retried503 Service Unavailableline, and every record queued at that moment is gone for goodAfter: the same requests all show up in Log Analytics, and a short outage delays them instead of losing them
LiteLLM_CL | where TimeGenerated > ago(10m)lists all seven requestsAzure Sentinel API Error - Payload too large for a single recordand only that one record is skipped503 Service Unavailableline per flush, and once the endpoint is back the queued records land in Log Analytics on the next flushRelevant issues
Fixes #26450
Linear ticket
Resolves LIT-5899
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Screenshots / Proof of Fix
Datadog UI evidence: public dashboard and dashboard screenshot. The dashboard reads six real Datadog Logs events from both the base proxy and fixed head after the intake returned 403 for 30 seconds, then recovered
Azure portal evidence from the dedicated Log Analytics workspace:
Reactive live-413 readback: the forwarder returned 413 above 300KB and relayed smaller requests to the real Azure endpoint; the base delivered 1 of 6 records while the fixed head delivered all 6. Open the full image
Real Azure proactive-split readback: the workspace contains every final-head large-batch and small-sibling record; the base's burst result varies with flush timing and dropped mixed oversize batches in the recorded run. Open the full image
Audit stream parity readback: both base and head delivered the small audit record, confirming the audit callback path remains intact. Open the full image
Azure portal UI steps: open
https://portal.azure.com, select Log Analytics workspaces, openlit5899-wsin resource grouplitellm-lit5899-rg, choose Logs, paste theLiteLLM_CLquery below, set the time range to Last hour, and run it. The query shows the final-headLIT5899_R6_S1_HEAD,LIT5899_R6_S3_HEAD, andLIT5899_R6_S4SMALL_HEADrows. Run theLiteLLMAudit_CLquery below for the audit markerShared setup, identical for both runs. Real proxy on a fresh Postgres, real OpenAI, real Azure Monitor Logs Ingestion endpoint (a dedicated Log Analytics workspace with a
Custom-LiteLLMand aCustom-LiteLLM-Auditstream), one workerBefore (aea5358)
One small request followed by six concurrent 240KB requests
curl -sS -o /dev/null -w 'status=%{http_code}\n' http://127.0.0.1:4000/v1/chat/completions -H 'Authorization: Bearer sk-lit5899' -H 'Content-Type: application/json' -d @s1.jsonseq 1 6 | xargs -P 6 -I{} curl -sS -o /dev/null -w 'status=%{http_code}\n' http://127.0.0.1:4000/v1/chat/completions -H 'Authorization: Bearer sk-lit5899' -H 'Content-Type: application/json' -d @s3.json | sort | uniq -cProxy log at the next flush
LiteLLM_CLreadback: no rows forLIT5899_R2_S1_BASEorLIT5899_R2_S3_BASE. All seven records were dropped, including the small one that happened to share the batchOne 1.1MB request plus three small ones
curl ... -d @s4big.json & seq 1 3 | xargs -P 3 -I{} curl ... -d @s4small.json; waitThe 400 is OpenAI rejecting the 1.1MB prompt, which the proxy still logs as a failure record
Proxy log at the next flush
LiteLLM_CLreadback: no rows forLIT5899_R2_S4SMALL_BASE. The three small records were dropped along with the one oversize recordTeam creation audit record
curl -sS -o /dev/null -w 'status=%{http_code}\n' http://127.0.0.1:4000/team/new -H 'Authorization: Bearer sk-lit5899' -H 'Content-Type: application/json' -d '{"team_alias":"LIT5899_R2_S5_base"}'LiteLLMAudit_CLreadbackSmall audit batches were never affected, this case is the parity check
Ingestion endpoint answers 503 for 25 seconds, then recovers
Point
AZURE_SENTINEL_ENDPOINTat a local forwarder that answers 503 while/tmp/fault.flagexists and otherwise relays the request unchanged to the real endpoint.touch /tmp/fault.flag, then send three chat completions with markerLIT5899_REC_baseand onePOST /team/newwith aliasLIT5899_REC_S5_baseForwarder log after 22 seconds, then
rm /tmp/fault.flagand 15 more secondsOne attempt per stream, nothing after the fault is lifted
Proxy log
Readback with the
LIT5899_REC_markers: no rows inLiteLLM_CL, no rows inLiteLLMAudit_CLAfter (934a0d1)
One small request followed by six concurrent 240KB requests
curl -sS -o /dev/null -w 'status=%{http_code}\n' http://127.0.0.1:4000/v1/chat/completions -H 'Authorization: Bearer sk-lit5899' -H 'Content-Type: application/json' -d @s1.jsonseq 1 6 | xargs -P 6 -I{} curl -sS -o /dev/null -w 'status=%{http_code}\n' http://127.0.0.1:4000/v1/chat/completions -H 'Authorization: Bearer sk-lit5899' -H 'Content-Type: application/json' -d @s3.json | sort | uniq -cProxy log at the next flush: no error line
LiteLLM_CLreadbackOne 1.1MB request plus three small ones
curl ... -d @s4big.json & seq 1 3 | xargs -P 3 -I{} curl ... -d @s4small.json; waitProxy log at the next flush
LiteLLM_CLreadbackThe three small records land. Only the single record that is over 1MB on its own is skipped, and the log names it
Team creation audit record
curl -sS -o /dev/null -w 'status=%{http_code}\n' http://127.0.0.1:4000/team/new -H 'Authorization: Bearer sk-lit5899' -H 'Content-Type: application/json' -d '{"team_alias":"LIT5899_R2_S5_head"}'LiteLLMAudit_CLreadbackIngestion endpoint answers 503 for 25 seconds, then recovers
Same forwarder,
touch /tmp/fault.flag, three chat completions with markerLIT5899_REC_headand onePOST /team/newwith aliasLIT5899_REC_S5_headForwarder log after 22 seconds, then
rm /tmp/fault.flagand 15 more secondsOne retry per flush interval while the fault is on, then both batches delivered with 204 as soon as it lifts
Proxy log
(one pair per flush, the audit stream's twin lines omitted for brevity)
Readback with the
LIT5899_REC_markersRetry cadence and concurrent threshold sends with
DEFAULT_BATCH_SIZE=4(396f0ee)Same forwarder, with
DEFAULT_BATCH_SIZE=4exported on all proxies. The new leg sent concurrent requests while the endpoint returned 503, then read back the real workspaceTwelve concurrent requests per side, one worker, 22 seconds under fault, then 15 seconds after recovery
Forty-eight concurrent requests per side, four workers, 22 seconds under fault, then 15 seconds after recovery
The previous head let concurrent threshold waiters resend the growing queue after the first failure. The new head rechecks the retry flag and threshold under the flush lock, so one failed threshold attempt is followed by timer retries only, and recovery delivers every record exactly once
The earlier sequential leg remains covered: base indexed 17 records for 9 requests under overlapping flushes, while the fixed head indexed exactly 9; during a 22-second outage, the fixed head retried once per 5-second flush interval and delivered all standard and audit records after recovery
Reactive halving on a live 413 (4b78699)
Azure's cap equals litellm's 1MB estimate, so the real destination never exercises the halve-on-413 branch. For this leg the forwarder answered 413 for any body over 300KB and relayed smaller bodies to the real ingestion endpoint. Six concurrent 240KB requests per side,
DEFAULT_BATCH_SIZE=8The head split the six records into two 756KB halves, each 413 was halved again, and every single-record body was accepted; no error line was logged. The base dropped its five-record batch on the first 413
Type
🐛 Bug Fix
Caveats (if any)
Medium
max_queue_size(default 50000), oldest dropped firstLow
batch_sizeno longer triggers a send; the periodic flush is the only retry path, so a failed backlog is attempted once perflush_interval(default 5s) instead of once per new eventCustomBatchLoggersubclasses still decide their batch-size sends outside the flush lock, and this PR only changes Azure Sentinel, so that stays for a follow-upDatadog delivered N recordsandDatadog API error: status_code=Datadog: delivered N eventsandDatadog: unexpected response status_code=undelivered_after_http_errorexplicitlybatch_sizerecords, and cancellation requeues only the unsent suffixFinal exhaustive live matrix (4b78699)
The final head was re-driven base-versus-head across every remaining Azure Sentinel and Datadog branch against real Azure Monitor and Datadog destinations. HTTP faults were injected only at destination or OAuth boundaries
Final Attestation
Note
Medium Risk
Changes core logging delivery semantics for Azure Sentinel (retry vs drop on 4xx, queue behavior under failure) and refactors Datadog’s batch send path; misconfiguration could now drop permanent 4xx batches in Sentinel where Datadog still requeues.
Overview
Fixes Azure Sentinel (and hardens Datadog) so proxy logs are not lost when batches exceed Azure Monitor’s ~1MB limit or when ingestion/OAuth fails transiently.
Shared
batch_utils: Newsend_batch_with_413_splitproactively splits oversized batches, halves on 413, drops only a single record that still 413s, isolates unserializable records, and on cancel requeues only the suffix that was never accepted. HTTP handling distinguishes retryable errors (5xx, 408, 429) from permanent 4xx viaundelivered_after_http_error(Sentinel) vs Datadog’s existing “requeue all non-413” behavior.Azure Sentinel: Sends through the shared helper with
AZURE_SENTINEL_MAX_PAYLOAD_SIZE_BYTES(1MB) and per-requestbatch_sizecaps. Queues are detached before send and merged back with undelivered items (withmax_queue_sizetrimming).logs_awaiting_retry/audit_logs_awaiting_retryblock batch-size triggers while a backlog is pending so retries run on the periodic flush only. Threshold sends useflush_lockwith rechecks to avoid duplicate sends and ordering bugs under concurrency.Datadog: Replaces inline 413/split loop with the same helper; adds tests for serialization failures and mid-split cancellation.
Reviewed by Cursor Bugbot for commit 4b78699. Bugbot is set up for automated code reviews on this repo. Configure here.
ran /live-pr-risk and found no regressions/backward incompatible risks