test(azure_sentinel): pin batch_size as a per-request bound under concurrent events

[devin-ai-integration]

TLDR

Problem this solves:

• Nothing pinned Azure Sentinel's per-request record cap under concurrent traffic
• LIT-6920's concurrency defect was fixed as a side effect of fix(azure_sentinel): split batches under the 1MB ingestion cap #39880 (LIT-5899)
• A later change to the size split could quietly bring the concurrency bug back

How it solves it:

• Adds one regression test to the mapped Azure Sentinel test file
• 40 records land concurrently at batch_size 5 while the first send is held open on an event
• Asserts the held send carried exactly 5 records and the other 35 are still queued
• Asserts no request exceeds batch_size, every record arrives exactly once, queue drains
• Runs for both the standard log queue and the audit log queue
• Fails on the tree before fix(azure_sentinel): split batches under the 1MB ingestion cap #39880, passes on current staging
• No production code changes

User Flow

Before: a platform team on a release without #39880 lowers DEFAULT_BATCH_SIZE to 50 to stay under Sentinel's 1 MB ingestion cap, and still loses records

1. They set DEFAULT_BATCH_SIZE=50 and callbacks: ["azure_sentinel"], then restart the proxy
2. Their app sends 400 concurrent POST http://litellm-domain/v1/chat/completions calls for gpt-5-mini, all 400 return 200 with a chatcmpl-... id
3. The proxy sends 145 ingestion requests to Sentinel, some carrying 92 records and 1.29 MB, and Sentinel answers 90 of them with HTTP 413
4. They query the Sentinel table for those 400 ids: 292 are present, many of them several times over, and 108 never arrive

After: the same team on a release with #39880 sees every request stay under batch_size and every record arrive once

1. They set DEFAULT_BATCH_SIZE=50 and callbacks: ["azure_sentinel"], then restart the proxy
2. Their app sends the same 400 concurrent POST http://litellm-domain/v1/chat/completions calls, all 400 return 200
3. The proxy sends 12 ingestion requests to Sentinel, none over 50 records or 711 KB, and Sentinel answers every one with 204
4. They query the Sentinel table for those 400 ids: all 400 are present exactly once

This PR adds the test that keeps the After flow from regressing. The merge base already behaves like After

Relevant issues

Pylon #7652

Linear ticket

Resolves LIT-6920

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Shared setup. Real proxy per tree, real Postgres (--use_prisma_db_push), 4 uvicorn workers on both sides, real OpenAI gpt-5-mini calls. PYTHONPATH is pinned to the tree under test and litellm.__file__ is asserted to sit inside it before boot. Sentinel is a local HTTP ingestion endpoint on 127.0.0.1 that takes 2 s per request, answers 204, and answers 413 for any body over 1,000,000 bytes, which is Azure's documented cap. It logs bytes, record count, status, and record ids per request, so the same numbers a customer reads off Sentinel diagnostics are read back here

The merge base d9b63ef already contains the fix from #39880, so a Before at the merge base would match After. Before is captured at 56a61cf, the last staging commit without that behavior, which is the tree the ticket reproduced against

No screenshots: the behavior this test pins, how many records ride in each Sentinel ingestion request, is visible only at the ingestion endpoint. The Admin UI Logs page shows the same 400 rows on both trees and no dashboard page renders ingestion batches, so the endpoint-side tally above is the proof

Config:

model_list:
  - model_name: gpt-5-mini
    litellm_params:
      model: openai/gpt-5-mini
      api_key: os.environ/OPENAI_API_KEY

litellm_settings:
  callbacks: ["azure_sentinel"]

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

Environment on both sides: DEFAULT_BATCH_SIZE=50 DEFAULT_FLUSH_INTERVAL_SECONDS=30 AZURE_SENTINEL_ENDPOINT=http://127.0.0.1:25920 AZURE_SENTINEL_AUTHORITY_HOST=http://127.0.0.1:25920 plus the usual AZURE_SENTINEL_* ids

Boot (per tree):

export PYTHONPATH=$PWD
python -c "import litellm, os; assert litellm.__file__.startswith(os.getcwd())"
python -m litellm.proxy.proxy_cli --config proxy_config.yaml --port 20920 --use_prisma_db_push --detailed_debug --num_workers 4

Burst (400 concurrent chat completions, each response id is the Sentinel record id):

seq 1 400 | xargs -P 400 -I{} bash -c '
  curl -sS "http://127.0.0.1:20920/v1/chat/completions" \
    -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
    -d "{\"model\":\"gpt-5-mini\",\"messages\":[{\"role\":\"user\",\"content\":\"Request {}. Reply with the single word ok. Ignore this filler: '"$(head -c 2000 /dev/zero | tr '\0' x)"'\"}],\"max_tokens\":64,\"reasoning_effort\":\"minimal\"}" \
    | grep -oE "\"id\":\"[^\"]+\"" | head -1'

Then wait 40 s for the periodic flush and compare the 400 response ids against the ids Sentinel accepted

Before (56a61cf)

1. grep -c "Started server process" proxy.log

   4
   

2. Burst

   burst(base4_slow): done in 19s; statuses:     400 200
   

3. Sentinel-side tally after the flush

   [base4_slow] proxy 200s=400 sentinel_requests=145 max_body_bytes=1294130 max_records_per_body=92 (batch_size=50)
   [base4_slow] bodies_over_batch_size=144 responses_413=90 accepted_records=3535 unique=292 duplicates=3243 lost=108
   [base4_slow] VERDICT: UNBOUNDED | lost=108 | duplicates=3243
   

4. Records per body for the first 12 ingestion requests

   [52, 52, 52, 54, 53, 55, 56, 57, 58, 59, 61, 62]
   

5. grep -c "413 Request Entity Too Large" proxy.log

   270
   

After (2a9c860)

1. grep -c "Started server process" proxy.log

   4
   

2. Burst

   burst(head4_tip2): done in 20s; statuses:     400 200
   

3. Sentinel-side tally after the flush

   [head4_tip2] proxy 200s=400 sentinel_requests=12 max_body_bytes=710819 max_records_per_body=50 (batch_size=50)
   [head4_tip2] bodies_over_batch_size=0 responses_413=0 accepted_records=400 unique=400 duplicates=0 lost=0
   [head4_tip2] VERDICT: BOUNDED | lost=0 | duplicates=0
   

4. Records per body for all 12 ingestion requests

   [50, 50, 50, 50, 21, 44, 37, 39, 22, 13, 19, 5]
   

5. grep -c "413 Request Entity Too Large" proxy.log

   0
   

Regression evidence for the test itself: at 2a9c860 the new test fails on 56a61cf (2 failed, both parametrizations, the held first send carried 26 records instead of 5) and passes on staging (85 passed for the file, 3 repeated runs). Mutants that drop the flush lock, drop the count split, or send from the live queue instead of a detached copy each fail at least one test in the file

Type

✅ Test

Caveats (if any)

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Low Risk
Test-only change in the Azure Sentinel integration test suite; no runtime or configuration behavior is modified.

Overview
Adds a regression test so Azure Sentinel ingestion never sends more than batch_size records in one HTTP request when new events keep arriving while a batch is still on the wire.

The test fires 40 concurrent log/audit callbacks with batch_size=5, blocks the first ingestion call until mid-flight assertions run, then checks the held request carried exactly five IDs, the remainder stayed queued, every later request stayed ≤5 records, all 40 IDs were delivered once, and the queue drained. It runs for both standard and audit queues via existing QUEUE_CASES parametrization.

No production code changes—this locks in behavior from the prior concurrency fix (LIT-6920) so future batching changes cannot reintroduce oversized bodies, 413s, duplicates, or lost records.

^{Reviewed by Cursor Bugbot for commit 2a9c860. Bugbot is set up for automated code reviews on this repo. Configure here.}

Link to Devin session: https://app.devin.ai/sessions/7fdc48a030e94795bb06e1ad3b0af942
Open in Devin Desktop: https://app.devin.ai/desktop/session/7fdc48a030e94795bb06e1ad3b0af942?variant=devin
Requested by: @yucheng-berri

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[greptile-apps]

Greptile Summary

This test-only PR strengthens the Azure Sentinel batching regression coverage by replacing timing-based overlap with explicit event synchronization.

• Holds the first ingestion request open while concurrent events continue to queue.
• Verifies the first request contains exactly batch_size records.
• Verifies subsequent requests remain bounded, every record is delivered once in order, and both supported queues drain.
• Makes no production runtime changes.

Confidence Score: 5/5

The PR appears safe to merge because the revised synchronization directly exercises the in-flight-send condition without introducing production changes or new actionable issues.

No new defects or applicable rule violations remain. The earlier timing-dependence thread was manually resolved without explanation; the current event-based synchronization nevertheless addresses its concern. The other previous findings were correctly withdrawn after repository scope and local test conventions were clarified.

Important Files Changed

Filename	Overview
tests/test_litellm/integrations/test_azure_sentinel.py	Adds deterministic synchronization and intermediate assertions to the concurrent batch-size regression test for standard and audit log queues.

_{Reviews (2): Last reviewed commit: "test(azure_sentinel): gate the first sen..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[yucheng-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[devin-ai-integration]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 2a9c860. Configure here.}