feat(proxy): per-worker admission control that rejects excess requests with 503

[devin-ai-integration]

Description

TLDR

Problem this solves:

• A saturated worker keeps accepting requests and clients hang with no overload signal
• Liveness probes share the loop, so k8s restarts overloaded pods and cascades load
• global_max_parallel_requests needs Redis and is ignored by the v3 limiter (LIT-5460)

How it solves it:

• Outermost ASGI middleware caps in-flight and queued requests per worker process
• Excess requests get an immediate 503 overloaded_error with retry-after: 1
• Probe and metrics paths bypass the gate so liveness stays honest under load, including behind SERVER_ROOT_PATH
• Invalid limits (nonpositive cap or timeout, negative queue) are logged once and leave the gate off
• New Prometheus metrics for admitted, queued, and rejected counts, plus /health/backlog fields
• Env-gated one-worker Granian saturation benchmark under tests/load_tests/

Configuration (all under general_settings, disabled unless max_in_flight_requests_per_worker is set):

general_settings:
  max_in_flight_requests_per_worker: 64   # concurrent requests admitted per worker process
  max_queued_requests_per_worker: 64      # waiting for a slot, defaults to the in-flight cap
  admission_queue_timeout_seconds: 1.0    # a queued request is rejected after this long

The limit is per worker process, so a pod with --num_workers 4 admits 4x that. It works the same on uvicorn and Granian and complements the Redis-backed global_max_parallel_requests: that one bounds a whole deployment, this one keeps any single event loop from drowning while an HPA catches up. Docs PR to follow in litellm-docs

User Flow

Before: a spike lands on a one-worker Granian pod and every caller just waits

1. An operator runs the proxy with --run_granian --num_workers 1 and a load spike sends 200 concurrent POST https://litellm-domain/v1/chat/completions
2. Every request is accepted and sits in the event loop; none fail, callers wait for many seconds with no signal to back off
3. GET https://litellm-domain/health/liveliness also takes seconds, so Kubernetes restarts the pod and the load moves to the remaining replicas
4. GET https://litellm-domain/health/backlog only shows {"in_flight_requests": 200}

After: the same spike gets a bounded number of requests through and tells the rest to retry

1. The operator adds max_in_flight_requests_per_worker: 8 (and optionally the queue size and timeout) to general_settings, restarts, and the same 200 concurrent POST https://litellm-domain/v1/chat/completions arrive
2. 8 requests run, 8 wait up to the queue timeout, and the rest immediately get 503 {"error":{"message":"Worker at capacity: 8 in-flight, 8 queued requests. Retry later.","type":"overloaded_error","code":"503"}} with a retry-after: 1 header
3. GET https://litellm-domain/health/liveliness keeps answering in milliseconds, so the pod stays in rotation
4. GET https://litellm-domain/health/backlog now returns in_flight_requests, admitted_requests, queued_requests, and rejected_requests, and /metrics exposes litellm_admission_admitted_requests, litellm_admission_queued_requests, and litellm_admission_rejected_requests_total{reason="queue_full"|"queue_timeout"}

Relevant issues

Linear ticket

Resolves LIT-6561

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live proxy, one Granian worker, real Anthropic calls. Config used for both runs (key values omitted):

model_list:
  - model_name: claude
    litellm_params:
      model: anthropic/claude-opus-5
      api_key: os.environ/ANTHROPIC_API_KEY
general_settings:
  master_key: sk-...
  max_in_flight_requests_per_worker: 2
  max_queued_requests_per_worker: 1
  admission_queue_timeout_seconds: 0.5
litellm_settings:
  callbacks: ["prometheus"]

Server: python litellm/proxy/proxy_cli.py --config proof_config.yaml --run_granian --num_workers 1 --port 4000

Before (55e9e4c)

Six concurrent completions

1. for i in $(seq 6); do curl -s -o /dev/null -w "%{http_code} %{time_total}s\n" http://localhost:4000/v1/chat/completions -H "Authorization: Bearer sk-..." -H "Content-Type: application/json" -d '{"model":"claude","messages":[{"role":"user","content":"Write a 300 word story"}]}' & done; wait
2. All six are accepted and the settings are ignored, every caller waits 14 to 18 seconds:

200 14.454359s
200 14.646097s
200 15.567830s
200 15.817365s
200 16.379516s
200 18.561153s

Backlog and metrics

1. curl -s http://localhost:4000/health/backlog -H "Authorization: Bearer sk-..."
2. {"in_flight_requests":1}
3. curl -sL http://localhost:4000/metrics -H "Authorization: Bearer sk-..." | grep litellm_admission returns nothing

After (6131186, behavior identical to a5c5c20 where this run was captured; later commits add /metrics/ to the exempt paths, strip root_path before the exempt check, log and ignore invalid settings, move counters into an injected AdmissionControlState, route new arrivals behind pending queue waiters instead of past them, and cache settings parsing off the hot path, plus extra tests)

Six concurrent completions

1. Same loop as Before
2. Two run, one waits the 0.5s queue timeout, three are rejected in about 5ms:

503 0.006648s
503 0.005549s
503 0.005015s
503 0.510173s
200 14.001931s
200 15.692811s

3. Headers of a rejected request (curl -si while 3 others are in flight):

HTTP/1.1 503 Service Unavailable
content-type: application/json
content-length: 127
retry-after: 1
server: granian

{"error":{"message":"Worker at capacity: 2 in-flight, 1 queued requests. Retry later.","type":"overloaded_error","code":"503"}}

Streaming holds a slot

1. Two concurrent "stream": true requests piped through head -c 200, then a third non-stream request 100ms later
2. Both streams deliver chunks, the third is rejected after the queue timeout:

third-non-stream 503 0.501896s
stream-1: data: {"id":"chatcmpl-53cfb691-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"reasoning_content":"", ...
stream-2: data: {"id":"chatcmpl-4ad0ccda-...","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"reasoning_content":"", ...

Backlog and metrics

1. curl -s http://localhost:4000/health/backlog -H "Authorization: Bearer sk-..."
2. {"in_flight_requests":1,"admitted_requests":0,"queued_requests":0,"rejected_requests":7} (the one in-flight request is the backlog call itself, which is exempt)
3. curl -sL http://localhost:4000/metrics -H "Authorization: Bearer sk-..." | grep litellm_admission

litellm_admission_admitted_requests 1.0
litellm_admission_queued_requests 0.0
litellm_admission_rejected_requests_total{reason="queue_full"} 4.0
litellm_admission_rejected_requests_total{reason="queue_timeout"} 3.0

Saturation benchmark

1. LITELLM_RUN_SATURATION_BENCHMARK=1 uv run --no-sync pytest tests/load_tests/test_granian_admission_saturation.py -s (one Granian worker, 200 concurrent requests against a local fake OpenAI endpoint, cap 8 in-flight and 8 queued, 0.5s queue timeout)
2. Output:

metric          value
rps             54.94
200 count       8
503 count       192
p99             3.640s
liveness p95    0.018s

Type

🆕 New Feature

Caveats (if any)

Medium

• Limits are parsed on each request (cached), but the in-flight semaphore is sized once per event loop, so restart after changing max_in_flight_requests_per_worker
• Per process only: size the cap per worker, not per pod or deployment

Low

• Rejections happen before auth, so they are not attributed to a key in spend logs
• /health (model health check) is not exempt, only the probe paths are

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/079cbff62c9e41e38376d31c27df4dcf
Open in Devin Desktop: https://app.devin.ai/desktop/session/079cbff62c9e41e38376d31c27df4dcf?variant=devin
Requested by: @yassin-berriai

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[devin-ai-integration]

PR #39352 (BerriAI/litellm, author devin-ai-integration[bot]) has no enterprise label — out of scope; no changes made. Final risk label: none applied; routing not run.

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_lit6561_granian_admission_control (6131186) with litellm_internal_staging (a06d63f)}

[greptile-apps]

Greptile Summary

The PR adds opt-in, per-worker admission control that bounds active and queued HTTP requests while exempting operational probe and metrics routes.

• Returns immediate OpenAI-style 503 overload responses with retry guidance when capacity is exhausted.
• Adds per-process admission metrics and backlog-health statistics.
• Validates and caches admission settings, disabling the gate for invalid configurations.
• Adds isolated middleware tests and an environment-gated Granian saturation benchmark.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
litellm/proxy/middleware/admission_control_middleware.py	Implements the admission gate, root-path-aware exemptions, injected process-local state, cached validated settings, metrics, and overload responses; the previously reported issues are addressed.
litellm/proxy/proxy_server.py	Registers the admission middleware around the proxy and exposes the new settings through general-settings metadata.
litellm/proxy/health_endpoints/_health_endpoints.py	Extends the authenticated backlog endpoint with admitted, queued, and rejected request statistics.
litellm/proxy/_types.py	Adds typed and constrained configuration fields for the per-worker admission limits and queue timeout.
tests/test_litellm/proxy/middleware/test_admission_control_middleware.py	Covers capacity, queue ordering, timeout and cancellation cleanup, prefixed exemptions, streaming lifetime, metrics, invalid settings, and state isolation.
tests/load_tests/test_granian_admission_saturation.py	Adds an opt-in single-worker saturation benchmark validating overload responses and liveness latency.

_{Reviews (2): Last reviewed commit: "refactor(proxy): simplify invalid admiss..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 97.36842% with 5 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...m/proxy/middleware/admission_control_middleware.py	98.28%	3 Missing ⚠️
litellm/proxy/proxy_server.py	60.00%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[devin-ai-integration]

Fixed in 8d3e9c7: the middleware now strips root_path before checking exempt paths, with a parametrized test for /proxy/health/liveliness.

[devin-ai-integration]

Fixed in 8d3e9c7: negative queue size and nonpositive timeout are rejected via constrained Pydantic adapters and Field bounds, with tests.

[devin-ai-integration]

Fixed in 8d3e9c7: the overload response is now a Starlette JSONResponse instead of hand-built ASGI messages.

[devin-ai-integration]

Fixed in 8d3e9c7: counters and semaphore moved into an injected AdmissionControlState, so tests build their own state per case.

[devin-ai-integration]

@greptileai please re-review: new arrivals now queue behind pending waiters, settings parsing is cached, invalid limits are logged