fix(proxy): enforce max_parallel_requests as a per-slot concurrency gauge

[yassin-berriai]

Relevant issues

Fixes #16011

Linear ticket

Resolves LIT-4259

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

All runs below are against a live proxy (python litellm/proxy/proxy_cli.py --config config.yaml --port 4259) with real Postgres and Redis containers. Measuring backend concurrency requires an upstream slow enough to keep requests in flight across rate-limit windows, so the concurrency measurement uses a local OpenAI-compatible upstream that sleeps 12s per request and records its own max concurrency (a real provider answers in under a second, which hides the window-roll bug the issue reports; vLLM under load behaves like the slow upstream). A real OpenAI call is included at the end to show end-user behavior against a real provider

Setup: virtual key with the limit from the issue

curl -s -X POST http://127.0.0.1:4259/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
  -d '{"max_parallel_requests": 5, "key_alias": "litfix-4259-mpr"}'

Load: 60 concurrent closed-loop workers for 45s (the issue's oha scenario), LITELLM_RATE_LIMIT_WINDOW_SIZE=10 so in-flight requests span window rolls

Before the fix (current litellm_internal_staging):

status counts: {429: 435, 200: 238}
mock upstream stats: {'active': 0, 'max_active': 60, 'total': 238}

The upstream saw 60 concurrent requests on a key limited to 5; concurrency climbed monotonically (new max concurrency: 1 ... 60 in the upstream log) exactly as the vLLM logs in the issue show Running: 200 reqs on a limited key

After the fix (same proxy command, same key, same load):

status counts: {429: 9711, 200: 20}
mock upstream stats: {'active': 0, 'max_active': 5, 'total': 20}

Backend concurrency never exceeds the configured limit of 5, and throughput matches the theoretical ceiling for 12s requests (45s / 12s x 5 slots is about 18 completions)

Controlled burst, 8 simultaneous requests against the limit-5 key:

for i in $(seq 1 8); do curl -s -m 30 http://127.0.0.1:4259/v1/chat/completions \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"model":"slow-model","messages":[{"role":"user","content":"hi"}]}' -o /dev/null -w "%{http_code}\n" & done; wait
# exactly 5x200 + 3x429; upstream stats: {"active":0,"max_active":5,"total":5}

The gauge in Redis is now a per-slot registry instead of a windowed counter (three requests in flight):

$ redis-cli TYPE '{api_key:7c4cf8...c4e4}:max_parallel_requests'
zset
$ redis-cli ZRANGE '{api_key:7c4cf8...c4e4}:max_parallel_requests' 0 -1 WITHSCORES
ac1bc6db59f54510993e446dde31b7d6  1783493545
b47623b1d89648279cf6cdba584fe65e  1783493545
d70947d356bc402c9bc8cdb3964ab8ab  1783493545
$ redis-cli TTL '{api_key:7c4cf8...c4e4}:max_parallel_requests'
3597

Real provider (OpenAI gpt-5.4-mini, real API key, real money), key with max_parallel_requests: 2, 8 simultaneous requests:

   2 200
   6 429

and a single call after the burst drains, showing the response and the parallel-request headers:

content: parallel test ok
x-ratelimit-api_key-remaining-max_parallel_requests: 1
x-ratelimit-api_key-limit-max_parallel_requests: 2

Independent end to end verification

A separate agent session verified the fix from a fresh clone with its own Postgres, Redis, slow mock upstream, and virtual key. On the base branch litellm_internal_staging at 6f6bd45 the bug reproduced cleanly: backend concurrency climbed past the limit and pinned at 60 on a key limited to 5 (670x429, 180x200). On the PR branch at c681833, with Redis flushed and the same config and key, the identical 60-worker load held backend max_active at exactly 5 for the entire run (3765x429, 15x200), the 8-request burst returned exactly 5x200 and 3x429 with upstream max_active 5 during the burst, and the gauge key in Redis was a zset whose ZCARD was 5 with exactly 5 slot ids scored by acquire time and a TTL of 3596 seconds

Type

🐛 Bug Fix

Changes

max_parallel_requests is a concurrency gauge, but the v3 limiter enforced it through the same sliding-window counters as RPM/TPM. Three compounding defects made the limit meaningless under sustained load: every window roll reset the counter to 1 while requests were still in flight, so each roll admitted a fresh limit of concurrency on top of what was already running; the completion decrements for those forgotten requests then drove the counter negative, admitting limit + N more; and the increment-before-check pattern left a stranded +1 for every 429-rejected request that no callback ever released

This PR moves max_parallel_requests out of the windowed counters onto a per-slot registry. In Redis the gauge key becomes a sorted set of per-request slot ids scored by acquire time, and admission runs a Lua script that prunes expired slots and registers a new slot id only when in_flight + 1 <= limit, so a rejected request never occupies a slot. Without Redis, an asyncio-locked in-memory registry provides the same semantics per worker. The acquired slot id is stashed in the request metadata channels (the same plumbing the TPM reservation uses), and the success, failure, and client-disconnect paths release exactly that slot id. Releasing by id makes double-fired callbacks and proxy-side rejections structurally harmless: removing an absent member is a no-op, so a release can never free a slot owned by another in-flight request. Slots leaked by a crashed worker are pruned individually after PARALLEL_REQUEST_SLOT_TTL_SECONDS even while the key stays busy, which the old whole-key TTL could not do

The windowed RPM/TPM logic is unchanged; should_rate_limit runs the gauge phase after the windowed check so a windowed rejection never strands an acquired slot. Regression tests cover the window roll not resetting the gauge, rejected requests not consuming slots, releases without a matching acquire not freeing foreign slots, idempotent double releases, the Lua argument contract, and the disconnect release path

Two follow-up commits address review findings. The acquisition stash now carries the gauge counter keys alongside the slot id so the release paths free the slot from every gauge it was registered under, the disconnect release keys off the stashed acquisition rather than the key object's mutable configuration, and the in-memory fallback treats a cached integer (the count mirrored from the last successful Redis script call) as real occupancy instead of restarting from an empty registry during a Redis outage. async_post_call_failure_hook now releases the stashed slot before the TPM reservation guard, since it is the only callback that fires when a downstream hook such as a guardrail or budget check rejects a request after the slot was acquired; without it those slots lingered for the full slot TTL

During a rolling upgrade, pods on the previous build treat the gauge key as a string counter while upgraded pods use a sorted set; both sides catch the resulting WRONGTYPE script failure and degrade to per-pod in-memory enforcement, and the conflict clears itself once the previous build's keys expire after the rollout completes

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codecov]

Codecov Report

❌ Patch coverage is 93.60731% with 14 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/hooks/parallel_request_limiter_v3.py	93.48%	14 Missing ⚠️

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR fixes a long-standing bug where max_parallel_requests was enforced through the same sliding-window counters as RPM/TPM, causing the concurrency limit to be meaningless under sustained load (window rolls reset the counter while requests were in flight, rejected requests consumed slots, and decrements drove counters negative).

• Core change in parallel_request_limiter_v3.py: replaces the windowed counter approach with a Redis sorted-set gauge per descriptor. Three atomic Lua scripts (PARALLEL_ACQUIRE_SCRIPT, PARALLEL_RELEASE_SCRIPT, PARALLEL_COUNT_SCRIPT) handle check-then-acquire, slot removal by ID, and in-flight counting with expired-slot pruning. A UUID slot ID is generated per request at pre-call, stashed in request metadata, and used to release exactly that slot in all four exit paths (success, LLM failure, proxy-level rejection via async_post_call_failure_hook, and streaming client disconnect). _clear_parallel_slot_marker makes all release paths idempotent: a ZREM on an absent member is a no-op in Redis, and the marker clear prevents subsequent callbacks from attempting a second release.
• Bug previously noted in comments: async_post_call_failure_hook previously exited early at reserved_tokens ≤ 0 without releasing the parallel slot; the new code releases the slot before that check.
• Windowed RPM/TPM logic is unchanged: _collect_windowed_keys_and_gauges separates descriptors into windowed keys and gauges; the gauge check runs only after the windowed check passes, so a windowed 429 never strands an acquired slot.

Confidence Score: 5/5

Safe to merge — the concurrency gauge design is structurally sound, all four release paths are wired correctly, and the atomic Lua scripts prevent the check/acquire race that made the old windowed approach ineffective.

The redesign addresses all three compounding defects described in the PR. The pre-call hook acquires a slot only after the windowed RPM/TPM check passes, so a windowed rejection never strands an acquired slot. The stashed slot ID ensures that success, failure, proxy-rejection, and disconnect callbacks each release exactly the slot they own. The in-memory fallback correctly treats a cached integer as real occupancy. The slot-not-released issue in async_post_call_failure_hook is fixed. The structural multi-gauge release concern is also addressed: the stashed acquisition carries all counter_keys. Test coverage is thorough.

No files require special attention — the change is well-isolated to the parallel request limiter and its direct callers.

Important Files Changed

Filename	Overview
litellm/proxy/hooks/parallel_request_limiter_v3.py	Core change: moves max_parallel_requests from windowed counters to Redis sorted-set gauge with atomic Lua scripts; slot acquire, release, and idempotency logic are correct; all four release paths (success, failure, proxy rejection, disconnect) now properly release the acquired slot; previous issue with async_post_call_failure_hook is fixed; in-memory fallback correctly handles integer mirrors from Redis; structural multi-gauge release is correct.
tests/test_litellm/proxy/hooks/test_parallel_request_limiter_v3.py	Tests updated to reflect the slot-registry model with new coverage for window-roll correctness, rejected requests not consuming slots, Redis script args contract, in-memory fallback with mirrored integers, post-call failure hook idempotency, and disconnect with stale key config.
litellm/proxy/utils.py	_release_max_parallel_requests_on_disconnect now accepts request_data to pass the stashed acquisition to the limiter, fixing the old guard that keyed on the key object's current max_parallel_requests value rather than whether a slot was actually acquired.
litellm/proxy/common_request_processing.py	One-line change: passes request_data to _release_max_parallel_requests_on_disconnect so the disconnect handler can locate the stashed slot acquisition.
litellm/proxy/proxy_server.py	One-line change: passes request_data to the disconnect release in async_data_generator.

_{Reviews (3): Last reviewed commit: "fix(proxy): release the parallel slot on..." | Re-trigger Greptile}

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_max_parallel_requests_concurrency_gauge (d29a777) with litellm_internal_staging (2162da5)}

[yassin-berriai]

@greptileai please review the current head 2de5429

[yassin-berriai]

@greptileai please review the current head c681833

[galatolofederico]

Any ETA on this merge?