feat(rate-limiter): allow opting out of v3 TPM reservation and Redis circuit breaker

[yassin-berriai]

Relevant issues

A user reported Redis timeouts and latency regressions after deploying v1.82.0 or greater of the v3 rate limiter.

Linear ticket

Resolves LIT-3612

Pre-Submission checklist

• I have added meaningful tests
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

This was run against a live proxy talking to the real OpenAI API (gpt-4o-mini), with a real Redis backing the v3 rate limiter and a real Postgres for key storage. The two new toggles default to the current behavior, so an existing deployment is unaffected unless it opts out.

Setup

docker run -d --name pr30211-redis -p 6379:6379 redis:7-alpine
docker run -d --name pr30211-pg -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=litellm -p 5432:5432 postgres:16-alpine

Config (/tmp/pr30211_config.yaml):

model_list:
  - model_name: gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY

litellm_settings:
  cache: true
  cache_params:
    type: redis
    host: localhost
    port: 6379

general_settings:
  master_key: sk-1234

A virtual key with a deliberately small TPM cap so the reservation behavior is easy to see:

curl -s -X POST http://localhost:4000/key/generate \
  -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
  -d '{"models":["gpt-4o-mini"],"tpm_limit":200,"max_budget":5}'
# -> {"key":"sk-...", "tpm_limit":200, ...}

Scenario A: default behavior, upfront TPM reservation ON

Started the proxy with neither toggle set, so reservation is on (the v1.82+ behavior). A single request with max_tokens=120 reserves the full estimated budget up front even though the model only returns a handful of tokens.

single request, max_tokens=120 -> http=200
actual total_tokens = 15
proxy log: TPM reservation estimate: input=4, max_tokens=120 (explicit=True), total=124
proxy log: limit_remaining': 76, 'rate_limit_type': 'tokens'

So 124 of the 200 token budget is consumed at admission time against an actual usage of 15. Under a concurrent burst the reservations stack and the limiter sheds requests that would otherwise fit:

for i in 1 2 3 4; do
  curl -s -o /dev/null -w "req$i http=%{http_code}\n" -X POST http://localhost:4000/v1/chat/completions \
    -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
    -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Say hi in one word."}],"max_tokens":120}' &
done; wait

req4 http=200
req2 http=429
req1 http=429
req3 http=200

{"error":{"message":"Rate limit exceeded for api_key: <hash>. Limit type: tokens. Current limit: 200, Remaining: 76. Limit resets at: 2026-06-11 10:12:27 UTC","type":"throttling_error","param":null,"code":"429"}}

Two of four concurrent requests are rejected with a token limit error even though their combined actual usage is well under 200, because each in-flight request reserves about 124 tokens up front.

Scenario B: opt out, reservation OFF and circuit breaker OFF

Restarted the proxy with both toggles disabled:

LITELLM_TPM_TOKEN_RESERVATION_ENABLED=false REDIS_CIRCUIT_BREAKER_ENABLED=false \
  python litellm/proxy/proxy_cli.py --config /tmp/pr30211_config.yaml --port 4000 --detailed_debug

The exact same concurrent burst now all succeeds, no reservation runs, and the Redis :tokens counter reflects only actual post-call usage:

req3 http=200
req1 http=200
req2 http=200
req4 http=200

reservation-estimate log lines: 0
{api_key:<hash>}:tokens = 64
{model_per_key:<hash>:gpt-4o-mini}:tokens = 60

TPM enforcement still holds, it is just charged from real usage after the call rather than reserved up front. Firing sequential requests against the same key shows the counter climbing by actual tokens until it crosses the cap, at which point a 429 is returned:

req1  http=200  :tokens_after=16
req2  http=200  :tokens_after=32
...
req12 http=200  :tokens_after=192
req13 http=200  :tokens_after=208
req14 http=429  :tokens_after=209

{"error":{"message":"Rate limit exceeded for api_key: <hash>. Limit type: tokens. Current limit: 200, Remaining: 0. Limit resets at: 2026-06-11 10:13:13 UTC","type":"throttling_error","param":null,"code":"429"}}

Redis circuit breaker toggle

The circuit breaker only fires when Redis calls fail, so its effect is shown directly against the constant the proxy resolves at startup. With the toggle off the breaker never opens regardless of failures, so the limiter keeps using Redis instead of fast-failing to the in-memory fallback.

REDIS_CIRCUIT_BREAKER_ENABLED=false -> const=False, is_open() after 5 failures = False
default (enabled)                    -> const=True,  breaker OPENS after 2 failures, is_open() = True

Cleanup

docker rm -f pr30211-redis pr30211-pg

Type

🐛 Bug Fix

Changes

The v3 rate limiter's upfront TPM token reservation (PR #27001) atomically debits each request's estimated token budget before the upstream call. That costs an extra Redis Lua round-trip per request and, when Redis fails, serializes the in-memory fallback through a process-global asyncio.Lock. Against a shared limit pool with thousands of concurrent users all hitting the same {api_key:pool}:tokens key, this compounds into occasional Lua timeouts, which trip the Redis circuit breaker open, which then fans every async Redis call out to the locked in-memory fallback and serializes the event loop.

Rather than fork a _v4 handler, this gates the two pieces behind environment variables so the deployment can shed the overhead while everyone else is unaffected. Both default to the current behavior.

LITELLM_TPM_TOKEN_RESERVATION_ENABLED=false skips the reservation path entirely. The first sliding-window pass then enforces TPM directly (it no longer sets skip_tpm_check), and TPM is reconciled post-call from actual usage; this is the pre-v1.82 accounting model, which the post-call reconciliation already supports unchanged (no reservation stashed means every scope is charged the full actual usage). No extra Lua call, and the global-lock in-memory fallback is no longer on the per-request TPM path.

REDIS_CIRCUIT_BREAKER_ENABLED=false keeps the breaker permanently closed: is_open() always returns False, and record_failure() / record_success() are no-ops, so a burst of Lua timeouts can't open it and force every Redis call through the locked fallback. The existing REDIS_CIRCUIT_BREAKER_FAILURE_THRESHOLD / _RECOVERY_TIMEOUT knobs are unchanged.

Tests cover the breaker staying closed under sustained failures (and the guard always invoking the wrapped method), and the pre-call hook reserving vs. skipping based on the flag, including that no reservation stash leaks into request metadata when disabled.

Companion docs PR documenting the two env vars (unblocks the documentation and code-quality CI checks): BerriAI/litellm-docs#334

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[yassin-berriai]

@greptileai

[greptile-apps]

Greptile Summary

This PR adds two opt-out environment variables to the v3 rate limiter to address Redis latency regressions introduced in v1.82: REDIS_CIRCUIT_BREAKER_ENABLED=false keeps the circuit breaker permanently closed (no OPEN state from Lua timeouts), and LITELLM_TPM_TOKEN_RESERVATION_ENABLED=false skips the upfront token reservation, restoring the pre-v1.82 post-call accounting model at the cost of less precise concurrent TPM enforcement.

• RedisCircuitBreaker gains an enabled flag with early-return guards in is_open(), record_failure(), and record_success(); the constant is read from REDIS_CIRCUIT_BREAKER_ENABLED in constants.py.
• _PROXY_MaxParallelRequestsHandler_v3.__init__ reads LITELLM_TPM_TOKEN_RESERVATION_ENABLED and gates the reservation block and skip_tpm_check logic on self.tpm_reservation_enabled; when disabled, post-call reconciliation falls back cleanly to charging full actual usage since reserved_tokens=0 and reserved_scopes=[].
• Both features are covered by new unit tests; a live-proxy integration test is noted as in progress in the PR description.

Confidence Score: 5/5

Safe to merge; all changed code paths are guarded by opt-in flags that default to the current behavior, and post-call reconciliation already handled the no-reservation case before this PR.

Both new flags default to the pre-existing behavior, so every existing deployment is unaffected. The circuit-breaker guard is correctly applied to all three state-machine methods. The disabled-reservation path restores the pre-v1.82 accounting model, and _build_reservation_aware_tpm_ops already handled reserved_tokens=0 / empty reserved_scopes correctly, so post-call charging works without modification. Unit tests cover all new branches thoroughly.

No files require special attention; all changes are additive and flag-gated.

Important Files Changed

Filename	Overview
litellm/proxy/hooks/parallel_request_limiter_v3.py	Adds tpm_reservation_enabled flag (read inline via os.getenv) and gates the skip_tpm_check value and reservation block on it; post-call reconciliation handles the no-reservation case correctly via existing reserved_tokens=0 path.
litellm/caching/redis_cache.py	Adds enabled parameter to RedisCircuitBreaker with early-return guards in all three state-machine methods (is_open, record_failure, record_success), passed from the new REDIS_CIRCUIT_BREAKER_ENABLED constant.
litellm/constants.py	Adds REDIS_CIRCUIT_BREAKER_ENABLED module-level constant following the same pattern as the existing circuit-breaker threshold/timeout constants.
tests/test_litellm/caching/test_dual_cache.py	Three new unit tests cover the disabled circuit-breaker path: breaker never opens under sustained failures, record_success leaves state untouched, and the guard always invokes the wrapped method.
tests/test_litellm/proxy/hooks/test_parallel_request_limiter_v3.py	Five new unit tests cover TPM reservation enabled/disabled defaults, env-var parsing, and pre-call hook behavior (reservation called with correct args when enabled; reservation skipped and skip_tpm_check=False when disabled).
scripts/health_check/health_check_client.py	Trivial comment change: removes an internal company name from the custom_auth_header example string.

_{Reviews (2): Last reviewed commit: "feat(rate-limiter): allow opting out of ..." | Re-trigger Greptile}

[yassin-berriai]

@greptileai addressed the record_success guard asymmetry in b32c4bd; all three state-machine methods now short-circuit when the breaker is disabled

[greptile-apps]

The fix in b32c4bddc9 looks correct. All three state-machine methods are now symmetric:

def is_open(self) -> bool:
    if not self.enabled:
        return False
    ...

def record_failure(self) -> None:
    if not self.enabled:
        return
    ...

def record_success(self) -> None:
    if not self.enabled:   # ← added in b32c4bddc9
        return
    ...

The test is well-constructed too — by forcing _state = "open" and _failure_count = 3 on a disabled breaker before calling record_success(), it rules out a false pass where the method happens to be a no-op by coincidence (i.e., the HALF_OPEN branch not taken → state unchanged). It's a solid regression test that will catch any future side-effects added to record_success (logging, metrics, etc.) that would otherwise silently execute when the breaker is disabled.

Good to merge from the circuit-breaker symmetry perspective.

[yassin-berriai]

@greptileai please re-review; the branch was squashed to a single commit that includes the record_success guard fix