perf(proxy): request-scoped RedisBatch plan for pre-call identity, spend, rate limit and routing reads

[devin-ai-integration]

TLDR

Problem this solves:

• Pre-call still pays one serial Redis round trip per subsystem
• Auth MGET, spend MGET, routing MGET, rate limit Lua each go alone
• Warm chat request: 8 sequential pre-call Redis round trips

How it solves it:

• RedisBatch: declare mget/get/evalsha/incrbyfloat/set/delete, one pipeline(transaction=False)
• RedisRequestPlan (ContextVar, per request): subsystems declare, first await runs the round
• Auth prefetch, spend counters, routing prefetch ride one round; rate limit Lua order kept
• Warm chat request: 4 sequential pre-call rounds, decisions unchanged

Root cause in one sentence: each pre-call owner (auth prefetch, spend counter batch, rate limiter, routing read) issues its own Redis call even though every key is derivable before any of them awaits, so independent reads serialize into separate round trips

User Flow

Before: a developer's chat request is admitted, but the proxy spends eight serial Redis hops before it even picks a deployment

1. They send POST http://localhost:4000/v1/chat/completions with a virtual key, "model": "gpt-group" and "user": "perf-enduser"
2. The proxy reads the key's team and user, then the end-user spend, then the team membership, then rate limits, then deployment cooldowns, one Redis hop after another
3. They get HTTP 200 with the completion; with Redis a few ms away, most of the pre-call latency was those hops

After: the same request is admitted with the same decisions, and the independent reads leave in one hop

1. They send the same POST http://localhost:4000/v1/chat/completions with the same key and body
2. The proxy reads team, user, end-user spend and deployment cooldowns in one pipelined hop, then runs the rate limit scripts in the same order as before
3. They get the same HTTP 200 with the same completion

Linear ticket

Resolves LIT-8882

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Stacked on #43369 (base fe94831). Same fixture bytes for both arms, real Redis at 127.0.0.1:6379, real Postgres, mock deployments so the only network in the request path is Redis. Both proxies ran from their own checkout with PYTHONPATH=<checkout>:<checkout>/enterprise plus a sitecustomize.py that wraps redis.asyncio.client.Redis.execute_command and Pipeline.execute and appends one line per round trip (a pipeline or a Lua call counts once). Timer jobs (_sync_in_memory_spend_with_redis) are filtered out. Config: usage-based-routing-v2 with router_settings.redis_host, litellm_settings.cache redis, enable_redis_auth_cache: true, a team, user, end user and key each with budgets and tpm/rpm limits

model_list:
  - model_name: gpt-group
    litellm_params: {model: openai/gpt-dep-a, api_key: fake, mock_response: "mock reply from a", tpm: 1000000, rpm: 100000}
    model_info: {id: gpt-dep-a}
  - model_name: gpt-group
    litellm_params: {model: openai/gpt-dep-b, api_key: fake, mock_response: "mock reply from b", tpm: 1000000, rpm: 100000}
    model_info: {id: gpt-dep-b}
router_settings:
  routing_strategy: usage-based-routing-v2
  redis_host: 127.0.0.1
  redis_port: 6379
  enable_pre_call_checks: true
litellm_settings:
  cache: true
  cache_params: {type: redis, host: 127.0.0.1, port: 6379}
  enable_redis_auth_cache: true
general_settings:
  master_key: sk-perf-master
  database_url: os.environ/DATABASE_URL

{"model":"gpt-group","user":"perf-enduser","messages":[{"role":"user","content":"Reply with the single word: pong B"}],"max_tokens":5}

Before (fe94831)

1. python litellm/proxy/proxy_cli.py --config ~/perf_rt/config_full.yaml --port 4010, one warm-up request, then:

curl -s -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  http://127.0.0.1:4010/v1/chat/completions -d @chat_warm.req.json

2. Observed: HTTP 200, "mock reply from a". Redis round trips in request order (pre-call above the line, 8 sequential hops):

cmd MGET perf-team_perf-user          auth_object_prefetch._fill_from_redis
cmd MGET spend:end_user:perf-enduser  spend_counter_batch._fetch
cmd GET  team_membership:perf-user:perf-team
cmd SET  team_membership:perf-user:perf-team
cmd EVALSHA 1e1d6e80...   parallel_request_limiter_v3 batch rate limiter script
cmd EVALSHA cbf6fb1b...   parallel_request_limiter_v3 per-descriptor TPM script
cmd EVALSHA cbf6fb1b...   parallel_request_limiter_v3 per-descriptor TPM script
cmd MGET deployment:gpt-dep-a:cooldown   routing shared read (cooldowns + usage)
------------------------------------------------ post-call, unchanged
pipeline 4:INCRBYFLOAT+EXPIRE+INCRBYFLOAT+EXPIRE global_router:...:tpm
pipeline 2:INCRBYFLOAT+TTL gpt-dep-a:gpt-dep-a:tpm
cmd MGET default_user_id:spend
cmd MGET global_router:gpt-dep-b:...:tpm
cmd EVALSHA bacdccaf...
pipeline 2:INCRBYFLOAT+EXPIRE ...:rpm

Total 14 round trips, 8 of them sequential pre-call. A colder run (L1 expired, auth objects re-read from Redis) measured 30

After (7ec1f03)

1. Same command on the fixed checkout, port 4011, same key and body:

curl -s -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  http://127.0.0.1:4011/v1/chat/completions -d @chat_warm.req.json

2. Observed: HTTP 200, "mock reply from b" (usage-based routing alternates deployments; same decision path). Redis round trips in request order:

pipeline 1:MGET deployment:gpt-dep-a:cooldown   redis_batch.execute  } same plan round, one pipeline per
pipeline 2:MGET+MGET spend:end_user:perf-enduser redis_batch.execute } Redis client, run concurrently
pipeline 1:EVALSHA 1e1d6e80...  redis_batch.execute  batch rate limiter script
pipeline 1:EVALSHA cbf6fb1b...  redis_batch.execute  TPM descriptor 1 (serial, stops at first reject)
pipeline 1:EVALSHA cbf6fb1b...  redis_batch.execute  TPM descriptor 2
------------------------------------------------ post-call, unchanged
pipeline 4:INCRBYFLOAT+EXPIRE+INCRBYFLOAT+EXPIRE global_router:...:tpm
pipeline 2:INCRBYFLOAT+TTL gpt-dep-b:gpt-dep-b:tpm
cmd MGET default_user_id:spend
pipeline 1:MGET spend:end_user:perf-enduser   redis_batch.execute (post-call spend read rides the plan)
cmd EXPIRE gpt-dep-b:gpt-dep-b:tpm
cmd EVALSHA bacdccaf...
cmd DEL team_membership:perf-user:perf-team

Pre-call: 4 sequential rounds (round 1 is two pipelines on two Redis clients, issued concurrently) against 6 on the base for the same keys (the team_membership GET+SET pair on the base leg is L1 state, present or absent on both arms depending on the previous request). On the earlier cold comparison the fixed build measured 18 total round trips against 30

3. Same request three times on each proxy: every leg HTTP 200, admitted, same team/user/end-user spend keys read, routing alternated between both deployments on both arms

4. Admin UI on the fixed proxy (http://127.0.0.1:4011/ui, logged in with the master key): Test Key page sends Reply with the single word: pong B to gpt-group and gets mock reply from b; Logs page shows the fixture request as Success with team perf-team, model gpt-group, end user perf-enduser

Regime

Warm L1 (key row cached), Redis standalone (not Cluster), two mock deployments, budgets and limits set but far from exhausted. Rejection paths (over budget, over rpm, cooled down deployment) are covered by the unit tests below, not by this live run

Type

🚄 Infrastructure

Caveats (if any)

Medium

• general_settings.redis_request_pipeline and the speculative mode from the ticket are not in this PR
  • This PR is the two_phase shape only: reads in round 1, rate limit Lua after, reservation writes unchanged
  • Follow-up ticket: speculative reservations + compensation pipeline, and the mode knob (config + Admin UI)
• Rate limit Lua keeps its serial order (batch script, then one round per descriptor, stopping at the first reject) so a rejected request never increments a later counter
• team_membership GET+SET each request is pre-existing: the spend writer deletes that key post-call, so it is cold every time

Low

• Live run had response caching on, so the mock LLM leg was a cache hit on both arms; the pre-call path measured here runs before the cache lookup and is identical in shape on both arms
• Routing prefetch declares usage keys for the full deployment list; the later read consumes it only when its keys are a subset, otherwise it reads again and applies the orphaned prefetch to memory
• Round 1 is one pipeline per RedisCache instance; router and cache Redis are separate clients in this config, so two concurrent pipelines
• Auth write-back SETs are resolved during auth, not deferred to request exit, so the invalidation race window is the same as today's awaited write
• A batch whose pipeline returns a per-command error logs a Redis failure service event but does not feed the circuit breaker (item errors are not connectivity failures)

Taxonomy audit

Greptile round 1 (on 0cf3bc6, 2/5): all four findings fixed in 7ec1f03: later descriptors incremented before an earlier reject was known (serial order restored), deferred auth write-back could restore an invalidated grant (resolved during auth), per-command pipeline errors reported as success (failure event logged), unused spend MGET flushed at request exit (RedisBatch.discard on close)

Audited the full diff against blocks A-BB before pushing. Fixed: B4 (a cancelled round left declared futures pending; execute now fails every future on BaseException and re-raises non-Exception), H4 (four lines over 120 chars), dead before_execute API removed. Judged not applicable: A1/A2 (adapters only read; write-back SETs use the same codec and TTL path), B1 (non-transactional pipeline carries independent commands, no read-modify-write), C series (async_batch_get_cache_shared signature and behavior kept, all public APIs additive), D/Z (prefetch is best-effort warm-up, enforcement getters still fail closed on their own reads), E series (refund semantics preserved and extended to later applied groups via the Lua status slot), M1/M4 (nothing credential-bearing is logged), T8 (new tests land in covered shards). H lint: no Optional[X], no bare dict/Any, no docs in source, no em dashes, grep litellm.proxy litellm/caching/redis_batch.py litellm/caching/redis_request_plan.py empty (no cyclic import into the proxy hub)

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/7d8fee3bf51542e38aaad9e757595014
Open in Devin Desktop: https://app.devin.ai/desktop/session/7d8fee3bf51542e38aaad9e757595014?variant=devin
Requested by: @yassin-berriai

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[devin-ai-integration]

@greptileai

[greptile-apps]

Confidence Score: 5/5

[High risk] Adds request-scoped Redis batching to proxy pre-call operations.

The PR has no newly established blocking issue, though the existing circuit-breaker finding remains unresolved.

Findings

1.  Failed commands count as success ▶

Summary

The PR introduces request-scoped Redis batches for auth, spend, and routing reads, and pipelines rate-limit scripts. The latest changes restore serial descriptor decisions, complete auth cache writes during authentication, report command-level failures, and discard unused spend reads.

_{Reviews (2) · Last reviewed commit: "perf(proxy): restore serial rate-limit L..."}

[greptile-apps]

Failed commands count as success

The pipeline returns individual command errors as result slots. This code fails the affected future but then records a circuit-breaker success and a successful batch event. Repeated command failures therefore do not count toward the breaker's failure threshold, and monitoring reports those batches as successful.

[devin-ai-integration]

Fixed in 7ec1f03: a per-command error now logs a Redis failure service event. Breaker stays connection-level by design, item errors are not connectivity failures

[codecov]

Codecov Report

❌ Patch coverage is 95.28302% with 20 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/caching/redis_batch.py	96.68%	5 Missing ⚠️
litellm/router_utils/routing_read_batch.py	91.66%	5 Missing ⚠️
litellm/caching/redis_cache.py	33.33%	2 Missing ⚠️
litellm/proxy/auth/auth_object_prefetch.py	90.90%	2 Missing ⚠️
litellm/proxy/auth/user_api_key_auth.py	75.00%	2 Missing ⚠️
.../proxy/middleware/redis_request_plan_middleware.py	89.47%	2 Missing ⚠️
...itellm/proxy/spend_tracking/spend_counter_batch.py	95.23%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[devin-ai-integration]

@greptileai

[devin-ai-integration]

Closing in favour of #43407, the P2 PR going forward for LIT-8882 (Yassin's call)