perf(proxy): one MGET and one pipeline for post-call spend counters, no team/user/org refetch on the response path

[devin-ai-integration]

Was stacked on #40834, which is now merged (squash f2e0a5d), so this PR targets litellm_internal_staging directly. The tip b4f81a7 is a merge of staging that resolves the conflicts with that squash; its diff against staging is byte-identical to git diff e75b541bec...0626401197, so every measurement and gate result below, taken at 0626401, describes the same code. At b4f81a7 the changed-file ruff, ruff format, and strict gate pass, and the touched test files re-run as before (Redis cache file alone 76 of 76, auth suites 480 passed, spend batch, Prometheus carried state and prefetch tests all green; the same isolation failure appears when the Redis cache and Prometheus modules are collected together)

TLDR

Problem this solves:

• After the LLM answers, the spend callback does one Redis GET per counter (8 to 12 GETs)
• Reservation reconcile reads the same counters again, one GET each
• Each INCRBYFLOAT is followed by a separate TTL or EXPIRE round trip
• The Prometheus success logger refetches team, user and org from Postgres after the response
• In prod this post-response tail was 6 parallel GETs at ~272 ms each, then 6 increments at ~119 ms each

How it solves it:

• One task-local batch per request: warm checks and reconcile share a single MGET
• All post-call increments go out in one pipeline; INCRBYFLOAT and its TTL command share a round trip
• Increment results are recorded, so later reads in the same request never go back to Redis
• Auth pins immutable team/user/org budget snapshots on the request; the pre-call setup writes them into metadata; Prometheus reads them back instead of calling get_*_object
• Every fallback stays: missing snapshot means the old lookup; a failed pipeline invalidates every counter it touched

User Flow

Before: a developer's chat request returns fine but the gateway keeps talking to Redis and Postgres long after the answer, so p95 under load is inflated

1. They send POST https://litellm-domain/v1/chat/completions with {"model": "gpt-5.6", "messages": [{"role": "user", "content": "hi"}]} and a key that belongs to a team, user and org
2. They get 200 and a normal chat completion body
3. After the body was returned, the gateway issues 9 to 12 single-key Redis GETs, 8 separate increments each followed by a TTL check, and 3 Postgres SELECTs for the team, user and org it authenticated 1 second earlier
4. Under 8 concurrent requests their p95 sits at 160 ms on the local rig (Rust admission on) even though the model call is mocked to ~0 ms

After: the same request returns the same body and the post-response tail is one MGET, one pipeline, no Postgres object lookups

1. They send the same POST https://litellm-domain/v1/chat/completions with the same body and key
2. They get 200 and the same chat completion body
3. After the body was returned, the gateway issues one MGET for every counter it will touch, one pipeline carrying every increment and its TTL, and no Postgres SELECT for team, user or org
4. Under the same 8 concurrent requests their p95 is 127 ms on the same rig, and Redis commands per request drop from 81.5 to 70.3
5. A key denied for the model still gets the same 403 with the byte-identical error body

Relevant issues

Follow-up to #40834 (auth phase). Together they cover the whole-request I/O reduction tracked in LIT-7478

Linear ticket

Resolves LIT-7644

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more (1926 passed across the nine touched files; two order-dependent tests unrelated to this diff pass in isolation, see Tests and gates)
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Design notes

The post-call path in _PROXY_track_cost_callback now runs inside spend_counter_batch_scope(redis, counter_keys=post_call_counter_keys(...)). _is_spend_counter_cache_warm asks the task-local SpendCounterBatch first; the first reader pays one MGET for every bound key, later readers hit the loaded mapping. Budget-window counters that are only known at read time bind on read and join the same MGET when they get there before it fires. Reservation reconcile (_set_reserved_entries_actual_cost) opens the same scope type; nested inside an open outer batch it binds into it instead of starting a second MGET

Writes: increment_spend_counters_pipeline(pending) sends every INCRBYFLOAT plus its EXPIRE in one pipeline and calls record_spend_counter_value with each returned value, so a counter incremented in this request is served from the batch on the next read. RedisCache.async_increment for a single key now sends INCRBYFLOAT and TTL (or EXPIRE) in one pipeline instead of two round trips, and only issues a third EXPIRE when the key turned out to have no TTL. On any pipeline failure every pending counter is invalidated (forget_spend_counter plus in-memory delete) before the error propagates, so a half-applied batch is never read as authoritative

Response path: carry_team_and_user_budget_state and carry_organization_budget_state pin frozen TeamBudgetSnapshot, UserBudgetSnapshot, OrgBudgetSnapshot on UserAPIKeyAuth right where auth already holds the objects. add_litellm_data_to_request writes them into metadata under the existing user_api_key_* names. Prometheus _set_team_budget_metrics_after_api_request, _set_user_budget_metrics_after_api_request and _set_org_budget_metrics_after_api_request build from the snapshot when it is present and otherwise call the same get_*_object they always did. Custom auth, unauthenticated routes and requests where a budget check was skipped therefore behave exactly as before

Not changed on purpose: the rate limiter's own post-call INCRBYFLOAT/TTL/EXPIRE triplets (9 each in the cold trace) and the pre-call rate limiter reads are left alone, as the brief excluded a rate-limiter redesign. Those are the material pre-call round trips left; see the PR (c) decision below

Redis pool diagnosis (why 6 parallel GETs each took ~272 ms in prod)

Measured with redis_pool_diag.py against the repo's RedisCache (redis-py BlockingConnectionPool, max_connections=50) and a local Redis under MONITOR:

condition	per-call ms	GETs seen by Redis	client connections used	arrival spread at Redis
idle event loop, 6 asyncio.gathered GETs	1.4, 1.4, 2.1, 2.0, 1.9, 1.8	6	6	1.28 ms
same 6 GETs, event loop blocked 250 ms right after they were sent	251.0, 251.0, 251.0, 250.8, 250.9, 250.8	6	6	0.02 ms

The GETs leave on six different connections within a millisecond of each other, so nothing serializes them on the Redis side or in the pool. What produces "six parallel calls, each ~272 ms" is the event loop not being free to read the replies: every reply is already in the socket buffer, and all six awaits resolve together once the loop gets control back. In the prod trace the neighbour is batch_write_to_db (433 ms) running in the same loop. The fix in this PR removes five of the six reads rather than the stall itself; the stall itself (what in _PROXY_track_cost_callback / the DB writer holds the loop) is written up as a follow-up on LIT-7644. A third leg that held the Redis server busy with a 200 ms Lua script timed out on the client and is not cited

Screenshots / Proof of Fix

Both runs use the same local rig: staging proxy from the given checkout, PostgreSQL 16.15 with log_statement=all and one application_name per proxy, Redis 7.4.11 under MONITOR, user_api_key_cache_ttl: 2, Prometheus callback enabled, plus the rate limiter v3 and budget reservation on. One master-key-created key with a team, user, org, end user and two tags, plus a second key whose models list only allows real-chat for the denied case. Before is the merge base e75b541 (the tip of #40834), After is this PR's tip 0626401. Both sides run two proxy processes (pod A on 20900, pod B on 20901) sharing the same Redis and Postgres, so the pod B cases show a key whose objects are already in Redis but not in that process's memory. Redis round trips are estimated from MONITOR by grouping commands that arrive on the same client connection within the same pipeline; Postgres counts are statements attributed to this request's application_name in the log, split into the auth, pre-call, post-call, DB-writer and response-path windows

The real-provider run below hits openai/gpt-5.6 (as real-chat) and anthropic/claude-sonnet-5 (as real-claude) with the provisioned provider keys and costs real money. The synthetic matrix further down uses mock_response so latency deltas isolate the gateway; it is labeled as synthetic

Shared setup for every case:

export KEY=$(cat /home/ubuntu/rig/key_live.txt)   # sk-... created via POST /key/generate with team_id, user_id, tags
export DENIED_KEY=$(cat /home/ubuntu/rig/key_live_denied.txt)   # sk-... created with models: ["real-chat"]
CHAT='{"model":"real-chat","messages":[{"role":"user","content":"hi"}],"user":"authio-enduser"}'
MSG='{"model":"real-claude","max_tokens":16,"messages":[{"role":"user","content":"hi"}],"metadata":{"user_id":"authio-enduser"}}'
# cold = proxy freshly restarted and Redis FLUSHALL; hot = one identical unmeasured request 1 s earlier
# measured request wrapped in: redis-cli -p 16900 MONITOR > $LABEL.monitor &  and the Postgres log window

Before (e75b541)

chat, cold cache, pod A (port 20900), real OpenAI call

1. curl -s -w '\n%{http_code} %{time_total}s' -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d "$CHAT" http://127.0.0.1:20900/v1/chat/completions
2. 200, 1430.3 ms, body {"id":"...","created":1789205475,"model":"real-chat","object":"chat.completion","choices":[{"finish_reason":"stop","index":0,"message":{"content":"Hi! How can I help?","role":"assistant","provider_specific_fiel...
3. Redis: 73 estimated round trips, 112 commands. By phase: auth 23, pre-call 19, post-call 26, DB writer 1, response path 4. Pre-call spend counters: GET 6, MGET 0, INCRBYFLOAT 6, EXPIRE 6. Post-call spend counters: GET 12, MGET 0, INCRBYFLOAT 8, EXPIRE 8, SET 1
4. Postgres: 29 statements. By phase: auth 11, pre-call 0, post-call 0, DB writer 14, response path 4. Response path: SELECT - 1, SELECT LiteLLM_ObjectPermissionTable 1, SELECT LiteLLM_ShadowEvalJob 1, SELECT LiteLLM_UserTable 1

chat, same key on pod B (port 20901, Redis warm, in-memory cold), real OpenAI call

1. curl -s -w '\n%{http_code} %{time_total}s' -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d "$CHAT" http://127.0.0.1:20901/v1/chat/completions
2. 200, 1098.7 ms, body {"id":"...","created":1789205498,"model":"real-chat","object":"chat.completion","choices":[{"finish_reason":"stop","index":0,"message":{"content":"Hi! How can I help?","role":"assistant","provider_specific_fiel...
3. Redis: 52 estimated round trips, 79 commands. By phase: auth 9, pre-call 16, post-call 24, DB writer 1, response path 2. Pre-call spend counters: GET 5, MGET 0, INCRBYFLOAT 5, EXPIRE 5. Post-call spend counters: GET 9, MGET 0, INCRBYFLOAT 8, EXPIRE 8
4. Postgres: 19 statements. By phase: auth 1, pre-call 1, post-call 0, DB writer 15, response path 2. Response path: SELECT LiteLLM_ObjectPermissionTable 1, SELECT LiteLLM_ShadowEvalJob 1

messages, cold cache, pod A, real Anthropic call

1. curl -s -w '\n%{http_code} %{time_total}s' -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d "$MSG" http://127.0.0.1:20900/v1/messages
2. 200, 1232.2 ms, body {"model":"real-claude","id":"...","type":"message","role":"assistant","content":[{"type":"text","text":"Hi! How can I help you today?"}],"container":null,"stop_reason":"end_turn","stop_sequence":null,"stop_deta...
3. Redis: 46 estimated round trips, 72 commands. By phase: auth 4, pre-call 16, post-call 23, DB writer 1, response path 2. Pre-call spend counters: GET 5, MGET 0, INCRBYFLOAT 5, EXPIRE 5. Post-call spend counters: GET 9, MGET 0, INCRBYFLOAT 8, EXPIRE 8
4. Postgres: 19 statements. By phase: auth 1, pre-call 0, post-call 1, DB writer 16, response path 1. Response path: SELECT LiteLLM_ShadowEvalJob 1

messages, same key on pod B, real Anthropic call

1. curl -s -w '\n%{http_code} %{time_total}s' -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d "$MSG" http://127.0.0.1:20901/v1/messages
2. 200, 1189.3 ms, body {"model":"real-claude","id":"...","type":"message","role":"assistant","content":[{"type":"text","text":"Hi! How can I help you today?"}],"container":null,"stop_reason":"end_turn","stop_sequence":null,"stop_deta...
3. Redis: 46 estimated round trips, 75 commands. By phase: auth 4, pre-call 16, post-call 23, DB writer 1, response path 2. Pre-call spend counters: GET 5, MGET 0, INCRBYFLOAT 5, EXPIRE 5. Post-call spend counters: GET 9, MGET 0, INCRBYFLOAT 8, EXPIRE 8
4. Postgres: 17 statements. By phase: auth 1, pre-call 0, post-call 0, DB writer 14, response path 2. Response path: SELECT - 1, SELECT LiteLLM_ShadowEvalJob 1

chat, denied (key not allowed for the model), pod A

1. curl -s -w '\n%{http_code}' -H "Authorization: Bearer $DENIED_KEY" -H 'Content-Type: application/json' -d '{"model":"real-claude","messages":[{"role":"user","content":"hi"}],"user":"authio-enduser"}' http://127.0.0.1:20900/v1/chat/completions
2. 403, 39.9 ms, body {"error":{"message":"key not allowed to access model. This key can only access models=['real-chat']. Tried to access real-claude","type":"key_model_access_denied","param":"model","code":"403"}}
3. Redis: 11 estimated round trips, 11 commands, all in auth. Postgres: 18 statements (6 auth, 12 DB writer). No provider call is made

After (0626401)

chat, cold cache, pod A (port 20900), real OpenAI call

1. curl -s -w '\n%{http_code} %{time_total}s' -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d "$CHAT" http://127.0.0.1:20900/v1/chat/completions
2. 200, 2131.6 ms, body {"id":"...","created":1789205251,"model":"real-chat","object":"chat.completion","choices":[{"finish_reason":"stop","index":0,"message":{"content":"Hi! How can I help?","role":"assistant","provider_specific_fiel...
3. Redis: 40 estimated round trips, 93 commands. By phase: auth 21, pre-call 7, post-call 7, DB writer 1, response path 4. Pre-call spend counters: GET 0, MGET 0, INCRBYFLOAT 6, EXPIRE 6. Post-call spend counters: GET 1, MGET 1, INCRBYFLOAT 8, EXPIRE 8, SET 1
4. Postgres: 27 statements. By phase: auth 10, pre-call 0, post-call 0, DB writer 14, response path 3. Response path: SELECT - 1, SELECT LiteLLM_ShadowEvalJob 1, SELECT LiteLLM_UserTable 1

chat, same key on pod B (port 20901, Redis warm, in-memory cold), real OpenAI call

1. curl -s -w '\n%{http_code} %{time_total}s' -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d "$CHAT" http://127.0.0.1:20901/v1/chat/completions
2. 200, 2273.6 ms, body {"id":"...","created":1789205276,"model":"real-chat","object":"chat.completion","choices":[{"finish_reason":"stop","index":0,"message":{"content":"Hi! How can I help?","role":"assistant","provider_specific_fiel...
3. Redis: 26 estimated round trips, 66 commands. By phase: auth 10, pre-call 6, post-call 7, DB writer 1, response path 2. Pre-call spend counters: GET 0, MGET 0, INCRBYFLOAT 5, EXPIRE 5. Post-call spend counters: GET 1, MGET 1, INCRBYFLOAT 8, EXPIRE 8
4. Postgres: 19 statements. By phase: auth 1, pre-call 1, post-call 0, DB writer 15, response path 2. Response path: SELECT LiteLLM_ObjectPermissionTable 1, SELECT LiteLLM_ShadowEvalJob 1

messages, cold cache, pod A, real Anthropic call

1. curl -s -w '\n%{http_code} %{time_total}s' -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d "$MSG" http://127.0.0.1:20900/v1/messages
2. 200, 1225.7 ms, body {"model":"real-claude","id":"...","type":"message","role":"assistant","content":[{"type":"text","text":"Hi! How can I help you today?"}],"container":null,"stop_reason":"end_turn","stop_sequence":null,"stop_deta...
3. Redis: 20 estimated round trips, 61 commands. By phase: auth 5, pre-call 6, post-call 6, DB writer 1, response path 2. Pre-call spend counters: GET 0, MGET 0, INCRBYFLOAT 5, EXPIRE 5. Post-call spend counters: GET 1, MGET 1, INCRBYFLOAT 8, EXPIRE 8
4. Postgres: 19 statements. By phase: auth 1, pre-call 2, post-call 0, DB writer 15, response path 1. Response path: SELECT LiteLLM_ShadowEvalJob 1

messages, same key on pod B, real Anthropic call

1. curl -s -w '\n%{http_code} %{time_total}s' -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d "$MSG" http://127.0.0.1:20901/v1/messages
2. 200, 1136.0 ms, body {"model":"real-claude","id":"...","type":"message","role":"assistant","content":[{"type":"text","text":"Hi there! How can I help you today?"}],"container":null,"stop_reason":"end_turn","stop_sequence":null,"sto...
3. Redis: 20 estimated round trips, 64 commands. By phase: auth 5, pre-call 6, post-call 6, DB writer 1, response path 2. Pre-call spend counters: GET 0, MGET 0, INCRBYFLOAT 5, EXPIRE 5. Post-call spend counters: GET 1, MGET 1, INCRBYFLOAT 8, EXPIRE 8
4. Postgres: 19 statements. By phase: auth 1, pre-call 0, post-call 0, DB writer 15, response path 3. Response path: SELECT - 1, SELECT LiteLLM_ObjectPermissionTable 1, SELECT LiteLLM_ShadowEvalJob 1

chat, denied (key not allowed for the model), pod A

1. curl -s -w '\n%{http_code}' -H "Authorization: Bearer $DENIED_KEY" -H 'Content-Type: application/json' -d '{"model":"real-claude","messages":[{"role":"user","content":"hi"}],"user":"authio-enduser"}' http://127.0.0.1:20900/v1/chat/completions
2. 403, 37.6 ms, body {"error":{"message":"key not allowed to access model. This key can only access models=['real-chat']. Tried to access real-claude","type":"key_model_access_denied","param":"model","code":"403"}}
3. Redis: 10 estimated round trips, 10 commands, all in auth. Postgres: 7 statements (6 auth, 1 DB writer). No provider call is made

Real-provider latency moved 1430 -> 2132 ms (chat cold, pod A), 1099 -> 2274 ms (chat, pod B), 1232 -> 1226 ms (messages cold, pod A), 1189 -> 1136 ms (messages, pod B). Single samples against live providers are dominated by provider time: in the MONITOR timestamps the gap between the last pre-call and the first post-call Redis command (the provider wait) is 1312 -> 2029 ms and 1016 -> 2199 ms for the two chat samples, while the proxy-side time from request receipt to the last pre-call command drops on every row (122 -> 107, 86 -> 78, 42 -> 35, 39 -> 37 ms), so the round-trip and statement counts are the signal: cold chat goes from 73 to 40 estimated Redis round trips (post-call 26 -> 7, pre-call 19 -> 7 because the reservation warm checks join the MGET), pod B chat from 52 to 26, messages from 46 to 20 on both pods. Denied requests stay at 403 with the same body, 10 or 11 Redis commands all in auth, and no provider call. The one remaining post-call GET is the spend-counter SET/GET for the reseed marker, not a warm check. The response-path Postgres statements left on both sides are the shadow-eval job lookup, the object-permission read and, on the cold pod, the spend-counter reseed described above; in this rig the Before side served the Prometheus team/user/org lookups from the in-process cache, so the delayed-response check below is where the removed get_*_object SELECTs show up

Synthetic before/after matrix (mock_response, same rig, gateway time only)

Latency is a single measured request per cell after the same warm-up; the concurrent-load rows are 200 requests at 8 in flight. "idle" is a request after 60 s of no traffic (in-memory caches expired, Redis still warm). The 3464 ms Before idle cell for chat/Rust off is a one-off Redis reconnect the old code hit after the idle gap and is not representative. Matrix captured at 6012071; the commits after it are the typing_extensions.Self import for Python 3.10, a fixture string rename, two extra batch tests, the typed reservation entry (annotation only) and the two merges of #40834 whose runtime change is the prefetch SQL alias fix, which the two-pod real-provider run above covers at the exact tip, so this synthetic section was not re-run

chat (Rust admission off), before -> after

regime	status	Redis rt auth	pre-call	post-call	db writer	response path	Redis rt total	Redis cmds	PG total	latency ms
cold	200	22 -> 21	25 -> 13	27 -> 7	1	4	79 -> 46	135 -> 116	29 -> 28	113.8 -> 116.7
hot	200	1 -> 2	22 -> 12	48 -> 7	1	2	74 -> 24	115 -> 74	17	19.1 -> 20.3
idle	200	0 -> 5	0 -> 12	0 -> 6	0 -> 1	0 -> 2	0 -> 26	0 -> 77	20 -> 23	3464.1 -> 32.3
expired	200	7 -> 8	22 -> 12	22 -> 8	1	2	54 -> 31	92 -> 85	18 -> 23	33.0 -> 33.9
denied	403	5	0	0	0 -> 1	0	5 -> 6	5 -> 6	5 -> 16	22.3 -> 22.1

Concurrent load: p50 115.4 -> 103.0 ms, p95 134.5 -> 114.4 ms, p99 442.5 -> 442.4 ms, 62.8 -> 69.3 rps, Redis commands/request 81.5 -> 70.3, PG statements/request 0.15 -> 0.09, errors none -> none

messages (Rust admission off), before -> after

regime	status	Redis rt auth	pre-call	post-call	db writer	response path	Redis rt total	Redis cmds	PG total	latency ms
cold	200	22 -> 21	25 -> 13	18 -> 5	1	2	68 -> 42	118 -> 107	31 -> 27	137.3 -> 123.3
hot	200	1 -> 2	22 -> 12	19 -> 5	1	2	45 -> 22	76 -> 66	17	20.3 -> 17.2
idle	200	3 -> 5	22 -> 12	18 -> 5	1	2	46 -> 25	79 -> 71	23 -> 19	36.2 -> 28.7
expired	200	7 -> 9	22 -> 12	18 -> 5	1	2	50 -> 29	83 -> 75	22 -> 18	34.7 -> 28.9
denied	403	5	0	0	1 -> 0	0	6 -> 5	6 -> 5	15 -> 4	21.6 -> 24.0

Concurrent load: p50 108.5 -> 102.1 ms, p95 123.4 -> 117.7 ms, p99 444.6 -> 441.1 ms, 66.0 -> 69.6 rps, Redis commands/request 72.3 -> 64.3, PG statements/request 0.12 -> 0.12, errors none -> none

chat (Rust admission on), before -> after

regime	status	Redis rt auth	pre-call	post-call	db writer	response path	Redis rt total	Redis cmds	PG total	latency ms
cold	200	22 -> 21	25 -> 13	27 -> 7	1	4	79 -> 46	135 -> 116	33 -> 27	402.3 -> 395.2
hot	200	1 -> 2	22 -> 12	23 -> 6	1	2	49 -> 23	85 -> 72	17 -> 18	18.2 -> 21.9
idle	200	3 -> 7	22 -> 12	22 -> 6	1	2	50 -> 28	88 -> 81	20 -> 22	35.4 -> 33.7
expired	200	7 -> 9	22 -> 12	22 -> 6	1	2	54 -> 30	92 -> 81	18	36.2 -> 33.2
denied	403	5	0	0	1	0	6	6	15 -> 17	21.7 -> 24.7

Concurrent load: p50 119.5 -> 107.3 ms, p95 160.6 -> 126.6 ms, p99 452.7 -> 442.5 ms, 59.8 -> 66.7 rps, Redis commands/request 81.5 -> 70.3, PG statements/request 0.12 -> 0.09, errors none -> none

messages (Rust admission on), before -> after

regime	status	Redis rt auth	pre-call	post-call	db writer	response path	Redis rt total	Redis cmds	PG total	latency ms
cold	200	22 -> 21	25 -> 13	18 -> 5	1	2	68 -> 42	118 -> 107	27 -> 26	410.4 -> 407.4
hot	200	1 -> 2	22 -> 12	19 -> 6	1	2	45 -> 23	76 -> 68	17	21.5 -> 18.9
idle	200	3 -> 5	22 -> 12	18 -> 5	1	2	46 -> 25	79 -> 71	21 -> 19	32.4 -> 29.1
expired	200	7 -> 9	22 -> 12	18 -> 5	1	2	50 -> 29	83 -> 75	17	34.0 -> 33.0
denied	403	5	0	0	0	0	5	5	5	20.6 -> 25.8

Concurrent load: p50 113.3 -> 104.8 ms, p95 245.2 -> 127.6 ms, p99 448.2 -> 465.4 ms, 61.5 -> 67.1 rps, Redis commands/request 72.4 -> 64.3, PG statements/request 0.20 -> 0.14, errors none -> none

Denied request, all four matrix cells: HTTP 403 -> 403, body byte-identical

Redis commands by call site, cold cache, chat, Rust on (before -> after), only the rows that changed

phase	call site	command	before	after
auth (reservation)	spend_counter	GET	1	0
auth (reservation)	spend_counter	MGET	1	2
auth (reservation)	spend_counter	SET	5	4
pre-call	spend_counter	GET	6	0
post-call	spend_counter	GET	12	1
post-call	spend_counter	MGET	0	1
post-call	other	CLIENT	2	0

Every other row (rate limiter INCRBYFLOAT/TTL/EXPIRE, router MGET, auth object GET/SET) is unchanged, which is the pre-call leftover discussed below

Synthetic delayed-response check of the response path

With mock_response plus a 3 s mock_delay so the post-response window is unambiguous, Rust on, cold cache. Postgres statements in the response-path window:

Before (897d561): SELECT LiteLLM_UserTable 2, SELECT LiteLLM_TeamTable 1, SELECT LiteLLM_OrganizationMembership 1, SELECT LiteLLM_ShadowEvalJob 1

After (6012071; the commits after it are the typing_extensions.Self import for Python 3.10, a fixture string rename, two extra batch tests, the typed reservation entry (annotation only) and the two merges of #40834 whose runtime change is the prefetch SQL alias fix, which the two-pod real-provider run above covers at the exact tip, so this synthetic section was not re-run): SELECT LiteLLM_UserTable 1 (the spend-counter reseed described above), SELECT LiteLLM_ShadowEvalJob 1

Tests and gates

New and extended tests (all uv run pytest, counting fakes over a real RedisCache interface, no monkeypatching of production classes):

• tests/test_litellm/proxy/spend_tracking/test_spend_counter_batch.py: nested scope joins the outer batch; record makes later reads skip Redis; forget sends the next read to Redis; exact MGET count for warm checks; post_call_counter_keys covers tag and access-group counters
• tests/test_litellm/proxy/proxy_server/test_spend_counters.py: one MGET plus one pipeline for the whole post-call path; pipeline failure invalidates every counter; inconsistent counter still reseeds; end-user, tag, access-group and budget-window counters keep their values
• tests/test_litellm/proxy/spend_tracking/test_budget_reservation.py, test_spend_counter_reseed.py: reconcile shares the post-call MGET, pre-call resize against an inconsistent counter still raises
• tests/test_litellm/caching/test_redis_cache.py: INCRBYFLOAT and TTL/EXPIRE in one pipeline, third EXPIRE only for a key without TTL
• tests/test_litellm/proxy/spend_tracking/test_carried_budget_state.py, tests/test_litellm/proxy/auth/test_auth_checks.py, test_user_api_key_auth.py: snapshots pinned from the objects auth holds, metadata round trip, from_metadata returns None on partial metadata
• tests/test_litellm/integrations/test_prometheus_carried_budget_state.py, tests/test_litellm/proxy/test_proxy_server.py: Prometheus reads carried state and calls no get_*_object; missing snapshot falls back to the lookup

Running the ten touched test files together at 0626401: 736 passed, 1 failed. The failure, test_breaker_metrics_track_state_and_failure_class, reads the Prometheus registry and fails whenever any Prometheus test module is collected in the same run: at the merge base e75b541 the same two-file run with the pre-existing test_prometheus_budget_metric_guard.py fails identically, and the file alone passes 76 of 76 at this tip, so it is a test-isolation defect that predates this branch. Denied-request bodies compared byte-for-byte before/after in every regime

State at 0626401:

• make check: PASS at 0626401 (ruff, ruff strict gate, type-discipline gate, basedpyright gate, test-quality gate, OpenAPI/dashboard type sync), full log read, no warnings
• CI: 82 checks passed, 0 failed, 1 skipped (the stage-mirror e2e job, which only runs when an e2e test changes) at 0626401; codecov/patch 96.29% of the diff hit against a 80.77% target; buildkite e2e-tests passed; CodSpeed does not run on this PR's changed files
• Greptile: Confidence 5/5 with the summary refreshed at 0626401 (last reviewed commit is the tip); both of its threads (fixture name, untyped reservation entry) were fixed in later commits and are resolved by Greptile, no findings outstanding
• Bugbot: review on 0626401 reads "found no new issues"; the earlier clean reviews at 189c563 and df2a305 are marked stale by that newer run, and no Bugbot thread is open. Veria posted no check or comment on this PR, so it is recorded as unavailable rather than passed
• Mutation check: 13 hand-written mutants over the carried snapshots (org carry dropped, team and user max_budget fill-in dropped, org spend not added, key reset dropped, team snapshot skipped, user snapshot skipped), the batch (record not caching, forget keeping the stale value, nested scope opening a second batch, tag counters dropped from the post-call keys) and the Redis pipeline (refresh EXPIRE replaced by TTL, third EXPIRE suppressed). At f7e03fb the two batch mutants survived, which is what the two tests in 189c563 add; re-run at 189c563, 13 of 13 are killed
• Concernloop: both bot threads (Greptile, fixture name and untyped reservation entry) were fixed in later commits and Greptile re-triggered at the tip; no Bugbot or Veria findings; /qa is the two-pod real-provider run above; /live-pr-risk is the section below and its checkbox at the end of this description

Live risk (what else depends on the changed code, and what was driven live)

Changed contracts: RedisCache.async_increment keeps its signature and return type and only changes how the TTL/EXPIRE follow-up travels; callers other than the spend path are DualCache.async_increment (parallel request limiter v1/v2, dynamic rate limiter, provider budget limiter), rate limiter v3, the MCP gateway DCR flow and the spend-counter reseed. UserAPIKeyAuth gains three exclude=True fields, so no response body, /key/info, dashboard or SDK type sees them. Request metadata gains seven user_api_key_* entries written through model_dump(mode='json'), so every downstream reader (spend log metadata allowlist, logging callbacks, Prometheus) receives strings and floats, matching the existing user_api_key_budget_reset_at entry. SpendCounterBatch changes are internal to the spend path

Driven live on both sides (base e75b541 vs head 0626401, two proxy processes per side sharing one Postgres and one Redis, rate limiter v3 and budget reservation on, Prometheus callback on, real OpenAI and Anthropic calls): /v1/chat/completions and /v1/messages cold on pod A and warm-Redis on pod B, a denied request, the post-call spend path, the reseed on the cold pod, the rate limiter's own INCRBYFLOAT/TTL/EXPIRE triplets through the new pipeline, and the Prometheus callback. Statuses and denied bodies match between sides, and the db-writer statement set (the LiteLLM_SpendLogs insert, the daily spend inserts, the team, user, org, key, membership and tag spend updates) is identical on both sides; only the Redis and Postgres counts differ, as tabulated above. The removed Prometheus get_*_object SELECTs show up in the synthetic delayed-response check because this rig served them from the in-process cache on the Before side. The synthetic matrix adds Rust admission on/off, expired-cache and 8-way concurrency

Covered by tests but not driven live: a Redis failure between INCRBYFLOAT and EXPIRE, a failed increment pipeline (every touched counter invalidated, test_spend_counters.py), a non-numeric counter payload and a failed MGET (batch hands the read back to the caller, test_spend_counter_batch.py), and a partial carried snapshot (Prometheus falls back to get_*_object, test_prometheus_carried_budget_state.py). Not verified: DualCache.async_increment callers outside the proxy request path (provider budget limiter, dynamic rate limiter, MCP DCR flow) were not exercised on the rig; they call the unchanged signature and the behavior change is one fewer round trip, so the risk is a Redis client that rejects pipeline(transaction=False), which redis-py has supported since 2.x. No lockfile, pyproject.toml, Dockerfile or schema.prisma change, so there is no packaging leg

PR (c) decision

Remaining pre-call round trips after (a) and (b), cold chat with Rust on: rate limiter 6 GET + 4 SET + 6 EXPIRE + 2 INCRBY + 2 TTL (mostly individually issued), 3 EVALSHA, 2 TIME, 1 router MGET. That is well over the two-round-trip threshold, but all of it sits inside the rate limiter v3 and the router's own cache, which the brief excluded from redesign. A PR (c) would be a rate-limiter change, so it is recorded on LIT-7478 as a candidate rather than opened here

Type

🚄 Infrastructure

Caveats (if any)

Medium

• The carried snapshot is taken at auth time; a team/user/org budget edited between auth and response is reflected on the next request, same window the in-memory object cache already had
• INCRBYFLOAT and EXPIRE now travel in one non-transactional pipeline; a Redis failure between them leaves the same state the old two-call path could leave, and the counter is invalidated either way

Low

• Org budget_reset_at is not carried, so the org remaining-hours gauge is set from None on the carried path. The fallback path produces None too, because LiteLLM_OrganizationTable.litellm_budget_table is typed as LiteLLM_BudgetTable, which has no budget_reset_at; no behavior change, noted for a follow-up
• Rate limiter post-call INCRBYFLOAT/TTL/EXPIRE triplets and the pre-call rate limiter reads are unchanged (see PR (c) decision)
• Live-provider latency is a single sample per cell and mostly provider time; use the synthetic concurrent-load rows for gateway latency

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR
• 0626401 passes /live-pr-risk (b4f81a7 carries the identical diff onto the merged base)

Link to Devin session: https://app.devin.ai/sessions/bbe614d754bf4e779ed46e0d131c407e
Open in Devin Desktop: https://app.devin.ai/desktop/session/bbe614d754bf4e779ed46e0d131c407e?variant=devin
Requested by: @yassin-berriai

---

Note

Medium Risk
Changes cross-pod spend counter reads/writes and budget reservation reconcile; incorrect batching or invalidation could skew enforcement, though tests and explicit invalidate-on-pipeline-failure mitigate that.

Overview
This PR cuts Redis and Postgres work on the post-response path: spend counter warm checks, reservation reconcile, and Prometheus budget gauges.

Spend counters: Post-call increment_spend_counters runs inside a task-local spend_counter_batch_scope so warm reads and budget reservation reconcile share one MGET instead of per-key GETs. Increments go through increment_spend_counters_pipeline (single INCRBYFLOAT+EXPIRE pipeline); record_spend_counter_value / forget_spend_counter keep later reads in the same request off Redis. RedisCache.async_increment now pipelines increment with TTL/EXPIRE via _incrbyfloat_with_ttl. Pipeline failures invalidate every touched counter before re-raising.

Prometheus / logging: Auth pins team/user/org budget snapshots on UserAPIKeyAuth and writes them into request metadata; success logging builds budget objects from KeyBudgetSnapshot / TeamBudgetSnapshot / etc. when present, skipping get_team_object / get_user_object / get_org_object on the happy path.

Reservation apply/reconcile and reseed paths are wired into the same batch helpers; behavior when snapshots are missing or Redis fails stays on the previous lookup and invalidation fallbacks.

^{Reviewed by Cursor Bugbot for commit b4f81a7. Bugbot is set up for automated code reviews on this repo. Configure here.}

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

This PR reduces post-call spend-accounting latency by batching Redis reads and writes and carrying auth-time budget state into response-path telemetry.

• Adds task-local spend-counter batches that share MGET results across warm checks, reservation reconciliation, and reseeding.
• Pipelines spend-counter increments with TTL operations and invalidates touched counters when a pipeline fails.
• Carries immutable team, user, organization, and key budget snapshots into request metadata so Prometheus can avoid redundant object lookups.
• Preserves fallback lookups when carried snapshots are unavailable and adds focused coverage for batching, invalidation, metadata, and Redis TTL behavior.

Confidence Score: 5/5

The PR appears safe to merge; no outstanding blocking or non-blocking code issue was established in the current changes.

The current implementation preserves fallback behavior, invalidates uncertain counter state after failed writes, and avoids treating partial carried metadata as authoritative. Both earlier Greptile threads were resolved without explanatory replies and therefore do not remain outstanding.

Important Files Changed

Filename	Overview
litellm/proxy/spend_tracking/spend_counter_batch.py	Extends task-local spend-counter batching with post-call key binding, cached write results, and invalidation support.
litellm/proxy/proxy_server.py	Runs post-call counter preparation and reservation reconciliation in one batch and applies increments through one Redis pipeline.
litellm/proxy/spend_tracking/budget_reservation.py	Batches reservation reconciliation reads and applies consistent adjustments together while preserving inconsistent-counter reseeding.
litellm/caching/redis_cache.py	Combines INCRBYFLOAT with TTL or EXPIRE in a non-transactional Redis pipeline without changing the public increment contract.
litellm/proxy/spend_tracking/carried_budget_state.py	Captures auth-resolved budget state and serializes available snapshots into request metadata.
litellm/integrations/prometheus.py	Consumes carried budget snapshots for post-response metrics and retains existing object-lookup fallbacks.

_{Reviews (7): Last reviewed commit: "Merge remote-tracking branch 'origin/lit..." | Re-trigger Greptile}

[devin-ai-integration]

@greptileai

[codecov]

Codecov Report

❌ Patch coverage is 96.29630% with 9 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/spend_tracking/budget_reservation.py	90.00%	6 Missing ⚠️
litellm/caching/redis_cache.py	92.85%	1 Missing ⚠️
litellm/proxy/db/spend_counter_reseed.py	88.88%	1 Missing ⚠️
...itellm/proxy/spend_tracking/spend_counter_batch.py	98.33%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[devin-ai-integration]

@greptileai

[devin-ai-integration]

@greptileai

[mateo-berri]

bugbot run

[devin-ai-integration]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[devin-ai-integration]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[devin-ai-integration]

@greptileai

[mateo-berri]

bugbot run

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_post_call_spend_batch (b4f81a7) with litellm_internal_staging (f2e0a5d)}

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit b4f81a7. Configure here.}