fix(router): await budget redis pipeline before sync reads (internal copy of #32618)

[devin-ai-integration]

Internal copy of #32618 by @emerzon at its head d3b1ddc, pushed to an origin litellm_ branch with the commits and authors intact because CircleCI never builds a fork head. The fix is the original author's work

TLDR

Problem this solves:

• Router budget sync replaces live spend with an older Redis value
• A cancelled or timed-out flush can drop spend already taken off the queue
• Virtual-key model budgets lack the queue lock the shared flush path expects

How it solves it:

• Await the Redis increment pipeline before reading spend back
• Shield the write and cleanup task; requeue the batch if Redis fails
• Skip the Redis-to-memory overwrite when the flush did not land
• Initialize the same queue lock on the virtual-key model budget limiter

User Flow

Before: a team running two proxy replicas behind one Redis with a provider budget gets a third completion through after the second one used the budget up

1. The admin sets provider_budget_config with an openai budget_limit of 0.00017 for 1d and boots two proxy instances sharing one Redis
2. Their app sends a burst of POST https://litellm-domain/v1/chat/completions requests at 5 per second, each costing about 0.00011, so two fit under the budget
3. The third request still returns 200, so three requests land where the budget covered two, and the 429s that follow report 0.0003372 >= 0.00017
4. A burst of POST https://litellm-domain/v1/responses against a gemini budget lets the same extra request through

After: the same burst is refused from the first request over the budget, on both instances

1. The admin sets the same provider_budget_config and boots the same two instances
2. Their app sends the same burst of POST https://litellm-domain/v1/chat/completions requests
3. The third request returns 429 Exceeded budget for provider openai: 0.0002248 >= 0.00017, and so does every later one
4. A burst of POST https://litellm-domain/v1/responses against the gemini budget refuses the third request too, and fresh connections to either instance are refused as well

Relevant issues

Fixes #32614

Refs #20886

Backports #20888 onto main

Supersedes #32618, whose commits this branch carries unchanged

Linear ticket

Resolves LIT-8390

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The focused tests covering this change pass locally: tests/test_litellm/router_strategy/test_budget_limiter_hotpath.py and tests/unit/proxy/hooks/test_unit_test_max_model_budget_limiter.py, 90 passed at 40e42ff
• My PR passes all required CI/CD checks at the current tip
• My PR scope is as isolated as possible
• The current tip has a Greptile Confidence Score of at least 4/5 (5/5 at 40e42ff)

Changes

A flush task owns the detached batch through the Redis write and its cleanup. Cancellation waits for that task to finish, including while the flush waits for its lock and after repeated cancellation, so an applied batch is not replayed. Failed writes coalesce retained increments by spend key, keep the latest TTL, and skip the Redis read. Synchronization merges spend queued during Redis I/O under the queue lock, and the flush lock serializes overlapping syncs. The virtual-key model budget limiter initializes the inherited queue state

The merge commit d3b1ddc brings the branch up to main and removes stale LIT013 lint markers the merge made unnecessary; it changes no behavior

The commit 40e42ff sums the pending increments by key once per sync and adds that total to the Redis values, instead of scanning the whole queue once per Redis key, so the queue lock is held for a single pass; the merged spend is the same

Screenshots / Proof of Fix

Every case ran the same topology on both sides: two proxy instances (--num_workers 2 each) on two random ports, sharing one Redis 7 on 127.0.0.1:21668 that was flushed before each leg, with real OpenAI, Gemini, and Anthropic calls. Instance A got a serial burst of 17 requests at 5 per second over one keep-alive connection, then fresh connections went B, A, B, A. Each request carries a 260-item filler list so it costs a measurable amount, and asks for the single word pong with max_tokens 5. The virtual key case adds a Postgres database so /key/generate works

Provider budget config, used by the three endpoint cases:

model_list:
  - model_name: openai-chat
    litellm_params:
      model: openai/gpt-5.6-luna
      api_key: os.environ/OPENAI_API_KEY
  - model_name: gemini-responses
    litellm_params:
      model: gemini/gemini-3.8-flash
      api_key: os.environ/GEMINI_API_KEY
  - model_name: anthropic-messages
    litellm_params:
      model: anthropic/claude-haiku-4-5
      api_key: os.environ/ANTHROPIC_API_KEY
router_settings:
  redis_host: 127.0.0.1
  redis_port: 21668
  provider_budget_config:
    openai:
      budget_limit: 0.00017
      time_period: 1d
    gemini:
      budget_limit: 0.00107
      time_period: 1d
    anthropic:
      budget_limit: 0.00124
      time_period: 1d
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
litellm_settings:
  telemetry: false

Deployment, tag, and provider reader config:

model_list:
  - model_name: openai-deploy
    litellm_params:
      model: openai/gpt-5.6-luna
      api_key: os.environ/OPENAI_API_KEY
      max_budget: 0.00017
      budget_duration: 1d
    model_info:
      id: openai-deploy-id
  - model_name: openai-tag
    litellm_params:
      model: openai/gpt-5.6-luna
      api_key: os.environ/OPENAI_API_KEY
  - model_name: gemini-responses
    litellm_params:
      model: gemini/gemini-3.8-flash
      api_key: os.environ/GEMINI_API_KEY
router_settings:
  redis_host: 127.0.0.1
  redis_port: 21668
  provider_budget_config:
    gemini:
      budget_limit: 0.00107
      time_period: 1d
litellm_settings:
  telemetry: false
  tag_budget_config:
    lit8390tag:
      budget_limit: 0.00017
      time_period: 1d
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

Virtual key config (coordination_redis is what gives the proxy's key spend counters a Redis; router_settings.redis_host alone leaves them in memory):

model_list:
  - model_name: openai-chat
    litellm_params:
      model: openai/gpt-5.6-luna
      api_key: os.environ/OPENAI_API_KEY
router_settings:
  redis_host: 127.0.0.1
  redis_port: 21668
litellm_settings:
  telemetry: false
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
  coordination_redis:
    host: 127.0.0.1
    port: 21668

Request bodies ($FILLER is item0 item1 ... item259); the deployment, tag, and virtual key cases use the first body with model set to openai-deploy, openai-tag, and openai-chat:

{"model":"openai-chat","messages":[{"role":"user","content":"Ignore this list: $FILLER. Reply with the single word pong."}],"max_tokens":5,"reasoning_effort":"none"}
{"model":"gemini-responses","input":"Ignore this list: $FILLER. Reply with the single word pong.","max_output_tokens":5,"reasoning":{"effort":"minimal"}}
{"model":"anthropic-messages","max_tokens":5,"messages":[{"role":"user","content":"Ignore this list: $FILLER. Reply with the single word pong."}]}

The burst and the fresh connections, run once per case with $A and $B set to the two instance ports, $TOKEN to the master key (or the generated virtual key), and $HDR empty except -H 'x-litellm-tags: lit8390tag' on the tag case:

args=(); for i in $(seq -w 1 17); do args+=(-o "A$i.json" "http://127.0.0.1:$A/v1/$ROUTE"); done
curl -sS --rate 5/s -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' $HDR -d @$BODY -w '%{http_code}\n' "${args[@]}" \
  | awk '{printf "A req%02d HTTP %s\n", NR, $1}'
sleep 2
n=0; for inst in B A B A; do n=$((n+1)); port=$A; [ "$inst" = B ] && port=$B
  curl -sS -o "${inst}x$n.json" -w "$inst fresh$n HTTP %{http_code}\n" "http://127.0.0.1:$port/v1/$ROUTE" \
    -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' $HDR -d @$BODY
done
cat *.json | grep -o 'Exceeded budget for [^"]*' | sort | uniq -c

The virtual key case first mints the key on A and then runs the burst above with it:

TOKEN=$(curl -sS "http://127.0.0.1:$A/key/generate" -H "Authorization: Bearer $MK" -H 'Content-Type: application/json' \
  -d '{"key_alias":"lit8390-'$(date +%s)'","model_max_budget":{"openai-chat":{"budget_limit":0.00017,"time_period":"1d"}}}' | jq -r .key)

The provider reader case sends two gemini requests to A and then reads both instances:

for n in 1 2; do curl -sS -o gem$n.json -w "A gemini$n HTTP %{http_code}\n" "http://127.0.0.1:$A/v1/responses" \
  -H "Authorization: Bearer $MK" -H 'Content-Type: application/json' -d @body_responses.json; done
sleep 3
for port in $A $B; do curl -sS "http://127.0.0.1:$port/provider/budgets" -H "Authorization: Bearer $MK" | jq -c .; done

Before (cbba146)

Provider budget on /v1/chat/completions

1. Flush Redis, boot both instances at cbba146 with the provider budget config, run the burst with ROUTE=chat/completions
2. Observed: A req01 HTTP 200, A req02 HTTP 200, A req03 HTTP 200, then HTTP 429 for req04 through req17
3. Fresh connections: B fresh1 HTTP 429, A fresh2 HTTP 429, B fresh3 HTTP 429, A fresh4 HTTP 429
4. Every 429 body says Exceeded budget for provider openai: 0.0003372 >= 0.00017 (18 of them): three requests got through where the budget covered two

Provider budget on /v1/responses

1. Same burst with ROUTE=responses and the gemini body
2. Observed: A req01 HTTP 200, A req02 HTTP 200, A req03 HTTP 200, then HTTP 429 for req04 through req17
3. Fresh connections: all four HTTP 429
4. Every 429 body says Exceeded budget for provider gemini: 0.002133 >= 0.00107 (18 of them): again one request more than the budget covered

Provider budget on /v1/messages

1. Same burst with ROUTE=messages and the anthropic body
2. Observed: all 17 burst requests and all 4 fresh connections HTTP 200, no budget error at all
3. After the leg, redis-cli -p 21668 GET provider_spend:anthropic:1d reads 0.0: this endpoint's spend never reaches provider budgets (see Caveats)

Deployment budget on /v1/chat/completions

1. Flush Redis, boot both instances at cbba146 with the deployment, tag, and provider reader config, run the burst with the openai-deploy body
2. Observed: A req01 HTTP 200, A req02 HTTP 200, A req03 HTTP 200, then HTTP 429 for req04 through req17 and all four fresh connections
3. Every 429 body says Exceeded budget for deployment model_name: openai-deploy, litellm_params.model: openai/gpt-5.6-luna, model_id: openai-deploy-id: 0.0003372 >= 1d (18 of them): one request more than the budget covered

Tag budget on /v1/chat/completions

1. Same instances, run the burst with the openai-tag body and -H 'x-litellm-tags: lit8390tag'
2. Observed: A req01 HTTP 200, A req02 HTTP 200, then HTTP 429 for req03 through req17 and all four fresh connections
3. Every 429 body says Exceeded budget for tag='lit8390tag', tag_spend=0.0002248, tag_budget_limit=0.00017 (19 of them)

GET /provider/budgets on both instances

1. Same instances, send two gemini /v1/responses requests to A (A gemini1 HTTP 200, A gemini2 HTTP 200), wait 3 seconds, read both instances
2. A returns {"providers":{"gemini":{"budget_limit":0.00107,"time_period":"1d","spend":0.001422,"budget_reset_at":"2026-09-26T04:13:39.843304+00:00"}}}
3. B returns the same spend of 0.001422 with its own budget_reset_at

Virtual key model budget on /v1/chat/completions

1. Flush Redis, boot both instances at cbba146 with the virtual key config and the database, mint the key on A, run the burst with it
2. Observed: A req01 HTTP 200, A req02 HTTP 200, A req03 HTTP 200, then HTTP 422 for req04 through req17
3. Fresh connections: B fresh1 HTTP 422, A fresh2 HTTP 422, B fresh3 HTTP 422, A fresh4 HTTP 422
4. Every 422 body says LiteLLM Virtual Key: <hash>, key_alias: lit8390-base2-1790311139, exceeded budget for model=openai-chat (18 of them), and redis-cli -p 21668 GET virtual_key_spend:<hash>:openai-chat:1d reads 0.0003372 after the leg

After (40e42ff)

Provider budget on /v1/chat/completions

1. Flush Redis, boot both instances at 40e42ff with the provider budget config, run the burst with ROUTE=chat/completions
2. Observed: A req01 HTTP 200, A req02 HTTP 200, then HTTP 429 for req03 through req17
3. Fresh connections: B fresh1 HTTP 429, A fresh2 HTTP 429, B fresh3 HTTP 429, A fresh4 HTTP 429
4. Every 429 body says Exceeded budget for provider openai: 0.0002248 >= 0.00017 (19 of them): the third request is refused, exactly the first one over the budget

Provider budget on /v1/responses

1. Same burst with ROUTE=responses and the gemini body
2. Observed: A req01 HTTP 200, A req02 HTTP 200, then HTTP 429 for req03 through req17
3. Fresh connections: all four HTTP 429
4. Every 429 body says Exceeded budget for provider gemini: 0.00142575 >= 0.00107 (19 of them)

Provider budget on /v1/messages

1. Same burst with ROUTE=messages and the anthropic body
2. Observed: all 17 burst requests and all 4 fresh connections HTTP 200, identical to Before
3. redis-cli -p 21668 GET provider_spend:anthropic:1d reads 0.0 again, so this endpoint is unchanged by the PR

Deployment budget on /v1/chat/completions

1. Flush Redis, boot both instances at 40e42ff with the deployment, tag, and provider reader config, run the burst with the openai-deploy body
2. Observed: A req01 HTTP 200, A req02 HTTP 200, then HTTP 429 for req03 through req17 and all four fresh connections
3. Every 429 body says Exceeded budget for deployment model_name: openai-deploy, litellm_params.model: openai/gpt-5.6-luna, model_id: openai-deploy-id: 0.0002248 >= 1d (19 of them): the third request is refused

Tag budget on /v1/chat/completions

1. Same instances, run the burst with the openai-tag body and -H 'x-litellm-tags: lit8390tag'
2. Observed: A req01 HTTP 200, A req02 HTTP 200, then HTTP 429 for req03 through req17 and all four fresh connections, identical to Before
3. Every 429 body says Exceeded budget for tag='lit8390tag', tag_spend=0.0002248, tag_budget_limit=0.00017 (19 of them)

GET /provider/budgets on both instances

1. Same instances, send two gemini /v1/responses requests to A (A gemini1 HTTP 200, A gemini2 HTTP 200), wait 3 seconds, read both instances
2. A returns {"providers":{"gemini":{"budget_limit":0.00107,"time_period":"1d","spend":0.00142575,"budget_reset_at":"2026-09-26T04:17:51.854185+00:00"}}}
3. B returns the same spend of 0.00142575 with its own budget_reset_at

Virtual key model budget on /v1/chat/completions

1. Flush Redis, boot both instances at 40e42ff with the virtual key config and the database, mint the key on A, run the burst with it
2. Observed: A req01 HTTP 200, A req02 HTTP 200, A req03 HTTP 200, then HTTP 422 for req04 through req17, identical to Before
3. Fresh connections: B fresh1 HTTP 422, A fresh2 HTTP 422, B fresh3 HTTP 422, A fresh4 HTTP 422
4. Every 422 body says LiteLLM Virtual Key: <hash>, key_alias: lit8390-tip2-1790311243, exceeded budget for model=openai-chat (18 of them), and redis-cli -p 21668 GET virtual_key_spend:<hash>:openai-chat:1d reads 0.0003372 after the leg

The three provider budget cases also ran on this branch merged into main at 0d47347 (merged tree 24c7a0abf4) with the same result as After: chat and responses refuse from the third request (19 x openai: 0.0002248 >= 0.00017, 19 x gemini: 0.0014295 >= 0.00107), messages all 200. Main has moved by three commits since, none touching the router, the budget limiter, or the caches. The legacy CircleCI suite tests/local_testing/test_router_budget_limiter.py reads 12 passed, 1 failed at both cbba146 and 40e42ff, the one failure being the Azure case in Caveats

Type

Bug Fix
Test

Caveats (if any)

Low

• On the virtual key case the third request passes on both sides: that check reads the key's spend straight from Redis and the second request's spend had not landed yet, and the path runs no periodic sync, so the overwrite race never applied there; the PR only seeds the queue state its new flush path expects
• The tag budget case refuses from the third request on both sides, so it checks for a regression rather than showing the race
• Spend from POST /v1/messages never reaches router provider budgets, on main or on this tip ([Bug]: provider budgets never count /v1/responses or /v1/messages spend, so agent traffic bypasses the cap entirely #38176, fix in fix(router): count provider budget spend on every API surface #38172), so that endpoint shows every request 200 on both sides
• The deployment budget error prints the duration where the limit belongs (0.0002248 >= 1d), unchanged main code, tracked in LIT-8634
• On the virtual key path each successful request now awaits its Redis increment inside the logging worker instead of scheduling it, which is how it worked before fix(caching): keep an open Redis circuit breaker open and quiet on the sync read and spend counter paths #40624; the worker runs callbacks concurrently under its semaphore and the Redis circuit breaker bounds an outage, so the client response path is unchanged
• The routing strategies keep their own spend queue in BaseRoutingStrategy, which still clears the queue when a Redis write fails; that path serves rpm and tpm counters, not budgets, and is out of scope here
• Greptile's batch-marker P1 on the original PR is in unchanged main code and is tracked in [Bug]: Completed batch can be skipped by model budget after logging cancellation #42627
• tests/local_testing/test_router_budget_limiter.py::test_provider_budgets_e2e_test fails identically at cbba146 and 40e42ff with an Azure NotFoundError for a deployment the test account no longer has, before any budget code runs
• CircleCI never produced a pipeline for the fork head of fix(router): await budget redis pipeline before sync reads #32618, even after cycling the run-ci label, which is why this copy exists
• CircleCI reds at 40e42ff (pipeline 90379 and its rerun from failed, workflow a88ffcf5) that are also red on main or were fixed on main after this branch's merge base:
  • unit: two test_auth_checks::test_get_end_user_object cases, ten test_jwt_key_mapping tests, and eleven load-sensitive tests outside the budget code (five off_the_event_loop timing checks, the three test_jev_logging_failure_preserves_verdict cases, two test_call_mcp_tool cases, and test_path_source_peak_stays_below_list_pipeline), all red on main pipelines 90363 and 90381 (unit jobs 2214837 and 2215836) and all passing in isolation at cbba146 and 40e42ff
  • local_testing_part1: test_get_model_info_bedrock_cross_region_capability_parity, red on main 90363 and 90381; test_get_model_info_bedrock_region cleared on the rerun
  • local_testing_part2: green on the rerun, so test_openai_stream_options_call_text_completion was a flake
  • build_and_test: two test_keys::test_key_model_list cases, red on main 90363 and fixed there by 1d51a8d (fix(proxy): authorize key model aliases the same way as team aliases #43049), which is newer than the merge base
  • build_and_test: two test_rust_python_harness OCR cases, whose modules e2302be (refactor(ocr): remove the Python OCR execution path and require the Rust route #43081) removed on main while the parametrization still lists them, red on main 90381 too
  • using_litellm_on_windows: the job fails with zero failing tests on both runs, same as main 90363 and 90381

Final Attestation

• Regression tests exercise the changed behavior and identified edge cases

• 40e42ff passes /live-pr-risk

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[greptile-apps]

Confidence Score: 5/5

[High risk] Adds synchronization locks to budget tracking Redis operations.

The PR appears safe to merge based on the reviewed changes

Findings

1.  Redis delays completed requests ▶
2.  Failed pipelines can double spend ▶
3.  Retries mix budget windows ▶

Summary

The PR makes router budget synchronization wait for Redis writes before copying shared spend into memory, retains failed batches for retry, and initializes the inherited queue state for virtual-key model budgets. The change since the previous review aggregates pending spend once per sync instead of scanning the queue for each Redis key

_{Reviews (2) · Last reviewed commit: "perf(router): sum pending budget spend b..."}

[greptile-apps]

Redis delays completed requests When Redis is slow, the awaited success callback waits for this write and any earlier flushes. Otherwise completed requests are delayed.

Knowledge Base Used: Observability and governance

[mateo-berri]

This callback runs on the background logging worker or a stream-end task after the response is sent, so awaiting delays spend recording, not the client

[greptile-apps]

You're right — in these invocation paths async_log_success_event runs after the response has been delivered (or in the background logging worker), so awaiting the Redis flush does not add latency to the client request. It can delay spend recording while Redis is slow, but that is the intended trade-off here to ensure the subsequent sync cannot overwrite memory with stale Redis data. I’m withdrawing the “delays completed requests” concern.

[greptile-apps]

Failed pipelines can double spend If Redis applies some increments before a connection failure, this non-transactional pipeline retries them all. Shared spend rises twice, potentially rejecting valid requests.

Knowledge Base Used: Routing and resilience

[mateo-berri]

A drop between send and reply replays at most the failed batch, an overcount the window reset clears; main dropped that spend, undercounting

[greptile-apps]

That’s fair. A Redis connection drop makes execution ambiguous, so re-queuing the detached batch deliberately favors conservative accounting: it can overcount a batch that Redis partially applied, but avoids the undercount/data loss in the previous fire-and-forget path, and the window reset bounds the effect. I was treating any possible duplicate as an unconditional correctness bug; given this budget-limiter trade-off, I’m withdrawing the finding.

[greptile-apps]

Retries mix budget windows If Redis remains unavailable across a window reset, grouping by key gives old and new spend one expiry. Expired spend can falsely exhaust the new budget.

Knowledge Base Used: Routing and resilience

[mateo-berri]

Only a Redis outage spanning a window reset replays old-window spend, bounded by what queued during the outage; main lost it entirely, undercounting

[greptile-apps]

You're right — this is limited to a Redis outage (or stalled flush) that crosses the budget-window boundary. The retained queue is bounded to increments accumulated during that outage; without this requeue, main drops those increments and undercounts spend. Since the window reset writes the new spend under the same Redis key, replaying the retained batch can temporarily mix old-window spend into the new window, but that is an outage-only edge case rather than the general behavior I implied. I’m withdrawing the broader finding.

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_internal_copy_32618 (40e42ff) with main (0d47347)}

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[codecov]

Codecov Report

❌ Patch coverage is 86.48649% with 15 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/router_strategy/budget_limiter.py	85.98%	15 Missing ⚠️

📢 Thoughts on this report? Let us know!

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 40e42ff. Configure here.}

[mateo-berri]

LGTM