fix(proxy): retry lock-timed-out daily spend batches in place so the shutdown flush keeps them

[mateo-berri]

TLDR

Problem this solves:

• CircleCI integration-accounting red on every scheduled main run since fix(proxy): bound daily spend rollup row-lock waits with lock_timeout and requeue 55P03 #44450
• A restarting multi-worker proxy drops daily user spend blocked behind a sibling commit

How it solves it:

• The daily spend writer retries a lock_timeout (55P03) batch in place
• Same retry rule fix(proxy): bound daily spend rollup row-lock waits with lock_timeout and requeue 55P03 #44450 already gave the other spend writers

User Flow

Before: an admin restarting a 2-worker proxy under load sees a user's daily request count fall short of the team's

1. A developer sends a burst of 30 POST http://localhost:4000/v1/chat/completions with a key owned by a user on a team
2. The admin sends SIGTERM to the proxy while one worker's daily spend write is still committing
3. GET http://localhost:4000/user/daily/activity reports 11 requests for that user while GET /team/daily/activity reports 31

After: the same restart keeps every request, so both views agree

1. A developer sends a burst of 30 POST http://localhost:4000/v1/chat/completions with a key owned by a user on a team
2. The admin sends SIGTERM to the proxy while one worker's daily spend write is still committing
3. GET http://localhost:4000/user/daily/activity and GET /team/daily/activity both report 31 requests

CircleCI

Red on main, job integration-accounting, failing test tests/integration/spend/test_shutdown_flush.py::test_daily_spend_burst_across_two_workers_survives_shutdown_during_commit_exactly_once

• Pipeline 91007 (2026-10-08 00:09 UTC): https://app.circleci.com/pipelines/github/BerriAI/litellm/91007/workflows/022d7430-e5bb-4d30-a99b-11059af1df04/jobs/2249769
• Pipeline 91000: https://app.circleci.com/pipelines/github/BerriAI/litellm/91000/workflows/50307c2c-3a5b-4eb6-ad5a-3f1fb5cd005f
• Pipeline 90996: https://app.circleci.com/pipelines/github/BerriAI/litellm/90996/workflows/50c7c9c8-4f29-4612-8842-c7a3aa18dfa3

Culprit: 2477635 fix(proxy): bound daily spend rollup row-lock waits with lock_timeout and requeue 55P03 (#44450). Bisect over the scheduled green/red range: its parent passes every time, it and every later commit fail most runs

Mechanism: #44450 runs every spend rollup transaction under lock_timeout=5s. In DBSpendUpdateWriter._update_daily_spend a 55P03 counts as requeue-safe, but the in-place retry loop only retries connection errors and deadlocks, so the batch leaves immediately and _flush_daily_spend_queue puts it back on the in-memory queue "for retry on next tick". The shutdown flush (flush_spend_counters_on_shutdown) is the last tick, so a daily batch waiting more than 5s on a row a sibling worker is still committing is lost. #44450 already documents 55P03 as retry-safe and retries it in _handle_spend_update_retry; the daily writer was missed

Fix: _update_daily_spend treats PrismaDBExceptionHandler.is_lock_timeout_error as retryable alongside deadlocks, with the existing backoff and n_retry_times. Once retries run out the batch is still requeued as before, so normal ticks keep #44450's bounded waits. New unit test test_update_daily_spend_retries_lock_timeout_errors

PR-branch CircleCI verification: https://app.circleci.com/pipelines/github/BerriAI/litellm/91047 at f56a891 (main 9f150f7 merged in, PR diff unchanged). On https://app.circleci.com/pipelines/github/BerriAI/litellm/91042 at 69d5750 test_shutdown_flush passed in integration-accounting (job 2251802) and every other failure matched main 91036 except intermittent replica and migration jobs; schema-migration passed on rerun and the Lens pagination flake is fixed on main by #45346, now merged in. Earlier https://app.circleci.com/pipelines/github/BerriAI/litellm/91023 at 436e83b (earlier 91018 at 4733842, where test_shutdown_flush passed while it failed on main pipeline 91016, and 91015 at 74d4a2c). Baseline on current main (03ba79d) for jobs this PR does not touch: https://app.circleci.com/pipelines/github/BerriAI/litellm/91016

Deferred (distinct root causes, equally red on main, left for the next run):

• integration-providers: test_cache_control_tool_call_marks_wire.py response-cache tests send 5+ message conversations that fix(caching): skip the cache past max_messages and keep tool_result text in semantic prompts #43878's max_messages=4 cache cap now skips by design; also usage-based routing Redis tests
• integration-accounting: test_vertex_image_regional_uplift.py burst tests, red since they landed in fix(vertex_ai): apply regional endpoint uplift on image generation cost path #44679
• integration-database, integration-database-replica: test_partition_transactions::test_real_partition_ddl_survives_witnessed_lock_and_is_idempotent
• integration-extensions: test_passthrough_upstream_error_chaos::test_passthrough_worker_sigkill_leaves_sibling_serving_and_logging
• integration-management: test_cli_sso_login_ui_disabled::test_burst_of_logins_recovers_from_an_idp_token_outage
• proxy-behavior: Lens test_scan_lifecycle_persists_results_and_revokes_worker, and e2e_ui_testing, e2e_ui_testing_server_root_path, red identically on main pipeline 91016
• integration-providers on 91023 also fails the test_decisions_basic_* and test_decisions_wire tests and a realtime transcription guardrail test that arrived with the main merge (feat(decisions): serve System One format at /v1/systemone and OpenAI format at /v1/decisions #45184, feat(decisions): add OpenAI as a Decisions provider behind a shared decisions format #45214), not in this diff
• GitHub rust-wheel (test_schema_setup_uses_configured_retention) fails on main d6e453f too; neither it nor rust-test is a required check

Pre-Submission checklist

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Live A/B on a real proxy (litellm --num_workers 2, real Postgres, real OpenAI gpt-4o-mini), merge base 0e09e99 vs 74d4a2c (later commits up to 436e83b only touch the unit test or merge main, product code is identical). A user key on a team sends 1 request, a 15s slow COMMIT is injected on that user's daily row (same fault as the CircleCI test), 30 concurrent POST /v1/chat/completions follow, the proxy gets SIGTERM mid-commit and is booted again, then GET /user/daily/activity and GET /team/daily/activity are read

Leg	Run	user api_requests	team api_requests
before	1	20	31
before	2	31	31
after	1	31	31
after	2	31	31

The loss is a race (a sibling commit must overlap the 5s lock_timeout), which is why CircleCI flakes rather than failing every run. Without a shutdown and with the same slow commit, both commits record exactly 40 requests for user and team (no loss, no double count), and /v1/chat/completions, /v1/messages and /v1/responses bill identically on both

CircleCI on this branch: https://app.circleci.com/pipelines/github/BerriAI/litellm/91047 (latest), https://app.circleci.com/pipelines/github/BerriAI/litellm/91042, https://app.circleci.com/pipelines/github/BerriAI/litellm/91023

Type

🐛 Bug Fix

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

• 436e83b passes /live-pr-risk (f56a891 only merges main, PR product diff unchanged)

---

Note

Medium Risk
Touches proxy spend accounting commit paths under contention; behavior change is narrowly scoped to retrying safe lock-timeout rollbacks, with a regression test.

Overview
Daily spend commits now retry Postgres lock_timeout (55P03) inside _update_daily_spend, matching the retry rules already used by the other spend writers via handle_spend_update_failure.

When a bulk daily upsert hits a lock timeout, the batch is retried with the existing backoff instead of failing out and relying on a later queue tick. That matters on shutdown flush, where there is no next tick and under-counted user daily activity could diverge from team totals after a multi-worker restart under load.

A unit test asserts that a simulated 55P03 on the first attempt triggers a second execute_raw and drains the in-memory transaction dict without invoking the failure handler.

^{Reviewed by Cursor Bugbot for commit f56a891. Bugbot is set up for automated code reviews on this repo. Configure here.}

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Confidence Score: 5/5

[Medium risk] Adds lock timeout retry logic to daily spend batch updates.

The PR appears safe to merge, with no blocking issues found

What we checked:

• Retry uses a fresh transaction: Each retry enters _spend_update_tx again. The failed transaction exits before the retry starts

Summary

The daily spend writer now retries 55P03 lock timeouts in place, using its existing retry limit and backoff. This gives a shutdown flush another chance to write a blocked batch

• Adds a test for a lock timeout followed by a successful retry
• Both earlier, unnumbered findings are fixed: the test has the required annotations and assertion messages
• No new actionable issues were found

_{Reviews (4) · Last reviewed commit: "Merge remote-tracking branch 'origin/mai..." · Reviewed by Greptile}

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_circleci_heal_daily_spend_lock_timeout (f56a891) with main (49266e3)¹}

Footnotes

1. No successful run was found on main (a81f507) during the generation of this report, so 49266e3 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

bugbot run

[mateo-berri]

@greptileai

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit f56a891. Configure here.}