feat(proxy): opt-in budget rollover carrying overage into the next window - #38514
Conversation
…ndow Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
|
PR #38514 is a Devin-authored PR in BerriAI/litellm, but it has no |
|
|
Greptile SummaryThis PR adds opt-in budget rollover across direct, linked, end-user, and per-window budget resets
Confidence Score: 5/5The PR appears safe to merge The previously reported cascade ordering issue is fixed by zeroing under-cap rows before decrementing over-cap rows, and no blocking failure remains
|
| Filename | Overview |
|---|---|
| litellm/proxy/common_utils/reset_budget_job.py | Implements rollover calculation, ordered cascade resets, direct resets, counter synchronization, and per-window rollover without leaving the previously reported ordering defect |
| litellm/repositories/unit_of_work.py | Adds atomic decrement payloads for direct and linked spend resets |
| tests/test_litellm/proxy/common_utils/test_reset_budget_job.py | Covers enabled and disabled rollover, direct and cascade paths, end users, counters, and sequential statement execution |
| litellm/proxy/proxy_server.py | Registers budget rollover in the Admin UI general-settings schema |
| litellm/constants.py | Allows the new rollover flag to be safely loaded from database-backed settings |
Reviews (2): Last reviewed commit: "fix(proxy): zero under-cap rows before d..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| for budget_id, cap in cascade.rollover_caps.items(): | ||
| writes.queue_spend_decrement( | ||
| where={"budget_id": budget_id, **extra, "spend": {"gt": cap}}, amount=cap | ||
| ) # mutable-ok: prisma where filter must be a dict | ||
| writes.queue_spend_zero( | ||
| where={"budget_id": budget_id, **extra, "spend": {"gt": 0, "lte": cap}} | ||
| ) # mutable-ok: prisma where filter must be a dict |
There was a problem hiding this comment.
When a linked row is above its cap but no more than twice the cap, the first sequential update_many decrements it into the 0 < spend <= cap range and the following update resets it to zero, forgiving the overage instead of carrying it forward. The end-user reset at lines 199-204 has the same overlapping predicates.
Knowledge Base Used: Spend, budgets, and rate limits
There was a problem hiding this comment.
Fixed in 9caa257: the zero now runs before the decrement, plus a sequential-replay regression test.
| @@ -1222,7 +1350,7 @@ async def _reset_budget_common( | |||
| still holds the pre-reset value, admitting requests past the cap. | |||
There was a problem hiding this comment.
Assigning the calculated rollover directly to item.spend mutates the helper's input parameter, introducing hidden caller-visible state changes and making the reset flow dependent on mutation order. Construct an immutable reset value instead, as required by the repository's immutability guidance.
Context Used: CLAUDE.md (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
There was a problem hiding this comment.
In-place mutation is this helper's documented contract; callers return the same object. Keeping it avoids a wider refactor here.
There was a problem hiding this comment.
You're right — given that _reset_budget_common is documented to mutate the supplied object and callers return that same object, the assignment is consistent with the helper's contract. My immutability concern doesn't apply here, so no change is needed for this comment.
…cascade resets Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
|
@greptileai please re-review at 9caa257: cascade write ordering fixed and sequential-replay tests added. |
…et (BerriAI#39726) Adapted for stable/1.99.x: this line predates budget rollover (BerriAI#38514), so each end user's spend counter is zeroed through the existing counter_keys list instead of counter_resets, and its cache entry is evicted via end_user_cache_key. Signed-off-by: amasen02 <amasen02@users.noreply.github.com> (cherry picked from commit daced81)
…BerriAI#40639) Adapted for stable/1.99.x: this line predates budget rollover (BerriAI#38514), so the fix is applied to _commit_budget_cascade_once directly. End users reset on the budget link plus a NULL budget_id branch for the default tier, which is what upstream's _queue_enduser_resets does with rollover off. The rollover test hunk is dropped. (cherry picked from commit 8a4fae0)
2 participants
TLDR
Problem this solves:
How it solves it:
budget_rolloverlitellm setting (default off, admin UI configurable)max_budgetcarries forward:carried = max(0, spend - max_budget)decrementso spend landing mid-reset is never erasedUser Flow
Before: a proxy admin caps a key at $100/month; a user who spends $150 gets a clean $100 again next month, so the overage is free
{"max_budget": 100, "budget_duration": "30d"}"spend": 0.0, so the $50 overage is forgivenAfter: with rollover enabled, the overage is deducted from the next window's allowance
budget_rolloverin the Admin UI general settings (orlitellm_settings.budget_rollover: true){"max_budget": 100, "budget_duration": "30d"}"spend": 50.0, so only $50 of budget remains for the new window"spend": 0.0exactly as beforeRelevant issues
Linear ticket
Resolves LIT-3085
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Screenshots / Proof of Fix
Live proxy on
localhost:4000(Postgres + real OpenAI calls),PROXY_BUDGET_RESCHEDULER_MIN_TIME=10 PROXY_BUDGET_RESCHEDULER_MAX_TIME=15so resets fire quickly. Fixture is byte-identical across both arms; the arm is identified by whether thebudget_rolloverfield exists inGET /config/list?config_type=general_settings, never by changing the payload. Both arms exercise the direct key reset path (reset_budget_for_litellm_keys); the cascade/linked-row and per-window paths are covered by the unit tests in this PR.Before (cd63c7e, merge base)
Over-cap spend is forgiven at reset
curl -s "http://localhost:4000/config/list?config_type=general_settings" -H "Authorization: Bearer sk-1234" | grep -c budget_rolloverreturns0(MARKER budget_rollover ABSENT: unfixed build)curl -s -X POST http://localhost:4000/key/generate -H "Authorization: Bearer sk-1234" -d '{"max_budget": 0.00001, "budget_duration": "30s", "key_alias": "lit3085-before"}'returns 200 with the keycurl -s -X POST http://localhost:4000/v1/chat/completions -H "Authorization: Bearer <key>" -d '{"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "say OK"}]}'returns 200 with a real completion; key spend reaches3.3e-05, over the1e-05capcurl -s "http://localhost:4000/key/info?key=<key>" -H "Authorization: Bearer sk-1234"shows"spend": 0.0,"budget_reset_at": "2026-08-27T13:00:30+00:00", so the2.3e-05overage is fully forgivenAfter (9caa257, PR tip)
Over-cap spend carries into the next window
curl -s "http://localhost:4000/config/list?config_type=general_settings" -H "Authorization: Bearer sk-1234" | grep -c budget_rolloverreturns1(MARKER budget_rollover PRESENT: fixed build, setting visible in Admin UI general settings)curl -s -X POST http://localhost:4000/config/field/update -H "Authorization: Bearer sk-1234" -d '{"field_name": "budget_rollover", "field_value": true, "config_type": "general_settings"}'returns{"message":"Field budget_rollover updated","status":"success"}(persists in DB and hot-reloads)curl -s -X POST http://localhost:4000/key/generate -H "Authorization: Bearer sk-1234" -d '{"max_budget": 0.000001, "budget_duration": "30s", "key_alias": "lit3085-tip2"}'returns 200curl -s "http://localhost:4000/key/info?key=<key>" -H "Authorization: Bearer sk-1234"shows pre-reset"spend": 2.55e-06against the1e-06cap"spend": 5.5e-07,"budget_reset_at": "2026-08-27T13:37:00+00:00", exactly2.55e-06 - 2 * 1e-06, one cap deducted per window with the remainder carried forwardUnder-cap spend still resets to zero
"max_budget": 0.00001; pre-reset"spend": 2.55e-06(under cap)"spend": 0.0,"budget_reset_at": "2026-08-27T13:35:30+00:00", unchanged legacy behaviorType
🆕 New Feature
Caveats (if any)
Medium
spendon the new window; no separate "carried over" field in API responsesLow
Final Attestation
@greptileai
Link to Devin session: https://app.devin.ai/sessions/327a7b4156a6492eabc012fb228bd913
Open in Devin Desktop: https://app.devin.ai/desktop/session/327a7b4156a6492eabc012fb228bd913?variant=devin
Requested by: @yassin-berriai