fix(proxy): make per-model budgets track spend, enforce, and report the same counter - #37736
Conversation
|
|
Greptile SummaryThis PR aligns per-model budget enforcement, spend increments, and usage reporting across key, user, compaction, and built-in passthrough paths
Confidence Score: 5/5The PR appears safe to merge No blocking failure remains
|
| Filename | Overview |
|---|---|
| litellm/proxy/auth/user_api_key_auth.py | Propagates and enforces user-level per-model budgets across virtual-key, JWT, and custom-auth request paths |
| litellm/proxy/hooks/model_max_budget_limiter.py | Unifies counter ownership under configured budget models and carries pre-upgrade spend for a bounded transition window |
| litellm/proxy/pass_through_endpoints/pass_through_endpoints.py | Preserves budget metadata for built-in provider passthrough routes while excluding user-defined passthrough handlers |
| litellm/proxy/management_endpoints/internal_user_endpoints.py | Persists user per-model budgets and includes current usage in user information responses |
| ui/litellm-dashboard/src/components/key_team_helpers/ModelMaxBudgetEditor.tsx | Adds the shared licensed editor used by key and internal-user budget forms |
Reviews (19): Last reviewed commit: "fix(proxy): make per-model budgets track..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
5b0c98f to
bd41a9b
Compare
bd41a9b to
0b15382
Compare
|
@greptileai both findings were real and are fixed on 0b15382: JWT path now enforces before it returns, plus a call-graph test. Please re-review. |
0b15382 to
b4c5564
Compare
b4c5564 to
559d0ff
Compare
|
@greptileai auto-register was right and is fixed on 559d0ff. Mapped-key is not reachable: it sets do_standard_jwt_auth=False and falls through to the virtual-key checks. Please re-review. |
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 4 · PR risk: 0/10 |
559d0ff to
5e362cc
Compare
|
@greptileai re-review 5e362cc: native passthrough now attaches the per-model budget metadata, and the dashboard editor is license-gated with unchanged budgets omitted. |
5e362cc to
a28fd62
Compare
|
@greptileai re-review a28fd62. Only change since the last request: the editor render test moved to the integration tier. |
a28fd62 to
cfbf88d
Compare
|
@greptileai re-review cfbf88d: the budget editor now reads either BudgetConfig spelling and carries through tpm_limit and rpm_limit, so editing one model cannot drop another. |
cfbf88d to
8607548
Compare
|
@greptileai re-review 8607548. The custom-auth finding is not correct, and failing closed there would be an outage rather than a hardening.
The virtual-key path already makes the identical call and swallows the identical exception, logging "Unable to get user from db/cache. Setting user_obj to None" ( There is also nothing to bypass. The budget being looked up lives on the row that could not be read, so failing closed refuses traffic on the possibility that a budget exists. Every sibling budget on this path, the key budget, the user Two tests now pin this, with your suggested change as the mutant: Also in this push: |
8607548 to
7ed409a
Compare
|
@greptileai re-review 7ed409a. Correcting one sentence of my previous comment: "there is nothing to bypass" holds only for an absent user row, not for an unreachable database, and the difference is worth stating precisely.
That is a real fail-open. It is pre-existing behaviour of So the rest of the original finding still does not hold: failing closed would 4xx every custom-auth request whose user has no row, which is the normal state for deployments that do not write users to the proxy DB. Three tests now pin all of it, the third asserting the outage case falls through unenforced so the limitation is recorded as known rather than accidental. |
|
bugbot run |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
8099fc1 to
0fb82e9
Compare
|
bugbot run |
|
@greptileai re-review 0fb82e9: the per-model budget editor is no longer offered in bulk user edit, where the payload forwards a fixed field list and would have discarded it silently. |
0fb82e9 to
8136f01
Compare
|
bugbot run |
|
@greptileai re-review 8136f01: the pass-through budget metadata is now attached only on the built-in provider routes, since get_model_from_request deliberately resolves no model for a user-defined pass-through and enforcement is skipped there. |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 8136f01. Configure here.
8136f01 to
3e6f6de
Compare
|
@greptileai re-review 3e6f6de. The premise here is right and the conclusion does not follow, so the built-in provider routes are unaffected. Right: Not right: that product is never registered as a route. It is constructed inside the decorated handler at request time and awaited immediately ( Two independent confirmations. The proof-of-fix leg in the PR description drives real traffic through
|
|
bugbot run |
3e6f6de to
b05e217
Compare
|
On b05e217. The general rule in this finding is right and it does not hold for this model, so the documented spelling is not bypassed. It did point at a real fragility though, which is now pinned. Measured against the installed Pydantic rather than argued: The third line is the control: an unrecognised key leaves Where you are right is that nothing in this repository guarantees it. |
|
bugbot run |
|
@greptileai re-review b05e217: adds regression tests pinning that BudgetConfig's budget_limit and time_period aliases survive model_validate, with a control, so a Pydantic upgrade that stopped applying them fails loudly instead of silently dropping every budget in the documented spelling. |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit b05e217. Configure here.
…he same counter Per-model budgets were three separate things pretending to be one. The enforcement check, the post-call increment and the info endpoints each derived their own cache key, so a budget could refuse traffic at 429 while /key/info reported zero usage, and a Bedrock model id never matched a budget keyed on the bare family name. /user/new echoed a model_max_budget back and stored an empty dict, and nothing enforced a user-scoped per-model budget at all. One owner now builds the counter key from the configured budget model, and enforcement, the increment and the info endpoints all read it. Bedrock ids resolve through the model-cost map. Auth carries the user's budget onto the token on every branch that reaches the spend hook, including JWT and auto-registration. Native passthrough attaches the three budget metadata keys its StandardLoggingUserAPIKeyMetadata does not carry, so /anthropic/... and /bedrock/... traffic is counted and capped like /v1/chat/completions. The dashboard gains the per-model budget editor it never had, on the key create, key edit and internal-user edit forms. It is read-only without an enterprise license, matching the write gate the proxy already enforces, and an untouched budget is left out of an update so an unrelated edit cannot trip that gate. The editor hydrates from either BudgetConfig spelling, since model_max_budget is a plain dict that the proxy stores exactly as the client sent it, and it carries through the fields it does not model. Without both, editing one model would drop another model row entirely and silently discard its tpm_limit and rpm_limit. /user/info refreshes its local copy of the user field by field after a save, so model_max_budget joins that list. Left out, a saved cap read back as the old one when the form was reopened, and clearing the row to recover would then wipe the value that had actually persisted. A zero-dollar cap is the strictest limit expressible, not the absence of one, so it is enforced rather than skipped on falsiness, spend exactly at the cap is refused the way every sibling budget check already refuses it, and a counter that was never written reads as zero spend rather than as unknown. The usage endpoints read every counter in one batched lookup, so a large model_max_budget cannot fan out into one concurrent cache call per configured model. Every auth path honours the same zero-cost skip flag, so none of them can refuse a free request that another serves. The custom-auth helper gains the flag it never had, which also changes its pre-existing key and end-user checks. The compaction summary gate checks the user scope alongside the key and end-user ones. This file propagates all three budgets into the summary subrequest, so enforcing only two let compaction increment a counter it could not be refused by. Custom auth attaches the user's budget to the token unconditionally, since the post-call spend hook reads it there: gating the attach on the same condition as enforcement left the counter uncharged whenever the request was not itself enforceable. An entry that will not validate is skipped rather than raised on, so one malformed scope cannot abort every other scope's increment or turn a config typo into a 500. The edit forms re-seed the budget editor when a different key or user is loaded. Its rows are seeded once and cannot re-read their own value prop, so without this a save wrote the previously loaded record's budgets onto the current one. Only the built-in provider pass-through routes carry the budget metadata. get_model_from_request deliberately resolves no model for a user-defined pass-through, since its body is forwarded verbatim and names an upstream model, so attaching there would charge a counter nothing on that route can refuse.
b05e217 to
81f64a1
Compare
|
@veria-ai please re-review 81f64a1. Your cache-key migration finding was right: the pre-upgrade counter is now carried, with a live three-build proof. |
|
@greptileai re-review 81f64a1: a per-model counter written by the previous release is now honoured for the rest of its window. |
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 81f64a1. Configure here.
tin-berri
left a comment
tin-berri
left a comment
There was a problem hiding this comment.
Reviewed via background agent given size (3654/1039 across 35 files) and billing-enforcement stakes. Verdict: the core claims all check out against the actual diff, not just the PR description.
- One counter key:
resolve_model_budget()/model_budget_spend_cache_key()is the sole matcher shared by enforcement (_is_entity_within_model_budget), the post-call increment, and the reporting endpoints (/key/info,/user/infonow read the samedual_cachethe limiter writes to) — this is the exact fix for "blocked at 429, reports zero usage." - Zero-cap bug: genuinely fixed — the comparison flips to
>=(spend-at-cap now refused) and the zero-cap short-circuit is corrected, with a direct test (test_a_zero_dollar_cap_blocks_the_model) assertingBudgetExceededError. - User-level enforcement: wired into all three auth surfaces (virtual-key, JWT/mapped-key, custom-auth) via the same resolver, each with a
BudgetExceededError-raising test — this was a real, previously-unenforced gap, now closed. - Bedrock resolution: correctly gates on
litellm_provider.startswith("bedrock")before splitting a dotted id (avoiding false positives on e.g.azure/gpt-4.1), and delegates the actual base-model resolution to the existing, already-battle-testedget_bedrock_base_model()rather than reinventing cross-region/ARN/provisioned-throughput handling. - Migration compat: real bounded shim (
_legacy_request_model_spend_cache_key) that carries a pre-upgrade counter for one process-lifetime window, not a silent reset — with the two real limitations (reporting lag, model-string match requirement) honestly documented and tested rather than hidden. - Passthrough spend: genuinely attached, gated on the same predicate used for tracking so tracking/enforcement can't disagree about which routes qualify, plus an explicit test that it can't be forged via request body.
- Several bot-flagged "P1"/"High" findings were spot-checked against actual head code (not just the labels) and are either already fixed or false positives (e.g. Cursor's pydantic alias-bypass claim doesn't reproduce against pydantic 2.13.4's actual
model_validatebehavior).
One non-blocking loose end: the per-model budget editor in user_edit_view.tsx seeds its state via useSeededState(user_id, ...), which only re-seeds on a user_id change — if userData refreshes with real budgets after an initial incomplete-data mount, the editor could show stale/empty rows and a save could clear existing budgets. UI-only, gated behind premium license + admin role, not a backend billing-bypass — worth a quick look, not a blocker.
CI green (72/72). Approved.
Two whole-tree test lints are red on litellm_internal_staging, which blocks the lint job on every PR into it. test_user_api_key_auth.py used pytest.raises(Exception) with no match=. B017 forbids that (enforced since #37731): any Exception subtype, including one from an unrelated regression, satisfies the assert and reads as a pass. Narrowed with match=r"(?i)budget", which preserves the original `assert "budget" in str(exc.value).lower()` it replaces. test_unit_test_max_model_budget_limiter.py wrapped an if/else with two different awaited calls inside pytest.raises(). PT012 forbids that (enforced since #37748): the block must hold a single simple statement, so a coroutine built in the wrong branch cannot silently never run. The coroutine is now built outside the block and awaited inside it. Both violations landed in #37736, one day before ruff-tests.toml began enforcing these rules whole-tree, so no delta-vs-base gate caught them. Verified: `ruff check --config ruff-tests.toml tests` is clean, both tests pass, and each still fails under an injected regression.
4 participants
TLDR
Problem this solves:
/user/newacceptsmodel_max_budgetand writes an empty dictHow it solves it:
/user/newpersists the budget it already echoed backBudgetConfigspelling and keeps the fields it does not modelUser Flow
Before: an admin caps Opus spend per model and the cap silently does nothing, while the dashboard shows the budget sitting at zero usage forever
"model_max_budget": {"claude-opus-4-8": {"budget_limit": 0.0001, "time_period": "18h"}}and get a key back carrying that budgetbedrock/anthropic.claude-opus-4-8with that keyspend: 0.00034next tomodel_max_budget_usage.claude-opus-4-8.current_spend: 0.0anthropic/claude-opus-4-8, and the second request now returns 429budget_exceeded, but/key/infostill reportscurrent_spend: 0.0, so the number they would put on a dashboard disagrees with the number that is rejecting their trafficmodel_max_budgetand a1moperiod, and the response echoes it back to themmodel_max_budget: {}, with no usage field at allAfter: the same caps hold, and the usage the API reports is the usage the proxy is enforcing
"model_max_budget": {"claude-opus-4-8": {"budget_limit": 0.0001, "time_period": "18h"}}bedrock/anthropic.claude-opus-4-8with that keybudget_exceedednamingmodel=bedrock/anthropic.claude-opus-4-8model_max_budget_usage.claude-opus-4-8.current_spend: 0.0002againstbudget_limit: 0.0001, which is exactly what the refusal is based onanthropic/claude-opus-4-8behaves identically, and/key/inforeports the same non-zero usage rather than zeromodel_max_budgetand a1moperiodmodel_max_budget_usagereporting current-window spend for that modelLiteLLM User: <user_id>, exceeded budget for model=claude-opus-4-8, and the cap holds across every key that user ownsmodel_max_budgetand is refused at 429 once the cap is blownRelevant issues
Linear ticket
Resolves LIT-5894
Pre-Submission checklist
uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Screenshots / Proof of Fix
Shared setup, identical on both sides. Postgres 16 in Docker, a proxy on port 4894 loaded from the worktree, real Anthropic traffic on a real key.
bedrock/anthropic.claude-opus-4-8is amodel_listalias pointing atanthropic/claude-opus-4-8, so the model NAME the budget has to match is the Bedrock one while the credential is one I hold.Every leg re-probes
/health/readinessafter its last request and printsINVALID: proxy died mid-captureif the process went away, so a partially-captured leg cannot be mistaken for a passing one. Both legs below ended healthy.Before (fc3b160)
A. Key budget keyed
claude-opus-4-8, traffic onbedrock/anthropic.claude-opus-4-8curl -X POST $P/key/generate -H "Authorization: Bearer sk-lit5894" -d '{"user_id": "...", "duration": "18h", "model_max_budget": {"claude-opus-4-8": {"budget_limit": 0.0001, "time_period": "18h"}}}'curl -X POST $P/v1/chat/completions -H "Authorization: Bearer $KEY" -d '{"model": "bedrock/anthropic.claude-opus-4-8", "messages": [{"role":"user","content":"Say the single word: ping"}], "max_tokens": 16}'curl -X GET "$P/key/info?key=$KEY" -H "Authorization: Bearer sk-lit5894"{ "spend": 0.00034, "model_max_budget": {"claude-opus-4-8": {"time_period": "18h", "budget_limit": 0.0001}}, "model_max_budget_usage": { "claude-opus-4-8": {"current_spend": 0.0, "budget_limit": 0.0001, "time_period": "18h"} } }Three requests served at 3.4x the cap, and the usage counter never moved.
B. Same key budget, traffic on
anthropic/claude-opus-4-8/key/generatecall, same budget/v1/chat/completionscall with"model": "anthropic/claude-opus-4-8"curl -X GET "$P/key/info?key=$KEY" -H "Authorization: Bearer sk-lit5894"{ "spend": 0.00017, "model_max_budget_usage": { "claude-opus-4-8": {"current_spend": 0.0, "budget_limit": 0.0001, "time_period": "18h"} } }This is the sharp one: the key is actively being refused at 429 and the endpoint an operator would build a dashboard on reports zero.
C. User-level budget, monthly window, key carries none
curl -X POST $P/user/new -H "Authorization: Bearer sk-lit5894" -d '{"user_id": "...", "model_max_budget": {"claude-opus-4-8": {"budget_limit": 0.0001, "time_period": "1mo"}}}'{"user_id": "lit5894-C-before-...", "model_max_budget": {"claude-opus-4-8": {"budget_limit": 0.0001, "time_period": "1mo"}}}curl -X GET "$P/user/info?user_id=..." -H "Authorization: Bearer sk-lit5894"{"model_max_budget": {}}The create call echoed the budget back; nothing was stored.
curl -X POST $P/key/generate -d '{"user_id": "...", "duration": "18h"}', then three times the/v1/chat/completionscall with"model": "claude-opus-4-8"curl -X GET "$P/user/info?user_id=..." -H "Authorization: Bearer sk-lit5894"{"spend": 0.00051, "model_max_budget": {}, "model_max_budget_usage": null}After (b05e217)
Legs A to D were captured on 559d0ff. The commits since add the passthrough fix that leg E covers and the dashboard license gate, neither of which touches the
/v1/chat/completionspath these four legs drive.Legs E and F were captured on b05e217, which is also the build leg G drives as its unfixed side. The head is 81f64a1, which adds only the pre-upgrade counter carry leg G covers. Every leg above ran against a Redis created after that build, so no pre-upgrade counter exists for the carry to find and their results stand unchanged.
A. Key budget keyed
claude-opus-4-8, traffic onbedrock/anthropic.claude-opus-4-8/key/generatecall as before/v1/chat/completionscalls forbedrock/anthropic.claude-opus-4-8/key/infocall{ "spend": 0.00017, "model_max_budget": {"claude-opus-4-8": {"time_period": "18h", "budget_limit": 0.0001}}, "model_max_budget_usage": { "claude-opus-4-8": {"current_spend": 0.0002, "budget_limit": 0.0001, "time_period": "18h"} } }B. Same key budget, traffic on
anthropic/claude-opus-4-8/key/generatecall as before/v1/chat/completionscalls foranthropic/claude-opus-4-8/key/infocall{ "spend": 0.00017, "model_max_budget_usage": { "claude-opus-4-8": {"current_spend": 0.0002, "budget_limit": 0.0001, "time_period": "18h"} } }The reported usage is now the number the refusal is computed from.
C. User-level budget, monthly window, key carries none
/user/newcall as before{"user_id": "lit5894-C-after-...", "model_max_budget": {"claude-opus-4-8": {"budget_limit": 0.0001, "time_period": "1mo"}}}/user/infocall{"model_max_budget": {"claude-opus-4-8": {"time_period": "1mo", "budget_limit": 0.0001}}}/key/generatefor that user with no per-model budget, then three/v1/chat/completionscalls forclaude-opus-4-8/user/infocall{ "spend": 0.00017, "model_max_budget": {"claude-opus-4-8": {"time_period": "1mo", "budget_limit": 0.0001}}, "model_max_budget_usage": { "claude-opus-4-8": {"current_spend": 0.0002, "budget_limit": 0.0001, "time_period": "1mo"} } }D. The dashboard control, driven end to end
Before: the admin UI has no per-model budget control at all. The only way to set
model_max_budgetis curl, and the Budget Fallbacks tooltip tells operators to "Configure per-model budgets in Advanced Settings", which does not exist.After, driven through the real dashboard served same-origin from the proxy, logged in as admin:
claude-opus-4-8, type0.0001, leave the window on Monthly{"model_max_budget_usage": {"claude-opus-4-8": {"current_spend": 0.0002, "budget_limit": 0.0001, "time_period": "30d"}}}E. Native passthrough,
/anthropic/v1/messagesSame key shape, same real Anthropic traffic, but routed through the native passthrough handler instead of
/v1/chat/completions. This leg covers the BUILT-IN provider routes only. A user-defined pass-through is deliberately excluded from both tracking and enforcement, becauseget_model_from_requestresolves no model there: the body is forwarded verbatim, so itsmodelnames an upstream model rather than a LiteLLM-managed one. That handler builds its logging metadata fromStandardLoggingUserAPIKeyMetadata, which carries no budget field, and never callsadd_litellm_data_to_request, so the post-call increment found nothing to increment.Before (unfixed passthrough):
curl -X POST $P/key/generate -H "Authorization: Bearer sk-lit5894" -d '{"key_alias": "lit5894-passthrough-before", "models": ["anthropic/*"], "model_max_budget": {"claude-opus-4-8": {"budget_limit": 10, "time_period": "30d"}}}'curl -X POST $P/anthropic/v1/messages -H "Authorization: Bearer $KEY" -H 'anthropic-version: 2023-06-01' -d '{"model": "claude-opus-4-8", "max_tokens": 16, "messages": [{"role": "user", "content": "Reply with the single word: budget"}]}'{"model": "claude-opus-4-8", "usage": {"input_tokens": 16, "output_tokens": 4}, "content": [{"type": "text", "text": "budget"}]}curl -X GET "$P/key/info?key=$KEY" -H "Authorization: Bearer sk-lit5894"Real money left the account and the counter the budget is computed from never moved.
After (b05e217), same three calls:
Tracking is only half of it, so the same leg run against a cap smaller than one request, to show the counter is the one that refuses:
curl -X POST $P/key/generate -d '{"key_alias": "lit5894-passthrough-enforce", "models": ["anthropic/*"], "model_max_budget": {"claude-opus-4-8": {"budget_limit": 0.0001, "time_period": "30d"}}}', then the same/anthropic/v1/messagescall twiceNegative control, so the 429 above cannot be read as something else doing the work. Identical key, identical
$0.0001cap, identical two requests, run on the unfixed handler:Twice over the cap, both served, counter still zero.
F. A zero-dollar cap, which is the strictest limit an operator can set
budget_limit: 0means nobody may spend anything on this model. The old check skipped any cap that was falsy or<= 0, so the strictest possible setting behaved as no setting at all. The dashboard editor added here can produce that value, which is how it surfaced.Before (zero-cap check skipped):
curl -X POST $P/key/generate -H "Authorization: Bearer sk-lit5894" -d '{"key_alias": "lit5894-zerocap-before", "models": ["anthropic/*"], "model_max_budget": {"claude-opus-4-8": {"budget_limit": 0, "time_period": "30d"}}}'POST $P/v1/chat/completionsforclaude-opus-4-8, the first request on the key, with nothing spent yet{"model": "claude-opus-4-8", "choices": [{"message": {"content": "ping", "role": "assistant"}}], "usage": {"total_tokens": 18}}curl -X GET "$P/key/info?key=$KEY" -H "Authorization: Bearer sk-lit5894"{"claude-opus-4-8": {"current_spend": 0.0002, "budget_limit": 0.0, "time_period": "30d"}}Served, real money spent, and the endpoint then reports the overspend against a cap of
0.0. An operator can watch the limit being exceeded with nothing enforcing it.After (b05e217), same two calls:
{"claude-opus-4-8": {"current_spend": 0.0, "budget_limit": 0.0, "time_period": "30d"}}Refused on the very first request, before any spend, which is what a
$0cap has to mean.G. A budget window that was already open when the proxy upgraded
Moving the counter key from the request model to the configured budget model orphans the counter an operator's traffic is already being charged to, so an upgrade mid-window would hand that key up to one more full allowance, the whole of it when every request used the prefixed spelling and the prefixed share of it otherwise. Enforcement now adds the pre-upgrade counter in for the remainder of its window.
One Postgres, one Redis, one port, three builds in sequence. Counters are Redis-backed here (
general_settings.coordination_redispluslitellm_settings.enable_redis_auth_cache: true), which is what lets them outlive the restart and makes an upgrade observable at all. Two keys are created on the old release and driven identically, so the two builds under test each start from a counter the old release wrote itself rather than from one this capture invented.fc3b160fb5,curl -X POST $P/key/generate -H "Authorization: Bearer sk-lit5894m" -d '{"key_alias": "...", "duration": "18h", "model_max_budget": {"claude-opus-4-8": {"budget_limit": 0.0001, "time_period": "18h"}}}', twice, for keys A and Bcurl -X POST $P/v1/chat/completions -H "Authorization: Bearer $KEY" -d '{"model": "anthropic/claude-opus-4-8", "messages": [{"role":"user","content":"Say the single word: ping"}], "max_tokens": 16}'Both keys are over their cap and being refused, on counters keyed by the model as REQUESTED.
b05e217b12, this PR before the fix below, against the same Redis and the same Postgres. Same request, key AA key that was being refused a minute earlier is served, and the third line is the mechanism: the same window is now being charged to a second counter that started empty.
Review notes
Correcting a claim I published on this PR earlier. Replying to the cache-key migration finding I wrote that "the old code wrote spend to a key that enforcement never read". That is true for a Bedrock id the previous release matched no budget for, and it is false for the provider-prefixed case the finding actually named:
_get_virtual_key_spend_for_modelread the request-model key FIRST and only then fell back to the stripped spelling, so a budget ongpt-4taking traffic foropenai/gpt-4was enforced onvirtual_key_spend:<id>:openai/gpt-4:<duration>. Leg G above is that counter being written and enforced by the previous release, then ignored after the upgrade. The finding was right and it is fixed here rather than argued with.Two bounds worth stating rather than leaving to be discovered. The carry is read-only and stops one budget window after start-up, since a pre-upgrade counter belongs to a window that was already open when this process replaced the one writing it. And
/key/infocannot include it: the reporting path knows the configured budget name and not which request spellings were charged against it, so during that one window reported usage can sit below the number enforcement is refusing on. It converges when the pre-upgrade counter expires.One more bound on the carry, since it is easier to state than to discover. The pre-upgrade counter was written under
model_group or modelwhile enforcement reconstructs it from the request model. Those are the same string on the router path and differ on a direct deployment with no model group, where the carry finds nothing and the request is admitted exactly as it is today.Type
🐛 Bug Fix
🆕 New Feature
Caveats (if any)
/key/infocannot name that counter, so it can lag enforcement for a windowbudget_limit200 becoming 0.5 reproduces nowhere/user/updatedoes not gatemodel_max_budget, unlike/user/newbudget_limit/time_periodspellingFinal Attestation
Note
High Risk
Changes auth-time budget enforcement and post-call spend accounting for keys, users, and end users. A mismatch here can over-serve traffic or refuse valid requests.
Overview
Fixes per-model budgets so enforcement, post-call spend, and
/key/info//user/infoall use the same counter, keyed on the configured budget model rather than the request spelling.User-level
model_max_budgetis now real: auth loads it ontoUserAPIKeyAuth, JWT / virtual-key / custom-auth paths enforce it,/user/newpersists it, and/user/inforeports usage. Compaction summaries and built-in provider passthrough attach the same metadata so those calls cannot skip the cap.Matching and windows: Bedrock ids resolve to family names via the cost map;
$0caps and spend-at-cap now refuse; budget windows are per model. For one window after upgrade, key/end-user enforcement still adds the old request-model Redis counter so a mid-window deploy does not grant a second allowance. User-defined pass-through routes stay untracked, as before.Reviewed by Cursor Bugbot for commit 81f64a1. Bugbot is set up for automated code reviews on this repo. Configure here.