fix(proxy): reserve the larger declared output budget for TPM limits

[yassin-berriai]

TLDR

Problem this solves:

• A request declaring both token-limit fields reserved the wrong one
• max_tokens=1 plus max_completion_tokens=10000 reserved 2 tokens
• The provider then emitted up to 10,000 against that reservation

How it solves it:

• Reserve against the larger declared budget, not the first one read
• Over-reserving is the safe direction; reconciliation refunds the difference

User Flow

Before: a caller holding a key capped at 100 TPM gets a full-size generation admitted by declaring a tiny max_tokens alongside a large max_completion_tokens

1. An admin issues a key with {"tpm_limit": 100} via POST https://litellm-domain/key/generate
2. The caller sends POST https://litellm-domain/v1/chat/completions with {"max_tokens": 1, "max_completion_tokens": 10000}
3. The gateway admits the request, having set aside 2 tokens of the 100 the key is allowed
4. The same caller sending only {"max_completion_tokens": 10000} is refused with HTTP 429, so the cap is enforced only against callers who declare their budget once

After: the same request is refused, and declaring the budget twice buys nothing

1. An admin issues a key with {"tpm_limit": 100} via POST https://litellm-domain/key/generate
2. The caller sends POST https://litellm-domain/v1/chat/completions with {"max_tokens": 1, "max_completion_tokens": 10000}
3. The gateway sets aside 10,001 tokens, sees that it exceeds the 100 the key is allowed, and returns HTTP 429 naming Limit type: tokens
4. The same caller sending only {"max_completion_tokens": 10000} is refused identically, so both spellings now cost the same

Relevant issues

Found while reviewing #36857, which makes azure/gpt-5-chat translate max_tokens into max_completion_tokens and so joins the families where the legacy field is dropped before the provider sees it. The under-reservation predates that change and already applied to azure/gpt-5, azure/o3-mini, openai/gpt-5 and openai/o3-mini, which is why the fix belongs here in the limiter rather than in any one provider's param mapping

Linear ticket

Pre-Submission checklist

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Live proxy on 127.0.0.1:45559 backed by its own Postgres on 127.0.0.1:55490, with one Azure deployment pointed at a local HTTP server on 127.0.0.1:45560 that records the request body. The rate limiter is the real one, the key is a real virtual key carrying tpm_limit: 100, and each leg uses a fresh key so no leg inherits another's window. --detailed_debug is on so the limiter prints the figure it reserved

Every leg runs the same three requests: the conflicting one, then two controls that declare the budget once

Before, at 40a4184 (this branch's base):

$ curl -s http://127.0.0.1:45559/health/liveliness
"I'm alive!"

$ curl -s -X POST http://127.0.0.1:45559/key/generate -H 'Authorization: Bearer sk-lit5549b' \
    -H 'Content-Type: application/json' -d '{"tpm_limit": 100, "models": ["gpt-5-chat"]}'
-> sk-1rMFzjtEFB1jZnkixMuZDg

CASE A, both fields
$ curl -s -w '\nHTTP %{http_code}\n' http://127.0.0.1:45559/v1/chat/completions \
    -H "Authorization: Bearer $KEY_A" -H 'Content-Type: application/json' \
    -d '{"model":"gpt-5-chat","messages":[{"role":"user","content":"hi"}],"max_tokens":1,"max_completion_tokens":10000}'
HTTP 400
TPM reservation estimate: input=1, max_tokens=1 (explicit=True), total=2

CONTROL B, only max_completion_tokens=10000, fresh key
HTTP 429  Rate limit exceeded ... Limit type: tokens. Current limit: 100
TPM reservation estimate: input=1, max_tokens=10000 (explicit=True), total=10001

CONTROL C, only max_tokens=1, fresh key
HTTP 400
TPM reservation estimate: input=1, max_tokens=1 (explicit=True), total=2

After, at 10af552 (this PR):

$ curl -s http://127.0.0.1:45559/health/liveliness
"I'm alive!"

CASE A, both fields
$ curl -s -w '\nHTTP %{http_code}\n' http://127.0.0.1:45559/v1/chat/completions \
    -H "Authorization: Bearer $KEY_A" -H 'Content-Type: application/json' \
    -d '{"model":"gpt-5-chat","messages":[{"role":"user","content":"hi"}],"max_tokens":1,"max_completion_tokens":10000}'
{"error":{"message":"Rate limit exceeded for api_key: 6ea9af16... Limit type: tokens. Current limit: 100, Remaining: 100. ...","type":"throttling_error","code":"429"}}
HTTP 429
TPM reservation estimate: input=1, max_tokens=10000 (explicit=True), total=10001

CONTROL B, only max_completion_tokens=10000, fresh key
HTTP 429  Rate limit exceeded ... Limit type: tokens. Current limit: 100
TPM reservation estimate: input=1, max_tokens=10000 (explicit=True), total=10001

CONTROL C, only max_tokens=1, fresh key
HTTP 400
TPM reservation estimate: input=1, max_tokens=1 (explicit=True), total=2

The reserved figure for the identical request moves from 2 to 10001, and case A moves from admitted to refused while both controls stay exactly where they were. Control B is the load-bearing one: it shows the harness can produce a 429 at this limit before the fix, so case A being admitted was the conflicting pair winning rather than the cap being unreachable. The HTTP 400 on case A before, and on control C in both runs, is the separate Azure defect in #36857, which this branch does not carry; it is what a request looks like when the limiter lets it through to the provider

Type

🐛 Bug Fix

Caveats (if any)

• Over-reservation is deliberate; reconciliation refunds the unused difference
• max is right even when max_tokens is the larger declared value
• Line 2597 already reads both fields, so 607 was the lone offender

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

^{Cursor Bugbot is generating a summary for commit 10af552. Configure here.}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

The PR updates TPM reservation estimates to use the larger successfully parsed output-token budget when both supported fields are declared

• Adds budget coercion for integer, float, and numeric-string values
• Adds regression coverage for conflicting fields, non-integer representations, and pre-call TPM rejection

Confidence Score: 5/5

The PR appears safe to merge

No blocking failure remains

Important Files Changed

Filename	Overview
litellm/proxy/hooks/parallel_request_limiter_v3.py	Coerces supported budget representations and reserves against the largest declared output limit
tests/test_litellm/proxy/hooks/test_parallel_request_limiter_v3.py	Adds focused reservation and TPM enforcement tests while correcting the previously reported broad typing

_{Reviews (3): Last reviewed commit: "fix(proxy): reserve the larger declared ..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 81.81818% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/hooks/parallel_request_limiter_v3.py	81.81%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_lit5549b_tpm_reservation_conflicting_token_limits (3397a39) with litellm_internal_staging (1a183ef)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (6c2dcb8) during the generation of this report, so 1a183ef was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[yassin-berriai]

bugbot run

[yassin-berriai]

@greptileai please review the current head 47e1ab5, the test helper now returns Dict[str, Dict[str, object]] instead of Any

[yassin-berriai]

@greptileai please review the current head 3397a39, it fixes the bugbot Medium on float and numeric-string budgets skipping reservation

[yassin-berriai]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 3397a39. Configure here.}