feat(proxy): add project-level ITPM and OTPM quotas by shivijain2323 · Pull Request #35110 · BerriAI/litellm

shivijain2323 · 2026-07-29T17:36:00Z

TLDR

Problem this solves:

Projects cannot cap input and output tokens separately
Bedrock Mantle-style quotas need distinct ITPM and OTPM buckets
Combined TPM lets output-heavy jobs starve prompt-heavy ones

How it solves it:

Adds model_itpm_limit and model_otpm_limit maps to projects
Reserves estimated input and output tokens before dispatch
Reconciles reservations against real usage after the call
Cached prompt reads stay free of the input bucket

User Flow

Before: a proxy admin cannot give a project separate input and output token budgets, so a single combined TPM cap is the only lever

The admin sends POST http://localhost:4000/project/new with {"project_alias": "search-ranking", "models": ["MODEL"], "model_itpm_limit": {"MODEL": 300}, "model_otpm_limit": {"MODEL": 60}} and the response drops both fields: the created project's metadata carries neither limit
A developer with a key on that project sends POST http://localhost:4000/v1/chat/completions with a prompt far larger than 300 tokens and it returns 200: nothing enforces an input budget
The same developer sends "max_tokens": 200 requests all minute long and every one returns 200: nothing enforces an output budget

After: the same project carries per-model ITPM and OTPM buckets and requests over either one get a 429 naming that bucket

The admin sends the same POST http://localhost:4000/project/new and the response echoes model_itpm_limit and model_otpm_limit in the project's metadata
The developer's oversized prompt now returns 429 with Rate limit exceeded for model_per_project_itpm
A request with "max_tokens": 200 against the 60-token output bucket returns 429 with Rate limit exceeded for model_per_project_otpm
An under-limit request returns 200 with x-ratelimit-model_per_project_itpm-remaining-tokens and x-ratelimit-model_per_project_otpm-remaining-tokens headers showing each bucket drain independently
The same enforcement holds when the developer calls POST /v1/messages or POST /v1/responses with the same key

Relevant issues

Builds on #31952

Linear ticket

Resolves LIT-5646

Pre-Submission checklist

I have added meaningful tests
My PR passes all CI/CD checks (e.g., lint, format, unit tests)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live proxy QA against real Anthropic claude-fable-5 calls. Shared setup, run once per hash with a fresh Postgres DB:

python litellm/proxy/proxy_cli.py --config qa_config.yaml --port $PORT   # model_list: claude-fable-5 -> anthropic/claude-fable-5

curl -X POST $H/team/new -H "$MASTER" -d '{"team_alias": "qa-io-quota-team", "models": ["claude-fable-5"]}'
curl -X POST $H/project/new -H "$MASTER" -d '{"project_alias": "qa-chat-itpm", "team_id": "'$TEAM_ID'", "models": ["claude-fable-5"], "model_itpm_limit": {"claude-fable-5": 300}, "model_otpm_limit": {"claude-fable-5": 100000}}'
curl -X POST $H/key/generate -H "$MASTER" -d '{"project_id": "'$PROJECT_ID'"}'

Each case below uses its own project (fresh per-minute windows) with the tight bucket set to 300 input or 60 output tokens and the other bucket at 100000

Before (`6d32d40`)

Project creation echoes the limits

POST /project/new with both limit maps returns 200 but drops them, the created project's metadata is empty

"metadata": {}

/v1/chat/completions input budget (itpm 300)

Prompt of roughly 700 tokens ("Summarize the history of aviation. " x 80, max_tokens 30) sails through the 300-token input bucket

HTTP/1.1 200 OK

Small prompt also 200, with only the combined key-level headers, nothing per project

HTTP/1.1 200 OK
x-ratelimit-limit-tokens: 4800000

/v1/chat/completions output budget (otpm 60)

"max_tokens": 200 against the 60-token output bucket is not blocked

HTTP/1.1 200 OK

/v1/messages input budget (itpm 300)

The same oversized prompt on the Anthropic surface also passes

HTTP/1.1 200 OK

/v1/responses output budget (otpm 60)

"max_output_tokens": 200 on the Responses surface also passes, the response body bills 200 output tokens

HTTP/1.1 200 OK

After (`c435c25`)

Project creation echoes the limits

The same POST /project/new now persists and echoes both maps

"metadata": {
  "model_itpm_limit": {"claude-fable-5": 300},
  "model_otpm_limit": {"claude-fable-5": 100000}
}

/v1/chat/completions input budget (itpm 300)

The oversized prompt is rejected before dispatch, naming the input bucket

HTTP/1.1 429 Too Many Requests
{"error":{"message":"Rate limit exceeded for model_per_project_itpm: 6fc333ce-...:claude-fable-5. Limit type: tokens. Current limit: 300, Remaining: 300. ...","code":"429"}}

A small prompt passes and both buckets drain independently in the headers

HTTP/1.1 200 OK
x-ratelimit-model_per_project_itpm-remaining-tokens: 287
x-ratelimit-model_per_project_itpm-limit-tokens: 300
x-ratelimit-model_per_project_otpm-remaining-tokens: 99970
x-ratelimit-model_per_project_otpm-limit-tokens: 100000

/v1/chat/completions output budget (otpm 60)

"max_tokens": 200 is rejected, naming the output bucket

HTTP/1.1 429 Too Many Requests
{"error":{"message":"Rate limit exceeded for model_per_project_otpm: f8841929-...:claude-fable-5. Limit type: tokens. Current limit: 60, Remaining: 60. ...","code":"429"}}

"max_tokens": 30 passes with the output bucket showing the reservation

HTTP/1.1 200 OK
x-ratelimit-model_per_project_otpm-remaining-tokens: 30
x-ratelimit-model_per_project_otpm-limit-tokens: 60

/v1/messages input budget (itpm 300)

The oversized prompt is now rejected on the Anthropic surface too

HTTP/1.1 429 Too Many Requests
{"error":{"message":"litellm.RateLimitError: Rate limit exceeded for model_per_project_itpm: 429aeb3e-...:claude-fable-5. ...","code":"429"}}

The small prompt passes (success responses on this endpoint carry no x-ratelimit headers, which matches base behavior for the existing combined limits)

HTTP/1.1 200 OK

/v1/responses output budget (otpm 60)

"max_output_tokens": 200 is rejected before dispatch

HTTP/1.1 429 Too Many Requests
{"error":{"message":"Rate limit exceeded for model_per_project_otpm: d1699a04-...:claude-fable-5. ...","code":"429"}}

"max_output_tokens": 30 passes with per-project headers

HTTP/1.1 200 OK
x-ratelimit-model_per_project_otpm-remaining-tokens: 30
x-ratelimit-model_per_project_otpm-limit-tokens: 60

Type

🆕 New Feature

Caveats (if any)

Limits live in project metadata, no DB migration
Estimates reserve up front and real usage reconciles after the call, except realtime WebSocket response.create frames and batch file rows, which charge their estimates without post-call reconciliation (documented in the frame hook's docstring)
Docs PRs: docs: add LITELLM_DEFAULT_AUDIO_TOKEN_ESTIMATE to env vars reference litellm-docs#638 (env-var row) and docs(proxy): document project-level ITPM and OTPM limits litellm-docs#934 (project-level section on the ITPM/OTPM page)
Review-round hardening in 72960d1: batch rows scale their output reservation by n / best_of, unparseable output caps fall back to the no-cap floor instead of 500ing inside the limiter (a string cap still fails later in the provider transformation, same as base), and disabled-reservation mode no longer double-charges the ITPM/OTPM buckets. 69ea1c6 makes batch rows coerce n / best_of the same way live traffic does (float or numeric-string counts now scale the reservation) by reusing the limiter's candidate-count helper, and 3c34c34 guards that coercion against float overflow so a crafted row like n: 1e309 can no longer crash the limiter (live requests get a clean 400, and batch rows fall back to a conservative estimate instead of skipping the quota charge through the batch hook's fail-open error handler)
3c34c34 passes /live-pr-risk
c435c25 passes /live-pr-risk (merge of litellm_internal_staging; base touches neither rate-limiter hook, QA and overflow probe re-run green at this tip)

Note

^{Cursor Bugbot is generating a summary for commit 72960d1. Configure here.}

codecov · 2026-07-29T17:44:48Z

Codecov Report

❌ Patch coverage is 90.64748% with 13 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/responses/streaming_iterator.py	78.94%	12 Missing ⚠️
litellm/proxy/hooks/batch_rate_limiter.py	98.59%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

codspeed-hq · 2026-07-29T18:13:02Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing shivijain2323:feature/bedrock-mantle-quota-project-itr1 (c435c25) with litellm_internal_staging (852368d)¹}

No successful run was found on litellm_internal_staging (2903d3a) during the generation of this report, so 852368d was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

shivijain2323 · 2026-07-30T03:56:12Z

@greptileai review

greptile-apps · 2026-07-30T03:59:30Z

Greptile Summary

Adds project-level per-model input and output token quotas

Persists ITPM and OTPM limits in project metadata
Reserves and reconciles separate input and output token counters across standard, streaming, WebSocket, and batch request paths
Adds quota headers, request-shape handling, and regression coverage for supported API surfaces

Confidence Score: 5/5

The PR appears safe to merge

No blocking failure remains

Important Files Changed

Filename	Overview
litellm/proxy/hooks/parallel_request_limiter_v3.py	Implements separate project ITPM and OTPM reservations, window-aware reconciliation, request estimation, and quota response headers
litellm/proxy/hooks/batch_rate_limiter.py	Applies per-row model-specific input and output quota estimates to batch files
litellm/responses/streaming_iterator.py	Propagates project quota enforcement through Responses API WebSocket frames
litellm/llms/custom_httpx/llm_http_handler.py	Discovers registered project quota callbacks and supplies them to native and managed WebSocket handlers
litellm/proxy/_types.py	Adds project ITPM and OTPM request fields and management metadata persistence
litellm/proxy/auth/auth_utils.py	Extends project metadata rate-limit lookup typing to the new quota maps
tests/test_litellm/proxy/hooks/test_tpm_concurrent.py	Covers split quota reservation, streaming reconciliation, usage shapes, cache reads, and window transitions

_{Reviews (23): Last reviewed commit: "Merge remote-tracking branch 'origin/lit..." | Re-trigger Greptile}

veria-ai · 2026-07-30T04:02:05Z

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 20 · PR risk: 0/10

shivijain2323 · 2026-07-30T04:25:19Z

@greptileai review

shivijain2323 · 2026-07-30T05:05:24Z

@greptileai review

shivijain2323 · 2026-07-31T04:43:30Z

@greptileai review

shivijain2323 · 2026-07-31T05:00:37Z

@greptileai review

shivijain2323 · 2026-07-31T05:19:58Z

@greptileai review

shivijain2323 · 2026-07-31T06:07:57Z

@greptileai review

shivijain2323 · 2026-07-31T06:23:50Z

@greptileai review

shivijain2323 · 2026-07-31T06:47:10Z

@greptileai review

shivijain2323 · 2026-07-31T07:15:26Z

@greptileai review

shivijain2323 · 2026-07-31T09:08:32Z

@greptileai review

shivijain2323 · 2026-07-31T11:03:55Z

@greptileai review

Embeddings rows were identified by body shape (has `input`, no `messages`/`prompt`), which also matches a `/v1/responses` batch row and reserved zero output tokens for it -- letting a project caller run large Responses generations against a quota-limited model without consuming OTPM. Classify embeddings by the row's own `url` instead, and read `max_output_tokens` as a Responses output cap alongside `max_tokens`/`max_completion_tokens`. Co-authored-by: Cursor <cursoragent@cursor.com>

…uota-project-itr1

Image, file, video, and previous_response_id requests reserved the whole project ITPM limit up front, so any window with existing usage rejected them and one in-flight multimodal request blocked the entire project. Reserve the token_counter estimate instead, like every other request; post-call reconciliation already charges actual usage.

mateo-berri · 2026-08-18T20:58:33Z

@greptileai

mateo-berri · 2026-08-18T20:58:34Z

bugbot run

- scale batch output-token reservations by the row's n / best_of candidate count - parse client-supplied output caps defensively instead of 500ing on unparseable values - exclude project IO descriptors from the first should_rate_limit pass when TPM reservation is disabled so their buckets are not double-charged

mateo-berri · 2026-08-18T21:45:43Z

bugbot run

mateo-berri · 2026-08-18T21:45:45Z

@greptileai

mateo-berri · 2026-08-18T22:04:32Z

bugbot run

mateo-berri · 2026-08-18T22:04:34Z

@greptileai

…t overflow

mateo-berri · 2026-08-18T22:42:07Z

bugbot run

mateo-berri · 2026-08-18T22:42:20Z

@greptileai

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 3c34c34. Configure here.}

…itellm_pr35110_itpm_otpm # Conflicts: # type-discipline-budget.json

mateo-berri · 2026-08-18T23:11:46Z

bugbot run

mateo-berri · 2026-08-18T23:11:47Z

@greptileai

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit c435c25. Configure here.}

mateo-berri · 2026-08-18T23:46:08Z

@greptileai

mateo-berri

LGTM. Thanks for the contribution!

shivijain2323 marked this pull request as ready for review July 30, 2026 03:56

greptile-apps Bot reviewed Jul 30, 2026

View reviewed changes