feat(rate_limit): support per-tag rpm limiting on a single key

[yassin-berriai]

Relevant issues

Resolves LIT-3147

Linear ticket

LIT-3147

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Run against a live proxy (real Postgres, real OpenAI gpt-4o-mini calls) on this branch

Create a key with a per-tag RPM limit of 2 on cell-1 and a generous key-level limit of 50

curl -s -X POST http://127.0.0.1:4147/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
  -d '{"models":["gpt-4o-mini"],"rpm_limit":50,"tag_rpm_limit":{"cell-1":2}}'

# /key/info read-back confirms the limit persists in the key metadata (no DB migration):
metadata.tag_rpm_limit= {'cell-1': 2}

Each tag is tracked independently. cell-1 returns 429 once it hits its own limit, while cell-2 (no configured tag limit) keeps flowing, and an untagged request falls back to the key-level limit

[cell-1 #1] tag=cell-1 -> HTTP 200
[cell-1 #2] tag=cell-1 -> HTTP 200
[cell-1 #3] tag=cell-1 -> HTTP 429  Rate limit exceeded for tag_per_key: 89f1043448bca888...
[cell-2 #1] tag=cell-2 -> HTTP 200
[cell-2 #2] tag=cell-2 -> HTTP 200
[cell-2 #3] tag=cell-2 -> HTTP 200
[no-tag]    tag=none   -> HTTP 200

Response headers on the cell-1 429

HTTP/1.1 429 Too Many Requests
retry-after: 60
rate_limit_type: requests

Updating the limits through /key/update (the path the edit UI uses) works too, and the new limit is enforced on the next request

curl -s -X POST http://127.0.0.1:4147/key/update \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
  -d '{"key":"sk-...","tag_rpm_limit":{"cell-1":5,"cell-9":1}}'

# read-back:
tag_rpm_limit= {'cell-1': 5, 'cell-9': 1}

# cell-9 now has a limit of 1:
cell-9 #1 -> HTTP 200
cell-9 #2 -> HTTP 429

UI: open the key create modal (Advanced Settings) or the key edit view, add a "Per-Tag Rate Limits" row with a tag name plus RPM, and save. The key info view then shows the configured Tag RPM Limits

End-to-end UI demo

Recorded against the freshly built dashboard on this branch. The flow creates a key per-tag-demo-key with a per-tag limit of cell-1 = 2 RPM through the create-key form, opens the key's Settings tab (which shows Tag RPM Limits: {"cell-1":2} and the tag_rpm_limit metadata), then drives live requests through the proxy. cell-1 gets two 200s and a 429 on the third; a different tag cell-2 stays at 200 for all three, confirming the counters are independent

Type

🆕 New Feature

Changes

A customer asked whether a single key's rate limit could be divided per request tag so each cell/group it operates is tracked independently instead of sharing one key-wide limit. Imagine ten cells that should each get 1000 RPM: today that means one key at 10000 RPM shared across all cells (a burst in one cell starves the rest) or ten separate keys to manage. This adds a per-tag RPM limit on a single key instead

A new field, tag_rpm_limit, can be set on a key. It maps a request tag to an RPM limit and is stored in the key's metadata JSON, mirroring how model_rpm_limit and mcp_rpm_limit already work, so there is no schema migration. At request time the v3 rate limiter reads the request's tags and, for any tag that has a configured limit, enforces an independent counter keyed by tag_per_key / {api_key}:{tag}. Tags without a configured limit, and untagged requests, are governed only by the existing key-level limit, so behavior is unchanged for keys that do not opt in

On the backend the work is three small pieces: the field on GenerateRequestBase (added to LiteLLM_ManagementEndpoint_MetadataFields so it folds into metadata), a get_key_tag_rpm_limit helper in auth_utils, and a tag_per_key descriptor in parallel_request_limiter_v3 that reuses the existing per-scope RPM machinery, so enforcement needs no bespoke logic

The dashboard gets a "Per-Tag Rate Limits" row editor on key create and edit, and the key info view displays the configured map

Link to Devin session: https://app.devin.ai/sessions/60c6ad134652487687282119acff22a0

Documentation

Documented in BerriAI/litellm-docs#458.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[yassin-berriai]

@greptileai

[codecov]

Codecov Report

❌ Patch coverage is 90.90909% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...y/management_endpoints/key_management_endpoints.py	33.33%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR adds per-tag RPM limiting on a single API key, letting callers scope independent rate-limit counters to request tags (e.g. cell-1, cell-2) rather than sharing one key-wide budget. The implementation mirrors the existing mcp_rpm_limit / model_rpm_limit patterns: the field is stored in key metadata (no schema migration), read by a new get_key_tag_rpm_limit helper, and enforced via a new tag_per_key descriptor in the v3 parallel-request limiter's pre-call path.

• Backend: tag_rpm_limit added to GenerateRequestBase, LiteLLM_ManagementEndpoint_MetadataFields, and generate_key_helper_fn; _add_tag_per_key_rate_limit_descriptor short-circuits cleanly when no tag limits are configured, so untagged keys pay no overhead.
• Dashboard: New TagRateLimitEditor component (with stable row IDs) wired into both the key create modal and key edit view; the info view displays the map with a proper "Unlimited" fallback for empty configs.
• Tests: Four new unit tests cover independent counters, descriptor presence/absence, and untagged-request fallback; a regression test guards the NewUserRequest → generate_key_helper_fn forwarding path.

Confidence Score: 4/5

Safe to merge for the RPM enforcement path; two open issues in the post-call TPM tracking path (flagged in previous review rounds) remain unaddressed and should be resolved before the feature is considered complete.

The pre-call RPM enforcement is correct and well-tested. The open issues from previous rounds — unconditional tag_per_key TPM counter writes on every tagged request regardless of whether any tag TPM limit is configured, and per-tag TPM limits being silently unenforced when LITELLM_TPM_TOKEN_RESERVATION_ENABLED=false — are real defects in the post-call path that could cause unexpected Redis write amplification at scale and leave a documented limit permanently unenforced in non-default configurations.

litellm/proxy/hooks/parallel_request_limiter_v3.py — specifically the _collect_tpm_scope_targets method and the TPM enforcement path when reservation is disabled.

Important Files Changed

Filename	Overview
litellm/proxy/_types.py	Adds tag_rpm_limit: Optional[dict[str, int]] to GenerateRequestBase and appends "tag_rpm_limit" to LiteLLM_ManagementEndpoint_MetadataFields so the field is automatically folded into key metadata on create/update. Clean, minimal change consistent with how mcp_rpm_limit was added.
litellm/proxy/auth/auth_utils.py	Adds get_key_tag_rpm_limit helper that reads tag_rpm_limit from key metadata. Pattern is identical to the existing get_key_mcp_rpm_limit helper.
litellm/proxy/hooks/parallel_request_limiter_v3.py	Adds _add_tag_per_key_rate_limit_descriptor for pre-call RPM enforcement; the pre-call path correctly short-circuits when no tag limits are configured. However, the post-call _collect_tpm_scope_targets emits tag_per_key TPM entries unconditionally for all tagged requests (flagged in previous review round), and per-tag TPM enforcement is silently absent when LITELLM_TPM_TOKEN_RESERVATION_ENABLED=false (also previously flagged).
litellm/proxy/management_endpoints/key_management_endpoints.py	Adds tag_rpm_limit parameter to generate_key_helper_fn and stores it in metadata when provided, mirroring the existing mcp_rpm_limit handling. Docstrings for generate_key_fn and update_key_fn are updated accordingly.
ui/litellm-dashboard/src/components/key_team_helpers/TagRateLimitEditor.tsx	New reusable editor component for per-tag RPM rows. Uses stable id field (not array index) for React list keys, and exposes tagRowsToLimits / tagLimitsToRows converters. Module-level nextRowId counter provides unique IDs across remounts.
ui/litellm-dashboard/src/components/templates/key_edit_view.tsx	Integrates TagRateLimitEditor into the key edit form. Always submits tag_rpm_limit (even {}) so clearing all rows removes the stored limits. State is initialized from keyData.metadata.tag_rpm_limit at mount; the useEffect that re-syncs Ant Design form fields when keyData changes does not re-sync tagRateLimits, but this is benign because setIsEditing(false) unmounts the component after every successful save.
tests/test_litellm/proxy/hooks/test_parallel_request_limiter_v3.py	Four new tests cover independent tag counters, descriptor creation, absence of descriptors without config, and untagged fallback to key-level limit — all meaningful behavioral assertions.

_{Reviews (12): Last reviewed commit: "feat(rate_limit): support per-tag rpm li..." | Re-trigger Greptile}

[yassin-berriai]

@greptileai

[veria-ai]

Medium: Caller-controlled tags bypass tag limits

The tag descriptor is only added for tags supplied in the request body. A client using a key with tag_rpm_limit={"cell-1": 2} can omit metadata.tags or send an unconfigured tag and avoid the per-tag counter entirely, falling back to the broader key limit. If tag limits are intended as an enforcement boundary, derive the tag from trusted key/team configuration or fail closed when a key has tag limits but the request has no authorized matching tag.

[yassin-berriai]

Per-tag limits are designed as opt-in sub-limits layered under the key-level rpm/tpm ceiling, not as a standalone enforcement boundary. A tag with no configured limit, or a request that carries no tag at all, is still governed by the key-level rpm_limit/tpm_limit, which stays the hard ceiling for the key. Dropping or changing the tag cannot lift a caller above the key's overall budget; it only forfeits the finer per-tag bucket and falls back to the broader key limit. That fallback is the behavior the feature is built around (the proof-of-fix shows an untagged request returning 200 under the key limit), and a fail-closed-on-missing-tag rule would reject legitimate untagged traffic the key is entitled to send

If a deployment wants tags to act as a hard boundary, it sets a key-level rpm_limit/tpm_limit as the ceiling and the per-tag limits subdivide it. I added a regression test, test_per_tag_untagged_request_governed_by_key_limit_v3, that pins this: an untagged or unconfigured-tag request is bounded by the key-level limit and is never rejected by a tag counter, so a future fail-closed change fails the test instead of silently breaking the documented behavior

[yassin-berriai]

Confirmed by design, and unchanged now that this PR is narrowed to per-tag RPM only. Per-tag limits are opt-in sub-limits beneath the key-level rpm_limit ceiling, not a standalone enforcement boundary. Omitting or changing a tag only forfeits the finer per-tag bucket and falls back to the key-level limit, which stays the hard ceiling and cannot be exceeded; it cannot lift a caller above the key overall budget. Failing closed on a missing tag would reject legitimate untagged traffic the key is entitled to send (the live proof shows an untagged request returning 200 under the key limit). test_per_tag_untagged_request_governed_by_key_limit_v3 pins this so a future fail-closed change fails the test instead of silently changing the documented behavior

[veria-ai]

Medium: Key owners can clear tag limits through key updates

UpdateKeyRequest inherits these fields, and the update path treats metadata/rate-limit metadata as a non-budget change for key owners and authorized team members. A non-admin who can update their own key can send tag_rpm_limit: {} / tag_tpm_limit: {} (or the same keys inside metadata) to remove an admin-assigned per-tag throttle, then exceed that limit. Gate changes to these fields like other rate-limit/budget controls and validate them against team/org ceilings before writing them to key metadata.

[yassin-berriai]

tag_rpm_limit/tag_tpm_limit are rate-limit fields and follow the same /key/update authorization path as the existing rpm_limit/tpm_limit. In _validate_update_key_data only max_budget, spend, and budget_limits are admin-gated; rpm_limit/tpm_limit and the other non-budget fields are already editable by the key owner or an authorized team member without the admin check. A key owner who can clear tag_rpm_limit can equally clear their own rpm_limit today, so gating the tag maps more tightly than the key-level limit they subdivide would be inconsistent. /key/update also requires key-management route permission, which a plain inference key does not carry, so this is not reachable by the data-plane key that the rate limit applies to

If admin-only rate-limit ceilings are wanted, that belongs in the shared budget/upperbound machinery for rpm_limit/tpm_limit in general rather than special-casing the tag maps in this PR

[veria-ai]

Thanks for the detail. I can't automatically confirm this is safe to dismiss, so I'm leaving the thread open for a maintainer to make the call.

[yassin-berriai]

tag_rpm_limit follows the same /key/update authorization path as the existing rpm_limit. /key/update requires key-management route permission, which a plain data-plane inference key (the one the rate limit actually applies to) does not carry, so this is not reachable by that key. For a caller who can reach /key/update, only max_budget, spend, and budget_limits are admin-gated in _validate_update_key_data; rpm_limit/tpm_limit and the other non-budget fields are already editable by the key owner or an authorized team member. A key owner who can clear tag_rpm_limit can equally clear their own rpm_limit today, so gating the tag map more tightly than the key-level limit it subdivides would be inconsistent. An admin-only ceiling for rate limits belongs in the shared upperbound machinery for rpm_limit/tpm_limit in general rather than a special case for the tag map in this PR

[veria-ai]

Thanks for the detail. I can't automatically confirm this is safe to dismiss, so I'm leaving the thread open for a maintainer to make the call.

[yassin-berriai]

Following up with the reachability and consistency picture, since I think both point the same way.

Reachability: /key/update requires key-management route permission. The key a tag rate limit actually governs is a data-plane inference key, which does not carry that permission, so the credential the throttle applies to cannot reach this endpoint to clear its own limit. Changing the limit requires a separately privileged management credential, not the rate-limited key itself.

Consistency: in _validate_update_key_data only max_budget, spend, and budget_limits are admin-gated. rpm_limit, tpm_limit, and the other non-budget fields are already editable by the key owner or an authorized team member, and tag_rpm_limit is a rate-limit field that behaves identically. A key owner who can clear tag_rpm_limit can equally clear their own rpm_limit today, so this PR does not add a new class of bypass; it adds one more field to an existing, uniformly applied set.

I do think there is a fair product question underneath this: should rate-limit ceilings be admin-only at all. That is a general decision about rpm_limit/tpm_limit, and the consistent place to make it is the shared budget/upperbound machinery applied uniformly to every rate-limit field, rather than special-casing the tag map in this feature PR. If a maintainer wants that boundary, I am happy to open a follow-up that gates all rate-limit fields together so the behavior stays consistent across the board

[veria-ai]

PR overview

This PR adds support for applying per-tag rate limits to a single API key, including new request/type fields and enforcement logic in the parallel request limiter. It appears to let keys carry tag-specific RPM/TPM limits alongside existing broader key-level limits.

Two security issues remain open around enforcement of the new tag-based limits. Requests can avoid the tag-specific counter by omitting or changing caller-supplied tags, and non-admin key owners may be able to clear admin-assigned tag limits through key updates. No issues have been addressed yet, so the current posture still allows practical bypass of the new throttling controls.

Open issues (2)

• Medium: Caller-controlled tags bypass tag limits — litellm/proxy/hooks/parallel_request_limiter_v3.py:1422
• Medium: Key owners can clear tag limits through key updates — litellm/proxy/_types.py:1063

Fixed/addressed: 0 · PR risk: 6/10

[yassin-berriai]

@greptileai

[yassin-berriai]

@greptileai

[yassin-berriai]

@greptileai

[yassin-berriai]

@greptileai

[yassin-berriai]

@greptileai

[yassin-berriai]

@greptileai

[yassin-berriai]

@greptileai

[yassin-berriai]

@greptileai

[yassin-berriai]

CI note: the only red check is lint, specifically the basedpyright reportArgumentType per-rule budget; it is not a defect in the diff. Adding the tag_rpm_limit field to the key/user request models adds one reportArgumentType at each pre-existing site that spreads a loosely-typed dict into a model constructor (for example GenerateKeyResponse(**response)), which is the same cost every existing field such as mcp_rpm_limit and model_rpm_limit already incurs. Across the codebase that is +4

The ceiling is also stale on the base branch: litellm_internal_staging already sits above this rule cap before this change, and a full re-ratchet (make lint-budget-update) shows the rule has drifted well beyond its committed baseline since it was last updated, only a handful of which is this PR. Re-ratcheting belongs in a separate budget PR on litellm_internal_staging rather than absorbing unrelated drift here, so I left the ceiling untouched. Happy to bump just this one rule by the minimal amount if a maintainer would rather see the check green on this PR

[devin-ai-integration]

Documentation PR for this feature: #31855

Includes a docs page with API examples (/key/generate and /key/update with tag_rpm_limit), UI screenshot of the per-tag rate limit editor, and usage examples for sending tagged requests via x-litellm-tags header.

The docs files (docs/proxy/per_tag_rate_limits.md and docs/img/per_tag_rate_limits_create_key.png) are intended for the litellm-docs repo. The sidebars.js entry should go under "Budgets + Rate Limits", after proxy/tag_budgets.

[codspeed-hq]

Merging this PR will improve performance by 11.46%

⚡ 2 improved benchmarks
❌ 1 regressed benchmark
✅ 27 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Benchmark	BASE	HEAD	Efficiency
❌	test_completion_multi_turn	3.1 ms	4.2 ms	-25.3%
⚡	test_completion_simple_message	4.6 ms	3.2 ms	+41.34%
⚡	test_completion_with_tools	4.2 ms	3.2 ms	+31.14%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing litellm_lit-3147-per-tag-rate-limit (730c674) with litellm_internal_staging (d6cbf6e)}