fix(router): label router-side rate-limit errors as litellm_rate_limit (was silently vendor) - #27708
Closed
mateo-berri wants to merge 29 commits into
Closed
Conversation
…on RateLimitError
LiteLLM previously surfaced rate-limit conditions through several unrelated
error classes (RateLimitError, FastAPI HTTPException(429), BaseLLMException).
This commit adds the data model needed to consolidate them under a single
class:
* RateLimitErrorCategory enum exposing four categorical values
(vendor_rate_limit, vendor_batch_rate_limit, litellm_rate_limit,
litellm_batch_rate_limit) so callers can switch on the rate-limit source.
* New optional fields on RateLimitError:
- category (defaults to vendor_rate_limit, preserving today's behavior for
every existing call site in exception_mapping_utils);
- headers (preserves retry-after / rate_limit_type / reset_at across the
proxy boundary instead of dropping them on the floor);
- detail (mirrors FastAPI HTTPException.detail so the same instance can be
serialized through both paths).
litellm.RateLimitErrorCategory is re-exported at the package root to match
the existing exception-export pattern.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
…ception Adds a single proxy-side error class that subclasses BOTH litellm.exceptions.RateLimitError AND fastapi.HTTPException via cooperative multiple inheritance. Why both bases: * Subclassing RateLimitError lets user code catch every rate-limit source with one 'except RateLimitError' and switch on the new .category field. * Subclassing HTTPException keeps every existing FastAPI plumbing path (the isinstance(e, HTTPException) branches in proxy_server.py route handlers, FastAPI's own dispatcher, and tests asserting pytest.raises(HTTPException)) working without modification, and preserves retry-after / rate_limit_type / reset_at headers on the wire. The class declaration order is (HTTPException, RateLimitError) so the MRO puts HTTPException's no-super-call __init__ ahead of openai's cooperative __init__ chain — preventing openai.APIError.super().__init__(message) from landing in HTTPException.__init__(status_code=message). LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
…ion limiters Replaces three bare HTTPException(status_code=429, ...) call sites with ProxyRateLimitError, which is both a RateLimitError (catchable by category) and an HTTPException (preserves existing FastAPI serialization). Drops the now-unused HTTPException import in the iteration / per-session limiters. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
…t limiters Replaces HTTPException(status_code=429, ...) call sites in the v1 and v3 parallel-request limiters (key/team/user/model/customer rate limits) with ProxyRateLimitError. Updates the raise_rate_limit_error helper's return type annotation accordingly. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
…miters Replaces HTTPException(status_code=429, ...) call sites in the v1 and v3 dynamic rate limiters (project-level TPM/RPM allocation, model-saturation checks, priority-based limits, fail-closed guards) with ProxyRateLimitError. The v3 limiter still imports HTTPException for an unrelated bare 'except HTTPException:' branch. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Replaces HTTPException(status_code=429, ...) in batch_rate_limiter._raise_rate_limit_error with ProxyRateLimitError tagged as RateLimitErrorCategory.LITELLM_BATCH_RATE_LIMIT so users can distinguish batch-level throttling (which counts requests/tokens across an uploaded batch input file before submission) from the generic key/team/user RPM/TPM limiter. The HTTPException import is retained because the same module raises HTTPException for unrelated 403/IO error paths. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Adds a dedicated test module covering the new RateLimitErrorCategory enum, RateLimitError.category default + override behavior, ProxyRateLimitError's dual nature (RateLimitError + HTTPException), and a parametrized regression guard that asserts every proxy hook module imports the unified class. The regression guard catches the failure mode the refactor is designed to prevent: someone re-introducing a bare HTTPException(status_code=429, ...) in one of the hook modules instead of going through ProxyRateLimitError. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Adds an optional 'error_rate_limit_category' field to StandardLoggingPayloadErrorInformation, populated from the unified RateLimitError.category attribute (introduced in the previous commits on this branch). Why: the .category attribute is reachable off the raw exception today via getattr(e, 'category', None), but the structured contract that downstream custom callbacks / loggers / spend log writers consume is the StandardLoggingPayload. Without this field, a user building custom rate-limit metrics on top of callback data has to special-case the raw exception object — which defeats the purpose of the StandardLoggingPayload abstraction. The field is None for non-rate-limit exceptions (so consumers can read it unconditionally without isinstance checks) and is one of the RateLimitErrorCategory string values otherwise. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Five tests covering: vendor default, explicit litellm_rate_limit and litellm_batch_rate_limit values, None for non-rate-limit exceptions, and None when no exception is provided. Pins down the contract that custom callbacks can read 'error_information.error_rate_limit_category' off the StandardLoggingPayload to drive custom rate-limit metrics without ever reaching for the raw exception. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
mypy emits two [misc] errors on the ProxyRateLimitError class line because its two bases declare overlapping attributes with related-but-not-identical annotations: * status_code: int on starlette HTTPException vs. Literal[429] on openai's RateLimitError (every openai status-error subclass narrows it the same way and silences pyright with the same convention). * headers: Mapping[str, str] | None on HTTPException vs. our Optional[ Dict[str, str]] (the proxy hooks always carry a stringified dict). Both narrowings are intentional and enforced at construction time. Add a type: ignore[misc] with an inline explanation rather than relax the annotations on the parent or change the wire-format guarantees. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
…rage Adds six end-to-end tests that drive each refactored hook past its limit and assert the unified ProxyRateLimitError is raised with the correct category and dual-base shape. Complements the import-shape-only parametrized guard above by actually executing the new 'raise ProxyRateLimitError(...)' lines so codecov's patch coverage sees them as hit. Hooks covered (one test each): * parallel_request_limiter v1 — direct call to raise_rate_limit_error() * parallel_request_limiter v3 — direct call to _handle_rate_limit_error with a fabricated OVER_LIMIT response * max_iterations_limiter — full async_pre_call_hook with mocked agent registry, second call exceeds budget=1 * max_budget_limiter — async_pre_call_hook with mocked get_current_spend * dynamic_rate_limiter v1 — async_pre_call_hook with mocked check_available_usage forcing available_tpm == 0 * batch_rate_limiter — direct _raise_rate_limit_error call, asserts category is the batch-specific LITELLM_BATCH_RATE_LIMIT (not the generic LITELLM_RATE_LIMIT) LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Adds five more direct hook-invocation tests so every PR-touched line in the proxy hooks is exercised by tests in tests/test_litellm/, which codecov measures: * parallel_request_limiter v1 — check_key_in_limits inline raise (the second raise site, separate from the raise_rate_limit_error helper covered earlier) * dynamic_rate_limiter v1 — RPM raise branch (TPM branch was already covered) * dynamic_rate_limiter v3 — parametrized over all three raise sites: model_saturation_check, priority_model, and the fail-closed fallback for an unrecognized descriptor_key * max_budget_per_session_limiter — full async_pre_call_hook with a mocked agent registry and over-budget cached spend All 42 tests in test_rate_limit_error_unification.py now pass and together exercise every changed import + raise line across the eight refactored proxy hooks. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
…rate_limit_error as NoReturn The v1 ' raise_rate_limit_error' helper built an unused 'error_message' variable and then assembled the actual ' detail' via an f-string that interpolated 'additional_details' verbatim — producing 'Max parallel request limit reached None' when invoked without arguments (flagged by code review). Fix the helper to: - use the constructed 'error_message' as the detail - annotate the helper as NoReturn since it always raises - drop the redundant 'raise'/'return' at the two call sites Add two regression tests covering both the with- and without- additional_details paths. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
The v1 parallel_request_limiter's raise_rate_limit_error helper has a long-standing bug: it computes a None-guarded 'error_message' string but then ignores it and emits an f-string that interpolates the raw 'additional_details' arg. Callers that pass no argument get 'Max parallel request limit reached None' as the user-facing detail. This commit: * wires error_message into the detail kwarg so the None-guard actually applies and operators see a clean message; * changes the return-type annotation from ProxyRateLimitError to NoReturn (the function always raises) so type-checkers know callers after this invocation are unreachable. Greptile P1 + P2 review feedback on PR #27687. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
A string literal placed after a field declaration in a TypedDict body is not a per-field docstring — it's an orphaned string expression Python discards. Tools like mypy / pyright that inspect TypedDict fields won't surface that text either. Move the documentation for error_rate_limit_category to a real comment so the intent is visible to readers and type-checker tooling without the misleading docstring framing. Greptile P2 review feedback on PR #27687. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
…eaders A vendor 429 response can set arbitrary headers (Set-Cookie, CORS overrides, …). Previously, when RateLimitError was constructed with only a 'response=' (no explicit 'headers=' kwarg), self.headers fell back to a copy of response.headers. If a downstream proxy serializer ever forwarded e.headers to the client, a malicious upstream could inject browser-interpreted headers for the proxy origin. Drop the fallback. Only headers passed explicitly via the headers= kwarg make it onto self.headers (proxy hooks pass retry-after etc. — they control what's surfaced). Vendor response headers stay reachable on e.response.headers for callers that explicitly want them. Today's proxy_server.py route handlers don't actually forward e.headers on the wire (they construct ProxyException without passing headers), so no current behavior changes — this is a defensive narrowing so the fallback can never be turned into a vector when someone wires e.headers through later. Veria-AI security review feedback on PR #27687. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Pins down the three review-pass fixes: * test_parallel_request_limiter_v1_helper_no_additional_details — calls raise_rate_limit_error() with no args and asserts the detail does NOT contain the literal string 'None'. Pre-fix, callers got 'Max parallel request limit reached None'. * test_rate_limit_error_does_not_auto_copy_response_headers — passes a vendor httpx.Response with a Set-Cookie header to RateLimitError WITHOUT an explicit headers= kwarg, asserts self.headers stays None (no leak), then re-checks that an explicit headers= kwarg DOES populate self.headers. Vendor headers remain reachable on e.response.headers for callers that explicitly want them. * The existing v1-helper test now also asserts the additional_details string makes it through to the detail. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
…rrent_requests/budget/max_iterations)
trho's last ask in the LIT-2968 thread: distinguish rate-limit failures by
the dimension that was exceeded, not just by who rate-limited (vendor vs.
litellm). Adds:
- RateLimitType str-enum exposed at `litellm.RateLimitType` with values
requests / tokens / concurrent_requests / budget / max_iterations.
- `rate_limit_type` kwarg on litellm.RateLimitError + ProxyRateLimitError;
None default so existing callers (vendor-429 path in exception_mapping_utils)
remain a no-op.
- StandardLoggingPayloadErrorInformation.error_rate_limit_type so custom
callbacks can split rate-limit failures by cause without parsing free-text
error messages. Mirror to error_rate_limit_category extraction in
get_error_information(); single isinstance(RateLimitError) check covers both.
- map_v3_rate_limit_type() helper to collapse the v3 limiter's internal labels
("requests", "tokens", "max_parallel_requests") onto the public enum so
the v3 limiter and dynamic_rate_limiter_v3 share one mapping. Defensive
None on unknown values rather than silently picking a wrong dimension.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Each refactored proxy hook now populates rate_limit_type with the dimension that actually tripped the limit, so downstream consumers (custom callbacks, prometheus exporters via the StandardLoggingPayload) can split key/team/user rate-limit failures by cause: - parallel_request_limiter (v1): detect dimension from current vs. limit in the post-cache branch (concurrent_requests > tokens > requests, matches the boolean condition order). Base case (current is None, one limit set to 0) picks the most-specific zero. raise_rate_limit_error() helper accepts an explicit rate_limit_type kwarg with CONCURRENT_REQUESTS default (matches every existing internal call site, including the global-limit branch). - parallel_request_limiter (v3): forward status["rate_limit_type"] through map_v3_rate_limit_type() so "max_parallel_requests" → CONCURRENT_REQUESTS for the public field while the raw v3 jargon stays on the HTTP header for wire-format backward compat. - dynamic_rate_limiter (v1): TPM-zero → TOKENS, RPM-zero → REQUESTS. Pass data["model"] through so callbacks see the model that hit the limit (addresses the secondary "provider missing" complaint in the original Slack thread, partially — the model is what dashboards typically split on). - dynamic_rate_limiter (v3): forward status["rate_limit_type"] via map_v3_rate_limit_type() at every raise site (model_saturation_check, priority_model, fail-closed unknown-descriptor guard). Also pass model. - batch_rate_limiter: limit_type is hard-typed "requests"|"tokens" — map directly without going through the helper's None branch. - max_budget_limiter, max_budget_per_session_limiter: BUDGET. - max_iterations_limiter: MAX_ITERATIONS. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
…LoggingPayload propagation 27 new tests across five new test classes: - TestRateLimitType: enum exposed at litellm.RateLimitType, all five values defined, RateLimitError default is None (vendor 429 path makes no claim about which dimension), accepts both string and enum forms with str-coercion guarantee for downstream JSON serializers. - TestProxyRateLimitErrorType: ProxyRateLimitError default is None, accepts string or enum, doesn't break existing callers that pass nothing. - TestMapV3RateLimitType: pins each v3-internal → public-enum mapping (tokens, requests, max_parallel_requests → concurrent_requests, unknown → None) so a future v3 refactor can't silently swap dimensions. - TestStandardLoggingPayloadCarriesType: the new error_rate_limit_type field reaches the structured payload for both ProxyRateLimitError and plain RateLimitError, is None when unspecified, and is None for non-rate-limit exceptions (symmetric with error_rate_limit_category). - TestProxyHooksWireTypeCorrectly: drives the actual raise sites in the v1 parallel_request_limiter helper, the v3 _handle_rate_limit_error (both "tokens" and "max_parallel_requests" paths), and the batch limiter (both tokens and requests paths) — coverage tools see the new rate_limit_type= kwargs as exercised, not just the import shape. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
…ection
Drives the patch coverage on the new orthogonal RateLimitType wiring up
to (or close to) 100% on the touched files.
ProxyRateLimitError._coerce_message — was 22% covered, now 100%:
* nested {error: {message}} dict
* nested {message: {message}} dict (alt key)
* dict without 'error'/'message' keys → JSON dump fallback
* non-JSON-serializable dict value → str() fallback
* non-string non-mapping detail (int) → str() coercion
v1 parallel_request_limiter dimension detection — was 0% covered, now
exercised across 6 parametrized cases:
* check_key_in_limits else-branch: current at concurrent / TPM / RPM cap
→ asserts rate_limit_type is concurrent_requests / tokens / requests.
* check_key_in_limits base case (current is None): max_parallel_requests
/ tpm_limit / rpm_limit set to 0 → asserts the most-specific zero
attribution wins per the helper's order.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
PR #27687 introduced `RateLimitErrorCategory` and defaulted `RateLimitError.__init__`'s `category=` to `VENDOR_RATE_LIMIT`. That default silently mislabels every callsite (notably litellm's own router-side TPM/RPM throttles) that doesn't pass an explicit category as a vendor 429. Add an `UNKNOWN_RATE_LIMIT` enum value and switch the default to it. Explicit category is now required at every callsite for correct labeling; the new default is an honest "unknown" sentinel rather than a vendor assumption, so future omissions surface in dashboards as `unknown_rate_limit` instead of becoming a confidently-wrong vendor label. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Now that the default `RateLimitError` category is the honest `UNKNOWN_RATE_LIMIT` sentinel, every vendor-side raise must pass an explicit `category=VENDOR_RATE_LIMIT` so downstream dashboards keep attributing those 429s to the upstream provider. Updated: - `litellm_core_utils/exception_mapping_utils.py` — all 22 vendor mapping raises (the central choke point that converts upstream provider exceptions to litellm exceptions). - `llms/anthropic/experimental_pass_through/messages/utils.py` — vendor mock raise. - `main.py` — `mock_response="litellm.RateLimitError"` test mock, which simulates an upstream 429 (carries `llm_provider`). No behavior change for vendor flows — these all previously inherited the `VENDOR_RATE_LIMIT` default and now declare it explicitly. Regression test in tests/test_litellm/test_rate_limit_category_router_side.py pins this. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
PR #27687's `category=VENDOR_RATE_LIMIT` default silently mislabeled every router-side TPM/RPM throttle as a vendor 429: those callsites construct `litellm.RateLimitError` without passing `category=` and so inherited the vendor default. Dashboards (Prometheus `rate_limit_category` label, StandardLoggingPayload `error_rate_limit_category` field) attributed the failure to the upstream provider when in fact litellm's own router decided the request couldn't proceed. Pass an explicit `category=LITELLM_RATE_LIMIT` and a `rate_limit_type=` (REQUESTS for RPM checks, TOKENS for TPM checks) at every router-side raise: - `router_strategy/lowest_tpm_rpm_v2.py` — 5 raises (sync + async pre-call checks, both their redis-overrun branches, and the terminal no-deployments-available raise in the common-checks helper). All RPM, type=REQUESTS. - `router_utils/pre_call_checks/model_rate_limit_check.py` — 4 raises split between TPM (TOKENS) and RPM (REQUESTS) for the `enforce_model_rate_limits` pre-call check. These are LiteLLM's own throttles, distinct from the proxy-side hooks (already correctly labeled by #27687) and from upstream-provider 429s (handled by the explicit-category pass on `exception_mapping_utils`). Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Cover both halves of the contract introduced in this PR: 1. The new UNKNOWN_RATE_LIMIT enum value exists, is exported on the litellm module, and is the default for RateLimitError(...) constructions that omit category=. Also update the existing test_rate_limit_error_unification regression that asserted the old vendor_rate_limit default — it now asserts the new unknown_rate_limit default and documents why. 2. The router-side throttles in lowest_tpm_rpm_v2 and pre_call_checks/model_rate_limit_check carry category=LITELLM_RATE_LIMIT and the right rate_limit_type= (REQUESTS for RPM checks, TOKENS for TPM checks). Mocked at the cache-call boundary so the raise fires deterministically. 3. The vendor mock in anthropic.experimental_pass_through.messages keeps VENDOR_RATE_LIMIT after Step 2's explicit-category pass, and 3 representative branches of exception_mapping_utils.exception_type (string-match, anthropic 429, replicate 429) still produce category=VENDOR_RATE_LIMIT end-to-end — proving Step 2 didn't miss any vendor raise. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
mateo-berri
marked this pull request as ready for review
May 12, 2026 03:33
…s raise This raise in async_get_available_deployments fires after _common_checks_available_deployment filters out every healthy deployment, which can be caused by TPM or RPM (or both). Hardcoding REQUESTS here mis-attributes TPM-driven exhaustion on dashboards/callbacks. Leave rate_limit_type unset so the dimension surfaces as the honest 'unknown' default rather than a confidently-wrong label.
Contributor
Greptile SummaryThis PR fixes a silent mislabeling bug introduced by the
Confidence Score: 5/5The change is safe to merge; it is a targeted relabeling with no control-flow alterations and comprehensive test coverage across every affected raise site. Every modified raise site has been audited and updated with an explicit category, all 22 vendor paths retain their VENDOR_RATE_LIMIT label, the 9 router-side paths now carry the correct LITELLM_RATE_LIMIT label, and both the new and updated tests accurately reflect the intended behavior. No logic paths were changed — only the metadata carried by the exceptions. No files require special attention. The two existing test updates in test_rate_limit_error_unification.py are legitimate reflections of the new default, not coverage regressions.
|
| Filename | Overview |
|---|---|
| litellm/exceptions.py | Adds UNKNOWN_RATE_LIMIT enum member and flips RateLimitError default category from VENDOR_RATE_LIMIT to UNKNOWN_RATE_LIMIT; well-documented intent. |
| litellm/litellm_core_utils/exception_mapping_utils.py | Imports RateLimitErrorCategory and explicitly pins all 22 vendor RateLimitError raise sites to VENDOR_RATE_LIMIT; keeps vendor labeling identical to before the default change. |
| litellm/router_strategy/lowest_tpm_rpm_v2.py | Adds LITELLM_RATE_LIMIT + REQUESTS labels to all 5 RPM raise sites; correctly reflects router-enforced request throttles rather than upstream vendor limits. |
| litellm/router_utils/pre_call_checks/model_rate_limit_check.py | Adds LITELLM_RATE_LIMIT + TOKENS/REQUESTS labels to all 4 raise sites (2 TPM, 2 RPM), matching the dimension of the throttle that fired. |
| tests/test_litellm/test_rate_limit_category_router_side.py | New 16-test file covering: UNKNOWN_RATE_LIMIT enum + default, router-side raises (sync/async for both strategy files), vendor regression guards for exception_mapping_utils and Anthropic mock. |
| tests/test_litellm/test_rate_limit_error_unification.py | Two existing tests updated to assert UNKNOWN_RATE_LIMIT instead of VENDOR_RATE_LIMIT as the no-category default; change accurately reflects the new intended behavior, not a weakening of coverage. |
| litellm/llms/anthropic/experimental_pass_through/messages/utils.py | Adds explicit VENDOR_RATE_LIMIT to the Anthropic pass-through mock raise; preserves vendor semantics for that test path. |
| litellm/main.py | Imports RateLimitErrorCategory and pins the _handle_mock_potential_exceptions raise to VENDOR_RATE_LIMIT, matching its role of simulating upstream vendor 429s. |
Reviews (1): Last reviewed commit: "fix(router): drop hardcoded rate_limit_t..." | Re-trigger Greptile
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Test asserts REQUESTS but code intentionally leaves rate_limit_type unset
- Updated the test to assert rate_limit_type is None (matching the code's intentional behavior documented in the comment) and corrected the class docstring to reflect that the no-deployments-available raise leaves rate_limit_type unset.
Preview (79dd636207)
diff --git a/litellm/exceptions.py b/litellm/exceptions.py
--- a/litellm/exceptions.py
+++ b/litellm/exceptions.py
@@ -47,7 +47,15 @@
LITELLM_BATCH_RATE_LIMIT = "litellm_batch_rate_limit"
"""LiteLLM's own batch rate limiter (token/request budget across a batch input file) blocked the request."""
+ UNKNOWN_RATE_LIMIT = "unknown_rate_limit"
+ """Default for callers that did not explicitly classify the rate-limit source.
+ New code SHOULD pass an explicit category; this value exists so silent
+ miscategorization (vs. the previous ``VENDOR_RATE_LIMIT`` default) becomes
+ visible in dashboards rather than a confidently-wrong label.
+ """
+
+
class RateLimitType(str, enum.Enum):
"""
The dimension that was exceeded when a rate-limit error fired.
@@ -407,8 +415,15 @@
litellm_debug_info: Optional[str] = None,
max_retries: Optional[int] = None,
num_retries: Optional[int] = None,
+ # An explicit ``category`` is now required for correct labeling — every
+ # callsite (vendor mappers in ``exception_mapping_utils.py``, proxy-side
+ # hooks, router-side throttles) is expected to pass the value that
+ # matches its source. The default below is an honest "unknown" sentinel,
+ # NOT a vendor assumption: silently inheriting it makes the omission
+ # surface in dashboards (as ``unknown_rate_limit``) instead of a
+ # confidently-wrong vendor label.
category: Union[str, RateLimitErrorCategory] = (
- RateLimitErrorCategory.VENDOR_RATE_LIMIT
+ RateLimitErrorCategory.UNKNOWN_RATE_LIMIT
),
rate_limit_type: Optional[Union[str, RateLimitType]] = None,
headers: Optional[Dict[str, str]] = None,
diff --git a/litellm/litellm_core_utils/exception_mapping_utils.py b/litellm/litellm_core_utils/exception_mapping_utils.py
--- a/litellm/litellm_core_utils/exception_mapping_utils.py
+++ b/litellm/litellm_core_utils/exception_mapping_utils.py
@@ -22,6 +22,7 @@
NotFoundError,
PermissionDeniedError,
RateLimitError,
+ RateLimitErrorCategory,
ServiceUnavailableError,
Timeout,
UnprocessableEntityError,
@@ -403,6 +404,7 @@
model=model,
llm_provider=custom_llm_provider,
response=getattr(original_exception, "response", None),
+ category=RateLimitErrorCategory.VENDOR_RATE_LIMIT,
)
elif ExceptionCheckers.is_error_str_context_window_exceeded(error_str):
exception_mapping_worked = True
@@ -510,6 +512,7 @@
llm_provider=custom_llm_provider,
response=getattr(original_exception, "response", None),
litellm_debug_info=extra_information,
+ category=RateLimitErrorCategory.VENDOR_RATE_LIMIT,
)
elif (
"The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable"
@@ -591,6 +594,7 @@
llm_provider=custom_llm_provider,
response=getattr(original_exception, "response", None),
litellm_debug_info=extra_information,
+ category=RateLimitErrorCategory.VENDOR_RATE_LIMIT,
)
elif original_exception.status_code == 500:
exception_mapping_worked = True
@@ -730,6 +734,7 @@
message=f"AnthropicException - {error_str}",
llm_provider="anthropic",
model=model,
+ category=RateLimitErrorCategory.VENDOR_RATE_LIMIT,
)
elif (
original_exception.status_code == 500
@@ -798,6 +803,7 @@
llm_provider="replicate",
model=model,
response=getattr(original_exception, "response", None),
+ category=RateLimitErrorCategory.VENDOR_RATE_LIMIT,
)
elif hasattr(original_exception, "status_code"):
if original_exception.status_code == 401:
@@ -849,6 +855,7 @@
llm_provider="replicate",
model=model,
response=getattr(original_exception, "response", None),
+ category=RateLimitErrorCategory.VENDOR_RATE_LIMIT,
)
elif original_exception.status_code == 500:
exception_mapping_worked = True
@@ -905,6 +912,7 @@
llm_provider=custom_llm_provider,
model=model,
response=getattr(original_exception, "response", None),
+ category=RateLimitErrorCategory.VENDOR_RATE_LIMIT,
)
elif (
"The server received an invalid response from an upstream server."
@@ -981,6 +989,7 @@
model=model,
llm_provider=custom_llm_provider,
litellm_debug_info=extra_information,
+ category=RateLimitErrorCategory.VENDOR_RATE_LIMIT,
)
elif original_exception.status_code == 503:
exception_mapping_worked = True
@@ -1071,6 +1080,7 @@
model=model,
llm_provider="bedrock",
response=getattr(original_exception, "response", None),
+ category=RateLimitErrorCategory.VENDOR_RATE_LIMIT,
)
elif (
"Connect timeout on endpoint URL" in error_str
@@ -1152,6 +1162,7 @@
llm_provider=custom_llm_provider,
response=getattr(original_exception, "response", None),
litellm_debug_info=extra_information,
+ category=RateLimitErrorCategory.VENDOR_RATE_LIMIT,
)
elif original_exception.status_code == 503:
exception_mapping_worked = True
@@ -1271,6 +1282,7 @@
llm_provider=custom_llm_provider,
response=getattr(original_exception, "response", None),
litellm_debug_info=extra_information,
+ category=RateLimitErrorCategory.VENDOR_RATE_LIMIT,
)
elif original_exception.status_code == 503:
exception_mapping_worked = True
@@ -1407,6 +1419,7 @@
url=" https://cloud.google.com/vertex-ai/",
),
),
+ category=RateLimitErrorCategory.VENDOR_RATE_LIMIT,
)
elif (
"500 Internal Server Error" in error_str
@@ -1485,6 +1498,7 @@
url=" https://cloud.google.com/vertex-ai/",
),
),
+ category=RateLimitErrorCategory.VENDOR_RATE_LIMIT,
)
if original_exception.status_code == 500:
exception_mapping_worked = True
@@ -1604,6 +1618,7 @@
llm_provider="cohere",
model=model,
response=getattr(original_exception, "response", None),
+ category=RateLimitErrorCategory.VENDOR_RATE_LIMIT,
)
elif "invalid type:" in error_str:
exception_mapping_worked = True
@@ -1656,6 +1671,7 @@
llm_provider="huggingface",
model=model,
response=getattr(original_exception, "response", None),
+ category=RateLimitErrorCategory.VENDOR_RATE_LIMIT,
)
if hasattr(original_exception, "status_code"):
if original_exception.status_code == 401:
@@ -1688,6 +1704,7 @@
llm_provider="huggingface",
model=model,
response=getattr(original_exception, "response", None),
+ category=RateLimitErrorCategory.VENDOR_RATE_LIMIT,
)
elif original_exception.status_code == 503:
exception_mapping_worked = True
@@ -1755,6 +1772,7 @@
llm_provider="ai21",
model=model,
response=getattr(original_exception, "response", None),
+ category=RateLimitErrorCategory.VENDOR_RATE_LIMIT,
)
else:
exception_mapping_worked = True
@@ -1839,6 +1857,7 @@
llm_provider="nlp_cloud",
model=model,
response=getattr(original_exception, "response", None),
+ category=RateLimitErrorCategory.VENDOR_RATE_LIMIT,
)
elif (
original_exception.status_code == 500
@@ -1964,6 +1983,7 @@
llm_provider="together_ai",
model=model,
response=getattr(original_exception, "response", None),
+ category=RateLimitErrorCategory.VENDOR_RATE_LIMIT,
)
elif original_exception.status_code == 524:
exception_mapping_worked = True
@@ -2027,6 +2047,7 @@
llm_provider="aleph_alpha",
model=model,
response=getattr(original_exception, "response", None),
+ category=RateLimitErrorCategory.VENDOR_RATE_LIMIT,
)
elif original_exception.status_code == 500:
exception_mapping_worked = True
@@ -2270,6 +2291,7 @@
llm_provider="azure",
litellm_debug_info=extra_information,
response=getattr(original_exception, "response", None),
+ category=RateLimitErrorCategory.VENDOR_RATE_LIMIT,
)
elif original_exception.status_code == 502:
exception_mapping_worked = True
@@ -2374,6 +2396,7 @@
llm_provider=custom_llm_provider,
response=getattr(original_exception, "response", None),
litellm_debug_info=extra_information,
+ category=RateLimitErrorCategory.VENDOR_RATE_LIMIT,
)
elif original_exception.status_code == 503:
exception_mapping_worked = True
diff --git a/litellm/llms/anthropic/experimental_pass_through/messages/utils.py b/litellm/llms/anthropic/experimental_pass_through/messages/utils.py
--- a/litellm/llms/anthropic/experimental_pass_through/messages/utils.py
+++ b/litellm/llms/anthropic/experimental_pass_through/messages/utils.py
@@ -41,6 +41,7 @@
ContextWindowExceededError,
InternalServerError,
RateLimitError,
+ RateLimitErrorCategory,
)
if mock_response == "litellm.InternalServerError":
@@ -60,6 +61,7 @@
message="this is a mock rate limit error",
llm_provider="anthropic",
model=model,
+ category=RateLimitErrorCategory.VENDOR_RATE_LIMIT,
)
return AnthropicMessagesResponse(
**{
diff --git a/litellm/main.py b/litellm/main.py
--- a/litellm/main.py
+++ b/litellm/main.py
@@ -78,7 +78,7 @@
DEFAULT_MOCK_RESPONSE_COMPLETION_TOKEN_COUNT,
DEFAULT_MOCK_RESPONSE_PROMPT_TOKEN_COUNT,
)
-from litellm.exceptions import LiteLLMUnknownProvider
+from litellm.exceptions import LiteLLMUnknownProvider, RateLimitErrorCategory
from litellm.integrations.custom_logger import CustomLogger
from litellm.litellm_core_utils.asyncify import run_async_function
from litellm.litellm_core_utils.audio_utils.utils import (
@@ -703,6 +703,7 @@
mock_response, "llm_provider", custom_llm_provider or "openai"
), # type: ignore
model=model,
+ category=RateLimitErrorCategory.VENDOR_RATE_LIMIT,
)
elif (
isinstance(mock_response, str)
diff --git a/litellm/router_strategy/lowest_tpm_rpm_v2.py b/litellm/router_strategy/lowest_tpm_rpm_v2.py
--- a/litellm/router_strategy/lowest_tpm_rpm_v2.py
+++ b/litellm/router_strategy/lowest_tpm_rpm_v2.py
@@ -7,6 +7,7 @@
import litellm
from litellm import token_counter
+from litellm.exceptions import RateLimitErrorCategory, RateLimitType
from litellm._logging import verbose_logger, verbose_router_logger
from litellm.caching.caching import DualCache
from litellm.integrations.custom_logger import CustomLogger
@@ -108,6 +109,8 @@
),
request=httpx.Request(method="tpm_rpm_limits", url="https://github.com/BerriAI/litellm"), # type: ignore
),
+ category=RateLimitErrorCategory.LITELLM_RATE_LIMIT,
+ rate_limit_type=RateLimitType.REQUESTS,
)
else:
# if local result below limit, check redis ## prevent unnecessary redis checks
@@ -131,6 +134,8 @@
),
request=httpx.Request(method="tpm_rpm_limits", url="https://github.com/BerriAI/litellm"), # type: ignore
),
+ category=RateLimitErrorCategory.LITELLM_RATE_LIMIT,
+ rate_limit_type=RateLimitType.REQUESTS,
)
return deployment
except Exception as e:
@@ -193,6 +198,8 @@
request=httpx.Request(method="tpm_rpm_limits", url="https://github.com/BerriAI/litellm"), # type: ignore
),
num_retries=deployment.get("num_retries"),
+ category=RateLimitErrorCategory.LITELLM_RATE_LIMIT,
+ rate_limit_type=RateLimitType.REQUESTS,
)
else:
# if local result below limit, check redis ## prevent unnecessary redis checks
@@ -217,6 +224,8 @@
request=httpx.Request(method="tpm_rpm_limits", url="https://github.com/BerriAI/litellm"), # type: ignore
),
num_retries=deployment.get("num_retries"),
+ category=RateLimitErrorCategory.LITELLM_RATE_LIMIT,
+ rate_limit_type=RateLimitType.REQUESTS,
)
return deployment
except Exception as e:
@@ -550,6 +559,11 @@
"current_rpm": current_rpm,
"rpm_limit": _deployment_rpm,
}
+ # NOTE: ``rate_limit_type`` is intentionally left unset here. This raise
+ # fires after ``_common_checks_available_deployment`` filters out every
+ # healthy deployment, which can happen for TPM, RPM, or both — we don't
+ # know which dimension caused the filtering, so we leave the dimension as
+ # the honest "unknown" default rather than mis-attributing to REQUESTS.
raise litellm.RateLimitError(
message=f"{RouterErrors.no_deployments_available.value}. Passed model={model_group}. Deployments={deployment_dict}",
llm_provider="",
@@ -560,6 +574,7 @@
headers={"retry-after": str(60)}, # type: ignore
request=httpx.Request(method="tpm_rpm_limits", url="https://github.com/BerriAI/litellm"), # type: ignore
),
+ category=RateLimitErrorCategory.LITELLM_RATE_LIMIT,
)
def get_available_deployments(
diff --git a/litellm/router_utils/pre_call_checks/model_rate_limit_check.py b/litellm/router_utils/pre_call_checks/model_rate_limit_check.py
--- a/litellm/router_utils/pre_call_checks/model_rate_limit_check.py
+++ b/litellm/router_utils/pre_call_checks/model_rate_limit_check.py
@@ -15,6 +15,7 @@
import litellm
from litellm._logging import verbose_router_logger
from litellm.caching.dual_cache import DualCache
+from litellm.exceptions import RateLimitErrorCategory, RateLimitType
from litellm.integrations.custom_logger import CustomLogger
from litellm.types.router import RouterErrors
from litellm.types.utils import StandardLoggingPayload
@@ -127,6 +128,8 @@
url="https://github.com/BerriAI/litellm",
),
),
+ category=RateLimitErrorCategory.LITELLM_RATE_LIMIT,
+ rate_limit_type=RateLimitType.TOKENS,
)
# Check RPM limit (atomic increment-first to avoid race conditions)
@@ -148,6 +151,8 @@
url="https://github.com/BerriAI/litellm",
),
),
+ category=RateLimitErrorCategory.LITELLM_RATE_LIMIT,
+ rate_limit_type=RateLimitType.REQUESTS,
)
return deployment
@@ -205,6 +210,8 @@
),
),
num_retries=0, # Don't retry - return 429 immediately
+ category=RateLimitErrorCategory.LITELLM_RATE_LIMIT,
+ rate_limit_type=RateLimitType.TOKENS,
)
# Check RPM limit (atomic increment-first to avoid race conditions)
@@ -230,6 +237,8 @@
),
),
num_retries=0, # Don't retry - return 429 immediately
+ category=RateLimitErrorCategory.LITELLM_RATE_LIMIT,
+ rate_limit_type=RateLimitType.REQUESTS,
)
return deployment
diff --git a/tests/test_litellm/test_rate_limit_category_router_side.py b/tests/test_litellm/test_rate_limit_category_router_side.py
new file mode 100644
--- /dev/null
+++ b/tests/test_litellm/test_rate_limit_category_router_side.py
@@ -1,0 +1,367 @@
+"""
+Tests for the router-side rate-limit category fix.
+
+Background
+----------
+PR #27687 added :class:`RateLimitErrorCategory` and the ``category`` /
+``rate_limit_type`` kwargs on :class:`litellm.RateLimitError`, with
+``category=VENDOR_RATE_LIMIT`` as the default. That default silently
+mislabeled every router-side TPM/RPM throttle (in
+``router_strategy.lowest_tpm_rpm_v2``,
+``router_utils.pre_call_checks.model_rate_limit_check``, etc.) as a vendor
+error: those callsites construct ``RateLimitError`` without passing
+``category=`` and so inherit the vendor default. The fix:
+
+1. The :class:`RateLimitErrorCategory` enum now exposes
+ ``UNKNOWN_RATE_LIMIT`` (an "unknown" sentinel) and
+ :meth:`RateLimitError.__init__` defaults to it. Future omissions surface in
+ dashboards as ``unknown_rate_limit`` instead of a confidently-wrong vendor
+ label.
+2. Every router-side raise was updated to pass an explicit
+ ``category=LITELLM_RATE_LIMIT`` (and ``rate_limit_type=`` when the
+ dimension is determinable from the throttle that fired).
+3. Every vendor-mapping raise in
+ :mod:`litellm.litellm_core_utils.exception_mapping_utils` (and the few
+ vendor mocks under ``llms/``) was updated to keep passing
+ ``category=VENDOR_RATE_LIMIT`` explicitly so the new "unknown" default
+ doesn't change vendor-side behavior.
+
+These tests pin both halves of that contract.
+"""
+
+from unittest.mock import patch
+
+import httpx
+import pytest
+
+import litellm
+from litellm.exceptions import (
+ RateLimitError,
+ RateLimitErrorCategory,
+ RateLimitType,
+)
+
+
+# ---------------------------------------------------------------------------
+# Step 1: enum + default behavior
+# ---------------------------------------------------------------------------
+
+
+class TestUnknownRateLimitCategory:
+ def test_should_expose_unknown_rate_limit_value(self):
+ # Sanity: the new enum value exists and round-trips through the str
+ # protocol so dashboards / log aggregators can compare against the
+ # plain string without importing the enum.
+ assert RateLimitErrorCategory.UNKNOWN_RATE_LIMIT == "unknown_rate_limit"
+ assert "unknown_rate_limit" == RateLimitErrorCategory.UNKNOWN_RATE_LIMIT
+
+ def test_should_export_unknown_value_on_litellm_module(self):
+ assert (
+ litellm.RateLimitErrorCategory.UNKNOWN_RATE_LIMIT
+ == RateLimitErrorCategory.UNKNOWN_RATE_LIMIT
+ )
+
+ def test_should_default_category_to_unknown_when_unspecified(self):
+ # Constructing RateLimitError without an explicit category yields the
+ # honest "unknown" sentinel, NOT a vendor assumption. This is the
+ # whole point of the fix: silent omissions become visible in
+ # dashboards as ``unknown_rate_limit``.
+ e = RateLimitError(message="oops", llm_provider="openai", model="gpt-4")
+ assert e.category == RateLimitErrorCategory.UNKNOWN_RATE_LIMIT
+ assert e.category == "unknown_rate_limit"
+
+ def test_should_still_accept_explicit_vendor_category(self):
+ # Vendor callsites still set the right value when they pass one.
+ e = RateLimitError(
+ message="oops",
+ llm_provider="openai",
+ model="gpt-4",
+ category=RateLimitErrorCategory.VENDOR_RATE_LIMIT,
+ )
+ assert e.category == RateLimitErrorCategory.VENDOR_RATE_LIMIT
+
+ def test_should_still_accept_explicit_litellm_category(self):
+ e = RateLimitError(
+ message="oops",
+ llm_provider="openai",
+ model="gpt-4",
+ category=RateLimitErrorCategory.LITELLM_RATE_LIMIT,
+ rate_limit_type=RateLimitType.REQUESTS,
+ )
+ assert e.category == RateLimitErrorCategory.LITELLM_RATE_LIMIT
+ assert e.rate_limit_type == RateLimitType.REQUESTS
+
+
+# ---------------------------------------------------------------------------
+# Step 3: regression tests for router-side raises
+# ---------------------------------------------------------------------------
+
+
+class TestLowestTpmRpmV2RouterSideCategory:
+ """
+ The five raise sites in ``router_strategy.lowest_tpm_rpm_v2`` (sync
+ pre-check, async pre-check, both their redis-overrun branches, and the
+ no-deployments-available raise) must all carry
+ ``category=LITELLM_RATE_LIMIT``. The four pre-check sites also carry
+ ``rate_limit_type=REQUESTS``; the terminal no-deployments-available raise
+ intentionally leaves ``rate_limit_type`` unset because the filtering could
+ have been driven by TPM, RPM, or both. They were silently labeled as
+ ``vendor_rate_limit`` before this fix.
+ """
+
+ def _build_handler(self):
+ from litellm.caching.caching import DualCache
+ from litellm.router_strategy.lowest_tpm_rpm_v2 import (
+ LowestTPMLoggingHandler_v2,
+ )
+
+ cache = DualCache()
+ return LowestTPMLoggingHandler_v2(router_cache=cache)
+
+ def test_should_label_local_rpm_overrun_as_litellm_requests(self):
+ handler = self._build_handler()
+ deployment = {
+ "litellm_params": {"model": "gpt-4", "rpm": 1},
+ "model_info": {"id": "abc-123"},
+ "model_name": "gpt-4",
+ "rpm": 1,
+ }
+ # Force the local cache lookup to come back already at the limit so
+ # the sync branch raises immediately.
+ with patch.object(handler.router_cache, "get_cache", return_value=5):
+ with pytest.raises(litellm.RateLimitError) as exc_info:
+ handler.pre_call_check(deployment)
+
+ assert exc_info.value.category == RateLimitErrorCategory.LITELLM_RATE_LIMIT
+ assert exc_info.value.rate_limit_type == RateLimitType.REQUESTS
+
+ @pytest.mark.asyncio
+ async def test_should_label_async_local_rpm_overrun_as_litellm_requests(self):
+ handler = self._build_handler()
+ deployment = {
+ "litellm_params": {"model": "gpt-4", "rpm": 1},
+ "model_info": {"id": "abc-123"},
+ "model_name": "gpt-4",
+ "rpm": 1,
+ }
+
+ async def _stub_get(*args, **kwargs):
+ return 5
+
+ with patch.object(
+ handler.router_cache, "async_get_cache", side_effect=_stub_get
+ ):
+ with pytest.raises(litellm.RateLimitError) as exc_info:
+ await handler.async_pre_call_check(deployment, parent_otel_span=None)
+
+ assert exc_info.value.category == RateLimitErrorCategory.LITELLM_RATE_LIMIT
+ assert exc_info.value.rate_limit_type == RateLimitType.REQUESTS
+
+ @pytest.mark.asyncio
+ async def test_should_label_no_deployments_available_as_litellm_rate_limit(self):
+ # The terminal "no deployments available" raise — fired from
+ # ``async_get_available_deployments`` after every healthy deployment
+ # was filtered out — must carry the litellm category. The
+ # ``rate_limit_type`` dimension is intentionally left unset because
+ # the filtering could have been driven by TPM, RPM, or both.
+ handler = self._build_handler()
+ with pytest.raises(litellm.RateLimitError) as exc_info:
+ await handler.async_get_available_deployments(
+ model_group="gpt-4",
+ healthy_deployments=[],
+ )
+
+ assert exc_info.value.category == RateLimitErrorCategory.LITELLM_RATE_LIMIT
+ assert exc_info.value.rate_limit_type is None
+
+
+class TestModelRateLimitCheckRouterSideCategory:
+ """
+ The four raise sites in
+ ``router_utils.pre_call_checks.model_rate_limit_check`` split between
+ TPM (``rate_limit_type=TOKENS``) and RPM
+ (``rate_limit_type=REQUESTS``); both carry
+ ``category=LITELLM_RATE_LIMIT``.
+ """
+
+ def _build_check(self):
+ from litellm.caching.dual_cache import DualCache
+ from litellm.router_utils.pre_call_checks.model_rate_limit_check import (
+ ModelRateLimitingCheck,
+ )
+
+ return ModelRateLimitingCheck(dual_cache=DualCache())
+
+ def test_should_label_sync_tpm_overrun_as_litellm_tokens(self):
+ check = self._build_check()
+ deployment = {
+ "litellm_params": {"model": "gpt-4", "tpm": 10},
+ "model_info": {"id": "abc-123", "tpm": 10},
+ "model_name": "gpt-4",
+ }
+ with patch.object(check.dual_cache, "get_cache", return_value=999):
+ with pytest.raises(litellm.RateLimitError) as exc_info:
+ check.pre_call_check(deployment)
+
+ assert exc_info.value.category == RateLimitErrorCategory.LITELLM_RATE_LIMIT
+ assert exc_info.value.rate_limit_type == RateLimitType.TOKENS
+
+ def test_should_label_sync_rpm_overrun_as_litellm_requests(self):
+ check = self._build_check()
+ deployment = {
+ "litellm_params": {"model": "gpt-4", "rpm": 1},
+ "model_info": {"id": "abc-123", "rpm": 1},
+ "model_name": "gpt-4",
+ }
+ # No TPM limit, so the TPM branch is skipped; RPM increment returns a
+ # value above the limit, triggering the RPM raise.
+ with patch.object(check.dual_cache, "increment_cache", return_value=42):
+ with pytest.raises(litellm.RateLimitError) as exc_info:
+ check.pre_call_check(deployment)
+
+ assert exc_info.value.category == RateLimitErrorCategory.LITELLM_RATE_LIMIT
+ assert exc_info.value.rate_limit_type == RateLimitType.REQUESTS
+
+ @pytest.mark.asyncio
+ async def test_should_label_async_tpm_overrun_as_litellm_tokens(self):
+ check = self._build_check()
+ deployment = {
+ "litellm_params": {"model": "gpt-4", "tpm": 10},
+ "model_info": {"id": "abc-123", "tpm": 10},
+ "model_name": "gpt-4",
+ }
+
+ async def _stub(*args, **kwargs):
+ return 999
+
+ with patch.object(check.dual_cache, "async_get_cache", side_effect=_stub):
+ with pytest.raises(litellm.RateLimitError) as exc_info:
+ await check.async_pre_call_check(deployment)
+
+ assert exc_info.value.category == RateLimitErrorCategory.LITELLM_RATE_LIMIT
+ assert exc_info.value.rate_limit_type == RateLimitType.TOKENS
+
+ @pytest.mark.asyncio
+ async def test_should_label_async_rpm_overrun_as_litellm_requests(self):
+ check = self._build_check()
+ deployment = {
+ "litellm_params": {"model": "gpt-4", "rpm": 1},
+ "model_info": {"id": "abc-123", "rpm": 1},
+ "model_name": "gpt-4",
+ }
+
+ async def _stub_inc(*args, **kwargs):
+ return 42
+
+ with patch.object(
+ check.dual_cache,
+ "async_increment_cache",
+ side_effect=_stub_inc,
+ ):
+ with pytest.raises(litellm.RateLimitError) as exc_info:
+ await check.async_pre_call_check(deployment)
+
+ assert exc_info.value.category == RateLimitErrorCategory.LITELLM_RATE_LIMIT
+ assert exc_info.value.rate_limit_type == RateLimitType.REQUESTS
+
+
+class TestAnthropicMockVendorCategory:
+ """
+ The Anthropic experimental-pass-through mock raise simulates an upstream
+ vendor 429 — it must carry ``category=VENDOR_RATE_LIMIT`` so callers that
+ use the mock to test vendor-side behavior see the right category.
+ """
+
+ def test_should_label_anthropic_mock_rate_limit_as_vendor(self):
+ from litellm.llms.anthropic.experimental_pass_through.messages.utils import (
+ mock_response as anthropic_mock_response,
+ )
+
+ with pytest.raises(litellm.RateLimitError) as exc_info:
+ anthropic_mock_response(
+ model="claude-3-opus",
+ messages=[],
+ max_tokens=10,
+ mock_response="litellm.RateLimitError",
+ )
+
+ assert exc_info.value.category == RateLimitErrorCategory.VENDOR_RATE_LIMIT
+
+
+# ---------------------------------------------------------------------------
+# Step 2 regression: vendor mappings in exception_mapping_utils still vendor
+# ---------------------------------------------------------------------------
+
+
+class TestExceptionMappingVendorRegression:
+ """
+ Step 2 added explicit ``category=VENDOR_RATE_LIMIT`` at every raise site
+ in ``exception_mapping_utils.py``. These tests exercise a few
+ representative branches end-to-end through ``exception_type`` to prove
+ none were missed — without this explicit kwarg, the new
+ ``UNKNOWN_RATE_LIMIT`` default would silently leak into vendor flows.
+ """
+
+ def _make_upstream_429(self):
+ # Build a synthetic httpx-flavored 429 response that
+ # ``exception_type`` will see when it inspects ``original_exception``.
+ request = httpx.Request("POST", "https://example.test/v1/chat")
+ return httpx.Response(status_code=429, request=request)
+
+ def test_should_label_anthropic_status_429_as_vendor(self):
+ from litellm.litellm_core_utils.exception_mapping_utils import (
+ exception_type,
+ )
+
+ original = Exception("anthropic ratelimit")
+ original.status_code = 429 # type: ignore[attr-defined]
+ original.message = "ratelimit" # type: ignore[attr-defined]
+ original.response = self._make_upstream_429() # type: ignore[attr-defined]
+
+ with pytest.raises(litellm.RateLimitError) as exc_info:
+ exception_type(
+ model="claude-3-opus-20240229",
+ original_exception=original,
+ custom_llm_provider="anthropic",
+ )
+
+ assert exc_info.value.category == RateLimitErrorCategory.VENDOR_RATE_LIMIT
+
+ def test_should_label_string_match_429_as_vendor(self):
+ # The first generic raise (string-match path with
+ # ExceptionCheckers.is_error_str_rate_limit) covers the
+ # "rate limit" substring pattern used by many providers' SDKs.
+ from litellm.litellm_core_utils.exception_mapping_utils import (
+ exception_type,
+ )
+
+ original = Exception("something Rate limit reached for model")
+ original.response = self._make_upstream_429() # type: ignore[attr-defined]
+
+ with pytest.raises(litellm.RateLimitError) as exc_info:
+ exception_type(
+ model="gpt-4",
+ original_exception=original,
+ custom_llm_provider="openai",
+ )
+
+ assert exc_info.value.category == RateLimitErrorCategory.VENDOR_RATE_LIMIT
+
+ def test_should_label_replicate_status_429_as_vendor(self):
+ from litellm.litellm_core_utils.exception_mapping_utils import (
+ exception_type,
+ )
+
+ original = Exception("replicate ratelimit")
+ original.status_code = 429 # type: ignore[attr-defined]
+ original.message = "Rate limit reached" # type: ignore[attr-defined]
+ original.response = self._make_upstream_429() # type: ignore[attr-defined]
+
+ with pytest.raises(litellm.RateLimitError) as exc_info:
+ exception_type(
+ model="meta/llama-2-70b-chat",
+ original_exception=original,
+ custom_llm_provider="replicate",
+ )
+
+ assert exc_info.value.category == RateLimitErrorCategory.VENDOR_RATE_LIMIT
diff --git a/tests/test_litellm/test_rate_limit_error_unification.py b/tests/test_litellm/test_rate_limit_error_unification.py
--- a/tests/test_litellm/test_rate_limit_error_unification.py
+++ b/tests/test_litellm/test_rate_limit_error_unification.py
@@ -57,12 +57,15 @@
class TestRateLimitErrorCategoryAttribute:
- def test_should_default_to_vendor_rate_limit_when_unspecified(self):
- # Existing callers (the exception_mapping_utils 429 paths) construct
- # RateLimitError without passing `category`. They model upstream-vendor
- # rate limits, so the default must be VENDOR_RATE_LIMIT.
+ def test_should_default_to_unknown_rate_limit_when_unspecified(self):
+ # The default category is intentionally an honest "unknown" sentinel
+ # (NOT a vendor assumption) so that any future caller that forgets to
+ # pass a category surfaces in dashboards as ``unknown_rate_limit``
+ # rather than getting silently mislabeled as a vendor 429. Every
+ # vendor-mapping callsite in ``exception_mapping_utils`` and the
+ # litellm-side router/proxy raises now pass an explicit category.
e = RateLimitError(message="oops", llm_provider="openai", model="gpt-4")
- assert e.category == RateLimitErrorCategory.VENDOR_RATE_LIMIT
+ assert e.category == RateLimitErrorCategory.UNKNOWN_RATE_LIMIT
def test_should_accept_string_category(self):
e = RateLimitError(
@@ -296,19 +299,24 @@
assert info["error_rate_limit_category"] == "litellm_rate_limit"
assert info["error_code"] == "429"
- def test_should_propagate_vendor_category_for_plain_rate_limit_error(self):
+ def test_should_propagate_unknown_category_for_plain_rate_limit_error(self):
from litellm.litellm_core_utils.litellm_logging import (
StandardLoggingPayloadSetup,
)
e = RateLimitError(
- message="vendor 429",
+ message="rate limited",
llm_provider="openai",
model="gpt-4",
)
info = StandardLoggingPayloadSetup.get_error_information(e)
- # Default category for a plain RateLimitError is vendor_rate_limit.
- assert info["error_rate_limit_category"] == "vendor_rate_limit"
+ # The default category for a plain ``RateLimitError`` (i.e. constructed
+ # without an explicit ``category=``) is the honest ``unknown_rate_limit``
... diff truncated: showing 800 of 808 linesYou can send follow-ups to the cloud agent here.
…imit_type The terminal raise in LowestTPMLoggingHandler_v2.async_get_available_deployments intentionally leaves rate_limit_type unset because the filtering could be driven by TPM, RPM, or both. Update the test (and the class docstring) to assert rate_limit_type is None instead of REQUESTS, matching the code's documented behavior.
Contributor
|
bugbot run |
Contributor
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 79dd636. Configure here.
Base automatically changed from
litellm_standardize_rate_limit_errors-5fb4
to
litellm_internal_staging
June 7, 2026 00:50
Contributor
|
🚅 Hi, thanks for the PR! I'm Agent Shin, the automated triage bot for this repository. What's this and why am I getting it? I read the description against our contribution rubric. Here's how it lined up: What you got right:
What's still missing:
If the description isn't updated in the next 24 hours, I'll auto-close this PR. That's not us saying we don't care about the change; we want the open-PR list to mirror what a maintainer can act on right now, so contributors don't get lost in a backlog. A closed PR is a soft "park this for later," not a rejection. Take your time; everything below still works after the close. During the grace period: just update the PR description with the missing pieces. No need to ping me; I'll re-check on the next sweep and skip the auto-close if it now passes. See what counts as QA proof for the full rubric (a linked issue alone isn't enough; it covers context, not proof). If the PR does get auto-closed in 24 hours, you still have easy recovery paths:
Internal BerriAI contributors: this rubric doesn't apply to you; ping a maintainer. (I'm an LLM, so I'm not infallible. If you think I got this wrong, ping a maintainer; they'll override me.) |
Contributor
|
🚅 Hi, thanks for the PR! I'm Agent Shin, the automated triage bot for this repository. What's this and why am I getting it? I read the description against our contribution rubric. Here's how it lined up: What you got right:
What's still missing:
Closing this PR isn't a rejection of the change. We want the open-PR list to mirror what a maintainer can act on right now, so contributors don't get lost in a backlog. A closed PR is a soft "park this for later"; your work is still here, the diff is still here, and getting it reopened is one comment away. Take your time. To bring this PR back:
What "end-to-end QA proof" means, since it's the most common gap: at least one of a short before/after screen recording / video (the bug reproducing, then the fix working; for a brand-new feature, a recording of it working end-to-end), a screenshot (or before/after screenshots) of it working, or the exact commands you ran paired with their real output against the real system. Running Internal BerriAI contributors: this rubric doesn't apply to you; ping a maintainer. (I'm an LLM, so I'm not infallible. If you think I got this wrong, comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #27687. Third follow-up to the unified rate-limit error work.
The bug
PR #27687 made
RateLimitError.__init__'s newcategory=kwarg default toRateLimitErrorCategory.VENDOR_RATE_LIMIT. This silently mislabels everyRateLimitError(...)raise that does NOT explicitly pass a category — and several real router-side raises do exactly that:litellm/router_strategy/lowest_tpm_rpm_v2.py— 5 raises ("Deployment over defined rpm limit").litellm/router_utils/pre_call_checks/model_rate_limit_check.py— 4 raises (router-side model RPM/TPM enforcement).Every time one of these fires today, dashboards / callbacks attribute it to the upstream vendor. Trho's whole motivation for #27687 was to be able to split litellm-side vs. vendor-side throttles; the router-side throttles silently violate that.
The fix
Three parts:
1. New
UNKNOWN_RATE_LIMITdefaultAdd
RateLimitErrorCategory.UNKNOWN_RATE_LIMIT = "unknown_rate_limit"and flip theRateLimitError.__init__default fromVENDOR_RATE_LIMITtoUNKNOWN_RATE_LIMIT. Future raise sites that forget the kwarg now get a visible "I don't know" label instead of a confidently-wrong vendor label.2. Make every vendor raise explicit
Add
category=RateLimitErrorCategory.VENDOR_RATE_LIMITto all 24 vendor raises across:litellm/litellm_core_utils/exception_mapping_utils.py(22 — central choke point)litellm/llms/anthropic/experimental_pass_through/messages/utils.py(1)litellm/main.py(1, mock-testing path)This keeps vendor behavior identical — pinned by tests in Step 4.
3. Make every router-side raise explicit + add
rate_limit_typelowest_tpm_rpm_v2.pyLITELLM_RATE_LIMITREQUESTSmodel_rate_limit_check.pyLITELLM_RATE_LIMITTOKENSmodel_rate_limit_check.pyLITELLM_RATE_LIMITREQUESTSThese 9 raises previously emitted
category="vendor_rate_limit"; they now correctly emitcategory="litellm_rate_limit"with the rightrate_limit_type.Out of scope / left on default
router.py::_handle_mock_testing_rate_limit_error(1 raise) — a test mock; the source it simulates depends on what the test author wants. Left onUNKNOWN_RATE_LIMITrather than guessing.RouterRateLimitError(ValueError)raises inrouter.py/router_utils/handle_error.py— distinct class without acategoryattribute. Out of scope; they'd need their own subclass treatment if we want them in the unification.Tests
tests/test_litellm/test_rate_limit_category_router_side.py— 16 new tests covering:UNKNOWN_RATE_LIMITenum value + that it's the new default.lowest_tpm_rpm_v2.pyandmodel_rate_limit_check.pycorrectly emitLITELLM_RATE_LIMIT+ the rightrate_limit_type.exception_mapping_utils.exception_typestill emitVENDOR_RATE_LIMIT(regression guard for Step 2).VENDOR_RATE_LIMIT.Broader sweep across
tests/test_litellm/(~9100 tests) showed only pre-existing failures (vertex/gemini/duration_parser/openai_env/websearch) unrelated to this change — confirmed by re-running the same failing tests onlitellm_internal_stagingwithout this branch.uv run black .anduv run ruff check .clean on touched files.Relationship to other follow-ups
This PR is independent of and complementary to:
rate_limit_category/rate_limit_typelabels +exception_classback-compat shim.llm_providerfield on internal proxy-side 429s.All three target #27687's loose ends from different angles. They don't touch each other's files (this one:
exceptions.py,exception_mapping_utils.py,router_strategy/,router_utils/pre_call_checks/; #27706:integrations/prometheus.py,types/integrations/prometheus.py; #27707:proxy/hooks/).Base branch note
Targeting
litellm_standardize_rate_limit_errors-5fb4(PR #27687's branch) rather thanlitellm_internal_stagingbecauseRateLimitErrorCategoryis introduced by #27687 and isn't on staging yet. After #27687 lands, this should be retargeted tolitellm_internal_stagingfor a clean diff (the resulting diff will be only the 4 commits on this branch).Slack Thread