feat: standardize rate limit errors with category, rate_limit_type, model, and llm_provider fields - #27687
Merged
mateo-berri merged 59 commits intoJun 7, 2026
Conversation
|
|
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
mateo-berri
marked this pull request as ready for review
May 11, 2026 22:57
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Contributor
Greptile SummaryThis PR unifies all rate-limit error paths across litellm's proxy-side hooks behind a single
Confidence Score: 5/5The PR is safe to merge. All proxy hook raise paths are covered by the new unified class; backward-compat shims preserve existing Prometheus dashboard labels and FastAPI exception-handler behavior; new fields are additive and optional with safe defaults. The dual-base MRO is explicitly initialized with no cooperative super() surprises, the resolver helper has three-layer defensive fallback so it can never mask the rate-limit error it supports, and every changed raise site is covered by focused unit tests. The label-caching approach is sound for the documented restart-to-change semantics, and the back-compat shims keep existing dashboards working. No data-loss or security-boundary issues were found. litellm/integrations/prometheus.py — the _cached_metric_labels lazy-population means litellm_proxy_failed_requests_metric's label set is snapshotted on the first failure event rather than at logger init, which could behave unexpectedly if prometheus_emit_rate_limit_labels is toggled between startup and the first 429.
|
| Filename | Overview |
|---|---|
| litellm/exceptions.py | Adds RateLimitErrorCategory/RateLimitType enums, validation helpers, and new optional kwargs (category, rate_limit_type, headers, detail) to RateLimitError; BudgetExceededError gains llm_provider and exposes unified category/rate_limit_type fields without joining the RateLimitError hierarchy. |
| litellm/proxy/common_utils/proxy_rate_limit_error.py | New file defining ProxyRateLimitError (dual-base HTTPException + RateLimitError) and map_v3_rate_limit_type helper; explicit attribute restoration after dual init calls is correct and well-commented. |
| litellm/proxy/hooks/rate_limiter_utils.py | Adds resolve_llm_provider_for_rate_limit with router-alias fallback and triple-layer exception safety; deferred proxy_server import is appropriate for the proxy folder. |
| litellm/integrations/prometheus.py | Adds _cached_metric_labels snapshot at init, _extract_rate_limit_labels helper, BudgetExceededError back-compat short-circuit, and prometheus_exception_class_name marker support; label caching works correctly but the cache for litellm_proxy_failed_requests_metric is lazily populated rather than pre-warmed at init. |
| litellm/proxy/hooks/parallel_request_limiter.py | Refactored to raise ProxyRateLimitError with correct triggered dimension; raise_rate_limit_error now annotated NoReturn and raises internally (outer raise removed). |
| litellm/proxy/auth/auth_exception_handler.py | Resolves llm_provider on BudgetExceededError before the post-call failure hook runs, ensuring StandardLoggingPayload gets provider attribution for budget-exceeded 429s raised during auth. |
| litellm/types/integrations/prometheus.py | Adds RATE_LIMIT_CATEGORY/RATE_LIMIT_TYPE constants, UserAPIKeyLabelNames enum members, optional label fields on UserAPIKeyLabelValues, and conditional label appending in get_labels guarded by prometheus_emit_rate_limit_labels. |
Reviews (12): Last reviewed commit: "refactor(prometheus): drop transitive fa..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
Contributor
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 1 · PR risk: 0/10 |
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
Autofix Details
Bugbot Autofix resolved 1 of the 2 issues found in the latest run.
- ✅ Fixed: Unused
error_messagevariable; detail embeds literalNone- Wired the computed
error_message(with proper None guard) into theProxyRateLimitError(detail=...)argument, removing the literalNoneinterpolation and the dead variable.
- Wired the computed
Preview (bc239e9b07)
diff --git a/litellm/__init__.py b/litellm/__init__.py
--- a/litellm/__init__.py
+++ b/litellm/__init__.py
@@ -1255,6 +1255,7 @@
NotFoundError,
PermissionDeniedError,
RateLimitError,
+ RateLimitErrorCategory,
ServiceUnavailableError,
BadGatewayError,
OpenAIError,
diff --git a/litellm/exceptions.py b/litellm/exceptions.py
--- a/litellm/exceptions.py
+++ b/litellm/exceptions.py
@@ -9,13 +9,45 @@
## LiteLLM versions of the OpenAI Exception Types
-from typing import Any, Dict, Optional
+import enum
+from typing import Any, Dict, Optional, Union
import httpx
import openai
from litellm.types.utils import LiteLLMCommonStrings
+
+class RateLimitErrorCategory(str, enum.Enum):
+ """
+ Category of a rate limit error, allowing callers to distinguish where the rate
+ limit originated. Exposed on every :class:`RateLimitError` instance via the
+ ``category`` attribute.
+
+ Use these values to switch on the rate limit source, e.g.::
+
+ try:
+ ...
+ except litellm.RateLimitError as e:
+ if e.category == RateLimitErrorCategory.LITELLM_RATE_LIMIT:
+ ... # litellm's own limiter (key/team/user/model RPM/TPM/budget)
+ elif e.category == RateLimitErrorCategory.VENDOR_RATE_LIMIT:
+ ... # the upstream LLM provider returned 429
+ """
+
+ VENDOR_RATE_LIMIT = "vendor_rate_limit"
+ """The upstream LLM provider returned a rate-limit response (e.g. OpenAI 429)."""
+
+ VENDOR_BATCH_RATE_LIMIT = "vendor_batch_rate_limit"
+ """The upstream LLM provider returned a rate-limit response on a batch endpoint."""
+
+ LITELLM_RATE_LIMIT = "litellm_rate_limit"
+ """LiteLLM's own rate limiter (key/team/user/model RPM/TPM, budget, parallel-requests, etc.) blocked the request."""
+
+ LITELLM_BATCH_RATE_LIMIT = "litellm_batch_rate_limit"
+ """LiteLLM's own batch rate limiter (token/request budget across a batch input file) blocked the request."""
+
+
_MINIMAL_ERROR_RESPONSE: Optional[httpx.Response] = None
@@ -321,6 +353,18 @@
class RateLimitError(openai.RateLimitError): # type: ignore
+ """
+ Unified rate-limit error.
+
+ Every rate-limit condition surfaced by litellm — whether it originated from
+ an upstream LLM provider, a vendor batch endpoint, or one of litellm's own
+ proxy-side limiters (parallel-requests, dynamic-rate, batch-rate, budget,
+ max-iterations, etc.) — is raised as an instance of this class.
+
+ The :attr:`category` attribute lets callers distinguish the source. See
+ :class:`RateLimitErrorCategory` for the available values.
+ """
+
def __init__(
self,
message,
@@ -330,6 +374,11 @@
litellm_debug_info: Optional[str] = None,
max_retries: Optional[int] = None,
num_retries: Optional[int] = None,
+ category: Union[str, RateLimitErrorCategory] = (
+ RateLimitErrorCategory.VENDOR_RATE_LIMIT
+ ),
+ headers: Optional[Dict[str, str]] = None,
+ detail: Any = None,
):
self.status_code = 429
self.message = "litellm.RateLimitError: {}".format(message)
@@ -338,9 +387,21 @@
self.litellm_debug_info = litellm_debug_info
self.max_retries = max_retries
self.num_retries = num_retries
+ self.category = (
+ category.value if isinstance(category, RateLimitErrorCategory) else category
+ )
+ # Headers carried with the error (e.g. retry-after, rate_limit_type,
+ # reset_at). Preserved across the proxy boundary so clients can react
+ # appropriately.
_response_headers = (
getattr(response, "headers", None) if response is not None else None
)
+ self.headers: Optional[Dict[str, str]] = (
+ {k: str(v) for k, v in headers.items()} if headers else None
+ ) or (dict(_response_headers) if _response_headers else None)
+ # Mirrors FastAPI HTTPException.detail so the same instance can be
+ # serialized through both the ProxyException and HTTPException paths.
+ self.detail = detail if detail is not None else self.message
self.response = httpx.Response(
status_code=429,
headers=_response_headers,
diff --git a/litellm/litellm_core_utils/litellm_logging.py b/litellm/litellm_core_utils/litellm_logging.py
--- a/litellm/litellm_core_utils/litellm_logging.py
+++ b/litellm/litellm_core_utils/litellm_logging.py
@@ -5159,12 +5159,23 @@
# Get additional error details
error_message = str(original_exception)
+ # For rate-limit errors (litellm.RateLimitError + the proxy-side
+ # ProxyRateLimitError subclass), surface the unified `category` field
+ # so callbacks can distinguish vendor vs. litellm rate limits without
+ # reaching for the raw exception object.
+ rate_limit_category: Optional[str] = (
+ getattr(original_exception, "category", None)
+ if isinstance(original_exception, litellm.RateLimitError)
+ else None
+ )
+
return StandardLoggingPayloadErrorInformation(
error_code=error_status,
error_class=error_class,
llm_provider=_llm_provider_in_exception,
traceback=traceback_info,
error_message=error_message if original_exception else "",
+ error_rate_limit_category=rate_limit_category,
)
@staticmethod
diff --git a/litellm/proxy/common_utils/proxy_rate_limit_error.py b/litellm/proxy/common_utils/proxy_rate_limit_error.py
new file mode 100644
--- /dev/null
+++ b/litellm/proxy/common_utils/proxy_rate_limit_error.py
@@ -1,0 +1,156 @@
+"""
+ProxyRateLimitError — a unified rate-limit exception used by litellm's
+proxy-side hooks.
+
+Background
+----------
+LiteLLM previously surfaced rate-limit conditions through *several* unrelated
+exception types:
+
+* :class:`litellm.exceptions.RateLimitError` — raised by exception mapping when
+ an upstream LLM provider returns 429.
+* :class:`fastapi.HTTPException` (status 429) — raised directly by proxy hooks
+ such as ``parallel_request_limiter``, ``dynamic_rate_limiter``,
+ ``batch_rate_limiter``, ``max_budget_limiter``, ``max_iterations_limiter``,
+ etc.
+* :class:`litellm.llms.base_llm.chat.transformation.BaseLLMException` (status
+ 429) — raised by some provider transports.
+
+This made it impossible for downstream code (and end users) to express
+"is this a rate limit?" with a single ``except`` clause, and impossible to
+distinguish *where* the rate limit originated (vendor vs. litellm, batch vs.
+chat) without ad-hoc string-matching on the message.
+
+This module provides a single proxy-side error class that:
+
+1. Is a subclass of :class:`litellm.exceptions.RateLimitError`, so user code
+ that catches ``RateLimitError`` works for *every* rate-limit source.
+2. Is also a subclass of :class:`fastapi.HTTPException`, so existing proxy
+ plumbing (``isinstance(e, HTTPException)`` branches in route handlers and
+ FastAPI's own dispatcher) continues to behave the same way and the
+ ``retry-after`` / ``rate_limit_type`` / ``reset_at`` headers are preserved
+ on the wire.
+3. Carries a :attr:`category` field (one of
+ :class:`litellm.exceptions.RateLimitErrorCategory`) so callers can switch on
+ the rate limit source.
+"""
+
+import json
+from typing import Any, Dict, Mapping, Optional, Union
+
+from fastapi import HTTPException
+
+from litellm.exceptions import RateLimitError, RateLimitErrorCategory
+
+
+def _coerce_message(detail: Any) -> str:
+ """Best-effort, JSON-friendly stringification of an HTTPException-style detail."""
+ if isinstance(detail, str):
+ return detail
+ if isinstance(detail, Mapping):
+ for key in ("error", "message"):
+ if isinstance(detail.get(key), str):
+ return detail[key]
+ inner = detail.get(key)
+ if isinstance(inner, Mapping) and isinstance(inner.get("message"), str):
+ return inner["message"]
+ try:
+ return json.dumps(detail)
+ except (TypeError, ValueError):
+ return str(detail)
+ return str(detail)
+
+
+# NOTE: mypy emits two `[misc]` errors on the class line below because the
+# bases declare overlapping attributes with related-but-not-identical
+# annotations:
+# * `status_code` is `int` on starlette HTTPException but `Literal[429]` on
+# openai.RateLimitError (every openai status-error subclass narrows it
+# this way and silences pyright with the same convention).
+# * `headers` is `Mapping[str, str] | None` on HTTPException; we narrow it
+# to `Optional[Dict[str, str]]` on RateLimitError because we always carry
+# a stringified dict.
+# Both narrowings are intentional and handled at construction time — every
+# instance always has status_code == 429 and a Dict-typed headers — so we
+# silence the ATTR-overlap check rather than relax the annotations.
+class ProxyRateLimitError(HTTPException, RateLimitError): # type: ignore[misc]
+ """
+ A 429 raised by litellm's proxy-side rate limiting hooks.
+
+ This class deliberately inherits from BOTH
+ :class:`litellm.exceptions.RateLimitError` and :class:`fastapi.HTTPException`
+ so the same instance can flow through:
+
+ * ``except RateLimitError`` (user / SDK code that wants a category-aware
+ handler), and
+ * ``isinstance(e, HTTPException)`` (FastAPI / proxy_server.py route
+ handlers that need to forward ``status_code``, ``detail`` and
+ ``headers`` back to the client).
+
+ Downstream code should prefer this class over
+ ``raise HTTPException(status_code=429, ...)`` for litellm-internal rate
+ limits.
+
+ Parameters
+ ----------
+ detail:
+ The structured error payload. Forwarded as ``HTTPException.detail`` so
+ FastAPI's default exception handler will serialize it verbatim.
+ headers:
+ Optional response headers (e.g. ``retry-after``). Values are stringified
+ to satisfy FastAPI's typing.
+ category:
+ One of :class:`RateLimitErrorCategory`. Defaults to
+ ``LITELLM_RATE_LIMIT`` since this class is only used by litellm's own
+ proxy-side limiters; pass ``LITELLM_BATCH_RATE_LIMIT`` for the batch
+ limiter, etc.
+ model / llm_provider:
+ Optional context, propagated to the inherited ``RateLimitError`` for
+ compatibility with logging / standard payload extraction.
+ """
+
+ def __init__(
+ self,
+ detail: Any,
+ headers: Optional[Mapping[str, Any]] = None,
+ category: Union[
+ str, RateLimitErrorCategory
+ ] = RateLimitErrorCategory.LITELLM_RATE_LIMIT,
+ model: Optional[str] = None,
+ llm_provider: str = "litellm_proxy",
+ ):
+ message = _coerce_message(detail)
+ stringified_headers: Optional[Dict[str, str]] = (
+ {k: str(v) for k, v in headers.items()} if headers else None
+ )
+
+ # Initialize the FastAPI HTTPException portion first so its attributes
+ # (status_code, detail, headers) are already on the instance before
+ # RateLimitError.__init__ runs and possibly overrides them.
+ HTTPException.__init__(
+ self,
+ status_code=429,
+ detail=detail,
+ headers=stringified_headers,
+ )
+
+ # Now initialize the litellm RateLimitError portion. We deliberately
+ # pass the structured detail through so RateLimitError preserves it as
+ # its `.detail` attribute too — keeping both sides of the MRO
+ # consistent.
+ RateLimitError.__init__(
+ self,
+ message=message,
+ llm_provider=llm_provider,
+ model=model or "",
+ category=category,
+ headers=stringified_headers,
+ detail=detail,
+ )
+ # RateLimitError.__init__ overwrites self.headers with its own copy and
+ # leaves self.status_code at 429 — restore the HTTPException-style
+ # headers value so downstream code that pulls headers off the
+ # instance gets back exactly what the limiter passed in.
+ self.headers = stringified_headers
+ self.detail = detail
+ self.status_code = 429
diff --git a/litellm/proxy/hooks/batch_rate_limiter.py b/litellm/proxy/hooks/batch_rate_limiter.py
--- a/litellm/proxy/hooks/batch_rate_limiter.py
+++ b/litellm/proxy/hooks/batch_rate_limiter.py
@@ -29,8 +29,10 @@
_get_file_content_as_dictionary,
_get_models_from_batch_input_file_content,
)
+from litellm.exceptions import RateLimitErrorCategory
from litellm.integrations.custom_logger import CustomLogger
from litellm.proxy._types import UserAPIKeyAuth
+from litellm.proxy.common_utils.proxy_rate_limit_error import ProxyRateLimitError
if TYPE_CHECKING:
from opentelemetry.trace import Span as _Span
@@ -105,7 +107,7 @@
batch_usage: BatchFileUsage,
limit_type: str,
) -> None:
- """Raise HTTPException for rate limit exceeded."""
+ """Raise :class:`ProxyRateLimitError` (a 429) for batch rate limit exceeded."""
from datetime import datetime
# Find the descriptor for this status
@@ -148,14 +150,14 @@
f"Limit resets at: {reset_time_formatted}"
)
- raise HTTPException(
- status_code=429,
+ raise ProxyRateLimitError(
detail=detail,
headers={
"retry-after": str(window_size),
"rate_limit_type": limit_type,
"reset_at": reset_time_formatted,
},
+ category=RateLimitErrorCategory.LITELLM_BATCH_RATE_LIMIT,
)
async def _check_and_increment_batch_counters(
diff --git a/litellm/proxy/hooks/dynamic_rate_limiter.py b/litellm/proxy/hooks/dynamic_rate_limiter.py
--- a/litellm/proxy/hooks/dynamic_rate_limiter.py
+++ b/litellm/proxy/hooks/dynamic_rate_limiter.py
@@ -6,14 +6,13 @@
import os
from typing import List, Optional, Tuple, Union
-from fastapi import HTTPException
-
import litellm
from litellm import ModelResponse, Router
from litellm._logging import verbose_proxy_logger
from litellm.caching.caching import DualCache
from litellm.integrations.custom_logger import CustomLogger
from litellm.proxy._types import UserAPIKeyAuth
+from litellm.proxy.common_utils.proxy_rate_limit_error import ProxyRateLimitError
from litellm.types.router import ModelGroupInfo
from litellm.types.utils import CallTypesLiteral
from litellm.utils import get_utc_datetime
@@ -218,8 +217,7 @@
)
### CHECK TPM ###
if available_tpm is not None and available_tpm == 0:
- raise HTTPException(
- status_code=429,
+ raise ProxyRateLimitError(
detail={
"error": "Key={} over available TPM={}. Model TPM={}, Active keys={}".format(
user_api_key_dict.api_key,
@@ -231,8 +229,7 @@
)
### CHECK RPM ###
elif available_rpm is not None and available_rpm == 0:
- raise HTTPException(
- status_code=429,
+ raise ProxyRateLimitError(
detail={
"error": "Key={} over available RPM={}. Model RPM={}, Active keys={}".format(
user_api_key_dict.api_key,
diff --git a/litellm/proxy/hooks/dynamic_rate_limiter_v3.py b/litellm/proxy/hooks/dynamic_rate_limiter_v3.py
--- a/litellm/proxy/hooks/dynamic_rate_limiter_v3.py
+++ b/litellm/proxy/hooks/dynamic_rate_limiter_v3.py
@@ -14,6 +14,7 @@
from litellm.caching.caching import DualCache
from litellm.integrations.custom_logger import CustomLogger
from litellm.proxy._types import UserAPIKeyAuth
+from litellm.proxy.common_utils.proxy_rate_limit_error import ProxyRateLimitError
from litellm.proxy.hooks.parallel_request_limiter_v3 import (
RateLimitDescriptor,
RateLimitDescriptorRateLimitObject,
@@ -492,8 +493,7 @@
continue
descriptor_key = status["descriptor_key"]
if descriptor_key == "model_saturation_check":
- raise HTTPException(
- status_code=429,
+ raise ProxyRateLimitError(
detail={
"error": f"Model capacity reached for {model}. "
f"Priority: {priority}, "
@@ -513,8 +513,7 @@
f"Enforcing priority limits for {model}, saturation: {saturation:.1%}, "
f"priority: {priority}"
)
- raise HTTPException(
- status_code=429,
+ raise ProxyRateLimitError(
detail={
"error": f"Priority-based rate limit exceeded. "
f"Model: {model}, "
@@ -547,8 +546,7 @@
f"Dynamic rate limiter: OVER_LIMIT response with unknown "
f"descriptor_key(s) — refusing request. response={atomic_response}"
)
- raise HTTPException(
- status_code=429,
+ raise ProxyRateLimitError(
detail={
"error": "Rate limit exceeded",
"descriptor_key": (
diff --git a/litellm/proxy/hooks/max_budget_limiter.py b/litellm/proxy/hooks/max_budget_limiter.py
--- a/litellm/proxy/hooks/max_budget_limiter.py
+++ b/litellm/proxy/hooks/max_budget_limiter.py
@@ -5,6 +5,7 @@
from litellm.caching.caching import DualCache
from litellm.integrations.custom_logger import CustomLogger
from litellm.proxy._types import UserAPIKeyAuth
+from litellm.proxy.common_utils.proxy_rate_limit_error import ProxyRateLimitError
class _PROXY_MaxBudgetLimiter(CustomLogger):
@@ -63,7 +64,7 @@
# CHECK IF REQUEST ALLOWED
if curr_spend >= max_budget:
- raise HTTPException(status_code=429, detail="Max budget limit reached.")
+ raise ProxyRateLimitError(detail="Max budget limit reached.")
except HTTPException as e:
raise e
except Exception as e:
diff --git a/litellm/proxy/hooks/max_budget_per_session_limiter.py b/litellm/proxy/hooks/max_budget_per_session_limiter.py
--- a/litellm/proxy/hooks/max_budget_per_session_limiter.py
+++ b/litellm/proxy/hooks/max_budget_per_session_limiter.py
@@ -17,12 +17,11 @@
import os
from typing import TYPE_CHECKING, Any, Optional, Union
-from fastapi import HTTPException
-
from litellm import DualCache
from litellm._logging import verbose_proxy_logger
from litellm.integrations.custom_logger import CustomLogger
from litellm.proxy._types import UserAPIKeyAuth
+from litellm.proxy.common_utils.proxy_rate_limit_error import ProxyRateLimitError
if TYPE_CHECKING:
from litellm.proxy.utils import InternalUsageCache as _InternalUsageCache
@@ -112,8 +111,7 @@
)
if current_spend >= max_budget:
- raise HTTPException(
- status_code=429,
+ raise ProxyRateLimitError(
detail=(
f"Session budget exceeded for session {session_id}. "
f"Current spend: ${current_spend:.4f}, "
diff --git a/litellm/proxy/hooks/max_iterations_limiter.py b/litellm/proxy/hooks/max_iterations_limiter.py
--- a/litellm/proxy/hooks/max_iterations_limiter.py
+++ b/litellm/proxy/hooks/max_iterations_limiter.py
@@ -13,12 +13,11 @@
import os
from typing import TYPE_CHECKING, Any, Optional, Union
-from fastapi import HTTPException
-
from litellm import DualCache
from litellm._logging import verbose_proxy_logger
from litellm.integrations.custom_logger import CustomLogger
from litellm.proxy._types import UserAPIKeyAuth
+from litellm.proxy.common_utils.proxy_rate_limit_error import ProxyRateLimitError
if TYPE_CHECKING:
from litellm.proxy.utils import InternalUsageCache as _InternalUsageCache
@@ -116,8 +115,7 @@
current_count = await self._increment_and_get(cache_key)
if current_count > max_iterations:
- raise HTTPException(
- status_code=429,
+ raise ProxyRateLimitError(
detail=(
f"Max iterations exceeded for session {session_id}. "
f"Current count: {current_count}, max_iterations: {max_iterations}."
diff --git a/litellm/proxy/hooks/parallel_request_limiter.py b/litellm/proxy/hooks/parallel_request_limiter.py
--- a/litellm/proxy/hooks/parallel_request_limiter.py
+++ b/litellm/proxy/hooks/parallel_request_limiter.py
@@ -3,7 +3,6 @@
from datetime import datetime, timedelta
from typing import TYPE_CHECKING, Any, List, Literal, Optional, Tuple, Union
-from fastapi import HTTPException
from pydantic import BaseModel
from typing_extensions import TypedDict
@@ -17,6 +16,7 @@
get_key_model_rpm_limit,
get_key_model_tpm_limit,
)
+from litellm.proxy.common_utils.proxy_rate_limit_error import ProxyRateLimitError
if TYPE_CHECKING:
from opentelemetry.trace import Span as _Span
@@ -95,8 +95,7 @@
values_to_update_in_cache.append((request_count_api_key, new_val))
else:
- raise HTTPException(
- status_code=429,
+ raise ProxyRateLimitError(
detail=f"LiteLLM Rate Limit Handler for rate limit type = {rate_limit_type}. {CommonProxyErrors.max_parallel_request_limit_reached.value}. current rpm: {current['current_rpm']}, rpm limit: {rpm_limit}, current tpm: {current['current_tpm']}, tpm limit: {tpm_limit}, current max_parallel_requests: {current['current_requests']}, max_parallel_requests: {max_parallel_requests}",
headers={"retry-after": str(self.time_to_next_minute())},
)
@@ -123,16 +122,20 @@
def raise_rate_limit_error(
self, additional_details: Optional[str] = None
- ) -> HTTPException:
+ ) -> ProxyRateLimitError:
"""
- Raise an HTTPException with a 429 status code and a retry-after header
+ Raise a 429 with a retry-after header for litellm-proxy parallel-request limits.
+
+ Returns a :class:`ProxyRateLimitError`, which is both a
+ :class:`litellm.RateLimitError` (so callers can catch by category) and a
+ :class:`fastapi.HTTPException` (so the FastAPI dispatcher serializes it
+ correctly with status 429 and the supplied headers).
"""
error_message = "Max parallel request limit reached"
if additional_details is not None:
error_message = error_message + " " + additional_details
- raise HTTPException(
- status_code=429,
- detail=f"Max parallel request limit reached {additional_details}",
+ raise ProxyRateLimitError(
+ detail=error_message,
headers={"retry-after": str(self.time_to_next_minute())},
)
diff --git a/litellm/proxy/hooks/parallel_request_limiter_v3.py b/litellm/proxy/hooks/parallel_request_limiter_v3.py
--- a/litellm/proxy/hooks/parallel_request_limiter_v3.py
+++ b/litellm/proxy/hooks/parallel_request_limiter_v3.py
@@ -23,8 +23,6 @@
cast,
)
-from fastapi import HTTPException
-
from litellm import DualCache
from litellm._logging import verbose_proxy_logger
from litellm.constants import DYNAMIC_RATE_LIMIT_ERROR_THRESHOLD_PER_MINUTE
@@ -34,6 +32,7 @@
)
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.auth.auth_utils import get_model_rate_limit_from_metadata
+from litellm.proxy.common_utils.proxy_rate_limit_error import ProxyRateLimitError
from litellm.types.caching import RedisPipelineIncrementOperation
from litellm.types.llms.openai import BaseLiteLLMOpenAIResponseObject
from litellm.types.utils import ModelResponse, Usage
@@ -1838,7 +1837,7 @@
response: RateLimitResponse,
descriptors: List[RateLimitDescriptor],
) -> None:
- """Handle rate limit exceeded error by raising HTTPException."""
+ """Handle rate limit exceeded by raising :class:`ProxyRateLimitError` (a 429)."""
for status in response["statuses"]:
if status["code"] == "OVER_LIMIT":
descriptor_key = status["descriptor_key"]
@@ -1869,8 +1868,7 @@
f"Limit resets at: {reset_time_formatted}"
)
- raise HTTPException(
- status_code=429,
+ raise ProxyRateLimitError(
detail=detail,
headers={
"retry-after": str(self.window_size),
diff --git a/litellm/types/utils.py b/litellm/types/utils.py
--- a/litellm/types/utils.py
+++ b/litellm/types/utils.py
@@ -2697,8 +2697,18 @@
llm_provider: Optional[str]
traceback: Optional[str]
error_message: Optional[str]
+ error_rate_limit_category: Optional[str]
+ """
+ For 429 / rate-limit errors, the source of the rate limit. One of the
+ string values defined by :class:`litellm.exceptions.RateLimitErrorCategory`
+ (``vendor_rate_limit``, ``vendor_batch_rate_limit``, ``litellm_rate_limit``,
+ ``litellm_batch_rate_limit``). ``None`` for non-rate-limit exceptions.
+ Surfaced here so custom callbacks / metrics consumers can switch on the
+ rate-limit source without reaching for the raw exception.
+ """
+
class GuardrailMode(TypedDict, total=False):
tags: Optional[Dict[str, Union[str, List[str]]]]
default: Optional[Union[str, List[str]]]
diff --git a/tests/test_litellm/test_rate_limit_error_unification.py b/tests/test_litellm/test_rate_limit_error_unification.py
new file mode 100644
--- /dev/null
+++ b/tests/test_litellm/test_rate_limit_error_unification.py
@@ -1,0 +1,750 @@
+"""
+Tests for the unified rate-limit error model introduced by LIT-2968.
+
+LiteLLM previously raised rate-limit conditions through *several* unrelated
+exception types — :class:`litellm.RateLimitError` (vendor 429s),
+:class:`fastapi.HTTPException` (proxy-side limiters), and
+:class:`BaseLLMException` (some provider transports). These tests pin down
+the new behavior:
+
+1. Every rate-limit exception is a :class:`litellm.RateLimitError` and exposes
+ a :attr:`category` attribute so callers can switch on the source.
+2. Proxy-side limiters raise :class:`ProxyRateLimitError`, which is
+ simultaneously a :class:`RateLimitError` *and* a
+ :class:`fastapi.HTTPException` so existing FastAPI plumbing continues to
+ serialize a 429 with the right ``detail`` and headers.
+3. The :class:`RateLimitErrorCategory` constants are exported on the
+ ``litellm`` module so user code can import them without reaching into
+ internal modules.
+"""
+
+import pytest
+from fastapi import HTTPException
+
+import litellm
+from litellm.exceptions import RateLimitError, RateLimitErrorCategory
+from litellm.proxy.common_utils.proxy_rate_limit_error import (
+ ProxyRateLimitError,
+)
+
+
+class TestRateLimitErrorCategory:
+ def test_should_export_category_enum_on_litellm_module(self):
+ assert hasattr(litellm, "RateLimitErrorCategory")
+ assert litellm.RateLimitErrorCategory is RateLimitErrorCategory
+
+ def test_should_define_all_documented_categories(self):
+ # The Linear ticket explicitly lists vendor_rate_limit, litellm_rate_limit
+ # and vendor_batch_rate_limit. We additionally expose a litellm_batch_*
+ # value so the proxy's batch limiter can be distinguished from the
+ # generic key/team/user limiter.
+ assert RateLimitErrorCategory.VENDOR_RATE_LIMIT == "vendor_rate_limit"
+ assert (
+ RateLimitErrorCategory.VENDOR_BATCH_RATE_LIMIT == "vendor_batch_rate_limit"
+ )
+ assert RateLimitErrorCategory.LITELLM_RATE_LIMIT == "litellm_rate_limit"
+ assert (
+ RateLimitErrorCategory.LITELLM_BATCH_RATE_LIMIT
+ == "litellm_batch_rate_limit"
+ )
+
+ def test_should_str_compare_for_easy_user_switching(self):
+ # Storing the value as a str-enum lets users compare against a plain
+ # string without importing the enum, e.g. `if e.category == "vendor_rate_limit":`
+ assert RateLimitErrorCategory.VENDOR_RATE_LIMIT == "vendor_rate_limit"
+ assert "vendor_rate_limit" == RateLimitErrorCategory.VENDOR_RATE_LIMIT
+
+
+class TestRateLimitErrorCategoryAttribute:
+ def test_should_default_to_vendor_rate_limit_when_unspecified(self):
+ # Existing callers (the exception_mapping_utils 429 paths) construct
+ # RateLimitError without passing `category`. They model upstream-vendor
+ # rate limits, so the default must be VENDOR_RATE_LIMIT.
+ e = RateLimitError(message="oops", llm_provider="openai", model="gpt-4")
+ assert e.category == RateLimitErrorCategory.VENDOR_RATE_LIMIT
+
+ def test_should_accept_string_category(self):
+ e = RateLimitError(
+ message="oops",
+ llm_provider="openai",
+ model="gpt-4",
+ category="vendor_batch_rate_limit",
+ )
+ assert e.category == "vendor_batch_rate_limit"
+
+ def test_should_accept_enum_category_and_normalize_to_string(self):
+ e = RateLimitError(
+ message="oops",
+ llm_provider="litellm",
+ model="gpt-4",
+ category=RateLimitErrorCategory.LITELLM_RATE_LIMIT,
+ )
+ # The .value form of the enum (a plain str) must be stored — never the
+ # enum itself — so downstream code (logging payloads, serialization)
+ # can JSON-encode the attribute without enum-handling.
+ assert e.category == "litellm_rate_limit"
+ assert isinstance(e.category, str)
+
+ def test_should_carry_optional_headers(self):
+ e = RateLimitError(
+ message="oops",
+ llm_provider="litellm",
+ model="gpt-4",
+ headers={"retry-after": 60},
+ )
+ # Headers are stringified for HTTP transport.
+ assert e.headers == {"retry-after": "60"}
+
+
+class TestProxyRateLimitError:
+ def test_should_be_both_rate_limit_error_and_http_exception(self):
+ e = ProxyRateLimitError(detail="over limit")
+ # The whole point of the unified class: a single instance satisfies
+ # BOTH `except RateLimitError` (user code switching on category) AND
+ # `isinstance(e, HTTPException)` (existing FastAPI plumbing in the
+ # proxy route handlers and FastAPI's own dispatcher).
+ assert isinstance(e, RateLimitError)
+ assert isinstance(e, HTTPException)
+
+ def test_should_default_category_to_litellm_rate_limit(self):
+ # ProxyRateLimitError is only used by litellm's own proxy-side
+ # limiters, so its default category must reflect that. The vendor
+ # default lives on the parent RateLimitError.
+ e = ProxyRateLimitError(detail="over limit")
+ assert e.category == RateLimitErrorCategory.LITELLM_RATE_LIMIT
+
+ def test_should_accept_litellm_batch_rate_limit_category(self):
+ e = ProxyRateLimitError(
+ detail="batch over limit",
+ category=RateLimitErrorCategory.LITELLM_BATCH_RATE_LIMIT,
+ )
+ assert e.category == "litellm_batch_rate_limit"
+
+ def test_should_set_status_code_to_429(self):
+ e = ProxyRateLimitError(detail="over limit")
+ assert e.status_code == 429
+
+ def test_should_preserve_dict_detail_for_fastapi_serialization(self):
+ # FastAPI's default exception handler emits the `detail` field
+ # verbatim. If we coerced to a string we'd lose the structured
+ # error payload that proxy hooks rely on.
+ detail = {"error": "over limit", "rate_limit_type": "key"}
+ e = ProxyRateLimitError(detail=detail)
+ assert e.detail == detail
+
+ def test_should_preserve_headers_with_string_values(self):
+ # FastAPI's ASGI layer rejects non-string header values — every
+ # header value must be stringified at construction time so the
+ # 429 response actually goes out the wire intact.
+ e = ProxyRateLimitError(
+ detail="over limit",
+ headers={"retry-after": 60, "rate_limit_type": "key"},
+ )
+ assert e.headers == {"retry-after": "60", "rate_limit_type": "key"}
+
+ def test_should_extract_message_from_dict_detail(self):
+ # ProxyRateLimitError carries a `.message` (from RateLimitError) AND a
+ # structured `.detail` (from HTTPException). When detail is a dict in
+ # the canonical {"error": "..."} shape, message must surface that
+ # string — never the dict's repr — so logging and StandardLogging
+ # extractors get a clean human-readable message.
+ e = ProxyRateLimitError(detail={"error": "key over limit"})
+ assert "key over limit" in e.message
+
+ def test_should_be_catchable_as_rate_limit_error(self):
+ with pytest.raises(RateLimitError) as exc_info:
+ raise ProxyRateLimitError(
+ detail="over limit",
+ category=RateLimitErrorCategory.LITELLM_RATE_LIMIT,
+ )
+ assert exc_info.value.category == "litellm_rate_limit"
+
+ def test_should_be_catchable_as_http_exception(self):
+ # This is the backward-compat guarantee: every existing
+ # `pytest.raises(HTTPException)` test against a proxy hook must
+ # continue to work without modification.
+ with pytest.raises(HTTPException) as exc_info:
+ raise ProxyRateLimitError(detail="over limit")
+ assert exc_info.value.status_code == 429
+ assert exc_info.value.detail == "over limit"
+
... diff truncated: showing 800 of 1380 linesYou can send follow-ups to the cloud agent here.
Contributor
Author
|
bugbot run |
Contributor
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
2 issues from previous reviews remain unresolved.
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit a3fd231f1289c005f0b669692aefe48579fc9b12. Configure here.
Contributor
Author
…on RateLimitError
LiteLLM previously surfaced rate-limit conditions through several unrelated
error classes (RateLimitError, FastAPI HTTPException(429), BaseLLMException).
This commit adds the data model needed to consolidate them under a single
class:
* RateLimitErrorCategory enum exposing four categorical values
(vendor_rate_limit, vendor_batch_rate_limit, litellm_rate_limit,
litellm_batch_rate_limit) so callers can switch on the rate-limit source.
* New optional fields on RateLimitError:
- category (defaults to vendor_rate_limit, preserving today's behavior for
every existing call site in exception_mapping_utils);
- headers (preserves retry-after / rate_limit_type / reset_at across the
proxy boundary instead of dropping them on the floor);
- detail (mirrors FastAPI HTTPException.detail so the same instance can be
serialized through both paths).
litellm.RateLimitErrorCategory is re-exported at the package root to match
the existing exception-export pattern.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
…ception Adds a single proxy-side error class that subclasses BOTH litellm.exceptions.RateLimitError AND fastapi.HTTPException via cooperative multiple inheritance. Why both bases: * Subclassing RateLimitError lets user code catch every rate-limit source with one 'except RateLimitError' and switch on the new .category field. * Subclassing HTTPException keeps every existing FastAPI plumbing path (the isinstance(e, HTTPException) branches in proxy_server.py route handlers, FastAPI's own dispatcher, and tests asserting pytest.raises(HTTPException)) working without modification, and preserves retry-after / rate_limit_type / reset_at headers on the wire. The class declaration order is (HTTPException, RateLimitError) so the MRO puts HTTPException's no-super-call __init__ ahead of openai's cooperative __init__ chain — preventing openai.APIError.super().__init__(message) from landing in HTTPException.__init__(status_code=message). LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
…ion limiters Replaces three bare HTTPException(status_code=429, ...) call sites with ProxyRateLimitError, which is both a RateLimitError (catchable by category) and an HTTPException (preserves existing FastAPI serialization). Drops the now-unused HTTPException import in the iteration / per-session limiters. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
…t limiters Replaces HTTPException(status_code=429, ...) call sites in the v1 and v3 parallel-request limiters (key/team/user/model/customer rate limits) with ProxyRateLimitError. Updates the raise_rate_limit_error helper's return type annotation accordingly. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
…miters Replaces HTTPException(status_code=429, ...) call sites in the v1 and v3 dynamic rate limiters (project-level TPM/RPM allocation, model-saturation checks, priority-based limits, fail-closed guards) with ProxyRateLimitError. The v3 limiter still imports HTTPException for an unrelated bare 'except HTTPException:' branch. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Replaces HTTPException(status_code=429, ...) in batch_rate_limiter._raise_rate_limit_error with ProxyRateLimitError tagged as RateLimitErrorCategory.LITELLM_BATCH_RATE_LIMIT so users can distinguish batch-level throttling (which counts requests/tokens across an uploaded batch input file before submission) from the generic key/team/user RPM/TPM limiter. The HTTPException import is retained because the same module raises HTTPException for unrelated 403/IO error paths. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Adds a dedicated test module covering the new RateLimitErrorCategory enum, RateLimitError.category default + override behavior, ProxyRateLimitError's dual nature (RateLimitError + HTTPException), and a parametrized regression guard that asserts every proxy hook module imports the unified class. The regression guard catches the failure mode the refactor is designed to prevent: someone re-introducing a bare HTTPException(status_code=429, ...) in one of the hook modules instead of going through ProxyRateLimitError. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Adds an optional 'error_rate_limit_category' field to StandardLoggingPayloadErrorInformation, populated from the unified RateLimitError.category attribute (introduced in the previous commits on this branch). Why: the .category attribute is reachable off the raw exception today via getattr(e, 'category', None), but the structured contract that downstream custom callbacks / loggers / spend log writers consume is the StandardLoggingPayload. Without this field, a user building custom rate-limit metrics on top of callback data has to special-case the raw exception object — which defeats the purpose of the StandardLoggingPayload abstraction. The field is None for non-rate-limit exceptions (so consumers can read it unconditionally without isinstance checks) and is one of the RateLimitErrorCategory string values otherwise. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Five tests covering: vendor default, explicit litellm_rate_limit and litellm_batch_rate_limit values, None for non-rate-limit exceptions, and None when no exception is provided. Pins down the contract that custom callbacks can read 'error_information.error_rate_limit_category' off the StandardLoggingPayload to drive custom rate-limit metrics without ever reaching for the raw exception. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
mypy emits two [misc] errors on the ProxyRateLimitError class line because its two bases declare overlapping attributes with related-but-not-identical annotations: * status_code: int on starlette HTTPException vs. Literal[429] on openai's RateLimitError (every openai status-error subclass narrows it the same way and silences pyright with the same convention). * headers: Mapping[str, str] | None on HTTPException vs. our Optional[ Dict[str, str]] (the proxy hooks always carry a stringified dict). Both narrowings are intentional and enforced at construction time. Add a type: ignore[misc] with an inline explanation rather than relax the annotations on the parent or change the wire-format guarantees. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
…rage Adds six end-to-end tests that drive each refactored hook past its limit and assert the unified ProxyRateLimitError is raised with the correct category and dual-base shape. Complements the import-shape-only parametrized guard above by actually executing the new 'raise ProxyRateLimitError(...)' lines so codecov's patch coverage sees them as hit. Hooks covered (one test each): * parallel_request_limiter v1 — direct call to raise_rate_limit_error() * parallel_request_limiter v3 — direct call to _handle_rate_limit_error with a fabricated OVER_LIMIT response * max_iterations_limiter — full async_pre_call_hook with mocked agent registry, second call exceeds budget=1 * max_budget_limiter — async_pre_call_hook with mocked get_current_spend * dynamic_rate_limiter v1 — async_pre_call_hook with mocked check_available_usage forcing available_tpm == 0 * batch_rate_limiter — direct _raise_rate_limit_error call, asserts category is the batch-specific LITELLM_BATCH_RATE_LIMIT (not the generic LITELLM_RATE_LIMIT) LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Adds five more direct hook-invocation tests so every PR-touched line in the proxy hooks is exercised by tests in tests/test_litellm/, which codecov measures: * parallel_request_limiter v1 — check_key_in_limits inline raise (the second raise site, separate from the raise_rate_limit_error helper covered earlier) * dynamic_rate_limiter v1 — RPM raise branch (TPM branch was already covered) * dynamic_rate_limiter v3 — parametrized over all three raise sites: model_saturation_check, priority_model, and the fail-closed fallback for an unrecognized descriptor_key * max_budget_per_session_limiter — full async_pre_call_hook with a mocked agent registry and over-budget cached spend All 42 tests in test_rate_limit_error_unification.py now pass and together exercise every changed import + raise line across the eight refactored proxy hooks. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
…rate_limit_error as NoReturn The v1 ' raise_rate_limit_error' helper built an unused 'error_message' variable and then assembled the actual ' detail' via an f-string that interpolated 'additional_details' verbatim — producing 'Max parallel request limit reached None' when invoked without arguments (flagged by code review). Fix the helper to: - use the constructed 'error_message' as the detail - annotate the helper as NoReturn since it always raises - drop the redundant 'raise'/'return' at the two call sites Add two regression tests covering both the with- and without- additional_details paths. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
…_limit_errors-5fb4
Contributor
Author
Proxy-side rate limits (litellm_rate_limit, budget, max_iterations) are rejected at the gate before any upstream call, so async_post_call_failure_hook tags the synthetic failure log with LITELLM_LOGGING_NO_UPSTREAM_LLM_CALL and the v2 OTel logger never opens an LLM-call span for them; the litellm.error.rate_limit_category / litellm.error.rate_limit_type attributes were dead for exactly the cases they were meant to surface. The only failure that does open an LLM-call span carrying a RateLimitError is a vendor 429, where rate_limit_type is always None and the category just restates error.type=RateLimitError. The decomposition still reaches downstream consumers through StandardLoggingPayload.error_information.error_rate_limit_* and the prometheus rate_limit_category / rate_limit_type labels, both unchanged. Removes the SpanError fields, the _parse_error reads, the genai mapper attributes, the semconv keys, and the three span tests that asserted a scenario that never reaches the mapper in production.
Contributor
Author
Contributor
Author
|
bugbot run |
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Batch mislabels parallel limit type
- Replaced the inline tokens/else-requests branch in
_raise_rate_limit_errorwithmap_v3_rate_limit_type(limit_type)so amax_parallel_requestsv3 status maps toRateLimitType.CONCURRENT_REQUESTSfor structured logging and Prometheus.
- Replaced the inline tokens/else-requests branch in
Preview (9c92921479)
diff --git a/litellm/__init__.py b/litellm/__init__.py
--- a/litellm/__init__.py
+++ b/litellm/__init__.py
@@ -432,6 +432,13 @@
custom_prometheus_tags: List[str] = []
prometheus_metrics_config: Optional[List] = None
prometheus_emit_stream_label: bool = False
+# Opt-in: emit `rate_limit_category` and `rate_limit_type` labels on
+# `litellm_proxy_failed_requests_metric`. Off by default to preserve the
+# pre-unification label set so existing dashboards / recording rules keyed on
+# that metric keep matching after upgrade. Enable when downstream consumers
+# are ready to split 429s by source (vendor vs. litellm) and dimension
+# (RPM/TPM/concurrent/budget).
+prometheus_emit_rate_limit_labels: bool = False
prometheus_user_budget_label_include_email_alias: bool = False
prometheus_end_user_metrics_max_series_per_metric: Optional[int] = 10000
prometheus_end_user_metrics_ttl_seconds: Optional[float] = 3600.0
@@ -1277,6 +1284,8 @@
NotFoundError,
PermissionDeniedError,
RateLimitError,
+ RateLimitErrorCategory,
+ RateLimitType,
ServiceUnavailableError,
BadGatewayError,
OpenAIError,
diff --git a/litellm/exceptions.py b/litellm/exceptions.py
--- a/litellm/exceptions.py
+++ b/litellm/exceptions.py
@@ -9,13 +9,109 @@
## LiteLLM versions of the OpenAI Exception Types
-from typing import Any, Dict, Optional
+import enum
+from typing import Any, Dict, Optional, Union
import httpx
import openai
from litellm.types.utils import LiteLLMCommonStrings
+
+class RateLimitErrorCategory(str, enum.Enum):
+ """
+ Category of a rate limit error, allowing callers to distinguish where the rate
+ limit originated. Exposed on every :class:`RateLimitError` instance via the
+ ``category`` attribute.
+
+ Use these values to switch on the rate limit source, e.g.::
+
+ try:
+ ...
+ except litellm.RateLimitError as e:
+ if e.category == RateLimitErrorCategory.LITELLM_RATE_LIMIT:
+ ... # litellm's own limiter (key/team/user/model RPM/TPM/budget)
+ elif e.category == RateLimitErrorCategory.VENDOR_RATE_LIMIT:
+ ... # the upstream LLM provider returned 429
+ """
+
+ VENDOR_RATE_LIMIT = "vendor_rate_limit"
+ """The upstream LLM provider returned a rate-limit response (e.g. OpenAI 429)."""
+
+ VENDOR_BATCH_RATE_LIMIT = "vendor_batch_rate_limit"
+ """The upstream LLM provider returned a rate-limit response on a batch endpoint."""
+
+ LITELLM_RATE_LIMIT = "litellm_rate_limit"
+ """LiteLLM's own rate limiter (key/team/user/model RPM/TPM, budget, parallel-requests, etc.) blocked the request."""
+
+ LITELLM_BATCH_RATE_LIMIT = "litellm_batch_rate_limit"
+ """LiteLLM's own batch rate limiter (token/request budget across a batch input file) blocked the request."""
+
+
+class RateLimitType(str, enum.Enum):
+ """
+ The dimension that was exceeded when a rate-limit error fired.
+
+ This is orthogonal to :class:`RateLimitErrorCategory` — *category* tells
+ callers **who** rate-limited the request (the upstream vendor vs. one of
+ litellm's own limiters), while *type* tells them **which limit dimension**
+ was exceeded (an RPM ceiling, a TPM ceiling, a max-parallel-requests
+ ceiling, a budget cap, or a max-iterations cap).
+
+ Surfaced both on every :class:`RateLimitError` instance via the
+ ``rate_limit_type`` attribute and on the structured
+ ``StandardLoggingPayload.error_information.error_rate_limit_type`` field
+ so custom callbacks / metrics consumers can split rate-limit failures by
+ cause without parsing free-text error messages.
+ """
+
+ REQUESTS = "requests"
+ """Requests-per-minute (RPM) or requests-per-window ceiling exceeded."""
+
+ TOKENS = "tokens"
+ """Tokens-per-minute (TPM) or tokens-per-window ceiling exceeded."""
+
+ CONCURRENT_REQUESTS = "concurrent_requests"
+ """``max_parallel_requests`` — too many in-flight requests at once."""
+
+ BUDGET = "budget"
+ """Spend budget cap reached (key, team, user, or per-session)."""
+
+ MAX_ITERATIONS = "max_iterations"
+ """Per-session max-iterations cap reached (agent-style flows)."""
+
+
+_RATE_LIMIT_CATEGORY_VALUES = frozenset(c.value for c in RateLimitErrorCategory)
+_RATE_LIMIT_TYPE_VALUES = frozenset(t.value for t in RateLimitType)
+
+
+def validate_rate_limit_category(value: Any) -> Optional[str]:
+ """Return ``value`` only if it matches a known :class:`RateLimitErrorCategory`.
+
+ Used at duck-typed read sites (StandardLoggingPayload extraction, Prometheus
+ labels) to reject `.category` strings set by unrelated third-party exceptions
+ — otherwise those would leak into custom-callback payloads and Prometheus
+ label cardinality.
+ """
+ if isinstance(value, RateLimitErrorCategory):
+ return value.value
+ if isinstance(value, str) and value in _RATE_LIMIT_CATEGORY_VALUES:
+ return value
+ return None
+
+
+def validate_rate_limit_type(value: Any) -> Optional[str]:
+ """Return ``value`` only if it matches a known :class:`RateLimitType`.
+
+ See :func:`validate_rate_limit_category` for the rationale.
+ """
+ if isinstance(value, RateLimitType):
+ return value.value
+ if isinstance(value, str) and value in _RATE_LIMIT_TYPE_VALUES:
+ return value
+ return None
+
+
_MINIMAL_ERROR_RESPONSE: Optional[httpx.Response] = None
@@ -321,6 +417,18 @@
class RateLimitError(openai.RateLimitError): # type: ignore
+ """
+ Unified rate-limit error.
+
+ Every rate-limit condition surfaced by litellm — whether it originated from
+ an upstream LLM provider, a vendor batch endpoint, or one of litellm's own
+ proxy-side limiters (parallel-requests, dynamic-rate, batch-rate, budget,
+ max-iterations, etc.) — is raised as an instance of this class.
+
+ The :attr:`category` attribute lets callers distinguish the source. See
+ :class:`RateLimitErrorCategory` for the available values.
+ """
+
def __init__(
self,
message,
@@ -330,6 +438,12 @@
litellm_debug_info: Optional[str] = None,
max_retries: Optional[int] = None,
num_retries: Optional[int] = None,
+ category: Union[str, RateLimitErrorCategory] = (
+ RateLimitErrorCategory.VENDOR_RATE_LIMIT
+ ),
+ rate_limit_type: Optional[Union[str, RateLimitType]] = None,
+ headers: Optional[Dict[str, str]] = None,
+ detail: Any = None,
):
self.status_code = 429
self.message = "litellm.RateLimitError: {}".format(message)
@@ -338,9 +452,39 @@
self.litellm_debug_info = litellm_debug_info
self.max_retries = max_retries
self.num_retries = num_retries
+ self.category = (
+ category.value if isinstance(category, RateLimitErrorCategory) else category
+ )
+ # Which dimension was exceeded — request count, token count, parallel
+ # requests, budget, max iterations. None when the source didn't
+ # classify the failure (e.g. legacy vendor 429 with no header hints).
+ self.rate_limit_type: Optional[str] = (
+ rate_limit_type.value
+ if isinstance(rate_limit_type, RateLimitType)
+ else rate_limit_type
+ )
+ # Headers explicitly attached to the error (e.g. retry-after,
+ # rate_limit_type, reset_at). Preserved across the proxy boundary so
+ # clients can react appropriately.
+ #
+ # IMPORTANT: we deliberately do NOT auto-populate self.headers from
+ # response.headers when only `response` is provided. A vendor 429 can
+ # set arbitrary response headers (Set-Cookie, CORS overrides, …); if
+ # those leaked into e.headers and a downstream proxy serializer
+ # forwarded them to the client, a malicious upstream could inject
+ # browser-interpreted headers for the proxy origin. Vendor response
+ # headers stay reachable on `e.response.headers` for callers that
+ # explicitly want them; only the proxy-supplied `headers=` kwarg
+ # makes it onto `self.headers`.
_response_headers = (
getattr(response, "headers", None) if response is not None else None
)
+ self.headers: Optional[Dict[str, str]] = (
+ {k: str(v) for k, v in headers.items()} if headers else None
+ )
+ # Mirrors FastAPI HTTPException.detail so the same instance can be
+ # serialized through both the ProxyException and HTTPException paths.
+ self.detail = detail if detail is not None else self.message
self.response = httpx.Response(
status_code=429,
headers=_response_headers,
@@ -843,11 +987,24 @@
class BudgetExceededError(Exception):
def __init__(
- self, current_cost: float, max_budget: float, message: Optional[str] = None
+ self,
+ current_cost: float,
+ max_budget: float,
+ message: Optional[str] = None,
+ llm_provider: Optional[str] = None,
):
self.current_cost = current_cost
self.max_budget = max_budget
self.status_code = 429
+ self.llm_provider = llm_provider or ""
+ # Surface unified rate-limit fields without joining the RateLimitError
+ # hierarchy so existing `except BudgetExceededError:` handlers keep
+ # working; custom callbacks reading StandardLoggingPayload pick these
+ # up via the same `category` / `rate_limit_type` attributes the rest
+ # of the unified rate-limit error path uses. Stored as plain strings
+ # to match the normalization RateLimitError.__init__ performs.
+ self.category: str = RateLimitErrorCategory.LITELLM_RATE_LIMIT.value
+ self.rate_limit_type: str = RateLimitType.BUDGET.value
message = (
message
or f"Budget has been exceeded! Current cost: {current_cost}, Max budget: {max_budget}"
diff --git a/litellm/integrations/prometheus.py b/litellm/integrations/prometheus.py
--- a/litellm/integrations/prometheus.py
+++ b/litellm/integrations/prometheus.py
@@ -24,6 +24,10 @@
import litellm
from litellm._logging import print_verbose, verbose_logger
+from litellm.exceptions import (
+ validate_rate_limit_category,
+ validate_rate_limit_type,
+)
from litellm.integrations.custom_logger import CustomLogger
from litellm.integrations.prometheus_helpers.bounded_prometheus_series_tracker import (
BoundedPrometheusSeriesTracker,
@@ -78,6 +82,20 @@
# Always initialize label_filters, even for non-premium users
self.label_filters = self._parse_prometheus_config()
+ # Cache resolved label sets per metric. Several entries in
+ # ``PrometheusMetricLabels.get_labels`` read module-level toggles
+ # (e.g. ``litellm.prometheus_emit_stream_label``,
+ # ``litellm.prometheus_emit_rate_limit_labels``) that can be
+ # changed at runtime. Prometheus counters/gauges/histograms are
+ # created with a *fixed* ``labelnames`` set; if a runtime call
+ # to ``get_labels_for_metric`` returned a different set, the
+ # subsequent ``counter.labels(**_labels)`` would raise a
+ # ``ValueError`` from the prometheus client. Snapshotting at
+ # logger init time pins the label set for the lifetime of the
+ # logger so toggling these flags only takes effect after a
+ # restart, keeping init-time and runtime label sets in sync.
+ self._cached_metric_labels: Dict[str, List[str]] = {}
+
_custom_buckets = litellm.prometheus_latency_buckets
self.latency_buckets = (
tuple(_custom_buckets)
@@ -1033,13 +1051,27 @@
self, metric_name: DEFINED_PROMETHEUS_METRICS
) -> List[str]:
"""
- Get the labels for a metric, filtered if configured
+ Get the labels for a metric, filtered if configured.
+
+ The result is cached on the instance so the label set used to
+ construct each Prometheus metric at ``__init__`` time stays in lock
+ step with the label set passed to ``counter.labels(...)`` at
+ runtime, even if the underlying module-level toggles consulted by
+ :meth:`PrometheusMetricLabels.get_labels` (e.g.
+ ``litellm.prometheus_emit_rate_limit_labels``,
+ ``litellm.prometheus_emit_stream_label``) are flipped after the
+ logger has been created.
"""
+ cached = self._cached_metric_labels.get(metric_name)
+ if cached is not None:
+ return cached
+
# Get default labels for this metric from PrometheusMetricLabels
default_labels = PrometheusMetricLabels.get_labels(metric_name)
# If no label filtering is configured for this metric, use default labels
if metric_name not in self.label_filters:
+ self._cached_metric_labels[metric_name] = default_labels
return default_labels
# Get configured labels for this metric
@@ -1050,6 +1082,7 @@
label for label in default_labels if label in configured_labels
]
+ self._cached_metric_labels[metric_name] = filtered_labels
return filtered_labels
def _track_end_user_metric_series(
@@ -2029,14 +2062,8 @@
Proxy level tracking - failed client side requests
- labelnames=[
- "end_user",
- "hashed_api_key",
- "api_key_alias",
- REQUESTED_MODEL,
- "team",
- "team_alias",
- ] + EXCEPTION_LABELS,
+ See :attr:`PrometheusMetricLabels.litellm_proxy_failed_requests_metric`
+ for the authoritative list of labels emitted on this metric.
"""
from litellm.litellm_core_utils.litellm_logging import (
StandardLoggingPayloadSetup,
@@ -2059,6 +2086,9 @@
model_id = _metadata.get("model_info", {}).get("id") or request_data.get(
"model_info", {}
).get("id")
+ rate_limit_category, rate_limit_type = self._extract_rate_limit_labels(
+ original_exception
+ )
enum_values = UserAPIKeyLabelValues(
end_user=user_api_key_dict.end_user_id,
user=user_api_key_dict.user_id,
@@ -2073,6 +2103,8 @@
status_code=str(status_code),
exception_status=str(status_code),
exception_class=self._get_exception_class_name(original_exception),
+ rate_limit_category=rate_limit_category,
+ rate_limit_type=rate_limit_type,
tags=_tags,
route=user_api_key_dict.request_route,
client_ip=_metadata.get("requester_ip_address"),
@@ -2843,6 +2875,50 @@
@staticmethod
def _get_exception_class_name(exception: Exception) -> str:
+ # Back-compat: ProxyRateLimitError multi-inherits from
+ # fastapi.HTTPException + litellm.RateLimitError. Before the unified
+ # rate-limit error class landed, every proxy-side 429 surfaced on this
+ # label as the literal string "HTTPException", and existing dashboards
+ # / alerts (e.g. trho's HubSpot dashboard) key off that exact value.
+ # Renaming the class would silently break those queries, so we keep
+ # emitting "HTTPException" here. Callers that need to distinguish
+ # vendor vs. litellm rate limits should use the new
+ # ``rate_limit_category`` / ``rate_limit_type`` labels instead.
+ #
+ # The import is intentionally lazy + ImportError-tolerant: this method
+ # is also called from router-side fallback events
+ # (``log_success_fallback_event`` / ``log_failure_fallback_event``)
+ # which can run in non-proxy installs where ``fastapi`` (a transitive
+ # dep of ``proxy_rate_limit_error``) is not installed.
+ try:
+ from litellm.proxy.common_utils.proxy_rate_limit_error import (
+ ProxyRateLimitError,
+ )
+ except ImportError:
+ ProxyRateLimitError = None # type: ignore[assignment,misc]
+
+ if ProxyRateLimitError is not None and isinstance(
+ exception, ProxyRateLimitError
+ ):
+ return "HTTPException"
+
+ # Same back-compat reasoning for ``BudgetExceededError``: the unified
+ # rate-limit error work attached ``.llm_provider`` to budget errors
+ # too (so callbacks reading ``StandardLoggingPayload`` get provider
+ # attribution). Without this short-circuit, the provider prefix below
+ # would silently flip the label from "BudgetExceededError" to e.g.
+ # "Openai.BudgetExceededError" and break dashboards keyed on the
+ # original value.
+ try:
+ from litellm.exceptions import BudgetExceededError
+ except ImportError:
+ BudgetExceededError = None # type: ignore[assignment,misc]
+
+ if BudgetExceededError is not None and isinstance(
+ exception, BudgetExceededError
+ ):
+ return "BudgetExceededError"
+
exception_class_name = ""
if hasattr(exception, "llm_provider"):
exception_class_name = getattr(exception, "llm_provider") or ""
@@ -2857,6 +2933,27 @@
exception_class_name += exception.__class__.__name__
return exception_class_name
+ @staticmethod
+ def _extract_rate_limit_labels(
+ exception: Optional[Exception],
+ ) -> Tuple[Optional[str], Optional[str]]:
+ """
+ Pull the unified ``category`` / ``rate_limit_type`` fields off any
+ exception that declares them (``litellm.RateLimitError`` and bare-
+ Exception subclasses like ``BudgetExceededError``).
+
+ Values are validated against the :class:`RateLimitErrorCategory` /
+ :class:`RateLimitType` enums so unrelated third-party exceptions that
+ happen to declare ``.category`` / ``.rate_limit_type`` string attributes
+ can't leak garbage into Prometheus label cardinality.
+ """
+ if exception is None:
+ return None, None
+ return (
+ validate_rate_limit_category(getattr(exception, "category", None)),
+ validate_rate_limit_type(getattr(exception, "rate_limit_type", None)),
+ )
+
async def log_success_fallback_event(
self, original_model_group: str, kwargs: dict, original_exception: Exception
):
diff --git a/litellm/litellm_core_utils/litellm_logging.py b/litellm/litellm_core_utils/litellm_logging.py
--- a/litellm/litellm_core_utils/litellm_logging.py
+++ b/litellm/litellm_core_utils/litellm_logging.py
@@ -37,6 +37,10 @@
turn_off_message_logging,
)
from litellm._logging import _is_debugging_on, _redact_string, verbose_logger
+from litellm.exceptions import (
+ validate_rate_limit_category,
+ validate_rate_limit_type,
+)
from litellm._uuid import uuid
from litellm.batches.batch_utils import _handle_completed_batch
from litellm.caching.caching import DualCache, InMemoryCache
@@ -5307,12 +5311,27 @@
else str(original_exception)
)
+ # Duck-typed read so bare-Exception subclasses like
+ # `litellm.BudgetExceededError` can participate without joining the
+ # RateLimitError hierarchy (which would break `except BudgetExceededError`).
+ # Validated against the enum value sets so a third-party exception that
+ # happens to declare a `.category` or `.rate_limit_type` string attribute
+ # can't leak garbage into the payload or Prometheus label cardinality.
+ rate_limit_category = validate_rate_limit_category(
+ getattr(original_exception, "category", None)
+ )
+ rate_limit_type = validate_rate_limit_type(
+ getattr(original_exception, "rate_limit_type", None)
+ )
+
return StandardLoggingPayloadErrorInformation(
error_code=error_status,
error_class=error_class,
llm_provider=_llm_provider_in_exception,
traceback=traceback_info,
error_message=error_message if original_exception else "",
+ error_rate_limit_category=rate_limit_category,
+ error_rate_limit_type=rate_limit_type,
)
@staticmethod
diff --git a/litellm/proxy/auth/auth_exception_handler.py b/litellm/proxy/auth/auth_exception_handler.py
--- a/litellm/proxy/auth/auth_exception_handler.py
+++ b/litellm/proxy/auth/auth_exception_handler.py
@@ -106,6 +106,21 @@
api_key=api_key,
request_route=route,
)
+
+ # Budget checks live in tenant-scoped helpers (key / team / org / tag)
+ # that don't see the request model, so the BudgetExceededError they
+ # raise carries `llm_provider=""`. Resolve it here off `request_data`
+ # so custom-callback consumers reading StandardLoggingPayload get
+ # the same `llm_provider` attribution as for RPM/TPM 429s.
+ if isinstance(e, litellm.BudgetExceededError) and not e.llm_provider:
+ from litellm.proxy.hooks.rate_limiter_utils import (
+ resolve_llm_provider_for_rate_limit,
+ )
+
+ _, e.llm_provider = resolve_llm_provider_for_rate_limit(
+ request_data.get("model")
+ )
+
# Allow callbacks to transform the error response
transformed_exception = await proxy_logging_obj.post_call_failure_hook(
request_data=request_data,
diff --git a/litellm/proxy/common_utils/proxy_rate_limit_error.py b/litellm/proxy/common_utils/proxy_rate_limit_error.py
new file mode 100644
--- /dev/null
+++ b/litellm/proxy/common_utils/proxy_rate_limit_error.py
@@ -1,0 +1,189 @@
+"""
+ProxyRateLimitError — a unified rate-limit exception used by litellm's
+proxy-side hooks.
+
+Background
+----------
+LiteLLM previously surfaced rate-limit conditions through *several* unrelated
+exception types:
+
+* :class:`litellm.exceptions.RateLimitError` — raised by exception mapping when
+ an upstream LLM provider returns 429.
+* :class:`fastapi.HTTPException` (status 429) — raised directly by proxy hooks
+ such as ``parallel_request_limiter``, ``dynamic_rate_limiter``,
+ ``batch_rate_limiter``, ``max_budget_limiter``, ``max_iterations_limiter``,
+ etc.
+* :class:`litellm.llms.base_llm.chat.transformation.BaseLLMException` (status
+ 429) — raised by some provider transports.
+
+This made it impossible for downstream code (and end users) to express
+"is this a rate limit?" with a single ``except`` clause, and impossible to
+distinguish *where* the rate limit originated (vendor vs. litellm, batch vs.
+chat) without ad-hoc string-matching on the message.
+
+This module provides a single proxy-side error class that:
+
+1. Is a subclass of :class:`litellm.exceptions.RateLimitError`, so user code
+ that catches ``RateLimitError`` works for *every* rate-limit source.
+2. Is also a subclass of :class:`fastapi.HTTPException`, so existing proxy
+ plumbing (``isinstance(e, HTTPException)`` branches in route handlers and
+ FastAPI's own dispatcher) continues to behave the same way and the
+ ``retry-after`` / ``rate_limit_type`` / ``reset_at`` headers are preserved
+ on the wire.
+3. Carries a :attr:`category` field (one of
+ :class:`litellm.exceptions.RateLimitErrorCategory`) so callers can switch on
+ the rate limit source.
+"""
+
+import json
+from typing import Any, Dict, Mapping, Optional, Union
+
+from fastapi import HTTPException
+
+from litellm.exceptions import RateLimitError, RateLimitErrorCategory, RateLimitType
+
+
+def map_v3_rate_limit_type(
+ v3_value: Optional[str],
+) -> Optional[RateLimitType]:
+ """
+ Map the v3 rate limiter's internal `status["rate_limit_type"]` strings
+ onto the public :class:`RateLimitType` enum.
+
+ The v3 limiter uses the literal values ``"requests"``, ``"tokens"``, and
+ ``"max_parallel_requests"``. We collapse the last one onto
+ :attr:`RateLimitType.CONCURRENT_REQUESTS` because that's the public name
+ documented for users and dashboards. Unrecognized values return ``None``
+ so the field stays absent rather than carrying garbage downstream.
+ """
+ if v3_value == "tokens":
+ return RateLimitType.TOKENS
+ if v3_value == "max_parallel_requests":
+ return RateLimitType.CONCURRENT_REQUESTS
+ if v3_value == "requests":
+ return RateLimitType.REQUESTS
+ return None
+
+
+def _coerce_message(detail: Any) -> str:
+ """Best-effort, JSON-friendly stringification of an HTTPException-style detail."""
+ if detail is None:
+ return ""
+ if isinstance(detail, str):
+ return detail
+ if isinstance(detail, Mapping):
+ for key in ("error", "message"):
+ if isinstance(detail.get(key), str):
+ return detail[key]
+ inner = detail.get(key)
+ if isinstance(inner, Mapping) and isinstance(inner.get("message"), str):
+ return inner["message"]
+ try:
+ return json.dumps(detail)
+ except (TypeError, ValueError):
+ return str(detail)
+ return str(detail)
+
+
+# NOTE: mypy emits two `[misc]` errors on the class line below because the
+# bases declare overlapping attributes with related-but-not-identical
+# annotations:
+# * `status_code` is `int` on starlette HTTPException but `Literal[429]` on
+# openai.RateLimitError (every openai status-error subclass narrows it
+# this way and silences pyright with the same convention).
+# * `headers` is `Mapping[str, str] | None` on HTTPException; we narrow it
+# to `Optional[Dict[str, str]]` on RateLimitError because we always carry
+# a stringified dict.
+# Both narrowings are intentional and handled at construction time — every
+# instance always has status_code == 429 and a Dict-typed headers — so we
+# silence the ATTR-overlap check rather than relax the annotations.
+class ProxyRateLimitError(HTTPException, RateLimitError): # type: ignore[misc]
+ """
+ A 429 raised by litellm's proxy-side rate limiting hooks.
+
+ This class deliberately inherits from BOTH
+ :class:`litellm.exceptions.RateLimitError` and :class:`fastapi.HTTPException`
+ so the same instance can flow through:
+
+ * ``except RateLimitError`` (user / SDK code that wants a category-aware
+ handler), and
+ * ``isinstance(e, HTTPException)`` (FastAPI / proxy_server.py route
+ handlers that need to forward ``status_code``, ``detail`` and
+ ``headers`` back to the client).
+
+ Downstream code should prefer this class over
+ ``raise HTTPException(status_code=429, ...)`` for litellm-internal rate
+ limits.
+
+ Parameters
+ ----------
+ detail:
+ The structured error payload. Forwarded as ``HTTPException.detail`` so
+ FastAPI's default exception handler will serialize it verbatim.
+ headers:
+ Optional response headers (e.g. ``retry-after``). Values are stringified
+ to satisfy FastAPI's typing.
+ category:
+ One of :class:`RateLimitErrorCategory`. Defaults to
+ ``LITELLM_RATE_LIMIT`` since this class is only used by litellm's own
+ proxy-side limiters; pass ``LITELLM_BATCH_RATE_LIMIT`` for the batch
+ limiter, etc.
+ model / llm_provider:
+ Optional context, propagated to the inherited ``RateLimitError`` for
+ compatibility with logging / standard payload extraction.
+ """
+
+ def __init__(
+ self,
+ detail: Any,
+ headers: Optional[Mapping[str, Any]] = None,
+ category: Union[
+ str, RateLimitErrorCategory
+ ] = RateLimitErrorCategory.LITELLM_RATE_LIMIT,
+ rate_limit_type: Optional[Union[str, RateLimitType]] = None,
+ model: Optional[str] = None,
+ llm_provider: Optional[str] = "litellm_proxy",
+ ):
+ # Normalize None → safe defaults so callers (and the resolver helper
+ # in `rate_limiter_utils`) can pass `None` without producing an
+ # instance whose `.llm_provider` attribute is `None` — that would
+ # break Prometheus' `_get_exception_class_name` (it calls
+ # `.capitalize()` on the provider string).
+ model = model or ""
+ llm_provider = llm_provider or "litellm_proxy"
+ message = _coerce_message(detail)
+ stringified_headers: Optional[Dict[str, str]] = (
+ {k: str(v) for k, v in headers.items()} if headers else None
+ )
+
+ # Initialize the FastAPI HTTPException portion first so its attributes
+ # (status_code, detail, headers) are already on the instance before
+ # RateLimitError.__init__ runs and possibly overrides them.
+ HTTPException.__init__(
+ self,
+ status_code=429,
+ detail=detail,
+ headers=stringified_headers,
+ )
+
+ # Now initialize the litellm RateLimitError portion. We deliberately
+ # pass the structured detail through so RateLimitError preserves it as
+ # its `.detail` attribute too — keeping both sides of the MRO
+ # consistent.
+ RateLimitError.__init__(
+ self,
+ message=message,
+ llm_provider=llm_provider,
+ model=model,
+ category=category,
+ rate_limit_type=rate_limit_type,
+ headers=stringified_headers,
+ detail=detail,
+ )
+ # RateLimitError.__init__ overwrites self.headers with its own copy and
+ # leaves self.status_code at 429 — restore the HTTPException-style
+ # headers value so downstream code that pulls headers off the
+ # instance gets back exactly what the limiter passed in.
+ self.headers = stringified_headers
+ self.detail = detail
+ self.status_code = 429
diff --git a/litellm/proxy/hooks/batch_rate_limiter.py b/litellm/proxy/hooks/batch_rate_limiter.py
--- a/litellm/proxy/hooks/batch_rate_limiter.py
+++ b/litellm/proxy/hooks/batch_rate_limiter.py
@@ -17,7 +17,17 @@
- async_log_success_event() fires on GET /v1/batches/{id} (batch completion)
"""
-from typing import TYPE_CHECKING, Any, Dict, List, Literal, Optional, Tuple, Union
+from typing import (
+ TYPE_CHECKING,
+ Any,
+ Dict,
+ List,
+ Literal,
+ NoReturn,
+ Optional,
+ Tuple,
+ Union,
+)
from fastapi import HTTPException
from pydantic import BaseModel
@@ -30,8 +40,14 @@
_get_file_content_as_dictionary,
_get_models_from_batch_input_file_content,
)
+from litellm.exceptions import RateLimitErrorCategory
from litellm.integrations.custom_logger import CustomLogger
from litellm.proxy._types import SpecialModelNames, UserAPIKeyAuth
+from litellm.proxy.common_utils.proxy_rate_limit_error import (
+ ProxyRateLimitError,
+ map_v3_rate_limit_type,
+)
+from litellm.proxy.hooks.rate_limiter_utils import resolve_llm_provider_for_rate_limit
if TYPE_CHECKING:
from opentelemetry.trace import Span as _Span
@@ -375,8 +391,9 @@
descriptors: List["RateLimitDescriptor"],
batch_usage: BatchFileUsage,
limit_type: str,
- ) -> None:
- """Raise HTTPException for rate limit exceeded."""
+ requested_model: Optional[str] = None,
+ ) -> NoReturn:
+ """Raise :class:`ProxyRateLimitError` (a 429) for batch rate limit exceeded."""
from datetime import datetime
# Find the descriptor for this status
@@ -419,14 +436,20 @@
f"Limit resets at: {reset_time_formatted}"
)
- raise HTTPException(
- status_code=429,
+ resolved_model, llm_provider = resolve_llm_provider_for_rate_limit(
+ requested_model
+ )
+ raise ProxyRateLimitError(
detail=detail,
headers={
"retry-after": str(window_size),
"rate_limit_type": limit_type,
"reset_at": reset_time_formatted,
},
+ category=RateLimitErrorCategory.LITELLM_BATCH_RATE_LIMIT,
+ rate_limit_type=map_v3_rate_limit_type(limit_type),
+ model=resolved_model,
+ llm_provider=llm_provider,
)
async def _check_and_increment_batch_counters(
@@ -470,6 +493,7 @@
)
if rate_limit_response["overall_code"] == "OVER_LIMIT":
+ requested_model = data.get("model") if data else None
for status in rate_limit_response["statuses"]:
if status["code"] == "OVER_LIMIT":
self._raise_rate_limit_error(
@@ -477,6 +501,7 @@
descriptors,
batch_usage,
status["rate_limit_type"],
+ requested_model=requested_model,
)
async def count_input_file_usage(
diff --git a/litellm/proxy/hooks/dynamic_rate_limiter.py b/litellm/proxy/hooks/dynamic_rate_limiter.py
--- a/litellm/proxy/hooks/dynamic_rate_limiter.py
+++ b/litellm/proxy/hooks/dynamic_rate_limiter.py
@@ -6,21 +6,23 @@
import os
from typing import List, Optional, Tuple, Union
-from fastapi import HTTPException
-
import litellm
from litellm import ModelResponse, Router
from litellm._logging import verbose_proxy_logger
from litellm.caching.caching import DualCache
from litellm.integrations.custom_logger import CustomLogger
+from litellm.exceptions import RateLimitType
from litellm.proxy._types import UserAPIKeyAuth
+from litellm.proxy.common_utils.proxy_rate_limit_error import ProxyRateLimitError
+from litellm.proxy.hooks.rate_limiter_utils import (
+ convert_priority_to_percent,
... diff truncated: showing 800 of 4919 linesYou can send follow-ups to the cloud agent here.
Reviewed by Cursor Bugbot for commit e04e4c2. Configure here.
| RateLimitType.TOKENS | ||
| if limit_type == "tokens" | ||
| else RateLimitType.REQUESTS | ||
| ), |
Contributor
There was a problem hiding this comment.
Batch mislabels parallel limit type
Medium Severity
When batch rate limiting trips on a max_parallel_requests descriptor, the v3 status reports rate_limit_type as max_parallel_requests, but _raise_rate_limit_error maps any non-tokens value to requests. Structured logging and Prometheus then show requests instead of concurrent_requests for the same failure.
Reviewed by Cursor Bugbot for commit e04e4c2. Configure here.
Contributor
Author
…ion_class_name Read the legacy exception_class label from a prometheus_exception_class_name marker on ProxyRateLimitError instead of importing the proxy module, keeping the integrations layer free of a transitive fastapi dependency.
Contributor
Author
…itellm_standardize_rate_limit_errors-5fb4 # Conflicts: # litellm/proxy/hooks/batch_rate_limiter.py # litellm/proxy/hooks/dynamic_rate_limiter.py # litellm/proxy/hooks/dynamic_rate_limiter_v3.py # litellm/proxy/hooks/max_budget_limiter.py # litellm/proxy/hooks/max_budget_per_session_limiter.py # litellm/proxy/hooks/max_iterations_limiter.py # litellm/proxy/hooks/parallel_request_limiter.py # litellm/proxy/hooks/parallel_request_limiter_v3.py # litellm/proxy/hooks/rate_limiter_utils.py # tests/test_litellm/proxy/hooks/test_proxy_rate_limit_provider_field.py
Contributor
Author
Proof of fix (AFTER): standardized rate-limit 429sVerified on a real OpenAI path (
HTTP responses (curl)StandardLoggingPayload.error_information (custom callback)// RPM 429
{"error_code": "429", "error_class": "ProxyRateLimitError", "llm_provider": "openai",
"error_rate_limit_category": "litellm_rate_limit", "error_rate_limit_type": "requests"}
// Budget 429
{"error_code": "429", "error_class": "BudgetExceededError", "llm_provider": "openai",
"error_rate_limit_category": "litellm_rate_limit", "error_rate_limit_type": "budget"}The RPM error is raised from Prometheus litellm_proxy_failed_requests_metricEnabled with What the OTel v2 traces showThese traces come from the v2 OpenTelemetry integration ( The For the budget 429 the Screenshots (otel-desktop-viewer)Trace list, the two successful calls are 7-span traces (server, auth, DB, the LLM call, spend write) and the two gate-rejected 429s are 4-span traces with no upstream span; the budget 429 carries the 
RPM 429 detail, the FastAPI server span at 
|
…itellm_standardize_rate_limit_errors-5fb4 # Conflicts: # litellm/proxy/auth/auth_exception_handler.py
The ProxyRateLimitError docstring flows into the proxy OpenAPI spec's 429 response description, so the generated dashboard types were out of sync. Regenerated via npm run gen:api (Check UI API Types Sync).
yassin-berriai
approved these changes
Jun 7, 2026
4 tasks
michaelxer
pushed a commit
to michaelxer/litellm
that referenced
this pull request
Jun 17, 2026
…odel, and llm_provider fields (BerriAI#27687) * feat(exceptions): add RateLimitErrorCategory + headers/detail fields on RateLimitError LiteLLM previously surfaced rate-limit conditions through several unrelated error classes (RateLimitError, FastAPI HTTPException(429), BaseLLMException). This commit adds the data model needed to consolidate them under a single class: * RateLimitErrorCategory enum exposing four categorical values (vendor_rate_limit, vendor_batch_rate_limit, litellm_rate_limit, litellm_batch_rate_limit) so callers can switch on the rate-limit source. * New optional fields on RateLimitError: - category (defaults to vendor_rate_limit, preserving today's behavior for every existing call site in exception_mapping_utils); - headers (preserves retry-after / rate_limit_type / reset_at across the proxy boundary instead of dropping them on the floor); - detail (mirrors FastAPI HTTPException.detail so the same instance can be serialized through both paths). litellm.RateLimitErrorCategory is re-exported at the package root to match the existing exception-export pattern. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(proxy): add ProxyRateLimitError unifying RateLimitError + HTTPException Adds a single proxy-side error class that subclasses BOTH litellm.exceptions.RateLimitError AND fastapi.HTTPException via cooperative multiple inheritance. Why both bases: * Subclassing RateLimitError lets user code catch every rate-limit source with one 'except RateLimitError' and switch on the new .category field. * Subclassing HTTPException keeps every existing FastAPI plumbing path (the isinstance(e, HTTPException) branches in proxy_server.py route handlers, FastAPI's own dispatcher, and tests asserting pytest.raises(HTTPException)) working without modification, and preserves retry-after / rate_limit_type / reset_at headers on the wire. The class declaration order is (HTTPException, RateLimitError) so the MRO puts HTTPException's no-super-call __init__ ahead of openai's cooperative __init__ chain — preventing openai.APIError.super().__init__(message) from landing in HTTPException.__init__(status_code=message). LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from budget + iteration limiters Replaces three bare HTTPException(status_code=429, ...) call sites with ProxyRateLimitError, which is both a RateLimitError (catchable by category) and an HTTPException (preserves existing FastAPI serialization). Drops the now-unused HTTPException import in the iteration / per-session limiters. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from parallel-request limiters Replaces HTTPException(status_code=429, ...) call sites in the v1 and v3 parallel-request limiters (key/team/user/model/customer rate limits) with ProxyRateLimitError. Updates the raise_rate_limit_error helper's return type annotation accordingly. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from dynamic rate limiters Replaces HTTPException(status_code=429, ...) call sites in the v1 and v3 dynamic rate limiters (project-level TPM/RPM allocation, model-saturation checks, priority-based limits, fail-closed guards) with ProxyRateLimitError. The v3 limiter still imports HTTPException for an unrelated bare 'except HTTPException:' branch. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from batch rate limiter Replaces HTTPException(status_code=429, ...) in batch_rate_limiter._raise_rate_limit_error with ProxyRateLimitError tagged as RateLimitErrorCategory.LITELLM_BATCH_RATE_LIMIT so users can distinguish batch-level throttling (which counts requests/tokens across an uploaded batch input file before submission) from the generic key/team/user RPM/TPM limiter. The HTTPException import is retained because the same module raises HTTPException for unrelated 403/IO error paths. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): pin down unified rate-limit error contract Adds a dedicated test module covering the new RateLimitErrorCategory enum, RateLimitError.category default + override behavior, ProxyRateLimitError's dual nature (RateLimitError + HTTPException), and a parametrized regression guard that asserts every proxy hook module imports the unified class. The regression guard catches the failure mode the refactor is designed to prevent: someone re-introducing a bare HTTPException(status_code=429, ...) in one of the hook modules instead of going through ProxyRateLimitError. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(logging): expose rate-limit category via StandardLoggingPayload Adds an optional 'error_rate_limit_category' field to StandardLoggingPayloadErrorInformation, populated from the unified RateLimitError.category attribute (introduced in the previous commits on this branch). Why: the .category attribute is reachable off the raw exception today via getattr(e, 'category', None), but the structured contract that downstream custom callbacks / loggers / spend log writers consume is the StandardLoggingPayload. Without this field, a user building custom rate-limit metrics on top of callback data has to special-case the raw exception object — which defeats the purpose of the StandardLoggingPayload abstraction. The field is None for non-rate-limit exceptions (so consumers can read it unconditionally without isinstance checks) and is one of the RateLimitErrorCategory string values otherwise. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): assert StandardLoggingPayload carries the category Five tests covering: vendor default, explicit litellm_rate_limit and litellm_batch_rate_limit values, None for non-rate-limit exceptions, and None when no exception is provided. Pins down the contract that custom callbacks can read 'error_information.error_rate_limit_category' off the StandardLoggingPayload to drive custom rate-limit metrics without ever reaching for the raw exception. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(types): silence mypy [misc] on intentional dual-base attr overlap mypy emits two [misc] errors on the ProxyRateLimitError class line because its two bases declare overlapping attributes with related-but-not-identical annotations: * status_code: int on starlette HTTPException vs. Literal[429] on openai's RateLimitError (every openai status-error subclass narrows it the same way and silences pyright with the same convention). * headers: Mapping[str, str] | None on HTTPException vs. our Optional[ Dict[str, str]] (the proxy hooks always carry a stringified dict). Both narrowings are intentional and enforced at construction time. Add a type: ignore[misc] with an inline explanation rather than relax the annotations on the parent or change the wire-format guarantees. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): add direct hook-invocation tests to lift patch coverage Adds six end-to-end tests that drive each refactored hook past its limit and assert the unified ProxyRateLimitError is raised with the correct category and dual-base shape. Complements the import-shape-only parametrized guard above by actually executing the new 'raise ProxyRateLimitError(...)' lines so codecov's patch coverage sees them as hit. Hooks covered (one test each): * parallel_request_limiter v1 — direct call to raise_rate_limit_error() * parallel_request_limiter v3 — direct call to _handle_rate_limit_error with a fabricated OVER_LIMIT response * max_iterations_limiter — full async_pre_call_hook with mocked agent registry, second call exceeds budget=1 * max_budget_limiter — async_pre_call_hook with mocked get_current_spend * dynamic_rate_limiter v1 — async_pre_call_hook with mocked check_available_usage forcing available_tpm == 0 * batch_rate_limiter — direct _raise_rate_limit_error call, asserts category is the batch-specific LITELLM_BATCH_RATE_LIMIT (not the generic LITELLM_RATE_LIMIT) LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix: guard rate_limit_category extraction with isinstance check * test(rate-limit): cover remaining hook raise sites for codecov Adds five more direct hook-invocation tests so every PR-touched line in the proxy hooks is exercised by tests in tests/test_litellm/, which codecov measures: * parallel_request_limiter v1 — check_key_in_limits inline raise (the second raise site, separate from the raise_rate_limit_error helper covered earlier) * dynamic_rate_limiter v1 — RPM raise branch (TPM branch was already covered) * dynamic_rate_limiter v3 — parametrized over all three raise sites: model_saturation_check, priority_model, and the fail-closed fallback for an unrecognized descriptor_key * max_budget_per_session_limiter — full async_pre_call_hook with a mocked agent registry and over-budget cached spend All 42 tests in test_rate_limit_error_unification.py now pass and together exercise every changed import + raise line across the eight refactored proxy hooks. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix: use computed error_message in ProxyRateLimitError detail * fix(parallel-request-limiter): drop None from detail; annotate raise_rate_limit_error as NoReturn The v1 ' raise_rate_limit_error' helper built an unused 'error_message' variable and then assembled the actual ' detail' via an f-string that interpolated 'additional_details' verbatim — producing 'Max parallel request limit reached None' when invoked without arguments (flagged by code review). Fix the helper to: - use the constructed 'error_message' as the detail - annotate the helper as NoReturn since it always raises - drop the redundant 'raise'/'return' at the two call sites Add two regression tests covering both the with- and without- additional_details paths. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): drop literal 'None' from raise_rate_limit_error detail The v1 parallel_request_limiter's raise_rate_limit_error helper has a long-standing bug: it computes a None-guarded 'error_message' string but then ignores it and emits an f-string that interpolates the raw 'additional_details' arg. Callers that pass no argument get 'Max parallel request limit reached None' as the user-facing detail. This commit: * wires error_message into the detail kwarg so the None-guard actually applies and operators see a clean message; * changes the return-type annotation from ProxyRateLimitError to NoReturn (the function always raises) so type-checkers know callers after this invocation are unreachable. Greptile P1 + P2 review feedback on PR BerriAI#27687. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(types): demote TypedDict floating string to a # comment A string literal placed after a field declaration in a TypedDict body is not a per-field docstring — it's an orphaned string expression Python discards. Tools like mypy / pyright that inspect TypedDict fields won't surface that text either. Move the documentation for error_rate_limit_category to a real comment so the intent is visible to readers and type-checker tooling without the misleading docstring framing. Greptile P2 review feedback on PR BerriAI#27687. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * security(exceptions): do not auto-copy vendor response headers to e.headers A vendor 429 response can set arbitrary headers (Set-Cookie, CORS overrides, …). Previously, when RateLimitError was constructed with only a 'response=' (no explicit 'headers=' kwarg), self.headers fell back to a copy of response.headers. If a downstream proxy serializer ever forwarded e.headers to the client, a malicious upstream could inject browser-interpreted headers for the proxy origin. Drop the fallback. Only headers passed explicitly via the headers= kwarg make it onto self.headers (proxy hooks pass retry-after etc. — they control what's surfaced). Vendor response headers stay reachable on e.response.headers for callers that explicitly want them. Today's proxy_server.py route handlers don't actually forward e.headers on the wire (they construct ProxyException without passing headers), so no current behavior changes — this is a defensive narrowing so the fallback can never be turned into a vector when someone wires e.headers through later. Veria-AI security review feedback on PR BerriAI#27687. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): regression guards for review-pass fixes Pins down the three review-pass fixes: * test_parallel_request_limiter_v1_helper_no_additional_details — calls raise_rate_limit_error() with no args and asserts the detail does NOT contain the literal string 'None'. Pre-fix, callers got 'Max parallel request limit reached None'. * test_rate_limit_error_does_not_auto_copy_response_headers — passes a vendor httpx.Response with a Set-Cookie header to RateLimitError WITHOUT an explicit headers= kwarg, asserts self.headers stays None (no leak), then re-checks that an explicit headers= kwarg DOES populate self.headers. Vendor headers remain reachable on e.response.headers for callers that explicitly want them. * The existing v1-helper test now also asserts the additional_details string makes it through to the detail. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(rate-limit): add orthogonal RateLimitType (requests/tokens/concurrent_requests/budget/max_iterations) trho's last ask in the LIT-2968 thread: distinguish rate-limit failures by the dimension that was exceeded, not just by who rate-limited (vendor vs. litellm). Adds: - RateLimitType str-enum exposed at `litellm.RateLimitType` with values requests / tokens / concurrent_requests / budget / max_iterations. - `rate_limit_type` kwarg on litellm.RateLimitError + ProxyRateLimitError; None default so existing callers (vendor-429 path in exception_mapping_utils) remain a no-op. - StandardLoggingPayloadErrorInformation.error_rate_limit_type so custom callbacks can split rate-limit failures by cause without parsing free-text error messages. Mirror to error_rate_limit_category extraction in get_error_information(); single isinstance(RateLimitError) check covers both. - map_v3_rate_limit_type() helper to collapse the v3 limiter's internal labels ("requests", "tokens", "max_parallel_requests") onto the public enum so the v3 limiter and dynamic_rate_limiter_v3 share one mapping. Defensive None on unknown values rather than silently picking a wrong dimension. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(proxy/hooks): wire rate_limit_type onto every limiter raise site Each refactored proxy hook now populates rate_limit_type with the dimension that actually tripped the limit, so downstream consumers (custom callbacks, prometheus exporters via the StandardLoggingPayload) can split key/team/user rate-limit failures by cause: - parallel_request_limiter (v1): detect dimension from current vs. limit in the post-cache branch (concurrent_requests > tokens > requests, matches the boolean condition order). Base case (current is None, one limit set to 0) picks the most-specific zero. raise_rate_limit_error() helper accepts an explicit rate_limit_type kwarg with CONCURRENT_REQUESTS default (matches every existing internal call site, including the global-limit branch). - parallel_request_limiter (v3): forward status["rate_limit_type"] through map_v3_rate_limit_type() so "max_parallel_requests" → CONCURRENT_REQUESTS for the public field while the raw v3 jargon stays on the HTTP header for wire-format backward compat. - dynamic_rate_limiter (v1): TPM-zero → TOKENS, RPM-zero → REQUESTS. Pass data["model"] through so callbacks see the model that hit the limit (addresses the secondary "provider missing" complaint in the original Slack thread, partially — the model is what dashboards typically split on). - dynamic_rate_limiter (v3): forward status["rate_limit_type"] via map_v3_rate_limit_type() at every raise site (model_saturation_check, priority_model, fail-closed unknown-descriptor guard). Also pass model. - batch_rate_limiter: limit_type is hard-typed "requests"|"tokens" — map directly without going through the helper's None branch. - max_budget_limiter, max_budget_per_session_limiter: BUDGET. - max_iterations_limiter: MAX_ITERATIONS. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): cover RateLimitType enum, hook wiring, and StandardLoggingPayload propagation 27 new tests across five new test classes: - TestRateLimitType: enum exposed at litellm.RateLimitType, all five values defined, RateLimitError default is None (vendor 429 path makes no claim about which dimension), accepts both string and enum forms with str-coercion guarantee for downstream JSON serializers. - TestProxyRateLimitErrorType: ProxyRateLimitError default is None, accepts string or enum, doesn't break existing callers that pass nothing. - TestMapV3RateLimitType: pins each v3-internal → public-enum mapping (tokens, requests, max_parallel_requests → concurrent_requests, unknown → None) so a future v3 refactor can't silently swap dimensions. - TestStandardLoggingPayloadCarriesType: the new error_rate_limit_type field reaches the structured payload for both ProxyRateLimitError and plain RateLimitError, is None when unspecified, and is None for non-rate-limit exceptions (symmetric with error_rate_limit_category). - TestProxyHooksWireTypeCorrectly: drives the actual raise sites in the v1 parallel_request_limiter helper, the v3 _handle_rate_limit_error (both "tokens" and "max_parallel_requests" paths), and the batch limiter (both tokens and requests paths) — coverage tools see the new rate_limit_type= kwargs as exercised, not just the import shape. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): cover _coerce_message branches and v1 dimension detection Drives the patch coverage on the new orthogonal RateLimitType wiring up to (or close to) 100% on the touched files. ProxyRateLimitError._coerce_message — was 22% covered, now 100%: * nested {error: {message}} dict * nested {message: {message}} dict (alt key) * dict without 'error'/'message' keys → JSON dump fallback * non-JSON-serializable dict value → str() fallback * non-string non-mapping detail (int) → str() coercion v1 parallel_request_limiter dimension detection — was 0% covered, now exercised across 6 parametrized cases: * check_key_in_limits else-branch: current at concurrent / TPM / RPM cap → asserts rate_limit_type is concurrent_requests / tokens / requests. * check_key_in_limits base case (current is None): max_parallel_requests / tpm_limit / rpm_limit set to 0 → asserts the most-specific zero attribution wins per the helper's order. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(proxy/hooks): add ProxyHTTPRateLimitError + provider resolver Introduces a small helper layer used by every proxy-side rate-limit hook so that the 429 they raise carries a populated llm_provider / model — instead of an empty exception.llm_provider that downstream loggers (Prometheus failure metric, observability callbacks) read as 'no provider attribution'. ProxyHTTPRateLimitError inherits from both fastapi.HTTPException (so the proxy server still renders it as a 429) and litellm.exceptions.RateLimitError (so isinstance checks and PrometheusLogger._get_exception_class_name pick up llm_provider). We deliberately don't call RateLimitError.__init__ — it constructs an httpx.Response we don't need and would just add failure surface; attribute parity is what downstream consumers care about. resolve_llm_provider_for_rate_limit() wraps litellm.get_llm_provider defensively. Internal limiter hooks fire from async_pre_call_hook — well before get_llm_provider runs anywhere else in the request lifecycle — so we have to call it ourselves at raise time. If the model is missing or unparseable (alias, router-only model) we fall back to llm_provider='litellm_proxy' rather than letting a second exception leak out and break the request path. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on parallel-request 429s Both v1 and v3 parallel-request limiters fired bare HTTPException(429) from inside async_pre_call_hook. The downstream Prometheus failure metric reads exception.llm_provider via _get_exception_class_name — the empty value showed up as exception_class='HTTPException' and left model_id='None' on the time series. Threads requested_model through every raise site in: * parallel_request_limiter.py: - check_key_in_limits (the per-key/per-model/per-user/per-team/ per-customer over-limit path) - raise_rate_limit_error (zero-limit + global_max_parallel_requests paths) — now takes an optional requested_model kwarg * parallel_request_limiter_v3.py: - _handle_rate_limit_error (the OVER_LIMIT translator), called from both the should_rate_limit pre-check and the TPM reservation path Resolved via resolve_llm_provider_for_rate_limit so unknown / missing models silently fall back to llm_provider='litellm_proxy' instead of breaking the request path with a second exception. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on dynamic-rate-limit 429s Same plumbing change as the parallel limiters, applied to both dynamic_rate_limiter (v1) and dynamic_rate_limiter_v3: * v1: TPM-zero and RPM-zero paths in async_pre_call_hook now resolve data['model'] -> (model, llm_provider) once and pass it into both raises. * v3: All three raise sites in _check_rate_limits — the model_saturation_check enforced raise, the priority_model enforced raise, and the fail-closed unknown-descriptor branch — now attribute the 429 to the actual provider. Falls back to llm_provider='litellm_proxy' when the model can't be resolved. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on batch-rate-limit 429s batch_rate_limiter._raise_rate_limit_error now takes a requested_model kwarg threaded from data['model'] in _check_and_increment_batch_counters. The batch-creation 429 is what gets raised when the input file's tokens/requests count would push the per-key TPM/RPM window over its limit. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on budget/iterations 429s Final batch of internal raise sites — the user/session-budget and max-iterations hooks. Same pattern: resolve data['model'] once at raise time, attach to ProxyHTTPRateLimitError so Prometheus and observability callbacks can attribute the 429. Hooks updated: * max_budget_limiter (per-user max_budget exceeded) * max_iterations_limiter (per-session agent iteration cap) * max_budget_per_session_limiter (per-session dollar cap) All three fall back to llm_provider='litellm_proxy' when data['model'] is missing or unparseable. Drops the now-unused HTTPException import from each module. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(proxy/hooks): pin provider field on internal rate-limit 429s Regression coverage for the 'provider field missing' bug across every proxy-side rate-limit hook + the helper layer: * ProxyHTTPRateLimitError class shape (HTTPException + RateLimitError, dict-detail stringification, None-provider normalization). * resolve_llm_provider_for_rate_limit happy paths (gpt-4o-mini, anthropic/..., bedrock/...) plus all three fallback branches (None, '', unknown name) plus a 'get_llm_provider raises' case that asserts we swallow the secondary exception. * For each limiter (parallel v1/v3, dynamic v1/v3, batch, max_budget, max_iterations, max_budget_per_session): assert the raised exception is a RateLimitError carrying the resolved model + llm_provider, and a sibling test that asserts the fallback path returns 'litellm_proxy' without leaking a second exception. * Two PrometheusLogger._get_exception_class_name pins so the Prometheus failure metric label flips from 'HTTPException' to 'Openai.ProxyHTTPRateLimitError' (or 'Litellm_proxy.*' on fallback) — that's what dashboards consume. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * perf(proxy/hooks): defer provider resolution to over-limit branches * fix: use error_message in raise_rate_limit_error to avoid literal 'None' in detail * Consolidate rate_limiter_utils imports in dynamic_rate_limiter * fix(proxy): set num_retries/max_retries on ProxyHTTPRateLimitError ProxyHTTPRateLimitError inherits from RateLimitError but did not call RateLimitError.__init__, so num_retries/max_retries were never set. When Starlette's HTTPException lacks __str__, MRO falls through to RateLimitError.__str__, which unconditionally reads these attributes and raises AttributeError during logging/traceback formatting. Initialize them to None defensively. * fix(mypy): silence base-class status_code conflict on ProxyHTTPRateLimitError HTTPException declares 'status_code: int' while openai.RateLimitError (via APIStatusError) declares 'status_code: Literal[429] = 429'. Mypy flags the multi-base override as [misc] in CI lint. The runtime semantics are fine (we set self.status_code in __init__), so silence the class-level annotation conflict with a targeted ignore. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix: annotate batch limiter _raise_rate_limit_error as NoReturn * feat(prometheus): rate-limit category/type labels + exception_class back-compat (follow-up to BerriAI#27687) (BerriAI#27706) * feat(prometheus): add rate_limit_category and rate_limit_type labels Adds two new labels to litellm_proxy_failed_requests_metric so dashboards can split 429s by rate-limit source (vendor vs. litellm-internal) and by the dimension that was exceeded (requests/tokens/concurrent_requests/ budget/max_iterations) without parsing free-text error messages. Closes the Prometheus side of LIT-2718. The unified RateLimitError.category and .rate_limit_type fields landed in PR BerriAI#27687 but were only surfaced on StandardLoggingPayload (custom-callback channel); this exposes them on the metric label set as well. Both labels are populated only when the underlying exception is a litellm.RateLimitError; non-rate-limit failures keep them empty. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(prometheus): populate rate-limit labels + preserve exception_class back-compat Two coupled changes in the Prometheus integration: 1. async_post_call_failure_hook now extracts the new RateLimitError .category / .rate_limit_type fields (added in PR BerriAI#27687) via a _extract_rate_limit_labels helper and forwards them through UserAPIKeyLabelValues onto litellm_proxy_failed_requests_metric. Empty for non-rate-limit failures. 2. _get_exception_class_name special-cases ProxyRateLimitError and keeps emitting 'HTTPException' for the exception_class label. Without this shim, ProxyRateLimitError (which multi-inherits from HTTPException + RateLimitError) would silently flip the label from 'HTTPException' (the historical value for proxy-side 429s) to 'ProxyRateLimitError', breaking existing dashboards / alerts that key off exception_class='HTTPException'. Distinguishing vendor vs. litellm 429s is now the job of the new rate_limit_category label. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(prometheus): cover rate-limit labels and exception_class back-compat Adds 19 tests across: - enum / label-list registration - _extract_rate_limit_labels for vendor RateLimitError, ProxyRateLimitError, non-rate-limit and None inputs (incl. parametrized over every RateLimitErrorCategory x RateLimitType combo) - _get_exception_class_name back-compat: ProxyRateLimitError keeps the legacy 'HTTPException' string while vendor RateLimitError keeps the historical 'Provider.ClassName' format - end-to-end through async_post_call_failure_hook with both ProxyRateLimitError and vendor RateLimitError, asserting both new labels populate and exception_class stays back-compat Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(prometheus): tolerate missing fastapi in lazy ProxyRateLimitError import Address greptile feedback: - async_post_call_failure_hook docstring: drop the stale labelnames listing and reference PrometheusMetricLabels.litellm_proxy_failed_requests_metric as the source of truth so the doc cannot drift from the actual labelset. - _get_exception_class_name: guard the lazy ProxyRateLimitError import with ImportError so router-side fallback callsites don't blow up in non-proxy installs that don't have fastapi (a transitive dep of proxy.common_utils.proxy_rate_limit_error). Behavior is unchanged when fastapi is available. Also fix the existing enterprise callback test that asserted the old labelset on litellm_proxy_failed_requests_metric — it now expects the new rate_limit_category / rate_limit_type labels populated for vendor 429s. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(bugbot): simplify rate-limit label coercion + guard None detail - prometheus.py _extract_rate_limit_labels: RateLimitError.__init__ already normalizes category/rate_limit_type to plain str, so the getattr(.value) + isinstance dance was dead code. Reduce to str(value) if not None. - proxy_rate_limit_error.py _coerce_message: short-circuit None to '' instead of falling through to str(None) = 'None', which produced the literal message 'litellm.RateLimitError: None'. * fix(rate-limit): surface unified category/type fields on BudgetExceededError The most common budget cap (virtual-key max_budget enforcement in auth_checks.py) raises litellm.BudgetExceededError, a bare Exception subclass that bypassed the unified rate-limit error class introduced by PR BerriAI#27687. Custom callbacks reading StandardLoggingPayload.error_information saw category=None and rate_limit_type=None for these 429s, missing the most common budget case (team / org / end-user budgets all hit the same code path). Surface the fields off BudgetExceededError as plain attributes: - category = RateLimitErrorCategory.LITELLM_RATE_LIMIT - rate_limit_type = RateLimitType.BUDGET - llm_provider = "" (or caller-supplied) Switch get_error_information and _extract_rate_limit_labels from isinstance(RateLimitError) gating to duck-typed attribute reads, guarded by membership in the rate-limit enums so unrelated third-party exceptions exposing a .category attribute can't leak garbage values into the payload. This is strictly additive: BudgetExceededError keeps its bare-Exception base class, so `except BudgetExceededError:` handlers keep firing and `except RateLimitError:` does not start catching budget errors. * fix(rate-limit): validate enum membership at duck-typed read sites + enrich BudgetExceededError llm_provider Two follow-ups uncovered during the second QA pass on PR BerriAI#27687: 1. Guard third-party `.category` / `.rate_limit_type` attribute leakage. The duck-typed read in `get_error_information` and `_extract_rate_limit_labels` would forward any string attribute named `category` / `rate_limit_type` on an unrelated third-party exception into the StandardLoggingPayload and Prometheus labels — silently mislabeling custom-callback payloads and blowing out Prometheus label cardinality. Add `validate_rate_limit_category` / `validate_rate_limit_type` helpers that gate on the documented enum value sets; non-matching values are dropped to None. 2. Enrich BudgetExceededError.llm_provider from request_data. Budget checks live in tenant-scoped helpers (key / team / org / tag / end-user / project) that don't see the request model, so the BudgetExceededError they raise carried llm_provider="" — leaving custom-metrics consumers without provider attribution for the most common 429 case. Resolve it once at the central UserAPIKeyAuthExceptionHandler seam, before post_call_failure_hook fires, so the StandardLoggingPayload the callback sees has the same provider attribution as RPM/TPM 429s. Regression tests pin both: 4 leakage tests + 4 enrichment tests. The leakage tests would fail under the pre-validation version of either read site; the enrichment tests would fail if the handler skipped the resolver call. * fix(rate-limit): resolve router model_name aliases to real provider (BerriAI#27914) * fix(rate-limit): resolve router model_name aliases to real provider For nearly every real LiteLLM proxy deployment the request model is a router model_name alias (e.g. 'tpm-locked' -> litellm_params.model: openai/gpt-4o-mini), and 'litellm.get_llm_provider' doesn't know about router aliases — it raises 'LLMProviderNotProvidedError'. The resolver then fell through to the defensive 'litellm_proxy' fallback, so the 'llm_provider' field this PR adds was effectively always 'litellm_proxy' in the field, defeating its purpose for the most common proxy configuration. Add a router-alias fallback step: when 'get_llm_provider' raises, scan the active 'llm_router.model_list' for a deployment whose 'model_name' matches the request model and resolve from its 'litellm_params.model' instead. If multiple deployments share the same alias (load-balancing case) the first one wins — every deployment under one alias should agree on provider in any sensible config, and 'first' is deterministic so the Prometheus label stays stable. Defensive throughout: an uninitialized router, a malformed deployment, a 'litellm_params.model' that itself fails 'get_llm_provider' — every branch falls through to the existing 'litellm_proxy' fallback rather than letting a secondary exception escape and mask the rate-limit error we're trying to surface. Tests: - test_router_alias_resolves_to_underlying_provider: alias 'tpm-locked' -> 'openai/gpt-4o-mini' produces provider='openai', model='gpt-4o-mini'. - test_router_alias_with_multiple_deployments_uses_first. - test_router_alias_unknown_falls_back. - test_router_alias_with_malformed_deployment_falls_back. - Existing fallback test updated to also stub 'litellm.proxy.proxy_server.llm_router' so it exercises the full 'no resolution anywhere' path. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(rate-limit): harden router alias resolver + test isolation - Wrap _resolve_provider_from_router_alias loop in top-level try/except so a non-iterable model_list / unexpected deployment shape can't escape and mask the 429 with a 500. - Type-check litellm_params before .get() to handle non-dict truthy values. - Patch llm_router=None in the parametrized fallback test so a router left by another test in the session can't redirect the unknown-model path. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(bugbot): preserve "BudgetExceededError" Prometheus label Adding llm_provider to BudgetExceededError (so callbacks get provider attribution from StandardLoggingPayload) made the provider-prefix step in _get_exception_class_name silently flip the label from "BudgetExceededError" to e.g. "Openai.BudgetExceededError", breaking dashboards keyed on the historical value. Short-circuit BudgetExceededError in _get_exception_class_name the same way ProxyRateLimitError already is. Provider/category attribution still lands on the new rate_limit_category / rate_limit_type labels. * test: fix invalid 'rpm' rate_limit_type in v3 limiter test mocks The v3 rate limiter only emits 'requests', 'tokens', or 'max_parallel_requests'. Using 'rpm' caused map_v3_rate_limit_type to return None, leaving the expected RateLimitType.REQUESTS untested. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(bugbot): hoist provider resolver + opt-in prom rate-limit labels - dynamic_rate_limiter.py: hoist resolve_llm_provider_for_rate_limit above the TPM/RPM if/elif so the lookup runs once per request, matching the pattern in dynamic_rate_limiter_v3.py. - prometheus.py: gate the new rate_limit_category / rate_limit_type labels on litellm_proxy_failed_requests_metric behind litellm.prometheus_emit_rate_limit_labels (default False). Mirrors the existing prometheus_emit_stream_label opt-in. Preserves the metric's pre-unification label set so existing dashboards / recording rules keep matching after upgrade; operators can enable the new labels once downstream consumers include them. - Tests updated: default-off back-compat case, opt-in path enables the flag before asserting label presence. * fix: stabilize prometheus label sets and drop redundant model normalization - Cache PrometheusLogger.get_labels_for_metric per metric_name so that the label set used to construct counters at __init__ time stays in sync with the label set used at increment time, even if module-level toggles like prometheus_emit_rate_limit_labels or prometheus_emit_stream_label are flipped at runtime. Without this, toggling these flags after the logger was created would cause ValueError from prometheus_client because the runtime labels would not match the counter's declared labelnames. - Drop redundant 'model or ""' guard in ProxyRateLimitError.__init__ where model is already normalized one step earlier. Co-authored-by: Yassin Kortam <yassin@berri.ai> * perf(dynamic_rate_limiter): only resolve provider when rate limit hit Co-authored-by: Yassin Kortam <yassin@berri.ai> * test(prometheus): clear cached metric labels after toggling rate-limit flag The PrometheusLogger caches each metric's label set at construction time so that labels used at counter.labels(...) time stay consistent with the labels the metric was registered with. The enterprise async_post_call_failure_hook test toggles litellm.prometheus_emit_rate_limit_labels = True AFTER the fixture has already built the logger, so without invalidating the cache the rate_limit_category / rate_limit_type labels never reach the mocked counter and the assert_called_once_with check fails. Co-authored-by: Yassin Kortam <yassin@berri.ai> * test: fix CI failures from prom label cache + flaky time-window assertion PrometheusLogger.get_labels_for_metric now caches the per-metric label set at first read so the labels passed to counter.labels(...) stay in lock step with the labels the counter was registered with. This broke two existing test patterns: - test_prometheus_labels.py: tests bind the real method onto a MagicMock, but MagicMock auto-creates a Mock for _cached_metric_labels whose .get(...) returns a truthy Mock — treated as a populated cache and returned as the label set, producing empty filtered labels and KeyError on labels["requested_model"] / ["route"]. Seed real {} containers for _cached_metric_labels and label_filters before binding. - test_prometheus_logging_callbacks.py::test_set_team_budget_metrics_with_custom_labels: the fixture builds the logger before the test monkeypatches litellm.custom_prometheus_metadata_labels, so the cached label set never picks up the new metadata labels. Clear the cache after the monkeypatch (same pattern already used for the rate-limit toggle in test_async_post_call_failure_hook). UI: view_logs/index.test.tsx "Last Minute" window assertion is off by one at the minute boundary. start_date is floored to the minute, so the dropped sub-minute fraction can push the truncated-seconds diff up to (minMinutes+1)*60 exactly when the click lands near a minute rollover. Switch the upper bound to toBeLessThanOrEqual. * feat(otel-v2): surface rate_limit_category + rate_limit_type on failed LLM-call spans PR BerriAI#28909 introduced the typed v2 OTel engine that builds spans from StandardLoggingPayload, with SpanError carrying error_type + message and the genai mapper stamping error.type onto every failed LLM-call span. This PR's earlier commits added error_rate_limit_category and error_rate_limit_type to the same StandardLoggingPayload.error_information the v2 engine reads — but neither field reached a span attribute, so v2 OTel traces stayed opaque about *why* a 429 fired (vendor vs litellm, RPM vs TPM vs concurrent vs budget vs max_iterations) even after the custom-callback and prometheus surfaces gained that decomposition. Three coupled changes: 1. semconv.py: add LiteLLM.ERROR_RATE_LIMIT_CATEGORY / LiteLLM.ERROR_RATE_LIMIT_TYPE under the litellm.* vendor namespace (no GenAI semconv equivalent exists for who-rate-limited / which-dimension). 2. payloads.py: extend SpanError with rate_limit_category + rate_limit_type, populated by _parse_error() from the same error_information.error_rate_limit_* fields the custom-callback channel and prometheus rate_limit_category / rate_limit_type labels read. Single source of truth across all three observability surfaces. 3. mappers/genai.py: stamp the two attributes on the LLM-call span when present. drop_none guarantees they stay absent (not 'None') for non-rate-limit failures so trace consumers can read them unconditionally. Three regression tests in test_otel_v2_emitter.py pin: a vendor / litellm-internal RateLimitError lands category=litellm_rate_limit + rate_limit_type=requests on the span; a BudgetExceededError lands rate_limit_type=budget; a non-rate-limit failure (BadRequestError) keeps the rate_limit_* attributes absent. Mutation-tested against reverting either the SpanError extension or the _parse_error read site — both new tests fail under either mutation. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test: align prometheus user-budget + logs quick-select tests with merged code The merge into this branch left two test patterns out of step with the code they exercise. test_set_user_budget_metrics_includes_user_email_and_alias_labels_when_opted_in flipped litellm.prometheus_user_budget_label_include_email_alias after the fixture had already built the PrometheusLogger. get_labels_for_metric now snapshots each metric's label set at construction time, so the runtime flip no longer reached the cached labels. Enable the flag before constructing the logger, matching how the proxy applies config at startup. view_logs/index.test.tsx referenced uiSpendLogsCall and moment without importing them, and the merged index.tsx now fetches through useLogFilterLogic (the hook the file stubs out) rather than calling uiSpendLogsCall directly. Add the imports and restore the real hook for the Quick Select window assertions so the call is actually observed. * refactor(otel/v2): drop rate-limit decomposition from the LLM-call span Proxy-side rate limits (litellm_rate_limit, budget, max_iterations) are rejected at the gate before any upstream call, so async_post_call_failure_hook tags the synthetic failure log with LITELLM_LOGGING_NO_UPSTREAM_LLM_CALL and the v2 OTel logger never opens an LLM-call span for them; the litellm.error.rate_limit_category / litellm.error.rate_limit_type attributes were dead for exactly the cases they were meant to surface. The only failure that does open an LLM-call span carrying a RateLimitError is a vendor 429, where rate_limit_type is always None and the category just restates error.type=RateLimitError. The decomposition still reaches downstream consumers through StandardLoggingPayload.error_information.error_rate_limit_* and the prometheus rate_limit_category / rate_limit_type labels, both unchanged. Removes the SpanError fields, the _parse_error reads, the genai mapper attributes, the semconv keys, and the three span tests that asserted a scenario that never reaches the mapper in production. * fix(batch_rate_limiter): map max_parallel_requests to concurrent_requests * refactor(prometheus): drop transitive fastapi import from _get_exception_class_name Read the legacy exception_class label from a prometheus_exception_class_name marker on ProxyRateLimitError instead of importing the proxy module, keeping the integrations layer free of a transitive fastapi dependency. * chore(ui): sync schema.d.ts with unified rate-limit error spec The ProxyRateLimitError docstring flows into the proxy OpenAPI spec's 429 response description, so the generated dashboard types were out of sync. Regenerated via npm run gen:api (Check UI API Types Sync). --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> Co-authored-by: Yassin Kortam <yassin@berri.ai>
michaelxer
pushed a commit
to michaelxer/litellm
that referenced
this pull request
Jun 17, 2026
…odel, and llm_provider fields (BerriAI#27687) * feat(exceptions): add RateLimitErrorCategory + headers/detail fields on RateLimitError LiteLLM previously surfaced rate-limit conditions through several unrelated error classes (RateLimitError, FastAPI HTTPException(429), BaseLLMException). This commit adds the data model needed to consolidate them under a single class: * RateLimitErrorCategory enum exposing four categorical values (vendor_rate_limit, vendor_batch_rate_limit, litellm_rate_limit, litellm_batch_rate_limit) so callers can switch on the rate-limit source. * New optional fields on RateLimitError: - category (defaults to vendor_rate_limit, preserving today's behavior for every existing call site in exception_mapping_utils); - headers (preserves retry-after / rate_limit_type / reset_at across the proxy boundary instead of dropping them on the floor); - detail (mirrors FastAPI HTTPException.detail so the same instance can be serialized through both paths). litellm.RateLimitErrorCategory is re-exported at the package root to match the existing exception-export pattern. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(proxy): add ProxyRateLimitError unifying RateLimitError + HTTPException Adds a single proxy-side error class that subclasses BOTH litellm.exceptions.RateLimitError AND fastapi.HTTPException via cooperative multiple inheritance. Why both bases: * Subclassing RateLimitError lets user code catch every rate-limit source with one 'except RateLimitError' and switch on the new .category field. * Subclassing HTTPException keeps every existing FastAPI plumbing path (the isinstance(e, HTTPException) branches in proxy_server.py route handlers, FastAPI's own dispatcher, and tests asserting pytest.raises(HTTPException)) working without modification, and preserves retry-after / rate_limit_type / reset_at headers on the wire. The class declaration order is (HTTPException, RateLimitError) so the MRO puts HTTPException's no-super-call __init__ ahead of openai's cooperative __init__ chain — preventing openai.APIError.super().__init__(message) from landing in HTTPException.__init__(status_code=message). LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from budget + iteration limiters Replaces three bare HTTPException(status_code=429, ...) call sites with ProxyRateLimitError, which is both a RateLimitError (catchable by category) and an HTTPException (preserves existing FastAPI serialization). Drops the now-unused HTTPException import in the iteration / per-session limiters. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from parallel-request limiters Replaces HTTPException(status_code=429, ...) call sites in the v1 and v3 parallel-request limiters (key/team/user/model/customer rate limits) with ProxyRateLimitError. Updates the raise_rate_limit_error helper's return type annotation accordingly. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from dynamic rate limiters Replaces HTTPException(status_code=429, ...) call sites in the v1 and v3 dynamic rate limiters (project-level TPM/RPM allocation, model-saturation checks, priority-based limits, fail-closed guards) with ProxyRateLimitError. The v3 limiter still imports HTTPException for an unrelated bare 'except HTTPException:' branch. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from batch rate limiter Replaces HTTPException(status_code=429, ...) in batch_rate_limiter._raise_rate_limit_error with ProxyRateLimitError tagged as RateLimitErrorCategory.LITELLM_BATCH_RATE_LIMIT so users can distinguish batch-level throttling (which counts requests/tokens across an uploaded batch input file before submission) from the generic key/team/user RPM/TPM limiter. The HTTPException import is retained because the same module raises HTTPException for unrelated 403/IO error paths. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): pin down unified rate-limit error contract Adds a dedicated test module covering the new RateLimitErrorCategory enum, RateLimitError.category default + override behavior, ProxyRateLimitError's dual nature (RateLimitError + HTTPException), and a parametrized regression guard that asserts every proxy hook module imports the unified class. The regression guard catches the failure mode the refactor is designed to prevent: someone re-introducing a bare HTTPException(status_code=429, ...) in one of the hook modules instead of going through ProxyRateLimitError. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(logging): expose rate-limit category via StandardLoggingPayload Adds an optional 'error_rate_limit_category' field to StandardLoggingPayloadErrorInformation, populated from the unified RateLimitError.category attribute (introduced in the previous commits on this branch). Why: the .category attribute is reachable off the raw exception today via getattr(e, 'category', None), but the structured contract that downstream custom callbacks / loggers / spend log writers consume is the StandardLoggingPayload. Without this field, a user building custom rate-limit metrics on top of callback data has to special-case the raw exception object — which defeats the purpose of the StandardLoggingPayload abstraction. The field is None for non-rate-limit exceptions (so consumers can read it unconditionally without isinstance checks) and is one of the RateLimitErrorCategory string values otherwise. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): assert StandardLoggingPayload carries the category Five tests covering: vendor default, explicit litellm_rate_limit and litellm_batch_rate_limit values, None for non-rate-limit exceptions, and None when no exception is provided. Pins down the contract that custom callbacks can read 'error_information.error_rate_limit_category' off the StandardLoggingPayload to drive custom rate-limit metrics without ever reaching for the raw exception. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(types): silence mypy [misc] on intentional dual-base attr overlap mypy emits two [misc] errors on the ProxyRateLimitError class line because its two bases declare overlapping attributes with related-but-not-identical annotations: * status_code: int on starlette HTTPException vs. Literal[429] on openai's RateLimitError (every openai status-error subclass narrows it the same way and silences pyright with the same convention). * headers: Mapping[str, str] | None on HTTPException vs. our Optional[ Dict[str, str]] (the proxy hooks always carry a stringified dict). Both narrowings are intentional and enforced at construction time. Add a type: ignore[misc] with an inline explanation rather than relax the annotations on the parent or change the wire-format guarantees. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): add direct hook-invocation tests to lift patch coverage Adds six end-to-end tests that drive each refactored hook past its limit and assert the unified ProxyRateLimitError is raised with the correct category and dual-base shape. Complements the import-shape-only parametrized guard above by actually executing the new 'raise ProxyRateLimitError(...)' lines so codecov's patch coverage sees them as hit. Hooks covered (one test each): * parallel_request_limiter v1 — direct call to raise_rate_limit_error() * parallel_request_limiter v3 — direct call to _handle_rate_limit_error with a fabricated OVER_LIMIT response * max_iterations_limiter — full async_pre_call_hook with mocked agent registry, second call exceeds budget=1 * max_budget_limiter — async_pre_call_hook with mocked get_current_spend * dynamic_rate_limiter v1 — async_pre_call_hook with mocked check_available_usage forcing available_tpm == 0 * batch_rate_limiter — direct _raise_rate_limit_error call, asserts category is the batch-specific LITELLM_BATCH_RATE_LIMIT (not the generic LITELLM_RATE_LIMIT) LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix: guard rate_limit_category extraction with isinstance check * test(rate-limit): cover remaining hook raise sites for codecov Adds five more direct hook-invocation tests so every PR-touched line in the proxy hooks is exercised by tests in tests/test_litellm/, which codecov measures: * parallel_request_limiter v1 — check_key_in_limits inline raise (the second raise site, separate from the raise_rate_limit_error helper covered earlier) * dynamic_rate_limiter v1 — RPM raise branch (TPM branch was already covered) * dynamic_rate_limiter v3 — parametrized over all three raise sites: model_saturation_check, priority_model, and the fail-closed fallback for an unrecognized descriptor_key * max_budget_per_session_limiter — full async_pre_call_hook with a mocked agent registry and over-budget cached spend All 42 tests in test_rate_limit_error_unification.py now pass and together exercise every changed import + raise line across the eight refactored proxy hooks. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix: use computed error_message in ProxyRateLimitError detail * fix(parallel-request-limiter): drop None from detail; annotate raise_rate_limit_error as NoReturn The v1 ' raise_rate_limit_error' helper built an unused 'error_message' variable and then assembled the actual ' detail' via an f-string that interpolated 'additional_details' verbatim — producing 'Max parallel request limit reached None' when invoked without arguments (flagged by code review). Fix the helper to: - use the constructed 'error_message' as the detail - annotate the helper as NoReturn since it always raises - drop the redundant 'raise'/'return' at the two call sites Add two regression tests covering both the with- and without- additional_details paths. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): drop literal 'None' from raise_rate_limit_error detail The v1 parallel_request_limiter's raise_rate_limit_error helper has a long-standing bug: it computes a None-guarded 'error_message' string but then ignores it and emits an f-string that interpolates the raw 'additional_details' arg. Callers that pass no argument get 'Max parallel request limit reached None' as the user-facing detail. This commit: * wires error_message into the detail kwarg so the None-guard actually applies and operators see a clean message; * changes the return-type annotation from ProxyRateLimitError to NoReturn (the function always raises) so type-checkers know callers after this invocation are unreachable. Greptile P1 + P2 review feedback on PR BerriAI#27687. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(types): demote TypedDict floating string to a # comment A string literal placed after a field declaration in a TypedDict body is not a per-field docstring — it's an orphaned string expression Python discards. Tools like mypy / pyright that inspect TypedDict fields won't surface that text either. Move the documentation for error_rate_limit_category to a real comment so the intent is visible to readers and type-checker tooling without the misleading docstring framing. Greptile P2 review feedback on PR BerriAI#27687. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * security(exceptions): do not auto-copy vendor response headers to e.headers A vendor 429 response can set arbitrary headers (Set-Cookie, CORS overrides, …). Previously, when RateLimitError was constructed with only a 'response=' (no explicit 'headers=' kwarg), self.headers fell back to a copy of response.headers. If a downstream proxy serializer ever forwarded e.headers to the client, a malicious upstream could inject browser-interpreted headers for the proxy origin. Drop the fallback. Only headers passed explicitly via the headers= kwarg make it onto self.headers (proxy hooks pass retry-after etc. — they control what's surfaced). Vendor response headers stay reachable on e.response.headers for callers that explicitly want them. Today's proxy_server.py route handlers don't actually forward e.headers on the wire (they construct ProxyException without passing headers), so no current behavior changes — this is a defensive narrowing so the fallback can never be turned into a vector when someone wires e.headers through later. Veria-AI security review feedback on PR BerriAI#27687. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): regression guards for review-pass fixes Pins down the three review-pass fixes: * test_parallel_request_limiter_v1_helper_no_additional_details — calls raise_rate_limit_error() with no args and asserts the detail does NOT contain the literal string 'None'. Pre-fix, callers got 'Max parallel request limit reached None'. * test_rate_limit_error_does_not_auto_copy_response_headers — passes a vendor httpx.Response with a Set-Cookie header to RateLimitError WITHOUT an explicit headers= kwarg, asserts self.headers stays None (no leak), then re-checks that an explicit headers= kwarg DOES populate self.headers. Vendor headers remain reachable on e.response.headers for callers that explicitly want them. * The existing v1-helper test now also asserts the additional_details string makes it through to the detail. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(rate-limit): add orthogonal RateLimitType (requests/tokens/concurrent_requests/budget/max_iterations) trho's last ask in the LIT-2968 thread: distinguish rate-limit failures by the dimension that was exceeded, not just by who rate-limited (vendor vs. litellm). Adds: - RateLimitType str-enum exposed at `litellm.RateLimitType` with values requests / tokens / concurrent_requests / budget / max_iterations. - `rate_limit_type` kwarg on litellm.RateLimitError + ProxyRateLimitError; None default so existing callers (vendor-429 path in exception_mapping_utils) remain a no-op. - StandardLoggingPayloadErrorInformation.error_rate_limit_type so custom callbacks can split rate-limit failures by cause without parsing free-text error messages. Mirror to error_rate_limit_category extraction in get_error_information(); single isinstance(RateLimitError) check covers both. - map_v3_rate_limit_type() helper to collapse the v3 limiter's internal labels ("requests", "tokens", "max_parallel_requests") onto the public enum so the v3 limiter and dynamic_rate_limiter_v3 share one mapping. Defensive None on unknown values rather than silently picking a wrong dimension. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(proxy/hooks): wire rate_limit_type onto every limiter raise site Each refactored proxy hook now populates rate_limit_type with the dimension that actually tripped the limit, so downstream consumers (custom callbacks, prometheus exporters via the StandardLoggingPayload) can split key/team/user rate-limit failures by cause: - parallel_request_limiter (v1): detect dimension from current vs. limit in the post-cache branch (concurrent_requests > tokens > requests, matches the boolean condition order). Base case (current is None, one limit set to 0) picks the most-specific zero. raise_rate_limit_error() helper accepts an explicit rate_limit_type kwarg with CONCURRENT_REQUESTS default (matches every existing internal call site, including the global-limit branch). - parallel_request_limiter (v3): forward status["rate_limit_type"] through map_v3_rate_limit_type() so "max_parallel_requests" → CONCURRENT_REQUESTS for the public field while the raw v3 jargon stays on the HTTP header for wire-format backward compat. - dynamic_rate_limiter (v1): TPM-zero → TOKENS, RPM-zero → REQUESTS. Pass data["model"] through so callbacks see the model that hit the limit (addresses the secondary "provider missing" complaint in the original Slack thread, partially — the model is what dashboards typically split on). - dynamic_rate_limiter (v3): forward status["rate_limit_type"] via map_v3_rate_limit_type() at every raise site (model_saturation_check, priority_model, fail-closed unknown-descriptor guard). Also pass model. - batch_rate_limiter: limit_type is hard-typed "requests"|"tokens" — map directly without going through the helper's None branch. - max_budget_limiter, max_budget_per_session_limiter: BUDGET. - max_iterations_limiter: MAX_ITERATIONS. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): cover RateLimitType enum, hook wiring, and StandardLoggingPayload propagation 27 new tests across five new test classes: - TestRateLimitType: enum exposed at litellm.RateLimitType, all five values defined, RateLimitError default is None (vendor 429 path makes no claim about which dimension), accepts both string and enum forms with str-coercion guarantee for downstream JSON serializers. - TestProxyRateLimitErrorType: ProxyRateLimitError default is None, accepts string or enum, doesn't break existing callers that pass nothing. - TestMapV3RateLimitType: pins each v3-internal → public-enum mapping (tokens, requests, max_parallel_requests → concurrent_requests, unknown → None) so a future v3 refactor can't silently swap dimensions. - TestStandardLoggingPayloadCarriesType: the new error_rate_limit_type field reaches the structured payload for both ProxyRateLimitError and plain RateLimitError, is None when unspecified, and is None for non-rate-limit exceptions (symmetric with error_rate_limit_category). - TestProxyHooksWireTypeCorrectly: drives the actual raise sites in the v1 parallel_request_limiter helper, the v3 _handle_rate_limit_error (both "tokens" and "max_parallel_requests" paths), and the batch limiter (both tokens and requests paths) — coverage tools see the new rate_limit_type= kwargs as exercised, not just the import shape. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): cover _coerce_message branches and v1 dimension detection Drives the patch coverage on the new orthogonal RateLimitType wiring up to (or close to) 100% on the touched files. ProxyRateLimitError._coerce_message — was 22% covered, now 100%: * nested {error: {message}} dict * nested {message: {message}} dict (alt key) * dict without 'error'/'message' keys → JSON dump fallback * non-JSON-serializable dict value → str() fallback * non-string non-mapping detail (int) → str() coercion v1 parallel_request_limiter dimension detection — was 0% covered, now exercised across 6 parametrized cases: * check_key_in_limits else-branch: current at concurrent / TPM / RPM cap → asserts rate_limit_type is concurrent_requests / tokens / requests. * check_key_in_limits base case (current is None): max_parallel_requests / tpm_limit / rpm_limit set to 0 → asserts the most-specific zero attribution wins per the helper's order. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(proxy/hooks): add ProxyHTTPRateLimitError + provider resolver Introduces a small helper layer used by every proxy-side rate-limit hook so that the 429 they raise carries a populated llm_provider / model — instead of an empty exception.llm_provider that downstream loggers (Prometheus failure metric, observability callbacks) read as 'no provider attribution'. ProxyHTTPRateLimitError inherits from both fastapi.HTTPException (so the proxy server still renders it as a 429) and litellm.exceptions.RateLimitError (so isinstance checks and PrometheusLogger._get_exception_class_name pick up llm_provider). We deliberately don't call RateLimitError.__init__ — it constructs an httpx.Response we don't need and would just add failure surface; attribute parity is what downstream consumers care about. resolve_llm_provider_for_rate_limit() wraps litellm.get_llm_provider defensively. Internal limiter hooks fire from async_pre_call_hook — well before get_llm_provider runs anywhere else in the request lifecycle — so we have to call it ourselves at raise time. If the model is missing or unparseable (alias, router-only model) we fall back to llm_provider='litellm_proxy' rather than letting a second exception leak out and break the request path. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on parallel-request 429s Both v1 and v3 parallel-request limiters fired bare HTTPException(429) from inside async_pre_call_hook. The downstream Prometheus failure metric reads exception.llm_provider via _get_exception_class_name — the empty value showed up as exception_class='HTTPException' and left model_id='None' on the time series. Threads requested_model through every raise site in: * parallel_request_limiter.py: - check_key_in_limits (the per-key/per-model/per-user/per-team/ per-customer over-limit path) - raise_rate_limit_error (zero-limit + global_max_parallel_requests paths) — now takes an optional requested_model kwarg * parallel_request_limiter_v3.py: - _handle_rate_limit_error (the OVER_LIMIT translator), called from both the should_rate_limit pre-check and the TPM reservation path Resolved via resolve_llm_provider_for_rate_limit so unknown / missing models silently fall back to llm_provider='litellm_proxy' instead of breaking the request path with a second exception. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on dynamic-rate-limit 429s Same plumbing change as the parallel limiters, applied to both dynamic_rate_limiter (v1) and dynamic_rate_limiter_v3: * v1: TPM-zero and RPM-zero paths in async_pre_call_hook now resolve data['model'] -> (model, llm_provider) once and pass it into both raises. * v3: All three raise sites in _check_rate_limits — the model_saturation_check enforced raise, the priority_model enforced raise, and the fail-closed unknown-descriptor branch — now attribute the 429 to the actual provider. Falls back to llm_provider='litellm_proxy' when the model can't be resolved. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on batch-rate-limit 429s batch_rate_limiter._raise_rate_limit_error now takes a requested_model kwarg threaded from data['model'] in _check_and_increment_batch_counters. The batch-creation 429 is what gets raised when the input file's tokens/requests count would push the per-key TPM/RPM window over its limit. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on budget/iterations 429s Final batch of internal raise sites — the user/session-budget and max-iterations hooks. Same pattern: resolve data['model'] once at raise time, attach to ProxyHTTPRateLimitError so Prometheus and observability callbacks can attribute the 429. Hooks updated: * max_budget_limiter (per-user max_budget exceeded) * max_iterations_limiter (per-session agent iteration cap) * max_budget_per_session_limiter (per-session dollar cap) All three fall back to llm_provider='litellm_proxy' when data['model'] is missing or unparseable. Drops the now-unused HTTPException import from each module. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(proxy/hooks): pin provider field on internal rate-limit 429s Regression coverage for the 'provider field missing' bug across every proxy-side rate-limit hook + the helper layer: * ProxyHTTPRateLimitError class shape (HTTPException + RateLimitError, dict-detail stringification, None-provider normalization). * resolve_llm_provider_for_rate_limit happy paths (gpt-4o-mini, anthropic/..., bedrock/...) plus all three fallback branches (None, '', unknown name) plus a 'get_llm_provider raises' case that asserts we swallow the secondary exception. * For each limiter (parallel v1/v3, dynamic v1/v3, batch, max_budget, max_iterations, max_budget_per_session): assert the raised exception is a RateLimitError carrying the resolved model + llm_provider, and a sibling test that asserts the fallback path returns 'litellm_proxy' without leaking a second exception. * Two PrometheusLogger._get_exception_class_name pins so the Prometheus failure metric label flips from 'HTTPException' to 'Openai.ProxyHTTPRateLimitError' (or 'Litellm_proxy.*' on fallback) — that's what dashboards consume. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * perf(proxy/hooks): defer provider resolution to over-limit branches * fix: use error_message in raise_rate_limit_error to avoid literal 'None' in detail * Consolidate rate_limiter_utils imports in dynamic_rate_limiter * fix(proxy): set num_retries/max_retries on ProxyHTTPRateLimitError ProxyHTTPRateLimitError inherits from RateLimitError but did not call RateLimitError.__init__, so num_retries/max_retries were never set. When Starlette's HTTPException lacks __str__, MRO falls through to RateLimitError.__str__, which unconditionally reads these attributes and raises AttributeError during logging/traceback formatting. Initialize them to None defensively. * fix(mypy): silence base-class status_code conflict on ProxyHTTPRateLimitError HTTPException declares 'status_code: int' while openai.RateLimitError (via APIStatusError) declares 'status_code: Literal[429] = 429'. Mypy flags the multi-base override as [misc] in CI lint. The runtime semantics are fine (we set self.status_code in __init__), so silence the class-level annotation conflict with a targeted ignore. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix: annotate batch limiter _raise_rate_limit_error as NoReturn * feat(prometheus): rate-limit category/type labels + exception_class back-compat (follow-up to BerriAI#27687) (BerriAI#27706) * feat(prometheus): add rate_limit_category and rate_limit_type labels Adds two new labels to litellm_proxy_failed_requests_metric so dashboards can split 429s by rate-limit source (vendor vs. litellm-internal) and by the dimension that was exceeded (requests/tokens/concurrent_requests/ budget/max_iterations) without parsing free-text error messages. Closes the Prometheus side of LIT-2718. The unified RateLimitError.category and .rate_limit_type fields landed in PR BerriAI#27687 but were only surfaced on StandardLoggingPayload (custom-callback channel); this exposes them on the metric label set as well. Both labels are populated only when the underlying exception is a litellm.RateLimitError; non-rate-limit failures keep them empty. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(prometheus): populate rate-limit labels + preserve exception_class back-compat Two coupled changes in the Prometheus integration: 1. async_post_call_failure_hook now extracts the new RateLimitError .category / .rate_limit_type fields (added in PR BerriAI#27687) via a _extract_rate_limit_labels helper and forwards them through UserAPIKeyLabelValues onto litellm_proxy_failed_requests_metric. Empty for non-rate-limit failures. 2. _get_exception_class_name special-cases ProxyRateLimitError and keeps emitting 'HTTPException' for the exception_class label. Without this shim, ProxyRateLimitError (which multi-inherits from HTTPException + RateLimitError) would silently flip the label from 'HTTPException' (the historical value for proxy-side 429s) to 'ProxyRateLimitError', breaking existing dashboards / alerts that key off exception_class='HTTPException'. Distinguishing vendor vs. litellm 429s is now the job of the new rate_limit_category label. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(prometheus): cover rate-limit labels and exception_class back-compat Adds 19 tests across: - enum / label-list registration - _extract_rate_limit_labels for vendor RateLimitError, ProxyRateLimitError, non-rate-limit and None inputs (incl. parametrized over every RateLimitErrorCategory x RateLimitType combo) - _get_exception_class_name back-compat: ProxyRateLimitError keeps the legacy 'HTTPException' string while vendor RateLimitError keeps the historical 'Provider.ClassName' format - end-to-end through async_post_call_failure_hook with both ProxyRateLimitError and vendor RateLimitError, asserting both new labels populate and exception_class stays back-compat Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(prometheus): tolerate missing fastapi in lazy ProxyRateLimitError import Address greptile feedback: - async_post_call_failure_hook docstring: drop the stale labelnames listing and reference PrometheusMetricLabels.litellm_proxy_failed_requests_metric as the source of truth so the doc cannot drift from the actual labelset. - _get_exception_class_name: guard the lazy ProxyRateLimitError import with ImportError so router-side fallback callsites don't blow up in non-proxy installs that don't have fastapi (a transitive dep of proxy.common_utils.proxy_rate_limit_error). Behavior is unchanged when fastapi is available. Also fix the existing enterprise callback test that asserted the old labelset on litellm_proxy_failed_requests_metric — it now expects the new rate_limit_category / rate_limit_type labels populated for vendor 429s. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(bugbot): simplify rate-limit label coercion + guard None detail - prometheus.py _extract_rate_limit_labels: RateLimitError.__init__ already normalizes category/rate_limit_type to plain str, so the getattr(.value) + isinstance dance was dead code. Reduce to str(value) if not None. - proxy_rate_limit_error.py _coerce_message: short-circuit None to '' instead of falling through to str(None) = 'None', which produced the literal message 'litellm.RateLimitError: None'. * fix(rate-limit): surface unified category/type fields on BudgetExceededError The most common budget cap (virtual-key max_budget enforcement in auth_checks.py) raises litellm.BudgetExceededError, a bare Exception subclass that bypassed the unified rate-limit error class introduced by PR BerriAI#27687. Custom callbacks reading StandardLoggingPayload.error_information saw category=None and rate_limit_type=None for these 429s, missing the most common budget case (team / org / end-user budgets all hit the same code path). Surface the fields off BudgetExceededError as plain attributes: - category = RateLimitErrorCategory.LITELLM_RATE_LIMIT - rate_limit_type = RateLimitType.BUDGET - llm_provider = "" (or caller-supplied) Switch get_error_information and _extract_rate_limit_labels from isinstance(RateLimitError) gating to duck-typed attribute reads, guarded by membership in the rate-limit enums so unrelated third-party exceptions exposing a .category attribute can't leak garbage values into the payload. This is strictly additive: BudgetExceededError keeps its bare-Exception base class, so `except BudgetExceededError:` handlers keep firing and `except RateLimitError:` does not start catching budget errors. * fix(rate-limit): validate enum membership at duck-typed read sites + enrich BudgetExceededError llm_provider Two follow-ups uncovered during the second QA pass on PR BerriAI#27687: 1. Guard third-party `.category` / `.rate_limit_type` attribute leakage. The duck-typed read in `get_error_information` and `_extract_rate_limit_labels` would forward any string attribute named `category` / `rate_limit_type` on an unrelated third-party exception into the StandardLoggingPayload and Prometheus labels — silently mislabeling custom-callback payloads and blowing out Prometheus label cardinality. Add `validate_rate_limit_category` / `validate_rate_limit_type` helpers that gate on the documented enum value sets; non-matching values are dropped to None. 2. Enrich BudgetExceededError.llm_provider from request_data. Budget checks live in tenant-scoped helpers (key / team / org / tag / end-user / project) that don't see the request model, so the BudgetExceededError they raise carried llm_provider="" — leaving custom-metrics consumers without provider attribution for the most common 429 case. Resolve it once at the central UserAPIKeyAuthExceptionHandler seam, before post_call_failure_hook fires, so the StandardLoggingPayload the callback sees has the same provider attribution as RPM/TPM 429s. Regression tests pin both: 4 leakage tests + 4 enrichment tests. The leakage tests would fail under the pre-validation version of either read site; the enrichment tests would fail if the handler skipped the resolver call. * fix(rate-limit): resolve router model_name aliases to real provider (BerriAI#27914) * fix(rate-limit): resolve router model_name aliases to real provider For nearly every real LiteLLM proxy deployment the request model is a router model_name alias (e.g. 'tpm-locked' -> litellm_params.model: openai/gpt-4o-mini), and 'litellm.get_llm_provider' doesn't know about router aliases — it raises 'LLMProviderNotProvidedError'. The resolver then fell through to the defensive 'litellm_proxy' fallback, so the 'llm_provider' field this PR adds was effectively always 'litellm_proxy' in the field, defeating its purpose for the most common proxy configuration. Add a router-alias fallback step: when 'get_llm_provider' raises, scan the active 'llm_router.model_list' for a deployment whose 'model_name' matches the request model and resolve from its 'litellm_params.model' instead. If multiple deployments share the same alias (load-balancing case) the first one wins — every deployment under one alias should agree on provider in any sensible config, and 'first' is deterministic so the Prometheus label stays stable. Defensive throughout: an uninitialized router, a malformed deployment, a 'litellm_params.model' that itself fails 'get_llm_provider' — every branch falls through to the existing 'litellm_proxy' fallback rather than letting a secondary exception escape and mask the rate-limit error we're trying to surface. Tests: - test_router_alias_resolves_to_underlying_provider: alias 'tpm-locked' -> 'openai/gpt-4o-mini' produces provider='openai', model='gpt-4o-mini'. - test_router_alias_with_multiple_deployments_uses_first. - test_router_alias_unknown_falls_back. - test_router_alias_with_malformed_deployment_falls_back. - Existing fallback test updated to also stub 'litellm.proxy.proxy_server.llm_router' so it exercises the full 'no resolution anywhere' path. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(rate-limit): harden router alias resolver + test isolation - Wrap _resolve_provider_from_router_alias loop in top-level try/except so a non-iterable model_list / unexpected deployment shape can't escape and mask the 429 with a 500. - Type-check litellm_params before .get() to handle non-dict truthy values. - Patch llm_router=None in the parametrized fallback test so a router left by another test in the session can't redirect the unknown-model path. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(bugbot): preserve "BudgetExceededError" Prometheus label Adding llm_provider to BudgetExceededError (so callbacks get provider attribution from StandardLoggingPayload) made the provider-prefix step in _get_exception_class_name silently flip the label from "BudgetExceededError" to e.g. "Openai.BudgetExceededError", breaking dashboards keyed on the historical value. Short-circuit BudgetExceededError in _get_exception_class_name the same way ProxyRateLimitError already is. Provider/category attribution still lands on the new rate_limit_category / rate_limit_type labels. * test: fix invalid 'rpm' rate_limit_type in v3 limiter test mocks The v3 rate limiter only emits 'requests', 'tokens', or 'max_parallel_requests'. Using 'rpm' caused map_v3_rate_limit_type to return None, leaving the expected RateLimitType.REQUESTS untested. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(bugbot): hoist provider resolver + opt-in prom rate-limit labels - dynamic_rate_limiter.py: hoist resolve_llm_provider_for_rate_limit above the TPM/RPM if/elif so the lookup runs once per request, matching the pattern in dynamic_rate_limiter_v3.py. - prometheus.py: gate the new rate_limit_category / rate_limit_type labels on litellm_proxy_failed_requests_metric behind litellm.prometheus_emit_rate_limit_labels (default False). Mirrors the existing prometheus_emit_stream_label opt-in. Preserves the metric's pre-unification label set so existing dashboards / recording rules keep matching after upgrade; operators can enable the new labels once downstream consumers include them. - Tests updated: default-off back-compat case, opt-in path enables the flag before asserting label presence. * fix: stabilize prometheus label sets and drop redundant model normalization - Cache PrometheusLogger.get_labels_for_metric per metric_name so that the label set used to construct counters at __init__ time stays in sync with the label set used at increment time, even if module-level toggles like prometheus_emit_rate_limit_labels or prometheus_emit_stream_label are flipped at runtime. Without this, toggling these flags after the logger was created would cause ValueError from prometheus_client because the runtime labels would not match the counter's declared labelnames. - Drop redundant 'model or ""' guard in ProxyRateLimitError.__init__ where model is already normalized one step earlier. Co-authored-by: Yassin Kortam <yassin@berri.ai> * perf(dynamic_rate_limiter): only resolve provider when rate limit hit Co-authored-by: Yassin Kortam <yassin@berri.ai> * test(prometheus): clear cached metric labels after toggling rate-limit flag The PrometheusLogger caches each metric's label set at construction time so that labels used at counter.labels(...) time stay consistent with the labels the metric was registered with. The enterprise async_post_call_failure_hook test toggles litellm.prometheus_emit_rate_limit_labels = True AFTER the fixture has already built the logger, so without invalidating the cache the rate_limit_category / rate_limit_type labels never reach the mocked counter and the assert_called_once_with check fails. Co-authored-by: Yassin Kortam <yassin@berri.ai> * test: fix CI failures from prom label cache + flaky time-window assertion PrometheusLogger.get_labels_for_metric now caches the per-metric label set at first read so the labels passed to counter.labels(...) stay in lock step with the labels the counter was registered with. This broke two existing test patterns: - test_prometheus_labels.py: tests bind the real method onto a MagicMock, but MagicMock auto-creates a Mock for _cached_metric_labels whose .get(...) returns a truthy Mock — treated as a populated cache and returned as the label set, producing empty filtered labels and KeyError on labels["requested_model"] / ["route"]. Seed real {} containers for _cached_metric_labels and label_filters before binding. - test_prometheus_logging_callbacks.py::test_set_team_budget_metrics_with_custom_labels: the fixture builds the logger before the test monkeypatches litellm.custom_prometheus_metadata_labels, so the cached label set never picks up the new metadata labels. Clear the cache after the monkeypatch (same pattern already used for the rate-limit toggle in test_async_post_call_failure_hook). UI: view_logs/index.test.tsx "Last Minute" window assertion is off by one at the minute boundary. start_date is floored to the minute, so the dropped sub-minute fraction can push the truncated-seconds diff up to (minMinutes+1)*60 exactly when the click lands near a minute rollover. Switch the upper bound to toBeLessThanOrEqual. * feat(otel-v2): surface rate_limit_category + rate_limit_type on failed LLM-call spans PR BerriAI#28909 introduced the typed v2 OTel engine that builds spans from StandardLoggingPayload, with SpanError carrying error_type + message and the genai mapper stamping error.type onto every failed LLM-call span. This PR's earlier commits added error_rate_limit_category and error_rate_limit_type to the same StandardLoggingPayload.error_information the v2 engine reads — but neither field reached a span attribute, so v2 OTel traces stayed opaque about *why* a 429 fired (vendor vs litellm, RPM vs TPM vs concurrent vs budget vs max_iterations) even after the custom-callback and prometheus surfaces gained that decomposition. Three coupled changes: 1. semconv.py: add LiteLLM.ERROR_RATE_LIMIT_CATEGORY / LiteLLM.ERROR_RATE_LIMIT_TYPE under the litellm.* vendor namespace (no GenAI semconv equivalent exists for who-rate-limited / which-dimension). 2. payloads.py: extend SpanError with rate_limit_category + rate_limit_type, populated by _parse_error() from the same error_information.error_rate_limit_* fields the custom-callback channel and prometheus rate_limit_category / rate_limit_type labels read. Single source of truth across all three observability surfaces. 3. mappers/genai.py: stamp the two attributes on the LLM-call span when present. drop_none guarantees they stay absent (not 'None') for non-rate-limit failures so trace consumers can read them unconditionally. Three regression tests in test_otel_v2_emitter.py pin: a vendor / litellm-internal RateLimitError lands category=litellm_rate_limit + rate_limit_type=requests on the span; a BudgetExceededError lands rate_limit_type=budget; a non-rate-limit failure (BadRequestError) keeps the rate_limit_* attributes absent. Mutation-tested against reverting either the SpanError extension or the _parse_error read site — both new tests fail under either mutation. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test: align prometheus user-budget + logs quick-select tests with merged code The merge into this branch left two test patterns out of step with the code they exercise. test_set_user_budget_metrics_includes_user_email_and_alias_labels_when_opted_in flipped litellm.prometheus_user_budget_label_include_email_alias after the fixture had already built the PrometheusLogger. get_labels_for_metric now snapshots each metric's label set at construction time, so the runtime flip no longer reached the cached labels. Enable the flag before constructing the logger, matching how the proxy applies config at startup. view_logs/index.test.tsx referenced uiSpendLogsCall and moment without importing them, and the merged index.tsx now fetches through useLogFilterLogic (the hook the file stubs out) rather than calling uiSpendLogsCall directly. Add the imports and restore the real hook for the Quick Select window assertions so the call is actually observed. * refactor(otel/v2): drop rate-limit decomposition from the LLM-call span Proxy-side rate limits (litellm_rate_limit, budget, max_iterations) are rejected at the gate before any upstream call, so async_post_call_failure_hook tags the synthetic failure log with LITELLM_LOGGING_NO_UPSTREAM_LLM_CALL and the v2 OTel logger never opens an LLM-call span for them; the litellm.error.rate_limit_category / litellm.error.rate_limit_type attributes were dead for exactly the cases they were meant to surface. The only failure that does open an LLM-call span carrying a RateLimitError is a vendor 429, where rate_limit_type is always None and the category just restates error.type=RateLimitError. The decomposition still reaches downstream consumers through StandardLoggingPayload.error_information.error_rate_limit_* and the prometheus rate_limit_category / rate_limit_type labels, both unchanged. Removes the SpanError fields, the _parse_error reads, the genai mapper attributes, the semconv keys, and the three span tests that asserted a scenario that never reaches the mapper in production. * fix(batch_rate_limiter): map max_parallel_requests to concurrent_requests * refactor(prometheus): drop transitive fastapi import from _get_exception_class_name Read the legacy exception_class label from a prometheus_exception_class_name marker on ProxyRateLimitError instead of importing the proxy module, keeping the integrations layer free of a transitive fastapi dependency. * chore(ui): sync schema.d.ts with unified rate-limit error spec The ProxyRateLimitError docstring flows into the proxy OpenAPI spec's 429 response description, so the generated dashboard types were out of sync. Regenerated via npm run gen:api (Check UI API Types Sync). --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> Co-authored-by: Yassin Kortam <yassin@berri.ai>
koladefaj
pushed a commit
to koladefaj/litellm
that referenced
this pull request
Jun 17, 2026
…odel, and llm_provider fields (BerriAI#27687) * feat(exceptions): add RateLimitErrorCategory + headers/detail fields on RateLimitError LiteLLM previously surfaced rate-limit conditions through several unrelated error classes (RateLimitError, FastAPI HTTPException(429), BaseLLMException). This commit adds the data model needed to consolidate them under a single class: * RateLimitErrorCategory enum exposing four categorical values (vendor_rate_limit, vendor_batch_rate_limit, litellm_rate_limit, litellm_batch_rate_limit) so callers can switch on the rate-limit source. * New optional fields on RateLimitError: - category (defaults to vendor_rate_limit, preserving today's behavior for every existing call site in exception_mapping_utils); - headers (preserves retry-after / rate_limit_type / reset_at across the proxy boundary instead of dropping them on the floor); - detail (mirrors FastAPI HTTPException.detail so the same instance can be serialized through both paths). litellm.RateLimitErrorCategory is re-exported at the package root to match the existing exception-export pattern. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(proxy): add ProxyRateLimitError unifying RateLimitError + HTTPException Adds a single proxy-side error class that subclasses BOTH litellm.exceptions.RateLimitError AND fastapi.HTTPException via cooperative multiple inheritance. Why both bases: * Subclassing RateLimitError lets user code catch every rate-limit source with one 'except RateLimitError' and switch on the new .category field. * Subclassing HTTPException keeps every existing FastAPI plumbing path (the isinstance(e, HTTPException) branches in proxy_server.py route handlers, FastAPI's own dispatcher, and tests asserting pytest.raises(HTTPException)) working without modification, and preserves retry-after / rate_limit_type / reset_at headers on the wire. The class declaration order is (HTTPException, RateLimitError) so the MRO puts HTTPException's no-super-call __init__ ahead of openai's cooperative __init__ chain — preventing openai.APIError.super().__init__(message) from landing in HTTPException.__init__(status_code=message). LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from budget + iteration limiters Replaces three bare HTTPException(status_code=429, ...) call sites with ProxyRateLimitError, which is both a RateLimitError (catchable by category) and an HTTPException (preserves existing FastAPI serialization). Drops the now-unused HTTPException import in the iteration / per-session limiters. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from parallel-request limiters Replaces HTTPException(status_code=429, ...) call sites in the v1 and v3 parallel-request limiters (key/team/user/model/customer rate limits) with ProxyRateLimitError. Updates the raise_rate_limit_error helper's return type annotation accordingly. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from dynamic rate limiters Replaces HTTPException(status_code=429, ...) call sites in the v1 and v3 dynamic rate limiters (project-level TPM/RPM allocation, model-saturation checks, priority-based limits, fail-closed guards) with ProxyRateLimitError. The v3 limiter still imports HTTPException for an unrelated bare 'except HTTPException:' branch. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from batch rate limiter Replaces HTTPException(status_code=429, ...) in batch_rate_limiter._raise_rate_limit_error with ProxyRateLimitError tagged as RateLimitErrorCategory.LITELLM_BATCH_RATE_LIMIT so users can distinguish batch-level throttling (which counts requests/tokens across an uploaded batch input file before submission) from the generic key/team/user RPM/TPM limiter. The HTTPException import is retained because the same module raises HTTPException for unrelated 403/IO error paths. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): pin down unified rate-limit error contract Adds a dedicated test module covering the new RateLimitErrorCategory enum, RateLimitError.category default + override behavior, ProxyRateLimitError's dual nature (RateLimitError + HTTPException), and a parametrized regression guard that asserts every proxy hook module imports the unified class. The regression guard catches the failure mode the refactor is designed to prevent: someone re-introducing a bare HTTPException(status_code=429, ...) in one of the hook modules instead of going through ProxyRateLimitError. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(logging): expose rate-limit category via StandardLoggingPayload Adds an optional 'error_rate_limit_category' field to StandardLoggingPayloadErrorInformation, populated from the unified RateLimitError.category attribute (introduced in the previous commits on this branch). Why: the .category attribute is reachable off the raw exception today via getattr(e, 'category', None), but the structured contract that downstream custom callbacks / loggers / spend log writers consume is the StandardLoggingPayload. Without this field, a user building custom rate-limit metrics on top of callback data has to special-case the raw exception object — which defeats the purpose of the StandardLoggingPayload abstraction. The field is None for non-rate-limit exceptions (so consumers can read it unconditionally without isinstance checks) and is one of the RateLimitErrorCategory string values otherwise. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): assert StandardLoggingPayload carries the category Five tests covering: vendor default, explicit litellm_rate_limit and litellm_batch_rate_limit values, None for non-rate-limit exceptions, and None when no exception is provided. Pins down the contract that custom callbacks can read 'error_information.error_rate_limit_category' off the StandardLoggingPayload to drive custom rate-limit metrics without ever reaching for the raw exception. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(types): silence mypy [misc] on intentional dual-base attr overlap mypy emits two [misc] errors on the ProxyRateLimitError class line because its two bases declare overlapping attributes with related-but-not-identical annotations: * status_code: int on starlette HTTPException vs. Literal[429] on openai's RateLimitError (every openai status-error subclass narrows it the same way and silences pyright with the same convention). * headers: Mapping[str, str] | None on HTTPException vs. our Optional[ Dict[str, str]] (the proxy hooks always carry a stringified dict). Both narrowings are intentional and enforced at construction time. Add a type: ignore[misc] with an inline explanation rather than relax the annotations on the parent or change the wire-format guarantees. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): add direct hook-invocation tests to lift patch coverage Adds six end-to-end tests that drive each refactored hook past its limit and assert the unified ProxyRateLimitError is raised with the correct category and dual-base shape. Complements the import-shape-only parametrized guard above by actually executing the new 'raise ProxyRateLimitError(...)' lines so codecov's patch coverage sees them as hit. Hooks covered (one test each): * parallel_request_limiter v1 — direct call to raise_rate_limit_error() * parallel_request_limiter v3 — direct call to _handle_rate_limit_error with a fabricated OVER_LIMIT response * max_iterations_limiter — full async_pre_call_hook with mocked agent registry, second call exceeds budget=1 * max_budget_limiter — async_pre_call_hook with mocked get_current_spend * dynamic_rate_limiter v1 — async_pre_call_hook with mocked check_available_usage forcing available_tpm == 0 * batch_rate_limiter — direct _raise_rate_limit_error call, asserts category is the batch-specific LITELLM_BATCH_RATE_LIMIT (not the generic LITELLM_RATE_LIMIT) LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix: guard rate_limit_category extraction with isinstance check * test(rate-limit): cover remaining hook raise sites for codecov Adds five more direct hook-invocation tests so every PR-touched line in the proxy hooks is exercised by tests in tests/test_litellm/, which codecov measures: * parallel_request_limiter v1 — check_key_in_limits inline raise (the second raise site, separate from the raise_rate_limit_error helper covered earlier) * dynamic_rate_limiter v1 — RPM raise branch (TPM branch was already covered) * dynamic_rate_limiter v3 — parametrized over all three raise sites: model_saturation_check, priority_model, and the fail-closed fallback for an unrecognized descriptor_key * max_budget_per_session_limiter — full async_pre_call_hook with a mocked agent registry and over-budget cached spend All 42 tests in test_rate_limit_error_unification.py now pass and together exercise every changed import + raise line across the eight refactored proxy hooks. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix: use computed error_message in ProxyRateLimitError detail * fix(parallel-request-limiter): drop None from detail; annotate raise_rate_limit_error as NoReturn The v1 ' raise_rate_limit_error' helper built an unused 'error_message' variable and then assembled the actual ' detail' via an f-string that interpolated 'additional_details' verbatim — producing 'Max parallel request limit reached None' when invoked without arguments (flagged by code review). Fix the helper to: - use the constructed 'error_message' as the detail - annotate the helper as NoReturn since it always raises - drop the redundant 'raise'/'return' at the two call sites Add two regression tests covering both the with- and without- additional_details paths. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): drop literal 'None' from raise_rate_limit_error detail The v1 parallel_request_limiter's raise_rate_limit_error helper has a long-standing bug: it computes a None-guarded 'error_message' string but then ignores it and emits an f-string that interpolates the raw 'additional_details' arg. Callers that pass no argument get 'Max parallel request limit reached None' as the user-facing detail. This commit: * wires error_message into the detail kwarg so the None-guard actually applies and operators see a clean message; * changes the return-type annotation from ProxyRateLimitError to NoReturn (the function always raises) so type-checkers know callers after this invocation are unreachable. Greptile P1 + P2 review feedback on PR BerriAI#27687. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(types): demote TypedDict floating string to a # comment A string literal placed after a field declaration in a TypedDict body is not a per-field docstring — it's an orphaned string expression Python discards. Tools like mypy / pyright that inspect TypedDict fields won't surface that text either. Move the documentation for error_rate_limit_category to a real comment so the intent is visible to readers and type-checker tooling without the misleading docstring framing. Greptile P2 review feedback on PR BerriAI#27687. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * security(exceptions): do not auto-copy vendor response headers to e.headers A vendor 429 response can set arbitrary headers (Set-Cookie, CORS overrides, …). Previously, when RateLimitError was constructed with only a 'response=' (no explicit 'headers=' kwarg), self.headers fell back to a copy of response.headers. If a downstream proxy serializer ever forwarded e.headers to the client, a malicious upstream could inject browser-interpreted headers for the proxy origin. Drop the fallback. Only headers passed explicitly via the headers= kwarg make it onto self.headers (proxy hooks pass retry-after etc. — they control what's surfaced). Vendor response headers stay reachable on e.response.headers for callers that explicitly want them. Today's proxy_server.py route handlers don't actually forward e.headers on the wire (they construct ProxyException without passing headers), so no current behavior changes — this is a defensive narrowing so the fallback can never be turned into a vector when someone wires e.headers through later. Veria-AI security review feedback on PR BerriAI#27687. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): regression guards for review-pass fixes Pins down the three review-pass fixes: * test_parallel_request_limiter_v1_helper_no_additional_details — calls raise_rate_limit_error() with no args and asserts the detail does NOT contain the literal string 'None'. Pre-fix, callers got 'Max parallel request limit reached None'. * test_rate_limit_error_does_not_auto_copy_response_headers — passes a vendor httpx.Response with a Set-Cookie header to RateLimitError WITHOUT an explicit headers= kwarg, asserts self.headers stays None (no leak), then re-checks that an explicit headers= kwarg DOES populate self.headers. Vendor headers remain reachable on e.response.headers for callers that explicitly want them. * The existing v1-helper test now also asserts the additional_details string makes it through to the detail. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(rate-limit): add orthogonal RateLimitType (requests/tokens/concurrent_requests/budget/max_iterations) trho's last ask in the LIT-2968 thread: distinguish rate-limit failures by the dimension that was exceeded, not just by who rate-limited (vendor vs. litellm). Adds: - RateLimitType str-enum exposed at `litellm.RateLimitType` with values requests / tokens / concurrent_requests / budget / max_iterations. - `rate_limit_type` kwarg on litellm.RateLimitError + ProxyRateLimitError; None default so existing callers (vendor-429 path in exception_mapping_utils) remain a no-op. - StandardLoggingPayloadErrorInformation.error_rate_limit_type so custom callbacks can split rate-limit failures by cause without parsing free-text error messages. Mirror to error_rate_limit_category extraction in get_error_information(); single isinstance(RateLimitError) check covers both. - map_v3_rate_limit_type() helper to collapse the v3 limiter's internal labels ("requests", "tokens", "max_parallel_requests") onto the public enum so the v3 limiter and dynamic_rate_limiter_v3 share one mapping. Defensive None on unknown values rather than silently picking a wrong dimension. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(proxy/hooks): wire rate_limit_type onto every limiter raise site Each refactored proxy hook now populates rate_limit_type with the dimension that actually tripped the limit, so downstream consumers (custom callbacks, prometheus exporters via the StandardLoggingPayload) can split key/team/user rate-limit failures by cause: - parallel_request_limiter (v1): detect dimension from current vs. limit in the post-cache branch (concurrent_requests > tokens > requests, matches the boolean condition order). Base case (current is None, one limit set to 0) picks the most-specific zero. raise_rate_limit_error() helper accepts an explicit rate_limit_type kwarg with CONCURRENT_REQUESTS default (matches every existing internal call site, including the global-limit branch). - parallel_request_limiter (v3): forward status["rate_limit_type"] through map_v3_rate_limit_type() so "max_parallel_requests" → CONCURRENT_REQUESTS for the public field while the raw v3 jargon stays on the HTTP header for wire-format backward compat. - dynamic_rate_limiter (v1): TPM-zero → TOKENS, RPM-zero → REQUESTS. Pass data["model"] through so callbacks see the model that hit the limit (addresses the secondary "provider missing" complaint in the original Slack thread, partially — the model is what dashboards typically split on). - dynamic_rate_limiter (v3): forward status["rate_limit_type"] via map_v3_rate_limit_type() at every raise site (model_saturation_check, priority_model, fail-closed unknown-descriptor guard). Also pass model. - batch_rate_limiter: limit_type is hard-typed "requests"|"tokens" — map directly without going through the helper's None branch. - max_budget_limiter, max_budget_per_session_limiter: BUDGET. - max_iterations_limiter: MAX_ITERATIONS. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): cover RateLimitType enum, hook wiring, and StandardLoggingPayload propagation 27 new tests across five new test classes: - TestRateLimitType: enum exposed at litellm.RateLimitType, all five values defined, RateLimitError default is None (vendor 429 path makes no claim about which dimension), accepts both string and enum forms with str-coercion guarantee for downstream JSON serializers. - TestProxyRateLimitErrorType: ProxyRateLimitError default is None, accepts string or enum, doesn't break existing callers that pass nothing. - TestMapV3RateLimitType: pins each v3-internal → public-enum mapping (tokens, requests, max_parallel_requests → concurrent_requests, unknown → None) so a future v3 refactor can't silently swap dimensions. - TestStandardLoggingPayloadCarriesType: the new error_rate_limit_type field reaches the structured payload for both ProxyRateLimitError and plain RateLimitError, is None when unspecified, and is None for non-rate-limit exceptions (symmetric with error_rate_limit_category). - TestProxyHooksWireTypeCorrectly: drives the actual raise sites in the v1 parallel_request_limiter helper, the v3 _handle_rate_limit_error (both "tokens" and "max_parallel_requests" paths), and the batch limiter (both tokens and requests paths) — coverage tools see the new rate_limit_type= kwargs as exercised, not just the import shape. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): cover _coerce_message branches and v1 dimension detection Drives the patch coverage on the new orthogonal RateLimitType wiring up to (or close to) 100% on the touched files. ProxyRateLimitError._coerce_message — was 22% covered, now 100%: * nested {error: {message}} dict * nested {message: {message}} dict (alt key) * dict without 'error'/'message' keys → JSON dump fallback * non-JSON-serializable dict value → str() fallback * non-string non-mapping detail (int) → str() coercion v1 parallel_request_limiter dimension detection — was 0% covered, now exercised across 6 parametrized cases: * check_key_in_limits else-branch: current at concurrent / TPM / RPM cap → asserts rate_limit_type is concurrent_requests / tokens / requests. * check_key_in_limits base case (current is None): max_parallel_requests / tpm_limit / rpm_limit set to 0 → asserts the most-specific zero attribution wins per the helper's order. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(proxy/hooks): add ProxyHTTPRateLimitError + provider resolver Introduces a small helper layer used by every proxy-side rate-limit hook so that the 429 they raise carries a populated llm_provider / model — instead of an empty exception.llm_provider that downstream loggers (Prometheus failure metric, observability callbacks) read as 'no provider attribution'. ProxyHTTPRateLimitError inherits from both fastapi.HTTPException (so the proxy server still renders it as a 429) and litellm.exceptions.RateLimitError (so isinstance checks and PrometheusLogger._get_exception_class_name pick up llm_provider). We deliberately don't call RateLimitError.__init__ — it constructs an httpx.Response we don't need and would just add failure surface; attribute parity is what downstream consumers care about. resolve_llm_provider_for_rate_limit() wraps litellm.get_llm_provider defensively. Internal limiter hooks fire from async_pre_call_hook — well before get_llm_provider runs anywhere else in the request lifecycle — so we have to call it ourselves at raise time. If the model is missing or unparseable (alias, router-only model) we fall back to llm_provider='litellm_proxy' rather than letting a second exception leak out and break the request path. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on parallel-request 429s Both v1 and v3 parallel-request limiters fired bare HTTPException(429) from inside async_pre_call_hook. The downstream Prometheus failure metric reads exception.llm_provider via _get_exception_class_name — the empty value showed up as exception_class='HTTPException' and left model_id='None' on the time series. Threads requested_model through every raise site in: * parallel_request_limiter.py: - check_key_in_limits (the per-key/per-model/per-user/per-team/ per-customer over-limit path) - raise_rate_limit_error (zero-limit + global_max_parallel_requests paths) — now takes an optional requested_model kwarg * parallel_request_limiter_v3.py: - _handle_rate_limit_error (the OVER_LIMIT translator), called from both the should_rate_limit pre-check and the TPM reservation path Resolved via resolve_llm_provider_for_rate_limit so unknown / missing models silently fall back to llm_provider='litellm_proxy' instead of breaking the request path with a second exception. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on dynamic-rate-limit 429s Same plumbing change as the parallel limiters, applied to both dynamic_rate_limiter (v1) and dynamic_rate_limiter_v3: * v1: TPM-zero and RPM-zero paths in async_pre_call_hook now resolve data['model'] -> (model, llm_provider) once and pass it into both raises. * v3: All three raise sites in _check_rate_limits — the model_saturation_check enforced raise, the priority_model enforced raise, and the fail-closed unknown-descriptor branch — now attribute the 429 to the actual provider. Falls back to llm_provider='litellm_proxy' when the model can't be resolved. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on batch-rate-limit 429s batch_rate_limiter._raise_rate_limit_error now takes a requested_model kwarg threaded from data['model'] in _check_and_increment_batch_counters. The batch-creation 429 is what gets raised when the input file's tokens/requests count would push the per-key TPM/RPM window over its limit. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on budget/iterations 429s Final batch of internal raise sites — the user/session-budget and max-iterations hooks. Same pattern: resolve data['model'] once at raise time, attach to ProxyHTTPRateLimitError so Prometheus and observability callbacks can attribute the 429. Hooks updated: * max_budget_limiter (per-user max_budget exceeded) * max_iterations_limiter (per-session agent iteration cap) * max_budget_per_session_limiter (per-session dollar cap) All three fall back to llm_provider='litellm_proxy' when data['model'] is missing or unparseable. Drops the now-unused HTTPException import from each module. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(proxy/hooks): pin provider field on internal rate-limit 429s Regression coverage for the 'provider field missing' bug across every proxy-side rate-limit hook + the helper layer: * ProxyHTTPRateLimitError class shape (HTTPException + RateLimitError, dict-detail stringification, None-provider normalization). * resolve_llm_provider_for_rate_limit happy paths (gpt-4o-mini, anthropic/..., bedrock/...) plus all three fallback branches (None, '', unknown name) plus a 'get_llm_provider raises' case that asserts we swallow the secondary exception. * For each limiter (parallel v1/v3, dynamic v1/v3, batch, max_budget, max_iterations, max_budget_per_session): assert the raised exception is a RateLimitError carrying the resolved model + llm_provider, and a sibling test that asserts the fallback path returns 'litellm_proxy' without leaking a second exception. * Two PrometheusLogger._get_exception_class_name pins so the Prometheus failure metric label flips from 'HTTPException' to 'Openai.ProxyHTTPRateLimitError' (or 'Litellm_proxy.*' on fallback) — that's what dashboards consume. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * perf(proxy/hooks): defer provider resolution to over-limit branches * fix: use error_message in raise_rate_limit_error to avoid literal 'None' in detail * Consolidate rate_limiter_utils imports in dynamic_rate_limiter * fix(proxy): set num_retries/max_retries on ProxyHTTPRateLimitError ProxyHTTPRateLimitError inherits from RateLimitError but did not call RateLimitError.__init__, so num_retries/max_retries were never set. When Starlette's HTTPException lacks __str__, MRO falls through to RateLimitError.__str__, which unconditionally reads these attributes and raises AttributeError during logging/traceback formatting. Initialize them to None defensively. * fix(mypy): silence base-class status_code conflict on ProxyHTTPRateLimitError HTTPException declares 'status_code: int' while openai.RateLimitError (via APIStatusError) declares 'status_code: Literal[429] = 429'. Mypy flags the multi-base override as [misc] in CI lint. The runtime semantics are fine (we set self.status_code in __init__), so silence the class-level annotation conflict with a targeted ignore. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix: annotate batch limiter _raise_rate_limit_error as NoReturn * feat(prometheus): rate-limit category/type labels + exception_class back-compat (follow-up to BerriAI#27687) (BerriAI#27706) * feat(prometheus): add rate_limit_category and rate_limit_type labels Adds two new labels to litellm_proxy_failed_requests_metric so dashboards can split 429s by rate-limit source (vendor vs. litellm-internal) and by the dimension that was exceeded (requests/tokens/concurrent_requests/ budget/max_iterations) without parsing free-text error messages. Closes the Prometheus side of LIT-2718. The unified RateLimitError.category and .rate_limit_type fields landed in PR BerriAI#27687 but were only surfaced on StandardLoggingPayload (custom-callback channel); this exposes them on the metric label set as well. Both labels are populated only when the underlying exception is a litellm.RateLimitError; non-rate-limit failures keep them empty. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(prometheus): populate rate-limit labels + preserve exception_class back-compat Two coupled changes in the Prometheus integration: 1. async_post_call_failure_hook now extracts the new RateLimitError .category / .rate_limit_type fields (added in PR BerriAI#27687) via a _extract_rate_limit_labels helper and forwards them through UserAPIKeyLabelValues onto litellm_proxy_failed_requests_metric. Empty for non-rate-limit failures. 2. _get_exception_class_name special-cases ProxyRateLimitError and keeps emitting 'HTTPException' for the exception_class label. Without this shim, ProxyRateLimitError (which multi-inherits from HTTPException + RateLimitError) would silently flip the label from 'HTTPException' (the historical value for proxy-side 429s) to 'ProxyRateLimitError', breaking existing dashboards / alerts that key off exception_class='HTTPException'. Distinguishing vendor vs. litellm 429s is now the job of the new rate_limit_category label. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(prometheus): cover rate-limit labels and exception_class back-compat Adds 19 tests across: - enum / label-list registration - _extract_rate_limit_labels for vendor RateLimitError, ProxyRateLimitError, non-rate-limit and None inputs (incl. parametrized over every RateLimitErrorCategory x RateLimitType combo) - _get_exception_class_name back-compat: ProxyRateLimitError keeps the legacy 'HTTPException' string while vendor RateLimitError keeps the historical 'Provider.ClassName' format - end-to-end through async_post_call_failure_hook with both ProxyRateLimitError and vendor RateLimitError, asserting both new labels populate and exception_class stays back-compat Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(prometheus): tolerate missing fastapi in lazy ProxyRateLimitError import Address greptile feedback: - async_post_call_failure_hook docstring: drop the stale labelnames listing and reference PrometheusMetricLabels.litellm_proxy_failed_requests_metric as the source of truth so the doc cannot drift from the actual labelset. - _get_exception_class_name: guard the lazy ProxyRateLimitError import with ImportError so router-side fallback callsites don't blow up in non-proxy installs that don't have fastapi (a transitive dep of proxy.common_utils.proxy_rate_limit_error). Behavior is unchanged when fastapi is available. Also fix the existing enterprise callback test that asserted the old labelset on litellm_proxy_failed_requests_metric — it now expects the new rate_limit_category / rate_limit_type labels populated for vendor 429s. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(bugbot): simplify rate-limit label coercion + guard None detail - prometheus.py _extract_rate_limit_labels: RateLimitError.__init__ already normalizes category/rate_limit_type to plain str, so the getattr(.value) + isinstance dance was dead code. Reduce to str(value) if not None. - proxy_rate_limit_error.py _coerce_message: short-circuit None to '' instead of falling through to str(None) = 'None', which produced the literal message 'litellm.RateLimitError: None'. * fix(rate-limit): surface unified category/type fields on BudgetExceededError The most common budget cap (virtual-key max_budget enforcement in auth_checks.py) raises litellm.BudgetExceededError, a bare Exception subclass that bypassed the unified rate-limit error class introduced by PR BerriAI#27687. Custom callbacks reading StandardLoggingPayload.error_information saw category=None and rate_limit_type=None for these 429s, missing the most common budget case (team / org / end-user budgets all hit the same code path). Surface the fields off BudgetExceededError as plain attributes: - category = RateLimitErrorCategory.LITELLM_RATE_LIMIT - rate_limit_type = RateLimitType.BUDGET - llm_provider = "" (or caller-supplied) Switch get_error_information and _extract_rate_limit_labels from isinstance(RateLimitError) gating to duck-typed attribute reads, guarded by membership in the rate-limit enums so unrelated third-party exceptions exposing a .category attribute can't leak garbage values into the payload. This is strictly additive: BudgetExceededError keeps its bare-Exception base class, so `except BudgetExceededError:` handlers keep firing and `except RateLimitError:` does not start catching budget errors. * fix(rate-limit): validate enum membership at duck-typed read sites + enrich BudgetExceededError llm_provider Two follow-ups uncovered during the second QA pass on PR BerriAI#27687: 1. Guard third-party `.category` / `.rate_limit_type` attribute leakage. The duck-typed read in `get_error_information` and `_extract_rate_limit_labels` would forward any string attribute named `category` / `rate_limit_type` on an unrelated third-party exception into the StandardLoggingPayload and Prometheus labels — silently mislabeling custom-callback payloads and blowing out Prometheus label cardinality. Add `validate_rate_limit_category` / `validate_rate_limit_type` helpers that gate on the documented enum value sets; non-matching values are dropped to None. 2. Enrich BudgetExceededError.llm_provider from request_data. Budget checks live in tenant-scoped helpers (key / team / org / tag / end-user / project) that don't see the request model, so the BudgetExceededError they raise carried llm_provider="" — leaving custom-metrics consumers without provider attribution for the most common 429 case. Resolve it once at the central UserAPIKeyAuthExceptionHandler seam, before post_call_failure_hook fires, so the StandardLoggingPayload the callback sees has the same provider attribution as RPM/TPM 429s. Regression tests pin both: 4 leakage tests + 4 enrichment tests. The leakage tests would fail under the pre-validation version of either read site; the enrichment tests would fail if the handler skipped the resolver call. * fix(rate-limit): resolve router model_name aliases to real provider (BerriAI#27914) * fix(rate-limit): resolve router model_name aliases to real provider For nearly every real LiteLLM proxy deployment the request model is a router model_name alias (e.g. 'tpm-locked' -> litellm_params.model: openai/gpt-4o-mini), and 'litellm.get_llm_provider' doesn't know about router aliases — it raises 'LLMProviderNotProvidedError'. The resolver then fell through to the defensive 'litellm_proxy' fallback, so the 'llm_provider' field this PR adds was effectively always 'litellm_proxy' in the field, defeating its purpose for the most common proxy configuration. Add a router-alias fallback step: when 'get_llm_provider' raises, scan the active 'llm_router.model_list' for a deployment whose 'model_name' matches the request model and resolve from its 'litellm_params.model' instead. If multiple deployments share the same alias (load-balancing case) the first one wins — every deployment under one alias should agree on provider in any sensible config, and 'first' is deterministic so the Prometheus label stays stable. Defensive throughout: an uninitialized router, a malformed deployment, a 'litellm_params.model' that itself fails 'get_llm_provider' — every branch falls through to the existing 'litellm_proxy' fallback rather than letting a secondary exception escape and mask the rate-limit error we're trying to surface. Tests: - test_router_alias_resolves_to_underlying_provider: alias 'tpm-locked' -> 'openai/gpt-4o-mini' produces provider='openai', model='gpt-4o-mini'. - test_router_alias_with_multiple_deployments_uses_first. - test_router_alias_unknown_falls_back. - test_router_alias_with_malformed_deployment_falls_back. - Existing fallback test updated to also stub 'litellm.proxy.proxy_server.llm_router' so it exercises the full 'no resolution anywhere' path. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(rate-limit): harden router alias resolver + test isolation - Wrap _resolve_provider_from_router_alias loop in top-level try/except so a non-iterable model_list / unexpected deployment shape can't escape and mask the 429 with a 500. - Type-check litellm_params before .get() to handle non-dict truthy values. - Patch llm_router=None in the parametrized fallback test so a router left by another test in the session can't redirect the unknown-model path. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(bugbot): preserve "BudgetExceededError" Prometheus label Adding llm_provider to BudgetExceededError (so callbacks get provider attribution from StandardLoggingPayload) made the provider-prefix step in _get_exception_class_name silently flip the label from "BudgetExceededError" to e.g. "Openai.BudgetExceededError", breaking dashboards keyed on the historical value. Short-circuit BudgetExceededError in _get_exception_class_name the same way ProxyRateLimitError already is. Provider/category attribution still lands on the new rate_limit_category / rate_limit_type labels. * test: fix invalid 'rpm' rate_limit_type in v3 limiter test mocks The v3 rate limiter only emits 'requests', 'tokens', or 'max_parallel_requests'. Using 'rpm' caused map_v3_rate_limit_type to return None, leaving the expected RateLimitType.REQUESTS untested. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(bugbot): hoist provider resolver + opt-in prom rate-limit labels - dynamic_rate_limiter.py: hoist resolve_llm_provider_for_rate_limit above the TPM/RPM if/elif so the lookup runs once per request, matching the pattern in dynamic_rate_limiter_v3.py. - prometheus.py: gate the new rate_limit_category / rate_limit_type labels on litellm_proxy_failed_requests_metric behind litellm.prometheus_emit_rate_limit_labels (default False). Mirrors the existing prometheus_emit_stream_label opt-in. Preserves the metric's pre-unification label set so existing dashboards / recording rules keep matching after upgrade; operators can enable the new labels once downstream consumers include them. - Tests updated: default-off back-compat case, opt-in path enables the flag before asserting label presence. * fix: stabilize prometheus label sets and drop redundant model normalization - Cache PrometheusLogger.get_labels_for_metric per metric_name so that the label set used to construct counters at __init__ time stays in sync with the label set used at increment time, even if module-level toggles like prometheus_emit_rate_limit_labels or prometheus_emit_stream_label are flipped at runtime. Without this, toggling these flags after the logger was created would cause ValueError from prometheus_client because the runtime labels would not match the counter's declared labelnames. - Drop redundant 'model or ""' guard in ProxyRateLimitError.__init__ where model is already normalized one step earlier. Co-authored-by: Yassin Kortam <yassin@berri.ai> * perf(dynamic_rate_limiter): only resolve provider when rate limit hit Co-authored-by: Yassin Kortam <yassin@berri.ai> * test(prometheus): clear cached metric labels after toggling rate-limit flag The PrometheusLogger caches each metric's label set at construction time so that labels used at counter.labels(...) time stay consistent with the labels the metric was registered with. The enterprise async_post_call_failure_hook test toggles litellm.prometheus_emit_rate_limit_labels = True AFTER the fixture has already built the logger, so without invalidating the cache the rate_limit_category / rate_limit_type labels never reach the mocked counter and the assert_called_once_with check fails. Co-authored-by: Yassin Kortam <yassin@berri.ai> * test: fix CI failures from prom label cache + flaky time-window assertion PrometheusLogger.get_labels_for_metric now caches the per-metric label set at first read so the labels passed to counter.labels(...) stay in lock step with the labels the counter was registered with. This broke two existing test patterns: - test_prometheus_labels.py: tests bind the real method onto a MagicMock, but MagicMock auto-creates a Mock for _cached_metric_labels whose .get(...) returns a truthy Mock — treated as a populated cache and returned as the label set, producing empty filtered labels and KeyError on labels["requested_model"] / ["route"]. Seed real {} containers for _cached_metric_labels and label_filters before binding. - test_prometheus_logging_callbacks.py::test_set_team_budget_metrics_with_custom_labels: the fixture builds the logger before the test monkeypatches litellm.custom_prometheus_metadata_labels, so the cached label set never picks up the new metadata labels. Clear the cache after the monkeypatch (same pattern already used for the rate-limit toggle in test_async_post_call_failure_hook). UI: view_logs/index.test.tsx "Last Minute" window assertion is off by one at the minute boundary. start_date is floored to the minute, so the dropped sub-minute fraction can push the truncated-seconds diff up to (minMinutes+1)*60 exactly when the click lands near a minute rollover. Switch the upper bound to toBeLessThanOrEqual. * feat(otel-v2): surface rate_limit_category + rate_limit_type on failed LLM-call spans PR BerriAI#28909 introduced the typed v2 OTel engine that builds spans from StandardLoggingPayload, with SpanError carrying error_type + message and the genai mapper stamping error.type onto every failed LLM-call span. This PR's earlier commits added error_rate_limit_category and error_rate_limit_type to the same StandardLoggingPayload.error_information the v2 engine reads — but neither field reached a span attribute, so v2 OTel traces stayed opaque about *why* a 429 fired (vendor vs litellm, RPM vs TPM vs concurrent vs budget vs max_iterations) even after the custom-callback and prometheus surfaces gained that decomposition. Three coupled changes: 1. semconv.py: add LiteLLM.ERROR_RATE_LIMIT_CATEGORY / LiteLLM.ERROR_RATE_LIMIT_TYPE under the litellm.* vendor namespace (no GenAI semconv equivalent exists for who-rate-limited / which-dimension). 2. payloads.py: extend SpanError with rate_limit_category + rate_limit_type, populated by _parse_error() from the same error_information.error_rate_limit_* fields the custom-callback channel and prometheus rate_limit_category / rate_limit_type labels read. Single source of truth across all three observability surfaces. 3. mappers/genai.py: stamp the two attributes on the LLM-call span when present. drop_none guarantees they stay absent (not 'None') for non-rate-limit failures so trace consumers can read them unconditionally. Three regression tests in test_otel_v2_emitter.py pin: a vendor / litellm-internal RateLimitError lands category=litellm_rate_limit + rate_limit_type=requests on the span; a BudgetExceededError lands rate_limit_type=budget; a non-rate-limit failure (BadRequestError) keeps the rate_limit_* attributes absent. Mutation-tested against reverting either the SpanError extension or the _parse_error read site — both new tests fail under either mutation. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test: align prometheus user-budget + logs quick-select tests with merged code The merge into this branch left two test patterns out of step with the code they exercise. test_set_user_budget_metrics_includes_user_email_and_alias_labels_when_opted_in flipped litellm.prometheus_user_budget_label_include_email_alias after the fixture had already built the PrometheusLogger. get_labels_for_metric now snapshots each metric's label set at construction time, so the runtime flip no longer reached the cached labels. Enable the flag before constructing the logger, matching how the proxy applies config at startup. view_logs/index.test.tsx referenced uiSpendLogsCall and moment without importing them, and the merged index.tsx now fetches through useLogFilterLogic (the hook the file stubs out) rather than calling uiSpendLogsCall directly. Add the imports and restore the real hook for the Quick Select window assertions so the call is actually observed. * refactor(otel/v2): drop rate-limit decomposition from the LLM-call span Proxy-side rate limits (litellm_rate_limit, budget, max_iterations) are rejected at the gate before any upstream call, so async_post_call_failure_hook tags the synthetic failure log with LITELLM_LOGGING_NO_UPSTREAM_LLM_CALL and the v2 OTel logger never opens an LLM-call span for them; the litellm.error.rate_limit_category / litellm.error.rate_limit_type attributes were dead for exactly the cases they were meant to surface. The only failure that does open an LLM-call span carrying a RateLimitError is a vendor 429, where rate_limit_type is always None and the category just restates error.type=RateLimitError. The decomposition still reaches downstream consumers through StandardLoggingPayload.error_information.error_rate_limit_* and the prometheus rate_limit_category / rate_limit_type labels, both unchanged. Removes the SpanError fields, the _parse_error reads, the genai mapper attributes, the semconv keys, and the three span tests that asserted a scenario that never reaches the mapper in production. * fix(batch_rate_limiter): map max_parallel_requests to concurrent_requests * refactor(prometheus): drop transitive fastapi import from _get_exception_class_name Read the legacy exception_class label from a prometheus_exception_class_name marker on ProxyRateLimitError instead of importing the proxy module, keeping the integrations layer free of a transitive fastapi dependency. * chore(ui): sync schema.d.ts with unified rate-limit error spec The ProxyRateLimitError docstring flows into the proxy OpenAPI spec's 429 response description, so the generated dashboard types were out of sync. Regenerated via npm run gen:api (Check UI API Types Sync). --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> Co-authored-by: Yassin Kortam <yassin@berri.ai>
factnn
pushed a commit
to factnn/litellm
that referenced
this pull request
Jun 18, 2026
…odel, and llm_provider fields (BerriAI#27687) * feat(exceptions): add RateLimitErrorCategory + headers/detail fields on RateLimitError LiteLLM previously surfaced rate-limit conditions through several unrelated error classes (RateLimitError, FastAPI HTTPException(429), BaseLLMException). This commit adds the data model needed to consolidate them under a single class: * RateLimitErrorCategory enum exposing four categorical values (vendor_rate_limit, vendor_batch_rate_limit, litellm_rate_limit, litellm_batch_rate_limit) so callers can switch on the rate-limit source. * New optional fields on RateLimitError: - category (defaults to vendor_rate_limit, preserving today's behavior for every existing call site in exception_mapping_utils); - headers (preserves retry-after / rate_limit_type / reset_at across the proxy boundary instead of dropping them on the floor); - detail (mirrors FastAPI HTTPException.detail so the same instance can be serialized through both paths). litellm.RateLimitErrorCategory is re-exported at the package root to match the existing exception-export pattern. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(proxy): add ProxyRateLimitError unifying RateLimitError + HTTPException Adds a single proxy-side error class that subclasses BOTH litellm.exceptions.RateLimitError AND fastapi.HTTPException via cooperative multiple inheritance. Why both bases: * Subclassing RateLimitError lets user code catch every rate-limit source with one 'except RateLimitError' and switch on the new .category field. * Subclassing HTTPException keeps every existing FastAPI plumbing path (the isinstance(e, HTTPException) branches in proxy_server.py route handlers, FastAPI's own dispatcher, and tests asserting pytest.raises(HTTPException)) working without modification, and preserves retry-after / rate_limit_type / reset_at headers on the wire. The class declaration order is (HTTPException, RateLimitError) so the MRO puts HTTPException's no-super-call __init__ ahead of openai's cooperative __init__ chain — preventing openai.APIError.super().__init__(message) from landing in HTTPException.__init__(status_code=message). LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from budget + iteration limiters Replaces three bare HTTPException(status_code=429, ...) call sites with ProxyRateLimitError, which is both a RateLimitError (catchable by category) and an HTTPException (preserves existing FastAPI serialization). Drops the now-unused HTTPException import in the iteration / per-session limiters. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from parallel-request limiters Replaces HTTPException(status_code=429, ...) call sites in the v1 and v3 parallel-request limiters (key/team/user/model/customer rate limits) with ProxyRateLimitError. Updates the raise_rate_limit_error helper's return type annotation accordingly. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from dynamic rate limiters Replaces HTTPException(status_code=429, ...) call sites in the v1 and v3 dynamic rate limiters (project-level TPM/RPM allocation, model-saturation checks, priority-based limits, fail-closed guards) with ProxyRateLimitError. The v3 limiter still imports HTTPException for an unrelated bare 'except HTTPException:' branch. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from batch rate limiter Replaces HTTPException(status_code=429, ...) in batch_rate_limiter._raise_rate_limit_error with ProxyRateLimitError tagged as RateLimitErrorCategory.LITELLM_BATCH_RATE_LIMIT so users can distinguish batch-level throttling (which counts requests/tokens across an uploaded batch input file before submission) from the generic key/team/user RPM/TPM limiter. The HTTPException import is retained because the same module raises HTTPException for unrelated 403/IO error paths. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): pin down unified rate-limit error contract Adds a dedicated test module covering the new RateLimitErrorCategory enum, RateLimitError.category default + override behavior, ProxyRateLimitError's dual nature (RateLimitError + HTTPException), and a parametrized regression guard that asserts every proxy hook module imports the unified class. The regression guard catches the failure mode the refactor is designed to prevent: someone re-introducing a bare HTTPException(status_code=429, ...) in one of the hook modules instead of going through ProxyRateLimitError. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(logging): expose rate-limit category via StandardLoggingPayload Adds an optional 'error_rate_limit_category' field to StandardLoggingPayloadErrorInformation, populated from the unified RateLimitError.category attribute (introduced in the previous commits on this branch). Why: the .category attribute is reachable off the raw exception today via getattr(e, 'category', None), but the structured contract that downstream custom callbacks / loggers / spend log writers consume is the StandardLoggingPayload. Without this field, a user building custom rate-limit metrics on top of callback data has to special-case the raw exception object — which defeats the purpose of the StandardLoggingPayload abstraction. The field is None for non-rate-limit exceptions (so consumers can read it unconditionally without isinstance checks) and is one of the RateLimitErrorCategory string values otherwise. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): assert StandardLoggingPayload carries the category Five tests covering: vendor default, explicit litellm_rate_limit and litellm_batch_rate_limit values, None for non-rate-limit exceptions, and None when no exception is provided. Pins down the contract that custom callbacks can read 'error_information.error_rate_limit_category' off the StandardLoggingPayload to drive custom rate-limit metrics without ever reaching for the raw exception. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(types): silence mypy [misc] on intentional dual-base attr overlap mypy emits two [misc] errors on the ProxyRateLimitError class line because its two bases declare overlapping attributes with related-but-not-identical annotations: * status_code: int on starlette HTTPException vs. Literal[429] on openai's RateLimitError (every openai status-error subclass narrows it the same way and silences pyright with the same convention). * headers: Mapping[str, str] | None on HTTPException vs. our Optional[ Dict[str, str]] (the proxy hooks always carry a stringified dict). Both narrowings are intentional and enforced at construction time. Add a type: ignore[misc] with an inline explanation rather than relax the annotations on the parent or change the wire-format guarantees. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): add direct hook-invocation tests to lift patch coverage Adds six end-to-end tests that drive each refactored hook past its limit and assert the unified ProxyRateLimitError is raised with the correct category and dual-base shape. Complements the import-shape-only parametrized guard above by actually executing the new 'raise ProxyRateLimitError(...)' lines so codecov's patch coverage sees them as hit. Hooks covered (one test each): * parallel_request_limiter v1 — direct call to raise_rate_limit_error() * parallel_request_limiter v3 — direct call to _handle_rate_limit_error with a fabricated OVER_LIMIT response * max_iterations_limiter — full async_pre_call_hook with mocked agent registry, second call exceeds budget=1 * max_budget_limiter — async_pre_call_hook with mocked get_current_spend * dynamic_rate_limiter v1 — async_pre_call_hook with mocked check_available_usage forcing available_tpm == 0 * batch_rate_limiter — direct _raise_rate_limit_error call, asserts category is the batch-specific LITELLM_BATCH_RATE_LIMIT (not the generic LITELLM_RATE_LIMIT) LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix: guard rate_limit_category extraction with isinstance check * test(rate-limit): cover remaining hook raise sites for codecov Adds five more direct hook-invocation tests so every PR-touched line in the proxy hooks is exercised by tests in tests/test_litellm/, which codecov measures: * parallel_request_limiter v1 — check_key_in_limits inline raise (the second raise site, separate from the raise_rate_limit_error helper covered earlier) * dynamic_rate_limiter v1 — RPM raise branch (TPM branch was already covered) * dynamic_rate_limiter v3 — parametrized over all three raise sites: model_saturation_check, priority_model, and the fail-closed fallback for an unrecognized descriptor_key * max_budget_per_session_limiter — full async_pre_call_hook with a mocked agent registry and over-budget cached spend All 42 tests in test_rate_limit_error_unification.py now pass and together exercise every changed import + raise line across the eight refactored proxy hooks. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix: use computed error_message in ProxyRateLimitError detail * fix(parallel-request-limiter): drop None from detail; annotate raise_rate_limit_error as NoReturn The v1 ' raise_rate_limit_error' helper built an unused 'error_message' variable and then assembled the actual ' detail' via an f-string that interpolated 'additional_details' verbatim — producing 'Max parallel request limit reached None' when invoked without arguments (flagged by code review). Fix the helper to: - use the constructed 'error_message' as the detail - annotate the helper as NoReturn since it always raises - drop the redundant 'raise'/'return' at the two call sites Add two regression tests covering both the with- and without- additional_details paths. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): drop literal 'None' from raise_rate_limit_error detail The v1 parallel_request_limiter's raise_rate_limit_error helper has a long-standing bug: it computes a None-guarded 'error_message' string but then ignores it and emits an f-string that interpolates the raw 'additional_details' arg. Callers that pass no argument get 'Max parallel request limit reached None' as the user-facing detail. This commit: * wires error_message into the detail kwarg so the None-guard actually applies and operators see a clean message; * changes the return-type annotation from ProxyRateLimitError to NoReturn (the function always raises) so type-checkers know callers after this invocation are unreachable. Greptile P1 + P2 review feedback on PR BerriAI#27687. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(types): demote TypedDict floating string to a # comment A string literal placed after a field declaration in a TypedDict body is not a per-field docstring — it's an orphaned string expression Python discards. Tools like mypy / pyright that inspect TypedDict fields won't surface that text either. Move the documentation for error_rate_limit_category to a real comment so the intent is visible to readers and type-checker tooling without the misleading docstring framing. Greptile P2 review feedback on PR BerriAI#27687. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * security(exceptions): do not auto-copy vendor response headers to e.headers A vendor 429 response can set arbitrary headers (Set-Cookie, CORS overrides, …). Previously, when RateLimitError was constructed with only a 'response=' (no explicit 'headers=' kwarg), self.headers fell back to a copy of response.headers. If a downstream proxy serializer ever forwarded e.headers to the client, a malicious upstream could inject browser-interpreted headers for the proxy origin. Drop the fallback. Only headers passed explicitly via the headers= kwarg make it onto self.headers (proxy hooks pass retry-after etc. — they control what's surfaced). Vendor response headers stay reachable on e.response.headers for callers that explicitly want them. Today's proxy_server.py route handlers don't actually forward e.headers on the wire (they construct ProxyException without passing headers), so no current behavior changes — this is a defensive narrowing so the fallback can never be turned into a vector when someone wires e.headers through later. Veria-AI security review feedback on PR BerriAI#27687. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): regression guards for review-pass fixes Pins down the three review-pass fixes: * test_parallel_request_limiter_v1_helper_no_additional_details — calls raise_rate_limit_error() with no args and asserts the detail does NOT contain the literal string 'None'. Pre-fix, callers got 'Max parallel request limit reached None'. * test_rate_limit_error_does_not_auto_copy_response_headers — passes a vendor httpx.Response with a Set-Cookie header to RateLimitError WITHOUT an explicit headers= kwarg, asserts self.headers stays None (no leak), then re-checks that an explicit headers= kwarg DOES populate self.headers. Vendor headers remain reachable on e.response.headers for callers that explicitly want them. * The existing v1-helper test now also asserts the additional_details string makes it through to the detail. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(rate-limit): add orthogonal RateLimitType (requests/tokens/concurrent_requests/budget/max_iterations) trho's last ask in the LIT-2968 thread: distinguish rate-limit failures by the dimension that was exceeded, not just by who rate-limited (vendor vs. litellm). Adds: - RateLimitType str-enum exposed at `litellm.RateLimitType` with values requests / tokens / concurrent_requests / budget / max_iterations. - `rate_limit_type` kwarg on litellm.RateLimitError + ProxyRateLimitError; None default so existing callers (vendor-429 path in exception_mapping_utils) remain a no-op. - StandardLoggingPayloadErrorInformation.error_rate_limit_type so custom callbacks can split rate-limit failures by cause without parsing free-text error messages. Mirror to error_rate_limit_category extraction in get_error_information(); single isinstance(RateLimitError) check covers both. - map_v3_rate_limit_type() helper to collapse the v3 limiter's internal labels ("requests", "tokens", "max_parallel_requests") onto the public enum so the v3 limiter and dynamic_rate_limiter_v3 share one mapping. Defensive None on unknown values rather than silently picking a wrong dimension. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(proxy/hooks): wire rate_limit_type onto every limiter raise site Each refactored proxy hook now populates rate_limit_type with the dimension that actually tripped the limit, so downstream consumers (custom callbacks, prometheus exporters via the StandardLoggingPayload) can split key/team/user rate-limit failures by cause: - parallel_request_limiter (v1): detect dimension from current vs. limit in the post-cache branch (concurrent_requests > tokens > requests, matches the boolean condition order). Base case (current is None, one limit set to 0) picks the most-specific zero. raise_rate_limit_error() helper accepts an explicit rate_limit_type kwarg with CONCURRENT_REQUESTS default (matches every existing internal call site, including the global-limit branch). - parallel_request_limiter (v3): forward status["rate_limit_type"] through map_v3_rate_limit_type() so "max_parallel_requests" → CONCURRENT_REQUESTS for the public field while the raw v3 jargon stays on the HTTP header for wire-format backward compat. - dynamic_rate_limiter (v1): TPM-zero → TOKENS, RPM-zero → REQUESTS. Pass data["model"] through so callbacks see the model that hit the limit (addresses the secondary "provider missing" complaint in the original Slack thread, partially — the model is what dashboards typically split on). - dynamic_rate_limiter (v3): forward status["rate_limit_type"] via map_v3_rate_limit_type() at every raise site (model_saturation_check, priority_model, fail-closed unknown-descriptor guard). Also pass model. - batch_rate_limiter: limit_type is hard-typed "requests"|"tokens" — map directly without going through the helper's None branch. - max_budget_limiter, max_budget_per_session_limiter: BUDGET. - max_iterations_limiter: MAX_ITERATIONS. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): cover RateLimitType enum, hook wiring, and StandardLoggingPayload propagation 27 new tests across five new test classes: - TestRateLimitType: enum exposed at litellm.RateLimitType, all five values defined, RateLimitError default is None (vendor 429 path makes no claim about which dimension), accepts both string and enum forms with str-coercion guarantee for downstream JSON serializers. - TestProxyRateLimitErrorType: ProxyRateLimitError default is None, accepts string or enum, doesn't break existing callers that pass nothing. - TestMapV3RateLimitType: pins each v3-internal → public-enum mapping (tokens, requests, max_parallel_requests → concurrent_requests, unknown → None) so a future v3 refactor can't silently swap dimensions. - TestStandardLoggingPayloadCarriesType: the new error_rate_limit_type field reaches the structured payload for both ProxyRateLimitError and plain RateLimitError, is None when unspecified, and is None for non-rate-limit exceptions (symmetric with error_rate_limit_category). - TestProxyHooksWireTypeCorrectly: drives the actual raise sites in the v1 parallel_request_limiter helper, the v3 _handle_rate_limit_error (both "tokens" and "max_parallel_requests" paths), and the batch limiter (both tokens and requests paths) — coverage tools see the new rate_limit_type= kwargs as exercised, not just the import shape. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): cover _coerce_message branches and v1 dimension detection Drives the patch coverage on the new orthogonal RateLimitType wiring up to (or close to) 100% on the touched files. ProxyRateLimitError._coerce_message — was 22% covered, now 100%: * nested {error: {message}} dict * nested {message: {message}} dict (alt key) * dict without 'error'/'message' keys → JSON dump fallback * non-JSON-serializable dict value → str() fallback * non-string non-mapping detail (int) → str() coercion v1 parallel_request_limiter dimension detection — was 0% covered, now exercised across 6 parametrized cases: * check_key_in_limits else-branch: current at concurrent / TPM / RPM cap → asserts rate_limit_type is concurrent_requests / tokens / requests. * check_key_in_limits base case (current is None): max_parallel_requests / tpm_limit / rpm_limit set to 0 → asserts the most-specific zero attribution wins per the helper's order. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(proxy/hooks): add ProxyHTTPRateLimitError + provider resolver Introduces a small helper layer used by every proxy-side rate-limit hook so that the 429 they raise carries a populated llm_provider / model — instead of an empty exception.llm_provider that downstream loggers (Prometheus failure metric, observability callbacks) read as 'no provider attribution'. ProxyHTTPRateLimitError inherits from both fastapi.HTTPException (so the proxy server still renders it as a 429) and litellm.exceptions.RateLimitError (so isinstance checks and PrometheusLogger._get_exception_class_name pick up llm_provider). We deliberately don't call RateLimitError.__init__ — it constructs an httpx.Response we don't need and would just add failure surface; attribute parity is what downstream consumers care about. resolve_llm_provider_for_rate_limit() wraps litellm.get_llm_provider defensively. Internal limiter hooks fire from async_pre_call_hook — well before get_llm_provider runs anywhere else in the request lifecycle — so we have to call it ourselves at raise time. If the model is missing or unparseable (alias, router-only model) we fall back to llm_provider='litellm_proxy' rather than letting a second exception leak out and break the request path. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on parallel-request 429s Both v1 and v3 parallel-request limiters fired bare HTTPException(429) from inside async_pre_call_hook. The downstream Prometheus failure metric reads exception.llm_provider via _get_exception_class_name — the empty value showed up as exception_class='HTTPException' and left model_id='None' on the time series. Threads requested_model through every raise site in: * parallel_request_limiter.py: - check_key_in_limits (the per-key/per-model/per-user/per-team/ per-customer over-limit path) - raise_rate_limit_error (zero-limit + global_max_parallel_requests paths) — now takes an optional requested_model kwarg * parallel_request_limiter_v3.py: - _handle_rate_limit_error (the OVER_LIMIT translator), called from both the should_rate_limit pre-check and the TPM reservation path Resolved via resolve_llm_provider_for_rate_limit so unknown / missing models silently fall back to llm_provider='litellm_proxy' instead of breaking the request path with a second exception. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on dynamic-rate-limit 429s Same plumbing change as the parallel limiters, applied to both dynamic_rate_limiter (v1) and dynamic_rate_limiter_v3: * v1: TPM-zero and RPM-zero paths in async_pre_call_hook now resolve data['model'] -> (model, llm_provider) once and pass it into both raises. * v3: All three raise sites in _check_rate_limits — the model_saturation_check enforced raise, the priority_model enforced raise, and the fail-closed unknown-descriptor branch — now attribute the 429 to the actual provider. Falls back to llm_provider='litellm_proxy' when the model can't be resolved. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on batch-rate-limit 429s batch_rate_limiter._raise_rate_limit_error now takes a requested_model kwarg threaded from data['model'] in _check_and_increment_batch_counters. The batch-creation 429 is what gets raised when the input file's tokens/requests count would push the per-key TPM/RPM window over its limit. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on budget/iterations 429s Final batch of internal raise sites — the user/session-budget and max-iterations hooks. Same pattern: resolve data['model'] once at raise time, attach to ProxyHTTPRateLimitError so Prometheus and observability callbacks can attribute the 429. Hooks updated: * max_budget_limiter (per-user max_budget exceeded) * max_iterations_limiter (per-session agent iteration cap) * max_budget_per_session_limiter (per-session dollar cap) All three fall back to llm_provider='litellm_proxy' when data['model'] is missing or unparseable. Drops the now-unused HTTPException import from each module. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(proxy/hooks): pin provider field on internal rate-limit 429s Regression coverage for the 'provider field missing' bug across every proxy-side rate-limit hook + the helper layer: * ProxyHTTPRateLimitError class shape (HTTPException + RateLimitError, dict-detail stringification, None-provider normalization). * resolve_llm_provider_for_rate_limit happy paths (gpt-4o-mini, anthropic/..., bedrock/...) plus all three fallback branches (None, '', unknown name) plus a 'get_llm_provider raises' case that asserts we swallow the secondary exception. * For each limiter (parallel v1/v3, dynamic v1/v3, batch, max_budget, max_iterations, max_budget_per_session): assert the raised exception is a RateLimitError carrying the resolved model + llm_provider, and a sibling test that asserts the fallback path returns 'litellm_proxy' without leaking a second exception. * Two PrometheusLogger._get_exception_class_name pins so the Prometheus failure metric label flips from 'HTTPException' to 'Openai.ProxyHTTPRateLimitError' (or 'Litellm_proxy.*' on fallback) — that's what dashboards consume. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * perf(proxy/hooks): defer provider resolution to over-limit branches * fix: use error_message in raise_rate_limit_error to avoid literal 'None' in detail * Consolidate rate_limiter_utils imports in dynamic_rate_limiter * fix(proxy): set num_retries/max_retries on ProxyHTTPRateLimitError ProxyHTTPRateLimitError inherits from RateLimitError but did not call RateLimitError.__init__, so num_retries/max_retries were never set. When Starlette's HTTPException lacks __str__, MRO falls through to RateLimitError.__str__, which unconditionally reads these attributes and raises AttributeError during logging/traceback formatting. Initialize them to None defensively. * fix(mypy): silence base-class status_code conflict on ProxyHTTPRateLimitError HTTPException declares 'status_code: int' while openai.RateLimitError (via APIStatusError) declares 'status_code: Literal[429] = 429'. Mypy flags the multi-base override as [misc] in CI lint. The runtime semantics are fine (we set self.status_code in __init__), so silence the class-level annotation conflict with a targeted ignore. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix: annotate batch limiter _raise_rate_limit_error as NoReturn * feat(prometheus): rate-limit category/type labels + exception_class back-compat (follow-up to BerriAI#27687) (BerriAI#27706) * feat(prometheus): add rate_limit_category and rate_limit_type labels Adds two new labels to litellm_proxy_failed_requests_metric so dashboards can split 429s by rate-limit source (vendor vs. litellm-internal) and by the dimension that was exceeded (requests/tokens/concurrent_requests/ budget/max_iterations) without parsing free-text error messages. Closes the Prometheus side of LIT-2718. The unified RateLimitError.category and .rate_limit_type fields landed in PR BerriAI#27687 but were only surfaced on StandardLoggingPayload (custom-callback channel); this exposes them on the metric label set as well. Both labels are populated only when the underlying exception is a litellm.RateLimitError; non-rate-limit failures keep them empty. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(prometheus): populate rate-limit labels + preserve exception_class back-compat Two coupled changes in the Prometheus integration: 1. async_post_call_failure_hook now extracts the new RateLimitError .category / .rate_limit_type fields (added in PR BerriAI#27687) via a _extract_rate_limit_labels helper and forwards them through UserAPIKeyLabelValues onto litellm_proxy_failed_requests_metric. Empty for non-rate-limit failures. 2. _get_exception_class_name special-cases ProxyRateLimitError and keeps emitting 'HTTPException' for the exception_class label. Without this shim, ProxyRateLimitError (which multi-inherits from HTTPException + RateLimitError) would silently flip the label from 'HTTPException' (the historical value for proxy-side 429s) to 'ProxyRateLimitError', breaking existing dashboards / alerts that key off exception_class='HTTPException'. Distinguishing vendor vs. litellm 429s is now the job of the new rate_limit_category label. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(prometheus): cover rate-limit labels and exception_class back-compat Adds 19 tests across: - enum / label-list registration - _extract_rate_limit_labels for vendor RateLimitError, ProxyRateLimitError, non-rate-limit and None inputs (incl. parametrized over every RateLimitErrorCategory x RateLimitType combo) - _get_exception_class_name back-compat: ProxyRateLimitError keeps the legacy 'HTTPException' string while vendor RateLimitError keeps the historical 'Provider.ClassName' format - end-to-end through async_post_call_failure_hook with both ProxyRateLimitError and vendor RateLimitError, asserting both new labels populate and exception_class stays back-compat Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(prometheus): tolerate missing fastapi in lazy ProxyRateLimitError import Address greptile feedback: - async_post_call_failure_hook docstring: drop the stale labelnames listing and reference PrometheusMetricLabels.litellm_proxy_failed_requests_metric as the source of truth so the doc cannot drift from the actual labelset. - _get_exception_class_name: guard the lazy ProxyRateLimitError import with ImportError so router-side fallback callsites don't blow up in non-proxy installs that don't have fastapi (a transitive dep of proxy.common_utils.proxy_rate_limit_error). Behavior is unchanged when fastapi is available. Also fix the existing enterprise callback test that asserted the old labelset on litellm_proxy_failed_requests_metric — it now expects the new rate_limit_category / rate_limit_type labels populated for vendor 429s. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(bugbot): simplify rate-limit label coercion + guard None detail - prometheus.py _extract_rate_limit_labels: RateLimitError.__init__ already normalizes category/rate_limit_type to plain str, so the getattr(.value) + isinstance dance was dead code. Reduce to str(value) if not None. - proxy_rate_limit_error.py _coerce_message: short-circuit None to '' instead of falling through to str(None) = 'None', which produced the literal message 'litellm.RateLimitError: None'. * fix(rate-limit): surface unified category/type fields on BudgetExceededError The most common budget cap (virtual-key max_budget enforcement in auth_checks.py) raises litellm.BudgetExceededError, a bare Exception subclass that bypassed the unified rate-limit error class introduced by PR BerriAI#27687. Custom callbacks reading StandardLoggingPayload.error_information saw category=None and rate_limit_type=None for these 429s, missing the most common budget case (team / org / end-user budgets all hit the same code path). Surface the fields off BudgetExceededError as plain attributes: - category = RateLimitErrorCategory.LITELLM_RATE_LIMIT - rate_limit_type = RateLimitType.BUDGET - llm_provider = "" (or caller-supplied) Switch get_error_information and _extract_rate_limit_labels from isinstance(RateLimitError) gating to duck-typed attribute reads, guarded by membership in the rate-limit enums so unrelated third-party exceptions exposing a .category attribute can't leak garbage values into the payload. This is strictly additive: BudgetExceededError keeps its bare-Exception base class, so `except BudgetExceededError:` handlers keep firing and `except RateLimitError:` does not start catching budget errors. * fix(rate-limit): validate enum membership at duck-typed read sites + enrich BudgetExceededError llm_provider Two follow-ups uncovered during the second QA pass on PR BerriAI#27687: 1. Guard third-party `.category` / `.rate_limit_type` attribute leakage. The duck-typed read in `get_error_information` and `_extract_rate_limit_labels` would forward any string attribute named `category` / `rate_limit_type` on an unrelated third-party exception into the StandardLoggingPayload and Prometheus labels — silently mislabeling custom-callback payloads and blowing out Prometheus label cardinality. Add `validate_rate_limit_category` / `validate_rate_limit_type` helpers that gate on the documented enum value sets; non-matching values are dropped to None. 2. Enrich BudgetExceededError.llm_provider from request_data. Budget checks live in tenant-scoped helpers (key / team / org / tag / end-user / project) that don't see the request model, so the BudgetExceededError they raise carried llm_provider="" — leaving custom-metrics consumers without provider attribution for the most common 429 case. Resolve it once at the central UserAPIKeyAuthExceptionHandler seam, before post_call_failure_hook fires, so the StandardLoggingPayload the callback sees has the same provider attribution as RPM/TPM 429s. Regression tests pin both: 4 leakage tests + 4 enrichment tests. The leakage tests would fail under the pre-validation version of either read site; the enrichment tests would fail if the handler skipped the resolver call. * fix(rate-limit): resolve router model_name aliases to real provider (BerriAI#27914) * fix(rate-limit): resolve router model_name aliases to real provider For nearly every real LiteLLM proxy deployment the request model is a router model_name alias (e.g. 'tpm-locked' -> litellm_params.model: openai/gpt-4o-mini), and 'litellm.get_llm_provider' doesn't know about router aliases — it raises 'LLMProviderNotProvidedError'. The resolver then fell through to the defensive 'litellm_proxy' fallback, so the 'llm_provider' field this PR adds was effectively always 'litellm_proxy' in the field, defeating its purpose for the most common proxy configuration. Add a router-alias fallback step: when 'get_llm_provider' raises, scan the active 'llm_router.model_list' for a deployment whose 'model_name' matches the request model and resolve from its 'litellm_params.model' instead. If multiple deployments share the same alias (load-balancing case) the first one wins — every deployment under one alias should agree on provider in any sensible config, and 'first' is deterministic so the Prometheus label stays stable. Defensive throughout: an uninitialized router, a malformed deployment, a 'litellm_params.model' that itself fails 'get_llm_provider' — every branch falls through to the existing 'litellm_proxy' fallback rather than letting a secondary exception escape and mask the rate-limit error we're trying to surface. Tests: - test_router_alias_resolves_to_underlying_provider: alias 'tpm-locked' -> 'openai/gpt-4o-mini' produces provider='openai', model='gpt-4o-mini'. - test_router_alias_with_multiple_deployments_uses_first. - test_router_alias_unknown_falls_back. - test_router_alias_with_malformed_deployment_falls_back. - Existing fallback test updated to also stub 'litellm.proxy.proxy_server.llm_router' so it exercises the full 'no resolution anywhere' path. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(rate-limit): harden router alias resolver + test isolation - Wrap _resolve_provider_from_router_alias loop in top-level try/except so a non-iterable model_list / unexpected deployment shape can't escape and mask the 429 with a 500. - Type-check litellm_params before .get() to handle non-dict truthy values. - Patch llm_router=None in the parametrized fallback test so a router left by another test in the session can't redirect the unknown-model path. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(bugbot): preserve "BudgetExceededError" Prometheus label Adding llm_provider to BudgetExceededError (so callbacks get provider attribution from StandardLoggingPayload) made the provider-prefix step in _get_exception_class_name silently flip the label from "BudgetExceededError" to e.g. "Openai.BudgetExceededError", breaking dashboards keyed on the historical value. Short-circuit BudgetExceededError in _get_exception_class_name the same way ProxyRateLimitError already is. Provider/category attribution still lands on the new rate_limit_category / rate_limit_type labels. * test: fix invalid 'rpm' rate_limit_type in v3 limiter test mocks The v3 rate limiter only emits 'requests', 'tokens', or 'max_parallel_requests'. Using 'rpm' caused map_v3_rate_limit_type to return None, leaving the expected RateLimitType.REQUESTS untested. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(bugbot): hoist provider resolver + opt-in prom rate-limit labels - dynamic_rate_limiter.py: hoist resolve_llm_provider_for_rate_limit above the TPM/RPM if/elif so the lookup runs once per request, matching the pattern in dynamic_rate_limiter_v3.py. - prometheus.py: gate the new rate_limit_category / rate_limit_type labels on litellm_proxy_failed_requests_metric behind litellm.prometheus_emit_rate_limit_labels (default False). Mirrors the existing prometheus_emit_stream_label opt-in. Preserves the metric's pre-unification label set so existing dashboards / recording rules keep matching after upgrade; operators can enable the new labels once downstream consumers include them. - Tests updated: default-off back-compat case, opt-in path enables the flag before asserting label presence. * fix: stabilize prometheus label sets and drop redundant model normalization - Cache PrometheusLogger.get_labels_for_metric per metric_name so that the label set used to construct counters at __init__ time stays in sync with the label set used at increment time, even if module-level toggles like prometheus_emit_rate_limit_labels or prometheus_emit_stream_label are flipped at runtime. Without this, toggling these flags after the logger was created would cause ValueError from prometheus_client because the runtime labels would not match the counter's declared labelnames. - Drop redundant 'model or ""' guard in ProxyRateLimitError.__init__ where model is already normalized one step earlier. Co-authored-by: Yassin Kortam <yassin@berri.ai> * perf(dynamic_rate_limiter): only resolve provider when rate limit hit Co-authored-by: Yassin Kortam <yassin@berri.ai> * test(prometheus): clear cached metric labels after toggling rate-limit flag The PrometheusLogger caches each metric's label set at construction time so that labels used at counter.labels(...) time stay consistent with the labels the metric was registered with. The enterprise async_post_call_failure_hook test toggles litellm.prometheus_emit_rate_limit_labels = True AFTER the fixture has already built the logger, so without invalidating the cache the rate_limit_category / rate_limit_type labels never reach the mocked counter and the assert_called_once_with check fails. Co-authored-by: Yassin Kortam <yassin@berri.ai> * test: fix CI failures from prom label cache + flaky time-window assertion PrometheusLogger.get_labels_for_metric now caches the per-metric label set at first read so the labels passed to counter.labels(...) stay in lock step with the labels the counter was registered with. This broke two existing test patterns: - test_prometheus_labels.py: tests bind the real method onto a MagicMock, but MagicMock auto-creates a Mock for _cached_metric_labels whose .get(...) returns a truthy Mock — treated as a populated cache and returned as the label set, producing empty filtered labels and KeyError on labels["requested_model"] / ["route"]. Seed real {} containers for _cached_metric_labels and label_filters before binding. - test_prometheus_logging_callbacks.py::test_set_team_budget_metrics_with_custom_labels: the fixture builds the logger before the test monkeypatches litellm.custom_prometheus_metadata_labels, so the cached label set never picks up the new metadata labels. Clear the cache after the monkeypatch (same pattern already used for the rate-limit toggle in test_async_post_call_failure_hook). UI: view_logs/index.test.tsx "Last Minute" window assertion is off by one at the minute boundary. start_date is floored to the minute, so the dropped sub-minute fraction can push the truncated-seconds diff up to (minMinutes+1)*60 exactly when the click lands near a minute rollover. Switch the upper bound to toBeLessThanOrEqual. * feat(otel-v2): surface rate_limit_category + rate_limit_type on failed LLM-call spans PR BerriAI#28909 introduced the typed v2 OTel engine that builds spans from StandardLoggingPayload, with SpanError carrying error_type + message and the genai mapper stamping error.type onto every failed LLM-call span. This PR's earlier commits added error_rate_limit_category and error_rate_limit_type to the same StandardLoggingPayload.error_information the v2 engine reads — but neither field reached a span attribute, so v2 OTel traces stayed opaque about *why* a 429 fired (vendor vs litellm, RPM vs TPM vs concurrent vs budget vs max_iterations) even after the custom-callback and prometheus surfaces gained that decomposition. Three coupled changes: 1. semconv.py: add LiteLLM.ERROR_RATE_LIMIT_CATEGORY / LiteLLM.ERROR_RATE_LIMIT_TYPE under the litellm.* vendor namespace (no GenAI semconv equivalent exists for who-rate-limited / which-dimension). 2. payloads.py: extend SpanError with rate_limit_category + rate_limit_type, populated by _parse_error() from the same error_information.error_rate_limit_* fields the custom-callback channel and prometheus rate_limit_category / rate_limit_type labels read. Single source of truth across all three observability surfaces. 3. mappers/genai.py: stamp the two attributes on the LLM-call span when present. drop_none guarantees they stay absent (not 'None') for non-rate-limit failures so trace consumers can read them unconditionally. Three regression tests in test_otel_v2_emitter.py pin: a vendor / litellm-internal RateLimitError lands category=litellm_rate_limit + rate_limit_type=requests on the span; a BudgetExceededError lands rate_limit_type=budget; a non-rate-limit failure (BadRequestError) keeps the rate_limit_* attributes absent. Mutation-tested against reverting either the SpanError extension or the _parse_error read site — both new tests fail under either mutation. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test: align prometheus user-budget + logs quick-select tests with merged code The merge into this branch left two test patterns out of step with the code they exercise. test_set_user_budget_metrics_includes_user_email_and_alias_labels_when_opted_in flipped litellm.prometheus_user_budget_label_include_email_alias after the fixture had already built the PrometheusLogger. get_labels_for_metric now snapshots each metric's label set at construction time, so the runtime flip no longer reached the cached labels. Enable the flag before constructing the logger, matching how the proxy applies config at startup. view_logs/index.test.tsx referenced uiSpendLogsCall and moment without importing them, and the merged index.tsx now fetches through useLogFilterLogic (the hook the file stubs out) rather than calling uiSpendLogsCall directly. Add the imports and restore the real hook for the Quick Select window assertions so the call is actually observed. * refactor(otel/v2): drop rate-limit decomposition from the LLM-call span Proxy-side rate limits (litellm_rate_limit, budget, max_iterations) are rejected at the gate before any upstream call, so async_post_call_failure_hook tags the synthetic failure log with LITELLM_LOGGING_NO_UPSTREAM_LLM_CALL and the v2 OTel logger never opens an LLM-call span for them; the litellm.error.rate_limit_category / litellm.error.rate_limit_type attributes were dead for exactly the cases they were meant to surface. The only failure that does open an LLM-call span carrying a RateLimitError is a vendor 429, where rate_limit_type is always None and the category just restates error.type=RateLimitError. The decomposition still reaches downstream consumers through StandardLoggingPayload.error_information.error_rate_limit_* and the prometheus rate_limit_category / rate_limit_type labels, both unchanged. Removes the SpanError fields, the _parse_error reads, the genai mapper attributes, the semconv keys, and the three span tests that asserted a scenario that never reaches the mapper in production. * fix(batch_rate_limiter): map max_parallel_requests to concurrent_requests * refactor(prometheus): drop transitive fastapi import from _get_exception_class_name Read the legacy exception_class label from a prometheus_exception_class_name marker on ProxyRateLimitError instead of importing the proxy module, keeping the integrations layer free of a transitive fastapi dependency. * chore(ui): sync schema.d.ts with unified rate-limit error spec The ProxyRateLimitError docstring flows into the proxy OpenAPI spec's 429 response description, so the generated dashboard types were out of sync. Regenerated via npm run gen:api (Check UI API Types Sync). --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> Co-authored-by: Yassin Kortam <yassin@berri.ai>
fzowl
pushed a commit
to fzowl/litellm
that referenced
this pull request
Jun 24, 2026
…odel, and llm_provider fields (BerriAI#27687) * feat(exceptions): add RateLimitErrorCategory + headers/detail fields on RateLimitError LiteLLM previously surfaced rate-limit conditions through several unrelated error classes (RateLimitError, FastAPI HTTPException(429), BaseLLMException). This commit adds the data model needed to consolidate them under a single class: * RateLimitErrorCategory enum exposing four categorical values (vendor_rate_limit, vendor_batch_rate_limit, litellm_rate_limit, litellm_batch_rate_limit) so callers can switch on the rate-limit source. * New optional fields on RateLimitError: - category (defaults to vendor_rate_limit, preserving today's behavior for every existing call site in exception_mapping_utils); - headers (preserves retry-after / rate_limit_type / reset_at across the proxy boundary instead of dropping them on the floor); - detail (mirrors FastAPI HTTPException.detail so the same instance can be serialized through both paths). litellm.RateLimitErrorCategory is re-exported at the package root to match the existing exception-export pattern. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(proxy): add ProxyRateLimitError unifying RateLimitError + HTTPException Adds a single proxy-side error class that subclasses BOTH litellm.exceptions.RateLimitError AND fastapi.HTTPException via cooperative multiple inheritance. Why both bases: * Subclassing RateLimitError lets user code catch every rate-limit source with one 'except RateLimitError' and switch on the new .category field. * Subclassing HTTPException keeps every existing FastAPI plumbing path (the isinstance(e, HTTPException) branches in proxy_server.py route handlers, FastAPI's own dispatcher, and tests asserting pytest.raises(HTTPException)) working without modification, and preserves retry-after / rate_limit_type / reset_at headers on the wire. The class declaration order is (HTTPException, RateLimitError) so the MRO puts HTTPException's no-super-call __init__ ahead of openai's cooperative __init__ chain — preventing openai.APIError.super().__init__(message) from landing in HTTPException.__init__(status_code=message). LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from budget + iteration limiters Replaces three bare HTTPException(status_code=429, ...) call sites with ProxyRateLimitError, which is both a RateLimitError (catchable by category) and an HTTPException (preserves existing FastAPI serialization). Drops the now-unused HTTPException import in the iteration / per-session limiters. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from parallel-request limiters Replaces HTTPException(status_code=429, ...) call sites in the v1 and v3 parallel-request limiters (key/team/user/model/customer rate limits) with ProxyRateLimitError. Updates the raise_rate_limit_error helper's return type annotation accordingly. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from dynamic rate limiters Replaces HTTPException(status_code=429, ...) call sites in the v1 and v3 dynamic rate limiters (project-level TPM/RPM allocation, model-saturation checks, priority-based limits, fail-closed guards) with ProxyRateLimitError. The v3 limiter still imports HTTPException for an unrelated bare 'except HTTPException:' branch. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from batch rate limiter Replaces HTTPException(status_code=429, ...) in batch_rate_limiter._raise_rate_limit_error with ProxyRateLimitError tagged as RateLimitErrorCategory.LITELLM_BATCH_RATE_LIMIT so users can distinguish batch-level throttling (which counts requests/tokens across an uploaded batch input file before submission) from the generic key/team/user RPM/TPM limiter. The HTTPException import is retained because the same module raises HTTPException for unrelated 403/IO error paths. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): pin down unified rate-limit error contract Adds a dedicated test module covering the new RateLimitErrorCategory enum, RateLimitError.category default + override behavior, ProxyRateLimitError's dual nature (RateLimitError + HTTPException), and a parametrized regression guard that asserts every proxy hook module imports the unified class. The regression guard catches the failure mode the refactor is designed to prevent: someone re-introducing a bare HTTPException(status_code=429, ...) in one of the hook modules instead of going through ProxyRateLimitError. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(logging): expose rate-limit category via StandardLoggingPayload Adds an optional 'error_rate_limit_category' field to StandardLoggingPayloadErrorInformation, populated from the unified RateLimitError.category attribute (introduced in the previous commits on this branch). Why: the .category attribute is reachable off the raw exception today via getattr(e, 'category', None), but the structured contract that downstream custom callbacks / loggers / spend log writers consume is the StandardLoggingPayload. Without this field, a user building custom rate-limit metrics on top of callback data has to special-case the raw exception object — which defeats the purpose of the StandardLoggingPayload abstraction. The field is None for non-rate-limit exceptions (so consumers can read it unconditionally without isinstance checks) and is one of the RateLimitErrorCategory string values otherwise. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): assert StandardLoggingPayload carries the category Five tests covering: vendor default, explicit litellm_rate_limit and litellm_batch_rate_limit values, None for non-rate-limit exceptions, and None when no exception is provided. Pins down the contract that custom callbacks can read 'error_information.error_rate_limit_category' off the StandardLoggingPayload to drive custom rate-limit metrics without ever reaching for the raw exception. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(types): silence mypy [misc] on intentional dual-base attr overlap mypy emits two [misc] errors on the ProxyRateLimitError class line because its two bases declare overlapping attributes with related-but-not-identical annotations: * status_code: int on starlette HTTPException vs. Literal[429] on openai's RateLimitError (every openai status-error subclass narrows it the same way and silences pyright with the same convention). * headers: Mapping[str, str] | None on HTTPException vs. our Optional[ Dict[str, str]] (the proxy hooks always carry a stringified dict). Both narrowings are intentional and enforced at construction time. Add a type: ignore[misc] with an inline explanation rather than relax the annotations on the parent or change the wire-format guarantees. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): add direct hook-invocation tests to lift patch coverage Adds six end-to-end tests that drive each refactored hook past its limit and assert the unified ProxyRateLimitError is raised with the correct category and dual-base shape. Complements the import-shape-only parametrized guard above by actually executing the new 'raise ProxyRateLimitError(...)' lines so codecov's patch coverage sees them as hit. Hooks covered (one test each): * parallel_request_limiter v1 — direct call to raise_rate_limit_error() * parallel_request_limiter v3 — direct call to _handle_rate_limit_error with a fabricated OVER_LIMIT response * max_iterations_limiter — full async_pre_call_hook with mocked agent registry, second call exceeds budget=1 * max_budget_limiter — async_pre_call_hook with mocked get_current_spend * dynamic_rate_limiter v1 — async_pre_call_hook with mocked check_available_usage forcing available_tpm == 0 * batch_rate_limiter — direct _raise_rate_limit_error call, asserts category is the batch-specific LITELLM_BATCH_RATE_LIMIT (not the generic LITELLM_RATE_LIMIT) LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix: guard rate_limit_category extraction with isinstance check * test(rate-limit): cover remaining hook raise sites for codecov Adds five more direct hook-invocation tests so every PR-touched line in the proxy hooks is exercised by tests in tests/test_litellm/, which codecov measures: * parallel_request_limiter v1 — check_key_in_limits inline raise (the second raise site, separate from the raise_rate_limit_error helper covered earlier) * dynamic_rate_limiter v1 — RPM raise branch (TPM branch was already covered) * dynamic_rate_limiter v3 — parametrized over all three raise sites: model_saturation_check, priority_model, and the fail-closed fallback for an unrecognized descriptor_key * max_budget_per_session_limiter — full async_pre_call_hook with a mocked agent registry and over-budget cached spend All 42 tests in test_rate_limit_error_unification.py now pass and together exercise every changed import + raise line across the eight refactored proxy hooks. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix: use computed error_message in ProxyRateLimitError detail * fix(parallel-request-limiter): drop None from detail; annotate raise_rate_limit_error as NoReturn The v1 ' raise_rate_limit_error' helper built an unused 'error_message' variable and then assembled the actual ' detail' via an f-string that interpolated 'additional_details' verbatim — producing 'Max parallel request limit reached None' when invoked without arguments (flagged by code review). Fix the helper to: - use the constructed 'error_message' as the detail - annotate the helper as NoReturn since it always raises - drop the redundant 'raise'/'return' at the two call sites Add two regression tests covering both the with- and without- additional_details paths. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): drop literal 'None' from raise_rate_limit_error detail The v1 parallel_request_limiter's raise_rate_limit_error helper has a long-standing bug: it computes a None-guarded 'error_message' string but then ignores it and emits an f-string that interpolates the raw 'additional_details' arg. Callers that pass no argument get 'Max parallel request limit reached None' as the user-facing detail. This commit: * wires error_message into the detail kwarg so the None-guard actually applies and operators see a clean message; * changes the return-type annotation from ProxyRateLimitError to NoReturn (the function always raises) so type-checkers know callers after this invocation are unreachable. Greptile P1 + P2 review feedback on PR BerriAI#27687. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(types): demote TypedDict floating string to a # comment A string literal placed after a field declaration in a TypedDict body is not a per-field docstring — it's an orphaned string expression Python discards. Tools like mypy / pyright that inspect TypedDict fields won't surface that text either. Move the documentation for error_rate_limit_category to a real comment so the intent is visible to readers and type-checker tooling without the misleading docstring framing. Greptile P2 review feedback on PR BerriAI#27687. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * security(exceptions): do not auto-copy vendor response headers to e.headers A vendor 429 response can set arbitrary headers (Set-Cookie, CORS overrides, …). Previously, when RateLimitError was constructed with only a 'response=' (no explicit 'headers=' kwarg), self.headers fell back to a copy of response.headers. If a downstream proxy serializer ever forwarded e.headers to the client, a malicious upstream could inject browser-interpreted headers for the proxy origin. Drop the fallback. Only headers passed explicitly via the headers= kwarg make it onto self.headers (proxy hooks pass retry-after etc. — they control what's surfaced). Vendor response headers stay reachable on e.response.headers for callers that explicitly want them. Today's proxy_server.py route handlers don't actually forward e.headers on the wire (they construct ProxyException without passing headers), so no current behavior changes — this is a defensive narrowing so the fallback can never be turned into a vector when someone wires e.headers through later. Veria-AI security review feedback on PR BerriAI#27687. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): regression guards for review-pass fixes Pins down the three review-pass fixes: * test_parallel_request_limiter_v1_helper_no_additional_details — calls raise_rate_limit_error() with no args and asserts the detail does NOT contain the literal string 'None'. Pre-fix, callers got 'Max parallel request limit reached None'. * test_rate_limit_error_does_not_auto_copy_response_headers — passes a vendor httpx.Response with a Set-Cookie header to RateLimitError WITHOUT an explicit headers= kwarg, asserts self.headers stays None (no leak), then re-checks that an explicit headers= kwarg DOES populate self.headers. Vendor headers remain reachable on e.response.headers for callers that explicitly want them. * The existing v1-helper test now also asserts the additional_details string makes it through to the detail. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(rate-limit): add orthogonal RateLimitType (requests/tokens/concurrent_requests/budget/max_iterations) trho's last ask in the LIT-2968 thread: distinguish rate-limit failures by the dimension that was exceeded, not just by who rate-limited (vendor vs. litellm). Adds: - RateLimitType str-enum exposed at `litellm.RateLimitType` with values requests / tokens / concurrent_requests / budget / max_iterations. - `rate_limit_type` kwarg on litellm.RateLimitError + ProxyRateLimitError; None default so existing callers (vendor-429 path in exception_mapping_utils) remain a no-op. - StandardLoggingPayloadErrorInformation.error_rate_limit_type so custom callbacks can split rate-limit failures by cause without parsing free-text error messages. Mirror to error_rate_limit_category extraction in get_error_information(); single isinstance(RateLimitError) check covers both. - map_v3_rate_limit_type() helper to collapse the v3 limiter's internal labels ("requests", "tokens", "max_parallel_requests") onto the public enum so the v3 limiter and dynamic_rate_limiter_v3 share one mapping. Defensive None on unknown values rather than silently picking a wrong dimension. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(proxy/hooks): wire rate_limit_type onto every limiter raise site Each refactored proxy hook now populates rate_limit_type with the dimension that actually tripped the limit, so downstream consumers (custom callbacks, prometheus exporters via the StandardLoggingPayload) can split key/team/user rate-limit failures by cause: - parallel_request_limiter (v1): detect dimension from current vs. limit in the post-cache branch (concurrent_requests > tokens > requests, matches the boolean condition order). Base case (current is None, one limit set to 0) picks the most-specific zero. raise_rate_limit_error() helper accepts an explicit rate_limit_type kwarg with CONCURRENT_REQUESTS default (matches every existing internal call site, including the global-limit branch). - parallel_request_limiter (v3): forward status["rate_limit_type"] through map_v3_rate_limit_type() so "max_parallel_requests" → CONCURRENT_REQUESTS for the public field while the raw v3 jargon stays on the HTTP header for wire-format backward compat. - dynamic_rate_limiter (v1): TPM-zero → TOKENS, RPM-zero → REQUESTS. Pass data["model"] through so callbacks see the model that hit the limit (addresses the secondary "provider missing" complaint in the original Slack thread, partially — the model is what dashboards typically split on). - dynamic_rate_limiter (v3): forward status["rate_limit_type"] via map_v3_rate_limit_type() at every raise site (model_saturation_check, priority_model, fail-closed unknown-descriptor guard). Also pass model. - batch_rate_limiter: limit_type is hard-typed "requests"|"tokens" — map directly without going through the helper's None branch. - max_budget_limiter, max_budget_per_session_limiter: BUDGET. - max_iterations_limiter: MAX_ITERATIONS. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): cover RateLimitType enum, hook wiring, and StandardLoggingPayload propagation 27 new tests across five new test classes: - TestRateLimitType: enum exposed at litellm.RateLimitType, all five values defined, RateLimitError default is None (vendor 429 path makes no claim about which dimension), accepts both string and enum forms with str-coercion guarantee for downstream JSON serializers. - TestProxyRateLimitErrorType: ProxyRateLimitError default is None, accepts string or enum, doesn't break existing callers that pass nothing. - TestMapV3RateLimitType: pins each v3-internal → public-enum mapping (tokens, requests, max_parallel_requests → concurrent_requests, unknown → None) so a future v3 refactor can't silently swap dimensions. - TestStandardLoggingPayloadCarriesType: the new error_rate_limit_type field reaches the structured payload for both ProxyRateLimitError and plain RateLimitError, is None when unspecified, and is None for non-rate-limit exceptions (symmetric with error_rate_limit_category). - TestProxyHooksWireTypeCorrectly: drives the actual raise sites in the v1 parallel_request_limiter helper, the v3 _handle_rate_limit_error (both "tokens" and "max_parallel_requests" paths), and the batch limiter (both tokens and requests paths) — coverage tools see the new rate_limit_type= kwargs as exercised, not just the import shape. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): cover _coerce_message branches and v1 dimension detection Drives the patch coverage on the new orthogonal RateLimitType wiring up to (or close to) 100% on the touched files. ProxyRateLimitError._coerce_message — was 22% covered, now 100%: * nested {error: {message}} dict * nested {message: {message}} dict (alt key) * dict without 'error'/'message' keys → JSON dump fallback * non-JSON-serializable dict value → str() fallback * non-string non-mapping detail (int) → str() coercion v1 parallel_request_limiter dimension detection — was 0% covered, now exercised across 6 parametrized cases: * check_key_in_limits else-branch: current at concurrent / TPM / RPM cap → asserts rate_limit_type is concurrent_requests / tokens / requests. * check_key_in_limits base case (current is None): max_parallel_requests / tpm_limit / rpm_limit set to 0 → asserts the most-specific zero attribution wins per the helper's order. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(proxy/hooks): add ProxyHTTPRateLimitError + provider resolver Introduces a small helper layer used by every proxy-side rate-limit hook so that the 429 they raise carries a populated llm_provider / model — instead of an empty exception.llm_provider that downstream loggers (Prometheus failure metric, observability callbacks) read as 'no provider attribution'. ProxyHTTPRateLimitError inherits from both fastapi.HTTPException (so the proxy server still renders it as a 429) and litellm.exceptions.RateLimitError (so isinstance checks and PrometheusLogger._get_exception_class_name pick up llm_provider). We deliberately don't call RateLimitError.__init__ — it constructs an httpx.Response we don't need and would just add failure surface; attribute parity is what downstream consumers care about. resolve_llm_provider_for_rate_limit() wraps litellm.get_llm_provider defensively. Internal limiter hooks fire from async_pre_call_hook — well before get_llm_provider runs anywhere else in the request lifecycle — so we have to call it ourselves at raise time. If the model is missing or unparseable (alias, router-only model) we fall back to llm_provider='litellm_proxy' rather than letting a second exception leak out and break the request path. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on parallel-request 429s Both v1 and v3 parallel-request limiters fired bare HTTPException(429) from inside async_pre_call_hook. The downstream Prometheus failure metric reads exception.llm_provider via _get_exception_class_name — the empty value showed up as exception_class='HTTPException' and left model_id='None' on the time series. Threads requested_model through every raise site in: * parallel_request_limiter.py: - check_key_in_limits (the per-key/per-model/per-user/per-team/ per-customer over-limit path) - raise_rate_limit_error (zero-limit + global_max_parallel_requests paths) — now takes an optional requested_model kwarg * parallel_request_limiter_v3.py: - _handle_rate_limit_error (the OVER_LIMIT translator), called from both the should_rate_limit pre-check and the TPM reservation path Resolved via resolve_llm_provider_for_rate_limit so unknown / missing models silently fall back to llm_provider='litellm_proxy' instead of breaking the request path with a second exception. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on dynamic-rate-limit 429s Same plumbing change as the parallel limiters, applied to both dynamic_rate_limiter (v1) and dynamic_rate_limiter_v3: * v1: TPM-zero and RPM-zero paths in async_pre_call_hook now resolve data['model'] -> (model, llm_provider) once and pass it into both raises. * v3: All three raise sites in _check_rate_limits — the model_saturation_check enforced raise, the priority_model enforced raise, and the fail-closed unknown-descriptor branch — now attribute the 429 to the actual provider. Falls back to llm_provider='litellm_proxy' when the model can't be resolved. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on batch-rate-limit 429s batch_rate_limiter._raise_rate_limit_error now takes a requested_model kwarg threaded from data['model'] in _check_and_increment_batch_counters. The batch-creation 429 is what gets raised when the input file's tokens/requests count would push the per-key TPM/RPM window over its limit. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on budget/iterations 429s Final batch of internal raise sites — the user/session-budget and max-iterations hooks. Same pattern: resolve data['model'] once at raise time, attach to ProxyHTTPRateLimitError so Prometheus and observability callbacks can attribute the 429. Hooks updated: * max_budget_limiter (per-user max_budget exceeded) * max_iterations_limiter (per-session agent iteration cap) * max_budget_per_session_limiter (per-session dollar cap) All three fall back to llm_provider='litellm_proxy' when data['model'] is missing or unparseable. Drops the now-unused HTTPException import from each module. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(proxy/hooks): pin provider field on internal rate-limit 429s Regression coverage for the 'provider field missing' bug across every proxy-side rate-limit hook + the helper layer: * ProxyHTTPRateLimitError class shape (HTTPException + RateLimitError, dict-detail stringification, None-provider normalization). * resolve_llm_provider_for_rate_limit happy paths (gpt-4o-mini, anthropic/..., bedrock/...) plus all three fallback branches (None, '', unknown name) plus a 'get_llm_provider raises' case that asserts we swallow the secondary exception. * For each limiter (parallel v1/v3, dynamic v1/v3, batch, max_budget, max_iterations, max_budget_per_session): assert the raised exception is a RateLimitError carrying the resolved model + llm_provider, and a sibling test that asserts the fallback path returns 'litellm_proxy' without leaking a second exception. * Two PrometheusLogger._get_exception_class_name pins so the Prometheus failure metric label flips from 'HTTPException' to 'Openai.ProxyHTTPRateLimitError' (or 'Litellm_proxy.*' on fallback) — that's what dashboards consume. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * perf(proxy/hooks): defer provider resolution to over-limit branches * fix: use error_message in raise_rate_limit_error to avoid literal 'None' in detail * Consolidate rate_limiter_utils imports in dynamic_rate_limiter * fix(proxy): set num_retries/max_retries on ProxyHTTPRateLimitError ProxyHTTPRateLimitError inherits from RateLimitError but did not call RateLimitError.__init__, so num_retries/max_retries were never set. When Starlette's HTTPException lacks __str__, MRO falls through to RateLimitError.__str__, which unconditionally reads these attributes and raises AttributeError during logging/traceback formatting. Initialize them to None defensively. * fix(mypy): silence base-class status_code conflict on ProxyHTTPRateLimitError HTTPException declares 'status_code: int' while openai.RateLimitError (via APIStatusError) declares 'status_code: Literal[429] = 429'. Mypy flags the multi-base override as [misc] in CI lint. The runtime semantics are fine (we set self.status_code in __init__), so silence the class-level annotation conflict with a targeted ignore. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix: annotate batch limiter _raise_rate_limit_error as NoReturn * feat(prometheus): rate-limit category/type labels + exception_class back-compat (follow-up to BerriAI#27687) (BerriAI#27706) * feat(prometheus): add rate_limit_category and rate_limit_type labels Adds two new labels to litellm_proxy_failed_requests_metric so dashboards can split 429s by rate-limit source (vendor vs. litellm-internal) and by the dimension that was exceeded (requests/tokens/concurrent_requests/ budget/max_iterations) without parsing free-text error messages. Closes the Prometheus side of LIT-2718. The unified RateLimitError.category and .rate_limit_type fields landed in PR BerriAI#27687 but were only surfaced on StandardLoggingPayload (custom-callback channel); this exposes them on the metric label set as well. Both labels are populated only when the underlying exception is a litellm.RateLimitError; non-rate-limit failures keep them empty. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(prometheus): populate rate-limit labels + preserve exception_class back-compat Two coupled changes in the Prometheus integration: 1. async_post_call_failure_hook now extracts the new RateLimitError .category / .rate_limit_type fields (added in PR BerriAI#27687) via a _extract_rate_limit_labels helper and forwards them through UserAPIKeyLabelValues onto litellm_proxy_failed_requests_metric. Empty for non-rate-limit failures. 2. _get_exception_class_name special-cases ProxyRateLimitError and keeps emitting 'HTTPException' for the exception_class label. Without this shim, ProxyRateLimitError (which multi-inherits from HTTPException + RateLimitError) would silently flip the label from 'HTTPException' (the historical value for proxy-side 429s) to 'ProxyRateLimitError', breaking existing dashboards / alerts that key off exception_class='HTTPException'. Distinguishing vendor vs. litellm 429s is now the job of the new rate_limit_category label. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(prometheus): cover rate-limit labels and exception_class back-compat Adds 19 tests across: - enum / label-list registration - _extract_rate_limit_labels for vendor RateLimitError, ProxyRateLimitError, non-rate-limit and None inputs (incl. parametrized over every RateLimitErrorCategory x RateLimitType combo) - _get_exception_class_name back-compat: ProxyRateLimitError keeps the legacy 'HTTPException' string while vendor RateLimitError keeps the historical 'Provider.ClassName' format - end-to-end through async_post_call_failure_hook with both ProxyRateLimitError and vendor RateLimitError, asserting both new labels populate and exception_class stays back-compat Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(prometheus): tolerate missing fastapi in lazy ProxyRateLimitError import Address greptile feedback: - async_post_call_failure_hook docstring: drop the stale labelnames listing and reference PrometheusMetricLabels.litellm_proxy_failed_requests_metric as the source of truth so the doc cannot drift from the actual labelset. - _get_exception_class_name: guard the lazy ProxyRateLimitError import with ImportError so router-side fallback callsites don't blow up in non-proxy installs that don't have fastapi (a transitive dep of proxy.common_utils.proxy_rate_limit_error). Behavior is unchanged when fastapi is available. Also fix the existing enterprise callback test that asserted the old labelset on litellm_proxy_failed_requests_metric — it now expects the new rate_limit_category / rate_limit_type labels populated for vendor 429s. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(bugbot): simplify rate-limit label coercion + guard None detail - prometheus.py _extract_rate_limit_labels: RateLimitError.__init__ already normalizes category/rate_limit_type to plain str, so the getattr(.value) + isinstance dance was dead code. Reduce to str(value) if not None. - proxy_rate_limit_error.py _coerce_message: short-circuit None to '' instead of falling through to str(None) = 'None', which produced the literal message 'litellm.RateLimitError: None'. * fix(rate-limit): surface unified category/type fields on BudgetExceededError The most common budget cap (virtual-key max_budget enforcement in auth_checks.py) raises litellm.BudgetExceededError, a bare Exception subclass that bypassed the unified rate-limit error class introduced by PR BerriAI#27687. Custom callbacks reading StandardLoggingPayload.error_information saw category=None and rate_limit_type=None for these 429s, missing the most common budget case (team / org / end-user budgets all hit the same code path). Surface the fields off BudgetExceededError as plain attributes: - category = RateLimitErrorCategory.LITELLM_RATE_LIMIT - rate_limit_type = RateLimitType.BUDGET - llm_provider = "" (or caller-supplied) Switch get_error_information and _extract_rate_limit_labels from isinstance(RateLimitError) gating to duck-typed attribute reads, guarded by membership in the rate-limit enums so unrelated third-party exceptions exposing a .category attribute can't leak garbage values into the payload. This is strictly additive: BudgetExceededError keeps its bare-Exception base class, so `except BudgetExceededError:` handlers keep firing and `except RateLimitError:` does not start catching budget errors. * fix(rate-limit): validate enum membership at duck-typed read sites + enrich BudgetExceededError llm_provider Two follow-ups uncovered during the second QA pass on PR BerriAI#27687: 1. Guard third-party `.category` / `.rate_limit_type` attribute leakage. The duck-typed read in `get_error_information` and `_extract_rate_limit_labels` would forward any string attribute named `category` / `rate_limit_type` on an unrelated third-party exception into the StandardLoggingPayload and Prometheus labels — silently mislabeling custom-callback payloads and blowing out Prometheus label cardinality. Add `validate_rate_limit_category` / `validate_rate_limit_type` helpers that gate on the documented enum value sets; non-matching values are dropped to None. 2. Enrich BudgetExceededError.llm_provider from request_data. Budget checks live in tenant-scoped helpers (key / team / org / tag / end-user / project) that don't see the request model, so the BudgetExceededError they raise carried llm_provider="" — leaving custom-metrics consumers without provider attribution for the most common 429 case. Resolve it once at the central UserAPIKeyAuthExceptionHandler seam, before post_call_failure_hook fires, so the StandardLoggingPayload the callback sees has the same provider attribution as RPM/TPM 429s. Regression tests pin both: 4 leakage tests + 4 enrichment tests. The leakage tests would fail under the pre-validation version of either read site; the enrichment tests would fail if the handler skipped the resolver call. * fix(rate-limit): resolve router model_name aliases to real provider (BerriAI#27914) * fix(rate-limit): resolve router model_name aliases to real provider For nearly every real LiteLLM proxy deployment the request model is a router model_name alias (e.g. 'tpm-locked' -> litellm_params.model: openai/gpt-4o-mini), and 'litellm.get_llm_provider' doesn't know about router aliases — it raises 'LLMProviderNotProvidedError'. The resolver then fell through to the defensive 'litellm_proxy' fallback, so the 'llm_provider' field this PR adds was effectively always 'litellm_proxy' in the field, defeating its purpose for the most common proxy configuration. Add a router-alias fallback step: when 'get_llm_provider' raises, scan the active 'llm_router.model_list' for a deployment whose 'model_name' matches the request model and resolve from its 'litellm_params.model' instead. If multiple deployments share the same alias (load-balancing case) the first one wins — every deployment under one alias should agree on provider in any sensible config, and 'first' is deterministic so the Prometheus label stays stable. Defensive throughout: an uninitialized router, a malformed deployment, a 'litellm_params.model' that itself fails 'get_llm_provider' — every branch falls through to the existing 'litellm_proxy' fallback rather than letting a secondary exception escape and mask the rate-limit error we're trying to surface. Tests: - test_router_alias_resolves_to_underlying_provider: alias 'tpm-locked' -> 'openai/gpt-4o-mini' produces provider='openai', model='gpt-4o-mini'. - test_router_alias_with_multiple_deployments_uses_first. - test_router_alias_unknown_falls_back. - test_router_alias_with_malformed_deployment_falls_back. - Existing fallback test updated to also stub 'litellm.proxy.proxy_server.llm_router' so it exercises the full 'no resolution anywhere' path. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(rate-limit): harden router alias resolver + test isolation - Wrap _resolve_provider_from_router_alias loop in top-level try/except so a non-iterable model_list / unexpected deployment shape can't escape and mask the 429 with a 500. - Type-check litellm_params before .get() to handle non-dict truthy values. - Patch llm_router=None in the parametrized fallback test so a router left by another test in the session can't redirect the unknown-model path. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(bugbot): preserve "BudgetExceededError" Prometheus label Adding llm_provider to BudgetExceededError (so callbacks get provider attribution from StandardLoggingPayload) made the provider-prefix step in _get_exception_class_name silently flip the label from "BudgetExceededError" to e.g. "Openai.BudgetExceededError", breaking dashboards keyed on the historical value. Short-circuit BudgetExceededError in _get_exception_class_name the same way ProxyRateLimitError already is. Provider/category attribution still lands on the new rate_limit_category / rate_limit_type labels. * test: fix invalid 'rpm' rate_limit_type in v3 limiter test mocks The v3 rate limiter only emits 'requests', 'tokens', or 'max_parallel_requests'. Using 'rpm' caused map_v3_rate_limit_type to return None, leaving the expected RateLimitType.REQUESTS untested. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(bugbot): hoist provider resolver + opt-in prom rate-limit labels - dynamic_rate_limiter.py: hoist resolve_llm_provider_for_rate_limit above the TPM/RPM if/elif so the lookup runs once per request, matching the pattern in dynamic_rate_limiter_v3.py. - prometheus.py: gate the new rate_limit_category / rate_limit_type labels on litellm_proxy_failed_requests_metric behind litellm.prometheus_emit_rate_limit_labels (default False). Mirrors the existing prometheus_emit_stream_label opt-in. Preserves the metric's pre-unification label set so existing dashboards / recording rules keep matching after upgrade; operators can enable the new labels once downstream consumers include them. - Tests updated: default-off back-compat case, opt-in path enables the flag before asserting label presence. * fix: stabilize prometheus label sets and drop redundant model normalization - Cache PrometheusLogger.get_labels_for_metric per metric_name so that the label set used to construct counters at __init__ time stays in sync with the label set used at increment time, even if module-level toggles like prometheus_emit_rate_limit_labels or prometheus_emit_stream_label are flipped at runtime. Without this, toggling these flags after the logger was created would cause ValueError from prometheus_client because the runtime labels would not match the counter's declared labelnames. - Drop redundant 'model or ""' guard in ProxyRateLimitError.__init__ where model is already normalized one step earlier. Co-authored-by: Yassin Kortam <yassin@berri.ai> * perf(dynamic_rate_limiter): only resolve provider when rate limit hit Co-authored-by: Yassin Kortam <yassin@berri.ai> * test(prometheus): clear cached metric labels after toggling rate-limit flag The PrometheusLogger caches each metric's label set at construction time so that labels used at counter.labels(...) time stay consistent with the labels the metric was registered with. The enterprise async_post_call_failure_hook test toggles litellm.prometheus_emit_rate_limit_labels = True AFTER the fixture has already built the logger, so without invalidating the cache the rate_limit_category / rate_limit_type labels never reach the mocked counter and the assert_called_once_with check fails. Co-authored-by: Yassin Kortam <yassin@berri.ai> * test: fix CI failures from prom label cache + flaky time-window assertion PrometheusLogger.get_labels_for_metric now caches the per-metric label set at first read so the labels passed to counter.labels(...) stay in lock step with the labels the counter was registered with. This broke two existing test patterns: - test_prometheus_labels.py: tests bind the real method onto a MagicMock, but MagicMock auto-creates a Mock for _cached_metric_labels whose .get(...) returns a truthy Mock — treated as a populated cache and returned as the label set, producing empty filtered labels and KeyError on labels["requested_model"] / ["route"]. Seed real {} containers for _cached_metric_labels and label_filters before binding. - test_prometheus_logging_callbacks.py::test_set_team_budget_metrics_with_custom_labels: the fixture builds the logger before the test monkeypatches litellm.custom_prometheus_metadata_labels, so the cached label set never picks up the new metadata labels. Clear the cache after the monkeypatch (same pattern already used for the rate-limit toggle in test_async_post_call_failure_hook). UI: view_logs/index.test.tsx "Last Minute" window assertion is off by one at the minute boundary. start_date is floored to the minute, so the dropped sub-minute fraction can push the truncated-seconds diff up to (minMinutes+1)*60 exactly when the click lands near a minute rollover. Switch the upper bound to toBeLessThanOrEqual. * feat(otel-v2): surface rate_limit_category + rate_limit_type on failed LLM-call spans PR BerriAI#28909 introduced the typed v2 OTel engine that builds spans from StandardLoggingPayload, with SpanError carrying error_type + message and the genai mapper stamping error.type onto every failed LLM-call span. This PR's earlier commits added error_rate_limit_category and error_rate_limit_type to the same StandardLoggingPayload.error_information the v2 engine reads — but neither field reached a span attribute, so v2 OTel traces stayed opaque about *why* a 429 fired (vendor vs litellm, RPM vs TPM vs concurrent vs budget vs max_iterations) even after the custom-callback and prometheus surfaces gained that decomposition. Three coupled changes: 1. semconv.py: add LiteLLM.ERROR_RATE_LIMIT_CATEGORY / LiteLLM.ERROR_RATE_LIMIT_TYPE under the litellm.* vendor namespace (no GenAI semconv equivalent exists for who-rate-limited / which-dimension). 2. payloads.py: extend SpanError with rate_limit_category + rate_limit_type, populated by _parse_error() from the same error_information.error_rate_limit_* fields the custom-callback channel and prometheus rate_limit_category / rate_limit_type labels read. Single source of truth across all three observability surfaces. 3. mappers/genai.py: stamp the two attributes on the LLM-call span when present. drop_none guarantees they stay absent (not 'None') for non-rate-limit failures so trace consumers can read them unconditionally. Three regression tests in test_otel_v2_emitter.py pin: a vendor / litellm-internal RateLimitError lands category=litellm_rate_limit + rate_limit_type=requests on the span; a BudgetExceededError lands rate_limit_type=budget; a non-rate-limit failure (BadRequestError) keeps the rate_limit_* attributes absent. Mutation-tested against reverting either the SpanError extension or the _parse_error read site — both new tests fail under either mutation. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test: align prometheus user-budget + logs quick-select tests with merged code The merge into this branch left two test patterns out of step with the code they exercise. test_set_user_budget_metrics_includes_user_email_and_alias_labels_when_opted_in flipped litellm.prometheus_user_budget_label_include_email_alias after the fixture had already built the PrometheusLogger. get_labels_for_metric now snapshots each metric's label set at construction time, so the runtime flip no longer reached the cached labels. Enable the flag before constructing the logger, matching how the proxy applies config at startup. view_logs/index.test.tsx referenced uiSpendLogsCall and moment without importing them, and the merged index.tsx now fetches through useLogFilterLogic (the hook the file stubs out) rather than calling uiSpendLogsCall directly. Add the imports and restore the real hook for the Quick Select window assertions so the call is actually observed. * refactor(otel/v2): drop rate-limit decomposition from the LLM-call span Proxy-side rate limits (litellm_rate_limit, budget, max_iterations) are rejected at the gate before any upstream call, so async_post_call_failure_hook tags the synthetic failure log with LITELLM_LOGGING_NO_UPSTREAM_LLM_CALL and the v2 OTel logger never opens an LLM-call span for them; the litellm.error.rate_limit_category / litellm.error.rate_limit_type attributes were dead for exactly the cases they were meant to surface. The only failure that does open an LLM-call span carrying a RateLimitError is a vendor 429, where rate_limit_type is always None and the category just restates error.type=RateLimitError. The decomposition still reaches downstream consumers through StandardLoggingPayload.error_information.error_rate_limit_* and the prometheus rate_limit_category / rate_limit_type labels, both unchanged. Removes the SpanError fields, the _parse_error reads, the genai mapper attributes, the semconv keys, and the three span tests that asserted a scenario that never reaches the mapper in production. * fix(batch_rate_limiter): map max_parallel_requests to concurrent_requests * refactor(prometheus): drop transitive fastapi import from _get_exception_class_name Read the legacy exception_class label from a prometheus_exception_class_name marker on ProxyRateLimitError instead of importing the proxy module, keeping the integrations layer free of a transitive fastapi dependency. * chore(ui): sync schema.d.ts with unified rate-limit error spec The ProxyRateLimitError docstring flows into the proxy OpenAPI spec's 429 response description, so the generated dashboard types were out of sync. Regenerated via npm run gen:api (Check UI API Types Sync). --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> Co-authored-by: Yassin Kortam <yassin@berri.ai>
blake-hamm
added a commit
to blake-hamm/bhamm-lab
that referenced
this pull request
Jun 28, 2026
…to v1.90.0 (#232) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [https://github.com/BerriAI/litellm.git](https://github.com/BerriAI/litellm) | minor | `v1.89.4` → `v1.90.0` | --- ### Release Notes <details> <summary>BerriAI/litellm (https://github.com/BerriAI/litellm.git)</summary> ### [`v1.90.0`](https://github.com/BerriAI/litellm/releases/tag/v1.90.0) [Compare Source](BerriAI/litellm@v1.89.4...v1.90.0-rc.1) #### Verify Docker Image Signature All LiteLLM Docker images are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](BerriAI/litellm@0112e53). **Verify using the pinned commit hash (recommended):** A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \ ghcr.io/berriai/litellm:v1.90.0 ``` **Verify using the release tag (convenience):** Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/v1.90.0/cosign.pub \ ghcr.io/berriai/litellm:v1.90.0 ``` Expected output: ``` The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key ``` *** #### What's Changed - fix(responses-bridge): map system-only chat request to system input item by [@​milan-berri](https://github.com/milan-berri) in [#​29817](BerriAI/litellm#29817) - feat(bedrock): forward strict and additionalProperties to Converse toolSpec by [@​mateo-berri](https://github.com/mateo-berri) in [#​29814](BerriAI/litellm#29814) - fix(mcp): highlight MCP cards red when the logged-in user is missing per-user env vars by [@​mateo-berri](https://github.com/mateo-berri) in [#​29856](BerriAI/litellm#29856) - feat(ui): add budget duration to edit team member form by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29717](BerriAI/litellm#29717) - fix(ui): make workflow runs page fill full width by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29868](BerriAI/litellm#29868) - feat: standardize rate limit errors with category, rate\_limit\_type, model, and llm\_provider fields by [@​mateo-berri](https://github.com/mateo-berri) in [#​27687](BerriAI/litellm#27687) - fix(ui): default guardrails page to the Guardrails tab by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29872](BerriAI/litellm#29872) - docs(readme): add Deploy on AWS/GCP Terraform section and fix deploy button rendering by [@​mateo-berri](https://github.com/mateo-berri) in [#​29879](BerriAI/litellm#29879) - refactor(bedrock): build Converse toolSpec via a BedrockToolSpec dict subclass by [@​mateo-berri](https://github.com/mateo-berri) in [#​29869](BerriAI/litellm#29869) - feat(litellm): add models and repository layers by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​29686](BerriAI/litellm#29686) - feat(ui): include internal routes in the dashboard's generated OpenAPI types by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29885](BerriAI/litellm#29885) - feat(proxy): publish /v2/model/info in Swagger OpenAPI spec by [@​Sameerlite](https://github.com/Sameerlite) in [#​29900](BerriAI/litellm#29900) - refactor(ui): single source of truth for migrated-page routing by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29949](BerriAI/litellm#29949) - fix(ui/model-hub): render provider icons on the public model hub by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29958](BerriAI/litellm#29958) - fix(ui): keep create guardrail modal open on outside click by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29871](BerriAI/litellm#29871) - fix(ui): label default key type as "Full Access" on key edit page by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29870](BerriAI/litellm#29870) - fix(ui): unify migrated-route URLs and migrate the API Reference page by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29953](BerriAI/litellm#29953) - fix(mcp): let non-creator users OAuth into OBO-mode MCP servers from the Tools page by [@​tin-berri](https://github.com/tin-berri) in [#​29867](BerriAI/litellm#29867) - Litellm oss staging 080626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​29932](BerriAI/litellm#29932) - feat(galileo): add health check support for UI callback test by [@​Sameerlite](https://github.com/Sameerlite) in [#​29908](BerriAI/litellm#29908) - fix(model-management): allow deleting a BYOK model after its team is deleted by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29875](BerriAI/litellm#29875) - feat(jwt-auth): opt-in fallback to DB team on unresolved JWT claim by [@​milan-berri](https://github.com/milan-berri) in [#​28913](BerriAI/litellm#28913) - fix(team\_endpoints): don't block /team/update on unchanged team budget by [@​milan-berri](https://github.com/milan-berri) in [#​29525](BerriAI/litellm#29525) - fix(fireworks): enable tool calling for glm-5p1 in model cost map by [@​milan-berri](https://github.com/milan-berri) in [#​29697](BerriAI/litellm#29697) - fix(vertex): propagate Vertex AI metadata in streaming success callbacks by [@​Sameerlite](https://github.com/Sameerlite) in [#​29899](BerriAI/litellm#29899) - fix(ui): show team projects to internal users on key creation by [@​milan-berri](https://github.com/milan-berri) in [#​28855](BerriAI/litellm#28855) - build(deps): bump pyjwt to 2.13.0 and ws override to 8.20.1 by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29982](BerriAI/litellm#29982) - fix(team-management): delete a team's BYOK models when the team is deleted by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29977](BerriAI/litellm#29977) - feat(vantage): include organization metadata in FOCUS Tags export by [@​milan-berri](https://github.com/milan-berri) in [#​28184](BerriAI/litellm#28184) - fix(guardrails): read CrowdStrike AIDR identity from both metadata bags by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29991](BerriAI/litellm#29991) - fix(mcp): mirror upstream token lifetime instead of forcing a 1h OBO expiry by [@​tin-berri](https://github.com/tin-berri) in [#​29951](BerriAI/litellm#29951) - feat(azure\_ai): add MAI-Image-2.5 image generation support by [@​Sameerlite](https://github.com/Sameerlite) in [#​29688](BerriAI/litellm#29688) - fix(mcp): load MCP tool configuration tools via the OBO/passthrough-aware GET path by [@​tin-berri](https://github.com/tin-berri) in [#​29960](BerriAI/litellm#29960) - fix(team): reserve team budget raises for proxy admins on /team/update by [@​milan-berri](https://github.com/milan-berri) in [#​30030](BerriAI/litellm#30030) - test(ui): data-driven App Router migration E2E smoke (default + server-root-path) by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29974](BerriAI/litellm#29974) - fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through by [@​michelligabriele](https://github.com/michelligabriele) in [#​24232](BerriAI/litellm#24232) - chore(ui): remove dead App Router route stubs under (dashboard) by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30045](BerriAI/litellm#30045) - fix(ui/mcp): reset OAuth state on create-server modal close so a prior server's token no longer leaks into the next add-server session by [@​tin-berri](https://github.com/tin-berri) in [#​30000](BerriAI/litellm#30000) - fix(mcp): allow team access-group grants in OAuth authorize/token access check by [@​tin-berri](https://github.com/tin-berri) in [#​30041](BerriAI/litellm#30041) - docs(security): require a reproduction video for vulnerability reports by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30063](BerriAI/litellm#30063) - feat(ui): add admin flag to disable in-product UI nudges for everyone by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29796](BerriAI/litellm#29796) - chore(ui): remove dead dashboard files and unused dependencies by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30047](BerriAI/litellm#30047) - fix(proxy): authorize batch files using upload target\_model\_names (LIT-3593) by [@​Sameerlite](https://github.com/Sameerlite) in [#​30009](BerriAI/litellm#30009) - Add Claude Fable 5 across Anthropic, Bedrock, Vertex AI, and Azure AI by [@​mateo-berri](https://github.com/mateo-berri) in [#​30064](BerriAI/litellm#30064) - Add Claude Fable 5 cost map entries (data-only hotfix for the hosted map) by [@​mateo-berri](https://github.com/mateo-berri) in [#​30076](BerriAI/litellm#30076) - fix(caching): restore stored prompt\_tokens on embedding cache hits instead of recomputing by [@​michelligabriele](https://github.com/michelligabriele) in [#​30046](BerriAI/litellm#30046) - Litellm oss 090626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​30021](BerriAI/litellm#30021) - fix(proxy): self-heal startup/reload prisma reads on engine disconnect by [@​michelligabriele](https://github.com/michelligabriele) in [#​28803](BerriAI/litellm#28803) - chore(ui): make knip recognize .mjs scripts and openapi-typescript by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30052](BerriAI/litellm#30052) - fix(register\_model): preserve built-in cache pricing when registering custom overrides under unmapped keys by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30044](BerriAI/litellm#30044) - \[internal copy of [#​28007](BerriAI/litellm#28007)] Fix/gcp model garden streaming by [@​mateo-berri](https://github.com/mateo-berri) in [#​28363](BerriAI/litellm#28363) - feat(cli): per-agent `lite claude` / `codex` / `opencode` commands that wrap coding agents through the proxy by [@​mateo-berri](https://github.com/mateo-berri) in [#​29850](BerriAI/litellm#29850) - fix(callbacks): forward callback\_settings to callback initializers and guard consumers against non-dict values by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30161](BerriAI/litellm#30161) - fix(mcp): drop orphaned per-user credential rows when an MCP server is deleted by [@​tin-berri](https://github.com/tin-berri) in [#​30141](BerriAI/litellm#30141) - fix(proxy): recover from cached-plan errors by reconnecting the Prisma client by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​29983](BerriAI/litellm#29983) - feat(proxy): add option to disable server-side prepared statements for DB lookups by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​29984](BerriAI/litellm#29984) - fix(release): stop backport releases from overwriting the latest badge by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30005](BerriAI/litellm#30005) - feat: add conventional commits and coding guidelines by [@​mateo-berri](https://github.com/mateo-berri) in [#​30159](BerriAI/litellm#30159) - fix(proxy): return 5xx on DB infra errors during auth; reserve 401 for genuine auth failures by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​29986](BerriAI/litellm#29986) - fix(ui): dev server 404s on migrated-page links because uiBase hardcodes /ui by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30169](BerriAI/litellm#30169) - refactor(ui): consolidate dashboard to one shell in the (dashboard) layout by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30166](BerriAI/litellm#30166) - fix(proxy): align /v1/model/info with router deployments by [@​Sameerlite](https://github.com/Sameerlite) in [#​30025](BerriAI/litellm#30025) - fix: completion\_cost AttributeError on streaming Anthropic web\_search responses ([#​26153](BerriAI/litellm#26153)) by [@​ishaan-berri](https://github.com/ishaan-berri) in [#​27346](BerriAI/litellm#27346) - \[internal copy of [#​30137](BerriAI/litellm#30137)] perf(realtime): eliminate redundant per-frame JSON work on OpenAI realtime relay by [@​mateo-berri](https://github.com/mateo-berri) in [#​30142](BerriAI/litellm#30142) - feat(bedrock): aws\_bedrock\_project\_id for bedrock-mantle project / workspace association by [@​mateo-berri](https://github.com/mateo-berri) in [#​30163](BerriAI/litellm#30163) - chore(hooks): enforce Conventional Commits and Conventional Branches by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30174](BerriAI/litellm#30174) - feat(rate-limiter): allow opting out of v3 TPM reservation and Redis circuit breaker by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30211](BerriAI/litellm#30211) - feat(spend\_logs): opt-in native Postgres partitioning for SpendLogs retention by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​29466](BerriAI/litellm#29466) - feat(ui): migrate playground to path routing and colocate its files by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30185](BerriAI/litellm#30185) - feat(ui): migrate projects and access-groups to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30226](BerriAI/litellm#30226) - fix(proxy): coalesce NULL rollup metrics in aggregated daily-activity by [@​michelligabriele](https://github.com/michelligabriele) in [#​30151](BerriAI/litellm#30151) - fix(anthropic\_passthrough): resolve costing model from message\_start chunk, litellm\_params and model\_group instead of 'unknown' by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30160](BerriAI/litellm#30160) - feat(ui): migrate budgets, workflows, and guardrails-monitor to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30236](BerriAI/litellm#30236) - feat(ui): migrate mcp-servers, search-tools, tag-management, vector-stores, and memory to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30261](BerriAI/litellm#30261) - fix(a2a): forward agent\_extra\_headers through completion bridge by [@​mateo-berri](https://github.com/mateo-berri) in [#​28277](BerriAI/litellm#28277) - fix(gemini-live): forward audio buffer commit and correct Vertex PCM rate by [@​Sameerlite](https://github.com/Sameerlite) in [#​29946](BerriAI/litellm#29946) - fix(proxy): skip double-wrapping unified batch output file ids on retrieve by [@​Sameerlite](https://github.com/Sameerlite) in [#​30011](BerriAI/litellm#30011) - feat: litellm oss 110626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​30202](BerriAI/litellm#30202) - fix(docker): copy only runtime artifacts into the final image by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30243](BerriAI/litellm#30243) - feat(proxy): enforce key/team guardrails on bedrock passthrough routes by [@​Sameerlite](https://github.com/Sameerlite) in [#​30194](BerriAI/litellm#30194) - feat(gemini): forward web search tools in image generation by [@​Sameerlite](https://github.com/Sameerlite) in [#​30119](BerriAI/litellm#30119) - fix: bedrock mantle fixes by [@​Sameerlite](https://github.com/Sameerlite) in [#​30083](BerriAI/litellm#30083) - feat(proxy): add require\_managed\_files setting for file uploads by [@​Sameerlite](https://github.com/Sameerlite) in [#​30186](BerriAI/litellm#30186) - fix(mcp): honor server\_id for REST tool calls with shared upstream URLs by [@​Sameerlite](https://github.com/Sameerlite) in [#​30184](BerriAI/litellm#30184) - fix(responses): presidio PII masking for Azure WebSocket and streaming by [@​Sameerlite](https://github.com/Sameerlite) in [#​30003](BerriAI/litellm#30003) - feat(passthrough): add configurable pass-through request timeouts by [@​Sameerlite](https://github.com/Sameerlite) in [#​30266](BerriAI/litellm#30266) - fix(google\_genai): preserve complete SSE events in Vertex/Gemini image streaming by [@​Sameerlite](https://github.com/Sameerlite) in [#​30270](BerriAI/litellm#30270) - fix(proxy): populate access\_via\_team\_ids on /v1/model/info by [@​Sameerlite](https://github.com/Sameerlite) in [#​30274](BerriAI/litellm#30274) - chore(oss): litellm oss staging 120626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​30292](BerriAI/litellm#30292) - feat(ui): migrate policies, guardrails, prompts, tool-policies, and skills to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30263](BerriAI/litellm#30263) - feat(ui): migrate caching, cost-tracking, transform-request, ui-theme, and logs to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30267](BerriAI/litellm#30267) - fix(ui): gate dashboard layout on ui config load so deep links work under SERVER\_ROOT\_PATH by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30312](BerriAI/litellm#30312) - feat(ui): migrate admin-panel, logging-and-alerts, model-hub-table, and usage to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30268](BerriAI/litellm#30268) - fix(otel): cap metric attribute cardinality with include/exclude lists by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30257](BerriAI/litellm#30257) - fix(proxy): grace-period key rotation 401s; return deprecated-key lookup result directly by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30327](BerriAI/litellm#30327) - chore(deps): bump vitest, brace-expansion, pypdf and tornado by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30220](BerriAI/litellm#30220) - refactor(ui): remove unreachable /chat page by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30178](BerriAI/litellm#30178) - feat(ui): migrate agents and router-settings to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30323](BerriAI/litellm#30323) - feat: strengthen coding conventions in CLAUDE.md by [@​mateo-berri](https://github.com/mateo-berri) in [#​30333](BerriAI/litellm#30333) - feat(ui): cut the users page over to the /ui/users path route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30334](BerriAI/litellm#30334) - feat: ruff strict-rule suppressions baseline gate by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30303](BerriAI/litellm#30303) - feat(guardrails): add Cisco AI Defense integration ([#​28249](BerriAI/litellm#28249)) by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30338](BerriAI/litellm#30338) - chore(ui): remove dead UI components unreferenced by any page by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30340](BerriAI/litellm#30340) - ci: add osv-scanner lockfile scan workflow by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30222](BerriAI/litellm#30222) - fix(otel): record full error message on standard exception event in otel v2 by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30380](BerriAI/litellm#30380) - test(fireworks): mock whisper transcription tests instead of live calls by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30391](BerriAI/litellm#30391) - build(ui): pin esbuild to 0.28.1 via overrides by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30390](BerriAI/litellm#30390) - feat(ui): cut the organizations page over to the /ui/organizations path route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30336](BerriAI/litellm#30336) - fix(proxy): support SMTP implicit SSL (port 465) by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30395](BerriAI/litellm#30395) - fix(mcp): default Linear MCP registry entry to streamable HTTP by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30396](BerriAI/litellm#30396) - fix(ui): stop Virtual Keys page from infinite render loop by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30397](BerriAI/litellm#30397) - fix(streaming): guard raise\_on\_model\_repetition against empty choices by [@​shivamrawat1](https://github.com/shivamrawat1) in [#​30485](BerriAI/litellm#30485) - feat(otel-v2): emit the 6 gen\_ai.client.\* metrics at parity with v1 by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30326](BerriAI/litellm#30326) - fix(mcp): drop phantom 401 span on delegated OAuth2 tool calls by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30494](BerriAI/litellm#30494) - feat(ui): cut the teams page over to the /ui/teams path route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30343](BerriAI/litellm#30343) - fix(integrations): cap Anthropic cache\_control injection at 4 blocks by [@​shivamrawat1](https://github.com/shivamrawat1) in [#​30480](BerriAI/litellm#30480) - chore(codecov): add Batches, Videos, and Realtime components by [@​Sameerlite](https://github.com/Sameerlite) in [#​30517](BerriAI/litellm#30517) - test(batches): move orphan tests into tests/test\_litellm for CI coverage by [@​Sameerlite](https://github.com/Sameerlite) in [#​30510](BerriAI/litellm#30510) - fix(guardrails): run pre\_call hook once for model-level guardrails by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30543](BerriAI/litellm#30543) - fix(guardrails): stop re-initializing DB guardrails on every poll by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30542](BerriAI/litellm#30542) - chore(oss): litellm oss staging 150626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​30463](BerriAI/litellm#30463) - ci(lint): add blanket-noqa, dataclass-default, and unused-noqa Ruff rules by [@​mateo-berri](https://github.com/mateo-berri) in [#​30516](BerriAI/litellm#30516) - ci: ratchet lint and type-check gates (ruff preview, ANN, mypy, basedpyright) by [@​mateo-berri](https://github.com/mateo-berri) in [#​30379](BerriAI/litellm#30379) - fix(proxy): allow internal roles to access vector store CRUD routes by [@​shivamrawat1](https://github.com/shivamrawat1) in [#​30503](BerriAI/litellm#30503) - fix(otel): stamp gen\_ai.input/output.messages on v2 spans by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30548](BerriAI/litellm#30548) - fix(otel): export v2 gen\_ai client metrics to the configured meter provider by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30549](BerriAI/litellm#30549) - fix(bedrock): preserve cache\_control for ARN models in /v1/messages adapter by [@​mateo-berri](https://github.com/mateo-berri) in [#​29823](BerriAI/litellm#29823) - fix: greatly increase basedpyright slack by [@​mateo-berri](https://github.com/mateo-berri) in [#​30563](BerriAI/litellm#30563) - fix(budget): recompute budget\_reset\_at when budget\_duration changes on /budget/update by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30555](BerriAI/litellm#30555) - fix(otel): accept UPPER\_SNAKE\_CASE OTEL\_INSTRUMENTATION\_GENAI\_CAPTURE\_MESSAGE\_CONTENT in v2 by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30562](BerriAI/litellm#30562) - chore(lint): remove PLR0915 too-many-statements ruff rule by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30574](BerriAI/litellm#30574) - ci(lint): ratcheted type-discipline gate (mutable collections, casts, guards, kwargs, suppressions) by [@​mateo-berri](https://github.com/mateo-berri) in [#​30500](BerriAI/litellm#30500) - feat(proxy): add verification\_uri\_complete to CLI SSO device flow by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30571](BerriAI/litellm#30571) - chore: litellm oss staging160626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​30527](BerriAI/litellm#30527) - fix(guardrails): return 400 not 500 when AIM blocks a request by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30573](BerriAI/litellm#30573) - ci(lint): grandfather any-discipline with a per-file ratchet budget (50% headroom) by [@​mateo-berri](https://github.com/mateo-berri) in [#​30582](BerriAI/litellm#30582) - fix(audio): don't override explicit response\_format with verbose\_json by [@​mateo-berri](https://github.com/mateo-berri) in [#​30599](BerriAI/litellm#30599) - fix(anthropic): price and surface response service\_tier in cost tracking by [@​mateo-berri](https://github.com/mateo-berri) in [#​30558](BerriAI/litellm#30558) - feat: add dev and wildcard proxy configs for local testing by [@​mateo-berri](https://github.com/mateo-berri) in [#​30556](BerriAI/litellm#30556) - fix(proxy): list public team model name in /v1/models by [@​ishaan-berri](https://github.com/ishaan-berri) in [#​30588](BerriAI/litellm#30588) - ci: drop mypy entirely, standardize type checking on basedpyright by [@​mateo-berri](https://github.com/mateo-berri) in [#​30648](BerriAI/litellm#30648) - feat(guardrails): surface OpenAI moderation violation\_categories on guardrail traces by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30659](BerriAI/litellm#30659) - fix(proxy): resolve list files credentials from team BYOK deployments by [@​shivamrawat1](https://github.com/shivamrawat1) in [#​30495](BerriAI/litellm#30495) - feat(proxy): add --max\_requests\_before\_restart\_jitter to stagger worker restarts by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30601](BerriAI/litellm#30601) - fix(health): correct bedrock embedding health checks by [@​mateo-berri](https://github.com/mateo-berri) in [#​30583](BerriAI/litellm#30583) - test: harden remaining pass-through CI flakes (image-gen spend poll, ruby assistants timeout) by [@​mateo-berri](https://github.com/mateo-berri) in [#​30685](BerriAI/litellm#30685) - test(pass\_through): harden vertex spendlog poll against transient empty reads by [@​mateo-berri](https://github.com/mateo-berri) in [#​30683](BerriAI/litellm#30683) - fix(cost): stop non-string service\_tier from silently dropping cost tracking by [@​mateo-berri](https://github.com/mateo-berri) in [#​30690](BerriAI/litellm#30690) - feat(proxy): warn at startup when custom\_auth skips common\_checks enforcement by [@​tin-berri](https://github.com/tin-berri) in [#​30665](BerriAI/litellm#30665) - fix(pod\_lock): release cron lock by matching async\_set\_cache JSON encoding by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30600](BerriAI/litellm#30600) - ci: run a local fake OpenAI endpoint instead of the shared Railway mock by [@​mateo-berri](https://github.com/mateo-berri) in [#​30695](BerriAI/litellm#30695) - ci(windows): pin uv to Python 3.11 so it ignores the preinstalled 3.14 by [@​mateo-berri](https://github.com/mateo-berri) in [#​30704](BerriAI/litellm#30704) - feat(ui): migrate models page to App Router path route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30677](BerriAI/litellm#30677) - refactor(ui): remove orphaned pass-through-settings route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30692](BerriAI/litellm#30692) - fix(cost): stop non-string response service\_tier from dropping cost tracking by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30706](BerriAI/litellm#30706) - feat(agent-shin): automated PR/issue triage, low-quality auto-close, and review-gate label lifecycle by [@​mateo-berri](https://github.com/mateo-berri) in [#​30433](BerriAI/litellm#30433) - chore: litellm oss 170626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​30637](BerriAI/litellm#30637) - fix(bedrock\_mantle): add SigV4 fallback to chat completions auth by [@​mateo-berri](https://github.com/mateo-berri) in [#​30714](BerriAI/litellm#30714) - feat(search): add TinyFish as search provider by [@​simantak-dabhade](https://github.com/simantak-dabhade) in [#​30634](BerriAI/litellm#30634) - feat(ui): migrate old usage report to App Router path route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30694](BerriAI/litellm#30694) - fix(proxy): enforce budgets against authoritative DB spend when the cross-pod counter is stale by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30684](BerriAI/litellm#30684) - chore(ci): remove Agent Shin pull\_request\_target workflows by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30784](BerriAI/litellm#30784) - chore: litellm oss staging by [@​Sameerlite](https://github.com/Sameerlite) in [#​30745](BerriAI/litellm#30745) - ci(zizmor): also run on litellm\_internal\_staging by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30789](BerriAI/litellm#30789) - fix(test): drop references to removed Agent Shin workflows by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30791](https://github.com/BerriAI/litellm/pull/30791) - chore: remove in-product survey and Claude Code feedback nudges by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30773](https://github.com/BerriAI/litellm/pull/30773) - feat(ui): migrate api-keys landing to App Router path route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30699](https://github.com/BerriAI/litellm/pull/30699) - feat(proxy): configurable response headers and login-page hint by [@​yucheng-berri](https://github.com/yucheng-berri) in [#​30792](https://github.com/BerriAI/litellm/pull/30792) - ci(zizmor): gate PRs on medium+ findings and clear existing ones by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30797](https://github.com/BerriAI/litellm/pull/30797) - fix(proxy): use e.request\_data for logging\_obj in ModifyResponseException streaming passthrough by [@​mateo-berri](https://github.com/mateo-berri) in [#​30800](https://github.com/BerriAI/litellm/pull/30800) - chore: make pr template linear portion clearer by [@​mateo-berri](https://github.com/mateo-berri) in [#​30766](https://github.com/BerriAI/litellm/pull/30766) - chore(typing): add boto3/botocore stubs so basedpyright resolves the AWS SDK by [@​mateo-berri](https://github.com/mateo-berri) in [#​30815](https://github.com/BerriAI/litellm/pull/30815) - fix(otel): one v2 logger owns the global provider; scope tenant OTLP creds per exporter by [@​yucheng-berri](https://github.com/yucheng-berri) in [#​30590](https://github.com/BerriAI/litellm/pull/30590) - fix(passthrough): recover output tokens for interrupted anthropic streams by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30787](https://github.com/BerriAI/litellm/pull/30787) - fix(proxy): record partial spend on the failure row for interrupted streams by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30788](https://github.com/BerriAI/litellm/pull/30788) - fix(ui): repoint dead usage guide link to cost tracking docs by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30859](https://github.com/BerriAI/litellm/pull/30859) - fix(ui): warn that team models are deleted in the delete-team modal by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29990](https://github.com/BerriAI/litellm/pull/29990) - feat(caching): add valkey-semantic cache backend and fix semantic cache scope keys by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30675](https://github.com/BerriAI/litellm/pull/30675) - test(ui): isolate OldTeams delete-warning tests from leaked mock by [@​mateo-berri](https://github.com/mateo-berri) in [#​30871](https://github.com/BerriAI/litellm/pull/30871) - feat: add lint-gate target and truncation-proof summary to the strict ruff gate by [@​mateo-berri](https://github.com/mateo-berri) in [#​30877](https://github.com/BerriAI/litellm/pull/30877) - chore(ui): rebuild ui for release by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30894](https://github.com/BerriAI/litellm/pull/30894) - chore(ci): bump deps by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30899](https://github.com/BerriAI/litellm/pull/30899) - fix(watsonx): wrap string embedding input in array for WatsonX API by [@​shivamrawat1](https://github.com/shivamrawat1) in [#​30897](https://github.com/BerriAI/litellm/pull/30897) - test: point router/completion/triton tests at the local fake OpenAI endpoint by [@​mateo-berri](https://github.com/mateo-berri) in [#​30900](https://github.com/BerriAI/litellm/pull/30900) - feat(sandbox): e2b code execution primitive by [@​krrish-berri-2](https://github.com/krrish-berri-2) in [#​30898](https://github.com/BerriAI/litellm/pull/30898) - fix(ui): source api-keys identity from useAuthorized to stop "User ID is not set" by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30903](https://github.com/BerriAI/litellm/pull/30903) - chore(ui): rebuild ui by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30906](https://github.com/BerriAI/litellm/pull/30906) - chore(ci): promote internal staging to main by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30907](https://github.com/BerriAI/litellm/pull/30907) - fix(redis): prevent forcing SSLConnection when ssl=False in connection pool by [@​Jacopos311](https://github.com/Jacopos311) in [#​30770](https://github.com/BerriAI/litellm/pull/30770) - fix(proxy): log UI setup failures instead of silently swallowing by [@​sarvesh1327](https://github.com/sarvesh1327) in [#​30819](https://github.com/BerriAI/litellm/pull/30819) #### New Contributors - [@​simantak-dabhade](https://github.com/simantak-dabhade) made their first contribution in [#​30634](BerriAI/litellm#30634) - [@​Jacopos311](https://github.com/Jacopos311) made their first contribution in [#​30770](https://github.com/BerriAI/litellm/pull/30770) - [@​sarvesh1327](https://github.com/sarvesh1327) made their first contribution in [#​30819](https://github.com/BerriAI/litellm/pull/30819) **Full Changelog**: <BerriAI/litellm@v1.89.0...v1.90.0> </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yMjAuMCIsInVwZGF0ZWRJblZlciI6IjQzLjIyMC4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJkZXBlbmRlbmNpZXMiXX0=--> Co-authored-by: Renovate Bot <renovate@bhamm-lab.com> Reviewed-on: https://codeberg.org/blake-hamm/bhamm-lab/pulls/232
blake-hamm
added a commit
to blake-hamm/bhamm-lab
that referenced
this pull request
Jun 28, 2026
This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [ghcr.io/berriai/litellm](https://images.chainguard.dev/directory/image/wolfi-base/overview) ([source](https://github.com/BerriAI/litellm)) | final | minor | `v1.85.1` → `v1.90.0` | --- ### Release Notes <details> <summary>BerriAI/litellm (ghcr.io/berriai/litellm)</summary> ### [`v1.90.0`](https://github.com/BerriAI/litellm/releases/tag/v1.90.0) [Compare Source](https://github.com/BerriAI/litellm/compare/v1.90.0...v1.90.0) ##### Verify Docker Image Signature All LiteLLM Docker images are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](https://github.com/BerriAI/litellm/commit/0112e53046018d726492c814b3644b7d376029d0). **Verify using the pinned commit hash (recommended):** A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \ ghcr.io/berriai/litellm:v1.90.0 ``` **Verify using the release tag (convenience):** Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/v1.90.0/cosign.pub \ ghcr.io/berriai/litellm:v1.90.0 ``` Expected output: ``` The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key ``` *** ##### What's Changed - fix(responses-bridge): map system-only chat request to system input item by [@​milan-berri](https://github.com/milan-berri) in [#​29817](https://github.com/BerriAI/litellm/pull/29817) - feat(bedrock): forward strict and additionalProperties to Converse toolSpec by [@​mateo-berri](https://github.com/mateo-berri) in [#​29814](https://github.com/BerriAI/litellm/pull/29814) - fix(mcp): highlight MCP cards red when the logged-in user is missing per-user env vars by [@​mateo-berri](https://github.com/mateo-berri) in [#​29856](https://github.com/BerriAI/litellm/pull/29856) - feat(ui): add budget duration to edit team member form by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29717](https://github.com/BerriAI/litellm/pull/29717) - fix(ui): make workflow runs page fill full width by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29868](https://github.com/BerriAI/litellm/pull/29868) - feat: standardize rate limit errors with category, rate\_limit\_type, model, and llm\_provider fields by [@​mateo-berri](https://github.com/mateo-berri) in [#​27687](https://github.com/BerriAI/litellm/pull/27687) - fix(ui): default guardrails page to the Guardrails tab by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29872](https://github.com/BerriAI/litellm/pull/29872) - docs(readme): add Deploy on AWS/GCP Terraform section and fix deploy button rendering by [@​mateo-berri](https://github.com/mateo-berri) in [#​29879](https://github.com/BerriAI/litellm/pull/29879) - refactor(bedrock): build Converse toolSpec via a BedrockToolSpec dict subclass by [@​mateo-berri](https://github.com/mateo-berri) in [#​29869](https://github.com/BerriAI/litellm/pull/29869) - feat(litellm): add models and repository layers by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​29686](https://github.com/BerriAI/litellm/pull/29686) - feat(ui): include internal routes in the dashboard's generated OpenAPI types by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29885](https://github.com/BerriAI/litellm/pull/29885) - feat(proxy): publish /v2/model/info in Swagger OpenAPI spec by [@​Sameerlite](https://github.com/Sameerlite) in [#​29900](https://github.com/BerriAI/litellm/pull/29900) - refactor(ui): single source of truth for migrated-page routing by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29949](https://github.com/BerriAI/litellm/pull/29949) - fix(ui/model-hub): render provider icons on the public model hub by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29958](https://github.com/BerriAI/litellm/pull/29958) - fix(ui): keep create guardrail modal open on outside click by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29871](https://github.com/BerriAI/litellm/pull/29871) - fix(ui): label default key type as "Full Access" on key edit page by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29870](https://github.com/BerriAI/litellm/pull/29870) - fix(ui): unify migrated-route URLs and migrate the API Reference page by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29953](https://github.com/BerriAI/litellm/pull/29953) - fix(mcp): let non-creator users OAuth into OBO-mode MCP servers from the Tools page by [@​tin-berri](https://github.com/tin-berri) in [#​29867](https://github.com/BerriAI/litellm/pull/29867) - Litellm oss staging 080626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​29932](https://github.com/BerriAI/litellm/pull/29932) - feat(galileo): add health check support for UI callback test by [@​Sameerlite](https://github.com/Sameerlite) in [#​29908](https://github.com/BerriAI/litellm/pull/29908) - fix(model-management): allow deleting a BYOK model after its team is deleted by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29875](https://github.com/BerriAI/litellm/pull/29875) - feat(jwt-auth): opt-in fallback to DB team on unresolved JWT claim by [@​milan-berri](https://github.com/milan-berri) in [#​28913](https://github.com/BerriAI/litellm/pull/28913) - fix(team\_endpoints): don't block /team/update on unchanged team budget by [@​milan-berri](https://github.com/milan-berri) in [#​29525](https://github.com/BerriAI/litellm/pull/29525) - fix(fireworks): enable tool calling for glm-5p1 in model cost map by [@​milan-berri](https://github.com/milan-berri) in [#​29697](https://github.com/BerriAI/litellm/pull/29697) - fix(vertex): propagate Vertex AI metadata in streaming success callbacks by [@​Sameerlite](https://github.com/Sameerlite) in [#​29899](https://github.com/BerriAI/litellm/pull/29899) - fix(ui): show team projects to internal users on key creation by [@​milan-berri](https://github.com/milan-berri) in [#​28855](https://github.com/BerriAI/litellm/pull/28855) - build(deps): bump pyjwt to 2.13.0 and ws override to 8.20.1 by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29982](https://github.com/BerriAI/litellm/pull/29982) - fix(team-management): delete a team's BYOK models when the team is deleted by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29977](https://github.com/BerriAI/litellm/pull/29977) - feat(vantage): include organization metadata in FOCUS Tags export by [@​milan-berri](https://github.com/milan-berri) in [#​28184](https://github.com/BerriAI/litellm/pull/28184) - fix(guardrails): read CrowdStrike AIDR identity from both metadata bags by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29991](https://github.com/BerriAI/litellm/pull/29991) - fix(mcp): mirror upstream token lifetime instead of forcing a 1h OBO expiry by [@​tin-berri](https://github.com/tin-berri) in [#​29951](https://github.com/BerriAI/litellm/pull/29951) - feat(azure\_ai): add MAI-Image-2.5 image generation support by [@​Sameerlite](https://github.com/Sameerlite) in [#​29688](https://github.com/BerriAI/litellm/pull/29688) - fix(mcp): load MCP tool configuration tools via the OBO/passthrough-aware GET path by [@​tin-berri](https://github.com/tin-berri) in [#​29960](https://github.com/BerriAI/litellm/pull/29960) - fix(team): reserve team budget raises for proxy admins on /team/update by [@​milan-berri](https://github.com/milan-berri) in [#​30030](https://github.com/BerriAI/litellm/pull/30030) - test(ui): data-driven App Router migration E2E smoke (default + server-root-path) by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29974](https://github.com/BerriAI/litellm/pull/29974) - fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through by [@​michelligabriele](https://github.com/michelligabriele) in [#​24232](https://github.com/BerriAI/litellm/pull/24232) - chore(ui): remove dead App Router route stubs under (dashboard) by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30045](https://github.com/BerriAI/litellm/pull/30045) - fix(ui/mcp): reset OAuth state on create-server modal close so a prior server's token no longer leaks into the next add-server session by [@​tin-berri](https://github.com/tin-berri) in [#​30000](https://github.com/BerriAI/litellm/pull/30000) - fix(mcp): allow team access-group grants in OAuth authorize/token access check by [@​tin-berri](https://github.com/tin-berri) in [#​30041](https://github.com/BerriAI/litellm/pull/30041) - docs(security): require a reproduction video for vulnerability reports by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30063](https://github.com/BerriAI/litellm/pull/30063) - feat(ui): add admin flag to disable in-product UI nudges for everyone by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29796](https://github.com/BerriAI/litellm/pull/29796) - chore(ui): remove dead dashboard files and unused dependencies by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30047](https://github.com/BerriAI/litellm/pull/30047) - fix(proxy): authorize batch files using upload target\_model\_names (LIT-3593) by [@​Sameerlite](https://github.com/Sameerlite) in [#​30009](https://github.com/BerriAI/litellm/pull/30009) - Add Claude Fable 5 across Anthropic, Bedrock, Vertex AI, and Azure AI by [@​mateo-berri](https://github.com/mateo-berri) in [#​30064](https://github.com/BerriAI/litellm/pull/30064) - Add Claude Fable 5 cost map entries (data-only hotfix for the hosted map) by [@​mateo-berri](https://github.com/mateo-berri) in [#​30076](https://github.com/BerriAI/litellm/pull/30076) - fix(caching): restore stored prompt\_tokens on embedding cache hits instead of recomputing by [@​michelligabriele](https://github.com/michelligabriele) in [#​30046](https://github.com/BerriAI/litellm/pull/30046) - Litellm oss 090626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​30021](https://github.com/BerriAI/litellm/pull/30021) - fix(proxy): self-heal startup/reload prisma reads on engine disconnect by [@​michelligabriele](https://github.com/michelligabriele) in [#​28803](https://github.com/BerriAI/litellm/pull/28803) - chore(ui): make knip recognize .mjs scripts and openapi-typescript by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30052](https://github.com/BerriAI/litellm/pull/30052) - fix(register\_model): preserve built-in cache pricing when registering custom overrides under unmapped keys by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30044](https://github.com/BerriAI/litellm/pull/30044) - \[internal copy of [#​28007](https://github.com/BerriAI/litellm/issues/28007)] Fix/gcp model garden streaming by [@​mateo-berri](https://github.com/mateo-berri) in [#​28363](https://github.com/BerriAI/litellm/pull/28363) - feat(cli): per-agent `lite claude` / `codex` / `opencode` commands that wrap coding agents through the proxy by [@​mateo-berri](https://github.com/mateo-berri) in [#​29850](https://github.com/BerriAI/litellm/pull/29850) - fix(callbacks): forward callback\_settings to callback initializers and guard consumers against non-dict values by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30161](https://github.com/BerriAI/litellm/pull/30161) - fix(mcp): drop orphaned per-user credential rows when an MCP server is deleted by [@​tin-berri](https://github.com/tin-berri) in [#​30141](https://github.com/BerriAI/litellm/pull/30141) - fix(proxy): recover from cached-plan errors by reconnecting the Prisma client by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​29983](https://github.com/BerriAI/litellm/pull/29983) - feat(proxy): add option to disable server-side prepared statements for DB lookups by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​29984](https://github.com/BerriAI/litellm/pull/29984) - fix(release): stop backport releases from overwriting the latest badge by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30005](https://github.com/BerriAI/litellm/pull/30005) - feat: add conventional commits and coding guidelines by [@​mateo-berri](https://github.com/mateo-berri) in [#​30159](https://github.com/BerriAI/litellm/pull/30159) - fix(proxy): return 5xx on DB infra errors during auth; reserve 401 for genuine auth failures by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​29986](https://github.com/BerriAI/litellm/pull/29986) - fix(ui): dev server 404s on migrated-page links because uiBase hardcodes /ui by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30169](https://github.com/BerriAI/litellm/pull/30169) - refactor(ui): consolidate dashboard to one shell in the (dashboard) layout by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30166](https://github.com/BerriAI/litellm/pull/30166) - fix(proxy): align /v1/model/info with router deployments by [@​Sameerlite](https://github.com/Sameerlite) in [#​30025](https://github.com/BerriAI/litellm/pull/30025) - fix: completion\_cost AttributeError on streaming Anthropic web\_search responses ([#​26153](https://github.com/BerriAI/litellm/issues/26153)) by [@​ishaan-berri](https://github.com/ishaan-berri) in [#​27346](https://github.com/BerriAI/litellm/pull/27346) - \[internal copy of [#​30137](https://github.com/BerriAI/litellm/issues/30137)] perf(realtime): eliminate redundant per-frame JSON work on OpenAI realtime relay by [@​mateo-berri](https://github.com/mateo-berri) in [#​30142](https://github.com/BerriAI/litellm/pull/30142) - feat(bedrock): aws\_bedrock\_project\_id for bedrock-mantle project / workspace association by [@​mateo-berri](https://github.com/mateo-berri) in [#​30163](https://github.com/BerriAI/litellm/pull/30163) - chore(hooks): enforce Conventional Commits and Conventional Branches by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30174](https://github.com/BerriAI/litellm/pull/30174) - feat(rate-limiter): allow opting out of v3 TPM reservation and Redis circuit breaker by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30211](https://github.com/BerriAI/litellm/pull/30211) - feat(spend\_logs): opt-in native Postgres partitioning for SpendLogs retention by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​29466](https://github.com/BerriAI/litellm/pull/29466) - feat(ui): migrate playground to path routing and colocate its files by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30185](https://github.com/BerriAI/litellm/pull/30185) - feat(ui): migrate projects and access-groups to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30226](https://github.com/BerriAI/litellm/pull/30226) - fix(proxy): coalesce NULL rollup metrics in aggregated daily-activity by [@​michelligabriele](https://github.com/michelligabriele) in [#​30151](https://github.com/BerriAI/litellm/pull/30151) - fix(anthropic\_passthrough): resolve costing model from message\_start chunk, litellm\_params and model\_group instead of 'unknown' by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30160](https://github.com/BerriAI/litellm/pull/30160) - feat(ui): migrate budgets, workflows, and guardrails-monitor to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30236](https://github.com/BerriAI/litellm/pull/30236) - feat(ui): migrate mcp-servers, search-tools, tag-management, vector-stores, and memory to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30261](https://github.com/BerriAI/litellm/pull/30261) - fix(a2a): forward agent\_extra\_headers through completion bridge by [@​mateo-berri](https://github.com/mateo-berri) in [#​28277](https://github.com/BerriAI/litellm/pull/28277) - fix(gemini-live): forward audio buffer commit and correct Vertex PCM rate by [@​Sameerlite](https://github.com/Sameerlite) in [#​29946](https://github.com/BerriAI/litellm/pull/29946) - fix(proxy): skip double-wrapping unified batch output file ids on retrieve by [@​Sameerlite](https://github.com/Sameerlite) in [#​30011](https://github.com/BerriAI/litellm/pull/30011) - feat: litellm oss 110626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​30202](https://github.com/BerriAI/litellm/pull/30202) - fix(docker): copy only runtime artifacts into the final image by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30243](https://github.com/BerriAI/litellm/pull/30243) - feat(proxy): enforce key/team guardrails on bedrock passthrough routes by [@​Sameerlite](https://github.com/Sameerlite) in [#​30194](https://github.com/BerriAI/litellm/pull/30194) - feat(gemini): forward web search tools in image generation by [@​Sameerlite](https://github.com/Sameerlite) in [#​30119](https://github.com/BerriAI/litellm/pull/30119) - fix: bedrock mantle fixes by [@​Sameerlite](https://github.com/Sameerlite) in [#​30083](https://github.com/BerriAI/litellm/pull/30083) - feat(proxy): add require\_managed\_files setting for file uploads by [@​Sameerlite](https://github.com/Sameerlite) in [#​30186](https://github.com/BerriAI/litellm/pull/30186) - fix(mcp): honor server\_id for REST tool calls with shared upstream URLs by [@​Sameerlite](https://github.com/Sameerlite) in [#​30184](https://github.com/BerriAI/litellm/pull/30184) - fix(responses): presidio PII masking for Azure WebSocket and streaming by [@​Sameerlite](https://github.com/Sameerlite) in [#​30003](https://github.com/BerriAI/litellm/pull/30003) - feat(passthrough): add configurable pass-through request timeouts by [@​Sameerlite](https://github.com/Sameerlite) in [#​30266](https://github.com/BerriAI/litellm/pull/30266) - fix(google\_genai): preserve complete SSE events in Vertex/Gemini image streaming by [@​Sameerlite](https://github.com/Sameerlite) in [#​30270](https://github.com/BerriAI/litellm/pull/30270) - fix(proxy): populate access\_via\_team\_ids on /v1/model/info by [@​Sameerlite](https://github.com/Sameerlite) in [#​30274](https://github.com/BerriAI/litellm/pull/30274) - chore(oss): litellm oss staging 120626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​30292](https://github.com/BerriAI/litellm/pull/30292) - feat(ui): migrate policies, guardrails, prompts, tool-policies, and skills to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30263](https://github.com/BerriAI/litellm/pull/30263) - feat(ui): migrate caching, cost-tracking, transform-request, ui-theme, and logs to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30267](https://github.com/BerriAI/litellm/pull/30267) - fix(ui): gate dashboard layout on ui config load so deep links work under SERVER\_ROOT\_PATH by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30312](https://github.com/BerriAI/litellm/pull/30312) - feat(ui): migrate admin-panel, logging-and-alerts, model-hub-table, and usage to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30268](https://github.com/BerriAI/litellm/pull/30268) - fix(otel): cap metric attribute cardinality with include/exclude lists by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30257](https://github.com/BerriAI/litellm/pull/30257) - fix(proxy): grace-period key rotation 401s; return deprecated-key lookup result directly by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30327](https://github.com/BerriAI/litellm/pull/30327) - chore(deps): bump vitest, brace-expansion, pypdf and tornado by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30220](https://github.com/BerriAI/litellm/pull/30220) - refactor(ui): remove unreachable /chat page by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30178](https://github.com/BerriAI/litellm/pull/30178) - feat(ui): migrate agents and router-settings to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30323](https://github.com/BerriAI/litellm/pull/30323) - feat: strengthen coding conventions in CLAUDE.md by [@​mateo-berri](https://github.com/mateo-berri) in [#​30333](https://github.com/BerriAI/litellm/pull/30333) - feat(ui): cut the users page over to the /ui/users path route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30334](https://github.com/BerriAI/litellm/pull/30334) - feat: ruff strict-rule suppressions baseline gate by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30303](https://github.com/BerriAI/litellm/pull/30303) - feat(guardrails): add Cisco AI Defense integration ([#​28249](https://github.com/BerriAI/litellm/issues/28249)) by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30338](https://github.com/BerriAI/litellm/pull/30338) - chore(ui): remove dead UI components unreferenced by any page by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30340](https://github.com/BerriAI/litellm/pull/30340) - ci: add osv-scanner lockfile scan workflow by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30222](https://github.com/BerriAI/litellm/pull/30222) - fix(otel): record full error message on standard exception event in otel v2 by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30380](https://github.com/BerriAI/litellm/pull/30380) - test(fireworks): mock whisper transcription tests instead of live calls by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30391](https://github.com/BerriAI/litellm/pull/30391) - build(ui): pin esbuild to 0.28.1 via overrides by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30390](https://github.com/BerriAI/litellm/pull/30390) - feat(ui): cut the organizations page over to the /ui/organizations path route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30336](https://github.com/BerriAI/litellm/pull/30336) - fix(proxy): support SMTP implicit SSL (port 465) by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30395](https://github.com/BerriAI/litellm/pull/30395) - fix(mcp): default Linear MCP registry entry to streamable HTTP by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30396](https://github.com/BerriAI/litellm/pull/30396) - fix(ui): stop Virtual Keys page from infinite render loop by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30397](https://github.com/BerriAI/litellm/pull/30397) - fix(streaming): guard raise\_on\_model\_repetition against empty choices by [@​shivamrawat1](https://github.com/shivamrawat1) in [#​30485](https://github.com/BerriAI/litellm/pull/30485) - feat(otel-v2): emit the 6 gen\_ai.client.\* metrics at parity with v1 by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30326](https://github.com/BerriAI/litellm/pull/30326) - fix(mcp): drop phantom 401 span on delegated OAuth2 tool calls by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30494](https://github.com/BerriAI/litellm/pull/30494) - feat(ui): cut the teams page over to the /ui/teams path route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30343](https://github.com/BerriAI/litellm/pull/30343) - fix(integrations): cap Anthropic cache\_control injection at 4 blocks by [@​shivamrawat1](https://github.com/shivamrawat1) in [#​30480](https://github.com/BerriAI/litellm/pull/30480) - chore(codecov): add Batches, Videos, and Realtime components by [@​Sameerlite](https://github.com/Sameerlite) in [#​30517](https://github.com/BerriAI/litellm/pull/30517) - test(batches): move orphan tests into tests/test\_litellm for CI coverage by [@​Sameerlite](https://github.com/Sameerlite) in [#​30510](https://github.com/BerriAI/litellm/pull/30510) - fix(guardrails): run pre\_call hook once for model-level guardrails by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30543](https://github.com/BerriAI/litellm/pull/30543) - fix(guardrails): stop re-initializing DB guardrails on every poll by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30542](https://github.com/BerriAI/litellm/pull/30542) - chore(oss): litellm oss staging 150626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​30463](https://github.com/BerriAI/litellm/pull/30463) - ci(lint): add blanket-noqa, dataclass-default, and unused-noqa Ruff rules by [@​mateo-berri](https://github.com/mateo-berri) in [#​30516](https://github.com/BerriAI/litellm/pull/30516) - ci: ratchet lint and type-check gates (ruff preview, ANN, mypy, basedpyright) by [@​mateo-berri](https://github.com/mateo-berri) in [#​30379](https://github.com/BerriAI/litellm/pull/30379) - fix(proxy): allow internal roles to access vector store CRUD routes by [@​shivamrawat1](https://github.com/shivamrawat1) in [#​30503](https://github.com/BerriAI/litellm/pull/30503) - fix(otel): stamp gen\_ai.input/output.messages on v2 spans by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30548](https://github.com/BerriAI/litellm/pull/30548) - fix(otel): export v2 gen\_ai client metrics to the configured meter provider by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30549](https://github.com/BerriAI/litellm/pull/30549) - fix(bedrock): preserve cache\_control for ARN models in /v1/messages adapter by [@​mateo-berri](https://github.com/mateo-berri) in [#​29823](https://github.com/BerriAI/litellm/pull/29823) - fix: greatly increase basedpyright slack by [@​mateo-berri](https://github.com/mateo-berri) in [#​30563](https://github.com/BerriAI/litellm/pull/30563) - fix(budget): recompute budget\_reset\_at when budget\_duration changes on /budget/update by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30555](https://github.com/BerriAI/litellm/pull/30555) - fix(otel): accept UPPER\_SNAKE\_CASE OTEL\_INSTRUMENTATION\_GENAI\_CAPTURE\_MESSAGE\_CONTENT in v2 by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30562](https://github.com/BerriAI/litellm/pull/30562) - chore(lint): remove PLR0915 too-many-statements ruff rule by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30574](https://github.com/BerriAI/litellm/pull/30574) - ci(lint): ratcheted type-discipline gate (mutable collections, casts, guards, kwargs, suppressions) by [@​mateo-berri](https://github.com/mateo-berri) in [#​30500](https://github.com/BerriAI/litellm/pull/30500) - feat(proxy): add verification\_uri\_complete to CLI SSO device flow by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30571](https://github.com/BerriAI/litellm/pull/30571) - chore: litellm oss staging160626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​30527](https://github.com/BerriAI/litellm/pull/30527) - fix(guardrails): return 400 not 500 when AIM blocks a request by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30573](https://github.com/BerriAI/litellm/pull/30573) - ci(lint): grandfather any-discipline with a per-file ratchet budget (50% headroom) by [@​mateo-berri](https://github.com/mateo-berri) in [#​30582](https://github.com/BerriAI/litellm/pull/30582) - fix(audio): don't override explicit response\_format with verbose\_json by [@​mateo-berri](https://github.com/mateo-berri) in [#​30599](https://github.com/BerriAI/litellm/pull/30599) - fix(anthropic): price and surface response service\_tier in cost tracking by [@​mateo-berri](https://github.com/mateo-berri) in [#​30558](https://github.com/BerriAI/litellm/pull/30558) - feat: add dev and wildcard proxy configs for local testing by [@​mateo-berri](https://github.com/mateo-berri) in [#​30556](https://github.com/BerriAI/litellm/pull/30556) - fix(proxy): list public team model name in /v1/models by [@​ishaan-berri](https://github.com/ishaan-berri) in [#​30588](https://github.com/BerriAI/litellm/pull/30588) - ci: drop mypy entirely, standardize type checking on basedpyright by [@​mateo-berri](https://github.com/mateo-berri) in [#​30648](https://github.com/BerriAI/litellm/pull/30648) - feat(guardrails): surface OpenAI moderation violation\_categories on guardrail traces by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30659](https://github.com/BerriAI/litellm/pull/30659) - fix(proxy): resolve list files credentials from team BYOK deployments by [@​shivamrawat1](https://github.com/shivamrawat1) in [#​30495](https://github.com/BerriAI/litellm/pull/30495) - feat(proxy): add --max\_requests\_before\_restart\_jitter to stagger worker restarts by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30601](https://github.com/BerriAI/litellm/pull/30601) - fix(health): correct bedrock embedding health checks by [@​mateo-berri](https://github.com/mateo-berri) in [#​30583](https://github.com/BerriAI/litellm/pull/30583) - test: harden remaining pass-through CI flakes (image-gen spend poll, ruby assistants timeout) by [@​mateo-berri](https://github.com/mateo-berri) in [#​30685](https://github.com/BerriAI/litellm/pull/30685) - test(pass\_through): harden vertex spendlog poll against transient empty reads by [@​mateo-berri](https://github.com/mateo-berri) in [#​30683](https://github.com/BerriAI/litellm/pull/30683) - fix(cost): stop non-string service\_tier from silently dropping cost tracking by [@​mateo-berri](https://github.com/mateo-berri) in [#​30690](https://github.com/BerriAI/litellm/pull/30690) - feat(proxy): warn at startup when custom\_auth skips common\_checks enforcement by [@​tin-berri](https://github.com/tin-berri) in [#​30665](https://github.com/BerriAI/litellm/pull/30665) - fix(pod\_lock): release cron lock by matching async\_set\_cache JSON encoding by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30600](https://github.com/BerriAI/litellm/pull/30600) - ci: run a local fake OpenAI endpoint instead of the shared Railway mock by [@​mateo-berri](https://github.com/mateo-berri) in [#​30695](https://github.com/BerriAI/litellm/pull/30695) - ci(windows): pin uv to Python 3.11 so it ignores the preinstalled 3.14 by [@​mateo-berri](https://github.com/mateo-berri) in [#​30704](https://github.com/BerriAI/litellm/pull/30704) - feat(ui): migrate models page to App Router path route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30677](https://github.com/BerriAI/litellm/pull/30677) - refactor(ui): remove orphaned pass-through-settings route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30692](https://github.com/BerriAI/litellm/pull/30692) - fix(cost): stop non-string response service\_tier from dropping cost tracking by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30706](https://github.com/BerriAI/litellm/pull/30706) - feat(agent-shin): automated PR/issue triage, low-quality auto-close, and review-gate label lifecycle by [@​mateo-berri](https://github.com/mateo-berri) in [#​30433](https://github.com/BerriAI/litellm/pull/30433) - chore: litellm oss 170626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​30637](https://github.com/BerriAI/litellm/pull/30637) - fix(bedrock\_mantle): add SigV4 fallback to chat completions auth by [@​mateo-berri](https://github.com/mateo-berri) in [#​30714](https://github.com/BerriAI/litellm/pull/30714) - feat(search): add TinyFish as search provider by [@​simantak-dabhade](https://github.com/simantak-dabhade) in [#​30634](https://github.com/BerriAI/litellm/pull/30634) - feat(ui): migrate old usage report to App Router path route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30694](https://github.com/BerriAI/litellm/pull/30694) - fix(proxy): enforce budgets against authoritative DB spend when the cross-pod counter is stale by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30684](https://github.com/BerriAI/litellm/pull/30684) - chore(ci): remove Agent Shin pull\_request\_target workflows by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30784](https://github.com/BerriAI/litellm/pull/30784) - chore: litellm oss staging by [@​Sameerlite](https://github.com/Sameerlite) in [#​30745](https://github.com/BerriAI/litellm/pull/30745) - ci(zizmor): also run on litellm\_internal\_staging by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30789](https://github.com/BerriAI/litellm/pull/30789) - fix(test): drop references to removed Agent Shin workflows by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30791](https://github.com/BerriAI/litellm/pull/30791) - chore: remove in-product survey and Claude Code feedback nudges by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30773](https://github.com/BerriAI/litellm/pull/30773) - feat(ui): migrate api-keys landing to App Router path route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30699](https://github.com/BerriAI/litellm/pull/30699) - feat(proxy): configurable response headers and login-page hint by [@​yucheng-berri](https://github.com/yucheng-berri) in [#​30792](https://github.com/BerriAI/litellm/pull/30792) - ci(zizmor): gate PRs on medium+ findings and clear existing ones by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30797](https://github.com/BerriAI/litellm/pull/30797) - fix(proxy): use e.request\_data for logging\_obj in ModifyResponseException streaming passthrough by [@​mateo-berri](https://github.com/mateo-berri) in [#​30800](https://github.com/BerriAI/litellm/pull/30800) - chore: make pr template linear portion clearer by [@​mateo-berri](https://github.com/mateo-berri) in [#​30766](https://github.com/BerriAI/litellm/pull/30766) - chore(typing): add boto3/botocore stubs so basedpyright resolves the AWS SDK by [@​mateo-berri](https://github.com/mateo-berri) in [#​30815](https://github.com/BerriAI/litellm/pull/30815) - fix(otel): one v2 logger owns the global provider; scope tenant OTLP creds per exporter by [@​yucheng-berri](https://github.com/yucheng-berri) in [#​30590](https://github.com/BerriAI/litellm/pull/30590) - fix(passthrough): recover output tokens for interrupted anthropic streams by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30787](https://github.com/BerriAI/litellm/pull/30787) - fix(proxy): record partial spend on the failure row for interrupted streams by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30788](https://github.com/BerriAI/litellm/pull/30788) - fix(ui): repoint dead usage guide link to cost tracking docs by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30859](https://github.com/BerriAI/litellm/pull/30859) - fix(ui): warn that team models are deleted in the delete-team modal by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29990](https://github.com/BerriAI/litellm/pull/29990) - feat(caching): add valkey-semantic cache backend and fix semantic cache scope keys by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30675](https://github.com/BerriAI/litellm/pull/30675) - test(ui): isolate OldTeams delete-warning tests from leaked mock by [@​mateo-berri](https://github.com/mateo-berri) in [#​30871](https://github.com/BerriAI/litellm/pull/30871) - feat: add lint-gate target and truncation-proof summary to the strict ruff gate by [@​mateo-berri](https://github.com/mateo-berri) in [#​30877](https://github.com/BerriAI/litellm/pull/30877) - chore(ui): rebuild ui for release by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30894](https://github.com/BerriAI/litellm/pull/30894) - chore(ci): bump deps by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30899](https://github.com/BerriAI/litellm/pull/30899) - fix(watsonx): wrap string embedding input in array for WatsonX API by [@​shivamrawat1](https://github.com/shivamrawat1) in [#​30897](https://github.com/BerriAI/litellm/pull/30897) - test: point router/completion/triton tests at the local fake OpenAI endpoint by [@​mateo-berri](https://github.com/mateo-berri) in [#​30900](https://github.com/BerriAI/litellm/pull/30900) - feat(sandbox): e2b code execution primitive by [@​krrish-berri-2](https://github.com/krrish-berri-2) in [#​30898](https://github.com/BerriAI/litellm/pull/30898) - fix(ui): source api-keys identity from useAuthorized to stop "User ID is not set" by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30903](https://github.com/BerriAI/litellm/pull/30903) - chore(ui): rebuild ui by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30906](https://github.com/BerriAI/litellm/pull/30906) - chore(ci): promote internal staging to main by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30907](https://github.com/BerriAI/litellm/pull/30907) - fix(redis): prevent forcing SSLConnection when ssl=False in connection pool by [@​Jacopos311](https://github.com/Jacopos311) in [#​30770](https://github.com/BerriAI/litellm/pull/30770) - fix(proxy): log UI setup failures instead of silently swallowing by [@​sarvesh1327](https://github.com/sarvesh1327) in [#​30819](https://github.com/BerriAI/litellm/pull/30819) ##### New Contributors - [@​simantak-dabhade](https://github.com/simantak-dabhade) made their first contribution in [#​30634](https://github.com/BerriAI/litellm/pull/30634) - [@​Jacopos311](https://github.com/Jacopos311) made their first contribution in [#​30770](https://github.com/BerriAI/litellm/pull/30770) - [@​sarvesh1327](https://github.com/sarvesh1327) made their first contribution in [#​30819](https://github.com/BerriAI/litellm/pull/30819) **Full Changelog**: <https://github.com/BerriAI/litellm/compare/v1.89.0...v1.90.0> ### [`v1.90.0`](https://github.com/BerriAI/litellm/releases/tag/v1.90.0) [Compare Source](https://github.com/BerriAI/litellm/compare/v1.89.4...v1.90.0) ##### Verify Docker Image Signature All LiteLLM Docker images are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](https://github.com/BerriAI/litellm/commit/0112e53046018d726492c814b3644b7d376029d0). **Verify using the pinned commit hash (recommended):** A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \ ghcr.io/berriai/litellm:v1.90.0 ``` **Verify using the release tag (convenience):** Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/v1.90.0/cosign.pub \ ghcr.io/berriai/litellm:v1.90.0 ``` Expected output: ``` The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key ``` *** ##### What's Changed - fix(responses-bridge): map system-only chat request to system input item by [@​milan-berri](https://github.com/milan-berri) in [#​29817](https://github.com/BerriAI/litellm/pull/29817) - feat(bedrock): forward strict and additionalProperties to Converse toolSpec by [@​mateo-berri](https://github.com/mateo-berri) in [#​29814](https://github.com/BerriAI/litellm/pull/29814) - fix(mcp): highlight MCP cards red when the logged-in user is missing per-user env vars by [@​mateo-berri](https://github.com/mateo-berri) in [#​29856](https://github.com/BerriAI/litellm/pull/29856) - feat(ui): add budget duration to edit team member form by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29717](https://github.com/BerriAI/litellm/pull/29717) - fix(ui): make workflow runs page fill full width by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29868](https://github.com/BerriAI/litellm/pull/29868) - feat: standardize rate limit errors with category, rate\_limit\_type, model, and llm\_provider fields by [@​mateo-berri](https://github.com/mateo-berri) in [#​27687](https://github.com/BerriAI/litellm/pull/27687) - fix(ui): default guardrails page to the Guardrails tab by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29872](https://github.com/BerriAI/litellm/pull/29872) - docs(readme): add Deploy on AWS/GCP Terraform section and fix deploy button rendering by [@​mateo-berri](https://github.com/mateo-berri) in [#​29879](https://github.com/BerriAI/litellm/pull/29879) - refactor(bedrock): build Converse toolSpec via a BedrockToolSpec dict subclass by [@​mateo-berri](https://github.com/mateo-berri) in [#​29869](https://github.com/BerriAI/litellm/pull/29869) - feat(litellm): add models and repository layers by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​29686](https://github.com/BerriAI/litellm/pull/29686) - feat(ui): include internal routes in the dashboard's generated OpenAPI types by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29885](https://github.com/BerriAI/litellm/pull/29885) - feat(proxy): publish /v2/model/info in Swagger OpenAPI spec by [@​Sameerlite](https://github.com/Sameerlite) in [#​29900](https://github.com/BerriAI/litellm/pull/29900) - refactor(ui): single source of truth for migrated-page routing by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29949](https://github.com/BerriAI/litellm/pull/29949) - fix(ui/model-hub): render provider icons on the public model hub by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29958](https://github.com/BerriAI/litellm/pull/29958) - fix(ui): keep create guardrail modal open on outside click by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29871](https://github.com/BerriAI/litellm/pull/29871) - fix(ui): label default key type as "Full Access" on key edit page by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29870](https://github.com/BerriAI/litellm/pull/29870) - fix(ui): unify migrated-route URLs and migrate the API Reference page by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29953](https://github.com/BerriAI/litellm/pull/29953) - fix(mcp): let non-creator users OAuth into OBO-mode MCP servers from the Tools page by [@​tin-berri](https://github.com/tin-berri) in [#​29867](https://github.com/BerriAI/litellm/pull/29867) - Litellm oss staging 080626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​29932](https://github.com/BerriAI/litellm/pull/29932) - feat(galileo): add health check support for UI callback test by [@​Sameerlite](https://github.com/Sameerlite) in [#​29908](https://github.com/BerriAI/litellm/pull/29908) - fix(model-management): allow deleting a BYOK model after its team is deleted by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29875](https://github.com/BerriAI/litellm/pull/29875) - feat(jwt-auth): opt-in fallback to DB team on unresolved JWT claim by [@​milan-berri](https://github.com/milan-berri) in [#​28913](https://github.com/BerriAI/litellm/pull/28913) - fix(team\_endpoints): don't block /team/update on unchanged team budget by [@​milan-berri](https://github.com/milan-berri) in [#​29525](https://github.com/BerriAI/litellm/pull/29525) - fix(fireworks): enable tool calling for glm-5p1 in model cost map by [@​milan-berri](https://github.com/milan-berri) in [#​29697](https://github.com/BerriAI/litellm/pull/29697) - fix(vertex): propagate Vertex AI metadata in streaming success callbacks by [@​Sameerlite](https://github.com/Sameerlite) in [#​29899](https://github.com/BerriAI/litellm/pull/29899) - fix(ui): show team projects to internal users on key creation by [@​milan-berri](https://github.com/milan-berri) in [#​28855](https://github.com/BerriAI/litellm/pull/28855) - build(deps): bump pyjwt to 2.13.0 and ws override to 8.20.1 by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29982](https://github.com/BerriAI/litellm/pull/29982) - fix(team-management): delete a team's BYOK models when the team is deleted by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29977](https://github.com/BerriAI/litellm/pull/29977) - feat(vantage): include organization metadata in FOCUS Tags export by [@​milan-berri](https://github.com/milan-berri) in [#​28184](https://github.com/BerriAI/litellm/pull/28184) - fix(guardrails): read CrowdStrike AIDR identity from both metadata bags by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29991](https://github.com/BerriAI/litellm/pull/29991) - fix(mcp): mirror upstream token lifetime instead of forcing a 1h OBO expiry by [@​tin-berri](https://github.com/tin-berri) in [#​29951](https://github.com/BerriAI/litellm/pull/29951) - feat(azure\_ai): add MAI-Image-2.5 image generation support by [@​Sameerlite](https://github.com/Sameerlite) in [#​29688](https://github.com/BerriAI/litellm/pull/29688) - fix(mcp): load MCP tool configuration tools via the OBO/passthrough-aware GET path by [@​tin-berri](https://github.com/tin-berri) in [#​29960](https://github.com/BerriAI/litellm/pull/29960) - fix(team): reserve team budget raises for proxy admins on /team/update by [@​milan-berri](https://github.com/milan-berri) in [#​30030](https://github.com/BerriAI/litellm/pull/30030) - test(ui): data-driven App Router migration E2E smoke (default + server-root-path) by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29974](https://github.com/BerriAI/litellm/pull/29974) - fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through by [@​michelligabriele](https://github.com/michelligabriele) in [#​24232](https://github.com/BerriAI/litellm/pull/24232) - chore(ui): remove dead App Router route stubs under (dashboard) by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30045](https://github.com/BerriAI/litellm/pull/30045) - fix(ui/mcp): reset OAuth state on create-server modal close so a prior server's token no longer leaks into the next add-server session by [@​tin-berri](https://github.com/tin-berri) in [#​30000](https://github.com/BerriAI/litellm/pull/30000) - fix(mcp): allow team access-group grants in OAuth authorize/token access check by [@​tin-berri](https://github.com/tin-berri) in [#​30041](https://github.com/BerriAI/litellm/pull/30041) - docs(security): require a reproduction video for vulnerability reports by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30063](https://github.com/BerriAI/litellm/pull/30063) - feat(ui): add admin flag to disable in-product UI nudges for everyone by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29796](https://github.com/BerriAI/litellm/pull/29796) - chore(ui): remove dead dashboard files and unused dependencies by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30047](https://github.com/BerriAI/litellm/pull/30047) - fix(proxy): authorize batch files using upload target\_model\_names (LIT-3593) by [@​Sameerlite](https://github.com/Sameerlite) in [#​30009](https://github.com/BerriAI/litellm/pull/30009) - Add Claude Fable 5 across Anthropic, Bedrock, Vertex AI, and Azure AI by [@​mateo-berri](https://github.com/mateo-berri) in [#​30064](https://github.com/BerriAI/litellm/pull/30064) - Add Claude Fable 5 cost map entries (data-only hotfix for the hosted map) by [@​mateo-berri](https://github.com/mateo-berri) in [#​30076](https://github.com/BerriAI/litellm/pull/30076) - fix(caching): restore stored prompt\_tokens on embedding cache hits instead of recomputing by [@​michelligabriele](https://github.com/michelligabriele) in [#​30046](https://github.com/BerriAI/litellm/pull/30046) - Litellm oss 090626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​30021](https://github.com/BerriAI/litellm/pull/30021) - fix(proxy): self-heal startup/reload prisma reads on engine disconnect by [@​michelligabriele](https://github.com/michelligabriele) in [#​28803](https://github.com/BerriAI/litellm/pull/28803) - chore(ui): make knip recognize .mjs scripts and openapi-typescript by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30052](https://github.com/BerriAI/litellm/pull/30052) - fix(register\_model): preserve built-in cache pricing when registering custom overrides under unmapped keys by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30044](https://github.com/BerriAI/litellm/pull/30044) - \[internal copy of [#​28007](https://github.com/BerriAI/litellm/issues/28007)] Fix/gcp model garden streaming by [@​mateo-berri](https://github.com/mateo-berri) in [#​28363](https://github.com/BerriAI/litellm/pull/28363) - feat(cli): per-agent `lite claude` / `codex` / `opencode` commands that wrap coding agents through the proxy by [@​mateo-berri](https://github.com/mateo-berri) in [#​29850](https://github.com/BerriAI/litellm/pull/29850) - fix(callbacks): forward callback\_settings to callback initializers and guard consumers against non-dict values by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30161](https://github.com/BerriAI/litellm/pull/30161) - fix(mcp): drop orphaned per-user credential rows when an MCP server is deleted by [@​tin-berri](https://github.com/tin-berri) in [#​30141](https://github.com/BerriAI/litellm/pull/30141) - fix(proxy): recover from cached-plan errors by reconnecting the Prisma client by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​29983](https://github.com/BerriAI/litellm/pull/29983) - feat(proxy): add option to disable server-side prepared statements for DB lookups by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​29984](https://github.com/BerriAI/litellm/pull/29984) - fix(release): stop backport releases from overwriting the latest badge by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30005](https://github.com/BerriAI/litellm/pull/30005) - feat: add conventional commits and coding guidelines by [@​mateo-berri](https://github.com/mateo-berri) in [#​30159](https://github.com/BerriAI/litellm/pull/30159) - fix(proxy): return 5xx on DB infra errors during auth; reserve 401 for genuine auth failures by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​29986](https://github.com/BerriAI/litellm/pull/29986) - fix(ui): dev server 404s on migrated-page links because uiBase hardcodes /ui by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30169](https://github.com/BerriAI/litellm/pull/30169) - refactor(ui): consolidate dashboard to one shell in the (dashboard) layout by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30166](https://github.com/BerriAI/litellm/pull/30166) - fix(proxy): align /v1/model/info with router deployments by [@​Sameerlite](https://github.com/Sameerlite) in [#​30025](https://github.com/BerriAI/litellm/pull/30025) - fix: completion\_cost AttributeError on streaming Anthropic web\_search responses ([#​26153](https://github.com/BerriAI/litellm/issues/26153)) by [@​ishaan-berri](https://github.com/ishaan-berri) in [#​27346](https://github.com/BerriAI/litellm/pull/27346) - \[internal copy of [#​30137](https://github.com/BerriAI/litellm/issues/30137)] perf(realtime): eliminate redundant per-frame JSON work on OpenAI realtime relay by [@​mateo-berri](https://github.com/mateo-berri) in [#​30142](https://github.com/BerriAI/litellm/pull/30142) - feat(bedrock): aws\_bedrock\_project\_id for bedrock-mantle project / workspace association by [@​mateo-berri](https://github.com/mateo-berri) in [#​30163](https://github.com/BerriAI/litellm/pull/30163) - chore(hooks): enforce Conventional Commits and Conventional Branches by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30174](https://github.com/BerriAI/litellm/pull/30174) - feat(rate-limiter): allow opting out of v3 TPM reservation and Redis circuit breaker by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30211](https://github.com/BerriAI/litellm/pull/30211) - feat(spend\_logs): opt-in native Postgres partitioning for SpendLogs retention by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​29466](https://github.com/BerriAI/litellm/pull/29466) - feat(ui): migrate playground to path routing and colocate its files by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30185](https://github.com/BerriAI/litellm/pull/30185) - feat(ui): migrate projects and access-groups to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30226](https://github.com/BerriAI/litellm/pull/30226) - fix(proxy): coalesce NULL rollup metrics in aggregated daily-activity by [@​michelligabriele](https://github.com/michelligabriele) in [#​30151](https://github.com/BerriAI/litellm/pull/30151) - fix(anthropic\_passthrough): resolve costing model from message\_start chunk, litellm\_params and model\_group instead of 'unknown' by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30160](https://github.com/BerriAI/litellm/pull/30160) - feat(ui): migrate budgets, workflows, and guardrails-monitor to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30236](https://github.com/BerriAI/litellm/pull/30236) - feat(ui): migrate mcp-servers, search-tools, tag-management, vector-stores, and memory to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30261](https://github.com/BerriAI/litellm/pull/30261) - fix(a2a): forward agent\_extra\_headers through completion bridge by [@​mateo-berri](https://github.com/mateo-berri) in [#​28277](https://github.com/BerriAI/litellm/pull/28277) - fix(gemini-live): forward audio buffer commit and correct Vertex PCM rate by [@​Sameerlite](https://github.com/Sameerlite) in [#​29946](https://github.com/BerriAI/litellm/pull/29946) - fix(proxy): skip double-wrapping unified batch output file ids on retrieve by [@​Sameerlite](https://github.com/Sameerlite) in [#​30011](https://github.com/BerriAI/litellm/pull/30011) - feat: litellm oss 110626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​30202](https://github.com/BerriAI/litellm/pull/30202) - fix(docker): copy only runtime artifacts into the final image by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30243](https://github.com/BerriAI/litellm/pull/30243) - feat(proxy): enforce key/team guardrails on bedrock passthrough routes by [@​Sameerlite](https://github.com/Sameerlite) in [#​30194](https://github.com/BerriAI/litellm/pull/30194) - feat(gemini): forward web search tools in image generation by [@​Sameerlite](https://github.com/Sameerlite) in [#​30119](https://github.com/BerriAI/litellm/pull/30119) - fix: bedrock mantle fixes by [@​Sameerlite](https://github.com/Sameerlite) in [#​30083](https://github.com/BerriAI/litellm/pull/30083) - feat(proxy): add require\_managed\_files setting for file uploads by [@​Sameerlite](https://github.com/Sameerlite) in [#​30186](https://github.com/BerriAI/litellm/pull/30186) - fix(mcp): honor server\_id for REST tool calls with shared upstream URLs by [@​Sameerlite](https://github.com/Sameerlite) in [#​30184](https://github.com/BerriAI/litellm/pull/30184) - fix(responses): presidio PII masking for Azure WebSocket and streaming by [@​Sameerlite](https://github.com/Sameerlite) in [#​30003](https://github.com/BerriAI/litellm/pull/30003) - feat(passthrough): add configurable pass-through request timeouts by [@​Sameerlite](https://github.com/Sameerlite) in [#​30266](https://github.com/BerriAI/litellm/pull/30266) - fix(google\_genai): preserve complete SSE events in Vertex/Gemini image streaming by [@​Sameerlite](https://github.com/Sameerlite) in [#​30270](https://github.com/BerriAI/litellm/pull/30270) - fix(proxy): populate access\_via\_team\_ids on /v1/model/info by [@​Sameerlite](https://github.com/Sameerlite) in [#​30274](https://github.com/BerriAI/litellm/pull/30274) - chore(oss): litellm oss staging 120626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​30292](https://github.com/BerriAI/litellm/pull/30292) - feat(ui): migrate policies, guardrails, prompts, tool-policies, and skills to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30263](https://github.com/BerriAI/litellm/pull/30263) - feat(ui): migrate caching, cost-tracking, transform-request, ui-theme, and logs to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30267](https://github.com/BerriAI/litellm/pull/30267) - fix(ui): gate dashboard layout on ui config load so deep links work under SERVER\_ROOT\_PATH by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30312](https://github.com/BerriAI/litellm/pull/30312) - feat(ui): migrate admin-panel, logging-and-alerts, model-hub-table, and usage to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30268](https://github.com/BerriAI/litellm/pull/30268) - fix(otel): cap metric attribute cardinality with include/exclude lists by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30257](https://github.com/BerriAI/litellm/pull/30257) - fix(proxy): grace-period key rotation 401s; return deprecated-key lookup result directly by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30327](https://github.com/BerriAI/litellm/pull/30327) - chore(deps): bump vitest, brace-expansion, pypdf and tornado by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30220](https://github.com/BerriAI/litellm/pull/30220) - refactor(ui): remove unreachable /chat page by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30178](https://github.com/BerriAI/litellm/pull/30178) - feat(ui): migrate agents and router-settings to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30323](https://github.com/BerriAI/litellm/pull/30323) - feat: strengthen coding conventions in CLAUDE.md by [@​mateo-berri](https://github.com/mateo-berri) in [#​30333](https://github.com/BerriAI/litellm/pull/30333) - feat(ui): cut the users page over to the /ui/users path route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30334](https://github.com/BerriAI/litellm/pull/30334) - feat: ruff strict-rule suppressions baseline gate by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30303](https://github.com/BerriAI/litellm/pull/30303) - feat(guardrails): add Cisco AI Defense integration ([#​28249](https://github.com/BerriAI/litellm/issues/28249)) by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30338](https://github.com/BerriAI/litellm/pull/30338) - chore(ui): remove dead UI components unreferenced by any page by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30340](https://github.com/BerriAI/litellm/pull/30340) - ci: add osv-scanner lockfile scan workflow by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30222](https://github.com/BerriAI/litellm/pull/30222) - fix(otel): record full error message on standard exception event in otel v2 by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30380](https://github.com/BerriAI/litellm/pull/30380) - test(fireworks): mock whisper transcription tests instead of live calls by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30391](https://github.com/BerriAI/litellm/pull/30391) - build(ui): pin esbuild to 0.28.1 via overrides by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30390](https://github.com/BerriAI/litellm/pull/30390) - feat(ui): cut the organizations page over to the /ui/organizations path route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30336](https://github.com/BerriAI/litellm/pull/30336) - fix(proxy): support SMTP implicit SSL (port 465) by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30395](https://github.com/BerriAI/litellm/pull/30395) - fix(mcp): default Linear MCP registry entry to streamable HTTP by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30396](https://github.com/BerriAI/litellm/pull/30396) - fix(ui): stop Virtual Keys page from infinite render loop by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30397](https://github.com/BerriAI/litellm/pull/30397) - fix(streaming): guard raise\_on\_model\_repetition against empty choices by [@​shivamrawat1](https://github.com/shivamrawat1) in [#​30485](https://github.com/BerriAI/litellm/pull/30485) - feat(otel-v2): emit the 6 gen\_ai.client.\* metrics at parity with v1 by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30326](https://github.com/BerriAI/litellm/pull/30326) - fix(mcp): drop phantom 401 span on delegated OAuth2 tool calls by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30494](https://github.com/BerriAI/litellm/pull/30494) - feat(ui): cut the teams page over to the /ui/teams path route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30343](https://github.com/BerriAI/litellm/pull/30343) - fix(integrations): cap Anthropic cache\_control injection at 4 blocks by [@​shivamrawat1](https://github.com/shivamrawat1) in [#​30480](https://github.com/BerriAI/litellm/pull/30480) - chore(codecov): add Batches, Videos, and Realtime components by [@​Sameerlite](https://github.com/Sameerlite) in [#​30517](https://github.com/BerriAI/litellm/pull/30517) - test(batches): move orphan tests into tests/test\_litellm for CI coverage by [@​Sameerlite](https://github.com/Sameerlite) in [#​30510](https://github.com/BerriAI/litellm/pull/30510) - fix(guardrails): run pre\_call hook once for model-level guardrails by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30543](http…
15 tasks
mubashir1osmani
added a commit
to mubashir1osmani/litellm
that referenced
this pull request
Jul 27, 2026
Two shipped regressions that had no live coverage, both about a number a customer reconciles against something external. Bedrock's regional Claude variants are priced above the global variant, and the cost-map mappings regressed so a regional deployment resolved the global rate (LIT-3912), silently under-billing every regional call. The test registers the global, us and eu variants of one model and pins each resolved rate from /model/info, asserting the exact expected uplift rather than regional > global: a mapping that landed on some other model's rate would satisfy an inequality while being just as wrong. It reads pricing only, so it needs no AWS credentials and spends nothing. A 429 is actionable only if it says who produced it, because a vendor rejection means back off or fail over while a gateway rejection means the key's own limit is too low. exception_class is what separates them, HTTPException for the gateway's limiter against a provider error class for the vendor, and api_provider is what tells a multi-provider deployment which upstream was involved (PR BerriAI#27687). The test drives a key past its own RPM limit, which is unambiguously gateway-side, and requires the resulting failure series to be labelled that way and to name a provider. Both were mutation-checked against the live proxy: claiming no uplift, claiming the gateway 429 carries a vendor error class, scraping for an alias that made no calls, and requiring an empty provider each fail the relevant test, and both pass restored.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Status
Rebased onto current
litellm_internal_staging(post-#28909 typed v2 OTel refactor). 5 mechanical merge conflicts resolved (textual collisions withprometheus_user_budget_label_include_email_alias, the gpt-3.5-turbo to gpt-5-mini CI-model rename, batch_rate_limiter imports, the auth-not-ready guarddescribeblock; no semantic overlap). Branch is now mergeable.The v2 OTel span integration that earlier revisions layered on #28909 has been dropped after review (yassin-berriai): proxy-side rate limits are rejected at the gate before any upstream call, so the synthetic failure log is tagged
LITELLM_LOGGING_NO_UPSTREAM_LLM_CALLand the v2 logger never opens an LLM-call span for them, which left those attributes dead for exactly the cases they were meant to surface. The decomposition still reaches downstream consumers through the StandardLoggingPayload custom-callback fields and the Prometheus labels.QA Proof
Stood up the proxy from this branch on
localhost:4000against realapi.openai.com/gpt-4o-mini, withprometheus_emit_rate_limit_labels: trueplus a tiny custom callback that dumpsStandardLoggingPayload.error_informationon every failure. Ran the same two scenarios twice in one session: once on the parent commitbaa68ebb12(BEFORE state, no PR commits applied), once on this PR's HEAD276714bdcb(AFTER), with the proxy actually restarted between them. Samecurlboth runs.Scenario A: virtual key
rpm_limit=1, two/v1/chat/completions; the second 429s through the parallel-request limiter. Scenario B: virtual keymax_budget=1e-08; the second call 429s throughBudgetExceededError.BEFORE (parent commit
baa68ebb12)The custom callback's structured payload only carries
[error_class, error_code, error_message, llm_provider, traceback], withllm_provider=""empty. The Prometheus row carriesexception_class="HTTPException"(or"BudgetExceededError") and norate_limit_category/rate_limit_typelabels at all.BEFORE state: callback dump shows empty llm_provider and no rate_limit_* fields; Prometheus rows have no rate_limit_category / rate_limit_type labels
AFTER (this PR's HEAD
276714bdcb)Both scenarios produce
error_rate_limit_category="litellm_rate_limit"pluserror_rate_limit_type="requests"(RPM case) /"budget"(budget case), withllm_provider="openai"populated.exception_typeis"ProxyRateLimitError"(RPM) /"BudgetExceededError"(budget). The Prometheus row now carriesrate_limit_category="litellm_rate_limit"andrate_limit_type="requests"/"budget"whileexception_class="HTTPException"/"BudgetExceededError"is preserved by the back-compat shim. Wire-format headers (retry-after: 60,rate_limit_type: requests,reset_at: ...) survive on the 429 in both states.AFTER state: pretty-printed JSON shows error_rate_limit_category=litellm_rate_limit and error_rate_limit_type=requests|budget; Prometheus rows have the new labels
Summary table
error_information.error_rate_limit_categoryon callback"litellm_rate_limit"error_information.error_rate_limit_typeon callback"requests"/"budget"error_information.llm_provideron callback"""openai"error_information.error_classon callback (RPM 429)"HTTPException""ProxyRateLimitError"rate_limit_categorylabel"litellm_rate_limit"rate_limit_typelabel"requests"/"budget"exception_classlabel"HTTPException"/"Budget…"retry-after,rate_limit_type,reset_at)Relevant issues
Linear ticket
Resolves LIT-2968
Pre-Submission checklist
tests/test_litellm/(test_rate_limit_error_unification.py— 71 tests total: 30 from the original consolidation pass + 41 new tests for therate_limit_typefield, the v3 → public mapping helper, and the per-hook wiring;test_proxy_rate_limit_provider_field.py— 34 tests for themodel/llm_providerresolver wiring)Type
🧹 Refactoring + 🆕 minor New Feature (the
category+rate_limit_typeAPI onRateLimitError, plusmodel+llm_providerpopulated on every internal proxy-side 429)Changes
Background
LiteLLM previously surfaced rate-limit conditions through several unrelated error classes:
litellm.exceptions.RateLimitError— raised by exception mapping when an upstream LLM provider returns 429.fastapi.HTTPException(status_code=429, ...)— raised directly by every proxy-side limiter (parallel_request_limiter,parallel_request_limiter_v3,dynamic_rate_limiter,dynamic_rate_limiter_v3,batch_rate_limiter,max_budget_limiter,max_budget_per_session_limiter,max_iterations_limiter).BaseLLMException(status 429) — raised by some provider transports.This made it impossible for downstream / user code to express "is this a rate limit?" with a single
exceptclause, and impossible to distinguish where the rate limit originated (vendor vs. litellm, batch vs. chat) or which dimension tripped the limit (RPM vs TPM vs concurrent vs budget vs max-iterations) or which provider the limited request was destined for, without ad-hoc string matching.What this PR does
Unifies the class hierarchy. Every rate-limit condition in litellm is now an instance of
litellm.RateLimitError. Proxy-side limiters raiseProxyRateLimitError(HTTPException, RateLimitError)— a single instance that satisfies bothexcept RateLimitErrorandisinstance(e, HTTPException), so existing FastAPI plumbing keeps working unchanged and theretry-after/rate_limit_type/reset_atheaders are preserved on the wire.Adds three orthogonal classification fields:
category(RateLimitErrorCategoryenum) — who rate-limited the request:vendor_rate_limit— upstream LLM provider returned 429vendor_batch_rate_limit— upstream provider returned 429 on a batch endpointlitellm_rate_limit— litellm's own limiter (key/team/user/model RPM/TPM/parallel-requests/budget) blocked the requestlitellm_batch_rate_limit— litellm's batch limiter blocked a batch submissionrate_limit_type(RateLimitTypeenum) — which dimension was exceeded:requests— RPM ceilingtokens— TPM ceilingconcurrent_requests—max_parallel_requestscapbudget— spend cap (key, team, user, or per-session)max_iterations— per-session max-iterations cap (agent flows)model+llm_provider— populated on every internal proxy-side rate-limit error via a defensive resolver (resolve_llm_provider_for_rate_limit) that runslitellm.get_llm_providerondata["model"]. Falls back to("", "litellm_proxy")when the model is missing or unparseable so the rate-limit raise is never masked by a secondary exception. (This piece was originally #27707 — folded into this PR so a single 429 carriescategory+rate_limit_type+model+llm_providerin one place.)Surfaces both classification fields through the StandardLoggingPayload (
error_information.error_rate_limit_categoryanderror_information.error_rate_limit_type) so custom callbacks / metrics builders can split rate-limit failures by source AND cause without ever reaching for the raw exception.Usage
Custom callback access (for custom-metrics builders)
Both classification fields are also surfaced through the structured callback contract —
StandardLoggingPayload.error_information.{error_rate_limit_category,error_rate_limit_type}— so custom callbacks can drive their own rate-limit metrics without ever reaching for the raw exception:Both fields are
Nonefor non-rate-limit errors so consumers can read them unconditionally.Backward compatibility
A naive consolidation would have broken every proxy hook test that asserts
pytest.raises(HTTPException), every FastAPI handler branch that doesif isinstance(e, HTTPException): ..., and every wire-format expectation aroundretry-afterheaders.To avoid that, the proxy-side surface (
ProxyRateLimitError) inherits from both bases via cooperative multiple inheritance:class ProxyRateLimitError(HTTPException, RateLimitError). The result: the same instance satisfiesexcept RateLimitError,isinstance(e, HTTPException), and serializes correctly through FastAPI's default 429 dispatcher with theretry-after/rate_limit_type/reset_atheaders preserved (which the previousHTTPException → ProxyExceptionpath silently dropped).Both new kwargs —
categoryandrate_limit_type— are optional with safe defaults:RateLimitError(...)defaultscategory=VENDOR_RATE_LIMIT(matches the existingexception_mapping_utils429 paths) andrate_limit_type=None(vendor 429 makes no claim about which dimension was hit).ProxyRateLimitError(...)defaultscategory=LITELLM_RATE_LIMIT(since this class is only used by litellm-side limiters) andrate_limit_type=None(callers populate it explicitly).Existing Prometheus dashboards continue to see
exception_class="HTTPException"for proxy-side 429s —_get_exception_class_namecarries a back-compat shim that returns the historical literal for anyProxyRateLimitErrorinstance. Newrate_limit_category/rate_limit_typePrometheus labels surface the new fields without breaking the oldexception_classqueries.Files touched
Core:
litellm/exceptions.py— addsRateLimitErrorCategory+RateLimitTypeenums andcategory/rate_limit_type/headers/detailparameters onRateLimitError. Defaults preserve today's behavior at every existing call site.litellm/__init__.py— re-exportsRateLimitErrorCategoryandRateLimitTypealongsideRateLimitError.Logging payload:
litellm/types/utils.py— addserror_rate_limit_categoryanderror_rate_limit_type(bothOptional[str]) toStandardLoggingPayloadErrorInformation.litellm/litellm_core_utils/litellm_logging.py— populates the new fields inget_error_information()fromgetattr(exception, "category"/"rate_limit_type", None), gated on a singleisinstance(litellm.RateLimitError)check.New file:
litellm/proxy/common_utils/proxy_rate_limit_error.py— definesProxyRateLimitError(HTTPException, RateLimitError)with structured-detail message extraction and stringified header values, plus amap_v3_rate_limit_type()helper that collapses the v3 limiter's internal labels ("requests","tokens","max_parallel_requests") onto the publicRateLimitTypeenum (so the v3 jargonmax_parallel_requestsbecomes the publicconcurrent_requests, while the raw v3 string stays on the HTTP header for wire-format compat). Constructor normalizesNonemodel/llm_providerto safe defaults.Provider-resolution helper:
litellm/proxy/hooks/rate_limiter_utils.py— addsresolve_llm_provider_for_rate_limit(model)which wrapslitellm.get_llm_providerdefensively and returns(resolved_model, llm_provider)for the rate-limit raise site. Falls back to("", "litellm_proxy")when the model is missing or unparseable so the rate-limit raise is never masked.Proxy hooks (refactored to raise
ProxyRateLimitErrorwithcategory,rate_limit_type,model, andllm_providerpopulated):litellm/proxy/hooks/parallel_request_limiter.py— detects which dimension actually tripped the limit (concurrent → tokens → requests, mirroring the boolean condition order); base case picks the most-specific zero limit.raise_rate_limit_error()now also acceptsrequested_modeland runs the resolver.litellm/proxy/hooks/parallel_request_limiter_v3.py— forwardsstatus["rate_limit_type"]throughmap_v3_rate_limit_type(); resolver runs at the raise site with the requested model.litellm/proxy/hooks/dynamic_rate_limiter.py— TPM-zero →TOKENS, RPM-zero →REQUESTS. Resolver runs per branch.litellm/proxy/hooks/dynamic_rate_limiter_v3.py— usesmap_v3_rate_limit_type()at every raise site (model_saturation_check,priority_model, fail-closed unknown-descriptor guard); resolver runs once before the loop and feeds every raise.litellm/proxy/hooks/batch_rate_limiter.py— usesLITELLM_BATCH_RATE_LIMITcategory;limit_type(already typed"requests"|"tokens") maps directly to the enum._raise_rate_limit_erroracceptsrequested_model.litellm/proxy/hooks/max_budget_limiter.py,max_budget_per_session_limiter.py—BUDGET. Both run the resolver.litellm/proxy/hooks/max_iterations_limiter.py—MAX_ITERATIONS. Runs the resolver.Tests:
tests/test_litellm/test_rate_limit_error_unification.py— 71 tests across nine test classes, covering: enum exports, default values, string ↔ enum coercion,ProxyRateLimitError's dual-base nature, wire-format preservation (detail,headers,status_code,retry-after),StandardLoggingPayloadpropagation for both fields, themap_v3_rate_limit_type()mapping (including the unknown→None defensive branch), end-to-end raise-site coverage for every refactored hook (v1 + v3 parallel, v1 + v3 dynamic, batch tokens/requests, max-budget, max-iterations), and a parametrized regression guard that asserts every proxy hook module imports the unified class.tests/test_litellm/proxy/hooks/test_proxy_rate_limit_provider_field.py— 34 tests pinning themodel/llm_providerresolver behavior end-to-end at every limiter raise site (happy path with known providers, fallback path with unknown / missing models, and the Prometheusexception_classback-compat shim).Test results
Note
Medium Risk
Touches every proxy rate-limit raise path and exception/Prometheus labeling; defaults and shims limit breakage, but misconfigured
prometheus_emit_rate_limit_labelsor reliance on old exception shapes still needs careful rollout.Overview
This PR unifies rate-limit handling so vendor and proxy-side 429s share one exception model with structured metadata, while keeping FastAPI and Prometheus behavior stable for existing operators.
RateLimitErrorgainscategory(vendor vs litellm, chat vs batch),rate_limit_type(RPM, TPM, concurrent, budget, max iterations), optional explicitheaders(vendor response headers are no longer copied onto the error automatically), anddetail. New enumsRateLimitErrorCategoryandRateLimitTypeare exported, with validators so arbitrary third-party.categorystrings do not pollute logs or metrics.BudgetExceededErrorkeeps its ownexcepttype but exposes the same category/type fields and optionalllm_provider.Proxy limiters stop raising bare
HTTPException(429)and instead raiseProxyRateLimitError(bothHTTPExceptionandRateLimitError), withresolve_llm_provider_for_rate_limitfillingmodel/llm_providerfrom the request (including router alias fallback). Hooks updated include parallel (v1/v3), dynamic (v1/v3), batch, max budget, per-session budget, and max iterations.Observability:
StandardLoggingPayloadaddserror_rate_limit_categoryanderror_rate_limit_type. Prometheus can opt in viaprometheus_emit_rate_limit_labelsforrate_limit_category/rate_limit_typeonlitellm_proxy_failed_requests_metric;exception_classstaysHTTPExceptionforProxyRateLimitErrorandBudgetExceededErrorfor dashboard compatibility. Label sets are cached at Prometheus logger init so runtime toggles do not desync metric registration from.labels()calls. Auth fillsllm_provideron budget errors when missing.Reviewed by Cursor Bugbot for commit e04e4c2. Bugbot is set up for automated code reviews on this repo. Configure here.