feat(otel): typed semconv-aligned OpenTelemetry instrumentation by yassin-berriai · Pull Request #28909 · BerriAI/litellm

yassin-berriai · 2026-05-26T22:41:07Z

Summary

Opt-in OpenTelemetry instrumentation for LiteLLM, gated by the off-by-default
LITELLM_OTEL_V2 environment variable. When the flag is unset, nothing in this
package runs and existing logging is unaffected.

It follows the standard OTel-for-a-web-app pattern: the proxy's FastAPI app is
instrumented directly, so HTTP server spans are owned by
opentelemetry-instrumentation-fastapi, and a small typed engine emits the
gen-ai spans (LLM calls, guardrails, internal service/DB calls) that nest
under them. Everything lives under litellm/integrations/otel/ (see the package
README.md).

How it works

A traced proxy request produces one trace. The request-level gen-ai spans are
siblings under the server span — a guardrail is part of the request
lifecycle (a pre-call guardrail runs before the LLM call even starts), so it
parents to the server span alongside the LLM call, not under it. Auth is given a
live phase span so the DB lookups it triggers nest under it instead of
flattening onto the server span:

SERVER span  "POST /v1/chat/completions"        ← FastAPI instrumentation
├── INTERNAL span  "auth /v1/chat/completions"   ← auth phase     ┐
│   ├── CLIENT span  "postgres get_key_object"    ← datastore call │
│   └── CLIENT span  "postgres get_team_membership"                │
├── INTERNAL span  "execute_guardrail …"         ← guardrail       │ this package
├── CLIENT span    "chat gpt-4o"                  ← LLM call        │
└── CLIENT span    "batch_write_to_db …"          ← spend write    ┘

Server spans are created by the FastAPI instrumentation. It stamps
http.* attributes and extracts inbound traceparent headers. Request routes
never create or modify spans.
Gen-ai spans come from a CustomLogger adapter that turns LiteLLM's
logging callbacks into typed span data and hands it to the engine.
Parenting is explicit, not purely ambient. Request-level spans (LLM call,
guardrail) anchor to an explicit request root span — set_request_root_span
captures the server span once at the auth boundary, and resolve_request_span_context
reads it — rather than to whatever span is momentarily active. Pure-ambient
parenting was wrong at two boundaries: inside the live auth span the active
span is auth (the LLM span would wrongly nest under auth), and a pass-through
request closes its span from a detached asyncio.create_task where the server
span is no longer active (the span would orphan into its own trace). The anchor
is a contextvar inherited by those child tasks, giving a stable parent in both
cases. DB/service spans keep ambient parenting, so an auth DB lookup still
nests under auth.
The LLM-call span is born at the boundary. log_pre_api_call runs
synchronously in the request task just before the upstream call and opens
the span (parented to the anchor); the async success/failure callback builds
the typed LLMCallSpanData (token usage/cost are only known by then), stamps
attributes, sets status, and closes it. The open span is held in a bounded
cache keyed by litellm_call_id — no live Span ever travels through a
litellm_params metadata dict. When pre_call runs off the request task (a
sync-only provider on a thread pool, where the anchor contextvar doesn't
follow), creation is deferred to the async callback, whose worker context
was copied from the request task at enqueue.
One shared provider. FastAPI middleware can only be added before the app
serves, so the app is instrumented at import time with no provider — it
binds to the OTel global ProxyTracerProvider. After config (and callbacks)
load, the proxy publishes the chosen logger's TracerProvider as the global
via trace.set_tracer_provider(...); the server spans then delegate to it, so
both kinds share one provider and the same trace. With a preset callback, that
preset's provider becomes the global, so server spans export to its backend too.

Which service calls become spans

LiteLLM's service-logging layer instruments many internal functions; only some
are traceable units of work (spans.span_role_for_service):

DB_CALL (CLIENT) — outbound datastore calls (redis, postgres,
batch_write_to_db), carrying db.system.name / db.operation.name semconv.
SERVICE (INTERNAL) — genuine internal work worth a span (background
budget/reset jobs, pod-lock manager).
metrics-only (no span) — self (the track_llm_api_timing wrapper, which
duplicates the LLM-call span), router (duplicates the request), and
proxy_pre_call (whose real span is execute_guardrail …). These still feed
Prometheus/Datadog through their own hooks; they just never enter the trace.
auth is excluded here because it gets a live phase span instead.

Architecture (`litellm/integrations/otel/`)

Sources of truth (model/, no opentelemetry import — importable without the SDK):

model/semconv.py — attribute-key constants + GenAI operation/provider mapping
model/spans.py — the span registry (role → kind, hierarchy, name builder, span_role_for_service)
model/payloads.py — frozen typed span-data dataclasses built from logging payloads; sanitize_event_metadata
model/config.py — OpenTelemetryV2Config (pydantic-settings, reads OTEL_* / LITELLM_OTEL_* envs) + the feature gate
model/baggage.py — the single definition of which request-identity values are promoted into Baggage and under which keys
model/metadata.py, model/utils.py — metadata extraction, value coercion, JSON serialization shared across the package

Engine:

emitter.py — SpanEmitter.emit(role, data): dedupe → start → mapper chain → status → end; owns no attribute keys. The dedupe set is a bounded LRU.
mappers/ — each mapper turns span data into a flat {key: value} dict; they compose, layering multiple attribute vocabularies onto one span:
- genai (canonical OTel GenAI, always present), legacy (semconv-ai / Traceloop key names), and the vendor vocabularies openinference, langfuse, weave, langtrace

Plumbing (plumbing/, imports only model/): providers.py (TracerProvider + exporters + Baggage span processor + an exporter-factory registry), context.py (trace-context + Baggage + request-anchor helpers), routing.py (TenantTracerCache for per-request multi-tenant credentials), metrics.py.

Adapter: logger.py — OpenTelemetryV2(CustomLogger), registers itself into litellm.input_callback so Logging.pre_call fires the boundary hook.

Mount: mount.py — instrument_fastapi_app(app), the single site that attaches opentelemetry-instrumentation-fastapi for SERVER spans, owns the default route-exclusion set and the passthrough span-naming hook. Runtime: runtime.py — startup wiring that publishes the chosen provider as the OTel global.

Presets: presets/ — each reads one integration's env vars and returns an OpenTelemetryV2Config (exporter destination + mapper vocabularies + resource attributes). PRESET_BY_CALLBACK maps a callback name to its preset; the factory in litellm_logging.py resolves the name and builds the logger. Presets do no network I/O at build time.

Callback name	Mapper vocabularies	Destination	Dynamic creds
`arize`	`openinference`	Arize OTLP gRPC	✓
`arize_phoenix`	`openinference`	Phoenix OTLP + `openinference.project.name` resource attr	—
`langfuse_otel`	`langfuse`	Langfuse OTLP HTTP	✓
`weave_otel`	`openinference` + `weave`	W&B OTLP HTTP	✓
`agentops`	—	AgentOps OTLP HTTP (JWT minted lazily on first export, off the event loop)	—
`levo`	—	Levo OTLP HTTP	—
`langtrace`	`langtrace`	customer's existing OTLP collector	—

Multi-tenant: when a request carries team/key-scoped vendor credentials
(standard_callback_dynamic_params), TenantTracerCache routes its spans
through a credential-scoped TracerProvider (a bounded LRU, cached per
credential set and flushed/shut down on eviction), so one logger serves many
tenants without unbounded provider/thread growth. Integrations that support this
provide a per-request OTLP header builder (DYNAMIC_HEADERS_BY_CALLBACK).

Excluded routes

Noisy non-LLM routes are dropped from tracing by default
(mount._DEFAULT_EXCLUDED_ROUTES): health checks (/health*), the Prometheus
scrape (/metrics), and static UI/docs assets (/_next, /ui, /swagger,
/docs, /redoc, /openapi.json, /.well-known, …). Entries are
substring-matched. Override the whole set with OTEL_PYTHON_FASTAPI_EXCLUDED_URLS
("" to trace everything, or a comma-separated path list).

Content capture

Prompt and response bodies are written as span attributes only when
capture_span_content opts in (span_only / span_and_event). It defaults
to no_content, and the gate is applied at the payload source so every
vendor mapper (openinference, langfuse, weave, langtrace) is covered at once —
a user request can't force its prompt/completion into the backend while capture
is disabled. Caller-supplied event_metadata is sanitized (primitives only, no
live objects, no secrets/headers, bounded) before it reaches a span.

Review feedback addressed

Correct parenting at every boundary — request-level spans anchor to an explicit captured request-root span instead of the momentarily-active span, fixing LLM spans nesting under the live auth span and pass-through spans orphaning out of the request trace; DB/service spans keep ambient parenting so auth DB lookups still nest under auth.
No live Span through metadata dicts — the open LLM-call span is held in a bounded litellm_call_id-keyed cache, not threaded through litellm_params.
Service/DB spans now register — litellm._service_logger recognizes the V2 OpenTelemetryV2 logger (a plain CustomLogger, not a subclass of the legacy OpenTelemetry) and the adapter classifies each call into DB_CALL / SERVICE / metrics-only.
Unbounded growth fixed — the emitter dedupe set and the per-tenant provider cache are both bounded LRUs; evicted tenant providers are flushed and shutdown() so their BatchSpanProcessor threads are reclaimed.
No shared-client corruption — the AgentOps auth call owns a short-lived httpx.Client instead of closing the process-wide cached handler.
No event-loop blocking — AgentOps mints its JWT lazily inside a custom exporter on the first export (in the BatchSpanProcessor worker thread), never during callback construction.
One trace, one backend — server spans and gen-ai spans share a provider via the deferred-global-provider pattern above.
Noisy routes excluded — health checks, the metrics scrape, and static UI/docs assets are dropped by default; override with OTEL_PYTHON_FASTAPI_EXCLUDED_URLS.
Configurable Baggage — promoted/metadata key allowlists come from LITELLM_OTEL_BAGGAGE_* env vars or callback_settings.otel.* in config.yaml instead of being hard-coded.

Test plan

OTel unit suite under tests/test_litellm/integrations/otel/ — 166 tests (engine, mappers, presets, dynamic routing + LRU eviction, content-capture gating, AgentOps lazy JWT, baggage promotion + parenting + guardrails, FastAPI-mount server-span + nesting, multi-backend, sources-of-truth invariants)
ruff / mypy / black clean across the package
End-to-end smoke test: ran the proxy locally with LITELLM_OTEL_V2=true, console export, a mock model, and a local guardrail. A chat/completions request produced a single trace — POST /v1/chat/completions (SERVER) with the auth phase span (and its postgres DB calls), chat gpt-4o (CLIENT), execute_guardrail … (INTERNAL), and the spend-write nested under it, carrying the expected gen_ai.* / http.* / litellm.* attributes.
Import-safety: importing litellm.integrations.otel (and the model/ sources of truth) does not require the OTel SDK; the FastAPI instrumentation dependency is optional and imported lazily.
CI green on this branch

🤖 Generated with Claude Code

CLAassistant · 2026-05-26T22:41:13Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

codecov · 2026-05-26T23:12:45Z

Codecov Report

❌ Patch coverage is 88.89582% with 178 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/litellm_core_utils/litellm_logging.py	2.00%	49 Missing ⚠️
litellm/integrations/otel/model/payloads.py	89.21%	22 Missing ⚠️
litellm/integrations/otel/logger.py	89.59%	18 Missing ⚠️
litellm/integrations/otel/presets/arize.py	62.50%	12 Missing ⚠️
litellm/proxy/proxy_server.py	47.05%	9 Missing ⚠️
litellm/integrations/otel/mappers/utils.py	81.08%	7 Missing ⚠️
litellm/integrations/otel/model/metadata.py	93.68%	6 Missing ⚠️
litellm/integrations/otel/presets/phoenix.py	64.28%	5 Missing ⚠️
litellm/integrations/otel/presets/utils.py	28.57%	5 Missing ⚠️
litellm/integrations/otel/runtime.py	70.58%	5 Missing ⚠️
... and 17 more

📢 Thoughts on this report? Let us know!

yassin-berriai · 2026-05-29T01:03:08Z

@greptileai

greptile-apps · 2026-05-29T01:09:02Z

Greptile Summary

This PR introduces an opt-in (LITELLM_OTEL_V2=true) OpenTelemetry v2 instrumentation layer under litellm/integrations/otel/. It is fully gated and leaves all existing logging paths untouched when the flag is off.

New OTel package (litellm/integrations/otel/): typed semconv-aligned span engine, mapper chain, preset configs for Arize/Langfuse/Weave/AgentOps/Levo/LangTrace, multi-tenant credential routing with bounded LRU eviction, and an explicit request-root-span anchor that fixes parenting at the auth/pass-through boundaries.
Proxy integration: FastAPI server spans via opentelemetry-instrumentation-fastapi (lazy import, mounted once at app creation); auth phase wrapped in a live INTERNAL span so DB lookups nest under it; set_tracer_provider deferred to startup so server and gen-ai spans share one provider; log_db_metrics event_metadata sanitized to avoid leaking auth tokens/live objects onto spans.
Previous-round issues addressed: service_callback registration, unbounded emitter/provider caches (now bounded LRUs with thread cleanup on eviction), shared-client closure in AgentOps, blocking JWT fetch (now lazy in worker thread), and the dual-provider split between server and gen-ai spans.

Confidence Score: 5/5

Safe to merge — entirely opt-in behind LITELLM_OTEL_V2, no changes to existing request or logging paths when the flag is off.

All changes are gated by LITELLM_OTEL_V2 and no-op without it. The previous round's correctness issues (unbounded caches, shared-client closure, service_callback registration, blocking JWT, dual providers) are addressed with bounded LRUs and correct lifecycle management. The only concern is the fragile private _session attribute access in the AgentOps exporter, which would cause a silent auth failure on SDK upgrade rather than a crash.

litellm/integrations/otel/presets/agentops.py — the _LazyAuthAgentOpsExporter accesses _session (a private OTLPSpanExporter attribute) to inject the Authorization header; this is the only fragile spot in the package.

Important Files Changed

Filename	Overview
litellm/integrations/otel/logger.py	Core CustomLogger adapter; LLM-call span lifecycle (open at pre_call, close on success/failure), bounded open-call map, service hooks, guardrail spans, and request identity seeding — all implemented correctly with proper deduplication and bounded memory.
litellm/integrations/otel/emitter.py	Span engine with bounded LRU dedup cache (10,000 entries); correctly evicts oldest key on overflow. Previous unbounded-set issue is fixed.
litellm/integrations/otel/plumbing/routing.py	TenantTracerCache uses a bounded OrderedDict LRU (256 entries) with _shutdown_provider on eviction to reclaim BatchSpanProcessor threads. Previous unbounded-growth issue is fixed.
litellm/integrations/otel/presets/agentops.py	Lazy JWT fetch (in BatchSpanProcessor worker thread, not the event loop) using a dedicated short-lived httpx.Client. Uses private _session attribute on OTLPSpanExporter to inject the Authorization header after minting — fragile against OTel SDK internals changes.
litellm/integrations/otel/mount.py	FastAPI server-span instrumentation, gated by LITELLM_OTEL_V2. FastAPIInstrumentor import is lazy (inside the function body), so the optional package is never required when the gate is off.
litellm/integrations/otel/plumbing/context.py	Trace-context helpers: explicit request-root-span anchor (ContextVar), Baggage helpers, resolve_request_span_context for request-level spans vs resolve_parent_context for ambient DB/service spans. Parenting logic looks correct.
litellm/_service_logger.py	_resolve_otel_service_logger now recognizes both legacy OpenTelemetry and OpenTelemetryV2 (a plain CustomLogger, not a subclass). Previous issue where V2 service spans were silently dropped is fixed via direct isinstance check against lazily-imported class.
litellm/proxy/auth/user_api_key_auth.py	Entire auth flow wrapped in phase_span("auth {route}") so DB lookups nest under it; seed_request_identity called after the span closes so Baggage persists past auth. SDK-free wrappers in runtime.py ensure no-op without OTel.
litellm/litellm_core_utils/litellm_logging.py	_maybe_construct_otel_v2 intercepts each preset callback (arize, langfuse_otel, weave_otel, etc.) when LITELLM_OTEL_V2 is on and returns a correctly-configured OpenTelemetryV2 instead of the legacy logger. Dedup loop prevents double-registration.
litellm/proxy/proxy_server.py	V2 startup: searches service_callback for an already-registered preset logger (arize, langfuse, …); falls back to a generic OpenTelemetryV2() only if none found. set_tracer_provider is called once, making server spans and gen-ai spans share one provider.
litellm/proxy/db/log_db_metrics.py	event_metadata no longer dumps function_kwargs/function_args (which carried live Prisma objects, OTel spans, and auth tokens); only the table_name string is surfaced. Correct and avoids data leakage onto spans.

_{Reviews (5): Last reviewed commit: "fix: refactor" | Re-trigger Greptile}

veria-ai · 2026-05-29T01:24:33Z

PR overview

The PR has one remaining security concern: request-supplied OpenTelemetry baggage can be accepted as LiteLLM metadata, allowing an authenticated caller to forge metadata on exported spans. This primarily affects observability and audit/attribution integrity rather than direct application data access, and the issue should be addressed by separating server-derived metadata from inbound baggage or filtering it before export. Two previously flagged issues have already been fixed, so the overall security posture is improving but not yet clean.

Open issues (1)

litellm/integrations/otel/providers.py:62 — Medium: Untrusted baggage metadata export FastAPI instrumentation extracts the request's W3C Baggage header into the active OpenTeleme

Fixed/addressed: 2 · PR risk: 5/10

yassin-berriai · 2026-05-29T02:11:23Z

@greptileai

veria-ai · 2026-05-29T02:22:30Z

+    def __init__(
+        self,
+        allowed_keys: Iterable[str],
+        allowed_prefixes: tuple[str, ...] = (LiteLLM.METADATA_PREFIX,),


Medium: Untrusted baggage metadata export

FastAPI instrumentation extracts the request's W3C Baggage header into the active OpenTelemetry context before spans start, and this processor accepts any litellm.metadata.* key from that context. An authenticated caller can send Baggage: litellm.metadata.some_tenant_field=... and have forged LiteLLM metadata exported on the request's spans; use an internal-only context value for server-derived identity, or clear/filter inbound baggage before stamping LiteLLM attributes.

yassin-berriai · 2026-05-29T02:27:28Z

@greptileai

ryan-crabbe-berri

lgtm

…t_metadata PR #28909 narrowed the DB service log event_metadata to a non-sensitive table_name (dropping function_name/function_kwargs/function_args so live prisma clients, otel spans, and tokens never land on a span), but test_log_db_metrics_success still asserted function_name was present, so it raised TypeError on the now-None metadata. Pass a table_name and assert the metadata surfaces exactly that, matching _safe_db_event_metadata.

test_log_db_metrics_success still asserted the legacy event_metadata shape (function_name/function_kwargs/function_args), which #28909 intentionally removed so that live Prisma clients, OTel spans, and secrets never land on a service-log span. The decorator now emits only a sanitized payload: None when no table_name is present, and {"table_name": ...} when it is. Update the test to verify both branches of that contract.

…Opus 4.7 self-heal) (#29344) * test(logging): align DB metrics event_metadata assertions with safe redaction PR #28909 hardened log_db_metrics to emit a minimal, non-sensitive event_metadata (only table_name when present, otherwise None) instead of dumping function_name, function_kwargs, and function_args onto the span. The test in test_log_db_redis_services was not updated and still asserted "function_name" in event_metadata, which raised TypeError (argument of type 'NoneType' is not iterable) and turned the logging_testing CI job red on litellm_internal_staging. Update test_log_db_metrics_success to assert event_metadata is None when no table_name is passed, and add test_log_db_metrics_event_metadata_is_safe as a regression guard verifying that only the table name surfaces and that sensitive kwargs (tokens, prisma client) are never dumped. * test(bedrock): self-heal opus-4-7 grid cells when unentitled on CI The bedrock-claude-opus-4-7 converse cells are unentitled on the Bedrock CI account, so they were marked xfail. xfail keeps reporting them as expected failures even after access is granted, so the wire translation never gets verified again. Now the cell makes the call and skips only when Bedrock replies "is not available for this account"; the moment the model is entitled the same cells run their full assertions with no edit. A focused unit test pins the tolerance predicate so any other failure still surfaces loudly and the available path still runs the assertions.

…utes (#29327) * test(logging): align DB metrics event_metadata assertions with safe redaction PR #28909 hardened log_db_metrics to emit a minimal, non-sensitive event_metadata (only table_name when present, otherwise None) instead of dumping function_name, function_kwargs, and function_args onto the span. The test in test_log_db_redis_services was not updated and still asserted "function_name" in event_metadata, which raised TypeError (argument of type 'NoneType' is not iterable) and turned the logging_testing CI job red on litellm_internal_staging. Update test_log_db_metrics_success to assert event_metadata is None when no table_name is passed, and add test_log_db_metrics_event_metadata_is_safe as a regression guard verifying that only the table name surfaces and that sensitive kwargs (tokens, prisma client) are never dumped. * test(bedrock): self-heal opus-4-7 grid cells when unentitled on CI The bedrock-claude-opus-4-7 converse cells are unentitled on the Bedrock CI account, so they were marked xfail. xfail keeps reporting them as expected failures even after access is granted, so the wire translation never gets verified again. Now the cell makes the call and skips only when Bedrock replies "is not available for this account"; the moment the model is entitled the same cells run their full assertions with no edit. A focused unit test pins the tolerance predicate so any other failure still surfaces loudly and the available path still runs the assertions. * test(reasoning-effort-grid): add claude-opus-4-8 across provider routes Adds claude-opus-4-8 to the anthropic, azure, vertex and bedrock-converse routes (275 cells total) so the reasoning-effort wire translation is covered for the new model. The bedrock opus-4-8 and opus-4-7 cells reuse the self-heal path: they run the call and skip only on Bedrock's "is not available for this account" reply, then assert in full once the model is entitled. The azure and vertex opus-4-8 cells stay xfail until a Foundry deployment exists and Vertex availability is confirmed. The shared xhigh+max capability set is renamed to _CAPS_XHIGH_MAX now that more than one model uses it.

…29160) * feat(pass_through): extend passthrough_managed_object_ids to Azure Adds managed ID minting/resolution for Azure passthrough endpoints (/azure/...) alongside the existing OpenAI passthrough support. Key changes: - pass_through_endpoints.py: detect azure/azure_ai custom_llm_provider (string or enum) to set _is_managed_id_provider and _managed_id_provider; both INPUT and OUTPUT rewrite blocks now fire for Azure. - llm_passthrough_endpoints.py: forward custom_llm_provider into create_pass_through_route so it reaches pass_through_request (was None). - managed_id_rewriter.py: extend _PASSTHROUGH_PREFIX_RE and _canonical_path to strip /azure/openai prefix and add /v1/ for Azure paths that omit it; add ("azure", method, path) entries to BUILTIN_OUTPUT_ID_FIELD_MAP for files and batches endpoints. - managed_id_codec.py / types/utils.py: supporting codec and enum constant. - proxy_server.py: register llm_passthrough_router before batches_router to prevent route collision for /openai_passthrough/* paths. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(pass_through): remove unused imports for ruff F401 Co-authored-by: Cursor <cursoragent@cursor.com> * fix(pass_through): satisfy mypy for optional parsed_body Co-authored-by: Cursor <cursoragent@cursor.com> * fix(pass_through): compute query params string after managed-ID rewrite Move requested_query_params_str computation to after the managed-ID input rewrite block so logging_url reflects the rewritten raw-provider query params actually sent upstream, instead of the original managed IDs. Co-authored-by: Yassin Kortam <yassin@berri.ai> * Add support for managed ids for passthrough responses api * Add support for list batches and list files * style: run Black on passthrough managed ID files Fix CI formatting for managed_id_rewriter.py and pass_through_endpoints.py. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(passthrough): parse json file_object and implement before-cursor pagination - Parse row.file_object via json.loads when Prisma returns it as a string; mirrors openai_files_endpoints/common_utils.py so list responses keep all stored detail fields (status, timestamps, etc.). - Implement the previously-parsed-but-unused 'before' cursor for list pagination by flipping fetch order to ascending with a 'gt' bound on created_at, then reversing rows so the response stays newest-first. Co-authored-by: Yassin Kortam <yassin@berri.ai> * Remove logger * refactor: split list_passthrough_ids_from_db to fix PLR0915 Extract pagination, fetch, and serialization helpers so the main list function stays under the statement limit without changing behavior. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: scope passthrough managed ID dedup and list by provider Validate embedded provider before reusing deduped file/object rows so OpenAI and Azure cannot share the same managed ID for an identical raw ID. Filter list responses to rows whose managed IDs decode to the current provider, with over-fetch scanning when needed. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(managed_id_rewriter): cap pagination trim at effective limit When raw_limit > 100, fetch_limit is capped at 101 (one extra row to detect has_more), but trimming with rows[:raw_limit] failed to drop the sentinel row. Use the capped effective limit instead. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix: cross-provider object collision and fail-closed list error handling Greptile P1: move provider check before access check in _mint_or_reuse_object so a cross-provider raw ID collision (OpenAI and Azure share the same batch_ ID) falls through to mint a new provider-scoped row instead of raising 404. Veria-ai medium: _fetch_provider_scoped_list_rows now always returns (page, has_more) — DB errors break out of the scan loop and return matched rows so far. list_passthrough_ids_from_db never returns None for a recognised list route, so the caller can never fall through to the upstream provider. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: namespace passthrough model_object_id by provider to prevent unique violation Store model_object_id as 'passthrough:{provider}:{raw_id}' instead of the bare raw ID so OpenAI and Azure can each own a row for the same raw batch ID without hitting the @unique constraint. Dedup lookup uses the same namespaced key so it is implicitly provider-scoped and the _managed_id_matches_provider check is no longer needed on the object path. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: gate list interception on managed_files hook like input/output rewrites Without the hook no managed IDs are minted so the DB is empty. Intercepting GET /v1/files without the hook returned an empty list and hid the caller's real upstream files/batches. Matches the guard used by the input and output rewrite blocks. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: set has_more=True when scan cap is hit with a full final DB batch When max_scans (20) is exhausted and the last DB page was full-sized, there are almost certainly more rows beyond the scan window. Track last_batch_full across iterations so the scan_cap_hit condition sets has_more=True in that case, preventing silent pagination truncation in high-mixed-provider pools. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(managed_id_rewriter): scope pagination cursor lookup to caller-owned rows Prevent a cross-tenant timing oracle by constraining the after/before cursor row lookup to the caller's owner_filter, and cover the real Azure responses path form (no /v1/) in tests. * fix(managed_id_rewriter): align passthrough list metadata with direct GET Persist upstream file metadata when minting a managed file ID and rewrite nested batch file IDs before snapshotting the object, so DB-served file/batch list responses return the same fields and managed IDs as a direct endpoint GET. * fix(managed_id_rewriter): degrade to raw id on cross-owner object collision The OUTPUT (mint) path of _mint_or_reuse_object raised HTTPException(404) when a dedup hit on the namespaced model_object_id belonged to a different owner, converting a successful upstream batch/response creation into a 404 for the caller. Two upstream accounts under one provider name can issue the same raw id, so this is reachable in multi-tenant deployments. Return the caller's raw id unmanaged instead: the upstream create already succeeded, a new managed row can't be minted (model_object_id is @unique), and reusing the other owner's managed id would later fail the access check. * fix(managed_id_rewriter): scope list cursor by provider and cap body-rewrite recursion depth * perf(managed_id_rewriter): push batch list provider scope to DB and anchor canonical-path prefix Object (batch) list rows store model_object_id as passthrough:{provider}:{raw}, so the provider filter is now applied at the indexed DB column, collapsing the application-layer multi-scan to a single query for that table. File rows keep the decode-based scan since they have no provider column. Anchor the canonical-path prefix regex at a path boundary so routes such as /openai_realtime/... are no longer mis-stripped. * fix(managed_id_rewriter): refresh stored batch snapshot on reuse The dedup-reuse path in _mint_or_reuse_object returned the existing managed id without updating the stored file_object, so DB-served list responses kept the creation-time snapshot and showed null output_file_id/error_file_id even after the batch completed. Refresh the snapshot when an owned row is reused so the list reflects the batch's latest state. * fix(managed_id_rewriter): deny cross-owner object access on retrieve/cancel/delete Returning the raw id when can_access_resource fails only made sense for create responses, where the caller's own upstream create succeeded under a raw id that a different owner already holds. On retrieve/cancel/delete the caller reaches that branch only by supplying another tenant's raw id (which bypasses the managed-id input gate), so echoing the upstream object back leaked it cross-tenant. Restrict the raw fallback to create routes and return 404 otherwise. * fix(managed_id_rewriter): deny cross-owner file access on retrieve/delete _mint_or_reuse_file scoped the raw file dedup lookup to the current caller, so a raw file-... id belonging to another tenant was never found and the OUTPUT path minted a fresh managed id for that same upstream file under the caller. A raw id only reaches this path by skipping the managed-id input gate (raw provider ids are opt-out), so a different-owner row means the caller is touching someone else's file. Look up flat_model_file_ids globally and run can_access_resource; deny with 404 on retrieve/delete and leave the raw id unmanaged on create, which mirrors the cross-owner handling already in _mint_or_reuse_object. * fix(managed_id_rewriter): deterministic provider-scoped file dedup Replace the unscoped find_first in _mint_or_reuse_file with a find_many ordered by created_at and an application-layer provider filter. The file table has no provider column, so a raw file id shared across OpenAI and Azure could map to one row per provider; find_first then picked a row non-deterministically and, on a provider mismatch, minted a fresh managed row on every call, accumulating duplicates. Selecting the oldest matching same-provider row the caller can access keeps reuse stable and prevents duplicate rows while preserving the cross-tenant deny/leave-raw behaviour. * refactor(pass_through): scope passthrough managed IDs on the explicit provider Move the openai/azure detection out of pass_through_request into resolve_passthrough_managed_id_provider in llms/base_llm/managed_resources, and key managed-ID rewriting on the forwarded custom_llm_provider rather than the upstream URL's EndpointType. The helper documents why azure and azure_ai collapse to one "azure" scope (they share the same Azure OpenAI files/batches surface, so an ID minted on one must resolve on the other) and returns None for any other provider so a third-party OpenAI-compatible endpoint never triggers managed-ID minting. Add TestManagedIdProviderScope covering the azure_ai -> azure collapse and the non-openai/azure exclusion. * test(log_db_metrics): assert sanitized event_metadata contract test_log_db_metrics_success still asserted the legacy event_metadata shape (function_name/function_kwargs/function_args), which #28909 intentionally removed so that live Prisma clients, OTel spans, and secrets never land on a service-log span. The decorator now emits only a sanitized payload: None when no table_name is present, and {"table_name": ...} when it is. Update the test to verify both branches of that contract. * fix(managed_id_rewriter): page provider-scoped file list by offset The file list scan advanced its cursor with a strict created_at boundary. When several rows shared a created_at timestamp and a non-matching provider row sat on the page boundary, the next query skipped the remaining rows at that timestamp, dropping matching files from the response. Page by a stable offset over a total order (created_at plus the unique id column) so tied rows are never skipped or repeated. * fix(managed_id_rewriter): push file-list provider scope to the DB The file-list helper had no provider column to query, so it scanned the table and filtered by decoding each managed ID in the application layer, capped at 20 pages. For an admin with a large mixed-provider file pool that cap could truncate a page. Mint now writes a _passthrough_provider:{provider} marker into flat_model_file_ids, giving the file table the same DB-queryable provider scope object rows already get from the namespaced model_object_id. The list helper pushes the scope into the query so a single round-trip serves the page. The scan loop, the offset paging, and the cap are gone, so pages can no longer truncate, leak the other provider, or skip rows that share a created_at timestamp. * fix(managed_id_rewriter): deny raw provider IDs that map to another tenant's managed resource Clients only ever receive managed IDs on passthrough, so a raw file/batch/response ID for another tenant's managed object can only be recovered by decoding that tenant's managed ID. Raw IDs were forwarded upstream untouched (deliberate opt-out), which on a retrieve/cancel/delete executed upstream before the response-side ownership check ran, leaking a cross-tenant action. Guard raw provider IDs on the input path: when a raw file-/batch_/resp_ ID resolves to a managed row the caller cannot access, return 404 before forwarding. Genuinely unmanaged raw IDs (no DB row) and IDs the caller owns are left untouched, preserving the opt-out. * test(managed_id_rewriter): cover azure_ai and pre-versioned azure passthrough paths * fix(managed_id_rewriter): fall back to raw id when persistence fails A DB persistence failure after a successful upstream create left the client holding a minted managed ID with no backing row, so every later resolve returned 404 and the resource was permanently unreachable. Mint the managed ID only when the row is stored; on persistence failure return the raw provider id, matching the no-persistence-available fallback, so the freshly-created resource stays reachable. * fix(managed_id_rewriter): log only rewritten query param keys * fix(managed_id_rewriter): use compound (created_at, id) list cursor boundary A timestamp-only lt/gt cursor boundary skips list rows that share the cursor row's created_at across a page boundary, silently dropping them. Compare the unique id (the secondary sort key) alongside created_at so the page walk stays complete when timestamps tie. * fix(managed_id_rewriter): converge concurrent object creates on one managed id * fix(managed_id_rewriter): bound raw-id guard DB lookups per request The INPUT guard fired one DB lookup for every file-/batch_/resp_ prefixed string in the path, query, and body. The file-id guard is an unindexed array-containment scan over LiteLLM_ManagedFileTable, so an authenticated caller could amplify a single passthrough request into thousands of full-table scans by packing a body with id-shaped strings. De-dupe raw ids within a request and cap the distinct guard lookups, failing closed with 400 instead of skipping the guard. Legitimate callers hold managed ids (resolved via an indexed unified_*_id lookup, not the guard), so the cap only trips under abuse. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>

…d LLM-call spans PR #28909 introduced the typed v2 OTel engine that builds spans from StandardLoggingPayload, with SpanError carrying error_type + message and the genai mapper stamping error.type onto every failed LLM-call span. This PR's earlier commits added error_rate_limit_category and error_rate_limit_type to the same StandardLoggingPayload.error_information the v2 engine reads — but neither field reached a span attribute, so v2 OTel traces stayed opaque about *why* a 429 fired (vendor vs litellm, RPM vs TPM vs concurrent vs budget vs max_iterations) even after the custom-callback and prometheus surfaces gained that decomposition. Three coupled changes: 1. semconv.py: add LiteLLM.ERROR_RATE_LIMIT_CATEGORY / LiteLLM.ERROR_RATE_LIMIT_TYPE under the litellm.* vendor namespace (no GenAI semconv equivalent exists for who-rate-limited / which-dimension). 2. payloads.py: extend SpanError with rate_limit_category + rate_limit_type, populated by _parse_error() from the same error_information.error_rate_limit_* fields the custom-callback channel and prometheus rate_limit_category / rate_limit_type labels read. Single source of truth across all three observability surfaces. 3. mappers/genai.py: stamp the two attributes on the LLM-call span when present. drop_none guarantees they stay absent (not 'None') for non-rate-limit failures so trace consumers can read them unconditionally. Three regression tests in test_otel_v2_emitter.py pin: a vendor / litellm-internal RateLimitError lands category=litellm_rate_limit + rate_limit_type=requests on the span; a BudgetExceededError lands rate_limit_type=budget; a non-rate-limit failure (BadRequestError) keeps the rate_limit_* attributes absent. Mutation-tested against reverting either the SpanError extension or the _parse_error read site — both new tests fail under either mutation. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>

…odel, and llm_provider fields (#27687) * feat(exceptions): add RateLimitErrorCategory + headers/detail fields on RateLimitError LiteLLM previously surfaced rate-limit conditions through several unrelated error classes (RateLimitError, FastAPI HTTPException(429), BaseLLMException). This commit adds the data model needed to consolidate them under a single class: * RateLimitErrorCategory enum exposing four categorical values (vendor_rate_limit, vendor_batch_rate_limit, litellm_rate_limit, litellm_batch_rate_limit) so callers can switch on the rate-limit source. * New optional fields on RateLimitError: - category (defaults to vendor_rate_limit, preserving today's behavior for every existing call site in exception_mapping_utils); - headers (preserves retry-after / rate_limit_type / reset_at across the proxy boundary instead of dropping them on the floor); - detail (mirrors FastAPI HTTPException.detail so the same instance can be serialized through both paths). litellm.RateLimitErrorCategory is re-exported at the package root to match the existing exception-export pattern. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(proxy): add ProxyRateLimitError unifying RateLimitError + HTTPException Adds a single proxy-side error class that subclasses BOTH litellm.exceptions.RateLimitError AND fastapi.HTTPException via cooperative multiple inheritance. Why both bases: * Subclassing RateLimitError lets user code catch every rate-limit source with one 'except RateLimitError' and switch on the new .category field. * Subclassing HTTPException keeps every existing FastAPI plumbing path (the isinstance(e, HTTPException) branches in proxy_server.py route handlers, FastAPI's own dispatcher, and tests asserting pytest.raises(HTTPException)) working without modification, and preserves retry-after / rate_limit_type / reset_at headers on the wire. The class declaration order is (HTTPException, RateLimitError) so the MRO puts HTTPException's no-super-call __init__ ahead of openai's cooperative __init__ chain — preventing openai.APIError.super().__init__(message) from landing in HTTPException.__init__(status_code=message). LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from budget + iteration limiters Replaces three bare HTTPException(status_code=429, ...) call sites with ProxyRateLimitError, which is both a RateLimitError (catchable by category) and an HTTPException (preserves existing FastAPI serialization). Drops the now-unused HTTPException import in the iteration / per-session limiters. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from parallel-request limiters Replaces HTTPException(status_code=429, ...) call sites in the v1 and v3 parallel-request limiters (key/team/user/model/customer rate limits) with ProxyRateLimitError. Updates the raise_rate_limit_error helper's return type annotation accordingly. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from dynamic rate limiters Replaces HTTPException(status_code=429, ...) call sites in the v1 and v3 dynamic rate limiters (project-level TPM/RPM allocation, model-saturation checks, priority-based limits, fail-closed guards) with ProxyRateLimitError. The v3 limiter still imports HTTPException for an unrelated bare 'except HTTPException:' branch. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from batch rate limiter Replaces HTTPException(status_code=429, ...) in batch_rate_limiter._raise_rate_limit_error with ProxyRateLimitError tagged as RateLimitErrorCategory.LITELLM_BATCH_RATE_LIMIT so users can distinguish batch-level throttling (which counts requests/tokens across an uploaded batch input file before submission) from the generic key/team/user RPM/TPM limiter. The HTTPException import is retained because the same module raises HTTPException for unrelated 403/IO error paths. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): pin down unified rate-limit error contract Adds a dedicated test module covering the new RateLimitErrorCategory enum, RateLimitError.category default + override behavior, ProxyRateLimitError's dual nature (RateLimitError + HTTPException), and a parametrized regression guard that asserts every proxy hook module imports the unified class. The regression guard catches the failure mode the refactor is designed to prevent: someone re-introducing a bare HTTPException(status_code=429, ...) in one of the hook modules instead of going through ProxyRateLimitError. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(logging): expose rate-limit category via StandardLoggingPayload Adds an optional 'error_rate_limit_category' field to StandardLoggingPayloadErrorInformation, populated from the unified RateLimitError.category attribute (introduced in the previous commits on this branch). Why: the .category attribute is reachable off the raw exception today via getattr(e, 'category', None), but the structured contract that downstream custom callbacks / loggers / spend log writers consume is the StandardLoggingPayload. Without this field, a user building custom rate-limit metrics on top of callback data has to special-case the raw exception object — which defeats the purpose of the StandardLoggingPayload abstraction. The field is None for non-rate-limit exceptions (so consumers can read it unconditionally without isinstance checks) and is one of the RateLimitErrorCategory string values otherwise. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): assert StandardLoggingPayload carries the category Five tests covering: vendor default, explicit litellm_rate_limit and litellm_batch_rate_limit values, None for non-rate-limit exceptions, and None when no exception is provided. Pins down the contract that custom callbacks can read 'error_information.error_rate_limit_category' off the StandardLoggingPayload to drive custom rate-limit metrics without ever reaching for the raw exception. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(types): silence mypy [misc] on intentional dual-base attr overlap mypy emits two [misc] errors on the ProxyRateLimitError class line because its two bases declare overlapping attributes with related-but-not-identical annotations: * status_code: int on starlette HTTPException vs. Literal[429] on openai's RateLimitError (every openai status-error subclass narrows it the same way and silences pyright with the same convention). * headers: Mapping[str, str] | None on HTTPException vs. our Optional[ Dict[str, str]] (the proxy hooks always carry a stringified dict). Both narrowings are intentional and enforced at construction time. Add a type: ignore[misc] with an inline explanation rather than relax the annotations on the parent or change the wire-format guarantees. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): add direct hook-invocation tests to lift patch coverage Adds six end-to-end tests that drive each refactored hook past its limit and assert the unified ProxyRateLimitError is raised with the correct category and dual-base shape. Complements the import-shape-only parametrized guard above by actually executing the new 'raise ProxyRateLimitError(...)' lines so codecov's patch coverage sees them as hit. Hooks covered (one test each): * parallel_request_limiter v1 — direct call to raise_rate_limit_error() * parallel_request_limiter v3 — direct call to _handle_rate_limit_error with a fabricated OVER_LIMIT response * max_iterations_limiter — full async_pre_call_hook with mocked agent registry, second call exceeds budget=1 * max_budget_limiter — async_pre_call_hook with mocked get_current_spend * dynamic_rate_limiter v1 — async_pre_call_hook with mocked check_available_usage forcing available_tpm == 0 * batch_rate_limiter — direct _raise_rate_limit_error call, asserts category is the batch-specific LITELLM_BATCH_RATE_LIMIT (not the generic LITELLM_RATE_LIMIT) LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix: guard rate_limit_category extraction with isinstance check * test(rate-limit): cover remaining hook raise sites for codecov Adds five more direct hook-invocation tests so every PR-touched line in the proxy hooks is exercised by tests in tests/test_litellm/, which codecov measures: * parallel_request_limiter v1 — check_key_in_limits inline raise (the second raise site, separate from the raise_rate_limit_error helper covered earlier) * dynamic_rate_limiter v1 — RPM raise branch (TPM branch was already covered) * dynamic_rate_limiter v3 — parametrized over all three raise sites: model_saturation_check, priority_model, and the fail-closed fallback for an unrecognized descriptor_key * max_budget_per_session_limiter — full async_pre_call_hook with a mocked agent registry and over-budget cached spend All 42 tests in test_rate_limit_error_unification.py now pass and together exercise every changed import + raise line across the eight refactored proxy hooks. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix: use computed error_message in ProxyRateLimitError detail * fix(parallel-request-limiter): drop None from detail; annotate raise_rate_limit_error as NoReturn The v1 ' raise_rate_limit_error' helper built an unused 'error_message' variable and then assembled the actual ' detail' via an f-string that interpolated 'additional_details' verbatim — producing 'Max parallel request limit reached None' when invoked without arguments (flagged by code review). Fix the helper to: - use the constructed 'error_message' as the detail - annotate the helper as NoReturn since it always raises - drop the redundant 'raise'/'return' at the two call sites Add two regression tests covering both the with- and without- additional_details paths. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): drop literal 'None' from raise_rate_limit_error detail The v1 parallel_request_limiter's raise_rate_limit_error helper has a long-standing bug: it computes a None-guarded 'error_message' string but then ignores it and emits an f-string that interpolates the raw 'additional_details' arg. Callers that pass no argument get 'Max parallel request limit reached None' as the user-facing detail. This commit: * wires error_message into the detail kwarg so the None-guard actually applies and operators see a clean message; * changes the return-type annotation from ProxyRateLimitError to NoReturn (the function always raises) so type-checkers know callers after this invocation are unreachable. Greptile P1 + P2 review feedback on PR #27687. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(types): demote TypedDict floating string to a # comment A string literal placed after a field declaration in a TypedDict body is not a per-field docstring — it's an orphaned string expression Python discards. Tools like mypy / pyright that inspect TypedDict fields won't surface that text either. Move the documentation for error_rate_limit_category to a real comment so the intent is visible to readers and type-checker tooling without the misleading docstring framing. Greptile P2 review feedback on PR #27687. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * security(exceptions): do not auto-copy vendor response headers to e.headers A vendor 429 response can set arbitrary headers (Set-Cookie, CORS overrides, …). Previously, when RateLimitError was constructed with only a 'response=' (no explicit 'headers=' kwarg), self.headers fell back to a copy of response.headers. If a downstream proxy serializer ever forwarded e.headers to the client, a malicious upstream could inject browser-interpreted headers for the proxy origin. Drop the fallback. Only headers passed explicitly via the headers= kwarg make it onto self.headers (proxy hooks pass retry-after etc. — they control what's surfaced). Vendor response headers stay reachable on e.response.headers for callers that explicitly want them. Today's proxy_server.py route handlers don't actually forward e.headers on the wire (they construct ProxyException without passing headers), so no current behavior changes — this is a defensive narrowing so the fallback can never be turned into a vector when someone wires e.headers through later. Veria-AI security review feedback on PR #27687. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): regression guards for review-pass fixes Pins down the three review-pass fixes: * test_parallel_request_limiter_v1_helper_no_additional_details — calls raise_rate_limit_error() with no args and asserts the detail does NOT contain the literal string 'None'. Pre-fix, callers got 'Max parallel request limit reached None'. * test_rate_limit_error_does_not_auto_copy_response_headers — passes a vendor httpx.Response with a Set-Cookie header to RateLimitError WITHOUT an explicit headers= kwarg, asserts self.headers stays None (no leak), then re-checks that an explicit headers= kwarg DOES populate self.headers. Vendor headers remain reachable on e.response.headers for callers that explicitly want them. * The existing v1-helper test now also asserts the additional_details string makes it through to the detail. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(rate-limit): add orthogonal RateLimitType (requests/tokens/concurrent_requests/budget/max_iterations) trho's last ask in the LIT-2968 thread: distinguish rate-limit failures by the dimension that was exceeded, not just by who rate-limited (vendor vs. litellm). Adds: - RateLimitType str-enum exposed at `litellm.RateLimitType` with values requests / tokens / concurrent_requests / budget / max_iterations. - `rate_limit_type` kwarg on litellm.RateLimitError + ProxyRateLimitError; None default so existing callers (vendor-429 path in exception_mapping_utils) remain a no-op. - StandardLoggingPayloadErrorInformation.error_rate_limit_type so custom callbacks can split rate-limit failures by cause without parsing free-text error messages. Mirror to error_rate_limit_category extraction in get_error_information(); single isinstance(RateLimitError) check covers both. - map_v3_rate_limit_type() helper to collapse the v3 limiter's internal labels ("requests", "tokens", "max_parallel_requests") onto the public enum so the v3 limiter and dynamic_rate_limiter_v3 share one mapping. Defensive None on unknown values rather than silently picking a wrong dimension. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(proxy/hooks): wire rate_limit_type onto every limiter raise site Each refactored proxy hook now populates rate_limit_type with the dimension that actually tripped the limit, so downstream consumers (custom callbacks, prometheus exporters via the StandardLoggingPayload) can split key/team/user rate-limit failures by cause: - parallel_request_limiter (v1): detect dimension from current vs. limit in the post-cache branch (concurrent_requests > tokens > requests, matches the boolean condition order). Base case (current is None, one limit set to 0) picks the most-specific zero. raise_rate_limit_error() helper accepts an explicit rate_limit_type kwarg with CONCURRENT_REQUESTS default (matches every existing internal call site, including the global-limit branch). - parallel_request_limiter (v3): forward status["rate_limit_type"] through map_v3_rate_limit_type() so "max_parallel_requests" → CONCURRENT_REQUESTS for the public field while the raw v3 jargon stays on the HTTP header for wire-format backward compat. - dynamic_rate_limiter (v1): TPM-zero → TOKENS, RPM-zero → REQUESTS. Pass data["model"] through so callbacks see the model that hit the limit (addresses the secondary "provider missing" complaint in the original Slack thread, partially — the model is what dashboards typically split on). - dynamic_rate_limiter (v3): forward status["rate_limit_type"] via map_v3_rate_limit_type() at every raise site (model_saturation_check, priority_model, fail-closed unknown-descriptor guard). Also pass model. - batch_rate_limiter: limit_type is hard-typed "requests"|"tokens" — map directly without going through the helper's None branch. - max_budget_limiter, max_budget_per_session_limiter: BUDGET. - max_iterations_limiter: MAX_ITERATIONS. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): cover RateLimitType enum, hook wiring, and StandardLoggingPayload propagation 27 new tests across five new test classes: - TestRateLimitType: enum exposed at litellm.RateLimitType, all five values defined, RateLimitError default is None (vendor 429 path makes no claim about which dimension), accepts both string and enum forms with str-coercion guarantee for downstream JSON serializers. - TestProxyRateLimitErrorType: ProxyRateLimitError default is None, accepts string or enum, doesn't break existing callers that pass nothing. - TestMapV3RateLimitType: pins each v3-internal → public-enum mapping (tokens, requests, max_parallel_requests → concurrent_requests, unknown → None) so a future v3 refactor can't silently swap dimensions. - TestStandardLoggingPayloadCarriesType: the new error_rate_limit_type field reaches the structured payload for both ProxyRateLimitError and plain RateLimitError, is None when unspecified, and is None for non-rate-limit exceptions (symmetric with error_rate_limit_category). - TestProxyHooksWireTypeCorrectly: drives the actual raise sites in the v1 parallel_request_limiter helper, the v3 _handle_rate_limit_error (both "tokens" and "max_parallel_requests" paths), and the batch limiter (both tokens and requests paths) — coverage tools see the new rate_limit_type= kwargs as exercised, not just the import shape. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): cover _coerce_message branches and v1 dimension detection Drives the patch coverage on the new orthogonal RateLimitType wiring up to (or close to) 100% on the touched files. ProxyRateLimitError._coerce_message — was 22% covered, now 100%: * nested {error: {message}} dict * nested {message: {message}} dict (alt key) * dict without 'error'/'message' keys → JSON dump fallback * non-JSON-serializable dict value → str() fallback * non-string non-mapping detail (int) → str() coercion v1 parallel_request_limiter dimension detection — was 0% covered, now exercised across 6 parametrized cases: * check_key_in_limits else-branch: current at concurrent / TPM / RPM cap → asserts rate_limit_type is concurrent_requests / tokens / requests. * check_key_in_limits base case (current is None): max_parallel_requests / tpm_limit / rpm_limit set to 0 → asserts the most-specific zero attribution wins per the helper's order. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(proxy/hooks): add ProxyHTTPRateLimitError + provider resolver Introduces a small helper layer used by every proxy-side rate-limit hook so that the 429 they raise carries a populated llm_provider / model — instead of an empty exception.llm_provider that downstream loggers (Prometheus failure metric, observability callbacks) read as 'no provider attribution'. ProxyHTTPRateLimitError inherits from both fastapi.HTTPException (so the proxy server still renders it as a 429) and litellm.exceptions.RateLimitError (so isinstance checks and PrometheusLogger._get_exception_class_name pick up llm_provider). We deliberately don't call RateLimitError.__init__ — it constructs an httpx.Response we don't need and would just add failure surface; attribute parity is what downstream consumers care about. resolve_llm_provider_for_rate_limit() wraps litellm.get_llm_provider defensively. Internal limiter hooks fire from async_pre_call_hook — well before get_llm_provider runs anywhere else in the request lifecycle — so we have to call it ourselves at raise time. If the model is missing or unparseable (alias, router-only model) we fall back to llm_provider='litellm_proxy' rather than letting a second exception leak out and break the request path. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on parallel-request 429s Both v1 and v3 parallel-request limiters fired bare HTTPException(429) from inside async_pre_call_hook. The downstream Prometheus failure metric reads exception.llm_provider via _get_exception_class_name — the empty value showed up as exception_class='HTTPException' and left model_id='None' on the time series. Threads requested_model through every raise site in: * parallel_request_limiter.py: - check_key_in_limits (the per-key/per-model/per-user/per-team/ per-customer over-limit path) - raise_rate_limit_error (zero-limit + global_max_parallel_requests paths) — now takes an optional requested_model kwarg * parallel_request_limiter_v3.py: - _handle_rate_limit_error (the OVER_LIMIT translator), called from both the should_rate_limit pre-check and the TPM reservation path Resolved via resolve_llm_provider_for_rate_limit so unknown / missing models silently fall back to llm_provider='litellm_proxy' instead of breaking the request path with a second exception. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on dynamic-rate-limit 429s Same plumbing change as the parallel limiters, applied to both dynamic_rate_limiter (v1) and dynamic_rate_limiter_v3: * v1: TPM-zero and RPM-zero paths in async_pre_call_hook now resolve data['model'] -> (model, llm_provider) once and pass it into both raises. * v3: All three raise sites in _check_rate_limits — the model_saturation_check enforced raise, the priority_model enforced raise, and the fail-closed unknown-descriptor branch — now attribute the 429 to the actual provider. Falls back to llm_provider='litellm_proxy' when the model can't be resolved. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on batch-rate-limit 429s batch_rate_limiter._raise_rate_limit_error now takes a requested_model kwarg threaded from data['model'] in _check_and_increment_batch_counters. The batch-creation 429 is what gets raised when the input file's tokens/requests count would push the per-key TPM/RPM window over its limit. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on budget/iterations 429s Final batch of internal raise sites — the user/session-budget and max-iterations hooks. Same pattern: resolve data['model'] once at raise time, attach to ProxyHTTPRateLimitError so Prometheus and observability callbacks can attribute the 429. Hooks updated: * max_budget_limiter (per-user max_budget exceeded) * max_iterations_limiter (per-session agent iteration cap) * max_budget_per_session_limiter (per-session dollar cap) All three fall back to llm_provider='litellm_proxy' when data['model'] is missing or unparseable. Drops the now-unused HTTPException import from each module. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(proxy/hooks): pin provider field on internal rate-limit 429s Regression coverage for the 'provider field missing' bug across every proxy-side rate-limit hook + the helper layer: * ProxyHTTPRateLimitError class shape (HTTPException + RateLimitError, dict-detail stringification, None-provider normalization). * resolve_llm_provider_for_rate_limit happy paths (gpt-4o-mini, anthropic/..., bedrock/...) plus all three fallback branches (None, '', unknown name) plus a 'get_llm_provider raises' case that asserts we swallow the secondary exception. * For each limiter (parallel v1/v3, dynamic v1/v3, batch, max_budget, max_iterations, max_budget_per_session): assert the raised exception is a RateLimitError carrying the resolved model + llm_provider, and a sibling test that asserts the fallback path returns 'litellm_proxy' without leaking a second exception. * Two PrometheusLogger._get_exception_class_name pins so the Prometheus failure metric label flips from 'HTTPException' to 'Openai.ProxyHTTPRateLimitError' (or 'Litellm_proxy.*' on fallback) — that's what dashboards consume. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * perf(proxy/hooks): defer provider resolution to over-limit branches * fix: use error_message in raise_rate_limit_error to avoid literal 'None' in detail * Consolidate rate_limiter_utils imports in dynamic_rate_limiter * fix(proxy): set num_retries/max_retries on ProxyHTTPRateLimitError ProxyHTTPRateLimitError inherits from RateLimitError but did not call RateLimitError.__init__, so num_retries/max_retries were never set. When Starlette's HTTPException lacks __str__, MRO falls through to RateLimitError.__str__, which unconditionally reads these attributes and raises AttributeError during logging/traceback formatting. Initialize them to None defensively. * fix(mypy): silence base-class status_code conflict on ProxyHTTPRateLimitError HTTPException declares 'status_code: int' while openai.RateLimitError (via APIStatusError) declares 'status_code: Literal[429] = 429'. Mypy flags the multi-base override as [misc] in CI lint. The runtime semantics are fine (we set self.status_code in __init__), so silence the class-level annotation conflict with a targeted ignore. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix: annotate batch limiter _raise_rate_limit_error as NoReturn * feat(prometheus): rate-limit category/type labels + exception_class back-compat (follow-up to #27687) (#27706) * feat(prometheus): add rate_limit_category and rate_limit_type labels Adds two new labels to litellm_proxy_failed_requests_metric so dashboards can split 429s by rate-limit source (vendor vs. litellm-internal) and by the dimension that was exceeded (requests/tokens/concurrent_requests/ budget/max_iterations) without parsing free-text error messages. Closes the Prometheus side of LIT-2718. The unified RateLimitError.category and .rate_limit_type fields landed in PR #27687 but were only surfaced on StandardLoggingPayload (custom-callback channel); this exposes them on the metric label set as well. Both labels are populated only when the underlying exception is a litellm.RateLimitError; non-rate-limit failures keep them empty. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(prometheus): populate rate-limit labels + preserve exception_class back-compat Two coupled changes in the Prometheus integration: 1. async_post_call_failure_hook now extracts the new RateLimitError .category / .rate_limit_type fields (added in PR #27687) via a _extract_rate_limit_labels helper and forwards them through UserAPIKeyLabelValues onto litellm_proxy_failed_requests_metric. Empty for non-rate-limit failures. 2. _get_exception_class_name special-cases ProxyRateLimitError and keeps emitting 'HTTPException' for the exception_class label. Without this shim, ProxyRateLimitError (which multi-inherits from HTTPException + RateLimitError) would silently flip the label from 'HTTPException' (the historical value for proxy-side 429s) to 'ProxyRateLimitError', breaking existing dashboards / alerts that key off exception_class='HTTPException'. Distinguishing vendor vs. litellm 429s is now the job of the new rate_limit_category label. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(prometheus): cover rate-limit labels and exception_class back-compat Adds 19 tests across: - enum / label-list registration - _extract_rate_limit_labels for vendor RateLimitError, ProxyRateLimitError, non-rate-limit and None inputs (incl. parametrized over every RateLimitErrorCategory x RateLimitType combo) - _get_exception_class_name back-compat: ProxyRateLimitError keeps the legacy 'HTTPException' string while vendor RateLimitError keeps the historical 'Provider.ClassName' format - end-to-end through async_post_call_failure_hook with both ProxyRateLimitError and vendor RateLimitError, asserting both new labels populate and exception_class stays back-compat Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(prometheus): tolerate missing fastapi in lazy ProxyRateLimitError import Address greptile feedback: - async_post_call_failure_hook docstring: drop the stale labelnames listing and reference PrometheusMetricLabels.litellm_proxy_failed_requests_metric as the source of truth so the doc cannot drift from the actual labelset. - _get_exception_class_name: guard the lazy ProxyRateLimitError import with ImportError so router-side fallback callsites don't blow up in non-proxy installs that don't have fastapi (a transitive dep of proxy.common_utils.proxy_rate_limit_error). Behavior is unchanged when fastapi is available. Also fix the existing enterprise callback test that asserted the old labelset on litellm_proxy_failed_requests_metric — it now expects the new rate_limit_category / rate_limit_type labels populated for vendor 429s. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(bugbot): simplify rate-limit label coercion + guard None detail - prometheus.py _extract_rate_limit_labels: RateLimitError.__init__ already normalizes category/rate_limit_type to plain str, so the getattr(.value) + isinstance dance was dead code. Reduce to str(value) if not None. - proxy_rate_limit_error.py _coerce_message: short-circuit None to '' instead of falling through to str(None) = 'None', which produced the literal message 'litellm.RateLimitError: None'. * fix(rate-limit): surface unified category/type fields on BudgetExceededError The most common budget cap (virtual-key max_budget enforcement in auth_checks.py) raises litellm.BudgetExceededError, a bare Exception subclass that bypassed the unified rate-limit error class introduced by PR #27687. Custom callbacks reading StandardLoggingPayload.error_information saw category=None and rate_limit_type=None for these 429s, missing the most common budget case (team / org / end-user budgets all hit the same code path). Surface the fields off BudgetExceededError as plain attributes: - category = RateLimitErrorCategory.LITELLM_RATE_LIMIT - rate_limit_type = RateLimitType.BUDGET - llm_provider = "" (or caller-supplied) Switch get_error_information and _extract_rate_limit_labels from isinstance(RateLimitError) gating to duck-typed attribute reads, guarded by membership in the rate-limit enums so unrelated third-party exceptions exposing a .category attribute can't leak garbage values into the payload. This is strictly additive: BudgetExceededError keeps its bare-Exception base class, so `except BudgetExceededError:` handlers keep firing and `except RateLimitError:` does not start catching budget errors. * fix(rate-limit): validate enum membership at duck-typed read sites + enrich BudgetExceededError llm_provider Two follow-ups uncovered during the second QA pass on PR #27687: 1. Guard third-party `.category` / `.rate_limit_type` attribute leakage. The duck-typed read in `get_error_information` and `_extract_rate_limit_labels` would forward any string attribute named `category` / `rate_limit_type` on an unrelated third-party exception into the StandardLoggingPayload and Prometheus labels — silently mislabeling custom-callback payloads and blowing out Prometheus label cardinality. Add `validate_rate_limit_category` / `validate_rate_limit_type` helpers that gate on the documented enum value sets; non-matching values are dropped to None. 2. Enrich BudgetExceededError.llm_provider from request_data. Budget checks live in tenant-scoped helpers (key / team / org / tag / end-user / project) that don't see the request model, so the BudgetExceededError they raise carried llm_provider="" — leaving custom-metrics consumers without provider attribution for the most common 429 case. Resolve it once at the central UserAPIKeyAuthExceptionHandler seam, before post_call_failure_hook fires, so the StandardLoggingPayload the callback sees has the same provider attribution as RPM/TPM 429s. Regression tests pin both: 4 leakage tests + 4 enrichment tests. The leakage tests would fail under the pre-validation version of either read site; the enrichment tests would fail if the handler skipped the resolver call. * fix(rate-limit): resolve router model_name aliases to real provider (#27914) * fix(rate-limit): resolve router model_name aliases to real provider For nearly every real LiteLLM proxy deployment the request model is a router model_name alias (e.g. 'tpm-locked' -> litellm_params.model: openai/gpt-4o-mini), and 'litellm.get_llm_provider' doesn't know about router aliases — it raises 'LLMProviderNotProvidedError'. The resolver then fell through to the defensive 'litellm_proxy' fallback, so the 'llm_provider' field this PR adds was effectively always 'litellm_proxy' in the field, defeating its purpose for the most common proxy configuration. Add a router-alias fallback step: when 'get_llm_provider' raises, scan the active 'llm_router.model_list' for a deployment whose 'model_name' matches the request model and resolve from its 'litellm_params.model' instead. If multiple deployments share the same alias (load-balancing case) the first one wins — every deployment under one alias should agree on provider in any sensible config, and 'first' is deterministic so the Prometheus label stays stable. Defensive throughout: an uninitialized router, a malformed deployment, a 'litellm_params.model' that itself fails 'get_llm_provider' — every branch falls through to the existing 'litellm_proxy' fallback rather than letting a secondary exception escape and mask the rate-limit error we're trying to surface. Tests: - test_router_alias_resolves_to_underlying_provider: alias 'tpm-locked' -> 'openai/gpt-4o-mini' produces provider='openai', model='gpt-4o-mini'. - test_router_alias_with_multiple_deployments_uses_first. - test_router_alias_unknown_falls_back. - test_router_alias_with_malformed_deployment_falls_back. - Existing fallback test updated to also stub 'litellm.proxy.proxy_server.llm_router' so it exercises the full 'no resolution anywhere' path. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(rate-limit): harden router alias resolver + test isolation - Wrap _resolve_provider_from_router_alias loop in top-level try/except so a non-iterable model_list / unexpected deployment shape can't escape and mask the 429 with a 500. - Type-check litellm_params before .get() to handle non-dict truthy values. - Patch llm_router=None in the parametrized fallback test so a router left by another test in the session can't redirect the unknown-model path. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(bugbot): preserve "BudgetExceededError" Prometheus label Adding llm_provider to BudgetExceededError (so callbacks get provider attribution from StandardLoggingPayload) made the provider-prefix step in _get_exception_class_name silently flip the label from "BudgetExceededError" to e.g. "Openai.BudgetExceededError", breaking dashboards keyed on the historical value. Short-circuit BudgetExceededError in _get_exception_class_name the same way ProxyRateLimitError already is. Provider/category attribution still lands on the new rate_limit_category / rate_limit_type labels. * test: fix invalid 'rpm' rate_limit_type in v3 limiter test mocks The v3 rate limiter only emits 'requests', 'tokens', or 'max_parallel_requests'. Using 'rpm' caused map_v3_rate_limit_type to return None, leaving the expected RateLimitType.REQUESTS untested. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(bugbot): hoist provider resolver + opt-in prom rate-limit labels - dynamic_rate_limiter.py: hoist resolve_llm_provider_for_rate_limit above the TPM/RPM if/elif so the lookup runs once per request, matching the pattern in dynamic_rate_limiter_v3.py. - prometheus.py: gate the new rate_limit_category / rate_limit_type labels on litellm_proxy_failed_requests_metric behind litellm.prometheus_emit_rate_limit_labels (default False). Mirrors the existing prometheus_emit_stream_label opt-in. Preserves the metric's pre-unification label set so existing dashboards / recording rules keep matching after upgrade; operators can enable the new labels once downstream consumers include them. - Tests updated: default-off back-compat case, opt-in path enables the flag before asserting label presence. * fix: stabilize prometheus label sets and drop redundant model normalization - Cache PrometheusLogger.get_labels_for_metric per metric_name so that the label set used to construct counters at __init__ time stays in sync with the label set used at increment time, even if module-level toggles like prometheus_emit_rate_limit_labels or prometheus_emit_stream_label are flipped at runtime. Without this, toggling these flags after the logger was created would cause ValueError from prometheus_client because the runtime labels would not match the counter's declared labelnames. - Drop redundant 'model or ""' guard in ProxyRateLimitError.__init__ where model is already normalized one step earlier. Co-authored-by: Yassin Kortam <yassin@berri.ai> * perf(dynamic_rate_limiter): only resolve provider when rate limit hit Co-authored-by: Yassin Kortam <yassin@berri.ai> * test(prometheus): clear cached metric labels after toggling rate-limit flag The PrometheusLogger caches each metric's label set at construction time so that labels used at counter.labels(...) time stay consistent with the labels the metric was registered with. The enterprise async_post_call_failure_hook test toggles litellm.prometheus_emit_rate_limit_labels = True AFTER the fixture has already built the logger, so without invalidating the cache the rate_limit_category / rate_limit_type labels never reach the mocked counter and the assert_called_once_with check fails. Co-authored-by: Yassin Kortam <yassin@berri.ai> * test: fix CI failures from prom label cache + flaky time-window assertion PrometheusLogger.get_labels_for_metric now caches the per-metric label set at first read so the labels passed to counter.labels(...) stay in lock step with the labels the counter was registered with. This broke two existing test patterns: - test_prometheus_labels.py: tests bind the real method onto a MagicMock, but MagicMock auto-creates a Mock for _cached_metric_labels whose .get(...) returns a truthy Mock — treated as a populated cache and returned as the label set, producing empty filtered labels and KeyError on labels["requested_model"] / ["route"]. Seed real {} containers for _cached_metric_labels and label_filters before binding. - test_prometheus_logging_callbacks.py::test_set_team_budget_metrics_with_custom_labels: the fixture builds the logger before the test monkeypatches litellm.custom_prometheus_metadata_labels, so the cached label set never picks up the new metadata labels. Clear the cache after the monkeypatch (same pattern already used for the rate-limit toggle in test_async_post_call_failure_hook). UI: view_logs/index.test.tsx "Last Minute" window assertion is off by one at the minute boundary. start_date is floored to the minute, so the dropped sub-minute fraction can push the truncated-seconds diff up to (minMinutes+1)*60 exactly when the click lands near a minute rollover. Switch the upper bound to toBeLessThanOrEqual. * feat(otel-v2): surface rate_limit_category + rate_limit_type on failed LLM-call spans PR #28909 introduced the typed v2 OTel engine that builds spans from StandardLoggingPayload, with SpanError carrying error_type + message and the genai mapper stamping error.type onto every failed LLM-call span. This PR's earlier commits added error_rate_limit_category and error_rate_limit_type to the same StandardLoggingPayload.error_information the v2 engine reads — but neither field reached a span attribute, so v2 OTel traces stayed opaque about *why* a 429 fired (vendor vs litellm, RPM vs TPM vs concurrent vs budget vs max_iterations) even after the custom-callback and prometheus surfaces gained that decomposition. Three coupled changes: 1. semconv.py: add LiteLLM.ERROR_RATE_LIMIT_CATEGORY / LiteLLM.ERROR_RATE_LIMIT_TYPE under the litellm.* vendor namespace (no GenAI semconv equivalent exists for who-rate-limited / which-dimension). 2. payloads.py: extend SpanError with rate_limit_category + rate_limit_type, populated by _parse_error() from the same error_information.error_rate_limit_* fields the custom-callback channel and prometheus rate_limit_category / rate_limit_type labels read. Single source of truth across all three observability surfaces. 3. mappers/genai.py: stamp the two attributes on the LLM-call span when present. drop_none guarantees they stay absent (not 'None') for non-rate-limit failures so trace consumers can read them unconditionally. Three regression tests in test_otel_v2_emitter.py pin: a vendor / litellm-internal RateLimitError lands category=litellm_rate_limit + rate_limit_type=requests on the span; a BudgetExceededError lands rate_limit_type=budget; a non-rate-limit failure (BadRequestError) keeps the rate_limit_* attributes absent. Mutation-tested against reverting either the SpanError extension or the _parse_error read site — both new tests fail under either mutation. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test: align prometheus user-budget + logs quick-select tests with merged code The merge into this branch left two test patterns out of step with the code they exercise. test_set_user_budget_metrics_includes_user_email_and_alias_labels_when_opted_in flipped litellm.prometheus_user_budget_label_include_email_alias after the fixture had already built the PrometheusLogger. get_labels_for_metric now snapshots each metric's label set at construction time, so the runtime flip no longer reached the cached labels. Enable the flag before constructing the logger, matching how the proxy applies config at startup. view_logs/index.test.tsx referenced uiSpendLogsCall and moment without importing them, and the merged index.tsx now fetches through useLogFilterLogic (the hook the file stubs out) rather than calling uiSpendLogsCall directly. Add the imports and restore the real hook for the Quick Select window assertions so the call is actually observed. * refactor(otel/v2): drop rate-limit decomposition from the LLM-call span Proxy-side rate limits (litellm_rate_limit, budget, max_iterations) are rejected at the gate before any upstream call, so async_post_call_failure_hook tags the synthetic failure log with LITELLM_LOGGING_NO_UPSTREAM_LLM_CALL and the v2 OTel logger never opens an LLM-call span for them; the litellm.error.rate_limit_category / litellm.error.rate_limit_type attributes were dead for exactly the cases they were meant to surface. The only failure that does open an LLM-call span carrying a RateLimitError is a vendor 429, where rate_limit_type is always None and the category just restates error.type=RateLimitError. The decomposition still reaches downstream consumers through StandardLoggingPayload.error_information.error_rate_limit_* and the prometheus rate_limit_category / rate_limit_type labels, both unchanged. Removes the SpanError fields, the _parse_error reads, the genai mapper attributes, the semconv keys, and the three span tests that asserted a scenario that never reaches the mapper in production. * fix(batch_rate_limiter): map max_parallel_requests to concurrent_requests * refactor(prometheus): drop transitive fastapi import from _get_exception_class_name Read the legacy exception_class label from a prometheus_exception_class_name marker on ProxyRateLimitError instead of importing the proxy module, keeping the integrations layer free of a transitive fastapi dependency. * chore(ui): sync schema.d.ts with unified rate-limit error spec The ProxyRateLimitError docstring flows into the proxy OpenAPI spec's 429 response description, so the generated dashboard types were out of sync. Regenerated via npm run gen:api (Check UI API Types Sync). --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> Co-authored-by: Yassin Kortam <yassin@berri.ai>

…odel, and llm_provider fields (BerriAI#27687) * feat(exceptions): add RateLimitErrorCategory + headers/detail fields on RateLimitError LiteLLM previously surfaced rate-limit conditions through several unrelated error classes (RateLimitError, FastAPI HTTPException(429), BaseLLMException). This commit adds the data model needed to consolidate them under a single class: * RateLimitErrorCategory enum exposing four categorical values (vendor_rate_limit, vendor_batch_rate_limit, litellm_rate_limit, litellm_batch_rate_limit) so callers can switch on the rate-limit source. * New optional fields on RateLimitError: - category (defaults to vendor_rate_limit, preserving today's behavior for every existing call site in exception_mapping_utils); - headers (preserves retry-after / rate_limit_type / reset_at across the proxy boundary instead of dropping them on the floor); - detail (mirrors FastAPI HTTPException.detail so the same instance can be serialized through both paths). litellm.RateLimitErrorCategory is re-exported at the package root to match the existing exception-export pattern. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(proxy): add ProxyRateLimitError unifying RateLimitError + HTTPException Adds a single proxy-side error class that subclasses BOTH litellm.exceptions.RateLimitError AND fastapi.HTTPException via cooperative multiple inheritance. Why both bases: * Subclassing RateLimitError lets user code catch every rate-limit source with one 'except RateLimitError' and switch on the new .category field. * Subclassing HTTPException keeps every existing FastAPI plumbing path (the isinstance(e, HTTPException) branches in proxy_server.py route handlers, FastAPI's own dispatcher, and tests asserting pytest.raises(HTTPException)) working without modification, and preserves retry-after / rate_limit_type / reset_at headers on the wire. The class declaration order is (HTTPException, RateLimitError) so the MRO puts HTTPException's no-super-call __init__ ahead of openai's cooperative __init__ chain — preventing openai.APIError.super().__init__(message) from landing in HTTPException.__init__(status_code=message). LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from budget + iteration limiters Replaces three bare HTTPException(status_code=429, ...) call sites with ProxyRateLimitError, which is both a RateLimitError (catchable by category) and an HTTPException (preserves existing FastAPI serialization). Drops the now-unused HTTPException import in the iteration / per-session limiters. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from parallel-request limiters Replaces HTTPException(status_code=429, ...) call sites in the v1 and v3 parallel-request limiters (key/team/user/model/customer rate limits) with ProxyRateLimitError. Updates the raise_rate_limit_error helper's return type annotation accordingly. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from dynamic rate limiters Replaces HTTPException(status_code=429, ...) call sites in the v1 and v3 dynamic rate limiters (project-level TPM/RPM allocation, model-saturation checks, priority-based limits, fail-closed guards) with ProxyRateLimitError. The v3 limiter still imports HTTPException for an unrelated bare 'except HTTPException:' branch. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from batch rate limiter Replaces HTTPException(status_code=429, ...) in batch_rate_limiter._raise_rate_limit_error with ProxyRateLimitError tagged as RateLimitErrorCategory.LITELLM_BATCH_RATE_LIMIT so users can distinguish batch-level throttling (which counts requests/tokens across an uploaded batch input file before submission) from the generic key/team/user RPM/TPM limiter. The HTTPException import is retained because the same module raises HTTPException for unrelated 403/IO error paths. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): pin down unified rate-limit error contract Adds a dedicated test module covering the new RateLimitErrorCategory enum, RateLimitError.category default + override behavior, ProxyRateLimitError's dual nature (RateLimitError + HTTPException), and a parametrized regression guard that asserts every proxy hook module imports the unified class. The regression guard catches the failure mode the refactor is designed to prevent: someone re-introducing a bare HTTPException(status_code=429, ...) in one of the hook modules instead of going through ProxyRateLimitError. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(logging): expose rate-limit category via StandardLoggingPayload Adds an optional 'error_rate_limit_category' field to StandardLoggingPayloadErrorInformation, populated from the unified RateLimitError.category attribute (introduced in the previous commits on this branch). Why: the .category attribute is reachable off the raw exception today via getattr(e, 'category', None), but the structured contract that downstream custom callbacks / loggers / spend log writers consume is the StandardLoggingPayload. Without this field, a user building custom rate-limit metrics on top of callback data has to special-case the raw exception object — which defeats the purpose of the StandardLoggingPayload abstraction. The field is None for non-rate-limit exceptions (so consumers can read it unconditionally without isinstance checks) and is one of the RateLimitErrorCategory string values otherwise. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): assert StandardLoggingPayload carries the category Five tests covering: vendor default, explicit litellm_rate_limit and litellm_batch_rate_limit values, None for non-rate-limit exceptions, and None when no exception is provided. Pins down the contract that custom callbacks can read 'error_information.error_rate_limit_category' off the StandardLoggingPayload to drive custom rate-limit metrics without ever reaching for the raw exception. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(types): silence mypy [misc] on intentional dual-base attr overlap mypy emits two [misc] errors on the ProxyRateLimitError class line because its two bases declare overlapping attributes with related-but-not-identical annotations: * status_code: int on starlette HTTPException vs. Literal[429] on openai's RateLimitError (every openai status-error subclass narrows it the same way and silences pyright with the same convention). * headers: Mapping[str, str] | None on HTTPException vs. our Optional[ Dict[str, str]] (the proxy hooks always carry a stringified dict). Both narrowings are intentional and enforced at construction time. Add a type: ignore[misc] with an inline explanation rather than relax the annotations on the parent or change the wire-format guarantees. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): add direct hook-invocation tests to lift patch coverage Adds six end-to-end tests that drive each refactored hook past its limit and assert the unified ProxyRateLimitError is raised with the correct category and dual-base shape. Complements the import-shape-only parametrized guard above by actually executing the new 'raise ProxyRateLimitError(...)' lines so codecov's patch coverage sees them as hit. Hooks covered (one test each): * parallel_request_limiter v1 — direct call to raise_rate_limit_error() * parallel_request_limiter v3 — direct call to _handle_rate_limit_error with a fabricated OVER_LIMIT response * max_iterations_limiter — full async_pre_call_hook with mocked agent registry, second call exceeds budget=1 * max_budget_limiter — async_pre_call_hook with mocked get_current_spend * dynamic_rate_limiter v1 — async_pre_call_hook with mocked check_available_usage forcing available_tpm == 0 * batch_rate_limiter — direct _raise_rate_limit_error call, asserts category is the batch-specific LITELLM_BATCH_RATE_LIMIT (not the generic LITELLM_RATE_LIMIT) LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix: guard rate_limit_category extraction with isinstance check * test(rate-limit): cover remaining hook raise sites for codecov Adds five more direct hook-invocation tests so every PR-touched line in the proxy hooks is exercised by tests in tests/test_litellm/, which codecov measures: * parallel_request_limiter v1 — check_key_in_limits inline raise (the second raise site, separate from the raise_rate_limit_error helper covered earlier) * dynamic_rate_limiter v1 — RPM raise branch (TPM branch was already covered) * dynamic_rate_limiter v3 — parametrized over all three raise sites: model_saturation_check, priority_model, and the fail-closed fallback for an unrecognized descriptor_key * max_budget_per_session_limiter — full async_pre_call_hook with a mocked agent registry and over-budget cached spend All 42 tests in test_rate_limit_error_unification.py now pass and together exercise every changed import + raise line across the eight refactored proxy hooks. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix: use computed error_message in ProxyRateLimitError detail * fix(parallel-request-limiter): drop None from detail; annotate raise_rate_limit_error as NoReturn The v1 ' raise_rate_limit_error' helper built an unused 'error_message' variable and then assembled the actual ' detail' via an f-string that interpolated 'additional_details' verbatim — producing 'Max parallel request limit reached None' when invoked without arguments (flagged by code review). Fix the helper to: - use the constructed 'error_message' as the detail - annotate the helper as NoReturn since it always raises - drop the redundant 'raise'/'return' at the two call sites Add two regression tests covering both the with- and without- additional_details paths. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): drop literal 'None' from raise_rate_limit_error detail The v1 parallel_request_limiter's raise_rate_limit_error helper has a long-standing bug: it computes a None-guarded 'error_message' string but then ignores it and emits an f-string that interpolates the raw 'additional_details' arg. Callers that pass no argument get 'Max parallel request limit reached None' as the user-facing detail. This commit: * wires error_message into the detail kwarg so the None-guard actually applies and operators see a clean message; * changes the return-type annotation from ProxyRateLimitError to NoReturn (the function always raises) so type-checkers know callers after this invocation are unreachable. Greptile P1 + P2 review feedback on PR BerriAI#27687. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(types): demote TypedDict floating string to a # comment A string literal placed after a field declaration in a TypedDict body is not a per-field docstring — it's an orphaned string expression Python discards. Tools like mypy / pyright that inspect TypedDict fields won't surface that text either. Move the documentation for error_rate_limit_category to a real comment so the intent is visible to readers and type-checker tooling without the misleading docstring framing. Greptile P2 review feedback on PR BerriAI#27687. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * security(exceptions): do not auto-copy vendor response headers to e.headers A vendor 429 response can set arbitrary headers (Set-Cookie, CORS overrides, …). Previously, when RateLimitError was constructed with only a 'response=' (no explicit 'headers=' kwarg), self.headers fell back to a copy of response.headers. If a downstream proxy serializer ever forwarded e.headers to the client, a malicious upstream could inject browser-interpreted headers for the proxy origin. Drop the fallback. Only headers passed explicitly via the headers= kwarg make it onto self.headers (proxy hooks pass retry-after etc. — they control what's surfaced). Vendor response headers stay reachable on e.response.headers for callers that explicitly want them. Today's proxy_server.py route handlers don't actually forward e.headers on the wire (they construct ProxyException without passing headers), so no current behavior changes — this is a defensive narrowing so the fallback can never be turned into a vector when someone wires e.headers through later. Veria-AI security review feedback on PR BerriAI#27687. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): regression guards for review-pass fixes Pins down the three review-pass fixes: * test_parallel_request_limiter_v1_helper_no_additional_details — calls raise_rate_limit_error() with no args and asserts the detail does NOT contain the literal string 'None'. Pre-fix, callers got 'Max parallel request limit reached None'. * test_rate_limit_error_does_not_auto_copy_response_headers — passes a vendor httpx.Response with a Set-Cookie header to RateLimitError WITHOUT an explicit headers= kwarg, asserts self.headers stays None (no leak), then re-checks that an explicit headers= kwarg DOES populate self.headers. Vendor headers remain reachable on e.response.headers for callers that explicitly want them. * The existing v1-helper test now also asserts the additional_details string makes it through to the detail. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(rate-limit): add orthogonal RateLimitType (requests/tokens/concurrent_requests/budget/max_iterations) trho's last ask in the LIT-2968 thread: distinguish rate-limit failures by the dimension that was exceeded, not just by who rate-limited (vendor vs. litellm). Adds: - RateLimitType str-enum exposed at `litellm.RateLimitType` with values requests / tokens / concurrent_requests / budget / max_iterations. - `rate_limit_type` kwarg on litellm.RateLimitError + ProxyRateLimitError; None default so existing callers (vendor-429 path in exception_mapping_utils) remain a no-op. - StandardLoggingPayloadErrorInformation.error_rate_limit_type so custom callbacks can split rate-limit failures by cause without parsing free-text error messages. Mirror to error_rate_limit_category extraction in get_error_information(); single isinstance(RateLimitError) check covers both. - map_v3_rate_limit_type() helper to collapse the v3 limiter's internal labels ("requests", "tokens", "max_parallel_requests") onto the public enum so the v3 limiter and dynamic_rate_limiter_v3 share one mapping. Defensive None on unknown values rather than silently picking a wrong dimension. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(proxy/hooks): wire rate_limit_type onto every limiter raise site Each refactored proxy hook now populates rate_limit_type with the dimension that actually tripped the limit, so downstream consumers (custom callbacks, prometheus exporters via the StandardLoggingPayload) can split key/team/user rate-limit failures by cause: - parallel_request_limiter (v1): detect dimension from current vs. limit in the post-cache branch (concurrent_requests > tokens > requests, matches the boolean condition order). Base case (current is None, one limit set to 0) picks the most-specific zero. raise_rate_limit_error() helper accepts an explicit rate_limit_type kwarg with CONCURRENT_REQUESTS default (matches every existing internal call site, including the global-limit branch). - parallel_request_limiter (v3): forward status["rate_limit_type"] through map_v3_rate_limit_type() so "max_parallel_requests" → CONCURRENT_REQUESTS for the public field while the raw v3 jargon stays on the HTTP header for wire-format backward compat. - dynamic_rate_limiter (v1): TPM-zero → TOKENS, RPM-zero → REQUESTS. Pass data["model"] through so callbacks see the model that hit the limit (addresses the secondary "provider missing" complaint in the original Slack thread, partially — the model is what dashboards typically split on). - dynamic_rate_limiter (v3): forward status["rate_limit_type"] via map_v3_rate_limit_type() at every raise site (model_saturation_check, priority_model, fail-closed unknown-descriptor guard). Also pass model. - batch_rate_limiter: limit_type is hard-typed "requests"|"tokens" — map directly without going through the helper's None branch. - max_budget_limiter, max_budget_per_session_limiter: BUDGET. - max_iterations_limiter: MAX_ITERATIONS. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): cover RateLimitType enum, hook wiring, and StandardLoggingPayload propagation 27 new tests across five new test classes: - TestRateLimitType: enum exposed at litellm.RateLimitType, all five values defined, RateLimitError default is None (vendor 429 path makes no claim about which dimension), accepts both string and enum forms with str-coercion guarantee for downstream JSON serializers. - TestProxyRateLimitErrorType: ProxyRateLimitError default is None, accepts string or enum, doesn't break existing callers that pass nothing. - TestMapV3RateLimitType: pins each v3-internal → public-enum mapping (tokens, requests, max_parallel_requests → concurrent_requests, unknown → None) so a future v3 refactor can't silently swap dimensions. - TestStandardLoggingPayloadCarriesType: the new error_rate_limit_type field reaches the structured payload for both ProxyRateLimitError and plain RateLimitError, is None when unspecified, and is None for non-rate-limit exceptions (symmetric with error_rate_limit_category). - TestProxyHooksWireTypeCorrectly: drives the actual raise sites in the v1 parallel_request_limiter helper, the v3 _handle_rate_limit_error (both "tokens" and "max_parallel_requests" paths), and the batch limiter (both tokens and requests paths) — coverage tools see the new rate_limit_type= kwargs as exercised, not just the import shape. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): cover _coerce_message branches and v1 dimension detection Drives the patch coverage on the new orthogonal RateLimitType wiring up to (or close to) 100% on the touched files. ProxyRateLimitError._coerce_message — was 22% covered, now 100%: * nested {error: {message}} dict * nested {message: {message}} dict (alt key) * dict without 'error'/'message' keys → JSON dump fallback * non-JSON-serializable dict value → str() fallback * non-string non-mapping detail (int) → str() coercion v1 parallel_request_limiter dimension detection — was 0% covered, now exercised across 6 parametrized cases: * check_key_in_limits else-branch: current at concurrent / TPM / RPM cap → asserts rate_limit_type is concurrent_requests / tokens / requests. * check_key_in_limits base case (current is None): max_parallel_requests / tpm_limit / rpm_limit set to 0 → asserts the most-specific zero attribution wins per the helper's order. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(proxy/hooks): add ProxyHTTPRateLimitError + provider resolver Introduces a small helper layer used by every proxy-side rate-limit hook so that the 429 they raise carries a populated llm_provider / model — instead of an empty exception.llm_provider that downstream loggers (Prometheus failure metric, observability callbacks) read as 'no provider attribution'. ProxyHTTPRateLimitError inherits from both fastapi.HTTPException (so the proxy server still renders it as a 429) and litellm.exceptions.RateLimitError (so isinstance checks and PrometheusLogger._get_exception_class_name pick up llm_provider). We deliberately don't call RateLimitError.__init__ — it constructs an httpx.Response we don't need and would just add failure surface; attribute parity is what downstream consumers care about. resolve_llm_provider_for_rate_limit() wraps litellm.get_llm_provider defensively. Internal limiter hooks fire from async_pre_call_hook — well before get_llm_provider runs anywhere else in the request lifecycle — so we have to call it ourselves at raise time. If the model is missing or unparseable (alias, router-only model) we fall back to llm_provider='litellm_proxy' rather than letting a second exception leak out and break the request path. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on parallel-request 429s Both v1 and v3 parallel-request limiters fired bare HTTPException(429) from inside async_pre_call_hook. The downstream Prometheus failure metric reads exception.llm_provider via _get_exception_class_name — the empty value showed up as exception_class='HTTPException' and left model_id='None' on the time series. Threads requested_model through every raise site in: * parallel_request_limiter.py: - check_key_in_limits (the per-key/per-model/per-user/per-team/ per-customer over-limit path) - raise_rate_limit_error (zero-limit + global_max_parallel_requests paths) — now takes an optional requested_model kwarg * parallel_request_limiter_v3.py: - _handle_rate_limit_error (the OVER_LIMIT translator), called from both the should_rate_limit pre-check and the TPM reservation path Resolved via resolve_llm_provider_for_rate_limit so unknown / missing models silently fall back to llm_provider='litellm_proxy' instead of breaking the request path with a second exception. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on dynamic-rate-limit 429s Same plumbing change as the parallel limiters, applied to both dynamic_rate_limiter (v1) and dynamic_rate_limiter_v3: * v1: TPM-zero and RPM-zero paths in async_pre_call_hook now resolve data['model'] -> (model, llm_provider) once and pass it into both raises. * v3: All three raise sites in _check_rate_limits — the model_saturation_check enforced raise, the priority_model enforced raise, and the fail-closed unknown-descriptor branch — now attribute the 429 to the actual provider. Falls back to llm_provider='litellm_proxy' when the model can't be resolved. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on batch-rate-limit 429s batch_rate_limiter._raise_rate_limit_error now takes a requested_model kwarg threaded from data['model'] in _check_and_increment_batch_counters. The batch-creation 429 is what gets raised when the input file's tokens/requests count would push the per-key TPM/RPM window over its limit. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on budget/iterations 429s Final batch of internal raise sites — the user/session-budget and max-iterations hooks. Same pattern: resolve data['model'] once at raise time, attach to ProxyHTTPRateLimitError so Prometheus and observability callbacks can attribute the 429. Hooks updated: * max_budget_limiter (per-user max_budget exceeded) * max_iterations_limiter (per-session agent iteration cap) * max_budget_per_session_limiter (per-session dollar cap) All three fall back to llm_provider='litellm_proxy' when data['model'] is missing or unparseable. Drops the now-unused HTTPException import from each module. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(proxy/hooks): pin provider field on internal rate-limit 429s Regression coverage for the 'provider field missing' bug across every proxy-side rate-limit hook + the helper layer: * ProxyHTTPRateLimitError class shape (HTTPException + RateLimitError, dict-detail stringification, None-provider normalization). * resolve_llm_provider_for_rate_limit happy paths (gpt-4o-mini, anthropic/..., bedrock/...) plus all three fallback branches (None, '', unknown name) plus a 'get_llm_provider raises' case that asserts we swallow the secondary exception. * For each limiter (parallel v1/v3, dynamic v1/v3, batch, max_budget, max_iterations, max_budget_per_session): assert the raised exception is a RateLimitError carrying the resolved model + llm_provider, and a sibling test that asserts the fallback path returns 'litellm_proxy' without leaking a second exception. * Two PrometheusLogger._get_exception_class_name pins so the Prometheus failure metric label flips from 'HTTPException' to 'Openai.ProxyHTTPRateLimitError' (or 'Litellm_proxy.*' on fallback) — that's what dashboards consume. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * perf(proxy/hooks): defer provider resolution to over-limit branches * fix: use error_message in raise_rate_limit_error to avoid literal 'None' in detail * Consolidate rate_limiter_utils imports in dynamic_rate_limiter * fix(proxy): set num_retries/max_retries on ProxyHTTPRateLimitError ProxyHTTPRateLimitError inherits from RateLimitError but did not call RateLimitError.__init__, so num_retries/max_retries were never set. When Starlette's HTTPException lacks __str__, MRO falls through to RateLimitError.__str__, which unconditionally reads these attributes and raises AttributeError during logging/traceback formatting. Initialize them to None defensively. * fix(mypy): silence base-class status_code conflict on ProxyHTTPRateLimitError HTTPException declares 'status_code: int' while openai.RateLimitError (via APIStatusError) declares 'status_code: Literal[429] = 429'. Mypy flags the multi-base override as [misc] in CI lint. The runtime semantics are fine (we set self.status_code in __init__), so silence the class-level annotation conflict with a targeted ignore. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix: annotate batch limiter _raise_rate_limit_error as NoReturn * feat(prometheus): rate-limit category/type labels + exception_class back-compat (follow-up to BerriAI#27687) (BerriAI#27706) * feat(prometheus): add rate_limit_category and rate_limit_type labels Adds two new labels to litellm_proxy_failed_requests_metric so dashboards can split 429s by rate-limit source (vendor vs. litellm-internal) and by the dimension that was exceeded (requests/tokens/concurrent_requests/ budget/max_iterations) without parsing free-text error messages. Closes the Prometheus side of LIT-2718. The unified RateLimitError.category and .rate_limit_type fields landed in PR BerriAI#27687 but were only surfaced on StandardLoggingPayload (custom-callback channel); this exposes them on the metric label set as well. Both labels are populated only when the underlying exception is a litellm.RateLimitError; non-rate-limit failures keep them empty. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(prometheus): populate rate-limit labels + preserve exception_class back-compat Two coupled changes in the Prometheus integration: 1. async_post_call_failure_hook now extracts the new RateLimitError .category / .rate_limit_type fields (added in PR BerriAI#27687) via a _extract_rate_limit_labels helper and forwards them through UserAPIKeyLabelValues onto litellm_proxy_failed_requests_metric. Empty for non-rate-limit failures. 2. _get_exception_class_name special-cases ProxyRateLimitError and keeps emitting 'HTTPException' for the exception_class label. Without this shim, ProxyRateLimitError (which multi-inherits from HTTPException + RateLimitError) would silently flip the label from 'HTTPException' (the historical value for proxy-side 429s) to 'ProxyRateLimitError', breaking existing dashboards / alerts that key off exception_class='HTTPException'. Distinguishing vendor vs. litellm 429s is now the job of the new rate_limit_category label. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(prometheus): cover rate-limit labels and exception_class back-compat Adds 19 tests across: - enum / label-list registration - _extract_rate_limit_labels for vendor RateLimitError, ProxyRateLimitError, non-rate-limit and None inputs (incl. parametrized over every RateLimitErrorCategory x RateLimitType combo) - _get_exception_class_name back-compat: ProxyRateLimitError keeps the legacy 'HTTPException' string while vendor RateLimitError keeps the historical 'Provider.ClassName' format - end-to-end through async_post_call_failure_hook with both ProxyRateLimitError and vendor RateLimitError, asserting both new labels populate and exception_class stays back-compat Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(prometheus): tolerate missing fastapi in lazy ProxyRateLimitError import Address greptile feedback: - async_post_call_failure_hook docstring: drop the stale labelnames listing and reference PrometheusMetricLabels.litellm_proxy_failed_requests_metric as the source of truth so the doc cannot drift from the actual labelset. - _get_exception_class_name: guard the lazy ProxyRateLimitError import with ImportError so router-side fallback callsites don't blow up in non-proxy installs that don't have fastapi (a transitive dep of proxy.common_utils.proxy_rate_limit_error). Behavior is unchanged when fastapi is available. Also fix the existing enterprise callback test that asserted the old labelset on litellm_proxy_failed_requests_metric — it now expects the new rate_limit_category / rate_limit_type labels populated for vendor 429s. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(bugbot): simplify rate-limit label coercion + guard None detail - prometheus.py _extract_rate_limit_labels: RateLimitError.__init__ already normalizes category/rate_limit_type to plain str, so the getattr(.value) + isinstance dance was dead code. Reduce to str(value) if not None. - proxy_rate_limit_error.py _coerce_message: short-circuit None to '' instead of falling through to str(None) = 'None', which produced the literal message 'litellm.RateLimitError: None'. * fix(rate-limit): surface unified category/type fields on BudgetExceededError The most common budget cap (virtual-key max_budget enforcement in auth_checks.py) raises litellm.BudgetExceededError, a bare Exception subclass that bypassed the unified rate-limit error class introduced by PR BerriAI#27687. Custom callbacks reading StandardLoggingPayload.error_information saw category=None and rate_limit_type=None for these 429s, missing the most common budget case (team / org / end-user budgets all hit the same code path). Surface the fields off BudgetExceededError as plain attributes: - category = RateLimitErrorCategory.LITELLM_RATE_LIMIT - rate_limit_type = RateLimitType.BUDGET - llm_provider = "" (or caller-supplied) Switch get_error_information and _extract_rate_limit_labels from isinstance(RateLimitError) gating to duck-typed attribute reads, guarded by membership in the rate-limit enums so unrelated third-party exceptions exposing a .category attribute can't leak garbage values into the payload. This is strictly additive: BudgetExceededError keeps its bare-Exception base class, so `except BudgetExceededError:` handlers keep firing and `except RateLimitError:` does not start catching budget errors. * fix(rate-limit): validate enum membership at duck-typed read sites + enrich BudgetExceededError llm_provider Two follow-ups uncovered during the second QA pass on PR BerriAI#27687: 1. Guard third-party `.category` / `.rate_limit_type` attribute leakage. The duck-typed read in `get_error_information` and `_extract_rate_limit_labels` would forward any string attribute named `category` / `rate_limit_type` on an unrelated third-party exception into the StandardLoggingPayload and Prometheus labels — silently mislabeling custom-callback payloads and blowing out Prometheus label cardinality. Add `validate_rate_limit_category` / `validate_rate_limit_type` helpers that gate on the documented enum value sets; non-matching values are dropped to None. 2. Enrich BudgetExceededError.llm_provider from request_data. Budget checks live in tenant-scoped helpers (key / team / org / tag / end-user / project) that don't see the request model, so the BudgetExceededError they raise carried llm_provider="" — leaving custom-metrics consumers without provider attribution for the most common 429 case. Resolve it once at the central UserAPIKeyAuthExceptionHandler seam, before post_call_failure_hook fires, so the StandardLoggingPayload the callback sees has the same provider attribution as RPM/TPM 429s. Regression tests pin both: 4 leakage tests + 4 enrichment tests. The leakage tests would fail under the pre-validation version of either read site; the enrichment tests would fail if the handler skipped the resolver call. * fix(rate-limit): resolve router model_name aliases to real provider (BerriAI#27914) * fix(rate-limit): resolve router model_name aliases to real provider For nearly every real LiteLLM proxy deployment the request model is a router model_name alias (e.g. 'tpm-locked' -> litellm_params.model: openai/gpt-4o-mini), and 'litellm.get_llm_provider' doesn't know about router aliases — it raises 'LLMProviderNotProvidedError'. The resolver then fell through to the defensive 'litellm_proxy' fallback, so the 'llm_provider' field this PR adds was effectively always 'litellm_proxy' in the field, defeating its purpose for the most common proxy configuration. Add a router-alias fallback step: when 'get_llm_provider' raises, scan the active 'llm_router.model_list' for a deployment whose 'model_name' matches the request model and resolve from its 'litellm_params.model' instead. If multiple deployments share the same alias (load-balancing case) the first one wins — every deployment under one alias should agree on provider in any sensible config, and 'first' is deterministic so the Prometheus label stays stable. Defensive throughout: an uninitialized router, a malformed deployment, a 'litellm_params.model' that itself fails 'get_llm_provider' — every branch falls through to the existing 'litellm_proxy' fallback rather than letting a secondary exception escape and mask the rate-limit error we're trying to surface. Tests: - test_router_alias_resolves_to_underlying_provider: alias 'tpm-locked' -> 'openai/gpt-4o-mini' produces provider='openai', model='gpt-4o-mini'. - test_router_alias_with_multiple_deployments_uses_first. - test_router_alias_unknown_falls_back. - test_router_alias_with_malformed_deployment_falls_back. - Existing fallback test updated to also stub 'litellm.proxy.proxy_server.llm_router' so it exercises the full 'no resolution anywhere' path. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(rate-limit): harden router alias resolver + test isolation - Wrap _resolve_provider_from_router_alias loop in top-level try/except so a non-iterable model_list / unexpected deployment shape can't escape and mask the 429 with a 500. - Type-check litellm_params before .get() to handle non-dict truthy values. - Patch llm_router=None in the parametrized fallback test so a router left by another test in the session can't redirect the unknown-model path. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(bugbot): preserve "BudgetExceededError" Prometheus label Adding llm_provider to BudgetExceededError (so callbacks get provider attribution from StandardLoggingPayload) made the provider-prefix step in _get_exception_class_name silently flip the label from "BudgetExceededError" to e.g. "Openai.BudgetExceededError", breaking dashboards keyed on the historical value. Short-circuit BudgetExceededError in _get_exception_class_name the same way ProxyRateLimitError already is. Provider/category attribution still lands on the new rate_limit_category / rate_limit_type labels. * test: fix invalid 'rpm' rate_limit_type in v3 limiter test mocks The v3 rate limiter only emits 'requests', 'tokens', or 'max_parallel_requests'. Using 'rpm' caused map_v3_rate_limit_type to return None, leaving the expected RateLimitType.REQUESTS untested. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(bugbot): hoist provider resolver + opt-in prom rate-limit labels - dynamic_rate_limiter.py: hoist resolve_llm_provider_for_rate_limit above the TPM/RPM if/elif so the lookup runs once per request, matching the pattern in dynamic_rate_limiter_v3.py. - prometheus.py: gate the new rate_limit_category / rate_limit_type labels on litellm_proxy_failed_requests_metric behind litellm.prometheus_emit_rate_limit_labels (default False). Mirrors the existing prometheus_emit_stream_label opt-in. Preserves the metric's pre-unification label set so existing dashboards / recording rules keep matching after upgrade; operators can enable the new labels once downstream consumers include them. - Tests updated: default-off back-compat case, opt-in path enables the flag before asserting label presence. * fix: stabilize prometheus label sets and drop redundant model normalization - Cache PrometheusLogger.get_labels_for_metric per metric_name so that the label set used to construct counters at __init__ time stays in sync with the label set used at increment time, even if module-level toggles like prometheus_emit_rate_limit_labels or prometheus_emit_stream_label are flipped at runtime. Without this, toggling these flags after the logger was created would cause ValueError from prometheus_client because the runtime labels would not match the counter's declared labelnames. - Drop redundant 'model or ""' guard in ProxyRateLimitError.__init__ where model is already normalized one step earlier. Co-authored-by: Yassin Kortam <yassin@berri.ai> * perf(dynamic_rate_limiter): only resolve provider when rate limit hit Co-authored-by: Yassin Kortam <yassin@berri.ai> * test(prometheus): clear cached metric labels after toggling rate-limit flag The PrometheusLogger caches each metric's label set at construction time so that labels used at counter.labels(...) time stay consistent with the labels the metric was registered with. The enterprise async_post_call_failure_hook test toggles litellm.prometheus_emit_rate_limit_labels = True AFTER the fixture has already built the logger, so without invalidating the cache the rate_limit_category / rate_limit_type labels never reach the mocked counter and the assert_called_once_with check fails. Co-authored-by: Yassin Kortam <yassin@berri.ai> * test: fix CI failures from prom label cache + flaky time-window assertion PrometheusLogger.get_labels_for_metric now caches the per-metric label set at first read so the labels passed to counter.labels(...) stay in lock step with the labels the counter was registered with. This broke two existing test patterns: - test_prometheus_labels.py: tests bind the real method onto a MagicMock, but MagicMock auto-creates a Mock for _cached_metric_labels whose .get(...) returns a truthy Mock — treated as a populated cache and returned as the label set, producing empty filtered labels and KeyError on labels["requested_model"] / ["route"]. Seed real {} containers for _cached_metric_labels and label_filters before binding. - test_prometheus_logging_callbacks.py::test_set_team_budget_metrics_with_custom_labels: the fixture builds the logger before the test monkeypatches litellm.custom_prometheus_metadata_labels, so the cached label set never picks up the new metadata labels. Clear the cache after the monkeypatch (same pattern already used for the rate-limit toggle in test_async_post_call_failure_hook). UI: view_logs/index.test.tsx "Last Minute" window assertion is off by one at the minute boundary. start_date is floored to the minute, so the dropped sub-minute fraction can push the truncated-seconds diff up to (minMinutes+1)*60 exactly when the click lands near a minute rollover. Switch the upper bound to toBeLessThanOrEqual. * feat(otel-v2): surface rate_limit_category + rate_limit_type on failed LLM-call spans PR BerriAI#28909 introduced the typed v2 OTel engine that builds spans from StandardLoggingPayload, with SpanError carrying error_type + message and the genai mapper stamping error.type onto every failed LLM-call span. This PR's earlier commits added error_rate_limit_category and error_rate_limit_type to the same StandardLoggingPayload.error_information the v2 engine reads — but neither field reached a span attribute, so v2 OTel traces stayed opaque about *why* a 429 fired (vendor vs litellm, RPM vs TPM vs concurrent vs budget vs max_iterations) even after the custom-callback and prometheus surfaces gained that decomposition. Three coupled changes: 1. semconv.py: add LiteLLM.ERROR_RATE_LIMIT_CATEGORY / LiteLLM.ERROR_RATE_LIMIT_TYPE under the litellm.* vendor namespace (no GenAI semconv equivalent exists for who-rate-limited / which-dimension). 2. payloads.py: extend SpanError with rate_limit_category + rate_limit_type, populated by _parse_error() from the same error_information.error_rate_limit_* fields the custom-callback channel and prometheus rate_limit_category / rate_limit_type labels read. Single source of truth across all three observability surfaces. 3. mappers/genai.py: stamp the two attributes on the LLM-call span when present. drop_none guarantees they stay absent (not 'None') for non-rate-limit failures so trace consumers can read them unconditionally. Three regression tests in test_otel_v2_emitter.py pin: a vendor / litellm-internal RateLimitError lands category=litellm_rate_limit + rate_limit_type=requests on the span; a BudgetExceededError lands rate_limit_type=budget; a non-rate-limit failure (BadRequestError) keeps the rate_limit_* attributes absent. Mutation-tested against reverting either the SpanError extension or the _parse_error read site — both new tests fail under either mutation. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test: align prometheus user-budget + logs quick-select tests with merged code The merge into this branch left two test patterns out of step with the code they exercise. test_set_user_budget_metrics_includes_user_email_and_alias_labels_when_opted_in flipped litellm.prometheus_user_budget_label_include_email_alias after the fixture had already built the PrometheusLogger. get_labels_for_metric now snapshots each metric's label set at construction time, so the runtime flip no longer reached the cached labels. Enable the flag before constructing the logger, matching how the proxy applies config at startup. view_logs/index.test.tsx referenced uiSpendLogsCall and moment without importing them, and the merged index.tsx now fetches through useLogFilterLogic (the hook the file stubs out) rather than calling uiSpendLogsCall directly. Add the imports and restore the real hook for the Quick Select window assertions so the call is actually observed. * refactor(otel/v2): drop rate-limit decomposition from the LLM-call span Proxy-side rate limits (litellm_rate_limit, budget, max_iterations) are rejected at the gate before any upstream call, so async_post_call_failure_hook tags the synthetic failure log with LITELLM_LOGGING_NO_UPSTREAM_LLM_CALL and the v2 OTel logger never opens an LLM-call span for them; the litellm.error.rate_limit_category / litellm.error.rate_limit_type attributes were dead for exactly the cases they were meant to surface. The only failure that does open an LLM-call span carrying a RateLimitError is a vendor 429, where rate_limit_type is always None and the category just restates error.type=RateLimitError. The decomposition still reaches downstream consumers through StandardLoggingPayload.error_information.error_rate_limit_* and the prometheus rate_limit_category / rate_limit_type labels, both unchanged. Removes the SpanError fields, the _parse_error reads, the genai mapper attributes, the semconv keys, and the three span tests that asserted a scenario that never reaches the mapper in production. * fix(batch_rate_limiter): map max_parallel_requests to concurrent_requests * refactor(prometheus): drop transitive fastapi import from _get_exception_class_name Read the legacy exception_class label from a prometheus_exception_class_name marker on ProxyRateLimitError instead of importing the proxy module, keeping the integrations layer free of a transitive fastapi dependency. * chore(ui): sync schema.d.ts with unified rate-limit error spec The ProxyRateLimitError docstring flows into the proxy OpenAPI spec's 429 response description, so the generated dashboard types were out of sync. Regenerated via npm run gen:api (Check UI API Types Sync). --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> Co-authored-by: Yassin Kortam <yassin@berri.ai>

…iAI#28909)

…Opus 4.7 self-heal) (BerriAI#29344) * test(logging): align DB metrics event_metadata assertions with safe redaction PR BerriAI#28909 hardened log_db_metrics to emit a minimal, non-sensitive event_metadata (only table_name when present, otherwise None) instead of dumping function_name, function_kwargs, and function_args onto the span. The test in test_log_db_redis_services was not updated and still asserted "function_name" in event_metadata, which raised TypeError (argument of type 'NoneType' is not iterable) and turned the logging_testing CI job red on litellm_internal_staging. Update test_log_db_metrics_success to assert event_metadata is None when no table_name is passed, and add test_log_db_metrics_event_metadata_is_safe as a regression guard verifying that only the table name surfaces and that sensitive kwargs (tokens, prisma client) are never dumped. * test(bedrock): self-heal opus-4-7 grid cells when unentitled on CI The bedrock-claude-opus-4-7 converse cells are unentitled on the Bedrock CI account, so they were marked xfail. xfail keeps reporting them as expected failures even after access is granted, so the wire translation never gets verified again. Now the cell makes the call and skips only when Bedrock replies "is not available for this account"; the moment the model is entitled the same cells run their full assertions with no edit. A focused unit test pins the tolerance predicate so any other failure still surfaces loudly and the available path still runs the assertions.

…utes (BerriAI#29327) * test(logging): align DB metrics event_metadata assertions with safe redaction PR BerriAI#28909 hardened log_db_metrics to emit a minimal, non-sensitive event_metadata (only table_name when present, otherwise None) instead of dumping function_name, function_kwargs, and function_args onto the span. The test in test_log_db_redis_services was not updated and still asserted "function_name" in event_metadata, which raised TypeError (argument of type 'NoneType' is not iterable) and turned the logging_testing CI job red on litellm_internal_staging. Update test_log_db_metrics_success to assert event_metadata is None when no table_name is passed, and add test_log_db_metrics_event_metadata_is_safe as a regression guard verifying that only the table name surfaces and that sensitive kwargs (tokens, prisma client) are never dumped. * test(bedrock): self-heal opus-4-7 grid cells when unentitled on CI The bedrock-claude-opus-4-7 converse cells are unentitled on the Bedrock CI account, so they were marked xfail. xfail keeps reporting them as expected failures even after access is granted, so the wire translation never gets verified again. Now the cell makes the call and skips only when Bedrock replies "is not available for this account"; the moment the model is entitled the same cells run their full assertions with no edit. A focused unit test pins the tolerance predicate so any other failure still surfaces loudly and the available path still runs the assertions. * test(reasoning-effort-grid): add claude-opus-4-8 across provider routes Adds claude-opus-4-8 to the anthropic, azure, vertex and bedrock-converse routes (275 cells total) so the reasoning-effort wire translation is covered for the new model. The bedrock opus-4-8 and opus-4-7 cells reuse the self-heal path: they run the call and skip only on Bedrock's "is not available for this account" reply, then assert in full once the model is entitled. The azure and vertex opus-4-8 cells stay xfail until a Foundry deployment exists and Vertex availability is confirmed. The shared xhigh+max capability set is renamed to _CAPS_XHIGH_MAX now that more than one model uses it.

…erriAI#29160) * feat(pass_through): extend passthrough_managed_object_ids to Azure Adds managed ID minting/resolution for Azure passthrough endpoints (/azure/...) alongside the existing OpenAI passthrough support. Key changes: - pass_through_endpoints.py: detect azure/azure_ai custom_llm_provider (string or enum) to set _is_managed_id_provider and _managed_id_provider; both INPUT and OUTPUT rewrite blocks now fire for Azure. - llm_passthrough_endpoints.py: forward custom_llm_provider into create_pass_through_route so it reaches pass_through_request (was None). - managed_id_rewriter.py: extend _PASSTHROUGH_PREFIX_RE and _canonical_path to strip /azure/openai prefix and add /v1/ for Azure paths that omit it; add ("azure", method, path) entries to BUILTIN_OUTPUT_ID_FIELD_MAP for files and batches endpoints. - managed_id_codec.py / types/utils.py: supporting codec and enum constant. - proxy_server.py: register llm_passthrough_router before batches_router to prevent route collision for /openai_passthrough/* paths. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(pass_through): remove unused imports for ruff F401 Co-authored-by: Cursor <cursoragent@cursor.com> * fix(pass_through): satisfy mypy for optional parsed_body Co-authored-by: Cursor <cursoragent@cursor.com> * fix(pass_through): compute query params string after managed-ID rewrite Move requested_query_params_str computation to after the managed-ID input rewrite block so logging_url reflects the rewritten raw-provider query params actually sent upstream, instead of the original managed IDs. Co-authored-by: Yassin Kortam <yassin@berri.ai> * Add support for managed ids for passthrough responses api * Add support for list batches and list files * style: run Black on passthrough managed ID files Fix CI formatting for managed_id_rewriter.py and pass_through_endpoints.py. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(passthrough): parse json file_object and implement before-cursor pagination - Parse row.file_object via json.loads when Prisma returns it as a string; mirrors openai_files_endpoints/common_utils.py so list responses keep all stored detail fields (status, timestamps, etc.). - Implement the previously-parsed-but-unused 'before' cursor for list pagination by flipping fetch order to ascending with a 'gt' bound on created_at, then reversing rows so the response stays newest-first. Co-authored-by: Yassin Kortam <yassin@berri.ai> * Remove logger * refactor: split list_passthrough_ids_from_db to fix PLR0915 Extract pagination, fetch, and serialization helpers so the main list function stays under the statement limit without changing behavior. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: scope passthrough managed ID dedup and list by provider Validate embedded provider before reusing deduped file/object rows so OpenAI and Azure cannot share the same managed ID for an identical raw ID. Filter list responses to rows whose managed IDs decode to the current provider, with over-fetch scanning when needed. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(managed_id_rewriter): cap pagination trim at effective limit When raw_limit > 100, fetch_limit is capped at 101 (one extra row to detect has_more), but trimming with rows[:raw_limit] failed to drop the sentinel row. Use the capped effective limit instead. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix: cross-provider object collision and fail-closed list error handling Greptile P1: move provider check before access check in _mint_or_reuse_object so a cross-provider raw ID collision (OpenAI and Azure share the same batch_ ID) falls through to mint a new provider-scoped row instead of raising 404. Veria-ai medium: _fetch_provider_scoped_list_rows now always returns (page, has_more) — DB errors break out of the scan loop and return matched rows so far. list_passthrough_ids_from_db never returns None for a recognised list route, so the caller can never fall through to the upstream provider. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: namespace passthrough model_object_id by provider to prevent unique violation Store model_object_id as 'passthrough:{provider}:{raw_id}' instead of the bare raw ID so OpenAI and Azure can each own a row for the same raw batch ID without hitting the @unique constraint. Dedup lookup uses the same namespaced key so it is implicitly provider-scoped and the _managed_id_matches_provider check is no longer needed on the object path. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: gate list interception on managed_files hook like input/output rewrites Without the hook no managed IDs are minted so the DB is empty. Intercepting GET /v1/files without the hook returned an empty list and hid the caller's real upstream files/batches. Matches the guard used by the input and output rewrite blocks. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: set has_more=True when scan cap is hit with a full final DB batch When max_scans (20) is exhausted and the last DB page was full-sized, there are almost certainly more rows beyond the scan window. Track last_batch_full across iterations so the scan_cap_hit condition sets has_more=True in that case, preventing silent pagination truncation in high-mixed-provider pools. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(managed_id_rewriter): scope pagination cursor lookup to caller-owned rows Prevent a cross-tenant timing oracle by constraining the after/before cursor row lookup to the caller's owner_filter, and cover the real Azure responses path form (no /v1/) in tests. * fix(managed_id_rewriter): align passthrough list metadata with direct GET Persist upstream file metadata when minting a managed file ID and rewrite nested batch file IDs before snapshotting the object, so DB-served file/batch list responses return the same fields and managed IDs as a direct endpoint GET. * fix(managed_id_rewriter): degrade to raw id on cross-owner object collision The OUTPUT (mint) path of _mint_or_reuse_object raised HTTPException(404) when a dedup hit on the namespaced model_object_id belonged to a different owner, converting a successful upstream batch/response creation into a 404 for the caller. Two upstream accounts under one provider name can issue the same raw id, so this is reachable in multi-tenant deployments. Return the caller's raw id unmanaged instead: the upstream create already succeeded, a new managed row can't be minted (model_object_id is @unique), and reusing the other owner's managed id would later fail the access check. * fix(managed_id_rewriter): scope list cursor by provider and cap body-rewrite recursion depth * perf(managed_id_rewriter): push batch list provider scope to DB and anchor canonical-path prefix Object (batch) list rows store model_object_id as passthrough:{provider}:{raw}, so the provider filter is now applied at the indexed DB column, collapsing the application-layer multi-scan to a single query for that table. File rows keep the decode-based scan since they have no provider column. Anchor the canonical-path prefix regex at a path boundary so routes such as /openai_realtime/... are no longer mis-stripped. * fix(managed_id_rewriter): refresh stored batch snapshot on reuse The dedup-reuse path in _mint_or_reuse_object returned the existing managed id without updating the stored file_object, so DB-served list responses kept the creation-time snapshot and showed null output_file_id/error_file_id even after the batch completed. Refresh the snapshot when an owned row is reused so the list reflects the batch's latest state. * fix(managed_id_rewriter): deny cross-owner object access on retrieve/cancel/delete Returning the raw id when can_access_resource fails only made sense for create responses, where the caller's own upstream create succeeded under a raw id that a different owner already holds. On retrieve/cancel/delete the caller reaches that branch only by supplying another tenant's raw id (which bypasses the managed-id input gate), so echoing the upstream object back leaked it cross-tenant. Restrict the raw fallback to create routes and return 404 otherwise. * fix(managed_id_rewriter): deny cross-owner file access on retrieve/delete _mint_or_reuse_file scoped the raw file dedup lookup to the current caller, so a raw file-... id belonging to another tenant was never found and the OUTPUT path minted a fresh managed id for that same upstream file under the caller. A raw id only reaches this path by skipping the managed-id input gate (raw provider ids are opt-out), so a different-owner row means the caller is touching someone else's file. Look up flat_model_file_ids globally and run can_access_resource; deny with 404 on retrieve/delete and leave the raw id unmanaged on create, which mirrors the cross-owner handling already in _mint_or_reuse_object. * fix(managed_id_rewriter): deterministic provider-scoped file dedup Replace the unscoped find_first in _mint_or_reuse_file with a find_many ordered by created_at and an application-layer provider filter. The file table has no provider column, so a raw file id shared across OpenAI and Azure could map to one row per provider; find_first then picked a row non-deterministically and, on a provider mismatch, minted a fresh managed row on every call, accumulating duplicates. Selecting the oldest matching same-provider row the caller can access keeps reuse stable and prevents duplicate rows while preserving the cross-tenant deny/leave-raw behaviour. * refactor(pass_through): scope passthrough managed IDs on the explicit provider Move the openai/azure detection out of pass_through_request into resolve_passthrough_managed_id_provider in llms/base_llm/managed_resources, and key managed-ID rewriting on the forwarded custom_llm_provider rather than the upstream URL's EndpointType. The helper documents why azure and azure_ai collapse to one "azure" scope (they share the same Azure OpenAI files/batches surface, so an ID minted on one must resolve on the other) and returns None for any other provider so a third-party OpenAI-compatible endpoint never triggers managed-ID minting. Add TestManagedIdProviderScope covering the azure_ai -> azure collapse and the non-openai/azure exclusion. * test(log_db_metrics): assert sanitized event_metadata contract test_log_db_metrics_success still asserted the legacy event_metadata shape (function_name/function_kwargs/function_args), which BerriAI#28909 intentionally removed so that live Prisma clients, OTel spans, and secrets never land on a service-log span. The decorator now emits only a sanitized payload: None when no table_name is present, and {"table_name": ...} when it is. Update the test to verify both branches of that contract. * fix(managed_id_rewriter): page provider-scoped file list by offset The file list scan advanced its cursor with a strict created_at boundary. When several rows shared a created_at timestamp and a non-matching provider row sat on the page boundary, the next query skipped the remaining rows at that timestamp, dropping matching files from the response. Page by a stable offset over a total order (created_at plus the unique id column) so tied rows are never skipped or repeated. * fix(managed_id_rewriter): push file-list provider scope to the DB The file-list helper had no provider column to query, so it scanned the table and filtered by decoding each managed ID in the application layer, capped at 20 pages. For an admin with a large mixed-provider file pool that cap could truncate a page. Mint now writes a _passthrough_provider:{provider} marker into flat_model_file_ids, giving the file table the same DB-queryable provider scope object rows already get from the namespaced model_object_id. The list helper pushes the scope into the query so a single round-trip serves the page. The scan loop, the offset paging, and the cap are gone, so pages can no longer truncate, leak the other provider, or skip rows that share a created_at timestamp. * fix(managed_id_rewriter): deny raw provider IDs that map to another tenant's managed resource Clients only ever receive managed IDs on passthrough, so a raw file/batch/response ID for another tenant's managed object can only be recovered by decoding that tenant's managed ID. Raw IDs were forwarded upstream untouched (deliberate opt-out), which on a retrieve/cancel/delete executed upstream before the response-side ownership check ran, leaking a cross-tenant action. Guard raw provider IDs on the input path: when a raw file-/batch_/resp_ ID resolves to a managed row the caller cannot access, return 404 before forwarding. Genuinely unmanaged raw IDs (no DB row) and IDs the caller owns are left untouched, preserving the opt-out. * test(managed_id_rewriter): cover azure_ai and pre-versioned azure passthrough paths * fix(managed_id_rewriter): fall back to raw id when persistence fails A DB persistence failure after a successful upstream create left the client holding a minted managed ID with no backing row, so every later resolve returned 404 and the resource was permanently unreachable. Mint the managed ID only when the row is stored; on persistence failure return the raw provider id, matching the no-persistence-available fallback, so the freshly-created resource stays reachable. * fix(managed_id_rewriter): log only rewritten query param keys * fix(managed_id_rewriter): use compound (created_at, id) list cursor boundary A timestamp-only lt/gt cursor boundary skips list rows that share the cursor row's created_at across a page boundary, silently dropping them. Compare the unique id (the secondary sort key) alongside created_at so the page walk stays complete when timestamps tie. * fix(managed_id_rewriter): converge concurrent object creates on one managed id * fix(managed_id_rewriter): bound raw-id guard DB lookups per request The INPUT guard fired one DB lookup for every file-/batch_/resp_ prefixed string in the path, query, and body. The file-id guard is an unindexed array-containment scan over LiteLLM_ManagedFileTable, so an authenticated caller could amplify a single passthrough request into thousands of full-table scans by packing a body with id-shaped strings. De-dupe raw ids within a request and cap the distinct guard lookups, failing closed with 400 instead of skipping the guard. Legitimate callers hold managed ids (resolved via an indexed unified_*_id lookup, not the guard), so the cap only trips under abuse. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>

…odel, and llm_provider fields (BerriAI#27687) * feat(exceptions): add RateLimitErrorCategory + headers/detail fields on RateLimitError LiteLLM previously surfaced rate-limit conditions through several unrelated error classes (RateLimitError, FastAPI HTTPException(429), BaseLLMException). This commit adds the data model needed to consolidate them under a single class: * RateLimitErrorCategory enum exposing four categorical values (vendor_rate_limit, vendor_batch_rate_limit, litellm_rate_limit, litellm_batch_rate_limit) so callers can switch on the rate-limit source. * New optional fields on RateLimitError: - category (defaults to vendor_rate_limit, preserving today's behavior for every existing call site in exception_mapping_utils); - headers (preserves retry-after / rate_limit_type / reset_at across the proxy boundary instead of dropping them on the floor); - detail (mirrors FastAPI HTTPException.detail so the same instance can be serialized through both paths). litellm.RateLimitErrorCategory is re-exported at the package root to match the existing exception-export pattern. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(proxy): add ProxyRateLimitError unifying RateLimitError + HTTPException Adds a single proxy-side error class that subclasses BOTH litellm.exceptions.RateLimitError AND fastapi.HTTPException via cooperative multiple inheritance. Why both bases: * Subclassing RateLimitError lets user code catch every rate-limit source with one 'except RateLimitError' and switch on the new .category field. * Subclassing HTTPException keeps every existing FastAPI plumbing path (the isinstance(e, HTTPException) branches in proxy_server.py route handlers, FastAPI's own dispatcher, and tests asserting pytest.raises(HTTPException)) working without modification, and preserves retry-after / rate_limit_type / reset_at headers on the wire. The class declaration order is (HTTPException, RateLimitError) so the MRO puts HTTPException's no-super-call __init__ ahead of openai's cooperative __init__ chain — preventing openai.APIError.super().__init__(message) from landing in HTTPException.__init__(status_code=message). LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from budget + iteration limiters Replaces three bare HTTPException(status_code=429, ...) call sites with ProxyRateLimitError, which is both a RateLimitError (catchable by category) and an HTTPException (preserves existing FastAPI serialization). Drops the now-unused HTTPException import in the iteration / per-session limiters. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from parallel-request limiters Replaces HTTPException(status_code=429, ...) call sites in the v1 and v3 parallel-request limiters (key/team/user/model/customer rate limits) with ProxyRateLimitError. Updates the raise_rate_limit_error helper's return type annotation accordingly. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from dynamic rate limiters Replaces HTTPException(status_code=429, ...) call sites in the v1 and v3 dynamic rate limiters (project-level TPM/RPM allocation, model-saturation checks, priority-based limits, fail-closed guards) with ProxyRateLimitError. The v3 limiter still imports HTTPException for an unrelated bare 'except HTTPException:' branch. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(proxy/hooks): raise ProxyRateLimitError from batch rate limiter Replaces HTTPException(status_code=429, ...) in batch_rate_limiter._raise_rate_limit_error with ProxyRateLimitError tagged as RateLimitErrorCategory.LITELLM_BATCH_RATE_LIMIT so users can distinguish batch-level throttling (which counts requests/tokens across an uploaded batch input file before submission) from the generic key/team/user RPM/TPM limiter. The HTTPException import is retained because the same module raises HTTPException for unrelated 403/IO error paths. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): pin down unified rate-limit error contract Adds a dedicated test module covering the new RateLimitErrorCategory enum, RateLimitError.category default + override behavior, ProxyRateLimitError's dual nature (RateLimitError + HTTPException), and a parametrized regression guard that asserts every proxy hook module imports the unified class. The regression guard catches the failure mode the refactor is designed to prevent: someone re-introducing a bare HTTPException(status_code=429, ...) in one of the hook modules instead of going through ProxyRateLimitError. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(logging): expose rate-limit category via StandardLoggingPayload Adds an optional 'error_rate_limit_category' field to StandardLoggingPayloadErrorInformation, populated from the unified RateLimitError.category attribute (introduced in the previous commits on this branch). Why: the .category attribute is reachable off the raw exception today via getattr(e, 'category', None), but the structured contract that downstream custom callbacks / loggers / spend log writers consume is the StandardLoggingPayload. Without this field, a user building custom rate-limit metrics on top of callback data has to special-case the raw exception object — which defeats the purpose of the StandardLoggingPayload abstraction. The field is None for non-rate-limit exceptions (so consumers can read it unconditionally without isinstance checks) and is one of the RateLimitErrorCategory string values otherwise. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): assert StandardLoggingPayload carries the category Five tests covering: vendor default, explicit litellm_rate_limit and litellm_batch_rate_limit values, None for non-rate-limit exceptions, and None when no exception is provided. Pins down the contract that custom callbacks can read 'error_information.error_rate_limit_category' off the StandardLoggingPayload to drive custom rate-limit metrics without ever reaching for the raw exception. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(types): silence mypy [misc] on intentional dual-base attr overlap mypy emits two [misc] errors on the ProxyRateLimitError class line because its two bases declare overlapping attributes with related-but-not-identical annotations: * status_code: int on starlette HTTPException vs. Literal[429] on openai's RateLimitError (every openai status-error subclass narrows it the same way and silences pyright with the same convention). * headers: Mapping[str, str] | None on HTTPException vs. our Optional[ Dict[str, str]] (the proxy hooks always carry a stringified dict). Both narrowings are intentional and enforced at construction time. Add a type: ignore[misc] with an inline explanation rather than relax the annotations on the parent or change the wire-format guarantees. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): add direct hook-invocation tests to lift patch coverage Adds six end-to-end tests that drive each refactored hook past its limit and assert the unified ProxyRateLimitError is raised with the correct category and dual-base shape. Complements the import-shape-only parametrized guard above by actually executing the new 'raise ProxyRateLimitError(...)' lines so codecov's patch coverage sees them as hit. Hooks covered (one test each): * parallel_request_limiter v1 — direct call to raise_rate_limit_error() * parallel_request_limiter v3 — direct call to _handle_rate_limit_error with a fabricated OVER_LIMIT response * max_iterations_limiter — full async_pre_call_hook with mocked agent registry, second call exceeds budget=1 * max_budget_limiter — async_pre_call_hook with mocked get_current_spend * dynamic_rate_limiter v1 — async_pre_call_hook with mocked check_available_usage forcing available_tpm == 0 * batch_rate_limiter — direct _raise_rate_limit_error call, asserts category is the batch-specific LITELLM_BATCH_RATE_LIMIT (not the generic LITELLM_RATE_LIMIT) LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix: guard rate_limit_category extraction with isinstance check * test(rate-limit): cover remaining hook raise sites for codecov Adds five more direct hook-invocation tests so every PR-touched line in the proxy hooks is exercised by tests in tests/test_litellm/, which codecov measures: * parallel_request_limiter v1 — check_key_in_limits inline raise (the second raise site, separate from the raise_rate_limit_error helper covered earlier) * dynamic_rate_limiter v1 — RPM raise branch (TPM branch was already covered) * dynamic_rate_limiter v3 — parametrized over all three raise sites: model_saturation_check, priority_model, and the fail-closed fallback for an unrecognized descriptor_key * max_budget_per_session_limiter — full async_pre_call_hook with a mocked agent registry and over-budget cached spend All 42 tests in test_rate_limit_error_unification.py now pass and together exercise every changed import + raise line across the eight refactored proxy hooks. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix: use computed error_message in ProxyRateLimitError detail * fix(parallel-request-limiter): drop None from detail; annotate raise_rate_limit_error as NoReturn The v1 ' raise_rate_limit_error' helper built an unused 'error_message' variable and then assembled the actual ' detail' via an f-string that interpolated 'additional_details' verbatim — producing 'Max parallel request limit reached None' when invoked without arguments (flagged by code review). Fix the helper to: - use the constructed 'error_message' as the detail - annotate the helper as NoReturn since it always raises - drop the redundant 'raise'/'return' at the two call sites Add two regression tests covering both the with- and without- additional_details paths. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): drop literal 'None' from raise_rate_limit_error detail The v1 parallel_request_limiter's raise_rate_limit_error helper has a long-standing bug: it computes a None-guarded 'error_message' string but then ignores it and emits an f-string that interpolates the raw 'additional_details' arg. Callers that pass no argument get 'Max parallel request limit reached None' as the user-facing detail. This commit: * wires error_message into the detail kwarg so the None-guard actually applies and operators see a clean message; * changes the return-type annotation from ProxyRateLimitError to NoReturn (the function always raises) so type-checkers know callers after this invocation are unreachable. Greptile P1 + P2 review feedback on PR BerriAI#27687. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(types): demote TypedDict floating string to a # comment A string literal placed after a field declaration in a TypedDict body is not a per-field docstring — it's an orphaned string expression Python discards. Tools like mypy / pyright that inspect TypedDict fields won't surface that text either. Move the documentation for error_rate_limit_category to a real comment so the intent is visible to readers and type-checker tooling without the misleading docstring framing. Greptile P2 review feedback on PR BerriAI#27687. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * security(exceptions): do not auto-copy vendor response headers to e.headers A vendor 429 response can set arbitrary headers (Set-Cookie, CORS overrides, …). Previously, when RateLimitError was constructed with only a 'response=' (no explicit 'headers=' kwarg), self.headers fell back to a copy of response.headers. If a downstream proxy serializer ever forwarded e.headers to the client, a malicious upstream could inject browser-interpreted headers for the proxy origin. Drop the fallback. Only headers passed explicitly via the headers= kwarg make it onto self.headers (proxy hooks pass retry-after etc. — they control what's surfaced). Vendor response headers stay reachable on e.response.headers for callers that explicitly want them. Today's proxy_server.py route handlers don't actually forward e.headers on the wire (they construct ProxyException without passing headers), so no current behavior changes — this is a defensive narrowing so the fallback can never be turned into a vector when someone wires e.headers through later. Veria-AI security review feedback on PR BerriAI#27687. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): regression guards for review-pass fixes Pins down the three review-pass fixes: * test_parallel_request_limiter_v1_helper_no_additional_details — calls raise_rate_limit_error() with no args and asserts the detail does NOT contain the literal string 'None'. Pre-fix, callers got 'Max parallel request limit reached None'. * test_rate_limit_error_does_not_auto_copy_response_headers — passes a vendor httpx.Response with a Set-Cookie header to RateLimitError WITHOUT an explicit headers= kwarg, asserts self.headers stays None (no leak), then re-checks that an explicit headers= kwarg DOES populate self.headers. Vendor headers remain reachable on e.response.headers for callers that explicitly want them. * The existing v1-helper test now also asserts the additional_details string makes it through to the detail. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(rate-limit): add orthogonal RateLimitType (requests/tokens/concurrent_requests/budget/max_iterations) trho's last ask in the LIT-2968 thread: distinguish rate-limit failures by the dimension that was exceeded, not just by who rate-limited (vendor vs. litellm). Adds: - RateLimitType str-enum exposed at `litellm.RateLimitType` with values requests / tokens / concurrent_requests / budget / max_iterations. - `rate_limit_type` kwarg on litellm.RateLimitError + ProxyRateLimitError; None default so existing callers (vendor-429 path in exception_mapping_utils) remain a no-op. - StandardLoggingPayloadErrorInformation.error_rate_limit_type so custom callbacks can split rate-limit failures by cause without parsing free-text error messages. Mirror to error_rate_limit_category extraction in get_error_information(); single isinstance(RateLimitError) check covers both. - map_v3_rate_limit_type() helper to collapse the v3 limiter's internal labels ("requests", "tokens", "max_parallel_requests") onto the public enum so the v3 limiter and dynamic_rate_limiter_v3 share one mapping. Defensive None on unknown values rather than silently picking a wrong dimension. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(proxy/hooks): wire rate_limit_type onto every limiter raise site Each refactored proxy hook now populates rate_limit_type with the dimension that actually tripped the limit, so downstream consumers (custom callbacks, prometheus exporters via the StandardLoggingPayload) can split key/team/user rate-limit failures by cause: - parallel_request_limiter (v1): detect dimension from current vs. limit in the post-cache branch (concurrent_requests > tokens > requests, matches the boolean condition order). Base case (current is None, one limit set to 0) picks the most-specific zero. raise_rate_limit_error() helper accepts an explicit rate_limit_type kwarg with CONCURRENT_REQUESTS default (matches every existing internal call site, including the global-limit branch). - parallel_request_limiter (v3): forward status["rate_limit_type"] through map_v3_rate_limit_type() so "max_parallel_requests" → CONCURRENT_REQUESTS for the public field while the raw v3 jargon stays on the HTTP header for wire-format backward compat. - dynamic_rate_limiter (v1): TPM-zero → TOKENS, RPM-zero → REQUESTS. Pass data["model"] through so callbacks see the model that hit the limit (addresses the secondary "provider missing" complaint in the original Slack thread, partially — the model is what dashboards typically split on). - dynamic_rate_limiter (v3): forward status["rate_limit_type"] via map_v3_rate_limit_type() at every raise site (model_saturation_check, priority_model, fail-closed unknown-descriptor guard). Also pass model. - batch_rate_limiter: limit_type is hard-typed "requests"|"tokens" — map directly without going through the helper's None branch. - max_budget_limiter, max_budget_per_session_limiter: BUDGET. - max_iterations_limiter: MAX_ITERATIONS. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): cover RateLimitType enum, hook wiring, and StandardLoggingPayload propagation 27 new tests across five new test classes: - TestRateLimitType: enum exposed at litellm.RateLimitType, all five values defined, RateLimitError default is None (vendor 429 path makes no claim about which dimension), accepts both string and enum forms with str-coercion guarantee for downstream JSON serializers. - TestProxyRateLimitErrorType: ProxyRateLimitError default is None, accepts string or enum, doesn't break existing callers that pass nothing. - TestMapV3RateLimitType: pins each v3-internal → public-enum mapping (tokens, requests, max_parallel_requests → concurrent_requests, unknown → None) so a future v3 refactor can't silently swap dimensions. - TestStandardLoggingPayloadCarriesType: the new error_rate_limit_type field reaches the structured payload for both ProxyRateLimitError and plain RateLimitError, is None when unspecified, and is None for non-rate-limit exceptions (symmetric with error_rate_limit_category). - TestProxyHooksWireTypeCorrectly: drives the actual raise sites in the v1 parallel_request_limiter helper, the v3 _handle_rate_limit_error (both "tokens" and "max_parallel_requests" paths), and the batch limiter (both tokens and requests paths) — coverage tools see the new rate_limit_type= kwargs as exercised, not just the import shape. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(rate-limit): cover _coerce_message branches and v1 dimension detection Drives the patch coverage on the new orthogonal RateLimitType wiring up to (or close to) 100% on the touched files. ProxyRateLimitError._coerce_message — was 22% covered, now 100%: * nested {error: {message}} dict * nested {message: {message}} dict (alt key) * dict without 'error'/'message' keys → JSON dump fallback * non-JSON-serializable dict value → str() fallback * non-string non-mapping detail (int) → str() coercion v1 parallel_request_limiter dimension detection — was 0% covered, now exercised across 6 parametrized cases: * check_key_in_limits else-branch: current at concurrent / TPM / RPM cap → asserts rate_limit_type is concurrent_requests / tokens / requests. * check_key_in_limits base case (current is None): max_parallel_requests / tpm_limit / rpm_limit set to 0 → asserts the most-specific zero attribution wins per the helper's order. LIT-2968 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(proxy/hooks): add ProxyHTTPRateLimitError + provider resolver Introduces a small helper layer used by every proxy-side rate-limit hook so that the 429 they raise carries a populated llm_provider / model — instead of an empty exception.llm_provider that downstream loggers (Prometheus failure metric, observability callbacks) read as 'no provider attribution'. ProxyHTTPRateLimitError inherits from both fastapi.HTTPException (so the proxy server still renders it as a 429) and litellm.exceptions.RateLimitError (so isinstance checks and PrometheusLogger._get_exception_class_name pick up llm_provider). We deliberately don't call RateLimitError.__init__ — it constructs an httpx.Response we don't need and would just add failure surface; attribute parity is what downstream consumers care about. resolve_llm_provider_for_rate_limit() wraps litellm.get_llm_provider defensively. Internal limiter hooks fire from async_pre_call_hook — well before get_llm_provider runs anywhere else in the request lifecycle — so we have to call it ourselves at raise time. If the model is missing or unparseable (alias, router-only model) we fall back to llm_provider='litellm_proxy' rather than letting a second exception leak out and break the request path. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on parallel-request 429s Both v1 and v3 parallel-request limiters fired bare HTTPException(429) from inside async_pre_call_hook. The downstream Prometheus failure metric reads exception.llm_provider via _get_exception_class_name — the empty value showed up as exception_class='HTTPException' and left model_id='None' on the time series. Threads requested_model through every raise site in: * parallel_request_limiter.py: - check_key_in_limits (the per-key/per-model/per-user/per-team/ per-customer over-limit path) - raise_rate_limit_error (zero-limit + global_max_parallel_requests paths) — now takes an optional requested_model kwarg * parallel_request_limiter_v3.py: - _handle_rate_limit_error (the OVER_LIMIT translator), called from both the should_rate_limit pre-check and the TPM reservation path Resolved via resolve_llm_provider_for_rate_limit so unknown / missing models silently fall back to llm_provider='litellm_proxy' instead of breaking the request path with a second exception. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on dynamic-rate-limit 429s Same plumbing change as the parallel limiters, applied to both dynamic_rate_limiter (v1) and dynamic_rate_limiter_v3: * v1: TPM-zero and RPM-zero paths in async_pre_call_hook now resolve data['model'] -> (model, llm_provider) once and pass it into both raises. * v3: All three raise sites in _check_rate_limits — the model_saturation_check enforced raise, the priority_model enforced raise, and the fail-closed unknown-descriptor branch — now attribute the 429 to the actual provider. Falls back to llm_provider='litellm_proxy' when the model can't be resolved. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on batch-rate-limit 429s batch_rate_limiter._raise_rate_limit_error now takes a requested_model kwarg threaded from data['model'] in _check_and_increment_batch_counters. The batch-creation 429 is what gets raised when the input file's tokens/requests count would push the per-key TPM/RPM window over its limit. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(proxy/hooks): populate llm_provider on budget/iterations 429s Final batch of internal raise sites — the user/session-budget and max-iterations hooks. Same pattern: resolve data['model'] once at raise time, attach to ProxyHTTPRateLimitError so Prometheus and observability callbacks can attribute the 429. Hooks updated: * max_budget_limiter (per-user max_budget exceeded) * max_iterations_limiter (per-session agent iteration cap) * max_budget_per_session_limiter (per-session dollar cap) All three fall back to llm_provider='litellm_proxy' when data['model'] is missing or unparseable. Drops the now-unused HTTPException import from each module. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(proxy/hooks): pin provider field on internal rate-limit 429s Regression coverage for the 'provider field missing' bug across every proxy-side rate-limit hook + the helper layer: * ProxyHTTPRateLimitError class shape (HTTPException + RateLimitError, dict-detail stringification, None-provider normalization). * resolve_llm_provider_for_rate_limit happy paths (gpt-4o-mini, anthropic/..., bedrock/...) plus all three fallback branches (None, '', unknown name) plus a 'get_llm_provider raises' case that asserts we swallow the secondary exception. * For each limiter (parallel v1/v3, dynamic v1/v3, batch, max_budget, max_iterations, max_budget_per_session): assert the raised exception is a RateLimitError carrying the resolved model + llm_provider, and a sibling test that asserts the fallback path returns 'litellm_proxy' without leaking a second exception. * Two PrometheusLogger._get_exception_class_name pins so the Prometheus failure metric label flips from 'HTTPException' to 'Openai.ProxyHTTPRateLimitError' (or 'Litellm_proxy.*' on fallback) — that's what dashboards consume. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * perf(proxy/hooks): defer provider resolution to over-limit branches * fix: use error_message in raise_rate_limit_error to avoid literal 'None' in detail * Consolidate rate_limiter_utils imports in dynamic_rate_limiter * fix(proxy): set num_retries/max_retries on ProxyHTTPRateLimitError ProxyHTTPRateLimitError inherits from RateLimitError but did not call RateLimitError.__init__, so num_retries/max_retries were never set. When Starlette's HTTPException lacks __str__, MRO falls through to RateLimitError.__str__, which unconditionally reads these attributes and raises AttributeError during logging/traceback formatting. Initialize them to None defensively. * fix(mypy): silence base-class status_code conflict on ProxyHTTPRateLimitError HTTPException declares 'status_code: int' while openai.RateLimitError (via APIStatusError) declares 'status_code: Literal[429] = 429'. Mypy flags the multi-base override as [misc] in CI lint. The runtime semantics are fine (we set self.status_code in __init__), so silence the class-level annotation conflict with a targeted ignore. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix: annotate batch limiter _raise_rate_limit_error as NoReturn * feat(prometheus): rate-limit category/type labels + exception_class back-compat (follow-up to BerriAI#27687) (BerriAI#27706) * feat(prometheus): add rate_limit_category and rate_limit_type labels Adds two new labels to litellm_proxy_failed_requests_metric so dashboards can split 429s by rate-limit source (vendor vs. litellm-internal) and by the dimension that was exceeded (requests/tokens/concurrent_requests/ budget/max_iterations) without parsing free-text error messages. Closes the Prometheus side of LIT-2718. The unified RateLimitError.category and .rate_limit_type fields landed in PR BerriAI#27687 but were only surfaced on StandardLoggingPayload (custom-callback channel); this exposes them on the metric label set as well. Both labels are populated only when the underlying exception is a litellm.RateLimitError; non-rate-limit failures keep them empty. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * feat(prometheus): populate rate-limit labels + preserve exception_class back-compat Two coupled changes in the Prometheus integration: 1. async_post_call_failure_hook now extracts the new RateLimitError .category / .rate_limit_type fields (added in PR BerriAI#27687) via a _extract_rate_limit_labels helper and forwards them through UserAPIKeyLabelValues onto litellm_proxy_failed_requests_metric. Empty for non-rate-limit failures. 2. _get_exception_class_name special-cases ProxyRateLimitError and keeps emitting 'HTTPException' for the exception_class label. Without this shim, ProxyRateLimitError (which multi-inherits from HTTPException + RateLimitError) would silently flip the label from 'HTTPException' (the historical value for proxy-side 429s) to 'ProxyRateLimitError', breaking existing dashboards / alerts that key off exception_class='HTTPException'. Distinguishing vendor vs. litellm 429s is now the job of the new rate_limit_category label. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(prometheus): cover rate-limit labels and exception_class back-compat Adds 19 tests across: - enum / label-list registration - _extract_rate_limit_labels for vendor RateLimitError, ProxyRateLimitError, non-rate-limit and None inputs (incl. parametrized over every RateLimitErrorCategory x RateLimitType combo) - _get_exception_class_name back-compat: ProxyRateLimitError keeps the legacy 'HTTPException' string while vendor RateLimitError keeps the historical 'Provider.ClassName' format - end-to-end through async_post_call_failure_hook with both ProxyRateLimitError and vendor RateLimitError, asserting both new labels populate and exception_class stays back-compat Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(prometheus): tolerate missing fastapi in lazy ProxyRateLimitError import Address greptile feedback: - async_post_call_failure_hook docstring: drop the stale labelnames listing and reference PrometheusMetricLabels.litellm_proxy_failed_requests_metric as the source of truth so the doc cannot drift from the actual labelset. - _get_exception_class_name: guard the lazy ProxyRateLimitError import with ImportError so router-side fallback callsites don't blow up in non-proxy installs that don't have fastapi (a transitive dep of proxy.common_utils.proxy_rate_limit_error). Behavior is unchanged when fastapi is available. Also fix the existing enterprise callback test that asserted the old labelset on litellm_proxy_failed_requests_metric — it now expects the new rate_limit_category / rate_limit_type labels populated for vendor 429s. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(bugbot): simplify rate-limit label coercion + guard None detail - prometheus.py _extract_rate_limit_labels: RateLimitError.__init__ already normalizes category/rate_limit_type to plain str, so the getattr(.value) + isinstance dance was dead code. Reduce to str(value) if not None. - proxy_rate_limit_error.py _coerce_message: short-circuit None to '' instead of falling through to str(None) = 'None', which produced the literal message 'litellm.RateLimitError: None'. * fix(rate-limit): surface unified category/type fields on BudgetExceededError The most common budget cap (virtual-key max_budget enforcement in auth_checks.py) raises litellm.BudgetExceededError, a bare Exception subclass that bypassed the unified rate-limit error class introduced by PR BerriAI#27687. Custom callbacks reading StandardLoggingPayload.error_information saw category=None and rate_limit_type=None for these 429s, missing the most common budget case (team / org / end-user budgets all hit the same code path). Surface the fields off BudgetExceededError as plain attributes: - category = RateLimitErrorCategory.LITELLM_RATE_LIMIT - rate_limit_type = RateLimitType.BUDGET - llm_provider = "" (or caller-supplied) Switch get_error_information and _extract_rate_limit_labels from isinstance(RateLimitError) gating to duck-typed attribute reads, guarded by membership in the rate-limit enums so unrelated third-party exceptions exposing a .category attribute can't leak garbage values into the payload. This is strictly additive: BudgetExceededError keeps its bare-Exception base class, so `except BudgetExceededError:` handlers keep firing and `except RateLimitError:` does not start catching budget errors. * fix(rate-limit): validate enum membership at duck-typed read sites + enrich BudgetExceededError llm_provider Two follow-ups uncovered during the second QA pass on PR BerriAI#27687: 1. Guard third-party `.category` / `.rate_limit_type` attribute leakage. The duck-typed read in `get_error_information` and `_extract_rate_limit_labels` would forward any string attribute named `category` / `rate_limit_type` on an unrelated third-party exception into the StandardLoggingPayload and Prometheus labels — silently mislabeling custom-callback payloads and blowing out Prometheus label cardinality. Add `validate_rate_limit_category` / `validate_rate_limit_type` helpers that gate on the documented enum value sets; non-matching values are dropped to None. 2. Enrich BudgetExceededError.llm_provider from request_data. Budget checks live in tenant-scoped helpers (key / team / org / tag / end-user / project) that don't see the request model, so the BudgetExceededError they raise carried llm_provider="" — leaving custom-metrics consumers without provider attribution for the most common 429 case. Resolve it once at the central UserAPIKeyAuthExceptionHandler seam, before post_call_failure_hook fires, so the StandardLoggingPayload the callback sees has the same provider attribution as RPM/TPM 429s. Regression tests pin both: 4 leakage tests + 4 enrichment tests. The leakage tests would fail under the pre-validation version of either read site; the enrichment tests would fail if the handler skipped the resolver call. * fix(rate-limit): resolve router model_name aliases to real provider (BerriAI#27914) * fix(rate-limit): resolve router model_name aliases to real provider For nearly every real LiteLLM proxy deployment the request model is a router model_name alias (e.g. 'tpm-locked' -> litellm_params.model: openai/gpt-4o-mini), and 'litellm.get_llm_provider' doesn't know about router aliases — it raises 'LLMProviderNotProvidedError'. The resolver then fell through to the defensive 'litellm_proxy' fallback, so the 'llm_provider' field this PR adds was effectively always 'litellm_proxy' in the field, defeating its purpose for the most common proxy configuration. Add a router-alias fallback step: when 'get_llm_provider' raises, scan the active 'llm_router.model_list' for a deployment whose 'model_name' matches the request model and resolve from its 'litellm_params.model' instead. If multiple deployments share the same alias (load-balancing case) the first one wins — every deployment under one alias should agree on provider in any sensible config, and 'first' is deterministic so the Prometheus label stays stable. Defensive throughout: an uninitialized router, a malformed deployment, a 'litellm_params.model' that itself fails 'get_llm_provider' — every branch falls through to the existing 'litellm_proxy' fallback rather than letting a secondary exception escape and mask the rate-limit error we're trying to surface. Tests: - test_router_alias_resolves_to_underlying_provider: alias 'tpm-locked' -> 'openai/gpt-4o-mini' produces provider='openai', model='gpt-4o-mini'. - test_router_alias_with_multiple_deployments_uses_first. - test_router_alias_unknown_falls_back. - test_router_alias_with_malformed_deployment_falls_back. - Existing fallback test updated to also stub 'litellm.proxy.proxy_server.llm_router' so it exercises the full 'no resolution anywhere' path. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(rate-limit): harden router alias resolver + test isolation - Wrap _resolve_provider_from_router_alias loop in top-level try/except so a non-iterable model_list / unexpected deployment shape can't escape and mask the 429 with a 500. - Type-check litellm_params before .get() to handle non-dict truthy values. - Patch llm_router=None in the parametrized fallback test so a router left by another test in the session can't redirect the unknown-model path. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(bugbot): preserve "BudgetExceededError" Prometheus label Adding llm_provider to BudgetExceededError (so callbacks get provider attribution from StandardLoggingPayload) made the provider-prefix step in _get_exception_class_name silently flip the label from "BudgetExceededError" to e.g. "Openai.BudgetExceededError", breaking dashboards keyed on the historical value. Short-circuit BudgetExceededError in _get_exception_class_name the same way ProxyRateLimitError already is. Provider/category attribution still lands on the new rate_limit_category / rate_limit_type labels. * test: fix invalid 'rpm' rate_limit_type in v3 limiter test mocks The v3 rate limiter only emits 'requests', 'tokens', or 'max_parallel_requests'. Using 'rpm' caused map_v3_rate_limit_type to return None, leaving the expected RateLimitType.REQUESTS untested. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(bugbot): hoist provider resolver + opt-in prom rate-limit labels - dynamic_rate_limiter.py: hoist resolve_llm_provider_for_rate_limit above the TPM/RPM if/elif so the lookup runs once per request, matching the pattern in dynamic_rate_limiter_v3.py. - prometheus.py: gate the new rate_limit_category / rate_limit_type labels on litellm_proxy_failed_requests_metric behind litellm.prometheus_emit_rate_limit_labels (default False). Mirrors the existing prometheus_emit_stream_label opt-in. Preserves the metric's pre-unification label set so existing dashboards / recording rules keep matching after upgrade; operators can enable the new labels once downstream consumers include them. - Tests updated: default-off back-compat case, opt-in path enables the flag before asserting label presence. * fix: stabilize prometheus label sets and drop redundant model normalization - Cache PrometheusLogger.get_labels_for_metric per metric_name so that the label set used to construct counters at __init__ time stays in sync with the label set used at increment time, even if module-level toggles like prometheus_emit_rate_limit_labels or prometheus_emit_stream_label are flipped at runtime. Without this, toggling these flags after the logger was created would cause ValueError from prometheus_client because the runtime labels would not match the counter's declared labelnames. - Drop redundant 'model or ""' guard in ProxyRateLimitError.__init__ where model is already normalized one step earlier. Co-authored-by: Yassin Kortam <yassin@berri.ai> * perf(dynamic_rate_limiter): only resolve provider when rate limit hit Co-authored-by: Yassin Kortam <yassin@berri.ai> * test(prometheus): clear cached metric labels after toggling rate-limit flag The PrometheusLogger caches each metric's label set at construction time so that labels used at counter.labels(...) time stay consistent with the labels the metric was registered with. The enterprise async_post_call_failure_hook test toggles litellm.prometheus_emit_rate_limit_labels = True AFTER the fixture has already built the logger, so without invalidating the cache the rate_limit_category / rate_limit_type labels never reach the mocked counter and the assert_called_once_with check fails. Co-authored-by: Yassin Kortam <yassin@berri.ai> * test: fix CI failures from prom label cache + flaky time-window assertion PrometheusLogger.get_labels_for_metric now caches the per-metric label set at first read so the labels passed to counter.labels(...) stay in lock step with the labels the counter was registered with. This broke two existing test patterns: - test_prometheus_labels.py: tests bind the real method onto a MagicMock, but MagicMock auto-creates a Mock for _cached_metric_labels whose .get(...) returns a truthy Mock — treated as a populated cache and returned as the label set, producing empty filtered labels and KeyError on labels["requested_model"] / ["route"]. Seed real {} containers for _cached_metric_labels and label_filters before binding. - test_prometheus_logging_callbacks.py::test_set_team_budget_metrics_with_custom_labels: the fixture builds the logger before the test monkeypatches litellm.custom_prometheus_metadata_labels, so the cached label set never picks up the new metadata labels. Clear the cache after the monkeypatch (same pattern already used for the rate-limit toggle in test_async_post_call_failure_hook). UI: view_logs/index.test.tsx "Last Minute" window assertion is off by one at the minute boundary. start_date is floored to the minute, so the dropped sub-minute fraction can push the truncated-seconds diff up to (minMinutes+1)*60 exactly when the click lands near a minute rollover. Switch the upper bound to toBeLessThanOrEqual. * feat(otel-v2): surface rate_limit_category + rate_limit_type on failed LLM-call spans PR BerriAI#28909 introduced the typed v2 OTel engine that builds spans from StandardLoggingPayload, with SpanError carrying error_type + message and the genai mapper stamping error.type onto every failed LLM-call span. This PR's earlier commits added error_rate_limit_category and error_rate_limit_type to the same StandardLoggingPayload.error_information the v2 engine reads — but neither field reached a span attribute, so v2 OTel traces stayed opaque about *why* a 429 fired (vendor vs litellm, RPM vs TPM vs concurrent vs budget vs max_iterations) even after the custom-callback and prometheus surfaces gained that decomposition. Three coupled changes: 1. semconv.py: add LiteLLM.ERROR_RATE_LIMIT_CATEGORY / LiteLLM.ERROR_RATE_LIMIT_TYPE under the litellm.* vendor namespace (no GenAI semconv equivalent exists for who-rate-limited / which-dimension). 2. payloads.py: extend SpanError with rate_limit_category + rate_limit_type, populated by _parse_error() from the same error_information.error_rate_limit_* fields the custom-callback channel and prometheus rate_limit_category / rate_limit_type labels read. Single source of truth across all three observability surfaces. 3. mappers/genai.py: stamp the two attributes on the LLM-call span when present. drop_none guarantees they stay absent (not 'None') for non-rate-limit failures so trace consumers can read them unconditionally. Three regression tests in test_otel_v2_emitter.py pin: a vendor / litellm-internal RateLimitError lands category=litellm_rate_limit + rate_limit_type=requests on the span; a BudgetExceededError lands rate_limit_type=budget; a non-rate-limit failure (BadRequestError) keeps the rate_limit_* attributes absent. Mutation-tested against reverting either the SpanError extension or the _parse_error read site — both new tests fail under either mutation. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test: align prometheus user-budget + logs quick-select tests with merged code The merge into this branch left two test patterns out of step with the code they exercise. test_set_user_budget_metrics_includes_user_email_and_alias_labels_when_opted_in flipped litellm.prometheus_user_budget_label_include_email_alias after the fixture had already built the PrometheusLogger. get_labels_for_metric now snapshots each metric's label set at construction time, so the runtime flip no longer reached the cached labels. Enable the flag before constructing the logger, matching how the proxy applies config at startup. view_logs/index.test.tsx referenced uiSpendLogsCall and moment without importing them, and the merged index.tsx now fetches through useLogFilterLogic (the hook the file stubs out) rather than calling uiSpendLogsCall directly. Add the imports and restore the real hook for the Quick Select window assertions so the call is actually observed. * refactor(otel/v2): drop rate-limit decomposition from the LLM-call span Proxy-side rate limits (litellm_rate_limit, budget, max_iterations) are rejected at the gate before any upstream call, so async_post_call_failure_hook tags the synthetic failure log with LITELLM_LOGGING_NO_UPSTREAM_LLM_CALL and the v2 OTel logger never opens an LLM-call span for them; the litellm.error.rate_limit_category / litellm.error.rate_limit_type attributes were dead for exactly the cases they were meant to surface. The only failure that does open an LLM-call span carrying a RateLimitError is a vendor 429, where rate_limit_type is always None and the category just restates error.type=RateLimitError. The decomposition still reaches downstream consumers through StandardLoggingPayload.error_information.error_rate_limit_* and the prometheus rate_limit_category / rate_limit_type labels, both unchanged. Removes the SpanError fields, the _parse_error reads, the genai mapper attributes, the semconv keys, and the three span tests that asserted a scenario that never reaches the mapper in production. * fix(batch_rate_limiter): map max_parallel_requests to concurrent_requests * refactor(prometheus): drop transitive fastapi import from _get_exception_class_name Read the legacy exception_class label from a prometheus_exception_class_name marker on ProxyRateLimitError instead of importing the proxy module, keeping the integrations layer free of a transitive fastapi dependency. * chore(ui): sync schema.d.ts with unified rate-limit error spec The ProxyRateLimitError docstring flows into the proxy OpenAPI spec's 429 response description, so the generated dashboard types were out of sync. Regenerated via npm run gen:api (Check UI API Types Sync). --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> Co-authored-by: Yassin Kortam <yassin@berri.ai>

yassin-berriai mentioned this pull request May 26, 2026

docs(otel): RFC for typed semconv-aligned OpenTelemetry rewrite #28889

Closed

2 tasks

yassin-berriai force-pushed the litellm_fix/kind-ptolemy-CNozr branch from c2c090b to f6a2551 Compare May 28, 2026 20:06

yassin-berriai changed the title ~~feat(otel): typed semconv-aligned OpenTelemetry instrumentation (RFC + Phase 0-1)~~ feat(otel): typed semconv-aligned OpenTelemetry instrumentation May 28, 2026

yassin-berriai force-pushed the litellm_fix/kind-ptolemy-CNozr branch 3 times, most recently from c528458 to aa9a357 Compare May 29, 2026 01:02

yassin-berriai marked this pull request as ready for review May 29, 2026 01:03

yassin-berriai requested a review from a team May 29, 2026 01:03

greptile-apps Bot reviewed May 29, 2026

View reviewed changes

veria-ai Bot reviewed May 29, 2026

View reviewed changes

Comment thread litellm/integrations/otel/mappers/openinference.py

yassin-berriai force-pushed the litellm_fix/kind-ptolemy-CNozr branch from 40e0119 to a76b513 Compare May 29, 2026 01:32

veria-ai Bot reviewed May 29, 2026

View reviewed changes

Comment thread litellm/integrations/otel/plumbing/routing.py

yassin-berriai force-pushed the litellm_fix/kind-ptolemy-CNozr branch from a76b513 to 0c81162 Compare May 29, 2026 02:11

veria-ai Bot reviewed May 29, 2026

View reviewed changes

yassin-berriai force-pushed the litellm_fix/kind-ptolemy-CNozr branch from 0c81162 to 5e77678 Compare May 29, 2026 02:27

BerriAI deleted a comment from greptile-apps Bot May 29, 2026

yassin-berriai marked this pull request as draft May 29, 2026 02:54

yassin-berriai force-pushed the litellm_fix/kind-ptolemy-CNozr branch 2 times, most recently from 948203e to ab489eb Compare May 29, 2026 03:24

yassin-berriai mentioned this pull request May 29, 2026

fix(otel): configurable baggage, pass-through + service spans, typed guardrails #29263

Merged

yuneng-berri approved these changes May 30, 2026

View reviewed changes

yassin-berriai force-pushed the litellm_fix/kind-ptolemy-CNozr branch 2 times, most recently from e523391 to 1e4ccf9 Compare May 30, 2026 05:56

yassin-berriai enabled auto-merge (squash) May 30, 2026 06:04

yassin-berriai mentioned this pull request May 30, 2026

docs: add OpenTelemetry v2 tracing guide BerriAI/litellm-docs#260

Merged

feat(otel): typed semconv-aligned OpenTelemetry instrumentation

f485deb

yassin-berriai force-pushed the litellm_fix/kind-ptolemy-CNozr branch from 1e4ccf9 to f485deb Compare May 30, 2026 06:09

ryan-crabbe-berri approved these changes May 30, 2026

View reviewed changes

yassin-berriai merged commit d82eb33 into litellm_internal_staging May 30, 2026
101 of 107 checks passed

yassin-berriai mentioned this pull request May 30, 2026

feat(otel): add team_metadata, http.route, and model names to inference spans #29319

Merged

4 tasks

greptile-apps Bot mentioned this pull request May 30, 2026

fix(ci): make litellm_internal_staging green (logging test + Bedrock Opus 4.7 self-heal) #29344

Merged

7 tasks

1w2w3y mentioned this pull request Jun 1, 2026

[Feature]: Bump pinned OpenTelemetry SDK from 1.28.0 to the 1.40 line (or relax to a range) #29407

Open

cursor Bot mentioned this pull request Jun 3, 2026

feat: standardize rate limit errors with category, rate_limit_type, model, and llm_provider fields #27687

Merged

4 tasks

choogeboom mentioned this pull request Jun 10, 2026

feat(otel): emit v2 cost breakdown + stamp tracer scope version #30156

Merged

4 tasks

fzowl pushed a commit to fzowl/litellm that referenced this pull request Jun 24, 2026

feat(otel): typed semconv-aligned OpenTelemetry instrumentation (Berr…

de9552c

…iAI#28909)

Uh oh!

Uh oh!

Conversation

yassin-berriai commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How it works

Which service calls become spans

Architecture (litellm/integrations/otel/)

Excluded routes

Content capture

Review feedback addressed

Test plan

Uh oh!

CLAassistant commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

yassin-berriai commented May 29, 2026

Uh oh!

greptile-apps Bot commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

veria-ai Bot commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR overview

Open issues (1)

Uh oh!

Uh oh!

yassin-berriai commented May 29, 2026

Uh oh!

veria-ai Bot May 29, 2026

Choose a reason for hiding this comment

Uh oh!

yassin-berriai commented May 29, 2026

Uh oh!

ryan-crabbe-berri left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yassin-berriai commented May 26, 2026 •

edited

Loading

Architecture (`litellm/integrations/otel/`)

CLAassistant commented May 26, 2026 •

edited

Loading

codecov Bot commented May 26, 2026 •

edited

Loading

greptile-apps Bot commented May 29, 2026 •

edited

Loading

veria-ai Bot commented May 29, 2026 •

edited

Loading