chore(ci): promote internal staging to main - #32400
Merged
Merged
Conversation
When pre-call hooks (parallel_request_limiter, dynamic_rate_limiter_v3) reject a request with ProxyRateLimitError, the router's fallback logic was never reached because the exception was raised before route_request was called. Add _pre_call_with_fallbacks that catches ProxyRateLimitError, resolves configured fallbacks (key-level router_settings -> router-level), and retries with each fallback model in order. If all fallbacks are also rate-limited, the original error is re-raised.
…ack loop Addresses Greptile review feedback: wrap the fallback loop in try/except BaseException to always restore self.data['model'] to the original value when a non-ProxyRateLimitError exception escapes a fallback attempt. Add regression test for this edge case
The typed cache-settings form already renders a Redis URL and a Database Index field, but the backend never defined them, so GET /cache/settings could not round-trip a saved value into the form and the "URL takes precedence over Host/Port/Password/Database Index" help text the UI shows was not actually enforced anywhere. Add the url and db entries to CACHE_SETTINGS_FIELDS so the endpoint knows about them, and add _resolve_cache_url_precedence: when a non-empty url is present it wins and the discrete host/port/db/password fields are dropped before the settings are tested or persisted, matching how litellm._redis resolves the connection at runtime (redis.Redis.from_url ignores them). Cluster mode is exempt because it authenticates via the discrete fields rather than a url. Both test and save paths go through the resolver so the stored config is unambiguous. This finishes LIT-3996: operators can now isolate the cache into a logical database (e.g. redis://host:6379/1) entirely from the Admin UI instead of hardcoding REDIS_URL in the environment.
…der url precedence Two follow-ups from review of the url/db work. A Redis/Valkey url can embed a password (redis://:secret@host:6379/1), but _CACHE_SENSITIVE_FIELDS only masked the discrete password and sentinel_password, so a stored password-bearing url came back in plaintext from every GET /cache/settings. Add url to the masked set so it gets the same masked-on-read treatment as password. The url-precedence resolver dropped host/port/db/password but not username, even though a url can encode a username too (redis://user:pass@host). Left in, the discrete username rode along and could contradict the url. Add username to the overridden set and update the Redis URL help text to list it among the fields url takes precedence over. Tests: GET masks a password-bearing url (secret never returned verbatim) while a non-credential field is untouched, and the resolver drops a discrete username when a url is present.
) * feat(mcp): add entra_obo profile to the token_exchange (OBO) arm Microsoft Entra On-Behalf-Of uses the RFC 7523 jwt-bearer grant rather than RFC 8693, so the existing token_exchange arm cannot mint tokens against Entra. This adds an entra_obo profile on TokenExchangeConfig that switches the request form to Entra's jwt-bearer OBO dialect (the inbound token as assertion, the target resource carried in scope, and requested_token_use=on_behalf_of), while reusing the shared caching, single-flight, TTL, and fail-closed machinery. The exchanger is renamed from Rfc8693TokenExchanger to OboTokenExchanger since it now serves both dialects The profile is threaded through the config-load path, the DB credentials blob, and the MCPCredentials request schema, so an operator can select it from YAML or the management API. It rides in the existing credentials JSON blob, so there is no new auth_type and no DB migration Resolves LIT-4163 * feat(mcp): propagate the Entra Conditional Access step-up challenge on the OBO 401 An entra_obo exchange that the IdP rejects for Conditional Access returns a 4xx with error=interaction_required and a claims blob the client must satisfy to step up. The arm dropped both and emitted a static RFC 9728 challenge, so a CA-protected Entra upstream was unreachable through the gateway. The provider now reads the RFC 6749 error code and the claims string off the rejection body (error_description is still never carried; it can leak IdP internals), threads them through SubjectTokenRejected -> CredError.unauthorized, and the challenge builder folds them into WWW-Authenticate: the machine error only when it is a plain OAuth token (guards against header injection from a hostile body) and the claims base64-encoded in a claims parameter, the convention MSAL-family clients decode. With neither field the header is byte-identical to the static challenge. The multi-server aggregate still absorbs a step-up 401 to an empty listing; only single-server routes surface it * fix(mcp): use error=insufficient_claims for the Entra step-up challenge Per Microsoft's claims-challenge format, a WWW-Authenticate carrying a claims challenge must set error=insufficient_claims (the value MSAL-family clients key on to recognize the challenge and replay the claims), not the raw token-endpoint code. The challenge now sets insufficient_claims whenever a claims blob is present and keeps invalid_token otherwise, with a step-up-accurate error_description in the claims case. The presence of claims now drives the error value, so the raw oauth_error no longer needs threading from the provider through CredError to the edge; that plumbing is removed (the provider still reads the error code for its gateway-fault classification). Both the error value and the base64 claims are fixed-alphabet, so nothing from the IdP body reaches the header unescaped. Cross-checked field-by-field against Microsoft Learn; a bogus jwt-bearer OBO POST to the real login.microsoftonline.com/common endpoint confirmed Entra recognizes the grant and returns the error shape the parser reads. Follow-up: also emit authorization_uri alongside resource_metadata for strict non-MCP MSAL clients (RFC 9728 resource_metadata already serves MCP) * fix(mcp): filter blank scopes on the config-load path so entra_obo fails closed The DB-build path already runs YAML/DB scopes through _extract_scopes (which drops blanks), but the config-load path read server_config["scopes"] raw. A YAML `scopes: [""]` therefore reached the exchanger as a ("",) tuple: non-empty, so the entra_obo `not config.scopes` precondition skipped its fail-closed misconfigured path and the form builder POSTed an empty scope to the IdP instead of failing before any network call. Config-load now filters blanks the same way, so an all-blank list normalizes to None and the precondition fails closed. Regression test loads a blank-scope entra_obo server and asserts the exchange returns misconfigured without POSTing
…ore comments (#32152) * fix: zero out crash-class basedpyright rules across litellm/ * feat(lint): add LIT009 banning inert type: ignore comments * docs: require bracketed rule and reason on every suppression * chore(lint): ratchet budgets down and zero crash-class pyright limits * fix: narrow auto router routelayer through a local before calling * test: add regression tests for crash-class fixes * fix: drop dead AZURE_AD_TOKEN lookups and word-bound the type-ignore regex
…"allow" (#32158) * fix(headroom guardrail): log real token/compression stats instead of "allow" The headroom guardrail fetched tokens_before/tokens_after/compression_ratio from Headroom's /v1/compress response but only surfaced them via a debug-level log line, so spend_logs.guardrail_information showed guardrail_response: "allow" with no way to tell whether compression actually ran or by how much. _call_compress now returns the token/compression stats alongside the compressed messages and success flag, and apply_guardrail logs them via add_standard_logging_guardrail_information_to_request_data when compression succeeds. Raw message content is intentionally excluded from what's logged - only token counts, compression ratio, and applied transform names. * fix(ci): apply ruff format to headroom.py * fix(review): remove comment per repo's no-comments-unless-asked convention Addresses codex review feedback - CLAUDE.md says not to add comments unless explicitly asked; the sensitive-logging guarantee is already expressed by the stats dict only pulling specific keys, not messages.
…ng /v1/messages responses (#32160) * fix(anthropic_messages): forward provider response headers on streaming /v1/messages responses * fix(anthropic_messages): forward aclose to inner streaming iterator * fix(anthropic_messages): forward aclose through the streaming response wrapper The proxy's streaming cleanup closes the handler's return value via hasattr(response, "aclose"); the new wrapper hid the upstream generator's aclose, so provider connections could linger on client disconnect. The wrapper now delegates aclose to the wrapped stream and AgenticAnthropicStreamingIterator closes its inner and follow-up streams. Also adds test coverage for the agentic streaming branch --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
…ithout message_stop (#32159) * fix(bedrock): emit SSE error event when invoke Messages stream ends without message_stop * fix(bedrock): tighten stream-terminal detection to avoid false positives and double errors The bytes branch of _is_message_stop_chunk used a plain substring match, so a content_block_delta whose partial_json contained the literal text message_stop would look like a real terminal event and suppress the synthetic incomplete-stream error. Match the SSE event header line instead. Also treat a provider-emitted error event as terminal so a stream that ends with an upstream error is not followed by a second, contradictory synthetic incomplete-stream error. * test(bedrock): lock in that the synthetic truncation error event is excluded from logged chunks --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
…ests (#32016) * test(e2e): migrate access-control and inference-endpoint regression tests Move the access-control and non-chat inference-endpoint cases from litellm-regression-tests onto the shared e2e harness so a regression in either fails here first access_control/ asserts the gateway's authorization and error-shape contract: a key limited to one model is denied 403 (key_model_access_denied) when it calls another, a key scoped to allowed_routes=["llm_api_routes"] is forbidden 403 from a management route, and an unknown model is rejected 400 before any provider is called. The source asserted 401 for the disallowed-model case against an older proxy; the live contract is now a 403, so the guard tracks current behavior llm_translation/ gains one file per non-chat inference endpoint (/v1/responses, /v1/messages, /embeddings, /v1/rerank, /v1/audio/speech, /v1/images/generations). Each test registers the deployment it needs through /model/new, drives real provider traffic, asserts the parsed body carries real content instead of just a 200, then deletes the model on teardown, so nothing is hardcoded into the gateway config * Update endpoints_client.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
…es, batches, prometheus, and langfuse eviction (#32165) * fix(e2e): define SpendTagsResponse/TagSpend so spend suite collects spend_tracking/spend_e2e_client.py imported SpendTagsResponse and TagSpend from models, but neither was ever defined, so importing the client raised ImportError and pytest aborted collection for the whole e2e session. The tag-spend tests had never run. Model /spend/tags as it actually answers: a bare array of per-tag aggregates, so SpendTagsResponse is a RootModel[list[TagSpend]] like the existing SpendLogs. spend_by_tags read a nonexistent spend_per_tag field that also wouldn't match the array shape; it now reads .root, matching how spend_logs consumes its RootModel. * test(e2e): close coverage gaps across chat/responses, provider features, batches, prometheus, and langfuse eviction Adds regression nets and gap-surfacing tests: A1 (llm_translation/test_deepseek_reasoning_e2e.py): control case proves the DeepSeek reasoner returns reasoning_content; two xfail(strict) cases document that reasoning_effort='none' and thinking type='disabled' are silently dropped (LIT-3686 / GH #27453) A2 (llm_translation/test_chat_completions_regression_e2e.py and test_responses_e2e.py): parametrized regression net asserting real completion content, not just a 200, across the configured providers for /chat/completions and /responses (GH #28991) A3 (llm_translation/test_provider_features_e2e.py): asserts service_tier is honored and prompt-cache read tokens grow on a repeated cacheable prefix A4 (batches/test_batches_e2e.py): mints a rate-limited key so the batch pre-call rate limiter runs, then asserts no unattributed spend row is left behind by the internal input-file retrieval (LIT-3266) A5 (logging/test_prometheus_cardinality_e2e.py): drives one chat per distinct key_alias and asserts each alias gets its own labeled series on /metrics A6 (test_litellm/.../specialty_caches/test_dynamic_logging_cache.py): xfail(strict) regression proving eviction must not close an httpx client still held by an in-flight caller (LIT-3221 / GH #13034) Extends tests/e2e/models.py with the typed request and response fields these tests read (reasoning_effort, thinking, service_tier, key_alias, cache usage fields, spend-log api_key) Co-authored-by: Cursor <cursoragent@cursor.com> * test(e2e): drop unused litellm-regression-tests submodule The e2e suite migrated the regression cases into this repo; nothing imports the submodule at runtime (only a provenance comment references it), so the .gitmodules entry and gitlink pointing at a personal repo would just make upstream CI init a submodule it never uses. Remove both to keep the change test-only. * test(e2e): drop A6 langfuse-eviction xfail; keep PR to live e2e coverage The dynamic_logging_cache strict-xfail documented an unfixed shared-httpx-client close-on-eviction bug (LIT-3221 / GH #13034). That is a non-trivial fix (thread cleanup vs shared client teardown) and belongs in its own PR, not this e2e coverage PR, so revert the file to its base state. --------- Co-authored-by: Cursor <cursoragent@cursor.com>
* chore: add latest model rule to CLAUDE.md * chore: correct grammar mistake * chore: make the rule more concise * chore: replace rule instead * chore: revise wording to override memories, etc. * chore: slightly adjust wording to be more precise
The test posted /config/update, slept a fixed 20s, then fired a single chat request with no readiness check or retry. When the single-process proxy was momentarily not accepting connections in that window, the request failed with a bare openai.APIConnectionError and took the whole job down, since the suite runs against one shared container with pytest -x Gate the chat request behind a /health/liveliness poll, retry it on connection errors only so real HTTP errors and the Langfuse assertion still fail the test, close the previously leaked aiohttp session, and target 127.0.0.1 instead of the 0.0.0.0 bind address. In CI, give the proxy container --restart on-failure so an intermittent crash recovers instead of leaving the port dead for the rest of the run
…itellm_/suspicious-jennings-5b6ef7
* ci: gate CircleCI jobs on changed paths Every CircleCI job used to run on every PR. Now each job starts with a lightweight `skip_if_unrelated_changes` step that inspects the PR diff and halts the job as successful when nothing relevant changed. Docs-only PRs (*.md, *.mdx, docs/) run nothing, UI-only PRs (ui/) run just the frontend jobs, and any backend change still runs both the backend and frontend jobs. The decision logic lives in .circleci/scripts/classify_changes.sh (pure, reads the changed-file list on stdin) so it can be unit tested, while path_filter.sh handles the git plumbing and fails open (runs the job) on any uncertainty such as a missing merge base or a non-PR pipeline. Halting via `circleci-agent step halt` keeps the job green, so required status checks are never left pending. The Windows smoke job is intentionally left ungated to avoid cross-platform shell fragility * fix(ci): keep path filter fail-open when classifier errors Guard the classify_changes.sh invocation with `|| run_full` so a broken or non-zero classifier runs the job instead of falling through to a silent halt, and mark the advisory logging pipe best-effort with `|| true`. Add path_filter.sh regression tests covering the docs-only halt, backend run, non-PR fail-open, and classifier-failure fail-open paths
…registration fix(e2e): register batch + rust OCR deployments via /model/new
* fix: pass websearch tool params * fix: load db websearch tool params * fix: merge search tools in proxy * fix: satisfy websearch lint budget * fix: enforce websearch tool auth * fix: preserve search tools on empty sync * chore: rerun circleci
The group_by=team branch queried spend for every team in the date range regardless of team_id; team_id was only honored in the separate branch that also required a customer_id. Route the team report through a new get_spend_by_team helper (sibling of get_spend_by_team_and_customer) that binds team_id as an optional $3 predicate, so a provided team_id narrows the result to that team and a null team_id still returns every team.
…itellm_/suspicious-yonath-c3c703
…b6ef7 test: de-flake langfuse callbacks-in-db e2e test
* feat(router): add separate ITPM/OTPM deployment rate limits Support input/output tokens per minute on deployments via enforce_model_rate_limits, with reservation, reconciliation, refund on failure, and rate-limit headers. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(router): keep ITPM/OTPM diff minimal in router.py Drop unrelated Black reformatting from router.py and types/router.py so the PR only contains functional ITPM/OTPM changes. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): make ITPM/OTPM limits separate and atomic Address Greptile review on separate ITPM/OTPM deployment rate limits. - OTPM is now reserved atomically pre-call with rollback, matching the ITPM path, so concurrent requests can no longer overshoot the configured output limit before reconciliation - ITPM counts input tokens only; it no longer accumulates completion tokens, so the input-token limit and x-ratelimit-limit-input-tokens header describe input usage as their names imply - _read_reservation_from_kwargs only falls back to litellm_params.metadata when the top-level metadata channel is absent, so production requests carrying a litellm_params.metadata dict still reconcile and refund their reservation Adds regression tests for OTPM atomicity under concurrency, input-only ITPM enforcement, and reservation lookup when litellm_params.metadata is present. * fix(router): subtract input tokens only from remaining-input-tokens header The in-flight replay for x-ratelimit-remaining-input-tokens subtracted total tokens (input + output) instead of input tokens only, so clients saw remaining input quota understated by the completion token count on every response. Now consistent with the input-only ITPM counter. * fix(router): make itpm/otpm vs tpm/rpm precedence explicit When a deployment configures itpm/otpm alongside tpm/rpm, the io-token path takes over and the tpm/rpm limits are not enforced. Log a warning the first time such a conflicting deployment is seen so the supersession is not silent, and document the mutual exclusivity. Post-call reconciliation now only trues up a counter that was actually reserved against, so the itpm/otpm keys are no longer incremented for deployments that never configured that limit. * fix(router): track actual io-token usage on the reservation-minute key Post-call reconciliation now keys off the exact cache key stashed at pre-call time rather than one recomputed from the response-time minute. This fixes two issues: a request whose pre-call estimate was 0 now still writes its actual billable input to the ITPM counter (previously it was skipped, leaving the limit unenforceable for that request), and a call that finishes in a later minute reconciles against the minute it reserved against instead of pushing a negative delta into the next minute. Counters are only touched when their limit is configured. * fix(router): run io-token reconciliation before the model_id guard async_log_success_event gated IO reconciliation behind the model_id guard that only the TPM tracking path needs. Since reconciliation works entirely from the cache keys stashed in kwargs, a success event whose standard_logging_object lacks model_id would skip reconciliation and leave the reservation on the counter until the TTL expired, wasting quota. Route the IO path first. * fix(router): don't replay in-flight delta for itpm/otpm headers For ITPM/OTPM model groups the counter is incremented at reservation time (pre-call), so the remaining values returned by get_remaining_model_group_usage already account for the current request. Replaying the in-flight delta on top double-counted it and understated x-ratelimit-remaining-input/output-tokens by up to max_tokens on every response. Skip the delta for io-token groups; the legacy TPM/RPM replay path is unchanged. * fix(router): clear io-token reservation after reconcile/refund async_io_token_refund_failure and async_io_token_reconcile_success now clear the stashed reservation keys from the request metadata once done. Otherwise, on a model group mixing IO-limited and non-IO deployments, a failed IO call that retries on a non-IO fallback left the stale sentinel in the shared request metadata; the fallback's success handler would divert into IO reconciliation against the already-refunded key, driving the ITPM counter negative and skipping the non-IO deployment's TPM tracking. * fix(router): tidy reservation channel lookup and header guard Consolidate the reservation channel lookup into a single ordered helper shared by read and clear, so top-level metadata always wins over litellm_params metadata without the tangled per-iteration fallback. Also stop gating the router rate-limit header block on the presence of x-ratelimit-remaining-input/output-tokens. That block only emits those headers for ITPM/OTPM groups; for a non-IO group backed by a provider that natively returns input/output token headers, the extra conditions suppressed the router's own remaining-tokens/requests headers. * fix(router): strip client-supplied io-token reservation keys The reservation sentinels (_litellm_itpm_reserved, _litellm_itpm_cache_key, and the otpm equivalents) are server-only, but metadata is caller-controlled on proxy requests. An authenticated caller could forge these fields with an arbitrary cache key so the post-call reconcile/refund path would decrement any deployment's ITPM/OTPM counter and let it exceed the configured limit. Strip the reserved keys from the request metadata in set_io_token_rate_limit_request_kwargs, which runs before the router stashes its own reservation, so only a genuine server-side reservation is ever read post-call. * fix(router): track TPM routing load for io-limited deployments deployment_callback_on_success early-returned for any deployment with itpm/otpm set, so its total-token usage never landed in the router's TPM routing counter. TPM-aware routing strategies then saw 0 load for IO deployments and over-routed to them in mixed model groups. Only skip tracking when neither tpm/rpm nor itpm/otpm are configured; itpm/otpm enforcement still runs separately in ModelRateLimitingCheck, so the routing counter and the enforcement counters stay independent. * fix(router): expose standard tpm/rpm headers for io-limited groups get_remaining_model_group_usage returned early for ITPM/OTPM groups, so a group that also set tpm/rpm never emitted x-ratelimit-remaining-tokens / -requests; clients and prometheus gauges reading those saw no data. Build both header sets instead of returning early. Also simplify the in-flight header replay: only the tpm/rpm counters are incremented post-response, so the delta now adjusts just those. The itpm/otpm counters are incremented at reservation time (pre-call), so the input/output token headers already reflect the request and are left untouched - which removes the need for the separate io-group special case. * fix(router): roll back ITPM on any OTPM reservation error; dedup warning per instance Two follow-ups from review. The pre-call OTPM reservation only rolled back the ITPM reservation on a RateLimitError, so a transient cache error while reserving OTPM left the ITPM counter inflated until the TTL expired; catch any exception, release the ITPM reservation, then re-raise. Replace the module-level lru_cache warn-once (caching a logging side effect, which never re-warns in a long-lived process) with an instance-scoped set of already-warned deployment ids on ModelRateLimitingCheck. * fix(router): always clear reservation stash on reconcile; don't collapse id-less warning dedup Clear the reservation in a finally block so a mid-reconciliation cache error still removes the stash and a duplicate success event can't re-process it. Dedup the itpm/otpm-vs-tpm/rpm conflict warning per real deployment id; a deployment with no id no longer collapses every id-less deployment onto the str(None) key (which would suppress all but the first warning). * fix(router): skip io reservation when deployment can't be keyed _get_cache_keys returned a shared 'global_router:None:None:...' key when a deployment was missing model_info.id or litellm_params.model, so misconfigured deployments could share one rate-limit bucket. Return None in that case and skip io reservation for the request. * fix(router): honor explicit max_tokens=0 in io reservation _resolve_max_tokens used 'max_tokens or max_completion_tokens', so an explicit max_tokens=0 fell through to the model default. Only fall back to max_completion_tokens when max_tokens is absent. * fix(ci): satisfy lint budget, router coverage, and dashboard schema sync - Modernize the new itpm/otpm module's type hints to PEP 585 lowercase generics (Dict/Tuple/List -> dict/tuple/list) to clear the added UP006 violations; ratchet ruff-strict-budget.json's UP006 ceiling down to match. - Replace three try/except Exception blocks that must stay broad by design (token_counter and litellm.get_model_info raise untyped exceptions, and an io-token refund failure must never break the logging pipeline) with contextlib.suppress(Exception), matching the codebase's existing resolution for this exact BLE001 pattern. - Add direct unit tests for get_model_group_io_token_usage (multi-deployment aggregation and the empty-model-list case) in test_router_helper_utils.py, satisfying the router function-coverage check. - Regenerate the dashboard's schema.d.ts so the new itpm/otpm fields on GenericLiteLLMParams and ModelGroupInfo are reflected in the OpenAPI types. * fix: enforce io token rate limits consistently * fix: honor zero max tokens in otpm reservation * fix(lint): fix UP007 violation and resync ruff-strict-budget.json to base Convert Union[_Span, Any] to _Span | Any (safe on this repo's Python >=3.10 floor) to clear the new UP007 violation from the TYPE_CHECKING-gated Span alias. The previously committed ruff-strict-budget.json ratcheted UP006 down from a stale base; litellm_internal_staging has since tightened that same ceiling further on its own. Reset the file to the current base's committed values and re-ratchet from there so the budget only ever moves down relative to the actual merge-base, never against a stale snapshot. * fix(router): attach ITPM/OTPM headers on dict responses and harden reservation Strip itpm/otpm from provider kwargs, ensure messages are available for ITPM estimation, honor max_output_tokens on /v1/responses, and propagate rate-limit headers through /v1/messages dict responses via _hidden_params. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): attach ITPM/OTPM headers to streaming /v1/messages responses Wrap bare async iterators in HiddenParamsAsyncIteratorWrapper so set_response_headers can attach rate-limit headers to streaming Anthropic messages responses that lack a _hidden_params slot. Co-authored-by: Cursor <cursoragent@cursor.com> * style: ruff format add_retry_fallback_headers.py Fix CI ruff format check failure on get_hidden_params_dict call site. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(router): extract set_response_headers helpers to fix C901 budget Move header-attachment logic into add_retry_fallback_headers helpers so set_response_headers stays under the strict complexity ceiling. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: keep IO token reservation when response usage is missing Missing usage was reconciled as zero and fully refunded the pre-call reservation, allowing limit bypass on repeated successful calls. Only adjust counters when usage is resolved from the response or standard logging fields; otherwise keep the reservation until TTL expires. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: enforce RPM/TPM alongside IO-token limits on mixed deployments Deployments with both itpm/otpm and tpm/rpm previously returned after the IO reservation and skipped RPM/TPM checks. Run both paths and refund the IO reservation only when RPM/TPM rejects after a successful reservation. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: track TPM usage on success for mixed IO+TPM deployments The early return after IO-token reconciliation in log_success_event and async_log_success_event skipped the TPM counter increment, so the tpm_key the pre-call check reads was never written and tpm_limit was never actually enforced on deployments that also configure itpm/otpm. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: treat total-only usage as unresolved in IO-token reconcile usage/standard_logging_object entries carrying only total_tokens (no prompt/completion or input/output breakdown) were treated as resolved usage, resolving to (0, 0) and refunding the full reservation. Both _usage_is_present and the standard_logging_object fallback now require an actual input/output breakdown before reconciling, keeping the reservation otherwise. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: reserve minimal token when input/output estimation fails _reservation_value(0, limit) reserved the entire limit whenever token estimation failed (empty/unsupported input, tokenizer error), letting one such request claim the whole bucket and 429 every concurrent request to the deployment until it completed. Reserve 1 token instead so estimation failures no longer serialize traffic. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: refund IO reservation synchronously before retry deployment pick On retry, set_io_token_rate_limit_request_kwargs clears reservation sentinels from the shared kwargs dict before a background failure handler can refund them, stranding the counter until TTL. Refund and clear any stale reservation in _update_kwargs_with_deployment before stripping sentinels for the next attempt. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(io_token_rate_limit_check): use model-specific tokenizer for ITPM estimate; document sync-refund Redis ceiling Pass the deployment litellm_params.model to token_counter so it uses the model's native tokenizer instead of the generic fallback, narrowing the reservation over/under-estimate window between pre-call and post-call reconcile. Add a ponytail: comment to refund_stale_reservation_before_retry explaining the known ceiling: the synchronous DualCache.increment_cache issues a blocking Redis INCR when a Redis backend is configured. This only fires on streaming mid-stream retries (non-streaming failures await their failure handler before the retry picks a new deployment, leaving no sentinels to refund). Upgrade path: make _update_kwargs_with_deployment async. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…treams (#32237) * fix(streaming): surface in-body error payloads on OpenAI-compatible streams vLLM and sglang return HTTP 200 streams whose SSE body carries the error, e.g. data: {"error": {"message": "...", "code": 400}}. The OpenAI-compatible chunk parser had no detection for this shape: since #23931 the payload parsed into an empty chunk (choices=[]) and the stream ended silently with 200, losing the provider's error and never attempting configured fallbacks. Detect the payload in OpenAIChatCompletionStreamingHandler.chunk_parser and raise OpenAIError with the upstream message and status code. The existing mid-stream gate then applies: 4xx surface directly to the client, 5xx wrap into MidStreamFallbackError so the router can run configured fallbacks. Fixes #25492 * fix(streaming): serialize messageless error payloads as JSON Address review feedback: an error dict without a message field now serializes via json.dumps instead of Python dict repr
…ad (#32145) * fix(ui): reflect persisted store_prompts_in_spend_logs toggle on load Co-Authored-By: bot_apk <apk@cognition.ai> * test(ui): avoid new no-explicit-any in logging settings regression test Co-Authored-By: bot_apk <apk@cognition.ai> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: bot_apk <apk@cognition.ai>
* feat(ui): add cost optimization feedback banner to models page Surfaces a dismissible banner on Models + Endpoints prompting users to share cost optimization feedback (routing, budgets, etc) via a GitHub discussion. * test(ui): add regression test for cost optimization feedback banner * test(ui): update Models+Endpoints banner tests for cost optimization banner Missing Provider banner tests are replaced since that banner was removed in favor of the new always-on cost optimization feedback banner.
fix(spend): filter /global/spend/report by team_id when group_by=team
…CrowdStrike AIDR (#31974) Previously, every guardrail request forwarded the full conversation history to CrowdStrike AIDR. In a multi-turn conversation this means every prior message gets re-scanned on every new call, even though those messages were already evaluated in earlier turns. CrowdStrike AIDR internally has a conversation boundary optimization in place for just this scenario (ref. <https://aidr-docs.crowdstrike.com/docs/aidr/apis#messages-array-optional---array-of-message-objects-containing-a-conversation-segment-with-the-ai-system>). However, it is nevertheless wasteful to send so much data to the API when only a subset of it will be processed. It also risks hitting the documented 1 MiB request size limit. So now we filter down to system messages plus either the messages after the last assistant turn, or the last assistant message itself when that is what is being guarded. We also preserve the original, full message history within the guardrail in order to stitch back any transformations. Co-authored-by: Kenan Yildirim <kenan@kenany.me>
…vices chart (#32233) The componentized chart at helm/litellm had no way to mount extra volumes into its deployments, so custom callback or SSO handler code could not be mounted the way the docs describe for the monolithic chart. Adds per-component volumes and volumeMounts values for gateway, backend, and ui, merged with the existing gateway-config volume, plus a helm-unittest suite for the chart wired into the helm unit test workflow Resolves LIT-4209
…urced models PR #30867 removed request-time os.environ/ expansion in BaseAWSLLM.get_credentials to close LIT-3831. That relies on config-load paths pre-resolving os.environ/ refs, but the DB-load path (_resolve_db_litellm_param) only re-expands keys in _DB_LITELLM_PARAM_ENV_REF_KEYS, which covered api_key, aws_access_key_id, and aws_secret_access_key but not the other AWS auth fields. A model stored in Postgres with e.g. aws_role_name: os.environ/BEDROCK_ASSUME_ROLE_ARN lands on the router with the literal string, get_credentials no longer expands it, and STS returns ValidationError: os.environ/BEDROCK_ASSUME_ROLE_ARN is invalid Add the remaining AWS auth params to the allowlist so DB-sourced values resolve at model-load time (trusted, server-side), matching the YAML-config path. Team-scoped DB rows still get resolve_env_refs=False, so the LIT-3831 defense-in-depth path is unchanged and request-body injection is still blocked by _BANNED_REQUEST_BODY_PARAMS. Regression tests pin every added field as an os.environ/ DB value and assert it resolves on the router, plus a team-scoped pin that asserts env refs remain literal.
…how filtered-out count (#32285) * fix(mcp): apply semantic filter to expanded litellm_proxy tools and show filtered-out count * fix(mcp): guard expansion-path filtering on enabled flag and isolate metadata emission
…ess (#31300) Add an opt-in mode so a key that exceeds its own max_budget is throttled to a globally configured percentage of its TPM/RPM instead of being blocked entirely. A new litellm_settings global, budget_exceeded_throttle_percentage, sets the fraction (e.g. 0.1 = 10%). A per-key throttle_on_budget_exceeded flag (stored in key metadata via the existing management-endpoint metadata routing) opts the key in. When both are set and the key is over budget, the budget check records the percentage on a request-scoped budget_throttle_pct instead of raising, and the rate limiter scales the key's configured TPM/RPM by it. Keys without the flag keep hard-blocking; team/user/org budgets are unaffected. The throttle is recomputed from the key's original limits on every request and the decision is cleared before the auth object is cached, so it never compounds across requests. Both the budget read-time check and the budget reservation path honor the opt-in, and both the v3 and legacy rate limiters apply the scaling. Enabling throttle_on_budget_exceeded is proxy-admin only. It converts an admin-imposed hard budget block into a soft throttle that keeps spending past max_budget, so a non-admin must not be able to self-opt-in and bypass their own spend cap. Both /key/generate and /key/update reject a non-admin setting it to true (update only gates the transition to enabled, so a non-admin can still edit other fields and turn the flag off). This matches the feature being wholly proxy-admin operated: the global percentage is admin-only too. A key that opts in but has no TPM or RPM limit has nothing to scale, so it stays hard-blocked rather than serving unlimited requests past its budget (fail-safe). The global budget_exceeded_throttle_percentage is configurable from the admin UI (Settings -> General Settings), persisted through litellm_settings so it survives a restart, not only from config.yaml. Resolves LIT-3894. Scope for LIT-3893.
…cans (#31825) * fix(spend): bound the logs-tab pagination count to stop full-window scans The spend-logs UI list endpoint (/spend/logs/ui, /spend/logs/v2) computed an exact pagination total over the whole selected time window on every load. That was a standalone SELECT COUNT(*) FROM (SELECT request_id FROM LiteLLM_SpendLogs WHERE ...) which, on a multi-TB SpendLogs table, scans a huge number of rows and spikes Aurora ACU. The later LIT-4027 change folded it into COUNT(*) OVER (), but a window count still drains every matching row before the LIMIT applies, so the full-window scan remained. Compute the total with a bounded SELECT COUNT(*) FROM (SELECT 1 ... LIMIT $cap+1) that probes at most cap+1 rows, and drop the window count from the page query so the page query is a plain indexed ORDER BY ... LIMIT. When more than the cap match, report the cap and set total_is_capped so the UI renders "<cap>+". The bounded subquery terminates early rather than aggregating across all tablets, so it stays safe on sharded engines like YugabyteDB too. Resolves LIT-4119 * test(spend): make zero-total mock reflect real COUNT(*), add capped-total tooltip Address Greptile review on #31825: - the empty-result test now returns [{"total_count": 0}] for the bounded count query (real COUNT(*) always returns one row) instead of [], so the zero-total path exercises the normal branch rather than the defensive guard - the logs toolbar shows a tooltip explaining the cap when total_is_capped is set, so a disabled Next button at the cap boundary reads as intentional
* refactor(ui): switch shadcn primitives from Radix to Base UI shadcn made Base UI the default primitive library in July 2026 and our only shadcn component so far is the Button canary, so this is the last cheap moment to switch before the primitives phase adds the full set. components.json style moves from new-york (a legacy alias that resolves to the Radix variant) to base-vega. Button is regenerated from the base-vega registry with the same local adaptations as before: cva beta object form via lib/cva.config and a React 18 forwardRef wrapper. The polymorphic asChild prop becomes Base UI's render prop. radix-ui is replaced by @base-ui/react 1.6.0. Base UI optionally peers on date-fns 4 while tremor pins 3, so date-fns is bumped to 4.4.0 with an npm override; our only usage (add) is API-identical and the override can go away when tremor does. * refactor(ui): convert chat UI shadcn components from Radix to Base UI The chat UI migration landed 11 components/ui files generated against the old Radix registry config after this branch cut over to Base UI, which would have left them importing a deleted package. All 11 (dialog, alert-dialog, select, popover, tooltip, tabs, switch, scroll-area, collapsible, separator, label) are regenerated from the base-vega registry, with the repo conventions re-applied where relevant (cva beta object form from lib/cva.config in tabs; the Button canary keeps its React 18 forwardRef adaptation). Chat feature call sites move from the Radix asChild pattern to Base UI's render prop, and TooltipProvider delayDuration becomes delay. * fix(ui): restore security override pins clobbered by the date-fns override The date-fns 4 override was written by replacing the whole overrides object, dropping the ten security pins (prismjs, js-yaml, glob, minimatch, lodash, ws, braces, axios, postcss, esbuild) that keep patched versions in the lockfile; osv-scan caught the vulnerable versions resurfacing. Restores the pins alongside date-fns and regenerates the lockfile. * test(ui): pin tremor DateRangePicker behavior on date-fns 4 The date-fns 4 override forces react-day-picker 8 (authored against v3) onto v4 at runtime, which a build or lint pass cannot validate. This renders the shared UsageDatePicker wrapper, opens the calendar, checks the month grid, and selects a day, so a date-fns API break in the tremor date path fails tests instead of throwing in production. Delete alongside the override when tremor is removed. * fix(ui): close the alert dialog when AlertDialogAction is clicked The base-vega registry template renders AlertDialogAction as a plain Button with no Close binding, so confirm buttons fired their onClick but left the dialog open; both consumers (conversation delete, MCP credential revoke) were written against the Radix semantics where Action dismisses on click. Binds Action to AlertDialogPrimitive.Close via the render prop, mirroring AlertDialogCancel, and pins the behavior with a test so a future shadcn add --overwrite cannot silently reintroduce the template's non-closing Action.
…nfig; inference reduced to the request-time backstop (#32292) * refactor(mcp): read oauth2_flow verbatim from DB rows; inference stays config-only plus a logged backstop With every DB write site stamping oauth2_flow (#32283, #32288) and the startup backfill healing legacy null rows, the DB build no longer needs to re-derive the flow from field shape. build_mcp_server_from_table now reads the column verbatim via _explicit_oauth2_flow: unknown or null values resolve to None, which needs_user_oauth_token already treats as interactive, so an unstamped row degrades to the safe default instead of guessing M2M from a shape that a DCR-registered interactive server shares whenever discovery is down Field-shape inference survives in exactly two places. config.yaml-loaded servers keep it at load time: they are rebuilt from the config on every boot, so there is no row to backfill and load-time resolution is their write-time stamp. And the request-time backstop in _get_allowed_mcp_servers keeps a not-yet-backfilled M2M row blocking caller Authorization forwarding (the P1 property); it now logs a warning whenever it actually fires, which is the fire-rate signal for deleting it once deployments have booted past the backfill Regression tests pin that the DB build does not infer M2M from the credential shape and reads an explicit column value verbatim Fourth step of the oauth2_flow persistence sequence, stacked on the backfill * feat(mcp): deprecation warning when config-level M2M is inferred rather than declared A config.yaml oauth2 server whose credential shape decides client_credentials without an explicit oauth2_flow now logs a warning at load pointing the admin at the explicit declaration. First rung of the deprecation ladder: the docs make oauth2_flow the recommended path, the warning surfaces configs still relying on inference, and a future breaking release can turn it into a config validation error, at which point config-level shape inference dies entirely. Interactive omissions stay silent since the default matches inference there and nothing load-bearing is being guessed * feat(mcp)!: require explicit oauth2_flow for config-defined oauth2 servers A config.yaml server with auth_type oauth2 must now declare its flow; the load raises a config validation error naming both values and what each means: oauth2_flow: client_credentials for machine-to-machine (the proxy mints a shared token at token_url using client_id/client_secret) or oauth2_flow: authorization_code for interactive (per-user tokens via browser sign-in, including delegate_auth_to_upstream) This replaces the load-time shape inference for config servers entirely. The credential shape is genuinely ambiguous (a DCR-registered interactive server carries client creds + token_url with no authorization_url, identical to M2M), so the config asserts the answer instead of the proxy guessing it. With this, field-shape inference survives in exactly one place: the request-time security backstop, which is telemetry-gated for deletion BREAKING CHANGE: config-defined oauth2 MCP servers without oauth2_flow fail proxy startup with the error above. Add the one line to the server block; the error text says exactly which value to pick * test(mcp): pin the verbatim read for authorization_code alongside client_credentials Raised by review on the PR * fix(mcp): fail closed on the anonymous delegate gate for unstamped M2M-shaped servers Reading oauth2_flow verbatim (this PR) changed has_client_credentials from True to False for a legacy null-flow row that still carries the M2M credential shape. That value is what the anonymous upstream-delegate gate checks before skipping LiteLLM auth entirely, so an M2M-shaped delegate server that was never stamped would newly pass the gate: an unauthenticated caller could get it selected and then list/read upstream data using the client credentials the request-time backstop re-infers, running as LiteLLM's service account. This reopens the hole the gate's existing 'never delegate for M2M' guard was written to close The gate now resolves the flow (column first, shape fallback) instead of reading the bare column, mirroring the request-time backstop in _get_allowed_mcp_servers: both fail closed on the ambiguous M2M shape and are removed together once no null rows remain. A pure-PKCE delegate server (no stored credentials) resolves to a non-M2M flow and keeps its bypass, so the common delegate case is unaffected Tests: an unstamped M2M-shaped delegate server is denied the bypass (mutation-checked against the bare-column regression), and a pure-PKCE delegate server still bypasses Raised by review on the PR * fix(mcp): centralize the request-time oauth2_flow backstop across every security site Reading oauth2_flow verbatim made has_client_credentials unreliable for legacy null rows, and the backstop that compensates was applied at only one reader. Review found three more consequences of that per-site approach: - the anonymous-delegate allowlist in get_allowed_mcp_servers read the bare column, so an unstamped M2M-shape delegate server was surfaced to anonymous callers (High) - call_mcp_tool resolved allowed ids into MCPServer objects without the backstop, so a null-flow M2M-shape row kept has_client_credentials false on tool execution during a backfill gap, though the listing path was covered (High) - the request-time warning claimed the startup backfill would stamp the row next boot, but the backfill deliberately leaves the ambiguous M2M shape unstamped (Low) Rather than patch each site, introduce two helpers on MCPServerManager that are the single choke point for request-time resolution: effective_oauth2_flow(server) for the enum/boolean decisions (allowlist filter, anonymous-delegate gate) and resolve_oauth2_flow_for_request(server) for the egress object copy (listing and tool call). Both fail closed on the M2M shape and leave stamped rows and pure-PKCE rows untouched. The gate now shares effective_oauth2_flow instead of its inline resolution, and the corrected warning lives once inside resolve_oauth2_flow_for_request, so deleting the whole transitional layer later is a single-site change. Tests: helper unit coverage (stamped verbatim, null M2M-shape resolves, pure-PKCE stays None, stamped/pure-PKCE return the same object, corrected warning text), the anonymous allowlist excludes an unstamped M2M-shape delegate server, and the call path resolves the flow like the listing path. The two security-integration tests are mutation-checked against the bare-column regression. Raised by review on the PR
… streaming (#32141) * fix(bedrock): preserve stream param and decode SSE for bedrock mantle streaming * refactor(bedrock): pass self positionally in mantle messages streaming delegation
bump: litellm-enterprise 0.1.47 -> 0.1.48, litellm 1.92.0 -> 1.93.0
…el_cost pricing (#32163) * fix(main): stop per-request custom pricing from clobbering shared model_cost pricing A request routed through a wildcard deployment with explicit zero pricing (e.g. openai/* with input_cost_per_token: 0) registered that pricing on the shared {provider}/{model} key in litellm.model_cost, so sibling deployments relying on built-in pricing logged $0 until process restart (LIT-3991). Request-time registration in completion()/embedding() now mirrors the router-startup isolation: router-originated requests register full pricing under the deployment's unique model id only, while the shared backend key receives the entry with custom pricing fields stripped. Direct SDK calls without a router deployment id keep the legacy shared-key registration. The stripping logic is shared via CustomPricingLiteLLMParams.strip_custom_pricing_fields and reused by Router._create_deployment and Router.add_deployment. * test: update legacy tests that asserted per-request pricing leaking into shared model_cost test_router_fallbacks_with_custom_model_costs asserted the shared claude-sonnet-4-5-20250929 entry ends up with the deployment's 30/60 pricing, which is exactly the cross-deployment leak this PR removes; it now asserts the shared key keeps the built-in pricing, matching the test's stated goal. test_cost_calc.py::test_run computed streaming cost via completion_cost(response), which only matched the non-stream cost while the shared gpt-3.5-turbo entry was poisoned with the per-request 2/token pricing; it now passes the request's custom pricing explicitly via custom_cost_per_token.
#32348) UI session tokens carry the virtual team_id litellm-dashboard (UI_TEAM_ID), which is never persisted. The MCP team-permission helpers passed it to get_team_object anyway, so every dashboard MCP listing raised a 404 per lookup that was swallowed into per-server 'Failed to get allowed tools for server' warnings (plus the sibling 'allowed MCP servers for team' and 'MCP access groups for team' warnings) and wasted DB queries. The 404 also escaped past the key-level permission handling in get_allowed_tools_for_server, dropping key tool restrictions for such sessions. Short-circuit the virtual team before the DB lookup in the three helpers, mirroring the existing UI_TEAM_ID handling in agent_permission_handler. Also reject /team/new with the reserved team_id, since a real row would bind its budget and permissions to every UI session
* feat(ui): OAuth flow selector on the MCP edit page
The edit form had no flow selector: oauth_flow_type was watched but never registered,
so isM2MFlow was always false in edit mode and the flow could only be changed over
REST. That left the backfill's remediation for ambiguous legacy rows (client creds +
token_url, no interactive signal, left unstamped) without a dashboard path
The oauth2 section now opens with an OAuth Flow Type select. Explicit rows prefill
their stored value and re-persist it on save; legacy null rows show a placeholder
instead of a fake preselection, and an untouched save still writes nothing, so the
form never guesses on the admin's behalf. Choosing Machine-to-Machine (M2M) persists
oauth2_flow=client_credentials, choosing Interactive (PKCE) persists
authorization_code, which is exactly the assertion the backfill warning asks for.
Registering the field also brings the existing isM2MFlow gating in the edit form to
life, so M2M rows stop showing the interactive-only token-validation fields
Tests cover the prefill round-trip for both explicit values, the untouched null row
writing nothing, and both selections persisting on a legacy null-flow row
* fix(mcp): registry-to-table conversions must carry oauth2_flow
_build_mcp_server_table and the health-check table builder dropped oauth2_flow when
converting registry servers for GET /v1/mcp/server (list and by-id), so the dashboard
never received the persisted flow: the edit page could not prefill the selector, M2M
gating never activated, and the tools page classifier saw every oauth2 server as
interactive regardless of the column. Found live while proving the edit-selector
persistence path end to end; the write side was fine (PUT persists and the column
reads back correctly), the read side was dropping the field at the conversion
Both builders now carry oauth2_flow; regression test pins the conversion
* docs(mcp): flag _resolve_oauth2_flow as security-sensitive in its docstring
The prior wording ('not called directly by security sites') could read as if the
function has no security relevance, when it is the shape-inference engine both
request-time security helpers delegate to. Reword to state that plainly: it decides
M2M-vs-interactive for an unstamped row, must always be reached through
effective_oauth2_flow or resolve_oauth2_flow_for_request, and its M2M-shape branch
must not be weakened without accounting for those callers. Docstring-only; no logic
change
Raised by review on the stacked PR
* refactor(ui): extract oauth2FlowToFormValue helper for the MCP OAuth flow prefill
The edit form derived the OAuth Flow Type select value from the stored oauth2_flow
with a nested ternary duplicated at two call sites. Extract the mapping into a named
helper in types.tsx (next to getMcpOAuthMode and the flow constants): client_credentials
-> M2M, authorization_code -> Interactive, null/unset -> undefined so the select shows
its placeholder instead of a guessed default. The tool-config call site keeps its
null -> Interactive display fallback via a trailing ?? OAUTH_FLOW.INTERACTIVE, so
behavior is unchanged. Adds unit tests for the helper; the existing prefill/save tests
already cover the call sites
* feat(ui): surface and warn on an unset MCP oauth2_flow (server card + edit page)
An oauth2 MCP server whose oauth2_flow was never classified (legacy null row the
backfill left ambiguous) now advertises that it needs attention instead of silently
falling back. The server card shows an 'OAuth flow not set' warning tag for any
auth_type=oauth2 server with no oauth2_flow, so admins can spot them in the list
without opening each one. The edit page shows a warning alert directly under the new
OAuth Flow Type selector while the flow is unset, and it clears the moment a flow is
picked.
Delegate (delegate_auth_to_upstream) servers are excluded from both: they authenticate
via upstream PKCE passthrough and route to passthrough regardless of oauth2_flow, so
the M2M-vs-interactive classification does not apply and prompting for it would be a
false alarm. The edit page reads the delegate state from the watched switch when it is
mounted and falls back to the stored value otherwise (useWatch returns undefined for an
unmounted field).
Also adds end-to-end coverage of the null-flow chain the selector depends on:
build_mcp_server_from_table carries oauth2_flow=None verbatim into the GET response,
so the dashboard maps it to undefined and shows the placeholder rather than a guessed
default. Tests: backend null carry, the select prefill display for all three states,
the edit-page warning show/hide/clear-on-select and delegate exclusion, and the card
badge across oauth2/non-oauth2, stamped/unstamped, and delegate
* test(ui): characterize DataTable behavior before shadcn reskin Pins the shared view_logs DataTable contract with library-agnostic queries ahead of the tremor-to-shadcn table migration: loading and empty states, TanStack column defs with custom cell renderers, onRowClick payload, both expansion render paths (colspan sub-component and sibling child rows), the getRowCanExpand gate, and client-side sorting on and off. These must pass unchanged after the reskin. * test(ui): assert child rows hidden before expansion in DataTable test
Introduce the e2e coverage denominator: 282 behavior cells across the six tracking modules (LLMs, MCPs, Management/UI, Reliability & Performance, Logging & Guardrails, Other), one validated YAML row each, plus a collector that diffs the registry against @pytest.mark.covers markers and reports coverage per module. The registry rows validate against a pydantic discriminated union so a row cannot carry a field from another module. The collector is static: a collect-only pass reads the markers, so it runs no test and needs no live proxy. Register the covers marker suite-wide so that pass works under --strict-markers. This is a draft for review. Tiers are proposed rather than signed off, and a few cells still need a support check or a prune.
…ath partner models with custom api_base (#32380)
…ashboard (#31772) * feat(mcp): support oauth2_token_exchange auth type via REST API and dashboard OAuth 2.0 Token Exchange (RFC 8693, a.k.a. OBO) for MCP servers could previously only be configured through config.yaml; the create/update REST API and the dashboard had no way to express it. This wires token_exchange_endpoint, audience, and subject_token_type end to end. These three are persisted as dedicated columns on LiteLLM_MCPServerTable, mirroring how token_url and oauth2_flow are stored, so the edit form prefills them and they are unaffected by the credentials-blob clearing on auth_type change. build_mcp_server_from_table reads the columns first and falls back to the credentials blob so servers persisted before the columns existed still load. client_id and client_secret continue to ride the existing encrypted credentials path. On the dashboard, "OAuth Token Exchange (OBO)" is a distinct auth-type option with its own field section. The McpOAuthMode classifier gains a token_exchange arm keyed off auth_type; the previous catch-all oauth2 mode was renamed from "obo" to "authorization_code" so the two on-behalf-of mechanisms are no longer conflated. The token-exchange IdP endpoint and audience are scrubbed from non-admin and virtual-key responses, matching how token_url is treated. * fix(ui): gate the MCP list-401 re-auth on authorization_code, not token_exchange The renamed 401 gate keyed off isTokenExchange, but that condition exists for authorization_code: when a stored per-user credential is present yet the tools/list 401s (the backend's refresh could not mint a token), the user must re-authorize via the browser flow. token_exchange has no gateway-side authorize step, so the Authorize gate never applied to it. The prior isObo flag was undefined (a compile error) and, per this file's convention and its tests, meant authorization_code; renaming it to isTokenExchange changed the behavior and broke the mcp_tools auth-gate test for an authorization_code server whose token expired with no usable refresh. Gate on isAuthorizationCode instead and drop the now-unused isTokenExchange * fix(mcp): clear flow-scoped endpoint config when a server's auth_type changes Switching an existing oauth2 server to oauth2_token_exchange left the old flow's token_url on the row. The OBO resolver treats token_exchange_endpoint or token_url as the configured exchange endpoint, so the stale value both suppressed the RFC 9728/8414 discovery this PR adds and sent the exchange grant (client credentials plus the user's subject token) to the previous flow's token endpoint update_mcp_server now mirrors its existing stale-credentials rule for the flow-scoped columns (authorization_url, token_url, registration_url, oauth2_flow, token_exchange_endpoint, audience, subject_token_type): when auth_type changes, each one is cleared unless the same request explicitly provides it, so a deliberate override in the switch request still wins. Updates that keep the auth_type never touch these columns, which keeps legacy OBO rows that use token_url as their exchange endpoint working The edit form sends explicit nulls for the previous flow's fields on an auth type switch; antd preserves unmounted field values by default, so without this the old token_url would be re-sent verbatim and read as an explicit override. Transitions are detected against the persisted auth_type, so saves that keep the auth type send nothing extra Reported by Cursor Bugbot on the PR * fix(mcp): lift legacy blob token-exchange settings into their columns on every write The three token-exchange settings live in dedicated columns but also exist on MCPCredentials as the pre-column REST shape. Writes now lift incoming blob values into the columns (an explicit top-level value wins, including an explicit null) and strip them from the stored blob; the same-auth credentials merge migrates legacy rows the same way. The read-time column-or-blob fallback then only ever serves rows current code has never written, so clearing a column to re-enable RFC 9728/8414 discovery can no longer be silently undone by a stale blob copy. Also asserts the auth-switch clearing fires on the external fields_set path (PUT /v1/mcp/server). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(mcp): single source for the RFC 8693 default subject_token_type The default was applied at four egress build sites plus two model defaults, each with its own copy of the literal. All sites now share DEFAULT_SUBJECT_TOKEN_TYPE from litellm.types.mcp. A DB-level DEFAULT is deliberately not used: Prisma writes explicit values on insert, so a column default would rarely apply, and NULL-means-RFC-default keeps existing rows correct. Also documents two review decisions in place: the audience column keeps the RFC 8693 parameter name (RFC 8707 resource indicators are already a separate concept named resource in the v2 egress types), and the migration's out-of-order timestamp is safe under prisma migrate deploy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: fix import sort order in outbound_credentials/types.py (I001 strict budget) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(mcp): purge legacy blob copies when a token-exchange column is written without credentials The migrate-on-write in the credentials merge lifts blob values into null columns, which is correct for legacy rows but could repopulate a column an admin had cleared in an earlier no-credentials update (that path never touched the blob, so the stale copy survived to be lifted later). An explicit token-exchange column write (set or clear) now migrates the row even when the update carries no credentials: untouched null columns are lifted, every blob copy is stripped, and unrelated blob keys stay as-is. A cleared column can then never be resurrected, because no write path leaves a blob copy behind. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(mcp): state the blob-to-column lift contract on the legacy credential keys The three token-exchange keys on MCPCredentials are the pre-column REST shape (the only REST shape from 2026-05 until this PR). Document on both the blob type and the request models that the dedicated columns are authoritative and that writes lift blob values into them and strip the stored copy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(mcp): scrub subject_token_type in the non-admin and virtual-key sanitizers The other two token-exchange fields were cleared while subject_token_type was left visible. It is a public RFC 8693 URN with no disclosure value, but the sanitizers' rule is that these views receive no token-exchange config at all — cleared for uniformity. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…d of model refusal wording (#32388) test_text_message_blocked_by_guardrail_no_ai_response classified the model's reply against a safe_markers keyword list to decide whether the guardrail had blocked the message. gpt-realtime words its refusal of the guardrail's "say exactly" voice prompt nondeterministically, so any new phrasing outside the list turned CI red on unrelated PRs; the list had already been extended in #28191, #28200 and #29477, and drifted again to "Sorry, I can't comply with that request" (11 of the 13 failed realtime_translation_testing runs since 2026-06-24, e.g. CircleCI job 2009316 on #32380). Record every frame the proxy sends to the backend through a RecordingBackendWebSocket wrapper and assert the invariant the product actually guarantees: the blocked phrase never reaches OpenAI, only the guardrail's own conversation.item.create and response.create are forwarded (the client's reflexive response.create is dropped), and the blocked phrase never appears in AI output. Replace the fixed 0.3s/3.0s sleeps with an event-driven wait for response.done; client frames are processed sequentially so no inter-message sleep is needed. Verified by mutation: disabling the response.create drop fails the response.create count assertion, and disabling the guardrail fails the guardrail_violation assertion.
…strip-and-retry re-sign (#32371) * fix(bedrock): stop stale SigV4 headers clobbering fresh signature on re-sign When the Anthropic /v1/messages strip-thinking-and-retry path re-signs a Bedrock request, _sign_request received attempt 1's already-signed headers and copied the old Authorization and X-Amz-Date back over the freshly computed SigV4 signature, so the retry POSTed the stripped body with a signature for the original body and AWS returned 403 SignatureDoesNotMatch. Skip SigV4-computed headers (authorization, x-amz-date, x-amz-security-token, date) when restoring caller headers after signing, and only preserve a caller-supplied Authorization that is not itself a SigV4 header so bearer-token setups keep working. * fix(bedrock): apply the same stale-header guard to get_request_headers
The token_exchange (OBO) branch of _call_regular_mcp_tool built its coroutine by calling _obo_call_tool_with_retry directly, outside the _limit_outbound_concurrency context manager that the regular branch uses. OBO tool calls (and the internal re-mint retry, which issues a second upstream call_tool) therefore bypassed the per-server max_concurrent_requests semaphore, so an authenticated caller could run unlimited concurrent tool calls against an OBO MCP server despite an admin-configured limit. Wrap the OBO coroutine in _limit_outbound_concurrency the same way the regular path does, holding one permit across the initial call, the on-401 re-mint, and the retry, so OBO calls honor the configured cap.
…hange auth type (#32385) * test(ui): pin that the token-exchange fields render only for the oauth2_token_exchange auth type No form section asserted the visibility contract: the token-exchange fields (Token Exchange Endpoint, Audience, Subject Token Type) must appear when 'OAuth Token Exchange (OBO)' is selected and for no other auth type. Assert hidden under plain OAuth, shown under token exchange, hidden again after switching to API Key. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(ui): assert the stdio transport switch unmounts the token-exchange fields The create form gates the whole Authentication section on non-stdio transport, so selecting OAuth Token Exchange (OBO) and then switching to stdio removes the token-exchange fields (and their required-credential rules, which antd does not validate while unmounted). Pin that sequence so the section-level gate cannot regress silently. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…he UI and API (#32144) * feat(mcp): let users select the entra_obo token_exchange profile in the UI and API The backend token_exchange arm supports two wire dialects via token_exchange_profile ("rfc8693" default, or "entra_obo" for Microsoft Entra's On-Behalf-Of, the RFC 7523 jwt-bearer grant), but it could only be set through config.yaml. This surfaces it to the create/update REST API and the dashboard so an admin can create an entra_obo server there, completing the parity started in the parent PR for the other token-exchange fields. token_exchange_profile becomes a dedicated column on LiteLLM_MCPServerTable, mirroring the sibling fields: it is added to the request models, read column-first in build_mcp_server_from_table with the credentials-blob as a back-compat fallback and a default of rfc8693, and carried through both runtime-to-table builders so registry round-trips preserve it. It is a non-secret dialect selector, so it is not scrubbed from non-admin or virtual-key responses. In the dashboard a Profile dropdown (RFC 8693 vs Microsoft Entra OBO) is added to the token-exchange section. Entra OBO carries the target resource in the scope, so selecting it makes the scope required and hints the api://<app-id>/.default form, while audience and subject_token_type (which that dialect ignores) are hidden. * fix(mcp): extend the blob-to-column lift and non-admin scrubbing to token_exchange_profile token_exchange_profile gets the same storage contract as the other three token-exchange settings: the column is authoritative, a blob copy is the legacy shape — lifted into the column on every write and stripped from the stored blob — and switching auth_type away from token exchange clears it (_AUTH_FLOW_SCOPED_FIELDS). Both restricted-view sanitizers scrub it for uniformity, and the edit form's auth-switch payload nulling includes it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(mcp): assert every token-exchange setting is configurable via config.yaml Pins the config surface: token_exchange_endpoint, audience, subject_token_type and token_exchange_profile load from top-level config keys onto the built server and through to the resolver spec; omitted keys resolve to their documented defaults (RFC 8693 subject token type, rfc8693 profile), and token_exchange servers need no oauth2_flow. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
Too many files changed for review. ( Bypass the limit by tagging |
tin-berri
approved these changes
Jul 7, 2026
ryan-crabbe-berri
approved these changes
Jul 8, 2026
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Contributor
Merging this PR will not alter performance
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ❌ | test_completion_simple_message |
3.2 ms | 4.8 ms | -32.55% |
| ⚡ | test_completion_multi_turn |
4.2 ms | 3.1 ms | +33.07% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing litellm_internal_staging (db24027) with main (88e03e5)1
Footnotes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relevant issues
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Type
🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test
Changes