chore(oss): litellm oss staging 120626 - #30292
Merged
mateo-berri merged 29 commits intoJun 12, 2026
Merged
Conversation
* feat(bedrock): add bedrock mantle gemma 4 models * test(bedrock): harden mantle local cost fixture
…#30209) * feat(responses): enable the responses API for the Tensormesh provider * Update litellm/llms/openai_like/providers.json Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
…es (#30240) stream_chunk_size is a LiteLLM-internal knob for re-chunking the HTTP response stream. The invoke transformations splat optional_params into the provider request body without dropping it, and Bedrock rejects unknown fields, so any bedrock/invoke request that sets the parameter fails with ValidationException: stream_chunk_size: Extra inputs are not permitted. Drop it in the invoke dispatcher (covers cohere, titan, mistral, meta, ai21) and in the Claude messages-format request builder (the route used for bedrock/invoke Anthropic models)
* fix(bedrock): stop buffering streamed tool-call argument deltas Two issues made Bedrock tool-use streaming arrive as a single end-of-stream burst through LiteLLM while plain text streamed fine. First, the anthropic-beta allowlist mapped fine-grained-tool-streaming-2025-05-14 to null for bedrock and bedrock_converse, so the header was silently stripped. Without that beta, Anthropic models on Bedrock buffer tool input server-side and emit all toolUse.input deltas at once (verified against converse-stream and invoke-with-response-stream directly). Bedrock accepts the beta via additionalModelRequestFields.anthropic_beta, so it is now forwarded. Second, the streaming reads re-chunked the AWS event stream with iter_bytes(chunk_size=1024). httpx's ByteChunker only releases full 1024-byte blocks, so the small early events (messageStart, contentBlockStart, first deltas) sat in the buffer until enough bytes accumulated, pushing time-to-first-byte from ~1.4s to ~8.5s on buffered tool-use streams. The default is now no re-chunking; an explicit stream_chunk_size is still honored. * test(bedrock): cover explicit stream_chunk_size on sync invoke path * test(bedrock): cover stream_chunk_size plumbing through converse completion * test(bedrock): cover stream_chunk_size default in legacy BedrockLLM streaming * test(bedrock): merge converse handler tests into existing mapped test file pytest imports test modules by basename in non-package test dirs, so the new tests/test_litellm/llms/bedrock/chat/test_converse_handler.py collided with the pre-existing tests/test_litellm/llms/chat/test_converse_handler.py and broke collection in CI. Move the new tests into the existing file
Read the StandardLoggingPayload cost_breakdown into a typed LLMCost on LLMCallSpanData and emit each component under litellm.cost.* (absent components omitted, so spans stay sparse). Stamp litellm.__version__ as the instrumentation scope version so every v2 span carries a deterministic scope.version. Tests under tests/test_litellm/integrations/otel/.
…t (opt-in) (#30223) * fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in) On the non-streaming path, base_process_llm_request awaited the LLM call with no disconnect monitoring; when the HTTP client went away the upstream request kept running until completion or request_timeout (6000s default), holding a backend slot (e.g. a vLLM GPU slot) for output nobody would read Add an opt-in general_settings.cancel_on_disconnect flag, default off, so the default code path is unchanged. When enabled, a receive-based watcher task observes http.disconnect and cancels the asyncio.gather driving the upstream call. The resulting CancelledError is converted to HTTPException 499 only when the disconnect event is set, so server-initiated cancellations still propagate as-is. The 499 then flows through _handle_llm_api_exception like any other failure, meaning post_call_failure_hook still releases max_parallel_requests slots and fires spend and alerting callbacks; it is logged at info level instead of a full traceback Also removes the dead check_request_disconnection helper in proxy_server.py (zero call sites) along with its behavior-pin tests Builds on the receive-based design from #25776 Addresses #13774. Re-fixes #22805 (regressed after the #14295 revert) Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> * fix(proxy): scope 499 quiet logging to disconnects and harden watcher Address the two P2 findings from the Greptile review on #30223. The info-level logging in _log_llm_api_exception now applies only to the disconnect-specific HTTPException (status 499 plus the shared _CLIENT_DISCONNECT_DETAIL message), so any other 499 raised by hooks or guardrails keeps its full traceback. The disconnect watcher now catches exceptions from request.receive() (e.g. a transport reset) and logs a warning instead of dying silently, making the degradation to no-op visible; a test pins that the LLM call is not cancelled in that case --------- Co-authored-by: kursad <kursad.lacin@brado.net> Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com>
…or claude_platform (#30200) (#30205) The inline STS session policy passed to assume_role_with_web_identity acts as an IAM PERMISSION CEILING — effective permissions are the intersection of the role's identity policies and this policy. Any action not listed is silently denied even when the IAM role grants it. #27678 added the bedrock/claude_platform/<model> route but its service-side action namespace is aws-external-anthropic:*, not bedrock:*. Without a matching statement here, every claude_platform request via OIDC (GCP federation, EKS Pod Identity webhook, etc.) 403s with 'no session policy allows the aws-external-anthropic:CreateInference action' — even with a fully permissive identity policy. Add a second ClaudePlatformLiteLLM statement covering CreateInference, CreateBatchInference, CancelBatchInference, DeleteBatchInference, CountTokens, Get*, List*. Keep aws:SecureTransport=true parity with the bedrock statement. Static creds + IRSA flow through different code paths and are not affected. Fixes #30200
…ses (#30098) * Set Retry-After header on RouterRateLimitError responses When all deployments for a model are in cooldown, the proxy returns a 429 whose cooldown timing is only available by parsing the error message string. RouterRateLimitError already carries cooldown_time, so expose it as a standard retry-after header in _handle_llm_api_exception. The value is rounded up so clients never retry before the cooldown window ends. Fixes #27823. * Set Retry-After after response-headers hook so cooldown wins The cooldown-derived retry-after was assigned before the post_call_response_headers_hook merge, so a callback returning a retry-after key (including a stale or empty value) silently clobbered it. Move the RouterRateLimitError block after the callback merge so the cooldown value is authoritative for this error type.
) * fix(router): route aspeech through async_function_with_fallbacks Router.aspeech selected a deployment and awaited litellm.aspeech directly, so TTS requests got no retry on failure and no failover to backup deployments; the except block only fired an exception alert and re-raised. Every other router endpoint (acompletion, aembedding, atranscription, arerank) already delegates to async_function_with_fallbacks Mirror the atranscription pattern: move deployment selection and the litellm.aspeech call into a private _aspeech method, then have the public aspeech set kwargs["original_function"] = self._aspeech and await self.async_function_with_fallbacks(**kwargs). _aspeech also picks up the shared _get_async_openai_model_client helper and the same total/success/fail call accounting the sibling endpoints use Fixes #27778. * fix(router): apply deployment kwargs and rpm semaphore in _aspeech Bring _aspeech fully in line with _atranscription: call _update_kwargs_with_deployment so deployment metadata, model_info, timeout, and default litellm params flow into the request, and wrap the litellm.aspeech call with the max_parallel_requests semaphore plus async_routing_strategy_pre_call_checks so TTS respects rpm limits the same way the other router endpoints do Also add a unit test that exercises _aspeech directly and asserts the deployment metadata reaches the underlying call
…equests below the alerting threshold (#30106) * fix(slack_alerting): skip hanging request alerts below the threshold The hanging request check alerted on any cached request whose completion status was not yet recorded, with no minimum age check. Since the background loop runs every alerting_threshold / 2 seconds, any request that happened to be in flight at a check fired a "hanging - Ns+ request time" alert even if it was only seconds old, producing a steady stream of false positives. Add a created_at timestamp to HangingRequestData, stamped when the request enters the hanging request cache, and skip requests younger than alerting_threshold without evicting them, so a later check can still alert if they never complete. Extend the cache TTL from threshold + 60s to 1.5x threshold + 60s; with the age check, entries only become alertable after threshold seconds, and the check period is threshold / 2, so the old TTL could evict a genuinely hanging request before any check saw it cross the threshold. Fixes #27855. * fix(slack_alerting): alert once per hanging request The min-age gate stops false positives for young in-flight requests, but a genuinely hanging request still re-alerted on every checker tick within the cache TTL. With the wider TTL (1.5x threshold + 60s) that is 1-2 extra Slack notifications per stuck request at the default 600s threshold. Flag a HangingRequestData entry as alerted once its alert fires and skip flagged entries on later ticks, so each hang produces exactly one alert. The cache reference is mutated in place, so the TTL is untouched and still handles cleanup. Adds a regression test asserting one alert across multiple ticks. Fixes #27855.
…30087) * fix(health): treat all-proxy-models keys as unrestricted in /health A key granted all model permissions stores the literal "all-proxy-models" marker in its models list. The /health access filter compared that marker against real model_names, so the model list filtered down to nothing and the WebUI health check returned healthy_count=0, unhealthy_count=0 with HTTP 503. Skip the filter (both the live path and the background-cache model_id scoping) when the marker is present, matching how auth_checks treats SpecialModelNames.all_proxy_models. Fixes #29744. * fix(health): resolve all-team-models sentinel to the team allowlist Same failure shape as the all-proxy-models case: a key carrying the literal "all-team-models" entry matches no real model_name, so the /health access filter would zero out the model list. Resolve the sentinel to the key's team models when team_id is set, matching get_key_models in model_checks.py. Without a team_id the sentinel stays unresolved and matches nothing, denying rather than widening access, mirroring _resolve_key_models_for_auth_check.
* feat(proxy): auto-enable drop_params for Claude Code requests Claude Code identifies itself with a claude-cli/<version> user agent and sends Anthropic-specific params (top_k, thinking, etc.) on every request. When the proxy routes those requests to a non-Anthropic provider, the unsupported params fail the call unless drop_params is configured. Detect the Claude Code user agent in add_litellm_data_to_request and default drop_params to true for those requests, without overriding an explicit drop_params value sent by the caller. * feat(proxy): respect operator litellm_settings drop_params over Claude Code default An explicit drop_params in the operator's litellm_settings (true or false) now suppresses the Claude Code user agent default, so an operator who deliberately configured drop_params: false keeps strict param validation for Claude Code clients too. The auto-default only fills the gap when neither the request body nor the config sets a value.
…ude models (#29964) * fix(snowflake): migrate to native Cortex REST API endpoints Replaces the legacy /api/v2/cortex/inference:complete endpoint with the native OpenAI-compatible /api/v2/cortex/v1/chat/completions endpoint, fixing error 390142 (Incoming request does not contain a valid payload) when using model: snowflake/<model> in LiteLLM proxy. Changes: - litellm/llms/snowflake/chat/transformation.py: route to native /cortex/v1/chat/completions, remove Snowflake-specific tool_spec payload transformation, remove content_list response handling, add stream to supported params - litellm/llms/snowflake/anthropic/transformation.py (new): SnowflakeCortexAnthropicConfig routes Claude models to /cortex/v1/messages with anthropic-version header and Anthropic->OpenAI response transform - tests: 29 unit tests covering URL routing, auth headers, payload format, and response parsing * fix(snowflake): map max_tokens to max_completion_tokens for native endpoint * fix: handle multi-turn tool conversations and OpenAI→Anthropic tool format conversion - _extract_system_and_messages now preserves tool_calls from assistant messages and converts them to Anthropic tool_use content blocks - tool role messages are converted to user role with tool_result content blocks (as required by Anthropic Messages API) - Added _transform_tools_to_anthropic() to convert OpenAI tool format (type/function/parameters) to Anthropic format (name/input_schema) - Added comprehensive tests for multi-turn tool conversations Addresses review feedback on PR #29964 * test: add coverage for malformed JSON and non-string tool arguments * fix(tests): update chat transformation tests for native OpenAI-compatible endpoint * style: apply black formatting * fix: resolve mypy type errors in anthropic transformation * fix: correct mypy type: ignore error codes (attr-defined) * fix: use max_tokens instead of max_completion_tokens for Snowflake endpoint compatibility * refactor: merge Anthropic config into unified SnowflakeConfig with auto-routing - Remove separate SnowflakeCortexAnthropicConfig and anthropic/ directory - SnowflakeConfig now auto-routes based on model name: - Claude models → /messages endpoint (Anthropic format) - All others → /chat/completions endpoint (OpenAI format) - No new provider needed (stays as SNOWFLAKE = 'snowflake') - Tool message transformation for Claude: tool_calls → tool_use blocks, tool role → user with tool_result - OpenAI → Anthropic tool format conversion (parameters → input_schema) - Addresses Greptile feedback about unwired SnowflakeCortexAnthropicConfig * fix: use max_completion_tokens for /chat/completions (Snowflake deprecated max_tokens on this endpoint) * fix(tests): update assertions for Claude auto-routing to /messages endpoint * fix(snowflake): add tool_choice conversion and preserve max_completion_tokens in Anthropic path * fix(snowflake): use ChatCompletionMessageToolCall objects and strip model prefix on OpenAI path * fix(snowflake): collect multiple system messages to prevent guardrail override * chore: remove committed .pyc files and add __pycache__ to .gitignore * fix: remove unused Union import * fix: restore original .gitignore (accidentally replaced in earlier commit) * feat(snowflake): add streaming response handler for both Anthropic and OpenAI SSE formats * fix: remove unused AsyncIterator and Iterator imports * fix: add missing total_tokens to ChatCompletionUsageBlock * fix(snowflake): coalesce consecutive tool results into single user message for Anthropic * fix(snowflake): handle message_start event for streaming input_tokens tracking
* fix: evict last deleted model in multi-instance deployments _delete_deployment had an early return when db_models was empty, preventing eviction of the last deleted model during reconciliation. - Remove len(db_models)==0 early return from _delete_deployment - Return None (not []) from _get_models_from_db on DB failure so callers can distinguish a transient failure from a genuinely empty DB - Guard _update_llm_router against None to skip updates on DB failure Fixes #28443 * test: remove dead MagicMock assignment in type_mismatch test * fix: update test to pass [] not None to _update_llm_router test_ProxyConfig__update_llm_router_bad_proxy_logging_raises was passing None as new_models to get through to the proxy_logging_obj check, but the None guard we added now returns early before reaching that path. Pass [] instead so the test exercises the intended AttributeError case. Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> * chore: regenerate API types to sync schema.d.ts with proxy OpenAPI spec Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> --------- Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com>
* fix: set Redis spend counter to reset_to value on /key/reset_spend Previously, the Redis spend counter was always set to 0.0 after a reset, even when reset_to was a non-zero value (partial reset). This caused the budget to be under-enforced for up to 60 seconds until the counter expired and fell through to the DB. Now the counter is set to the actual reset_to value, so partial resets are reflected correctly and budget enforcement is consistent. * test: update reset_key_spend test to match direct cache set The implementation now sets spend_counter_cache directly instead of calling _invalidate_spend_counter. Update the test to verify the in_memory_cache.set_cache call with the correct key, value, and ttl. --------- Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>
* fix: Add embeddings support for Scaleway provider * fix: resolve merge conflicts
…30196) * fix(main): clarify backend route handling for Swagger static assets * fix(allowlist): add BACKEND_MOUNT_PATHS for Swagger static assets
* fix(voyage): route multimodal embeddings to correct endpoint * test(voyage): cover multimodal embedding edge cases * test(voyage): cover api key fallback * fix(voyage): raise early on missing api key and malformed image url * test(voyage): cover utils routing and helper * fix(voyage): route supported openai params for multimodal models * style: apply black formatting
* fix(ui): infer Azure API version from API base * fix(ui): address Azure API version feedback
Sameerlite
changed the title
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Sameerlite
changed the title
Contributor
Greptile SummaryThis is a large OSS staging roll-up (69 files) introducing several independent features and bug fixes across the proxy, router, integrations, and LLM providers.
Confidence Score: 5/5Safe to merge; all changed paths are well-tested with mocks, and the one behaviour change that silently affects existing Claude Code users is low-risk because it only drops unsupported params rather than corrupting results. The changes are well-scoped with unit test coverage. The litellm/proxy/litellm_pre_call_utils.py — the new
|
| Filename | Overview |
|---|---|
| litellm/proxy/common_request_processing.py | Replaces polling-based check_request_disconnection with event-driven ASGI disconnect detection; adds retry-after header for router rate-limit errors. Both changes are well-structured. |
| litellm/proxy/litellm_pre_call_utils.py | Adds auto-drop-params for Claude Code user-agent requests — a backwards-incompatible silent behaviour change without a dedicated opt-out flag. |
| litellm/proxy/management_endpoints/team_endpoints.py | Replaces Python-level model merge with an atomic PostgreSQL execute_raw array append; follow-up update() is write-routed to avoid stale reads. Logic is correct for the PostgreSQL-only production setup. |
| litellm/proxy/management_endpoints/key_management_endpoints.py | Replaces _invalidate_spend_counter with a direct set_cache to reset_to value; correctly updates both in-memory and Redis caches with a 60 s TTL. |
| litellm/router.py | Refactors aspeech to split outer fallback wrapper (aspeech) and inner deployment handler (_aspeech), matching the pattern used for acompletion. RPM semaphore and pre-call checks are correctly handled. |
| litellm/integrations/datadog/datadog_team_handler.py | New DataDogHandler for team-scoped DD credentials, following the LangFuse per-team pattern. allow_env_credentials correctly prevents the proxy's global DD_API_KEY from being forwarded to team-controlled hosts. |
| litellm/integrations/datadog/datadog.py | Adds constructor kwargs for explicit credentials; credential resolution order (kwarg → env var) is consistent and the allow_env_credentials guard is applied correctly in both agent and direct-API paths. |
| litellm/litellm_core_utils/litellm_logging.py | Reorders process_dynamic_callbacks() after initialize_standard_callback_dynamic_params() so team DD credentials are available when callbacks are initialised. DD-specific init args are filtered correctly. |
| litellm/proxy/proxy_server.py | Removes the polling-based check_request_disconnection function; fixes _update_llm_router to guard against None from _get_models_from_db; removes the early-exit on len(db_models)==0 so legitimately empty DB evicts stale deployments. |
| litellm/llms/bedrock/base_aws_llm.py | Expands the OIDC session policy to include aws-external-anthropic:* actions required for the Claude Platform on AWS route; well-commented and correct. |
| litellm/integrations/SlackAlerting/hanging_request_check.py | Adds per-request age check and alerted flag to prevent repeated firing; extends cache TTL to 1.5× threshold so alerts aren't missed. Both fixes are correct. |
| litellm/llms/voyage/embedding/transformation_multimodal.py | New transformation class for Voyage multimodal embeddings; routes to /v1/multimodalembeddings, handles image_url → image_base64 normalisation, and maps dimensions → output_dimension. |
| litellm/integrations/otel/model/payloads.py | Adds LLMCost dataclass and wires it into LLMCallSpanData.cost from cost_breakdown; per-component costs are omitted (not zeroed) when absent, keeping spans sparse. |
| litellm/proxy/health_endpoints/_health_endpoints.py | Correctly handles SpecialModelNames.all_proxy_models sentinel so keys with unrestricted access don't get filtered to an empty model list on the /health endpoint. |
Reviews (6): Last reviewed commit: "fix(datadog): never fall back to env DD_..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Contributor
Author
Sameerlite
changed the title
Enable teams to configure their own Datadog credentials via
POST /team/{team_id}/callback, following the same pattern as Langfuse.
fix(proxy): atomic merge for team model aliases and team.models on BYOK create
Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com>
- Black-format snowflake/chat/transformation.py to fix lint failure - Update Anthropic config test to expect default max_tokens of 4096 (matches implementation) - Add AsyncMock + execute_raw mock to team_model_add cache-refresh pin test - Add model_dump mock and patch cache/logging in test_uses_atomic_array_append_with_dedup Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…yment logic _delete_deployment no longer short-circuits on empty db_models — it now treats [] as a valid empty-DB state and proceeds to check config models. Mock get_config to return the two router deployments so they appear in combined_id_list and are protected, which matches the real-world scenario where a DB error occurs but the models are config-backed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
Author
…nd config list (#30295) * feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list Follow-up to #30223 per maintainer review: documents the flag in ConfigGeneralSettings with a short description and adds it to allowed_args in get_config_list so the UI and /config/list expose it. A test pins that /config/list returns the field with type Boolean, which requires both registrations to be present * chore(ui): regenerate schema.d.ts for cancel_on_disconnect --------- Co-authored-by: kursad <kursad.lacin@brado.net>
Contributor
Author
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
Contributor
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 2 · PR risk: 0/10 |
…itellm_oss_staging_120626 # Conflicts: # tests/test_litellm/proxy/test_common_request_processing.py
Contributor
mateo-berri
changed the title
…estinations Team/key-scoped Datadog loggers could be pointed at an arbitrary dd_agent_host or dd_site while omitting dd_api_key, causing the proxy's global DD_API_KEY to be sent as the DD-API-KEY header to that destination. Gate the env-var fallback behind an allow_env_credentials flag, set to False when the destination is caller-supplied, mirroring the existing langfuse/langsmith pattern.
Contributor
3 tasks
michaelxer
added a commit
to michaelxer/litellm
that referenced
this pull request
Jun 17, 2026
* feat(bedrock): add bedrock mantle gemma 4 models (BerriAI#30264) * feat(bedrock): add bedrock mantle gemma 4 models * test(bedrock): harden mantle local cost fixture * feat(responses): enable the responses API for the Tensormesh provider (BerriAI#30209) * feat(responses): enable the responses API for the Tensormesh provider * Update litellm/llms/openai_like/providers.json Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(langfuse_otel): mark LLM spans as generations (BerriAI#30250) * fix(bedrock): stop stream_chunk_size leaking into invoke request bodies (BerriAI#30240) stream_chunk_size is a LiteLLM-internal knob for re-chunking the HTTP response stream. The invoke transformations splat optional_params into the provider request body without dropping it, and Bedrock rejects unknown fields, so any bedrock/invoke request that sets the parameter fails with ValidationException: stream_chunk_size: Extra inputs are not permitted. Drop it in the invoke dispatcher (covers cohere, titan, mistral, meta, ai21) and in the Claude messages-format request builder (the route used for bedrock/invoke Anthropic models) * fix(bedrock): stop buffering streamed tool-call argument deltas (BerriAI#30231) * fix(bedrock): stop buffering streamed tool-call argument deltas Two issues made Bedrock tool-use streaming arrive as a single end-of-stream burst through LiteLLM while plain text streamed fine. First, the anthropic-beta allowlist mapped fine-grained-tool-streaming-2025-05-14 to null for bedrock and bedrock_converse, so the header was silently stripped. Without that beta, Anthropic models on Bedrock buffer tool input server-side and emit all toolUse.input deltas at once (verified against converse-stream and invoke-with-response-stream directly). Bedrock accepts the beta via additionalModelRequestFields.anthropic_beta, so it is now forwarded. Second, the streaming reads re-chunked the AWS event stream with iter_bytes(chunk_size=1024). httpx's ByteChunker only releases full 1024-byte blocks, so the small early events (messageStart, contentBlockStart, first deltas) sat in the buffer until enough bytes accumulated, pushing time-to-first-byte from ~1.4s to ~8.5s on buffered tool-use streams. The default is now no re-chunking; an explicit stream_chunk_size is still honored. * test(bedrock): cover explicit stream_chunk_size on sync invoke path * test(bedrock): cover stream_chunk_size plumbing through converse completion * test(bedrock): cover stream_chunk_size default in legacy BedrockLLM streaming * test(bedrock): merge converse handler tests into existing mapped test file pytest imports test modules by basename in non-package test dirs, so the new tests/test_litellm/llms/bedrock/chat/test_converse_handler.py collided with the pre-existing tests/test_litellm/llms/chat/test_converse_handler.py and broke collection in CI. Move the new tests into the existing file * feat(otel): emit v2 cost breakdown + stamp tracer scope version (BerriAI#30156) Read the StandardLoggingPayload cost_breakdown into a typed LLMCost on LLMCallSpanData and emit each component under litellm.cost.* (absent components omitted, so spans stay sparse). Stamp litellm.__version__ as the instrumentation scope version so every v2 span carries a deterministic scope.version. Tests under tests/test_litellm/integrations/otel/. * fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in) (BerriAI#30223) * fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in) On the non-streaming path, base_process_llm_request awaited the LLM call with no disconnect monitoring; when the HTTP client went away the upstream request kept running until completion or request_timeout (6000s default), holding a backend slot (e.g. a vLLM GPU slot) for output nobody would read Add an opt-in general_settings.cancel_on_disconnect flag, default off, so the default code path is unchanged. When enabled, a receive-based watcher task observes http.disconnect and cancels the asyncio.gather driving the upstream call. The resulting CancelledError is converted to HTTPException 499 only when the disconnect event is set, so server-initiated cancellations still propagate as-is. The 499 then flows through _handle_llm_api_exception like any other failure, meaning post_call_failure_hook still releases max_parallel_requests slots and fires spend and alerting callbacks; it is logged at info level instead of a full traceback Also removes the dead check_request_disconnection helper in proxy_server.py (zero call sites) along with its behavior-pin tests Builds on the receive-based design from BerriAI#25776 Addresses BerriAI#13774. Re-fixes BerriAI#22805 (regressed after the BerriAI#14295 revert) Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> * fix(proxy): scope 499 quiet logging to disconnects and harden watcher Address the two P2 findings from the Greptile review on BerriAI#30223. The info-level logging in _log_llm_api_exception now applies only to the disconnect-specific HTTPException (status 499 plus the shared _CLIENT_DISCONNECT_DETAIL message), so any other 499 raised by hooks or guardrails keeps its full traceback. The disconnect watcher now catches exceptions from request.receive() (e.g. a transport reset) and logs a warning instead of dying silently, making the degradation to no-op visible; a test pins that the LLM call is not cancelled in that case --------- Co-authored-by: kursad <kursad.lacin@brado.net> Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> * fix(bedrock): grant aws-external-anthropic:* in OIDC session policy for claude_platform (BerriAI#30200) (BerriAI#30205) The inline STS session policy passed to assume_role_with_web_identity acts as an IAM PERMISSION CEILING — effective permissions are the intersection of the role's identity policies and this policy. Any action not listed is silently denied even when the IAM role grants it. BerriAI#27678 added the bedrock/claude_platform/<model> route but its service-side action namespace is aws-external-anthropic:*, not bedrock:*. Without a matching statement here, every claude_platform request via OIDC (GCP federation, EKS Pod Identity webhook, etc.) 403s with 'no session policy allows the aws-external-anthropic:CreateInference action' — even with a fully permissive identity policy. Add a second ClaudePlatformLiteLLM statement covering CreateInference, CreateBatchInference, CancelBatchInference, DeleteBatchInference, CountTokens, Get*, List*. Keep aws:SecureTransport=true parity with the bedrock statement. Static creds + IRSA flow through different code paths and are not affected. Fixes BerriAI#30200 * fix(proxy): set Retry-After header on RouterRateLimitError 429 responses (BerriAI#30098) * Set Retry-After header on RouterRateLimitError responses When all deployments for a model are in cooldown, the proxy returns a 429 whose cooldown timing is only available by parsing the error message string. RouterRateLimitError already carries cooldown_time, so expose it as a standard retry-after header in _handle_llm_api_exception. The value is rounded up so clients never retry before the cooldown window ends. Fixes BerriAI#27823. * Set Retry-After after response-headers hook so cooldown wins The cooldown-derived retry-after was assigned before the post_call_response_headers_hook merge, so a callback returning a retry-after key (including a stale or empty value) silently clobbered it. Move the RouterRateLimitError block after the callback merge so the cooldown value is authoritative for this error type. * fix(router): route aspeech through async_function_with_fallbacks (BerriAI#30104) * fix(router): route aspeech through async_function_with_fallbacks Router.aspeech selected a deployment and awaited litellm.aspeech directly, so TTS requests got no retry on failure and no failover to backup deployments; the except block only fired an exception alert and re-raised. Every other router endpoint (acompletion, aembedding, atranscription, arerank) already delegates to async_function_with_fallbacks Mirror the atranscription pattern: move deployment selection and the litellm.aspeech call into a private _aspeech method, then have the public aspeech set kwargs["original_function"] = self._aspeech and await self.async_function_with_fallbacks(**kwargs). _aspeech also picks up the shared _get_async_openai_model_client helper and the same total/success/fail call accounting the sibling endpoints use Fixes BerriAI#27778. * fix(router): apply deployment kwargs and rpm semaphore in _aspeech Bring _aspeech fully in line with _atranscription: call _update_kwargs_with_deployment so deployment metadata, model_info, timeout, and default litellm params flow into the request, and wrap the litellm.aspeech call with the max_parallel_requests semaphore plus async_routing_strategy_pre_call_checks so TTS respects rpm limits the same way the other router endpoints do Also add a unit test that exercises _aspeech directly and asserts the deployment metadata reaches the underlying call * fix(slack_alerting): stop false-positive hanging request alerts for requests below the alerting threshold (BerriAI#30106) * fix(slack_alerting): skip hanging request alerts below the threshold The hanging request check alerted on any cached request whose completion status was not yet recorded, with no minimum age check. Since the background loop runs every alerting_threshold / 2 seconds, any request that happened to be in flight at a check fired a "hanging - Ns+ request time" alert even if it was only seconds old, producing a steady stream of false positives. Add a created_at timestamp to HangingRequestData, stamped when the request enters the hanging request cache, and skip requests younger than alerting_threshold without evicting them, so a later check can still alert if they never complete. Extend the cache TTL from threshold + 60s to 1.5x threshold + 60s; with the age check, entries only become alertable after threshold seconds, and the check period is threshold / 2, so the old TTL could evict a genuinely hanging request before any check saw it cross the threshold. Fixes BerriAI#27855. * fix(slack_alerting): alert once per hanging request The min-age gate stops false positives for young in-flight requests, but a genuinely hanging request still re-alerted on every checker tick within the cache TTL. With the wider TTL (1.5x threshold + 60s) that is 1-2 extra Slack notifications per stuck request at the default 600s threshold. Flag a HangingRequestData entry as alerted once its alert fires and skip flagged entries on later ticks, so each hang produces exactly one alert. The cache reference is mutated in place, so the TTL is untouched and still handles cleanup. Adds a regression test asserting one alert across multiple ticks. Fixes BerriAI#27855. * fix(health): treat all-proxy-models keys as unrestricted in /health (BerriAI#30087) * fix(health): treat all-proxy-models keys as unrestricted in /health A key granted all model permissions stores the literal "all-proxy-models" marker in its models list. The /health access filter compared that marker against real model_names, so the model list filtered down to nothing and the WebUI health check returned healthy_count=0, unhealthy_count=0 with HTTP 503. Skip the filter (both the live path and the background-cache model_id scoping) when the marker is present, matching how auth_checks treats SpecialModelNames.all_proxy_models. Fixes BerriAI#29744. * fix(health): resolve all-team-models sentinel to the team allowlist Same failure shape as the all-proxy-models case: a key carrying the literal "all-team-models" entry matches no real model_name, so the /health access filter would zero out the model list. Resolve the sentinel to the key's team models when team_id is set, matching get_key_models in model_checks.py. Without a team_id the sentinel stays unresolved and matches nothing, denying rather than widening access, mirroring _resolve_key_models_for_auth_check. * feat(proxy): auto-enable drop_params for Claude Code requests (BerriAI#30218) * feat(proxy): auto-enable drop_params for Claude Code requests Claude Code identifies itself with a claude-cli/<version> user agent and sends Anthropic-specific params (top_k, thinking, etc.) on every request. When the proxy routes those requests to a non-Anthropic provider, the unsupported params fail the call unless drop_params is configured. Detect the Claude Code user agent in add_litellm_data_to_request and default drop_params to true for those requests, without overriding an explicit drop_params value sent by the caller. * feat(proxy): respect operator litellm_settings drop_params over Claude Code default An explicit drop_params in the operator's litellm_settings (true or false) now suppresses the Claude Code user agent default, so an operator who deliberately configured drop_params: false keeps strict param validation for Claude Code clients too. The auto-default only fills the gap when neither the request body nor the config sets a value. * fix(snowflake): migrate to native endpoints with auto-routing for Claude models (BerriAI#29964) * fix(snowflake): migrate to native Cortex REST API endpoints Replaces the legacy /api/v2/cortex/inference:complete endpoint with the native OpenAI-compatible /api/v2/cortex/v1/chat/completions endpoint, fixing error 390142 (Incoming request does not contain a valid payload) when using model: snowflake/<model> in LiteLLM proxy. Changes: - litellm/llms/snowflake/chat/transformation.py: route to native /cortex/v1/chat/completions, remove Snowflake-specific tool_spec payload transformation, remove content_list response handling, add stream to supported params - litellm/llms/snowflake/anthropic/transformation.py (new): SnowflakeCortexAnthropicConfig routes Claude models to /cortex/v1/messages with anthropic-version header and Anthropic->OpenAI response transform - tests: 29 unit tests covering URL routing, auth headers, payload format, and response parsing * fix(snowflake): map max_tokens to max_completion_tokens for native endpoint * fix: handle multi-turn tool conversations and OpenAI→Anthropic tool format conversion - _extract_system_and_messages now preserves tool_calls from assistant messages and converts them to Anthropic tool_use content blocks - tool role messages are converted to user role with tool_result content blocks (as required by Anthropic Messages API) - Added _transform_tools_to_anthropic() to convert OpenAI tool format (type/function/parameters) to Anthropic format (name/input_schema) - Added comprehensive tests for multi-turn tool conversations Addresses review feedback on PR BerriAI#29964 * test: add coverage for malformed JSON and non-string tool arguments * fix(tests): update chat transformation tests for native OpenAI-compatible endpoint * style: apply black formatting * fix: resolve mypy type errors in anthropic transformation * fix: correct mypy type: ignore error codes (attr-defined) * fix: use max_tokens instead of max_completion_tokens for Snowflake endpoint compatibility * refactor: merge Anthropic config into unified SnowflakeConfig with auto-routing - Remove separate SnowflakeCortexAnthropicConfig and anthropic/ directory - SnowflakeConfig now auto-routes based on model name: - Claude models → /messages endpoint (Anthropic format) - All others → /chat/completions endpoint (OpenAI format) - No new provider needed (stays as SNOWFLAKE = 'snowflake') - Tool message transformation for Claude: tool_calls → tool_use blocks, tool role → user with tool_result - OpenAI → Anthropic tool format conversion (parameters → input_schema) - Addresses Greptile feedback about unwired SnowflakeCortexAnthropicConfig * fix: use max_completion_tokens for /chat/completions (Snowflake deprecated max_tokens on this endpoint) * fix(tests): update assertions for Claude auto-routing to /messages endpoint * fix(snowflake): add tool_choice conversion and preserve max_completion_tokens in Anthropic path * fix(snowflake): use ChatCompletionMessageToolCall objects and strip model prefix on OpenAI path * fix(snowflake): collect multiple system messages to prevent guardrail override * chore: remove committed .pyc files and add __pycache__ to .gitignore * fix: remove unused Union import * fix: restore original .gitignore (accidentally replaced in earlier commit) * feat(snowflake): add streaming response handler for both Anthropic and OpenAI SSE formats * fix: remove unused AsyncIterator and Iterator imports * fix: add missing total_tokens to ChatCompletionUsageBlock * fix(snowflake): coalesce consecutive tool results into single user message for Anthropic * fix(snowflake): handle message_start event for streaming input_tokens tracking * fix: evict last deleted model in multi-instance deployments (BerriAI#28608) * fix: evict last deleted model in multi-instance deployments _delete_deployment had an early return when db_models was empty, preventing eviction of the last deleted model during reconciliation. - Remove len(db_models)==0 early return from _delete_deployment - Return None (not []) from _get_models_from_db on DB failure so callers can distinguish a transient failure from a genuinely empty DB - Guard _update_llm_router against None to skip updates on DB failure Fixes BerriAI#28443 * test: remove dead MagicMock assignment in type_mismatch test * fix: update test to pass [] not None to _update_llm_router test_ProxyConfig__update_llm_router_bad_proxy_logging_raises was passing None as new_models to get through to the proxy_logging_obj check, but the None guard we added now returns early before reaching that path. Pass [] instead so the test exercises the intended AttributeError case. Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> * chore: regenerate API types to sync schema.d.ts with proxy OpenAPI spec Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> --------- Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> * fix: invalidate Redis spend counter on /key/reset_spend (BerriAI#29694) * fix: set Redis spend counter to reset_to value on /key/reset_spend Previously, the Redis spend counter was always set to 0.0 after a reset, even when reset_to was a non-zero value (partial reset). This caused the budget to be under-enforced for up to 60 seconds until the counter expired and fell through to the DB. Now the counter is set to the actual reset_to value, so partial resets are reflected correctly and budget enforcement is consistent. * test: update reset_key_spend test to match direct cache set The implementation now sets spend_counter_cache directly instead of calling _invalidate_spend_counter. Update the test to verify the in_memory_cache.set_cache call with the correct key, value, and ttl. --------- Co-authored-by: michaelxer <michaelxer@users.noreply.github.com> * fix: add scaleway models pricing (BerriAI#27659) * fix: Add embeddings support for Scaleway provider * fix: resolve merge conflicts * fix(main): clarify backend route handling for Swagger static assets (BerriAI#30196) * fix(main): clarify backend route handling for Swagger static assets * fix(allowlist): add BACKEND_MOUNT_PATHS for Swagger static assets * fix(voyage): route multimodal embeddings to correct endpoint (BerriAI#30193) * fix(voyage): route multimodal embeddings to correct endpoint * test(voyage): cover multimodal embedding edge cases * test(voyage): cover api key fallback * fix(voyage): raise early on missing api key and malformed image url * test(voyage): cover utils routing and helper * fix(voyage): route supported openai params for multimodal models * style: apply black formatting * fix(ui): infer Azure API version from API base (BerriAI#30204) * fix(ui): infer Azure API version from API base * fix(ui): address Azure API version feedback * Update litellm/llms/snowflake/chat/transformation.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * feat(datadog): add team-scoped Datadog callback support (BerriAI#29947) Enable teams to configure their own Datadog credentials via POST /team/{team_id}/callback, following the same pattern as Langfuse. * Merge pull request BerriAI#29528 from aanchal22/litellm_byok-alias-merge fix(proxy): atomic merge for team model aliases and team.models on BYOK create * feat: add EmpirioLabs as an OpenAI-compatible provider (BerriAI#30278) Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com> * fix: resolve failing tests and lint in snowflake/team endpoints - Black-format snowflake/chat/transformation.py to fix lint failure - Update Anthropic config test to expect default max_tokens of 4096 (matches implementation) - Add AsyncMock + execute_raw mock to team_model_add cache-refresh pin test - Add model_dump mock and patch cache/logging in test_uses_atomic_array_append_with_dedup Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(test): update test_db_error_new_model_check for new _delete_deployment logic _delete_deployment no longer short-circuits on empty db_models — it now treats [] as a valid empty-DB state and proceeds to check config models. Mock get_config to return the two router deployments so they appear in combined_id_list and are protected, which matches the real-world scenario where a DB error occurs but the models are config-backed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list (BerriAI#30295) * feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list Follow-up to BerriAI#30223 per maintainer review: documents the flag in ConfigGeneralSettings with a short description and adds it to allowed_args in get_config_list so the UI and /config/list expose it. A test pins that /config/list returns the field with type Boolean, which requires both registrations to be present * chore(ui): regenerate schema.d.ts for cancel_on_disconnect --------- Co-authored-by: kursad <kursad.lacin@brado.net> * fix(datadog): never fall back to env DD_API_KEY for caller-supplied destinations Team/key-scoped Datadog loggers could be pointed at an arbitrary dd_agent_host or dd_site while omitting dd_api_key, causing the proxy's global DD_API_KEY to be sent as the DD-API-KEY header to that destination. Gate the env-var fallback behind an allow_env_credentials flag, set to False when the destination is caller-supplied, mirroring the existing langfuse/langsmith pattern. --------- Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: daitran-tensormesh <dai@tensormesh.ai> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Muspi Merol <me@promplate.dev> Co-authored-by: fangkang <fangkangm@gmail.com> Co-authored-by: Chris Hoogeboom <chris.hoogeboom@gmail.com> Co-authored-by: kursadlacin <kursadlacin@gmail.com> Co-authored-by: kursad <kursad.lacin@brado.net> Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> Co-authored-by: hcl <chenglunhu@gmail.com> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: sfc-gh-nashukla <navnit.shukla@snowflake.com> Co-authored-by: Rudra Dudhat <contact.rdudhat@gmail.com> Co-authored-by: Michael <52305679+michaelxer@users.noreply.github.com> Co-authored-by: michaelxer <michaelxer@users.noreply.github.com> Co-authored-by: Quentin Champenois <26109239+Quentinchampenois@users.noreply.github.com> Co-authored-by: mauriceberentsen <mauriceberentsen@live.nl> Co-authored-by: lost9999 <56498264+lost9999@users.noreply.github.com> Co-authored-by: GaetanVDB07 <86427581+GaetanVDB07@users.noreply.github.com> Co-authored-by: Aanchal Khandelwal <aan2210khandelwal@gmail.com> Co-authored-by: Adam Dalloul <adam_dalloul@icloud.com> Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
michaelxer
added a commit
to michaelxer/litellm
that referenced
this pull request
Jun 17, 2026
* feat(bedrock): add bedrock mantle gemma 4 models (BerriAI#30264) * feat(bedrock): add bedrock mantle gemma 4 models * test(bedrock): harden mantle local cost fixture * feat(responses): enable the responses API for the Tensormesh provider (BerriAI#30209) * feat(responses): enable the responses API for the Tensormesh provider * Update litellm/llms/openai_like/providers.json Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(langfuse_otel): mark LLM spans as generations (BerriAI#30250) * fix(bedrock): stop stream_chunk_size leaking into invoke request bodies (BerriAI#30240) stream_chunk_size is a LiteLLM-internal knob for re-chunking the HTTP response stream. The invoke transformations splat optional_params into the provider request body without dropping it, and Bedrock rejects unknown fields, so any bedrock/invoke request that sets the parameter fails with ValidationException: stream_chunk_size: Extra inputs are not permitted. Drop it in the invoke dispatcher (covers cohere, titan, mistral, meta, ai21) and in the Claude messages-format request builder (the route used for bedrock/invoke Anthropic models) * fix(bedrock): stop buffering streamed tool-call argument deltas (BerriAI#30231) * fix(bedrock): stop buffering streamed tool-call argument deltas Two issues made Bedrock tool-use streaming arrive as a single end-of-stream burst through LiteLLM while plain text streamed fine. First, the anthropic-beta allowlist mapped fine-grained-tool-streaming-2025-05-14 to null for bedrock and bedrock_converse, so the header was silently stripped. Without that beta, Anthropic models on Bedrock buffer tool input server-side and emit all toolUse.input deltas at once (verified against converse-stream and invoke-with-response-stream directly). Bedrock accepts the beta via additionalModelRequestFields.anthropic_beta, so it is now forwarded. Second, the streaming reads re-chunked the AWS event stream with iter_bytes(chunk_size=1024). httpx's ByteChunker only releases full 1024-byte blocks, so the small early events (messageStart, contentBlockStart, first deltas) sat in the buffer until enough bytes accumulated, pushing time-to-first-byte from ~1.4s to ~8.5s on buffered tool-use streams. The default is now no re-chunking; an explicit stream_chunk_size is still honored. * test(bedrock): cover explicit stream_chunk_size on sync invoke path * test(bedrock): cover stream_chunk_size plumbing through converse completion * test(bedrock): cover stream_chunk_size default in legacy BedrockLLM streaming * test(bedrock): merge converse handler tests into existing mapped test file pytest imports test modules by basename in non-package test dirs, so the new tests/test_litellm/llms/bedrock/chat/test_converse_handler.py collided with the pre-existing tests/test_litellm/llms/chat/test_converse_handler.py and broke collection in CI. Move the new tests into the existing file * feat(otel): emit v2 cost breakdown + stamp tracer scope version (BerriAI#30156) Read the StandardLoggingPayload cost_breakdown into a typed LLMCost on LLMCallSpanData and emit each component under litellm.cost.* (absent components omitted, so spans stay sparse). Stamp litellm.__version__ as the instrumentation scope version so every v2 span carries a deterministic scope.version. Tests under tests/test_litellm/integrations/otel/. * fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in) (BerriAI#30223) * fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in) On the non-streaming path, base_process_llm_request awaited the LLM call with no disconnect monitoring; when the HTTP client went away the upstream request kept running until completion or request_timeout (6000s default), holding a backend slot (e.g. a vLLM GPU slot) for output nobody would read Add an opt-in general_settings.cancel_on_disconnect flag, default off, so the default code path is unchanged. When enabled, a receive-based watcher task observes http.disconnect and cancels the asyncio.gather driving the upstream call. The resulting CancelledError is converted to HTTPException 499 only when the disconnect event is set, so server-initiated cancellations still propagate as-is. The 499 then flows through _handle_llm_api_exception like any other failure, meaning post_call_failure_hook still releases max_parallel_requests slots and fires spend and alerting callbacks; it is logged at info level instead of a full traceback Also removes the dead check_request_disconnection helper in proxy_server.py (zero call sites) along with its behavior-pin tests Builds on the receive-based design from BerriAI#25776 Addresses BerriAI#13774. Re-fixes BerriAI#22805 (regressed after the BerriAI#14295 revert) Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> * fix(proxy): scope 499 quiet logging to disconnects and harden watcher Address the two P2 findings from the Greptile review on BerriAI#30223. The info-level logging in _log_llm_api_exception now applies only to the disconnect-specific HTTPException (status 499 plus the shared _CLIENT_DISCONNECT_DETAIL message), so any other 499 raised by hooks or guardrails keeps its full traceback. The disconnect watcher now catches exceptions from request.receive() (e.g. a transport reset) and logs a warning instead of dying silently, making the degradation to no-op visible; a test pins that the LLM call is not cancelled in that case --------- Co-authored-by: kursad <kursad.lacin@brado.net> Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> * fix(bedrock): grant aws-external-anthropic:* in OIDC session policy for claude_platform (BerriAI#30200) (BerriAI#30205) The inline STS session policy passed to assume_role_with_web_identity acts as an IAM PERMISSION CEILING — effective permissions are the intersection of the role's identity policies and this policy. Any action not listed is silently denied even when the IAM role grants it. BerriAI#27678 added the bedrock/claude_platform/<model> route but its service-side action namespace is aws-external-anthropic:*, not bedrock:*. Without a matching statement here, every claude_platform request via OIDC (GCP federation, EKS Pod Identity webhook, etc.) 403s with 'no session policy allows the aws-external-anthropic:CreateInference action' — even with a fully permissive identity policy. Add a second ClaudePlatformLiteLLM statement covering CreateInference, CreateBatchInference, CancelBatchInference, DeleteBatchInference, CountTokens, Get*, List*. Keep aws:SecureTransport=true parity with the bedrock statement. Static creds + IRSA flow through different code paths and are not affected. Fixes BerriAI#30200 * fix(proxy): set Retry-After header on RouterRateLimitError 429 responses (BerriAI#30098) * Set Retry-After header on RouterRateLimitError responses When all deployments for a model are in cooldown, the proxy returns a 429 whose cooldown timing is only available by parsing the error message string. RouterRateLimitError already carries cooldown_time, so expose it as a standard retry-after header in _handle_llm_api_exception. The value is rounded up so clients never retry before the cooldown window ends. Fixes BerriAI#27823. * Set Retry-After after response-headers hook so cooldown wins The cooldown-derived retry-after was assigned before the post_call_response_headers_hook merge, so a callback returning a retry-after key (including a stale or empty value) silently clobbered it. Move the RouterRateLimitError block after the callback merge so the cooldown value is authoritative for this error type. * fix(router): route aspeech through async_function_with_fallbacks (BerriAI#30104) * fix(router): route aspeech through async_function_with_fallbacks Router.aspeech selected a deployment and awaited litellm.aspeech directly, so TTS requests got no retry on failure and no failover to backup deployments; the except block only fired an exception alert and re-raised. Every other router endpoint (acompletion, aembedding, atranscription, arerank) already delegates to async_function_with_fallbacks Mirror the atranscription pattern: move deployment selection and the litellm.aspeech call into a private _aspeech method, then have the public aspeech set kwargs["original_function"] = self._aspeech and await self.async_function_with_fallbacks(**kwargs). _aspeech also picks up the shared _get_async_openai_model_client helper and the same total/success/fail call accounting the sibling endpoints use Fixes BerriAI#27778. * fix(router): apply deployment kwargs and rpm semaphore in _aspeech Bring _aspeech fully in line with _atranscription: call _update_kwargs_with_deployment so deployment metadata, model_info, timeout, and default litellm params flow into the request, and wrap the litellm.aspeech call with the max_parallel_requests semaphore plus async_routing_strategy_pre_call_checks so TTS respects rpm limits the same way the other router endpoints do Also add a unit test that exercises _aspeech directly and asserts the deployment metadata reaches the underlying call * fix(slack_alerting): stop false-positive hanging request alerts for requests below the alerting threshold (BerriAI#30106) * fix(slack_alerting): skip hanging request alerts below the threshold The hanging request check alerted on any cached request whose completion status was not yet recorded, with no minimum age check. Since the background loop runs every alerting_threshold / 2 seconds, any request that happened to be in flight at a check fired a "hanging - Ns+ request time" alert even if it was only seconds old, producing a steady stream of false positives. Add a created_at timestamp to HangingRequestData, stamped when the request enters the hanging request cache, and skip requests younger than alerting_threshold without evicting them, so a later check can still alert if they never complete. Extend the cache TTL from threshold + 60s to 1.5x threshold + 60s; with the age check, entries only become alertable after threshold seconds, and the check period is threshold / 2, so the old TTL could evict a genuinely hanging request before any check saw it cross the threshold. Fixes BerriAI#27855. * fix(slack_alerting): alert once per hanging request The min-age gate stops false positives for young in-flight requests, but a genuinely hanging request still re-alerted on every checker tick within the cache TTL. With the wider TTL (1.5x threshold + 60s) that is 1-2 extra Slack notifications per stuck request at the default 600s threshold. Flag a HangingRequestData entry as alerted once its alert fires and skip flagged entries on later ticks, so each hang produces exactly one alert. The cache reference is mutated in place, so the TTL is untouched and still handles cleanup. Adds a regression test asserting one alert across multiple ticks. Fixes BerriAI#27855. * fix(health): treat all-proxy-models keys as unrestricted in /health (BerriAI#30087) * fix(health): treat all-proxy-models keys as unrestricted in /health A key granted all model permissions stores the literal "all-proxy-models" marker in its models list. The /health access filter compared that marker against real model_names, so the model list filtered down to nothing and the WebUI health check returned healthy_count=0, unhealthy_count=0 with HTTP 503. Skip the filter (both the live path and the background-cache model_id scoping) when the marker is present, matching how auth_checks treats SpecialModelNames.all_proxy_models. Fixes BerriAI#29744. * fix(health): resolve all-team-models sentinel to the team allowlist Same failure shape as the all-proxy-models case: a key carrying the literal "all-team-models" entry matches no real model_name, so the /health access filter would zero out the model list. Resolve the sentinel to the key's team models when team_id is set, matching get_key_models in model_checks.py. Without a team_id the sentinel stays unresolved and matches nothing, denying rather than widening access, mirroring _resolve_key_models_for_auth_check. * feat(proxy): auto-enable drop_params for Claude Code requests (BerriAI#30218) * feat(proxy): auto-enable drop_params for Claude Code requests Claude Code identifies itself with a claude-cli/<version> user agent and sends Anthropic-specific params (top_k, thinking, etc.) on every request. When the proxy routes those requests to a non-Anthropic provider, the unsupported params fail the call unless drop_params is configured. Detect the Claude Code user agent in add_litellm_data_to_request and default drop_params to true for those requests, without overriding an explicit drop_params value sent by the caller. * feat(proxy): respect operator litellm_settings drop_params over Claude Code default An explicit drop_params in the operator's litellm_settings (true or false) now suppresses the Claude Code user agent default, so an operator who deliberately configured drop_params: false keeps strict param validation for Claude Code clients too. The auto-default only fills the gap when neither the request body nor the config sets a value. * fix(snowflake): migrate to native endpoints with auto-routing for Claude models (BerriAI#29964) * fix(snowflake): migrate to native Cortex REST API endpoints Replaces the legacy /api/v2/cortex/inference:complete endpoint with the native OpenAI-compatible /api/v2/cortex/v1/chat/completions endpoint, fixing error 390142 (Incoming request does not contain a valid payload) when using model: snowflake/<model> in LiteLLM proxy. Changes: - litellm/llms/snowflake/chat/transformation.py: route to native /cortex/v1/chat/completions, remove Snowflake-specific tool_spec payload transformation, remove content_list response handling, add stream to supported params - litellm/llms/snowflake/anthropic/transformation.py (new): SnowflakeCortexAnthropicConfig routes Claude models to /cortex/v1/messages with anthropic-version header and Anthropic->OpenAI response transform - tests: 29 unit tests covering URL routing, auth headers, payload format, and response parsing * fix(snowflake): map max_tokens to max_completion_tokens for native endpoint * fix: handle multi-turn tool conversations and OpenAI→Anthropic tool format conversion - _extract_system_and_messages now preserves tool_calls from assistant messages and converts them to Anthropic tool_use content blocks - tool role messages are converted to user role with tool_result content blocks (as required by Anthropic Messages API) - Added _transform_tools_to_anthropic() to convert OpenAI tool format (type/function/parameters) to Anthropic format (name/input_schema) - Added comprehensive tests for multi-turn tool conversations Addresses review feedback on PR BerriAI#29964 * test: add coverage for malformed JSON and non-string tool arguments * fix(tests): update chat transformation tests for native OpenAI-compatible endpoint * style: apply black formatting * fix: resolve mypy type errors in anthropic transformation * fix: correct mypy type: ignore error codes (attr-defined) * fix: use max_tokens instead of max_completion_tokens for Snowflake endpoint compatibility * refactor: merge Anthropic config into unified SnowflakeConfig with auto-routing - Remove separate SnowflakeCortexAnthropicConfig and anthropic/ directory - SnowflakeConfig now auto-routes based on model name: - Claude models → /messages endpoint (Anthropic format) - All others → /chat/completions endpoint (OpenAI format) - No new provider needed (stays as SNOWFLAKE = 'snowflake') - Tool message transformation for Claude: tool_calls → tool_use blocks, tool role → user with tool_result - OpenAI → Anthropic tool format conversion (parameters → input_schema) - Addresses Greptile feedback about unwired SnowflakeCortexAnthropicConfig * fix: use max_completion_tokens for /chat/completions (Snowflake deprecated max_tokens on this endpoint) * fix(tests): update assertions for Claude auto-routing to /messages endpoint * fix(snowflake): add tool_choice conversion and preserve max_completion_tokens in Anthropic path * fix(snowflake): use ChatCompletionMessageToolCall objects and strip model prefix on OpenAI path * fix(snowflake): collect multiple system messages to prevent guardrail override * chore: remove committed .pyc files and add __pycache__ to .gitignore * fix: remove unused Union import * fix: restore original .gitignore (accidentally replaced in earlier commit) * feat(snowflake): add streaming response handler for both Anthropic and OpenAI SSE formats * fix: remove unused AsyncIterator and Iterator imports * fix: add missing total_tokens to ChatCompletionUsageBlock * fix(snowflake): coalesce consecutive tool results into single user message for Anthropic * fix(snowflake): handle message_start event for streaming input_tokens tracking * fix: evict last deleted model in multi-instance deployments (BerriAI#28608) * fix: evict last deleted model in multi-instance deployments _delete_deployment had an early return when db_models was empty, preventing eviction of the last deleted model during reconciliation. - Remove len(db_models)==0 early return from _delete_deployment - Return None (not []) from _get_models_from_db on DB failure so callers can distinguish a transient failure from a genuinely empty DB - Guard _update_llm_router against None to skip updates on DB failure Fixes BerriAI#28443 * test: remove dead MagicMock assignment in type_mismatch test * fix: update test to pass [] not None to _update_llm_router test_ProxyConfig__update_llm_router_bad_proxy_logging_raises was passing None as new_models to get through to the proxy_logging_obj check, but the None guard we added now returns early before reaching that path. Pass [] instead so the test exercises the intended AttributeError case. Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> * chore: regenerate API types to sync schema.d.ts with proxy OpenAPI spec Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> --------- Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> * fix: invalidate Redis spend counter on /key/reset_spend (BerriAI#29694) * fix: set Redis spend counter to reset_to value on /key/reset_spend Previously, the Redis spend counter was always set to 0.0 after a reset, even when reset_to was a non-zero value (partial reset). This caused the budget to be under-enforced for up to 60 seconds until the counter expired and fell through to the DB. Now the counter is set to the actual reset_to value, so partial resets are reflected correctly and budget enforcement is consistent. * test: update reset_key_spend test to match direct cache set The implementation now sets spend_counter_cache directly instead of calling _invalidate_spend_counter. Update the test to verify the in_memory_cache.set_cache call with the correct key, value, and ttl. --------- Co-authored-by: michaelxer <michaelxer@users.noreply.github.com> * fix: add scaleway models pricing (BerriAI#27659) * fix: Add embeddings support for Scaleway provider * fix: resolve merge conflicts * fix(main): clarify backend route handling for Swagger static assets (BerriAI#30196) * fix(main): clarify backend route handling for Swagger static assets * fix(allowlist): add BACKEND_MOUNT_PATHS for Swagger static assets * fix(voyage): route multimodal embeddings to correct endpoint (BerriAI#30193) * fix(voyage): route multimodal embeddings to correct endpoint * test(voyage): cover multimodal embedding edge cases * test(voyage): cover api key fallback * fix(voyage): raise early on missing api key and malformed image url * test(voyage): cover utils routing and helper * fix(voyage): route supported openai params for multimodal models * style: apply black formatting * fix(ui): infer Azure API version from API base (BerriAI#30204) * fix(ui): infer Azure API version from API base * fix(ui): address Azure API version feedback * Update litellm/llms/snowflake/chat/transformation.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * feat(datadog): add team-scoped Datadog callback support (BerriAI#29947) Enable teams to configure their own Datadog credentials via POST /team/{team_id}/callback, following the same pattern as Langfuse. * Merge pull request BerriAI#29528 from aanchal22/litellm_byok-alias-merge fix(proxy): atomic merge for team model aliases and team.models on BYOK create * feat: add EmpirioLabs as an OpenAI-compatible provider (BerriAI#30278) Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com> * fix: resolve failing tests and lint in snowflake/team endpoints - Black-format snowflake/chat/transformation.py to fix lint failure - Update Anthropic config test to expect default max_tokens of 4096 (matches implementation) - Add AsyncMock + execute_raw mock to team_model_add cache-refresh pin test - Add model_dump mock and patch cache/logging in test_uses_atomic_array_append_with_dedup Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(test): update test_db_error_new_model_check for new _delete_deployment logic _delete_deployment no longer short-circuits on empty db_models — it now treats [] as a valid empty-DB state and proceeds to check config models. Mock get_config to return the two router deployments so they appear in combined_id_list and are protected, which matches the real-world scenario where a DB error occurs but the models are config-backed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list (BerriAI#30295) * feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list Follow-up to BerriAI#30223 per maintainer review: documents the flag in ConfigGeneralSettings with a short description and adds it to allowed_args in get_config_list so the UI and /config/list expose it. A test pins that /config/list returns the field with type Boolean, which requires both registrations to be present * chore(ui): regenerate schema.d.ts for cancel_on_disconnect --------- Co-authored-by: kursad <kursad.lacin@brado.net> * fix(datadog): never fall back to env DD_API_KEY for caller-supplied destinations Team/key-scoped Datadog loggers could be pointed at an arbitrary dd_agent_host or dd_site while omitting dd_api_key, causing the proxy's global DD_API_KEY to be sent as the DD-API-KEY header to that destination. Gate the env-var fallback behind an allow_env_credentials flag, set to False when the destination is caller-supplied, mirroring the existing langfuse/langsmith pattern. --------- Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: daitran-tensormesh <dai@tensormesh.ai> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Muspi Merol <me@promplate.dev> Co-authored-by: fangkang <fangkangm@gmail.com> Co-authored-by: Chris Hoogeboom <chris.hoogeboom@gmail.com> Co-authored-by: kursadlacin <kursadlacin@gmail.com> Co-authored-by: kursad <kursad.lacin@brado.net> Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> Co-authored-by: hcl <chenglunhu@gmail.com> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: sfc-gh-nashukla <navnit.shukla@snowflake.com> Co-authored-by: Rudra Dudhat <contact.rdudhat@gmail.com> Co-authored-by: Michael <52305679+michaelxer@users.noreply.github.com> Co-authored-by: michaelxer <michaelxer@users.noreply.github.com> Co-authored-by: Quentin Champenois <26109239+Quentinchampenois@users.noreply.github.com> Co-authored-by: mauriceberentsen <mauriceberentsen@live.nl> Co-authored-by: lost9999 <56498264+lost9999@users.noreply.github.com> Co-authored-by: GaetanVDB07 <86427581+GaetanVDB07@users.noreply.github.com> Co-authored-by: Aanchal Khandelwal <aan2210khandelwal@gmail.com> Co-authored-by: Adam Dalloul <adam_dalloul@icloud.com> Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
koladefaj
pushed a commit
to koladefaj/litellm
that referenced
this pull request
Jun 17, 2026
* feat(bedrock): add bedrock mantle gemma 4 models (BerriAI#30264) * feat(bedrock): add bedrock mantle gemma 4 models * test(bedrock): harden mantle local cost fixture * feat(responses): enable the responses API for the Tensormesh provider (BerriAI#30209) * feat(responses): enable the responses API for the Tensormesh provider * Update litellm/llms/openai_like/providers.json Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(langfuse_otel): mark LLM spans as generations (BerriAI#30250) * fix(bedrock): stop stream_chunk_size leaking into invoke request bodies (BerriAI#30240) stream_chunk_size is a LiteLLM-internal knob for re-chunking the HTTP response stream. The invoke transformations splat optional_params into the provider request body without dropping it, and Bedrock rejects unknown fields, so any bedrock/invoke request that sets the parameter fails with ValidationException: stream_chunk_size: Extra inputs are not permitted. Drop it in the invoke dispatcher (covers cohere, titan, mistral, meta, ai21) and in the Claude messages-format request builder (the route used for bedrock/invoke Anthropic models) * fix(bedrock): stop buffering streamed tool-call argument deltas (BerriAI#30231) * fix(bedrock): stop buffering streamed tool-call argument deltas Two issues made Bedrock tool-use streaming arrive as a single end-of-stream burst through LiteLLM while plain text streamed fine. First, the anthropic-beta allowlist mapped fine-grained-tool-streaming-2025-05-14 to null for bedrock and bedrock_converse, so the header was silently stripped. Without that beta, Anthropic models on Bedrock buffer tool input server-side and emit all toolUse.input deltas at once (verified against converse-stream and invoke-with-response-stream directly). Bedrock accepts the beta via additionalModelRequestFields.anthropic_beta, so it is now forwarded. Second, the streaming reads re-chunked the AWS event stream with iter_bytes(chunk_size=1024). httpx's ByteChunker only releases full 1024-byte blocks, so the small early events (messageStart, contentBlockStart, first deltas) sat in the buffer until enough bytes accumulated, pushing time-to-first-byte from ~1.4s to ~8.5s on buffered tool-use streams. The default is now no re-chunking; an explicit stream_chunk_size is still honored. * test(bedrock): cover explicit stream_chunk_size on sync invoke path * test(bedrock): cover stream_chunk_size plumbing through converse completion * test(bedrock): cover stream_chunk_size default in legacy BedrockLLM streaming * test(bedrock): merge converse handler tests into existing mapped test file pytest imports test modules by basename in non-package test dirs, so the new tests/test_litellm/llms/bedrock/chat/test_converse_handler.py collided with the pre-existing tests/test_litellm/llms/chat/test_converse_handler.py and broke collection in CI. Move the new tests into the existing file * feat(otel): emit v2 cost breakdown + stamp tracer scope version (BerriAI#30156) Read the StandardLoggingPayload cost_breakdown into a typed LLMCost on LLMCallSpanData and emit each component under litellm.cost.* (absent components omitted, so spans stay sparse). Stamp litellm.__version__ as the instrumentation scope version so every v2 span carries a deterministic scope.version. Tests under tests/test_litellm/integrations/otel/. * fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in) (BerriAI#30223) * fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in) On the non-streaming path, base_process_llm_request awaited the LLM call with no disconnect monitoring; when the HTTP client went away the upstream request kept running until completion or request_timeout (6000s default), holding a backend slot (e.g. a vLLM GPU slot) for output nobody would read Add an opt-in general_settings.cancel_on_disconnect flag, default off, so the default code path is unchanged. When enabled, a receive-based watcher task observes http.disconnect and cancels the asyncio.gather driving the upstream call. The resulting CancelledError is converted to HTTPException 499 only when the disconnect event is set, so server-initiated cancellations still propagate as-is. The 499 then flows through _handle_llm_api_exception like any other failure, meaning post_call_failure_hook still releases max_parallel_requests slots and fires spend and alerting callbacks; it is logged at info level instead of a full traceback Also removes the dead check_request_disconnection helper in proxy_server.py (zero call sites) along with its behavior-pin tests Builds on the receive-based design from BerriAI#25776 Addresses BerriAI#13774. Re-fixes BerriAI#22805 (regressed after the BerriAI#14295 revert) Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> * fix(proxy): scope 499 quiet logging to disconnects and harden watcher Address the two P2 findings from the Greptile review on BerriAI#30223. The info-level logging in _log_llm_api_exception now applies only to the disconnect-specific HTTPException (status 499 plus the shared _CLIENT_DISCONNECT_DETAIL message), so any other 499 raised by hooks or guardrails keeps its full traceback. The disconnect watcher now catches exceptions from request.receive() (e.g. a transport reset) and logs a warning instead of dying silently, making the degradation to no-op visible; a test pins that the LLM call is not cancelled in that case --------- Co-authored-by: kursad <kursad.lacin@brado.net> Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> * fix(bedrock): grant aws-external-anthropic:* in OIDC session policy for claude_platform (BerriAI#30200) (BerriAI#30205) The inline STS session policy passed to assume_role_with_web_identity acts as an IAM PERMISSION CEILING — effective permissions are the intersection of the role's identity policies and this policy. Any action not listed is silently denied even when the IAM role grants it. BerriAI#27678 added the bedrock/claude_platform/<model> route but its service-side action namespace is aws-external-anthropic:*, not bedrock:*. Without a matching statement here, every claude_platform request via OIDC (GCP federation, EKS Pod Identity webhook, etc.) 403s with 'no session policy allows the aws-external-anthropic:CreateInference action' — even with a fully permissive identity policy. Add a second ClaudePlatformLiteLLM statement covering CreateInference, CreateBatchInference, CancelBatchInference, DeleteBatchInference, CountTokens, Get*, List*. Keep aws:SecureTransport=true parity with the bedrock statement. Static creds + IRSA flow through different code paths and are not affected. Fixes BerriAI#30200 * fix(proxy): set Retry-After header on RouterRateLimitError 429 responses (BerriAI#30098) * Set Retry-After header on RouterRateLimitError responses When all deployments for a model are in cooldown, the proxy returns a 429 whose cooldown timing is only available by parsing the error message string. RouterRateLimitError already carries cooldown_time, so expose it as a standard retry-after header in _handle_llm_api_exception. The value is rounded up so clients never retry before the cooldown window ends. Fixes BerriAI#27823. * Set Retry-After after response-headers hook so cooldown wins The cooldown-derived retry-after was assigned before the post_call_response_headers_hook merge, so a callback returning a retry-after key (including a stale or empty value) silently clobbered it. Move the RouterRateLimitError block after the callback merge so the cooldown value is authoritative for this error type. * fix(router): route aspeech through async_function_with_fallbacks (BerriAI#30104) * fix(router): route aspeech through async_function_with_fallbacks Router.aspeech selected a deployment and awaited litellm.aspeech directly, so TTS requests got no retry on failure and no failover to backup deployments; the except block only fired an exception alert and re-raised. Every other router endpoint (acompletion, aembedding, atranscription, arerank) already delegates to async_function_with_fallbacks Mirror the atranscription pattern: move deployment selection and the litellm.aspeech call into a private _aspeech method, then have the public aspeech set kwargs["original_function"] = self._aspeech and await self.async_function_with_fallbacks(**kwargs). _aspeech also picks up the shared _get_async_openai_model_client helper and the same total/success/fail call accounting the sibling endpoints use Fixes BerriAI#27778. * fix(router): apply deployment kwargs and rpm semaphore in _aspeech Bring _aspeech fully in line with _atranscription: call _update_kwargs_with_deployment so deployment metadata, model_info, timeout, and default litellm params flow into the request, and wrap the litellm.aspeech call with the max_parallel_requests semaphore plus async_routing_strategy_pre_call_checks so TTS respects rpm limits the same way the other router endpoints do Also add a unit test that exercises _aspeech directly and asserts the deployment metadata reaches the underlying call * fix(slack_alerting): stop false-positive hanging request alerts for requests below the alerting threshold (BerriAI#30106) * fix(slack_alerting): skip hanging request alerts below the threshold The hanging request check alerted on any cached request whose completion status was not yet recorded, with no minimum age check. Since the background loop runs every alerting_threshold / 2 seconds, any request that happened to be in flight at a check fired a "hanging - Ns+ request time" alert even if it was only seconds old, producing a steady stream of false positives. Add a created_at timestamp to HangingRequestData, stamped when the request enters the hanging request cache, and skip requests younger than alerting_threshold without evicting them, so a later check can still alert if they never complete. Extend the cache TTL from threshold + 60s to 1.5x threshold + 60s; with the age check, entries only become alertable after threshold seconds, and the check period is threshold / 2, so the old TTL could evict a genuinely hanging request before any check saw it cross the threshold. Fixes BerriAI#27855. * fix(slack_alerting): alert once per hanging request The min-age gate stops false positives for young in-flight requests, but a genuinely hanging request still re-alerted on every checker tick within the cache TTL. With the wider TTL (1.5x threshold + 60s) that is 1-2 extra Slack notifications per stuck request at the default 600s threshold. Flag a HangingRequestData entry as alerted once its alert fires and skip flagged entries on later ticks, so each hang produces exactly one alert. The cache reference is mutated in place, so the TTL is untouched and still handles cleanup. Adds a regression test asserting one alert across multiple ticks. Fixes BerriAI#27855. * fix(health): treat all-proxy-models keys as unrestricted in /health (BerriAI#30087) * fix(health): treat all-proxy-models keys as unrestricted in /health A key granted all model permissions stores the literal "all-proxy-models" marker in its models list. The /health access filter compared that marker against real model_names, so the model list filtered down to nothing and the WebUI health check returned healthy_count=0, unhealthy_count=0 with HTTP 503. Skip the filter (both the live path and the background-cache model_id scoping) when the marker is present, matching how auth_checks treats SpecialModelNames.all_proxy_models. Fixes BerriAI#29744. * fix(health): resolve all-team-models sentinel to the team allowlist Same failure shape as the all-proxy-models case: a key carrying the literal "all-team-models" entry matches no real model_name, so the /health access filter would zero out the model list. Resolve the sentinel to the key's team models when team_id is set, matching get_key_models in model_checks.py. Without a team_id the sentinel stays unresolved and matches nothing, denying rather than widening access, mirroring _resolve_key_models_for_auth_check. * feat(proxy): auto-enable drop_params for Claude Code requests (BerriAI#30218) * feat(proxy): auto-enable drop_params for Claude Code requests Claude Code identifies itself with a claude-cli/<version> user agent and sends Anthropic-specific params (top_k, thinking, etc.) on every request. When the proxy routes those requests to a non-Anthropic provider, the unsupported params fail the call unless drop_params is configured. Detect the Claude Code user agent in add_litellm_data_to_request and default drop_params to true for those requests, without overriding an explicit drop_params value sent by the caller. * feat(proxy): respect operator litellm_settings drop_params over Claude Code default An explicit drop_params in the operator's litellm_settings (true or false) now suppresses the Claude Code user agent default, so an operator who deliberately configured drop_params: false keeps strict param validation for Claude Code clients too. The auto-default only fills the gap when neither the request body nor the config sets a value. * fix(snowflake): migrate to native endpoints with auto-routing for Claude models (BerriAI#29964) * fix(snowflake): migrate to native Cortex REST API endpoints Replaces the legacy /api/v2/cortex/inference:complete endpoint with the native OpenAI-compatible /api/v2/cortex/v1/chat/completions endpoint, fixing error 390142 (Incoming request does not contain a valid payload) when using model: snowflake/<model> in LiteLLM proxy. Changes: - litellm/llms/snowflake/chat/transformation.py: route to native /cortex/v1/chat/completions, remove Snowflake-specific tool_spec payload transformation, remove content_list response handling, add stream to supported params - litellm/llms/snowflake/anthropic/transformation.py (new): SnowflakeCortexAnthropicConfig routes Claude models to /cortex/v1/messages with anthropic-version header and Anthropic->OpenAI response transform - tests: 29 unit tests covering URL routing, auth headers, payload format, and response parsing * fix(snowflake): map max_tokens to max_completion_tokens for native endpoint * fix: handle multi-turn tool conversations and OpenAI→Anthropic tool format conversion - _extract_system_and_messages now preserves tool_calls from assistant messages and converts them to Anthropic tool_use content blocks - tool role messages are converted to user role with tool_result content blocks (as required by Anthropic Messages API) - Added _transform_tools_to_anthropic() to convert OpenAI tool format (type/function/parameters) to Anthropic format (name/input_schema) - Added comprehensive tests for multi-turn tool conversations Addresses review feedback on PR BerriAI#29964 * test: add coverage for malformed JSON and non-string tool arguments * fix(tests): update chat transformation tests for native OpenAI-compatible endpoint * style: apply black formatting * fix: resolve mypy type errors in anthropic transformation * fix: correct mypy type: ignore error codes (attr-defined) * fix: use max_tokens instead of max_completion_tokens for Snowflake endpoint compatibility * refactor: merge Anthropic config into unified SnowflakeConfig with auto-routing - Remove separate SnowflakeCortexAnthropicConfig and anthropic/ directory - SnowflakeConfig now auto-routes based on model name: - Claude models → /messages endpoint (Anthropic format) - All others → /chat/completions endpoint (OpenAI format) - No new provider needed (stays as SNOWFLAKE = 'snowflake') - Tool message transformation for Claude: tool_calls → tool_use blocks, tool role → user with tool_result - OpenAI → Anthropic tool format conversion (parameters → input_schema) - Addresses Greptile feedback about unwired SnowflakeCortexAnthropicConfig * fix: use max_completion_tokens for /chat/completions (Snowflake deprecated max_tokens on this endpoint) * fix(tests): update assertions for Claude auto-routing to /messages endpoint * fix(snowflake): add tool_choice conversion and preserve max_completion_tokens in Anthropic path * fix(snowflake): use ChatCompletionMessageToolCall objects and strip model prefix on OpenAI path * fix(snowflake): collect multiple system messages to prevent guardrail override * chore: remove committed .pyc files and add __pycache__ to .gitignore * fix: remove unused Union import * fix: restore original .gitignore (accidentally replaced in earlier commit) * feat(snowflake): add streaming response handler for both Anthropic and OpenAI SSE formats * fix: remove unused AsyncIterator and Iterator imports * fix: add missing total_tokens to ChatCompletionUsageBlock * fix(snowflake): coalesce consecutive tool results into single user message for Anthropic * fix(snowflake): handle message_start event for streaming input_tokens tracking * fix: evict last deleted model in multi-instance deployments (BerriAI#28608) * fix: evict last deleted model in multi-instance deployments _delete_deployment had an early return when db_models was empty, preventing eviction of the last deleted model during reconciliation. - Remove len(db_models)==0 early return from _delete_deployment - Return None (not []) from _get_models_from_db on DB failure so callers can distinguish a transient failure from a genuinely empty DB - Guard _update_llm_router against None to skip updates on DB failure Fixes BerriAI#28443 * test: remove dead MagicMock assignment in type_mismatch test * fix: update test to pass [] not None to _update_llm_router test_ProxyConfig__update_llm_router_bad_proxy_logging_raises was passing None as new_models to get through to the proxy_logging_obj check, but the None guard we added now returns early before reaching that path. Pass [] instead so the test exercises the intended AttributeError case. Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> * chore: regenerate API types to sync schema.d.ts with proxy OpenAPI spec Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> --------- Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> * fix: invalidate Redis spend counter on /key/reset_spend (BerriAI#29694) * fix: set Redis spend counter to reset_to value on /key/reset_spend Previously, the Redis spend counter was always set to 0.0 after a reset, even when reset_to was a non-zero value (partial reset). This caused the budget to be under-enforced for up to 60 seconds until the counter expired and fell through to the DB. Now the counter is set to the actual reset_to value, so partial resets are reflected correctly and budget enforcement is consistent. * test: update reset_key_spend test to match direct cache set The implementation now sets spend_counter_cache directly instead of calling _invalidate_spend_counter. Update the test to verify the in_memory_cache.set_cache call with the correct key, value, and ttl. --------- Co-authored-by: michaelxer <michaelxer@users.noreply.github.com> * fix: add scaleway models pricing (BerriAI#27659) * fix: Add embeddings support for Scaleway provider * fix: resolve merge conflicts * fix(main): clarify backend route handling for Swagger static assets (BerriAI#30196) * fix(main): clarify backend route handling for Swagger static assets * fix(allowlist): add BACKEND_MOUNT_PATHS for Swagger static assets * fix(voyage): route multimodal embeddings to correct endpoint (BerriAI#30193) * fix(voyage): route multimodal embeddings to correct endpoint * test(voyage): cover multimodal embedding edge cases * test(voyage): cover api key fallback * fix(voyage): raise early on missing api key and malformed image url * test(voyage): cover utils routing and helper * fix(voyage): route supported openai params for multimodal models * style: apply black formatting * fix(ui): infer Azure API version from API base (BerriAI#30204) * fix(ui): infer Azure API version from API base * fix(ui): address Azure API version feedback * Update litellm/llms/snowflake/chat/transformation.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * feat(datadog): add team-scoped Datadog callback support (BerriAI#29947) Enable teams to configure their own Datadog credentials via POST /team/{team_id}/callback, following the same pattern as Langfuse. * Merge pull request BerriAI#29528 from aanchal22/litellm_byok-alias-merge fix(proxy): atomic merge for team model aliases and team.models on BYOK create * feat: add EmpirioLabs as an OpenAI-compatible provider (BerriAI#30278) Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com> * fix: resolve failing tests and lint in snowflake/team endpoints - Black-format snowflake/chat/transformation.py to fix lint failure - Update Anthropic config test to expect default max_tokens of 4096 (matches implementation) - Add AsyncMock + execute_raw mock to team_model_add cache-refresh pin test - Add model_dump mock and patch cache/logging in test_uses_atomic_array_append_with_dedup Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(test): update test_db_error_new_model_check for new _delete_deployment logic _delete_deployment no longer short-circuits on empty db_models — it now treats [] as a valid empty-DB state and proceeds to check config models. Mock get_config to return the two router deployments so they appear in combined_id_list and are protected, which matches the real-world scenario where a DB error occurs but the models are config-backed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list (BerriAI#30295) * feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list Follow-up to BerriAI#30223 per maintainer review: documents the flag in ConfigGeneralSettings with a short description and adds it to allowed_args in get_config_list so the UI and /config/list expose it. A test pins that /config/list returns the field with type Boolean, which requires both registrations to be present * chore(ui): regenerate schema.d.ts for cancel_on_disconnect --------- Co-authored-by: kursad <kursad.lacin@brado.net> * fix(datadog): never fall back to env DD_API_KEY for caller-supplied destinations Team/key-scoped Datadog loggers could be pointed at an arbitrary dd_agent_host or dd_site while omitting dd_api_key, causing the proxy's global DD_API_KEY to be sent as the DD-API-KEY header to that destination. Gate the env-var fallback behind an allow_env_credentials flag, set to False when the destination is caller-supplied, mirroring the existing langfuse/langsmith pattern. --------- Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: daitran-tensormesh <dai@tensormesh.ai> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Muspi Merol <me@promplate.dev> Co-authored-by: fangkang <fangkangm@gmail.com> Co-authored-by: Chris Hoogeboom <chris.hoogeboom@gmail.com> Co-authored-by: kursadlacin <kursadlacin@gmail.com> Co-authored-by: kursad <kursad.lacin@brado.net> Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> Co-authored-by: hcl <chenglunhu@gmail.com> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: sfc-gh-nashukla <navnit.shukla@snowflake.com> Co-authored-by: Rudra Dudhat <contact.rdudhat@gmail.com> Co-authored-by: Michael <52305679+michaelxer@users.noreply.github.com> Co-authored-by: michaelxer <michaelxer@users.noreply.github.com> Co-authored-by: Quentin Champenois <26109239+Quentinchampenois@users.noreply.github.com> Co-authored-by: mauriceberentsen <mauriceberentsen@live.nl> Co-authored-by: lost9999 <56498264+lost9999@users.noreply.github.com> Co-authored-by: GaetanVDB07 <86427581+GaetanVDB07@users.noreply.github.com> Co-authored-by: Aanchal Khandelwal <aan2210khandelwal@gmail.com> Co-authored-by: Adam Dalloul <adam_dalloul@icloud.com> Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
factnn
pushed a commit
to factnn/litellm
that referenced
this pull request
Jun 18, 2026
* feat(bedrock): add bedrock mantle gemma 4 models (BerriAI#30264) * feat(bedrock): add bedrock mantle gemma 4 models * test(bedrock): harden mantle local cost fixture * feat(responses): enable the responses API for the Tensormesh provider (BerriAI#30209) * feat(responses): enable the responses API for the Tensormesh provider * Update litellm/llms/openai_like/providers.json Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(langfuse_otel): mark LLM spans as generations (BerriAI#30250) * fix(bedrock): stop stream_chunk_size leaking into invoke request bodies (BerriAI#30240) stream_chunk_size is a LiteLLM-internal knob for re-chunking the HTTP response stream. The invoke transformations splat optional_params into the provider request body without dropping it, and Bedrock rejects unknown fields, so any bedrock/invoke request that sets the parameter fails with ValidationException: stream_chunk_size: Extra inputs are not permitted. Drop it in the invoke dispatcher (covers cohere, titan, mistral, meta, ai21) and in the Claude messages-format request builder (the route used for bedrock/invoke Anthropic models) * fix(bedrock): stop buffering streamed tool-call argument deltas (BerriAI#30231) * fix(bedrock): stop buffering streamed tool-call argument deltas Two issues made Bedrock tool-use streaming arrive as a single end-of-stream burst through LiteLLM while plain text streamed fine. First, the anthropic-beta allowlist mapped fine-grained-tool-streaming-2025-05-14 to null for bedrock and bedrock_converse, so the header was silently stripped. Without that beta, Anthropic models on Bedrock buffer tool input server-side and emit all toolUse.input deltas at once (verified against converse-stream and invoke-with-response-stream directly). Bedrock accepts the beta via additionalModelRequestFields.anthropic_beta, so it is now forwarded. Second, the streaming reads re-chunked the AWS event stream with iter_bytes(chunk_size=1024). httpx's ByteChunker only releases full 1024-byte blocks, so the small early events (messageStart, contentBlockStart, first deltas) sat in the buffer until enough bytes accumulated, pushing time-to-first-byte from ~1.4s to ~8.5s on buffered tool-use streams. The default is now no re-chunking; an explicit stream_chunk_size is still honored. * test(bedrock): cover explicit stream_chunk_size on sync invoke path * test(bedrock): cover stream_chunk_size plumbing through converse completion * test(bedrock): cover stream_chunk_size default in legacy BedrockLLM streaming * test(bedrock): merge converse handler tests into existing mapped test file pytest imports test modules by basename in non-package test dirs, so the new tests/test_litellm/llms/bedrock/chat/test_converse_handler.py collided with the pre-existing tests/test_litellm/llms/chat/test_converse_handler.py and broke collection in CI. Move the new tests into the existing file * feat(otel): emit v2 cost breakdown + stamp tracer scope version (BerriAI#30156) Read the StandardLoggingPayload cost_breakdown into a typed LLMCost on LLMCallSpanData and emit each component under litellm.cost.* (absent components omitted, so spans stay sparse). Stamp litellm.__version__ as the instrumentation scope version so every v2 span carries a deterministic scope.version. Tests under tests/test_litellm/integrations/otel/. * fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in) (BerriAI#30223) * fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in) On the non-streaming path, base_process_llm_request awaited the LLM call with no disconnect monitoring; when the HTTP client went away the upstream request kept running until completion or request_timeout (6000s default), holding a backend slot (e.g. a vLLM GPU slot) for output nobody would read Add an opt-in general_settings.cancel_on_disconnect flag, default off, so the default code path is unchanged. When enabled, a receive-based watcher task observes http.disconnect and cancels the asyncio.gather driving the upstream call. The resulting CancelledError is converted to HTTPException 499 only when the disconnect event is set, so server-initiated cancellations still propagate as-is. The 499 then flows through _handle_llm_api_exception like any other failure, meaning post_call_failure_hook still releases max_parallel_requests slots and fires spend and alerting callbacks; it is logged at info level instead of a full traceback Also removes the dead check_request_disconnection helper in proxy_server.py (zero call sites) along with its behavior-pin tests Builds on the receive-based design from BerriAI#25776 Addresses BerriAI#13774. Re-fixes BerriAI#22805 (regressed after the BerriAI#14295 revert) Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> * fix(proxy): scope 499 quiet logging to disconnects and harden watcher Address the two P2 findings from the Greptile review on BerriAI#30223. The info-level logging in _log_llm_api_exception now applies only to the disconnect-specific HTTPException (status 499 plus the shared _CLIENT_DISCONNECT_DETAIL message), so any other 499 raised by hooks or guardrails keeps its full traceback. The disconnect watcher now catches exceptions from request.receive() (e.g. a transport reset) and logs a warning instead of dying silently, making the degradation to no-op visible; a test pins that the LLM call is not cancelled in that case --------- Co-authored-by: kursad <kursad.lacin@brado.net> Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> * fix(bedrock): grant aws-external-anthropic:* in OIDC session policy for claude_platform (BerriAI#30200) (BerriAI#30205) The inline STS session policy passed to assume_role_with_web_identity acts as an IAM PERMISSION CEILING — effective permissions are the intersection of the role's identity policies and this policy. Any action not listed is silently denied even when the IAM role grants it. BerriAI#27678 added the bedrock/claude_platform/<model> route but its service-side action namespace is aws-external-anthropic:*, not bedrock:*. Without a matching statement here, every claude_platform request via OIDC (GCP federation, EKS Pod Identity webhook, etc.) 403s with 'no session policy allows the aws-external-anthropic:CreateInference action' — even with a fully permissive identity policy. Add a second ClaudePlatformLiteLLM statement covering CreateInference, CreateBatchInference, CancelBatchInference, DeleteBatchInference, CountTokens, Get*, List*. Keep aws:SecureTransport=true parity with the bedrock statement. Static creds + IRSA flow through different code paths and are not affected. Fixes BerriAI#30200 * fix(proxy): set Retry-After header on RouterRateLimitError 429 responses (BerriAI#30098) * Set Retry-After header on RouterRateLimitError responses When all deployments for a model are in cooldown, the proxy returns a 429 whose cooldown timing is only available by parsing the error message string. RouterRateLimitError already carries cooldown_time, so expose it as a standard retry-after header in _handle_llm_api_exception. The value is rounded up so clients never retry before the cooldown window ends. Fixes BerriAI#27823. * Set Retry-After after response-headers hook so cooldown wins The cooldown-derived retry-after was assigned before the post_call_response_headers_hook merge, so a callback returning a retry-after key (including a stale or empty value) silently clobbered it. Move the RouterRateLimitError block after the callback merge so the cooldown value is authoritative for this error type. * fix(router): route aspeech through async_function_with_fallbacks (BerriAI#30104) * fix(router): route aspeech through async_function_with_fallbacks Router.aspeech selected a deployment and awaited litellm.aspeech directly, so TTS requests got no retry on failure and no failover to backup deployments; the except block only fired an exception alert and re-raised. Every other router endpoint (acompletion, aembedding, atranscription, arerank) already delegates to async_function_with_fallbacks Mirror the atranscription pattern: move deployment selection and the litellm.aspeech call into a private _aspeech method, then have the public aspeech set kwargs["original_function"] = self._aspeech and await self.async_function_with_fallbacks(**kwargs). _aspeech also picks up the shared _get_async_openai_model_client helper and the same total/success/fail call accounting the sibling endpoints use Fixes BerriAI#27778. * fix(router): apply deployment kwargs and rpm semaphore in _aspeech Bring _aspeech fully in line with _atranscription: call _update_kwargs_with_deployment so deployment metadata, model_info, timeout, and default litellm params flow into the request, and wrap the litellm.aspeech call with the max_parallel_requests semaphore plus async_routing_strategy_pre_call_checks so TTS respects rpm limits the same way the other router endpoints do Also add a unit test that exercises _aspeech directly and asserts the deployment metadata reaches the underlying call * fix(slack_alerting): stop false-positive hanging request alerts for requests below the alerting threshold (BerriAI#30106) * fix(slack_alerting): skip hanging request alerts below the threshold The hanging request check alerted on any cached request whose completion status was not yet recorded, with no minimum age check. Since the background loop runs every alerting_threshold / 2 seconds, any request that happened to be in flight at a check fired a "hanging - Ns+ request time" alert even if it was only seconds old, producing a steady stream of false positives. Add a created_at timestamp to HangingRequestData, stamped when the request enters the hanging request cache, and skip requests younger than alerting_threshold without evicting them, so a later check can still alert if they never complete. Extend the cache TTL from threshold + 60s to 1.5x threshold + 60s; with the age check, entries only become alertable after threshold seconds, and the check period is threshold / 2, so the old TTL could evict a genuinely hanging request before any check saw it cross the threshold. Fixes BerriAI#27855. * fix(slack_alerting): alert once per hanging request The min-age gate stops false positives for young in-flight requests, but a genuinely hanging request still re-alerted on every checker tick within the cache TTL. With the wider TTL (1.5x threshold + 60s) that is 1-2 extra Slack notifications per stuck request at the default 600s threshold. Flag a HangingRequestData entry as alerted once its alert fires and skip flagged entries on later ticks, so each hang produces exactly one alert. The cache reference is mutated in place, so the TTL is untouched and still handles cleanup. Adds a regression test asserting one alert across multiple ticks. Fixes BerriAI#27855. * fix(health): treat all-proxy-models keys as unrestricted in /health (BerriAI#30087) * fix(health): treat all-proxy-models keys as unrestricted in /health A key granted all model permissions stores the literal "all-proxy-models" marker in its models list. The /health access filter compared that marker against real model_names, so the model list filtered down to nothing and the WebUI health check returned healthy_count=0, unhealthy_count=0 with HTTP 503. Skip the filter (both the live path and the background-cache model_id scoping) when the marker is present, matching how auth_checks treats SpecialModelNames.all_proxy_models. Fixes BerriAI#29744. * fix(health): resolve all-team-models sentinel to the team allowlist Same failure shape as the all-proxy-models case: a key carrying the literal "all-team-models" entry matches no real model_name, so the /health access filter would zero out the model list. Resolve the sentinel to the key's team models when team_id is set, matching get_key_models in model_checks.py. Without a team_id the sentinel stays unresolved and matches nothing, denying rather than widening access, mirroring _resolve_key_models_for_auth_check. * feat(proxy): auto-enable drop_params for Claude Code requests (BerriAI#30218) * feat(proxy): auto-enable drop_params for Claude Code requests Claude Code identifies itself with a claude-cli/<version> user agent and sends Anthropic-specific params (top_k, thinking, etc.) on every request. When the proxy routes those requests to a non-Anthropic provider, the unsupported params fail the call unless drop_params is configured. Detect the Claude Code user agent in add_litellm_data_to_request and default drop_params to true for those requests, without overriding an explicit drop_params value sent by the caller. * feat(proxy): respect operator litellm_settings drop_params over Claude Code default An explicit drop_params in the operator's litellm_settings (true or false) now suppresses the Claude Code user agent default, so an operator who deliberately configured drop_params: false keeps strict param validation for Claude Code clients too. The auto-default only fills the gap when neither the request body nor the config sets a value. * fix(snowflake): migrate to native endpoints with auto-routing for Claude models (BerriAI#29964) * fix(snowflake): migrate to native Cortex REST API endpoints Replaces the legacy /api/v2/cortex/inference:complete endpoint with the native OpenAI-compatible /api/v2/cortex/v1/chat/completions endpoint, fixing error 390142 (Incoming request does not contain a valid payload) when using model: snowflake/<model> in LiteLLM proxy. Changes: - litellm/llms/snowflake/chat/transformation.py: route to native /cortex/v1/chat/completions, remove Snowflake-specific tool_spec payload transformation, remove content_list response handling, add stream to supported params - litellm/llms/snowflake/anthropic/transformation.py (new): SnowflakeCortexAnthropicConfig routes Claude models to /cortex/v1/messages with anthropic-version header and Anthropic->OpenAI response transform - tests: 29 unit tests covering URL routing, auth headers, payload format, and response parsing * fix(snowflake): map max_tokens to max_completion_tokens for native endpoint * fix: handle multi-turn tool conversations and OpenAI→Anthropic tool format conversion - _extract_system_and_messages now preserves tool_calls from assistant messages and converts them to Anthropic tool_use content blocks - tool role messages are converted to user role with tool_result content blocks (as required by Anthropic Messages API) - Added _transform_tools_to_anthropic() to convert OpenAI tool format (type/function/parameters) to Anthropic format (name/input_schema) - Added comprehensive tests for multi-turn tool conversations Addresses review feedback on PR BerriAI#29964 * test: add coverage for malformed JSON and non-string tool arguments * fix(tests): update chat transformation tests for native OpenAI-compatible endpoint * style: apply black formatting * fix: resolve mypy type errors in anthropic transformation * fix: correct mypy type: ignore error codes (attr-defined) * fix: use max_tokens instead of max_completion_tokens for Snowflake endpoint compatibility * refactor: merge Anthropic config into unified SnowflakeConfig with auto-routing - Remove separate SnowflakeCortexAnthropicConfig and anthropic/ directory - SnowflakeConfig now auto-routes based on model name: - Claude models → /messages endpoint (Anthropic format) - All others → /chat/completions endpoint (OpenAI format) - No new provider needed (stays as SNOWFLAKE = 'snowflake') - Tool message transformation for Claude: tool_calls → tool_use blocks, tool role → user with tool_result - OpenAI → Anthropic tool format conversion (parameters → input_schema) - Addresses Greptile feedback about unwired SnowflakeCortexAnthropicConfig * fix: use max_completion_tokens for /chat/completions (Snowflake deprecated max_tokens on this endpoint) * fix(tests): update assertions for Claude auto-routing to /messages endpoint * fix(snowflake): add tool_choice conversion and preserve max_completion_tokens in Anthropic path * fix(snowflake): use ChatCompletionMessageToolCall objects and strip model prefix on OpenAI path * fix(snowflake): collect multiple system messages to prevent guardrail override * chore: remove committed .pyc files and add __pycache__ to .gitignore * fix: remove unused Union import * fix: restore original .gitignore (accidentally replaced in earlier commit) * feat(snowflake): add streaming response handler for both Anthropic and OpenAI SSE formats * fix: remove unused AsyncIterator and Iterator imports * fix: add missing total_tokens to ChatCompletionUsageBlock * fix(snowflake): coalesce consecutive tool results into single user message for Anthropic * fix(snowflake): handle message_start event for streaming input_tokens tracking * fix: evict last deleted model in multi-instance deployments (BerriAI#28608) * fix: evict last deleted model in multi-instance deployments _delete_deployment had an early return when db_models was empty, preventing eviction of the last deleted model during reconciliation. - Remove len(db_models)==0 early return from _delete_deployment - Return None (not []) from _get_models_from_db on DB failure so callers can distinguish a transient failure from a genuinely empty DB - Guard _update_llm_router against None to skip updates on DB failure Fixes BerriAI#28443 * test: remove dead MagicMock assignment in type_mismatch test * fix: update test to pass [] not None to _update_llm_router test_ProxyConfig__update_llm_router_bad_proxy_logging_raises was passing None as new_models to get through to the proxy_logging_obj check, but the None guard we added now returns early before reaching that path. Pass [] instead so the test exercises the intended AttributeError case. Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> * chore: regenerate API types to sync schema.d.ts with proxy OpenAPI spec Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> --------- Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> * fix: invalidate Redis spend counter on /key/reset_spend (BerriAI#29694) * fix: set Redis spend counter to reset_to value on /key/reset_spend Previously, the Redis spend counter was always set to 0.0 after a reset, even when reset_to was a non-zero value (partial reset). This caused the budget to be under-enforced for up to 60 seconds until the counter expired and fell through to the DB. Now the counter is set to the actual reset_to value, so partial resets are reflected correctly and budget enforcement is consistent. * test: update reset_key_spend test to match direct cache set The implementation now sets spend_counter_cache directly instead of calling _invalidate_spend_counter. Update the test to verify the in_memory_cache.set_cache call with the correct key, value, and ttl. --------- Co-authored-by: michaelxer <michaelxer@users.noreply.github.com> * fix: add scaleway models pricing (BerriAI#27659) * fix: Add embeddings support for Scaleway provider * fix: resolve merge conflicts * fix(main): clarify backend route handling for Swagger static assets (BerriAI#30196) * fix(main): clarify backend route handling for Swagger static assets * fix(allowlist): add BACKEND_MOUNT_PATHS for Swagger static assets * fix(voyage): route multimodal embeddings to correct endpoint (BerriAI#30193) * fix(voyage): route multimodal embeddings to correct endpoint * test(voyage): cover multimodal embedding edge cases * test(voyage): cover api key fallback * fix(voyage): raise early on missing api key and malformed image url * test(voyage): cover utils routing and helper * fix(voyage): route supported openai params for multimodal models * style: apply black formatting * fix(ui): infer Azure API version from API base (BerriAI#30204) * fix(ui): infer Azure API version from API base * fix(ui): address Azure API version feedback * Update litellm/llms/snowflake/chat/transformation.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * feat(datadog): add team-scoped Datadog callback support (BerriAI#29947) Enable teams to configure their own Datadog credentials via POST /team/{team_id}/callback, following the same pattern as Langfuse. * Merge pull request BerriAI#29528 from aanchal22/litellm_byok-alias-merge fix(proxy): atomic merge for team model aliases and team.models on BYOK create * feat: add EmpirioLabs as an OpenAI-compatible provider (BerriAI#30278) Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com> * fix: resolve failing tests and lint in snowflake/team endpoints - Black-format snowflake/chat/transformation.py to fix lint failure - Update Anthropic config test to expect default max_tokens of 4096 (matches implementation) - Add AsyncMock + execute_raw mock to team_model_add cache-refresh pin test - Add model_dump mock and patch cache/logging in test_uses_atomic_array_append_with_dedup Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(test): update test_db_error_new_model_check for new _delete_deployment logic _delete_deployment no longer short-circuits on empty db_models — it now treats [] as a valid empty-DB state and proceeds to check config models. Mock get_config to return the two router deployments so they appear in combined_id_list and are protected, which matches the real-world scenario where a DB error occurs but the models are config-backed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list (BerriAI#30295) * feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list Follow-up to BerriAI#30223 per maintainer review: documents the flag in ConfigGeneralSettings with a short description and adds it to allowed_args in get_config_list so the UI and /config/list expose it. A test pins that /config/list returns the field with type Boolean, which requires both registrations to be present * chore(ui): regenerate schema.d.ts for cancel_on_disconnect --------- Co-authored-by: kursad <kursad.lacin@brado.net> * fix(datadog): never fall back to env DD_API_KEY for caller-supplied destinations Team/key-scoped Datadog loggers could be pointed at an arbitrary dd_agent_host or dd_site while omitting dd_api_key, causing the proxy's global DD_API_KEY to be sent as the DD-API-KEY header to that destination. Gate the env-var fallback behind an allow_env_credentials flag, set to False when the destination is caller-supplied, mirroring the existing langfuse/langsmith pattern. --------- Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: daitran-tensormesh <dai@tensormesh.ai> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Muspi Merol <me@promplate.dev> Co-authored-by: fangkang <fangkangm@gmail.com> Co-authored-by: Chris Hoogeboom <chris.hoogeboom@gmail.com> Co-authored-by: kursadlacin <kursadlacin@gmail.com> Co-authored-by: kursad <kursad.lacin@brado.net> Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> Co-authored-by: hcl <chenglunhu@gmail.com> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: sfc-gh-nashukla <navnit.shukla@snowflake.com> Co-authored-by: Rudra Dudhat <contact.rdudhat@gmail.com> Co-authored-by: Michael <52305679+michaelxer@users.noreply.github.com> Co-authored-by: michaelxer <michaelxer@users.noreply.github.com> Co-authored-by: Quentin Champenois <26109239+Quentinchampenois@users.noreply.github.com> Co-authored-by: mauriceberentsen <mauriceberentsen@live.nl> Co-authored-by: lost9999 <56498264+lost9999@users.noreply.github.com> Co-authored-by: GaetanVDB07 <86427581+GaetanVDB07@users.noreply.github.com> Co-authored-by: Aanchal Khandelwal <aan2210khandelwal@gmail.com> Co-authored-by: Adam Dalloul <adam_dalloul@icloud.com> Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
fzowl
pushed a commit
to fzowl/litellm
that referenced
this pull request
Jun 24, 2026
* feat(bedrock): add bedrock mantle gemma 4 models (BerriAI#30264) * feat(bedrock): add bedrock mantle gemma 4 models * test(bedrock): harden mantle local cost fixture * feat(responses): enable the responses API for the Tensormesh provider (BerriAI#30209) * feat(responses): enable the responses API for the Tensormesh provider * Update litellm/llms/openai_like/providers.json Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(langfuse_otel): mark LLM spans as generations (BerriAI#30250) * fix(bedrock): stop stream_chunk_size leaking into invoke request bodies (BerriAI#30240) stream_chunk_size is a LiteLLM-internal knob for re-chunking the HTTP response stream. The invoke transformations splat optional_params into the provider request body without dropping it, and Bedrock rejects unknown fields, so any bedrock/invoke request that sets the parameter fails with ValidationException: stream_chunk_size: Extra inputs are not permitted. Drop it in the invoke dispatcher (covers cohere, titan, mistral, meta, ai21) and in the Claude messages-format request builder (the route used for bedrock/invoke Anthropic models) * fix(bedrock): stop buffering streamed tool-call argument deltas (BerriAI#30231) * fix(bedrock): stop buffering streamed tool-call argument deltas Two issues made Bedrock tool-use streaming arrive as a single end-of-stream burst through LiteLLM while plain text streamed fine. First, the anthropic-beta allowlist mapped fine-grained-tool-streaming-2025-05-14 to null for bedrock and bedrock_converse, so the header was silently stripped. Without that beta, Anthropic models on Bedrock buffer tool input server-side and emit all toolUse.input deltas at once (verified against converse-stream and invoke-with-response-stream directly). Bedrock accepts the beta via additionalModelRequestFields.anthropic_beta, so it is now forwarded. Second, the streaming reads re-chunked the AWS event stream with iter_bytes(chunk_size=1024). httpx's ByteChunker only releases full 1024-byte blocks, so the small early events (messageStart, contentBlockStart, first deltas) sat in the buffer until enough bytes accumulated, pushing time-to-first-byte from ~1.4s to ~8.5s on buffered tool-use streams. The default is now no re-chunking; an explicit stream_chunk_size is still honored. * test(bedrock): cover explicit stream_chunk_size on sync invoke path * test(bedrock): cover stream_chunk_size plumbing through converse completion * test(bedrock): cover stream_chunk_size default in legacy BedrockLLM streaming * test(bedrock): merge converse handler tests into existing mapped test file pytest imports test modules by basename in non-package test dirs, so the new tests/test_litellm/llms/bedrock/chat/test_converse_handler.py collided with the pre-existing tests/test_litellm/llms/chat/test_converse_handler.py and broke collection in CI. Move the new tests into the existing file * feat(otel): emit v2 cost breakdown + stamp tracer scope version (BerriAI#30156) Read the StandardLoggingPayload cost_breakdown into a typed LLMCost on LLMCallSpanData and emit each component under litellm.cost.* (absent components omitted, so spans stay sparse). Stamp litellm.__version__ as the instrumentation scope version so every v2 span carries a deterministic scope.version. Tests under tests/test_litellm/integrations/otel/. * fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in) (BerriAI#30223) * fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in) On the non-streaming path, base_process_llm_request awaited the LLM call with no disconnect monitoring; when the HTTP client went away the upstream request kept running until completion or request_timeout (6000s default), holding a backend slot (e.g. a vLLM GPU slot) for output nobody would read Add an opt-in general_settings.cancel_on_disconnect flag, default off, so the default code path is unchanged. When enabled, a receive-based watcher task observes http.disconnect and cancels the asyncio.gather driving the upstream call. The resulting CancelledError is converted to HTTPException 499 only when the disconnect event is set, so server-initiated cancellations still propagate as-is. The 499 then flows through _handle_llm_api_exception like any other failure, meaning post_call_failure_hook still releases max_parallel_requests slots and fires spend and alerting callbacks; it is logged at info level instead of a full traceback Also removes the dead check_request_disconnection helper in proxy_server.py (zero call sites) along with its behavior-pin tests Builds on the receive-based design from BerriAI#25776 Addresses BerriAI#13774. Re-fixes BerriAI#22805 (regressed after the BerriAI#14295 revert) Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> * fix(proxy): scope 499 quiet logging to disconnects and harden watcher Address the two P2 findings from the Greptile review on BerriAI#30223. The info-level logging in _log_llm_api_exception now applies only to the disconnect-specific HTTPException (status 499 plus the shared _CLIENT_DISCONNECT_DETAIL message), so any other 499 raised by hooks or guardrails keeps its full traceback. The disconnect watcher now catches exceptions from request.receive() (e.g. a transport reset) and logs a warning instead of dying silently, making the degradation to no-op visible; a test pins that the LLM call is not cancelled in that case --------- Co-authored-by: kursad <kursad.lacin@brado.net> Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> * fix(bedrock): grant aws-external-anthropic:* in OIDC session policy for claude_platform (BerriAI#30200) (BerriAI#30205) The inline STS session policy passed to assume_role_with_web_identity acts as an IAM PERMISSION CEILING — effective permissions are the intersection of the role's identity policies and this policy. Any action not listed is silently denied even when the IAM role grants it. BerriAI#27678 added the bedrock/claude_platform/<model> route but its service-side action namespace is aws-external-anthropic:*, not bedrock:*. Without a matching statement here, every claude_platform request via OIDC (GCP federation, EKS Pod Identity webhook, etc.) 403s with 'no session policy allows the aws-external-anthropic:CreateInference action' — even with a fully permissive identity policy. Add a second ClaudePlatformLiteLLM statement covering CreateInference, CreateBatchInference, CancelBatchInference, DeleteBatchInference, CountTokens, Get*, List*. Keep aws:SecureTransport=true parity with the bedrock statement. Static creds + IRSA flow through different code paths and are not affected. Fixes BerriAI#30200 * fix(proxy): set Retry-After header on RouterRateLimitError 429 responses (BerriAI#30098) * Set Retry-After header on RouterRateLimitError responses When all deployments for a model are in cooldown, the proxy returns a 429 whose cooldown timing is only available by parsing the error message string. RouterRateLimitError already carries cooldown_time, so expose it as a standard retry-after header in _handle_llm_api_exception. The value is rounded up so clients never retry before the cooldown window ends. Fixes BerriAI#27823. * Set Retry-After after response-headers hook so cooldown wins The cooldown-derived retry-after was assigned before the post_call_response_headers_hook merge, so a callback returning a retry-after key (including a stale or empty value) silently clobbered it. Move the RouterRateLimitError block after the callback merge so the cooldown value is authoritative for this error type. * fix(router): route aspeech through async_function_with_fallbacks (BerriAI#30104) * fix(router): route aspeech through async_function_with_fallbacks Router.aspeech selected a deployment and awaited litellm.aspeech directly, so TTS requests got no retry on failure and no failover to backup deployments; the except block only fired an exception alert and re-raised. Every other router endpoint (acompletion, aembedding, atranscription, arerank) already delegates to async_function_with_fallbacks Mirror the atranscription pattern: move deployment selection and the litellm.aspeech call into a private _aspeech method, then have the public aspeech set kwargs["original_function"] = self._aspeech and await self.async_function_with_fallbacks(**kwargs). _aspeech also picks up the shared _get_async_openai_model_client helper and the same total/success/fail call accounting the sibling endpoints use Fixes BerriAI#27778. * fix(router): apply deployment kwargs and rpm semaphore in _aspeech Bring _aspeech fully in line with _atranscription: call _update_kwargs_with_deployment so deployment metadata, model_info, timeout, and default litellm params flow into the request, and wrap the litellm.aspeech call with the max_parallel_requests semaphore plus async_routing_strategy_pre_call_checks so TTS respects rpm limits the same way the other router endpoints do Also add a unit test that exercises _aspeech directly and asserts the deployment metadata reaches the underlying call * fix(slack_alerting): stop false-positive hanging request alerts for requests below the alerting threshold (BerriAI#30106) * fix(slack_alerting): skip hanging request alerts below the threshold The hanging request check alerted on any cached request whose completion status was not yet recorded, with no minimum age check. Since the background loop runs every alerting_threshold / 2 seconds, any request that happened to be in flight at a check fired a "hanging - Ns+ request time" alert even if it was only seconds old, producing a steady stream of false positives. Add a created_at timestamp to HangingRequestData, stamped when the request enters the hanging request cache, and skip requests younger than alerting_threshold without evicting them, so a later check can still alert if they never complete. Extend the cache TTL from threshold + 60s to 1.5x threshold + 60s; with the age check, entries only become alertable after threshold seconds, and the check period is threshold / 2, so the old TTL could evict a genuinely hanging request before any check saw it cross the threshold. Fixes BerriAI#27855. * fix(slack_alerting): alert once per hanging request The min-age gate stops false positives for young in-flight requests, but a genuinely hanging request still re-alerted on every checker tick within the cache TTL. With the wider TTL (1.5x threshold + 60s) that is 1-2 extra Slack notifications per stuck request at the default 600s threshold. Flag a HangingRequestData entry as alerted once its alert fires and skip flagged entries on later ticks, so each hang produces exactly one alert. The cache reference is mutated in place, so the TTL is untouched and still handles cleanup. Adds a regression test asserting one alert across multiple ticks. Fixes BerriAI#27855. * fix(health): treat all-proxy-models keys as unrestricted in /health (BerriAI#30087) * fix(health): treat all-proxy-models keys as unrestricted in /health A key granted all model permissions stores the literal "all-proxy-models" marker in its models list. The /health access filter compared that marker against real model_names, so the model list filtered down to nothing and the WebUI health check returned healthy_count=0, unhealthy_count=0 with HTTP 503. Skip the filter (both the live path and the background-cache model_id scoping) when the marker is present, matching how auth_checks treats SpecialModelNames.all_proxy_models. Fixes BerriAI#29744. * fix(health): resolve all-team-models sentinel to the team allowlist Same failure shape as the all-proxy-models case: a key carrying the literal "all-team-models" entry matches no real model_name, so the /health access filter would zero out the model list. Resolve the sentinel to the key's team models when team_id is set, matching get_key_models in model_checks.py. Without a team_id the sentinel stays unresolved and matches nothing, denying rather than widening access, mirroring _resolve_key_models_for_auth_check. * feat(proxy): auto-enable drop_params for Claude Code requests (BerriAI#30218) * feat(proxy): auto-enable drop_params for Claude Code requests Claude Code identifies itself with a claude-cli/<version> user agent and sends Anthropic-specific params (top_k, thinking, etc.) on every request. When the proxy routes those requests to a non-Anthropic provider, the unsupported params fail the call unless drop_params is configured. Detect the Claude Code user agent in add_litellm_data_to_request and default drop_params to true for those requests, without overriding an explicit drop_params value sent by the caller. * feat(proxy): respect operator litellm_settings drop_params over Claude Code default An explicit drop_params in the operator's litellm_settings (true or false) now suppresses the Claude Code user agent default, so an operator who deliberately configured drop_params: false keeps strict param validation for Claude Code clients too. The auto-default only fills the gap when neither the request body nor the config sets a value. * fix(snowflake): migrate to native endpoints with auto-routing for Claude models (BerriAI#29964) * fix(snowflake): migrate to native Cortex REST API endpoints Replaces the legacy /api/v2/cortex/inference:complete endpoint with the native OpenAI-compatible /api/v2/cortex/v1/chat/completions endpoint, fixing error 390142 (Incoming request does not contain a valid payload) when using model: snowflake/<model> in LiteLLM proxy. Changes: - litellm/llms/snowflake/chat/transformation.py: route to native /cortex/v1/chat/completions, remove Snowflake-specific tool_spec payload transformation, remove content_list response handling, add stream to supported params - litellm/llms/snowflake/anthropic/transformation.py (new): SnowflakeCortexAnthropicConfig routes Claude models to /cortex/v1/messages with anthropic-version header and Anthropic->OpenAI response transform - tests: 29 unit tests covering URL routing, auth headers, payload format, and response parsing * fix(snowflake): map max_tokens to max_completion_tokens for native endpoint * fix: handle multi-turn tool conversations and OpenAI→Anthropic tool format conversion - _extract_system_and_messages now preserves tool_calls from assistant messages and converts them to Anthropic tool_use content blocks - tool role messages are converted to user role with tool_result content blocks (as required by Anthropic Messages API) - Added _transform_tools_to_anthropic() to convert OpenAI tool format (type/function/parameters) to Anthropic format (name/input_schema) - Added comprehensive tests for multi-turn tool conversations Addresses review feedback on PR BerriAI#29964 * test: add coverage for malformed JSON and non-string tool arguments * fix(tests): update chat transformation tests for native OpenAI-compatible endpoint * style: apply black formatting * fix: resolve mypy type errors in anthropic transformation * fix: correct mypy type: ignore error codes (attr-defined) * fix: use max_tokens instead of max_completion_tokens for Snowflake endpoint compatibility * refactor: merge Anthropic config into unified SnowflakeConfig with auto-routing - Remove separate SnowflakeCortexAnthropicConfig and anthropic/ directory - SnowflakeConfig now auto-routes based on model name: - Claude models → /messages endpoint (Anthropic format) - All others → /chat/completions endpoint (OpenAI format) - No new provider needed (stays as SNOWFLAKE = 'snowflake') - Tool message transformation for Claude: tool_calls → tool_use blocks, tool role → user with tool_result - OpenAI → Anthropic tool format conversion (parameters → input_schema) - Addresses Greptile feedback about unwired SnowflakeCortexAnthropicConfig * fix: use max_completion_tokens for /chat/completions (Snowflake deprecated max_tokens on this endpoint) * fix(tests): update assertions for Claude auto-routing to /messages endpoint * fix(snowflake): add tool_choice conversion and preserve max_completion_tokens in Anthropic path * fix(snowflake): use ChatCompletionMessageToolCall objects and strip model prefix on OpenAI path * fix(snowflake): collect multiple system messages to prevent guardrail override * chore: remove committed .pyc files and add __pycache__ to .gitignore * fix: remove unused Union import * fix: restore original .gitignore (accidentally replaced in earlier commit) * feat(snowflake): add streaming response handler for both Anthropic and OpenAI SSE formats * fix: remove unused AsyncIterator and Iterator imports * fix: add missing total_tokens to ChatCompletionUsageBlock * fix(snowflake): coalesce consecutive tool results into single user message for Anthropic * fix(snowflake): handle message_start event for streaming input_tokens tracking * fix: evict last deleted model in multi-instance deployments (BerriAI#28608) * fix: evict last deleted model in multi-instance deployments _delete_deployment had an early return when db_models was empty, preventing eviction of the last deleted model during reconciliation. - Remove len(db_models)==0 early return from _delete_deployment - Return None (not []) from _get_models_from_db on DB failure so callers can distinguish a transient failure from a genuinely empty DB - Guard _update_llm_router against None to skip updates on DB failure Fixes BerriAI#28443 * test: remove dead MagicMock assignment in type_mismatch test * fix: update test to pass [] not None to _update_llm_router test_ProxyConfig__update_llm_router_bad_proxy_logging_raises was passing None as new_models to get through to the proxy_logging_obj check, but the None guard we added now returns early before reaching that path. Pass [] instead so the test exercises the intended AttributeError case. Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> * chore: regenerate API types to sync schema.d.ts with proxy OpenAPI spec Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> --------- Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> * fix: invalidate Redis spend counter on /key/reset_spend (BerriAI#29694) * fix: set Redis spend counter to reset_to value on /key/reset_spend Previously, the Redis spend counter was always set to 0.0 after a reset, even when reset_to was a non-zero value (partial reset). This caused the budget to be under-enforced for up to 60 seconds until the counter expired and fell through to the DB. Now the counter is set to the actual reset_to value, so partial resets are reflected correctly and budget enforcement is consistent. * test: update reset_key_spend test to match direct cache set The implementation now sets spend_counter_cache directly instead of calling _invalidate_spend_counter. Update the test to verify the in_memory_cache.set_cache call with the correct key, value, and ttl. --------- Co-authored-by: michaelxer <michaelxer@users.noreply.github.com> * fix: add scaleway models pricing (BerriAI#27659) * fix: Add embeddings support for Scaleway provider * fix: resolve merge conflicts * fix(main): clarify backend route handling for Swagger static assets (BerriAI#30196) * fix(main): clarify backend route handling for Swagger static assets * fix(allowlist): add BACKEND_MOUNT_PATHS for Swagger static assets * fix(voyage): route multimodal embeddings to correct endpoint (BerriAI#30193) * fix(voyage): route multimodal embeddings to correct endpoint * test(voyage): cover multimodal embedding edge cases * test(voyage): cover api key fallback * fix(voyage): raise early on missing api key and malformed image url * test(voyage): cover utils routing and helper * fix(voyage): route supported openai params for multimodal models * style: apply black formatting * fix(ui): infer Azure API version from API base (BerriAI#30204) * fix(ui): infer Azure API version from API base * fix(ui): address Azure API version feedback * Update litellm/llms/snowflake/chat/transformation.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * feat(datadog): add team-scoped Datadog callback support (BerriAI#29947) Enable teams to configure their own Datadog credentials via POST /team/{team_id}/callback, following the same pattern as Langfuse. * Merge pull request BerriAI#29528 from aanchal22/litellm_byok-alias-merge fix(proxy): atomic merge for team model aliases and team.models on BYOK create * feat: add EmpirioLabs as an OpenAI-compatible provider (BerriAI#30278) Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com> * fix: resolve failing tests and lint in snowflake/team endpoints - Black-format snowflake/chat/transformation.py to fix lint failure - Update Anthropic config test to expect default max_tokens of 4096 (matches implementation) - Add AsyncMock + execute_raw mock to team_model_add cache-refresh pin test - Add model_dump mock and patch cache/logging in test_uses_atomic_array_append_with_dedup * fix(test): update test_db_error_new_model_check for new _delete_deployment logic _delete_deployment no longer short-circuits on empty db_models — it now treats [] as a valid empty-DB state and proceeds to check config models. Mock get_config to return the two router deployments so they appear in combined_id_list and are protected, which matches the real-world scenario where a DB error occurs but the models are config-backed. * feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list (BerriAI#30295) * feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list Follow-up to BerriAI#30223 per maintainer review: documents the flag in ConfigGeneralSettings with a short description and adds it to allowed_args in get_config_list so the UI and /config/list expose it. A test pins that /config/list returns the field with type Boolean, which requires both registrations to be present * chore(ui): regenerate schema.d.ts for cancel_on_disconnect --------- Co-authored-by: kursad <kursad.lacin@brado.net> * fix(datadog): never fall back to env DD_API_KEY for caller-supplied destinations Team/key-scoped Datadog loggers could be pointed at an arbitrary dd_agent_host or dd_site while omitting dd_api_key, causing the proxy's global DD_API_KEY to be sent as the DD-API-KEY header to that destination. Gate the env-var fallback behind an allow_env_credentials flag, set to False when the destination is caller-supplied, mirroring the existing langfuse/langsmith pattern. --------- Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: daitran-tensormesh <dai@tensormesh.ai> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Muspi Merol <me@promplate.dev> Co-authored-by: fangkang <fangkangm@gmail.com> Co-authored-by: Chris Hoogeboom <chris.hoogeboom@gmail.com> Co-authored-by: kursadlacin <kursadlacin@gmail.com> Co-authored-by: kursad <kursad.lacin@brado.net> Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> Co-authored-by: hcl <chenglunhu@gmail.com> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: sfc-gh-nashukla <navnit.shukla@snowflake.com> Co-authored-by: Rudra Dudhat <contact.rdudhat@gmail.com> Co-authored-by: Michael <52305679+michaelxer@users.noreply.github.com> Co-authored-by: michaelxer <michaelxer@users.noreply.github.com> Co-authored-by: Quentin Champenois <26109239+Quentinchampenois@users.noreply.github.com> Co-authored-by: mauriceberentsen <mauriceberentsen@live.nl> Co-authored-by: lost9999 <56498264+lost9999@users.noreply.github.com> Co-authored-by: GaetanVDB07 <86427581+GaetanVDB07@users.noreply.github.com> Co-authored-by: Aanchal Khandelwal <aan2210khandelwal@gmail.com> Co-authored-by: Adam Dalloul <adam_dalloul@icloud.com> Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com>
blake-hamm
added a commit
to blake-hamm/bhamm-lab
that referenced
this pull request
Jun 28, 2026
…to v1.90.0 (#232) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [https://github.com/BerriAI/litellm.git](https://github.com/BerriAI/litellm) | minor | `v1.89.4` → `v1.90.0` | --- ### Release Notes <details> <summary>BerriAI/litellm (https://github.com/BerriAI/litellm.git)</summary> ### [`v1.90.0`](https://github.com/BerriAI/litellm/releases/tag/v1.90.0) [Compare Source](BerriAI/litellm@v1.89.4...v1.90.0-rc.1) #### Verify Docker Image Signature All LiteLLM Docker images are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](BerriAI/litellm@0112e53). **Verify using the pinned commit hash (recommended):** A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \ ghcr.io/berriai/litellm:v1.90.0 ``` **Verify using the release tag (convenience):** Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/v1.90.0/cosign.pub \ ghcr.io/berriai/litellm:v1.90.0 ``` Expected output: ``` The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key ``` *** #### What's Changed - fix(responses-bridge): map system-only chat request to system input item by [@​milan-berri](https://github.com/milan-berri) in [#​29817](BerriAI/litellm#29817) - feat(bedrock): forward strict and additionalProperties to Converse toolSpec by [@​mateo-berri](https://github.com/mateo-berri) in [#​29814](BerriAI/litellm#29814) - fix(mcp): highlight MCP cards red when the logged-in user is missing per-user env vars by [@​mateo-berri](https://github.com/mateo-berri) in [#​29856](BerriAI/litellm#29856) - feat(ui): add budget duration to edit team member form by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29717](BerriAI/litellm#29717) - fix(ui): make workflow runs page fill full width by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29868](BerriAI/litellm#29868) - feat: standardize rate limit errors with category, rate\_limit\_type, model, and llm\_provider fields by [@​mateo-berri](https://github.com/mateo-berri) in [#​27687](BerriAI/litellm#27687) - fix(ui): default guardrails page to the Guardrails tab by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29872](BerriAI/litellm#29872) - docs(readme): add Deploy on AWS/GCP Terraform section and fix deploy button rendering by [@​mateo-berri](https://github.com/mateo-berri) in [#​29879](BerriAI/litellm#29879) - refactor(bedrock): build Converse toolSpec via a BedrockToolSpec dict subclass by [@​mateo-berri](https://github.com/mateo-berri) in [#​29869](BerriAI/litellm#29869) - feat(litellm): add models and repository layers by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​29686](BerriAI/litellm#29686) - feat(ui): include internal routes in the dashboard's generated OpenAPI types by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29885](BerriAI/litellm#29885) - feat(proxy): publish /v2/model/info in Swagger OpenAPI spec by [@​Sameerlite](https://github.com/Sameerlite) in [#​29900](BerriAI/litellm#29900) - refactor(ui): single source of truth for migrated-page routing by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29949](BerriAI/litellm#29949) - fix(ui/model-hub): render provider icons on the public model hub by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29958](BerriAI/litellm#29958) - fix(ui): keep create guardrail modal open on outside click by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29871](BerriAI/litellm#29871) - fix(ui): label default key type as "Full Access" on key edit page by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29870](BerriAI/litellm#29870) - fix(ui): unify migrated-route URLs and migrate the API Reference page by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29953](BerriAI/litellm#29953) - fix(mcp): let non-creator users OAuth into OBO-mode MCP servers from the Tools page by [@​tin-berri](https://github.com/tin-berri) in [#​29867](BerriAI/litellm#29867) - Litellm oss staging 080626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​29932](BerriAI/litellm#29932) - feat(galileo): add health check support for UI callback test by [@​Sameerlite](https://github.com/Sameerlite) in [#​29908](BerriAI/litellm#29908) - fix(model-management): allow deleting a BYOK model after its team is deleted by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29875](BerriAI/litellm#29875) - feat(jwt-auth): opt-in fallback to DB team on unresolved JWT claim by [@​milan-berri](https://github.com/milan-berri) in [#​28913](BerriAI/litellm#28913) - fix(team\_endpoints): don't block /team/update on unchanged team budget by [@​milan-berri](https://github.com/milan-berri) in [#​29525](BerriAI/litellm#29525) - fix(fireworks): enable tool calling for glm-5p1 in model cost map by [@​milan-berri](https://github.com/milan-berri) in [#​29697](BerriAI/litellm#29697) - fix(vertex): propagate Vertex AI metadata in streaming success callbacks by [@​Sameerlite](https://github.com/Sameerlite) in [#​29899](BerriAI/litellm#29899) - fix(ui): show team projects to internal users on key creation by [@​milan-berri](https://github.com/milan-berri) in [#​28855](BerriAI/litellm#28855) - build(deps): bump pyjwt to 2.13.0 and ws override to 8.20.1 by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29982](BerriAI/litellm#29982) - fix(team-management): delete a team's BYOK models when the team is deleted by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29977](BerriAI/litellm#29977) - feat(vantage): include organization metadata in FOCUS Tags export by [@​milan-berri](https://github.com/milan-berri) in [#​28184](BerriAI/litellm#28184) - fix(guardrails): read CrowdStrike AIDR identity from both metadata bags by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29991](BerriAI/litellm#29991) - fix(mcp): mirror upstream token lifetime instead of forcing a 1h OBO expiry by [@​tin-berri](https://github.com/tin-berri) in [#​29951](BerriAI/litellm#29951) - feat(azure\_ai): add MAI-Image-2.5 image generation support by [@​Sameerlite](https://github.com/Sameerlite) in [#​29688](BerriAI/litellm#29688) - fix(mcp): load MCP tool configuration tools via the OBO/passthrough-aware GET path by [@​tin-berri](https://github.com/tin-berri) in [#​29960](BerriAI/litellm#29960) - fix(team): reserve team budget raises for proxy admins on /team/update by [@​milan-berri](https://github.com/milan-berri) in [#​30030](BerriAI/litellm#30030) - test(ui): data-driven App Router migration E2E smoke (default + server-root-path) by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29974](BerriAI/litellm#29974) - fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through by [@​michelligabriele](https://github.com/michelligabriele) in [#​24232](BerriAI/litellm#24232) - chore(ui): remove dead App Router route stubs under (dashboard) by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30045](BerriAI/litellm#30045) - fix(ui/mcp): reset OAuth state on create-server modal close so a prior server's token no longer leaks into the next add-server session by [@​tin-berri](https://github.com/tin-berri) in [#​30000](BerriAI/litellm#30000) - fix(mcp): allow team access-group grants in OAuth authorize/token access check by [@​tin-berri](https://github.com/tin-berri) in [#​30041](BerriAI/litellm#30041) - docs(security): require a reproduction video for vulnerability reports by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30063](BerriAI/litellm#30063) - feat(ui): add admin flag to disable in-product UI nudges for everyone by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29796](BerriAI/litellm#29796) - chore(ui): remove dead dashboard files and unused dependencies by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30047](BerriAI/litellm#30047) - fix(proxy): authorize batch files using upload target\_model\_names (LIT-3593) by [@​Sameerlite](https://github.com/Sameerlite) in [#​30009](BerriAI/litellm#30009) - Add Claude Fable 5 across Anthropic, Bedrock, Vertex AI, and Azure AI by [@​mateo-berri](https://github.com/mateo-berri) in [#​30064](BerriAI/litellm#30064) - Add Claude Fable 5 cost map entries (data-only hotfix for the hosted map) by [@​mateo-berri](https://github.com/mateo-berri) in [#​30076](BerriAI/litellm#30076) - fix(caching): restore stored prompt\_tokens on embedding cache hits instead of recomputing by [@​michelligabriele](https://github.com/michelligabriele) in [#​30046](BerriAI/litellm#30046) - Litellm oss 090626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​30021](BerriAI/litellm#30021) - fix(proxy): self-heal startup/reload prisma reads on engine disconnect by [@​michelligabriele](https://github.com/michelligabriele) in [#​28803](BerriAI/litellm#28803) - chore(ui): make knip recognize .mjs scripts and openapi-typescript by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30052](BerriAI/litellm#30052) - fix(register\_model): preserve built-in cache pricing when registering custom overrides under unmapped keys by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30044](BerriAI/litellm#30044) - \[internal copy of [#​28007](BerriAI/litellm#28007)] Fix/gcp model garden streaming by [@​mateo-berri](https://github.com/mateo-berri) in [#​28363](BerriAI/litellm#28363) - feat(cli): per-agent `lite claude` / `codex` / `opencode` commands that wrap coding agents through the proxy by [@​mateo-berri](https://github.com/mateo-berri) in [#​29850](BerriAI/litellm#29850) - fix(callbacks): forward callback\_settings to callback initializers and guard consumers against non-dict values by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30161](BerriAI/litellm#30161) - fix(mcp): drop orphaned per-user credential rows when an MCP server is deleted by [@​tin-berri](https://github.com/tin-berri) in [#​30141](BerriAI/litellm#30141) - fix(proxy): recover from cached-plan errors by reconnecting the Prisma client by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​29983](BerriAI/litellm#29983) - feat(proxy): add option to disable server-side prepared statements for DB lookups by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​29984](BerriAI/litellm#29984) - fix(release): stop backport releases from overwriting the latest badge by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30005](BerriAI/litellm#30005) - feat: add conventional commits and coding guidelines by [@​mateo-berri](https://github.com/mateo-berri) in [#​30159](BerriAI/litellm#30159) - fix(proxy): return 5xx on DB infra errors during auth; reserve 401 for genuine auth failures by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​29986](BerriAI/litellm#29986) - fix(ui): dev server 404s on migrated-page links because uiBase hardcodes /ui by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30169](BerriAI/litellm#30169) - refactor(ui): consolidate dashboard to one shell in the (dashboard) layout by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30166](BerriAI/litellm#30166) - fix(proxy): align /v1/model/info with router deployments by [@​Sameerlite](https://github.com/Sameerlite) in [#​30025](BerriAI/litellm#30025) - fix: completion\_cost AttributeError on streaming Anthropic web\_search responses ([#​26153](BerriAI/litellm#26153)) by [@​ishaan-berri](https://github.com/ishaan-berri) in [#​27346](BerriAI/litellm#27346) - \[internal copy of [#​30137](BerriAI/litellm#30137)] perf(realtime): eliminate redundant per-frame JSON work on OpenAI realtime relay by [@​mateo-berri](https://github.com/mateo-berri) in [#​30142](BerriAI/litellm#30142) - feat(bedrock): aws\_bedrock\_project\_id for bedrock-mantle project / workspace association by [@​mateo-berri](https://github.com/mateo-berri) in [#​30163](BerriAI/litellm#30163) - chore(hooks): enforce Conventional Commits and Conventional Branches by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30174](BerriAI/litellm#30174) - feat(rate-limiter): allow opting out of v3 TPM reservation and Redis circuit breaker by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30211](BerriAI/litellm#30211) - feat(spend\_logs): opt-in native Postgres partitioning for SpendLogs retention by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​29466](BerriAI/litellm#29466) - feat(ui): migrate playground to path routing and colocate its files by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30185](BerriAI/litellm#30185) - feat(ui): migrate projects and access-groups to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30226](BerriAI/litellm#30226) - fix(proxy): coalesce NULL rollup metrics in aggregated daily-activity by [@​michelligabriele](https://github.com/michelligabriele) in [#​30151](BerriAI/litellm#30151) - fix(anthropic\_passthrough): resolve costing model from message\_start chunk, litellm\_params and model\_group instead of 'unknown' by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30160](BerriAI/litellm#30160) - feat(ui): migrate budgets, workflows, and guardrails-monitor to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30236](BerriAI/litellm#30236) - feat(ui): migrate mcp-servers, search-tools, tag-management, vector-stores, and memory to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30261](BerriAI/litellm#30261) - fix(a2a): forward agent\_extra\_headers through completion bridge by [@​mateo-berri](https://github.com/mateo-berri) in [#​28277](BerriAI/litellm#28277) - fix(gemini-live): forward audio buffer commit and correct Vertex PCM rate by [@​Sameerlite](https://github.com/Sameerlite) in [#​29946](BerriAI/litellm#29946) - fix(proxy): skip double-wrapping unified batch output file ids on retrieve by [@​Sameerlite](https://github.com/Sameerlite) in [#​30011](BerriAI/litellm#30011) - feat: litellm oss 110626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​30202](BerriAI/litellm#30202) - fix(docker): copy only runtime artifacts into the final image by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30243](BerriAI/litellm#30243) - feat(proxy): enforce key/team guardrails on bedrock passthrough routes by [@​Sameerlite](https://github.com/Sameerlite) in [#​30194](BerriAI/litellm#30194) - feat(gemini): forward web search tools in image generation by [@​Sameerlite](https://github.com/Sameerlite) in [#​30119](BerriAI/litellm#30119) - fix: bedrock mantle fixes by [@​Sameerlite](https://github.com/Sameerlite) in [#​30083](BerriAI/litellm#30083) - feat(proxy): add require\_managed\_files setting for file uploads by [@​Sameerlite](https://github.com/Sameerlite) in [#​30186](BerriAI/litellm#30186) - fix(mcp): honor server\_id for REST tool calls with shared upstream URLs by [@​Sameerlite](https://github.com/Sameerlite) in [#​30184](BerriAI/litellm#30184) - fix(responses): presidio PII masking for Azure WebSocket and streaming by [@​Sameerlite](https://github.com/Sameerlite) in [#​30003](BerriAI/litellm#30003) - feat(passthrough): add configurable pass-through request timeouts by [@​Sameerlite](https://github.com/Sameerlite) in [#​30266](BerriAI/litellm#30266) - fix(google\_genai): preserve complete SSE events in Vertex/Gemini image streaming by [@​Sameerlite](https://github.com/Sameerlite) in [#​30270](BerriAI/litellm#30270) - fix(proxy): populate access\_via\_team\_ids on /v1/model/info by [@​Sameerlite](https://github.com/Sameerlite) in [#​30274](BerriAI/litellm#30274) - chore(oss): litellm oss staging 120626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​30292](BerriAI/litellm#30292) - feat(ui): migrate policies, guardrails, prompts, tool-policies, and skills to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30263](BerriAI/litellm#30263) - feat(ui): migrate caching, cost-tracking, transform-request, ui-theme, and logs to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30267](BerriAI/litellm#30267) - fix(ui): gate dashboard layout on ui config load so deep links work under SERVER\_ROOT\_PATH by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30312](BerriAI/litellm#30312) - feat(ui): migrate admin-panel, logging-and-alerts, model-hub-table, and usage to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30268](BerriAI/litellm#30268) - fix(otel): cap metric attribute cardinality with include/exclude lists by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30257](BerriAI/litellm#30257) - fix(proxy): grace-period key rotation 401s; return deprecated-key lookup result directly by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30327](BerriAI/litellm#30327) - chore(deps): bump vitest, brace-expansion, pypdf and tornado by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30220](BerriAI/litellm#30220) - refactor(ui): remove unreachable /chat page by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30178](BerriAI/litellm#30178) - feat(ui): migrate agents and router-settings to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30323](BerriAI/litellm#30323) - feat: strengthen coding conventions in CLAUDE.md by [@​mateo-berri](https://github.com/mateo-berri) in [#​30333](BerriAI/litellm#30333) - feat(ui): cut the users page over to the /ui/users path route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30334](BerriAI/litellm#30334) - feat: ruff strict-rule suppressions baseline gate by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30303](BerriAI/litellm#30303) - feat(guardrails): add Cisco AI Defense integration ([#​28249](BerriAI/litellm#28249)) by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30338](BerriAI/litellm#30338) - chore(ui): remove dead UI components unreferenced by any page by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30340](BerriAI/litellm#30340) - ci: add osv-scanner lockfile scan workflow by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30222](BerriAI/litellm#30222) - fix(otel): record full error message on standard exception event in otel v2 by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30380](BerriAI/litellm#30380) - test(fireworks): mock whisper transcription tests instead of live calls by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30391](BerriAI/litellm#30391) - build(ui): pin esbuild to 0.28.1 via overrides by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30390](BerriAI/litellm#30390) - feat(ui): cut the organizations page over to the /ui/organizations path route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30336](BerriAI/litellm#30336) - fix(proxy): support SMTP implicit SSL (port 465) by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30395](BerriAI/litellm#30395) - fix(mcp): default Linear MCP registry entry to streamable HTTP by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30396](BerriAI/litellm#30396) - fix(ui): stop Virtual Keys page from infinite render loop by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30397](BerriAI/litellm#30397) - fix(streaming): guard raise\_on\_model\_repetition against empty choices by [@​shivamrawat1](https://github.com/shivamrawat1) in [#​30485](BerriAI/litellm#30485) - feat(otel-v2): emit the 6 gen\_ai.client.\* metrics at parity with v1 by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30326](BerriAI/litellm#30326) - fix(mcp): drop phantom 401 span on delegated OAuth2 tool calls by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30494](BerriAI/litellm#30494) - feat(ui): cut the teams page over to the /ui/teams path route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30343](BerriAI/litellm#30343) - fix(integrations): cap Anthropic cache\_control injection at 4 blocks by [@​shivamrawat1](https://github.com/shivamrawat1) in [#​30480](BerriAI/litellm#30480) - chore(codecov): add Batches, Videos, and Realtime components by [@​Sameerlite](https://github.com/Sameerlite) in [#​30517](BerriAI/litellm#30517) - test(batches): move orphan tests into tests/test\_litellm for CI coverage by [@​Sameerlite](https://github.com/Sameerlite) in [#​30510](BerriAI/litellm#30510) - fix(guardrails): run pre\_call hook once for model-level guardrails by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30543](BerriAI/litellm#30543) - fix(guardrails): stop re-initializing DB guardrails on every poll by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30542](BerriAI/litellm#30542) - chore(oss): litellm oss staging 150626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​30463](BerriAI/litellm#30463) - ci(lint): add blanket-noqa, dataclass-default, and unused-noqa Ruff rules by [@​mateo-berri](https://github.com/mateo-berri) in [#​30516](BerriAI/litellm#30516) - ci: ratchet lint and type-check gates (ruff preview, ANN, mypy, basedpyright) by [@​mateo-berri](https://github.com/mateo-berri) in [#​30379](BerriAI/litellm#30379) - fix(proxy): allow internal roles to access vector store CRUD routes by [@​shivamrawat1](https://github.com/shivamrawat1) in [#​30503](BerriAI/litellm#30503) - fix(otel): stamp gen\_ai.input/output.messages on v2 spans by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30548](BerriAI/litellm#30548) - fix(otel): export v2 gen\_ai client metrics to the configured meter provider by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30549](BerriAI/litellm#30549) - fix(bedrock): preserve cache\_control for ARN models in /v1/messages adapter by [@​mateo-berri](https://github.com/mateo-berri) in [#​29823](BerriAI/litellm#29823) - fix: greatly increase basedpyright slack by [@​mateo-berri](https://github.com/mateo-berri) in [#​30563](BerriAI/litellm#30563) - fix(budget): recompute budget\_reset\_at when budget\_duration changes on /budget/update by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30555](BerriAI/litellm#30555) - fix(otel): accept UPPER\_SNAKE\_CASE OTEL\_INSTRUMENTATION\_GENAI\_CAPTURE\_MESSAGE\_CONTENT in v2 by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30562](BerriAI/litellm#30562) - chore(lint): remove PLR0915 too-many-statements ruff rule by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30574](BerriAI/litellm#30574) - ci(lint): ratcheted type-discipline gate (mutable collections, casts, guards, kwargs, suppressions) by [@​mateo-berri](https://github.com/mateo-berri) in [#​30500](BerriAI/litellm#30500) - feat(proxy): add verification\_uri\_complete to CLI SSO device flow by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30571](BerriAI/litellm#30571) - chore: litellm oss staging160626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​30527](BerriAI/litellm#30527) - fix(guardrails): return 400 not 500 when AIM blocks a request by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30573](BerriAI/litellm#30573) - ci(lint): grandfather any-discipline with a per-file ratchet budget (50% headroom) by [@​mateo-berri](https://github.com/mateo-berri) in [#​30582](BerriAI/litellm#30582) - fix(audio): don't override explicit response\_format with verbose\_json by [@​mateo-berri](https://github.com/mateo-berri) in [#​30599](BerriAI/litellm#30599) - fix(anthropic): price and surface response service\_tier in cost tracking by [@​mateo-berri](https://github.com/mateo-berri) in [#​30558](BerriAI/litellm#30558) - feat: add dev and wildcard proxy configs for local testing by [@​mateo-berri](https://github.com/mateo-berri) in [#​30556](BerriAI/litellm#30556) - fix(proxy): list public team model name in /v1/models by [@​ishaan-berri](https://github.com/ishaan-berri) in [#​30588](BerriAI/litellm#30588) - ci: drop mypy entirely, standardize type checking on basedpyright by [@​mateo-berri](https://github.com/mateo-berri) in [#​30648](BerriAI/litellm#30648) - feat(guardrails): surface OpenAI moderation violation\_categories on guardrail traces by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30659](BerriAI/litellm#30659) - fix(proxy): resolve list files credentials from team BYOK deployments by [@​shivamrawat1](https://github.com/shivamrawat1) in [#​30495](BerriAI/litellm#30495) - feat(proxy): add --max\_requests\_before\_restart\_jitter to stagger worker restarts by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30601](BerriAI/litellm#30601) - fix(health): correct bedrock embedding health checks by [@​mateo-berri](https://github.com/mateo-berri) in [#​30583](BerriAI/litellm#30583) - test: harden remaining pass-through CI flakes (image-gen spend poll, ruby assistants timeout) by [@​mateo-berri](https://github.com/mateo-berri) in [#​30685](BerriAI/litellm#30685) - test(pass\_through): harden vertex spendlog poll against transient empty reads by [@​mateo-berri](https://github.com/mateo-berri) in [#​30683](BerriAI/litellm#30683) - fix(cost): stop non-string service\_tier from silently dropping cost tracking by [@​mateo-berri](https://github.com/mateo-berri) in [#​30690](BerriAI/litellm#30690) - feat(proxy): warn at startup when custom\_auth skips common\_checks enforcement by [@​tin-berri](https://github.com/tin-berri) in [#​30665](BerriAI/litellm#30665) - fix(pod\_lock): release cron lock by matching async\_set\_cache JSON encoding by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30600](BerriAI/litellm#30600) - ci: run a local fake OpenAI endpoint instead of the shared Railway mock by [@​mateo-berri](https://github.com/mateo-berri) in [#​30695](BerriAI/litellm#30695) - ci(windows): pin uv to Python 3.11 so it ignores the preinstalled 3.14 by [@​mateo-berri](https://github.com/mateo-berri) in [#​30704](BerriAI/litellm#30704) - feat(ui): migrate models page to App Router path route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30677](BerriAI/litellm#30677) - refactor(ui): remove orphaned pass-through-settings route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30692](BerriAI/litellm#30692) - fix(cost): stop non-string response service\_tier from dropping cost tracking by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30706](BerriAI/litellm#30706) - feat(agent-shin): automated PR/issue triage, low-quality auto-close, and review-gate label lifecycle by [@​mateo-berri](https://github.com/mateo-berri) in [#​30433](BerriAI/litellm#30433) - chore: litellm oss 170626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​30637](BerriAI/litellm#30637) - fix(bedrock\_mantle): add SigV4 fallback to chat completions auth by [@​mateo-berri](https://github.com/mateo-berri) in [#​30714](BerriAI/litellm#30714) - feat(search): add TinyFish as search provider by [@​simantak-dabhade](https://github.com/simantak-dabhade) in [#​30634](BerriAI/litellm#30634) - feat(ui): migrate old usage report to App Router path route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30694](BerriAI/litellm#30694) - fix(proxy): enforce budgets against authoritative DB spend when the cross-pod counter is stale by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30684](BerriAI/litellm#30684) - chore(ci): remove Agent Shin pull\_request\_target workflows by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30784](BerriAI/litellm#30784) - chore: litellm oss staging by [@​Sameerlite](https://github.com/Sameerlite) in [#​30745](BerriAI/litellm#30745) - ci(zizmor): also run on litellm\_internal\_staging by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30789](BerriAI/litellm#30789) - fix(test): drop references to removed Agent Shin workflows by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30791](https://github.com/BerriAI/litellm/pull/30791) - chore: remove in-product survey and Claude Code feedback nudges by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30773](https://github.com/BerriAI/litellm/pull/30773) - feat(ui): migrate api-keys landing to App Router path route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30699](https://github.com/BerriAI/litellm/pull/30699) - feat(proxy): configurable response headers and login-page hint by [@​yucheng-berri](https://github.com/yucheng-berri) in [#​30792](https://github.com/BerriAI/litellm/pull/30792) - ci(zizmor): gate PRs on medium+ findings and clear existing ones by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30797](https://github.com/BerriAI/litellm/pull/30797) - fix(proxy): use e.request\_data for logging\_obj in ModifyResponseException streaming passthrough by [@​mateo-berri](https://github.com/mateo-berri) in [#​30800](https://github.com/BerriAI/litellm/pull/30800) - chore: make pr template linear portion clearer by [@​mateo-berri](https://github.com/mateo-berri) in [#​30766](https://github.com/BerriAI/litellm/pull/30766) - chore(typing): add boto3/botocore stubs so basedpyright resolves the AWS SDK by [@​mateo-berri](https://github.com/mateo-berri) in [#​30815](https://github.com/BerriAI/litellm/pull/30815) - fix(otel): one v2 logger owns the global provider; scope tenant OTLP creds per exporter by [@​yucheng-berri](https://github.com/yucheng-berri) in [#​30590](https://github.com/BerriAI/litellm/pull/30590) - fix(passthrough): recover output tokens for interrupted anthropic streams by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30787](https://github.com/BerriAI/litellm/pull/30787) - fix(proxy): record partial spend on the failure row for interrupted streams by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30788](https://github.com/BerriAI/litellm/pull/30788) - fix(ui): repoint dead usage guide link to cost tracking docs by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30859](https://github.com/BerriAI/litellm/pull/30859) - fix(ui): warn that team models are deleted in the delete-team modal by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29990](https://github.com/BerriAI/litellm/pull/29990) - feat(caching): add valkey-semantic cache backend and fix semantic cache scope keys by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30675](https://github.com/BerriAI/litellm/pull/30675) - test(ui): isolate OldTeams delete-warning tests from leaked mock by [@​mateo-berri](https://github.com/mateo-berri) in [#​30871](https://github.com/BerriAI/litellm/pull/30871) - feat: add lint-gate target and truncation-proof summary to the strict ruff gate by [@​mateo-berri](https://github.com/mateo-berri) in [#​30877](https://github.com/BerriAI/litellm/pull/30877) - chore(ui): rebuild ui for release by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30894](https://github.com/BerriAI/litellm/pull/30894) - chore(ci): bump deps by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30899](https://github.com/BerriAI/litellm/pull/30899) - fix(watsonx): wrap string embedding input in array for WatsonX API by [@​shivamrawat1](https://github.com/shivamrawat1) in [#​30897](https://github.com/BerriAI/litellm/pull/30897) - test: point router/completion/triton tests at the local fake OpenAI endpoint by [@​mateo-berri](https://github.com/mateo-berri) in [#​30900](https://github.com/BerriAI/litellm/pull/30900) - feat(sandbox): e2b code execution primitive by [@​krrish-berri-2](https://github.com/krrish-berri-2) in [#​30898](https://github.com/BerriAI/litellm/pull/30898) - fix(ui): source api-keys identity from useAuthorized to stop "User ID is not set" by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30903](https://github.com/BerriAI/litellm/pull/30903) - chore(ui): rebuild ui by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30906](https://github.com/BerriAI/litellm/pull/30906) - chore(ci): promote internal staging to main by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30907](https://github.com/BerriAI/litellm/pull/30907) - fix(redis): prevent forcing SSLConnection when ssl=False in connection pool by [@​Jacopos311](https://github.com/Jacopos311) in [#​30770](https://github.com/BerriAI/litellm/pull/30770) - fix(proxy): log UI setup failures instead of silently swallowing by [@​sarvesh1327](https://github.com/sarvesh1327) in [#​30819](https://github.com/BerriAI/litellm/pull/30819) #### New Contributors - [@​simantak-dabhade](https://github.com/simantak-dabhade) made their first contribution in [#​30634](BerriAI/litellm#30634) - [@​Jacopos311](https://github.com/Jacopos311) made their first contribution in [#​30770](https://github.com/BerriAI/litellm/pull/30770) - [@​sarvesh1327](https://github.com/sarvesh1327) made their first contribution in [#​30819](https://github.com/BerriAI/litellm/pull/30819) **Full Changelog**: <BerriAI/litellm@v1.89.0...v1.90.0> </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yMjAuMCIsInVwZGF0ZWRJblZlciI6IjQzLjIyMC4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJkZXBlbmRlbmNpZXMiXX0=--> Co-authored-by: Renovate Bot <renovate@bhamm-lab.com> Reviewed-on: https://codeberg.org/blake-hamm/bhamm-lab/pulls/232
blake-hamm
added a commit
to blake-hamm/bhamm-lab
that referenced
this pull request
Jun 28, 2026
This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [ghcr.io/berriai/litellm](https://images.chainguard.dev/directory/image/wolfi-base/overview) ([source](https://github.com/BerriAI/litellm)) | final | minor | `v1.85.1` → `v1.90.0` | --- ### Release Notes <details> <summary>BerriAI/litellm (ghcr.io/berriai/litellm)</summary> ### [`v1.90.0`](https://github.com/BerriAI/litellm/releases/tag/v1.90.0) [Compare Source](https://github.com/BerriAI/litellm/compare/v1.90.0...v1.90.0) ##### Verify Docker Image Signature All LiteLLM Docker images are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](https://github.com/BerriAI/litellm/commit/0112e53046018d726492c814b3644b7d376029d0). **Verify using the pinned commit hash (recommended):** A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \ ghcr.io/berriai/litellm:v1.90.0 ``` **Verify using the release tag (convenience):** Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/v1.90.0/cosign.pub \ ghcr.io/berriai/litellm:v1.90.0 ``` Expected output: ``` The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key ``` *** ##### What's Changed - fix(responses-bridge): map system-only chat request to system input item by [@​milan-berri](https://github.com/milan-berri) in [#​29817](https://github.com/BerriAI/litellm/pull/29817) - feat(bedrock): forward strict and additionalProperties to Converse toolSpec by [@​mateo-berri](https://github.com/mateo-berri) in [#​29814](https://github.com/BerriAI/litellm/pull/29814) - fix(mcp): highlight MCP cards red when the logged-in user is missing per-user env vars by [@​mateo-berri](https://github.com/mateo-berri) in [#​29856](https://github.com/BerriAI/litellm/pull/29856) - feat(ui): add budget duration to edit team member form by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29717](https://github.com/BerriAI/litellm/pull/29717) - fix(ui): make workflow runs page fill full width by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29868](https://github.com/BerriAI/litellm/pull/29868) - feat: standardize rate limit errors with category, rate\_limit\_type, model, and llm\_provider fields by [@​mateo-berri](https://github.com/mateo-berri) in [#​27687](https://github.com/BerriAI/litellm/pull/27687) - fix(ui): default guardrails page to the Guardrails tab by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29872](https://github.com/BerriAI/litellm/pull/29872) - docs(readme): add Deploy on AWS/GCP Terraform section and fix deploy button rendering by [@​mateo-berri](https://github.com/mateo-berri) in [#​29879](https://github.com/BerriAI/litellm/pull/29879) - refactor(bedrock): build Converse toolSpec via a BedrockToolSpec dict subclass by [@​mateo-berri](https://github.com/mateo-berri) in [#​29869](https://github.com/BerriAI/litellm/pull/29869) - feat(litellm): add models and repository layers by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​29686](https://github.com/BerriAI/litellm/pull/29686) - feat(ui): include internal routes in the dashboard's generated OpenAPI types by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29885](https://github.com/BerriAI/litellm/pull/29885) - feat(proxy): publish /v2/model/info in Swagger OpenAPI spec by [@​Sameerlite](https://github.com/Sameerlite) in [#​29900](https://github.com/BerriAI/litellm/pull/29900) - refactor(ui): single source of truth for migrated-page routing by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29949](https://github.com/BerriAI/litellm/pull/29949) - fix(ui/model-hub): render provider icons on the public model hub by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29958](https://github.com/BerriAI/litellm/pull/29958) - fix(ui): keep create guardrail modal open on outside click by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29871](https://github.com/BerriAI/litellm/pull/29871) - fix(ui): label default key type as "Full Access" on key edit page by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29870](https://github.com/BerriAI/litellm/pull/29870) - fix(ui): unify migrated-route URLs and migrate the API Reference page by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29953](https://github.com/BerriAI/litellm/pull/29953) - fix(mcp): let non-creator users OAuth into OBO-mode MCP servers from the Tools page by [@​tin-berri](https://github.com/tin-berri) in [#​29867](https://github.com/BerriAI/litellm/pull/29867) - Litellm oss staging 080626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​29932](https://github.com/BerriAI/litellm/pull/29932) - feat(galileo): add health check support for UI callback test by [@​Sameerlite](https://github.com/Sameerlite) in [#​29908](https://github.com/BerriAI/litellm/pull/29908) - fix(model-management): allow deleting a BYOK model after its team is deleted by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29875](https://github.com/BerriAI/litellm/pull/29875) - feat(jwt-auth): opt-in fallback to DB team on unresolved JWT claim by [@​milan-berri](https://github.com/milan-berri) in [#​28913](https://github.com/BerriAI/litellm/pull/28913) - fix(team\_endpoints): don't block /team/update on unchanged team budget by [@​milan-berri](https://github.com/milan-berri) in [#​29525](https://github.com/BerriAI/litellm/pull/29525) - fix(fireworks): enable tool calling for glm-5p1 in model cost map by [@​milan-berri](https://github.com/milan-berri) in [#​29697](https://github.com/BerriAI/litellm/pull/29697) - fix(vertex): propagate Vertex AI metadata in streaming success callbacks by [@​Sameerlite](https://github.com/Sameerlite) in [#​29899](https://github.com/BerriAI/litellm/pull/29899) - fix(ui): show team projects to internal users on key creation by [@​milan-berri](https://github.com/milan-berri) in [#​28855](https://github.com/BerriAI/litellm/pull/28855) - build(deps): bump pyjwt to 2.13.0 and ws override to 8.20.1 by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29982](https://github.com/BerriAI/litellm/pull/29982) - fix(team-management): delete a team's BYOK models when the team is deleted by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29977](https://github.com/BerriAI/litellm/pull/29977) - feat(vantage): include organization metadata in FOCUS Tags export by [@​milan-berri](https://github.com/milan-berri) in [#​28184](https://github.com/BerriAI/litellm/pull/28184) - fix(guardrails): read CrowdStrike AIDR identity from both metadata bags by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29991](https://github.com/BerriAI/litellm/pull/29991) - fix(mcp): mirror upstream token lifetime instead of forcing a 1h OBO expiry by [@​tin-berri](https://github.com/tin-berri) in [#​29951](https://github.com/BerriAI/litellm/pull/29951) - feat(azure\_ai): add MAI-Image-2.5 image generation support by [@​Sameerlite](https://github.com/Sameerlite) in [#​29688](https://github.com/BerriAI/litellm/pull/29688) - fix(mcp): load MCP tool configuration tools via the OBO/passthrough-aware GET path by [@​tin-berri](https://github.com/tin-berri) in [#​29960](https://github.com/BerriAI/litellm/pull/29960) - fix(team): reserve team budget raises for proxy admins on /team/update by [@​milan-berri](https://github.com/milan-berri) in [#​30030](https://github.com/BerriAI/litellm/pull/30030) - test(ui): data-driven App Router migration E2E smoke (default + server-root-path) by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29974](https://github.com/BerriAI/litellm/pull/29974) - fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through by [@​michelligabriele](https://github.com/michelligabriele) in [#​24232](https://github.com/BerriAI/litellm/pull/24232) - chore(ui): remove dead App Router route stubs under (dashboard) by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30045](https://github.com/BerriAI/litellm/pull/30045) - fix(ui/mcp): reset OAuth state on create-server modal close so a prior server's token no longer leaks into the next add-server session by [@​tin-berri](https://github.com/tin-berri) in [#​30000](https://github.com/BerriAI/litellm/pull/30000) - fix(mcp): allow team access-group grants in OAuth authorize/token access check by [@​tin-berri](https://github.com/tin-berri) in [#​30041](https://github.com/BerriAI/litellm/pull/30041) - docs(security): require a reproduction video for vulnerability reports by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30063](https://github.com/BerriAI/litellm/pull/30063) - feat(ui): add admin flag to disable in-product UI nudges for everyone by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29796](https://github.com/BerriAI/litellm/pull/29796) - chore(ui): remove dead dashboard files and unused dependencies by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30047](https://github.com/BerriAI/litellm/pull/30047) - fix(proxy): authorize batch files using upload target\_model\_names (LIT-3593) by [@​Sameerlite](https://github.com/Sameerlite) in [#​30009](https://github.com/BerriAI/litellm/pull/30009) - Add Claude Fable 5 across Anthropic, Bedrock, Vertex AI, and Azure AI by [@​mateo-berri](https://github.com/mateo-berri) in [#​30064](https://github.com/BerriAI/litellm/pull/30064) - Add Claude Fable 5 cost map entries (data-only hotfix for the hosted map) by [@​mateo-berri](https://github.com/mateo-berri) in [#​30076](https://github.com/BerriAI/litellm/pull/30076) - fix(caching): restore stored prompt\_tokens on embedding cache hits instead of recomputing by [@​michelligabriele](https://github.com/michelligabriele) in [#​30046](https://github.com/BerriAI/litellm/pull/30046) - Litellm oss 090626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​30021](https://github.com/BerriAI/litellm/pull/30021) - fix(proxy): self-heal startup/reload prisma reads on engine disconnect by [@​michelligabriele](https://github.com/michelligabriele) in [#​28803](https://github.com/BerriAI/litellm/pull/28803) - chore(ui): make knip recognize .mjs scripts and openapi-typescript by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30052](https://github.com/BerriAI/litellm/pull/30052) - fix(register\_model): preserve built-in cache pricing when registering custom overrides under unmapped keys by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30044](https://github.com/BerriAI/litellm/pull/30044) - \[internal copy of [#​28007](https://github.com/BerriAI/litellm/issues/28007)] Fix/gcp model garden streaming by [@​mateo-berri](https://github.com/mateo-berri) in [#​28363](https://github.com/BerriAI/litellm/pull/28363) - feat(cli): per-agent `lite claude` / `codex` / `opencode` commands that wrap coding agents through the proxy by [@​mateo-berri](https://github.com/mateo-berri) in [#​29850](https://github.com/BerriAI/litellm/pull/29850) - fix(callbacks): forward callback\_settings to callback initializers and guard consumers against non-dict values by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30161](https://github.com/BerriAI/litellm/pull/30161) - fix(mcp): drop orphaned per-user credential rows when an MCP server is deleted by [@​tin-berri](https://github.com/tin-berri) in [#​30141](https://github.com/BerriAI/litellm/pull/30141) - fix(proxy): recover from cached-plan errors by reconnecting the Prisma client by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​29983](https://github.com/BerriAI/litellm/pull/29983) - feat(proxy): add option to disable server-side prepared statements for DB lookups by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​29984](https://github.com/BerriAI/litellm/pull/29984) - fix(release): stop backport releases from overwriting the latest badge by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30005](https://github.com/BerriAI/litellm/pull/30005) - feat: add conventional commits and coding guidelines by [@​mateo-berri](https://github.com/mateo-berri) in [#​30159](https://github.com/BerriAI/litellm/pull/30159) - fix(proxy): return 5xx on DB infra errors during auth; reserve 401 for genuine auth failures by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​29986](https://github.com/BerriAI/litellm/pull/29986) - fix(ui): dev server 404s on migrated-page links because uiBase hardcodes /ui by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30169](https://github.com/BerriAI/litellm/pull/30169) - refactor(ui): consolidate dashboard to one shell in the (dashboard) layout by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30166](https://github.com/BerriAI/litellm/pull/30166) - fix(proxy): align /v1/model/info with router deployments by [@​Sameerlite](https://github.com/Sameerlite) in [#​30025](https://github.com/BerriAI/litellm/pull/30025) - fix: completion\_cost AttributeError on streaming Anthropic web\_search responses ([#​26153](https://github.com/BerriAI/litellm/issues/26153)) by [@​ishaan-berri](https://github.com/ishaan-berri) in [#​27346](https://github.com/BerriAI/litellm/pull/27346) - \[internal copy of [#​30137](https://github.com/BerriAI/litellm/issues/30137)] perf(realtime): eliminate redundant per-frame JSON work on OpenAI realtime relay by [@​mateo-berri](https://github.com/mateo-berri) in [#​30142](https://github.com/BerriAI/litellm/pull/30142) - feat(bedrock): aws\_bedrock\_project\_id for bedrock-mantle project / workspace association by [@​mateo-berri](https://github.com/mateo-berri) in [#​30163](https://github.com/BerriAI/litellm/pull/30163) - chore(hooks): enforce Conventional Commits and Conventional Branches by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30174](https://github.com/BerriAI/litellm/pull/30174) - feat(rate-limiter): allow opting out of v3 TPM reservation and Redis circuit breaker by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30211](https://github.com/BerriAI/litellm/pull/30211) - feat(spend\_logs): opt-in native Postgres partitioning for SpendLogs retention by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​29466](https://github.com/BerriAI/litellm/pull/29466) - feat(ui): migrate playground to path routing and colocate its files by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30185](https://github.com/BerriAI/litellm/pull/30185) - feat(ui): migrate projects and access-groups to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30226](https://github.com/BerriAI/litellm/pull/30226) - fix(proxy): coalesce NULL rollup metrics in aggregated daily-activity by [@​michelligabriele](https://github.com/michelligabriele) in [#​30151](https://github.com/BerriAI/litellm/pull/30151) - fix(anthropic\_passthrough): resolve costing model from message\_start chunk, litellm\_params and model\_group instead of 'unknown' by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30160](https://github.com/BerriAI/litellm/pull/30160) - feat(ui): migrate budgets, workflows, and guardrails-monitor to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30236](https://github.com/BerriAI/litellm/pull/30236) - feat(ui): migrate mcp-servers, search-tools, tag-management, vector-stores, and memory to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30261](https://github.com/BerriAI/litellm/pull/30261) - fix(a2a): forward agent\_extra\_headers through completion bridge by [@​mateo-berri](https://github.com/mateo-berri) in [#​28277](https://github.com/BerriAI/litellm/pull/28277) - fix(gemini-live): forward audio buffer commit and correct Vertex PCM rate by [@​Sameerlite](https://github.com/Sameerlite) in [#​29946](https://github.com/BerriAI/litellm/pull/29946) - fix(proxy): skip double-wrapping unified batch output file ids on retrieve by [@​Sameerlite](https://github.com/Sameerlite) in [#​30011](https://github.com/BerriAI/litellm/pull/30011) - feat: litellm oss 110626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​30202](https://github.com/BerriAI/litellm/pull/30202) - fix(docker): copy only runtime artifacts into the final image by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30243](https://github.com/BerriAI/litellm/pull/30243) - feat(proxy): enforce key/team guardrails on bedrock passthrough routes by [@​Sameerlite](https://github.com/Sameerlite) in [#​30194](https://github.com/BerriAI/litellm/pull/30194) - feat(gemini): forward web search tools in image generation by [@​Sameerlite](https://github.com/Sameerlite) in [#​30119](https://github.com/BerriAI/litellm/pull/30119) - fix: bedrock mantle fixes by [@​Sameerlite](https://github.com/Sameerlite) in [#​30083](https://github.com/BerriAI/litellm/pull/30083) - feat(proxy): add require\_managed\_files setting for file uploads by [@​Sameerlite](https://github.com/Sameerlite) in [#​30186](https://github.com/BerriAI/litellm/pull/30186) - fix(mcp): honor server\_id for REST tool calls with shared upstream URLs by [@​Sameerlite](https://github.com/Sameerlite) in [#​30184](https://github.com/BerriAI/litellm/pull/30184) - fix(responses): presidio PII masking for Azure WebSocket and streaming by [@​Sameerlite](https://github.com/Sameerlite) in [#​30003](https://github.com/BerriAI/litellm/pull/30003) - feat(passthrough): add configurable pass-through request timeouts by [@​Sameerlite](https://github.com/Sameerlite) in [#​30266](https://github.com/BerriAI/litellm/pull/30266) - fix(google\_genai): preserve complete SSE events in Vertex/Gemini image streaming by [@​Sameerlite](https://github.com/Sameerlite) in [#​30270](https://github.com/BerriAI/litellm/pull/30270) - fix(proxy): populate access\_via\_team\_ids on /v1/model/info by [@​Sameerlite](https://github.com/Sameerlite) in [#​30274](https://github.com/BerriAI/litellm/pull/30274) - chore(oss): litellm oss staging 120626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​30292](https://github.com/BerriAI/litellm/pull/30292) - feat(ui): migrate policies, guardrails, prompts, tool-policies, and skills to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30263](https://github.com/BerriAI/litellm/pull/30263) - feat(ui): migrate caching, cost-tracking, transform-request, ui-theme, and logs to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30267](https://github.com/BerriAI/litellm/pull/30267) - fix(ui): gate dashboard layout on ui config load so deep links work under SERVER\_ROOT\_PATH by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30312](https://github.com/BerriAI/litellm/pull/30312) - feat(ui): migrate admin-panel, logging-and-alerts, model-hub-table, and usage to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30268](https://github.com/BerriAI/litellm/pull/30268) - fix(otel): cap metric attribute cardinality with include/exclude lists by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30257](https://github.com/BerriAI/litellm/pull/30257) - fix(proxy): grace-period key rotation 401s; return deprecated-key lookup result directly by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30327](https://github.com/BerriAI/litellm/pull/30327) - chore(deps): bump vitest, brace-expansion, pypdf and tornado by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30220](https://github.com/BerriAI/litellm/pull/30220) - refactor(ui): remove unreachable /chat page by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30178](https://github.com/BerriAI/litellm/pull/30178) - feat(ui): migrate agents and router-settings to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30323](https://github.com/BerriAI/litellm/pull/30323) - feat: strengthen coding conventions in CLAUDE.md by [@​mateo-berri](https://github.com/mateo-berri) in [#​30333](https://github.com/BerriAI/litellm/pull/30333) - feat(ui): cut the users page over to the /ui/users path route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30334](https://github.com/BerriAI/litellm/pull/30334) - feat: ruff strict-rule suppressions baseline gate by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30303](https://github.com/BerriAI/litellm/pull/30303) - feat(guardrails): add Cisco AI Defense integration ([#​28249](https://github.com/BerriAI/litellm/issues/28249)) by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30338](https://github.com/BerriAI/litellm/pull/30338) - chore(ui): remove dead UI components unreferenced by any page by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30340](https://github.com/BerriAI/litellm/pull/30340) - ci: add osv-scanner lockfile scan workflow by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30222](https://github.com/BerriAI/litellm/pull/30222) - fix(otel): record full error message on standard exception event in otel v2 by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30380](https://github.com/BerriAI/litellm/pull/30380) - test(fireworks): mock whisper transcription tests instead of live calls by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30391](https://github.com/BerriAI/litellm/pull/30391) - build(ui): pin esbuild to 0.28.1 via overrides by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30390](https://github.com/BerriAI/litellm/pull/30390) - feat(ui): cut the organizations page over to the /ui/organizations path route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30336](https://github.com/BerriAI/litellm/pull/30336) - fix(proxy): support SMTP implicit SSL (port 465) by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30395](https://github.com/BerriAI/litellm/pull/30395) - fix(mcp): default Linear MCP registry entry to streamable HTTP by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30396](https://github.com/BerriAI/litellm/pull/30396) - fix(ui): stop Virtual Keys page from infinite render loop by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30397](https://github.com/BerriAI/litellm/pull/30397) - fix(streaming): guard raise\_on\_model\_repetition against empty choices by [@​shivamrawat1](https://github.com/shivamrawat1) in [#​30485](https://github.com/BerriAI/litellm/pull/30485) - feat(otel-v2): emit the 6 gen\_ai.client.\* metrics at parity with v1 by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30326](https://github.com/BerriAI/litellm/pull/30326) - fix(mcp): drop phantom 401 span on delegated OAuth2 tool calls by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30494](https://github.com/BerriAI/litellm/pull/30494) - feat(ui): cut the teams page over to the /ui/teams path route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30343](https://github.com/BerriAI/litellm/pull/30343) - fix(integrations): cap Anthropic cache\_control injection at 4 blocks by [@​shivamrawat1](https://github.com/shivamrawat1) in [#​30480](https://github.com/BerriAI/litellm/pull/30480) - chore(codecov): add Batches, Videos, and Realtime components by [@​Sameerlite](https://github.com/Sameerlite) in [#​30517](https://github.com/BerriAI/litellm/pull/30517) - test(batches): move orphan tests into tests/test\_litellm for CI coverage by [@​Sameerlite](https://github.com/Sameerlite) in [#​30510](https://github.com/BerriAI/litellm/pull/30510) - fix(guardrails): run pre\_call hook once for model-level guardrails by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30543](https://github.com/BerriAI/litellm/pull/30543) - fix(guardrails): stop re-initializing DB guardrails on every poll by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30542](https://github.com/BerriAI/litellm/pull/30542) - chore(oss): litellm oss staging 150626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​30463](https://github.com/BerriAI/litellm/pull/30463) - ci(lint): add blanket-noqa, dataclass-default, and unused-noqa Ruff rules by [@​mateo-berri](https://github.com/mateo-berri) in [#​30516](https://github.com/BerriAI/litellm/pull/30516) - ci: ratchet lint and type-check gates (ruff preview, ANN, mypy, basedpyright) by [@​mateo-berri](https://github.com/mateo-berri) in [#​30379](https://github.com/BerriAI/litellm/pull/30379) - fix(proxy): allow internal roles to access vector store CRUD routes by [@​shivamrawat1](https://github.com/shivamrawat1) in [#​30503](https://github.com/BerriAI/litellm/pull/30503) - fix(otel): stamp gen\_ai.input/output.messages on v2 spans by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30548](https://github.com/BerriAI/litellm/pull/30548) - fix(otel): export v2 gen\_ai client metrics to the configured meter provider by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30549](https://github.com/BerriAI/litellm/pull/30549) - fix(bedrock): preserve cache\_control for ARN models in /v1/messages adapter by [@​mateo-berri](https://github.com/mateo-berri) in [#​29823](https://github.com/BerriAI/litellm/pull/29823) - fix: greatly increase basedpyright slack by [@​mateo-berri](https://github.com/mateo-berri) in [#​30563](https://github.com/BerriAI/litellm/pull/30563) - fix(budget): recompute budget\_reset\_at when budget\_duration changes on /budget/update by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30555](https://github.com/BerriAI/litellm/pull/30555) - fix(otel): accept UPPER\_SNAKE\_CASE OTEL\_INSTRUMENTATION\_GENAI\_CAPTURE\_MESSAGE\_CONTENT in v2 by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30562](https://github.com/BerriAI/litellm/pull/30562) - chore(lint): remove PLR0915 too-many-statements ruff rule by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30574](https://github.com/BerriAI/litellm/pull/30574) - ci(lint): ratcheted type-discipline gate (mutable collections, casts, guards, kwargs, suppressions) by [@​mateo-berri](https://github.com/mateo-berri) in [#​30500](https://github.com/BerriAI/litellm/pull/30500) - feat(proxy): add verification\_uri\_complete to CLI SSO device flow by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30571](https://github.com/BerriAI/litellm/pull/30571) - chore: litellm oss staging160626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​30527](https://github.com/BerriAI/litellm/pull/30527) - fix(guardrails): return 400 not 500 when AIM blocks a request by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30573](https://github.com/BerriAI/litellm/pull/30573) - ci(lint): grandfather any-discipline with a per-file ratchet budget (50% headroom) by [@​mateo-berri](https://github.com/mateo-berri) in [#​30582](https://github.com/BerriAI/litellm/pull/30582) - fix(audio): don't override explicit response\_format with verbose\_json by [@​mateo-berri](https://github.com/mateo-berri) in [#​30599](https://github.com/BerriAI/litellm/pull/30599) - fix(anthropic): price and surface response service\_tier in cost tracking by [@​mateo-berri](https://github.com/mateo-berri) in [#​30558](https://github.com/BerriAI/litellm/pull/30558) - feat: add dev and wildcard proxy configs for local testing by [@​mateo-berri](https://github.com/mateo-berri) in [#​30556](https://github.com/BerriAI/litellm/pull/30556) - fix(proxy): list public team model name in /v1/models by [@​ishaan-berri](https://github.com/ishaan-berri) in [#​30588](https://github.com/BerriAI/litellm/pull/30588) - ci: drop mypy entirely, standardize type checking on basedpyright by [@​mateo-berri](https://github.com/mateo-berri) in [#​30648](https://github.com/BerriAI/litellm/pull/30648) - feat(guardrails): surface OpenAI moderation violation\_categories on guardrail traces by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30659](https://github.com/BerriAI/litellm/pull/30659) - fix(proxy): resolve list files credentials from team BYOK deployments by [@​shivamrawat1](https://github.com/shivamrawat1) in [#​30495](https://github.com/BerriAI/litellm/pull/30495) - feat(proxy): add --max\_requests\_before\_restart\_jitter to stagger worker restarts by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30601](https://github.com/BerriAI/litellm/pull/30601) - fix(health): correct bedrock embedding health checks by [@​mateo-berri](https://github.com/mateo-berri) in [#​30583](https://github.com/BerriAI/litellm/pull/30583) - test: harden remaining pass-through CI flakes (image-gen spend poll, ruby assistants timeout) by [@​mateo-berri](https://github.com/mateo-berri) in [#​30685](https://github.com/BerriAI/litellm/pull/30685) - test(pass\_through): harden vertex spendlog poll against transient empty reads by [@​mateo-berri](https://github.com/mateo-berri) in [#​30683](https://github.com/BerriAI/litellm/pull/30683) - fix(cost): stop non-string service\_tier from silently dropping cost tracking by [@​mateo-berri](https://github.com/mateo-berri) in [#​30690](https://github.com/BerriAI/litellm/pull/30690) - feat(proxy): warn at startup when custom\_auth skips common\_checks enforcement by [@​tin-berri](https://github.com/tin-berri) in [#​30665](https://github.com/BerriAI/litellm/pull/30665) - fix(pod\_lock): release cron lock by matching async\_set\_cache JSON encoding by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30600](https://github.com/BerriAI/litellm/pull/30600) - ci: run a local fake OpenAI endpoint instead of the shared Railway mock by [@​mateo-berri](https://github.com/mateo-berri) in [#​30695](https://github.com/BerriAI/litellm/pull/30695) - ci(windows): pin uv to Python 3.11 so it ignores the preinstalled 3.14 by [@​mateo-berri](https://github.com/mateo-berri) in [#​30704](https://github.com/BerriAI/litellm/pull/30704) - feat(ui): migrate models page to App Router path route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30677](https://github.com/BerriAI/litellm/pull/30677) - refactor(ui): remove orphaned pass-through-settings route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30692](https://github.com/BerriAI/litellm/pull/30692) - fix(cost): stop non-string response service\_tier from dropping cost tracking by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30706](https://github.com/BerriAI/litellm/pull/30706) - feat(agent-shin): automated PR/issue triage, low-quality auto-close, and review-gate label lifecycle by [@​mateo-berri](https://github.com/mateo-berri) in [#​30433](https://github.com/BerriAI/litellm/pull/30433) - chore: litellm oss 170626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​30637](https://github.com/BerriAI/litellm/pull/30637) - fix(bedrock\_mantle): add SigV4 fallback to chat completions auth by [@​mateo-berri](https://github.com/mateo-berri) in [#​30714](https://github.com/BerriAI/litellm/pull/30714) - feat(search): add TinyFish as search provider by [@​simantak-dabhade](https://github.com/simantak-dabhade) in [#​30634](https://github.com/BerriAI/litellm/pull/30634) - feat(ui): migrate old usage report to App Router path route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30694](https://github.com/BerriAI/litellm/pull/30694) - fix(proxy): enforce budgets against authoritative DB spend when the cross-pod counter is stale by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30684](https://github.com/BerriAI/litellm/pull/30684) - chore(ci): remove Agent Shin pull\_request\_target workflows by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30784](https://github.com/BerriAI/litellm/pull/30784) - chore: litellm oss staging by [@​Sameerlite](https://github.com/Sameerlite) in [#​30745](https://github.com/BerriAI/litellm/pull/30745) - ci(zizmor): also run on litellm\_internal\_staging by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30789](https://github.com/BerriAI/litellm/pull/30789) - fix(test): drop references to removed Agent Shin workflows by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30791](https://github.com/BerriAI/litellm/pull/30791) - chore: remove in-product survey and Claude Code feedback nudges by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30773](https://github.com/BerriAI/litellm/pull/30773) - feat(ui): migrate api-keys landing to App Router path route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30699](https://github.com/BerriAI/litellm/pull/30699) - feat(proxy): configurable response headers and login-page hint by [@​yucheng-berri](https://github.com/yucheng-berri) in [#​30792](https://github.com/BerriAI/litellm/pull/30792) - ci(zizmor): gate PRs on medium+ findings and clear existing ones by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30797](https://github.com/BerriAI/litellm/pull/30797) - fix(proxy): use e.request\_data for logging\_obj in ModifyResponseException streaming passthrough by [@​mateo-berri](https://github.com/mateo-berri) in [#​30800](https://github.com/BerriAI/litellm/pull/30800) - chore: make pr template linear portion clearer by [@​mateo-berri](https://github.com/mateo-berri) in [#​30766](https://github.com/BerriAI/litellm/pull/30766) - chore(typing): add boto3/botocore stubs so basedpyright resolves the AWS SDK by [@​mateo-berri](https://github.com/mateo-berri) in [#​30815](https://github.com/BerriAI/litellm/pull/30815) - fix(otel): one v2 logger owns the global provider; scope tenant OTLP creds per exporter by [@​yucheng-berri](https://github.com/yucheng-berri) in [#​30590](https://github.com/BerriAI/litellm/pull/30590) - fix(passthrough): recover output tokens for interrupted anthropic streams by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30787](https://github.com/BerriAI/litellm/pull/30787) - fix(proxy): record partial spend on the failure row for interrupted streams by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30788](https://github.com/BerriAI/litellm/pull/30788) - fix(ui): repoint dead usage guide link to cost tracking docs by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30859](https://github.com/BerriAI/litellm/pull/30859) - fix(ui): warn that team models are deleted in the delete-team modal by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29990](https://github.com/BerriAI/litellm/pull/29990) - feat(caching): add valkey-semantic cache backend and fix semantic cache scope keys by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30675](https://github.com/BerriAI/litellm/pull/30675) - test(ui): isolate OldTeams delete-warning tests from leaked mock by [@​mateo-berri](https://github.com/mateo-berri) in [#​30871](https://github.com/BerriAI/litellm/pull/30871) - feat: add lint-gate target and truncation-proof summary to the strict ruff gate by [@​mateo-berri](https://github.com/mateo-berri) in [#​30877](https://github.com/BerriAI/litellm/pull/30877) - chore(ui): rebuild ui for release by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30894](https://github.com/BerriAI/litellm/pull/30894) - chore(ci): bump deps by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30899](https://github.com/BerriAI/litellm/pull/30899) - fix(watsonx): wrap string embedding input in array for WatsonX API by [@​shivamrawat1](https://github.com/shivamrawat1) in [#​30897](https://github.com/BerriAI/litellm/pull/30897) - test: point router/completion/triton tests at the local fake OpenAI endpoint by [@​mateo-berri](https://github.com/mateo-berri) in [#​30900](https://github.com/BerriAI/litellm/pull/30900) - feat(sandbox): e2b code execution primitive by [@​krrish-berri-2](https://github.com/krrish-berri-2) in [#​30898](https://github.com/BerriAI/litellm/pull/30898) - fix(ui): source api-keys identity from useAuthorized to stop "User ID is not set" by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30903](https://github.com/BerriAI/litellm/pull/30903) - chore(ui): rebuild ui by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30906](https://github.com/BerriAI/litellm/pull/30906) - chore(ci): promote internal staging to main by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30907](https://github.com/BerriAI/litellm/pull/30907) - fix(redis): prevent forcing SSLConnection when ssl=False in connection pool by [@​Jacopos311](https://github.com/Jacopos311) in [#​30770](https://github.com/BerriAI/litellm/pull/30770) - fix(proxy): log UI setup failures instead of silently swallowing by [@​sarvesh1327](https://github.com/sarvesh1327) in [#​30819](https://github.com/BerriAI/litellm/pull/30819) ##### New Contributors - [@​simantak-dabhade](https://github.com/simantak-dabhade) made their first contribution in [#​30634](https://github.com/BerriAI/litellm/pull/30634) - [@​Jacopos311](https://github.com/Jacopos311) made their first contribution in [#​30770](https://github.com/BerriAI/litellm/pull/30770) - [@​sarvesh1327](https://github.com/sarvesh1327) made their first contribution in [#​30819](https://github.com/BerriAI/litellm/pull/30819) **Full Changelog**: <https://github.com/BerriAI/litellm/compare/v1.89.0...v1.90.0> ### [`v1.90.0`](https://github.com/BerriAI/litellm/releases/tag/v1.90.0) [Compare Source](https://github.com/BerriAI/litellm/compare/v1.89.4...v1.90.0) ##### Verify Docker Image Signature All LiteLLM Docker images are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](https://github.com/BerriAI/litellm/commit/0112e53046018d726492c814b3644b7d376029d0). **Verify using the pinned commit hash (recommended):** A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \ ghcr.io/berriai/litellm:v1.90.0 ``` **Verify using the release tag (convenience):** Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/v1.90.0/cosign.pub \ ghcr.io/berriai/litellm:v1.90.0 ``` Expected output: ``` The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key ``` *** ##### What's Changed - fix(responses-bridge): map system-only chat request to system input item by [@​milan-berri](https://github.com/milan-berri) in [#​29817](https://github.com/BerriAI/litellm/pull/29817) - feat(bedrock): forward strict and additionalProperties to Converse toolSpec by [@​mateo-berri](https://github.com/mateo-berri) in [#​29814](https://github.com/BerriAI/litellm/pull/29814) - fix(mcp): highlight MCP cards red when the logged-in user is missing per-user env vars by [@​mateo-berri](https://github.com/mateo-berri) in [#​29856](https://github.com/BerriAI/litellm/pull/29856) - feat(ui): add budget duration to edit team member form by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29717](https://github.com/BerriAI/litellm/pull/29717) - fix(ui): make workflow runs page fill full width by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29868](https://github.com/BerriAI/litellm/pull/29868) - feat: standardize rate limit errors with category, rate\_limit\_type, model, and llm\_provider fields by [@​mateo-berri](https://github.com/mateo-berri) in [#​27687](https://github.com/BerriAI/litellm/pull/27687) - fix(ui): default guardrails page to the Guardrails tab by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29872](https://github.com/BerriAI/litellm/pull/29872) - docs(readme): add Deploy on AWS/GCP Terraform section and fix deploy button rendering by [@​mateo-berri](https://github.com/mateo-berri) in [#​29879](https://github.com/BerriAI/litellm/pull/29879) - refactor(bedrock): build Converse toolSpec via a BedrockToolSpec dict subclass by [@​mateo-berri](https://github.com/mateo-berri) in [#​29869](https://github.com/BerriAI/litellm/pull/29869) - feat(litellm): add models and repository layers by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​29686](https://github.com/BerriAI/litellm/pull/29686) - feat(ui): include internal routes in the dashboard's generated OpenAPI types by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29885](https://github.com/BerriAI/litellm/pull/29885) - feat(proxy): publish /v2/model/info in Swagger OpenAPI spec by [@​Sameerlite](https://github.com/Sameerlite) in [#​29900](https://github.com/BerriAI/litellm/pull/29900) - refactor(ui): single source of truth for migrated-page routing by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29949](https://github.com/BerriAI/litellm/pull/29949) - fix(ui/model-hub): render provider icons on the public model hub by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29958](https://github.com/BerriAI/litellm/pull/29958) - fix(ui): keep create guardrail modal open on outside click by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29871](https://github.com/BerriAI/litellm/pull/29871) - fix(ui): label default key type as "Full Access" on key edit page by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29870](https://github.com/BerriAI/litellm/pull/29870) - fix(ui): unify migrated-route URLs and migrate the API Reference page by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29953](https://github.com/BerriAI/litellm/pull/29953) - fix(mcp): let non-creator users OAuth into OBO-mode MCP servers from the Tools page by [@​tin-berri](https://github.com/tin-berri) in [#​29867](https://github.com/BerriAI/litellm/pull/29867) - Litellm oss staging 080626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​29932](https://github.com/BerriAI/litellm/pull/29932) - feat(galileo): add health check support for UI callback test by [@​Sameerlite](https://github.com/Sameerlite) in [#​29908](https://github.com/BerriAI/litellm/pull/29908) - fix(model-management): allow deleting a BYOK model after its team is deleted by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29875](https://github.com/BerriAI/litellm/pull/29875) - feat(jwt-auth): opt-in fallback to DB team on unresolved JWT claim by [@​milan-berri](https://github.com/milan-berri) in [#​28913](https://github.com/BerriAI/litellm/pull/28913) - fix(team\_endpoints): don't block /team/update on unchanged team budget by [@​milan-berri](https://github.com/milan-berri) in [#​29525](https://github.com/BerriAI/litellm/pull/29525) - fix(fireworks): enable tool calling for glm-5p1 in model cost map by [@​milan-berri](https://github.com/milan-berri) in [#​29697](https://github.com/BerriAI/litellm/pull/29697) - fix(vertex): propagate Vertex AI metadata in streaming success callbacks by [@​Sameerlite](https://github.com/Sameerlite) in [#​29899](https://github.com/BerriAI/litellm/pull/29899) - fix(ui): show team projects to internal users on key creation by [@​milan-berri](https://github.com/milan-berri) in [#​28855](https://github.com/BerriAI/litellm/pull/28855) - build(deps): bump pyjwt to 2.13.0 and ws override to 8.20.1 by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29982](https://github.com/BerriAI/litellm/pull/29982) - fix(team-management): delete a team's BYOK models when the team is deleted by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29977](https://github.com/BerriAI/litellm/pull/29977) - feat(vantage): include organization metadata in FOCUS Tags export by [@​milan-berri](https://github.com/milan-berri) in [#​28184](https://github.com/BerriAI/litellm/pull/28184) - fix(guardrails): read CrowdStrike AIDR identity from both metadata bags by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​29991](https://github.com/BerriAI/litellm/pull/29991) - fix(mcp): mirror upstream token lifetime instead of forcing a 1h OBO expiry by [@​tin-berri](https://github.com/tin-berri) in [#​29951](https://github.com/BerriAI/litellm/pull/29951) - feat(azure\_ai): add MAI-Image-2.5 image generation support by [@​Sameerlite](https://github.com/Sameerlite) in [#​29688](https://github.com/BerriAI/litellm/pull/29688) - fix(mcp): load MCP tool configuration tools via the OBO/passthrough-aware GET path by [@​tin-berri](https://github.com/tin-berri) in [#​29960](https://github.com/BerriAI/litellm/pull/29960) - fix(team): reserve team budget raises for proxy admins on /team/update by [@​milan-berri](https://github.com/milan-berri) in [#​30030](https://github.com/BerriAI/litellm/pull/30030) - test(ui): data-driven App Router migration E2E smoke (default + server-root-path) by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29974](https://github.com/BerriAI/litellm/pull/29974) - fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through by [@​michelligabriele](https://github.com/michelligabriele) in [#​24232](https://github.com/BerriAI/litellm/pull/24232) - chore(ui): remove dead App Router route stubs under (dashboard) by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30045](https://github.com/BerriAI/litellm/pull/30045) - fix(ui/mcp): reset OAuth state on create-server modal close so a prior server's token no longer leaks into the next add-server session by [@​tin-berri](https://github.com/tin-berri) in [#​30000](https://github.com/BerriAI/litellm/pull/30000) - fix(mcp): allow team access-group grants in OAuth authorize/token access check by [@​tin-berri](https://github.com/tin-berri) in [#​30041](https://github.com/BerriAI/litellm/pull/30041) - docs(security): require a reproduction video for vulnerability reports by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30063](https://github.com/BerriAI/litellm/pull/30063) - feat(ui): add admin flag to disable in-product UI nudges for everyone by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​29796](https://github.com/BerriAI/litellm/pull/29796) - chore(ui): remove dead dashboard files and unused dependencies by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30047](https://github.com/BerriAI/litellm/pull/30047) - fix(proxy): authorize batch files using upload target\_model\_names (LIT-3593) by [@​Sameerlite](https://github.com/Sameerlite) in [#​30009](https://github.com/BerriAI/litellm/pull/30009) - Add Claude Fable 5 across Anthropic, Bedrock, Vertex AI, and Azure AI by [@​mateo-berri](https://github.com/mateo-berri) in [#​30064](https://github.com/BerriAI/litellm/pull/30064) - Add Claude Fable 5 cost map entries (data-only hotfix for the hosted map) by [@​mateo-berri](https://github.com/mateo-berri) in [#​30076](https://github.com/BerriAI/litellm/pull/30076) - fix(caching): restore stored prompt\_tokens on embedding cache hits instead of recomputing by [@​michelligabriele](https://github.com/michelligabriele) in [#​30046](https://github.com/BerriAI/litellm/pull/30046) - Litellm oss 090626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​30021](https://github.com/BerriAI/litellm/pull/30021) - fix(proxy): self-heal startup/reload prisma reads on engine disconnect by [@​michelligabriele](https://github.com/michelligabriele) in [#​28803](https://github.com/BerriAI/litellm/pull/28803) - chore(ui): make knip recognize .mjs scripts and openapi-typescript by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30052](https://github.com/BerriAI/litellm/pull/30052) - fix(register\_model): preserve built-in cache pricing when registering custom overrides under unmapped keys by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30044](https://github.com/BerriAI/litellm/pull/30044) - \[internal copy of [#​28007](https://github.com/BerriAI/litellm/issues/28007)] Fix/gcp model garden streaming by [@​mateo-berri](https://github.com/mateo-berri) in [#​28363](https://github.com/BerriAI/litellm/pull/28363) - feat(cli): per-agent `lite claude` / `codex` / `opencode` commands that wrap coding agents through the proxy by [@​mateo-berri](https://github.com/mateo-berri) in [#​29850](https://github.com/BerriAI/litellm/pull/29850) - fix(callbacks): forward callback\_settings to callback initializers and guard consumers against non-dict values by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30161](https://github.com/BerriAI/litellm/pull/30161) - fix(mcp): drop orphaned per-user credential rows when an MCP server is deleted by [@​tin-berri](https://github.com/tin-berri) in [#​30141](https://github.com/BerriAI/litellm/pull/30141) - fix(proxy): recover from cached-plan errors by reconnecting the Prisma client by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​29983](https://github.com/BerriAI/litellm/pull/29983) - feat(proxy): add option to disable server-side prepared statements for DB lookups by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​29984](https://github.com/BerriAI/litellm/pull/29984) - fix(release): stop backport releases from overwriting the latest badge by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30005](https://github.com/BerriAI/litellm/pull/30005) - feat: add conventional commits and coding guidelines by [@​mateo-berri](https://github.com/mateo-berri) in [#​30159](https://github.com/BerriAI/litellm/pull/30159) - fix(proxy): return 5xx on DB infra errors during auth; reserve 401 for genuine auth failures by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​29986](https://github.com/BerriAI/litellm/pull/29986) - fix(ui): dev server 404s on migrated-page links because uiBase hardcodes /ui by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30169](https://github.com/BerriAI/litellm/pull/30169) - refactor(ui): consolidate dashboard to one shell in the (dashboard) layout by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30166](https://github.com/BerriAI/litellm/pull/30166) - fix(proxy): align /v1/model/info with router deployments by [@​Sameerlite](https://github.com/Sameerlite) in [#​30025](https://github.com/BerriAI/litellm/pull/30025) - fix: completion\_cost AttributeError on streaming Anthropic web\_search responses ([#​26153](https://github.com/BerriAI/litellm/issues/26153)) by [@​ishaan-berri](https://github.com/ishaan-berri) in [#​27346](https://github.com/BerriAI/litellm/pull/27346) - \[internal copy of [#​30137](https://github.com/BerriAI/litellm/issues/30137)] perf(realtime): eliminate redundant per-frame JSON work on OpenAI realtime relay by [@​mateo-berri](https://github.com/mateo-berri) in [#​30142](https://github.com/BerriAI/litellm/pull/30142) - feat(bedrock): aws\_bedrock\_project\_id for bedrock-mantle project / workspace association by [@​mateo-berri](https://github.com/mateo-berri) in [#​30163](https://github.com/BerriAI/litellm/pull/30163) - chore(hooks): enforce Conventional Commits and Conventional Branches by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30174](https://github.com/BerriAI/litellm/pull/30174) - feat(rate-limiter): allow opting out of v3 TPM reservation and Redis circuit breaker by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30211](https://github.com/BerriAI/litellm/pull/30211) - feat(spend\_logs): opt-in native Postgres partitioning for SpendLogs retention by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​29466](https://github.com/BerriAI/litellm/pull/29466) - feat(ui): migrate playground to path routing and colocate its files by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30185](https://github.com/BerriAI/litellm/pull/30185) - feat(ui): migrate projects and access-groups to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30226](https://github.com/BerriAI/litellm/pull/30226) - fix(proxy): coalesce NULL rollup metrics in aggregated daily-activity by [@​michelligabriele](https://github.com/michelligabriele) in [#​30151](https://github.com/BerriAI/litellm/pull/30151) - fix(anthropic\_passthrough): resolve costing model from message\_start chunk, litellm\_params and model\_group instead of 'unknown' by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30160](https://github.com/BerriAI/litellm/pull/30160) - feat(ui): migrate budgets, workflows, and guardrails-monitor to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30236](https://github.com/BerriAI/litellm/pull/30236) - feat(ui): migrate mcp-servers, search-tools, tag-management, vector-stores, and memory to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30261](https://github.com/BerriAI/litellm/pull/30261) - fix(a2a): forward agent\_extra\_headers through completion bridge by [@​mateo-berri](https://github.com/mateo-berri) in [#​28277](https://github.com/BerriAI/litellm/pull/28277) - fix(gemini-live): forward audio buffer commit and correct Vertex PCM rate by [@​Sameerlite](https://github.com/Sameerlite) in [#​29946](https://github.com/BerriAI/litellm/pull/29946) - fix(proxy): skip double-wrapping unified batch output file ids on retrieve by [@​Sameerlite](https://github.com/Sameerlite) in [#​30011](https://github.com/BerriAI/litellm/pull/30011) - feat: litellm oss 110626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​30202](https://github.com/BerriAI/litellm/pull/30202) - fix(docker): copy only runtime artifacts into the final image by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30243](https://github.com/BerriAI/litellm/pull/30243) - feat(proxy): enforce key/team guardrails on bedrock passthrough routes by [@​Sameerlite](https://github.com/Sameerlite) in [#​30194](https://github.com/BerriAI/litellm/pull/30194) - feat(gemini): forward web search tools in image generation by [@​Sameerlite](https://github.com/Sameerlite) in [#​30119](https://github.com/BerriAI/litellm/pull/30119) - fix: bedrock mantle fixes by [@​Sameerlite](https://github.com/Sameerlite) in [#​30083](https://github.com/BerriAI/litellm/pull/30083) - feat(proxy): add require\_managed\_files setting for file uploads by [@​Sameerlite](https://github.com/Sameerlite) in [#​30186](https://github.com/BerriAI/litellm/pull/30186) - fix(mcp): honor server\_id for REST tool calls with shared upstream URLs by [@​Sameerlite](https://github.com/Sameerlite) in [#​30184](https://github.com/BerriAI/litellm/pull/30184) - fix(responses): presidio PII masking for Azure WebSocket and streaming by [@​Sameerlite](https://github.com/Sameerlite) in [#​30003](https://github.com/BerriAI/litellm/pull/30003) - feat(passthrough): add configurable pass-through request timeouts by [@​Sameerlite](https://github.com/Sameerlite) in [#​30266](https://github.com/BerriAI/litellm/pull/30266) - fix(google\_genai): preserve complete SSE events in Vertex/Gemini image streaming by [@​Sameerlite](https://github.com/Sameerlite) in [#​30270](https://github.com/BerriAI/litellm/pull/30270) - fix(proxy): populate access\_via\_team\_ids on /v1/model/info by [@​Sameerlite](https://github.com/Sameerlite) in [#​30274](https://github.com/BerriAI/litellm/pull/30274) - chore(oss): litellm oss staging 120626 by [@​Sameerlite](https://github.com/Sameerlite) in [#​30292](https://github.com/BerriAI/litellm/pull/30292) - feat(ui): migrate policies, guardrails, prompts, tool-policies, and skills to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30263](https://github.com/BerriAI/litellm/pull/30263) - feat(ui): migrate caching, cost-tracking, transform-request, ui-theme, and logs to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30267](https://github.com/BerriAI/litellm/pull/30267) - fix(ui): gate dashboard layout on ui config load so deep links work under SERVER\_ROOT\_PATH by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30312](https://github.com/BerriAI/litellm/pull/30312) - feat(ui): migrate admin-panel, logging-and-alerts, model-hub-table, and usage to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30268](https://github.com/BerriAI/litellm/pull/30268) - fix(otel): cap metric attribute cardinality with include/exclude lists by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30257](https://github.com/BerriAI/litellm/pull/30257) - fix(proxy): grace-period key rotation 401s; return deprecated-key lookup result directly by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30327](https://github.com/BerriAI/litellm/pull/30327) - chore(deps): bump vitest, brace-expansion, pypdf and tornado by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30220](https://github.com/BerriAI/litellm/pull/30220) - refactor(ui): remove unreachable /chat page by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30178](https://github.com/BerriAI/litellm/pull/30178) - feat(ui): migrate agents and router-settings to path routes by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30323](https://github.com/BerriAI/litellm/pull/30323) - feat: strengthen coding conventions in CLAUDE.md by [@​mateo-berri](https://github.com/mateo-berri) in [#​30333](https://github.com/BerriAI/litellm/pull/30333) - feat(ui): cut the users page over to the /ui/users path route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30334](https://github.com/BerriAI/litellm/pull/30334) - feat: ruff strict-rule suppressions baseline gate by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30303](https://github.com/BerriAI/litellm/pull/30303) - feat(guardrails): add Cisco AI Defense integration ([#​28249](https://github.com/BerriAI/litellm/issues/28249)) by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30338](https://github.com/BerriAI/litellm/pull/30338) - chore(ui): remove dead UI components unreferenced by any page by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30340](https://github.com/BerriAI/litellm/pull/30340) - ci: add osv-scanner lockfile scan workflow by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30222](https://github.com/BerriAI/litellm/pull/30222) - fix(otel): record full error message on standard exception event in otel v2 by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30380](https://github.com/BerriAI/litellm/pull/30380) - test(fireworks): mock whisper transcription tests instead of live calls by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30391](https://github.com/BerriAI/litellm/pull/30391) - build(ui): pin esbuild to 0.28.1 via overrides by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30390](https://github.com/BerriAI/litellm/pull/30390) - feat(ui): cut the organizations page over to the /ui/organizations path route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30336](https://github.com/BerriAI/litellm/pull/30336) - fix(proxy): support SMTP implicit SSL (port 465) by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30395](https://github.com/BerriAI/litellm/pull/30395) - fix(mcp): default Linear MCP registry entry to streamable HTTP by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30396](https://github.com/BerriAI/litellm/pull/30396) - fix(ui): stop Virtual Keys page from infinite render loop by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30397](https://github.com/BerriAI/litellm/pull/30397) - fix(streaming): guard raise\_on\_model\_repetition against empty choices by [@​shivamrawat1](https://github.com/shivamrawat1) in [#​30485](https://github.com/BerriAI/litellm/pull/30485) - feat(otel-v2): emit the 6 gen\_ai.client.\* metrics at parity with v1 by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30326](https://github.com/BerriAI/litellm/pull/30326) - fix(mcp): drop phantom 401 span on delegated OAuth2 tool calls by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30494](https://github.com/BerriAI/litellm/pull/30494) - feat(ui): cut the teams page over to the /ui/teams path route by [@​ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#​30343](https://github.com/BerriAI/litellm/pull/30343) - fix(integrations): cap Anthropic cache\_control injection at 4 blocks by [@​shivamrawat1](https://github.com/shivamrawat1) in [#​30480](https://github.com/BerriAI/litellm/pull/30480) - chore(codecov): add Batches, Videos, and Realtime components by [@​Sameerlite](https://github.com/Sameerlite) in [#​30517](https://github.com/BerriAI/litellm/pull/30517) - test(batches): move orphan tests into tests/test\_litellm for CI coverage by [@​Sameerlite](https://github.com/Sameerlite) in [#​30510](https://github.com/BerriAI/litellm/pull/30510) - fix(guardrails): run pre\_call hook once for model-level guardrails by [@​yassin-berriai](https://github.com/yassin-berriai) in [#​30543](http…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
19 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relevant issues
Fixes LIT-3697
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
make test-unit@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewDelays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
CI (LiteLLM team)
Branch creation CI run
Link:
CI run for the last commit
Link:
Merge / cherry-pick CI run
Links:
Screenshots / Proof of Fix
Type
🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test
Changes