test(helm): cover default image tag rendering - #29370
Closed
koriyoshi2041 wants to merge 67 commits into
Closed
Conversation
… fixes, model catalog) (BerriAI#28223) * fix(opentelemetry): JSON-serialize dict metadata fields for OTEL span attributes (#27451) (#27455) Squash-merged by litellm-agent from Anai-Guo's PR. * feat(dashscope): add embeddings and reranks(qwen3-rerank) support via OpenAI-compatible endpoint (#27508) Squash-merged by litellm-agent from yimao's PR. * fix(vertex_ai/gemini): raise BadRequestError when image_url or url fi… (#24550) Squash-merged by litellm-agent from krisxia0506's PR. * fix(vertex_ai): raise error on mid-stream 429/error chunks instead of silently swallowing (#23711) Squash-merged by litellm-agent from krisxia0506's PR. * fix: raise BadRequestError for file content blocks missing 'file' sub… (#24503) Squash-merged by litellm-agent from krisxia0506's PR. * Fix Gemini MIME detection for extensionless GCS URIs (#27278) Squash-merged by litellm-agent from krisxia0506's PR. * fix(vertex_ai/partner_models): drop unused vertexai SDK gate from count_tokens (closes #28084) (#28107) Squash-merged by litellm-agent from voidborne-d's PR. * feat(chart): add support for autoscaling behavior in HPA (#27990) Squash-merged by litellm-agent from FabrizioCafolla's PR. * feat(proxy): add blocked flag to models for pause/resume from the UI (#27927) Squash-merged by litellm-agent from Cyberfilo's PR. * fix: pass socket timeouts to Redis cluster clients (#27920) Squash-merged by litellm-agent from tomdee's PR. * Fix/cache token (#28009) Squash-merged by litellm-agent from escon1004's PR. * fix(deepseek): forward reasoning_content in multi-turn thinking mode conversations (#28080) Squash-merged by litellm-agent from Divyansh8321's PR. * fix(guardrails): return HTTP 400 instead of 500 for blocked requests (#27617) * fix: reset org and tag budgets (#27326) * reset org budgets * reset tag budgets --------- Co-authored-by: Michael Riad Zaky <michaelr@Mac.localdomain> * fix(ui): omit allowed_routes from key edit save when unchanged (#27553) * fix(ui): omit allowed_routes from key edit save when unchanged When a team admin opens Edit Settings on a key with key_type=AI APIs and saves without changing anything, the UI re-sends the existing allowed_routes value, which the backend's _check_allowed_routes_caller_permission gate rejects for non-proxy-admins (LIT-2681). Strip allowed_routes from the patch in handleSubmit when it deep-equals the original keyData.allowed_routes. The backend treats absence as "leave alone," so no-op saves now succeed for non-admins. Admins explicitly editing the field still send the new value. * fix(ui): order-insensitive allowed_routes diff + cover null-original case Address Greptile review: - Switch the "is allowed_routes unchanged" check to a Set-based comparison so a server-side reorder of the array doesn't register as a user edit and re-trigger LIT-2681. - Add two regression tests: (1) keyData.allowed_routes is null and the form is untouched — patch should strip the field; (2) server returned routes in a different order than the user originally entered — patch should still recognize the value as unchanged. * chore(ui): strip ticket refs and tighten comments in key edit fix - Remove internal-tracker references from in-code comments - Tighten the WHY comment in handleSubmit to two lines - Drop redundant test-block comments — test names already describe the case * fix(ui): annotate Set<string> generic in allowed_routes diff to fix tsc * fix(guardrails): return HTTP 400 instead of 500 for guardrail-blocked requests GuardrailRaisedException and BlockedPiiEntityError both lacked a status_code attribute. When these exceptions reached the proxy exception handler (getattr(e, 'status_code', 500)), the fallback defaulted to HTTP 500 — making intentional guardrail blocks indistinguishable from server errors and causing unnecessary client retries. Changes: - Add status_code=400 (keyword-only) to GuardrailRaisedException - Add status_code=400 (keyword-only) to BlockedPiiEntityError - Update _is_guardrail_intervention() to recognize both exceptions so downstream loggers record 'guardrail_intervened' instead of 'guardrail_failed_to_respond' - Add 6 unit tests for default/custom status codes and getattr pattern - Strengthen existing blocked-action test with status_code assertion Fixes #24348 --------- Co-authored-by: Michael-RZ-Berri <michael@berri.ai> Co-authored-by: Michael Riad Zaky <michaelr@Mac.localdomain> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> Co-authored-by: Krrish Dholakia <krrish+github@berri.ai> * fix(router/proxy): address Greptile P1+P2 review comments on PR #28161 - router: raise ServiceUnavailableError (503) instead of RouterRateLimitErrorBasic (429) when a specifically-addressed deployment is administratively blocked; 429 misleads retry-enabled clients into spinning forever against a paused model - proxy_server: compute get_fully_blocked_model_names() once before both branches in model_list() instead of duplicating the call in each branch - deepseek: upgrade silent debug log to warning when injecting placeholder reasoning_content so callers are clearly notified of degraded multi-turn quality - tests: update two blocked-deployment assertions to expect ServiceUnavailableError Co-authored-by: Cursor <cursoragent@cursor.com> * fix: address bug detection findings (cache token order, mutable defaults) Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix: address bugs in async pass-through, anthropic cache token detection, rerank tests - async_get_available_deployment_for_pass_through: enforce blocked check on specific deployments - cost_calculator: detect anthropic-style usage by attribute presence (not truthiness) to avoid mixing OpenAI cached_tokens into anthropic normalization when read=0 - dashscope rerank tests: pass request to httpx.Response constructions for consistency Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix code qa * fix(vertex_ai/gemini): strip MIME parameters from GCS contentType GCS object metadata's contentType field can include parameters such as 'text/html; charset=utf-8'. Strip them in _apply_gemini_mime_type_aliases so downstream get_file_extension_from_mime_type sees a bare MIME type. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(vertex_ai/gemini): clarify mime-type error message string concatenation Co-authored-by: Yassin Kortam <yassin@berri.ai> * feat(oci): add embeddings, fix streaming/reasoning, expand model catalog - Add OCIEmbedConfig with full Cohere embed support (7 models, batch up to 96) - Fix sync streaming: split SSE events on \n\n before JSON parsing - Fix reasoning models (Gemini 2.5, xAI Grok): make completionTokens and message optional in OCIResponseChoice to handle max_tokens exhausted on reasoning - Fix compartment_id resolution in chat transform to use resolve_oci_credentials - Fix tool call id: make OCIToolCall.id optional, generate UUID fallback for providers (Google via OCI) that omit it - Add OCI_KEY env var support for inline PEM keys - Fix datetime.utcnow() deprecation in request signing - Expand model catalog: 29 OCI models including Llama 4, Gemini 2.5, xAI Grok, Cohere Command A, and all Cohere embed variants - Add 37 live integration tests: sync/async completions for Meta/Google/xAI/Cohere, sync/async embeddings, tool use across all vendors, streaming, env var auth - Add 23 embed unit tests covering all transform and validation paths * fix(oci): remove dead OCI elif branch in utils.py, align async split_chunks with sync version * test(oci): add unit tests for split_chunks fix and no-duplicate-OCI-branch guard * fix(oci): address remaining bugs from issue #25082 — streaming signed body, Cohere stop sequences, hardcoded defaults - Bug 1: sync and async streaming paths now use signed_json_body when provided instead of re-serializing data with json.dumps() — the OCI RSA-SHA256 signature covers the exact request body bytes, so re-serializing produces an invalid sig - Bug 3: Cohere stop sequences now map to 'stopSequences' (was incorrectly 'stop') - Bug 4: removed hardcoded Cohere defaults (maxTokens=600, temperature=1, topK=0, topP=0.75, frequencyPenalty=0) that silently overrode user intent on every call - Added 6 unit tests covering all three fixes * fix(oci): comprehensive code quality pass — bugs, tests, schema accuracy - Fix Cohere tool call IDs (was always call_0; now UUID per call) - Fix TOOL_CALL finish reason mapping in both sync and streaming paths - Fix Cohere stop parameter mapping (stop → stopSequences) - Remove hardcoded Cohere defaults (maxTokens/topK/topP/frequencyPenalty) - Fix content[0] safety guard against empty content arrays - Fix streaming signed body used consistently (not re-serialized) - Raise OCIError (not bare Exception/ValueError) throughout - Centralize OCI_API_VERSION constant; import uuid at module level - Fix embed get_complete_url to strip trailing slashes from api_base - Fix OCIEmbedResponse schema: add inputTextTokenCounts (actual OCI field) - Fix embed usage computed from inputTextTokenCounts (sum of per-input counts) - Fix Cohere toolCallId included in tool result messages - Add OCIToolCall.id as Optional (absent in Google/xAI streaming chunks) - Update tests to reflect correct behavior (no hardcoded defaults, UUID ids, deferred credential validation, OCIError vs ValueError, real response schema) * test(oci): move integration tests to tests/llm_translation/ Addresses greptile P1: tests/test_litellm/ is for mock-only unit tests (make test-unit target). Real-network OCI tests now live in the correct location alongside other provider integration tests. * fix(oci): align types and transformation with official OCI SDK - Remove OCIVendors.GEMINI — apiFormat="GEMINI" is invalid; all non-Cohere models use apiFormat="GENERIC" - Add toolChoice, logitBias, logProbs to OCIChatRequestPayload so params present in the mapping are no longer silently dropped by Pydantic - Exclude n→numGenerations from Cohere param map (not a Cohere API field) - Fix CohereToolResult: change callId/result to call/outputs matching the OCI SDK's CohereToolResult structure - Fix CohereToolMessage: replace non-existent toolCallId with toolResults list; update adapt_messages_to_cohere_standard to build proper tool-result history entries by resolving tool call name+params from preceding assistant messages - Map generic-model stream finish reasons to OpenAI convention (COMPLETE→stop, MAX_TOKENS→length, TOOL_CALLS→tool_calls), consistent with the existing Cohere streaming path - Add optional id field to OCIEmbedResponse so valid API responses carrying an id are not rejected by the Pydantic model * fix(oci): use 'output' key in Cohere tool result outputs (matches reference impl) * fix(oci): port schema/type utilities from langchain-oracle reference impl - Add resolve_oci_schema_refs: inline $ref/$defs — OCI rejects JSON Schema refs - Add resolve_oci_schema_anyof: flatten Optional[T] anyOf (Pydantic v2 emits these) - Add sanitize_oci_schema: strip title, normalise null types, ensure array items - Add OCI_JSON_TO_PYTHON_TYPES: Cohere expects Python type names (str/int/float), not JSON Schema names (string/integer/number) - Add enrich_cohere_param_description: embed enum/format/range/pattern constraints into description since CohereParameterDefinition has no dedicated fields - Apply all of the above in adapt_tool_definitions_to_cohere_standard and adapt_tool_definition_to_oci_standard - Fix toolChoice conversion: map OpenAI string ('auto','none','required') to OCI dict form ({"type":"AUTO"} etc.) — the API rejects plain strings - Update unit test expectations to match correct Python type names and enriched descriptions * refactor(oci): split transformation.py into cohere.py and generic.py transformation.py was 1 243 lines doing too many jobs. Split along the same boundaries as the langchain-oracle reference (providers/cohere.py, providers/generic.py): chat/cohere.py — Cohere message/tool building, response + stream parsing chat/generic.py — Generic message/tool building, response + stream parsing transformation.py — thin OCIChatConfig orchestrator + OCIStreamWrapper Public symbols (OCIChatConfig, OCIStreamWrapper, adapt_messages_to_*, OCIRequestWrapper, version, …) remain importable from transformation.py for backward compatibility. OCIStreamWrapper gains delegating shims for _handle_cohere_stream_chunk and _handle_generic_stream_chunk so existing test call sites keep working unchanged. transformation.py: 1 243 → 620 lines * refactor(oci): principal-level code quality pass - Remove _extract_text_content duplication — single definition in cohere.py, imported where needed; instance method on OCIChatConfig eliminated - Move cryptography imports to module level with _CRYPTOGRAPHY_AVAILABLE flag and _require_cryptography() guard; no more re-import on every signing call - Move litellm version import to module level via litellm._version; remove inline import inside validate_oci_environment - sign_with_manual_credentials now returns Tuple[dict, bytes] matching sign_with_oci_signer — asymmetry eliminated, Optional[bytes] guards removed throughout stream wrappers (signed_json_body: bytes = b"") - Rename _openai_to_oci_cohere_param_map → openai_to_oci_cohere_param_map for consistency with openai_to_oci_generic_param_map - Remove double-key bug in map_openai_params where responseFormat was stored under both OCI and OpenAI key names simultaneously - Remove delegating shims (adapt_messages_to_cohere_standard, adapt_tool_definitions_to_cohere_standard, _handle_generic_stream_chunk) from OCIChatConfig/OCIStreamWrapper; tests now import directly from cohere.py and generic.py where symbols live - Trim __all__ to 7 genuine public symbols; remove the 13-symbol list that existed only to support test imports - Collapse per-model integration test classes into pytest.mark.parametrize; CHAT_MODELS list is the single source of truth for model-specific config - Black + Ruff clean across all OCI files * fix(oci): address PR review findings - types/llms/oci.py: add "TOOL_CALL" to CohereChatResponse.finishReason Literal so Pydantic does not raise ValidationError on non-streaming Cohere tool-use calls (Greptile P1) - test_oci_cohere_tool_calls.py: add test covering TOOL_CALL finish reason - model_prices_and_context_window.json: remove 6 duplicate oci/cohere.embed-* keys that were silently overridden by the more complete entries already present in the file (Greptile P1) - common_utils.py: move OCI_API_VERSION here from chat/transformation.py so embed/transformation.py does not need to import chat/transformation; change Protocol stub body from ... to pass (CodeQL "statement no effect"); add comment to sha256_base64 clarifying it implements OCI HTTP signing spec, not password hashing (CodeQL false positive) - chat/transformation.py: import CustomStreamWrapper from litellm_core_utils.streaming_handler instead of litellm.utils to reduce import cycle depth (CodeQL cyclic import) - chat/cohere.py, chat/generic.py: import Usage and ChatCompletionMessageToolCall from litellm.types.utils instead of litellm.utils for the same reason - embed/transformation.py: import OCI_API_VERSION from common_utils instead of chat/transformation (removes the embed→chat import edge) * test(oci): add unit tests to improve patch coverage - test_oci_common_utils.py (new): covers sha256_base64, build_signature_string, OCIRequestWrapper.path_url, resolve_oci_credentials, get_oci_base_url, validate_oci_environment, sign_with_oci_signer error paths, sign_oci_request routing, load_private_key_from_file error paths, resolve_oci_schema_refs (including circular ref and external $ref), resolve_oci_schema_anyof, sanitize_oci_schema (all branches), enrich_cohere_param_description - test_oci_generic_chat.py (new): covers content-message error paths (non-dict item, unsupported type, non-string text, invalid image_url), tool-call validation error paths, adapt_messages_to_generic_oci_standard error paths, handle_generic_response (None message, text content, tool calls), handle_generic_stream_chunk (finish reasons, streaming tool calls), OCIStreamWrapper non-string chunk error - test_oci_chat_transformation.py: add error paths for validate_environment (empty messages), transform_request (missing compartment_id, Cohere without user messages), transform_response (error key), map_openai_params (unsupported param with and without drop_params), tool_choice string mapping - test_oci_cohere_tool_calls.py: add edge cases for stream chunk finish reasons (TOOL_CALL, MAX_TOKENS, unknown), _extract_text_content with non-dict list items and non-string input, adapt_messages_to_cohere_standard with malformed JSON tool arguments * fix(oci): rename supports_streaming to supports_native_streaming in model prices The JSON schema for model_prices_and_context_window.json uses `supports_native_streaming` (not `supports_streaming`) and has `additionalProperties: false`. Rename the field across all OCI entries to pass the schema validation test. * test(oci): add 67 tests targeting uncovered happy paths for coverage Boost patch coverage on the four lowest-coverage OCI files: - common_utils.py: sign_with_manual_credentials (oci_key / oci_key_file paths), sign_oci_request routing, _require_cryptography - generic.py: adapt_messages_to_generic_oci_standard (all roles), adapt_tool_definition_to_oci_standard, adapt_tools_to_openai_standard, handle_generic_stream_chunk text/finish-reason paths - cohere.py: _extract_text_content, adapt_messages_to_cohere_standard (all roles including tool results), handle_cohere_response / handle_cohere_stream_chunk all finish-reason branches - transformation.py: get_vendor_from_model, OCIChatConfig._get_optional_params (toolChoice string→dict, responseFormat, tools for both vendors), transform_request for GENERIC model, get_sync/async_custom_stream_wrapper with mocked HTTP, OCIStreamWrapper.chunk_creator happy paths * fix(oci): suppress CodeQL false positive on sha256_base64 (OCI HTTP signing, not password hashing) * fix(oci): remove 6 duplicate model price entries and reconcile conflicting values Six OCI chat model keys appeared twice in model_prices_and_context_window.json with conflicting pricing/context data (JSON parsers silently discard the first). Remove the first-occurrence entries and update the surviving entries: - meta.llama-4-maverick / llama-4-scout: keep updated entries (free preview pricing, larger context windows, vision support) - meta.llama-3.1-70b: keep original pricing, restore supports_native_streaming - google.gemini-2.5-{flash,pro,flash-lite}: keep OCI pricing page values, restore supports_native_streaming * fix(oci): route GPT-5 family to maxCompletionTokens GPT-5 / GPT-5-mini / GPT-5-nano / GPT-5.5 on OCI reject "maxTokens" with HTTP 400: Invalid 'maxTokens': Unsupported parameter: 'maxTokens' is not supported with this model. Use 'maxCompletionTokens' instead. (Same convention as OpenAI's reasoning-API contract.) Add a model-aware rename in OCIChatConfig._get_optional_params so the request payload uses maxCompletionTokens when the model id starts with openai.gpt-5. Regular Llama / Cohere / Gemini / GPT-4.x continue to use maxTokens unchanged. Also widen OCIChatRequestPayload to carry the new optional field so it survives Pydantic serialization. Verified live against OCI us-chicago-1: - openai.gpt-5, gpt-5-mini, gpt-5-nano, gpt-5.5 all return 200 - Full feature sweep on gpt-5.5 (basic, system, multi-turn, streaming, tools, usage) all green - meta.llama-3.3-70b-instruct still uses maxTokens (no regression) 4 new unit tests cover the helper, the routing in both pre- and post-translation states, and Pydantic serialization. * ci(oci): fix CI failures — black formatting + recursive_detector ignore - Run black on litellm/llms/oci/common_utils.py + 3 OCI test files that drifted out of black-compliance during the rebase. - Add the three bounded recursive functions in oci/common_utils.py (`_resolve`, `resolve_oci_schema_anyof`, `sanitize_oci_schema`) to the recursive_detector IGNORE_FUNCTIONS list. All three are bounded: `_resolve` uses a `resolving_stack` cycle guard; the other two are bounded by JSON-schema tree depth (no cycles in well-formed input), matching the pattern of the existing OCI/Vertex schema walkers already on the list. * fix(oci): silence MyPy errors in cohere.py — typed-dict access Two errors flagged by `lint` CI: llms/oci/chat/cohere.py:73: "object" has no attribute "__iter__" llms/oci/chat/cohere.py:119: No overload variant of "get" of "dict" matches argument types "object", "CohereToolCall" Both stem from `msg.get("tool_calls")` / `msg.get("tool_call_id")` returning `object` per the AllMessageValues TypedDict union. Bind to `Any` locally for the iteration and coerce the lookup key with `str()`, removing the now-unused `# type: ignore` on those lines. No behaviour change — pure type-narrowing for the type checker. * fix(oci): silence CodeQL py/weak-sensitive-data-hashing on sha256_base64 CodeQL's taint analysis traces request bodies back to environment-loaded secrets and flags `hashlib.sha256(body).digest()` as `py/weak-sensitive-data-hashing` — even though SHA-256 is the algorithm mandated by the OCI HTTP request signing spec for the `x-content-sha256` header (not a password/secret hash). The previous suppression used legacy `# lgtm[...]` syntax which the modern CodeQL action ignores. Switch to Python's standard `hashlib.sha256(..., usedforsecurity=False)` (Python 3.9+) which CodeQL honours as a non-security declaration. Behaviour unchanged. * feat(oci): add reasoning_effort passthrough — only true missing primitive OCI's GenericChatRequest exposes a reasoningEffort field (NONE/MINIMAL/LOW/MEDIUM/HIGH) that's the single biggest cost knob for reasoning-capable models on the service: - GPT-5 family - Gemini 2.5 - Grok reasoning variants (3-mini, 4-fast, 4.20) - Cohere Command-A-Reasoning Setting reasoning_effort=LOW typically cuts reasoning-token spend 5-10× vs the default. Without exposing this, litellm users had no way to tune cost-vs-quality on these models. The other GenericChatRequest fields (verbosity, parallel_tool_calls, logit_bias, n, metadata, web_search_options, prediction) are not exposed because they are not missing primitives — they either duplicate prompt-engineering, framework-level controls, or are too niche to justify the maintenance surface. We only ship what users genuinely can't accomplish another way. Excluded from the Cohere v1 param map: CohereChatRequest has no reasoningEffort field, and Cohere reasoning models (cohere.command-a-reasoning) use COHEREV2 which is a separate request type not covered by this PR. Verified live: GPT-5.5 + reasoning_effort="HIGH" sends {"reasoningEffort": "HIGH"} on the wire and OCI accepts the request. * feat(oci): reasoning_effort + reasoning_tokens for OCI GenAI Three small additions for OCI reasoning models, requested by users testing the PR in production fork builds: 1. **reasoning_effort param mapping (GENERIC vendors).** OCI expects uppercase levels ("LOW"/"MEDIUM"/"HIGH"/"NONE") on `reasoningEffort`, but OpenAI-compatible clients send lowercase. Mapped + uppercased in `_get_optional_params`. Marked unsupported on Cohere V1/V2 since OCI Cohere has no reasoning models (avoids Pydantic validation failure on CohereChatRequest). 2. **"disable" → "NONE" mapping.** OpenAI uses "disable" to turn off reasoning; OCI uses "NONE". Without this, callers get a 400. 3. **reasoning_tokens propagated to Usage.** OCI returns `completionTokensDetails.reasoningTokens` but it wasn't being passed to LiteLLM's Usage object. Now flows through to `Usage.completion_tokens_details.reasoning_tokens` so callers can track reasoning token consumption for cost/observability. Tests: 7 new unit tests in TestOCIReasoningEffort covering upper/lower case, "disable"→"NONE", Cohere drop/raise paths, and reasoning_tokens extraction (with and without completionTokensDetails). 5 new live integration tests against xai.grok-3-mini in us-chicago-1 verifying the full request/response loop end-to-end. Existing test_transform_response_simple_text assertion that completion_tokens_details was None has been updated to assert reasoning_tokens flows through. Verified live on xai.grok-3-mini: reasoning_effort=low → OCI accepts "LOW", returns reasoningTokens=316 in usage. reasoning_effort=disable → OCI accepts "NONE". Full suite: 370/370 unit + 51/51 integration. * fix(codeql): re-scope py/weak-sensitive-data-hashing exclusion to OCI signing file CodeQL's taint analysis re-fires the `py/weak-sensitive-data-hashing` alert at `litellm/llms/oci/common_utils.py:103` whenever upstream code paths into the OCI signing module change (touching `transformation.py` opens new flow paths that CodeQL re-evaluates from scratch). The `hashlib.sha256(..., usedforsecurity=False)` declaration silences the direct-call form of the query but not the taint-flow form. SHA-256 here is mandated by the OCI HTTP signing specification for the x-content-sha256 content-integrity header — not for password storage: https://docs.oracle.com/en-us/iaas/Content/API/Concepts/signingrequests.htm CodeQL has no per-query path filter and GitHub Code Scanning ignores inline lgtm/codeql comments, so path-ignoring this single ~560-line signing utility file is the narrowest available suppression. All other files retain full coverage of py/weak-sensitive-data-hashing — including litellm/proxy/utils.py where the rule legitimately applies. This restores the NEUTRAL CodeQL state the PR had on prior commits (see `2111c98af7` for the same approach on the previous branch evolution that the cherry-pick was rebased onto a different baseline). * fix(oci): drop duplicate text on Cohere streaming terminal chunk OCI Cohere's terminal SSE event re-sends the full assembled response in `text` alongside a populated `chatHistory`. Emitting that text as another delta concatenates the entire response onto the already-streamed output (e.g. "How can I help?How can I help?"). Use `chatHistory is not None` as the discriminator for the consolidated terminal event — `finishReason` is a weaker signal that could in principle appear on a non-consolidated chunk. The two coincide today; this preserves correctness if OCI ever ships finishReason on an incremental chunk. Adds a live-OCI integration regression test that compares streamed vs non-streamed length and asserts the response prefix appears only once. Verified to fail under the previous code with the exact reported reproduction: 'Hello! How can I help you today?Hello! How can I help you today?'. Reported by @gotsysdba on PR #25177. * fix(oci): buffer SSE stream across HTTP read boundaries The old split_chunks helper split each individual HTTP read on "\n\n", which assumed SSE event boundaries always aligned with read boundaries. In practice the OCI streaming endpoint delivers events that may: - straddle two reads (chunk_creator gets a truncated JSON and crashes) - arrive separated by a single "\n" instead of "\n\n" - share a read with multiple complete events Replace the inline split with module-level helpers _iter_sse_events (sync) / _aiter_sse_events (async) that maintain a buffer across reads, split on any newline, and yield only complete "data:" lines. Add 25 regression tests covering event-split-across-reads, tiny-chunk reads, single-newline separators, keepalive/comment lines, trailing partial events flushed at EOF, "\r\n" line endings, and an end-to-end smoke test that feeds an awkwardly-chopped payload through the splitter into OCIStreamWrapper.chunk_creator. Reported by John Lathouwers. * test(oci): repoint TestOCIKeyNormalization to sign_with_manual_credentials The signing helper moved from OCIChatConfig._sign_with_manual_credentials to a module-level sign_with_manual_credentials in common_utils.py. Four tests in TestOCIKeyNormalization still called the old method: - 2 failed outright with AttributeError - 2 passed by accident because they used pytest.raises(Exception), which happily caught the AttributeError instead of exercising the intended OCIError path Repoint all four to the new module-level function so they exercise the actual oci_key type-validation branch. * fix(oci): validate oci_region before URL interpolation to prevent SSRF Anchor oci_region to ^[a-z][a-z0-9-]{0,30}[a-z0-9]$ inside get_oci_base_url so user-supplied regions that would redirect the signed request to an attacker-controlled host (e.g. 'evil.com/#') fail with HTTP 400 before the URL or signature is built. Empty string still falls back to the us-ashburn-1 default, so existing callers are unaffected. * test(audio): skip when gpt-4o-audio-preview is unavailable upstream OpenAI retired `gpt-4o-audio-preview` (404 model_not_found in CI as of 2026-05-19), and the existing try/except in these tests only re-raised on 'openai-internal' errors. Other exceptions were silently swallowed, so the next line ran with an unbound `response`/`completion` and failed with an unrelated UnboundLocalError that masked the real cause. Extend the skip condition to also cover model_not_found / 'does not exist' so the suite reports the upstream outage cleanly, matching the pattern used in ce87c41 for the realtime and nvidia_nim rerank tests. Re-raise unknown exceptions instead of falling through. * fix(oci/router): catalog-driven maxCompletionTokens; generic blocked-deployment message - Drive OCI maxCompletionTokens via supports_reasoning from the model catalog instead of a hardcoded openai.gpt-5 prefix. Add OCI GPT-5 family entries (gpt-5, gpt-5-mini, gpt-5-nano) with supports_reasoning: true. Gate the override to non-Cohere vendor so Cohere reasoning models keep maxTokens (Cohere endpoint does not accept maxCompletionTokens). - Replace proxy-specific 'Contact your proxy admin' phrasing in the four Router blocked-deployment ServiceUnavailableError messages with neutral SDK-appropriate text. * fix(oci/cohere): guard handle_cohere_response against missing usage * fix(oci): address bug review findings in chat transformation - Cohere param map: keep tool_choice/n as False (not omitted) so unsupported params are dropped or rejected rather than silently passed through. - get_complete_url: when an explicit api_base/litellm.api_base is provided, use it as-is instead of unconditionally appending /20231130/actions/chat (mirrors the embed config behavior). - Cohere stream: require both chatHistory and finishReason to be present to identify a terminal consolidation chunk, avoiding silent text suppression if chatHistory ever appears on a non-terminal chunk. - Generic usage: use 'is not None' for reasoningTokens so a legitimate value of 0 is preserved instead of being treated as absent. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci/cohere): emit tool calls in streaming and null content when text empty handle_cohere_response now sets message.content to None when the Cohere response text is empty, matching the OpenAI convention for tool-call-only responses. handle_cohere_stream_chunk now extracts toolCalls — both directly from the chunk and from the terminal chunk's chatHistory CHATBOT message — and emits them in the delta. Previously, CohereStreamChunk lacked a toolCalls field, so any tool calls in the stream were silently dropped. * fix(oci): preserve tool results, embed URL path, and generic finish reason - Use SerializeAsAny on CohereChatRequest.chatHistory so subclass-specific fields like CohereToolMessage.toolResults are not dropped during Pydantic v2 serialization. - Make OCIEmbedConfig.get_complete_url append the /20231130/actions/embedText action path consistently with chat, so setting litellm.api_base to the region inference base URL no longer posts to the bare hostname. - Map OCI finishReason (COMPLETE / MAX_TOKENS / TOOL_CALLS) to OpenAI finish_reason values in handle_generic_response, mirroring the streaming handler and the Cohere non-streaming handler. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci/generic): silence mypy assignment error on dynamic finish_reason * fix(oci/embed): always set usage on embedding response Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci/chat): append /20231130/actions/chat to explicit api_base Restore the embed-style behavior so OCIChatConfig.get_complete_url always appends the OCI GenAI chat path. Routing through get_oci_base_url ensures the optional explicit api_base has its trailing slash stripped before the suffix is joined, matching the embed config and the test_respects_explicit_api_base expectation. * fix(oci/cohere): mark logprobs/logit_bias unsupported and normalize unknown stream finish reasons Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci/cohere): preserve trailing tool result in chatHistory When the last message in the OpenAI-format input is a tool result (the standard agentic continuation pattern), the prior messages[:-1] slice silently dropped that tool result from chatHistory and the model never saw it. Excluding the last user message by index instead keeps tool results that trail the last user turn intact. * fix(main): remove dead OCI embedding elif block The earlier elif at line 5119 already routes OCI embeddings through the base HTTP handler with the headers None-guard, so the later identical block was unreachable dead code. * test(oci): move integration tests out of llm_translation mock-only folder Greptile flags tests/llm_translation/ as mock-only via a project-specific rule; relocate the live-network OCI integration suite to tests/integration/ and adjust the in-file sys.path / run instructions accordingly. * fix(oci/cohere): suppress tool calls on stream terminal consolidation chunk The terminal SSE event re-sends the full assembled response in both `text` and `chatHistory`. The existing logic already suppresses `text` to avoid double-emit, but tool calls extracted from the terminal chunk (via `typed_chunk.toolCalls` or the `chatHistory` CHATBOT fallback) would still be re-emitted with fresh uuid4 IDs. If OCI Cohere ever streams tool calls progressively in intermediate chunks (now possible since CohereStreamChunk has a toolCalls field), this would cause downstream agentic frameworks to execute each tool call twice. Suppress tool calls on the terminal consolidation chunk for the same reason `text` is suppressed. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci,httpx): normalize finish_reason, preserve response_format, fix sync embed JSON content-type - cohere.py / generic.py: normalize unknown OCI finishReason values (ERROR, ERROR_TOXIC, CONTENT_FILTERED, USER_CANCEL, ...) to 'stop' in non-streaming and streaming generic handlers, matching the streaming Cohere handler so downstream consumers switching on finish_reason aren't broken by raw OCI values. - transformation.py: restore the dual-key alias so optional_params still carries the original 'response_format' key alongside the OCI-mapped 'responseFormat'. Downstream litellm framework code (json_mode detection, logging) inspects 'response_format' after map_openai_params runs. - llm_http_handler.py: make the sync embedding path mirror the async path — when sign_request returns no signed_body, send via json=data (which sets Content-Type: application/json) instead of data=json.dumps(data) which doesn't. Removes a sync/async behavioural asymmetry for non-OCI providers that adopt the sign_request pattern. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci): clean up OCIChatConfig init, normalize generic stream finish reasons, correct embed sign_request return type - Replace fragile setattr(self.__class__, ...) pattern in OCIChatConfig.__init__ with a @property for has_custom_stream_wrapper, matching the pattern used by other providers. - Normalize unknown OCI finish reasons (e.g. ERROR, ERROR_TOXIC, USER_CANCEL) to 'stop' in handle_generic_stream_chunk, matching the existing Cohere stream handler behaviour. - Tighten OCIEmbedConfig.sign_request return type from Tuple[dict, Optional[bytes]] to Tuple[dict, bytes] — sign_oci_request never returns None for the body, and this matches OCIChatConfig.sign_request. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci): strip trailing action path in get_oci_base_url to avoid URL doubling A fully-formed OCI endpoint URL (e.g. https://inference.generativeai.us-chicago-1.oci.oraclecloud.com/20231130/actions/chat) passed via api_base previously had the action path appended a second time by get_complete_url in both chat and embed configs, yielding a 404. get_oci_base_url now strips a trailing /20231130/actions/<name> so callers can always append the action path safely. * fix(httpx): preserve sync embed data= kwarg to avoid breaking mock-based tests The earlier sync_httpx_client.post() call passed data=json.dumps(data), which downstream embedding tests assert on (e.g. tests for hosted_vllm, jina_ai, watsonx). Switching to json=data changed the kwarg name and broke those tests. The OCI signed_body path keeps using data=signed_body and is unaffected. * fix(oci): stable tool-call ids across stream chunks; lenient Cohere finishReason - Replace random uuid4 per chunk with a deterministic content-derived digest for synthetic tool-call ids in both Cohere and Generic OCI handlers. Previously, when OCI omitted 'id' (always for Cohere, often for Generic streaming deltas), every chunk for the same logical tool call received a new uuid, causing downstream stream-mergers (which key off id) to treat each fragment as a distinct call. - Relax CohereChatResponse.finishReason from a strict Literal[...] to Optional[str], matching CohereStreamChunk.finishReason. The handle_cohere_response 'elif oci_finish_reason is not None' fallback was previously unreachable because Pydantic raised ValidationError on any unknown value before the fallback executed. Now non-streaming responses degrade unknown reasons to 'stop' just like the streaming path. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci/embed): validate OCI credentials in validate_environment Mirror OCIChatConfig.validate_environment so embedding requests fail fast with a clear error when oci_user/oci_fingerprint/oci_tenancy/ oci_compartment_id or an oci_key/oci_key_file is missing, instead of deferring the failure until sign_request. Co-authored-by: Yassin Kortam <yassin@berri.ai> * test(oci/embed): expect OCIError from validate_environment when credentials are missing OCIEmbedConfig.validate_environment now raises eagerly (mirroring OCIChatConfig) when oci_user/oci_fingerprint/oci_tenancy/oci_compartment_id or oci_key/oci_key_file is missing. Update the test to match. * fix(oci): polish stream chunk handling and signed body default - cohere stream terminal consolidation now emits content=None instead of "" - drop redundant index truthiness check (None is already replaced with 0) - accept both "TOOL_CALL" and "TOOL_CALLS" finish reasons in cohere - signed_json_body defaults to None and uses explicit None check, so an explicitly empty bytes body wouldn't be silently re-serialized Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci/chat): catch pydantic ValidationError when parsing OCI responses Pydantic v2 raises ValidationError (not TypeError) when field validation fails, so malformed OCI completion responses or stream chunks would propagate unhandled out of handle_generic_response, handle_generic_stream_chunk, and handle_cohere_stream_chunk. Widen the except clauses to also catch ValidationError so callers get a clean OCIError. * fix(oci/catalog): real prices for Llama 4, drop zero-cost OCI OpenAI entries Zero-cost catalog entries (input_cost_per_token=0, output_cost_per_token=0) make proxy spend tracking silently report $0 for these paid OCI models, so any caller can drive them without decrementing a budget. For Llama 4 Maverick and Scout, OCI charges the same character-based rate as Llama 3.3 70B ($0.0018 per 10,000 characters), so use the same per-token price as the existing oci/meta.llama-3.3-70b-instruct entry (7.2e-07 in/out). For oci/openai.gpt-5, gpt-5-mini, gpt-5-nano, gpt-oss-120b, and gpt-oss-20b, no public per-token pricing is available; drop the entries so operators must register them with explicit custom pricing. The existing GPT-5 reasoning test fixture already injects synthetic entries when the catalog omits them, so the chat transformation's supports_reasoning lookup keeps working in tests. * fix(oci/chat): wrap CohereChatResult construction in try/except Match the handle_generic_response pattern: surface OCIError with the upstream status code instead of letting a raw pydantic.ValidationError propagate when the Cohere response payload is malformed. * fix(oci): harden Cohere stream/finish-reason and dedupe maxTokens param mapping - Cohere stream: track per-stream tool-call emission and only suppress the terminal consolidation chunk's tool calls once they've been seen earlier. Prevents silent drop if tool calls are delivered exclusively on the terminal chunk. - Cohere stream: emit content=None (not "") on non-terminal text-free chunks (e.g. tool-call-only / keep-alive) so downstream consumers that distinguish missing vs explicitly-empty deltas behave correctly. - Generic handlers: accept singular TOOL_CALL finish reason in addition to TOOL_CALLS, matching the Cohere handlers. - _get_optional_params: when both max_tokens and max_completion_tokens are provided, explicitly prefer max_completion_tokens instead of relying on dict iteration order. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci): emit content=None instead of empty string for text-free generic stream chunks Co-authored-by: Yassin Kortam <yassin@berri.ai> * test(oci): expect content=None for text-free generic stream chunks handle_generic_stream_chunk now emits content=None instead of empty string when a chunk carries no text parts. Update the corresponding no-message test to match. * codeql: narrow OCI sha256 suppression to query-filter, not whole file paths-ignore was suppressing every CodeQL query on litellm/llms/oci/common_utils.py, hiding all future findings in a security-critical file (private key loading, credential resolution, URL construction, RSA signing). Move the suppression for py/weak-sensitive-data-hashing into query-filters so common_utils.py remains fully analyzed by every other query. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci): use locale-independent RFC 7231 date for manual signing email.utils.formatdate(usegmt=True) emits canonical English weekday/ month abbreviations regardless of system locale, so signature verification doesn't break on non-en_US deployments. * fix(oci): strip 'oci/' prefix in get_vendor_from_model Previously, get_vendor_from_model split on '.' without stripping the optional 'oci/' provider prefix, so 'oci/cohere.command-a-03-2025' was routed through the GENERIC pipeline instead of COHERE. Co-authored-by: Yassin Kortam <yassin@berri.ai> * codeql: scope OCI sha256 suppression to common_utils.py via filter-sarif Replace the global query-filters exclude for py/weak-sensitive-data-hashing with a SARIF post-filter that only drops the alert when it originates from litellm/llms/oci/common_utils.py, keeping the rule active on every other SHA-256 callsite in the repository. * Fix OCI chat bugs: tool_calls None key, dead max_tokens dedup, single-event stream text suppression - handle_cohere_response: omit tool_calls key from message dict when None, matching the generic handler's behaviour and avoiding tripping consumers that key off 'tool_calls' in message. - _get_optional_params: remove dead prefer_max_completion branch. By the time this helper runs, map_openai_params has already collapsed max_tokens/max_completion_tokens onto the OCI alias, so the OpenAI-key membership check is unreachable. - handle_cohere_stream_chunk: add prior_text_emitted parameter mirroring prior_tool_calls_emitted. The terminal consolidation chunk's text is only suppressed when prior deltas already emitted text — otherwise (degenerate single-event stream) the text passes through so the response content isn't silently lost. OCIStreamWrapper now tracks emitted text alongside emitted tool calls. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci): preserve all text parts in generic response and emit SYSTEM role for Cohere - handle_generic_response: iterate all content parts and concatenate text (matches the streaming handler) so non-leading text parts are not lost and a leading non-text part does not suppress trailing text. - adapt_messages_to_cohere_standard: emit CohereSystemMessage for system messages so direct callers do not silently drop them. The Cohere request builder filters system messages before calling this helper to avoid duplicating preambleOverride content into chatHistory. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci): normalise dict-format tool_choice to OCI flat uppercase shape The OCI Generative AI API only accepts toolChoice values of the form {"type": "AUTO"|"NONE"|"REQUIRED"} or {"type": "FUNCTION", "name": "<fn>"}. The previous conversion only handled string tool_choice values, so OpenAI's standard dict shape {"type": "function", "function": {"name": "<fn>"}} passed through unchanged and was rejected by OCI with a 400. Normalise the dict shape by uppercasing the discriminator and hoisting the function name to the top level. Also accept dict variants of the non-function selectors (e.g. {"type": "auto"}). * test(oci): exercise system-message filtering at transform_request boundary adapt_messages_to_cohere_standard now emits SYSTEM-role entries by design so direct callers don't silently drop system content. The Cohere request builder filters system messages before calling the helper and routes them into preambleOverride, so the user-visible 'no SYSTEM in chatHistory' guarantee holds at the transform_request boundary, where the test should live. * fix(oci/chat): extract tool_choice/response_format helpers to satisfy PLR0915 _get_optional_params exceeded ruff's 50-statement cap. The toolChoice and responseFormat normalisation blocks are self-contained mutations, so move them to module-level helpers. * fix(oci): normalize None finishReason in generic non-streaming handler; drop dead Cohere system-role branch Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci/generic): silence mypy assignment error on cleared finish_reason * fix(docker): install libatomic in builder for prisma nodeenv binary The prebuilt node binary that prisma-python's nodeenv downloads links against libatomic.so.1, which Wolfi does not pull in via gcc/nodejs. Without this, fresh Docker builds (no GHA cache hit) fail at `prisma generate` with: node: error while loading shared libraries: libatomic.so.1 * fix(oci): raise on invalid tool_choice instead of silently passing OpenAI shape _normalize_tool_choice previously left an OpenAI-format dict in selected_params['toolChoice'] when the type was unrecognized or when 'FUNCTION' was given with a missing/empty name. OCI would then reject the request with a non-obvious error. Raise ValueError with a clear message in these cases. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci): raise OCIError instead of ValueError in _normalize_tool_choice Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci/generic): declare non-security intent on sha256 for synthetic tool-call id * fix(oci): simplify _get_optional_params and reject invalid tool_choice types - Collapse the two-loop _get_optional_params into a single pass with clear precedence (OpenAI key wins over OCI alias; first OpenAI key reaching a given OCI target wins). Removes the redundant maxTokens special-case in the second loop and makes the map_openai_params / transform_request handoff easier to reason about. - Raise OCIError when _normalize_tool_choice sees an unexpected type (list, bool, int, ...) instead of silently letting it through to the OCI API where it would produce an opaque server-side error. Co-authored-by: Yassin Kortam <yassin@berri.ai> * Remove no-op data['stream'] deletion in OCI stream wrappers Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci): always send Cohere isStream field explicitly Match OCIChatRequestPayload by defaulting CohereChatRequest.isStream to False instead of None so model_dump(exclude_none=True) does not silently omit the field on non-streaming requests. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci): revert Cohere isStream to Optional[bool]=None to preserve omission semantics Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci/generic): raise OCIError on empty choices instead of IndexError Pydantic accepts an empty choices list when validating OCICompletionResponse, so accessing chatResponse.choices[0] could raise an unhandled IndexError. Surface it as OCIError so the response error path is consistent with the existing (TypeError, ValidationError) guard. * fix(oci/cohere): map top_k -> topK so Cohere topK param is settable The Cohere param map (derived from the GENERIC map) had no entry for topK. Since the simplified _get_optional_params only iterates over param_map entries, callers had no way to pass topK to CohereChatRequest (neither via an OpenAI-style key nor via the OCI alias). Add 'top_k': 'topK' to the Cohere map only — OCIChatRequestPayload (GENERIC) has no topK field. _get_optional_params accepts both the OpenAI key (top_k) and the OCI alias (topK) in optional_params, so this covers both calling conventions. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci): tighten cohere stream dedup flags and forward stream args in embed signing Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci/chat): reorder dict guard and wrap stream chunk json.loads - Move isinstance(response_json, dict) check before .get("error") so the guard runs before the attribute access it is supposed to protect. - Wrap json.loads in OCIStreamWrapper.chunk_creator with try/except so malformed SSE payloads surface as OCIError instead of a raw JSONDecodeError propagating out of the stream loop. * fix(oci/cohere stream): only flag text emitted on non-empty content An intermediate Cohere SSE chunk carrying text="" was flipping _cohere_text_emitted via the "is not None" check, which then caused the terminal consolidation chunk to drop its real text as a duplicate. Use a truthy check so only actual content marks the stream as having emitted text. * test(oci): end-to-end proxy integration test against real OCI GenAI Spins up the litellm proxy via the console-script entrypoint with a minimal OCI-only config and drives real OpenAI-shaped HTTP requests through it against OCI GenAI. Covers non-streaming chat, streaming chat, embeddings, and /v1/models for Cohere, Llama, Gemini, and Grok. Skips automatically when ~/.oci/config is absent or when the active profile uses session-token auth (the OCI provider currently only consumes OCI_* env vars; session tokens would need an in-process signer). API-key profiles work out of the box. * test(oci): move proxy integration test to tests/integration/ tests/llm_translation/ is mock-only; the OCI proxy integration test spawns a real proxy subprocess and makes live HTTP calls, so move it (and the companion config) to tests/integration/ alongside the existing test_oci_integration.py. * fix(oci): dedupe finish-reason mapping and batch Cohere tool results - Extract _normalize_oci_finish_reason helper so the four chat handlers (Cohere/GENERIC, sync/stream) share one OCI->OpenAI mapping instead of four near-identical if/elif chains. - Merge consecutive OpenAI tool-role messages into a single CohereToolMessage with multiple toolResults entries, matching the OCI Cohere API's expectation for parallel tool calls in one assistant turn. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci): drop dead Cohere toolChoice field and emit GENERIC tool-call dicts inline - Remove the unreachable toolChoice field from CohereChatRequest. The Cohere param map explicitly marks tool_choice as unsupported, so the field can never be populated through the normal optional_params flow and only confused the public model surface. - Build GENERIC stream tool-call dicts inline (id/type/function shape) instead of round-tripping through ChatCompletionMessageToolCall and model_dump(). Matches handle_cohere_stream_chunk so downstream stream-mergers see the same minimal payload regardless of which vendor produced the chunk. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(docker): drop redundant libatomic from non_root builder litellm_internal_staging already fixes the prisma `nodeenv` build failure at the root cause by restoring `npm` to the builder (#28519): with npm on PATH, prisma-python uses the system Node and never downloads the nodeenv binary that links against libatomic.so.1. After merging internal_staging the libatomic line is dead weight, so remove it. https://claude.ai/code/session_01SwKzxRxgUhLFyyEf4UV812 * fix(oci/catalog): add openai.gpt-5{,-mini,-nano} entries with supports_reasoning Without these catalog entries, supports_reasoning(model='openai.gpt-5*', custom_llm_provider='oci') returned False, so _model_uses_max_completion_tokens fell back to the default and OCI rejected the request with HTTP 400 ('Use maxCompletionTokens instead.'). Add the three entries so the catalog-driven maxCompletionTokens routing works against a stock LiteLLM install. Also reword the test fixture docstring — the bundled backup now actually ships these entries, so the fixture is only a fallback for environments that loaded their cost map from a stale remote source. --------- Co-authored-by: Tai An <antai12232931@outlook.com> Co-authored-by: Vincent <yimao1231@gmail.com> Co-authored-by: Kris Xia <xiajiayi0506@gmail.com> Co-authored-by: d 🔹 <liusway405@gmail.com> Co-authored-by: Fabrizio Cafolla <developer@fabriziocafolla.com> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Tom Denham <tom@tomdee.co.uk> Co-authored-by: escon1004 <70471150+escon1004@users.noreply.github.com> Co-authored-by: Divyansh Singhal <97736786+Divyansh8321@users.noreply.github.com> Co-authored-by: robin-fiddler <robin@fiddler.ai> Co-authored-by: Michael-RZ-Berri <michael@berri.ai> Co-authored-by: Michael Riad Zaky <michaelr@Mac.localdomain> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> Co-authored-by: Krrish Dholakia <krrish+github@berri.ai> Co-authored-by: Sameer Kankute <sameer@berri.ai> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: Federico Kamelhar <federico.kamelhar@oracle.com> Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Michael Riad Zaky <michaelr@Michaels-MacBook-Air.local> Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
…aming hot paths (BerriAI#28289) * perf: reduce per-request and per-chunk overhead across Anthropic streaming hot paths - Introduce pure-text fast-path in `_build_complete_streaming_response` that collapses O(N) `content_block_delta` events into a single equivalent SSE event before conversion, eliminating per-output-token Pydantic `ModelResponseStream` construction; non-text streams (tool_use, thinking, citations) fall back to the unchanged legacy path - Skip agentic streaming wrapper entirely when no callback overrides `async_should_run_agentic_loop`; the wrapper buffered every chunk and rebuilt the SSE response only to call hooks that all return `(False, {})` — a pure no-op for the default config - Serialize request body once (`json.dumps`) for both the pre-call log input and the wire, instead of twice; avoids a full O(payload) scan per request, significant for long-context Claude Code histories - Add fast path in `async_streaming_data_generator` that bypasses the per-chunk `async_post_call_streaming_hook` coroutine await, response-string materialization, and cost-injection call when no callback/guardrail/cost-injection is active (the default config) - Resolve `_DD_STREAMING_TRACE_ENABLED` once at import time; eliminate per-chunk `NullSpan` context manager allocation when Datadog tracing is disabled (the default) - Memoize `get_type_hints(AnthropicMessagesRequestOptionalParams)` with `@lru_cache(maxsize=1)` — resolves once per process instead of once per `/v1/messages` request (~80µs each) - Hoist `cost_injection_active` out of the per-chunk loop in `chunk_processor`; eliminates repeated `getattr` + endpoint-type checks on every streamed byte chunk - Extract `_build_passthrough_logging_result` from `_route_streaming_logging_to_handler` as a standalone static method to facilitate future off-loop dispatch - Convert `async_sse_data_generator` from an `async for: yield` trampoline to a direct return of the underlying generator, removing one async-generator layer per streamed chunk - Skip redundant `strip_empty_text_blocks_from_anthropic_messages` scan in `anthropic_messages_handler` when the async wrapper already sanitized (signalled via `_litellm_messages_presanitized` sentinel, popped before reaching provider params) - Gate debug log `f-string` evaluation behind `isEnabledFor(DEBUG)` in both the streaming generator and the transformation layer to avoid serializing entire message payloads on every request at non-debug log levels - Add benchmark script (`scripts/benchmark_anthropic_messages_perf.py`) with a local mock Anthropic SSE provider for reproducible TTFT and TPM measurement across commits/branches - Add parity tests asserting fast-path and legacy-path produce byte-identical logged/billed payloads, plus unit tests for agentic hook detection, pre-serialized body reuse, and memoized key resolution * perf: address greptile review for anthropic streaming hot path - Bail to legacy in `_collapse_pure_text_chunks` when content_block_delta events from different block indexes are observed without an intervening flush. Anthropic sends blocks strictly sequentially, but defensive bail prevents silent text-merging if the protocol ever interleaves. - Replace leaf-class `__dict__` check for `async_post_call_streaming_hook` in `_callback_capabilities` with a function-identity comparison that walks the MRO. A vendor base class can carry the override and the registered class can add nothing else; before this PR the hook was unconditionally invoked, so an inherited-override miss would silently drop the hook on the streaming path. - Add unit tests for both behaviors. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(mypy): narrow model_name to str in cost-injection branch The hoisted cost_injection_active flag in chunk_processor encodes the `bool(model_name)` requirement but mypy can't track that invariant through the local, so the per-chunk `_process_chunk_with_cost_injection( chunk, model_name)` calls flagged Optional[str] vs str. Pin a typed non-None local inside the cost-injection branch so mypy narrows correctly without changing runtime behavior. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: oss-agent-shin <279349115+oss-agent-shin@users.noreply.github.com> Co-authored-by: ishaan-berri <ishaan-berri@users.noreply.github.com>
… management endpoints (BerriAI#28681) * test(proxy): phase-4 payload behavior pinning for tier-2/3 key + team management endpoints Extends the Phase 1–3 behavior-pin suite at tests/proxy_behavior/management/ with a second axis: payload-shape pinning. Phase 1–3 held payload minimal and pinned (actor, target) → status across 37 routes; Phase 4 holds the caller fixed at an authorized actor, varies the payload shape, and asserts the observable DB effect (on accept) or the named guard / row-unchanged (on reject). Faithfulness contract from Phase 1–3 is unchanged. Six families + one gap-closer (59 new scenarios, 620 → 679 total): * F1 — key budget / rate-limit (test_key_budget_limits.py, 18) * F2 — key↔team reassignment (test_key_team_change.py, 6) * F3 — team budget / rate-limit (test_team_budget_limits.py, 15) * F4 — member-info validation (test_team_member_info_validation.py, 5) * F5 — permission batching (test_team_permissions_bulk_update.py, 6) * F6 — org-scoped team access (+2 detail-string pins in existing files) * F7 — coverage gap-closer (test_f7_coverage_closeout.py, 7) Harness extensions in conftest.py (additive only): * create_scratch_org() seeder with its own scratch-prefixed budget row * budget / limit fields on create_scratch_team() * scratch teardown also sweeps litellm_organizationtable Coverage telemetry (behavior-suite-only): * key_management_endpoints.py 60 % → 65 % (+82 lines) * team_endpoints.py 62 % → 72 % (+137 lines, crosses 70 % stretch) Key lands under 70 % per plan §7 escape hatch — the gap is dominated by routes outside F1–F6 scope (key list/info v2 internals) and structurally dead org-budget guards (call sites at lines 889 + 2310 + 985 + 1751 load the org without include_budget_table=True, so org.litellm_budget_table is None at guard time and the aggregate guard no-ops). Pinned as observed no-op behavior so a future fix that flips the flag turns these into reds. Zero source-code changes; pyproject.toml diff is empty; test_route_coverage.py stays green untouched; G3 grep guards still green; local wall-time 14 s for the full suite (no coverage), 22 s with coverage. G4 regression-replay protocol executed against three representative fix-PR parents (410ce76, 0bd49ec, 8bbc61e): all Phase 4 tests PASS at pre-fix SHAs — confirming the F1–F7 layer is a helper-body pin, not a regression-replay layer for those specific historical bypass shapes. Targeted RED-bait scenarios for each fix are left for a follow-up PR. * test(proxy): push key_management_endpoints.py past the 70% stretch (F7-extension) Adds 24 more payload-pin scenarios in test_f7_key_coverage_push.py following the same accepted-effect / rejected-guard pattern. Each scenario cites the file:line range it pins; same anti-snapshot rules apply. Target ranges (all reachable via HTTP-boundary payload variation): * 5942-6063 /key/health with metadata.logging → test_key_logging body * 4565-4692 /key/reset_spend happy + 404 + non-admin gate + value validation * 4421-4533 /key/regenerate ghost-404 + happy + new_key + grace_period * 4168-4202 _insert_deprecated_key body via grace_period * 6118-6133 _enforce_unique_key_alias duplicate-alias rejection * 6148-6169 validate_model_max_budget malformed-payload rejection * 4708-4789 validate_key_list_check user/team/org/key_hash branches * 2622-2733 /key/bulk_update mixed success/failure + admin gate + size limits * 2797-2950 /team/key/bulk_update all-keys path + explicit-keys dedupe + 404 * 5108-5207 /key/aliases admin + scoped + search-filter branches * 3253-3303 /key/info ghost + explicit-key + no-key-uses-auth-header * 3427-3436 generate_key_helper_fn budget_limits initialization * 1794-1815 prepare_key_update_data duration + budget_duration paths * 5280-5388 _build_filter_conditions across include_created_by_keys/team/sort/alias Coverage telemetry — full PR4 dataset: key_management_endpoints.py: 60 % → 71 % (+11 pts, +194 lines) team_endpoints.py: 62 % → 72 % (+10 pts, +137 lines) Both files now over the plan §7 PR4.M4 70 % stretch as a side effect of pinning real payload behavior. 721 tests pass in 19 s local (full suite, no coverage); 27 s with coverage. Zero source-code changes; pyproject.toml diff still empty; test_route_coverage.py + G3 grep guards still green. Honest finding (kept from the prior commit's body): four structurally-dead org-budget guards remain pinned as observed no-op behavior — they fire only when get_org_object is called with include_budget_table=True, which none of the four management-endpoint call sites currently do. Pinned so a future change that flips the flag turns these into reds. Two helper guards are honest-ceiling: _validate_reset_spend_value's isinstance check at line 4568 is unreachable from HTTP because Pydantic 422s non-float before the helper runs; same shape for /team/key/bulk_update's missing team_id / no-selector pre-handler guards. * test(proxy): address PR review — try/finally cleanup + loosen 500 envelope pins + Optional annotations Greptile review feedback on PR BerriAI#28681: 1. Wrap manual budget-row cleanup in try/finally so an assertion failure doesn't leave non-scratch-prefixed budget rows orphaned across CI re-runs (test_team_new_with_team_member_budget_creates_budget_row and test_team_update_team_member_budget_upserts). 2. Loosen the two 500-status pins to in (400, 422, 500) — the named-guard substring is the real pin; the outer ValueError-wrap envelope is an implementation detail that a future improvement should be free to fix to a proper 400/422 without flipping these tests red. 3. Add missing Optional annotations on _seed_token's max_budget / metadata / team_id keyword args (they default to None). Greptile's typo flag on 'read-world' in the conftest comment is declined — 'read-world' is the project's established term for the immutable seeded world fixture (see other usages in conftest.py and actors.py). 721 tests still pass in 17 s.
…iAI#28372) (BerriAI#28378) * feat(prometheus): emit per-token-type detail metrics (LIT-3220) (BerriAI#28372) Adds five sparse counter metrics that break out the token detail fields providers already report in `usage.prompt_tokens_details` and `usage.completion_tokens_details`: - litellm_input_cached_tokens_metric (provider prompt-cache reads) - litellm_input_cache_creation_tokens_metric (Anthropic prompt-cache writes) - litellm_input_audio_tokens_metric (audio input tokens) - litellm_output_reasoning_tokens_metric (reasoning tokens) - litellm_output_audio_tokens_metric (audio output tokens) These are additive — existing input/output/total counters are unchanged, so no dashboards break. Each new counter is only incremented when the underlying detail is populated and > 0, keeping scrape output sparse for providers that don't report a given field. Data is read from the canonical Usage dict that `get_standard_logging_object_payload` already attaches at `standard_logging_payload["metadata"]["usage_object"]`, so no new plumbing through the logging pipeline is required. Tests: 10 new unit tests covering registration, label-set parity, all-types increment, zero/None/negative skip behaviour, and the no-metadata/no-usage_object no-op paths. Closes LIT-3220 Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Claude <noreply@anthropic.com> * chore: remove proof folder image --------- Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai> Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
…rriAI#28405) * fix(otel): stamp http.response.status_code on all error responses httpx.HTTPStatusError exposes status under .response.status_code, not as a top-level attr, so unified-endpoint 5xx failures left the SERVER span without a status. The admin hooks only wrote a child span and never stamped or ended the parent at all, so admin 4xx/5xx (and success) responses were invisible to dashboards. Adds a fallback to .response.status_code in get_error_information, and ends the parent SERVER span in async_management_endpoint_{success,failure}_hook with the same _record_exception_on_span helper the unified path uses. Resolves LIT-3193 * test(otel): exercise httpx.HTTPStatusError through admin path Pins the contract that get_error_information's response.status_code fallback is reachable from any entry point — without this, a future refactor that bypasses _record_exception_on_span in the admin hooks could regress for httpx-wrapped exceptions while the unified suite still passes. * chore(otel): trim verbose comments in LIT-3193 changes Tighten docstrings and remove redundant section dividers/inline narration. Behavior is unchanged. * fix(otel): set span.status on management hook parent SERVER span Mirror the unified failure path: stamp StatusCode.ERROR on the parent SERVER span before recording the exception, and StatusCode.OK before ending it on success. Without this, OTEL backends filtering on span status (the idiomatic primitive) miss admin-endpoint failures even though the http.response.status_code attribute is correct. Extend assert_server_span_attrs to assert span.status.status_code matches the expected outcome so the gap can't regress. * fix(otel): close SERVER span on body-validation and unhandled errors Stash the SERVER span on request.state in auth so FastAPI exception handlers can finish it for failures that occur after auth but before the route handler (e.g. /model/new TypeError, /key/generate RequestValidationError). Without this, those requests left dangling spans missing http.response.status_code. Resolves LIT-3193 * fix(otel): generic 500 body, log exception details server-side Don't leak str(exc) and type(exc).__name__ to clients on uncaught exceptions. The full traceback is logged via verbose_proxy_logger and the SERVER span still gets http.response.status_code=500. Resolves LIT-3193 * fix(otel): stamp http.response.status_code on every SERVER span path Closes three remaining gaps where the proxy SERVER span ended without the http.response.status_code attribute: 1. ProxyException raised from _read_request_body (e.g. invalid JSON body) bubbled out of user_api_key_auth before the SERVER span was created, so the FastAPI handler had nothing to close and the trace never reached the backend. Hoist the span creation to a new idempotent _ensure_parent_otel_span_on_request_state helper called at the top of user_api_key_auth; wire openai_exception_handler to close the dangling span. Covers /v1/chat/completions, /v1/messages, /v1/responses (shared handler). 2. /v1/responses success — _handle_success ends the proxy span before async_post_call_success_hook fires on this path, so the hook's set_response_status_code_attribute(200) silently no-op'd against an ended span. Stamp 200 + set OK status at the close site in _handle_success / _end_proxy_span_from_kwargs via a shared _close_proxy_span_ok helper, so the attribute lands regardless of which success hook runs first. 3. Failure path for exceptions without code/status_code (e.g. a bare TypeError surfacing through _handle_llm_api_exception) — empty error_information.error_code → _record_exception_on_span skips the stamp → the hook ends the span. Default to 500 in async_post_call_failure_hook so the attribute is always set. Resolves LIT-3193
* fix(helm): drop main- prefix from default image tag
The default image tag in the deployment + migrations-job templates was
`main-{{ .Chart.AppVersion }}`. The current release pipeline publishes
content tags without the `main-` prefix (e.g. `v1.85.1` / `1.85.1`,
`v1.86.0-rc.1` / `1.86.0-rc.1`), so the rendered ref points at a tag
that does not exist on GHCR or DockerHub and installs fail with
ImagePullBackOff.
- templates/deployment.yaml, templates/migrations-job.yaml: render
`.Chart.AppVersion` directly instead of `main-<AppVersion>`.
- Chart.yaml: bump stale `appVersion: v1.80.12` (not on either
registry) to `v1.85.1` so local-checkout installs also resolve.
- values.yaml: update the commented tag-override hint to match.
* fix(helm): use :latest in tag override example, not pinned version
Per review: ghcr.io/berriai/litellm-database:latest is a floating
alias for the most recent stable (same digest as :main-stable),
maintained by the release pipeline's UPDATE_LATEST advance step.
Better example than a pinned version that goes stale.
…AI#28708) The schema in test_aaamodel_prices_and_context_window_json_is_valid uses additionalProperties: false. The azure/speech/azure-stt entry added in BerriAI#27482 introduced an audio_transcription_config field that the schema did not whitelist, so the test fails on every branch built on top of staging. Add the field as a string property.
chore(ci): promote internal staging to main
…rriAI#28683) * fix(team): refresh team cache on team_model_add/delete (LIT-3244) team_model_add and team_model_delete wrote to the DB but did not invalidate the in-memory LiteLLM_TeamTableCachedObj used by common_checks. After the v1.83.14 common_checks centralization made team.models authoritative on /v1/files and /v1/vector_stores/*, adding a Team-BYOK model silently failed to grant the new public model name to team members until the cache TTL expired (and a removed model kept working until then on the symmetric path). Extract the cache-refresh snippet from update_team into a small helper and apply it consistently at all three team-write sites. * test: also assert updated models in team-cache-refresh pin Strengthens the LIT-3244 regression test to also assert `call_kwargs["team_table"].models` matches the updated row, not just `team_id`. Both `existing_team` and `updated_team` share `team_id` in the test setup, so the previous assertion would have passed even if the implementation accidentally cached the pre-mutation row. Greptile review feedback. * fix(team): hydrate object_permission on cache-refreshing team updates The Prisma update calls in update_team, team_model_add, and team_model_delete returned a team row with object_permission_id set but object_permission=None (the relation was not requested via include=). _refresh_cached_team then wrote that to the in-memory LiteLLM_TeamTableCachedObj, and the cache-hit path in get_team_object returns the cached object without re-hydrating. Downstream consumers (validate_key_search_tools_against_team, the MCP/agent authz paths) treat a missing object_permission as no team-level restriction, so a team-write op silently dropped object-permission enforcement until the cache TTL expired or a DB-fetch path re-hydrated it. Add include={"object_permission": True} to all three updates so the refresh writes a complete cached team. Extend the LIT-3244 regression test to pin both the cached object_permission and the include shape on the Prisma call. Surfaced in PR review of LIT-3244.
… Anthropic (BerriAI#28723) `getProviderModels()` matched a model into a provider's dropdown when the model's `litellm_provider` string *contained* the provider key as a substring. The intent was to admit suffix variants (e.g. `anthropic_text`, `bedrock_converse`), but the substring check is too loose: it also pulls in unrelated providers whose name happens to contain the key, most visibly `vertex_ai-anthropic_models` matching `anthropic` and `vertex_ai-openai_models` matching `openai`. Replace `.includes()` with separator-anchored prefix matching (`startsWith(provider + "_")` / `startsWith(provider + "-")`). All legitimate variants in `model_prices_and_context_window.json` still match (`anthropic_text`, `azure_text`, `azure_ai`, `bedrock_converse`, `bedrock_mantle`, `cohere_chat`, `fireworks_ai-embedding-models`, `vertex_ai-*`, `vertex_ai_beta`), and the cross-provider leak is closed. Tests: update one assertion that pinned the buggy substring behavior (`custom_openai_endpoint` matching `openai` — not a real provider value); add 6 new tests covering the leak regressions and the variant-preservation contract for vertex_ai/bedrock/fireworks.
Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: ryan-crabbe-berri <ryan-crabbe-berri@users.noreply.github.com>
…rs and signed request body (BerriAI#27526) * Fix Bedrock KB pass-through SigV4 headers and signed body Coerce botocore HeadersDict to a dict for pass-through routes. When forward_headers is true, drop request headers that collide case-insensitively with signed headers so client Bearer auth does not shadow AWS SigV4. Send prepped.body as raw content so the outbound payload matches the signature after logging hooks mutate the parsed dict. Co-authored-by: Cursor <cursoragent@cursor.com> * Simplify pass-through raw body handling Read the SigV4-signed bytes directly from request.state inside pass_through_request instead of threading a custom_raw_body argument through three functions. Helper methods are restored to their original signatures, and the new branch lives in one place at each httpx call site. Co-authored-by: Cursor <cursoragent@cursor.com> * Harden pass-through raw body read from request.state Guard missing request.state (test fixtures) and ignore non-bytes/str values so MagicMock does not trigger the SigV4 raw-body path. Co-authored-by: Cursor <cursoragent@cursor.com> * Test pass_through_request state_raw_body uses httpx content= Cover non-streaming (async_client.request) and streaming (build_request) paths so SigV4 bytes on request.state are not replaced by json= of a hook-mutated dict. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…#28728) * chore(tests): migrate Bedrock CI from AWS account 888602223428 to 941277531214 The original account (888602223428) was put under a security restriction by AWS after a root access key leaked in a PR comment. While that account works its way through the AWS Support unlock process, Bedrock-touching CI tests have been migrated to a fresh account (941277531214). Changes: - Replace 26 hardcoded references to 888602223428 with 941277531214 across 8 files (provisioned-model ARNs, imported-model ARNs, AgentCore runtime ARNs, batch execution role ARN, and example proxy config). - The provisioned-model and imported-model ARNs are referenced only from mocked unit tests — no AWS resources to recreate. - The batch execution IAM role has been recreated in the new account with the same name and equivalent permissions. - The two AgentCore runtimes (hosted_agent_r9jvp-3ySZuRHjLC, hosted_agent_13sf6-cALnp38iZD) are being recreated in the new account under the same names — see tools/agentcore-deploy/ in a follow-up. CircleCI env vars AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_REGION_NAME were updated separately via the CircleCI API to point at the new account. Smoke-tested locally against the new account: aws bedrock-runtime converse --region us-west-2 \ --model-id us.anthropic.claude-sonnet-4-5-20250929-v1:0 \ --messages '[{"role":"user","content":[{"text":"ping"}]}]' → 200, model returned 'pong' Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(tests): refresh AgentCore ARN suffixes to match newly-deployed runtimes The first migration commit replaced just the account ID, but AgentCore auto-assigns a random 10-char suffix to every runtime on creation — we can't reuse the original suffixes (`3ySZuRHjLC`, `cALnp38iZD`) in the new account. Updated the AgentCore-runtime ARNs in the three files that reference real runtime IDs (not the mock-based unit-test ARNs). Deployed runtimes: arn:aws:bedrock-agentcore:us-west-2:941277531214:runtime/hosted_agent_r9jvp-Rq79QFC2fp arn:aws:bedrock-agentcore:us-west-2:941277531214:runtime/hosted_agent_13sf6-4046UzHSwy Both runtimes are status=READY and pass a smoke invoke: $ aws bedrock-agentcore invoke-agent-runtime --agent-runtime-arn ... --payload '{"prompt":"ping"}' → 200, {"result": "echo: ping"} The agent is a minimal echo (see /tmp/agentcore_deploy/agent.py for the deploy artifacts). Tests that only verify the SDK wiring will pass; if any test asserts on agent output content, swap the echo for the real agent. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(tests): point Bedrock batch tests at new-account S3 bucket The account migration (888602223428 -> 941277531214) was a flat account-ID swap, which only rewrites ARNs that embed the account number. S3 bucket names carry no account ID, so the live Bedrock batch tests still uploaded to `litellm-proxy` — a bucket that lives in the old account. S3 names are globally unique, and the old account still holds that name, so it can't be recreated in the new account. Rename to `litellm-proxy-941277531214` (account-ID suffix guarantees global uniqueness). The bucket must be created in 941277531214 and the batch execution role granted s3:GetObject/PutObject/ListBucket on it before this job is run in CI. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(tests): point live S3 logging test at new-account bucket Same account-ID-free blind spot as the batch bucket: `load-testing-oct` lives in the old account and its name can't be reused globally. The `logging_testing` CI job is wired into the workflow and runs test_basic_s3_logging, which uploads to this bucket with the CI env creds, then lists and deletes objects — a live dependency. Rename to `load-testing-oct-941277531214`. The bucket must exist in the new account with the CI IAM principal granted s3:PutObject/GetObject/ListBucket/DeleteObject before this job runs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(tests): repoint Bedrock guardrail IDs to new-account guardrails The migration left guardrail IDs untouched (no account ID in them), so all live guardrail tests failed with "guardrail identifier or version does not exist" against 941277531214. Recreated both guardrails in the new account and updated the hardcoded IDs: - wf0hkdb5x07f -> zgkmukebruil (PII mask: PHONE + CREDIT_DEBIT_CARD, with explicit inputAction=ANONYMIZE so masking applies to INPUT, which is the source litellm's moderation hook sends) - ff6ujrregl1q -> 4w3d1di3snt5 (blocks "coffee"; blocked message set to the exact string the tests assert on) Updated test_bedrock_guardrails.py, otel_test_config.yaml, and the guardrailConfig in test_bedrock_completion.py. Verified locally: the 5 previously-failing guardrail tests now pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(bedrock): migrate legacy models to current inference profiles The new CI account (941277531214) cannot invoke legacy Bedrock models (AWS gates them: "marked by provider as Legacy... not actively using in the last 30 days"). Migrated the live-call tests: - anthropic.claude-3-sonnet-20240229 -> us.anthropic.claude-sonnet-4-5-20250929-v1:0 - anthropic.claude-3-haiku-20240307 -> us.anthropic.claude-haiku-4-5-20251001-v1:0 Current Claude models on Bedrock require the us. inference-profile prefix (bare on-demand ids are rejected). cohere.command-r-plus has no working replacement (all Cohere is legacy- gated in the new account): swapped to claude-haiku-4-5 in provider- agnostic param lists. amazon.titan-image-generator skipped (no working replacement). Mocked/transformation/cost tests that reference the legacy strings are intentionally left unchanged. Verified live against the new account. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(bedrock): repoint SageMaker + Knowledge Base to new-account resources These referenced account-scoped resources by hardcoded id that only existed in the old account, so the migration's account-ID swap missed them. Recreated in 941277531214 and repointed: - SageMaker endpoint jumpstart-dft-hf-textgeneration1-mp-20240815-185614 -> litellm-ci-textgen (gpt2 on a TGI container, ml.g5.xlarge) - Bedrock Knowledge Base T37J8R4WTM -> LCYXFBR2TU (OpenSearch Serverless vector store + titan-embed-text-v2, seeded with a LiteLLM doc) Verified live: test_sagemaker.py (12 passed) and test_bedrock_knowledgebase_hook.py (12 passed). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(reasoning_effort_grid): skip bedrock claude-opus-4-7 cells (not entitled on 941277531214) claude-opus-4-7 is listed in the new Bedrock CI account's foundation models but invoke is denied (AccessDeniedException: "not available for this account"). Bedrock access to the flagship Opus requires an AWS Sales request, not the self-serve model-access toggle, so it can't be enabled inline with the rest of the account migration. Add an optional `skip_reason` to ModelEntry and set it on the bedrock-claude-opus-4-7 entry; the grid test honors it via pytest.skip. Cell count (231) and route coverage are unchanged, so the structural asserts still pass. Restore coverage by deleting the one skip_reason line once access is granted. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(bedrock): swap/skip legacy-gated models unavailable on new CI account The migrated AWS account (941277531214) cannot access several models that the old account could, so the remaining red CI jobs were hitting real Bedrock "Access denied / Legacy" and "account not authorized" errors: - image_gen: skip both Nova Canvas test classes (amazon.nova-canvas-v1:0 is legacy-gated), matching the existing titan skip. - batches: skip test_async_file_and_batch (Bedrock batch inference is not authorized on the new account; requires an AWS support case). - litellm_overhead: swap legacy claude-3-5-haiku for the active us.anthropic.claude-haiku-4-5 inference profile. - test_completion_claude_3_function_call: swap legacy claude-3-sonnet for the active us.anthropic.claude-sonnet-4-5 inference profile. https://claude.ai/code/session_01Y7zgHYu9GX29YRwV4yiWAa * test(bedrock): fix remaining e2e legacy-model + batch failures on new CI account - e2e_openai_endpoints: skip test_bedrock_batches_api (Bedrock batch inference is not authorized on account 941277531214) and migrate the missed s3_bucket_name in oai_misc_config.yaml to litellm-proxy-941277531214. - build_and_test: swap legacy bedrock claude-3-sonnet for the active us.anthropic.claude-sonnet-4-5 inference profile in the proxy structured output e2e test. https://claude.ai/code/session_01Y7zgHYu9GX29YRwV4yiWAa * test(bedrock): make opus-4-7 + batch cells fail loudly and mock image-gen (BerriAI#28791) Replace the silent skips added for the new CI account with noisier behavior: - reasoning-effort grid: opus-4-7 cells now fail (when AWS creds are present) instead of skipping, so the missing entitlement stays visible in CI; they still skip when AWS creds are absent (local dev) - Bedrock batch inference tests: drop the skip so they run and fail until batch access is granted - Titan + Nova Canvas image-gen tests: mock the Bedrock HTTP call so the transform + cost-tracking path stays under test without live model access https://claude.ai/code/session_01MT7SWDnXUjv6e6EPG7BDjT Co-authored-by: Claude <noreply@anthropic.com> * test(bedrock): use pytest.xfail for known-failing opus-4-7 cells Replace pytest.fail with pytest.xfail when a model has a fail_reason, so known-broken cells stay visible as XFAIL without keeping CI red. Co-authored-by: Yassin Kortam <yassin@berri.ai> --------- Co-authored-by: Mateo <mateo@Mateos-MacBook-Pro.local> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai>
…http_request (BerriAI#28794) Co-authored-by: Yassin Kortam <yassinkortam@Yassins-MacBook-Pro.local>
* chore(proxy): route path-dependent call sites through get_request_route Replace direct ``request.url.path`` reads in auth, ACL, routing, and audit-log decisions with ``get_request_route(request)`` — the helper already added in ``auth/auth_utils.py`` that returns the ASGI ``scope["path"]`` with ``root_path`` stripped. Starlette reconstructs ``url.path`` from the Host header; ``scope["path"]`` is uvicorn's parse of the request line and matches what FastAPI dispatches on, so it's the authoritative route for any decision that should agree with the actual handler. Sites: - _experimental/mcp_server/auth/user_api_key_auth_mcp.py - management_endpoints/mcp_management_endpoints.py - vector_store_endpoints/utils.py - pass_through_endpoints/pass_through_endpoints.py - auth/route_checks.py - litellm_pre_call_utils.py - spend_tracking/spend_management_endpoints.py - common_utils/http_parsing_utils.py - management_helpers/utils.py - health_endpoints/_health_endpoints.py Adds regression tests in tests/proxy_unit_tests/test_proxy_routes.py that construct a Request with scope["path"] set to a benign route and the Host header crafted so url.path would resolve differently; each site's decision is asserted against scope["path"]. * chore(proxy): make get_request_route imports lazy at call sites Move the ``from litellm.proxy.auth.auth_utils import get_request_route`` imports added in the prior commit back to the function bodies that use them. The module-level form participates in a long-standing import cycle through ``auth_utils -> _types -> ...`` and was flagged by CodeQL on the PR; the lazy form matches the pattern the proxy already uses for ``user_api_key_auth`` and related helpers elsewhere in these files. Also drop the ``RouteChecks._is_assistants_api_request`` delegation in ``_get_metadata_variable_name`` introduced in the prior commit — the delegation pulled ``RouteChecks`` into the same cycle, and the call site reuses the resolved route for its other branches, so inlining the substring check is both cycle-free and avoids a redundant second ``get_request_route`` call. Comment in test_proxy_routes.py acknowledges that the two MCP table entries exercise ``get_request_route`` directly rather than the full production handler (which needs ASGI scope + MCP state to invoke). --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: user <70670632+stuxf@users.noreply.github.com>
* feat(dashboard): navbar hierarchy + Agent Platform notifications (BerriAI#27543) * feat(dashboard): refine navbar zones and Agent Platform notice Restructure the admin navbar for production users: clear product vs community vs personal columns with vertical dividers, icon-only Slack/GitHub in a shared chip, and Docs/Blog typography aligned on an 8px rhythm. Add a notifications bell with popover linking to the LiteLLM Agent Platform repo and optional mark-as-read persistence. Promote the account control with initials avatar, single-line display name, and navDisplayName mapping for placeholder user ids (e.g. default_user_id). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(dashboard): address PR review — AntD buttons, public page guard, dedupe regex - Replace raw <button> with AntD Button in BlogDropdown, NotificationsBell, UserDropdown, and test mock - Guard NotificationsBell + container behind !isPublicPage to avoid rendering on public pages - Remove redundant equality checks in navDisplayName (regex already covers them) - Remove unused `lower` variable after simplification Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * fix(dashboard): drop dead useHealthReadiness import in navbar The module was removed in BerriAI#27896 (replaced by useHealthReadinessDetails), but the import survived the rebase. The symbol is unused — only useHealthReadinessDetails is consumed in the file. Removing the dead import unblocks the UI TypeScript build. * fix(dashboard): align CommunityEngagementButtons test with icon-only aria-labels The component was refactored to an icon-only chip with aria-label='LiteLLM on GitHub' (squash BerriAI#27543), but the test still asserted /star us on github/i. Update the query to match the rendered accessible name. * refactor(dashboard): drop unused props from NavbarProps The navbar refactor moved user identity + dark-mode state to internal hooks (useAuthorized, useWorker), but the NavbarProps interface still declared userID, userEmail, userRole, premiumUser, isDarkMode, and toggleDarkMode as required, forcing every caller to thread them through. Drop them from the interface and all four call sites (page.tsx, (dashboard)/layout.tsx, public_model_hub.tsx, navbar.test.tsx). Also shrinks the destructure in layout.tsx so the now-unused locals stop being pulled out of useAuthorized(). * refactor(dashboard): use useSyncExternalStore for NotificationsBell dismiss flag Reads/writes of the litellmHideAgentPlatformBanner key were done directly inside NotificationsBell via a useEffect + useState pair. Every other localStorage-backed flag in the dashboard (Disable ShowPrompts, DisableBouncingIcon, DisableShowNewBadge, DisableUsageIndicator, DisableBlogPosts) is wrapped in a useSyncExternalStore hook over localStorageUtils so all mounted components stay in sync. Extract useHideAgentPlatformBanner to follow the same shape, swap NotificationsBell to consume it, and add a regression test that two sibling bells stay in sync without a remount when one is dismissed. * refactor: mask credential fields in proxy settings GET responses (BerriAI#28682) * refactor: mask credential fields in proxy settings GET responses Brings SSO settings, cache settings, and the email/Slack alerting view in /get/config/callbacks in line with the HashiCorp Vault config-override pattern, so persisted credentials are not transported back to the UI in plaintext. * refactor: harden short-value masking and hoist alerting var constant Closes two review observations: - mask_sensitive_keys now replaces short values (below the visible prefix+suffix length) with an all-mask string instead of returning them unchanged, so a 1-7 character credential is no longer round-tripped verbatim. - _ALERTING_SENSITIVE_VARS is moved out of get_config() to a module-level constant, matching the analogous _SSO_SENSITIVE_FIELDS and _CACHE_SENSITIVE_FIELDS in the SSO and cache endpoint files. --------- Co-authored-by: Krrish Dholakia <krrish+github@berri.ai> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…riAI#28809) The Key Info Overview tab's Spend card truncated sub-dollar budgets to "$0" because formatNumberWithCommas defaults to 0 decimals. The Settings tab passes 2; align the overview so a $0.10 budget renders as "$0.10". Resolves LIT-2845
…erriAI#28442) * feat(proxy): allow llm_api_routes virtual keys to list MCP servers Add a new `mcp_discovery_routes` group (GET /v1/mcp/server and GET /v1/mcp/server/{server_id}) and include it in `llm_api_routes` so that virtual keys configured with `allowed_routes=["llm_api_routes"]` can discover the MCP servers they have access to. Previously these calls failed with 'Virtual key is not allowed to call this route. Only allowed to call routes: [llm_api_routes]'. The GET handlers already sanitize the response for restricted virtual keys via `_sanitize_mcp_server_list_for_virtual_key`, stripping credential-bearing fields (url, headers, env). Write methods (POST/PUT/DELETE) on the same paths remain gated by the existing handler-level admin role checks. The new discovery list is intentionally kept OUT of `mcp_inference_routes`, so `is_llm_api_route()` still returns False for these paths — this preserves the existing contract that DISABLE_LLM_API_ENDPOINTS must not block the Admin UI from listing MCP servers. Co-authored-by: ryan-crabbe-berri <ryan-crabbe-berri@users.noreply.github.com> * refactor(proxy): make MCP discovery carve-out method-aware Replace the `mcp_discovery_routes` group in `llm_api_routes` with a method-aware special case inside `is_virtual_key_allowed_to_call_route`. Virtual keys with allowed_routes=["llm_api_routes"] are now permitted to call only GET /v1/mcp/server and GET /v1/mcp/server/{server_id} — non-GET methods and multi-segment admin sub-paths fall through to the existing 403. This keeps the general llm_api_routes list free of management paths and avoids accidentally exposing POST/PUT/DELETE writes through the route-check layer. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: ryan-crabbe-berri <ryan-crabbe-berri@users.noreply.github.com>
* chore(proxy): route path-dependent call sites through get_request_route Replace direct ``request.url.path`` reads in auth, ACL, routing, and audit-log decisions with ``get_request_route(request)`` — the helper already added in ``auth/auth_utils.py`` that returns the ASGI ``scope["path"]`` with ``root_path`` stripped. Starlette reconstructs ``url.path`` from the Host header; ``scope["path"]`` is uvicorn's parse of the request line and matches what FastAPI dispatches on, so it's the authoritative route for any decision that should agree with the actual handler. Sites: - _experimental/mcp_server/auth/user_api_key_auth_mcp.py - management_endpoints/mcp_management_endpoints.py - vector_store_endpoints/utils.py - pass_through_endpoints/pass_through_endpoints.py - auth/route_checks.py - litellm_pre_call_utils.py - spend_tracking/spend_management_endpoints.py - common_utils/http_parsing_utils.py - management_helpers/utils.py - health_endpoints/_health_endpoints.py Adds regression tests in tests/proxy_unit_tests/test_proxy_routes.py that construct a Request with scope["path"] set to a benign route and the Host header crafted so url.path would resolve differently; each site's decision is asserted against scope["path"]. * chore(proxy): make get_request_route imports lazy at call sites Move the ``from litellm.proxy.auth.auth_utils import get_request_route`` imports added in the prior commit back to the function bodies that use them. The module-level form participates in a long-standing import cycle through ``auth_utils -> _types -> ...`` and was flagged by CodeQL on the PR; the lazy form matches the pattern the proxy already uses for ``user_api_key_auth`` and related helpers elsewhere in these files. Also drop the ``RouteChecks._is_assistants_api_request`` delegation in ``_get_metadata_variable_name`` introduced in the prior commit — the delegation pulled ``RouteChecks`` into the same cycle, and the call site reuses the resolved route for its other branches, so inlining the substring check is both cycle-free and avoids a redundant second ``get_request_route`` call. Comment in test_proxy_routes.py acknowledges that the two MCP table entries exercise ``get_request_route`` directly rather than the full production handler (which needs ASGI scope + MCP state to invoke). --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: user <70670632+stuxf@users.noreply.github.com>
…BerriAI#28737) * fix(team): keep team_alias cache in sync on _cache_team_object writes _cache_team_object wrote only to the team_id:<id> cache key, but the JWT auth path that uses team_alias_jwt_field reads from a separate team_alias:<alias> key (get_team_object_by_alias caches under both keys on miss, but reads only the alias-keyed one). After any team-mutation endpoint (team_model_add, team_model_delete, update_team, the two access-group writes) the team_id cache was refreshed but the team_alias cache stayed stale until TTL — JWT callers using team_alias_jwt_field kept seeing the pre-mutation team for the full cache window. Mirror the write under the alias key inside _cache_team_object so every existing caller stays in sync without further changes. Skip the alias write when team_alias is None/empty so we don't collide across alias-less teams. Surfaced testing the LIT-3244 cherry-pick on patch/1.86.0: the LIT-3244 fix correctly invalidated the team_id cache but the customer's JWT used team_alias_jwt_field, so they kept hitting the stale alias-keyed entry. * fix(team): delete (not overwrite) team_alias cache on _cache_team_object The prior shape of this PR wrote both team_id:<id> AND team_alias:<alias> from _cache_team_object. team_alias is NOT unique in the schema (no @unique on LiteLLM_TeamTable.team_alias), and get_team_object_by_alias enforces uniqueness on its own DB-fetch path (len(teams) > 1 raises). Writing the alias-keyed cache from the generic refresh path bypassed that check: a team admin renaming their team to collide with another team's alias could silently overwrite the cached team for JWT-by-alias auth, swapping the resolved team under that alias for the cache window. Switch the alias-keyed operation from a write to a delete (mirroring the dual-cache delete pattern in _delete_cache_key_object). After every team write, the next JWT-by-alias reader cache-misses and falls through to get_team_object_by_alias, which (a) re-fetches the fresh team from DB, closing the LIT-3244 staleness gap that motivated this PR, and (b) enforces alias uniqueness before populating either cache key. team_id:<id> writes are unchanged — team_id is the table PK and is guaranteed unique. Surfaced in veria-ai review on BerriAI#28739. * fix(managed-files): anchor model_id regex so it doesn't match llm_output_file_model_id extract_model_id_from_unified_id used `re.search(r"model_id,([^;]+)", ...)` which substring-matches the `model_id,` inside the file-ID encoding's `llm_output_file_model_id,<deployment_uuid>` field. parse_unified_id then fed that deployment UUID back into the auth path as a model candidate via _extract_models_from_managed_resource_id, and every team-BYOK file attach 403'd with: team not allowed to access model. This team can only access models=['openai/*']. Tried to access <deployment-uuid> The team's models list correctly contains the public name (`openai/*`) that target_model_names matches, but the bogus UUID candidate fails the wildcard check first. Anchor the regex to a field boundary (`(?:^|;)model_id,`) so it matches the legitimate top-level `model_id,<value>` field on vector_store unified IDs and skips substring matches inside other fields. File-IDs (which have no top-level `model_id` field) now return None and contribute no spurious UUID candidate. Surfaced reproducing LIT-3244 on patch/1.86.0 with the customer's exact flow: team with openai/* BYOK deployment, JWT-scoped user, POST /v1/vector_stores/{id}/files attaching a file uploaded with target_model_names=openai/gpt-4o.
…erriAI#28822) * fix(proxy): hydrate wildcard discovery credentials * fix(proxy): constrain wildcard credential hydration Co-authored-by: Dibyo Mukherjee <dibyo@adobe.com>
Creates litellm_oss_agent_shin_MM_DD_YYYY from main every day at 00:00 UTC. Lets us retarget oss-agent-shin fork PRs onto a canonical branch so CircleCI runs with secrets, without granting the agent write access. Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
…I#28827) * test(proxy): add harness for proxy_server.py behavior-pinning Creates tests/test_litellm/proxy/proxy_server/ with: - conftest.py: 11 shared fixtures (app, client, mock_prisma, auth_as, mock_router with parametrized response builders, normalize, etc.) - _coverage_check.py: per-PR coverage gate (line + branch) against a baseline, self-selects target by inspecting which placeholder files have been filled - _pin_check.py: AST-based gate that verifies every pin-list item has >=1 happy + >=1 error test with a real assertion (no status-only) - test_harness_smoke.py: 19 smoke tests covering every fixture + both scripts end-to-end - 26 placeholder test files (one docstring each) reserved for follow-up PRs per the directory ownership in the Notion plan - .coverage_baseline pinned at 0% so future PRs measure deltas against new-tests-only and aren't entangled with the broader scattered test suite Adds a dedicated proxy-server job to test-unit-proxy-endpoints.yml so this directory's runtime + coverage are tracked independently. Plan: https://www.notion.so/36c43b8acdab81ee845fd5365128a2fc * ci(proxy-endpoints): allow workflow_dispatch Lets the workflow be triggered manually on a branch via `gh workflow run`, which is needed for the verify-first flow on workflow changes before opening a PR. * test(proxy): address review feedback on proxy_server harness - conftest.py: anchor sys.path insert to __file__ (Path(__file__).resolve().parents[4]) instead of CWD-relative os.path.abspath("../../../../") which resolved to the wrong directory when pytest is launched from the repo root. - _coverage_check.py: actually read .coverage_baseline and use it as the floor (line_min = max(target, baseline)). Closes the gap between the PR description's "delta semantics" and what the script was doing. With baseline=0.0 today this is a no-op; future PRs that update the baseline cause regressions (test deletions etc.) to trip the gate even if the static PR target is still met. - _pin_check.py: drop unreachable startswith("_") guard (test_*.py glob never yields underscore-prefixed names) and read each test file once instead of twice.
…sidency (BerriAI#28626) * feat(openai): apply regional-processing cost uplift for EU/US data residency OpenAI charges a 10% uplift on the latest GPT models when requests are served from a regionalized hostname (eu./us.api.openai.com). Infer the region from `api_base`, expose it on `kwargs["litellm_params"]["data_residency"]`, and multiply the computed cost by a per-model `regional_processing_uplift_multiplier_<region>` field. https://claude.ai/code/session_012ebH44s7ohYxjoix5CXzTW * test: allow regional_processing_uplift_multiplier_{eu,us} in model_prices schema * fix(cost): tighten data_residency inference and restore model_cost in tests - Only infer OpenAI data_residency when custom_llm_provider == "openai"; drop the implicit None fallback so non-OpenAI callers can't accidentally pick up a regional tag from a stray OpenAI hostname. - _local_model_cost_map fixture now snapshots and restores litellm.model_cost and LITELLM_LOCAL_MODEL_COST_MAP so tests don't leak state across the session. * refactor(openai): move data_residency helper under llms/openai * fix: thread data_residency through realtime stream cost calculation Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(cost): thread data_residency through batch_cost_calculator Apply the OpenAI regional-processing uplift multiplier to retrieve_batch cost paths so Batch API requests served via eu./us.api.openai.com are priced at the same uplifted token rates as completions/transcriptions. * refactor(openai): encapsulate provider check inside infer_openai_data_residency Move the custom_llm_provider == "openai" guard from get_litellm_params into the helper itself so the core utility no longer carries provider-specific dispatch logic. Callers pass through the provider unconditionally; the helper returns None for any non-OpenAI provider. * fix(responses): thread data_residency through Responses logging params The Responses API paths build their logging litellm_params dict after provider resolution but did not include data_residency, so cost calc saw None even when the effective api_base was a regional OpenAI host. --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai>
…erriAI#28112) * chore(admin-ui): regenerate static export with trailingSlash: true Rebuilds litellm/proxy/_experimental/out/ from ui/litellm-dashboard with `trailingSlash: true` enabled in next.config.mjs. Next.js now emits every route as <dir>/index.html (e.g. mcp/oauth/callback/index.html) instead of <dir>.html with a sibling metadata-only directory, which fixes the 404 on extensionless URLs served through FastAPI's StaticFiles(html=True) mount. This is the build artifact half of the fix; the config change, Dockerfile cleanup, and regression test live in the follow-up source PR that stacks on top of this branch. * fix(admin-ui): emit nested routes as <dir>/index.html (BerriAI#28106) Linear and other OAuth providers redirect the user back to /ui/mcp/oauth/callback?code=...&state=... after the consent step. The packaged Next.js static export only produced /ui/mcp/oauth/callback.html, so FastAPI's StaticFiles served a 404 on the extensionless URL and the OAuth handshake never completed. The Dockerfile.non_root build step tried to paper over this at image-build time with `for html_file in *.html; do ...`, but that shell glob does not recurse, so nested routes like mcp/oauth/callback.html were left stranded next to an empty mcp/oauth/callback/ directory containing only Next.js metadata. The runtime restructure step in proxy_server.py was then skipped because the .litellm_ui_ready marker had already been dropped. Set trailingSlash: true in the dashboard's Next.js config so the export emits every nested route as <dir>/index.html natively. The Dockerfile loop is now a no-op for the bundled UI and has been removed; the .litellm_ui_ready marker is still written so the proxy keeps skipping the redundant Python restructure step at startup. Stacks on top of the static export regeneration in the parent branch. * chore: restore origin/litellm_internal_staging out files
…AI#28627) * fix(azure): preserve AD token refresh in v1 OpenAI client path The /openai/v1/ code path (api_version in {"v1", "latest", "preview"}) constructs a plain OpenAI/AsyncOpenAI client, but only forwarded `api_key` from `azure_client_params`. When `enable_azure_ad_token_refresh` is set (or any AD-only auth), `api_key` is None and the client constructor raised "The api_key client option must be set...", breaking every Azure call with a v1 api_version. The OpenAI SDK (>=2.20.0) accepts a callable for `api_key` and re-invokes it on every request via `_refresh_api_key`, so we now forward `azure_ad_token_provider` directly — preserving the per-request token refresh behavior of the regular AzureOpenAI client and avoiding the expiry hole that resolving the token once at client-creation time would introduce. Static `azure_ad_token` strings fall through to `api_key`. For the async path we wrap the sync provider returned by azure-identity in an async function since AsyncOpenAI expects `Callable[[], Awaitable[str]]`. Fixes BerriAI#27945 https://claude.ai/code/session_01UnzrDSFUUgp5T2wRoPMxq5 * fix(azure): offload sync token provider to thread in v1 async wrapper * fix(azure): include AD credential identity in v1 client cache key --------- Co-authored-by: Claude <noreply@anthropic.com>
* fix(ui): route API Reference back to query-param page The path-based /ui/api-reference route was broken in practice — the page-local useProxySettings hook didn't match what the root page passes down. Remove api_ref from the migration maps (LEGACY_REDIRECTS in app/page.tsx, MIGRATED_PAGES in leftnav.tsx and (dashboard)/layout.tsx), point the leftnav item back at page="api_ref", and restore the api_ref render branch in the root page. The path-based page.tsx and the useProxySettings hook stay in place unchanged; only api_ref is moved back to query-param routing while the migration infrastructure is preserved for future page moves. * fix(ui): alias ?page=api-reference to api_ref branch Handles bookmarks of the hyphen-form query param that was live during the brief path-based migration window, so they render the working APIReferenceView instead of falling through to the default page.
* fix(mcp): resolve team.access_group_ids → MCP servers A virtual key whose team has an MCP-granting access group attached via /v1/access_group now sees that server through /v1/mcp/server (and can call tools on it) instead of getting an empty list. The runtime already resolves the key's unified access_group_ids; this adds the symmetric resolution on the team side, mirroring the model-side pattern in can_team_access_model — the group being on the team is itself the gate, so no assigned_team_ids re-check is needed. Resolves BerriAI#27657 * chore(mcp): address greptile review on team access-group resolver Forward already-imported prisma_client / user_api_key_cache / proxy_logging_obj to _get_mcp_server_ids_from_access_groups so it skips its lazy re-import path. Update test docstring + assertions to reflect that the resolver is invoked with [] (and short-circuits without DB access) rather than skipped entirely.
…AI#28652) * test(ui): e2e cover team model edit + admin identity in navbar Adds two Playwright tests as part of the manual-QA → e2e migration: "Edit team model selection" exercises the Settings tab Models multi-select + Save Changes flow on a seeded team, and the existing login test now opens the User dropdown and asserts the role and User ID render — guarding against regressions where login succeeds but the auth context is empty. Resolves LIT-3093 * test(ui): restore seeded models in team-edit test so retries don't fail The 'Edit team model selection' test removed fake-anthropic-claude from E2E_TEAM_CRUD_ID without restoring it. CI runs with retries: 2 and the seed script runs once before the suite, so a flake on this test would fail the retry at the "tag is visible" assertion. Wrap the test in try/finally and restore the seeded models via /team/update before and after. * test(e2e): fail loudly if team/update restore call fails Surfaces the real cause when the master key is wrong or the proxy is unreachable, instead of silently leaving the team in a stale state and failing later on the visibility assertion. * fix(e2e): match navbar account button by aria-label, not non-existent "User" text The previous trigger filter (hasText: /^User$/) didn't match the rendered UserDropdown button — its text is the displayName ("Account" for the master-key admin, an email for SSO users), never "User". The evaluate call then timed out after 15s in CI. Use the stable aria-label prefix the component always emits, and click directly since the dropdown is configured trigger=["click"] (the synthetic hover was unnecessary).
* test(e2e): cover add-fallback flow in Router Settings as proxy admin The Router Settings → Fallbacks → Add Fallbacks flow was an uncovered manual-QA path. This adds a test that opens the modal, picks a primary + fallback from the seeded mock models, saves, and verifies both render in the fallback table. * fix(e2e): make router-fallback test idempotent and pick antd options by text - Match `.ant-select-item-option` by text instead of `getByTitle(...)` — FallbackGroupConfig uses `options=` (not <Select.Option> children), so no `title` attribute is emitted and the title-based selector hangs. - Add before/after hooks that wipe any fallback for fake-openai-gpt-4 via /config/update so retries and local reruns don't trip on leftover state. - Tighten the success assertion to a single tbody row containing BOTH the primary and the fallback names — pre-existing rows can no longer vacuously satisfy the check. - Fix the stale "Three tabs" comment to "Four tabs". Addresses Greptile P2s on PR BerriAI#29069. * fix(e2e): keyboard-select fallback models + correct cleanup endpoint - Replace mouse-based option clicks with click-to-focus + type + Enter. FallbackGroupConfig's Selects use `options=` and a custom getPopupContainer, so locating options via `.ant-select-dropdown` hit several races: DOM-clicks left antd's popup state stale (the primary popup then intercepted the fallback click), `getByRole` matched always-mounted hidden options, and pointer stability fought the open animation. Typing into the showSearch input narrows the listbox to one option and Enter selects it cleanly. - Assert on dialog-side state changes (the active tab adopts the primary model name; the chain helper shows "1/10 used") instead of popup contents — these reflect the actual selection landing. - Cleanup helper now hits /get/config/callbacks (the real endpoint; /get/callbacks returns 404), so the before/after reset actually clears prior router_settings.fallbacks state.
* test(e2e): cover Team-BYOK add-model flow as proxy admin The team-only model + team assignment was an uncovered manual-QA path. This adds a premium-gated test that toggles Team-BYOK, picks the seeded E2E Team CRUD, submits, and verifies the model lands in All Models with the team alias attached. * test(e2e): apply greptile fixes to Team-BYOK test - Add the 2s networkidle settle that the sibling addModel tests use — networkidle fires before the All Models table finishes re-rendering, so the search input was racing with the render. - Assert on `models-results-count` before inspecting the table body so an empty search result fails with a clear "expected results count" message instead of timing out on a missing row. Addresses Greptile P2s on PR BerriAI#29068. * test(e2e): harden Team-BYOK test against flake and stale state - Add before/after cleanup that deletes any Cohere model already scoped to e2e-team-crud via /v2/model/info + /model/delete, so Playwright retries and local reruns don't accumulate rows. - Pick the team from the dropdown by role/option name instead of a global getByText match — avoids matching a previously-rendered tag elsewhere in the form. - Scope the "created successfully" assertion to .ant-notification so a stale toast from an earlier test in the same browser context can't vacuously satisfy it. - Tighten the All Models assertion: require a single row that contains BOTH the cohere model name AND the e2e-team-crud alias, so the team-less wildcard from the sibling "Add wildcard route" test can't satisfy the check.
…ma Json serialization (BerriAI#28990) * fix(containers): record ownership for service-account keys + fix Prisma Json field serialization - Track containers created implicitly via /v1/responses by extracting container IDs from the response output and calling record_container_owner for each one, so subsequent file-API calls from the same service account pass ownership checks. - Fix DataError: Prisma Python requires Json fields to be JSON strings; serialize file_object with json.dumps() before insert/update in LiteLLM_ManagedObjectTable. - Add collect_container_ids_from_responses_response utility to responses/utils.py that walks all output item shapes (code_interpreter_call, message annotations). - Tests: two new cases covering the responses-tracking path and the end-to-end record-then-assert flow for service accounts with team scope. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(containers): swallow all exceptions in ownership hook; tighten file_object_json type to str Co-authored-by: Cursor <cursoragent@cursor.com> * fix(containers): parse file_object JSON string in existing ownership test Co-authored-by: Cursor <cursoragent@cursor.com> * fix: container ownership recording bugs - Remove unreachable _aresponses_websocket from route_type set in base_process_llm_request; the WebSocket endpoint never flows through base_process_llm_request, so this branch was dead code that gave a false impression of coverage. - Drop the HTTPException re-raise in record_container_owners_from_responses_response so per-container failures (including HTTP 403/500 from conflicting ownership rows) no longer abort the batch and skip recording for the remaining container IDs in the same response. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(containers): record ownership for streaming /v1/responses too Streaming /v1/responses returns through the select_data_generator branch in base_process_llm_request and bypasses the non-streaming ownership tail, so code-interpreter containers created mid-stream were never written to LiteLLM_ManagedObjectTable. Follow-up file API calls would then 403. Wrap the SSE generator so container ownership is recorded once the upstream iterator finishes assembling completed_response. Also covers the background-polling path, which loops body_iterator end-to-end. Co-authored-by: Yassin Kortam <yassin@berri.ai> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai>
…riAI#29070) * test(e2e): cover add-MCP-server flow via discovery → custom form The "Add MCP server" manual-QA step was uncovered. This adds a test that opens the discovery modal, jumps into the custom-server form, fills name + Streamable HTTP transport + a placeholder URL + None auth, submits, and verifies both the success toast and the new row. * test(e2e): apply greptile fixes to MCP add-server test - Anchor the auth-type Select via its enclosing Collapse panel ("Authentication") instead of the placeholder text. The Form.Item has no label prop, so the previous `hasText: /auth type/i` filter was matching via "Select auth type" placeholder copy — fragile. - Document the intentional lack of teardown, matching the pattern used in addModel.spec.ts: the e2e runner discards the DB per invocation. Addresses Greptile P2s on PR BerriAI#29070. * test(e2e): scope MCP row assertion to the servers table Scope the post-create row lookup to `table tbody` so the form modal's `server_name` input — which still holds the timestamped value during its close animation — can't satisfy the assertion before the server actually lands in the list. * docs(e2e): note MCP coverage scope and link to tracker This spec only smoke-tests the happy-path Streamable HTTP + None auth flow. Add a top-of-file comment pointing at E2E_COVERAGE.md so future contributors can see what's still uncovered (other transports, all auth types, edit/delete, BYOK, tool list/call, access groups).
…erriAI#29071) * test(e2e): cover AI Hub make-public flow and public model_hub_table Three previously-uncovered manual-QA paths land in one spec: - Admin opens "Select Models to Make Public", advances through the multi-step modal, and verifies the success toast. - AI Hub tab strip exposes Model Hub / Agent Hub / MCP Hub / Skill Hub — note the manual-QA "Claude Code Plugin Marketplace" label was renamed to Skill Hub; the test pins the current name. - Anonymous /ui/model_hub_table loads with the master key as `?key=` and renders the Model Hub tab. Agent Hub / MCP Hub tabs are conditional on public data and are not asserted here. * test(e2e): harden AI Hub make-public + public hub assertions Address Greptile review: - Make-public test now asserts "Select All (N)" with N>=1 before clicking, so a missing-seed-data run surfaces immediately instead of timing out on the disabled Next button or the success toast. - Public model_hub_table test dismisses the feedback popup before the tab visibility assertion, matching the ordering used by navigateToPage so a popup race can't mask the tab mid-evaluation. * docs(e2e): explain admin vs public AI Hub tab asymmetry Greptile flagged the all-4-tabs assertion as a potential CI flake, inferring from the public-page comment that Agent Hub / MCP Hub might be data-conditional in the admin view too. They aren't — ModelHubTable renders all four tabs unconditionally for admins. Document the asymmetry inline so future readers (and future review passes) don't re-derive it.
…onfig (BerriAI#28898) * feat: support goal mode for claude on bedrock * fix failing lint test * addressing greptile comments * fixing failed test * address greptile: copy output_config and warn on dropped converse format * fix(bedrock): skip redundant output_config normalization on Converse reasoning_effort path When reasoning_effort is mapped via _handle_reasoning_effort_parameter, the resulting output_config is already normalized via normalize_bedrock_opus_output_config_effort. Mark it as normalized so _prepare_request_params can skip the redundant call (and the associated get_model_info lookup) on every request. Co-authored-by: Yassin Kortam <yassin@berri.ai> * test(reasoning-effort-grid): reflect Bedrock opus-4-6 xhigh→max clamping * fix(bedrock): stop leaking output_config marker and message-content mutation * fix(bedrock): guard effort key access in normalize_bedrock_opus_output_config_effort Defensively check that 'effort' is a valid key in _BEDROCK_OUTPUT_CONFIG_EFFORT_ORDER before indexing, to prevent a KeyError if the hardcoded guard tuple ever drifts from the order dict's keys. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(bedrock): drop dead second clause in effort normalization guard The 'effort not in _BEDROCK_OUTPUT_CONFIG_EFFORT_ORDER' check is unreachable once 'effort not in ("xhigh", "max")' has been ruled out, since both literals are present in the order dict. Keep the literal membership check and let the dict lookups below speak for themselves. * fix(bedrock): clamp output_config.effort against ceiling for any known value The early return when effort was not 'xhigh'/'max' meant a ceiling of 'low' or 'medium' would silently forward an out-of-range value. Gate on the known effort ordering instead so the ceiling comparison runs for every recognized effort. * test(grid_spec): use _CAPS_OPUS_4_7 for non-Bedrock opus-4-6 entries claude-opus-4-6 now declares supports_xhigh_reasoning_effort in the model map, so production accepts xhigh on Azure AI and Vertex AI routes. Update those grid_spec entries to match production capabilities so expected() predicts 200 for xhigh instead of 400. Co-authored-by: Yassin Kortam <yassin@berri.ai> * test(grid_spec): revert xhigh caps for non-Bedrock opus-4-6 azure_ai/claude-opus-4-6 and vertex_ai/claude-opus-4-6 do not declare supports_xhigh_reasoning_effort in model_prices_and_context_window.json. Azure AI upstream rejects xhigh with HTTP 400 ("Supported levels: high, low, max, medium"). Restore _CAPS_4_6 so the grid predicts 400 for xhigh, matching production capabilities. * fix: stop advertising xhigh effort on Opus 4.5/4.6 Only Opus 4.7 supports the xhigh reasoning effort level. Remove the supports_xhigh_reasoning_effort flag from every Opus 4.5 and Opus 4.6 entry (direct Anthropic, Bedrock, and regional variants) in both model catalog files. On the direct Anthropic path there is no effort clamp, so flagging 4.5/4.6 as xhigh-capable caused litellm to forward xhigh to a model that rejects it (and made get_model_info misreport the capability). xhigh now correctly degrades to high / raises on those models. Bedrock graceful degradation for Claude Code goal mode is unaffected: it relies solely on the bedrock_output_config_effort_ceiling clamp (4.5->high, 4.6->max, 4.7->xhigh), which runs before validation, so xhigh requests to older Bedrock Opus models are still silently lowered rather than rejected. Update effort-gating tests to reflect that 4.5/4.6 no longer accept xhigh. * fix: clamp xhigh effort on Bedrock Invoke /v1/messages instead of rejecting Claude Code "goal mode" sends output_config.effort=xhigh over the Anthropic /v1/messages API, which routes Bedrock models through AmazonAnthropicClaudeMessagesConfig. That path validated effort against the model's native capability and raised 400 for xhigh on Opus 4.6, while the chat-completions paths (Converse + Invoke) already clamp xhigh to the model's bedrock_output_config_effort_ceiling. That asymmetry broke goal mode on the exact API surface Claude Code uses. Apply the same ceiling clamp on the messages path before the shared effort gate runs, so xhigh degrades to max on Opus 4.6 (and stays xhigh on 4.7). Scoped to adaptive-thinking models and to models that declare a ceiling, so Sonnet 4.6 (no ceiling) and Opus 4.5 (budget mode) are unaffected and still reject xhigh. * fix(bedrock): preserve user output_config when applying reasoning_effort - Converse path: merge mapped effort into existing output_config via setdefault instead of overwriting it, matching the Anthropic Messages path. Prevents user-supplied output_config.format from being silently dropped when reasoning_effort is also provided. - tests: clear _get_local_model_cost_map lru_cache in the autouse fixture alongside get_bedrock_response_stream_shape to avoid stale cache leakage between tests. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(bedrock): pre-clamp reasoning_effort for chat invoke; correct test caps - Add _clamp_adaptive_reasoning_effort_for_bedrock to AmazonAnthropicClaudeConfig so raw reasoning_effort=xhigh degrades to the model's bedrock effort ceiling before AnthropicConfig.map_openai_params converts it to output_config. Mirrors converse path (_handle_reasoning_effort_parameter) and messages path (_clamp_adaptive_reasoning_effort_for_bedrock) so the three Bedrock paths are consistent. - grid_spec: restore caps=_CAPS_4_6 for Bedrock converse/invoke Opus 4.6 entries so the test reflects the model's actual JSON capabilities. Teach expected() to bypass the xhigh/max cap check when bedrock_effort_ceiling will clamp the wire effort, so the test still passes for Bedrock's graceful degradation contract without lying about native model caps. Co-authored-by: Yassin Kortam <yassin@berri.ai> --------- Co-authored-by: Dennis Henry <dennis.henry@okta.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai>
…erriAI#28970) * feat(guardrails): wire apply_guardrail into proxy logging callbacks Route /apply_guardrail through pre/post proxy hooks and LiteLLM success/failure handlers so Langfuse and OTEL integrations receive input/output on guardrail-only requests. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(guardrails): fix Greptile review comments on apply_guardrail logging Co-authored-by: Cursor <cursoragent@cursor.com> * fix(apply_guardrail): preserve original exception and capture modified response - Capture return value from post_call_success_hook so callback-modified responses propagate to the caller. - Wrap success/failure logging calls in defensive try/except so logging infrastructure failures don't replace the user-visible response or mask the original guardrail exception. Co-authored-by: Yassin Kortam <yassin@berri.ai> * Fix mypy * fix(apply_guardrail): isolate failure logging and use post-hook response for logging - Split async_failure_handler and post_call_failure_hook into independent try/except blocks so a callback bug in one does not silently skip the other. - Build response_for_logging inside _emit_guardrail_success_logs after post_call_success_hook runs, so logged data matches the response the caller actually receives when the hook modifies the response. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(apply_guardrail): fix black formatting and update tests for fastapi_request param - Run black on guardrail_endpoints.py to fix CI formatting check - Add _mock_proxy_logging() helper to enterprise guardrail tests to patch proxy-server globals imported at call time - Pass fastapi_request=Mock() in all direct apply_guardrail test calls to match updated function signature Co-authored-by: Cursor <cursoragent@cursor.com> * fix(guardrails): use transformed exception from post_call_failure_hook in apply_guardrail Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(guardrails): isolate sync/async logging handlers in apply_guardrail Separate each logging handler call into its own try/except so a failure in the async handler does not silently skip the sync handler submission (and vice versa). Matches the docstring's defensive intent. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(apply_guardrail): guard transformed_exception with isinstance check Co-authored-by: Cursor <cursoragent@cursor.com> * test(guardrails): mock proxy globals in not_found test and share apply_guardrail logging fixture - Add proxy-server global mocks to test_apply_guardrail_not_found so the failure-path post_call_failure_hook call doesn't touch the real proxy logging singleton. - Extract the duplicated _mock_proxy_logging context manager out of the two enterprise apply_guardrail test files into a shared conftest fixture so the helper stays in one place. * fix(guardrails): use update_messages to keep logging obj in sync Co-authored-by: Yassin Kortam <yassin@berri.ai> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* build(deps): bump next from 16.2.4 to 16.2.6 in /ui/litellm-dashboard (BerriAI#27665) Bumps [next](https://github.com/vercel/next.js) from 16.2.4 to 16.2.6. - [Release notes](https://github.com/vercel/next.js/releases) - [Changelog](https://github.com/vercel/next.js/blob/canary/release.js) - [Commits](vercel/next.js@v16.2.4...v16.2.6) --- updated-dependencies: - dependency-name: next dependency-version: 16.2.6 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump protobufjs in /tests/pass_through_tests (BerriAI#28296) Bumps [protobufjs](https://github.com/protobufjs/protobuf.js) from 7.5.6 to 7.6.0. - [Release notes](https://github.com/protobufjs/protobuf.js/releases) - [Changelog](https://github.com/protobufjs/protobuf.js/blob/protobufjs-v7.6.0/CHANGELOG.md) - [Commits](protobufjs/protobuf.js@protobufjs-v7.5.6...protobufjs-v7.6.0) --- updated-dependencies: - dependency-name: protobufjs dependency-version: 7.6.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump ws from 8.20.0 to 8.20.1 in /tests/pass_through_tests (BerriAI#28303) Bumps [ws](https://github.com/websockets/ws) from 8.20.0 to 8.20.1. - [Release notes](https://github.com/websockets/ws/releases) - [Commits](websockets/ws@8.20.0...8.20.1) --- updated-dependencies: - dependency-name: ws dependency-version: 8.20.1 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix(proxy): enforce tag budgets for key-level tags Merge API key metadata.tags into request_data before _tag_max_budget_check so per-tag budgets apply when tags are set on the key at creation time. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(auth): avoid false reject for key-inherited tags Run reject_clientside_metadata_tags before key-tag injection, then inject key metadata tags immediately before tag budget checks so key tags still enforce budgets without being treated as client-supplied tags. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…video edit (BerriAI#29098) * fix(vertex-ai): pass litellm_params to validate_environment in video handlers and implement video edit for Veo - Pass litellm_params to validate_environment in 11 video handler call sites (remix, create_character, get_character, edit, extension, delete) so DB-stored Vertex AI credentials are used instead of falling back to ADC - Implement transform_video_edit_request/response for VertexAI: fetches source video via fetchPredictOperation then submits a new predictLongRunning request with the video bytes/gcsUri + edit prompt Co-authored-by: Cursor <cursoragent@cursor.com> * fix(vertex-ai): hoist fetchPredictOperation into handlers to avoid blocking event loop - Add get_video_edit_prefetch_params() to BaseVideoConfig (returns None) - VertexAI overrides it to return the fetchPredictOperation URL/body - Both sync and async video_edit handlers call this and use their shared httpx client for the fetch, passing the result as prefetched_source_data - transform_video_edit_request is now a pure transform with no HTTP calls - Fix extra_body.pop() mutation by working on a shallow copy Co-authored-by: Cursor <cursoragent@cursor.com> * fix(vertex-ai): include prefetch call inside _handle_error try/except block Co-authored-by: Cursor <cursoragent@cursor.com> * fix(videos): add prefetched_source_data param to all transform_video_edit_request overrides Co-authored-by: Cursor <cursoragent@cursor.com> * fix(video_edit): keep transform/pre_call outside try so validation errors propagate Move transform_video_edit_request and logging_obj.pre_call outside the try/except that wraps HTTP calls in (async_)video_edit_handler so that ValueError validation errors (e.g. 'source video not complete yet') are not silently wrapped as 500s by _handle_error. The prefetch HTTP call keeps its own try/except so its errors are still mapped through the provider's error handler. Matches the pattern used by video_extension_handler and video_remix_handler. Co-authored-by: Yassin Kortam <yassin@berri.ai> * refactor(vertex_ai): delegate get_video_edit_prefetch_params to status retrieve Co-authored-by: Yassin Kortam <yassin@berri.ai> * Fix varia review * fix(video_edit): route transform errors through _handle_error Wrap transform_video_edit_request and pre_call in the same try/except as the HTTP call in sync and async handlers so validation failures (e.g. source video not complete) return typed LiteLLM exceptions. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai>
…st (BerriAI#28487) * fix(datadog): drain cost-management queue + opt-in FinOps tag allowlist * fix(datadog): guard non-dict callback_specific_params + log empty aggregation * fix(datadog): block user-controlled tags from overwriting reserved cost-attribution dimensions * fix(datadog): cast metadata to dict[str, Any] to satisfy mypy
… and UI (BerriAI#28712) * feat(helm): split per-component ServiceAccounts for gateway, backend, and UI Replace the single shared serviceAccount with three separate serviceAccounts (gateway, backend, ui) so operators can attach different IRSA / Workload Identity annotations per component without granting data-plane credentials to the UI pod. Key changes: - values.yaml: rename serviceAccount → serviceAccounts with gateway/backend/ui sub-keys; UI defaults to automount: false - _helpers.tpl: replace litellm.serviceAccountName with three component-scoped helpers (litellm.gateway/backend/ui.serviceAccountName) - serviceaccount.yaml: create up to three separate ServiceAccount objects with component labels and per-SA automountServiceAccountToken - gateway/backend deployments: use their respective SA helpers - ui deployment: use litellm.ui.serviceAccountName + explicit automountServiceAccountToken: false on the pod spec so the projected token is absent even when the SA itself allows it - migrations-job: share the backend SA (both need DB write access) Resolves LIT-3171 https://claude.ai/code/session_01QPy362WnjmEpeNuJaPUqmF * fix(helm): enforce automountServiceAccountToken on all pod specs; fix leading --- in serviceaccount.yaml - gateway/backend deployments: add explicit automountServiceAccountToken on the pod spec so serviceAccounts.*.automount is honoured regardless of whether the SA is chart-created or operator-supplied (previously the flag only took effect on the SA object when create: true, creating an asymmetry with the UI which already enforced it at pod-spec level) - serviceaccount.yaml: use a $prev sentinel to emit --- only between documents, preventing a leading --- when gateway SA is skipped but backend or ui SA is created (avoids lint/GitOps warnings from strict YAML parsers and tools like ArgoCD) https://claude.ai/code/session_01QPy362WnjmEpeNuJaPUqmF --------- Co-authored-by: Claude <noreply@anthropic.com>
* fix(deps): bump vulnerable proxy dependencies (starlette/fastapi, granian, pyarrow, semantic-router) Resolve known CVEs flagged by osv-scanner/grype against uv.lock. All bumped versions verified to resolve, install, and pass the proxy auth/route/middleware unit suites (717 tests) plus an import smoke on the new stack. - starlette 0.50.0 -> 1.1.0 (CVE-2026-48710 "BadHost", GHSA-86qp-5c8j-p5mr): versions <1.0.1 reconstruct request.url from the unvalidated Host header, poisoning request.url.path. Required raising fastapi 0.124.4 -> 0.136.3, which dropped fastapi's starlette<0.51.0 cap; an explicit starlette>=1.0.1 floor blocks regression to a vulnerable transitive resolution. The proxy's own auth already reads scope["path"] via get_request_route, but the locked starlette still flagged in container scanners and left other request.url consumers exposed. - granian 2.5.7 -> 2.7.4 (CVE-2026-42544, unauthenticated DoS via WebSocket subprotocol header panic; CVE-2026-42545, WSGI response-header-panic DoS). granian is a selectable proxy server (proxy_cli). - pyarrow 22.0.0 -> 23.0.1 (CVE-2026-25087 / PYSEC-2026-113). - semantic-router 0.1.12 -> 0.1.15: 0.1.12 was yanked (CVE-2026-42208 — its unbounded litellm pin could resolve a credential-exfiltrating litellm==1.82.8 wheel). Not fixable by bump: diskcache 5.6.3 (CVE-2025-69872, unsafe pickle deserialization) has no upstream fix and is left pinned; exploiting it requires write access to the local cache directory. Relock side effect: sse-starlette 3.4.2 -> 3.4.4. * deps: relax exact pins in optional extras to compatible ranges The proxy/optional extras exact-pinned every dependency, which (1) forces downstream `pip install litellm[proxy]` consumers into version lockstep and (2) blocks them from pulling transitive security patches without forking — the structural cause behind needing a litellm release to clear the starlette CVE in the previous commit. Convert the ordinary extras deps to `>=current,<next_major` ranges, mirroring the core [project].dependencies style. Reproducibility for litellm's own Docker/CI is unaffected: images install via `uv sync --frozen`, and the lock re-resolves to the identical versions (no locked version changed). Kept exact-pinned: - litellm-proxy-extras, litellm-enterprise — litellm's own sub-packages, versioned in lockstep with the release. - opentelemetry-api/sdk/exporter-otlp — must resolve to matching versions. - grpcio — supply-chain-pinned to a vetted, aged release. Also corrects the stale comment claiming the extras are exact-pinned for Docker reproducibility (the images use the lock, not these pins). * fix(ci): resolve license-check lookup version from the floor for ranged deps check_licenses.py derived the PyPI lookup version with `next(iter(req.specifier))`, which returns an arbitrary specifier clause. For a range like `>=0.12.1,<1.0` it picked the upper bound (`1.0`) — a version that doesn't exist on PyPI — so the license lookup 404'd and the package was flagged as having an unknown license. The previous commit's switch from exact pins to ranges exposed this for soundfile, pyroscope-io, redisvl, diskcache, and mlflow (the ranged deps not already in liccheck.ini's allowlist). Prefer a lower-bound/exact version (a real released version) for the lookup. * fix(proxy): set strict_content_type=False on the FastAPI app Starlette 1.0 / FastAPI 0.13x flipped the default to strict_content_type=True, which refuses to parse a JSON request body when the client omits the Content-Type header. The proxy previously accepted those requests, so the fastapi/starlette bump in this PR would silently break clients that don't send a Content-Type. Restore the prior lenient behavior explicitly. Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com>
…replay (BerriAI#29229) The Redis-backed VCR layer was recording and replaying the Google OAuth2/STS token-mint call. The replayed ya29.* access token is long-expired, but its recorded expires_in keeps credentials.expired False, so litellm never refreshes it and sends the stale token to a live Vertex/Gemini endpoint, which returns 401 ACCESS_TOKEN_EXPIRED. This broke live partner-model tests whose completion call is not itself cassette-backed (e.g. test_vertex_ai_llama_tool_calling). Force credential-exchange hosts to pass through live (never recorded, never replayed) by returning None from before_record_request, mirroring the existing telemetry passthrough, so a fresh token is minted each run. Regression from BerriAI#28826, which added OAuth-token matcher tolerance plus TTL-refresh-on-read so a stale token episode matched and never expired.
…AI#29234) Updates the gollem_go_agent_framework example to the current Go release. Clears stale Go stdlib advisories reported by osv-scanner against the older 1.25.1 directive. No source changes; the single pinned dependency (gollem v0.1.0) is backward compatible.
* bump: version 1.87.0 → 1.88.0 * uv lock
…BerriAI#29238) * feat(anthropic): add Claude Opus 4.8 and prune reasoning-effort flags Register claude-opus-4-8 across the anthropic/bedrock/vertex/azure cost-map entries, BEDROCK_CONVERSE_MODELS, and the setup-wizard provider list. Prune two reasoning-effort fields from the cost map: - Drop supports_minimal_reasoning_effort from the Claude fleet (58 entries). "minimal" is not a real Anthropic effort level (the API accepts only low/medium/high/xhigh/max), so LiteLLM degrades it to "low" regardless; the flag was inert and misleading on Anthropic. - Remove tool_use_system_prompt_tokens everywhere (103 entries). It is not in the ModelInfo type and is read by no production code. Update the affected config/schema tests; the reasoning-effort registry tests now assert the Claude fleet omits supports_minimal. * fix(anthropic): recognize output_config effort after minimal-flag prune Pruning supports_minimal_reasoning_effort from the Claude fleet removed the only "supports effort param" marker from 11 Opus 4.5 / mythos-preview map entries that lack supports_output_config. _model_supports_effort_param then returned False for them, so output_config was wrongly dropped under drop_params=True -- regressing test_anthropic_model_supports_effort_param_recognizes_supporting_models for claude-opus-4-5-20251101 and the mythos preview. - _model_supports_effort_param now treats supports_output_config as a sufficient signal, matching the bedrock-invoke call sites that already check supports_output_config OR a reasoning-effort flag. Shared map lookup extracted into _supports_model_capability. - Add supports_output_config: true to the 11 Opus 4.5 / mythos entries that lost their only marker, restoring prior effort-forwarding behavior without re-adding the inert minimal flag.
chore(ci): promote internal staging to main
Contributor
Greptile SummaryThis PR adds Helm unit tests to cover the default image tag rendering path — verifying that when
Confidence Score: 4/5Safe to merge; only adds chart unit tests with no changes to rendered manifests. The tests correctly cover the previously unprotected default image tag path. The only concern is the hardcoded v1.85.1 string in both equal assertions — whenever Chart.yaml appVersion is bumped, both test files must be manually updated or the suite will fail. Both deployment_tests.yaml and migrations-job_tests.yaml hardcode the chart version and will need updating on every appVersion bump.
|
| Filename | Overview |
|---|---|
| deploy/charts/litellm-helm/tests/deployment_tests.yaml | Adds a new test asserting the default image tag renders as the chart appVersion; hardcodes v1.85.1 which will require manual updates on every version bump |
| deploy/charts/litellm-helm/tests/migrations-job_tests.yaml | Adds a new test for the migrations-job default image tag path; same hardcoded v1.85.1 value and redundant notMatchRegex assertion as the deployment test |
Reviews (1): Last reviewed commit: "test helm default image tag" | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+9
to
+14
| - equal: | ||
| path: spec.template.spec.containers[0].image | ||
| value: ghcr.io/berriai/litellm-database:v1.85.1 | ||
| - notMatchRegex: | ||
| path: spec.template.spec.containers[0].image | ||
| pattern: :main- |
Contributor
There was a problem hiding this comment.
equal assertion pins the exact string v1.85.1, which will cause this test to fail every time Chart.yaml's appVersion is bumped — requiring a manual update in both test files on every release. A matchRegex pattern that validates the semver-shaped tag already proves the default path uses the chart version, without tying the test to a specific version number. The notMatchRegex: :main- assertion beneath it becomes load-bearing after this change and is no longer redundant.
Suggested change
| - equal: | |
| path: spec.template.spec.containers[0].image | |
| value: ghcr.io/berriai/litellm-database:v1.85.1 | |
| - notMatchRegex: | |
| path: spec.template.spec.containers[0].image | |
| pattern: :main- | |
| - matchRegex: | |
| path: spec.template.spec.containers[0].image | |
| pattern: ^ghcr\.io/berriai/litellm-database:v\d+\.\d+\.\d+$ | |
| - notMatchRegex: | |
| path: spec.template.spec.containers[0].image | |
| pattern: :main- |
Comment on lines
+11
to
+16
| - equal: | ||
| path: spec.template.spec.containers[0].image | ||
| value: ghcr.io/berriai/litellm-database:v1.85.1 | ||
| - notMatchRegex: | ||
| path: spec.template.spec.containers[0].image | ||
| pattern: :main- |
Contributor
There was a problem hiding this comment.
deployment_tests.yaml: pinning v1.85.1 means every appVersion bump breaks this test. Using a matchRegex for a semver-shaped tag is more resilient and still proves the default path does not produce a :main- tag.
Suggested change
| - equal: | |
| path: spec.template.spec.containers[0].image | |
| value: ghcr.io/berriai/litellm-database:v1.85.1 | |
| - notMatchRegex: | |
| path: spec.template.spec.containers[0].image | |
| pattern: :main- | |
| - matchRegex: | |
| path: spec.template.spec.containers[0].image | |
| pattern: ^ghcr\.io/berriai/litellm-database:v\d+\.\d+\.\d+$ | |
| - notMatchRegex: | |
| path: spec.template.spec.containers[0].image | |
| pattern: :main- |
Contributor
Congrats! CodSpeed is installed 🎉
You will start to see performance impacts in the reports once the benchmarks are run from your default branch.
|
Author
|
Closing this one because I initially opened it against , which makes the retarget diff much too large. Reopened the same fix on the correct OSS base in #29371. |
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Author
|
Clarifying the previous close note: I initially opened this against |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
11 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The stable Helm chart issue in #29348 shows why the default image tag path matters: if
image.tagis unset, the Deployment and migrations Job must render the published chart appVersion tag, not amain-<version>tag.The templates already use
.Chart.AppVersionafter #28710, but the Helm unit tests only covered explicitimage.tagoverrides. That left the default path unprotected.Fix
Add Helm unit coverage for the default image tag in both:
templates/deployment.yamltemplates/migrations-job.yamlEach test asserts the rendered image uses the chart appVersion and does not include the
:main-prefix.Test
make test-unit-helm-> 56 passedhelm lint ./deploy/charts/litellm-helm-> passedgit diff --check-> passedNote: with Helm 4 locally, I had to install the existing pinned
helm-unittestplugin with--verify=false; after that, the repo'smake test-unit-helmcommand ran successfully.Risk
Low. This only adds chart tests and does not change rendered manifests.