fix(proxy): remove ghost team BYOK models from /models on delete - #29527
Closed
aanchal22 wants to merge 112 commits into
Closed
fix(proxy): remove ghost team BYOK models from /models on delete#29527aanchal22 wants to merge 112 commits into
aanchal22 wants to merge 112 commits into
Conversation
… fixes, model catalog) (BerriAI#28223) * fix(opentelemetry): JSON-serialize dict metadata fields for OTEL span attributes (#27451) (#27455) Squash-merged by litellm-agent from Anai-Guo's PR. * feat(dashscope): add embeddings and reranks(qwen3-rerank) support via OpenAI-compatible endpoint (#27508) Squash-merged by litellm-agent from yimao's PR. * fix(vertex_ai/gemini): raise BadRequestError when image_url or url fi… (#24550) Squash-merged by litellm-agent from krisxia0506's PR. * fix(vertex_ai): raise error on mid-stream 429/error chunks instead of silently swallowing (#23711) Squash-merged by litellm-agent from krisxia0506's PR. * fix: raise BadRequestError for file content blocks missing 'file' sub… (#24503) Squash-merged by litellm-agent from krisxia0506's PR. * Fix Gemini MIME detection for extensionless GCS URIs (#27278) Squash-merged by litellm-agent from krisxia0506's PR. * fix(vertex_ai/partner_models): drop unused vertexai SDK gate from count_tokens (closes #28084) (#28107) Squash-merged by litellm-agent from voidborne-d's PR. * feat(chart): add support for autoscaling behavior in HPA (#27990) Squash-merged by litellm-agent from FabrizioCafolla's PR. * feat(proxy): add blocked flag to models for pause/resume from the UI (#27927) Squash-merged by litellm-agent from Cyberfilo's PR. * fix: pass socket timeouts to Redis cluster clients (#27920) Squash-merged by litellm-agent from tomdee's PR. * Fix/cache token (#28009) Squash-merged by litellm-agent from escon1004's PR. * fix(deepseek): forward reasoning_content in multi-turn thinking mode conversations (#28080) Squash-merged by litellm-agent from Divyansh8321's PR. * fix(guardrails): return HTTP 400 instead of 500 for blocked requests (#27617) * fix: reset org and tag budgets (#27326) * reset org budgets * reset tag budgets --------- Co-authored-by: Michael Riad Zaky <michaelr@Mac.localdomain> * fix(ui): omit allowed_routes from key edit save when unchanged (#27553) * fix(ui): omit allowed_routes from key edit save when unchanged When a team admin opens Edit Settings on a key with key_type=AI APIs and saves without changing anything, the UI re-sends the existing allowed_routes value, which the backend's _check_allowed_routes_caller_permission gate rejects for non-proxy-admins (LIT-2681). Strip allowed_routes from the patch in handleSubmit when it deep-equals the original keyData.allowed_routes. The backend treats absence as "leave alone," so no-op saves now succeed for non-admins. Admins explicitly editing the field still send the new value. * fix(ui): order-insensitive allowed_routes diff + cover null-original case Address Greptile review: - Switch the "is allowed_routes unchanged" check to a Set-based comparison so a server-side reorder of the array doesn't register as a user edit and re-trigger LIT-2681. - Add two regression tests: (1) keyData.allowed_routes is null and the form is untouched — patch should strip the field; (2) server returned routes in a different order than the user originally entered — patch should still recognize the value as unchanged. * chore(ui): strip ticket refs and tighten comments in key edit fix - Remove internal-tracker references from in-code comments - Tighten the WHY comment in handleSubmit to two lines - Drop redundant test-block comments — test names already describe the case * fix(ui): annotate Set<string> generic in allowed_routes diff to fix tsc * fix(guardrails): return HTTP 400 instead of 500 for guardrail-blocked requests GuardrailRaisedException and BlockedPiiEntityError both lacked a status_code attribute. When these exceptions reached the proxy exception handler (getattr(e, 'status_code', 500)), the fallback defaulted to HTTP 500 — making intentional guardrail blocks indistinguishable from server errors and causing unnecessary client retries. Changes: - Add status_code=400 (keyword-only) to GuardrailRaisedException - Add status_code=400 (keyword-only) to BlockedPiiEntityError - Update _is_guardrail_intervention() to recognize both exceptions so downstream loggers record 'guardrail_intervened' instead of 'guardrail_failed_to_respond' - Add 6 unit tests for default/custom status codes and getattr pattern - Strengthen existing blocked-action test with status_code assertion Fixes #24348 --------- Co-authored-by: Michael-RZ-Berri <michael@berri.ai> Co-authored-by: Michael Riad Zaky <michaelr@Mac.localdomain> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> Co-authored-by: Krrish Dholakia <krrish+github@berri.ai> * fix(router/proxy): address Greptile P1+P2 review comments on PR #28161 - router: raise ServiceUnavailableError (503) instead of RouterRateLimitErrorBasic (429) when a specifically-addressed deployment is administratively blocked; 429 misleads retry-enabled clients into spinning forever against a paused model - proxy_server: compute get_fully_blocked_model_names() once before both branches in model_list() instead of duplicating the call in each branch - deepseek: upgrade silent debug log to warning when injecting placeholder reasoning_content so callers are clearly notified of degraded multi-turn quality - tests: update two blocked-deployment assertions to expect ServiceUnavailableError Co-authored-by: Cursor <cursoragent@cursor.com> * fix: address bug detection findings (cache token order, mutable defaults) Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix: address bugs in async pass-through, anthropic cache token detection, rerank tests - async_get_available_deployment_for_pass_through: enforce blocked check on specific deployments - cost_calculator: detect anthropic-style usage by attribute presence (not truthiness) to avoid mixing OpenAI cached_tokens into anthropic normalization when read=0 - dashscope rerank tests: pass request to httpx.Response constructions for consistency Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix code qa * fix(vertex_ai/gemini): strip MIME parameters from GCS contentType GCS object metadata's contentType field can include parameters such as 'text/html; charset=utf-8'. Strip them in _apply_gemini_mime_type_aliases so downstream get_file_extension_from_mime_type sees a bare MIME type. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(vertex_ai/gemini): clarify mime-type error message string concatenation Co-authored-by: Yassin Kortam <yassin@berri.ai> * feat(oci): add embeddings, fix streaming/reasoning, expand model catalog - Add OCIEmbedConfig with full Cohere embed support (7 models, batch up to 96) - Fix sync streaming: split SSE events on \n\n before JSON parsing - Fix reasoning models (Gemini 2.5, xAI Grok): make completionTokens and message optional in OCIResponseChoice to handle max_tokens exhausted on reasoning - Fix compartment_id resolution in chat transform to use resolve_oci_credentials - Fix tool call id: make OCIToolCall.id optional, generate UUID fallback for providers (Google via OCI) that omit it - Add OCI_KEY env var support for inline PEM keys - Fix datetime.utcnow() deprecation in request signing - Expand model catalog: 29 OCI models including Llama 4, Gemini 2.5, xAI Grok, Cohere Command A, and all Cohere embed variants - Add 37 live integration tests: sync/async completions for Meta/Google/xAI/Cohere, sync/async embeddings, tool use across all vendors, streaming, env var auth - Add 23 embed unit tests covering all transform and validation paths * fix(oci): remove dead OCI elif branch in utils.py, align async split_chunks with sync version * test(oci): add unit tests for split_chunks fix and no-duplicate-OCI-branch guard * fix(oci): address remaining bugs from issue #25082 — streaming signed body, Cohere stop sequences, hardcoded defaults - Bug 1: sync and async streaming paths now use signed_json_body when provided instead of re-serializing data with json.dumps() — the OCI RSA-SHA256 signature covers the exact request body bytes, so re-serializing produces an invalid sig - Bug 3: Cohere stop sequences now map to 'stopSequences' (was incorrectly 'stop') - Bug 4: removed hardcoded Cohere defaults (maxTokens=600, temperature=1, topK=0, topP=0.75, frequencyPenalty=0) that silently overrode user intent on every call - Added 6 unit tests covering all three fixes * fix(oci): comprehensive code quality pass — bugs, tests, schema accuracy - Fix Cohere tool call IDs (was always call_0; now UUID per call) - Fix TOOL_CALL finish reason mapping in both sync and streaming paths - Fix Cohere stop parameter mapping (stop → stopSequences) - Remove hardcoded Cohere defaults (maxTokens/topK/topP/frequencyPenalty) - Fix content[0] safety guard against empty content arrays - Fix streaming signed body used consistently (not re-serialized) - Raise OCIError (not bare Exception/ValueError) throughout - Centralize OCI_API_VERSION constant; import uuid at module level - Fix embed get_complete_url to strip trailing slashes from api_base - Fix OCIEmbedResponse schema: add inputTextTokenCounts (actual OCI field) - Fix embed usage computed from inputTextTokenCounts (sum of per-input counts) - Fix Cohere toolCallId included in tool result messages - Add OCIToolCall.id as Optional (absent in Google/xAI streaming chunks) - Update tests to reflect correct behavior (no hardcoded defaults, UUID ids, deferred credential validation, OCIError vs ValueError, real response schema) * test(oci): move integration tests to tests/llm_translation/ Addresses greptile P1: tests/test_litellm/ is for mock-only unit tests (make test-unit target). Real-network OCI tests now live in the correct location alongside other provider integration tests. * fix(oci): align types and transformation with official OCI SDK - Remove OCIVendors.GEMINI — apiFormat="GEMINI" is invalid; all non-Cohere models use apiFormat="GENERIC" - Add toolChoice, logitBias, logProbs to OCIChatRequestPayload so params present in the mapping are no longer silently dropped by Pydantic - Exclude n→numGenerations from Cohere param map (not a Cohere API field) - Fix CohereToolResult: change callId/result to call/outputs matching the OCI SDK's CohereToolResult structure - Fix CohereToolMessage: replace non-existent toolCallId with toolResults list; update adapt_messages_to_cohere_standard to build proper tool-result history entries by resolving tool call name+params from preceding assistant messages - Map generic-model stream finish reasons to OpenAI convention (COMPLETE→stop, MAX_TOKENS→length, TOOL_CALLS→tool_calls), consistent with the existing Cohere streaming path - Add optional id field to OCIEmbedResponse so valid API responses carrying an id are not rejected by the Pydantic model * fix(oci): use 'output' key in Cohere tool result outputs (matches reference impl) * fix(oci): port schema/type utilities from langchain-oracle reference impl - Add resolve_oci_schema_refs: inline $ref/$defs — OCI rejects JSON Schema refs - Add resolve_oci_schema_anyof: flatten Optional[T] anyOf (Pydantic v2 emits these) - Add sanitize_oci_schema: strip title, normalise null types, ensure array items - Add OCI_JSON_TO_PYTHON_TYPES: Cohere expects Python type names (str/int/float), not JSON Schema names (string/integer/number) - Add enrich_cohere_param_description: embed enum/format/range/pattern constraints into description since CohereParameterDefinition has no dedicated fields - Apply all of the above in adapt_tool_definitions_to_cohere_standard and adapt_tool_definition_to_oci_standard - Fix toolChoice conversion: map OpenAI string ('auto','none','required') to OCI dict form ({"type":"AUTO"} etc.) — the API rejects plain strings - Update unit test expectations to match correct Python type names and enriched descriptions * refactor(oci): split transformation.py into cohere.py and generic.py transformation.py was 1 243 lines doing too many jobs. Split along the same boundaries as the langchain-oracle reference (providers/cohere.py, providers/generic.py): chat/cohere.py — Cohere message/tool building, response + stream parsing chat/generic.py — Generic message/tool building, response + stream parsing transformation.py — thin OCIChatConfig orchestrator + OCIStreamWrapper Public symbols (OCIChatConfig, OCIStreamWrapper, adapt_messages_to_*, OCIRequestWrapper, version, …) remain importable from transformation.py for backward compatibility. OCIStreamWrapper gains delegating shims for _handle_cohere_stream_chunk and _handle_generic_stream_chunk so existing test call sites keep working unchanged. transformation.py: 1 243 → 620 lines * refactor(oci): principal-level code quality pass - Remove _extract_text_content duplication — single definition in cohere.py, imported where needed; instance method on OCIChatConfig eliminated - Move cryptography imports to module level with _CRYPTOGRAPHY_AVAILABLE flag and _require_cryptography() guard; no more re-import on every signing call - Move litellm version import to module level via litellm._version; remove inline import inside validate_oci_environment - sign_with_manual_credentials now returns Tuple[dict, bytes] matching sign_with_oci_signer — asymmetry eliminated, Optional[bytes] guards removed throughout stream wrappers (signed_json_body: bytes = b"") - Rename _openai_to_oci_cohere_param_map → openai_to_oci_cohere_param_map for consistency with openai_to_oci_generic_param_map - Remove double-key bug in map_openai_params where responseFormat was stored under both OCI and OpenAI key names simultaneously - Remove delegating shims (adapt_messages_to_cohere_standard, adapt_tool_definitions_to_cohere_standard, _handle_generic_stream_chunk) from OCIChatConfig/OCIStreamWrapper; tests now import directly from cohere.py and generic.py where symbols live - Trim __all__ to 7 genuine public symbols; remove the 13-symbol list that existed only to support test imports - Collapse per-model integration test classes into pytest.mark.parametrize; CHAT_MODELS list is the single source of truth for model-specific config - Black + Ruff clean across all OCI files * fix(oci): address PR review findings - types/llms/oci.py: add "TOOL_CALL" to CohereChatResponse.finishReason Literal so Pydantic does not raise ValidationError on non-streaming Cohere tool-use calls (Greptile P1) - test_oci_cohere_tool_calls.py: add test covering TOOL_CALL finish reason - model_prices_and_context_window.json: remove 6 duplicate oci/cohere.embed-* keys that were silently overridden by the more complete entries already present in the file (Greptile P1) - common_utils.py: move OCI_API_VERSION here from chat/transformation.py so embed/transformation.py does not need to import chat/transformation; change Protocol stub body from ... to pass (CodeQL "statement no effect"); add comment to sha256_base64 clarifying it implements OCI HTTP signing spec, not password hashing (CodeQL false positive) - chat/transformation.py: import CustomStreamWrapper from litellm_core_utils.streaming_handler instead of litellm.utils to reduce import cycle depth (CodeQL cyclic import) - chat/cohere.py, chat/generic.py: import Usage and ChatCompletionMessageToolCall from litellm.types.utils instead of litellm.utils for the same reason - embed/transformation.py: import OCI_API_VERSION from common_utils instead of chat/transformation (removes the embed→chat import edge) * test(oci): add unit tests to improve patch coverage - test_oci_common_utils.py (new): covers sha256_base64, build_signature_string, OCIRequestWrapper.path_url, resolve_oci_credentials, get_oci_base_url, validate_oci_environment, sign_with_oci_signer error paths, sign_oci_request routing, load_private_key_from_file error paths, resolve_oci_schema_refs (including circular ref and external $ref), resolve_oci_schema_anyof, sanitize_oci_schema (all branches), enrich_cohere_param_description - test_oci_generic_chat.py (new): covers content-message error paths (non-dict item, unsupported type, non-string text, invalid image_url), tool-call validation error paths, adapt_messages_to_generic_oci_standard error paths, handle_generic_response (None message, text content, tool calls), handle_generic_stream_chunk (finish reasons, streaming tool calls), OCIStreamWrapper non-string chunk error - test_oci_chat_transformation.py: add error paths for validate_environment (empty messages), transform_request (missing compartment_id, Cohere without user messages), transform_response (error key), map_openai_params (unsupported param with and without drop_params), tool_choice string mapping - test_oci_cohere_tool_calls.py: add edge cases for stream chunk finish reasons (TOOL_CALL, MAX_TOKENS, unknown), _extract_text_content with non-dict list items and non-string input, adapt_messages_to_cohere_standard with malformed JSON tool arguments * fix(oci): rename supports_streaming to supports_native_streaming in model prices The JSON schema for model_prices_and_context_window.json uses `supports_native_streaming` (not `supports_streaming`) and has `additionalProperties: false`. Rename the field across all OCI entries to pass the schema validation test. * test(oci): add 67 tests targeting uncovered happy paths for coverage Boost patch coverage on the four lowest-coverage OCI files: - common_utils.py: sign_with_manual_credentials (oci_key / oci_key_file paths), sign_oci_request routing, _require_cryptography - generic.py: adapt_messages_to_generic_oci_standard (all roles), adapt_tool_definition_to_oci_standard, adapt_tools_to_openai_standard, handle_generic_stream_chunk text/finish-reason paths - cohere.py: _extract_text_content, adapt_messages_to_cohere_standard (all roles including tool results), handle_cohere_response / handle_cohere_stream_chunk all finish-reason branches - transformation.py: get_vendor_from_model, OCIChatConfig._get_optional_params (toolChoice string→dict, responseFormat, tools for both vendors), transform_request for GENERIC model, get_sync/async_custom_stream_wrapper with mocked HTTP, OCIStreamWrapper.chunk_creator happy paths * fix(oci): suppress CodeQL false positive on sha256_base64 (OCI HTTP signing, not password hashing) * fix(oci): remove 6 duplicate model price entries and reconcile conflicting values Six OCI chat model keys appeared twice in model_prices_and_context_window.json with conflicting pricing/context data (JSON parsers silently discard the first). Remove the first-occurrence entries and update the surviving entries: - meta.llama-4-maverick / llama-4-scout: keep updated entries (free preview pricing, larger context windows, vision support) - meta.llama-3.1-70b: keep original pricing, restore supports_native_streaming - google.gemini-2.5-{flash,pro,flash-lite}: keep OCI pricing page values, restore supports_native_streaming * fix(oci): route GPT-5 family to maxCompletionTokens GPT-5 / GPT-5-mini / GPT-5-nano / GPT-5.5 on OCI reject "maxTokens" with HTTP 400: Invalid 'maxTokens': Unsupported parameter: 'maxTokens' is not supported with this model. Use 'maxCompletionTokens' instead. (Same convention as OpenAI's reasoning-API contract.) Add a model-aware rename in OCIChatConfig._get_optional_params so the request payload uses maxCompletionTokens when the model id starts with openai.gpt-5. Regular Llama / Cohere / Gemini / GPT-4.x continue to use maxTokens unchanged. Also widen OCIChatRequestPayload to carry the new optional field so it survives Pydantic serialization. Verified live against OCI us-chicago-1: - openai.gpt-5, gpt-5-mini, gpt-5-nano, gpt-5.5 all return 200 - Full feature sweep on gpt-5.5 (basic, system, multi-turn, streaming, tools, usage) all green - meta.llama-3.3-70b-instruct still uses maxTokens (no regression) 4 new unit tests cover the helper, the routing in both pre- and post-translation states, and Pydantic serialization. * ci(oci): fix CI failures — black formatting + recursive_detector ignore - Run black on litellm/llms/oci/common_utils.py + 3 OCI test files that drifted out of black-compliance during the rebase. - Add the three bounded recursive functions in oci/common_utils.py (`_resolve`, `resolve_oci_schema_anyof`, `sanitize_oci_schema`) to the recursive_detector IGNORE_FUNCTIONS list. All three are bounded: `_resolve` uses a `resolving_stack` cycle guard; the other two are bounded by JSON-schema tree depth (no cycles in well-formed input), matching the pattern of the existing OCI/Vertex schema walkers already on the list. * fix(oci): silence MyPy errors in cohere.py — typed-dict access Two errors flagged by `lint` CI: llms/oci/chat/cohere.py:73: "object" has no attribute "__iter__" llms/oci/chat/cohere.py:119: No overload variant of "get" of "dict" matches argument types "object", "CohereToolCall" Both stem from `msg.get("tool_calls")` / `msg.get("tool_call_id")` returning `object` per the AllMessageValues TypedDict union. Bind to `Any` locally for the iteration and coerce the lookup key with `str()`, removing the now-unused `# type: ignore` on those lines. No behaviour change — pure type-narrowing for the type checker. * fix(oci): silence CodeQL py/weak-sensitive-data-hashing on sha256_base64 CodeQL's taint analysis traces request bodies back to environment-loaded secrets and flags `hashlib.sha256(body).digest()` as `py/weak-sensitive-data-hashing` — even though SHA-256 is the algorithm mandated by the OCI HTTP request signing spec for the `x-content-sha256` header (not a password/secret hash). The previous suppression used legacy `# lgtm[...]` syntax which the modern CodeQL action ignores. Switch to Python's standard `hashlib.sha256(..., usedforsecurity=False)` (Python 3.9+) which CodeQL honours as a non-security declaration. Behaviour unchanged. * feat(oci): add reasoning_effort passthrough — only true missing primitive OCI's GenericChatRequest exposes a reasoningEffort field (NONE/MINIMAL/LOW/MEDIUM/HIGH) that's the single biggest cost knob for reasoning-capable models on the service: - GPT-5 family - Gemini 2.5 - Grok reasoning variants (3-mini, 4-fast, 4.20) - Cohere Command-A-Reasoning Setting reasoning_effort=LOW typically cuts reasoning-token spend 5-10× vs the default. Without exposing this, litellm users had no way to tune cost-vs-quality on these models. The other GenericChatRequest fields (verbosity, parallel_tool_calls, logit_bias, n, metadata, web_search_options, prediction) are not exposed because they are not missing primitives — they either duplicate prompt-engineering, framework-level controls, or are too niche to justify the maintenance surface. We only ship what users genuinely can't accomplish another way. Excluded from the Cohere v1 param map: CohereChatRequest has no reasoningEffort field, and Cohere reasoning models (cohere.command-a-reasoning) use COHEREV2 which is a separate request type not covered by this PR. Verified live: GPT-5.5 + reasoning_effort="HIGH" sends {"reasoningEffort": "HIGH"} on the wire and OCI accepts the request. * feat(oci): reasoning_effort + reasoning_tokens for OCI GenAI Three small additions for OCI reasoning models, requested by users testing the PR in production fork builds: 1. **reasoning_effort param mapping (GENERIC vendors).** OCI expects uppercase levels ("LOW"/"MEDIUM"/"HIGH"/"NONE") on `reasoningEffort`, but OpenAI-compatible clients send lowercase. Mapped + uppercased in `_get_optional_params`. Marked unsupported on Cohere V1/V2 since OCI Cohere has no reasoning models (avoids Pydantic validation failure on CohereChatRequest). 2. **"disable" → "NONE" mapping.** OpenAI uses "disable" to turn off reasoning; OCI uses "NONE". Without this, callers get a 400. 3. **reasoning_tokens propagated to Usage.** OCI returns `completionTokensDetails.reasoningTokens` but it wasn't being passed to LiteLLM's Usage object. Now flows through to `Usage.completion_tokens_details.reasoning_tokens` so callers can track reasoning token consumption for cost/observability. Tests: 7 new unit tests in TestOCIReasoningEffort covering upper/lower case, "disable"→"NONE", Cohere drop/raise paths, and reasoning_tokens extraction (with and without completionTokensDetails). 5 new live integration tests against xai.grok-3-mini in us-chicago-1 verifying the full request/response loop end-to-end. Existing test_transform_response_simple_text assertion that completion_tokens_details was None has been updated to assert reasoning_tokens flows through. Verified live on xai.grok-3-mini: reasoning_effort=low → OCI accepts "LOW", returns reasoningTokens=316 in usage. reasoning_effort=disable → OCI accepts "NONE". Full suite: 370/370 unit + 51/51 integration. * fix(codeql): re-scope py/weak-sensitive-data-hashing exclusion to OCI signing file CodeQL's taint analysis re-fires the `py/weak-sensitive-data-hashing` alert at `litellm/llms/oci/common_utils.py:103` whenever upstream code paths into the OCI signing module change (touching `transformation.py` opens new flow paths that CodeQL re-evaluates from scratch). The `hashlib.sha256(..., usedforsecurity=False)` declaration silences the direct-call form of the query but not the taint-flow form. SHA-256 here is mandated by the OCI HTTP signing specification for the x-content-sha256 content-integrity header — not for password storage: https://docs.oracle.com/en-us/iaas/Content/API/Concepts/signingrequests.htm CodeQL has no per-query path filter and GitHub Code Scanning ignores inline lgtm/codeql comments, so path-ignoring this single ~560-line signing utility file is the narrowest available suppression. All other files retain full coverage of py/weak-sensitive-data-hashing — including litellm/proxy/utils.py where the rule legitimately applies. This restores the NEUTRAL CodeQL state the PR had on prior commits (see `2111c98af7` for the same approach on the previous branch evolution that the cherry-pick was rebased onto a different baseline). * fix(oci): drop duplicate text on Cohere streaming terminal chunk OCI Cohere's terminal SSE event re-sends the full assembled response in `text` alongside a populated `chatHistory`. Emitting that text as another delta concatenates the entire response onto the already-streamed output (e.g. "How can I help?How can I help?"). Use `chatHistory is not None` as the discriminator for the consolidated terminal event — `finishReason` is a weaker signal that could in principle appear on a non-consolidated chunk. The two coincide today; this preserves correctness if OCI ever ships finishReason on an incremental chunk. Adds a live-OCI integration regression test that compares streamed vs non-streamed length and asserts the response prefix appears only once. Verified to fail under the previous code with the exact reported reproduction: 'Hello! How can I help you today?Hello! How can I help you today?'. Reported by @gotsysdba on PR #25177. * fix(oci): buffer SSE stream across HTTP read boundaries The old split_chunks helper split each individual HTTP read on "\n\n", which assumed SSE event boundaries always aligned with read boundaries. In practice the OCI streaming endpoint delivers events that may: - straddle two reads (chunk_creator gets a truncated JSON and crashes) - arrive separated by a single "\n" instead of "\n\n" - share a read with multiple complete events Replace the inline split with module-level helpers _iter_sse_events (sync) / _aiter_sse_events (async) that maintain a buffer across reads, split on any newline, and yield only complete "data:" lines. Add 25 regression tests covering event-split-across-reads, tiny-chunk reads, single-newline separators, keepalive/comment lines, trailing partial events flushed at EOF, "\r\n" line endings, and an end-to-end smoke test that feeds an awkwardly-chopped payload through the splitter into OCIStreamWrapper.chunk_creator. Reported by John Lathouwers. * test(oci): repoint TestOCIKeyNormalization to sign_with_manual_credentials The signing helper moved from OCIChatConfig._sign_with_manual_credentials to a module-level sign_with_manual_credentials in common_utils.py. Four tests in TestOCIKeyNormalization still called the old method: - 2 failed outright with AttributeError - 2 passed by accident because they used pytest.raises(Exception), which happily caught the AttributeError instead of exercising the intended OCIError path Repoint all four to the new module-level function so they exercise the actual oci_key type-validation branch. * fix(oci): validate oci_region before URL interpolation to prevent SSRF Anchor oci_region to ^[a-z][a-z0-9-]{0,30}[a-z0-9]$ inside get_oci_base_url so user-supplied regions that would redirect the signed request to an attacker-controlled host (e.g. 'evil.com/#') fail with HTTP 400 before the URL or signature is built. Empty string still falls back to the us-ashburn-1 default, so existing callers are unaffected. * test(audio): skip when gpt-4o-audio-preview is unavailable upstream OpenAI retired `gpt-4o-audio-preview` (404 model_not_found in CI as of 2026-05-19), and the existing try/except in these tests only re-raised on 'openai-internal' errors. Other exceptions were silently swallowed, so the next line ran with an unbound `response`/`completion` and failed with an unrelated UnboundLocalError that masked the real cause. Extend the skip condition to also cover model_not_found / 'does not exist' so the suite reports the upstream outage cleanly, matching the pattern used in ce87c41 for the realtime and nvidia_nim rerank tests. Re-raise unknown exceptions instead of falling through. * fix(oci/router): catalog-driven maxCompletionTokens; generic blocked-deployment message - Drive OCI maxCompletionTokens via supports_reasoning from the model catalog instead of a hardcoded openai.gpt-5 prefix. Add OCI GPT-5 family entries (gpt-5, gpt-5-mini, gpt-5-nano) with supports_reasoning: true. Gate the override to non-Cohere vendor so Cohere reasoning models keep maxTokens (Cohere endpoint does not accept maxCompletionTokens). - Replace proxy-specific 'Contact your proxy admin' phrasing in the four Router blocked-deployment ServiceUnavailableError messages with neutral SDK-appropriate text. * fix(oci/cohere): guard handle_cohere_response against missing usage * fix(oci): address bug review findings in chat transformation - Cohere param map: keep tool_choice/n as False (not omitted) so unsupported params are dropped or rejected rather than silently passed through. - get_complete_url: when an explicit api_base/litellm.api_base is provided, use it as-is instead of unconditionally appending /20231130/actions/chat (mirrors the embed config behavior). - Cohere stream: require both chatHistory and finishReason to be present to identify a terminal consolidation chunk, avoiding silent text suppression if chatHistory ever appears on a non-terminal chunk. - Generic usage: use 'is not None' for reasoningTokens so a legitimate value of 0 is preserved instead of being treated as absent. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci/cohere): emit tool calls in streaming and null content when text empty handle_cohere_response now sets message.content to None when the Cohere response text is empty, matching the OpenAI convention for tool-call-only responses. handle_cohere_stream_chunk now extracts toolCalls — both directly from the chunk and from the terminal chunk's chatHistory CHATBOT message — and emits them in the delta. Previously, CohereStreamChunk lacked a toolCalls field, so any tool calls in the stream were silently dropped. * fix(oci): preserve tool results, embed URL path, and generic finish reason - Use SerializeAsAny on CohereChatRequest.chatHistory so subclass-specific fields like CohereToolMessage.toolResults are not dropped during Pydantic v2 serialization. - Make OCIEmbedConfig.get_complete_url append the /20231130/actions/embedText action path consistently with chat, so setting litellm.api_base to the region inference base URL no longer posts to the bare hostname. - Map OCI finishReason (COMPLETE / MAX_TOKENS / TOOL_CALLS) to OpenAI finish_reason values in handle_generic_response, mirroring the streaming handler and the Cohere non-streaming handler. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci/generic): silence mypy assignment error on dynamic finish_reason * fix(oci/embed): always set usage on embedding response Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci/chat): append /20231130/actions/chat to explicit api_base Restore the embed-style behavior so OCIChatConfig.get_complete_url always appends the OCI GenAI chat path. Routing through get_oci_base_url ensures the optional explicit api_base has its trailing slash stripped before the suffix is joined, matching the embed config and the test_respects_explicit_api_base expectation. * fix(oci/cohere): mark logprobs/logit_bias unsupported and normalize unknown stream finish reasons Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci/cohere): preserve trailing tool result in chatHistory When the last message in the OpenAI-format input is a tool result (the standard agentic continuation pattern), the prior messages[:-1] slice silently dropped that tool result from chatHistory and the model never saw it. Excluding the last user message by index instead keeps tool results that trail the last user turn intact. * fix(main): remove dead OCI embedding elif block The earlier elif at line 5119 already routes OCI embeddings through the base HTTP handler with the headers None-guard, so the later identical block was unreachable dead code. * test(oci): move integration tests out of llm_translation mock-only folder Greptile flags tests/llm_translation/ as mock-only via a project-specific rule; relocate the live-network OCI integration suite to tests/integration/ and adjust the in-file sys.path / run instructions accordingly. * fix(oci/cohere): suppress tool calls on stream terminal consolidation chunk The terminal SSE event re-sends the full assembled response in both `text` and `chatHistory`. The existing logic already suppresses `text` to avoid double-emit, but tool calls extracted from the terminal chunk (via `typed_chunk.toolCalls` or the `chatHistory` CHATBOT fallback) would still be re-emitted with fresh uuid4 IDs. If OCI Cohere ever streams tool calls progressively in intermediate chunks (now possible since CohereStreamChunk has a toolCalls field), this would cause downstream agentic frameworks to execute each tool call twice. Suppress tool calls on the terminal consolidation chunk for the same reason `text` is suppressed. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci,httpx): normalize finish_reason, preserve response_format, fix sync embed JSON content-type - cohere.py / generic.py: normalize unknown OCI finishReason values (ERROR, ERROR_TOXIC, CONTENT_FILTERED, USER_CANCEL, ...) to 'stop' in non-streaming and streaming generic handlers, matching the streaming Cohere handler so downstream consumers switching on finish_reason aren't broken by raw OCI values. - transformation.py: restore the dual-key alias so optional_params still carries the original 'response_format' key alongside the OCI-mapped 'responseFormat'. Downstream litellm framework code (json_mode detection, logging) inspects 'response_format' after map_openai_params runs. - llm_http_handler.py: make the sync embedding path mirror the async path — when sign_request returns no signed_body, send via json=data (which sets Content-Type: application/json) instead of data=json.dumps(data) which doesn't. Removes a sync/async behavioural asymmetry for non-OCI providers that adopt the sign_request pattern. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci): clean up OCIChatConfig init, normalize generic stream finish reasons, correct embed sign_request return type - Replace fragile setattr(self.__class__, ...) pattern in OCIChatConfig.__init__ with a @property for has_custom_stream_wrapper, matching the pattern used by other providers. - Normalize unknown OCI finish reasons (e.g. ERROR, ERROR_TOXIC, USER_CANCEL) to 'stop' in handle_generic_stream_chunk, matching the existing Cohere stream handler behaviour. - Tighten OCIEmbedConfig.sign_request return type from Tuple[dict, Optional[bytes]] to Tuple[dict, bytes] — sign_oci_request never returns None for the body, and this matches OCIChatConfig.sign_request. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci): strip trailing action path in get_oci_base_url to avoid URL doubling A fully-formed OCI endpoint URL (e.g. https://inference.generativeai.us-chicago-1.oci.oraclecloud.com/20231130/actions/chat) passed via api_base previously had the action path appended a second time by get_complete_url in both chat and embed configs, yielding a 404. get_oci_base_url now strips a trailing /20231130/actions/<name> so callers can always append the action path safely. * fix(httpx): preserve sync embed data= kwarg to avoid breaking mock-based tests The earlier sync_httpx_client.post() call passed data=json.dumps(data), which downstream embedding tests assert on (e.g. tests for hosted_vllm, jina_ai, watsonx). Switching to json=data changed the kwarg name and broke those tests. The OCI signed_body path keeps using data=signed_body and is unaffected. * fix(oci): stable tool-call ids across stream chunks; lenient Cohere finishReason - Replace random uuid4 per chunk with a deterministic content-derived digest for synthetic tool-call ids in both Cohere and Generic OCI handlers. Previously, when OCI omitted 'id' (always for Cohere, often for Generic streaming deltas), every chunk for the same logical tool call received a new uuid, causing downstream stream-mergers (which key off id) to treat each fragment as a distinct call. - Relax CohereChatResponse.finishReason from a strict Literal[...] to Optional[str], matching CohereStreamChunk.finishReason. The handle_cohere_response 'elif oci_finish_reason is not None' fallback was previously unreachable because Pydantic raised ValidationError on any unknown value before the fallback executed. Now non-streaming responses degrade unknown reasons to 'stop' just like the streaming path. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci/embed): validate OCI credentials in validate_environment Mirror OCIChatConfig.validate_environment so embedding requests fail fast with a clear error when oci_user/oci_fingerprint/oci_tenancy/ oci_compartment_id or an oci_key/oci_key_file is missing, instead of deferring the failure until sign_request. Co-authored-by: Yassin Kortam <yassin@berri.ai> * test(oci/embed): expect OCIError from validate_environment when credentials are missing OCIEmbedConfig.validate_environment now raises eagerly (mirroring OCIChatConfig) when oci_user/oci_fingerprint/oci_tenancy/oci_compartment_id or oci_key/oci_key_file is missing. Update the test to match. * fix(oci): polish stream chunk handling and signed body default - cohere stream terminal consolidation now emits content=None instead of "" - drop redundant index truthiness check (None is already replaced with 0) - accept both "TOOL_CALL" and "TOOL_CALLS" finish reasons in cohere - signed_json_body defaults to None and uses explicit None check, so an explicitly empty bytes body wouldn't be silently re-serialized Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci/chat): catch pydantic ValidationError when parsing OCI responses Pydantic v2 raises ValidationError (not TypeError) when field validation fails, so malformed OCI completion responses or stream chunks would propagate unhandled out of handle_generic_response, handle_generic_stream_chunk, and handle_cohere_stream_chunk. Widen the except clauses to also catch ValidationError so callers get a clean OCIError. * fix(oci/catalog): real prices for Llama 4, drop zero-cost OCI OpenAI entries Zero-cost catalog entries (input_cost_per_token=0, output_cost_per_token=0) make proxy spend tracking silently report $0 for these paid OCI models, so any caller can drive them without decrementing a budget. For Llama 4 Maverick and Scout, OCI charges the same character-based rate as Llama 3.3 70B ($0.0018 per 10,000 characters), so use the same per-token price as the existing oci/meta.llama-3.3-70b-instruct entry (7.2e-07 in/out). For oci/openai.gpt-5, gpt-5-mini, gpt-5-nano, gpt-oss-120b, and gpt-oss-20b, no public per-token pricing is available; drop the entries so operators must register them with explicit custom pricing. The existing GPT-5 reasoning test fixture already injects synthetic entries when the catalog omits them, so the chat transformation's supports_reasoning lookup keeps working in tests. * fix(oci/chat): wrap CohereChatResult construction in try/except Match the handle_generic_response pattern: surface OCIError with the upstream status code instead of letting a raw pydantic.ValidationError propagate when the Cohere response payload is malformed. * fix(oci): harden Cohere stream/finish-reason and dedupe maxTokens param mapping - Cohere stream: track per-stream tool-call emission and only suppress the terminal consolidation chunk's tool calls once they've been seen earlier. Prevents silent drop if tool calls are delivered exclusively on the terminal chunk. - Cohere stream: emit content=None (not "") on non-terminal text-free chunks (e.g. tool-call-only / keep-alive) so downstream consumers that distinguish missing vs explicitly-empty deltas behave correctly. - Generic handlers: accept singular TOOL_CALL finish reason in addition to TOOL_CALLS, matching the Cohere handlers. - _get_optional_params: when both max_tokens and max_completion_tokens are provided, explicitly prefer max_completion_tokens instead of relying on dict iteration order. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci): emit content=None instead of empty string for text-free generic stream chunks Co-authored-by: Yassin Kortam <yassin@berri.ai> * test(oci): expect content=None for text-free generic stream chunks handle_generic_stream_chunk now emits content=None instead of empty string when a chunk carries no text parts. Update the corresponding no-message test to match. * codeql: narrow OCI sha256 suppression to query-filter, not whole file paths-ignore was suppressing every CodeQL query on litellm/llms/oci/common_utils.py, hiding all future findings in a security-critical file (private key loading, credential resolution, URL construction, RSA signing). Move the suppression for py/weak-sensitive-data-hashing into query-filters so common_utils.py remains fully analyzed by every other query. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci): use locale-independent RFC 7231 date for manual signing email.utils.formatdate(usegmt=True) emits canonical English weekday/ month abbreviations regardless of system locale, so signature verification doesn't break on non-en_US deployments. * fix(oci): strip 'oci/' prefix in get_vendor_from_model Previously, get_vendor_from_model split on '.' without stripping the optional 'oci/' provider prefix, so 'oci/cohere.command-a-03-2025' was routed through the GENERIC pipeline instead of COHERE. Co-authored-by: Yassin Kortam <yassin@berri.ai> * codeql: scope OCI sha256 suppression to common_utils.py via filter-sarif Replace the global query-filters exclude for py/weak-sensitive-data-hashing with a SARIF post-filter that only drops the alert when it originates from litellm/llms/oci/common_utils.py, keeping the rule active on every other SHA-256 callsite in the repository. * Fix OCI chat bugs: tool_calls None key, dead max_tokens dedup, single-event stream text suppression - handle_cohere_response: omit tool_calls key from message dict when None, matching the generic handler's behaviour and avoiding tripping consumers that key off 'tool_calls' in message. - _get_optional_params: remove dead prefer_max_completion branch. By the time this helper runs, map_openai_params has already collapsed max_tokens/max_completion_tokens onto the OCI alias, so the OpenAI-key membership check is unreachable. - handle_cohere_stream_chunk: add prior_text_emitted parameter mirroring prior_tool_calls_emitted. The terminal consolidation chunk's text is only suppressed when prior deltas already emitted text — otherwise (degenerate single-event stream) the text passes through so the response content isn't silently lost. OCIStreamWrapper now tracks emitted text alongside emitted tool calls. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci): preserve all text parts in generic response and emit SYSTEM role for Cohere - handle_generic_response: iterate all content parts and concatenate text (matches the streaming handler) so non-leading text parts are not lost and a leading non-text part does not suppress trailing text. - adapt_messages_to_cohere_standard: emit CohereSystemMessage for system messages so direct callers do not silently drop them. The Cohere request builder filters system messages before calling this helper to avoid duplicating preambleOverride content into chatHistory. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci): normalise dict-format tool_choice to OCI flat uppercase shape The OCI Generative AI API only accepts toolChoice values of the form {"type": "AUTO"|"NONE"|"REQUIRED"} or {"type": "FUNCTION", "name": "<fn>"}. The previous conversion only handled string tool_choice values, so OpenAI's standard dict shape {"type": "function", "function": {"name": "<fn>"}} passed through unchanged and was rejected by OCI with a 400. Normalise the dict shape by uppercasing the discriminator and hoisting the function name to the top level. Also accept dict variants of the non-function selectors (e.g. {"type": "auto"}). * test(oci): exercise system-message filtering at transform_request boundary adapt_messages_to_cohere_standard now emits SYSTEM-role entries by design so direct callers don't silently drop system content. The Cohere request builder filters system messages before calling the helper and routes them into preambleOverride, so the user-visible 'no SYSTEM in chatHistory' guarantee holds at the transform_request boundary, where the test should live. * fix(oci/chat): extract tool_choice/response_format helpers to satisfy PLR0915 _get_optional_params exceeded ruff's 50-statement cap. The toolChoice and responseFormat normalisation blocks are self-contained mutations, so move them to module-level helpers. * fix(oci): normalize None finishReason in generic non-streaming handler; drop dead Cohere system-role branch Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci/generic): silence mypy assignment error on cleared finish_reason * fix(docker): install libatomic in builder for prisma nodeenv binary The prebuilt node binary that prisma-python's nodeenv downloads links against libatomic.so.1, which Wolfi does not pull in via gcc/nodejs. Without this, fresh Docker builds (no GHA cache hit) fail at `prisma generate` with: node: error while loading shared libraries: libatomic.so.1 * fix(oci): raise on invalid tool_choice instead of silently passing OpenAI shape _normalize_tool_choice previously left an OpenAI-format dict in selected_params['toolChoice'] when the type was unrecognized or when 'FUNCTION' was given with a missing/empty name. OCI would then reject the request with a non-obvious error. Raise ValueError with a clear message in these cases. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci): raise OCIError instead of ValueError in _normalize_tool_choice Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci/generic): declare non-security intent on sha256 for synthetic tool-call id * fix(oci): simplify _get_optional_params and reject invalid tool_choice types - Collapse the two-loop _get_optional_params into a single pass with clear precedence (OpenAI key wins over OCI alias; first OpenAI key reaching a given OCI target wins). Removes the redundant maxTokens special-case in the second loop and makes the map_openai_params / transform_request handoff easier to reason about. - Raise OCIError when _normalize_tool_choice sees an unexpected type (list, bool, int, ...) instead of silently letting it through to the OCI API where it would produce an opaque server-side error. Co-authored-by: Yassin Kortam <yassin@berri.ai> * Remove no-op data['stream'] deletion in OCI stream wrappers Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci): always send Cohere isStream field explicitly Match OCIChatRequestPayload by defaulting CohereChatRequest.isStream to False instead of None so model_dump(exclude_none=True) does not silently omit the field on non-streaming requests. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci): revert Cohere isStream to Optional[bool]=None to preserve omission semantics Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci/generic): raise OCIError on empty choices instead of IndexError Pydantic accepts an empty choices list when validating OCICompletionResponse, so accessing chatResponse.choices[0] could raise an unhandled IndexError. Surface it as OCIError so the response error path is consistent with the existing (TypeError, ValidationError) guard. * fix(oci/cohere): map top_k -> topK so Cohere topK param is settable The Cohere param map (derived from the GENERIC map) had no entry for topK. Since the simplified _get_optional_params only iterates over param_map entries, callers had no way to pass topK to CohereChatRequest (neither via an OpenAI-style key nor via the OCI alias). Add 'top_k': 'topK' to the Cohere map only — OCIChatRequestPayload (GENERIC) has no topK field. _get_optional_params accepts both the OpenAI key (top_k) and the OCI alias (topK) in optional_params, so this covers both calling conventions. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci): tighten cohere stream dedup flags and forward stream args in embed signing Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci/chat): reorder dict guard and wrap stream chunk json.loads - Move isinstance(response_json, dict) check before .get("error") so the guard runs before the attribute access it is supposed to protect. - Wrap json.loads in OCIStreamWrapper.chunk_creator with try/except so malformed SSE payloads surface as OCIError instead of a raw JSONDecodeError propagating out of the stream loop. * fix(oci/cohere stream): only flag text emitted on non-empty content An intermediate Cohere SSE chunk carrying text="" was flipping _cohere_text_emitted via the "is not None" check, which then caused the terminal consolidation chunk to drop its real text as a duplicate. Use a truthy check so only actual content marks the stream as having emitted text. * test(oci): end-to-end proxy integration test against real OCI GenAI Spins up the litellm proxy via the console-script entrypoint with a minimal OCI-only config and drives real OpenAI-shaped HTTP requests through it against OCI GenAI. Covers non-streaming chat, streaming chat, embeddings, and /v1/models for Cohere, Llama, Gemini, and Grok. Skips automatically when ~/.oci/config is absent or when the active profile uses session-token auth (the OCI provider currently only consumes OCI_* env vars; session tokens would need an in-process signer). API-key profiles work out of the box. * test(oci): move proxy integration test to tests/integration/ tests/llm_translation/ is mock-only; the OCI proxy integration test spawns a real proxy subprocess and makes live HTTP calls, so move it (and the companion config) to tests/integration/ alongside the existing test_oci_integration.py. * fix(oci): dedupe finish-reason mapping and batch Cohere tool results - Extract _normalize_oci_finish_reason helper so the four chat handlers (Cohere/GENERIC, sync/stream) share one OCI->OpenAI mapping instead of four near-identical if/elif chains. - Merge consecutive OpenAI tool-role messages into a single CohereToolMessage with multiple toolResults entries, matching the OCI Cohere API's expectation for parallel tool calls in one assistant turn. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(oci): drop dead Cohere toolChoice field and emit GENERIC tool-call dicts inline - Remove the unreachable toolChoice field from CohereChatRequest. The Cohere param map explicitly marks tool_choice as unsupported, so the field can never be populated through the normal optional_params flow and only confused the public model surface. - Build GENERIC stream tool-call dicts inline (id/type/function shape) instead of round-tripping through ChatCompletionMessageToolCall and model_dump(). Matches handle_cohere_stream_chunk so downstream stream-mergers see the same minimal payload regardless of which vendor produced the chunk. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(docker): drop redundant libatomic from non_root builder litellm_internal_staging already fixes the prisma `nodeenv` build failure at the root cause by restoring `npm` to the builder (#28519): with npm on PATH, prisma-python uses the system Node and never downloads the nodeenv binary that links against libatomic.so.1. After merging internal_staging the libatomic line is dead weight, so remove it. https://claude.ai/code/session_01SwKzxRxgUhLFyyEf4UV812 * fix(oci/catalog): add openai.gpt-5{,-mini,-nano} entries with supports_reasoning Without these catalog entries, supports_reasoning(model='openai.gpt-5*', custom_llm_provider='oci') returned False, so _model_uses_max_completion_tokens fell back to the default and OCI rejected the request with HTTP 400 ('Use maxCompletionTokens instead.'). Add the three entries so the catalog-driven maxCompletionTokens routing works against a stock LiteLLM install. Also reword the test fixture docstring — the bundled backup now actually ships these entries, so the fixture is only a fallback for environments that loaded their cost map from a stale remote source. --------- Co-authored-by: Tai An <antai12232931@outlook.com> Co-authored-by: Vincent <yimao1231@gmail.com> Co-authored-by: Kris Xia <xiajiayi0506@gmail.com> Co-authored-by: d 🔹 <liusway405@gmail.com> Co-authored-by: Fabrizio Cafolla <developer@fabriziocafolla.com> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Tom Denham <tom@tomdee.co.uk> Co-authored-by: escon1004 <70471150+escon1004@users.noreply.github.com> Co-authored-by: Divyansh Singhal <97736786+Divyansh8321@users.noreply.github.com> Co-authored-by: robin-fiddler <robin@fiddler.ai> Co-authored-by: Michael-RZ-Berri <michael@berri.ai> Co-authored-by: Michael Riad Zaky <michaelr@Mac.localdomain> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> Co-authored-by: Krrish Dholakia <krrish+github@berri.ai> Co-authored-by: Sameer Kankute <sameer@berri.ai> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: Federico Kamelhar <federico.kamelhar@oracle.com> Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Michael Riad Zaky <michaelr@Michaels-MacBook-Air.local> Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
…aming hot paths (BerriAI#28289) * perf: reduce per-request and per-chunk overhead across Anthropic streaming hot paths - Introduce pure-text fast-path in `_build_complete_streaming_response` that collapses O(N) `content_block_delta` events into a single equivalent SSE event before conversion, eliminating per-output-token Pydantic `ModelResponseStream` construction; non-text streams (tool_use, thinking, citations) fall back to the unchanged legacy path - Skip agentic streaming wrapper entirely when no callback overrides `async_should_run_agentic_loop`; the wrapper buffered every chunk and rebuilt the SSE response only to call hooks that all return `(False, {})` — a pure no-op for the default config - Serialize request body once (`json.dumps`) for both the pre-call log input and the wire, instead of twice; avoids a full O(payload) scan per request, significant for long-context Claude Code histories - Add fast path in `async_streaming_data_generator` that bypasses the per-chunk `async_post_call_streaming_hook` coroutine await, response-string materialization, and cost-injection call when no callback/guardrail/cost-injection is active (the default config) - Resolve `_DD_STREAMING_TRACE_ENABLED` once at import time; eliminate per-chunk `NullSpan` context manager allocation when Datadog tracing is disabled (the default) - Memoize `get_type_hints(AnthropicMessagesRequestOptionalParams)` with `@lru_cache(maxsize=1)` — resolves once per process instead of once per `/v1/messages` request (~80µs each) - Hoist `cost_injection_active` out of the per-chunk loop in `chunk_processor`; eliminates repeated `getattr` + endpoint-type checks on every streamed byte chunk - Extract `_build_passthrough_logging_result` from `_route_streaming_logging_to_handler` as a standalone static method to facilitate future off-loop dispatch - Convert `async_sse_data_generator` from an `async for: yield` trampoline to a direct return of the underlying generator, removing one async-generator layer per streamed chunk - Skip redundant `strip_empty_text_blocks_from_anthropic_messages` scan in `anthropic_messages_handler` when the async wrapper already sanitized (signalled via `_litellm_messages_presanitized` sentinel, popped before reaching provider params) - Gate debug log `f-string` evaluation behind `isEnabledFor(DEBUG)` in both the streaming generator and the transformation layer to avoid serializing entire message payloads on every request at non-debug log levels - Add benchmark script (`scripts/benchmark_anthropic_messages_perf.py`) with a local mock Anthropic SSE provider for reproducible TTFT and TPM measurement across commits/branches - Add parity tests asserting fast-path and legacy-path produce byte-identical logged/billed payloads, plus unit tests for agentic hook detection, pre-serialized body reuse, and memoized key resolution * perf: address greptile review for anthropic streaming hot path - Bail to legacy in `_collapse_pure_text_chunks` when content_block_delta events from different block indexes are observed without an intervening flush. Anthropic sends blocks strictly sequentially, but defensive bail prevents silent text-merging if the protocol ever interleaves. - Replace leaf-class `__dict__` check for `async_post_call_streaming_hook` in `_callback_capabilities` with a function-identity comparison that walks the MRO. A vendor base class can carry the override and the registered class can add nothing else; before this PR the hook was unconditionally invoked, so an inherited-override miss would silently drop the hook on the streaming path. - Add unit tests for both behaviors. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(mypy): narrow model_name to str in cost-injection branch The hoisted cost_injection_active flag in chunk_processor encodes the `bool(model_name)` requirement but mypy can't track that invariant through the local, so the per-chunk `_process_chunk_with_cost_injection( chunk, model_name)` calls flagged Optional[str] vs str. Pin a typed non-None local inside the cost-injection branch so mypy narrows correctly without changing runtime behavior. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: oss-agent-shin <279349115+oss-agent-shin@users.noreply.github.com> Co-authored-by: ishaan-berri <ishaan-berri@users.noreply.github.com>
… management endpoints (BerriAI#28681) * test(proxy): phase-4 payload behavior pinning for tier-2/3 key + team management endpoints Extends the Phase 1–3 behavior-pin suite at tests/proxy_behavior/management/ with a second axis: payload-shape pinning. Phase 1–3 held payload minimal and pinned (actor, target) → status across 37 routes; Phase 4 holds the caller fixed at an authorized actor, varies the payload shape, and asserts the observable DB effect (on accept) or the named guard / row-unchanged (on reject). Faithfulness contract from Phase 1–3 is unchanged. Six families + one gap-closer (59 new scenarios, 620 → 679 total): * F1 — key budget / rate-limit (test_key_budget_limits.py, 18) * F2 — key↔team reassignment (test_key_team_change.py, 6) * F3 — team budget / rate-limit (test_team_budget_limits.py, 15) * F4 — member-info validation (test_team_member_info_validation.py, 5) * F5 — permission batching (test_team_permissions_bulk_update.py, 6) * F6 — org-scoped team access (+2 detail-string pins in existing files) * F7 — coverage gap-closer (test_f7_coverage_closeout.py, 7) Harness extensions in conftest.py (additive only): * create_scratch_org() seeder with its own scratch-prefixed budget row * budget / limit fields on create_scratch_team() * scratch teardown also sweeps litellm_organizationtable Coverage telemetry (behavior-suite-only): * key_management_endpoints.py 60 % → 65 % (+82 lines) * team_endpoints.py 62 % → 72 % (+137 lines, crosses 70 % stretch) Key lands under 70 % per plan §7 escape hatch — the gap is dominated by routes outside F1–F6 scope (key list/info v2 internals) and structurally dead org-budget guards (call sites at lines 889 + 2310 + 985 + 1751 load the org without include_budget_table=True, so org.litellm_budget_table is None at guard time and the aggregate guard no-ops). Pinned as observed no-op behavior so a future fix that flips the flag turns these into reds. Zero source-code changes; pyproject.toml diff is empty; test_route_coverage.py stays green untouched; G3 grep guards still green; local wall-time 14 s for the full suite (no coverage), 22 s with coverage. G4 regression-replay protocol executed against three representative fix-PR parents (410ce76, 0bd49ec, 8bbc61e): all Phase 4 tests PASS at pre-fix SHAs — confirming the F1–F7 layer is a helper-body pin, not a regression-replay layer for those specific historical bypass shapes. Targeted RED-bait scenarios for each fix are left for a follow-up PR. * test(proxy): push key_management_endpoints.py past the 70% stretch (F7-extension) Adds 24 more payload-pin scenarios in test_f7_key_coverage_push.py following the same accepted-effect / rejected-guard pattern. Each scenario cites the file:line range it pins; same anti-snapshot rules apply. Target ranges (all reachable via HTTP-boundary payload variation): * 5942-6063 /key/health with metadata.logging → test_key_logging body * 4565-4692 /key/reset_spend happy + 404 + non-admin gate + value validation * 4421-4533 /key/regenerate ghost-404 + happy + new_key + grace_period * 4168-4202 _insert_deprecated_key body via grace_period * 6118-6133 _enforce_unique_key_alias duplicate-alias rejection * 6148-6169 validate_model_max_budget malformed-payload rejection * 4708-4789 validate_key_list_check user/team/org/key_hash branches * 2622-2733 /key/bulk_update mixed success/failure + admin gate + size limits * 2797-2950 /team/key/bulk_update all-keys path + explicit-keys dedupe + 404 * 5108-5207 /key/aliases admin + scoped + search-filter branches * 3253-3303 /key/info ghost + explicit-key + no-key-uses-auth-header * 3427-3436 generate_key_helper_fn budget_limits initialization * 1794-1815 prepare_key_update_data duration + budget_duration paths * 5280-5388 _build_filter_conditions across include_created_by_keys/team/sort/alias Coverage telemetry — full PR4 dataset: key_management_endpoints.py: 60 % → 71 % (+11 pts, +194 lines) team_endpoints.py: 62 % → 72 % (+10 pts, +137 lines) Both files now over the plan §7 PR4.M4 70 % stretch as a side effect of pinning real payload behavior. 721 tests pass in 19 s local (full suite, no coverage); 27 s with coverage. Zero source-code changes; pyproject.toml diff still empty; test_route_coverage.py + G3 grep guards still green. Honest finding (kept from the prior commit's body): four structurally-dead org-budget guards remain pinned as observed no-op behavior — they fire only when get_org_object is called with include_budget_table=True, which none of the four management-endpoint call sites currently do. Pinned so a future change that flips the flag turns these into reds. Two helper guards are honest-ceiling: _validate_reset_spend_value's isinstance check at line 4568 is unreachable from HTTP because Pydantic 422s non-float before the helper runs; same shape for /team/key/bulk_update's missing team_id / no-selector pre-handler guards. * test(proxy): address PR review — try/finally cleanup + loosen 500 envelope pins + Optional annotations Greptile review feedback on PR BerriAI#28681: 1. Wrap manual budget-row cleanup in try/finally so an assertion failure doesn't leave non-scratch-prefixed budget rows orphaned across CI re-runs (test_team_new_with_team_member_budget_creates_budget_row and test_team_update_team_member_budget_upserts). 2. Loosen the two 500-status pins to in (400, 422, 500) — the named-guard substring is the real pin; the outer ValueError-wrap envelope is an implementation detail that a future improvement should be free to fix to a proper 400/422 without flipping these tests red. 3. Add missing Optional annotations on _seed_token's max_budget / metadata / team_id keyword args (they default to None). Greptile's typo flag on 'read-world' in the conftest comment is declined — 'read-world' is the project's established term for the immutable seeded world fixture (see other usages in conftest.py and actors.py). 721 tests still pass in 17 s.
…iAI#28372) (BerriAI#28378) * feat(prometheus): emit per-token-type detail metrics (LIT-3220) (BerriAI#28372) Adds five sparse counter metrics that break out the token detail fields providers already report in `usage.prompt_tokens_details` and `usage.completion_tokens_details`: - litellm_input_cached_tokens_metric (provider prompt-cache reads) - litellm_input_cache_creation_tokens_metric (Anthropic prompt-cache writes) - litellm_input_audio_tokens_metric (audio input tokens) - litellm_output_reasoning_tokens_metric (reasoning tokens) - litellm_output_audio_tokens_metric (audio output tokens) These are additive — existing input/output/total counters are unchanged, so no dashboards break. Each new counter is only incremented when the underlying detail is populated and > 0, keeping scrape output sparse for providers that don't report a given field. Data is read from the canonical Usage dict that `get_standard_logging_object_payload` already attaches at `standard_logging_payload["metadata"]["usage_object"]`, so no new plumbing through the logging pipeline is required. Tests: 10 new unit tests covering registration, label-set parity, all-types increment, zero/None/negative skip behaviour, and the no-metadata/no-usage_object no-op paths. Closes LIT-3220 Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Claude <noreply@anthropic.com> * chore: remove proof folder image --------- Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai> Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
…rriAI#28405) * fix(otel): stamp http.response.status_code on all error responses httpx.HTTPStatusError exposes status under .response.status_code, not as a top-level attr, so unified-endpoint 5xx failures left the SERVER span without a status. The admin hooks only wrote a child span and never stamped or ended the parent at all, so admin 4xx/5xx (and success) responses were invisible to dashboards. Adds a fallback to .response.status_code in get_error_information, and ends the parent SERVER span in async_management_endpoint_{success,failure}_hook with the same _record_exception_on_span helper the unified path uses. Resolves LIT-3193 * test(otel): exercise httpx.HTTPStatusError through admin path Pins the contract that get_error_information's response.status_code fallback is reachable from any entry point — without this, a future refactor that bypasses _record_exception_on_span in the admin hooks could regress for httpx-wrapped exceptions while the unified suite still passes. * chore(otel): trim verbose comments in LIT-3193 changes Tighten docstrings and remove redundant section dividers/inline narration. Behavior is unchanged. * fix(otel): set span.status on management hook parent SERVER span Mirror the unified failure path: stamp StatusCode.ERROR on the parent SERVER span before recording the exception, and StatusCode.OK before ending it on success. Without this, OTEL backends filtering on span status (the idiomatic primitive) miss admin-endpoint failures even though the http.response.status_code attribute is correct. Extend assert_server_span_attrs to assert span.status.status_code matches the expected outcome so the gap can't regress. * fix(otel): close SERVER span on body-validation and unhandled errors Stash the SERVER span on request.state in auth so FastAPI exception handlers can finish it for failures that occur after auth but before the route handler (e.g. /model/new TypeError, /key/generate RequestValidationError). Without this, those requests left dangling spans missing http.response.status_code. Resolves LIT-3193 * fix(otel): generic 500 body, log exception details server-side Don't leak str(exc) and type(exc).__name__ to clients on uncaught exceptions. The full traceback is logged via verbose_proxy_logger and the SERVER span still gets http.response.status_code=500. Resolves LIT-3193 * fix(otel): stamp http.response.status_code on every SERVER span path Closes three remaining gaps where the proxy SERVER span ended without the http.response.status_code attribute: 1. ProxyException raised from _read_request_body (e.g. invalid JSON body) bubbled out of user_api_key_auth before the SERVER span was created, so the FastAPI handler had nothing to close and the trace never reached the backend. Hoist the span creation to a new idempotent _ensure_parent_otel_span_on_request_state helper called at the top of user_api_key_auth; wire openai_exception_handler to close the dangling span. Covers /v1/chat/completions, /v1/messages, /v1/responses (shared handler). 2. /v1/responses success — _handle_success ends the proxy span before async_post_call_success_hook fires on this path, so the hook's set_response_status_code_attribute(200) silently no-op'd against an ended span. Stamp 200 + set OK status at the close site in _handle_success / _end_proxy_span_from_kwargs via a shared _close_proxy_span_ok helper, so the attribute lands regardless of which success hook runs first. 3. Failure path for exceptions without code/status_code (e.g. a bare TypeError surfacing through _handle_llm_api_exception) — empty error_information.error_code → _record_exception_on_span skips the stamp → the hook ends the span. Default to 500 in async_post_call_failure_hook so the attribute is always set. Resolves LIT-3193
* fix(helm): drop main- prefix from default image tag
The default image tag in the deployment + migrations-job templates was
`main-{{ .Chart.AppVersion }}`. The current release pipeline publishes
content tags without the `main-` prefix (e.g. `v1.85.1` / `1.85.1`,
`v1.86.0-rc.1` / `1.86.0-rc.1`), so the rendered ref points at a tag
that does not exist on GHCR or DockerHub and installs fail with
ImagePullBackOff.
- templates/deployment.yaml, templates/migrations-job.yaml: render
`.Chart.AppVersion` directly instead of `main-<AppVersion>`.
- Chart.yaml: bump stale `appVersion: v1.80.12` (not on either
registry) to `v1.85.1` so local-checkout installs also resolve.
- values.yaml: update the commented tag-override hint to match.
* fix(helm): use :latest in tag override example, not pinned version
Per review: ghcr.io/berriai/litellm-database:latest is a floating
alias for the most recent stable (same digest as :main-stable),
maintained by the release pipeline's UPDATE_LATEST advance step.
Better example than a pinned version that goes stale.
…AI#28708) The schema in test_aaamodel_prices_and_context_window_json_is_valid uses additionalProperties: false. The azure/speech/azure-stt entry added in BerriAI#27482 introduced an audio_transcription_config field that the schema did not whitelist, so the test fails on every branch built on top of staging. Add the field as a string property.
chore(ci): promote internal staging to main
…rriAI#28683) * fix(team): refresh team cache on team_model_add/delete (LIT-3244) team_model_add and team_model_delete wrote to the DB but did not invalidate the in-memory LiteLLM_TeamTableCachedObj used by common_checks. After the v1.83.14 common_checks centralization made team.models authoritative on /v1/files and /v1/vector_stores/*, adding a Team-BYOK model silently failed to grant the new public model name to team members until the cache TTL expired (and a removed model kept working until then on the symmetric path). Extract the cache-refresh snippet from update_team into a small helper and apply it consistently at all three team-write sites. * test: also assert updated models in team-cache-refresh pin Strengthens the LIT-3244 regression test to also assert `call_kwargs["team_table"].models` matches the updated row, not just `team_id`. Both `existing_team` and `updated_team` share `team_id` in the test setup, so the previous assertion would have passed even if the implementation accidentally cached the pre-mutation row. Greptile review feedback. * fix(team): hydrate object_permission on cache-refreshing team updates The Prisma update calls in update_team, team_model_add, and team_model_delete returned a team row with object_permission_id set but object_permission=None (the relation was not requested via include=). _refresh_cached_team then wrote that to the in-memory LiteLLM_TeamTableCachedObj, and the cache-hit path in get_team_object returns the cached object without re-hydrating. Downstream consumers (validate_key_search_tools_against_team, the MCP/agent authz paths) treat a missing object_permission as no team-level restriction, so a team-write op silently dropped object-permission enforcement until the cache TTL expired or a DB-fetch path re-hydrated it. Add include={"object_permission": True} to all three updates so the refresh writes a complete cached team. Extend the LIT-3244 regression test to pin both the cached object_permission and the include shape on the Prisma call. Surfaced in PR review of LIT-3244.
… Anthropic (BerriAI#28723) `getProviderModels()` matched a model into a provider's dropdown when the model's `litellm_provider` string *contained* the provider key as a substring. The intent was to admit suffix variants (e.g. `anthropic_text`, `bedrock_converse`), but the substring check is too loose: it also pulls in unrelated providers whose name happens to contain the key, most visibly `vertex_ai-anthropic_models` matching `anthropic` and `vertex_ai-openai_models` matching `openai`. Replace `.includes()` with separator-anchored prefix matching (`startsWith(provider + "_")` / `startsWith(provider + "-")`). All legitimate variants in `model_prices_and_context_window.json` still match (`anthropic_text`, `azure_text`, `azure_ai`, `bedrock_converse`, `bedrock_mantle`, `cohere_chat`, `fireworks_ai-embedding-models`, `vertex_ai-*`, `vertex_ai_beta`), and the cross-provider leak is closed. Tests: update one assertion that pinned the buggy substring behavior (`custom_openai_endpoint` matching `openai` — not a real provider value); add 6 new tests covering the leak regressions and the variant-preservation contract for vertex_ai/bedrock/fireworks.
Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: ryan-crabbe-berri <ryan-crabbe-berri@users.noreply.github.com>
…rs and signed request body (BerriAI#27526) * Fix Bedrock KB pass-through SigV4 headers and signed body Coerce botocore HeadersDict to a dict for pass-through routes. When forward_headers is true, drop request headers that collide case-insensitively with signed headers so client Bearer auth does not shadow AWS SigV4. Send prepped.body as raw content so the outbound payload matches the signature after logging hooks mutate the parsed dict. Co-authored-by: Cursor <cursoragent@cursor.com> * Simplify pass-through raw body handling Read the SigV4-signed bytes directly from request.state inside pass_through_request instead of threading a custom_raw_body argument through three functions. Helper methods are restored to their original signatures, and the new branch lives in one place at each httpx call site. Co-authored-by: Cursor <cursoragent@cursor.com> * Harden pass-through raw body read from request.state Guard missing request.state (test fixtures) and ignore non-bytes/str values so MagicMock does not trigger the SigV4 raw-body path. Co-authored-by: Cursor <cursoragent@cursor.com> * Test pass_through_request state_raw_body uses httpx content= Cover non-streaming (async_client.request) and streaming (build_request) paths so SigV4 bytes on request.state are not replaced by json= of a hook-mutated dict. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…#28728) * chore(tests): migrate Bedrock CI from AWS account 888602223428 to 941277531214 The original account (888602223428) was put under a security restriction by AWS after a root access key leaked in a PR comment. While that account works its way through the AWS Support unlock process, Bedrock-touching CI tests have been migrated to a fresh account (941277531214). Changes: - Replace 26 hardcoded references to 888602223428 with 941277531214 across 8 files (provisioned-model ARNs, imported-model ARNs, AgentCore runtime ARNs, batch execution role ARN, and example proxy config). - The provisioned-model and imported-model ARNs are referenced only from mocked unit tests — no AWS resources to recreate. - The batch execution IAM role has been recreated in the new account with the same name and equivalent permissions. - The two AgentCore runtimes (hosted_agent_r9jvp-3ySZuRHjLC, hosted_agent_13sf6-cALnp38iZD) are being recreated in the new account under the same names — see tools/agentcore-deploy/ in a follow-up. CircleCI env vars AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_REGION_NAME were updated separately via the CircleCI API to point at the new account. Smoke-tested locally against the new account: aws bedrock-runtime converse --region us-west-2 \ --model-id us.anthropic.claude-sonnet-4-5-20250929-v1:0 \ --messages '[{"role":"user","content":[{"text":"ping"}]}]' → 200, model returned 'pong' Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(tests): refresh AgentCore ARN suffixes to match newly-deployed runtimes The first migration commit replaced just the account ID, but AgentCore auto-assigns a random 10-char suffix to every runtime on creation — we can't reuse the original suffixes (`3ySZuRHjLC`, `cALnp38iZD`) in the new account. Updated the AgentCore-runtime ARNs in the three files that reference real runtime IDs (not the mock-based unit-test ARNs). Deployed runtimes: arn:aws:bedrock-agentcore:us-west-2:941277531214:runtime/hosted_agent_r9jvp-Rq79QFC2fp arn:aws:bedrock-agentcore:us-west-2:941277531214:runtime/hosted_agent_13sf6-4046UzHSwy Both runtimes are status=READY and pass a smoke invoke: $ aws bedrock-agentcore invoke-agent-runtime --agent-runtime-arn ... --payload '{"prompt":"ping"}' → 200, {"result": "echo: ping"} The agent is a minimal echo (see /tmp/agentcore_deploy/agent.py for the deploy artifacts). Tests that only verify the SDK wiring will pass; if any test asserts on agent output content, swap the echo for the real agent. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(tests): point Bedrock batch tests at new-account S3 bucket The account migration (888602223428 -> 941277531214) was a flat account-ID swap, which only rewrites ARNs that embed the account number. S3 bucket names carry no account ID, so the live Bedrock batch tests still uploaded to `litellm-proxy` — a bucket that lives in the old account. S3 names are globally unique, and the old account still holds that name, so it can't be recreated in the new account. Rename to `litellm-proxy-941277531214` (account-ID suffix guarantees global uniqueness). The bucket must be created in 941277531214 and the batch execution role granted s3:GetObject/PutObject/ListBucket on it before this job is run in CI. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(tests): point live S3 logging test at new-account bucket Same account-ID-free blind spot as the batch bucket: `load-testing-oct` lives in the old account and its name can't be reused globally. The `logging_testing` CI job is wired into the workflow and runs test_basic_s3_logging, which uploads to this bucket with the CI env creds, then lists and deletes objects — a live dependency. Rename to `load-testing-oct-941277531214`. The bucket must exist in the new account with the CI IAM principal granted s3:PutObject/GetObject/ListBucket/DeleteObject before this job runs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(tests): repoint Bedrock guardrail IDs to new-account guardrails The migration left guardrail IDs untouched (no account ID in them), so all live guardrail tests failed with "guardrail identifier or version does not exist" against 941277531214. Recreated both guardrails in the new account and updated the hardcoded IDs: - wf0hkdb5x07f -> zgkmukebruil (PII mask: PHONE + CREDIT_DEBIT_CARD, with explicit inputAction=ANONYMIZE so masking applies to INPUT, which is the source litellm's moderation hook sends) - ff6ujrregl1q -> 4w3d1di3snt5 (blocks "coffee"; blocked message set to the exact string the tests assert on) Updated test_bedrock_guardrails.py, otel_test_config.yaml, and the guardrailConfig in test_bedrock_completion.py. Verified locally: the 5 previously-failing guardrail tests now pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(bedrock): migrate legacy models to current inference profiles The new CI account (941277531214) cannot invoke legacy Bedrock models (AWS gates them: "marked by provider as Legacy... not actively using in the last 30 days"). Migrated the live-call tests: - anthropic.claude-3-sonnet-20240229 -> us.anthropic.claude-sonnet-4-5-20250929-v1:0 - anthropic.claude-3-haiku-20240307 -> us.anthropic.claude-haiku-4-5-20251001-v1:0 Current Claude models on Bedrock require the us. inference-profile prefix (bare on-demand ids are rejected). cohere.command-r-plus has no working replacement (all Cohere is legacy- gated in the new account): swapped to claude-haiku-4-5 in provider- agnostic param lists. amazon.titan-image-generator skipped (no working replacement). Mocked/transformation/cost tests that reference the legacy strings are intentionally left unchanged. Verified live against the new account. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(bedrock): repoint SageMaker + Knowledge Base to new-account resources These referenced account-scoped resources by hardcoded id that only existed in the old account, so the migration's account-ID swap missed them. Recreated in 941277531214 and repointed: - SageMaker endpoint jumpstart-dft-hf-textgeneration1-mp-20240815-185614 -> litellm-ci-textgen (gpt2 on a TGI container, ml.g5.xlarge) - Bedrock Knowledge Base T37J8R4WTM -> LCYXFBR2TU (OpenSearch Serverless vector store + titan-embed-text-v2, seeded with a LiteLLM doc) Verified live: test_sagemaker.py (12 passed) and test_bedrock_knowledgebase_hook.py (12 passed). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(reasoning_effort_grid): skip bedrock claude-opus-4-7 cells (not entitled on 941277531214) claude-opus-4-7 is listed in the new Bedrock CI account's foundation models but invoke is denied (AccessDeniedException: "not available for this account"). Bedrock access to the flagship Opus requires an AWS Sales request, not the self-serve model-access toggle, so it can't be enabled inline with the rest of the account migration. Add an optional `skip_reason` to ModelEntry and set it on the bedrock-claude-opus-4-7 entry; the grid test honors it via pytest.skip. Cell count (231) and route coverage are unchanged, so the structural asserts still pass. Restore coverage by deleting the one skip_reason line once access is granted. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(bedrock): swap/skip legacy-gated models unavailable on new CI account The migrated AWS account (941277531214) cannot access several models that the old account could, so the remaining red CI jobs were hitting real Bedrock "Access denied / Legacy" and "account not authorized" errors: - image_gen: skip both Nova Canvas test classes (amazon.nova-canvas-v1:0 is legacy-gated), matching the existing titan skip. - batches: skip test_async_file_and_batch (Bedrock batch inference is not authorized on the new account; requires an AWS support case). - litellm_overhead: swap legacy claude-3-5-haiku for the active us.anthropic.claude-haiku-4-5 inference profile. - test_completion_claude_3_function_call: swap legacy claude-3-sonnet for the active us.anthropic.claude-sonnet-4-5 inference profile. https://claude.ai/code/session_01Y7zgHYu9GX29YRwV4yiWAa * test(bedrock): fix remaining e2e legacy-model + batch failures on new CI account - e2e_openai_endpoints: skip test_bedrock_batches_api (Bedrock batch inference is not authorized on account 941277531214) and migrate the missed s3_bucket_name in oai_misc_config.yaml to litellm-proxy-941277531214. - build_and_test: swap legacy bedrock claude-3-sonnet for the active us.anthropic.claude-sonnet-4-5 inference profile in the proxy structured output e2e test. https://claude.ai/code/session_01Y7zgHYu9GX29YRwV4yiWAa * test(bedrock): make opus-4-7 + batch cells fail loudly and mock image-gen (BerriAI#28791) Replace the silent skips added for the new CI account with noisier behavior: - reasoning-effort grid: opus-4-7 cells now fail (when AWS creds are present) instead of skipping, so the missing entitlement stays visible in CI; they still skip when AWS creds are absent (local dev) - Bedrock batch inference tests: drop the skip so they run and fail until batch access is granted - Titan + Nova Canvas image-gen tests: mock the Bedrock HTTP call so the transform + cost-tracking path stays under test without live model access https://claude.ai/code/session_01MT7SWDnXUjv6e6EPG7BDjT Co-authored-by: Claude <noreply@anthropic.com> * test(bedrock): use pytest.xfail for known-failing opus-4-7 cells Replace pytest.fail with pytest.xfail when a model has a fail_reason, so known-broken cells stay visible as XFAIL without keeping CI red. Co-authored-by: Yassin Kortam <yassin@berri.ai> --------- Co-authored-by: Mateo <mateo@Mateos-MacBook-Pro.local> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai>
…http_request (BerriAI#28794) Co-authored-by: Yassin Kortam <yassinkortam@Yassins-MacBook-Pro.local>
* chore(proxy): route path-dependent call sites through get_request_route Replace direct ``request.url.path`` reads in auth, ACL, routing, and audit-log decisions with ``get_request_route(request)`` — the helper already added in ``auth/auth_utils.py`` that returns the ASGI ``scope["path"]`` with ``root_path`` stripped. Starlette reconstructs ``url.path`` from the Host header; ``scope["path"]`` is uvicorn's parse of the request line and matches what FastAPI dispatches on, so it's the authoritative route for any decision that should agree with the actual handler. Sites: - _experimental/mcp_server/auth/user_api_key_auth_mcp.py - management_endpoints/mcp_management_endpoints.py - vector_store_endpoints/utils.py - pass_through_endpoints/pass_through_endpoints.py - auth/route_checks.py - litellm_pre_call_utils.py - spend_tracking/spend_management_endpoints.py - common_utils/http_parsing_utils.py - management_helpers/utils.py - health_endpoints/_health_endpoints.py Adds regression tests in tests/proxy_unit_tests/test_proxy_routes.py that construct a Request with scope["path"] set to a benign route and the Host header crafted so url.path would resolve differently; each site's decision is asserted against scope["path"]. * chore(proxy): make get_request_route imports lazy at call sites Move the ``from litellm.proxy.auth.auth_utils import get_request_route`` imports added in the prior commit back to the function bodies that use them. The module-level form participates in a long-standing import cycle through ``auth_utils -> _types -> ...`` and was flagged by CodeQL on the PR; the lazy form matches the pattern the proxy already uses for ``user_api_key_auth`` and related helpers elsewhere in these files. Also drop the ``RouteChecks._is_assistants_api_request`` delegation in ``_get_metadata_variable_name`` introduced in the prior commit — the delegation pulled ``RouteChecks`` into the same cycle, and the call site reuses the resolved route for its other branches, so inlining the substring check is both cycle-free and avoids a redundant second ``get_request_route`` call. Comment in test_proxy_routes.py acknowledges that the two MCP table entries exercise ``get_request_route`` directly rather than the full production handler (which needs ASGI scope + MCP state to invoke). --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: user <70670632+stuxf@users.noreply.github.com>
* feat(dashboard): navbar hierarchy + Agent Platform notifications (BerriAI#27543) * feat(dashboard): refine navbar zones and Agent Platform notice Restructure the admin navbar for production users: clear product vs community vs personal columns with vertical dividers, icon-only Slack/GitHub in a shared chip, and Docs/Blog typography aligned on an 8px rhythm. Add a notifications bell with popover linking to the LiteLLM Agent Platform repo and optional mark-as-read persistence. Promote the account control with initials avatar, single-line display name, and navDisplayName mapping for placeholder user ids (e.g. default_user_id). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(dashboard): address PR review — AntD buttons, public page guard, dedupe regex - Replace raw <button> with AntD Button in BlogDropdown, NotificationsBell, UserDropdown, and test mock - Guard NotificationsBell + container behind !isPublicPage to avoid rendering on public pages - Remove redundant equality checks in navDisplayName (regex already covers them) - Remove unused `lower` variable after simplification Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * fix(dashboard): drop dead useHealthReadiness import in navbar The module was removed in BerriAI#27896 (replaced by useHealthReadinessDetails), but the import survived the rebase. The symbol is unused — only useHealthReadinessDetails is consumed in the file. Removing the dead import unblocks the UI TypeScript build. * fix(dashboard): align CommunityEngagementButtons test with icon-only aria-labels The component was refactored to an icon-only chip with aria-label='LiteLLM on GitHub' (squash BerriAI#27543), but the test still asserted /star us on github/i. Update the query to match the rendered accessible name. * refactor(dashboard): drop unused props from NavbarProps The navbar refactor moved user identity + dark-mode state to internal hooks (useAuthorized, useWorker), but the NavbarProps interface still declared userID, userEmail, userRole, premiumUser, isDarkMode, and toggleDarkMode as required, forcing every caller to thread them through. Drop them from the interface and all four call sites (page.tsx, (dashboard)/layout.tsx, public_model_hub.tsx, navbar.test.tsx). Also shrinks the destructure in layout.tsx so the now-unused locals stop being pulled out of useAuthorized(). * refactor(dashboard): use useSyncExternalStore for NotificationsBell dismiss flag Reads/writes of the litellmHideAgentPlatformBanner key were done directly inside NotificationsBell via a useEffect + useState pair. Every other localStorage-backed flag in the dashboard (Disable ShowPrompts, DisableBouncingIcon, DisableShowNewBadge, DisableUsageIndicator, DisableBlogPosts) is wrapped in a useSyncExternalStore hook over localStorageUtils so all mounted components stay in sync. Extract useHideAgentPlatformBanner to follow the same shape, swap NotificationsBell to consume it, and add a regression test that two sibling bells stay in sync without a remount when one is dismissed. * refactor: mask credential fields in proxy settings GET responses (BerriAI#28682) * refactor: mask credential fields in proxy settings GET responses Brings SSO settings, cache settings, and the email/Slack alerting view in /get/config/callbacks in line with the HashiCorp Vault config-override pattern, so persisted credentials are not transported back to the UI in plaintext. * refactor: harden short-value masking and hoist alerting var constant Closes two review observations: - mask_sensitive_keys now replaces short values (below the visible prefix+suffix length) with an all-mask string instead of returning them unchanged, so a 1-7 character credential is no longer round-tripped verbatim. - _ALERTING_SENSITIVE_VARS is moved out of get_config() to a module-level constant, matching the analogous _SSO_SENSITIVE_FIELDS and _CACHE_SENSITIVE_FIELDS in the SSO and cache endpoint files. --------- Co-authored-by: Krrish Dholakia <krrish+github@berri.ai> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…riAI#28809) The Key Info Overview tab's Spend card truncated sub-dollar budgets to "$0" because formatNumberWithCommas defaults to 0 decimals. The Settings tab passes 2; align the overview so a $0.10 budget renders as "$0.10". Resolves LIT-2845
…erriAI#28442) * feat(proxy): allow llm_api_routes virtual keys to list MCP servers Add a new `mcp_discovery_routes` group (GET /v1/mcp/server and GET /v1/mcp/server/{server_id}) and include it in `llm_api_routes` so that virtual keys configured with `allowed_routes=["llm_api_routes"]` can discover the MCP servers they have access to. Previously these calls failed with 'Virtual key is not allowed to call this route. Only allowed to call routes: [llm_api_routes]'. The GET handlers already sanitize the response for restricted virtual keys via `_sanitize_mcp_server_list_for_virtual_key`, stripping credential-bearing fields (url, headers, env). Write methods (POST/PUT/DELETE) on the same paths remain gated by the existing handler-level admin role checks. The new discovery list is intentionally kept OUT of `mcp_inference_routes`, so `is_llm_api_route()` still returns False for these paths — this preserves the existing contract that DISABLE_LLM_API_ENDPOINTS must not block the Admin UI from listing MCP servers. Co-authored-by: ryan-crabbe-berri <ryan-crabbe-berri@users.noreply.github.com> * refactor(proxy): make MCP discovery carve-out method-aware Replace the `mcp_discovery_routes` group in `llm_api_routes` with a method-aware special case inside `is_virtual_key_allowed_to_call_route`. Virtual keys with allowed_routes=["llm_api_routes"] are now permitted to call only GET /v1/mcp/server and GET /v1/mcp/server/{server_id} — non-GET methods and multi-segment admin sub-paths fall through to the existing 403. This keeps the general llm_api_routes list free of management paths and avoids accidentally exposing POST/PUT/DELETE writes through the route-check layer. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: ryan-crabbe-berri <ryan-crabbe-berri@users.noreply.github.com>
* chore(proxy): route path-dependent call sites through get_request_route Replace direct ``request.url.path`` reads in auth, ACL, routing, and audit-log decisions with ``get_request_route(request)`` — the helper already added in ``auth/auth_utils.py`` that returns the ASGI ``scope["path"]`` with ``root_path`` stripped. Starlette reconstructs ``url.path`` from the Host header; ``scope["path"]`` is uvicorn's parse of the request line and matches what FastAPI dispatches on, so it's the authoritative route for any decision that should agree with the actual handler. Sites: - _experimental/mcp_server/auth/user_api_key_auth_mcp.py - management_endpoints/mcp_management_endpoints.py - vector_store_endpoints/utils.py - pass_through_endpoints/pass_through_endpoints.py - auth/route_checks.py - litellm_pre_call_utils.py - spend_tracking/spend_management_endpoints.py - common_utils/http_parsing_utils.py - management_helpers/utils.py - health_endpoints/_health_endpoints.py Adds regression tests in tests/proxy_unit_tests/test_proxy_routes.py that construct a Request with scope["path"] set to a benign route and the Host header crafted so url.path would resolve differently; each site's decision is asserted against scope["path"]. * chore(proxy): make get_request_route imports lazy at call sites Move the ``from litellm.proxy.auth.auth_utils import get_request_route`` imports added in the prior commit back to the function bodies that use them. The module-level form participates in a long-standing import cycle through ``auth_utils -> _types -> ...`` and was flagged by CodeQL on the PR; the lazy form matches the pattern the proxy already uses for ``user_api_key_auth`` and related helpers elsewhere in these files. Also drop the ``RouteChecks._is_assistants_api_request`` delegation in ``_get_metadata_variable_name`` introduced in the prior commit — the delegation pulled ``RouteChecks`` into the same cycle, and the call site reuses the resolved route for its other branches, so inlining the substring check is both cycle-free and avoids a redundant second ``get_request_route`` call. Comment in test_proxy_routes.py acknowledges that the two MCP table entries exercise ``get_request_route`` directly rather than the full production handler (which needs ASGI scope + MCP state to invoke). --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: user <70670632+stuxf@users.noreply.github.com>
…BerriAI#28737) * fix(team): keep team_alias cache in sync on _cache_team_object writes _cache_team_object wrote only to the team_id:<id> cache key, but the JWT auth path that uses team_alias_jwt_field reads from a separate team_alias:<alias> key (get_team_object_by_alias caches under both keys on miss, but reads only the alias-keyed one). After any team-mutation endpoint (team_model_add, team_model_delete, update_team, the two access-group writes) the team_id cache was refreshed but the team_alias cache stayed stale until TTL — JWT callers using team_alias_jwt_field kept seeing the pre-mutation team for the full cache window. Mirror the write under the alias key inside _cache_team_object so every existing caller stays in sync without further changes. Skip the alias write when team_alias is None/empty so we don't collide across alias-less teams. Surfaced testing the LIT-3244 cherry-pick on patch/1.86.0: the LIT-3244 fix correctly invalidated the team_id cache but the customer's JWT used team_alias_jwt_field, so they kept hitting the stale alias-keyed entry. * fix(team): delete (not overwrite) team_alias cache on _cache_team_object The prior shape of this PR wrote both team_id:<id> AND team_alias:<alias> from _cache_team_object. team_alias is NOT unique in the schema (no @unique on LiteLLM_TeamTable.team_alias), and get_team_object_by_alias enforces uniqueness on its own DB-fetch path (len(teams) > 1 raises). Writing the alias-keyed cache from the generic refresh path bypassed that check: a team admin renaming their team to collide with another team's alias could silently overwrite the cached team for JWT-by-alias auth, swapping the resolved team under that alias for the cache window. Switch the alias-keyed operation from a write to a delete (mirroring the dual-cache delete pattern in _delete_cache_key_object). After every team write, the next JWT-by-alias reader cache-misses and falls through to get_team_object_by_alias, which (a) re-fetches the fresh team from DB, closing the LIT-3244 staleness gap that motivated this PR, and (b) enforces alias uniqueness before populating either cache key. team_id:<id> writes are unchanged — team_id is the table PK and is guaranteed unique. Surfaced in veria-ai review on BerriAI#28739. * fix(managed-files): anchor model_id regex so it doesn't match llm_output_file_model_id extract_model_id_from_unified_id used `re.search(r"model_id,([^;]+)", ...)` which substring-matches the `model_id,` inside the file-ID encoding's `llm_output_file_model_id,<deployment_uuid>` field. parse_unified_id then fed that deployment UUID back into the auth path as a model candidate via _extract_models_from_managed_resource_id, and every team-BYOK file attach 403'd with: team not allowed to access model. This team can only access models=['openai/*']. Tried to access <deployment-uuid> The team's models list correctly contains the public name (`openai/*`) that target_model_names matches, but the bogus UUID candidate fails the wildcard check first. Anchor the regex to a field boundary (`(?:^|;)model_id,`) so it matches the legitimate top-level `model_id,<value>` field on vector_store unified IDs and skips substring matches inside other fields. File-IDs (which have no top-level `model_id` field) now return None and contribute no spurious UUID candidate. Surfaced reproducing LIT-3244 on patch/1.86.0 with the customer's exact flow: team with openai/* BYOK deployment, JWT-scoped user, POST /v1/vector_stores/{id}/files attaching a file uploaded with target_model_names=openai/gpt-4o.
…erriAI#28822) * fix(proxy): hydrate wildcard discovery credentials * fix(proxy): constrain wildcard credential hydration Co-authored-by: Dibyo Mukherjee <dibyo@adobe.com>
Creates litellm_oss_agent_shin_MM_DD_YYYY from main every day at 00:00 UTC. Lets us retarget oss-agent-shin fork PRs onto a canonical branch so CircleCI runs with secrets, without granting the agent write access. Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
…I#28827) * test(proxy): add harness for proxy_server.py behavior-pinning Creates tests/test_litellm/proxy/proxy_server/ with: - conftest.py: 11 shared fixtures (app, client, mock_prisma, auth_as, mock_router with parametrized response builders, normalize, etc.) - _coverage_check.py: per-PR coverage gate (line + branch) against a baseline, self-selects target by inspecting which placeholder files have been filled - _pin_check.py: AST-based gate that verifies every pin-list item has >=1 happy + >=1 error test with a real assertion (no status-only) - test_harness_smoke.py: 19 smoke tests covering every fixture + both scripts end-to-end - 26 placeholder test files (one docstring each) reserved for follow-up PRs per the directory ownership in the Notion plan - .coverage_baseline pinned at 0% so future PRs measure deltas against new-tests-only and aren't entangled with the broader scattered test suite Adds a dedicated proxy-server job to test-unit-proxy-endpoints.yml so this directory's runtime + coverage are tracked independently. Plan: https://www.notion.so/36c43b8acdab81ee845fd5365128a2fc * ci(proxy-endpoints): allow workflow_dispatch Lets the workflow be triggered manually on a branch via `gh workflow run`, which is needed for the verify-first flow on workflow changes before opening a PR. * test(proxy): address review feedback on proxy_server harness - conftest.py: anchor sys.path insert to __file__ (Path(__file__).resolve().parents[4]) instead of CWD-relative os.path.abspath("../../../../") which resolved to the wrong directory when pytest is launched from the repo root. - _coverage_check.py: actually read .coverage_baseline and use it as the floor (line_min = max(target, baseline)). Closes the gap between the PR description's "delta semantics" and what the script was doing. With baseline=0.0 today this is a no-op; future PRs that update the baseline cause regressions (test deletions etc.) to trip the gate even if the static PR target is still met. - _pin_check.py: drop unreachable startswith("_") guard (test_*.py glob never yields underscore-prefixed names) and read each test file once instead of twice.
…sidency (BerriAI#28626) * feat(openai): apply regional-processing cost uplift for EU/US data residency OpenAI charges a 10% uplift on the latest GPT models when requests are served from a regionalized hostname (eu./us.api.openai.com). Infer the region from `api_base`, expose it on `kwargs["litellm_params"]["data_residency"]`, and multiply the computed cost by a per-model `regional_processing_uplift_multiplier_<region>` field. https://claude.ai/code/session_012ebH44s7ohYxjoix5CXzTW * test: allow regional_processing_uplift_multiplier_{eu,us} in model_prices schema * fix(cost): tighten data_residency inference and restore model_cost in tests - Only infer OpenAI data_residency when custom_llm_provider == "openai"; drop the implicit None fallback so non-OpenAI callers can't accidentally pick up a regional tag from a stray OpenAI hostname. - _local_model_cost_map fixture now snapshots and restores litellm.model_cost and LITELLM_LOCAL_MODEL_COST_MAP so tests don't leak state across the session. * refactor(openai): move data_residency helper under llms/openai * fix: thread data_residency through realtime stream cost calculation Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(cost): thread data_residency through batch_cost_calculator Apply the OpenAI regional-processing uplift multiplier to retrieve_batch cost paths so Batch API requests served via eu./us.api.openai.com are priced at the same uplifted token rates as completions/transcriptions. * refactor(openai): encapsulate provider check inside infer_openai_data_residency Move the custom_llm_provider == "openai" guard from get_litellm_params into the helper itself so the core utility no longer carries provider-specific dispatch logic. Callers pass through the provider unconditionally; the helper returns None for any non-OpenAI provider. * fix(responses): thread data_residency through Responses logging params The Responses API paths build their logging litellm_params dict after provider resolution but did not include data_residency, so cost calc saw None even when the effective api_base was a regional OpenAI host. --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai>
…erriAI#28112) * chore(admin-ui): regenerate static export with trailingSlash: true Rebuilds litellm/proxy/_experimental/out/ from ui/litellm-dashboard with `trailingSlash: true` enabled in next.config.mjs. Next.js now emits every route as <dir>/index.html (e.g. mcp/oauth/callback/index.html) instead of <dir>.html with a sibling metadata-only directory, which fixes the 404 on extensionless URLs served through FastAPI's StaticFiles(html=True) mount. This is the build artifact half of the fix; the config change, Dockerfile cleanup, and regression test live in the follow-up source PR that stacks on top of this branch. * fix(admin-ui): emit nested routes as <dir>/index.html (BerriAI#28106) Linear and other OAuth providers redirect the user back to /ui/mcp/oauth/callback?code=...&state=... after the consent step. The packaged Next.js static export only produced /ui/mcp/oauth/callback.html, so FastAPI's StaticFiles served a 404 on the extensionless URL and the OAuth handshake never completed. The Dockerfile.non_root build step tried to paper over this at image-build time with `for html_file in *.html; do ...`, but that shell glob does not recurse, so nested routes like mcp/oauth/callback.html were left stranded next to an empty mcp/oauth/callback/ directory containing only Next.js metadata. The runtime restructure step in proxy_server.py was then skipped because the .litellm_ui_ready marker had already been dropped. Set trailingSlash: true in the dashboard's Next.js config so the export emits every nested route as <dir>/index.html natively. The Dockerfile loop is now a no-op for the bundled UI and has been removed; the .litellm_ui_ready marker is still written so the proxy keeps skipping the redundant Python restructure step at startup. Stacks on top of the static export regeneration in the parent branch. * chore: restore origin/litellm_internal_staging out files
…AI#28627) * fix(azure): preserve AD token refresh in v1 OpenAI client path The /openai/v1/ code path (api_version in {"v1", "latest", "preview"}) constructs a plain OpenAI/AsyncOpenAI client, but only forwarded `api_key` from `azure_client_params`. When `enable_azure_ad_token_refresh` is set (or any AD-only auth), `api_key` is None and the client constructor raised "The api_key client option must be set...", breaking every Azure call with a v1 api_version. The OpenAI SDK (>=2.20.0) accepts a callable for `api_key` and re-invokes it on every request via `_refresh_api_key`, so we now forward `azure_ad_token_provider` directly — preserving the per-request token refresh behavior of the regular AzureOpenAI client and avoiding the expiry hole that resolving the token once at client-creation time would introduce. Static `azure_ad_token` strings fall through to `api_key`. For the async path we wrap the sync provider returned by azure-identity in an async function since AsyncOpenAI expects `Callable[[], Awaitable[str]]`. Fixes BerriAI#27945 https://claude.ai/code/session_01UnzrDSFUUgp5T2wRoPMxq5 * fix(azure): offload sync token provider to thread in v1 async wrapper * fix(azure): include AD credential identity in v1 client cache key --------- Co-authored-by: Claude <noreply@anthropic.com>
* fix(ui): route API Reference back to query-param page The path-based /ui/api-reference route was broken in practice — the page-local useProxySettings hook didn't match what the root page passes down. Remove api_ref from the migration maps (LEGACY_REDIRECTS in app/page.tsx, MIGRATED_PAGES in leftnav.tsx and (dashboard)/layout.tsx), point the leftnav item back at page="api_ref", and restore the api_ref render branch in the root page. The path-based page.tsx and the useProxySettings hook stay in place unchanged; only api_ref is moved back to query-param routing while the migration infrastructure is preserved for future page moves. * fix(ui): alias ?page=api-reference to api_ref branch Handles bookmarks of the hyphen-form query param that was live during the brief path-based migration window, so they render the working APIReferenceView instead of falling through to the default page.
* test(proxy/proxy_server): pin forwarding routes (PR2) (BerriAI#28887) * test(proxy): pin proxy_server.py forwarding-route behavior PR2 of the proxy_server.py behavior-pinning project: fills the 12 forwarding-route test files added by the harness PR with happy + error pins for all 52 LLM-facing routes (models, chat/completions, completions, embeddings, moderations, audio, assistants, threads, utils, model-info, model-metrics, queue). Every happy-path test asserts the full response dict via normalize() so the gate enforces real shape pinning rather than status codes. * test(proxy): drop task-plumbing comments from PR2 test files * test(proxy): tighten PR2 error-path status-code pins Apply the same review feedback Greptile gave on PR1 (BerriAI#28856) and PR3 (BerriAI#28850) to PR2's forwarding-route tests: - Replace permissive `>= 400` / `in (X, Y)` status assertions with the exact 500/405 the handler actually returns, so a regression that silently shifts the code now fails the pin. - Add a body-presence check alongside each tightened status assertion to satisfy _pin_check.py's no-status-only rule. --------- Co-authored-by: Claude <noreply@anthropic.com> * test(proxy): pin proxy_server.py non-route surface behavior (PR1) (BerriAI#28856) * test(proxy): pin proxy_server.py non-route surface behavior (PR1) Fills the 7 PR1 placeholder files under tests/test_litellm/proxy/proxy_server/ with behavior pins for the non-route surface of proxy_server.py: lifecycle/init/shutdown, ProxyConfig class methods, DB-overlay config scrubbers, spend counters, background-health helpers, OpenAPI customization, exception handlers, and streaming-generator helpers. 233 tests cover 101 pin-list symbols (1+ happy + 1+ error each). New-tests-only coverage on litellm/proxy/proxy_server.py: 32.80% line / 20.91% branch (PR1 gate: 25% line / 18% branch). Full directory runs in ~22s with -n 4. Plan: https://www.notion.so/Plan-Pin-proxy_server-py-behavior-2026-05-25-36c43b8acdab81ee845fd5365128a2fc * test(proxy): address Greptile review comments on test_lifecycle.py - test_initialize_signature_is_async_with_expected_params: hard-code expected_param_count so a signature change actually trips the gate (previously both sides of the comparison were len(sig.parameters)). - test_check_request_disconnection_invalid_when_connected_times_out: patch asyncio.sleep so the test no longer spins for ~1.2 s of real wall-clock; timeout lowered to 0.05 s. --------- Co-authored-by: Claude <noreply@anthropic.com> * test(proxy/proxy_server): pin control-plane routes (PR3) (BerriAI#28850) * test(proxy/proxy_server): pin misc routes (PR3, partial) Adds happy + error tests for the misc control-plane routes: GET /, /routes, /adaptive_router/state, /get_logo_url, /get_image, /get_favicon. Also gitignores .pin_list.txt (used by the pin gate). * test(proxy/proxy_server): pin login/SSO routes (PR3, partial) Adds happy + error tests for the 5 login/SSO control-plane routes: GET /fallback/login, POST /login, POST /v2/login, POST /v3/login, POST /v3/login/exchange. Mocks authenticate_user and create_ui_token_object at their imported location. * test(proxy/proxy_server): pin onboarding routes (PR3, partial) Adds happy + error tests for the 2 onboarding control-plane routes: GET /onboarding/get_token, POST /onboarding/claim_token. Wires a MagicMock async context manager for prisma_client.db.tx() and signs the onboarding JWT with the patched master_key. * test(proxy/proxy_server): pin model_cost_map reload routes (PR3, partial) Adds happy + error tests for the 5 model-cost-map control-plane routes: POST /reload/model_cost_map, POST|DELETE|GET /schedule/model_cost_map_reload(/status), GET /model/cost_map/source. Attaches litellm_config to mock_prisma per-test (the table is not in the default _PRISMA_TABLES fixture). * test(proxy/proxy_server): pin anthropic_beta_headers reload routes (PR3, partial) Adds happy + error tests for the 4 anthropic-beta-headers control-plane routes: POST /reload/anthropic_beta_headers, POST|DELETE|GET /schedule/anthropic_beta_headers_reload(/status). Stubs db.litellm_config (not in default _PRISMA_TABLES) and monkeypatches reload_beta_headers_config so no network calls fire. * test(proxy/proxy_server): pin invitation routes (PR3, partial) Adds happy + error tests for the 4 invitation control-plane routes: POST /invitation/new, GET /invitation/info, POST /invitation/update, POST /invitation/delete. Patches _user_has_admin_privileges / _user_has_admin_view to avoid extensive get_user_object mocking. * test(proxy/proxy_server): pin config CRUD routes (PR3, partial) Adds happy + error tests for the 8 config-CRUD control-plane routes: POST /config/update, POST|GET /config/field/update|info, GET /config/list, POST /config/field/delete, POST /config/callback/delete, GET /get/config/callbacks, GET /config/yaml. Attaches litellm_config to mock_prisma per-test. * test(proxy/proxy_server): tighten pin assertions per review - test_routes_misc.py: `b"" in response.content` is trivially true; replace with `len(response.content) > 0` so an empty 405 body trips the gate. - test_routes_login_sso.py: `len(response.content) >= 0` is trivially true; tighten to `> 0`. - test_routes_anthropic_beta.py: replace brittle string-literal checks on the serialized JSON (`'"interval_hours": 12' in payload`) with `json.loads` + dict access so the assertion survives any serializer spacing. - test_routes_config.py: `assert status_code in (404, 500)` was too permissive; the handler re-raises HTTPException(404) verbatim, so pin 404 strictly. --------- Co-authored-by: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
…iAI#29083) * test(e2e): cover Internal User create-key flow when in no teams The seeded e2e-internal-user is in two teams, so the "no team" branch of the Create Key modal — where the team dropdown must render empty — was unreachable. Seeds a noteam@test.local user and adds a spec that logs in fresh, opens the modal, and asserts the dropdown has zero options. * test(e2e): harden no-team dropdown assertion + add with-teams counterpart Replace the one-shot count() check with a settled-empty assertion: wait for the dropdown's loaded "No teams found" state before asserting zero options, so the test can't pass on a transient empty frame while the team-options request is still in flight. Add internalUserWithTeams.spec.ts as the differential partner; it logs in as the seeded e2e-internal-user (two team memberships) and asserts the dropdown lists exactly those teams. Without it, the no-team spec's zero-options assertion would still pass against a regression that empties the dropdown for every user.
BerriAI#29077) * test(e2e): assert internal-user navbar identity is scoped to that user The existing login.spec.ts only checks the admin's navbar identity. This adds the symmetric check for the internal user — verifying the account button + dropdown surface the internal user's email, id, and role, and that no admin-scoped values leak through. * test(e2e): harden navbar identity test per review feedback Locate the user dropdown panel by a data-testid on the popupRender div instead of Ant Design internal + Tailwind class names, so styling refactors no longer risk breaking the identity-scoping assertions. Source the seeded user emails/ids from shared constants (match seed.sql) instead of hardcoding them inline.
… providers (BerriAI#28868) * feat(anthropic/messages): in-gateway context_management polyfill for non-Anthropic providers - Add `context_management/` module with `clear_tool_uses_20250919` editor dispatched before chat-completions translation on `/v1/messages` - Hard-protect most-recently completed tool_result from being cleared - Attach `context_management.applied_edits` to both non-streaming and streaming (final `message_delta`) responses - Bedrock Converse: forward `context_management`; filter to `compact_20260112`-only edits with `compact-2026-01-12` beta header - token_counter: guard Anthropic-format tools (no `function` key) to prevent AttributeError during polyfill token counting - Streaming: handle empty-choices usage-only trailing chunks - Skip polyfill when `litellm.drop_params = True` Co-authored-by: Cursor <cursoragent@cursor.com> * fix(bedrock): pop None context_management before sending to Bedrock Converse If context_management is forwarded as None (e.g. when mapping returns None for an invalid format), _filter_context_management_for_bedrock_converse previously returned early without removing the key, leaving "context_management": null in the request and causing a validation error. Pop the key when the value is not a dict. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(bedrock/converse): pop None context_management; extract helpers to fix PLR0915 Co-authored-by: Cursor <cursoragent@cursor.com> * fix(anthropic/messages): check per-request drop_params alongside global Co-authored-by: Cursor <cursoragent@cursor.com> * fix(anthropic/messages): preserve drop_params for downstream and respect explicit False Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix: lazy debug logging in clear_tool_uses; remove unused context_management constants Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(anthropic/messages): guard context_management polyfill with try/except Wrap apply_context_management() in a try/except so any failure (e.g. litellm.token_counter raising on an unknown tokenizer or unexpected message format) is logged but does not crash the underlying LLM request. The polyfill is a best-effort additive feature; on failure we forward the original messages without applied edits. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(token_counter): guard None input_schema in Anthropic tool fallback Use `or {}` instead of `.get(..., {})` so explicit null parameters do not raise AttributeError when formatting function definitions for token counting. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: minimize context_management polyfill threading - Use None (not empty list) for polyfill_applied_edits when context management isn't requested, so semantics of 'feature not requested' vs 'feature requested but no edits applied' are distinct. - In the streaming iterator, only pass applied_edits to the per-chunk translator on the final (finish_reason) chunk; intermediate chunks ignore it anyway, and this makes intent explicit on both sync and async paths. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(context_management): align tool_use counts and normalize list spec - _count_tool_uses now requires a string id, matching _collect_tool_use_ids_in_order so the tool_uses trigger can't fire on blocks that aren't clearable. - apply_context_management dispatcher now accepts the OpenAI list form and normalizes it via AnthropicConfig.map_openai_context_management_to_anthropic, so the polyfill path no longer silently no-ops on list input. Co-authored-by: Yassin Kortam <yassin@berri.ai> * feat(context_management): add compact_20260112 polyfill for non-Anthropic providers Implements an in-gateway compaction polyfill that summarizes long conversations using a configurable model when `compact_20260112` is requested for non-Anthropic targets (e.g. OpenAI, Gemini), matching Anthropic's context management beta behaviour for those providers. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(compact): skip tool_result-only user turns; bedrock: elif for context_management - compact_20260112 Phase D: when keeping the last user turn after a full summary, skip role=user turns whose content is exclusively tool_result blocks. Such turns translate to OpenAI tool-role messages with no preceding assistant tool_calls (those got summarized away), which non-Anthropic providers reject. Fall back to a synthetic continuation prompt if no eligible user question exists, so the downstream call always has a non-empty user message. - bedrock converse: chain the context_management param as elif so it follows the same if/elif pattern as the surrounding thinking/ reasoning_effort checks. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(anthropic): post-compaction question selection, system type, sync stream merge - compact.py: select last user question from effective_messages (post-compaction slice) instead of raw messages, so prior summarized turns aren't reintroduced - handler.py: widen _prepare_completion_kwargs system parameter type to Union[str, List[Dict]] matching PolyfillResult.system - streaming_iterator.py: mirror async hold-and-merge logic in sync __next__ so context_management is attached to the final merged message_delta when stop_reason and usage arrive in separate chunks Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(anthropic/messages): apply context_management on sync path; clear held stop_reason chunk in async iterator - Sync `anthropic_messages_handler` was silently dropping the `context_management` kwarg via `ANTHROPIC_ONLY_REQUEST_KEYS` after the polyfill was moved into the async handler. Bridge to the async dispatcher with `run_async_function` so `litellm.messages.create()` callers keep working (regressed e.g. `clear_tool_uses_20250919`). - In the streaming iterator's `__anext__` `StopIteration` handler, clear `self.holding_stop_reason_chunk` after capturing it (matches `__next__`) so a subsequent call doesn't re-emit the same chunk. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(bugfixes): bedrock None context_mgmt; stream per-instance queue; sync polyfill; trailing-chunk passthrough Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(anthropic): silently drop trailing chunks after usage; remove dead _polyfill_result key - streaming_iterator: in sync __next__, after the usage chunk has been merged and emitted, silently consume any trailing provider events via 'continue' instead of forwarding them through the queue. Trailing chunks would translate to content_block_delta or message_delta and violate Anthropic SSE ordering after the final message_delta. The async __anext__ already drops these via 'if not self.queued_usage_chunk:' gating, so this aligns sync and async behavior. - handler: drop unused '_polyfill_result' from ANTHROPIC_ONLY_REQUEST_KEYS. PolyfillResult is passed as an explicit arg to the adapter methods, never through extra_kwargs, so the entry was dead code. Co-authored-by: Yassin Kortam <yassin@berri.ai> * refactor(anthropic): extract usage-merge helper; guard empty slice-only compaction result - Extract the duplicated hold-and-merge usage logic from the sync __next__ and async __anext__ paths into a shared _merge_usage_into_held_stop_reason_chunk helper so the subtle cache-token / context_management attachment lives in exactly one place. - In the compact_20260112 slice-only path, fall back to _select_last_user_question when _strip_compaction_blocks produces an empty list (e.g. messages ending on an assistant turn whose only content was the compaction block) so the downstream API never receives an empty messages array. Co-authored-by: Yassin Kortam <yassin@berri.ai> * refactor(anthropic/context_management): streaming iterator compaction fixes and compact polyfill improvements - Extract usage-merge helper; guard empty slice-only compaction result - Silently drop trailing chunks after usage; remove dead _polyfill_result key - Fix bedrock None context_mgmt; stream per-instance queue; sync polyfill; trailing-chunk passthrough - Apply context_management on sync path; clear held stop_reason chunk in async iterator - Fix post-compaction question selection, system type, sync stream merge - Skip tool_result-only user turns; bedrock: elif for context_management - Add streaming iterator compaction test suite Co-authored-by: Cursor <cursoragent@cursor.com> * revert(html): restore flat *.html naming in _experimental/out Reverses the accidental rename from *.html → */index.html introduced in 15ea941. All 35 files moved back to their original flat paths so the directory structure matches litellm_internal_staging. Co-authored-by: Cursor <cursoragent@cursor.com> * revert(config): restore proxy_server_config.yaml to litellm_internal_staging Co-authored-by: Cursor <cursoragent@cursor.com> * Fix: skip client compaction pre-processing when compact_20260112 polyfill will run The _prepare_context_managed_request helper unconditionally applied apply_client_compaction_block_history before invoking the polyfill. When the request also configured a compact_20260112 spec, that pre-processing consumed the client-sent compaction block and collapsed the message history to just the latest user question, starving the polyfill of conversation context. The polyfill's own Phase A (_slice_around_compaction_block) already handles client compaction blocks correctly and inspects the full post-compaction tail for the token-threshold check, so the pre-processing is both redundant and destructive in this case. Now the pre-processing only runs when no compact_20260112 polyfill spec will execute (no spec, drop_params on, or only non-compact edits like clear_tool_uses_20250919). Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(anthropic): plug compaction-block leak + iteration-usage gaps in streaming adapter - handler: when polyfill_will_run skipped client-history pre-processing and the polyfill ultimately returned None (best-effort swallow on unexpected error), apply the slice-only fallback before returning so Anthropic-specific 'compaction' content blocks don't leak to non- Anthropic backends that would reject them. - streaming_iterator: precompute will_merge_into_held so we don't pass applied_edits into the translator when the resulting processed_chunk will be discarded by the held stop-reason merge path. - streaming_iterator: augment processed_chunk with iterations usage in the holding_chunk branch (sync and async) for parity with the other emission branches; ensures usage.iterations is attached on the rare message_delta-reaches-holding_chunk path. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(anthropic): correct streaming usage iteration + translate tools for token counting - streaming_iterator: skip the trailing "message" iteration entry in the final message_delta when the held stop_reason chunk carries placeholder zero usage (no separate usage chunk arrived). Reporting zero tokens was misleading and inconsistent with the non-streaming path which always has real usage data. - streaming_iterator: drop two redundant type checks inside branches that are already guarded by an outer message_delta type check. - compact._count_effective_tokens: translate Anthropic-shaped tools (input_schema) to OpenAI shape before passing to litellm.token_counter so threshold checks aren't skewed by tokenizer paths that expect the OpenAI tool wrapper. Co-authored-by: Yassin Kortam <yassin@berri.ai> * Fix lint * fix(anthropic): plug content drop, compaction SSE shape, and compaction leak - Sync streaming __next__ no longer drops a buffered holding_chunk when the usage-merge path has already fired. Restoring the prior unconditional flush behavior preserves provider-emitted content (the SSE-ordering nit of a trailing content delta is preferable to silent content loss). - compaction content_block_start now carries the full block shape ({"type": "compaction", "content": ""}) to match the text-block pattern and Anthropic's native streaming shape, so clients that key off content_block_start see the field. - apply_compact_20260112 now slices around / strips compaction blocks before the opt-in gate check. Previously, when summary_model was not configured the editor returned the raw messages, leaking Anthropic-only compaction content blocks to non-Anthropic providers that reject them. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(anthropic): resolve mypy types in context management polyfill Use AppliedEdit and CompactionBlock consistently in the dispatcher and streaming adapter. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(anthropic): flush held content chunk in async streaming path Mirror the sync __next__ behavior: always flush a buffered holding_chunk after the stream ends, even when usage was already merged + emitted. Previously the async __anext__ kept the flush inside the 'if not self.queued_usage_chunk:' guard, silently dropping the last content delta on the proxy's primary path. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(anthropic adapter): correct sync streaming, surface polyfill failures, decouple sync path from proxy router - translate_completion_output_params_streaming: add is_async flag so the sync handler returns Iterator[bytes] instead of an unusable AsyncIterator. Async callers keep the existing behavior via the default is_async=True. - _run_polyfill_if_enabled: when the polyfill crashes and the spec requested non-compact edits (e.g. clear_tool_uses_20250919), raise an AnthropicContextManagementError instead of silently returning None so those edits are not dropped without an error surface. The compaction-block-slicing safety net remains for compact-only specs. - anthropic_messages_handler (sync): stop auto-attaching the proxy llm_router. run_async_function bridges to a new thread's event loop; reusing the proxy's loop-bound httpx clients there causes 'Event loop is closed' errors. The summary editor falls back to litellm.acompletion when llm_router is None. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix: address bug detection findings in token counter and streaming iterator - token_counter: guard against non-dict 'function' field in tool dicts and skip tools missing a name to avoid emitting 'type None = ...' which would produce inaccurate token counts. - streaming_iterator: change sync __next__ generic-error path to raise StopIteration (was StopAsyncIteration), so sync iteration cleanly stops. - streaming_iterator: centralize context_management attachment so the held-stop_reason direct-flush path defensively re-attaches applied_edits to match the merge path's guarantee. Co-authored-by: Yassin Kortam <yassin@berri.ai> * Fix lint * fix: correct COMPACT_MIN_TRIGGER_TOKENS to 50_000 Co-authored-by: Yassin Kortam <yassin@berri.ai> * Fix lint * Fix lint * Fix lint * fix(compact): reduce to last user question when summary_model not configured but prior compaction block exists Aligns the summary_model_not_configured path with the under-threshold and client-compaction-block paths, which both reduce post-compaction messages to just the latest user question so the downstream provider doesn't get the summary on system prefix AND the full post-compaction history. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(compact): forward caller system prompt to summary model call The default summarization instructions reference "the initial task above" and "the raw history above", but the system prompt that holds that task was not being forwarded to the summary model. The summary call now prepends an OpenAI-shaped system message translated from the original Anthropic-shaped system (str or content-block list) so the summarizer has the agent role and initial task in scope. * fix(compact_20260112): set default max_tokens and merge prompt when last turn is user - Set COMPACT_SUMMARY_MAX_TOKENS default for the summary call so providers like Anthropic (which require max_tokens) don't silently fail and degrade to summary_call_failed. - When the trailing translated message is already a user turn, merge the summarization prompt into it instead of appending a second user turn. Avoids consecutive role=user messages that strict providers reject. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(anthropic adapter): move current_content_block_start to __init__ Move the default TextBlock dict from a class-level attribute to __init__ so concurrent stream instances don't share the same mutable dict. The class-level default could be mutated in-place via tool_block['name'] = original_name in _should_start_new_content_block, leaking state across streams. This mirrors the existing fix already applied to chunk_queue. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(compact_20260112): surface error states + strip tool_result blocks in last user question applied_edits_for_response() now includes compact_20260112 edits that carry an error field (summary_model_not_configured, summary_call_failed, summary_extraction_failed) so clients and operators can see why compaction was requested but not applied. _select_last_user_question() now strips tool_result blocks from mixed [tool_result, text] turns rather than passing them through as-is. After compaction the paired tool_use assistant turn no longer exists, so forwarding tool_result blocks translates to orphaned role=tool messages on non-Anthropic providers and produces a 400. * fix(compact_20260112): carry prior compaction summary into Phase C summary call When a request already contains a compaction block, Phase A slices `effective_messages` to the turns since that block. Previously Phase C passed the original `system` to the summary model, so multi-round compaction silently dropped accumulated history each time the polyfill fired. Pass `augmented_system` (original system + prior summary prefix) so the summary model can produce a comprehensive summary that incorporates both the prior round's context and the current slice. `summarized_system` for the downstream call stays built from the original `system` + new `summary_text`. * refactor: delegate handler spec normalization to dispatcher _normalize_spec_edits in adapters/handler.py duplicated the spec-shape normalization already implemented by _normalize_spec in context_management/dispatcher.py. The two could drift: a change in one (e.g. supporting a new spec shape) without the other would cause the handler's polyfill_will_run prediction to disagree with the dispatcher's actual behavior, breaking the client-history pre-processing skip. Have the handler delegate to the dispatcher's _normalize_spec while keeping handler-specific concerns (drop_params short-circuit, swallow mapping exceptions) at the wrapper level. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(compact_20260112): surface warning-only applied edits in response `applied_edits_for_response()` previously hid `compact_20260112` edits when they had only warnings (no compaction block, no error). This dropped diagnostically important warnings such as `unsupported_trigger_type_X_using_input_tokens` and `pause_after_compaction_ignored` whenever the conversation was under the trigger threshold. Operators now see these warnings in the response. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix: address two low-severity context_management edge cases - streaming_iterator: keep `sent_content_block_finish` in sync with the compaction block's emitted start/delta/stop lifecycle and reset it when the next text block's start is queued. - bedrock _map_context_management_param: match dispatcher `_normalize_spec` behavior — only run the OpenAI→Anthropic mapper on list inputs; pass dict inputs through unchanged so already-Anthropic-format values aren't silently dropped. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(compact_20260112): use beta-header constant; require type discriminator; skip sync bridge when idle - bedrock: replace hardcoded "compact-2026-01-12" beta string with ANTHROPIC_BETA_HEADER_VALUES.COMPACT_2026_01_12.value in both Converse (_filter_context_management_for_bedrock_converse) and Invoke (anthropic_claude3) compact-edit handlers. - types: mark the "type" discriminator as Required[...] on the new CompactionBlock and UsageIteration TypedDicts so the discriminator is not silently optional under total=False. - adapters/handler: short-circuit the sync /v1/messages adapter path before spawning the run_async_function worker-thread event loop when the request has no context_management spec and no client-sent compaction block in the message history. Test plan: - uv run pytest tests/test_litellm/llms/anthropic/experimental_pass_through/ tests/test_litellm/llms/bedrock/test_converse_context_management.py -q (370 + 10 = 380 passed) - uv run pytest tests/test_litellm/llms/azure_ai/claude/test_azure_anthropic_transformation.py tests/test_litellm/llms/vertex_ai/vertex_ai_partner_models/anthropic/test_vertex_ai_partner_models_anthropic_transformation.py -k compact (3 passed) * fix(compact_20260112): include system prompt tokens in threshold check The threshold check in Phase B previously counted only message tokens and the compaction-block content, omitting the system prompt entirely. When the system carried a prior compaction summary (via _augment_system_with_summary) or was otherwise large, the threshold could fire later than intended, allowing the conversation to exceed the model's context window before compaction activated. _count_effective_tokens now also counts the (augmented) system prompt text. The caller passes compaction_block=None when augmented_system already includes the prior summary, to avoid double-counting. Co-authored-by: Yassin Kortam <yassin@berri.ai> * Fix SSE ordering and compaction state machine bugs in AnthropicStreamWrapper - Suppress holding_chunk flush after final message_delta has been emitted (queued_usage_chunk == True) so a trailing content_block_delta cannot follow message_delta, which strict Anthropic SDK clients may reject. When usage has not yet been merged, flush the holding_chunk *before* the held stop_reason chunk so SSE ordering remains correct. - Replace _queue_compaction_block_events with _next_compaction_event, emitting the compaction start/delta/stop events one at a time. The state machine flags (sent_content_block_finish) and content block index now advance atomically with the terminal stop event actually being returned to the caller, eliminating the transient inconsistent state where flags say the block is finished while its stop event is still buffered. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(compact_20260112): enforce parent key/team allowlist on summary model The compact_20260112 polyfill summary subrequest used llm_router.acompletion directly, bypassing the proxy auth checks that gate model access for the parent key/team. A caller whose key/team was not authorized for the configured context_management_summary_model could still cause the proxy to invoke that model and return its output as a compaction block. Pull the parent's UserAPIKeyAuth out of litellm_metadata in the handler, thread it through the dispatcher into apply_compact_20260112, and gate the summary call on _can_object_call_model for both key-level and team-level allowlists. Failures land as applied_edits[0].error = summary_model_access_denied without raising. SDK callers (no UserAPIKeyAuth) remain unaffected. * fix(compact_20260112): distinguish access-denied from transient errors; greedy summary regex - _check_summary_model_access now catches ProxyException explicitly for access denials and logs unexpected exceptions separately. Both still fail closed, but operators can now tell a denied key/team apart from a router internal raising during the check. - _SUMMARY_TAG_RE switches from non-greedy to greedy so a stray </summary> inside the model's summary content no longer silently truncates the captured text. * fix(compact_20260112): type object_type as Literal for mypy * fix(compact_20260112): attribute summary subcall spend to parent key/team The compact_20260112 polyfill summary subrequest propagated metadata via the Anthropic-shape `metadata` parameter, which only carries `user_id`. The proxy auth fields used for spend attribution (`user_api_key`, `user_api_key_team_id`, `litellm_call_id`, ...) live in `data["litellm_metadata"]`. As a result, summary subcalls landed on the router with an empty propagated metadata and the resulting tokens were not attributed to the caller's key/team budget. Rename the polyfill chain's spend-propagation parameter to `litellm_metadata` and pull it from `kwargs["litellm_metadata"]` in both the async and sync handlers, so the post-call hooks see the parent key/team and bill the summary tokens accordingly. Add an `_extract_proxy_litellm_metadata` helper and refactor `_extract_user_api_key_auth` to use it. * chore(anthropic adapters): remove unused _extract_user_api_key_auth helper Co-authored-by: Yassin Kortam <yassin@berri.ai> * chore(compact_20260112): non-greedy summary regex; use COMPACT_EDIT_TYPE in bedrock filter - Make _SUMMARY_TAG_RE non-greedy so a response with multiple <summary> blocks captures only the first complete block. - Replace the hardcoded 'compact_20260112' literal in _filter_context_management_for_bedrock_converse with the shared COMPACT_EDIT_TYPE constant. * fix: bug fixes from PR review - streaming_iterator: don't set sent_content_block_finish during compaction block lifecycle; that flag tracks the regular text/tool_use/thinking block state machine, conflating the two leaks bad state to introspection paths. - compact._call_summary_model: send propagated proxy auth/spend-attribution fields as 'litellm_metadata' instead of 'metadata' so the router's post-call hooks attribute summary tokens to the caller's key/team budget. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(anthropic-streaming): insert content_block_stop between held delta and final message_delta When the stream exhausts with both `holding_chunk` (a content_block_delta) and `holding_stop_reason_chunk` (a message_delta) buffered, the after-loop cleanup previously emitted them back-to-back, producing the invalid Anthropic SSE sequence `content_block_delta -> message_delta`. Insert a `content_block_stop` between them in both the sync `__next__` and async `__anext__` paths so the emitted ordering remains `content_block_delta -> content_block_stop -> message_delta`. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(compact_20260112): propagate allowed_model_region to summary subrequest The router enforces region restrictions by reading allowed_model_region from top-level request kwargs (Router._common_checks_available_deployment), but the compact_20260112 summary subrequest only forwarded litellm_metadata. A region-restricted caller could trigger compaction and have their conversation summarized by a deployment outside the permitted region. Extract allowed_model_region from user_api_key_auth and pass it through _call_summary_model as a top-level kwarg so the router applies the same region constraints the parent request would. * fix(anthropic adapter): emit content_block_stop before held message_delta in drain paths Co-authored-by: Yassin Kortam <yassin@berri.ai> * feat(context_management): configurable summary max_tokens; surface ignored knobs - compact_20260112: read summary max_tokens from general_settings (context_management_summary_max_tokens) so operators can fit the chosen summary model's output budget; falls back to the compiled default for missing or invalid values. - clear_tool_uses_20250919: log unsupported knobs at warning level (was debug, which silently dropped misconfiguration) and surface them as warnings on the AppliedEdit so clients see what was ignored. * fix(compact_20260112): bound _call_summary_model with timeout A slow or unresponsive summary model previously hung the parent /v1/messages request with no escape hatch. Pass a 60s timeout on the litellm.acompletion / llm_router.acompletion subrequest; on timeout the existing summary_call_failed path forwards the request without compaction rather than blocking indefinitely. * fix(compact_20260112): preserve post-compaction tail on slice-only path When a prior compaction block is present and the request is under threshold, the polyfill was reducing downstream messages to just the latest user question. The prior summary only covers turns before the compaction block, so dropping the post-compaction tail silently lost recent context — a multi-turn conversation that stayed below the threshold would arrive at the model with no memory of any turn after the prior compaction. Forward the already-stripped post-compaction tail unchanged on both the under-threshold path and apply_client_compaction_block_history. Fall back to _select_last_user_question only when the strip leaves nothing for the downstream call to answer. * fix(compact_20260112): enforce user/project/team-member model scopes on summary subrequest The local gate previously only checked the parent key's and team's allowed-model lists. A caller restricted by a personal user, project, or per-team-member allowed_models scope could still trigger the configured summary model and receive its <summary> output as a compaction block, because llm_router.acompletion bypasses the proxy common_checks path. Extend _check_summary_model_access to also load the user_object, project_object, and team_membership and run the matching allowlist check at each scope before invoking the summary model. * fix(compact_20260112): enforce summary model per-model budget and propagate budget metadata * fix(compact_20260112): forward post-compaction tail when summary model unconfigured * fix(anthropic endpoints): run failure hook on 500-level context management errors * fix(compact_20260112): enforce summary model rate limit before summary call * fix(compact_20260112): propagate end-user/project budget scope to summary call --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…erriAI#28830) Allow self-hosted installs to override the default LiteLLM sender address via RESEND_FROM_EMAIL, matching SendGrid's SENDGRID_SENDER_EMAIL pattern. Co-authored-by: Cursor <cursoragent@cursor.com>
…erriAI#28728)" (BerriAI#29326) This reverts the Bedrock CI account migration (BerriAI#28728). The original account (888602223428) was put under an AWS security restriction after a leaked key and has since been reactivated, while the replacement account (941277531214) lacks access to several models the suites exercise (legacy Bedrock Claude 3 models, Cohere, Nova Canvas image gen, Bedrock batch inference, and flagship Opus). Pointing CI back at the reactivated account restores that coverage. This is the exact inverse of BerriAI#28728: all hardcoded 941277531214 references go back to 888602223428 (provisioned/imported-model ARNs, AgentCore runtime ARNs and their suffixes, batch execution role ARN, and the example proxy config), the S3 buckets revert to litellm-proxy and load-testing-oct, the guardrail IDs revert to wf0hkdb5x07f and ff6ujrregl1q, the SageMaker endpoint and Knowledge Base revert to their original ids, and the live-call tests go back to the legacy model strings. The grid_spec fail_reason workaround for the unentitled Opus cells is dropped while keeping the unrelated bedrock_effort_ceiling field added after the migration. The CircleCI AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY env vars still point at 941277531214 and must be set to the reactivated account's fresh credentials separately via the CircleCI API; AWS_REGION_NAME stays us-west-2.
…erriAI#29249) * fix(mcp): preserve source_url in GET /v1/mcp/server list responses The list endpoint builds responses from the in-memory registry, but source_url was dropped during the DB-to-registry roundtrip even though GET /v1/mcp/server/{id} returned it correctly from the database. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tests/mcp): set source_url on MagicMock table records MagicMock auto-creates source_url as a mock object, which fails MCPServer Pydantic validation after source_url was wired through build_mcp_server_from_table. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…es (BerriAI#29253) * fix(mcp): preserve omitted fields on PUT /v1/mcp/server partial updates Use model_dump(exclude_unset=True) for updates so schema defaults (transport=sse, allow_all_keys=false, etc.) are not written when callers omit them. Serialize JSON fields from the filtered dict and only force is_byok on create. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): allow explicit alias=None on MCP server partial updates Snapshot caller-provided fields before normalization so omitted alias is not written while an intentional alias=None still clears the stored value. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…Opus 4.7 self-heal) (BerriAI#29344) * test(logging): align DB metrics event_metadata assertions with safe redaction PR BerriAI#28909 hardened log_db_metrics to emit a minimal, non-sensitive event_metadata (only table_name when present, otherwise None) instead of dumping function_name, function_kwargs, and function_args onto the span. The test in test_log_db_redis_services was not updated and still asserted "function_name" in event_metadata, which raised TypeError (argument of type 'NoneType' is not iterable) and turned the logging_testing CI job red on litellm_internal_staging. Update test_log_db_metrics_success to assert event_metadata is None when no table_name is passed, and add test_log_db_metrics_event_metadata_is_safe as a regression guard verifying that only the table name surfaces and that sensitive kwargs (tokens, prisma client) are never dumped. * test(bedrock): self-heal opus-4-7 grid cells when unentitled on CI The bedrock-claude-opus-4-7 converse cells are unentitled on the Bedrock CI account, so they were marked xfail. xfail keeps reporting them as expected failures even after access is granted, so the wire translation never gets verified again. Now the cell makes the call and skips only when Bedrock replies "is not available for this account"; the moment the model is entitled the same cells run their full assertions with no edit. A focused unit test pins the tolerance predicate so any other failure still surfaces loudly and the available path still runs the assertions.
…rriAI#29343) * refactor(proxy/auth): normalize Bearer prefix in safe-hash helper UserAPIKeyAuth._safe_hash_litellm_api_key now strips a leading "Bearer "/"bearer " prefix before its existing sk-/JWT classification, so the helper produces the same hashed output regardless of whether the caller stripped the Authorization header prefix or passed the header value through unchanged. * refactor(proxy/auth): make Bearer-prefix strip case-insensitive Per RFC 7235 the HTTP authorization scheme token is case-insensitive. Replace the two-prefix loop with a single case-insensitive check so the helper normalizes "Bearer ", "bearer ", "BEARER ", and any mixed-case variant before classifying the remainder as sk- or JWT. The contract test gains coverage of "BEARER " and "BeArEr ". * test(mcp): align auth-handler test expectations with safe-hash helper The two MCP auth tests asserted that UserAPIKeyAuth(api_key="Bearer ...") retained the raw header bytes on the api_key field. _safe_hash_litellm_api_key now normalizes that input — stripping the Bearer prefix and hashing the resulting sk- key — so the expectations move to the normalized form: the bare token in the parametrize case, and hash_token("sk-...") in the backward-compat assertion. This matches what the real auth flow produces (the builder strips Bearer and the DB stores the hashed token), so the mocks now line up with production rather than with the un-normalized validator output.
…utes (BerriAI#29327) * test(logging): align DB metrics event_metadata assertions with safe redaction PR BerriAI#28909 hardened log_db_metrics to emit a minimal, non-sensitive event_metadata (only table_name when present, otherwise None) instead of dumping function_name, function_kwargs, and function_args onto the span. The test in test_log_db_redis_services was not updated and still asserted "function_name" in event_metadata, which raised TypeError (argument of type 'NoneType' is not iterable) and turned the logging_testing CI job red on litellm_internal_staging. Update test_log_db_metrics_success to assert event_metadata is None when no table_name is passed, and add test_log_db_metrics_event_metadata_is_safe as a regression guard verifying that only the table name surfaces and that sensitive kwargs (tokens, prisma client) are never dumped. * test(bedrock): self-heal opus-4-7 grid cells when unentitled on CI The bedrock-claude-opus-4-7 converse cells are unentitled on the Bedrock CI account, so they were marked xfail. xfail keeps reporting them as expected failures even after access is granted, so the wire translation never gets verified again. Now the cell makes the call and skips only when Bedrock replies "is not available for this account"; the moment the model is entitled the same cells run their full assertions with no edit. A focused unit test pins the tolerance predicate so any other failure still surfaces loudly and the available path still runs the assertions. * test(reasoning-effort-grid): add claude-opus-4-8 across provider routes Adds claude-opus-4-8 to the anthropic, azure, vertex and bedrock-converse routes (275 cells total) so the reasoning-effort wire translation is covered for the new model. The bedrock opus-4-8 and opus-4-7 cells reuse the self-heal path: they run the call and skip only on Bedrock's "is not available for this account" reply, then assert in full once the model is entitled. The azure and vertex opus-4-8 cells stay xfail until a Foundry deployment exists and Vertex availability is confirmed. The shared xhigh+max capability set is renamed to _CAPS_XHIGH_MAX now that more than one model uses it.
…erriAI#28418) * fix(guardrails): return HTTP 400 for litellm content filter blocks Align litellm_content_filter hard rejects with the standard guardrail block status code so clients receive 400 instead of 403. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(guardrails): return HTTP 400 for custom code guardrail blocks Pre-call custom code guardrail blocks now raise HTTPException(400) instead of using the passthrough ModifyResponseException path that returned a synthetic 200 response. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(guardrails): preserve custom code passthrough blocks Keep standalone custom code guardrail blocks on the passthrough contract while covering policy pipeline block handling for passthrough-style guardrail interventions. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…BerriAI#29202) * fix(proxy): restrict vector store index create/delete to proxy admins Prevent non-admin API keys from registering indexes via POST /v1/indexes or deleting Azure AI Search indexes through managed pass-through routes. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): tighten vector store index lifecycle checks Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…erriAI#29160) * feat(pass_through): extend passthrough_managed_object_ids to Azure Adds managed ID minting/resolution for Azure passthrough endpoints (/azure/...) alongside the existing OpenAI passthrough support. Key changes: - pass_through_endpoints.py: detect azure/azure_ai custom_llm_provider (string or enum) to set _is_managed_id_provider and _managed_id_provider; both INPUT and OUTPUT rewrite blocks now fire for Azure. - llm_passthrough_endpoints.py: forward custom_llm_provider into create_pass_through_route so it reaches pass_through_request (was None). - managed_id_rewriter.py: extend _PASSTHROUGH_PREFIX_RE and _canonical_path to strip /azure/openai prefix and add /v1/ for Azure paths that omit it; add ("azure", method, path) entries to BUILTIN_OUTPUT_ID_FIELD_MAP for files and batches endpoints. - managed_id_codec.py / types/utils.py: supporting codec and enum constant. - proxy_server.py: register llm_passthrough_router before batches_router to prevent route collision for /openai_passthrough/* paths. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(pass_through): remove unused imports for ruff F401 Co-authored-by: Cursor <cursoragent@cursor.com> * fix(pass_through): satisfy mypy for optional parsed_body Co-authored-by: Cursor <cursoragent@cursor.com> * fix(pass_through): compute query params string after managed-ID rewrite Move requested_query_params_str computation to after the managed-ID input rewrite block so logging_url reflects the rewritten raw-provider query params actually sent upstream, instead of the original managed IDs. Co-authored-by: Yassin Kortam <yassin@berri.ai> * Add support for managed ids for passthrough responses api * Add support for list batches and list files * style: run Black on passthrough managed ID files Fix CI formatting for managed_id_rewriter.py and pass_through_endpoints.py. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(passthrough): parse json file_object and implement before-cursor pagination - Parse row.file_object via json.loads when Prisma returns it as a string; mirrors openai_files_endpoints/common_utils.py so list responses keep all stored detail fields (status, timestamps, etc.). - Implement the previously-parsed-but-unused 'before' cursor for list pagination by flipping fetch order to ascending with a 'gt' bound on created_at, then reversing rows so the response stays newest-first. Co-authored-by: Yassin Kortam <yassin@berri.ai> * Remove logger * refactor: split list_passthrough_ids_from_db to fix PLR0915 Extract pagination, fetch, and serialization helpers so the main list function stays under the statement limit without changing behavior. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: scope passthrough managed ID dedup and list by provider Validate embedded provider before reusing deduped file/object rows so OpenAI and Azure cannot share the same managed ID for an identical raw ID. Filter list responses to rows whose managed IDs decode to the current provider, with over-fetch scanning when needed. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(managed_id_rewriter): cap pagination trim at effective limit When raw_limit > 100, fetch_limit is capped at 101 (one extra row to detect has_more), but trimming with rows[:raw_limit] failed to drop the sentinel row. Use the capped effective limit instead. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix: cross-provider object collision and fail-closed list error handling Greptile P1: move provider check before access check in _mint_or_reuse_object so a cross-provider raw ID collision (OpenAI and Azure share the same batch_ ID) falls through to mint a new provider-scoped row instead of raising 404. Veria-ai medium: _fetch_provider_scoped_list_rows now always returns (page, has_more) — DB errors break out of the scan loop and return matched rows so far. list_passthrough_ids_from_db never returns None for a recognised list route, so the caller can never fall through to the upstream provider. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: namespace passthrough model_object_id by provider to prevent unique violation Store model_object_id as 'passthrough:{provider}:{raw_id}' instead of the bare raw ID so OpenAI and Azure can each own a row for the same raw batch ID without hitting the @unique constraint. Dedup lookup uses the same namespaced key so it is implicitly provider-scoped and the _managed_id_matches_provider check is no longer needed on the object path. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: gate list interception on managed_files hook like input/output rewrites Without the hook no managed IDs are minted so the DB is empty. Intercepting GET /v1/files without the hook returned an empty list and hid the caller's real upstream files/batches. Matches the guard used by the input and output rewrite blocks. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: set has_more=True when scan cap is hit with a full final DB batch When max_scans (20) is exhausted and the last DB page was full-sized, there are almost certainly more rows beyond the scan window. Track last_batch_full across iterations so the scan_cap_hit condition sets has_more=True in that case, preventing silent pagination truncation in high-mixed-provider pools. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(managed_id_rewriter): scope pagination cursor lookup to caller-owned rows Prevent a cross-tenant timing oracle by constraining the after/before cursor row lookup to the caller's owner_filter, and cover the real Azure responses path form (no /v1/) in tests. * fix(managed_id_rewriter): align passthrough list metadata with direct GET Persist upstream file metadata when minting a managed file ID and rewrite nested batch file IDs before snapshotting the object, so DB-served file/batch list responses return the same fields and managed IDs as a direct endpoint GET. * fix(managed_id_rewriter): degrade to raw id on cross-owner object collision The OUTPUT (mint) path of _mint_or_reuse_object raised HTTPException(404) when a dedup hit on the namespaced model_object_id belonged to a different owner, converting a successful upstream batch/response creation into a 404 for the caller. Two upstream accounts under one provider name can issue the same raw id, so this is reachable in multi-tenant deployments. Return the caller's raw id unmanaged instead: the upstream create already succeeded, a new managed row can't be minted (model_object_id is @unique), and reusing the other owner's managed id would later fail the access check. * fix(managed_id_rewriter): scope list cursor by provider and cap body-rewrite recursion depth * perf(managed_id_rewriter): push batch list provider scope to DB and anchor canonical-path prefix Object (batch) list rows store model_object_id as passthrough:{provider}:{raw}, so the provider filter is now applied at the indexed DB column, collapsing the application-layer multi-scan to a single query for that table. File rows keep the decode-based scan since they have no provider column. Anchor the canonical-path prefix regex at a path boundary so routes such as /openai_realtime/... are no longer mis-stripped. * fix(managed_id_rewriter): refresh stored batch snapshot on reuse The dedup-reuse path in _mint_or_reuse_object returned the existing managed id without updating the stored file_object, so DB-served list responses kept the creation-time snapshot and showed null output_file_id/error_file_id even after the batch completed. Refresh the snapshot when an owned row is reused so the list reflects the batch's latest state. * fix(managed_id_rewriter): deny cross-owner object access on retrieve/cancel/delete Returning the raw id when can_access_resource fails only made sense for create responses, where the caller's own upstream create succeeded under a raw id that a different owner already holds. On retrieve/cancel/delete the caller reaches that branch only by supplying another tenant's raw id (which bypasses the managed-id input gate), so echoing the upstream object back leaked it cross-tenant. Restrict the raw fallback to create routes and return 404 otherwise. * fix(managed_id_rewriter): deny cross-owner file access on retrieve/delete _mint_or_reuse_file scoped the raw file dedup lookup to the current caller, so a raw file-... id belonging to another tenant was never found and the OUTPUT path minted a fresh managed id for that same upstream file under the caller. A raw id only reaches this path by skipping the managed-id input gate (raw provider ids are opt-out), so a different-owner row means the caller is touching someone else's file. Look up flat_model_file_ids globally and run can_access_resource; deny with 404 on retrieve/delete and leave the raw id unmanaged on create, which mirrors the cross-owner handling already in _mint_or_reuse_object. * fix(managed_id_rewriter): deterministic provider-scoped file dedup Replace the unscoped find_first in _mint_or_reuse_file with a find_many ordered by created_at and an application-layer provider filter. The file table has no provider column, so a raw file id shared across OpenAI and Azure could map to one row per provider; find_first then picked a row non-deterministically and, on a provider mismatch, minted a fresh managed row on every call, accumulating duplicates. Selecting the oldest matching same-provider row the caller can access keeps reuse stable and prevents duplicate rows while preserving the cross-tenant deny/leave-raw behaviour. * refactor(pass_through): scope passthrough managed IDs on the explicit provider Move the openai/azure detection out of pass_through_request into resolve_passthrough_managed_id_provider in llms/base_llm/managed_resources, and key managed-ID rewriting on the forwarded custom_llm_provider rather than the upstream URL's EndpointType. The helper documents why azure and azure_ai collapse to one "azure" scope (they share the same Azure OpenAI files/batches surface, so an ID minted on one must resolve on the other) and returns None for any other provider so a third-party OpenAI-compatible endpoint never triggers managed-ID minting. Add TestManagedIdProviderScope covering the azure_ai -> azure collapse and the non-openai/azure exclusion. * test(log_db_metrics): assert sanitized event_metadata contract test_log_db_metrics_success still asserted the legacy event_metadata shape (function_name/function_kwargs/function_args), which BerriAI#28909 intentionally removed so that live Prisma clients, OTel spans, and secrets never land on a service-log span. The decorator now emits only a sanitized payload: None when no table_name is present, and {"table_name": ...} when it is. Update the test to verify both branches of that contract. * fix(managed_id_rewriter): page provider-scoped file list by offset The file list scan advanced its cursor with a strict created_at boundary. When several rows shared a created_at timestamp and a non-matching provider row sat on the page boundary, the next query skipped the remaining rows at that timestamp, dropping matching files from the response. Page by a stable offset over a total order (created_at plus the unique id column) so tied rows are never skipped or repeated. * fix(managed_id_rewriter): push file-list provider scope to the DB The file-list helper had no provider column to query, so it scanned the table and filtered by decoding each managed ID in the application layer, capped at 20 pages. For an admin with a large mixed-provider file pool that cap could truncate a page. Mint now writes a _passthrough_provider:{provider} marker into flat_model_file_ids, giving the file table the same DB-queryable provider scope object rows already get from the namespaced model_object_id. The list helper pushes the scope into the query so a single round-trip serves the page. The scan loop, the offset paging, and the cap are gone, so pages can no longer truncate, leak the other provider, or skip rows that share a created_at timestamp. * fix(managed_id_rewriter): deny raw provider IDs that map to another tenant's managed resource Clients only ever receive managed IDs on passthrough, so a raw file/batch/response ID for another tenant's managed object can only be recovered by decoding that tenant's managed ID. Raw IDs were forwarded upstream untouched (deliberate opt-out), which on a retrieve/cancel/delete executed upstream before the response-side ownership check ran, leaking a cross-tenant action. Guard raw provider IDs on the input path: when a raw file-/batch_/resp_ ID resolves to a managed row the caller cannot access, return 404 before forwarding. Genuinely unmanaged raw IDs (no DB row) and IDs the caller owns are left untouched, preserving the opt-out. * test(managed_id_rewriter): cover azure_ai and pre-versioned azure passthrough paths * fix(managed_id_rewriter): fall back to raw id when persistence fails A DB persistence failure after a successful upstream create left the client holding a minted managed ID with no backing row, so every later resolve returned 404 and the resource was permanently unreachable. Mint the managed ID only when the row is stored; on persistence failure return the raw provider id, matching the no-persistence-available fallback, so the freshly-created resource stays reachable. * fix(managed_id_rewriter): log only rewritten query param keys * fix(managed_id_rewriter): use compound (created_at, id) list cursor boundary A timestamp-only lt/gt cursor boundary skips list rows that share the cursor row's created_at across a page boundary, silently dropping them. Compare the unique id (the secondary sort key) alongside created_at so the page walk stays complete when timestamps tie. * fix(managed_id_rewriter): converge concurrent object creates on one managed id * fix(managed_id_rewriter): bound raw-id guard DB lookups per request The INPUT guard fired one DB lookup for every file-/batch_/resp_ prefixed string in the path, query, and body. The file-id guard is an unindexed array-containment scan over LiteLLM_ManagedFileTable, so an authenticated caller could amplify a single passthrough request into thousands of full-table scans by packing a body with id-shaped strings. De-dupe raw ids within a request and cap the distinct guard lookups, failing closed with 400 instead of skipping the guard. Legitimate callers hold managed ids (resolved via an indexed unified_*_id lookup, not the guard), so the cap only trips under abuse. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
BerriAI#29256) * fix(proxy): enforce allowed_passthrough_routes for auth=true pass-through Pass-through endpoints with auth=true were injected into openai_routes, so teams with openai_routes access bypassed per-team allowed_passthrough_routes. Gate auth-enforced pass-through at JWT, virtual-key, and non-admin route checks. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): clarify JWT passthrough denial Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): make pass-through auth checks method-aware Prevent allowlist bypass when the same path is registered with different auth settings per HTTP method. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix passthrough route auth checks * fix(proxy): reject unregistered pass-through HTTP methods Enforce method-aware JWT checks and return 405 when stale FastAPI routes accept requests outside the current pass-through registry. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): remove duplicate request_method in JWT team lookup Fixes SyntaxError on proxy startup caused by passing request_method twice to find_team_with_model_access. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix passthrough route auth enforcement * fix(proxy): raise passthrough-specific 403 directly in virtual-key path * fix(proxy): load team for RBAC role-claim JWT passthrough gating * Revert "chore(tests): migrate Bedrock CI to AWS account 941277531214 (BerriAI#28728)" (BerriAI#29326) This reverts the Bedrock CI account migration (BerriAI#28728). The original account (888602223428) was put under an AWS security restriction after a leaked key and has since been reactivated, while the replacement account (941277531214) lacks access to several models the suites exercise (legacy Bedrock Claude 3 models, Cohere, Nova Canvas image gen, Bedrock batch inference, and flagship Opus). Pointing CI back at the reactivated account restores that coverage. This is the exact inverse of BerriAI#28728: all hardcoded 941277531214 references go back to 888602223428 (provisioned/imported-model ARNs, AgentCore runtime ARNs and their suffixes, batch execution role ARN, and the example proxy config), the S3 buckets revert to litellm-proxy and load-testing-oct, the guardrail IDs revert to wf0hkdb5x07f and ff6ujrregl1q, the SageMaker endpoint and Knowledge Base revert to their original ids, and the live-call tests go back to the legacy model strings. The grid_spec fail_reason workaround for the unentitled Opus cells is dropped while keeping the unrelated bedrock_effort_ceiling field added after the migration. The CircleCI AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY env vars still point at 941277531214 and must be set to the reactivated account's fresh credentials separately via the CircleCI API; AWS_REGION_NAME stays us-west-2. (cherry picked from commit f11c12d) * fix(proxy): scope pass-through 405 to registry routes; grant rerank passthrough in rpm tests The auth=true pass-through 405 guard fired for mapped provider routes (e.g. /assemblyai/*) that are not in the in-memory registry, since get_registered_pass_through_route returns None for them while is_registered_pass_through_route matches via mapped_pass_through_routes. Only raise 405 when the path is registered but the request method is not allowed, so mapped provider pass-throughs fall through to the default target params as before. The rpm-limit pass-through tests register /v1/rerank with auth=true but gave their keys no allowed_passthrough_routes, so the new default-deny returned 403 before the rate limiter ran (non-deterministically, depending on registry insertion order). Grant the keys explicit passthrough access so the tests exercise rate limiting under the new auth model. * fix(proxy): guard request method lookup against scopes without a method Starlette's Request.method property reads scope["method"] and raises KeyError when the scope omits it (e.g. minimally-constructed test requests). getattr only swallows AttributeError, so the new _get_request_method helper propagated the KeyError up through user_api_key_auth and surfaced as a ProxyException. Catch KeyError (and AttributeError) and fall back to None. * test(passthrough): pin SERVER_ROOT_PATH in unregistered-method test test_custom_proxy.py sets os.environ['SERVER_ROOT_PATH'] = '/my-custom-path' at module import with no cleanup. When that module is collected into the same xdist worker as this test, the leaked root path is prepended to registered pass-through paths, so is_registered_pass_through_route misses '/test/path' and the handler returns 404 instead of the expected 405 (order-dependent). Pin SERVER_ROOT_PATH to '' so the test is deterministic. * test(passthrough): restore regression coverage for non-auth-enforced pass-through via llm_api_routes * fix(proxy): record auth flag in pass-through registry for allowlist enforcement Auth-enforced pass-through detection inferred enforcement from the FastAPI dependency stored at registration time. The management create and update endpoints register routes with dependencies=None even though auth defaults to true, so is_auth_enforced_pass_through_route treated those DB-created routes as unenforced. A key allowed for llm_api_routes could then call a management-created auth-enabled pass-through route without matching allowed_passthrough_routes. Store the auth setting on each registry entry and read it directly when deciding whether the allowlist applies, instead of deriving it from dependency metadata. * fix(proxy): include bool in pass-through registry value type for auth flag The auth flag stored in _registered_pass_through_routes is a bool, which was not part of the registry value Union, so mypy rejected the dict literal. Add bool to the Union and narrow route_methods to a list before the membership check so the in-operator stays valid. * fix(proxy): preserve stored auth flag on pass-through endpoint update model_dump(exclude_none=True) re-included the auth=True default whenever a partial update omitted auth, silently flipping an existing auth=false pass-through to auth-enforced and 403ing every team/key without allowed_passthrough_routes. Merge only explicitly set fields via exclude_unset so omitted fields keep their stored value. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
…gnment (BerriAI#29313) * fix(mcp): make key.access_group_ids grants additive over team ceiling A key whose unified access_group_ids grant a private MCP server was having that grant intersected against its team's MCP ceiling, so a key in a team scoped to other servers (or with no own scope) lost the granted server entirely. Resolve access_group_ids once as ungated additive grants and union them on top of the key/team ceiling instead of folding them into the key scope that gets intersected. * test(mcp): align key access-group tests with additive-grant model The previous commit moved key.access_group_ids resolution out of the intersected key ceiling (_get_allowed_mcp_servers_for_key) and into the ungated additive grant path (_get_key_access_group_mcp_server_extras), unioned on top of the team ceiling. Five tests from BerriAI#28890/BerriAI#29195 still asserted the old gated / in-key-scope contract and failed: - _get_allowed_mcp_servers_for_key now returns the object_permission ceiling only and never resolves access_group_ids; two tests now assert the group resolver is not called from that path (with and without an object_permission present). - The extras path is ungated, so a group whose assigned_team_ids / assigned_key_ids exclude the caller still contributes its servers. - The end-to-end test asserts the grant surfaces via the extras path rather than the base key path. - Dropped test_key_access_group_ids_empty_returns_no_extras; the empty case is already covered by the extras family's no-groups test. * feat(auth): gate member access-group assignment on keys behind opt-in Non-admin team members could attach access_group_ids to keys they create or update, letting them self-grant resources (MCP servers/models) the team admin never intended. Add an opt-in KEY_ACCESS_GROUP_ASSIGNMENT team-member permission (default-deny) enforced at /key/generate and /key/update; proxy and team admins bypass. Surfaces automatically as a checkbox in the team Member Permissions UI. * fix(auth): gate access-group assignment on /key/regenerate too RegenerateKeyRequest inherits access_group_ids and prepare_key_update_data persists it, so a non-admin key owner could self-grant access groups by regenerating. Apply the same opt-in member gate using the existing key's team. * test(auth): cover member access-group gate and additive MCP grants Add unit tests for enforce_member_can_assign_access_groups (deny without opt-in, allow with opt-in, and proxy-admin / team-admin / non-team-key bypasses) and for _get_key_access_group_mcp_server_extras (no-auth and no-resolved-servers return empty, resolved ids are expanded, errors degrade to no grants).
…eroing counter (BerriAI#29358) * fix(reset_budget): write only {spend, budget_reset_at} and stop pre-zeroing counter ResetBudgetJob's batched update_data path shipped the full key/user/team model on each reset. Prisma rejects object_permission_id and budget_limits on the update input type, so any row carrying those fields detonated the entire batch -- spend never reset, budget_reset_at never advanced. After v1.84.0 started populating object_permission_id on UI-created keys, this fires routinely. _reset_budget_common also zeroed the cross-pod spend counter before the DB write, so failed resets left enforcement reading 0 from the counter while the DB still held the over-budget spend, admitting requests past the cap until the counter naturally re-saturated from new reservations. Switch the write to per-row narrow updates ({spend, budget_reset_at}) via db.batch_, and move the counter invalidation out of _reset_budget_common so it only fires after the DB write commits. On DB-write failure the counter is left untouched, enforcement continues to block, and the next scheduler tick can retry without leaving a bypass window. Fixes BerriAI#27730. * fix(reset_budget): address Greptile review on BerriAI#29358 - Strengthen the bypass-half regression test: replace the for-loop over call_args_list (vacuously true when empty) with assert_not_called(), so the test would actually flag a re-introduction of counter-zeroing via any code path. - Add the same explanatory docstring on _write_user_reset_updates and _write_team_reset_updates that _write_key_reset_updates already has, so all three helpers point future maintainers at BerriAI#27730. * test(reset_budget): update test_proxy_budget_reset for new batch-write path Same shape as the previous test_reset_budget_job.py update: keys/users/teams now write through prisma.db.batch_().<table>.update, not update_data, so the tests need a batcher mock and updated assertions. Adds: - _wire_batcher_for_test helper that returns a list which accumulates per-row batch updates captured from prisma_client.db.batch_(). - _attrify helper that wraps dict fixtures so getattr(item, "token") works alongside the dict item-access the fake_reset_* mocks rely on. The new narrow-write helpers use getattr to pull out the row's id, and would silently skip plain dicts otherwise. - Updates 3 partial_failure tests to assert against the batch-call list (rows by id, payload contains only {spend, budget_reset_at}) instead of update_data.assert_awaited_once + data_list inspection. - Updates test_reset_budget_continues_other_categories_on_failure: only budget + enduser still flow through update_data; key/user/team go through the batch path now. - Wires the batcher mock into 3 service_logger_*_success tests so commit() is actually awaitable and the success hook fires. These tests were silently passing locally only because the editable install in .venv pointed at the main repo, not the worktree — running pytest with PYTHONPATH overridden to the worktree (matching CI) reproduces the failures.
* test(e2e): cover PROXY_LOGOUT_URL redirect on Logout Env-gated spec mirroring the existing serverRootPathRedirect pattern: when the proxy is booted with PROXY_LOGOUT_URL set, clicking Logout in the navbar must navigate to that external URL. The standard run_e2e.sh exports an empty value so the rest of the suite is unaffected; this spec self-skips unless the env var is populated. * test(e2e): run PROXY_LOGOUT_URL spec in the suite + harden logout assertions Boot the e2e proxy with PROXY_LOGOUT_URL set (job-level env in CircleCI and run_e2e.sh) so proxyLogoutUrl.spec.ts actually runs instead of self-skipping. Nothing else in the suite performs a logout, so this only affects the behavior under test. Harden the spec to verify the logout flow rather than a URL substring: - wait for /sso/get/ui_settings before clicking so logoutUrl is populated (otherwise window.location.href = "" silently reloads same-origin) - assert a token cookie exists first, and is cleared after logout - locate the dropdown via getByRole instead of internal antd CSS classes - stub the external destination and assert on URL origin + path prefix * test(e2e): assert exact PROXY_LOGOUT_URL on logout redirect Replace the origin + startsWith(pathname) checks with a single normalized href comparison. With PROXY_LOGOUT_URL=https://www.example.com the path was "/", so startsWith("/") matched any path and left path/query/hash unchecked. Comparing normalized hrefs pins scheme, host, port, path, query and hash while still tolerating the browser's trailing-slash/default-port normalization.
When the user has visited both the dev UI (e.g. localhost:3000) and the
proxy UI (e.g. localhost:4000) in the same tab, logging out from the dev
origin produced an infinite logout/login redirect.
The proxy-side LoginPage's "is the user still authenticated?" check
was reading getCookie("token"), which falls back to sessionStorage when
document.cookie has no token. The cross-origin clearTokenCookies() call
from the dev origin can clear cookies on the shared hostname, but cannot
reach sessionStorage on the proxy origin (sessionStorage is per-origin),
so the fallback returned a stale token and LoginPage interpreted the
user as logged in, redirecting back to the dev origin. Dev origin then
saw no cookie and redirected to LoginPage, repeating ~20x per second.
This change introduces getCookieFromDocument(), a cookie-only read with
no sessionStorage fallback, and uses it in LoginPage's already-logged-in
check. The HttpOnly-reverse-proxy defense from PR BerriAI#23532 is unaffected:
storeLoginToken still writes both the JS cookie at /ui and the
sessionStorage backup, and getCookie still falls back for callers that
want the full read path.
…erriAI#29369) The Next.js admin UI is exported with trailingSlash: true, so the proxy serves /ui/login at /ui/login/index.html and 308s /ui/login → /ui/login/. The waitForURL predicate used endsWith("/ui/login"), which never matched the canonicalized URL and timed out after 15s. This was masked until the build artifacts were regenerated against the AuthContext fix: the prior bundles still hit the racy redirect path that fired before proxyBaseUrl was populated, producing /ui/login (no prefix, no proxy round-trip, no trailing slash) which fortuitously satisfied the predicate. The first PR to ship the corrected bundle exposed the assertion bug. Switch the predicate to includes("/ui/login"); the prefix assertion below still validates the SERVER_ROOT_PATH preservation that is the actual contract under test.
* bump: version 0.1.41 → 0.1.42 * uv lock
chore(ci): promote internal staging to main
delete_model passed the internal model_name (model_name_{team_id}_{uuid})
to delete_team_model_alias, which searched alias map values for it.
For BYOK models the alias map is typically empty, so the lookup always
failed and the public name was never removed from team.models; creating
a ghost model visible via /models indefinitely.
Adds _resolve_team_public_model_name to extract the user-visible name
from model_info.team_public_model_name. Adds _cleanup_team_model_references
which always strips the public name from team.models (regardless of alias
map state) and also cleans up any alias entries pointing to the deleted
model's internal name. Wires _refresh_cached_team so /models reflects
the deletion immediately.
Fixes BerriAI#22594
On behalf of Adobe Inc.
Contributor
|
Too many files changed for review. ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relevant issues
Fixes #22594
Related PRs: #22595 (dibyom, stale), #28833 (oss-agent-shin, has blockers), #29526 (companion create-side fix)
Pre-Submission checklist
make test-unit@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewScreenshots / Proof of Fix
Reproduction script creates a team with 2 BYOK models sequentially, then deletes them. Before the fix, both models persist as ghosts in
team.modelsand/modelsindefinitely. After the fix, deleted models are removed immediately.Also tested the alias-overwrite scenario: create model-1, create model-2 (overwrites model-1's alias entry), delete model-1. Before the fix, model-1 persists as a ghost. After the fix, model-1 is cleaned up and model-2 is preserved.
Type
Bug Fix
Changes
delete_modelpassedmodel_params.model_name(the internal name, e.g.model_name_{team_id}_{uuid}) todelete_team_model_alias, which searched alias map values for a match. For BYOK models the alias map is typically empty or was overwritten by a later model create, so the lookup always failed and the public name was never removed fromteam.models.This PR adds two helpers and rewires the delete path:
_resolve_team_public_model_name: extracts the user-visible name frommodel_info.team_public_model_name, falling back tomodel_params.model_namefor non-BYOK models._cleanup_team_model_references: builds anames_to_removeset starting with the public name (unconditionally), then adds any alias keys whose values match the internal name. Strips all matched names fromteam.modelsand cleans up the alias map in the same pass.After the DB cleanup, calls
_refresh_cached_team(already exists on main) so/modelsreflects the deletion without waiting for cache TTL. The cache refresh is wrapped in try/except so a Redis failure does not undo the DB delete.delete_team_model_aliasis left in place; other code paths may still reference it.