Skip to content

fix(proxy): remove ghost team BYOK models from /models on delete - #29527

Closed
aanchal22 wants to merge 112 commits into
BerriAI:litellm_oss_branchfrom
aanchal22:fix/team-byok-delete-ghost-models
Closed

fix(proxy): remove ghost team BYOK models from /models on delete#29527
aanchal22 wants to merge 112 commits into
BerriAI:litellm_oss_branchfrom
aanchal22:fix/team-byok-delete-ghost-models

Conversation

@aanchal22

Copy link
Copy Markdown
Contributor

Relevant issues

Fixes #22594

Related PRs: #22595 (dibyom, stale), #28833 (oss-agent-shin, has blockers), #29526 (companion create-side fix)

Pre-Submission checklist

  • I have added meaningful tests
  • My PR passes all unit tests on make test-unit
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Reproduction script creates a team with 2 BYOK models sequentially, then deletes them. Before the fix, both models persist as ghosts in team.models and /models indefinitely. After the fix, deleted models are removed immediately.

Before (v1.85.0):
  After deleting both models:
  team.models in DB: ['all-proxy-models', 'bug2-model-1', 'bug2-model-2']
  Ghosts: both models still visible via /models

After (this PR):
  After deleting model-2:
  team.models in DB: ['all-proxy-models', 'bug2-model-1']
  After deleting model-1:
  team.models in DB: ['all-proxy-models']
  No ghosts.

Also tested the alias-overwrite scenario: create model-1, create model-2 (overwrites model-1's alias entry), delete model-1. Before the fix, model-1 persists as a ghost. After the fix, model-1 is cleaned up and model-2 is preserved.

Type

Bug Fix

Changes

delete_model passed model_params.model_name (the internal name, e.g. model_name_{team_id}_{uuid}) to delete_team_model_alias, which searched alias map values for a match. For BYOK models the alias map is typically empty or was overwritten by a later model create, so the lookup always failed and the public name was never removed from team.models.

This PR adds two helpers and rewires the delete path:

  • _resolve_team_public_model_name: extracts the user-visible name from model_info.team_public_model_name, falling back to model_params.model_name for non-BYOK models.
  • _cleanup_team_model_references: builds a names_to_remove set starting with the public name (unconditionally), then adds any alias keys whose values match the internal name. Strips all matched names from team.models and cleans up the alias map in the same pass.

After the DB cleanup, calls _refresh_cached_team (already exists on main) so /models reflects the deletion without waiting for cache TTL. The cache refresh is wrapped in try/except so a Redis failure does not undo the DB delete.

delete_team_model_alias is left in place; other code paths may still reference it.

mateo-berri and others added 30 commits May 23, 2026 12:15
… fixes, model catalog) (BerriAI#28223)

* fix(opentelemetry): JSON-serialize dict metadata fields for OTEL span attributes (#27451) (#27455)

Squash-merged by litellm-agent from Anai-Guo's PR.

* feat(dashscope): add embeddings and reranks(qwen3-rerank) support via OpenAI-compatible endpoint (#27508)

Squash-merged by litellm-agent from yimao's PR.

* fix(vertex_ai/gemini): raise BadRequestError when image_url or url fi… (#24550)

Squash-merged by litellm-agent from krisxia0506's PR.

* fix(vertex_ai): raise error on mid-stream 429/error chunks instead of silently swallowing (#23711)

Squash-merged by litellm-agent from krisxia0506's PR.

* fix: raise BadRequestError for file content blocks missing 'file' sub… (#24503)

Squash-merged by litellm-agent from krisxia0506's PR.

* Fix Gemini MIME detection for extensionless GCS URIs (#27278)

Squash-merged by litellm-agent from krisxia0506's PR.

* fix(vertex_ai/partner_models): drop unused vertexai SDK gate from count_tokens (closes #28084) (#28107)

Squash-merged by litellm-agent from voidborne-d's PR.

* feat(chart): add support for autoscaling behavior in HPA (#27990)

Squash-merged by litellm-agent from FabrizioCafolla's PR.

* feat(proxy): add blocked flag to models for pause/resume from the UI (#27927)

Squash-merged by litellm-agent from Cyberfilo's PR.

* fix: pass socket timeouts to Redis cluster clients (#27920)

Squash-merged by litellm-agent from tomdee's PR.

* Fix/cache token (#28009)

Squash-merged by litellm-agent from escon1004's PR.

* fix(deepseek): forward reasoning_content in multi-turn thinking mode conversations (#28080)

Squash-merged by litellm-agent from Divyansh8321's PR.

* fix(guardrails): return HTTP 400 instead of 500 for blocked requests (#27617)

* fix: reset org and tag budgets (#27326)

* reset org budgets

* reset tag budgets

---------

Co-authored-by: Michael Riad Zaky <michaelr@Mac.localdomain>

* fix(ui): omit allowed_routes from key edit save when unchanged (#27553)

* fix(ui): omit allowed_routes from key edit save when unchanged

When a team admin opens Edit Settings on a key with key_type=AI APIs and
saves without changing anything, the UI re-sends the existing allowed_routes
value, which the backend's _check_allowed_routes_caller_permission gate
rejects for non-proxy-admins (LIT-2681).

Strip allowed_routes from the patch in handleSubmit when it deep-equals the
original keyData.allowed_routes. The backend treats absence as "leave alone,"
so no-op saves now succeed for non-admins. Admins explicitly editing the
field still send the new value.

* fix(ui): order-insensitive allowed_routes diff + cover null-original case

Address Greptile review:

- Switch the "is allowed_routes unchanged" check to a Set-based comparison so
  a server-side reorder of the array doesn't register as a user edit and
  re-trigger LIT-2681.
- Add two regression tests: (1) keyData.allowed_routes is null and the form
  is untouched — patch should strip the field; (2) server returned routes in
  a different order than the user originally entered — patch should still
  recognize the value as unchanged.

* chore(ui): strip ticket refs and tighten comments in key edit fix

- Remove internal-tracker references from in-code comments
- Tighten the WHY comment in handleSubmit to two lines
- Drop redundant test-block comments — test names already describe the case

* fix(ui): annotate Set<string> generic in allowed_routes diff to fix tsc

* fix(guardrails): return HTTP 400 instead of 500 for guardrail-blocked requests

GuardrailRaisedException and BlockedPiiEntityError both lacked a
status_code attribute.  When these exceptions reached the proxy
exception handler (getattr(e, 'status_code', 500)), the fallback
defaulted to HTTP 500 — making intentional guardrail blocks
indistinguishable from server errors and causing unnecessary client
retries.

Changes:
- Add status_code=400 (keyword-only) to GuardrailRaisedException
- Add status_code=400 (keyword-only) to BlockedPiiEntityError
- Update _is_guardrail_intervention() to recognize both exceptions
  so downstream loggers record 'guardrail_intervened' instead of
  'guardrail_failed_to_respond'
- Add 6 unit tests for default/custom status codes and getattr pattern
- Strengthen existing blocked-action test with status_code assertion

Fixes #24348

---------

Co-authored-by: Michael-RZ-Berri <michael@berri.ai>
Co-authored-by: Michael Riad Zaky <michaelr@Mac.localdomain>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Co-authored-by: Krrish Dholakia <krrish+github@berri.ai>

* fix(router/proxy): address Greptile P1+P2 review comments on PR #28161

- router: raise ServiceUnavailableError (503) instead of RouterRateLimitErrorBasic (429)
  when a specifically-addressed deployment is administratively blocked; 429 misleads
  retry-enabled clients into spinning forever against a paused model
- proxy_server: compute get_fully_blocked_model_names() once before both branches in
  model_list() instead of duplicating the call in each branch
- deepseek: upgrade silent debug log to warning when injecting placeholder
  reasoning_content so callers are clearly notified of degraded multi-turn quality
- tests: update two blocked-deployment assertions to expect ServiceUnavailableError

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: address bug detection findings (cache token order, mutable defaults)

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix: address bugs in async pass-through, anthropic cache token detection, rerank tests

- async_get_available_deployment_for_pass_through: enforce blocked check on specific deployments
- cost_calculator: detect anthropic-style usage by attribute presence (not truthiness) to avoid mixing OpenAI cached_tokens into anthropic normalization when read=0
- dashscope rerank tests: pass request to httpx.Response constructions for consistency

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix code qa

* fix(vertex_ai/gemini): strip MIME parameters from GCS contentType

GCS object metadata's contentType field can include parameters such as
'text/html; charset=utf-8'. Strip them in _apply_gemini_mime_type_aliases
so downstream get_file_extension_from_mime_type sees a bare MIME type.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(vertex_ai/gemini): clarify mime-type error message string concatenation

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* feat(oci): add embeddings, fix streaming/reasoning, expand model catalog

- Add OCIEmbedConfig with full Cohere embed support (7 models, batch up to 96)
- Fix sync streaming: split SSE events on \n\n before JSON parsing
- Fix reasoning models (Gemini 2.5, xAI Grok): make completionTokens and message
  optional in OCIResponseChoice to handle max_tokens exhausted on reasoning
- Fix compartment_id resolution in chat transform to use resolve_oci_credentials
- Fix tool call id: make OCIToolCall.id optional, generate UUID fallback for
  providers (Google via OCI) that omit it
- Add OCI_KEY env var support for inline PEM keys
- Fix datetime.utcnow() deprecation in request signing
- Expand model catalog: 29 OCI models including Llama 4, Gemini 2.5, xAI Grok,
  Cohere Command A, and all Cohere embed variants
- Add 37 live integration tests: sync/async completions for Meta/Google/xAI/Cohere,
  sync/async embeddings, tool use across all vendors, streaming, env var auth
- Add 23 embed unit tests covering all transform and validation paths

* fix(oci): remove dead OCI elif branch in utils.py, align async split_chunks with sync version

* test(oci): add unit tests for split_chunks fix and no-duplicate-OCI-branch guard

* fix(oci): address remaining bugs from issue #25082 — streaming signed body, Cohere stop sequences, hardcoded defaults

- Bug 1: sync and async streaming paths now use signed_json_body when provided
  instead of re-serializing data with json.dumps() — the OCI RSA-SHA256 signature
  covers the exact request body bytes, so re-serializing produces an invalid sig
- Bug 3: Cohere stop sequences now map to 'stopSequences' (was incorrectly 'stop')
- Bug 4: removed hardcoded Cohere defaults (maxTokens=600, temperature=1, topK=0,
  topP=0.75, frequencyPenalty=0) that silently overrode user intent on every call
- Added 6 unit tests covering all three fixes

* fix(oci): comprehensive code quality pass — bugs, tests, schema accuracy

- Fix Cohere tool call IDs (was always call_0; now UUID per call)
- Fix TOOL_CALL finish reason mapping in both sync and streaming paths
- Fix Cohere stop parameter mapping (stop → stopSequences)
- Remove hardcoded Cohere defaults (maxTokens/topK/topP/frequencyPenalty)
- Fix content[0] safety guard against empty content arrays
- Fix streaming signed body used consistently (not re-serialized)
- Raise OCIError (not bare Exception/ValueError) throughout
- Centralize OCI_API_VERSION constant; import uuid at module level
- Fix embed get_complete_url to strip trailing slashes from api_base
- Fix OCIEmbedResponse schema: add inputTextTokenCounts (actual OCI field)
- Fix embed usage computed from inputTextTokenCounts (sum of per-input counts)
- Fix Cohere toolCallId included in tool result messages
- Add OCIToolCall.id as Optional (absent in Google/xAI streaming chunks)
- Update tests to reflect correct behavior (no hardcoded defaults, UUID ids,
  deferred credential validation, OCIError vs ValueError, real response schema)

* test(oci): move integration tests to tests/llm_translation/

Addresses greptile P1: tests/test_litellm/ is for mock-only unit tests
(make test-unit target). Real-network OCI tests now live in the correct
location alongside other provider integration tests.

* fix(oci): align types and transformation with official OCI SDK

- Remove OCIVendors.GEMINI — apiFormat="GEMINI" is invalid; all non-Cohere
  models use apiFormat="GENERIC"
- Add toolChoice, logitBias, logProbs to OCIChatRequestPayload so params
  present in the mapping are no longer silently dropped by Pydantic
- Exclude n→numGenerations from Cohere param map (not a Cohere API field)
- Fix CohereToolResult: change callId/result to call/outputs matching
  the OCI SDK's CohereToolResult structure
- Fix CohereToolMessage: replace non-existent toolCallId with toolResults
  list; update adapt_messages_to_cohere_standard to build proper tool-result
  history entries by resolving tool call name+params from preceding assistant
  messages
- Map generic-model stream finish reasons to OpenAI convention
  (COMPLETE→stop, MAX_TOKENS→length, TOOL_CALLS→tool_calls), consistent
  with the existing Cohere streaming path
- Add optional id field to OCIEmbedResponse so valid API responses
  carrying an id are not rejected by the Pydantic model

* fix(oci): use 'output' key in Cohere tool result outputs (matches reference impl)

* fix(oci): port schema/type utilities from langchain-oracle reference impl

- Add resolve_oci_schema_refs: inline $ref/$defs — OCI rejects JSON Schema refs
- Add resolve_oci_schema_anyof: flatten Optional[T] anyOf (Pydantic v2 emits these)
- Add sanitize_oci_schema: strip title, normalise null types, ensure array items
- Add OCI_JSON_TO_PYTHON_TYPES: Cohere expects Python type names (str/int/float),
  not JSON Schema names (string/integer/number)
- Add enrich_cohere_param_description: embed enum/format/range/pattern constraints
  into description since CohereParameterDefinition has no dedicated fields
- Apply all of the above in adapt_tool_definitions_to_cohere_standard and
  adapt_tool_definition_to_oci_standard
- Fix toolChoice conversion: map OpenAI string ('auto','none','required') to OCI
  dict form ({"type":"AUTO"} etc.) — the API rejects plain strings
- Update unit test expectations to match correct Python type names and enriched
  descriptions

* refactor(oci): split transformation.py into cohere.py and generic.py

transformation.py was 1 243 lines doing too many jobs. Split along the
same boundaries as the langchain-oracle reference (providers/cohere.py,
providers/generic.py):

  chat/cohere.py   — Cohere message/tool building, response + stream parsing
  chat/generic.py  — Generic message/tool building, response + stream parsing
  transformation.py — thin OCIChatConfig orchestrator + OCIStreamWrapper

Public symbols (OCIChatConfig, OCIStreamWrapper, adapt_messages_to_*,
OCIRequestWrapper, version, …) remain importable from transformation.py
for backward compatibility. OCIStreamWrapper gains delegating shims for
_handle_cohere_stream_chunk and _handle_generic_stream_chunk so existing
test call sites keep working unchanged.

transformation.py: 1 243 → 620 lines

* refactor(oci): principal-level code quality pass

- Remove _extract_text_content duplication — single definition in cohere.py,
  imported where needed; instance method on OCIChatConfig eliminated
- Move cryptography imports to module level with _CRYPTOGRAPHY_AVAILABLE flag
  and _require_cryptography() guard; no more re-import on every signing call
- Move litellm version import to module level via litellm._version; remove
  inline import inside validate_oci_environment
- sign_with_manual_credentials now returns Tuple[dict, bytes] matching
  sign_with_oci_signer — asymmetry eliminated, Optional[bytes] guards removed
  throughout stream wrappers (signed_json_body: bytes = b"")
- Rename _openai_to_oci_cohere_param_map → openai_to_oci_cohere_param_map
  for consistency with openai_to_oci_generic_param_map
- Remove double-key bug in map_openai_params where responseFormat was stored
  under both OCI and OpenAI key names simultaneously
- Remove delegating shims (adapt_messages_to_cohere_standard,
  adapt_tool_definitions_to_cohere_standard, _handle_generic_stream_chunk)
  from OCIChatConfig/OCIStreamWrapper; tests now import directly from
  cohere.py and generic.py where symbols live
- Trim __all__ to 7 genuine public symbols; remove the 13-symbol list that
  existed only to support test imports
- Collapse per-model integration test classes into pytest.mark.parametrize;
  CHAT_MODELS list is the single source of truth for model-specific config
- Black + Ruff clean across all OCI files

* fix(oci): address PR review findings

- types/llms/oci.py: add "TOOL_CALL" to CohereChatResponse.finishReason
  Literal so Pydantic does not raise ValidationError on non-streaming
  Cohere tool-use calls (Greptile P1)
- test_oci_cohere_tool_calls.py: add test covering TOOL_CALL finish reason
- model_prices_and_context_window.json: remove 6 duplicate oci/cohere.embed-*
  keys that were silently overridden by the more complete entries already
  present in the file (Greptile P1)
- common_utils.py: move OCI_API_VERSION here from chat/transformation.py
  so embed/transformation.py does not need to import chat/transformation;
  change Protocol stub body from ... to pass (CodeQL "statement no effect");
  add comment to sha256_base64 clarifying it implements OCI HTTP signing
  spec, not password hashing (CodeQL false positive)
- chat/transformation.py: import CustomStreamWrapper from
  litellm_core_utils.streaming_handler instead of litellm.utils to reduce
  import cycle depth (CodeQL cyclic import)
- chat/cohere.py, chat/generic.py: import Usage and
  ChatCompletionMessageToolCall from litellm.types.utils instead of
  litellm.utils for the same reason
- embed/transformation.py: import OCI_API_VERSION from common_utils
  instead of chat/transformation (removes the embed→chat import edge)

* test(oci): add unit tests to improve patch coverage

- test_oci_common_utils.py (new): covers sha256_base64, build_signature_string,
  OCIRequestWrapper.path_url, resolve_oci_credentials, get_oci_base_url,
  validate_oci_environment, sign_with_oci_signer error paths, sign_oci_request
  routing, load_private_key_from_file error paths, resolve_oci_schema_refs
  (including circular ref and external $ref), resolve_oci_schema_anyof,
  sanitize_oci_schema (all branches), enrich_cohere_param_description
- test_oci_generic_chat.py (new): covers content-message error paths (non-dict
  item, unsupported type, non-string text, invalid image_url), tool-call
  validation error paths, adapt_messages_to_generic_oci_standard error paths,
  handle_generic_response (None message, text content, tool calls),
  handle_generic_stream_chunk (finish reasons, streaming tool calls),
  OCIStreamWrapper non-string chunk error
- test_oci_chat_transformation.py: add error paths for validate_environment
  (empty messages), transform_request (missing compartment_id, Cohere without
  user messages), transform_response (error key), map_openai_params
  (unsupported param with and without drop_params), tool_choice string mapping
- test_oci_cohere_tool_calls.py: add edge cases for stream chunk finish
  reasons (TOOL_CALL, MAX_TOKENS, unknown), _extract_text_content with
  non-dict list items and non-string input,
  adapt_messages_to_cohere_standard with malformed JSON tool arguments

* fix(oci): rename supports_streaming to supports_native_streaming in model prices

The JSON schema for model_prices_and_context_window.json uses
`supports_native_streaming` (not `supports_streaming`) and has
`additionalProperties: false`. Rename the field across all OCI
entries to pass the schema validation test.

* test(oci): add 67 tests targeting uncovered happy paths for coverage

Boost patch coverage on the four lowest-coverage OCI files:
- common_utils.py: sign_with_manual_credentials (oci_key / oci_key_file
  paths), sign_oci_request routing, _require_cryptography
- generic.py: adapt_messages_to_generic_oci_standard (all roles),
  adapt_tool_definition_to_oci_standard, adapt_tools_to_openai_standard,
  handle_generic_stream_chunk text/finish-reason paths
- cohere.py: _extract_text_content, adapt_messages_to_cohere_standard
  (all roles including tool results), handle_cohere_response /
  handle_cohere_stream_chunk all finish-reason branches
- transformation.py: get_vendor_from_model, OCIChatConfig._get_optional_params
  (toolChoice string→dict, responseFormat, tools for both vendors),
  transform_request for GENERIC model, get_sync/async_custom_stream_wrapper
  with mocked HTTP, OCIStreamWrapper.chunk_creator happy paths

* fix(oci): suppress CodeQL false positive on sha256_base64 (OCI HTTP signing, not password hashing)

* fix(oci): remove 6 duplicate model price entries and reconcile conflicting values

Six OCI chat model keys appeared twice in model_prices_and_context_window.json
with conflicting pricing/context data (JSON parsers silently discard the first).
Remove the first-occurrence entries and update the surviving entries:
- meta.llama-4-maverick / llama-4-scout: keep updated entries (free preview
  pricing, larger context windows, vision support)
- meta.llama-3.1-70b: keep original pricing, restore supports_native_streaming
- google.gemini-2.5-{flash,pro,flash-lite}: keep OCI pricing page values,
  restore supports_native_streaming

* fix(oci): route GPT-5 family to maxCompletionTokens

GPT-5 / GPT-5-mini / GPT-5-nano / GPT-5.5 on OCI reject "maxTokens"
with HTTP 400:

  Invalid 'maxTokens': Unsupported parameter: 'maxTokens' is not
  supported with this model. Use 'maxCompletionTokens' instead.

(Same convention as OpenAI's reasoning-API contract.)

Add a model-aware rename in OCIChatConfig._get_optional_params so the
request payload uses maxCompletionTokens when the model id starts with
openai.gpt-5. Regular Llama / Cohere / Gemini / GPT-4.x continue to use
maxTokens unchanged.

Also widen OCIChatRequestPayload to carry the new optional field so it
survives Pydantic serialization.

Verified live against OCI us-chicago-1:
- openai.gpt-5, gpt-5-mini, gpt-5-nano, gpt-5.5 all return 200
- Full feature sweep on gpt-5.5 (basic, system, multi-turn, streaming,
  tools, usage) all green
- meta.llama-3.3-70b-instruct still uses maxTokens (no regression)

4 new unit tests cover the helper, the routing in both pre- and
post-translation states, and Pydantic serialization.

* ci(oci): fix CI failures — black formatting + recursive_detector ignore

- Run black on litellm/llms/oci/common_utils.py + 3 OCI test files
  that drifted out of black-compliance during the rebase.
- Add the three bounded recursive functions in oci/common_utils.py
  (`_resolve`, `resolve_oci_schema_anyof`, `sanitize_oci_schema`) to
  the recursive_detector IGNORE_FUNCTIONS list. All three are bounded:
  `_resolve` uses a `resolving_stack` cycle guard; the other two are
  bounded by JSON-schema tree depth (no cycles in well-formed input),
  matching the pattern of the existing OCI/Vertex schema walkers
  already on the list.

* fix(oci): silence MyPy errors in cohere.py — typed-dict access

Two errors flagged by `lint` CI:

  llms/oci/chat/cohere.py:73:  "object" has no attribute "__iter__"
  llms/oci/chat/cohere.py:119: No overload variant of "get" of "dict"
                               matches argument types "object", "CohereToolCall"

Both stem from `msg.get("tool_calls")` / `msg.get("tool_call_id")`
returning `object` per the AllMessageValues TypedDict union. Bind to
`Any` locally for the iteration and coerce the lookup key with `str()`,
removing the now-unused `# type: ignore` on those lines.

No behaviour change — pure type-narrowing for the type checker.

* fix(oci): silence CodeQL py/weak-sensitive-data-hashing on sha256_base64

CodeQL's taint analysis traces request bodies back to environment-loaded
secrets and flags `hashlib.sha256(body).digest()` as
`py/weak-sensitive-data-hashing` — even though SHA-256 is the algorithm
mandated by the OCI HTTP request signing spec for the
`x-content-sha256` header (not a password/secret hash).

The previous suppression used legacy `# lgtm[...]` syntax which the
modern CodeQL action ignores. Switch to Python's standard
`hashlib.sha256(..., usedforsecurity=False)` (Python 3.9+) which CodeQL
honours as a non-security declaration. Behaviour unchanged.

* feat(oci): add reasoning_effort passthrough — only true missing primitive

OCI's GenericChatRequest exposes a reasoningEffort field
(NONE/MINIMAL/LOW/MEDIUM/HIGH) that's the single biggest cost knob for
reasoning-capable models on the service:

  - GPT-5 family
  - Gemini 2.5
  - Grok reasoning variants (3-mini, 4-fast, 4.20)
  - Cohere Command-A-Reasoning

Setting reasoning_effort=LOW typically cuts reasoning-token spend 5-10×
vs the default. Without exposing this, litellm users had no way to tune
cost-vs-quality on these models.

The other GenericChatRequest fields (verbosity, parallel_tool_calls,
logit_bias, n, metadata, web_search_options, prediction) are not
exposed because they are not missing primitives — they either duplicate
prompt-engineering, framework-level controls, or are too niche to
justify the maintenance surface. We only ship what users genuinely
can't accomplish another way.

Excluded from the Cohere v1 param map: CohereChatRequest has no
reasoningEffort field, and Cohere reasoning models
(cohere.command-a-reasoning) use COHEREV2 which is a separate request
type not covered by this PR.

Verified live: GPT-5.5 + reasoning_effort="HIGH" sends
{"reasoningEffort": "HIGH"} on the wire and OCI accepts the request.

* feat(oci): reasoning_effort + reasoning_tokens for OCI GenAI

Three small additions for OCI reasoning models, requested by users
testing the PR in production fork builds:

1. **reasoning_effort param mapping (GENERIC vendors).** OCI expects
   uppercase levels ("LOW"/"MEDIUM"/"HIGH"/"NONE") on `reasoningEffort`,
   but OpenAI-compatible clients send lowercase. Mapped + uppercased in
   `_get_optional_params`. Marked unsupported on Cohere V1/V2 since OCI
   Cohere has no reasoning models (avoids Pydantic validation failure
   on CohereChatRequest).

2. **"disable" → "NONE" mapping.** OpenAI uses "disable" to turn off
   reasoning; OCI uses "NONE". Without this, callers get a 400.

3. **reasoning_tokens propagated to Usage.** OCI returns
   `completionTokensDetails.reasoningTokens` but it wasn't being passed
   to LiteLLM's Usage object. Now flows through to
   `Usage.completion_tokens_details.reasoning_tokens` so callers can
   track reasoning token consumption for cost/observability.

Tests: 7 new unit tests in TestOCIReasoningEffort covering upper/lower
case, "disable"→"NONE", Cohere drop/raise paths, and reasoning_tokens
extraction (with and without completionTokensDetails). 5 new live
integration tests against xai.grok-3-mini in us-chicago-1 verifying the
full request/response loop end-to-end. Existing
test_transform_response_simple_text assertion that
completion_tokens_details was None has been updated to assert
reasoning_tokens flows through.

Verified live on xai.grok-3-mini: reasoning_effort=low → OCI accepts
"LOW", returns reasoningTokens=316 in usage. reasoning_effort=disable
→ OCI accepts "NONE". Full suite: 370/370 unit + 51/51 integration.

* fix(codeql): re-scope py/weak-sensitive-data-hashing exclusion to OCI signing file

CodeQL's taint analysis re-fires the `py/weak-sensitive-data-hashing`
alert at `litellm/llms/oci/common_utils.py:103` whenever upstream code
paths into the OCI signing module change (touching `transformation.py`
opens new flow paths that CodeQL re-evaluates from scratch). The
`hashlib.sha256(..., usedforsecurity=False)` declaration silences the
direct-call form of the query but not the taint-flow form.

SHA-256 here is mandated by the OCI HTTP signing specification for the
x-content-sha256 content-integrity header — not for password storage:
https://docs.oracle.com/en-us/iaas/Content/API/Concepts/signingrequests.htm

CodeQL has no per-query path filter and GitHub Code Scanning ignores
inline lgtm/codeql comments, so path-ignoring this single ~560-line
signing utility file is the narrowest available suppression. All other
files retain full coverage of py/weak-sensitive-data-hashing — including
litellm/proxy/utils.py where the rule legitimately applies.

This restores the NEUTRAL CodeQL state the PR had on prior commits
(see `2111c98af7` for the same approach on the previous branch
evolution that the cherry-pick was rebased onto a different baseline).

* fix(oci): drop duplicate text on Cohere streaming terminal chunk

OCI Cohere's terminal SSE event re-sends the full assembled response in
`text` alongside a populated `chatHistory`. Emitting that text as another
delta concatenates the entire response onto the already-streamed output
(e.g. "How can I help?How can I help?").

Use `chatHistory is not None` as the discriminator for the consolidated
terminal event — `finishReason` is a weaker signal that could in principle
appear on a non-consolidated chunk. The two coincide today; this preserves
correctness if OCI ever ships finishReason on an incremental chunk.

Adds a live-OCI integration regression test that compares streamed vs
non-streamed length and asserts the response prefix appears only once.
Verified to fail under the previous code with the exact reported
reproduction: 'Hello! How can I help you today?Hello! How can I help you today?'.

Reported by @gotsysdba on PR #25177.

* fix(oci): buffer SSE stream across HTTP read boundaries

The old split_chunks helper split each individual HTTP read on "\n\n",
which assumed SSE event boundaries always aligned with read boundaries.
In practice the OCI streaming endpoint delivers events that may:

  - straddle two reads (chunk_creator gets a truncated JSON and crashes)
  - arrive separated by a single "\n" instead of "\n\n"
  - share a read with multiple complete events

Replace the inline split with module-level helpers _iter_sse_events
(sync) / _aiter_sse_events (async) that maintain a buffer across reads,
split on any newline, and yield only complete "data:" lines.

Add 25 regression tests covering event-split-across-reads, tiny-chunk
reads, single-newline separators, keepalive/comment lines, trailing
partial events flushed at EOF, "\r\n" line endings, and an end-to-end
smoke test that feeds an awkwardly-chopped payload through the splitter
into OCIStreamWrapper.chunk_creator.

Reported by John Lathouwers.

* test(oci): repoint TestOCIKeyNormalization to sign_with_manual_credentials

The signing helper moved from OCIChatConfig._sign_with_manual_credentials
to a module-level sign_with_manual_credentials in common_utils.py. Four
tests in TestOCIKeyNormalization still called the old method:

  - 2 failed outright with AttributeError
  - 2 passed by accident because they used pytest.raises(Exception),
    which happily caught the AttributeError instead of exercising the
    intended OCIError path

Repoint all four to the new module-level function so they exercise the
actual oci_key type-validation branch.

* fix(oci): validate oci_region before URL interpolation to prevent SSRF

Anchor oci_region to ^[a-z][a-z0-9-]{0,30}[a-z0-9]$ inside get_oci_base_url
so user-supplied regions that would redirect the signed request to an
attacker-controlled host (e.g. 'evil.com/#') fail with HTTP 400 before
the URL or signature is built. Empty string still falls back to the
us-ashburn-1 default, so existing callers are unaffected.

* test(audio): skip when gpt-4o-audio-preview is unavailable upstream

OpenAI retired `gpt-4o-audio-preview` (404 model_not_found in CI as of
2026-05-19), and the existing try/except in these tests only re-raised
on 'openai-internal' errors. Other exceptions were silently swallowed,
so the next line ran with an unbound `response`/`completion` and
failed with an unrelated UnboundLocalError that masked the real cause.

Extend the skip condition to also cover model_not_found / 'does not exist'
so the suite reports the upstream outage cleanly, matching the pattern
used in ce87c41 for the realtime and nvidia_nim rerank tests.
Re-raise unknown exceptions instead of falling through.

* fix(oci/router): catalog-driven maxCompletionTokens; generic blocked-deployment message

- Drive OCI maxCompletionTokens via supports_reasoning from the model
  catalog instead of a hardcoded openai.gpt-5 prefix. Add OCI GPT-5 family
  entries (gpt-5, gpt-5-mini, gpt-5-nano) with supports_reasoning: true.
  Gate the override to non-Cohere vendor so Cohere reasoning models keep
  maxTokens (Cohere endpoint does not accept maxCompletionTokens).
- Replace proxy-specific 'Contact your proxy admin' phrasing in the four
  Router blocked-deployment ServiceUnavailableError messages with neutral
  SDK-appropriate text.

* fix(oci/cohere): guard handle_cohere_response against missing usage

* fix(oci): address bug review findings in chat transformation

- Cohere param map: keep tool_choice/n as False (not omitted) so unsupported
  params are dropped or rejected rather than silently passed through.
- get_complete_url: when an explicit api_base/litellm.api_base is provided,
  use it as-is instead of unconditionally appending /20231130/actions/chat
  (mirrors the embed config behavior).
- Cohere stream: require both chatHistory and finishReason to be present to
  identify a terminal consolidation chunk, avoiding silent text suppression
  if chatHistory ever appears on a non-terminal chunk.
- Generic usage: use 'is not None' for reasoningTokens so a legitimate value
  of 0 is preserved instead of being treated as absent.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(oci/cohere): emit tool calls in streaming and null content when text empty

handle_cohere_response now sets message.content to None when the Cohere
response text is empty, matching the OpenAI convention for tool-call-only
responses.

handle_cohere_stream_chunk now extracts toolCalls — both directly from
the chunk and from the terminal chunk's chatHistory CHATBOT message —
and emits them in the delta. Previously, CohereStreamChunk lacked a
toolCalls field, so any tool calls in the stream were silently dropped.

* fix(oci): preserve tool results, embed URL path, and generic finish reason

- Use SerializeAsAny on CohereChatRequest.chatHistory so subclass-specific
  fields like CohereToolMessage.toolResults are not dropped during Pydantic
  v2 serialization.
- Make OCIEmbedConfig.get_complete_url append the /20231130/actions/embedText
  action path consistently with chat, so setting litellm.api_base to the
  region inference base URL no longer posts to the bare hostname.
- Map OCI finishReason (COMPLETE / MAX_TOKENS / TOOL_CALLS) to OpenAI
  finish_reason values in handle_generic_response, mirroring the streaming
  handler and the Cohere non-streaming handler.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(oci/generic): silence mypy assignment error on dynamic finish_reason

* fix(oci/embed): always set usage on embedding response

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(oci/chat): append /20231130/actions/chat to explicit api_base

Restore the embed-style behavior so OCIChatConfig.get_complete_url always
appends the OCI GenAI chat path. Routing through get_oci_base_url ensures the
optional explicit api_base has its trailing slash stripped before the suffix is
joined, matching the embed config and the test_respects_explicit_api_base
expectation.

* fix(oci/cohere): mark logprobs/logit_bias unsupported and normalize unknown stream finish reasons

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(oci/cohere): preserve trailing tool result in chatHistory

When the last message in the OpenAI-format input is a tool result (the
standard agentic continuation pattern), the prior messages[:-1] slice
silently dropped that tool result from chatHistory and the model never
saw it. Excluding the last user message by index instead keeps tool
results that trail the last user turn intact.

* fix(main): remove dead OCI embedding elif block

The earlier elif at line 5119 already routes OCI embeddings through the
base HTTP handler with the headers None-guard, so the later identical
block was unreachable dead code.

* test(oci): move integration tests out of llm_translation mock-only folder

Greptile flags tests/llm_translation/ as mock-only via a project-specific
rule; relocate the live-network OCI integration suite to tests/integration/
and adjust the in-file sys.path / run instructions accordingly.

* fix(oci/cohere): suppress tool calls on stream terminal consolidation chunk

The terminal SSE event re-sends the full assembled response in both
`text` and `chatHistory`. The existing logic already suppresses
`text` to avoid double-emit, but tool calls extracted from the
terminal chunk (via `typed_chunk.toolCalls` or the `chatHistory`
CHATBOT fallback) would still be re-emitted with fresh uuid4 IDs.
If OCI Cohere ever streams tool calls progressively in intermediate
chunks (now possible since CohereStreamChunk has a toolCalls field),
this would cause downstream agentic frameworks to execute each tool
call twice.

Suppress tool calls on the terminal consolidation chunk for the same
reason `text` is suppressed.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(oci,httpx): normalize finish_reason, preserve response_format, fix sync embed JSON content-type

- cohere.py / generic.py: normalize unknown OCI finishReason values (ERROR,
  ERROR_TOXIC, CONTENT_FILTERED, USER_CANCEL, ...) to 'stop' in non-streaming
  and streaming generic handlers, matching the streaming Cohere handler so
  downstream consumers switching on finish_reason aren't broken by raw OCI
  values.
- transformation.py: restore the dual-key alias so optional_params still
  carries the original 'response_format' key alongside the OCI-mapped
  'responseFormat'. Downstream litellm framework code (json_mode detection,
  logging) inspects 'response_format' after map_openai_params runs.
- llm_http_handler.py: make the sync embedding path mirror the async path —
  when sign_request returns no signed_body, send via json=data (which sets
  Content-Type: application/json) instead of data=json.dumps(data) which
  doesn't. Removes a sync/async behavioural asymmetry for non-OCI providers
  that adopt the sign_request pattern.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(oci): clean up OCIChatConfig init, normalize generic stream finish reasons, correct embed sign_request return type

- Replace fragile setattr(self.__class__, ...) pattern in OCIChatConfig.__init__ with a @property for has_custom_stream_wrapper, matching the pattern used by other providers.
- Normalize unknown OCI finish reasons (e.g. ERROR, ERROR_TOXIC, USER_CANCEL) to 'stop' in handle_generic_stream_chunk, matching the existing Cohere stream handler behaviour.
- Tighten OCIEmbedConfig.sign_request return type from Tuple[dict, Optional[bytes]] to Tuple[dict, bytes] — sign_oci_request never returns None for the body, and this matches OCIChatConfig.sign_request.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(oci): strip trailing action path in get_oci_base_url to avoid URL doubling

A fully-formed OCI endpoint URL (e.g. https://inference.generativeai.us-chicago-1.oci.oraclecloud.com/20231130/actions/chat) passed via api_base previously had the action path appended a second time by get_complete_url in both chat and embed configs, yielding a 404. get_oci_base_url now strips a trailing /20231130/actions/<name> so callers can always append the action path safely.

* fix(httpx): preserve sync embed data= kwarg to avoid breaking mock-based tests

The earlier sync_httpx_client.post() call passed data=json.dumps(data),
which downstream embedding tests assert on (e.g. tests for hosted_vllm,
jina_ai, watsonx). Switching to json=data changed the kwarg name and broke
those tests. The OCI signed_body path keeps using data=signed_body and is
unaffected.

* fix(oci): stable tool-call ids across stream chunks; lenient Cohere finishReason

- Replace random uuid4 per chunk with a deterministic content-derived
  digest for synthetic tool-call ids in both Cohere and Generic OCI
  handlers. Previously, when OCI omitted 'id' (always for Cohere, often
  for Generic streaming deltas), every chunk for the same logical tool
  call received a new uuid, causing downstream stream-mergers (which key
  off id) to treat each fragment as a distinct call.

- Relax CohereChatResponse.finishReason from a strict Literal[...] to
  Optional[str], matching CohereStreamChunk.finishReason. The
  handle_cohere_response 'elif oci_finish_reason is not None' fallback
  was previously unreachable because Pydantic raised ValidationError on
  any unknown value before the fallback executed. Now non-streaming
  responses degrade unknown reasons to 'stop' just like the streaming
  path.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(oci/embed): validate OCI credentials in validate_environment

Mirror OCIChatConfig.validate_environment so embedding requests fail
fast with a clear error when oci_user/oci_fingerprint/oci_tenancy/
oci_compartment_id or an oci_key/oci_key_file is missing, instead of
deferring the failure until sign_request.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* test(oci/embed): expect OCIError from validate_environment when credentials are missing

OCIEmbedConfig.validate_environment now raises eagerly (mirroring OCIChatConfig)
when oci_user/oci_fingerprint/oci_tenancy/oci_compartment_id or oci_key/oci_key_file
is missing. Update the test to match.

* fix(oci): polish stream chunk handling and signed body default

- cohere stream terminal consolidation now emits content=None instead of ""
- drop redundant index truthiness check (None is already replaced with 0)
- accept both "TOOL_CALL" and "TOOL_CALLS" finish reasons in cohere
- signed_json_body defaults to None and uses explicit None check, so an
  explicitly empty bytes body wouldn't be silently re-serialized

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(oci/chat): catch pydantic ValidationError when parsing OCI responses

Pydantic v2 raises ValidationError (not TypeError) when field validation
fails, so malformed OCI completion responses or stream chunks would
propagate unhandled out of handle_generic_response,
handle_generic_stream_chunk, and handle_cohere_stream_chunk. Widen the
except clauses to also catch ValidationError so callers get a clean
OCIError.

* fix(oci/catalog): real prices for Llama 4, drop zero-cost OCI OpenAI entries

Zero-cost catalog entries (input_cost_per_token=0, output_cost_per_token=0)
make proxy spend tracking silently report $0 for these paid OCI models, so
any caller can drive them without decrementing a budget.

For Llama 4 Maverick and Scout, OCI charges the same character-based rate
as Llama 3.3 70B ($0.0018 per 10,000 characters), so use the same per-token
price as the existing oci/meta.llama-3.3-70b-instruct entry (7.2e-07 in/out).

For oci/openai.gpt-5, gpt-5-mini, gpt-5-nano, gpt-oss-120b, and gpt-oss-20b,
no public per-token pricing is available; drop the entries so operators must
register them with explicit custom pricing. The existing GPT-5 reasoning test
fixture already injects synthetic entries when the catalog omits them, so the
chat transformation's supports_reasoning lookup keeps working in tests.

* fix(oci/chat): wrap CohereChatResult construction in try/except

Match the handle_generic_response pattern: surface OCIError with the
upstream status code instead of letting a raw pydantic.ValidationError
propagate when the Cohere response payload is malformed.

* fix(oci): harden Cohere stream/finish-reason and dedupe maxTokens param mapping

- Cohere stream: track per-stream tool-call emission and only suppress the
  terminal consolidation chunk's tool calls once they've been seen earlier.
  Prevents silent drop if tool calls are delivered exclusively on the
  terminal chunk.
- Cohere stream: emit content=None (not "") on non-terminal text-free
  chunks (e.g. tool-call-only / keep-alive) so downstream consumers that
  distinguish missing vs explicitly-empty deltas behave correctly.
- Generic handlers: accept singular TOOL_CALL finish reason in addition to
  TOOL_CALLS, matching the Cohere handlers.
- _get_optional_params: when both max_tokens and max_completion_tokens are
  provided, explicitly prefer max_completion_tokens instead of relying on
  dict iteration order.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(oci): emit content=None instead of empty string for text-free generic stream chunks

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* test(oci): expect content=None for text-free generic stream chunks

handle_generic_stream_chunk now emits content=None instead of empty
string when a chunk carries no text parts. Update the corresponding
no-message test to match.

* codeql: narrow OCI sha256 suppression to query-filter, not whole file

paths-ignore was suppressing every CodeQL query on
litellm/llms/oci/common_utils.py, hiding all future findings in a
security-critical file (private key loading, credential resolution,
URL construction, RSA signing). Move the suppression for
py/weak-sensitive-data-hashing into query-filters so common_utils.py
remains fully analyzed by every other query.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(oci): use locale-independent RFC 7231 date for manual signing

email.utils.formatdate(usegmt=True) emits canonical English weekday/
month abbreviations regardless of system locale, so signature
verification doesn't break on non-en_US deployments.

* fix(oci): strip 'oci/' prefix in get_vendor_from_model

Previously, get_vendor_from_model split on '.' without stripping the
optional 'oci/' provider prefix, so 'oci/cohere.command-a-03-2025' was
routed through the GENERIC pipeline instead of COHERE.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* codeql: scope OCI sha256 suppression to common_utils.py via filter-sarif

Replace the global query-filters exclude for py/weak-sensitive-data-hashing
with a SARIF post-filter that only drops the alert when it originates from
litellm/llms/oci/common_utils.py, keeping the rule active on every other
SHA-256 callsite in the repository.

* Fix OCI chat bugs: tool_calls None key, dead max_tokens dedup, single-event stream text suppression

- handle_cohere_response: omit tool_calls key from message dict when None,
  matching the generic handler's behaviour and avoiding tripping consumers
  that key off 'tool_calls' in message.
- _get_optional_params: remove dead prefer_max_completion branch. By the
  time this helper runs, map_openai_params has already collapsed
  max_tokens/max_completion_tokens onto the OCI alias, so the OpenAI-key
  membership check is unreachable.
- handle_cohere_stream_chunk: add prior_text_emitted parameter mirroring
  prior_tool_calls_emitted. The terminal consolidation chunk's text is
  only suppressed when prior deltas already emitted text — otherwise
  (degenerate single-event stream) the text passes through so the
  response content isn't silently lost. OCIStreamWrapper now tracks
  emitted text alongside emitted tool calls.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(oci): preserve all text parts in generic response and emit SYSTEM role for Cohere

- handle_generic_response: iterate all content parts and concatenate text
  (matches the streaming handler) so non-leading text parts are not lost
  and a leading non-text part does not suppress trailing text.
- adapt_messages_to_cohere_standard: emit CohereSystemMessage for system
  messages so direct callers do not silently drop them. The Cohere
  request builder filters system messages before calling this helper to
  avoid duplicating preambleOverride content into chatHistory.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(oci): normalise dict-format tool_choice to OCI flat uppercase shape

The OCI Generative AI API only accepts toolChoice values of the form
{"type": "AUTO"|"NONE"|"REQUIRED"} or {"type": "FUNCTION",
"name": "<fn>"}. The previous conversion only handled string
tool_choice values, so OpenAI's standard dict shape
{"type": "function", "function": {"name": "<fn>"}} passed
through unchanged and was rejected by OCI with a 400.

Normalise the dict shape by uppercasing the discriminator and hoisting
the function name to the top level. Also accept dict variants of the
non-function selectors (e.g. {"type": "auto"}).

* test(oci): exercise system-message filtering at transform_request boundary

adapt_messages_to_cohere_standard now emits SYSTEM-role entries by design
so direct callers don't silently drop system content. The Cohere request
builder filters system messages before calling the helper and routes them
into preambleOverride, so the user-visible 'no SYSTEM in chatHistory'
guarantee holds at the transform_request boundary, where the test should
live.

* fix(oci/chat): extract tool_choice/response_format helpers to satisfy PLR0915

_get_optional_params exceeded ruff's 50-statement cap. The toolChoice and
responseFormat normalisation blocks are self-contained mutations, so move
them to module-level helpers.

* fix(oci): normalize None finishReason in generic non-streaming handler; drop dead Cohere system-role branch

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(oci/generic): silence mypy assignment error on cleared finish_reason

* fix(docker): install libatomic in builder for prisma nodeenv binary

The prebuilt node binary that prisma-python's nodeenv downloads links
against libatomic.so.1, which Wolfi does not pull in via gcc/nodejs.
Without this, fresh Docker builds (no GHA cache hit) fail at
`prisma generate` with:
  node: error while loading shared libraries: libatomic.so.1

* fix(oci): raise on invalid tool_choice instead of silently passing OpenAI shape

_normalize_tool_choice previously left an OpenAI-format dict in selected_params['toolChoice'] when the type was unrecognized or when 'FUNCTION' was given with a missing/empty name. OCI would then reject the request with a non-obvious error. Raise ValueError with a clear message in these cases.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(oci): raise OCIError instead of ValueError in _normalize_tool_choice

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(oci/generic): declare non-security intent on sha256 for synthetic tool-call id

* fix(oci): simplify _get_optional_params and reject invalid tool_choice types

- Collapse the two-loop _get_optional_params into a single pass with
  clear precedence (OpenAI key wins over OCI alias; first OpenAI key
  reaching a given OCI target wins). Removes the redundant maxTokens
  special-case in the second loop and makes the map_openai_params /
  transform_request handoff easier to reason about.
- Raise OCIError when _normalize_tool_choice sees an unexpected type
  (list, bool, int, ...) instead of silently letting it through to the
  OCI API where it would produce an opaque server-side error.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* Remove no-op data['stream'] deletion in OCI stream wrappers

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(oci): always send Cohere isStream field explicitly

Match OCIChatRequestPayload by defaulting CohereChatRequest.isStream to
False instead of None so model_dump(exclude_none=True) does not silently
omit the field on non-streaming requests.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(oci): revert Cohere isStream to Optional[bool]=None to preserve omission semantics

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(oci/generic): raise OCIError on empty choices instead of IndexError

Pydantic accepts an empty choices list when validating OCICompletionResponse, so accessing chatResponse.choices[0] could raise an unhandled IndexError. Surface it as OCIError so the response error path is consistent with the existing (TypeError, ValidationError) guard.

* fix(oci/cohere): map top_k -> topK so Cohere topK param is settable

The Cohere param map (derived from the GENERIC map) had no entry for
topK. Since the simplified _get_optional_params only iterates over
param_map entries, callers had no way to pass topK to CohereChatRequest
(neither via an OpenAI-style key nor via the OCI alias).

Add 'top_k': 'topK' to the Cohere map only — OCIChatRequestPayload
(GENERIC) has no topK field. _get_optional_params accepts both the
OpenAI key (top_k) and the OCI alias (topK) in optional_params, so this
covers both calling conventions.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(oci): tighten cohere stream dedup flags and forward stream args in embed signing

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(oci/chat): reorder dict guard and wrap stream chunk json.loads

- Move isinstance(response_json, dict) check before .get("error") so
  the guard runs before the attribute access it is supposed to protect.
- Wrap json.loads in OCIStreamWrapper.chunk_creator with try/except so
  malformed SSE payloads surface as OCIError instead of a raw
  JSONDecodeError propagating out of the stream loop.

* fix(oci/cohere stream): only flag text emitted on non-empty content

An intermediate Cohere SSE chunk carrying text="" was flipping
_cohere_text_emitted via the "is not None" check, which then caused
the terminal consolidation chunk to drop its real text as a duplicate.
Use a truthy check so only actual content marks the stream as having
emitted text.

* test(oci): end-to-end proxy integration test against real OCI GenAI

Spins up the litellm proxy via the console-script entrypoint with a
minimal OCI-only config and drives real OpenAI-shaped HTTP requests
through it against OCI GenAI. Covers non-streaming chat, streaming
chat, embeddings, and /v1/models for Cohere, Llama, Gemini, and Grok.

Skips automatically when ~/.oci/config is absent or when the active
profile uses session-token auth (the OCI provider currently only
consumes OCI_* env vars; session tokens would need an in-process
signer). API-key profiles work out of the box.

* test(oci): move proxy integration test to tests/integration/

tests/llm_translation/ is mock-only; the OCI proxy integration test
spawns a real proxy subprocess and makes live HTTP calls, so move
it (and the companion config) to tests/integration/ alongside the
existing test_oci_integration.py.

* fix(oci): dedupe finish-reason mapping and batch Cohere tool results

- Extract _normalize_oci_finish_reason helper so the four chat handlers
  (Cohere/GENERIC, sync/stream) share one OCI->OpenAI mapping instead of
  four near-identical if/elif chains.
- Merge consecutive OpenAI tool-role messages into a single
  CohereToolMessage with multiple toolResults entries, matching the OCI
  Cohere API's expectation for parallel tool calls in one assistant turn.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(oci): drop dead Cohere toolChoice field and emit GENERIC tool-call dicts inline

- Remove the unreachable toolChoice field from CohereChatRequest. The
  Cohere param map explicitly marks tool_choice as unsupported, so the
  field can never be populated through the normal optional_params flow
  and only confused the public model surface.
- Build GENERIC stream tool-call dicts inline (id/type/function shape)
  instead of round-tripping through ChatCompletionMessageToolCall and
  model_dump(). Matches handle_cohere_stream_chunk so downstream
  stream-mergers see the same minimal payload regardless of which
  vendor produced the chunk.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(docker): drop redundant libatomic from non_root builder

litellm_internal_staging already fixes the prisma `nodeenv` build
failure at the root cause by restoring `npm` to the builder (#28519):
with npm on PATH, prisma-python uses the system Node and never downloads
the nodeenv binary that links against libatomic.so.1. After merging
internal_staging the libatomic line is dead weight, so remove it.

https://claude.ai/code/session_01SwKzxRxgUhLFyyEf4UV812

* fix(oci/catalog): add openai.gpt-5{,-mini,-nano} entries with supports_reasoning

Without these catalog entries, supports_reasoning(model='openai.gpt-5*',
custom_llm_provider='oci') returned False, so _model_uses_max_completion_tokens
fell back to the default and OCI rejected the request with HTTP 400
('Use maxCompletionTokens instead.'). Add the three entries so the catalog-driven
maxCompletionTokens routing works against a stock LiteLLM install.

Also reword the test fixture docstring — the bundled backup now actually ships
these entries, so the fixture is only a fallback for environments that loaded
their cost map from a stale remote source.

---------

Co-authored-by: Tai An <antai12232931@outlook.com>
Co-authored-by: Vincent <yimao1231@gmail.com>
Co-authored-by: Kris Xia <xiajiayi0506@gmail.com>
Co-authored-by: d 🔹 <liusway405@gmail.com>
Co-authored-by: Fabrizio Cafolla <developer@fabriziocafolla.com>
Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com>
Co-authored-by: Tom Denham <tom@tomdee.co.uk>
Co-authored-by: escon1004 <70471150+escon1004@users.noreply.github.com>
Co-authored-by: Divyansh Singhal <97736786+Divyansh8321@users.noreply.github.com>
Co-authored-by: robin-fiddler <robin@fiddler.ai>
Co-authored-by: Michael-RZ-Berri <michael@berri.ai>
Co-authored-by: Michael Riad Zaky <michaelr@Mac.localdomain>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Co-authored-by: Krrish Dholakia <krrish+github@berri.ai>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Federico Kamelhar <federico.kamelhar@oracle.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Michael Riad Zaky <michaelr@Michaels-MacBook-Air.local>
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
…aming hot paths (BerriAI#28289)

* perf: reduce per-request and per-chunk overhead across Anthropic streaming hot paths

- Introduce pure-text fast-path in `_build_complete_streaming_response` that collapses O(N) `content_block_delta` events into a single equivalent SSE event before conversion, eliminating per-output-token Pydantic `ModelResponseStream` construction; non-text streams (tool_use, thinking, citations) fall back to the unchanged legacy path
- Skip agentic streaming wrapper entirely when no callback overrides `async_should_run_agentic_loop`; the wrapper buffered every chunk and rebuilt the SSE response only to call hooks that all return `(False, {})` — a pure no-op for the default config
- Serialize request body once (`json.dumps`) for both the pre-call log input and the wire, instead of twice; avoids a full O(payload) scan per request, significant for long-context Claude Code histories
- Add fast path in `async_streaming_data_generator` that bypasses the per-chunk `async_post_call_streaming_hook` coroutine await, response-string materialization, and cost-injection call when no callback/guardrail/cost-injection is active (the default config)
- Resolve `_DD_STREAMING_TRACE_ENABLED` once at import time; eliminate per-chunk `NullSpan` context manager allocation when Datadog tracing is disabled (the default)
- Memoize `get_type_hints(AnthropicMessagesRequestOptionalParams)` with `@lru_cache(maxsize=1)` — resolves once per process instead of once per `/v1/messages` request (~80µs each)
- Hoist `cost_injection_active` out of the per-chunk loop in `chunk_processor`; eliminates repeated `getattr` + endpoint-type checks on every streamed byte chunk
- Extract `_build_passthrough_logging_result` from `_route_streaming_logging_to_handler` as a standalone static method to facilitate future off-loop dispatch
- Convert `async_sse_data_generator` from an `async for: yield` trampoline to a direct return of the underlying generator, removing one async-generator layer per streamed chunk
- Skip redundant `strip_empty_text_blocks_from_anthropic_messages` scan in `anthropic_messages_handler` when the async wrapper already sanitized (signalled via `_litellm_messages_presanitized` sentinel, popped before reaching provider params)
- Gate debug log `f-string` evaluation behind `isEnabledFor(DEBUG)` in both the streaming generator and the transformation layer to avoid serializing entire message payloads on every request at non-debug log levels
- Add benchmark script (`scripts/benchmark_anthropic_messages_perf.py`) with a local mock Anthropic SSE provider for reproducible TTFT and TPM measurement across commits/branches
- Add parity tests asserting fast-path and legacy-path produce byte-identical logged/billed payloads, plus unit tests for agentic hook detection, pre-serialized body reuse, and memoized key resolution

* perf: address greptile review for anthropic streaming hot path

- Bail to legacy in `_collapse_pure_text_chunks` when content_block_delta
  events from different block indexes are observed without an intervening
  flush. Anthropic sends blocks strictly sequentially, but defensive bail
  prevents silent text-merging if the protocol ever interleaves.
- Replace leaf-class `__dict__` check for `async_post_call_streaming_hook`
  in `_callback_capabilities` with a function-identity comparison that
  walks the MRO. A vendor base class can carry the override and the
  registered class can add nothing else; before this PR the hook was
  unconditionally invoked, so an inherited-override miss would silently
  drop the hook on the streaming path.
- Add unit tests for both behaviors.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(mypy): narrow model_name to str in cost-injection branch

The hoisted cost_injection_active flag in chunk_processor encodes the
`bool(model_name)` requirement but mypy can't track that invariant
through the local, so the per-chunk `_process_chunk_with_cost_injection(
chunk, model_name)` calls flagged Optional[str] vs str. Pin a typed
non-None local inside the cost-injection branch so mypy narrows
correctly without changing runtime behavior.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: oss-agent-shin <279349115+oss-agent-shin@users.noreply.github.com>
Co-authored-by: ishaan-berri <ishaan-berri@users.noreply.github.com>
… management endpoints (BerriAI#28681)

* test(proxy): phase-4 payload behavior pinning for tier-2/3 key + team management endpoints

Extends the Phase 1–3 behavior-pin suite at tests/proxy_behavior/management/
with a second axis: payload-shape pinning. Phase 1–3 held payload minimal
and pinned (actor, target) → status across 37 routes; Phase 4 holds the
caller fixed at an authorized actor, varies the payload shape, and asserts
the observable DB effect (on accept) or the named guard / row-unchanged
(on reject). Faithfulness contract from Phase 1–3 is unchanged.

Six families + one gap-closer (59 new scenarios, 620 → 679 total):

  * F1 — key budget / rate-limit (test_key_budget_limits.py, 18)
  * F2 — key↔team reassignment   (test_key_team_change.py, 6)
  * F3 — team budget / rate-limit (test_team_budget_limits.py, 15)
  * F4 — member-info validation   (test_team_member_info_validation.py, 5)
  * F5 — permission batching      (test_team_permissions_bulk_update.py, 6)
  * F6 — org-scoped team access   (+2 detail-string pins in existing files)
  * F7 — coverage gap-closer      (test_f7_coverage_closeout.py, 7)

Harness extensions in conftest.py (additive only):
  * create_scratch_org() seeder with its own scratch-prefixed budget row
  * budget / limit fields on create_scratch_team()
  * scratch teardown also sweeps litellm_organizationtable

Coverage telemetry (behavior-suite-only):
  * key_management_endpoints.py  60 % → 65 % (+82 lines)
  * team_endpoints.py            62 % → 72 % (+137 lines, crosses 70 % stretch)

Key lands under 70 % per plan §7 escape hatch — the gap is dominated by
routes outside F1–F6 scope (key list/info v2 internals) and structurally
dead org-budget guards (call sites at lines 889 + 2310 + 985 + 1751 load
the org without include_budget_table=True, so org.litellm_budget_table is
None at guard time and the aggregate guard no-ops). Pinned as observed
no-op behavior so a future fix that flips the flag turns these into reds.

Zero source-code changes; pyproject.toml diff is empty;
test_route_coverage.py stays green untouched; G3 grep guards still green;
local wall-time 14 s for the full suite (no coverage), 22 s with coverage.

G4 regression-replay protocol executed against three representative
fix-PR parents (410ce76, 0bd49ec, 8bbc61e): all Phase 4 tests
PASS at pre-fix SHAs — confirming the F1–F7 layer is a helper-body pin,
not a regression-replay layer for those specific historical bypass
shapes. Targeted RED-bait scenarios for each fix are left for a
follow-up PR.

* test(proxy): push key_management_endpoints.py past the 70% stretch (F7-extension)

Adds 24 more payload-pin scenarios in test_f7_key_coverage_push.py
following the same accepted-effect / rejected-guard pattern. Each
scenario cites the file:line range it pins; same anti-snapshot rules
apply.

Target ranges (all reachable via HTTP-boundary payload variation):
  * 5942-6063  /key/health with metadata.logging → test_key_logging body
  * 4565-4692  /key/reset_spend happy + 404 + non-admin gate + value validation
  * 4421-4533  /key/regenerate ghost-404 + happy + new_key + grace_period
  * 4168-4202  _insert_deprecated_key body via grace_period
  * 6118-6133  _enforce_unique_key_alias duplicate-alias rejection
  * 6148-6169  validate_model_max_budget malformed-payload rejection
  * 4708-4789  validate_key_list_check user/team/org/key_hash branches
  * 2622-2733  /key/bulk_update mixed success/failure + admin gate + size limits
  * 2797-2950  /team/key/bulk_update all-keys path + explicit-keys dedupe + 404
  * 5108-5207  /key/aliases admin + scoped + search-filter branches
  * 3253-3303  /key/info ghost + explicit-key + no-key-uses-auth-header
  * 3427-3436  generate_key_helper_fn budget_limits initialization
  * 1794-1815  prepare_key_update_data duration + budget_duration paths
  * 5280-5388  _build_filter_conditions across include_created_by_keys/team/sort/alias

Coverage telemetry — full PR4 dataset:

  key_management_endpoints.py: 60 % → 71 %  (+11 pts, +194 lines)
  team_endpoints.py:           62 % → 72 %  (+10 pts, +137 lines)

Both files now over the plan §7 PR4.M4 70 % stretch as a side effect of
pinning real payload behavior. 721 tests pass in 19 s local (full suite,
no coverage); 27 s with coverage. Zero source-code changes; pyproject.toml
diff still empty; test_route_coverage.py + G3 grep guards still green.

Honest finding (kept from the prior commit's body): four structurally-dead
org-budget guards remain pinned as observed no-op behavior — they fire
only when get_org_object is called with include_budget_table=True, which
none of the four management-endpoint call sites currently do. Pinned so
a future change that flips the flag turns these into reds.

Two helper guards are honest-ceiling: _validate_reset_spend_value's
isinstance check at line 4568 is unreachable from HTTP because Pydantic
422s non-float before the helper runs; same shape for /team/key/bulk_update's
missing team_id / no-selector pre-handler guards.

* test(proxy): address PR review — try/finally cleanup + loosen 500 envelope pins + Optional annotations

Greptile review feedback on PR BerriAI#28681:

1. Wrap manual budget-row cleanup in try/finally so an assertion failure
   doesn't leave non-scratch-prefixed budget rows orphaned across CI re-runs
   (test_team_new_with_team_member_budget_creates_budget_row and
   test_team_update_team_member_budget_upserts).
2. Loosen the two 500-status pins to in (400, 422, 500) — the named-guard
   substring is the real pin; the outer ValueError-wrap envelope is an
   implementation detail that a future improvement should be free to fix
   to a proper 400/422 without flipping these tests red.
3. Add missing Optional annotations on _seed_token's max_budget / metadata
   / team_id keyword args (they default to None).

Greptile's typo flag on 'read-world' in the conftest comment is declined —
'read-world' is the project's established term for the immutable seeded
world fixture (see other usages in conftest.py and actors.py).

721 tests still pass in 17 s.
…iAI#28372) (BerriAI#28378)

* feat(prometheus): emit per-token-type detail metrics (LIT-3220) (BerriAI#28372)

Adds five sparse counter metrics that break out the token detail
fields providers already report in `usage.prompt_tokens_details` and
`usage.completion_tokens_details`:

  - litellm_input_cached_tokens_metric            (provider prompt-cache reads)
  - litellm_input_cache_creation_tokens_metric    (Anthropic prompt-cache writes)
  - litellm_input_audio_tokens_metric             (audio input tokens)
  - litellm_output_reasoning_tokens_metric        (reasoning tokens)
  - litellm_output_audio_tokens_metric            (audio output tokens)

These are additive — existing input/output/total counters are
unchanged, so no dashboards break. Each new counter is only
incremented when the underlying detail is populated and > 0, keeping
scrape output sparse for providers that don't report a given field.

Data is read from the canonical Usage dict that
`get_standard_logging_object_payload` already attaches at
`standard_logging_payload["metadata"]["usage_object"]`, so no new
plumbing through the logging pipeline is required.

Tests: 10 new unit tests covering registration, label-set parity,
all-types increment, zero/None/negative skip behaviour, and the
no-metadata/no-usage_object no-op paths.

Closes LIT-3220

Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Claude <noreply@anthropic.com>

* chore: remove proof folder image

---------

Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai>
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
…rriAI#28405)

* fix(otel): stamp http.response.status_code on all error responses

httpx.HTTPStatusError exposes status under .response.status_code, not as a
top-level attr, so unified-endpoint 5xx failures left the SERVER span without
a status. The admin hooks only wrote a child span and never stamped or ended
the parent at all, so admin 4xx/5xx (and success) responses were invisible
to dashboards. Adds a fallback to .response.status_code in get_error_information,
and ends the parent SERVER span in async_management_endpoint_{success,failure}_hook
with the same _record_exception_on_span helper the unified path uses.

Resolves LIT-3193

* test(otel): exercise httpx.HTTPStatusError through admin path

Pins the contract that get_error_information's response.status_code fallback
is reachable from any entry point — without this, a future refactor that
bypasses _record_exception_on_span in the admin hooks could regress for
httpx-wrapped exceptions while the unified suite still passes.

* chore(otel): trim verbose comments in LIT-3193 changes

Tighten docstrings and remove redundant section dividers/inline narration.
Behavior is unchanged.

* fix(otel): set span.status on management hook parent SERVER span

Mirror the unified failure path: stamp StatusCode.ERROR on the parent
SERVER span before recording the exception, and StatusCode.OK before
ending it on success. Without this, OTEL backends filtering on span
status (the idiomatic primitive) miss admin-endpoint failures even
though the http.response.status_code attribute is correct.

Extend assert_server_span_attrs to assert span.status.status_code
matches the expected outcome so the gap can't regress.

* fix(otel): close SERVER span on body-validation and unhandled errors

Stash the SERVER span on request.state in auth so FastAPI exception
handlers can finish it for failures that occur after auth but before
the route handler (e.g. /model/new TypeError, /key/generate
RequestValidationError). Without this, those requests left dangling
spans missing http.response.status_code.

Resolves LIT-3193

* fix(otel): generic 500 body, log exception details server-side

Don't leak str(exc) and type(exc).__name__ to clients on uncaught
exceptions. The full traceback is logged via verbose_proxy_logger and
the SERVER span still gets http.response.status_code=500.

Resolves LIT-3193

* fix(otel): stamp http.response.status_code on every SERVER span path

Closes three remaining gaps where the proxy SERVER span ended without
the http.response.status_code attribute:

1. ProxyException raised from _read_request_body (e.g. invalid JSON
   body) bubbled out of user_api_key_auth before the SERVER span was
   created, so the FastAPI handler had nothing to close and the trace
   never reached the backend. Hoist the span creation to a new
   idempotent _ensure_parent_otel_span_on_request_state helper called
   at the top of user_api_key_auth; wire openai_exception_handler to
   close the dangling span. Covers /v1/chat/completions, /v1/messages,
   /v1/responses (shared handler).

2. /v1/responses success — _handle_success ends the proxy span before
   async_post_call_success_hook fires on this path, so the hook's
   set_response_status_code_attribute(200) silently no-op'd against an
   ended span. Stamp 200 + set OK status at the close site in
   _handle_success / _end_proxy_span_from_kwargs via a shared
   _close_proxy_span_ok helper, so the attribute lands regardless of
   which success hook runs first.

3. Failure path for exceptions without code/status_code (e.g. a bare
   TypeError surfacing through _handle_llm_api_exception) — empty
   error_information.error_code → _record_exception_on_span skips the
   stamp → the hook ends the span. Default to 500 in
   async_post_call_failure_hook so the attribute is always set.

Resolves LIT-3193
* fix(helm): drop main- prefix from default image tag

The default image tag in the deployment + migrations-job templates was
`main-{{ .Chart.AppVersion }}`. The current release pipeline publishes
content tags without the `main-` prefix (e.g. `v1.85.1` / `1.85.1`,
`v1.86.0-rc.1` / `1.86.0-rc.1`), so the rendered ref points at a tag
that does not exist on GHCR or DockerHub and installs fail with
ImagePullBackOff.

- templates/deployment.yaml, templates/migrations-job.yaml: render
  `.Chart.AppVersion` directly instead of `main-<AppVersion>`.
- Chart.yaml: bump stale `appVersion: v1.80.12` (not on either
  registry) to `v1.85.1` so local-checkout installs also resolve.
- values.yaml: update the commented tag-override hint to match.

* fix(helm): use :latest in tag override example, not pinned version

Per review: ghcr.io/berriai/litellm-database:latest is a floating
alias for the most recent stable (same digest as :main-stable),
maintained by the release pipeline's UPDATE_LATEST advance step.
Better example than a pinned version that goes stale.
…AI#28708)

The schema in test_aaamodel_prices_and_context_window_json_is_valid uses
additionalProperties: false. The azure/speech/azure-stt entry added in
BerriAI#27482 introduced an audio_transcription_config field that the schema
did not whitelist, so the test fails on every branch built on top of
staging.

Add the field as a string property.
chore(ci): promote internal staging to main
…rriAI#28683)

* fix(team): refresh team cache on team_model_add/delete (LIT-3244)

team_model_add and team_model_delete wrote to the DB but did not
invalidate the in-memory LiteLLM_TeamTableCachedObj used by
common_checks. After the v1.83.14 common_checks centralization made
team.models authoritative on /v1/files and /v1/vector_stores/*,
adding a Team-BYOK model silently failed to grant the new public
model name to team members until the cache TTL expired (and a
removed model kept working until then on the symmetric path).

Extract the cache-refresh snippet from update_team into a small
helper and apply it consistently at all three team-write sites.

* test: also assert updated models in team-cache-refresh pin

Strengthens the LIT-3244 regression test to also assert
`call_kwargs["team_table"].models` matches the updated row,
not just `team_id`. Both `existing_team` and `updated_team`
share `team_id` in the test setup, so the previous assertion
would have passed even if the implementation accidentally cached
the pre-mutation row.

Greptile review feedback.

* fix(team): hydrate object_permission on cache-refreshing team updates

The Prisma update calls in update_team, team_model_add, and
team_model_delete returned a team row with object_permission_id set
but object_permission=None (the relation was not requested via
include=). _refresh_cached_team then wrote that to the in-memory
LiteLLM_TeamTableCachedObj, and the cache-hit path in get_team_object
returns the cached object without re-hydrating. Downstream consumers
(validate_key_search_tools_against_team, the MCP/agent authz paths)
treat a missing object_permission as no team-level restriction, so
a team-write op silently dropped object-permission enforcement until
the cache TTL expired or a DB-fetch path re-hydrated it.

Add include={"object_permission": True} to all three updates so the
refresh writes a complete cached team. Extend the LIT-3244 regression
test to pin both the cached object_permission and the include shape
on the Prisma call.

Surfaced in PR review of LIT-3244.
… Anthropic (BerriAI#28723)

`getProviderModels()` matched a model into a provider's dropdown when the
model's `litellm_provider` string *contained* the provider key as a
substring. The intent was to admit suffix variants (e.g. `anthropic_text`,
`bedrock_converse`), but the substring check is too loose: it also pulls in
unrelated providers whose name happens to contain the key, most visibly
`vertex_ai-anthropic_models` matching `anthropic` and `vertex_ai-openai_models`
matching `openai`.

Replace `.includes()` with separator-anchored prefix matching
(`startsWith(provider + "_")` / `startsWith(provider + "-")`). All legitimate
variants in `model_prices_and_context_window.json` still match
(`anthropic_text`, `azure_text`, `azure_ai`, `bedrock_converse`,
`bedrock_mantle`, `cohere_chat`, `fireworks_ai-embedding-models`,
`vertex_ai-*`, `vertex_ai_beta`), and the cross-provider leak is closed.

Tests: update one assertion that pinned the buggy substring behavior
(`custom_openai_endpoint` matching `openai` — not a real provider value);
add 6 new tests covering the leak regressions and the variant-preservation
contract for vertex_ai/bedrock/fireworks.
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: ryan-crabbe-berri <ryan-crabbe-berri@users.noreply.github.com>
…rs and signed request body (BerriAI#27526)

* Fix Bedrock KB pass-through SigV4 headers and signed body

Coerce botocore HeadersDict to a dict for pass-through routes. When
forward_headers is true, drop request headers that collide case-insensitively
with signed headers so client Bearer auth does not shadow AWS SigV4.
Send prepped.body as raw content so the outbound payload matches the
signature after logging hooks mutate the parsed dict.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Simplify pass-through raw body handling

Read the SigV4-signed bytes directly from request.state inside
pass_through_request instead of threading a custom_raw_body argument
through three functions. Helper methods are restored to their original
signatures, and the new branch lives in one place at each httpx call site.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Harden pass-through raw body read from request.state

Guard missing request.state (test fixtures) and ignore non-bytes/str
values so MagicMock does not trigger the SigV4 raw-body path.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Test pass_through_request state_raw_body uses httpx content=

Cover non-streaming (async_client.request) and streaming (build_request)
paths so SigV4 bytes on request.state are not replaced by json= of a
hook-mutated dict.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
…#28728)

* chore(tests): migrate Bedrock CI from AWS account 888602223428 to 941277531214

The original account (888602223428) was put under a security restriction by
AWS after a root access key leaked in a PR comment. While that account works
its way through the AWS Support unlock process, Bedrock-touching CI tests have
been migrated to a fresh account (941277531214).

Changes:
  - Replace 26 hardcoded references to 888602223428 with 941277531214 across
    8 files (provisioned-model ARNs, imported-model ARNs, AgentCore runtime
    ARNs, batch execution role ARN, and example proxy config).
  - The provisioned-model and imported-model ARNs are referenced only from
    mocked unit tests — no AWS resources to recreate.
  - The batch execution IAM role has been recreated in the new account with
    the same name and equivalent permissions.
  - The two AgentCore runtimes (hosted_agent_r9jvp-3ySZuRHjLC,
    hosted_agent_13sf6-cALnp38iZD) are being recreated in the new account
    under the same names — see tools/agentcore-deploy/ in a follow-up.

CircleCI env vars AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_REGION_NAME
were updated separately via the CircleCI API to point at the new account.

Smoke-tested locally against the new account:
  aws bedrock-runtime converse --region us-west-2 \
    --model-id us.anthropic.claude-sonnet-4-5-20250929-v1:0 \
    --messages '[{"role":"user","content":[{"text":"ping"}]}]'
  → 200, model returned 'pong'

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore(tests): refresh AgentCore ARN suffixes to match newly-deployed runtimes

The first migration commit replaced just the account ID, but AgentCore
auto-assigns a random 10-char suffix to every runtime on creation — we
can't reuse the original suffixes (`3ySZuRHjLC`, `cALnp38iZD`) in the
new account. Updated the AgentCore-runtime ARNs in the three files that
reference real runtime IDs (not the mock-based unit-test ARNs).

Deployed runtimes:
  arn:aws:bedrock-agentcore:us-west-2:941277531214:runtime/hosted_agent_r9jvp-Rq79QFC2fp
  arn:aws:bedrock-agentcore:us-west-2:941277531214:runtime/hosted_agent_13sf6-4046UzHSwy

Both runtimes are status=READY and pass a smoke invoke:
  $ aws bedrock-agentcore invoke-agent-runtime --agent-runtime-arn ... --payload '{"prompt":"ping"}'
  → 200, {"result": "echo: ping"}

The agent is a minimal echo (see /tmp/agentcore_deploy/agent.py for the
deploy artifacts). Tests that only verify the SDK wiring will pass; if any
test asserts on agent output content, swap the echo for the real agent.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* chore(tests): point Bedrock batch tests at new-account S3 bucket

The account migration (888602223428 -> 941277531214) was a flat
account-ID swap, which only rewrites ARNs that embed the account
number. S3 bucket names carry no account ID, so the live Bedrock
batch tests still uploaded to `litellm-proxy` — a bucket that lives
in the old account. S3 names are globally unique, and the old account
still holds that name, so it can't be recreated in the new account.

Rename to `litellm-proxy-941277531214` (account-ID suffix guarantees
global uniqueness). The bucket must be created in 941277531214 and the
batch execution role granted s3:GetObject/PutObject/ListBucket on it
before this job is run in CI.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(tests): point live S3 logging test at new-account bucket

Same account-ID-free blind spot as the batch bucket: `load-testing-oct`
lives in the old account and its name can't be reused globally. The
`logging_testing` CI job is wired into the workflow and runs
test_basic_s3_logging, which uploads to this bucket with the CI env
creds, then lists and deletes objects — a live dependency.

Rename to `load-testing-oct-941277531214`. The bucket must exist in the
new account with the CI IAM principal granted
s3:PutObject/GetObject/ListBucket/DeleteObject before this job runs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(tests): repoint Bedrock guardrail IDs to new-account guardrails

The migration left guardrail IDs untouched (no account ID in them), so
all live guardrail tests failed with "guardrail identifier or version
does not exist" against 941277531214. Recreated both guardrails in the
new account and updated the hardcoded IDs:
  - wf0hkdb5x07f -> zgkmukebruil (PII mask: PHONE + CREDIT_DEBIT_CARD,
    with explicit inputAction=ANONYMIZE so masking applies to INPUT,
    which is the source litellm's moderation hook sends)
  - ff6ujrregl1q -> 4w3d1di3snt5 (blocks "coffee"; blocked message set
    to the exact string the tests assert on)

Updated test_bedrock_guardrails.py, otel_test_config.yaml, and the
guardrailConfig in test_bedrock_completion.py. Verified locally: the 5
previously-failing guardrail tests now pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(bedrock): migrate legacy models to current inference profiles

The new CI account (941277531214) cannot invoke legacy Bedrock models
(AWS gates them: "marked by provider as Legacy... not actively using in
the last 30 days"). Migrated the live-call tests:
  - anthropic.claude-3-sonnet-20240229    -> us.anthropic.claude-sonnet-4-5-20250929-v1:0
  - anthropic.claude-3-haiku-20240307     -> us.anthropic.claude-haiku-4-5-20251001-v1:0
Current Claude models on Bedrock require the us. inference-profile prefix
(bare on-demand ids are rejected).

cohere.command-r-plus has no working replacement (all Cohere is legacy-
gated in the new account): swapped to claude-haiku-4-5 in provider-
agnostic param lists. amazon.titan-image-generator skipped (no working
replacement). Mocked/transformation/cost tests that reference the legacy
strings are intentionally left unchanged. Verified live against the new
account.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(bedrock): repoint SageMaker + Knowledge Base to new-account resources

These referenced account-scoped resources by hardcoded id that only
existed in the old account, so the migration's account-ID swap missed
them. Recreated in 941277531214 and repointed:
  - SageMaker endpoint jumpstart-dft-hf-textgeneration1-mp-20240815-185614
    -> litellm-ci-textgen (gpt2 on a TGI container, ml.g5.xlarge)
  - Bedrock Knowledge Base T37J8R4WTM -> LCYXFBR2TU (OpenSearch Serverless
    vector store + titan-embed-text-v2, seeded with a LiteLLM doc)
Verified live: test_sagemaker.py (12 passed) and
test_bedrock_knowledgebase_hook.py (12 passed).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(reasoning_effort_grid): skip bedrock claude-opus-4-7 cells (not entitled on 941277531214)

claude-opus-4-7 is listed in the new Bedrock CI account's foundation
models but invoke is denied (AccessDeniedException: "not available for
this account"). Bedrock access to the flagship Opus requires an AWS
Sales request, not the self-serve model-access toggle, so it can't be
enabled inline with the rest of the account migration.

Add an optional `skip_reason` to ModelEntry and set it on the
bedrock-claude-opus-4-7 entry; the grid test honors it via pytest.skip.
Cell count (231) and route coverage are unchanged, so the structural
asserts still pass. Restore coverage by deleting the one skip_reason
line once access is granted.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(bedrock): swap/skip legacy-gated models unavailable on new CI account

The migrated AWS account (941277531214) cannot access several models that
the old account could, so the remaining red CI jobs were hitting real
Bedrock "Access denied / Legacy" and "account not authorized" errors:

- image_gen: skip both Nova Canvas test classes (amazon.nova-canvas-v1:0 is
  legacy-gated), matching the existing titan skip.
- batches: skip test_async_file_and_batch (Bedrock batch inference is not
  authorized on the new account; requires an AWS support case).
- litellm_overhead: swap legacy claude-3-5-haiku for the active
  us.anthropic.claude-haiku-4-5 inference profile.
- test_completion_claude_3_function_call: swap legacy claude-3-sonnet for the
  active us.anthropic.claude-sonnet-4-5 inference profile.

https://claude.ai/code/session_01Y7zgHYu9GX29YRwV4yiWAa

* test(bedrock): fix remaining e2e legacy-model + batch failures on new CI account

- e2e_openai_endpoints: skip test_bedrock_batches_api (Bedrock batch inference
  is not authorized on account 941277531214) and migrate the missed
  s3_bucket_name in oai_misc_config.yaml to litellm-proxy-941277531214.
- build_and_test: swap legacy bedrock claude-3-sonnet for the active
  us.anthropic.claude-sonnet-4-5 inference profile in the proxy structured
  output e2e test.

https://claude.ai/code/session_01Y7zgHYu9GX29YRwV4yiWAa

* test(bedrock): make opus-4-7 + batch cells fail loudly and mock image-gen (BerriAI#28791)

Replace the silent skips added for the new CI account with noisier behavior:
- reasoning-effort grid: opus-4-7 cells now fail (when AWS creds are present)
  instead of skipping, so the missing entitlement stays visible in CI; they
  still skip when AWS creds are absent (local dev)
- Bedrock batch inference tests: drop the skip so they run and fail until
  batch access is granted
- Titan + Nova Canvas image-gen tests: mock the Bedrock HTTP call so the
  transform + cost-tracking path stays under test without live model access

https://claude.ai/code/session_01MT7SWDnXUjv6e6EPG7BDjT

Co-authored-by: Claude <noreply@anthropic.com>

* test(bedrock): use pytest.xfail for known-failing opus-4-7 cells

Replace pytest.fail with pytest.xfail when a model has a fail_reason,
so known-broken cells stay visible as XFAIL without keeping CI red.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

---------

Co-authored-by: Mateo <mateo@Mateos-MacBook-Pro.local>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
…http_request (BerriAI#28794)

Co-authored-by: Yassin Kortam <yassinkortam@Yassins-MacBook-Pro.local>
* chore(proxy): route path-dependent call sites through get_request_route

Replace direct ``request.url.path`` reads in auth, ACL, routing, and
audit-log decisions with ``get_request_route(request)`` — the helper
already added in ``auth/auth_utils.py`` that returns the ASGI
``scope["path"]`` with ``root_path`` stripped. Starlette reconstructs
``url.path`` from the Host header; ``scope["path"]`` is uvicorn's
parse of the request line and matches what FastAPI dispatches on, so
it's the authoritative route for any decision that should agree with
the actual handler.

Sites:
- _experimental/mcp_server/auth/user_api_key_auth_mcp.py
- management_endpoints/mcp_management_endpoints.py
- vector_store_endpoints/utils.py
- pass_through_endpoints/pass_through_endpoints.py
- auth/route_checks.py
- litellm_pre_call_utils.py
- spend_tracking/spend_management_endpoints.py
- common_utils/http_parsing_utils.py
- management_helpers/utils.py
- health_endpoints/_health_endpoints.py

Adds regression tests in tests/proxy_unit_tests/test_proxy_routes.py
that construct a Request with scope["path"] set to a benign route and
the Host header crafted so url.path would resolve differently; each
site's decision is asserted against scope["path"].

* chore(proxy): make get_request_route imports lazy at call sites

Move the ``from litellm.proxy.auth.auth_utils import get_request_route``
imports added in the prior commit back to the function bodies that use
them. The module-level form participates in a long-standing import
cycle through ``auth_utils -> _types -> ...`` and was flagged by CodeQL
on the PR; the lazy form matches the pattern the proxy already uses
for ``user_api_key_auth`` and related helpers elsewhere in these files.

Also drop the ``RouteChecks._is_assistants_api_request`` delegation in
``_get_metadata_variable_name`` introduced in the prior commit — the
delegation pulled ``RouteChecks`` into the same cycle, and the call
site reuses the resolved route for its other branches, so inlining
the substring check is both cycle-free and avoids a redundant second
``get_request_route`` call.

Comment in test_proxy_routes.py acknowledges that the two MCP table
entries exercise ``get_request_route`` directly rather than the full
production handler (which needs ASGI scope + MCP state to invoke).

---------

Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: user <70670632+stuxf@users.noreply.github.com>
* feat(dashboard): navbar hierarchy + Agent Platform notifications (BerriAI#27543)

* feat(dashboard): refine navbar zones and Agent Platform notice

Restructure the admin navbar for production users: clear product vs community
vs personal columns with vertical dividers, icon-only Slack/GitHub in a
shared chip, and Docs/Blog typography aligned on an 8px rhythm.

Add a notifications bell with popover linking to the LiteLLM Agent Platform
repo and optional mark-as-read persistence.

Promote the account control with initials avatar, single-line display name,
and navDisplayName mapping for placeholder user ids (e.g. default_user_id).

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(dashboard): address PR review — AntD buttons, public page guard, dedupe regex

- Replace raw <button> with AntD Button in BlogDropdown, NotificationsBell, UserDropdown, and test mock
- Guard NotificationsBell + container behind !isPublicPage to avoid rendering on public pages
- Remove redundant equality checks in navDisplayName (regex already covers them)
- Remove unused `lower` variable after simplification

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>

* fix(dashboard): drop dead useHealthReadiness import in navbar

The module was removed in BerriAI#27896 (replaced by useHealthReadinessDetails),
but the import survived the rebase. The symbol is unused — only
useHealthReadinessDetails is consumed in the file. Removing the dead
import unblocks the UI TypeScript build.

* fix(dashboard): align CommunityEngagementButtons test with icon-only aria-labels

The component was refactored to an icon-only chip with aria-label='LiteLLM
on GitHub' (squash BerriAI#27543), but the test still asserted /star us on
github/i. Update the query to match the rendered accessible name.

* refactor(dashboard): drop unused props from NavbarProps

The navbar refactor moved user identity + dark-mode state to internal
hooks (useAuthorized, useWorker), but the NavbarProps interface still
declared userID, userEmail, userRole, premiumUser, isDarkMode, and
toggleDarkMode as required, forcing every caller to thread them through.

Drop them from the interface and all four call sites (page.tsx,
(dashboard)/layout.tsx, public_model_hub.tsx, navbar.test.tsx). Also
shrinks the destructure in layout.tsx so the now-unused locals stop
being pulled out of useAuthorized().

* refactor(dashboard): use useSyncExternalStore for NotificationsBell dismiss flag

Reads/writes of the litellmHideAgentPlatformBanner key were done
directly inside NotificationsBell via a useEffect + useState pair.
Every other localStorage-backed flag in the dashboard (Disable
ShowPrompts, DisableBouncingIcon, DisableShowNewBadge,
DisableUsageIndicator, DisableBlogPosts) is wrapped in a
useSyncExternalStore hook over localStorageUtils so all mounted
components stay in sync.

Extract useHideAgentPlatformBanner to follow the same shape, swap
NotificationsBell to consume it, and add a regression test that
two sibling bells stay in sync without a remount when one is
dismissed.

* refactor: mask credential fields in proxy settings GET responses (BerriAI#28682)

* refactor: mask credential fields in proxy settings GET responses

Brings SSO settings, cache settings, and the email/Slack alerting view in
/get/config/callbacks in line with the HashiCorp Vault config-override
pattern, so persisted credentials are not transported back to the UI in
plaintext.

* refactor: harden short-value masking and hoist alerting var constant

Closes two review observations:

- mask_sensitive_keys now replaces short values (below the visible
  prefix+suffix length) with an all-mask string instead of returning them
  unchanged, so a 1-7 character credential is no longer round-tripped
  verbatim.
- _ALERTING_SENSITIVE_VARS is moved out of get_config() to a module-level
  constant, matching the analogous _SSO_SENSITIVE_FIELDS and
  _CACHE_SENSITIVE_FIELDS in the SSO and cache endpoint files.

---------

Co-authored-by: Krrish Dholakia <krrish+github@berri.ai>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…riAI#28809)

The Key Info Overview tab's Spend card truncated sub-dollar budgets to
"$0" because formatNumberWithCommas defaults to 0 decimals. The Settings
tab passes 2; align the overview so a $0.10 budget renders as "$0.10".

Resolves LIT-2845
…erriAI#28442)

* feat(proxy): allow llm_api_routes virtual keys to list MCP servers

Add a new `mcp_discovery_routes` group (GET /v1/mcp/server and GET
/v1/mcp/server/{server_id}) and include it in `llm_api_routes` so that
virtual keys configured with `allowed_routes=["llm_api_routes"]` can
discover the MCP servers they have access to. Previously these calls
failed with 'Virtual key is not allowed to call this route. Only allowed
to call routes: [llm_api_routes]'.

The GET handlers already sanitize the response for restricted virtual
keys via `_sanitize_mcp_server_list_for_virtual_key`, stripping
credential-bearing fields (url, headers, env). Write methods
(POST/PUT/DELETE) on the same paths remain gated by the existing
handler-level admin role checks.

The new discovery list is intentionally kept OUT of
`mcp_inference_routes`, so `is_llm_api_route()` still returns False
for these paths — this preserves the existing contract that
DISABLE_LLM_API_ENDPOINTS must not block the Admin UI from listing MCP
servers.

Co-authored-by: ryan-crabbe-berri <ryan-crabbe-berri@users.noreply.github.com>

* refactor(proxy): make MCP discovery carve-out method-aware

Replace the `mcp_discovery_routes` group in `llm_api_routes` with a
method-aware special case inside `is_virtual_key_allowed_to_call_route`.
Virtual keys with allowed_routes=["llm_api_routes"] are now permitted
to call only GET /v1/mcp/server and GET /v1/mcp/server/{server_id} —
non-GET methods and multi-segment admin sub-paths fall through to the
existing 403. This keeps the general llm_api_routes list free of
management paths and avoids accidentally exposing POST/PUT/DELETE
writes through the route-check layer.

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: ryan-crabbe-berri <ryan-crabbe-berri@users.noreply.github.com>
* chore(proxy): route path-dependent call sites through get_request_route

Replace direct ``request.url.path`` reads in auth, ACL, routing, and
audit-log decisions with ``get_request_route(request)`` — the helper
already added in ``auth/auth_utils.py`` that returns the ASGI
``scope["path"]`` with ``root_path`` stripped. Starlette reconstructs
``url.path`` from the Host header; ``scope["path"]`` is uvicorn's
parse of the request line and matches what FastAPI dispatches on, so
it's the authoritative route for any decision that should agree with
the actual handler.

Sites:
- _experimental/mcp_server/auth/user_api_key_auth_mcp.py
- management_endpoints/mcp_management_endpoints.py
- vector_store_endpoints/utils.py
- pass_through_endpoints/pass_through_endpoints.py
- auth/route_checks.py
- litellm_pre_call_utils.py
- spend_tracking/spend_management_endpoints.py
- common_utils/http_parsing_utils.py
- management_helpers/utils.py
- health_endpoints/_health_endpoints.py

Adds regression tests in tests/proxy_unit_tests/test_proxy_routes.py
that construct a Request with scope["path"] set to a benign route and
the Host header crafted so url.path would resolve differently; each
site's decision is asserted against scope["path"].

* chore(proxy): make get_request_route imports lazy at call sites

Move the ``from litellm.proxy.auth.auth_utils import get_request_route``
imports added in the prior commit back to the function bodies that use
them. The module-level form participates in a long-standing import
cycle through ``auth_utils -> _types -> ...`` and was flagged by CodeQL
on the PR; the lazy form matches the pattern the proxy already uses
for ``user_api_key_auth`` and related helpers elsewhere in these files.

Also drop the ``RouteChecks._is_assistants_api_request`` delegation in
``_get_metadata_variable_name`` introduced in the prior commit — the
delegation pulled ``RouteChecks`` into the same cycle, and the call
site reuses the resolved route for its other branches, so inlining
the substring check is both cycle-free and avoids a redundant second
``get_request_route`` call.

Comment in test_proxy_routes.py acknowledges that the two MCP table
entries exercise ``get_request_route`` directly rather than the full
production handler (which needs ASGI scope + MCP state to invoke).

---------

Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: user <70670632+stuxf@users.noreply.github.com>
…BerriAI#28737)

* fix(team): keep team_alias cache in sync on _cache_team_object writes

_cache_team_object wrote only to the team_id:<id> cache key, but the
JWT auth path that uses team_alias_jwt_field reads from a separate
team_alias:<alias> key (get_team_object_by_alias caches under both
keys on miss, but reads only the alias-keyed one). After any
team-mutation endpoint (team_model_add, team_model_delete,
update_team, the two access-group writes) the team_id cache was
refreshed but the team_alias cache stayed stale until TTL — JWT
callers using team_alias_jwt_field kept seeing the pre-mutation
team for the full cache window.

Mirror the write under the alias key inside _cache_team_object so
every existing caller stays in sync without further changes. Skip
the alias write when team_alias is None/empty so we don't collide
across alias-less teams.

Surfaced testing the LIT-3244 cherry-pick on patch/1.86.0: the
LIT-3244 fix correctly invalidated the team_id cache but the
customer's JWT used team_alias_jwt_field, so they kept hitting the
stale alias-keyed entry.

* fix(team): delete (not overwrite) team_alias cache on _cache_team_object

The prior shape of this PR wrote both team_id:<id> AND team_alias:<alias>
from _cache_team_object. team_alias is NOT unique in the schema
(no @unique on LiteLLM_TeamTable.team_alias), and get_team_object_by_alias
enforces uniqueness on its own DB-fetch path (len(teams) > 1 raises).
Writing the alias-keyed cache from the generic refresh path bypassed
that check: a team admin renaming their team to collide with another
team's alias could silently overwrite the cached team for JWT-by-alias
auth, swapping the resolved team under that alias for the cache window.

Switch the alias-keyed operation from a write to a delete (mirroring
the dual-cache delete pattern in _delete_cache_key_object). After every
team write, the next JWT-by-alias reader cache-misses and falls through
to get_team_object_by_alias, which (a) re-fetches the fresh team from
DB, closing the LIT-3244 staleness gap that motivated this PR, and
(b) enforces alias uniqueness before populating either cache key.

team_id:<id> writes are unchanged — team_id is the table PK and is
guaranteed unique.

Surfaced in veria-ai review on BerriAI#28739.

* fix(managed-files): anchor model_id regex so it doesn't match llm_output_file_model_id

extract_model_id_from_unified_id used `re.search(r"model_id,([^;]+)", ...)`
which substring-matches the `model_id,` inside the file-ID encoding's
`llm_output_file_model_id,<deployment_uuid>` field. parse_unified_id
then fed that deployment UUID back into the auth path as a model
candidate via _extract_models_from_managed_resource_id, and every
team-BYOK file attach 403'd with:

    team not allowed to access model. This team can only access
    models=['openai/*']. Tried to access <deployment-uuid>

The team's models list correctly contains the public name (`openai/*`)
that target_model_names matches, but the bogus UUID candidate fails
the wildcard check first.

Anchor the regex to a field boundary (`(?:^|;)model_id,`) so it
matches the legitimate top-level `model_id,<value>` field on
vector_store unified IDs and skips substring matches inside other
fields. File-IDs (which have no top-level `model_id` field) now
return None and contribute no spurious UUID candidate.

Surfaced reproducing LIT-3244 on patch/1.86.0 with the customer's
exact flow: team with openai/* BYOK deployment, JWT-scoped user,
POST /v1/vector_stores/{id}/files attaching a file uploaded with
target_model_names=openai/gpt-4o.
…erriAI#28822)

* fix(proxy): hydrate wildcard discovery credentials

* fix(proxy): constrain wildcard credential hydration

Co-authored-by: Dibyo Mukherjee <dibyo@adobe.com>
Creates litellm_oss_agent_shin_MM_DD_YYYY from main every day at 00:00 UTC.
Lets us retarget oss-agent-shin fork PRs onto a canonical branch so CircleCI runs with secrets, without granting the agent write access.

Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
…I#28827)

* test(proxy): add harness for proxy_server.py behavior-pinning

Creates tests/test_litellm/proxy/proxy_server/ with:
- conftest.py: 11 shared fixtures (app, client, mock_prisma, auth_as,
  mock_router with parametrized response builders, normalize, etc.)
- _coverage_check.py: per-PR coverage gate (line + branch) against a
  baseline, self-selects target by inspecting which placeholder files
  have been filled
- _pin_check.py: AST-based gate that verifies every pin-list item has
  >=1 happy + >=1 error test with a real assertion (no status-only)
- test_harness_smoke.py: 19 smoke tests covering every fixture +
  both scripts end-to-end
- 26 placeholder test files (one docstring each) reserved for
  follow-up PRs per the directory ownership in the Notion plan
- .coverage_baseline pinned at 0% so future PRs measure deltas
  against new-tests-only and aren't entangled with the broader
  scattered test suite

Adds a dedicated proxy-server job to test-unit-proxy-endpoints.yml
so this directory's runtime + coverage are tracked independently.

Plan: https://www.notion.so/36c43b8acdab81ee845fd5365128a2fc

* ci(proxy-endpoints): allow workflow_dispatch

Lets the workflow be triggered manually on a branch via
`gh workflow run`, which is needed for the verify-first
flow on workflow changes before opening a PR.

* test(proxy): address review feedback on proxy_server harness

- conftest.py: anchor sys.path insert to __file__ (Path(__file__).resolve().parents[4])
  instead of CWD-relative os.path.abspath("../../../../") which resolved
  to the wrong directory when pytest is launched from the repo root.
- _coverage_check.py: actually read .coverage_baseline and use it as
  the floor (line_min = max(target, baseline)). Closes the gap between
  the PR description's "delta semantics" and what the script was doing.
  With baseline=0.0 today this is a no-op; future PRs that update the
  baseline cause regressions (test deletions etc.) to trip the gate
  even if the static PR target is still met.
- _pin_check.py: drop unreachable startswith("_") guard
  (test_*.py glob never yields underscore-prefixed names) and read
  each test file once instead of twice.
…sidency (BerriAI#28626)

* feat(openai): apply regional-processing cost uplift for EU/US data residency

OpenAI charges a 10% uplift on the latest GPT models when requests are
served from a regionalized hostname (eu./us.api.openai.com).  Infer the
region from `api_base`, expose it on `kwargs["litellm_params"]["data_residency"]`,
and multiply the computed cost by a per-model
`regional_processing_uplift_multiplier_<region>` field.

https://claude.ai/code/session_012ebH44s7ohYxjoix5CXzTW

* test: allow regional_processing_uplift_multiplier_{eu,us} in model_prices schema

* fix(cost): tighten data_residency inference and restore model_cost in tests

- Only infer OpenAI data_residency when custom_llm_provider == "openai";
  drop the implicit None fallback so non-OpenAI callers can't accidentally
  pick up a regional tag from a stray OpenAI hostname.
- _local_model_cost_map fixture now snapshots and restores
  litellm.model_cost and LITELLM_LOCAL_MODEL_COST_MAP so tests don't leak
  state across the session.

* refactor(openai): move data_residency helper under llms/openai

* fix: thread data_residency through realtime stream cost calculation

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(cost): thread data_residency through batch_cost_calculator

Apply the OpenAI regional-processing uplift multiplier to retrieve_batch
cost paths so Batch API requests served via eu./us.api.openai.com are
priced at the same uplifted token rates as completions/transcriptions.

* refactor(openai): encapsulate provider check inside infer_openai_data_residency

Move the custom_llm_provider == "openai" guard from get_litellm_params
into the helper itself so the core utility no longer carries
provider-specific dispatch logic. Callers pass through the provider
unconditionally; the helper returns None for any non-OpenAI provider.

* fix(responses): thread data_residency through Responses logging params

The Responses API paths build their logging litellm_params dict after
provider resolution but did not include data_residency, so cost calc
saw None even when the effective api_base was a regional OpenAI host.

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
…erriAI#28112)

* chore(admin-ui): regenerate static export with trailingSlash: true

Rebuilds litellm/proxy/_experimental/out/ from ui/litellm-dashboard with
`trailingSlash: true` enabled in next.config.mjs. Next.js now emits every
route as <dir>/index.html (e.g. mcp/oauth/callback/index.html) instead of
<dir>.html with a sibling metadata-only directory, which fixes the 404 on
extensionless URLs served through FastAPI's StaticFiles(html=True) mount.

This is the build artifact half of the fix; the config change, Dockerfile
cleanup, and regression test live in the follow-up source PR that stacks
on top of this branch.

* fix(admin-ui): emit nested routes as <dir>/index.html (BerriAI#28106)

Linear and other OAuth providers redirect the user back to
/ui/mcp/oauth/callback?code=...&state=... after the consent step. The
packaged Next.js static export only produced /ui/mcp/oauth/callback.html,
so FastAPI's StaticFiles served a 404 on the extensionless URL and the
OAuth handshake never completed.

The Dockerfile.non_root build step tried to paper over this at image-build
time with `for html_file in *.html; do ...`, but that shell glob does not
recurse, so nested routes like mcp/oauth/callback.html were left stranded
next to an empty mcp/oauth/callback/ directory containing only Next.js
metadata. The runtime restructure step in proxy_server.py was then skipped
because the .litellm_ui_ready marker had already been dropped.

Set trailingSlash: true in the dashboard's Next.js config so the export
emits every nested route as <dir>/index.html natively. The Dockerfile loop
is now a no-op for the bundled UI and has been removed; the
.litellm_ui_ready marker is still written so the proxy keeps skipping the
redundant Python restructure step at startup. Stacks on top of the static
export regeneration in the parent branch.

* chore: restore origin/litellm_internal_staging out files
…AI#28627)

* fix(azure): preserve AD token refresh in v1 OpenAI client path

The /openai/v1/ code path (api_version in {"v1", "latest", "preview"})
constructs a plain OpenAI/AsyncOpenAI client, but only forwarded
`api_key` from `azure_client_params`. When `enable_azure_ad_token_refresh`
is set (or any AD-only auth), `api_key` is None and the client
constructor raised "The api_key client option must be set...", breaking
every Azure call with a v1 api_version.

The OpenAI SDK (>=2.20.0) accepts a callable for `api_key` and re-invokes
it on every request via `_refresh_api_key`, so we now forward
`azure_ad_token_provider` directly — preserving the per-request token
refresh behavior of the regular AzureOpenAI client and avoiding the
expiry hole that resolving the token once at client-creation time would
introduce. Static `azure_ad_token` strings fall through to `api_key`.

For the async path we wrap the sync provider returned by azure-identity
in an async function since AsyncOpenAI expects `Callable[[], Awaitable[str]]`.

Fixes BerriAI#27945

https://claude.ai/code/session_01UnzrDSFUUgp5T2wRoPMxq5

* fix(azure): offload sync token provider to thread in v1 async wrapper

* fix(azure): include AD credential identity in v1 client cache key

---------

Co-authored-by: Claude <noreply@anthropic.com>
* fix(ui): route API Reference back to query-param page

The path-based /ui/api-reference route was broken in practice — the
page-local useProxySettings hook didn't match what the root page passes
down. Remove api_ref from the migration maps (LEGACY_REDIRECTS in
app/page.tsx, MIGRATED_PAGES in leftnav.tsx and (dashboard)/layout.tsx),
point the leftnav item back at page="api_ref", and restore the api_ref
render branch in the root page. The path-based page.tsx and the
useProxySettings hook stay in place unchanged; only api_ref is moved
back to query-param routing while the migration infrastructure is
preserved for future page moves.

* fix(ui): alias ?page=api-reference to api_ref branch

Handles bookmarks of the hyphen-form query param that was live during
the brief path-based migration window, so they render the working
APIReferenceView instead of falling through to the default page.
yuneng-berri and others added 27 commits May 29, 2026 23:17
* test(proxy/proxy_server): pin forwarding routes (PR2) (BerriAI#28887)

* test(proxy): pin proxy_server.py forwarding-route behavior

PR2 of the proxy_server.py behavior-pinning project: fills the 12
forwarding-route test files added by the harness PR with happy + error
pins for all 52 LLM-facing routes (models, chat/completions, completions,
embeddings, moderations, audio, assistants, threads, utils, model-info,
model-metrics, queue). Every happy-path test asserts the full response
dict via normalize() so the gate enforces real shape pinning rather
than status codes.

* test(proxy): drop task-plumbing comments from PR2 test files

* test(proxy): tighten PR2 error-path status-code pins

Apply the same review feedback Greptile gave on PR1 (BerriAI#28856) and PR3
(BerriAI#28850) to PR2's forwarding-route tests:

- Replace permissive `>= 400` / `in (X, Y)` status assertions with the
  exact 500/405 the handler actually returns, so a regression that
  silently shifts the code now fails the pin.
- Add a body-presence check alongside each tightened status assertion
  to satisfy _pin_check.py's no-status-only rule.

---------

Co-authored-by: Claude <noreply@anthropic.com>

* test(proxy): pin proxy_server.py non-route surface behavior (PR1) (BerriAI#28856)

* test(proxy): pin proxy_server.py non-route surface behavior (PR1)

Fills the 7 PR1 placeholder files under tests/test_litellm/proxy/proxy_server/
with behavior pins for the non-route surface of proxy_server.py:
lifecycle/init/shutdown, ProxyConfig class methods, DB-overlay config scrubbers,
spend counters, background-health helpers, OpenAPI customization, exception
handlers, and streaming-generator helpers.

233 tests cover 101 pin-list symbols (1+ happy + 1+ error each). New-tests-only
coverage on litellm/proxy/proxy_server.py: 32.80% line / 20.91% branch (PR1
gate: 25% line / 18% branch). Full directory runs in ~22s with -n 4.

Plan: https://www.notion.so/Plan-Pin-proxy_server-py-behavior-2026-05-25-36c43b8acdab81ee845fd5365128a2fc

* test(proxy): address Greptile review comments on test_lifecycle.py

- test_initialize_signature_is_async_with_expected_params: hard-code
  expected_param_count so a signature change actually trips the gate
  (previously both sides of the comparison were len(sig.parameters)).
- test_check_request_disconnection_invalid_when_connected_times_out:
  patch asyncio.sleep so the test no longer spins for ~1.2 s of real
  wall-clock; timeout lowered to 0.05 s.

---------

Co-authored-by: Claude <noreply@anthropic.com>

* test(proxy/proxy_server): pin control-plane routes (PR3) (BerriAI#28850)

* test(proxy/proxy_server): pin misc routes (PR3, partial)

Adds happy + error tests for the misc control-plane routes:
GET /, /routes, /adaptive_router/state, /get_logo_url,
/get_image, /get_favicon.

Also gitignores .pin_list.txt (used by the pin gate).

* test(proxy/proxy_server): pin login/SSO routes (PR3, partial)

Adds happy + error tests for the 5 login/SSO control-plane routes:
GET /fallback/login, POST /login, POST /v2/login, POST /v3/login,
POST /v3/login/exchange. Mocks authenticate_user and
create_ui_token_object at their imported location.

* test(proxy/proxy_server): pin onboarding routes (PR3, partial)

Adds happy + error tests for the 2 onboarding control-plane routes:
GET /onboarding/get_token, POST /onboarding/claim_token. Wires a
MagicMock async context manager for prisma_client.db.tx() and
signs the onboarding JWT with the patched master_key.

* test(proxy/proxy_server): pin model_cost_map reload routes (PR3, partial)

Adds happy + error tests for the 5 model-cost-map control-plane routes:
POST /reload/model_cost_map, POST|DELETE|GET
/schedule/model_cost_map_reload(/status), GET /model/cost_map/source.
Attaches litellm_config to mock_prisma per-test (the table is not in
the default _PRISMA_TABLES fixture).

* test(proxy/proxy_server): pin anthropic_beta_headers reload routes (PR3, partial)

Adds happy + error tests for the 4 anthropic-beta-headers control-plane
routes: POST /reload/anthropic_beta_headers, POST|DELETE|GET
/schedule/anthropic_beta_headers_reload(/status). Stubs
db.litellm_config (not in default _PRISMA_TABLES) and monkeypatches
reload_beta_headers_config so no network calls fire.

* test(proxy/proxy_server): pin invitation routes (PR3, partial)

Adds happy + error tests for the 4 invitation control-plane routes:
POST /invitation/new, GET /invitation/info, POST /invitation/update,
POST /invitation/delete. Patches _user_has_admin_privileges /
_user_has_admin_view to avoid extensive get_user_object mocking.

* test(proxy/proxy_server): pin config CRUD routes (PR3, partial)

Adds happy + error tests for the 8 config-CRUD control-plane routes:
POST /config/update, POST|GET /config/field/update|info, GET /config/list,
POST /config/field/delete, POST /config/callback/delete,
GET /get/config/callbacks, GET /config/yaml. Attaches litellm_config
to mock_prisma per-test.

* test(proxy/proxy_server): tighten pin assertions per review

- test_routes_misc.py: `b"" in response.content` is trivially true;
  replace with `len(response.content) > 0` so an empty 405 body trips
  the gate.
- test_routes_login_sso.py: `len(response.content) >= 0` is trivially
  true; tighten to `> 0`.
- test_routes_anthropic_beta.py: replace brittle string-literal checks
  on the serialized JSON (`'"interval_hours": 12' in payload`) with
  `json.loads` + dict access so the assertion survives any serializer
  spacing.
- test_routes_config.py: `assert status_code in (404, 500)` was too
  permissive; the handler re-raises HTTPException(404) verbatim, so
  pin 404 strictly.

---------

Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
…iAI#29083)

* test(e2e): cover Internal User create-key flow when in no teams

The seeded e2e-internal-user is in two teams, so the "no team" branch
of the Create Key modal — where the team dropdown must render empty —
was unreachable. Seeds a noteam@test.local user and adds a spec that
logs in fresh, opens the modal, and asserts the dropdown has zero
options.

* test(e2e): harden no-team dropdown assertion + add with-teams counterpart

Replace the one-shot count() check with a settled-empty assertion: wait for
the dropdown's loaded "No teams found" state before asserting zero options,
so the test can't pass on a transient empty frame while the team-options
request is still in flight.

Add internalUserWithTeams.spec.ts as the differential partner; it logs in as
the seeded e2e-internal-user (two team memberships) and asserts the dropdown
lists exactly those teams. Without it, the no-team spec's zero-options
assertion would still pass against a regression that empties the dropdown for
every user.
BerriAI#29077)

* test(e2e): assert internal-user navbar identity is scoped to that user

The existing login.spec.ts only checks the admin's navbar identity.
This adds the symmetric check for the internal user — verifying the
account button + dropdown surface the internal user's email, id, and
role, and that no admin-scoped values leak through.

* test(e2e): harden navbar identity test per review feedback

Locate the user dropdown panel by a data-testid on the popupRender div
instead of Ant Design internal + Tailwind class names, so styling
refactors no longer risk breaking the identity-scoping assertions.
Source the seeded user emails/ids from shared constants (match seed.sql)
instead of hardcoding them inline.
… providers (BerriAI#28868)

* feat(anthropic/messages): in-gateway context_management polyfill for non-Anthropic providers

- Add `context_management/` module with `clear_tool_uses_20250919` editor
  dispatched before chat-completions translation on `/v1/messages`
- Hard-protect most-recently completed tool_result from being cleared
- Attach `context_management.applied_edits` to both non-streaming and
  streaming (final `message_delta`) responses
- Bedrock Converse: forward `context_management`; filter to
  `compact_20260112`-only edits with `compact-2026-01-12` beta header
- token_counter: guard Anthropic-format tools (no `function` key) to
  prevent AttributeError during polyfill token counting
- Streaming: handle empty-choices usage-only trailing chunks
- Skip polyfill when `litellm.drop_params = True`

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(bedrock): pop None context_management before sending to Bedrock Converse

If context_management is forwarded as None (e.g. when mapping returns
None for an invalid format), _filter_context_management_for_bedrock_converse
previously returned early without removing the key, leaving
"context_management": null in the request and causing a validation
error. Pop the key when the value is not a dict.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(bedrock/converse): pop None context_management; extract helpers to fix PLR0915

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(anthropic/messages): check per-request drop_params alongside global

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(anthropic/messages): preserve drop_params for downstream and respect explicit False

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix: lazy debug logging in clear_tool_uses; remove unused context_management constants

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(anthropic/messages): guard context_management polyfill with try/except

Wrap apply_context_management() in a try/except so any failure (e.g.
litellm.token_counter raising on an unknown tokenizer or unexpected
message format) is logged but does not crash the underlying LLM
request. The polyfill is a best-effort additive feature; on failure we
forward the original messages without applied edits.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(token_counter): guard None input_schema in Anthropic tool fallback

Use `or {}` instead of `.get(..., {})` so explicit null parameters do not
raise AttributeError when formatting function definitions for token counting.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: minimize context_management polyfill threading

- Use None (not empty list) for polyfill_applied_edits when context
  management isn't requested, so semantics of 'feature not requested'
  vs 'feature requested but no edits applied' are distinct.
- In the streaming iterator, only pass applied_edits to the per-chunk
  translator on the final (finish_reason) chunk; intermediate chunks
  ignore it anyway, and this makes intent explicit on both sync and
  async paths.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(context_management): align tool_use counts and normalize list spec

- _count_tool_uses now requires a string id, matching _collect_tool_use_ids_in_order so the tool_uses trigger can't fire on blocks that aren't clearable.
- apply_context_management dispatcher now accepts the OpenAI list form and normalizes it via AnthropicConfig.map_openai_context_management_to_anthropic, so the polyfill path no longer silently no-ops on list input.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* feat(context_management): add compact_20260112 polyfill for non-Anthropic providers

Implements an in-gateway compaction polyfill that summarizes long conversations
using a configurable model when `compact_20260112` is requested for non-Anthropic
targets (e.g. OpenAI, Gemini), matching Anthropic's context management beta
behaviour for those providers.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(compact): skip tool_result-only user turns; bedrock: elif for context_management

- compact_20260112 Phase D: when keeping the last user turn after a full
  summary, skip role=user turns whose content is exclusively tool_result
  blocks. Such turns translate to OpenAI tool-role messages with no
  preceding assistant tool_calls (those got summarized away), which
  non-Anthropic providers reject. Fall back to a synthetic continuation
  prompt if no eligible user question exists, so the downstream call
  always has a non-empty user message.
- bedrock converse: chain the context_management param as elif so it
  follows the same if/elif pattern as the surrounding thinking/
  reasoning_effort checks.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(anthropic): post-compaction question selection, system type, sync stream merge

- compact.py: select last user question from effective_messages (post-compaction slice) instead of raw messages, so prior summarized turns aren't reintroduced
- handler.py: widen _prepare_completion_kwargs system parameter type to Union[str, List[Dict]] matching PolyfillResult.system
- streaming_iterator.py: mirror async hold-and-merge logic in sync __next__ so context_management is attached to the final merged message_delta when stop_reason and usage arrive in separate chunks

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(anthropic/messages): apply context_management on sync path; clear held stop_reason chunk in async iterator

- Sync `anthropic_messages_handler` was silently dropping the
  `context_management` kwarg via `ANTHROPIC_ONLY_REQUEST_KEYS` after the
  polyfill was moved into the async handler. Bridge to the async
  dispatcher with `run_async_function` so `litellm.messages.create()`
  callers keep working (regressed e.g. `clear_tool_uses_20250919`).
- In the streaming iterator's `__anext__` `StopIteration` handler, clear
  `self.holding_stop_reason_chunk` after capturing it (matches `__next__`)
  so a subsequent call doesn't re-emit the same chunk.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(bugfixes): bedrock None context_mgmt; stream per-instance queue; sync polyfill; trailing-chunk passthrough

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(anthropic): silently drop trailing chunks after usage; remove dead _polyfill_result key

- streaming_iterator: in sync __next__, after the usage chunk has been
  merged and emitted, silently consume any trailing provider events
  via 'continue' instead of forwarding them through the queue. Trailing
  chunks would translate to content_block_delta or message_delta and
  violate Anthropic SSE ordering after the final message_delta. The
  async __anext__ already drops these via 'if not self.queued_usage_chunk:'
  gating, so this aligns sync and async behavior.

- handler: drop unused '_polyfill_result' from ANTHROPIC_ONLY_REQUEST_KEYS.
  PolyfillResult is passed as an explicit arg to the adapter methods, never
  through extra_kwargs, so the entry was dead code.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* refactor(anthropic): extract usage-merge helper; guard empty slice-only compaction result

- Extract the duplicated hold-and-merge usage logic from the sync __next__ and
  async __anext__ paths into a shared _merge_usage_into_held_stop_reason_chunk
  helper so the subtle cache-token / context_management attachment lives in
  exactly one place.
- In the compact_20260112 slice-only path, fall back to _select_last_user_question
  when _strip_compaction_blocks produces an empty list (e.g. messages ending on
  an assistant turn whose only content was the compaction block) so the
  downstream API never receives an empty messages array.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* refactor(anthropic/context_management): streaming iterator compaction fixes and compact polyfill improvements

- Extract usage-merge helper; guard empty slice-only compaction result
- Silently drop trailing chunks after usage; remove dead _polyfill_result key
- Fix bedrock None context_mgmt; stream per-instance queue; sync polyfill; trailing-chunk passthrough
- Apply context_management on sync path; clear held stop_reason chunk in async iterator
- Fix post-compaction question selection, system type, sync stream merge
- Skip tool_result-only user turns; bedrock: elif for context_management
- Add streaming iterator compaction test suite

Co-authored-by: Cursor <cursoragent@cursor.com>

* revert(html): restore flat *.html naming in _experimental/out

Reverses the accidental rename from *.html → */index.html introduced in
15ea941. All 35 files moved back to their original flat paths so the
directory structure matches litellm_internal_staging.

Co-authored-by: Cursor <cursoragent@cursor.com>

* revert(config): restore proxy_server_config.yaml to litellm_internal_staging

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix: skip client compaction pre-processing when compact_20260112 polyfill will run

The _prepare_context_managed_request helper unconditionally applied
apply_client_compaction_block_history before invoking the polyfill. When
the request also configured a compact_20260112 spec, that pre-processing
consumed the client-sent compaction block and collapsed the message history
to just the latest user question, starving the polyfill of conversation
context. The polyfill's own Phase A (_slice_around_compaction_block)
already handles client compaction blocks correctly and inspects the full
post-compaction tail for the token-threshold check, so the pre-processing
is both redundant and destructive in this case.

Now the pre-processing only runs when no compact_20260112 polyfill spec
will execute (no spec, drop_params on, or only non-compact edits like
clear_tool_uses_20250919).

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(anthropic): plug compaction-block leak + iteration-usage gaps in streaming adapter

- handler: when polyfill_will_run skipped client-history pre-processing
  and the polyfill ultimately returned None (best-effort swallow on
  unexpected error), apply the slice-only fallback before returning so
  Anthropic-specific 'compaction' content blocks don't leak to non-
  Anthropic backends that would reject them.
- streaming_iterator: precompute will_merge_into_held so we don't pass
  applied_edits into the translator when the resulting processed_chunk
  will be discarded by the held stop-reason merge path.
- streaming_iterator: augment processed_chunk with iterations usage in
  the holding_chunk branch (sync and async) for parity with the other
  emission branches; ensures usage.iterations is attached on the rare
  message_delta-reaches-holding_chunk path.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(anthropic): correct streaming usage iteration + translate tools for token counting

- streaming_iterator: skip the trailing "message" iteration entry in the
  final message_delta when the held stop_reason chunk carries placeholder
  zero usage (no separate usage chunk arrived). Reporting zero tokens was
  misleading and inconsistent with the non-streaming path which always
  has real usage data.
- streaming_iterator: drop two redundant type checks inside branches
  that are already guarded by an outer message_delta type check.
- compact._count_effective_tokens: translate Anthropic-shaped tools
  (input_schema) to OpenAI shape before passing to litellm.token_counter
  so threshold checks aren't skewed by tokenizer paths that expect the
  OpenAI tool wrapper.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* Fix lint

* fix(anthropic): plug content drop, compaction SSE shape, and compaction leak

- Sync streaming __next__ no longer drops a buffered holding_chunk when
  the usage-merge path has already fired. Restoring the prior unconditional
  flush behavior preserves provider-emitted content (the SSE-ordering nit
  of a trailing content delta is preferable to silent content loss).
- compaction content_block_start now carries the full block shape
  ({"type": "compaction", "content": ""}) to match the text-block
  pattern and Anthropic's native streaming shape, so clients that key off
  content_block_start see the field.
- apply_compact_20260112 now slices around / strips compaction blocks
  before the opt-in gate check. Previously, when summary_model was not
  configured the editor returned the raw messages, leaking Anthropic-only
  compaction content blocks to non-Anthropic providers that reject them.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(anthropic): resolve mypy types in context management polyfill

Use AppliedEdit and CompactionBlock consistently in the dispatcher and streaming adapter.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(anthropic): flush held content chunk in async streaming path

Mirror the sync __next__ behavior: always flush a buffered
holding_chunk after the stream ends, even when usage was already
merged + emitted. Previously the async __anext__ kept the flush
inside the 'if not self.queued_usage_chunk:' guard, silently
dropping the last content delta on the proxy's primary path.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(anthropic adapter): correct sync streaming, surface polyfill failures, decouple sync path from proxy router

- translate_completion_output_params_streaming: add is_async flag so the
  sync handler returns Iterator[bytes] instead of an unusable
  AsyncIterator. Async callers keep the existing behavior via the
  default is_async=True.
- _run_polyfill_if_enabled: when the polyfill crashes and the spec
  requested non-compact edits (e.g. clear_tool_uses_20250919), raise an
  AnthropicContextManagementError instead of silently returning None so
  those edits are not dropped without an error surface. The
  compaction-block-slicing safety net remains for compact-only specs.
- anthropic_messages_handler (sync): stop auto-attaching the proxy
  llm_router. run_async_function bridges to a new thread's event loop;
  reusing the proxy's loop-bound httpx clients there causes
  'Event loop is closed' errors. The summary editor falls back to
  litellm.acompletion when llm_router is None.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix: address bug detection findings in token counter and streaming iterator

- token_counter: guard against non-dict 'function' field in tool dicts
  and skip tools missing a name to avoid emitting 'type None = ...' which
  would produce inaccurate token counts.
- streaming_iterator: change sync __next__ generic-error path to raise
  StopIteration (was StopAsyncIteration), so sync iteration cleanly stops.
- streaming_iterator: centralize context_management attachment so the
  held-stop_reason direct-flush path defensively re-attaches applied_edits
  to match the merge path's guarantee.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* Fix lint

* fix: correct COMPACT_MIN_TRIGGER_TOKENS to 50_000

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* Fix lint

* Fix lint

* Fix lint

* fix(compact): reduce to last user question when summary_model not configured but prior compaction block exists

Aligns the summary_model_not_configured path with the under-threshold and
client-compaction-block paths, which both reduce post-compaction messages
to just the latest user question so the downstream provider doesn't get
the summary on system prefix AND the full post-compaction history.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(compact): forward caller system prompt to summary model call

The default summarization instructions reference "the initial task above"
and "the raw history above", but the system prompt that holds that task
was not being forwarded to the summary model. The summary call now
prepends an OpenAI-shaped system message translated from the original
Anthropic-shaped system (str or content-block list) so the summarizer
has the agent role and initial task in scope.

* fix(compact_20260112): set default max_tokens and merge prompt when last turn is user

- Set COMPACT_SUMMARY_MAX_TOKENS default for the summary call so providers
  like Anthropic (which require max_tokens) don't silently fail and degrade
  to summary_call_failed.
- When the trailing translated message is already a user turn, merge the
  summarization prompt into it instead of appending a second user turn.
  Avoids consecutive role=user messages that strict providers reject.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(anthropic adapter): move current_content_block_start to __init__

Move the default TextBlock dict from a class-level attribute to __init__ so
concurrent stream instances don't share the same mutable dict. The class-level
default could be mutated in-place via tool_block['name'] = original_name in
_should_start_new_content_block, leaking state across streams. This mirrors
the existing fix already applied to chunk_queue.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(compact_20260112): surface error states + strip tool_result blocks in last user question

applied_edits_for_response() now includes compact_20260112 edits that
carry an error field (summary_model_not_configured, summary_call_failed,
summary_extraction_failed) so clients and operators can see why
compaction was requested but not applied.

_select_last_user_question() now strips tool_result blocks from mixed
[tool_result, text] turns rather than passing them through as-is. After
compaction the paired tool_use assistant turn no longer exists, so
forwarding tool_result blocks translates to orphaned role=tool messages
on non-Anthropic providers and produces a 400.

* fix(compact_20260112): carry prior compaction summary into Phase C summary call

When a request already contains a compaction block, Phase A slices
`effective_messages` to the turns since that block. Previously Phase C
passed the original `system` to the summary model, so multi-round
compaction silently dropped accumulated history each time the polyfill
fired. Pass `augmented_system` (original system + prior summary
prefix) so the summary model can produce a comprehensive summary that
incorporates both the prior round's context and the current slice.
`summarized_system` for the downstream call stays built from the
original `system` + new `summary_text`.

* refactor: delegate handler spec normalization to dispatcher

_normalize_spec_edits in adapters/handler.py duplicated the spec-shape
normalization already implemented by _normalize_spec in
context_management/dispatcher.py. The two could drift: a change in one
(e.g. supporting a new spec shape) without the other would cause the
handler's polyfill_will_run prediction to disagree with the
dispatcher's actual behavior, breaking the client-history pre-processing
skip.

Have the handler delegate to the dispatcher's _normalize_spec while
keeping handler-specific concerns (drop_params short-circuit, swallow
mapping exceptions) at the wrapper level.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(compact_20260112): surface warning-only applied edits in response

`applied_edits_for_response()` previously hid `compact_20260112` edits when
they had only warnings (no compaction block, no error). This dropped
diagnostically important warnings such as
`unsupported_trigger_type_X_using_input_tokens` and
`pause_after_compaction_ignored` whenever the conversation was under the
trigger threshold. Operators now see these warnings in the response.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix: address two low-severity context_management edge cases

- streaming_iterator: keep `sent_content_block_finish` in sync with the
  compaction block's emitted start/delta/stop lifecycle and reset it when
  the next text block's start is queued.
- bedrock _map_context_management_param: match dispatcher `_normalize_spec`
  behavior — only run the OpenAI→Anthropic mapper on list inputs; pass
  dict inputs through unchanged so already-Anthropic-format values aren't
  silently dropped.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(compact_20260112): use beta-header constant; require type discriminator; skip sync bridge when idle

- bedrock: replace hardcoded "compact-2026-01-12" beta string with
  ANTHROPIC_BETA_HEADER_VALUES.COMPACT_2026_01_12.value in both
  Converse (_filter_context_management_for_bedrock_converse) and Invoke
  (anthropic_claude3) compact-edit handlers.
- types: mark the "type" discriminator as Required[...] on the new
  CompactionBlock and UsageIteration TypedDicts so the discriminator
  is not silently optional under total=False.
- adapters/handler: short-circuit the sync /v1/messages adapter path
  before spawning the run_async_function worker-thread event loop when
  the request has no context_management spec and no client-sent
  compaction block in the message history.

Test plan:
- uv run pytest tests/test_litellm/llms/anthropic/experimental_pass_through/     tests/test_litellm/llms/bedrock/test_converse_context_management.py -q
  (370 + 10 = 380 passed)
- uv run pytest tests/test_litellm/llms/azure_ai/claude/test_azure_anthropic_transformation.py     tests/test_litellm/llms/vertex_ai/vertex_ai_partner_models/anthropic/test_vertex_ai_partner_models_anthropic_transformation.py     -k compact (3 passed)

* fix(compact_20260112): include system prompt tokens in threshold check

The threshold check in Phase B previously counted only message tokens and
the compaction-block content, omitting the system prompt entirely. When
the system carried a prior compaction summary (via _augment_system_with_summary)
or was otherwise large, the threshold could fire later than intended,
allowing the conversation to exceed the model's context window before
compaction activated.

_count_effective_tokens now also counts the (augmented) system prompt
text. The caller passes compaction_block=None when augmented_system
already includes the prior summary, to avoid double-counting.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* Fix SSE ordering and compaction state machine bugs in AnthropicStreamWrapper

- Suppress holding_chunk flush after final message_delta has been emitted
  (queued_usage_chunk == True) so a trailing content_block_delta cannot
  follow message_delta, which strict Anthropic SDK clients may reject.
  When usage has not yet been merged, flush the holding_chunk *before*
  the held stop_reason chunk so SSE ordering remains correct.

- Replace _queue_compaction_block_events with _next_compaction_event,
  emitting the compaction start/delta/stop events one at a time. The
  state machine flags (sent_content_block_finish) and content block
  index now advance atomically with the terminal stop event actually
  being returned to the caller, eliminating the transient inconsistent
  state where flags say the block is finished while its stop event is
  still buffered.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(compact_20260112): enforce parent key/team allowlist on summary model

The compact_20260112 polyfill summary subrequest used llm_router.acompletion
directly, bypassing the proxy auth checks that gate model access for the
parent key/team. A caller whose key/team was not authorized for the
configured context_management_summary_model could still cause the proxy to
invoke that model and return its output as a compaction block.

Pull the parent's UserAPIKeyAuth out of litellm_metadata in the handler,
thread it through the dispatcher into apply_compact_20260112, and gate the
summary call on _can_object_call_model for both key-level and team-level
allowlists. Failures land as applied_edits[0].error =
summary_model_access_denied without raising. SDK callers (no UserAPIKeyAuth)
remain unaffected.

* fix(compact_20260112): distinguish access-denied from transient errors; greedy summary regex

- _check_summary_model_access now catches ProxyException explicitly for access
  denials and logs unexpected exceptions separately. Both still fail closed,
  but operators can now tell a denied key/team apart from a router internal
  raising during the check.
- _SUMMARY_TAG_RE switches from non-greedy to greedy so a stray </summary>
  inside the model's summary content no longer silently truncates the
  captured text.

* fix(compact_20260112): type object_type as Literal for mypy

* fix(compact_20260112): attribute summary subcall spend to parent key/team

The compact_20260112 polyfill summary subrequest propagated metadata via
the Anthropic-shape `metadata` parameter, which only carries `user_id`.
The proxy auth fields used for spend attribution (`user_api_key`,
`user_api_key_team_id`, `litellm_call_id`, ...) live in
`data["litellm_metadata"]`. As a result, summary subcalls landed on the
router with an empty propagated metadata and the resulting tokens were
not attributed to the caller's key/team budget.

Rename the polyfill chain's spend-propagation parameter to
`litellm_metadata` and pull it from `kwargs["litellm_metadata"]` in
both the async and sync handlers, so the post-call hooks see the parent
key/team and bill the summary tokens accordingly. Add an
`_extract_proxy_litellm_metadata` helper and refactor
`_extract_user_api_key_auth` to use it.

* chore(anthropic adapters): remove unused _extract_user_api_key_auth helper

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* chore(compact_20260112): non-greedy summary regex; use COMPACT_EDIT_TYPE in bedrock filter

- Make _SUMMARY_TAG_RE non-greedy so a response with multiple <summary>
  blocks captures only the first complete block.
- Replace the hardcoded 'compact_20260112' literal in
  _filter_context_management_for_bedrock_converse with the shared
  COMPACT_EDIT_TYPE constant.

* fix: bug fixes from PR review

- streaming_iterator: don't set sent_content_block_finish during compaction
  block lifecycle; that flag tracks the regular text/tool_use/thinking block
  state machine, conflating the two leaks bad state to introspection paths.
- compact._call_summary_model: send propagated proxy auth/spend-attribution
  fields as 'litellm_metadata' instead of 'metadata' so the router's post-call
  hooks attribute summary tokens to the caller's key/team budget.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(anthropic-streaming): insert content_block_stop between held delta and final message_delta

When the stream exhausts with both `holding_chunk` (a content_block_delta)
and `holding_stop_reason_chunk` (a message_delta) buffered, the after-loop
cleanup previously emitted them back-to-back, producing the invalid
Anthropic SSE sequence `content_block_delta -> message_delta`. Insert a
`content_block_stop` between them in both the sync `__next__` and async
`__anext__` paths so the emitted ordering remains
`content_block_delta -> content_block_stop -> message_delta`.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(compact_20260112): propagate allowed_model_region to summary subrequest

The router enforces region restrictions by reading allowed_model_region
from top-level request kwargs (Router._common_checks_available_deployment),
but the compact_20260112 summary subrequest only forwarded litellm_metadata.
A region-restricted caller could trigger compaction and have their conversation
summarized by a deployment outside the permitted region.

Extract allowed_model_region from user_api_key_auth and pass it through
_call_summary_model as a top-level kwarg so the router applies the same
region constraints the parent request would.

* fix(anthropic adapter): emit content_block_stop before held message_delta in drain paths

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* feat(context_management): configurable summary max_tokens; surface ignored knobs

- compact_20260112: read summary max_tokens from general_settings
  (context_management_summary_max_tokens) so operators can fit the
  chosen summary model's output budget; falls back to the compiled
  default for missing or invalid values.

- clear_tool_uses_20250919: log unsupported knobs at warning level
  (was debug, which silently dropped misconfiguration) and surface
  them as warnings on the AppliedEdit so clients see what was ignored.

* fix(compact_20260112): bound _call_summary_model with timeout

A slow or unresponsive summary model previously hung the parent
/v1/messages request with no escape hatch. Pass a 60s timeout on the
litellm.acompletion / llm_router.acompletion subrequest; on timeout the
existing summary_call_failed path forwards the request without
compaction rather than blocking indefinitely.

* fix(compact_20260112): preserve post-compaction tail on slice-only path

When a prior compaction block is present and the request is under threshold,
the polyfill was reducing downstream messages to just the latest user
question. The prior summary only covers turns before the compaction block,
so dropping the post-compaction tail silently lost recent context — a
multi-turn conversation that stayed below the threshold would arrive at the
model with no memory of any turn after the prior compaction.

Forward the already-stripped post-compaction tail unchanged on both the
under-threshold path and apply_client_compaction_block_history. Fall
back to _select_last_user_question only when the strip leaves nothing
for the downstream call to answer.

* fix(compact_20260112): enforce user/project/team-member model scopes on summary subrequest

The local gate previously only checked the parent key's and team's
allowed-model lists. A caller restricted by a personal user, project,
or per-team-member allowed_models scope could still trigger the
configured summary model and receive its <summary> output as a
compaction block, because llm_router.acompletion bypasses the proxy
common_checks path. Extend _check_summary_model_access to also load
the user_object, project_object, and team_membership and run the
matching allowlist check at each scope before invoking the summary
model.

* fix(compact_20260112): enforce summary model per-model budget and propagate budget metadata

* fix(compact_20260112): forward post-compaction tail when summary model unconfigured

* fix(anthropic endpoints): run failure hook on 500-level context management errors

* fix(compact_20260112): enforce summary model rate limit before summary call

* fix(compact_20260112): propagate end-user/project budget scope to summary call

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…erriAI#28830)

Allow self-hosted installs to override the default LiteLLM sender address
via RESEND_FROM_EMAIL, matching SendGrid's SENDGRID_SENDER_EMAIL pattern.

Co-authored-by: Cursor <cursoragent@cursor.com>
…erriAI#28728)" (BerriAI#29326)

This reverts the Bedrock CI account migration (BerriAI#28728). The original account
(888602223428) was put under an AWS security restriction after a leaked key
and has since been reactivated, while the replacement account (941277531214)
lacks access to several models the suites exercise (legacy Bedrock Claude 3
models, Cohere, Nova Canvas image gen, Bedrock batch inference, and flagship
Opus). Pointing CI back at the reactivated account restores that coverage.

This is the exact inverse of BerriAI#28728: all hardcoded 941277531214 references go
back to 888602223428 (provisioned/imported-model ARNs, AgentCore runtime ARNs
and their suffixes, batch execution role ARN, and the example proxy config),
the S3 buckets revert to litellm-proxy and load-testing-oct, the guardrail IDs
revert to wf0hkdb5x07f and ff6ujrregl1q, the SageMaker endpoint and Knowledge
Base revert to their original ids, and the live-call tests go back to the
legacy model strings. The grid_spec fail_reason workaround for the unentitled
Opus cells is dropped while keeping the unrelated bedrock_effort_ceiling field
added after the migration.

The CircleCI AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY env vars still point at
941277531214 and must be set to the reactivated account's fresh credentials
separately via the CircleCI API; AWS_REGION_NAME stays us-west-2.
…erriAI#29249)

* fix(mcp): preserve source_url in GET /v1/mcp/server list responses

The list endpoint builds responses from the in-memory registry, but
source_url was dropped during the DB-to-registry roundtrip even though
GET /v1/mcp/server/{id} returned it correctly from the database.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(tests/mcp): set source_url on MagicMock table records

MagicMock auto-creates source_url as a mock object, which fails MCPServer
Pydantic validation after source_url was wired through build_mcp_server_from_table.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
…es (BerriAI#29253)

* fix(mcp): preserve omitted fields on PUT /v1/mcp/server partial updates

Use model_dump(exclude_unset=True) for updates so schema defaults (transport=sse,
allow_all_keys=false, etc.) are not written when callers omit them. Serialize
JSON fields from the filtered dict and only force is_byok on create.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(mcp): allow explicit alias=None on MCP server partial updates

Snapshot caller-provided fields before normalization so omitted alias is not written while an intentional alias=None still clears the stored value.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
…Opus 4.7 self-heal) (BerriAI#29344)

* test(logging): align DB metrics event_metadata assertions with safe redaction

PR BerriAI#28909 hardened log_db_metrics to emit a minimal, non-sensitive
event_metadata (only table_name when present, otherwise None) instead of
dumping function_name, function_kwargs, and function_args onto the span. The
test in test_log_db_redis_services was not updated and still asserted
"function_name" in event_metadata, which raised TypeError (argument of type
'NoneType' is not iterable) and turned the logging_testing CI job red on
litellm_internal_staging.

Update test_log_db_metrics_success to assert event_metadata is None when no
table_name is passed, and add test_log_db_metrics_event_metadata_is_safe as a
regression guard verifying that only the table name surfaces and that sensitive
kwargs (tokens, prisma client) are never dumped.

* test(bedrock): self-heal opus-4-7 grid cells when unentitled on CI

The bedrock-claude-opus-4-7 converse cells are unentitled on the Bedrock CI
account, so they were marked xfail. xfail keeps reporting them as expected
failures even after access is granted, so the wire translation never gets
verified again. Now the cell makes the call and skips only when Bedrock
replies "is not available for this account"; the moment the model is
entitled the same cells run their full assertions with no edit.

A focused unit test pins the tolerance predicate so any other failure still
surfaces loudly and the available path still runs the assertions.
…rriAI#29343)

* refactor(proxy/auth): normalize Bearer prefix in safe-hash helper

UserAPIKeyAuth._safe_hash_litellm_api_key now strips a leading
"Bearer "/"bearer " prefix before its existing sk-/JWT classification, so
the helper produces the same hashed output regardless of whether the
caller stripped the Authorization header prefix or passed the header
value through unchanged.

* refactor(proxy/auth): make Bearer-prefix strip case-insensitive

Per RFC 7235 the HTTP authorization scheme token is case-insensitive.
Replace the two-prefix loop with a single case-insensitive check so the
helper normalizes "Bearer ", "bearer ", "BEARER ", and any mixed-case
variant before classifying the remainder as sk- or JWT. The contract
test gains coverage of "BEARER " and "BeArEr ".

* test(mcp): align auth-handler test expectations with safe-hash helper

The two MCP auth tests asserted that UserAPIKeyAuth(api_key="Bearer ...")
retained the raw header bytes on the api_key field. _safe_hash_litellm_api_key
now normalizes that input — stripping the Bearer prefix and hashing the
resulting sk- key — so the expectations move to the normalized form:
the bare token in the parametrize case, and hash_token("sk-...") in the
backward-compat assertion. This matches what the real auth flow produces
(the builder strips Bearer and the DB stores the hashed token), so the
mocks now line up with production rather than with the un-normalized
validator output.
…utes (BerriAI#29327)

* test(logging): align DB metrics event_metadata assertions with safe redaction

PR BerriAI#28909 hardened log_db_metrics to emit a minimal, non-sensitive
event_metadata (only table_name when present, otherwise None) instead of
dumping function_name, function_kwargs, and function_args onto the span. The
test in test_log_db_redis_services was not updated and still asserted
"function_name" in event_metadata, which raised TypeError (argument of type
'NoneType' is not iterable) and turned the logging_testing CI job red on
litellm_internal_staging.

Update test_log_db_metrics_success to assert event_metadata is None when no
table_name is passed, and add test_log_db_metrics_event_metadata_is_safe as a
regression guard verifying that only the table name surfaces and that sensitive
kwargs (tokens, prisma client) are never dumped.

* test(bedrock): self-heal opus-4-7 grid cells when unentitled on CI

The bedrock-claude-opus-4-7 converse cells are unentitled on the Bedrock CI
account, so they were marked xfail. xfail keeps reporting them as expected
failures even after access is granted, so the wire translation never gets
verified again. Now the cell makes the call and skips only when Bedrock
replies "is not available for this account"; the moment the model is
entitled the same cells run their full assertions with no edit.

A focused unit test pins the tolerance predicate so any other failure still
surfaces loudly and the available path still runs the assertions.

* test(reasoning-effort-grid): add claude-opus-4-8 across provider routes

Adds claude-opus-4-8 to the anthropic, azure, vertex and bedrock-converse
routes (275 cells total) so the reasoning-effort wire translation is
covered for the new model. The bedrock opus-4-8 and opus-4-7 cells reuse
the self-heal path: they run the call and skip only on Bedrock's "is not
available for this account" reply, then assert in full once the model is
entitled. The azure and vertex opus-4-8 cells stay xfail until a Foundry
deployment exists and Vertex availability is confirmed. The shared
xhigh+max capability set is renamed to _CAPS_XHIGH_MAX now that more than
one model uses it.
…erriAI#28418)

* fix(guardrails): return HTTP 400 for litellm content filter blocks

Align litellm_content_filter hard rejects with the standard guardrail block status code so clients receive 400 instead of 403.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(guardrails): return HTTP 400 for custom code guardrail blocks

Pre-call custom code guardrail blocks now raise HTTPException(400) instead of using the passthrough ModifyResponseException path that returned a synthetic 200 response.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(guardrails): preserve custom code passthrough blocks

Keep standalone custom code guardrail blocks on the passthrough contract while covering policy pipeline block handling for passthrough-style guardrail interventions.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
…BerriAI#29202)

* fix(proxy): restrict vector store index create/delete to proxy admins

Prevent non-admin API keys from registering indexes via POST /v1/indexes or deleting Azure AI Search indexes through managed pass-through routes.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(proxy): tighten vector store index lifecycle checks

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
…erriAI#29160)

* feat(pass_through): extend passthrough_managed_object_ids to Azure

Adds managed ID minting/resolution for Azure passthrough endpoints
(/azure/...) alongside the existing OpenAI passthrough support.

Key changes:
- pass_through_endpoints.py: detect azure/azure_ai custom_llm_provider
  (string or enum) to set _is_managed_id_provider and _managed_id_provider;
  both INPUT and OUTPUT rewrite blocks now fire for Azure.
- llm_passthrough_endpoints.py: forward custom_llm_provider into
  create_pass_through_route so it reaches pass_through_request (was None).
- managed_id_rewriter.py: extend _PASSTHROUGH_PREFIX_RE and _canonical_path
  to strip /azure/openai prefix and add /v1/ for Azure paths that omit it;
  add ("azure", method, path) entries to BUILTIN_OUTPUT_ID_FIELD_MAP for
  files and batches endpoints.
- managed_id_codec.py / types/utils.py: supporting codec and enum constant.
- proxy_server.py: register llm_passthrough_router before batches_router to
  prevent route collision for /openai_passthrough/* paths.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(pass_through): remove unused imports for ruff F401

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(pass_through): satisfy mypy for optional parsed_body

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(pass_through): compute query params string after managed-ID rewrite

Move requested_query_params_str computation to after the managed-ID
input rewrite block so logging_url reflects the rewritten raw-provider
query params actually sent upstream, instead of the original
managed IDs.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* Add support for managed ids for passthrough responses api

* Add support for list batches and list files

* style: run Black on passthrough managed ID files

Fix CI formatting for managed_id_rewriter.py and pass_through_endpoints.py.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(passthrough): parse json file_object and implement before-cursor pagination

- Parse row.file_object via json.loads when Prisma returns it as a string;
  mirrors openai_files_endpoints/common_utils.py so list responses keep all
  stored detail fields (status, timestamps, etc.).
- Implement the previously-parsed-but-unused 'before' cursor for list
  pagination by flipping fetch order to ascending with a 'gt' bound on
  created_at, then reversing rows so the response stays newest-first.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* Remove logger

* refactor: split list_passthrough_ids_from_db to fix PLR0915

Extract pagination, fetch, and serialization helpers so the main list
function stays under the statement limit without changing behavior.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: scope passthrough managed ID dedup and list by provider

Validate embedded provider before reusing deduped file/object rows so
OpenAI and Azure cannot share the same managed ID for an identical raw ID.
Filter list responses to rows whose managed IDs decode to the current
provider, with over-fetch scanning when needed.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(managed_id_rewriter): cap pagination trim at effective limit

When raw_limit > 100, fetch_limit is capped at 101 (one extra row to
detect has_more), but trimming with rows[:raw_limit] failed to drop
the sentinel row. Use the capped effective limit instead.

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix: cross-provider object collision and fail-closed list error handling

Greptile P1: move provider check before access check in _mint_or_reuse_object
so a cross-provider raw ID collision (OpenAI and Azure share the same batch_
ID) falls through to mint a new provider-scoped row instead of raising 404.

Veria-ai medium: _fetch_provider_scoped_list_rows now always returns
(page, has_more) — DB errors break out of the scan loop and return matched
rows so far. list_passthrough_ids_from_db never returns None for a recognised
list route, so the caller can never fall through to the upstream provider.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: namespace passthrough model_object_id by provider to prevent unique violation

Store model_object_id as 'passthrough:{provider}:{raw_id}' instead of the
bare raw ID so OpenAI and Azure can each own a row for the same raw batch ID
without hitting the @unique constraint. Dedup lookup uses the same namespaced
key so it is implicitly provider-scoped and the _managed_id_matches_provider
check is no longer needed on the object path.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: gate list interception on managed_files hook like input/output rewrites

Without the hook no managed IDs are minted so the DB is empty. Intercepting
GET /v1/files without the hook returned an empty list and hid the caller's
real upstream files/batches. Matches the guard used by the input and output
rewrite blocks.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: set has_more=True when scan cap is hit with a full final DB batch

When max_scans (20) is exhausted and the last DB page was full-sized,
there are almost certainly more rows beyond the scan window.  Track
last_batch_full across iterations so the scan_cap_hit condition sets
has_more=True in that case, preventing silent pagination truncation in
high-mixed-provider pools.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(managed_id_rewriter): scope pagination cursor lookup to caller-owned rows

Prevent a cross-tenant timing oracle by constraining the after/before
cursor row lookup to the caller's owner_filter, and cover the real Azure
responses path form (no /v1/) in tests.

* fix(managed_id_rewriter): align passthrough list metadata with direct GET

Persist upstream file metadata when minting a managed file ID and rewrite
nested batch file IDs before snapshotting the object, so DB-served file/batch
list responses return the same fields and managed IDs as a direct endpoint
GET.

* fix(managed_id_rewriter): degrade to raw id on cross-owner object collision

The OUTPUT (mint) path of _mint_or_reuse_object raised HTTPException(404)
when a dedup hit on the namespaced model_object_id belonged to a different
owner, converting a successful upstream batch/response creation into a 404
for the caller. Two upstream accounts under one provider name can issue the
same raw id, so this is reachable in multi-tenant deployments.

Return the caller's raw id unmanaged instead: the upstream create already
succeeded, a new managed row can't be minted (model_object_id is @unique),
and reusing the other owner's managed id would later fail the access check.

* fix(managed_id_rewriter): scope list cursor by provider and cap body-rewrite recursion depth

* perf(managed_id_rewriter): push batch list provider scope to DB and anchor canonical-path prefix

Object (batch) list rows store model_object_id as passthrough:{provider}:{raw},
so the provider filter is now applied at the indexed DB column, collapsing the
application-layer multi-scan to a single query for that table. File rows keep
the decode-based scan since they have no provider column.

Anchor the canonical-path prefix regex at a path boundary so routes such as
/openai_realtime/... are no longer mis-stripped.

* fix(managed_id_rewriter): refresh stored batch snapshot on reuse

The dedup-reuse path in _mint_or_reuse_object returned the existing managed id
without updating the stored file_object, so DB-served list responses kept the
creation-time snapshot and showed null output_file_id/error_file_id even after
the batch completed. Refresh the snapshot when an owned row is reused so the
list reflects the batch's latest state.

* fix(managed_id_rewriter): deny cross-owner object access on retrieve/cancel/delete

Returning the raw id when can_access_resource fails only made sense for create
responses, where the caller's own upstream create succeeded under a raw id that a
different owner already holds. On retrieve/cancel/delete the caller reaches that
branch only by supplying another tenant's raw id (which bypasses the managed-id
input gate), so echoing the upstream object back leaked it cross-tenant. Restrict
the raw fallback to create routes and return 404 otherwise.

* fix(managed_id_rewriter): deny cross-owner file access on retrieve/delete

_mint_or_reuse_file scoped the raw file dedup lookup to the current caller, so a
raw file-... id belonging to another tenant was never found and the OUTPUT path
minted a fresh managed id for that same upstream file under the caller. A raw id
only reaches this path by skipping the managed-id input gate (raw provider ids
are opt-out), so a different-owner row means the caller is touching someone
else's file. Look up flat_model_file_ids globally and run can_access_resource;
deny with 404 on retrieve/delete and leave the raw id unmanaged on create, which
mirrors the cross-owner handling already in _mint_or_reuse_object.

* fix(managed_id_rewriter): deterministic provider-scoped file dedup

Replace the unscoped find_first in _mint_or_reuse_file with a find_many
ordered by created_at and an application-layer provider filter. The file
table has no provider column, so a raw file id shared across OpenAI and
Azure could map to one row per provider; find_first then picked a row
non-deterministically and, on a provider mismatch, minted a fresh managed
row on every call, accumulating duplicates. Selecting the oldest matching
same-provider row the caller can access keeps reuse stable and prevents
duplicate rows while preserving the cross-tenant deny/leave-raw behaviour.

* refactor(pass_through): scope passthrough managed IDs on the explicit provider

Move the openai/azure detection out of pass_through_request into
resolve_passthrough_managed_id_provider in llms/base_llm/managed_resources,
and key managed-ID rewriting on the forwarded custom_llm_provider rather than
the upstream URL's EndpointType. The helper documents why azure and azure_ai
collapse to one "azure" scope (they share the same Azure OpenAI files/batches
surface, so an ID minted on one must resolve on the other) and returns None for
any other provider so a third-party OpenAI-compatible endpoint never triggers
managed-ID minting.

Add TestManagedIdProviderScope covering the azure_ai -> azure collapse and the
non-openai/azure exclusion.

* test(log_db_metrics): assert sanitized event_metadata contract

test_log_db_metrics_success still asserted the legacy event_metadata
shape (function_name/function_kwargs/function_args), which BerriAI#28909
intentionally removed so that live Prisma clients, OTel spans, and
secrets never land on a service-log span. The decorator now emits only
a sanitized payload: None when no table_name is present, and
{"table_name": ...} when it is. Update the test to verify both branches
of that contract.

* fix(managed_id_rewriter): page provider-scoped file list by offset

The file list scan advanced its cursor with a strict created_at boundary.
When several rows shared a created_at timestamp and a non-matching provider
row sat on the page boundary, the next query skipped the remaining rows at
that timestamp, dropping matching files from the response. Page by a stable
offset over a total order (created_at plus the unique id column) so tied
rows are never skipped or repeated.

* fix(managed_id_rewriter): push file-list provider scope to the DB

The file-list helper had no provider column to query, so it scanned the
table and filtered by decoding each managed ID in the application layer,
capped at 20 pages. For an admin with a large mixed-provider file pool
that cap could truncate a page.

Mint now writes a _passthrough_provider:{provider} marker into
flat_model_file_ids, giving the file table the same DB-queryable provider
scope object rows already get from the namespaced model_object_id. The
list helper pushes the scope into the query so a single round-trip serves
the page. The scan loop, the offset paging, and the cap are gone, so pages
can no longer truncate, leak the other provider, or skip rows that share a
created_at timestamp.

* fix(managed_id_rewriter): deny raw provider IDs that map to another tenant's managed resource

Clients only ever receive managed IDs on passthrough, so a raw file/batch/response ID for another tenant's managed object can only be recovered by decoding that tenant's managed ID. Raw IDs were forwarded upstream untouched (deliberate opt-out), which on a retrieve/cancel/delete executed upstream before the response-side ownership check ran, leaking a cross-tenant action.

Guard raw provider IDs on the input path: when a raw file-/batch_/resp_ ID resolves to a managed row the caller cannot access, return 404 before forwarding. Genuinely unmanaged raw IDs (no DB row) and IDs the caller owns are left untouched, preserving the opt-out.

* test(managed_id_rewriter): cover azure_ai and pre-versioned azure passthrough paths

* fix(managed_id_rewriter): fall back to raw id when persistence fails

A DB persistence failure after a successful upstream create left the
client holding a minted managed ID with no backing row, so every later
resolve returned 404 and the resource was permanently unreachable. Mint
the managed ID only when the row is stored; on persistence failure return
the raw provider id, matching the no-persistence-available fallback, so
the freshly-created resource stays reachable.

* fix(managed_id_rewriter): log only rewritten query param keys

* fix(managed_id_rewriter): use compound (created_at, id) list cursor boundary

A timestamp-only lt/gt cursor boundary skips list rows that share the
cursor row's created_at across a page boundary, silently dropping them.
Compare the unique id (the secondary sort key) alongside created_at so
the page walk stays complete when timestamps tie.

* fix(managed_id_rewriter): converge concurrent object creates on one managed id

* fix(managed_id_rewriter): bound raw-id guard DB lookups per request

The INPUT guard fired one DB lookup for every file-/batch_/resp_ prefixed
string in the path, query, and body. The file-id guard is an unindexed
array-containment scan over LiteLLM_ManagedFileTable, so an authenticated
caller could amplify a single passthrough request into thousands of
full-table scans by packing a body with id-shaped strings.

De-dupe raw ids within a request and cap the distinct guard lookups,
failing closed with 400 instead of skipping the guard. Legitimate callers
hold managed ids (resolved via an indexed unified_*_id lookup, not the
guard), so the cap only trips under abuse.

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
BerriAI#29256)

* fix(proxy): enforce allowed_passthrough_routes for auth=true pass-through

Pass-through endpoints with auth=true were injected into openai_routes,
so teams with openai_routes access bypassed per-team allowed_passthrough_routes.
Gate auth-enforced pass-through at JWT, virtual-key, and non-admin route checks.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(proxy): clarify JWT passthrough denial

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(proxy): make pass-through auth checks method-aware

Prevent allowlist bypass when the same path is registered with different auth settings per HTTP method.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix passthrough route auth checks

* fix(proxy): reject unregistered pass-through HTTP methods

Enforce method-aware JWT checks and return 405 when stale FastAPI routes accept requests outside the current pass-through registry.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(proxy): remove duplicate request_method in JWT team lookup

Fixes SyntaxError on proxy startup caused by passing request_method twice to find_team_with_model_access.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Fix passthrough route auth enforcement

* fix(proxy): raise passthrough-specific 403 directly in virtual-key path

* fix(proxy): load team for RBAC role-claim JWT passthrough gating

* Revert "chore(tests): migrate Bedrock CI to AWS account 941277531214 (BerriAI#28728)" (BerriAI#29326)

This reverts the Bedrock CI account migration (BerriAI#28728). The original account
(888602223428) was put under an AWS security restriction after a leaked key
and has since been reactivated, while the replacement account (941277531214)
lacks access to several models the suites exercise (legacy Bedrock Claude 3
models, Cohere, Nova Canvas image gen, Bedrock batch inference, and flagship
Opus). Pointing CI back at the reactivated account restores that coverage.

This is the exact inverse of BerriAI#28728: all hardcoded 941277531214 references go
back to 888602223428 (provisioned/imported-model ARNs, AgentCore runtime ARNs
and their suffixes, batch execution role ARN, and the example proxy config),
the S3 buckets revert to litellm-proxy and load-testing-oct, the guardrail IDs
revert to wf0hkdb5x07f and ff6ujrregl1q, the SageMaker endpoint and Knowledge
Base revert to their original ids, and the live-call tests go back to the
legacy model strings. The grid_spec fail_reason workaround for the unentitled
Opus cells is dropped while keeping the unrelated bedrock_effort_ceiling field
added after the migration.

The CircleCI AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY env vars still point at
941277531214 and must be set to the reactivated account's fresh credentials
separately via the CircleCI API; AWS_REGION_NAME stays us-west-2.

(cherry picked from commit f11c12d)

* fix(proxy): scope pass-through 405 to registry routes; grant rerank passthrough in rpm tests

The auth=true pass-through 405 guard fired for mapped provider routes
(e.g. /assemblyai/*) that are not in the in-memory registry, since
get_registered_pass_through_route returns None for them while
is_registered_pass_through_route matches via mapped_pass_through_routes.
Only raise 405 when the path is registered but the request method is not
allowed, so mapped provider pass-throughs fall through to the default
target params as before.

The rpm-limit pass-through tests register /v1/rerank with auth=true but
gave their keys no allowed_passthrough_routes, so the new default-deny
returned 403 before the rate limiter ran (non-deterministically,
depending on registry insertion order). Grant the keys explicit
passthrough access so the tests exercise rate limiting under the new
auth model.

* fix(proxy): guard request method lookup against scopes without a method

Starlette's Request.method property reads scope["method"] and raises
KeyError when the scope omits it (e.g. minimally-constructed test
requests). getattr only swallows AttributeError, so the new
_get_request_method helper propagated the KeyError up through
user_api_key_auth and surfaced as a ProxyException. Catch KeyError
(and AttributeError) and fall back to None.

* test(passthrough): pin SERVER_ROOT_PATH in unregistered-method test

test_custom_proxy.py sets os.environ['SERVER_ROOT_PATH'] = '/my-custom-path'
at module import with no cleanup. When that module is collected into the same
xdist worker as this test, the leaked root path is prepended to registered
pass-through paths, so is_registered_pass_through_route misses '/test/path'
and the handler returns 404 instead of the expected 405 (order-dependent).
Pin SERVER_ROOT_PATH to '' so the test is deterministic.

* test(passthrough): restore regression coverage for non-auth-enforced pass-through via llm_api_routes

* fix(proxy): record auth flag in pass-through registry for allowlist enforcement

Auth-enforced pass-through detection inferred enforcement from the FastAPI
dependency stored at registration time. The management create and update
endpoints register routes with dependencies=None even though auth defaults to
true, so is_auth_enforced_pass_through_route treated those DB-created routes as
unenforced. A key allowed for llm_api_routes could then call a management-created
auth-enabled pass-through route without matching allowed_passthrough_routes.

Store the auth setting on each registry entry and read it directly when deciding
whether the allowlist applies, instead of deriving it from dependency metadata.

* fix(proxy): include bool in pass-through registry value type for auth flag

The auth flag stored in _registered_pass_through_routes is a bool, which
was not part of the registry value Union, so mypy rejected the dict literal.
Add bool to the Union and narrow route_methods to a list before the
membership check so the in-operator stays valid.

* fix(proxy): preserve stored auth flag on pass-through endpoint update

model_dump(exclude_none=True) re-included the auth=True default whenever
a partial update omitted auth, silently flipping an existing auth=false
pass-through to auth-enforced and 403ing every team/key without
allowed_passthrough_routes. Merge only explicitly set fields via
exclude_unset so omitted fields keep their stored value.

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
…gnment (BerriAI#29313)

* fix(mcp): make key.access_group_ids grants additive over team ceiling

A key whose unified access_group_ids grant a private MCP server was
having that grant intersected against its team's MCP ceiling, so a key
in a team scoped to other servers (or with no own scope) lost the
granted server entirely. Resolve access_group_ids once as ungated
additive grants and union them on top of the key/team ceiling instead
of folding them into the key scope that gets intersected.

* test(mcp): align key access-group tests with additive-grant model

The previous commit moved key.access_group_ids resolution out of the
intersected key ceiling (_get_allowed_mcp_servers_for_key) and into the
ungated additive grant path (_get_key_access_group_mcp_server_extras),
unioned on top of the team ceiling. Five tests from BerriAI#28890/BerriAI#29195 still
asserted the old gated / in-key-scope contract and failed:

- _get_allowed_mcp_servers_for_key now returns the object_permission
  ceiling only and never resolves access_group_ids; two tests now assert
  the group resolver is not called from that path (with and without an
  object_permission present).
- The extras path is ungated, so a group whose assigned_team_ids /
  assigned_key_ids exclude the caller still contributes its servers.
- The end-to-end test asserts the grant surfaces via the extras path
  rather than the base key path.
- Dropped test_key_access_group_ids_empty_returns_no_extras; the empty
  case is already covered by the extras family's no-groups test.

* feat(auth): gate member access-group assignment on keys behind opt-in

Non-admin team members could attach access_group_ids to keys they create
or update, letting them self-grant resources (MCP servers/models) the team
admin never intended. Add an opt-in KEY_ACCESS_GROUP_ASSIGNMENT team-member
permission (default-deny) enforced at /key/generate and /key/update; proxy
and team admins bypass. Surfaces automatically as a checkbox in the team
Member Permissions UI.

* fix(auth): gate access-group assignment on /key/regenerate too

RegenerateKeyRequest inherits access_group_ids and prepare_key_update_data
persists it, so a non-admin key owner could self-grant access groups by
regenerating. Apply the same opt-in member gate using the existing key's
team.

* test(auth): cover member access-group gate and additive MCP grants

Add unit tests for enforce_member_can_assign_access_groups (deny without
opt-in, allow with opt-in, and proxy-admin / team-admin / non-team-key
bypasses) and for _get_key_access_group_mcp_server_extras (no-auth and
no-resolved-servers return empty, resolved ids are expanded, errors
degrade to no grants).
…eroing counter (BerriAI#29358)

* fix(reset_budget): write only {spend, budget_reset_at} and stop pre-zeroing counter

ResetBudgetJob's batched update_data path shipped the full key/user/team
model on each reset. Prisma rejects object_permission_id and budget_limits
on the update input type, so any row carrying those fields detonated the
entire batch -- spend never reset, budget_reset_at never advanced. After
v1.84.0 started populating object_permission_id on UI-created keys, this
fires routinely.

_reset_budget_common also zeroed the cross-pod spend counter before the
DB write, so failed resets left enforcement reading 0 from the counter
while the DB still held the over-budget spend, admitting requests past
the cap until the counter naturally re-saturated from new reservations.

Switch the write to per-row narrow updates ({spend, budget_reset_at})
via db.batch_, and move the counter invalidation out of
_reset_budget_common so it only fires after the DB write commits. On
DB-write failure the counter is left untouched, enforcement continues
to block, and the next scheduler tick can retry without leaving a
bypass window.

Fixes BerriAI#27730.

* fix(reset_budget): address Greptile review on BerriAI#29358

- Strengthen the bypass-half regression test: replace the for-loop over
  call_args_list (vacuously true when empty) with assert_not_called(),
  so the test would actually flag a re-introduction of counter-zeroing
  via any code path.
- Add the same explanatory docstring on _write_user_reset_updates and
  _write_team_reset_updates that _write_key_reset_updates already has,
  so all three helpers point future maintainers at BerriAI#27730.

* test(reset_budget): update test_proxy_budget_reset for new batch-write path

Same shape as the previous test_reset_budget_job.py update: keys/users/teams
now write through prisma.db.batch_().<table>.update, not update_data, so the
tests need a batcher mock and updated assertions. Adds:

- _wire_batcher_for_test helper that returns a list which accumulates per-row
  batch updates captured from prisma_client.db.batch_().
- _attrify helper that wraps dict fixtures so getattr(item, "token") works
  alongside the dict item-access the fake_reset_* mocks rely on. The new
  narrow-write helpers use getattr to pull out the row's id, and would
  silently skip plain dicts otherwise.
- Updates 3 partial_failure tests to assert against the batch-call list
  (rows by id, payload contains only {spend, budget_reset_at}) instead of
  update_data.assert_awaited_once + data_list inspection.
- Updates test_reset_budget_continues_other_categories_on_failure: only
  budget + enduser still flow through update_data; key/user/team go through
  the batch path now.
- Wires the batcher mock into 3 service_logger_*_success tests so commit()
  is actually awaitable and the success hook fires.

These tests were silently passing locally only because the editable install
in .venv pointed at the main repo, not the worktree — running pytest with
PYTHONPATH overridden to the worktree (matching CI) reproduces the failures.
* test(e2e): cover PROXY_LOGOUT_URL redirect on Logout

Env-gated spec mirroring the existing serverRootPathRedirect pattern:
when the proxy is booted with PROXY_LOGOUT_URL set, clicking Logout in
the navbar must navigate to that external URL. The standard run_e2e.sh
exports an empty value so the rest of the suite is unaffected; this
spec self-skips unless the env var is populated.

* test(e2e): run PROXY_LOGOUT_URL spec in the suite + harden logout assertions

Boot the e2e proxy with PROXY_LOGOUT_URL set (job-level env in CircleCI and
run_e2e.sh) so proxyLogoutUrl.spec.ts actually runs instead of self-skipping.
Nothing else in the suite performs a logout, so this only affects the behavior
under test.

Harden the spec to verify the logout flow rather than a URL substring:
- wait for /sso/get/ui_settings before clicking so logoutUrl is populated
  (otherwise window.location.href = "" silently reloads same-origin)
- assert a token cookie exists first, and is cleared after logout
- locate the dropdown via getByRole instead of internal antd CSS classes
- stub the external destination and assert on URL origin + path prefix

* test(e2e): assert exact PROXY_LOGOUT_URL on logout redirect

Replace the origin + startsWith(pathname) checks with a single normalized
href comparison. With PROXY_LOGOUT_URL=https://www.example.com the path was
"/", so startsWith("/") matched any path and left path/query/hash
unchecked. Comparing normalized hrefs pins scheme, host, port, path, query
and hash while still tolerating the browser's trailing-slash/default-port
normalization.
When the user has visited both the dev UI (e.g. localhost:3000) and the
proxy UI (e.g. localhost:4000) in the same tab, logging out from the dev
origin produced an infinite logout/login redirect.

The proxy-side LoginPage's "is the user still authenticated?" check
was reading getCookie("token"), which falls back to sessionStorage when
document.cookie has no token. The cross-origin clearTokenCookies() call
from the dev origin can clear cookies on the shared hostname, but cannot
reach sessionStorage on the proxy origin (sessionStorage is per-origin),
so the fallback returned a stale token and LoginPage interpreted the
user as logged in, redirecting back to the dev origin. Dev origin then
saw no cookie and redirected to LoginPage, repeating ~20x per second.

This change introduces getCookieFromDocument(), a cookie-only read with
no sessionStorage fallback, and uses it in LoginPage's already-logged-in
check. The HttpOnly-reverse-proxy defense from PR BerriAI#23532 is unaffected:
storeLoginToken still writes both the JS cookie at /ui and the
sessionStorage backup, and getCookie still falls back for callers that
want the full read path.
…erriAI#29369)

The Next.js admin UI is exported with trailingSlash: true, so the proxy
serves /ui/login at /ui/login/index.html and 308s /ui/login → /ui/login/.
The waitForURL predicate used endsWith("/ui/login"), which never matched
the canonicalized URL and timed out after 15s.

This was masked until the build artifacts were regenerated against the
AuthContext fix: the prior bundles still hit the racy redirect path that
fired before proxyBaseUrl was populated, producing /ui/login (no prefix,
no proxy round-trip, no trailing slash) which fortuitously satisfied the
predicate. The first PR to ship the corrected bundle exposed the
assertion bug.

Switch the predicate to includes("/ui/login"); the prefix assertion below
still validates the SERVER_ROOT_PATH preservation that is the actual
contract under test.
* bump: version 0.1.41 → 0.1.42

* uv lock
chore(ci): promote internal staging to main
delete_model passed the internal model_name (model_name_{team_id}_{uuid})
to delete_team_model_alias, which searched alias map values for it.
For BYOK models the alias map is typically empty, so the lookup always
failed and the public name was never removed from team.models; creating
a ghost model visible via /models indefinitely.

Adds _resolve_team_public_model_name to extract the user-visible name
from model_info.team_public_model_name. Adds _cleanup_team_model_references
which always strips the public name from team.models (regardless of alias
map state) and also cleans up any alias entries pointing to the deleted
model's internal name. Wires _refresh_cached_team so /models reflects
the deletion immediately.

Fixes BerriAI#22594

On behalf of Adobe Inc.
@aanchal22
aanchal22 requested a review from a team June 2, 2026 20:01
@greptile-apps

greptile-apps Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Too many files changed for review. (1166 files found, 100 file limit)

@aanchal22 aanchal22 closed this Jun 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.