Skip to content

feat(datadog): add team-scoped Datadog callback support - #29947

Merged
ryan-crabbe-berri merged 1 commit into
BerriAI:litellm_oss_branchfrom
aanchal22:litellm_team-scoped-datadog
Jun 11, 2026
Merged

feat(datadog): add team-scoped Datadog callback support#29947
ryan-crabbe-berri merged 1 commit into
BerriAI:litellm_oss_branchfrom
aanchal22:litellm_team-scoped-datadog

Conversation

@aanchal22

@aanchal22 aanchal22 commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Relevant issues

Issue addressed - #29939

Pre-Submission checklist

  • I have added meaningful tests
  • My PR passes all unit tests on make test-unit
  • My PR's scope is as isolated as possible; it only solves 1 specific problem

Type

Feature

Changes

Adds team-scoped Datadog callback support, following the same pattern as Langfuse. Teams can now configure their own Datadog credentials via POST /team/{team_id}/callback to route logs to their own Datadog org.

What was missing

  1. DataDogLogger only read DD_API_KEY/DD_SITE from env vars — no per-team credential support
  2. No client isolation — all traffic went to one DD org
  3. dd_api_key/dd_site not in StandardCallbackDynamicParams allow-list

What changed

  • DataDogLogger.__init__() now accepts dd_api_key, dd_site, dd_agent_host, dd_agent_port as kwargs (falls back to env vars for global use)
  • DataDogHandler (new) resolves per-team DataDogLogger instances using DynamicLoggingCache, same pattern as LangFuseHandler
  • StandardCallbackDynamicParams and _supported_callback_params include DD params
  • _init_custom_logger_compatible_class uses DataDogHandler when team credentials are present
  • 15 unit tests covering credential kwargs, handler resolution, caching, and isolation

Existing behavior unchanged

Global Datadog callback (env-var based) continues to work exactly as before.

Screenshots / Proof of Fix

Adding team-scoped Datadog callback:

$ curl -sS "$BASE/team/dd-test-team/callback" \
  -H "Authorization: Bearer $MASTER" \
  -H "Content-Type: application/json" \
  -d '{"callback_name":"datadog","callback_type":"success_and_failure","callback_vars":{"dd_api_key":"test-team-dd-key-123","dd_site":"us5.datadoghq.com"}}' | jq .status

"success"

Callback stored in team metadata:

$ curl -sS "$BASE/team/info?team_id=dd-test-team" \
  -H "Authorization: Bearer $MASTER" | jq '.team_info.metadata'

{
  "logging": [
    {
      "callback_name": "datadog",
      "callback_type": "success_and_failure",
      "callback_vars": {
        "dd_site": "us5.datadoghq.com",
        "dd_api_key": "test-team-dd-key-123"
      }
    }
  ]
}

Request with team key routes logs to team's DD site (403 expected with fake key):

$ curl -sS "$BASE/v1/chat/completions" \
  -H "Authorization: Bearer $TEAM_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"dd-test-model","messages":[{"role":"user","content":"test"}]}' | jq .choices[0].message.content

"hello from dd test"

Proxy log confirms team's DD site was used:

Datadog Error sending batch API - Client error '403 Forbidden'
  for url 'https://http-intake.logs.us5.datadoghq.com/api/v2/logs'

The logger correctly used us5.datadoghq.com (from team's dd_site) instead of any global default.

@aanchal22
aanchal22 requested a review from a team June 8, 2026 17:50
@CLAassistant

CLAassistant commented Jun 8, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@aanchal22

Copy link
Copy Markdown
Contributor Author

@greptileai

@codecov

codecov Bot commented Jun 8, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.06349% with 5 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
litellm/litellm_core_utils/litellm_logging.py 54.54% 5 Missing ⚠️

📢 Thoughts on this report? Let us know!

@greptile-apps

greptile-apps Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds team-scoped Datadog callback support by extending DataDogLogger.__init__ with explicit credential kwargs, introducing DataDogHandler for per-team logger resolution via DynamicLoggingCache, and wiring the routing logic through _init_custom_logger_compatible_class. The ordering fix in Logging.__init__ (initializing standard_callback_dynamic_params before process_dynamic_callbacks) is necessary and correct.

  • DataDogLogger now accepts dd_api_key/dd_site/dd_agent_host/dd_agent_port as constructor kwargs, falling back to env vars for global use — existing deployments are unaffected.
  • _process_dynamic_callback_list filters dd_* keys from standard_callback_dynamic_params and forwards them exclusively to the datadog callback path, avoiding cross-integration credential leakage.
  • DD params are added to both _supported_callback_params and _request_blocked_callback_params, which correctly allows team-metadata sourcing while preventing per-request injection.

Confidence Score: 5/5

The change is safe to merge — team credential routing is correctly isolated via DynamicLoggingCache, the global env-var path is untouched, and DD credentials are blocked from per-request injection.

The credential resolution, cache keying, and routing logic are all correct. The reordering of initialization in Logging.init is sound, the security boundary (blocking dd_* from request metadata) is properly enforced, and tests cover caching, isolation, and detection. The one open concern is a dead method that duplicates the inline credential detection check, but it does not affect runtime behavior.

datadog_team_handler.py — the unused _dynamic_datadog_credentials_are_passed method duplicates logic that lives inline in litellm_logging.py and could drift over time.

Important Files Changed

Filename Overview
litellm/integrations/datadog/datadog.py DataDogLogger.init extended with explicit dd_api_key/dd_site/dd_agent_host/dd_agent_port kwargs that fall back to env vars; _configure_dd_agent and _configure_dd_direct_api updated to accept the same kwargs.
litellm/integrations/datadog/datadog_team_handler.py New DataDogHandler class provides per-team DataDogLogger resolution via DynamicLoggingCache; _dynamic_datadog_credentials_are_passed is defined but never called from production code.
litellm/litellm_core_utils/litellm_logging.py Reordered Logging.init to initialize standard_callback_dynamic_params before process_dynamic_callbacks; process_dynamic_callback_list now forwards filtered dd* keys to _init_custom_logger_compatible_class for the datadog callback; datadog branch in _init_custom_logger_compatible_class routes to DataDogHandler when team credentials are present.
litellm/litellm_core_utils/initialize_dynamic_callback_params.py Adds dd_api_key, dd_site, dd_agent_host, dd_agent_port to both _supported_callback_params and _request_blocked_callback_params, correctly allowing team-config sourcing while blocking per-request injection.
litellm/types/utils.py Adds dd_api_key, dd_site, dd_agent_host, dd_agent_port to StandardCallbackDynamicParams TypedDict.
tests/test_litellm/integrations/datadog/test_datadog_team_handler.py 15 new unit tests covering credential kwargs, handler resolution, caching, isolation, and allow-list membership; all use mocks with no real network calls.

Reviews (3): Last reviewed commit: "feat(datadog): add team-scoped Datadog c..." | Re-trigger Greptile

Comment thread litellm/litellm_core_utils/litellm_logging.py Outdated
Comment thread litellm/integrations/datadog/datadog_team_handler.py Outdated
Comment thread litellm/litellm_core_utils/litellm_logging.py
@greptile-apps

greptile-apps Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds team-scoped Datadog callback support to LiteLLM, following the same pattern already established for LangFuse. DataDogLogger.__init__ now accepts per-team credentials as kwargs (falling back to env vars for the global case), and a new DataDogHandler class resolves and caches per-team logger instances via the shared DynamicLoggingCache.

  • StandardCallbackDynamicParams and _supported_callback_params gain four new dd_* fields so team credentials flow through the existing callback-params pipeline.
  • Logging.__init__ initialization order is fixed: standard_callback_dynamic_params is now set before process_dynamic_callbacks() runs, allowing _init_callback_list to surface team credentials to _init_custom_logger_compatible_class for per-request logger resolution.
  • 15 mock-only unit tests cover credential kwargs, handler resolution, cache hit/miss, and team isolation.

Confidence Score: 4/5

The change is additive and isolated to the Datadog logging path; global Datadog behavior and all other callbacks are unchanged.

The core logic is correct and the LangFuse pattern is followed closely. The two issues worth watching are a latent dual-logger risk in _return_global_datadog_logger (unreachable from the main flow today but a footgun for future callers) and a now-redundant hasattr guard that could mislead future readers about initialization invariants. Neither affects current production behavior.

litellm/integrations/datadog/datadog_team_handler.py — specifically the _return_global_datadog_logger fallback path and its relationship to _in_memory_loggers.

Important Files Changed

Filename Overview
litellm/integrations/datadog/datadog.py Adds dd_api_key/dd_site/dd_agent_host/dd_agent_port kwargs to DataDogLogger.init, with env-var fallback; configures agent or direct-API path based on resolved values.
litellm/integrations/datadog/datadog_team_handler.py New DataDogHandler class following the LangFuse pattern: resolves and caches per-team DataDogLogger instances via DynamicLoggingCache; _return_global_datadog_logger creates a logger in DynamicLoggingCache with empty credentials, parallel to _in_memory_loggers and unreachable from the main production flow.
litellm/litellm_core_utils/litellm_logging.py Initialization order fixed so standard_callback_dynamic_params is set before process_dynamic_callbacks; _init_callback_list passes params to _init_custom_logger_compatible_class; datadog branch routes to DataDogHandler when team credentials are present.
litellm/litellm_core_utils/initialize_dynamic_callback_params.py Adds dd_api_key, dd_site, dd_agent_host, dd_agent_port to _supported_callback_params allow-list; straightforward addition following existing pattern.
litellm/types/utils.py Adds four Datadog dynamic params to StandardCallbackDynamicParams TypedDict; no logic changes.
tests/test_litellm/integrations/datadog/test_datadog_team_handler.py 15 new mock-only unit tests covering credential kwargs, handler resolution, caching, team isolation, and fallback; no real network calls; asyncio.create_task is patched throughout.

Reviews (2): Last reviewed commit: "feat(datadog): add team-scoped Datadog c..." | Re-trigger Greptile

Comment thread litellm/integrations/datadog/datadog_team_handler.py Outdated
Comment thread litellm/integrations/datadog/datadog_team_handler.py Outdated
Comment thread litellm/litellm_core_utils/litellm_logging.py Outdated
Comment thread litellm/litellm_core_utils/initialize_dynamic_callback_params.py
@veria-ai

veria-ai Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

Enable teams to configure their own Datadog credentials via
POST /team/{team_id}/callback, following the same pattern as Langfuse.
@aanchal22
aanchal22 force-pushed the litellm_team-scoped-datadog branch from f212e03 to 9c049da Compare June 8, 2026 18:07
@aanchal22

Copy link
Copy Markdown
Contributor Author

@greptile-apps re review please

@ryan-crabbe-berri
ryan-crabbe-berri merged commit f5e6012 into BerriAI:litellm_oss_branch Jun 11, 2026
44 checks passed
mateo-berri pushed a commit that referenced this pull request Jun 11, 2026
Cherry-picked from the PR head 9c049da (single-commit PR, merged to
litellm_oss_branch). Applied cleanly; no conflicts.

Note: black --check in this worktree flags pre-existing multi-line string
formatting in litellm_core_utils/litellm_logging.py (lines ~1006-1050) that is
already present on the patch/v1.89.0-rc.1 base and is untouched by this pick --
left as-is to avoid reformatting unrelated lines.
mateo-berri added a commit that referenced this pull request Jun 11, 2026
…AIDR, Mantle SigV4, NetApp streaming-cost fix, and team-scoped Datadog toward v1.89.0-rc.3 (#30179)

* fix(proxy): authorize batch files using upload target_model_names (LIT-3593) (#30009)

* fix(proxy): authorize batch files using upload target_model_names (LIT-3593)

After replace_model_in_jsonl, body.model is a stripped provider id. Reverse-mapping it via resolve_model_name_from_model_id is first-match on model_list and caused false 403s when multiple deployments share the same stripped name. Use target_model_names from the unified file id instead.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593)

Restores the reverse-lookup for the JSONL body.model fallback path so that
legacy/pre-target_model_names managed files still map stripped provider IDs
back to proxy aliases before auth. Also cleans up redundant `or None`.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Revert "fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593)"

This reverts commit 30d2e96.

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
(cherry picked from commit 2cd7e87)

* feat(guardrails): capture user and model metadata in CrowdStrike AIDR

(cherry picked from commit 6fc715c)

* fix(guardrails): read CrowdStrike AIDR identity from both metadata bags (#29991)

Capture user_id and extra_info from metadata or litellm_metadata. The single-bag read dropped identity whenever a request carried a present litellm_metadata field (null or a user-supplied dict), since /chat/completions routes the authenticated identity into metadata while the guardrail read litellm_metadata first

(cherry picked from commit 1bbaf1c)

* feat(bedrock_mantle): add SigV4/IAM auth to Responses API route (#29788)

Applied as the squash diff of PR #29788 (head 9800b2f), which landed
upstream inside the litellm_oss_staging_080626 sync (32c88ca, #29932)
and has no standalone commit to cherry-pick. The rc line already carries
the prerequisite #29490 Responses route via the 040626 sync.

* fix: completion_cost AttributeError on streaming Anthropic web_search responses (#26153) (#27346)

Cherry-picked from staging squash 4a3860d.

The rc line predates the Usage.__init__ server_tool_use dict->ServerToolUse
coercion that staging carries (it landed via the squashed OSS sync #29932 /
32c88ca, not as a standalone commit). The calculate_usage
Usage(**returned_usage.model_dump()) round-trip re-serializes server_tool_use
to a plain dict, so without that coercion the rebuilt usage holds a dict and the
regression test asserting a ServerToolUse type fails. Restored the coercion in
litellm/types/utils.py to satisfy the prerequisite -- it matches #27346's own
first commit (coerce server_tool_use dict to ServerToolUse in Usage.__init__),
which was dropped from the squash only because staging already carried it.

* feat(datadog): add team-scoped Datadog callback support (#29947)

Cherry-picked from the PR head 9c049da (single-commit PR, merged to
litellm_oss_branch). Applied cleanly; no conflicts.

Note: black --check in this worktree flags pre-existing multi-line string
formatting in litellm_core_utils/litellm_logging.py (lines ~1006-1050) that is
already present on the patch/v1.89.0-rc.1 base and is untouched by this pick --
left as-is to avoid reformatting unrelated lines.

---------

Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Kenan Yildirim <kenan@kenany.me>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Kent <kingdooo@gmail.com>
Co-authored-by: ishaan-berri <155045088+ishaan-berri@users.noreply.github.com>
Co-authored-by: aanchal22 <12680748+aanchal22@users.noreply.github.com>
Sameerlite pushed a commit that referenced this pull request Jun 12, 2026
Enable teams to configure their own Datadog credentials via
POST /team/{team_id}/callback, following the same pattern as Langfuse.
mateo-berri added a commit that referenced this pull request Jun 12, 2026
* feat(bedrock): add bedrock mantle gemma 4 models (#30264)

* feat(bedrock): add bedrock mantle gemma 4 models

* test(bedrock): harden mantle local cost fixture

* feat(responses): enable the responses API for the Tensormesh provider (#30209)

* feat(responses): enable the responses API for the Tensormesh provider

* Update litellm/llms/openai_like/providers.json

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

---------

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* fix(langfuse_otel): mark LLM spans as generations (#30250)

* fix(bedrock): stop stream_chunk_size leaking into invoke request bodies (#30240)

stream_chunk_size is a LiteLLM-internal knob for re-chunking the HTTP
response stream. The invoke transformations splat optional_params into the
provider request body without dropping it, and Bedrock rejects unknown
fields, so any bedrock/invoke request that sets the parameter fails with
ValidationException: stream_chunk_size: Extra inputs are not permitted.
Drop it in the invoke dispatcher (covers cohere, titan, mistral, meta,
ai21) and in the Claude messages-format request builder (the route used
for bedrock/invoke Anthropic models)

* fix(bedrock): stop buffering streamed tool-call argument deltas (#30231)

* fix(bedrock): stop buffering streamed tool-call argument deltas

Two issues made Bedrock tool-use streaming arrive as a single end-of-stream
burst through LiteLLM while plain text streamed fine.

First, the anthropic-beta allowlist mapped fine-grained-tool-streaming-2025-05-14
to null for bedrock and bedrock_converse, so the header was silently stripped.
Without that beta, Anthropic models on Bedrock buffer tool input server-side and
emit all toolUse.input deltas at once (verified against converse-stream and
invoke-with-response-stream directly). Bedrock accepts the beta via
additionalModelRequestFields.anthropic_beta, so it is now forwarded.

Second, the streaming reads re-chunked the AWS event stream with
iter_bytes(chunk_size=1024). httpx's ByteChunker only releases full 1024-byte
blocks, so the small early events (messageStart, contentBlockStart, first
deltas) sat in the buffer until enough bytes accumulated, pushing
time-to-first-byte from ~1.4s to ~8.5s on buffered tool-use streams. The
default is now no re-chunking; an explicit stream_chunk_size is still honored.

* test(bedrock): cover explicit stream_chunk_size on sync invoke path

* test(bedrock): cover stream_chunk_size plumbing through converse completion

* test(bedrock): cover stream_chunk_size default in legacy BedrockLLM streaming

* test(bedrock): merge converse handler tests into existing mapped test file

pytest imports test modules by basename in non-package test dirs, so the new
tests/test_litellm/llms/bedrock/chat/test_converse_handler.py collided with
the pre-existing tests/test_litellm/llms/chat/test_converse_handler.py and
broke collection in CI. Move the new tests into the existing file

* feat(otel): emit v2 cost breakdown + stamp tracer scope version (#30156)

Read the StandardLoggingPayload cost_breakdown into a typed LLMCost on
LLMCallSpanData and emit each component under litellm.cost.* (absent
components omitted, so spans stay sparse). Stamp litellm.__version__ as
the instrumentation scope version so every v2 span carries a
deterministic scope.version.

Tests under tests/test_litellm/integrations/otel/.

* fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in) (#30223)

* fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in)

On the non-streaming path, base_process_llm_request awaited the LLM call
with no disconnect monitoring; when the HTTP client went away the
upstream request kept running until completion or request_timeout (6000s
default), holding a backend slot (e.g. a vLLM GPU slot) for output
nobody would read

Add an opt-in general_settings.cancel_on_disconnect flag, default off,
so the default code path is unchanged. When enabled, a receive-based
watcher task observes http.disconnect and cancels the asyncio.gather
driving the upstream call. The resulting CancelledError is converted to
HTTPException 499 only when the disconnect event is set, so
server-initiated cancellations still propagate as-is. The 499 then flows
through _handle_llm_api_exception like any other failure, meaning
post_call_failure_hook still releases max_parallel_requests slots and
fires spend and alerting callbacks; it is logged at info level instead
of a full traceback

Also removes the dead check_request_disconnection helper in
proxy_server.py (zero call sites) along with its behavior-pin tests

Builds on the receive-based design from #25776

Addresses #13774. Re-fixes #22805 (regressed after the #14295 revert)

Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com>

* fix(proxy): scope 499 quiet logging to disconnects and harden watcher

Address the two P2 findings from the Greptile review on #30223. The
info-level logging in _log_llm_api_exception now applies only to the
disconnect-specific HTTPException (status 499 plus the shared
_CLIENT_DISCONNECT_DETAIL message), so any other 499 raised by hooks or
guardrails keeps its full traceback. The disconnect watcher now catches
exceptions from request.receive() (e.g. a transport reset) and logs a
warning instead of dying silently, making the degradation to no-op
visible; a test pins that the LLM call is not cancelled in that case

---------

Co-authored-by: kursad <kursad.lacin@brado.net>
Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com>

* fix(bedrock): grant aws-external-anthropic:* in OIDC session policy for claude_platform (#30200) (#30205)

The inline STS session policy passed to assume_role_with_web_identity
acts as an IAM PERMISSION CEILING — effective permissions are the
intersection of the role's identity policies and this policy. Any
action not listed is silently denied even when the IAM role grants it.

#27678 added the bedrock/claude_platform/<model> route but its
service-side action namespace is aws-external-anthropic:*, not
bedrock:*. Without a matching statement here, every claude_platform
request via OIDC (GCP federation, EKS Pod Identity webhook, etc.) 403s
with 'no session policy allows the aws-external-anthropic:CreateInference
action' — even with a fully permissive identity policy.

Add a second ClaudePlatformLiteLLM statement covering CreateInference,
CreateBatchInference, CancelBatchInference, DeleteBatchInference,
CountTokens, Get*, List*. Keep aws:SecureTransport=true parity with the
bedrock statement.

Static creds + IRSA flow through different code paths and are not
affected.

Fixes #30200

* fix(proxy): set Retry-After header on RouterRateLimitError 429 responses (#30098)

* Set Retry-After header on RouterRateLimitError responses

When all deployments for a model are in cooldown, the proxy returns a
429 whose cooldown timing is only available by parsing the error
message string. RouterRateLimitError already carries cooldown_time, so
expose it as a standard retry-after header in
_handle_llm_api_exception. The value is rounded up so clients never
retry before the cooldown window ends.

Fixes #27823.

* Set Retry-After after response-headers hook so cooldown wins

The cooldown-derived retry-after was assigned before the
post_call_response_headers_hook merge, so a callback returning a
retry-after key (including a stale or empty value) silently clobbered
it. Move the RouterRateLimitError block after the callback merge so the
cooldown value is authoritative for this error type.

* fix(router): route aspeech through async_function_with_fallbacks (#30104)

* fix(router): route aspeech through async_function_with_fallbacks

Router.aspeech selected a deployment and awaited litellm.aspeech
directly, so TTS requests got no retry on failure and no failover to
backup deployments; the except block only fired an exception alert and
re-raised. Every other router endpoint (acompletion, aembedding,
atranscription, arerank) already delegates to
async_function_with_fallbacks

Mirror the atranscription pattern: move deployment selection and the
litellm.aspeech call into a private _aspeech method, then have the
public aspeech set kwargs["original_function"] = self._aspeech and
await self.async_function_with_fallbacks(**kwargs). _aspeech also picks
up the shared _get_async_openai_model_client helper and the same
total/success/fail call accounting the sibling endpoints use

Fixes #27778.

* fix(router): apply deployment kwargs and rpm semaphore in _aspeech

Bring _aspeech fully in line with _atranscription: call
_update_kwargs_with_deployment so deployment metadata, model_info,
timeout, and default litellm params flow into the request, and wrap
the litellm.aspeech call with the max_parallel_requests semaphore plus
async_routing_strategy_pre_call_checks so TTS respects rpm limits the
same way the other router endpoints do

Also add a unit test that exercises _aspeech directly and asserts the
deployment metadata reaches the underlying call

* fix(slack_alerting): stop false-positive hanging request alerts for requests below the alerting threshold (#30106)

* fix(slack_alerting): skip hanging request alerts below the threshold

The hanging request check alerted on any cached request whose
completion status was not yet recorded, with no minimum age check.
Since the background loop runs every alerting_threshold / 2 seconds,
any request that happened to be in flight at a check fired a
"hanging - Ns+ request time" alert even if it was only seconds old,
producing a steady stream of false positives.

Add a created_at timestamp to HangingRequestData, stamped when the
request enters the hanging request cache, and skip requests younger
than alerting_threshold without evicting them, so a later check can
still alert if they never complete. Extend the cache TTL from
threshold + 60s to 1.5x threshold + 60s; with the age check, entries
only become alertable after threshold seconds, and the check period
is threshold / 2, so the old TTL could evict a genuinely hanging
request before any check saw it cross the threshold.

Fixes #27855.

* fix(slack_alerting): alert once per hanging request

The min-age gate stops false positives for young in-flight requests, but
a genuinely hanging request still re-alerted on every checker tick within
the cache TTL. With the wider TTL (1.5x threshold + 60s) that is 1-2 extra
Slack notifications per stuck request at the default 600s threshold.

Flag a HangingRequestData entry as alerted once its alert fires and skip
flagged entries on later ticks, so each hang produces exactly one alert.
The cache reference is mutated in place, so the TTL is untouched and still
handles cleanup. Adds a regression test asserting one alert across multiple
ticks.

Fixes #27855.

* fix(health): treat all-proxy-models keys as unrestricted in /health (#30087)

* fix(health): treat all-proxy-models keys as unrestricted in /health

A key granted all model permissions stores the literal
"all-proxy-models" marker in its models list. The /health access
filter compared that marker against real model_names, so the model
list filtered down to nothing and the WebUI health check returned
healthy_count=0, unhealthy_count=0 with HTTP 503. Skip the filter
(both the live path and the background-cache model_id scoping) when
the marker is present, matching how auth_checks treats
SpecialModelNames.all_proxy_models.

Fixes #29744.

* fix(health): resolve all-team-models sentinel to the team allowlist

Same failure shape as the all-proxy-models case: a key carrying the
literal "all-team-models" entry matches no real model_name, so the
/health access filter would zero out the model list. Resolve the
sentinel to the key's team models when team_id is set, matching
get_key_models in model_checks.py. Without a team_id the sentinel
stays unresolved and matches nothing, denying rather than widening
access, mirroring _resolve_key_models_for_auth_check.

* feat(proxy): auto-enable drop_params for Claude Code requests (#30218)

* feat(proxy): auto-enable drop_params for Claude Code requests

Claude Code identifies itself with a claude-cli/<version> user agent and
sends Anthropic-specific params (top_k, thinking, etc.) on every request.
When the proxy routes those requests to a non-Anthropic provider, the
unsupported params fail the call unless drop_params is configured. Detect
the Claude Code user agent in add_litellm_data_to_request and default
drop_params to true for those requests, without overriding an explicit
drop_params value sent by the caller.

* feat(proxy): respect operator litellm_settings drop_params over Claude Code default

An explicit drop_params in the operator's litellm_settings (true or false)
now suppresses the Claude Code user agent default, so an operator who
deliberately configured drop_params: false keeps strict param validation
for Claude Code clients too. The auto-default only fills the gap when
neither the request body nor the config sets a value.

* fix(snowflake): migrate to native endpoints with auto-routing for Claude models (#29964)

* fix(snowflake): migrate to native Cortex REST API endpoints

Replaces the legacy /api/v2/cortex/inference:complete endpoint with the
native OpenAI-compatible /api/v2/cortex/v1/chat/completions endpoint,
fixing error 390142 (Incoming request does not contain a valid payload)
when using model: snowflake/<model> in LiteLLM proxy.

Changes:
- litellm/llms/snowflake/chat/transformation.py: route to native
  /cortex/v1/chat/completions, remove Snowflake-specific tool_spec
  payload transformation, remove content_list response handling,
  add stream to supported params
- litellm/llms/snowflake/anthropic/transformation.py (new):
  SnowflakeCortexAnthropicConfig routes Claude models to /cortex/v1/messages
  with anthropic-version header and Anthropic->OpenAI response transform
- tests: 29 unit tests covering URL routing, auth headers, payload
  format, and response parsing

* fix(snowflake): map max_tokens to max_completion_tokens for native endpoint

* fix: handle multi-turn tool conversations and OpenAI→Anthropic tool format conversion

- _extract_system_and_messages now preserves tool_calls from assistant messages
  and converts them to Anthropic tool_use content blocks
- tool role messages are converted to user role with tool_result content blocks
  (as required by Anthropic Messages API)
- Added _transform_tools_to_anthropic() to convert OpenAI tool format
  (type/function/parameters) to Anthropic format (name/input_schema)
- Added comprehensive tests for multi-turn tool conversations

Addresses review feedback on PR #29964

* test: add coverage for malformed JSON and non-string tool arguments

* fix(tests): update chat transformation tests for native OpenAI-compatible endpoint

* style: apply black formatting

* fix: resolve mypy type errors in anthropic transformation

* fix: correct mypy type: ignore error codes (attr-defined)

* fix: use max_tokens instead of max_completion_tokens for Snowflake endpoint compatibility

* refactor: merge Anthropic config into unified SnowflakeConfig with auto-routing

- Remove separate SnowflakeCortexAnthropicConfig and anthropic/ directory
- SnowflakeConfig now auto-routes based on model name:
  - Claude models → /messages endpoint (Anthropic format)
  - All others → /chat/completions endpoint (OpenAI format)
- No new provider needed (stays as SNOWFLAKE = 'snowflake')
- Tool message transformation for Claude: tool_calls → tool_use blocks,
  tool role → user with tool_result
- OpenAI → Anthropic tool format conversion (parameters → input_schema)
- Addresses Greptile feedback about unwired SnowflakeCortexAnthropicConfig

* fix: use max_completion_tokens for /chat/completions (Snowflake deprecated max_tokens on this endpoint)

* fix(tests): update assertions for Claude auto-routing to /messages endpoint

* fix(snowflake): add tool_choice conversion and preserve max_completion_tokens in Anthropic path

* fix(snowflake): use ChatCompletionMessageToolCall objects and strip model prefix on OpenAI path

* fix(snowflake): collect multiple system messages to prevent guardrail override

* chore: remove committed .pyc files and add __pycache__ to .gitignore

* fix: remove unused Union import

* fix: restore original .gitignore (accidentally replaced in earlier commit)

* feat(snowflake): add streaming response handler for both Anthropic and OpenAI SSE formats

* fix: remove unused AsyncIterator and Iterator imports

* fix: add missing total_tokens to ChatCompletionUsageBlock

* fix(snowflake): coalesce consecutive tool results into single user message for Anthropic

* fix(snowflake): handle message_start event for streaming input_tokens tracking

* fix: evict last deleted model in multi-instance deployments (#28608)

* fix: evict last deleted model in multi-instance deployments

_delete_deployment had an early return when db_models was empty,
preventing eviction of the last deleted model during reconciliation.

- Remove len(db_models)==0 early return from _delete_deployment
- Return None (not []) from _get_models_from_db on DB failure so
  callers can distinguish a transient failure from a genuinely empty DB
- Guard _update_llm_router against None to skip updates on DB failure

Fixes #28443

* test: remove dead MagicMock assignment in type_mismatch test

* fix: update test to pass [] not None to _update_llm_router

test_ProxyConfig__update_llm_router_bad_proxy_logging_raises was passing
None as new_models to get through to the proxy_logging_obj check, but
the None guard we added now returns early before reaching that path.
Pass [] instead so the test exercises the intended AttributeError case.

Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com>

* chore: regenerate API types to sync schema.d.ts with proxy OpenAPI spec

Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com>

---------

Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com>

* fix: invalidate Redis spend counter on /key/reset_spend (#29694)

* fix: set Redis spend counter to reset_to value on /key/reset_spend

Previously, the Redis spend counter was always set to 0.0 after a reset,
even when reset_to was a non-zero value (partial reset). This caused
the budget to be under-enforced for up to 60 seconds until the counter
expired and fell through to the DB.

Now the counter is set to the actual reset_to value, so partial resets
are reflected correctly and budget enforcement is consistent.

* test: update reset_key_spend test to match direct cache set

The implementation now sets spend_counter_cache directly instead of
calling _invalidate_spend_counter. Update the test to verify the
in_memory_cache.set_cache call with the correct key, value, and ttl.

---------

Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>

* fix: add scaleway models pricing (#27659)

* fix: Add embeddings support for Scaleway provider

* fix: resolve merge conflicts

* fix(main): clarify backend route handling for Swagger static assets (#30196)

* fix(main): clarify backend route handling for Swagger static assets

* fix(allowlist): add BACKEND_MOUNT_PATHS for Swagger static assets

* fix(voyage): route multimodal embeddings to correct endpoint (#30193)

* fix(voyage): route multimodal embeddings to correct endpoint

* test(voyage): cover multimodal embedding edge cases

* test(voyage): cover api key fallback

* fix(voyage): raise early on missing api key and malformed image url

* test(voyage): cover utils routing and helper

* fix(voyage): route supported openai params for multimodal models

* style: apply black formatting

* fix(ui): infer Azure API version from API base (#30204)

* fix(ui): infer Azure API version from API base

* fix(ui): address Azure API version feedback

* Update litellm/llms/snowflake/chat/transformation.py

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* feat(datadog): add team-scoped Datadog callback support (#29947)

Enable teams to configure their own Datadog credentials via
POST /team/{team_id}/callback, following the same pattern as Langfuse.

* Merge pull request #29528 from aanchal22/litellm_byok-alias-merge

fix(proxy): atomic merge for team model aliases and team.models on BYOK create

* feat: add EmpirioLabs as an OpenAI-compatible provider (#30278)

Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com>

* fix: resolve failing tests and lint in snowflake/team endpoints

- Black-format snowflake/chat/transformation.py to fix lint failure
- Update Anthropic config test to expect default max_tokens of 4096 (matches implementation)
- Add AsyncMock + execute_raw mock to team_model_add cache-refresh pin test
- Add model_dump mock and patch cache/logging in test_uses_atomic_array_append_with_dedup

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(test): update test_db_error_new_model_check for new _delete_deployment logic

_delete_deployment no longer short-circuits on empty db_models — it now
treats [] as a valid empty-DB state and proceeds to check config models.
Mock get_config to return the two router deployments so they appear in
combined_id_list and are protected, which matches the real-world scenario
where a DB error occurs but the models are config-backed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list (#30295)

* feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list

Follow-up to #30223 per maintainer review: documents the flag in
ConfigGeneralSettings with a short description and adds it to
allowed_args in get_config_list so the UI and /config/list expose it.
A test pins that /config/list returns the field with type Boolean,
which requires both registrations to be present

* chore(ui): regenerate schema.d.ts for cancel_on_disconnect

---------

Co-authored-by: kursad <kursad.lacin@brado.net>

* fix(datadog): never fall back to env DD_API_KEY for caller-supplied destinations

Team/key-scoped Datadog loggers could be pointed at an arbitrary dd_agent_host or
dd_site while omitting dd_api_key, causing the proxy's global DD_API_KEY to be sent
as the DD-API-KEY header to that destination. Gate the env-var fallback behind an
allow_env_credentials flag, set to False when the destination is caller-supplied,
mirroring the existing langfuse/langsmith pattern.

---------

Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com>
Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com>
Co-authored-by: daitran-tensormesh <dai@tensormesh.ai>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Muspi Merol <me@promplate.dev>
Co-authored-by: fangkang <fangkangm@gmail.com>
Co-authored-by: Chris Hoogeboom <chris.hoogeboom@gmail.com>
Co-authored-by: kursadlacin <kursadlacin@gmail.com>
Co-authored-by: kursad <kursad.lacin@brado.net>
Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com>
Co-authored-by: hcl <chenglunhu@gmail.com>
Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: sfc-gh-nashukla <navnit.shukla@snowflake.com>
Co-authored-by: Rudra Dudhat <contact.rdudhat@gmail.com>
Co-authored-by: Michael <52305679+michaelxer@users.noreply.github.com>
Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>
Co-authored-by: Quentin Champenois <26109239+Quentinchampenois@users.noreply.github.com>
Co-authored-by: mauriceberentsen <mauriceberentsen@live.nl>
Co-authored-by: lost9999 <56498264+lost9999@users.noreply.github.com>
Co-authored-by: GaetanVDB07 <86427581+GaetanVDB07@users.noreply.github.com>
Co-authored-by: Aanchal Khandelwal <aan2210khandelwal@gmail.com>
Co-authored-by: Adam Dalloul <adam_dalloul@icloud.com>
Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
mrgizmo212 pushed a commit to ttgaillc/ttginf that referenced this pull request Jun 16, 2026
Enable teams to configure their own Datadog credentials via
POST /team/{team_id}/callback, following the same pattern as Langfuse.

(cherry picked from commit f5e6012)
michaelxer added a commit to michaelxer/litellm that referenced this pull request Jun 17, 2026
* feat(bedrock): add bedrock mantle gemma 4 models (BerriAI#30264)

* feat(bedrock): add bedrock mantle gemma 4 models

* test(bedrock): harden mantle local cost fixture

* feat(responses): enable the responses API for the Tensormesh provider (BerriAI#30209)

* feat(responses): enable the responses API for the Tensormesh provider

* Update litellm/llms/openai_like/providers.json

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

---------

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* fix(langfuse_otel): mark LLM spans as generations (BerriAI#30250)

* fix(bedrock): stop stream_chunk_size leaking into invoke request bodies (BerriAI#30240)

stream_chunk_size is a LiteLLM-internal knob for re-chunking the HTTP
response stream. The invoke transformations splat optional_params into the
provider request body without dropping it, and Bedrock rejects unknown
fields, so any bedrock/invoke request that sets the parameter fails with
ValidationException: stream_chunk_size: Extra inputs are not permitted.
Drop it in the invoke dispatcher (covers cohere, titan, mistral, meta,
ai21) and in the Claude messages-format request builder (the route used
for bedrock/invoke Anthropic models)

* fix(bedrock): stop buffering streamed tool-call argument deltas (BerriAI#30231)

* fix(bedrock): stop buffering streamed tool-call argument deltas

Two issues made Bedrock tool-use streaming arrive as a single end-of-stream
burst through LiteLLM while plain text streamed fine.

First, the anthropic-beta allowlist mapped fine-grained-tool-streaming-2025-05-14
to null for bedrock and bedrock_converse, so the header was silently stripped.
Without that beta, Anthropic models on Bedrock buffer tool input server-side and
emit all toolUse.input deltas at once (verified against converse-stream and
invoke-with-response-stream directly). Bedrock accepts the beta via
additionalModelRequestFields.anthropic_beta, so it is now forwarded.

Second, the streaming reads re-chunked the AWS event stream with
iter_bytes(chunk_size=1024). httpx's ByteChunker only releases full 1024-byte
blocks, so the small early events (messageStart, contentBlockStart, first
deltas) sat in the buffer until enough bytes accumulated, pushing
time-to-first-byte from ~1.4s to ~8.5s on buffered tool-use streams. The
default is now no re-chunking; an explicit stream_chunk_size is still honored.

* test(bedrock): cover explicit stream_chunk_size on sync invoke path

* test(bedrock): cover stream_chunk_size plumbing through converse completion

* test(bedrock): cover stream_chunk_size default in legacy BedrockLLM streaming

* test(bedrock): merge converse handler tests into existing mapped test file

pytest imports test modules by basename in non-package test dirs, so the new
tests/test_litellm/llms/bedrock/chat/test_converse_handler.py collided with
the pre-existing tests/test_litellm/llms/chat/test_converse_handler.py and
broke collection in CI. Move the new tests into the existing file

* feat(otel): emit v2 cost breakdown + stamp tracer scope version (BerriAI#30156)

Read the StandardLoggingPayload cost_breakdown into a typed LLMCost on
LLMCallSpanData and emit each component under litellm.cost.* (absent
components omitted, so spans stay sparse). Stamp litellm.__version__ as
the instrumentation scope version so every v2 span carries a
deterministic scope.version.

Tests under tests/test_litellm/integrations/otel/.

* fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in) (BerriAI#30223)

* fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in)

On the non-streaming path, base_process_llm_request awaited the LLM call
with no disconnect monitoring; when the HTTP client went away the
upstream request kept running until completion or request_timeout (6000s
default), holding a backend slot (e.g. a vLLM GPU slot) for output
nobody would read

Add an opt-in general_settings.cancel_on_disconnect flag, default off,
so the default code path is unchanged. When enabled, a receive-based
watcher task observes http.disconnect and cancels the asyncio.gather
driving the upstream call. The resulting CancelledError is converted to
HTTPException 499 only when the disconnect event is set, so
server-initiated cancellations still propagate as-is. The 499 then flows
through _handle_llm_api_exception like any other failure, meaning
post_call_failure_hook still releases max_parallel_requests slots and
fires spend and alerting callbacks; it is logged at info level instead
of a full traceback

Also removes the dead check_request_disconnection helper in
proxy_server.py (zero call sites) along with its behavior-pin tests

Builds on the receive-based design from BerriAI#25776

Addresses BerriAI#13774. Re-fixes BerriAI#22805 (regressed after the BerriAI#14295 revert)

Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com>

* fix(proxy): scope 499 quiet logging to disconnects and harden watcher

Address the two P2 findings from the Greptile review on BerriAI#30223. The
info-level logging in _log_llm_api_exception now applies only to the
disconnect-specific HTTPException (status 499 plus the shared
_CLIENT_DISCONNECT_DETAIL message), so any other 499 raised by hooks or
guardrails keeps its full traceback. The disconnect watcher now catches
exceptions from request.receive() (e.g. a transport reset) and logs a
warning instead of dying silently, making the degradation to no-op
visible; a test pins that the LLM call is not cancelled in that case

---------

Co-authored-by: kursad <kursad.lacin@brado.net>
Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com>

* fix(bedrock): grant aws-external-anthropic:* in OIDC session policy for claude_platform (BerriAI#30200) (BerriAI#30205)

The inline STS session policy passed to assume_role_with_web_identity
acts as an IAM PERMISSION CEILING — effective permissions are the
intersection of the role's identity policies and this policy. Any
action not listed is silently denied even when the IAM role grants it.

BerriAI#27678 added the bedrock/claude_platform/<model> route but its
service-side action namespace is aws-external-anthropic:*, not
bedrock:*. Without a matching statement here, every claude_platform
request via OIDC (GCP federation, EKS Pod Identity webhook, etc.) 403s
with 'no session policy allows the aws-external-anthropic:CreateInference
action' — even with a fully permissive identity policy.

Add a second ClaudePlatformLiteLLM statement covering CreateInference,
CreateBatchInference, CancelBatchInference, DeleteBatchInference,
CountTokens, Get*, List*. Keep aws:SecureTransport=true parity with the
bedrock statement.

Static creds + IRSA flow through different code paths and are not
affected.

Fixes BerriAI#30200

* fix(proxy): set Retry-After header on RouterRateLimitError 429 responses (BerriAI#30098)

* Set Retry-After header on RouterRateLimitError responses

When all deployments for a model are in cooldown, the proxy returns a
429 whose cooldown timing is only available by parsing the error
message string. RouterRateLimitError already carries cooldown_time, so
expose it as a standard retry-after header in
_handle_llm_api_exception. The value is rounded up so clients never
retry before the cooldown window ends.

Fixes BerriAI#27823.

* Set Retry-After after response-headers hook so cooldown wins

The cooldown-derived retry-after was assigned before the
post_call_response_headers_hook merge, so a callback returning a
retry-after key (including a stale or empty value) silently clobbered
it. Move the RouterRateLimitError block after the callback merge so the
cooldown value is authoritative for this error type.

* fix(router): route aspeech through async_function_with_fallbacks (BerriAI#30104)

* fix(router): route aspeech through async_function_with_fallbacks

Router.aspeech selected a deployment and awaited litellm.aspeech
directly, so TTS requests got no retry on failure and no failover to
backup deployments; the except block only fired an exception alert and
re-raised. Every other router endpoint (acompletion, aembedding,
atranscription, arerank) already delegates to
async_function_with_fallbacks

Mirror the atranscription pattern: move deployment selection and the
litellm.aspeech call into a private _aspeech method, then have the
public aspeech set kwargs["original_function"] = self._aspeech and
await self.async_function_with_fallbacks(**kwargs). _aspeech also picks
up the shared _get_async_openai_model_client helper and the same
total/success/fail call accounting the sibling endpoints use

Fixes BerriAI#27778.

* fix(router): apply deployment kwargs and rpm semaphore in _aspeech

Bring _aspeech fully in line with _atranscription: call
_update_kwargs_with_deployment so deployment metadata, model_info,
timeout, and default litellm params flow into the request, and wrap
the litellm.aspeech call with the max_parallel_requests semaphore plus
async_routing_strategy_pre_call_checks so TTS respects rpm limits the
same way the other router endpoints do

Also add a unit test that exercises _aspeech directly and asserts the
deployment metadata reaches the underlying call

* fix(slack_alerting): stop false-positive hanging request alerts for requests below the alerting threshold (BerriAI#30106)

* fix(slack_alerting): skip hanging request alerts below the threshold

The hanging request check alerted on any cached request whose
completion status was not yet recorded, with no minimum age check.
Since the background loop runs every alerting_threshold / 2 seconds,
any request that happened to be in flight at a check fired a
"hanging - Ns+ request time" alert even if it was only seconds old,
producing a steady stream of false positives.

Add a created_at timestamp to HangingRequestData, stamped when the
request enters the hanging request cache, and skip requests younger
than alerting_threshold without evicting them, so a later check can
still alert if they never complete. Extend the cache TTL from
threshold + 60s to 1.5x threshold + 60s; with the age check, entries
only become alertable after threshold seconds, and the check period
is threshold / 2, so the old TTL could evict a genuinely hanging
request before any check saw it cross the threshold.

Fixes BerriAI#27855.

* fix(slack_alerting): alert once per hanging request

The min-age gate stops false positives for young in-flight requests, but
a genuinely hanging request still re-alerted on every checker tick within
the cache TTL. With the wider TTL (1.5x threshold + 60s) that is 1-2 extra
Slack notifications per stuck request at the default 600s threshold.

Flag a HangingRequestData entry as alerted once its alert fires and skip
flagged entries on later ticks, so each hang produces exactly one alert.
The cache reference is mutated in place, so the TTL is untouched and still
handles cleanup. Adds a regression test asserting one alert across multiple
ticks.

Fixes BerriAI#27855.

* fix(health): treat all-proxy-models keys as unrestricted in /health (BerriAI#30087)

* fix(health): treat all-proxy-models keys as unrestricted in /health

A key granted all model permissions stores the literal
"all-proxy-models" marker in its models list. The /health access
filter compared that marker against real model_names, so the model
list filtered down to nothing and the WebUI health check returned
healthy_count=0, unhealthy_count=0 with HTTP 503. Skip the filter
(both the live path and the background-cache model_id scoping) when
the marker is present, matching how auth_checks treats
SpecialModelNames.all_proxy_models.

Fixes BerriAI#29744.

* fix(health): resolve all-team-models sentinel to the team allowlist

Same failure shape as the all-proxy-models case: a key carrying the
literal "all-team-models" entry matches no real model_name, so the
/health access filter would zero out the model list. Resolve the
sentinel to the key's team models when team_id is set, matching
get_key_models in model_checks.py. Without a team_id the sentinel
stays unresolved and matches nothing, denying rather than widening
access, mirroring _resolve_key_models_for_auth_check.

* feat(proxy): auto-enable drop_params for Claude Code requests (BerriAI#30218)

* feat(proxy): auto-enable drop_params for Claude Code requests

Claude Code identifies itself with a claude-cli/<version> user agent and
sends Anthropic-specific params (top_k, thinking, etc.) on every request.
When the proxy routes those requests to a non-Anthropic provider, the
unsupported params fail the call unless drop_params is configured. Detect
the Claude Code user agent in add_litellm_data_to_request and default
drop_params to true for those requests, without overriding an explicit
drop_params value sent by the caller.

* feat(proxy): respect operator litellm_settings drop_params over Claude Code default

An explicit drop_params in the operator's litellm_settings (true or false)
now suppresses the Claude Code user agent default, so an operator who
deliberately configured drop_params: false keeps strict param validation
for Claude Code clients too. The auto-default only fills the gap when
neither the request body nor the config sets a value.

* fix(snowflake): migrate to native endpoints with auto-routing for Claude models (BerriAI#29964)

* fix(snowflake): migrate to native Cortex REST API endpoints

Replaces the legacy /api/v2/cortex/inference:complete endpoint with the
native OpenAI-compatible /api/v2/cortex/v1/chat/completions endpoint,
fixing error 390142 (Incoming request does not contain a valid payload)
when using model: snowflake/<model> in LiteLLM proxy.

Changes:
- litellm/llms/snowflake/chat/transformation.py: route to native
  /cortex/v1/chat/completions, remove Snowflake-specific tool_spec
  payload transformation, remove content_list response handling,
  add stream to supported params
- litellm/llms/snowflake/anthropic/transformation.py (new):
  SnowflakeCortexAnthropicConfig routes Claude models to /cortex/v1/messages
  with anthropic-version header and Anthropic->OpenAI response transform
- tests: 29 unit tests covering URL routing, auth headers, payload
  format, and response parsing

* fix(snowflake): map max_tokens to max_completion_tokens for native endpoint

* fix: handle multi-turn tool conversations and OpenAI→Anthropic tool format conversion

- _extract_system_and_messages now preserves tool_calls from assistant messages
  and converts them to Anthropic tool_use content blocks
- tool role messages are converted to user role with tool_result content blocks
  (as required by Anthropic Messages API)
- Added _transform_tools_to_anthropic() to convert OpenAI tool format
  (type/function/parameters) to Anthropic format (name/input_schema)
- Added comprehensive tests for multi-turn tool conversations

Addresses review feedback on PR BerriAI#29964

* test: add coverage for malformed JSON and non-string tool arguments

* fix(tests): update chat transformation tests for native OpenAI-compatible endpoint

* style: apply black formatting

* fix: resolve mypy type errors in anthropic transformation

* fix: correct mypy type: ignore error codes (attr-defined)

* fix: use max_tokens instead of max_completion_tokens for Snowflake endpoint compatibility

* refactor: merge Anthropic config into unified SnowflakeConfig with auto-routing

- Remove separate SnowflakeCortexAnthropicConfig and anthropic/ directory
- SnowflakeConfig now auto-routes based on model name:
  - Claude models → /messages endpoint (Anthropic format)
  - All others → /chat/completions endpoint (OpenAI format)
- No new provider needed (stays as SNOWFLAKE = 'snowflake')
- Tool message transformation for Claude: tool_calls → tool_use blocks,
  tool role → user with tool_result
- OpenAI → Anthropic tool format conversion (parameters → input_schema)
- Addresses Greptile feedback about unwired SnowflakeCortexAnthropicConfig

* fix: use max_completion_tokens for /chat/completions (Snowflake deprecated max_tokens on this endpoint)

* fix(tests): update assertions for Claude auto-routing to /messages endpoint

* fix(snowflake): add tool_choice conversion and preserve max_completion_tokens in Anthropic path

* fix(snowflake): use ChatCompletionMessageToolCall objects and strip model prefix on OpenAI path

* fix(snowflake): collect multiple system messages to prevent guardrail override

* chore: remove committed .pyc files and add __pycache__ to .gitignore

* fix: remove unused Union import

* fix: restore original .gitignore (accidentally replaced in earlier commit)

* feat(snowflake): add streaming response handler for both Anthropic and OpenAI SSE formats

* fix: remove unused AsyncIterator and Iterator imports

* fix: add missing total_tokens to ChatCompletionUsageBlock

* fix(snowflake): coalesce consecutive tool results into single user message for Anthropic

* fix(snowflake): handle message_start event for streaming input_tokens tracking

* fix: evict last deleted model in multi-instance deployments (BerriAI#28608)

* fix: evict last deleted model in multi-instance deployments

_delete_deployment had an early return when db_models was empty,
preventing eviction of the last deleted model during reconciliation.

- Remove len(db_models)==0 early return from _delete_deployment
- Return None (not []) from _get_models_from_db on DB failure so
  callers can distinguish a transient failure from a genuinely empty DB
- Guard _update_llm_router against None to skip updates on DB failure

Fixes BerriAI#28443

* test: remove dead MagicMock assignment in type_mismatch test

* fix: update test to pass [] not None to _update_llm_router

test_ProxyConfig__update_llm_router_bad_proxy_logging_raises was passing
None as new_models to get through to the proxy_logging_obj check, but
the None guard we added now returns early before reaching that path.
Pass [] instead so the test exercises the intended AttributeError case.

Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com>

* chore: regenerate API types to sync schema.d.ts with proxy OpenAPI spec

Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com>

---------

Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com>

* fix: invalidate Redis spend counter on /key/reset_spend (BerriAI#29694)

* fix: set Redis spend counter to reset_to value on /key/reset_spend

Previously, the Redis spend counter was always set to 0.0 after a reset,
even when reset_to was a non-zero value (partial reset). This caused
the budget to be under-enforced for up to 60 seconds until the counter
expired and fell through to the DB.

Now the counter is set to the actual reset_to value, so partial resets
are reflected correctly and budget enforcement is consistent.

* test: update reset_key_spend test to match direct cache set

The implementation now sets spend_counter_cache directly instead of
calling _invalidate_spend_counter. Update the test to verify the
in_memory_cache.set_cache call with the correct key, value, and ttl.

---------

Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>

* fix: add scaleway models pricing (BerriAI#27659)

* fix: Add embeddings support for Scaleway provider

* fix: resolve merge conflicts

* fix(main): clarify backend route handling for Swagger static assets (BerriAI#30196)

* fix(main): clarify backend route handling for Swagger static assets

* fix(allowlist): add BACKEND_MOUNT_PATHS for Swagger static assets

* fix(voyage): route multimodal embeddings to correct endpoint (BerriAI#30193)

* fix(voyage): route multimodal embeddings to correct endpoint

* test(voyage): cover multimodal embedding edge cases

* test(voyage): cover api key fallback

* fix(voyage): raise early on missing api key and malformed image url

* test(voyage): cover utils routing and helper

* fix(voyage): route supported openai params for multimodal models

* style: apply black formatting

* fix(ui): infer Azure API version from API base (BerriAI#30204)

* fix(ui): infer Azure API version from API base

* fix(ui): address Azure API version feedback

* Update litellm/llms/snowflake/chat/transformation.py

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* feat(datadog): add team-scoped Datadog callback support (BerriAI#29947)

Enable teams to configure their own Datadog credentials via
POST /team/{team_id}/callback, following the same pattern as Langfuse.

* Merge pull request BerriAI#29528 from aanchal22/litellm_byok-alias-merge

fix(proxy): atomic merge for team model aliases and team.models on BYOK create

* feat: add EmpirioLabs as an OpenAI-compatible provider (BerriAI#30278)

Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com>

* fix: resolve failing tests and lint in snowflake/team endpoints

- Black-format snowflake/chat/transformation.py to fix lint failure
- Update Anthropic config test to expect default max_tokens of 4096 (matches implementation)
- Add AsyncMock + execute_raw mock to team_model_add cache-refresh pin test
- Add model_dump mock and patch cache/logging in test_uses_atomic_array_append_with_dedup

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(test): update test_db_error_new_model_check for new _delete_deployment logic

_delete_deployment no longer short-circuits on empty db_models — it now
treats [] as a valid empty-DB state and proceeds to check config models.
Mock get_config to return the two router deployments so they appear in
combined_id_list and are protected, which matches the real-world scenario
where a DB error occurs but the models are config-backed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list (BerriAI#30295)

* feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list

Follow-up to BerriAI#30223 per maintainer review: documents the flag in
ConfigGeneralSettings with a short description and adds it to
allowed_args in get_config_list so the UI and /config/list expose it.
A test pins that /config/list returns the field with type Boolean,
which requires both registrations to be present

* chore(ui): regenerate schema.d.ts for cancel_on_disconnect

---------

Co-authored-by: kursad <kursad.lacin@brado.net>

* fix(datadog): never fall back to env DD_API_KEY for caller-supplied destinations

Team/key-scoped Datadog loggers could be pointed at an arbitrary dd_agent_host or
dd_site while omitting dd_api_key, causing the proxy's global DD_API_KEY to be sent
as the DD-API-KEY header to that destination. Gate the env-var fallback behind an
allow_env_credentials flag, set to False when the destination is caller-supplied,
mirroring the existing langfuse/langsmith pattern.

---------

Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com>
Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com>
Co-authored-by: daitran-tensormesh <dai@tensormesh.ai>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Muspi Merol <me@promplate.dev>
Co-authored-by: fangkang <fangkangm@gmail.com>
Co-authored-by: Chris Hoogeboom <chris.hoogeboom@gmail.com>
Co-authored-by: kursadlacin <kursadlacin@gmail.com>
Co-authored-by: kursad <kursad.lacin@brado.net>
Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com>
Co-authored-by: hcl <chenglunhu@gmail.com>
Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: sfc-gh-nashukla <navnit.shukla@snowflake.com>
Co-authored-by: Rudra Dudhat <contact.rdudhat@gmail.com>
Co-authored-by: Michael <52305679+michaelxer@users.noreply.github.com>
Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>
Co-authored-by: Quentin Champenois <26109239+Quentinchampenois@users.noreply.github.com>
Co-authored-by: mauriceberentsen <mauriceberentsen@live.nl>
Co-authored-by: lost9999 <56498264+lost9999@users.noreply.github.com>
Co-authored-by: GaetanVDB07 <86427581+GaetanVDB07@users.noreply.github.com>
Co-authored-by: Aanchal Khandelwal <aan2210khandelwal@gmail.com>
Co-authored-by: Adam Dalloul <adam_dalloul@icloud.com>
Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
michaelxer added a commit to michaelxer/litellm that referenced this pull request Jun 17, 2026
* feat(bedrock): add bedrock mantle gemma 4 models (BerriAI#30264)

* feat(bedrock): add bedrock mantle gemma 4 models

* test(bedrock): harden mantle local cost fixture

* feat(responses): enable the responses API for the Tensormesh provider (BerriAI#30209)

* feat(responses): enable the responses API for the Tensormesh provider

* Update litellm/llms/openai_like/providers.json

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

---------

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* fix(langfuse_otel): mark LLM spans as generations (BerriAI#30250)

* fix(bedrock): stop stream_chunk_size leaking into invoke request bodies (BerriAI#30240)

stream_chunk_size is a LiteLLM-internal knob for re-chunking the HTTP
response stream. The invoke transformations splat optional_params into the
provider request body without dropping it, and Bedrock rejects unknown
fields, so any bedrock/invoke request that sets the parameter fails with
ValidationException: stream_chunk_size: Extra inputs are not permitted.
Drop it in the invoke dispatcher (covers cohere, titan, mistral, meta,
ai21) and in the Claude messages-format request builder (the route used
for bedrock/invoke Anthropic models)

* fix(bedrock): stop buffering streamed tool-call argument deltas (BerriAI#30231)

* fix(bedrock): stop buffering streamed tool-call argument deltas

Two issues made Bedrock tool-use streaming arrive as a single end-of-stream
burst through LiteLLM while plain text streamed fine.

First, the anthropic-beta allowlist mapped fine-grained-tool-streaming-2025-05-14
to null for bedrock and bedrock_converse, so the header was silently stripped.
Without that beta, Anthropic models on Bedrock buffer tool input server-side and
emit all toolUse.input deltas at once (verified against converse-stream and
invoke-with-response-stream directly). Bedrock accepts the beta via
additionalModelRequestFields.anthropic_beta, so it is now forwarded.

Second, the streaming reads re-chunked the AWS event stream with
iter_bytes(chunk_size=1024). httpx's ByteChunker only releases full 1024-byte
blocks, so the small early events (messageStart, contentBlockStart, first
deltas) sat in the buffer until enough bytes accumulated, pushing
time-to-first-byte from ~1.4s to ~8.5s on buffered tool-use streams. The
default is now no re-chunking; an explicit stream_chunk_size is still honored.

* test(bedrock): cover explicit stream_chunk_size on sync invoke path

* test(bedrock): cover stream_chunk_size plumbing through converse completion

* test(bedrock): cover stream_chunk_size default in legacy BedrockLLM streaming

* test(bedrock): merge converse handler tests into existing mapped test file

pytest imports test modules by basename in non-package test dirs, so the new
tests/test_litellm/llms/bedrock/chat/test_converse_handler.py collided with
the pre-existing tests/test_litellm/llms/chat/test_converse_handler.py and
broke collection in CI. Move the new tests into the existing file

* feat(otel): emit v2 cost breakdown + stamp tracer scope version (BerriAI#30156)

Read the StandardLoggingPayload cost_breakdown into a typed LLMCost on
LLMCallSpanData and emit each component under litellm.cost.* (absent
components omitted, so spans stay sparse). Stamp litellm.__version__ as
the instrumentation scope version so every v2 span carries a
deterministic scope.version.

Tests under tests/test_litellm/integrations/otel/.

* fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in) (BerriAI#30223)

* fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in)

On the non-streaming path, base_process_llm_request awaited the LLM call
with no disconnect monitoring; when the HTTP client went away the
upstream request kept running until completion or request_timeout (6000s
default), holding a backend slot (e.g. a vLLM GPU slot) for output
nobody would read

Add an opt-in general_settings.cancel_on_disconnect flag, default off,
so the default code path is unchanged. When enabled, a receive-based
watcher task observes http.disconnect and cancels the asyncio.gather
driving the upstream call. The resulting CancelledError is converted to
HTTPException 499 only when the disconnect event is set, so
server-initiated cancellations still propagate as-is. The 499 then flows
through _handle_llm_api_exception like any other failure, meaning
post_call_failure_hook still releases max_parallel_requests slots and
fires spend and alerting callbacks; it is logged at info level instead
of a full traceback

Also removes the dead check_request_disconnection helper in
proxy_server.py (zero call sites) along with its behavior-pin tests

Builds on the receive-based design from BerriAI#25776

Addresses BerriAI#13774. Re-fixes BerriAI#22805 (regressed after the BerriAI#14295 revert)

Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com>

* fix(proxy): scope 499 quiet logging to disconnects and harden watcher

Address the two P2 findings from the Greptile review on BerriAI#30223. The
info-level logging in _log_llm_api_exception now applies only to the
disconnect-specific HTTPException (status 499 plus the shared
_CLIENT_DISCONNECT_DETAIL message), so any other 499 raised by hooks or
guardrails keeps its full traceback. The disconnect watcher now catches
exceptions from request.receive() (e.g. a transport reset) and logs a
warning instead of dying silently, making the degradation to no-op
visible; a test pins that the LLM call is not cancelled in that case

---------

Co-authored-by: kursad <kursad.lacin@brado.net>
Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com>

* fix(bedrock): grant aws-external-anthropic:* in OIDC session policy for claude_platform (BerriAI#30200) (BerriAI#30205)

The inline STS session policy passed to assume_role_with_web_identity
acts as an IAM PERMISSION CEILING — effective permissions are the
intersection of the role's identity policies and this policy. Any
action not listed is silently denied even when the IAM role grants it.

BerriAI#27678 added the bedrock/claude_platform/<model> route but its
service-side action namespace is aws-external-anthropic:*, not
bedrock:*. Without a matching statement here, every claude_platform
request via OIDC (GCP federation, EKS Pod Identity webhook, etc.) 403s
with 'no session policy allows the aws-external-anthropic:CreateInference
action' — even with a fully permissive identity policy.

Add a second ClaudePlatformLiteLLM statement covering CreateInference,
CreateBatchInference, CancelBatchInference, DeleteBatchInference,
CountTokens, Get*, List*. Keep aws:SecureTransport=true parity with the
bedrock statement.

Static creds + IRSA flow through different code paths and are not
affected.

Fixes BerriAI#30200

* fix(proxy): set Retry-After header on RouterRateLimitError 429 responses (BerriAI#30098)

* Set Retry-After header on RouterRateLimitError responses

When all deployments for a model are in cooldown, the proxy returns a
429 whose cooldown timing is only available by parsing the error
message string. RouterRateLimitError already carries cooldown_time, so
expose it as a standard retry-after header in
_handle_llm_api_exception. The value is rounded up so clients never
retry before the cooldown window ends.

Fixes BerriAI#27823.

* Set Retry-After after response-headers hook so cooldown wins

The cooldown-derived retry-after was assigned before the
post_call_response_headers_hook merge, so a callback returning a
retry-after key (including a stale or empty value) silently clobbered
it. Move the RouterRateLimitError block after the callback merge so the
cooldown value is authoritative for this error type.

* fix(router): route aspeech through async_function_with_fallbacks (BerriAI#30104)

* fix(router): route aspeech through async_function_with_fallbacks

Router.aspeech selected a deployment and awaited litellm.aspeech
directly, so TTS requests got no retry on failure and no failover to
backup deployments; the except block only fired an exception alert and
re-raised. Every other router endpoint (acompletion, aembedding,
atranscription, arerank) already delegates to
async_function_with_fallbacks

Mirror the atranscription pattern: move deployment selection and the
litellm.aspeech call into a private _aspeech method, then have the
public aspeech set kwargs["original_function"] = self._aspeech and
await self.async_function_with_fallbacks(**kwargs). _aspeech also picks
up the shared _get_async_openai_model_client helper and the same
total/success/fail call accounting the sibling endpoints use

Fixes BerriAI#27778.

* fix(router): apply deployment kwargs and rpm semaphore in _aspeech

Bring _aspeech fully in line with _atranscription: call
_update_kwargs_with_deployment so deployment metadata, model_info,
timeout, and default litellm params flow into the request, and wrap
the litellm.aspeech call with the max_parallel_requests semaphore plus
async_routing_strategy_pre_call_checks so TTS respects rpm limits the
same way the other router endpoints do

Also add a unit test that exercises _aspeech directly and asserts the
deployment metadata reaches the underlying call

* fix(slack_alerting): stop false-positive hanging request alerts for requests below the alerting threshold (BerriAI#30106)

* fix(slack_alerting): skip hanging request alerts below the threshold

The hanging request check alerted on any cached request whose
completion status was not yet recorded, with no minimum age check.
Since the background loop runs every alerting_threshold / 2 seconds,
any request that happened to be in flight at a check fired a
"hanging - Ns+ request time" alert even if it was only seconds old,
producing a steady stream of false positives.

Add a created_at timestamp to HangingRequestData, stamped when the
request enters the hanging request cache, and skip requests younger
than alerting_threshold without evicting them, so a later check can
still alert if they never complete. Extend the cache TTL from
threshold + 60s to 1.5x threshold + 60s; with the age check, entries
only become alertable after threshold seconds, and the check period
is threshold / 2, so the old TTL could evict a genuinely hanging
request before any check saw it cross the threshold.

Fixes BerriAI#27855.

* fix(slack_alerting): alert once per hanging request

The min-age gate stops false positives for young in-flight requests, but
a genuinely hanging request still re-alerted on every checker tick within
the cache TTL. With the wider TTL (1.5x threshold + 60s) that is 1-2 extra
Slack notifications per stuck request at the default 600s threshold.

Flag a HangingRequestData entry as alerted once its alert fires and skip
flagged entries on later ticks, so each hang produces exactly one alert.
The cache reference is mutated in place, so the TTL is untouched and still
handles cleanup. Adds a regression test asserting one alert across multiple
ticks.

Fixes BerriAI#27855.

* fix(health): treat all-proxy-models keys as unrestricted in /health (BerriAI#30087)

* fix(health): treat all-proxy-models keys as unrestricted in /health

A key granted all model permissions stores the literal
"all-proxy-models" marker in its models list. The /health access
filter compared that marker against real model_names, so the model
list filtered down to nothing and the WebUI health check returned
healthy_count=0, unhealthy_count=0 with HTTP 503. Skip the filter
(both the live path and the background-cache model_id scoping) when
the marker is present, matching how auth_checks treats
SpecialModelNames.all_proxy_models.

Fixes BerriAI#29744.

* fix(health): resolve all-team-models sentinel to the team allowlist

Same failure shape as the all-proxy-models case: a key carrying the
literal "all-team-models" entry matches no real model_name, so the
/health access filter would zero out the model list. Resolve the
sentinel to the key's team models when team_id is set, matching
get_key_models in model_checks.py. Without a team_id the sentinel
stays unresolved and matches nothing, denying rather than widening
access, mirroring _resolve_key_models_for_auth_check.

* feat(proxy): auto-enable drop_params for Claude Code requests (BerriAI#30218)

* feat(proxy): auto-enable drop_params for Claude Code requests

Claude Code identifies itself with a claude-cli/<version> user agent and
sends Anthropic-specific params (top_k, thinking, etc.) on every request.
When the proxy routes those requests to a non-Anthropic provider, the
unsupported params fail the call unless drop_params is configured. Detect
the Claude Code user agent in add_litellm_data_to_request and default
drop_params to true for those requests, without overriding an explicit
drop_params value sent by the caller.

* feat(proxy): respect operator litellm_settings drop_params over Claude Code default

An explicit drop_params in the operator's litellm_settings (true or false)
now suppresses the Claude Code user agent default, so an operator who
deliberately configured drop_params: false keeps strict param validation
for Claude Code clients too. The auto-default only fills the gap when
neither the request body nor the config sets a value.

* fix(snowflake): migrate to native endpoints with auto-routing for Claude models (BerriAI#29964)

* fix(snowflake): migrate to native Cortex REST API endpoints

Replaces the legacy /api/v2/cortex/inference:complete endpoint with the
native OpenAI-compatible /api/v2/cortex/v1/chat/completions endpoint,
fixing error 390142 (Incoming request does not contain a valid payload)
when using model: snowflake/<model> in LiteLLM proxy.

Changes:
- litellm/llms/snowflake/chat/transformation.py: route to native
  /cortex/v1/chat/completions, remove Snowflake-specific tool_spec
  payload transformation, remove content_list response handling,
  add stream to supported params
- litellm/llms/snowflake/anthropic/transformation.py (new):
  SnowflakeCortexAnthropicConfig routes Claude models to /cortex/v1/messages
  with anthropic-version header and Anthropic->OpenAI response transform
- tests: 29 unit tests covering URL routing, auth headers, payload
  format, and response parsing

* fix(snowflake): map max_tokens to max_completion_tokens for native endpoint

* fix: handle multi-turn tool conversations and OpenAI→Anthropic tool format conversion

- _extract_system_and_messages now preserves tool_calls from assistant messages
  and converts them to Anthropic tool_use content blocks
- tool role messages are converted to user role with tool_result content blocks
  (as required by Anthropic Messages API)
- Added _transform_tools_to_anthropic() to convert OpenAI tool format
  (type/function/parameters) to Anthropic format (name/input_schema)
- Added comprehensive tests for multi-turn tool conversations

Addresses review feedback on PR BerriAI#29964

* test: add coverage for malformed JSON and non-string tool arguments

* fix(tests): update chat transformation tests for native OpenAI-compatible endpoint

* style: apply black formatting

* fix: resolve mypy type errors in anthropic transformation

* fix: correct mypy type: ignore error codes (attr-defined)

* fix: use max_tokens instead of max_completion_tokens for Snowflake endpoint compatibility

* refactor: merge Anthropic config into unified SnowflakeConfig with auto-routing

- Remove separate SnowflakeCortexAnthropicConfig and anthropic/ directory
- SnowflakeConfig now auto-routes based on model name:
  - Claude models → /messages endpoint (Anthropic format)
  - All others → /chat/completions endpoint (OpenAI format)
- No new provider needed (stays as SNOWFLAKE = 'snowflake')
- Tool message transformation for Claude: tool_calls → tool_use blocks,
  tool role → user with tool_result
- OpenAI → Anthropic tool format conversion (parameters → input_schema)
- Addresses Greptile feedback about unwired SnowflakeCortexAnthropicConfig

* fix: use max_completion_tokens for /chat/completions (Snowflake deprecated max_tokens on this endpoint)

* fix(tests): update assertions for Claude auto-routing to /messages endpoint

* fix(snowflake): add tool_choice conversion and preserve max_completion_tokens in Anthropic path

* fix(snowflake): use ChatCompletionMessageToolCall objects and strip model prefix on OpenAI path

* fix(snowflake): collect multiple system messages to prevent guardrail override

* chore: remove committed .pyc files and add __pycache__ to .gitignore

* fix: remove unused Union import

* fix: restore original .gitignore (accidentally replaced in earlier commit)

* feat(snowflake): add streaming response handler for both Anthropic and OpenAI SSE formats

* fix: remove unused AsyncIterator and Iterator imports

* fix: add missing total_tokens to ChatCompletionUsageBlock

* fix(snowflake): coalesce consecutive tool results into single user message for Anthropic

* fix(snowflake): handle message_start event for streaming input_tokens tracking

* fix: evict last deleted model in multi-instance deployments (BerriAI#28608)

* fix: evict last deleted model in multi-instance deployments

_delete_deployment had an early return when db_models was empty,
preventing eviction of the last deleted model during reconciliation.

- Remove len(db_models)==0 early return from _delete_deployment
- Return None (not []) from _get_models_from_db on DB failure so
  callers can distinguish a transient failure from a genuinely empty DB
- Guard _update_llm_router against None to skip updates on DB failure

Fixes BerriAI#28443

* test: remove dead MagicMock assignment in type_mismatch test

* fix: update test to pass [] not None to _update_llm_router

test_ProxyConfig__update_llm_router_bad_proxy_logging_raises was passing
None as new_models to get through to the proxy_logging_obj check, but
the None guard we added now returns early before reaching that path.
Pass [] instead so the test exercises the intended AttributeError case.

Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com>

* chore: regenerate API types to sync schema.d.ts with proxy OpenAPI spec

Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com>

---------

Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com>

* fix: invalidate Redis spend counter on /key/reset_spend (BerriAI#29694)

* fix: set Redis spend counter to reset_to value on /key/reset_spend

Previously, the Redis spend counter was always set to 0.0 after a reset,
even when reset_to was a non-zero value (partial reset). This caused
the budget to be under-enforced for up to 60 seconds until the counter
expired and fell through to the DB.

Now the counter is set to the actual reset_to value, so partial resets
are reflected correctly and budget enforcement is consistent.

* test: update reset_key_spend test to match direct cache set

The implementation now sets spend_counter_cache directly instead of
calling _invalidate_spend_counter. Update the test to verify the
in_memory_cache.set_cache call with the correct key, value, and ttl.

---------

Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>

* fix: add scaleway models pricing (BerriAI#27659)

* fix: Add embeddings support for Scaleway provider

* fix: resolve merge conflicts

* fix(main): clarify backend route handling for Swagger static assets (BerriAI#30196)

* fix(main): clarify backend route handling for Swagger static assets

* fix(allowlist): add BACKEND_MOUNT_PATHS for Swagger static assets

* fix(voyage): route multimodal embeddings to correct endpoint (BerriAI#30193)

* fix(voyage): route multimodal embeddings to correct endpoint

* test(voyage): cover multimodal embedding edge cases

* test(voyage): cover api key fallback

* fix(voyage): raise early on missing api key and malformed image url

* test(voyage): cover utils routing and helper

* fix(voyage): route supported openai params for multimodal models

* style: apply black formatting

* fix(ui): infer Azure API version from API base (BerriAI#30204)

* fix(ui): infer Azure API version from API base

* fix(ui): address Azure API version feedback

* Update litellm/llms/snowflake/chat/transformation.py

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* feat(datadog): add team-scoped Datadog callback support (BerriAI#29947)

Enable teams to configure their own Datadog credentials via
POST /team/{team_id}/callback, following the same pattern as Langfuse.

* Merge pull request BerriAI#29528 from aanchal22/litellm_byok-alias-merge

fix(proxy): atomic merge for team model aliases and team.models on BYOK create

* feat: add EmpirioLabs as an OpenAI-compatible provider (BerriAI#30278)

Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com>

* fix: resolve failing tests and lint in snowflake/team endpoints

- Black-format snowflake/chat/transformation.py to fix lint failure
- Update Anthropic config test to expect default max_tokens of 4096 (matches implementation)
- Add AsyncMock + execute_raw mock to team_model_add cache-refresh pin test
- Add model_dump mock and patch cache/logging in test_uses_atomic_array_append_with_dedup

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(test): update test_db_error_new_model_check for new _delete_deployment logic

_delete_deployment no longer short-circuits on empty db_models — it now
treats [] as a valid empty-DB state and proceeds to check config models.
Mock get_config to return the two router deployments so they appear in
combined_id_list and are protected, which matches the real-world scenario
where a DB error occurs but the models are config-backed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list (BerriAI#30295)

* feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list

Follow-up to BerriAI#30223 per maintainer review: documents the flag in
ConfigGeneralSettings with a short description and adds it to
allowed_args in get_config_list so the UI and /config/list expose it.
A test pins that /config/list returns the field with type Boolean,
which requires both registrations to be present

* chore(ui): regenerate schema.d.ts for cancel_on_disconnect

---------

Co-authored-by: kursad <kursad.lacin@brado.net>

* fix(datadog): never fall back to env DD_API_KEY for caller-supplied destinations

Team/key-scoped Datadog loggers could be pointed at an arbitrary dd_agent_host or
dd_site while omitting dd_api_key, causing the proxy's global DD_API_KEY to be sent
as the DD-API-KEY header to that destination. Gate the env-var fallback behind an
allow_env_credentials flag, set to False when the destination is caller-supplied,
mirroring the existing langfuse/langsmith pattern.

---------

Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com>
Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com>
Co-authored-by: daitran-tensormesh <dai@tensormesh.ai>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Muspi Merol <me@promplate.dev>
Co-authored-by: fangkang <fangkangm@gmail.com>
Co-authored-by: Chris Hoogeboom <chris.hoogeboom@gmail.com>
Co-authored-by: kursadlacin <kursadlacin@gmail.com>
Co-authored-by: kursad <kursad.lacin@brado.net>
Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com>
Co-authored-by: hcl <chenglunhu@gmail.com>
Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: sfc-gh-nashukla <navnit.shukla@snowflake.com>
Co-authored-by: Rudra Dudhat <contact.rdudhat@gmail.com>
Co-authored-by: Michael <52305679+michaelxer@users.noreply.github.com>
Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>
Co-authored-by: Quentin Champenois <26109239+Quentinchampenois@users.noreply.github.com>
Co-authored-by: mauriceberentsen <mauriceberentsen@live.nl>
Co-authored-by: lost9999 <56498264+lost9999@users.noreply.github.com>
Co-authored-by: GaetanVDB07 <86427581+GaetanVDB07@users.noreply.github.com>
Co-authored-by: Aanchal Khandelwal <aan2210khandelwal@gmail.com>
Co-authored-by: Adam Dalloul <adam_dalloul@icloud.com>
Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
koladefaj pushed a commit to koladefaj/litellm that referenced this pull request Jun 17, 2026
* feat(bedrock): add bedrock mantle gemma 4 models (BerriAI#30264)

* feat(bedrock): add bedrock mantle gemma 4 models

* test(bedrock): harden mantle local cost fixture

* feat(responses): enable the responses API for the Tensormesh provider (BerriAI#30209)

* feat(responses): enable the responses API for the Tensormesh provider

* Update litellm/llms/openai_like/providers.json

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

---------

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* fix(langfuse_otel): mark LLM spans as generations (BerriAI#30250)

* fix(bedrock): stop stream_chunk_size leaking into invoke request bodies (BerriAI#30240)

stream_chunk_size is a LiteLLM-internal knob for re-chunking the HTTP
response stream. The invoke transformations splat optional_params into the
provider request body without dropping it, and Bedrock rejects unknown
fields, so any bedrock/invoke request that sets the parameter fails with
ValidationException: stream_chunk_size: Extra inputs are not permitted.
Drop it in the invoke dispatcher (covers cohere, titan, mistral, meta,
ai21) and in the Claude messages-format request builder (the route used
for bedrock/invoke Anthropic models)

* fix(bedrock): stop buffering streamed tool-call argument deltas (BerriAI#30231)

* fix(bedrock): stop buffering streamed tool-call argument deltas

Two issues made Bedrock tool-use streaming arrive as a single end-of-stream
burst through LiteLLM while plain text streamed fine.

First, the anthropic-beta allowlist mapped fine-grained-tool-streaming-2025-05-14
to null for bedrock and bedrock_converse, so the header was silently stripped.
Without that beta, Anthropic models on Bedrock buffer tool input server-side and
emit all toolUse.input deltas at once (verified against converse-stream and
invoke-with-response-stream directly). Bedrock accepts the beta via
additionalModelRequestFields.anthropic_beta, so it is now forwarded.

Second, the streaming reads re-chunked the AWS event stream with
iter_bytes(chunk_size=1024). httpx's ByteChunker only releases full 1024-byte
blocks, so the small early events (messageStart, contentBlockStart, first
deltas) sat in the buffer until enough bytes accumulated, pushing
time-to-first-byte from ~1.4s to ~8.5s on buffered tool-use streams. The
default is now no re-chunking; an explicit stream_chunk_size is still honored.

* test(bedrock): cover explicit stream_chunk_size on sync invoke path

* test(bedrock): cover stream_chunk_size plumbing through converse completion

* test(bedrock): cover stream_chunk_size default in legacy BedrockLLM streaming

* test(bedrock): merge converse handler tests into existing mapped test file

pytest imports test modules by basename in non-package test dirs, so the new
tests/test_litellm/llms/bedrock/chat/test_converse_handler.py collided with
the pre-existing tests/test_litellm/llms/chat/test_converse_handler.py and
broke collection in CI. Move the new tests into the existing file

* feat(otel): emit v2 cost breakdown + stamp tracer scope version (BerriAI#30156)

Read the StandardLoggingPayload cost_breakdown into a typed LLMCost on
LLMCallSpanData and emit each component under litellm.cost.* (absent
components omitted, so spans stay sparse). Stamp litellm.__version__ as
the instrumentation scope version so every v2 span carries a
deterministic scope.version.

Tests under tests/test_litellm/integrations/otel/.

* fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in) (BerriAI#30223)

* fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in)

On the non-streaming path, base_process_llm_request awaited the LLM call
with no disconnect monitoring; when the HTTP client went away the
upstream request kept running until completion or request_timeout (6000s
default), holding a backend slot (e.g. a vLLM GPU slot) for output
nobody would read

Add an opt-in general_settings.cancel_on_disconnect flag, default off,
so the default code path is unchanged. When enabled, a receive-based
watcher task observes http.disconnect and cancels the asyncio.gather
driving the upstream call. The resulting CancelledError is converted to
HTTPException 499 only when the disconnect event is set, so
server-initiated cancellations still propagate as-is. The 499 then flows
through _handle_llm_api_exception like any other failure, meaning
post_call_failure_hook still releases max_parallel_requests slots and
fires spend and alerting callbacks; it is logged at info level instead
of a full traceback

Also removes the dead check_request_disconnection helper in
proxy_server.py (zero call sites) along with its behavior-pin tests

Builds on the receive-based design from BerriAI#25776

Addresses BerriAI#13774. Re-fixes BerriAI#22805 (regressed after the BerriAI#14295 revert)

Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com>

* fix(proxy): scope 499 quiet logging to disconnects and harden watcher

Address the two P2 findings from the Greptile review on BerriAI#30223. The
info-level logging in _log_llm_api_exception now applies only to the
disconnect-specific HTTPException (status 499 plus the shared
_CLIENT_DISCONNECT_DETAIL message), so any other 499 raised by hooks or
guardrails keeps its full traceback. The disconnect watcher now catches
exceptions from request.receive() (e.g. a transport reset) and logs a
warning instead of dying silently, making the degradation to no-op
visible; a test pins that the LLM call is not cancelled in that case

---------

Co-authored-by: kursad <kursad.lacin@brado.net>
Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com>

* fix(bedrock): grant aws-external-anthropic:* in OIDC session policy for claude_platform (BerriAI#30200) (BerriAI#30205)

The inline STS session policy passed to assume_role_with_web_identity
acts as an IAM PERMISSION CEILING — effective permissions are the
intersection of the role's identity policies and this policy. Any
action not listed is silently denied even when the IAM role grants it.

BerriAI#27678 added the bedrock/claude_platform/<model> route but its
service-side action namespace is aws-external-anthropic:*, not
bedrock:*. Without a matching statement here, every claude_platform
request via OIDC (GCP federation, EKS Pod Identity webhook, etc.) 403s
with 'no session policy allows the aws-external-anthropic:CreateInference
action' — even with a fully permissive identity policy.

Add a second ClaudePlatformLiteLLM statement covering CreateInference,
CreateBatchInference, CancelBatchInference, DeleteBatchInference,
CountTokens, Get*, List*. Keep aws:SecureTransport=true parity with the
bedrock statement.

Static creds + IRSA flow through different code paths and are not
affected.

Fixes BerriAI#30200

* fix(proxy): set Retry-After header on RouterRateLimitError 429 responses (BerriAI#30098)

* Set Retry-After header on RouterRateLimitError responses

When all deployments for a model are in cooldown, the proxy returns a
429 whose cooldown timing is only available by parsing the error
message string. RouterRateLimitError already carries cooldown_time, so
expose it as a standard retry-after header in
_handle_llm_api_exception. The value is rounded up so clients never
retry before the cooldown window ends.

Fixes BerriAI#27823.

* Set Retry-After after response-headers hook so cooldown wins

The cooldown-derived retry-after was assigned before the
post_call_response_headers_hook merge, so a callback returning a
retry-after key (including a stale or empty value) silently clobbered
it. Move the RouterRateLimitError block after the callback merge so the
cooldown value is authoritative for this error type.

* fix(router): route aspeech through async_function_with_fallbacks (BerriAI#30104)

* fix(router): route aspeech through async_function_with_fallbacks

Router.aspeech selected a deployment and awaited litellm.aspeech
directly, so TTS requests got no retry on failure and no failover to
backup deployments; the except block only fired an exception alert and
re-raised. Every other router endpoint (acompletion, aembedding,
atranscription, arerank) already delegates to
async_function_with_fallbacks

Mirror the atranscription pattern: move deployment selection and the
litellm.aspeech call into a private _aspeech method, then have the
public aspeech set kwargs["original_function"] = self._aspeech and
await self.async_function_with_fallbacks(**kwargs). _aspeech also picks
up the shared _get_async_openai_model_client helper and the same
total/success/fail call accounting the sibling endpoints use

Fixes BerriAI#27778.

* fix(router): apply deployment kwargs and rpm semaphore in _aspeech

Bring _aspeech fully in line with _atranscription: call
_update_kwargs_with_deployment so deployment metadata, model_info,
timeout, and default litellm params flow into the request, and wrap
the litellm.aspeech call with the max_parallel_requests semaphore plus
async_routing_strategy_pre_call_checks so TTS respects rpm limits the
same way the other router endpoints do

Also add a unit test that exercises _aspeech directly and asserts the
deployment metadata reaches the underlying call

* fix(slack_alerting): stop false-positive hanging request alerts for requests below the alerting threshold (BerriAI#30106)

* fix(slack_alerting): skip hanging request alerts below the threshold

The hanging request check alerted on any cached request whose
completion status was not yet recorded, with no minimum age check.
Since the background loop runs every alerting_threshold / 2 seconds,
any request that happened to be in flight at a check fired a
"hanging - Ns+ request time" alert even if it was only seconds old,
producing a steady stream of false positives.

Add a created_at timestamp to HangingRequestData, stamped when the
request enters the hanging request cache, and skip requests younger
than alerting_threshold without evicting them, so a later check can
still alert if they never complete. Extend the cache TTL from
threshold + 60s to 1.5x threshold + 60s; with the age check, entries
only become alertable after threshold seconds, and the check period
is threshold / 2, so the old TTL could evict a genuinely hanging
request before any check saw it cross the threshold.

Fixes BerriAI#27855.

* fix(slack_alerting): alert once per hanging request

The min-age gate stops false positives for young in-flight requests, but
a genuinely hanging request still re-alerted on every checker tick within
the cache TTL. With the wider TTL (1.5x threshold + 60s) that is 1-2 extra
Slack notifications per stuck request at the default 600s threshold.

Flag a HangingRequestData entry as alerted once its alert fires and skip
flagged entries on later ticks, so each hang produces exactly one alert.
The cache reference is mutated in place, so the TTL is untouched and still
handles cleanup. Adds a regression test asserting one alert across multiple
ticks.

Fixes BerriAI#27855.

* fix(health): treat all-proxy-models keys as unrestricted in /health (BerriAI#30087)

* fix(health): treat all-proxy-models keys as unrestricted in /health

A key granted all model permissions stores the literal
"all-proxy-models" marker in its models list. The /health access
filter compared that marker against real model_names, so the model
list filtered down to nothing and the WebUI health check returned
healthy_count=0, unhealthy_count=0 with HTTP 503. Skip the filter
(both the live path and the background-cache model_id scoping) when
the marker is present, matching how auth_checks treats
SpecialModelNames.all_proxy_models.

Fixes BerriAI#29744.

* fix(health): resolve all-team-models sentinel to the team allowlist

Same failure shape as the all-proxy-models case: a key carrying the
literal "all-team-models" entry matches no real model_name, so the
/health access filter would zero out the model list. Resolve the
sentinel to the key's team models when team_id is set, matching
get_key_models in model_checks.py. Without a team_id the sentinel
stays unresolved and matches nothing, denying rather than widening
access, mirroring _resolve_key_models_for_auth_check.

* feat(proxy): auto-enable drop_params for Claude Code requests (BerriAI#30218)

* feat(proxy): auto-enable drop_params for Claude Code requests

Claude Code identifies itself with a claude-cli/<version> user agent and
sends Anthropic-specific params (top_k, thinking, etc.) on every request.
When the proxy routes those requests to a non-Anthropic provider, the
unsupported params fail the call unless drop_params is configured. Detect
the Claude Code user agent in add_litellm_data_to_request and default
drop_params to true for those requests, without overriding an explicit
drop_params value sent by the caller.

* feat(proxy): respect operator litellm_settings drop_params over Claude Code default

An explicit drop_params in the operator's litellm_settings (true or false)
now suppresses the Claude Code user agent default, so an operator who
deliberately configured drop_params: false keeps strict param validation
for Claude Code clients too. The auto-default only fills the gap when
neither the request body nor the config sets a value.

* fix(snowflake): migrate to native endpoints with auto-routing for Claude models (BerriAI#29964)

* fix(snowflake): migrate to native Cortex REST API endpoints

Replaces the legacy /api/v2/cortex/inference:complete endpoint with the
native OpenAI-compatible /api/v2/cortex/v1/chat/completions endpoint,
fixing error 390142 (Incoming request does not contain a valid payload)
when using model: snowflake/<model> in LiteLLM proxy.

Changes:
- litellm/llms/snowflake/chat/transformation.py: route to native
  /cortex/v1/chat/completions, remove Snowflake-specific tool_spec
  payload transformation, remove content_list response handling,
  add stream to supported params
- litellm/llms/snowflake/anthropic/transformation.py (new):
  SnowflakeCortexAnthropicConfig routes Claude models to /cortex/v1/messages
  with anthropic-version header and Anthropic->OpenAI response transform
- tests: 29 unit tests covering URL routing, auth headers, payload
  format, and response parsing

* fix(snowflake): map max_tokens to max_completion_tokens for native endpoint

* fix: handle multi-turn tool conversations and OpenAI→Anthropic tool format conversion

- _extract_system_and_messages now preserves tool_calls from assistant messages
  and converts them to Anthropic tool_use content blocks
- tool role messages are converted to user role with tool_result content blocks
  (as required by Anthropic Messages API)
- Added _transform_tools_to_anthropic() to convert OpenAI tool format
  (type/function/parameters) to Anthropic format (name/input_schema)
- Added comprehensive tests for multi-turn tool conversations

Addresses review feedback on PR BerriAI#29964

* test: add coverage for malformed JSON and non-string tool arguments

* fix(tests): update chat transformation tests for native OpenAI-compatible endpoint

* style: apply black formatting

* fix: resolve mypy type errors in anthropic transformation

* fix: correct mypy type: ignore error codes (attr-defined)

* fix: use max_tokens instead of max_completion_tokens for Snowflake endpoint compatibility

* refactor: merge Anthropic config into unified SnowflakeConfig with auto-routing

- Remove separate SnowflakeCortexAnthropicConfig and anthropic/ directory
- SnowflakeConfig now auto-routes based on model name:
  - Claude models → /messages endpoint (Anthropic format)
  - All others → /chat/completions endpoint (OpenAI format)
- No new provider needed (stays as SNOWFLAKE = 'snowflake')
- Tool message transformation for Claude: tool_calls → tool_use blocks,
  tool role → user with tool_result
- OpenAI → Anthropic tool format conversion (parameters → input_schema)
- Addresses Greptile feedback about unwired SnowflakeCortexAnthropicConfig

* fix: use max_completion_tokens for /chat/completions (Snowflake deprecated max_tokens on this endpoint)

* fix(tests): update assertions for Claude auto-routing to /messages endpoint

* fix(snowflake): add tool_choice conversion and preserve max_completion_tokens in Anthropic path

* fix(snowflake): use ChatCompletionMessageToolCall objects and strip model prefix on OpenAI path

* fix(snowflake): collect multiple system messages to prevent guardrail override

* chore: remove committed .pyc files and add __pycache__ to .gitignore

* fix: remove unused Union import

* fix: restore original .gitignore (accidentally replaced in earlier commit)

* feat(snowflake): add streaming response handler for both Anthropic and OpenAI SSE formats

* fix: remove unused AsyncIterator and Iterator imports

* fix: add missing total_tokens to ChatCompletionUsageBlock

* fix(snowflake): coalesce consecutive tool results into single user message for Anthropic

* fix(snowflake): handle message_start event for streaming input_tokens tracking

* fix: evict last deleted model in multi-instance deployments (BerriAI#28608)

* fix: evict last deleted model in multi-instance deployments

_delete_deployment had an early return when db_models was empty,
preventing eviction of the last deleted model during reconciliation.

- Remove len(db_models)==0 early return from _delete_deployment
- Return None (not []) from _get_models_from_db on DB failure so
  callers can distinguish a transient failure from a genuinely empty DB
- Guard _update_llm_router against None to skip updates on DB failure

Fixes BerriAI#28443

* test: remove dead MagicMock assignment in type_mismatch test

* fix: update test to pass [] not None to _update_llm_router

test_ProxyConfig__update_llm_router_bad_proxy_logging_raises was passing
None as new_models to get through to the proxy_logging_obj check, but
the None guard we added now returns early before reaching that path.
Pass [] instead so the test exercises the intended AttributeError case.

Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com>

* chore: regenerate API types to sync schema.d.ts with proxy OpenAPI spec

Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com>

---------

Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com>

* fix: invalidate Redis spend counter on /key/reset_spend (BerriAI#29694)

* fix: set Redis spend counter to reset_to value on /key/reset_spend

Previously, the Redis spend counter was always set to 0.0 after a reset,
even when reset_to was a non-zero value (partial reset). This caused
the budget to be under-enforced for up to 60 seconds until the counter
expired and fell through to the DB.

Now the counter is set to the actual reset_to value, so partial resets
are reflected correctly and budget enforcement is consistent.

* test: update reset_key_spend test to match direct cache set

The implementation now sets spend_counter_cache directly instead of
calling _invalidate_spend_counter. Update the test to verify the
in_memory_cache.set_cache call with the correct key, value, and ttl.

---------

Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>

* fix: add scaleway models pricing (BerriAI#27659)

* fix: Add embeddings support for Scaleway provider

* fix: resolve merge conflicts

* fix(main): clarify backend route handling for Swagger static assets (BerriAI#30196)

* fix(main): clarify backend route handling for Swagger static assets

* fix(allowlist): add BACKEND_MOUNT_PATHS for Swagger static assets

* fix(voyage): route multimodal embeddings to correct endpoint (BerriAI#30193)

* fix(voyage): route multimodal embeddings to correct endpoint

* test(voyage): cover multimodal embedding edge cases

* test(voyage): cover api key fallback

* fix(voyage): raise early on missing api key and malformed image url

* test(voyage): cover utils routing and helper

* fix(voyage): route supported openai params for multimodal models

* style: apply black formatting

* fix(ui): infer Azure API version from API base (BerriAI#30204)

* fix(ui): infer Azure API version from API base

* fix(ui): address Azure API version feedback

* Update litellm/llms/snowflake/chat/transformation.py

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* feat(datadog): add team-scoped Datadog callback support (BerriAI#29947)

Enable teams to configure their own Datadog credentials via
POST /team/{team_id}/callback, following the same pattern as Langfuse.

* Merge pull request BerriAI#29528 from aanchal22/litellm_byok-alias-merge

fix(proxy): atomic merge for team model aliases and team.models on BYOK create

* feat: add EmpirioLabs as an OpenAI-compatible provider (BerriAI#30278)

Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com>

* fix: resolve failing tests and lint in snowflake/team endpoints

- Black-format snowflake/chat/transformation.py to fix lint failure
- Update Anthropic config test to expect default max_tokens of 4096 (matches implementation)
- Add AsyncMock + execute_raw mock to team_model_add cache-refresh pin test
- Add model_dump mock and patch cache/logging in test_uses_atomic_array_append_with_dedup

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(test): update test_db_error_new_model_check for new _delete_deployment logic

_delete_deployment no longer short-circuits on empty db_models — it now
treats [] as a valid empty-DB state and proceeds to check config models.
Mock get_config to return the two router deployments so they appear in
combined_id_list and are protected, which matches the real-world scenario
where a DB error occurs but the models are config-backed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list (BerriAI#30295)

* feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list

Follow-up to BerriAI#30223 per maintainer review: documents the flag in
ConfigGeneralSettings with a short description and adds it to
allowed_args in get_config_list so the UI and /config/list expose it.
A test pins that /config/list returns the field with type Boolean,
which requires both registrations to be present

* chore(ui): regenerate schema.d.ts for cancel_on_disconnect

---------

Co-authored-by: kursad <kursad.lacin@brado.net>

* fix(datadog): never fall back to env DD_API_KEY for caller-supplied destinations

Team/key-scoped Datadog loggers could be pointed at an arbitrary dd_agent_host or
dd_site while omitting dd_api_key, causing the proxy's global DD_API_KEY to be sent
as the DD-API-KEY header to that destination. Gate the env-var fallback behind an
allow_env_credentials flag, set to False when the destination is caller-supplied,
mirroring the existing langfuse/langsmith pattern.

---------

Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com>
Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com>
Co-authored-by: daitran-tensormesh <dai@tensormesh.ai>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Muspi Merol <me@promplate.dev>
Co-authored-by: fangkang <fangkangm@gmail.com>
Co-authored-by: Chris Hoogeboom <chris.hoogeboom@gmail.com>
Co-authored-by: kursadlacin <kursadlacin@gmail.com>
Co-authored-by: kursad <kursad.lacin@brado.net>
Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com>
Co-authored-by: hcl <chenglunhu@gmail.com>
Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: sfc-gh-nashukla <navnit.shukla@snowflake.com>
Co-authored-by: Rudra Dudhat <contact.rdudhat@gmail.com>
Co-authored-by: Michael <52305679+michaelxer@users.noreply.github.com>
Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>
Co-authored-by: Quentin Champenois <26109239+Quentinchampenois@users.noreply.github.com>
Co-authored-by: mauriceberentsen <mauriceberentsen@live.nl>
Co-authored-by: lost9999 <56498264+lost9999@users.noreply.github.com>
Co-authored-by: GaetanVDB07 <86427581+GaetanVDB07@users.noreply.github.com>
Co-authored-by: Aanchal Khandelwal <aan2210khandelwal@gmail.com>
Co-authored-by: Adam Dalloul <adam_dalloul@icloud.com>
Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
factnn pushed a commit to factnn/litellm that referenced this pull request Jun 18, 2026
* feat(bedrock): add bedrock mantle gemma 4 models (BerriAI#30264)

* feat(bedrock): add bedrock mantle gemma 4 models

* test(bedrock): harden mantle local cost fixture

* feat(responses): enable the responses API for the Tensormesh provider (BerriAI#30209)

* feat(responses): enable the responses API for the Tensormesh provider

* Update litellm/llms/openai_like/providers.json

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

---------

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* fix(langfuse_otel): mark LLM spans as generations (BerriAI#30250)

* fix(bedrock): stop stream_chunk_size leaking into invoke request bodies (BerriAI#30240)

stream_chunk_size is a LiteLLM-internal knob for re-chunking the HTTP
response stream. The invoke transformations splat optional_params into the
provider request body without dropping it, and Bedrock rejects unknown
fields, so any bedrock/invoke request that sets the parameter fails with
ValidationException: stream_chunk_size: Extra inputs are not permitted.
Drop it in the invoke dispatcher (covers cohere, titan, mistral, meta,
ai21) and in the Claude messages-format request builder (the route used
for bedrock/invoke Anthropic models)

* fix(bedrock): stop buffering streamed tool-call argument deltas (BerriAI#30231)

* fix(bedrock): stop buffering streamed tool-call argument deltas

Two issues made Bedrock tool-use streaming arrive as a single end-of-stream
burst through LiteLLM while plain text streamed fine.

First, the anthropic-beta allowlist mapped fine-grained-tool-streaming-2025-05-14
to null for bedrock and bedrock_converse, so the header was silently stripped.
Without that beta, Anthropic models on Bedrock buffer tool input server-side and
emit all toolUse.input deltas at once (verified against converse-stream and
invoke-with-response-stream directly). Bedrock accepts the beta via
additionalModelRequestFields.anthropic_beta, so it is now forwarded.

Second, the streaming reads re-chunked the AWS event stream with
iter_bytes(chunk_size=1024). httpx's ByteChunker only releases full 1024-byte
blocks, so the small early events (messageStart, contentBlockStart, first
deltas) sat in the buffer until enough bytes accumulated, pushing
time-to-first-byte from ~1.4s to ~8.5s on buffered tool-use streams. The
default is now no re-chunking; an explicit stream_chunk_size is still honored.

* test(bedrock): cover explicit stream_chunk_size on sync invoke path

* test(bedrock): cover stream_chunk_size plumbing through converse completion

* test(bedrock): cover stream_chunk_size default in legacy BedrockLLM streaming

* test(bedrock): merge converse handler tests into existing mapped test file

pytest imports test modules by basename in non-package test dirs, so the new
tests/test_litellm/llms/bedrock/chat/test_converse_handler.py collided with
the pre-existing tests/test_litellm/llms/chat/test_converse_handler.py and
broke collection in CI. Move the new tests into the existing file

* feat(otel): emit v2 cost breakdown + stamp tracer scope version (BerriAI#30156)

Read the StandardLoggingPayload cost_breakdown into a typed LLMCost on
LLMCallSpanData and emit each component under litellm.cost.* (absent
components omitted, so spans stay sparse). Stamp litellm.__version__ as
the instrumentation scope version so every v2 span carries a
deterministic scope.version.

Tests under tests/test_litellm/integrations/otel/.

* fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in) (BerriAI#30223)

* fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in)

On the non-streaming path, base_process_llm_request awaited the LLM call
with no disconnect monitoring; when the HTTP client went away the
upstream request kept running until completion or request_timeout (6000s
default), holding a backend slot (e.g. a vLLM GPU slot) for output
nobody would read

Add an opt-in general_settings.cancel_on_disconnect flag, default off,
so the default code path is unchanged. When enabled, a receive-based
watcher task observes http.disconnect and cancels the asyncio.gather
driving the upstream call. The resulting CancelledError is converted to
HTTPException 499 only when the disconnect event is set, so
server-initiated cancellations still propagate as-is. The 499 then flows
through _handle_llm_api_exception like any other failure, meaning
post_call_failure_hook still releases max_parallel_requests slots and
fires spend and alerting callbacks; it is logged at info level instead
of a full traceback

Also removes the dead check_request_disconnection helper in
proxy_server.py (zero call sites) along with its behavior-pin tests

Builds on the receive-based design from BerriAI#25776

Addresses BerriAI#13774. Re-fixes BerriAI#22805 (regressed after the BerriAI#14295 revert)

Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com>

* fix(proxy): scope 499 quiet logging to disconnects and harden watcher

Address the two P2 findings from the Greptile review on BerriAI#30223. The
info-level logging in _log_llm_api_exception now applies only to the
disconnect-specific HTTPException (status 499 plus the shared
_CLIENT_DISCONNECT_DETAIL message), so any other 499 raised by hooks or
guardrails keeps its full traceback. The disconnect watcher now catches
exceptions from request.receive() (e.g. a transport reset) and logs a
warning instead of dying silently, making the degradation to no-op
visible; a test pins that the LLM call is not cancelled in that case

---------

Co-authored-by: kursad <kursad.lacin@brado.net>
Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com>

* fix(bedrock): grant aws-external-anthropic:* in OIDC session policy for claude_platform (BerriAI#30200) (BerriAI#30205)

The inline STS session policy passed to assume_role_with_web_identity
acts as an IAM PERMISSION CEILING — effective permissions are the
intersection of the role's identity policies and this policy. Any
action not listed is silently denied even when the IAM role grants it.

BerriAI#27678 added the bedrock/claude_platform/<model> route but its
service-side action namespace is aws-external-anthropic:*, not
bedrock:*. Without a matching statement here, every claude_platform
request via OIDC (GCP federation, EKS Pod Identity webhook, etc.) 403s
with 'no session policy allows the aws-external-anthropic:CreateInference
action' — even with a fully permissive identity policy.

Add a second ClaudePlatformLiteLLM statement covering CreateInference,
CreateBatchInference, CancelBatchInference, DeleteBatchInference,
CountTokens, Get*, List*. Keep aws:SecureTransport=true parity with the
bedrock statement.

Static creds + IRSA flow through different code paths and are not
affected.

Fixes BerriAI#30200

* fix(proxy): set Retry-After header on RouterRateLimitError 429 responses (BerriAI#30098)

* Set Retry-After header on RouterRateLimitError responses

When all deployments for a model are in cooldown, the proxy returns a
429 whose cooldown timing is only available by parsing the error
message string. RouterRateLimitError already carries cooldown_time, so
expose it as a standard retry-after header in
_handle_llm_api_exception. The value is rounded up so clients never
retry before the cooldown window ends.

Fixes BerriAI#27823.

* Set Retry-After after response-headers hook so cooldown wins

The cooldown-derived retry-after was assigned before the
post_call_response_headers_hook merge, so a callback returning a
retry-after key (including a stale or empty value) silently clobbered
it. Move the RouterRateLimitError block after the callback merge so the
cooldown value is authoritative for this error type.

* fix(router): route aspeech through async_function_with_fallbacks (BerriAI#30104)

* fix(router): route aspeech through async_function_with_fallbacks

Router.aspeech selected a deployment and awaited litellm.aspeech
directly, so TTS requests got no retry on failure and no failover to
backup deployments; the except block only fired an exception alert and
re-raised. Every other router endpoint (acompletion, aembedding,
atranscription, arerank) already delegates to
async_function_with_fallbacks

Mirror the atranscription pattern: move deployment selection and the
litellm.aspeech call into a private _aspeech method, then have the
public aspeech set kwargs["original_function"] = self._aspeech and
await self.async_function_with_fallbacks(**kwargs). _aspeech also picks
up the shared _get_async_openai_model_client helper and the same
total/success/fail call accounting the sibling endpoints use

Fixes BerriAI#27778.

* fix(router): apply deployment kwargs and rpm semaphore in _aspeech

Bring _aspeech fully in line with _atranscription: call
_update_kwargs_with_deployment so deployment metadata, model_info,
timeout, and default litellm params flow into the request, and wrap
the litellm.aspeech call with the max_parallel_requests semaphore plus
async_routing_strategy_pre_call_checks so TTS respects rpm limits the
same way the other router endpoints do

Also add a unit test that exercises _aspeech directly and asserts the
deployment metadata reaches the underlying call

* fix(slack_alerting): stop false-positive hanging request alerts for requests below the alerting threshold (BerriAI#30106)

* fix(slack_alerting): skip hanging request alerts below the threshold

The hanging request check alerted on any cached request whose
completion status was not yet recorded, with no minimum age check.
Since the background loop runs every alerting_threshold / 2 seconds,
any request that happened to be in flight at a check fired a
"hanging - Ns+ request time" alert even if it was only seconds old,
producing a steady stream of false positives.

Add a created_at timestamp to HangingRequestData, stamped when the
request enters the hanging request cache, and skip requests younger
than alerting_threshold without evicting them, so a later check can
still alert if they never complete. Extend the cache TTL from
threshold + 60s to 1.5x threshold + 60s; with the age check, entries
only become alertable after threshold seconds, and the check period
is threshold / 2, so the old TTL could evict a genuinely hanging
request before any check saw it cross the threshold.

Fixes BerriAI#27855.

* fix(slack_alerting): alert once per hanging request

The min-age gate stops false positives for young in-flight requests, but
a genuinely hanging request still re-alerted on every checker tick within
the cache TTL. With the wider TTL (1.5x threshold + 60s) that is 1-2 extra
Slack notifications per stuck request at the default 600s threshold.

Flag a HangingRequestData entry as alerted once its alert fires and skip
flagged entries on later ticks, so each hang produces exactly one alert.
The cache reference is mutated in place, so the TTL is untouched and still
handles cleanup. Adds a regression test asserting one alert across multiple
ticks.

Fixes BerriAI#27855.

* fix(health): treat all-proxy-models keys as unrestricted in /health (BerriAI#30087)

* fix(health): treat all-proxy-models keys as unrestricted in /health

A key granted all model permissions stores the literal
"all-proxy-models" marker in its models list. The /health access
filter compared that marker against real model_names, so the model
list filtered down to nothing and the WebUI health check returned
healthy_count=0, unhealthy_count=0 with HTTP 503. Skip the filter
(both the live path and the background-cache model_id scoping) when
the marker is present, matching how auth_checks treats
SpecialModelNames.all_proxy_models.

Fixes BerriAI#29744.

* fix(health): resolve all-team-models sentinel to the team allowlist

Same failure shape as the all-proxy-models case: a key carrying the
literal "all-team-models" entry matches no real model_name, so the
/health access filter would zero out the model list. Resolve the
sentinel to the key's team models when team_id is set, matching
get_key_models in model_checks.py. Without a team_id the sentinel
stays unresolved and matches nothing, denying rather than widening
access, mirroring _resolve_key_models_for_auth_check.

* feat(proxy): auto-enable drop_params for Claude Code requests (BerriAI#30218)

* feat(proxy): auto-enable drop_params for Claude Code requests

Claude Code identifies itself with a claude-cli/<version> user agent and
sends Anthropic-specific params (top_k, thinking, etc.) on every request.
When the proxy routes those requests to a non-Anthropic provider, the
unsupported params fail the call unless drop_params is configured. Detect
the Claude Code user agent in add_litellm_data_to_request and default
drop_params to true for those requests, without overriding an explicit
drop_params value sent by the caller.

* feat(proxy): respect operator litellm_settings drop_params over Claude Code default

An explicit drop_params in the operator's litellm_settings (true or false)
now suppresses the Claude Code user agent default, so an operator who
deliberately configured drop_params: false keeps strict param validation
for Claude Code clients too. The auto-default only fills the gap when
neither the request body nor the config sets a value.

* fix(snowflake): migrate to native endpoints with auto-routing for Claude models (BerriAI#29964)

* fix(snowflake): migrate to native Cortex REST API endpoints

Replaces the legacy /api/v2/cortex/inference:complete endpoint with the
native OpenAI-compatible /api/v2/cortex/v1/chat/completions endpoint,
fixing error 390142 (Incoming request does not contain a valid payload)
when using model: snowflake/<model> in LiteLLM proxy.

Changes:
- litellm/llms/snowflake/chat/transformation.py: route to native
  /cortex/v1/chat/completions, remove Snowflake-specific tool_spec
  payload transformation, remove content_list response handling,
  add stream to supported params
- litellm/llms/snowflake/anthropic/transformation.py (new):
  SnowflakeCortexAnthropicConfig routes Claude models to /cortex/v1/messages
  with anthropic-version header and Anthropic->OpenAI response transform
- tests: 29 unit tests covering URL routing, auth headers, payload
  format, and response parsing

* fix(snowflake): map max_tokens to max_completion_tokens for native endpoint

* fix: handle multi-turn tool conversations and OpenAI→Anthropic tool format conversion

- _extract_system_and_messages now preserves tool_calls from assistant messages
  and converts them to Anthropic tool_use content blocks
- tool role messages are converted to user role with tool_result content blocks
  (as required by Anthropic Messages API)
- Added _transform_tools_to_anthropic() to convert OpenAI tool format
  (type/function/parameters) to Anthropic format (name/input_schema)
- Added comprehensive tests for multi-turn tool conversations

Addresses review feedback on PR BerriAI#29964

* test: add coverage for malformed JSON and non-string tool arguments

* fix(tests): update chat transformation tests for native OpenAI-compatible endpoint

* style: apply black formatting

* fix: resolve mypy type errors in anthropic transformation

* fix: correct mypy type: ignore error codes (attr-defined)

* fix: use max_tokens instead of max_completion_tokens for Snowflake endpoint compatibility

* refactor: merge Anthropic config into unified SnowflakeConfig with auto-routing

- Remove separate SnowflakeCortexAnthropicConfig and anthropic/ directory
- SnowflakeConfig now auto-routes based on model name:
  - Claude models → /messages endpoint (Anthropic format)
  - All others → /chat/completions endpoint (OpenAI format)
- No new provider needed (stays as SNOWFLAKE = 'snowflake')
- Tool message transformation for Claude: tool_calls → tool_use blocks,
  tool role → user with tool_result
- OpenAI → Anthropic tool format conversion (parameters → input_schema)
- Addresses Greptile feedback about unwired SnowflakeCortexAnthropicConfig

* fix: use max_completion_tokens for /chat/completions (Snowflake deprecated max_tokens on this endpoint)

* fix(tests): update assertions for Claude auto-routing to /messages endpoint

* fix(snowflake): add tool_choice conversion and preserve max_completion_tokens in Anthropic path

* fix(snowflake): use ChatCompletionMessageToolCall objects and strip model prefix on OpenAI path

* fix(snowflake): collect multiple system messages to prevent guardrail override

* chore: remove committed .pyc files and add __pycache__ to .gitignore

* fix: remove unused Union import

* fix: restore original .gitignore (accidentally replaced in earlier commit)

* feat(snowflake): add streaming response handler for both Anthropic and OpenAI SSE formats

* fix: remove unused AsyncIterator and Iterator imports

* fix: add missing total_tokens to ChatCompletionUsageBlock

* fix(snowflake): coalesce consecutive tool results into single user message for Anthropic

* fix(snowflake): handle message_start event for streaming input_tokens tracking

* fix: evict last deleted model in multi-instance deployments (BerriAI#28608)

* fix: evict last deleted model in multi-instance deployments

_delete_deployment had an early return when db_models was empty,
preventing eviction of the last deleted model during reconciliation.

- Remove len(db_models)==0 early return from _delete_deployment
- Return None (not []) from _get_models_from_db on DB failure so
  callers can distinguish a transient failure from a genuinely empty DB
- Guard _update_llm_router against None to skip updates on DB failure

Fixes BerriAI#28443

* test: remove dead MagicMock assignment in type_mismatch test

* fix: update test to pass [] not None to _update_llm_router

test_ProxyConfig__update_llm_router_bad_proxy_logging_raises was passing
None as new_models to get through to the proxy_logging_obj check, but
the None guard we added now returns early before reaching that path.
Pass [] instead so the test exercises the intended AttributeError case.

Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com>

* chore: regenerate API types to sync schema.d.ts with proxy OpenAPI spec

Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com>

---------

Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com>

* fix: invalidate Redis spend counter on /key/reset_spend (BerriAI#29694)

* fix: set Redis spend counter to reset_to value on /key/reset_spend

Previously, the Redis spend counter was always set to 0.0 after a reset,
even when reset_to was a non-zero value (partial reset). This caused
the budget to be under-enforced for up to 60 seconds until the counter
expired and fell through to the DB.

Now the counter is set to the actual reset_to value, so partial resets
are reflected correctly and budget enforcement is consistent.

* test: update reset_key_spend test to match direct cache set

The implementation now sets spend_counter_cache directly instead of
calling _invalidate_spend_counter. Update the test to verify the
in_memory_cache.set_cache call with the correct key, value, and ttl.

---------

Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>

* fix: add scaleway models pricing (BerriAI#27659)

* fix: Add embeddings support for Scaleway provider

* fix: resolve merge conflicts

* fix(main): clarify backend route handling for Swagger static assets (BerriAI#30196)

* fix(main): clarify backend route handling for Swagger static assets

* fix(allowlist): add BACKEND_MOUNT_PATHS for Swagger static assets

* fix(voyage): route multimodal embeddings to correct endpoint (BerriAI#30193)

* fix(voyage): route multimodal embeddings to correct endpoint

* test(voyage): cover multimodal embedding edge cases

* test(voyage): cover api key fallback

* fix(voyage): raise early on missing api key and malformed image url

* test(voyage): cover utils routing and helper

* fix(voyage): route supported openai params for multimodal models

* style: apply black formatting

* fix(ui): infer Azure API version from API base (BerriAI#30204)

* fix(ui): infer Azure API version from API base

* fix(ui): address Azure API version feedback

* Update litellm/llms/snowflake/chat/transformation.py

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* feat(datadog): add team-scoped Datadog callback support (BerriAI#29947)

Enable teams to configure their own Datadog credentials via
POST /team/{team_id}/callback, following the same pattern as Langfuse.

* Merge pull request BerriAI#29528 from aanchal22/litellm_byok-alias-merge

fix(proxy): atomic merge for team model aliases and team.models on BYOK create

* feat: add EmpirioLabs as an OpenAI-compatible provider (BerriAI#30278)

Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com>

* fix: resolve failing tests and lint in snowflake/team endpoints

- Black-format snowflake/chat/transformation.py to fix lint failure
- Update Anthropic config test to expect default max_tokens of 4096 (matches implementation)
- Add AsyncMock + execute_raw mock to team_model_add cache-refresh pin test
- Add model_dump mock and patch cache/logging in test_uses_atomic_array_append_with_dedup

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(test): update test_db_error_new_model_check for new _delete_deployment logic

_delete_deployment no longer short-circuits on empty db_models — it now
treats [] as a valid empty-DB state and proceeds to check config models.
Mock get_config to return the two router deployments so they appear in
combined_id_list and are protected, which matches the real-world scenario
where a DB error occurs but the models are config-backed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list (BerriAI#30295)

* feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list

Follow-up to BerriAI#30223 per maintainer review: documents the flag in
ConfigGeneralSettings with a short description and adds it to
allowed_args in get_config_list so the UI and /config/list expose it.
A test pins that /config/list returns the field with type Boolean,
which requires both registrations to be present

* chore(ui): regenerate schema.d.ts for cancel_on_disconnect

---------

Co-authored-by: kursad <kursad.lacin@brado.net>

* fix(datadog): never fall back to env DD_API_KEY for caller-supplied destinations

Team/key-scoped Datadog loggers could be pointed at an arbitrary dd_agent_host or
dd_site while omitting dd_api_key, causing the proxy's global DD_API_KEY to be sent
as the DD-API-KEY header to that destination. Gate the env-var fallback behind an
allow_env_credentials flag, set to False when the destination is caller-supplied,
mirroring the existing langfuse/langsmith pattern.

---------

Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com>
Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com>
Co-authored-by: daitran-tensormesh <dai@tensormesh.ai>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Muspi Merol <me@promplate.dev>
Co-authored-by: fangkang <fangkangm@gmail.com>
Co-authored-by: Chris Hoogeboom <chris.hoogeboom@gmail.com>
Co-authored-by: kursadlacin <kursadlacin@gmail.com>
Co-authored-by: kursad <kursad.lacin@brado.net>
Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com>
Co-authored-by: hcl <chenglunhu@gmail.com>
Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: sfc-gh-nashukla <navnit.shukla@snowflake.com>
Co-authored-by: Rudra Dudhat <contact.rdudhat@gmail.com>
Co-authored-by: Michael <52305679+michaelxer@users.noreply.github.com>
Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>
Co-authored-by: Quentin Champenois <26109239+Quentinchampenois@users.noreply.github.com>
Co-authored-by: mauriceberentsen <mauriceberentsen@live.nl>
Co-authored-by: lost9999 <56498264+lost9999@users.noreply.github.com>
Co-authored-by: GaetanVDB07 <86427581+GaetanVDB07@users.noreply.github.com>
Co-authored-by: Aanchal Khandelwal <aan2210khandelwal@gmail.com>
Co-authored-by: Adam Dalloul <adam_dalloul@icloud.com>
Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
marioleonardo pushed a commit to marioleonardo/litellm-mongodb-patched that referenced this pull request Jun 19, 2026
Enable teams to configure their own Datadog credentials via
POST /team/{team_id}/callback, following the same pattern as Langfuse.

(cherry picked from commit f5e6012)
fzowl pushed a commit to fzowl/litellm that referenced this pull request Jun 24, 2026
* feat(bedrock): add bedrock mantle gemma 4 models (BerriAI#30264)

* feat(bedrock): add bedrock mantle gemma 4 models

* test(bedrock): harden mantle local cost fixture

* feat(responses): enable the responses API for the Tensormesh provider (BerriAI#30209)

* feat(responses): enable the responses API for the Tensormesh provider

* Update litellm/llms/openai_like/providers.json

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

---------

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* fix(langfuse_otel): mark LLM spans as generations (BerriAI#30250)

* fix(bedrock): stop stream_chunk_size leaking into invoke request bodies (BerriAI#30240)

stream_chunk_size is a LiteLLM-internal knob for re-chunking the HTTP
response stream. The invoke transformations splat optional_params into the
provider request body without dropping it, and Bedrock rejects unknown
fields, so any bedrock/invoke request that sets the parameter fails with
ValidationException: stream_chunk_size: Extra inputs are not permitted.
Drop it in the invoke dispatcher (covers cohere, titan, mistral, meta,
ai21) and in the Claude messages-format request builder (the route used
for bedrock/invoke Anthropic models)

* fix(bedrock): stop buffering streamed tool-call argument deltas (BerriAI#30231)

* fix(bedrock): stop buffering streamed tool-call argument deltas

Two issues made Bedrock tool-use streaming arrive as a single end-of-stream
burst through LiteLLM while plain text streamed fine.

First, the anthropic-beta allowlist mapped fine-grained-tool-streaming-2025-05-14
to null for bedrock and bedrock_converse, so the header was silently stripped.
Without that beta, Anthropic models on Bedrock buffer tool input server-side and
emit all toolUse.input deltas at once (verified against converse-stream and
invoke-with-response-stream directly). Bedrock accepts the beta via
additionalModelRequestFields.anthropic_beta, so it is now forwarded.

Second, the streaming reads re-chunked the AWS event stream with
iter_bytes(chunk_size=1024). httpx's ByteChunker only releases full 1024-byte
blocks, so the small early events (messageStart, contentBlockStart, first
deltas) sat in the buffer until enough bytes accumulated, pushing
time-to-first-byte from ~1.4s to ~8.5s on buffered tool-use streams. The
default is now no re-chunking; an explicit stream_chunk_size is still honored.

* test(bedrock): cover explicit stream_chunk_size on sync invoke path

* test(bedrock): cover stream_chunk_size plumbing through converse completion

* test(bedrock): cover stream_chunk_size default in legacy BedrockLLM streaming

* test(bedrock): merge converse handler tests into existing mapped test file

pytest imports test modules by basename in non-package test dirs, so the new
tests/test_litellm/llms/bedrock/chat/test_converse_handler.py collided with
the pre-existing tests/test_litellm/llms/chat/test_converse_handler.py and
broke collection in CI. Move the new tests into the existing file

* feat(otel): emit v2 cost breakdown + stamp tracer scope version (BerriAI#30156)

Read the StandardLoggingPayload cost_breakdown into a typed LLMCost on
LLMCallSpanData and emit each component under litellm.cost.* (absent
components omitted, so spans stay sparse). Stamp litellm.__version__ as
the instrumentation scope version so every v2 span carries a
deterministic scope.version.

Tests under tests/test_litellm/integrations/otel/.

* fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in) (BerriAI#30223)

* fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in)

On the non-streaming path, base_process_llm_request awaited the LLM call
with no disconnect monitoring; when the HTTP client went away the
upstream request kept running until completion or request_timeout (6000s
default), holding a backend slot (e.g. a vLLM GPU slot) for output
nobody would read

Add an opt-in general_settings.cancel_on_disconnect flag, default off,
so the default code path is unchanged. When enabled, a receive-based
watcher task observes http.disconnect and cancels the asyncio.gather
driving the upstream call. The resulting CancelledError is converted to
HTTPException 499 only when the disconnect event is set, so
server-initiated cancellations still propagate as-is. The 499 then flows
through _handle_llm_api_exception like any other failure, meaning
post_call_failure_hook still releases max_parallel_requests slots and
fires spend and alerting callbacks; it is logged at info level instead
of a full traceback

Also removes the dead check_request_disconnection helper in
proxy_server.py (zero call sites) along with its behavior-pin tests

Builds on the receive-based design from BerriAI#25776

Addresses BerriAI#13774. Re-fixes BerriAI#22805 (regressed after the BerriAI#14295 revert)

Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com>

* fix(proxy): scope 499 quiet logging to disconnects and harden watcher

Address the two P2 findings from the Greptile review on BerriAI#30223. The
info-level logging in _log_llm_api_exception now applies only to the
disconnect-specific HTTPException (status 499 plus the shared
_CLIENT_DISCONNECT_DETAIL message), so any other 499 raised by hooks or
guardrails keeps its full traceback. The disconnect watcher now catches
exceptions from request.receive() (e.g. a transport reset) and logs a
warning instead of dying silently, making the degradation to no-op
visible; a test pins that the LLM call is not cancelled in that case

---------

Co-authored-by: kursad <kursad.lacin@brado.net>
Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com>

* fix(bedrock): grant aws-external-anthropic:* in OIDC session policy for claude_platform (BerriAI#30200) (BerriAI#30205)

The inline STS session policy passed to assume_role_with_web_identity
acts as an IAM PERMISSION CEILING — effective permissions are the
intersection of the role's identity policies and this policy. Any
action not listed is silently denied even when the IAM role grants it.

BerriAI#27678 added the bedrock/claude_platform/<model> route but its
service-side action namespace is aws-external-anthropic:*, not
bedrock:*. Without a matching statement here, every claude_platform
request via OIDC (GCP federation, EKS Pod Identity webhook, etc.) 403s
with 'no session policy allows the aws-external-anthropic:CreateInference
action' — even with a fully permissive identity policy.

Add a second ClaudePlatformLiteLLM statement covering CreateInference,
CreateBatchInference, CancelBatchInference, DeleteBatchInference,
CountTokens, Get*, List*. Keep aws:SecureTransport=true parity with the
bedrock statement.

Static creds + IRSA flow through different code paths and are not
affected.

Fixes BerriAI#30200

* fix(proxy): set Retry-After header on RouterRateLimitError 429 responses (BerriAI#30098)

* Set Retry-After header on RouterRateLimitError responses

When all deployments for a model are in cooldown, the proxy returns a
429 whose cooldown timing is only available by parsing the error
message string. RouterRateLimitError already carries cooldown_time, so
expose it as a standard retry-after header in
_handle_llm_api_exception. The value is rounded up so clients never
retry before the cooldown window ends.

Fixes BerriAI#27823.

* Set Retry-After after response-headers hook so cooldown wins

The cooldown-derived retry-after was assigned before the
post_call_response_headers_hook merge, so a callback returning a
retry-after key (including a stale or empty value) silently clobbered
it. Move the RouterRateLimitError block after the callback merge so the
cooldown value is authoritative for this error type.

* fix(router): route aspeech through async_function_with_fallbacks (BerriAI#30104)

* fix(router): route aspeech through async_function_with_fallbacks

Router.aspeech selected a deployment and awaited litellm.aspeech
directly, so TTS requests got no retry on failure and no failover to
backup deployments; the except block only fired an exception alert and
re-raised. Every other router endpoint (acompletion, aembedding,
atranscription, arerank) already delegates to
async_function_with_fallbacks

Mirror the atranscription pattern: move deployment selection and the
litellm.aspeech call into a private _aspeech method, then have the
public aspeech set kwargs["original_function"] = self._aspeech and
await self.async_function_with_fallbacks(**kwargs). _aspeech also picks
up the shared _get_async_openai_model_client helper and the same
total/success/fail call accounting the sibling endpoints use

Fixes BerriAI#27778.

* fix(router): apply deployment kwargs and rpm semaphore in _aspeech

Bring _aspeech fully in line with _atranscription: call
_update_kwargs_with_deployment so deployment metadata, model_info,
timeout, and default litellm params flow into the request, and wrap
the litellm.aspeech call with the max_parallel_requests semaphore plus
async_routing_strategy_pre_call_checks so TTS respects rpm limits the
same way the other router endpoints do

Also add a unit test that exercises _aspeech directly and asserts the
deployment metadata reaches the underlying call

* fix(slack_alerting): stop false-positive hanging request alerts for requests below the alerting threshold (BerriAI#30106)

* fix(slack_alerting): skip hanging request alerts below the threshold

The hanging request check alerted on any cached request whose
completion status was not yet recorded, with no minimum age check.
Since the background loop runs every alerting_threshold / 2 seconds,
any request that happened to be in flight at a check fired a
"hanging - Ns+ request time" alert even if it was only seconds old,
producing a steady stream of false positives.

Add a created_at timestamp to HangingRequestData, stamped when the
request enters the hanging request cache, and skip requests younger
than alerting_threshold without evicting them, so a later check can
still alert if they never complete. Extend the cache TTL from
threshold + 60s to 1.5x threshold + 60s; with the age check, entries
only become alertable after threshold seconds, and the check period
is threshold / 2, so the old TTL could evict a genuinely hanging
request before any check saw it cross the threshold.

Fixes BerriAI#27855.

* fix(slack_alerting): alert once per hanging request

The min-age gate stops false positives for young in-flight requests, but
a genuinely hanging request still re-alerted on every checker tick within
the cache TTL. With the wider TTL (1.5x threshold + 60s) that is 1-2 extra
Slack notifications per stuck request at the default 600s threshold.

Flag a HangingRequestData entry as alerted once its alert fires and skip
flagged entries on later ticks, so each hang produces exactly one alert.
The cache reference is mutated in place, so the TTL is untouched and still
handles cleanup. Adds a regression test asserting one alert across multiple
ticks.

Fixes BerriAI#27855.

* fix(health): treat all-proxy-models keys as unrestricted in /health (BerriAI#30087)

* fix(health): treat all-proxy-models keys as unrestricted in /health

A key granted all model permissions stores the literal
"all-proxy-models" marker in its models list. The /health access
filter compared that marker against real model_names, so the model
list filtered down to nothing and the WebUI health check returned
healthy_count=0, unhealthy_count=0 with HTTP 503. Skip the filter
(both the live path and the background-cache model_id scoping) when
the marker is present, matching how auth_checks treats
SpecialModelNames.all_proxy_models.

Fixes BerriAI#29744.

* fix(health): resolve all-team-models sentinel to the team allowlist

Same failure shape as the all-proxy-models case: a key carrying the
literal "all-team-models" entry matches no real model_name, so the
/health access filter would zero out the model list. Resolve the
sentinel to the key's team models when team_id is set, matching
get_key_models in model_checks.py. Without a team_id the sentinel
stays unresolved and matches nothing, denying rather than widening
access, mirroring _resolve_key_models_for_auth_check.

* feat(proxy): auto-enable drop_params for Claude Code requests (BerriAI#30218)

* feat(proxy): auto-enable drop_params for Claude Code requests

Claude Code identifies itself with a claude-cli/<version> user agent and
sends Anthropic-specific params (top_k, thinking, etc.) on every request.
When the proxy routes those requests to a non-Anthropic provider, the
unsupported params fail the call unless drop_params is configured. Detect
the Claude Code user agent in add_litellm_data_to_request and default
drop_params to true for those requests, without overriding an explicit
drop_params value sent by the caller.

* feat(proxy): respect operator litellm_settings drop_params over Claude Code default

An explicit drop_params in the operator's litellm_settings (true or false)
now suppresses the Claude Code user agent default, so an operator who
deliberately configured drop_params: false keeps strict param validation
for Claude Code clients too. The auto-default only fills the gap when
neither the request body nor the config sets a value.

* fix(snowflake): migrate to native endpoints with auto-routing for Claude models (BerriAI#29964)

* fix(snowflake): migrate to native Cortex REST API endpoints

Replaces the legacy /api/v2/cortex/inference:complete endpoint with the
native OpenAI-compatible /api/v2/cortex/v1/chat/completions endpoint,
fixing error 390142 (Incoming request does not contain a valid payload)
when using model: snowflake/<model> in LiteLLM proxy.

Changes:
- litellm/llms/snowflake/chat/transformation.py: route to native
  /cortex/v1/chat/completions, remove Snowflake-specific tool_spec
  payload transformation, remove content_list response handling,
  add stream to supported params
- litellm/llms/snowflake/anthropic/transformation.py (new):
  SnowflakeCortexAnthropicConfig routes Claude models to /cortex/v1/messages
  with anthropic-version header and Anthropic->OpenAI response transform
- tests: 29 unit tests covering URL routing, auth headers, payload
  format, and response parsing

* fix(snowflake): map max_tokens to max_completion_tokens for native endpoint

* fix: handle multi-turn tool conversations and OpenAI→Anthropic tool format conversion

- _extract_system_and_messages now preserves tool_calls from assistant messages
  and converts them to Anthropic tool_use content blocks
- tool role messages are converted to user role with tool_result content blocks
  (as required by Anthropic Messages API)
- Added _transform_tools_to_anthropic() to convert OpenAI tool format
  (type/function/parameters) to Anthropic format (name/input_schema)
- Added comprehensive tests for multi-turn tool conversations

Addresses review feedback on PR BerriAI#29964

* test: add coverage for malformed JSON and non-string tool arguments

* fix(tests): update chat transformation tests for native OpenAI-compatible endpoint

* style: apply black formatting

* fix: resolve mypy type errors in anthropic transformation

* fix: correct mypy type: ignore error codes (attr-defined)

* fix: use max_tokens instead of max_completion_tokens for Snowflake endpoint compatibility

* refactor: merge Anthropic config into unified SnowflakeConfig with auto-routing

- Remove separate SnowflakeCortexAnthropicConfig and anthropic/ directory
- SnowflakeConfig now auto-routes based on model name:
  - Claude models → /messages endpoint (Anthropic format)
  - All others → /chat/completions endpoint (OpenAI format)
- No new provider needed (stays as SNOWFLAKE = 'snowflake')
- Tool message transformation for Claude: tool_calls → tool_use blocks,
  tool role → user with tool_result
- OpenAI → Anthropic tool format conversion (parameters → input_schema)
- Addresses Greptile feedback about unwired SnowflakeCortexAnthropicConfig

* fix: use max_completion_tokens for /chat/completions (Snowflake deprecated max_tokens on this endpoint)

* fix(tests): update assertions for Claude auto-routing to /messages endpoint

* fix(snowflake): add tool_choice conversion and preserve max_completion_tokens in Anthropic path

* fix(snowflake): use ChatCompletionMessageToolCall objects and strip model prefix on OpenAI path

* fix(snowflake): collect multiple system messages to prevent guardrail override

* chore: remove committed .pyc files and add __pycache__ to .gitignore

* fix: remove unused Union import

* fix: restore original .gitignore (accidentally replaced in earlier commit)

* feat(snowflake): add streaming response handler for both Anthropic and OpenAI SSE formats

* fix: remove unused AsyncIterator and Iterator imports

* fix: add missing total_tokens to ChatCompletionUsageBlock

* fix(snowflake): coalesce consecutive tool results into single user message for Anthropic

* fix(snowflake): handle message_start event for streaming input_tokens tracking

* fix: evict last deleted model in multi-instance deployments (BerriAI#28608)

* fix: evict last deleted model in multi-instance deployments

_delete_deployment had an early return when db_models was empty,
preventing eviction of the last deleted model during reconciliation.

- Remove len(db_models)==0 early return from _delete_deployment
- Return None (not []) from _get_models_from_db on DB failure so
  callers can distinguish a transient failure from a genuinely empty DB
- Guard _update_llm_router against None to skip updates on DB failure

Fixes BerriAI#28443

* test: remove dead MagicMock assignment in type_mismatch test

* fix: update test to pass [] not None to _update_llm_router

test_ProxyConfig__update_llm_router_bad_proxy_logging_raises was passing
None as new_models to get through to the proxy_logging_obj check, but
the None guard we added now returns early before reaching that path.
Pass [] instead so the test exercises the intended AttributeError case.

Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com>

* chore: regenerate API types to sync schema.d.ts with proxy OpenAPI spec

Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com>

---------

Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com>

* fix: invalidate Redis spend counter on /key/reset_spend (BerriAI#29694)

* fix: set Redis spend counter to reset_to value on /key/reset_spend

Previously, the Redis spend counter was always set to 0.0 after a reset,
even when reset_to was a non-zero value (partial reset). This caused
the budget to be under-enforced for up to 60 seconds until the counter
expired and fell through to the DB.

Now the counter is set to the actual reset_to value, so partial resets
are reflected correctly and budget enforcement is consistent.

* test: update reset_key_spend test to match direct cache set

The implementation now sets spend_counter_cache directly instead of
calling _invalidate_spend_counter. Update the test to verify the
in_memory_cache.set_cache call with the correct key, value, and ttl.

---------

Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>

* fix: add scaleway models pricing (BerriAI#27659)

* fix: Add embeddings support for Scaleway provider

* fix: resolve merge conflicts

* fix(main): clarify backend route handling for Swagger static assets (BerriAI#30196)

* fix(main): clarify backend route handling for Swagger static assets

* fix(allowlist): add BACKEND_MOUNT_PATHS for Swagger static assets

* fix(voyage): route multimodal embeddings to correct endpoint (BerriAI#30193)

* fix(voyage): route multimodal embeddings to correct endpoint

* test(voyage): cover multimodal embedding edge cases

* test(voyage): cover api key fallback

* fix(voyage): raise early on missing api key and malformed image url

* test(voyage): cover utils routing and helper

* fix(voyage): route supported openai params for multimodal models

* style: apply black formatting

* fix(ui): infer Azure API version from API base (BerriAI#30204)

* fix(ui): infer Azure API version from API base

* fix(ui): address Azure API version feedback

* Update litellm/llms/snowflake/chat/transformation.py

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* feat(datadog): add team-scoped Datadog callback support (BerriAI#29947)

Enable teams to configure their own Datadog credentials via
POST /team/{team_id}/callback, following the same pattern as Langfuse.

* Merge pull request BerriAI#29528 from aanchal22/litellm_byok-alias-merge

fix(proxy): atomic merge for team model aliases and team.models on BYOK create

* feat: add EmpirioLabs as an OpenAI-compatible provider (BerriAI#30278)

Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com>

* fix: resolve failing tests and lint in snowflake/team endpoints

- Black-format snowflake/chat/transformation.py to fix lint failure
- Update Anthropic config test to expect default max_tokens of 4096 (matches implementation)
- Add AsyncMock + execute_raw mock to team_model_add cache-refresh pin test
- Add model_dump mock and patch cache/logging in test_uses_atomic_array_append_with_dedup


* fix(test): update test_db_error_new_model_check for new _delete_deployment logic

_delete_deployment no longer short-circuits on empty db_models — it now
treats [] as a valid empty-DB state and proceeds to check config models.
Mock get_config to return the two router deployments so they appear in
combined_id_list and are protected, which matches the real-world scenario
where a DB error occurs but the models are config-backed.


* feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list (BerriAI#30295)

* feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list

Follow-up to BerriAI#30223 per maintainer review: documents the flag in
ConfigGeneralSettings with a short description and adds it to
allowed_args in get_config_list so the UI and /config/list expose it.
A test pins that /config/list returns the field with type Boolean,
which requires both registrations to be present

* chore(ui): regenerate schema.d.ts for cancel_on_disconnect

---------

Co-authored-by: kursad <kursad.lacin@brado.net>

* fix(datadog): never fall back to env DD_API_KEY for caller-supplied destinations

Team/key-scoped Datadog loggers could be pointed at an arbitrary dd_agent_host or
dd_site while omitting dd_api_key, causing the proxy's global DD_API_KEY to be sent
as the DD-API-KEY header to that destination. Gate the env-var fallback behind an
allow_env_credentials flag, set to False when the destination is caller-supplied,
mirroring the existing langfuse/langsmith pattern.

---------

Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com>
Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com>
Co-authored-by: daitran-tensormesh <dai@tensormesh.ai>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Muspi Merol <me@promplate.dev>
Co-authored-by: fangkang <fangkangm@gmail.com>
Co-authored-by: Chris Hoogeboom <chris.hoogeboom@gmail.com>
Co-authored-by: kursadlacin <kursadlacin@gmail.com>
Co-authored-by: kursad <kursad.lacin@brado.net>
Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com>
Co-authored-by: hcl <chenglunhu@gmail.com>
Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: sfc-gh-nashukla <navnit.shukla@snowflake.com>
Co-authored-by: Rudra Dudhat <contact.rdudhat@gmail.com>
Co-authored-by: Michael <52305679+michaelxer@users.noreply.github.com>
Co-authored-by: michaelxer <michaelxer@users.noreply.github.com>
Co-authored-by: Quentin Champenois <26109239+Quentinchampenois@users.noreply.github.com>
Co-authored-by: mauriceberentsen <mauriceberentsen@live.nl>
Co-authored-by: lost9999 <56498264+lost9999@users.noreply.github.com>
Co-authored-by: GaetanVDB07 <86427581+GaetanVDB07@users.noreply.github.com>
Co-authored-by: Aanchal Khandelwal <aan2210khandelwal@gmail.com>
Co-authored-by: Adam Dalloul <adam_dalloul@icloud.com>
Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants