feat(helm): support tpl rendering in podAnnotations - #28609
Conversation
Wrap toYaml with tpl (passing the root context $) in the deployment
template's podAnnotations block so users can reference Helm values and
templates (e.g. {{ include ... | sha256sum }}) inside pod annotations.
Primary use case: when users disable the chart's built-in ConfigMap
(proxyConfigMap.create=false) to provide their own, the built-in
checksum/config annotation is also disabled and ConfigMap changes no
longer trigger a rolling restart. With tpl support, users can
re-implement the checksum/config annotation themselves.
Matches the existing tpl pattern already used in this chart for
extraInitContainers, extraContainers (deployment.yaml:50,215) and the
migrations job (migrations-job.yaml:40,99). Direct precedent: commit
87d7e86 ("feat(helm): add tpl support to extraContainers and
extraInitContainers").
Adds a helm unittest case exercising root-Values access (proving the
root context is wired) and plain-string passthrough (backward-compat
canary). make test-unit-helm: 54/54 passing. helm lint: clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
|
Greptile SummaryAdds
Confidence Score: 5/5Safe to merge — the change is a one-liner that follows an established pattern already used twice in the same file, and the new test covers the relevant scenarios. The change is minimal and consistent with the existing No files require special attention.
|
| Filename | Overview |
|---|---|
| deploy/charts/litellm-helm/templates/deployment.yaml | One-line change: wraps podAnnotations rendering with tpl (toYaml .) $ to enable template expressions; follows the identical pattern used for extraInitContainers (line 50) and extraContainers (line 231). |
| deploy/charts/litellm-helm/tests/deployment_tests.yaml | Adds a single new helm unittest case covering template-expression resolution, multi-value resolution, and plain-string backward-compat pass-through in podAnnotations. |
Reviews (1): Last reviewed commit: "feat(helm): support tpl rendering in pod..." | Re-trigger Greptile
PR overviewMedium: Helm template execution in pod annotationsThis PR changes pod annotations from static values into Helm templates. A user who can influence Security review
Risk: 5/10 |
| {{- end }} | ||
| {{- with .Values.podAnnotations }} | ||
| {{- toYaml . | nindent 8 }} | ||
| {{- tpl (toYaml .) $ | nindent 8 }} |
There was a problem hiding this comment.
Medium: Helm template injection in pod annotations
A user who can supply podAnnotations can now execute Helm template functions during render and place the output into Pod annotations, for example using lookup to read Kubernetes objects when Helm renders with cluster credentials. Keep annotations as data, or add a dedicated constrained value for the checksum use case instead of evaluating the entire annotations map as a template.
2c144fc
into
BerriAI:shin_agent_oss_staging_05_22_2026
|
🤖 litellm-agent: Squash-merged into staging branch Triage Summary Merge Confidence: 5/5 ✅ READY All checks green. Greptile 5/5, no blocking pattern findings, no CircleCI runs (OSS-typical). |
Squash-merged by litellm-agent from devauxbr's PR.
* fix(mcp): handle OAuth IdP error responses in /callback (LIT-2750) Per RFC 6749 section 4.1.2.1, when the IdP rejects an OAuth authorization request it redirects back to the client with ?error=...&error_description=... and no code. The MCP /callback handler declared code and state as required query params, so FastAPI rejected such error responses with a 422 before the handler ran -- stranding the MCP client waiting on the loopback. This change: - Makes code and state optional and accepts the RFC-defined error, error_description, and error_uri params. - When state decodes to a trusted client redirect_uri, propagates the error params back to that URI with the client's original (un-wrapped) state preserved, so the client's OAuth library can surface the failure. - When state is missing/undecryptable or the encoded redirect_uri is no longer trusted, renders a 400 HTML page with the (HTML-escaped) error details instead of leaking to an attacker-controlled redirect. - Preserves the existing success path (code + state -> 302 to validated client redirect_uri with original state). Fixes LIT-2750. * test(mcp): regression tests for /callback handling IdP error responses (LIT-2750) Adds a new test module covering the LIT-2750 fix: the MCP OAuth /callback endpoint must accept IdP error responses (e.g. ?error=access_denied) per RFC 6749 section 4.1.2.1 instead of returning a 422 because ``code`` is missing. Coverage: - IdP error with no state -> 400 HTML page surfacing the error. - HTML escaping of user-controlled error / error_description fields. - IdP error with a trusted (loopback) state -> 302 propagating error / error_description / original client state to the client. - IdP error with an untrusted redirect_uri encoded in state -> 400 inline (no open-redirect to attacker-controlled origin). - IdP error with an undecryptable state -> 400 HTML fallback. - Bare GET /callback with no params -> 400 HTML (not Pydantic 422). - Success path (code + state) still 302 to validated client redirect_uri with the original (un-wrapped) state preserved. * refactor(mcp): drop unused _OAUTH_ERROR_PARAMS constant (Greptile P2) The tuple was leftover scaffolding from an earlier draft of the LIT-2750 fix; nothing references it. The explanatory RFC 6749 §4.1.2.1 comment block above the callback handler covers the same intent. * fix(mcp/oauth): preserve empty original_state and clarify missing-param error in /callback Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(thinking): handle None thinking param in is_thinking_enabled (#28598) Squash-merged by litellm-agent from Terrajlz's PR. * feat(helm): support tpl rendering in podAnnotations (#28609) Squash-merged by litellm-agent from devauxbr's PR. * fix: apply black formatting to base_llm chat transformation Fix CI black --check failure on is_thinking_enabled return formatting. Co-authored-by: Cursor <cursoragent@cursor.com> * merge main (#28836) * fix(proxy): Bedrock Knowledge Base pass-through: preserve SigV4 headers and signed request body (#27526) * Fix Bedrock KB pass-through SigV4 headers and signed body Coerce botocore HeadersDict to a dict for pass-through routes. When forward_headers is true, drop request headers that collide case-insensitively with signed headers so client Bearer auth does not shadow AWS SigV4. Send prepped.body as raw content so the outbound payload matches the signature after logging hooks mutate the parsed dict. Co-authored-by: Cursor <cursoragent@cursor.com> * Simplify pass-through raw body handling Read the SigV4-signed bytes directly from request.state inside pass_through_request instead of threading a custom_raw_body argument through three functions. Helper methods are restored to their original signatures, and the new branch lives in one place at each httpx call site. Co-authored-by: Cursor <cursoragent@cursor.com> * Harden pass-through raw body read from request.state Guard missing request.state (test fixtures) and ignore non-bytes/str values so MagicMock does not trigger the SigV4 raw-body path. Co-authored-by: Cursor <cursoragent@cursor.com> * Test pass_through_request state_raw_body uses httpx content= Cover non-streaming (async_client.request) and streaming (build_request) paths so SigV4 bytes on request.state are not replaced by json= of a hook-mutated dict. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> * chore(tests): migrate Bedrock CI to AWS account 941277531214 (#28728) * chore(tests): migrate Bedrock CI from AWS account 888602223428 to 941277531214 The original account (888602223428) was put under a security restriction by AWS after a root access key leaked in a PR comment. While that account works its way through the AWS Support unlock process, Bedrock-touching CI tests have been migrated to a fresh account (941277531214). Changes: - Replace 26 hardcoded references to 888602223428 with 941277531214 across 8 files (provisioned-model ARNs, imported-model ARNs, AgentCore runtime ARNs, batch execution role ARN, and example proxy config). - The provisioned-model and imported-model ARNs are referenced only from mocked unit tests — no AWS resources to recreate. - The batch execution IAM role has been recreated in the new account with the same name and equivalent permissions. - The two AgentCore runtimes (hosted_agent_r9jvp-3ySZuRHjLC, hosted_agent_13sf6-cALnp38iZD) are being recreated in the new account under the same names — see tools/agentcore-deploy/ in a follow-up. CircleCI env vars AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_REGION_NAME were updated separately via the CircleCI API to point at the new account. Smoke-tested locally against the new account: aws bedrock-runtime converse --region us-west-2 \ --model-id us.anthropic.claude-sonnet-4-5-20250929-v1:0 \ --messages '[{"role":"user","content":[{"text":"ping"}]}]' → 200, model returned 'pong' Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(tests): refresh AgentCore ARN suffixes to match newly-deployed runtimes The first migration commit replaced just the account ID, but AgentCore auto-assigns a random 10-char suffix to every runtime on creation — we can't reuse the original suffixes (`3ySZuRHjLC`, `cALnp38iZD`) in the new account. Updated the AgentCore-runtime ARNs in the three files that reference real runtime IDs (not the mock-based unit-test ARNs). Deployed runtimes: arn:aws:bedrock-agentcore:us-west-2:941277531214:runtime/hosted_agent_r9jvp-Rq79QFC2fp arn:aws:bedrock-agentcore:us-west-2:941277531214:runtime/hosted_agent_13sf6-4046UzHSwy Both runtimes are status=READY and pass a smoke invoke: $ aws bedrock-agentcore invoke-agent-runtime --agent-runtime-arn ... --payload '{"prompt":"ping"}' → 200, {"result": "echo: ping"} The agent is a minimal echo (see /tmp/agentcore_deploy/agent.py for the deploy artifacts). Tests that only verify the SDK wiring will pass; if any test asserts on agent output content, swap the echo for the real agent. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(tests): point Bedrock batch tests at new-account S3 bucket The account migration (888602223428 -> 941277531214) was a flat account-ID swap, which only rewrites ARNs that embed the account number. S3 bucket names carry no account ID, so the live Bedrock batch tests still uploaded to `litellm-proxy` — a bucket that lives in the old account. S3 names are globally unique, and the old account still holds that name, so it can't be recreated in the new account. Rename to `litellm-proxy-941277531214` (account-ID suffix guarantees global uniqueness). The bucket must be created in 941277531214 and the batch execution role granted s3:GetObject/PutObject/ListBucket on it before this job is run in CI. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(tests): point live S3 logging test at new-account bucket Same account-ID-free blind spot as the batch bucket: `load-testing-oct` lives in the old account and its name can't be reused globally. The `logging_testing` CI job is wired into the workflow and runs test_basic_s3_logging, which uploads to this bucket with the CI env creds, then lists and deletes objects — a live dependency. Rename to `load-testing-oct-941277531214`. The bucket must exist in the new account with the CI IAM principal granted s3:PutObject/GetObject/ListBucket/DeleteObject before this job runs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(tests): repoint Bedrock guardrail IDs to new-account guardrails The migration left guardrail IDs untouched (no account ID in them), so all live guardrail tests failed with "guardrail identifier or version does not exist" against 941277531214. Recreated both guardrails in the new account and updated the hardcoded IDs: - wf0hkdb5x07f -> zgkmukebruil (PII mask: PHONE + CREDIT_DEBIT_CARD, with explicit inputAction=ANONYMIZE so masking applies to INPUT, which is the source litellm's moderation hook sends) - ff6ujrregl1q -> 4w3d1di3snt5 (blocks "coffee"; blocked message set to the exact string the tests assert on) Updated test_bedrock_guardrails.py, otel_test_config.yaml, and the guardrailConfig in test_bedrock_completion.py. Verified locally: the 5 previously-failing guardrail tests now pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(bedrock): migrate legacy models to current inference profiles The new CI account (941277531214) cannot invoke legacy Bedrock models (AWS gates them: "marked by provider as Legacy... not actively using in the last 30 days"). Migrated the live-call tests: - anthropic.claude-3-sonnet-20240229 -> us.anthropic.claude-sonnet-4-5-20250929-v1:0 - anthropic.claude-3-haiku-20240307 -> us.anthropic.claude-haiku-4-5-20251001-v1:0 Current Claude models on Bedrock require the us. inference-profile prefix (bare on-demand ids are rejected). cohere.command-r-plus has no working replacement (all Cohere is legacy- gated in the new account): swapped to claude-haiku-4-5 in provider- agnostic param lists. amazon.titan-image-generator skipped (no working replacement). Mocked/transformation/cost tests that reference the legacy strings are intentionally left unchanged. Verified live against the new account. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(bedrock): repoint SageMaker + Knowledge Base to new-account resources These referenced account-scoped resources by hardcoded id that only existed in the old account, so the migration's account-ID swap missed them. Recreated in 941277531214 and repointed: - SageMaker endpoint jumpstart-dft-hf-textgeneration1-mp-20240815-185614 -> litellm-ci-textgen (gpt2 on a TGI container, ml.g5.xlarge) - Bedrock Knowledge Base T37J8R4WTM -> LCYXFBR2TU (OpenSearch Serverless vector store + titan-embed-text-v2, seeded with a LiteLLM doc) Verified live: test_sagemaker.py (12 passed) and test_bedrock_knowledgebase_hook.py (12 passed). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(reasoning_effort_grid): skip bedrock claude-opus-4-7 cells (not entitled on 941277531214) claude-opus-4-7 is listed in the new Bedrock CI account's foundation models but invoke is denied (AccessDeniedException: "not available for this account"). Bedrock access to the flagship Opus requires an AWS Sales request, not the self-serve model-access toggle, so it can't be enabled inline with the rest of the account migration. Add an optional `skip_reason` to ModelEntry and set it on the bedrock-claude-opus-4-7 entry; the grid test honors it via pytest.skip. Cell count (231) and route coverage are unchanged, so the structural asserts still pass. Restore coverage by deleting the one skip_reason line once access is granted. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(bedrock): swap/skip legacy-gated models unavailable on new CI account The migrated AWS account (941277531214) cannot access several models that the old account could, so the remaining red CI jobs were hitting real Bedrock "Access denied / Legacy" and "account not authorized" errors: - image_gen: skip both Nova Canvas test classes (amazon.nova-canvas-v1:0 is legacy-gated), matching the existing titan skip. - batches: skip test_async_file_and_batch (Bedrock batch inference is not authorized on the new account; requires an AWS support case). - litellm_overhead: swap legacy claude-3-5-haiku for the active us.anthropic.claude-haiku-4-5 inference profile. - test_completion_claude_3_function_call: swap legacy claude-3-sonnet for the active us.anthropic.claude-sonnet-4-5 inference profile. https://claude.ai/code/session_01Y7zgHYu9GX29YRwV4yiWAa * test(bedrock): fix remaining e2e legacy-model + batch failures on new CI account - e2e_openai_endpoints: skip test_bedrock_batches_api (Bedrock batch inference is not authorized on account 941277531214) and migrate the missed s3_bucket_name in oai_misc_config.yaml to litellm-proxy-941277531214. - build_and_test: swap legacy bedrock claude-3-sonnet for the active us.anthropic.claude-sonnet-4-5 inference profile in the proxy structured output e2e test. https://claude.ai/code/session_01Y7zgHYu9GX29YRwV4yiWAa * test(bedrock): make opus-4-7 + batch cells fail loudly and mock image-gen (#28791) Replace the silent skips added for the new CI account with noisier behavior: - reasoning-effort grid: opus-4-7 cells now fail (when AWS creds are present) instead of skipping, so the missing entitlement stays visible in CI; they still skip when AWS creds are absent (local dev) - Bedrock batch inference tests: drop the skip so they run and fail until batch access is granted - Titan + Nova Canvas image-gen tests: mock the Bedrock HTTP call so the transform + cost-tracking path stays under test without live model access https://claude.ai/code/session_01MT7SWDnXUjv6e6EPG7BDjT Co-authored-by: Claude <noreply@anthropic.com> * test(bedrock): use pytest.xfail for known-failing opus-4-7 cells Replace pytest.fail with pytest.xfail when a model has a fail_reason, so known-broken cells stay visible as XFAIL without keeping CI red. Co-authored-by: Yassin Kortam <yassin@berri.ai> --------- Co-authored-by: Mateo <mateo@Mateos-MacBook-Pro.local> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(otel): export SERVER span on management-endpoint success without http_request (#28794) Co-authored-by: Yassin Kortam <yassinkortam@Yassins-MacBook-Pro.local> * chore(ci): merge dev branch (#28801) * chore(proxy): route path-dependent call sites through get_request_route Replace direct ``request.url.path`` reads in auth, ACL, routing, and audit-log decisions with ``get_request_route(request)`` — the helper already added in ``auth/auth_utils.py`` that returns the ASGI ``scope["path"]`` with ``root_path`` stripped. Starlette reconstructs ``url.path`` from the Host header; ``scope["path"]`` is uvicorn's parse of the request line and matches what FastAPI dispatches on, so it's the authoritative route for any decision that should agree with the actual handler. Sites: - _experimental/mcp_server/auth/user_api_key_auth_mcp.py - management_endpoints/mcp_management_endpoints.py - vector_store_endpoints/utils.py - pass_through_endpoints/pass_through_endpoints.py - auth/route_checks.py - litellm_pre_call_utils.py - spend_tracking/spend_management_endpoints.py - common_utils/http_parsing_utils.py - management_helpers/utils.py - health_endpoints/_health_endpoints.py Adds regression tests in tests/proxy_unit_tests/test_proxy_routes.py that construct a Request with scope["path"] set to a benign route and the Host header crafted so url.path would resolve differently; each site's decision is asserted against scope["path"]. * chore(proxy): make get_request_route imports lazy at call sites Move the ``from litellm.proxy.auth.auth_utils import get_request_route`` imports added in the prior commit back to the function bodies that use them. The module-level form participates in a long-standing import cycle through ``auth_utils -> _types -> ...`` and was flagged by CodeQL on the PR; the lazy form matches the pattern the proxy already uses for ``user_api_key_auth`` and related helpers elsewhere in these files. Also drop the ``RouteChecks._is_assistants_api_request`` delegation in ``_get_metadata_variable_name`` introduced in the prior commit — the delegation pulled ``RouteChecks`` into the same cycle, and the call site reuses the resolved route for its other branches, so inlining the substring check is both cycle-free and avoids a redundant second ``get_request_route`` call. Comment in test_proxy_routes.py acknowledges that the two MCP table entries exercise ``get_request_route`` directly rather than the full production handler (which needs ASGI scope + MCP state to invoke). --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: user <70670632+stuxf@users.noreply.github.com> * chore(ci): merge dev branch (#28657) * feat(dashboard): navbar hierarchy + Agent Platform notifications (#27543) * feat(dashboard): refine navbar zones and Agent Platform notice Restructure the admin navbar for production users: clear product vs community vs personal columns with vertical dividers, icon-only Slack/GitHub in a shared chip, and Docs/Blog typography aligned on an 8px rhythm. Add a notifications bell with popover linking to the LiteLLM Agent Platform repo and optional mark-as-read persistence. Promote the account control with initials avatar, single-line display name, and navDisplayName mapping for placeholder user ids (e.g. default_user_id). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(dashboard): address PR review — AntD buttons, public page guard, dedupe regex - Replace raw <button> with AntD Button in BlogDropdown, NotificationsBell, UserDropdown, and test mock - Guard NotificationsBell + container behind !isPublicPage to avoid rendering on public pages - Remove redundant equality checks in navDisplayName (regex already covers them) - Remove unused `lower` variable after simplification Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * fix(dashboard): drop dead useHealthReadiness import in navbar The module was removed in #27896 (replaced by useHealthReadinessDetails), but the import survived the rebase. The symbol is unused — only useHealthReadinessDetails is consumed in the file. Removing the dead import unblocks the UI TypeScript build. * fix(dashboard): align CommunityEngagementButtons test with icon-only aria-labels The component was refactored to an icon-only chip with aria-label='LiteLLM on GitHub' (squash #27543), but the test still asserted /star us on github/i. Update the query to match the rendered accessible name. * refactor(dashboard): drop unused props from NavbarProps The navbar refactor moved user identity + dark-mode state to internal hooks (useAuthorized, useWorker), but the NavbarProps interface still declared userID, userEmail, userRole, premiumUser, isDarkMode, and toggleDarkMode as required, forcing every caller to thread them through. Drop them from the interface and all four call sites (page.tsx, (dashboard)/layout.tsx, public_model_hub.tsx, navbar.test.tsx). Also shrinks the destructure in layout.tsx so the now-unused locals stop being pulled out of useAuthorized(). * refactor(dashboard): use useSyncExternalStore for NotificationsBell dismiss flag Reads/writes of the litellmHideAgentPlatformBanner key were done directly inside NotificationsBell via a useEffect + useState pair. Every other localStorage-backed flag in the dashboard (Disable ShowPrompts, DisableBouncingIcon, DisableShowNewBadge, DisableUsageIndicator, DisableBlogPosts) is wrapped in a useSyncExternalStore hook over localStorageUtils so all mounted components stay in sync. Extract useHideAgentPlatformBanner to follow the same shape, swap NotificationsBell to consume it, and add a regression test that two sibling bells stay in sync without a remount when one is dismissed. * refactor: mask credential fields in proxy settings GET responses (#28682) * refactor: mask credential fields in proxy settings GET responses Brings SSO settings, cache settings, and the email/Slack alerting view in /get/config/callbacks in line with the HashiCorp Vault config-override pattern, so persisted credentials are not transported back to the UI in plaintext. * refactor: harden short-value masking and hoist alerting var constant Closes two review observations: - mask_sensitive_keys now replaces short values (below the visible prefix+suffix length) with an all-mask string instead of returning them unchanged, so a 1-7 character credential is no longer round-tripped verbatim. - _ALERTING_SENSITIVE_VARS is moved out of get_config() to a module-level constant, matching the analogous _SSO_SENSITIVE_FIELDS and _CACHE_SENSITIVE_FIELDS in the SSO and cache endpoint files. --------- Co-authored-by: Krrish Dholakia <krrish+github@berri.ai> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * fix(ui): show 2-decimal precision for max_budget on key overview (#28809) The Key Info Overview tab's Spend card truncated sub-dollar budgets to "$0" because formatNumberWithCommas defaults to 0 decimals. The Settings tab passes 2; align the overview so a $0.10 budget renders as "$0.10". Resolves LIT-2845 * feat(proxy): allow `llm_api_routes` virtual keys to list MCP servers (#28442) * feat(proxy): allow llm_api_routes virtual keys to list MCP servers Add a new `mcp_discovery_routes` group (GET /v1/mcp/server and GET /v1/mcp/server/{server_id}) and include it in `llm_api_routes` so that virtual keys configured with `allowed_routes=["llm_api_routes"]` can discover the MCP servers they have access to. Previously these calls failed with 'Virtual key is not allowed to call this route. Only allowed to call routes: [llm_api_routes]'. The GET handlers already sanitize the response for restricted virtual keys via `_sanitize_mcp_server_list_for_virtual_key`, stripping credential-bearing fields (url, headers, env). Write methods (POST/PUT/DELETE) on the same paths remain gated by the existing handler-level admin role checks. The new discovery list is intentionally kept OUT of `mcp_inference_routes`, so `is_llm_api_route()` still returns False for these paths — this preserves the existing contract that DISABLE_LLM_API_ENDPOINTS must not block the Admin UI from listing MCP servers. Co-authored-by: ryan-crabbe-berri <ryan-crabbe-berri@users.noreply.github.com> * refactor(proxy): make MCP discovery carve-out method-aware Replace the `mcp_discovery_routes` group in `llm_api_routes` with a method-aware special case inside `is_virtual_key_allowed_to_call_route`. Virtual keys with allowed_routes=["llm_api_routes"] are now permitted to call only GET /v1/mcp/server and GET /v1/mcp/server/{server_id} — non-GET methods and multi-segment admin sub-paths fall through to the existing 403. This keeps the general llm_api_routes list free of management paths and avoids accidentally exposing POST/PUT/DELETE writes through the route-check layer. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: ryan-crabbe-berri <ryan-crabbe-berri@users.noreply.github.com> * chore(ci): merge dev branch (#28807) * chore(proxy): route path-dependent call sites through get_request_route Replace direct ``request.url.path`` reads in auth, ACL, routing, and audit-log decisions with ``get_request_route(request)`` — the helper already added in ``auth/auth_utils.py`` that returns the ASGI ``scope["path"]`` with ``root_path`` stripped. Starlette reconstructs ``url.path`` from the Host header; ``scope["path"]`` is uvicorn's parse of the request line and matches what FastAPI dispatches on, so it's the authoritative route for any decision that should agree with the actual handler. Sites: - _experimental/mcp_server/auth/user_api_key_auth_mcp.py - management_endpoints/mcp_management_endpoints.py - vector_store_endpoints/utils.py - pass_through_endpoints/pass_through_endpoints.py - auth/route_checks.py - litellm_pre_call_utils.py - spend_tracking/spend_management_endpoints.py - common_utils/http_parsing_utils.py - management_helpers/utils.py - health_endpoints/_health_endpoints.py Adds regression tests in tests/proxy_unit_tests/test_proxy_routes.py that construct a Request with scope["path"] set to a benign route and the Host header crafted so url.path would resolve differently; each site's decision is asserted against scope["path"]. * chore(proxy): make get_request_route imports lazy at call sites Move the ``from litellm.proxy.auth.auth_utils import get_request_route`` imports added in the prior commit back to the function bodies that use them. The module-level form participates in a long-standing import cycle through ``auth_utils -> _types -> ...`` and was flagged by CodeQL on the PR; the lazy form matches the pattern the proxy already uses for ``user_api_key_auth`` and related helpers elsewhere in these files. Also drop the ``RouteChecks._is_assistants_api_request`` delegation in ``_get_metadata_variable_name`` introduced in the prior commit — the delegation pulled ``RouteChecks`` into the same cycle, and the call site reuses the resolved route for its other branches, so inlining the substring check is both cycle-free and avoids a redundant second ``get_request_route`` call. Comment in test_proxy_routes.py acknowledges that the two MCP table entries exercise ``get_request_route`` directly rather than the full production handler (which needs ASGI scope + MCP state to invoke). --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: user <70670632+stuxf@users.noreply.github.com> * fix(team): keep team_alias cache in sync on _cache_team_object writes (#28737) * fix(team): keep team_alias cache in sync on _cache_team_object writes _cache_team_object wrote only to the team_id:<id> cache key, but the JWT auth path that uses team_alias_jwt_field reads from a separate team_alias:<alias> key (get_team_object_by_alias caches under both keys on miss, but reads only the alias-keyed one). After any team-mutation endpoint (team_model_add, team_model_delete, update_team, the two access-group writes) the team_id cache was refreshed but the team_alias cache stayed stale until TTL — JWT callers using team_alias_jwt_field kept seeing the pre-mutation team for the full cache window. Mirror the write under the alias key inside _cache_team_object so every existing caller stays in sync without further changes. Skip the alias write when team_alias is None/empty so we don't collide across alias-less teams. Surfaced testing the LIT-3244 cherry-pick on patch/1.86.0: the LIT-3244 fix correctly invalidated the team_id cache but the customer's JWT used team_alias_jwt_field, so they kept hitting the stale alias-keyed entry. * fix(team): delete (not overwrite) team_alias cache on _cache_team_object The prior shape of this PR wrote both team_id:<id> AND team_alias:<alias> from _cache_team_object. team_alias is NOT unique in the schema (no @unique on LiteLLM_TeamTable.team_alias), and get_team_object_by_alias enforces uniqueness on its own DB-fetch path (len(teams) > 1 raises). Writing the alias-keyed cache from the generic refresh path bypassed that check: a team admin renaming their team to collide with another team's alias could silently overwrite the cached team for JWT-by-alias auth, swapping the resolved team under that alias for the cache window. Switch the alias-keyed operation from a write to a delete (mirroring the dual-cache delete pattern in _delete_cache_key_object). After every team write, the next JWT-by-alias reader cache-misses and falls through to get_team_object_by_alias, which (a) re-fetches the fresh team from DB, closing the LIT-3244 staleness gap that motivated this PR, and (b) enforces alias uniqueness before populating either cache key. team_id:<id> writes are unchanged — team_id is the table PK and is guaranteed unique. Surfaced in veria-ai review on #28739. * fix(managed-files): anchor model_id regex so it doesn't match llm_output_file_model_id extract_model_id_from_unified_id used `re.search(r"model_id,([^;]+)", ...)` which substring-matches the `model_id,` inside the file-ID encoding's `llm_output_file_model_id,<deployment_uuid>` field. parse_unified_id then fed that deployment UUID back into the auth path as a model candidate via _extract_models_from_managed_resource_id, and every team-BYOK file attach 403'd with: team not allowed to access model. This team can only access models=['openai/*']. Tried to access <deployment-uuid> The team's models list correctly contains the public name (`openai/*`) that target_model_names matches, but the bogus UUID candidate fails the wildcard check first. Anchor the regex to a field boundary (`(?:^|;)model_id,`) so it matches the legitimate top-level `model_id,<value>` field on vector_store unified IDs and skips substring matches inside other fields. File-IDs (which have no top-level `model_id` field) now return None and contribute no spurious UUID candidate. Surfaced reproducing LIT-3244 on patch/1.86.0 with the customer's exact flow: team with openai/* BYOK deployment, JWT-scoped user, POST /v1/vector_stores/{id}/files attaching a file uploaded with target_model_names=openai/gpt-4o. * fix(proxy): hydrate wildcard discovery credentials (#28284) (#28822) * fix(proxy): hydrate wildcard discovery credentials * fix(proxy): constrain wildcard credential hydration Co-authored-by: Dibyo Mukherjee <dibyo@adobe.com> * ci: add daily oss-agent-shin branch creation workflow (#28829) Creates litellm_oss_agent_shin_MM_DD_YYYY from main every day at 00:00 UTC. Lets us retarget oss-agent-shin fork PRs onto a canonical branch so CircleCI runs with secrets, without granting the agent write access. Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com> * test(proxy): add harness for proxy_server.py behavior-pinning (#28827) * test(proxy): add harness for proxy_server.py behavior-pinning Creates tests/test_litellm/proxy/proxy_server/ with: - conftest.py: 11 shared fixtures (app, client, mock_prisma, auth_as, mock_router with parametrized response builders, normalize, etc.) - _coverage_check.py: per-PR coverage gate (line + branch) against a baseline, self-selects target by inspecting which placeholder files have been filled - _pin_check.py: AST-based gate that verifies every pin-list item has >=1 happy + >=1 error test with a real assertion (no status-only) - test_harness_smoke.py: 19 smoke tests covering every fixture + both scripts end-to-end - 26 placeholder test files (one docstring each) reserved for follow-up PRs per the directory ownership in the Notion plan - .coverage_baseline pinned at 0% so future PRs measure deltas against new-tests-only and aren't entangled with the broader scattered test suite Adds a dedicated proxy-server job to test-unit-proxy-endpoints.yml so this directory's runtime + coverage are tracked independently. Plan: https://www.notion.so/36c43b8acdab81ee845fd5365128a2fc * ci(proxy-endpoints): allow workflow_dispatch Lets the workflow be triggered manually on a branch via `gh workflow run`, which is needed for the verify-first flow on workflow changes before opening a PR. * test(proxy): address review feedback on proxy_server harness - conftest.py: anchor sys.path insert to __file__ (Path(__file__).resolve().parents[4]) instead of CWD-relative os.path.abspath("../../../../") which resolved to the wrong directory when pytest is launched from the repo root. - _coverage_check.py: actually read .coverage_baseline and use it as the floor (line_min = max(target, baseline)). Closes the gap between the PR description's "delta semantics" and what the script was doing. With baseline=0.0 today this is a no-op; future PRs that update the baseline cause regressions (test deletions etc.) to trip the gate even if the static PR target is still met. - _pin_check.py: drop unreachable startswith("_") guard (test_*.py glob never yields underscore-prefixed names) and read each test file once instead of twice. * feat(openai): apply regional-processing cost uplift for EU/US data residency (#28626) * feat(openai): apply regional-processing cost uplift for EU/US data residency OpenAI charges a 10% uplift on the latest GPT models when requests are served from a regionalized hostname (eu./us.api.openai.com). Infer the region from `api_base`, expose it on `kwargs["litellm_params"]["data_residency"]`, and multiply the computed cost by a per-model `regional_processing_uplift_multiplier_<region>` field. https://claude.ai/code/session_012ebH44s7ohYxjoix5CXzTW * test: allow regional_processing_uplift_multiplier_{eu,us} in model_prices schema * fix(cost): tighten data_residency inference and restore model_cost in tests - Only infer OpenAI data_residency when custom_llm_provider == "openai"; drop the implicit None fallback so non-OpenAI callers can't accidentally pick up a regional tag from a stray OpenAI hostname. - _local_model_cost_map fixture now snapshots and restores litellm.model_cost and LITELLM_LOCAL_MODEL_COST_MAP so tests don't leak state across the session. * refactor(openai): move data_residency helper under llms/openai * fix: thread data_residency through realtime stream cost calculation Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(cost): thread data_residency through batch_cost_calculator Apply the OpenAI regional-processing uplift multiplier to retrieve_batch cost paths so Batch API requests served via eu./us.api.openai.com are priced at the same uplifted token rates as completions/transcriptions. * refactor(openai): encapsulate provider check inside infer_openai_data_residency Move the custom_llm_provider == "openai" guard from get_litellm_params into the helper itself so the core utility no longer carries provider-specific dispatch logic. Callers pass through the provider unconditionally; the helper returns None for any non-OpenAI provider. * fix(responses): thread data_residency through Responses logging params The Responses API paths build their logging litellm_params dict after provider resolution but did not include data_residency, so cost calc saw None even when the effective api_base was a regional OpenAI host. --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> --------- Co-authored-by: milan-berri <milan@berri.ai> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Mateo <mateo@Mateos-MacBook-Pro.local> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: Yassin Kortam <yassinkortam@Yassins-MacBook-Pro.local> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: user <70670632+stuxf@users.noreply.github.com> Co-authored-by: Krrish Dholakia <krrish+github@berri.ai> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> Co-authored-by: ryan-crabbe-berri <ryan-crabbe-berri@users.noreply.github.com> Co-authored-by: Dibyo Mukherjee <dibyo@adobe.com> Co-authored-by: ishaan-berri <155045088+ishaan-berri@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com> * fix: preserve OTEL response payload and remove duplicate constant - _emit_management_endpoint_otel_span now passes result as response on success - remove duplicate _CREDENTIAL_LITELLM_PARAM_FIELDS assignment in model_checks Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix: address bug detection findings - pass_through_endpoints: use request.method instead of hardcoded POST in streaming SigV4-signed request path for consistency with the non-streaming branch - llm_cost_calc/utils: hoist DataResidency value set to a module-level frozenset to avoid rebuilding it on every cost calculation - example_config_yaml/oai_misc_config: replace real-looking AWS account ID with placeholder 123456789012 in example bucket and role ARN Co-authored-by: Yassin Kortam <yassin@berri.ai> * chore(github_copilot): refresh model catalog from upstream /models API (#28055) Aligns the github_copilot catalog with values returned by Copilot's public /models endpoint (capabilities.limits + capabilities.supports + model.supported_endpoints). - Adds 10 new model entries: claude-opus-4.7, claude-sonnet-4.6, gemini-3-flash-preview, gemini-3.1-pro-preview, gpt-4-0125-preview, gpt-5.2-codex, gpt-5.4, gpt-5.4-mini, gpt-5.5, oswe-vscode-prime. - Updates max_input_tokens for existing entries to reflect each model's true context window (e.g. gpt-4o-mini 64000 -> 128000, gpt-5-mini 128000 -> 264000, gpt-5.3-codex 128000 -> 400000, claude-haiku-4.5 128000 -> 200000). - Adds supports_reasoning, supports_response_schema, supports_function_calling, supports_parallel_function_calling, supports_vision based on capabilities.supports. - Declares supported_endpoints for entries missing it (e.g. gpt-3.5-turbo, gpt-4o, embeddings). - For responses-only models (gpt-5.2-codex, gpt-5.4, gpt-5.4-mini, gpt-5.5), sets mode to 'responses'. - gpt-41-copilot.mode changes from 'completion' to 'chat' because Copilot reports capabilities.type = 'chat'. Revertible on request. Pricing fields and other manually-curated values are preserved. * feat(datadog): emit litellm.overhead.latency as a standalone Datadog metric (#28831) Adds a new `litellm.overhead.latency` gauge metric to `DatadogMetricsLogger` (the `/api/v2/series` path). The value is sourced from `hidden_params["litellm_overhead_time_ms"]` already computed in `ResponseMetadata` and exposed in `StandardLoggingPayload`. Matches the Prometheus integration which exposes the same value via `litellm_overhead_latency_metric`. Emitted in seconds (ms ÷ 1000) for consistency with the other latency series. Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Shin <shin@litellm.ai> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: ishaan-berri <155045088+ishaan-berri@users.noreply.github.com> * feat(arize): route Phoenix traces via per-project TracerProviders (#28876) Use LRU-cached TracerProviders with project-scoped OTEL Resources so team/key metadata routes traces correctly. On the proxy, project selection is limited to server-controlled user_api_key_auth_metadata; client metadata fields stay banned. * fix(arize_phoenix): skip _emit_semantic_logs on failure path Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(arize_phoenix): skip raw request logging and metrics on failure path Restores pre-refactor behavior: _handle_failure no longer emits raw-request sub-spans or records OTEL metrics, matching the original _handle_failure that did not call these helpers. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(security): close two medium telemetry trust-boundary issues Issue 1 (arize_phoenix.py — caller-controlled telemetry routing): - _is_proxy_request no longer detects proxy mode by checking user_api_key_auth_metadata in request metadata. That field is user-supplied, so an authenticated caller could fake proxy-mode detection and have _project_from_metadata_dict read their own dict for project selection, routing telemetry to arbitrary Arize/Phoenix projects. Proxy mode is now determined solely by the server-set proxy_server_request field in litellm_params. - auth_utils.py adds user_api_key_auth_metadata to the banned request body params list so the proxy rejects any attempt to supply the field at the HTTP layer. The field is server-reserved: it is written exclusively by add_user_api_key_auth_to_request_metadata from the authenticated key's database record after the ban check runs. Issue 2 (management_helpers/utils.py — API key in OTEL span): - _emit_management_endpoint_otel_span stripped plaintext credential fields (key, token, api_key, secret, …) from the response dict before passing it to the OTEL success hook. dict(result) on a Pydantic GenerateKeyResponse includes the freshly-generated key field, which would previously be written as a span attribute to every configured OTEL collector/backend. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: Terrajlz <info@jouleselectrictech.com> Co-authored-by: Bruno Devaux <devaux.br@gmail.com> Co-authored-by: milan-berri <milan@berri.ai> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Mateo <mateo@Mateos-MacBook-Pro.local> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Yassin Kortam <yassinkortam@Yassins-MacBook-Pro.local> Co-authored-by: user <70670632+stuxf@users.noreply.github.com> Co-authored-by: Krrish Dholakia <krrish+github@berri.ai> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> Co-authored-by: ryan-crabbe-berri <ryan-crabbe-berri@users.noreply.github.com> Co-authored-by: Dibyo Mukherjee <dibyo@adobe.com> Co-authored-by: ishaan-berri <155045088+ishaan-berri@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com> Co-authored-by: rinto <54238243+ririnto@users.noreply.github.com> Co-authored-by: Shin <shin@litellm.ai> Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
#28875) * fix(thinking): handle None thinking param in is_thinking_enabled (#28598) Squash-merged by litellm-agent from Terrajlz's PR. * feat(helm): support tpl rendering in podAnnotations (#28609) Squash-merged by litellm-agent from devauxbr's PR. * feat(bedrock-batch): route /v1/embeddings JSONL to Titan v2 modelInput Changes vs main: - BedrockFilesConfig now detects OpenAI batch JSONL lines whose `url` is /v1/embeddings (with body-shape fallback) and routes them through a new `_map_openai_embedding_to_bedrock_params` helper instead of the chat-completion transformer that silently produces an invalid body. - The embedding helper currently supports Amazon Titan Text Embeddings V2 only. Other embed models (Titan G1, Titan Multimodal, Cohere Embed, Nova Multimodal Embeddings) raise NotImplementedError with a clear message; each will get a dedicated branch + tests in follow-up PRs to keep schema-specific risks isolated. - Validation refuses pre-tokenized inputs (List[int], List[List[int]]) and multi-element string lists with explicit errors so callers emit one JSONL line per embedding instead of relying on us to fan out. - Titan v2 model id match tolerates "bedrock/" prefix, cross-region inference profile prefix ("us.", "eu.", etc.), and ARN forms; the marker boundary check rejects lookalikes like "titan-embed-text-v20". - Tests cover happy path (fixtures), dimensions/encoding_format mapping, body-shape fallback, single-element list unwrap, error paths (missing input, multi-element list, unsupported model, pre-tokenized), mixed chat+embedding batch, and the model-id boundary check. * fix(bedrock-batch): trust explicit url over body shape in record routing Greptile-flagged gap in `_is_embedding_record`: when an OpenAI batch JSONL line carries an explicit `url` pointing to a non-embedding endpoint (e.g. `/v1/chat/completions`) AND its body happens to have `input` without `messages`, the body-shape fallback would mis-route that record to the embedding transformer and corrupt the modelInput. Changes vs previous commit: - `_is_embedding_record` now short-circuits to NOT-embedding whenever `url` is non-empty and not equal to `/v1/embeddings`. The body-shape fallback only runs when `url` is missing or empty. Docstring updated to spell out the precedence rules. - Two new tests cover the case: direct helper assertion that an explicit chat url plus an input-bearing body returns False, plus an end-to-end check that the resulting modelInput contains no `inputText` key. A second test asserts the same short-circuit for arbitrary non-embeddings urls (`/v1/completions`, `/v1/responses`). 28/28 tests pass (was 26/26 before this commit + 2 new). * refactor(bedrock-batch): extract embedding input normalization helper Splits the input-shape validation out of `_map_openai_embedding_to_bedrock_params` into a new static helper `_coerce_embedding_input_to_string`. Same semantics; the goal is to make the validation testable in isolation and to give future embedding-provider branches (Titan G1, Cohere) a reusable shaping function instead of duplicating type checks. - Helper accepts `str`, single-element `list[str]`, and raises `ValueError` / `NotImplementedError` with actionable messages for None, multi-element lists, pre-tokenized inputs (`list[int]` / `list[list[int]]`), and other unsupported types. - New unit test exercises the helper directly across happy paths, None / missing input, multi-element string list, multi-element int list (caught as 'one input per JSONL record' since we can't disambiguate from 'multiple strings' without more context), pre-tokenized single-element list-of-list, single-element list of bare int, and dict input. 29/29 tests in the file still pass. * fix(bedrock-batch): layer registry mode check + drop unused provider param Greptile-flagged issues on Titan v2 model detection: 1. Hardcoded substring without registry consultation conflicts with the project convention of treating `model_prices_and_context_window.json` as the source of truth for model capability flags. Fix: `_is_titan_v2_embed_model` now layers a registry mode check on top of the existing marker boundary. When `get_model_info` resolves the id, we additionally require `mode == "embedding"` so a malformed id whose path-component matches the marker but whose registered mode is "chat" doesn't slip through. Registry silence (cross-region inference profile prefixes like `us.amazon.titan-embed-text-v2:0`, ARN forms) keeps the substring-only behavior because the registry genuinely can't normalize those ids today. The substring is still needed because `mode == "embedding"` alone doesn't distinguish Titan v2's InvokeModel schema from Cohere Embed, Nova Multimodal, or Titan G1 - all also embedding mode, all with incompatible bodies. 2. `provider` parameter on `_map_openai_embedding_to_bedrock_params` was accepted but never used. Fix: dropped from the signature and the call site. Also adds `_lookup_registry_mode` static helper (mirrors the one in the sibling batches transformer) so the registry try/except shape lives in one place instead of being inlined into the detector. 5 new tests pin the layered behavior: - registry mode=chat overrides the marker match (rejected) - registry mode=embedding + marker match (accepted) - registry silent + marker match for cross-region and ARN ids (accepted) - direct `_lookup_registry_mode` coverage across all return paths 33/33 tests in the file pass. * fix(bedrock-batch): drive Titan v2 detection from registry provider_specific_entry Addresses Greptile's remaining policy concern on PR #28875: the hardcoded `_TITAN_V2_EMBED_MODEL_MARKER` substring required a code change to register new Titan v2 variants. Now the registry is the source of truth. Changes: - model_prices_and_context_window.json + model_prices_and_context_window_backup.json Adds `provider_specific_entry.bedrock_invocation_schema = "titan_v2"` to the `amazon.titan-embed-text-v2:0` entry. Uses the existing `provider_specific_entry` escape-hatch field (already surfaced by `get_model_info` via `ModelInfo.provider_specific_entry`) rather than introducing a new top-level field that would need a corresponding change in `utils.py::get_model_info` to flow through. - BedrockFilesConfig._is_titan_v2_embed_model now reads `get_model_info(model).provider_specific_entry.bedrock_invocation_schema` first. When the registry resolves the id we trust the field; registered ids without (or with a different) schema value are rejected outright - no substring second-chance for registered ids. The substring fallback only runs when `get_model_info` raises, which covers cross-region inference profile prefixes (`us.amazon.titan-embed-text-v2:0`) and Bedrock ARN forms - neither of which the registry normalizes today. - _lookup_registry_mode helper renamed to _lookup_provider_specific_field and generalized: takes a field name, reads `provider_specific_entry[field]` defensively. Future embed-schema branches (Titan G1, Cohere, Nova Multimodal) will share this helper. - New module-level constants: _BEDROCK_INVOCATION_SCHEMA_FIELD names the registry key; _TITAN_V2_INVOCATION_SCHEMA names the value. - _map_openai_embedding_to_bedrock_params: dropped the unused `provider` parameter (separate Greptile flag). Tests updated for the new nested-field semantics. 34/34 tests pass; one end-to-end integration check confirms that with LITELLM_LOCAL_MODEL_COST_MAP=true, the registry returns the new field and all 7 representative model ids classify correctly. * chore(lint): apply Black formatting to is_thinking_enabled Pre-existing Black violation on the shin_agent_oss_staging_05_22_2026 base branch - the lint CI check on this PR fails on `litellm/llms/base_llm/chat/transformation.py` even with zero changes from our side. Applying the formatter's preferred line-break inside `is_thinking_enabled` unblocks the lint check without touching the method's logic. --------- Co-authored-by: Terrajlz <info@jouleselectrictech.com> Co-authored-by: Bruno Devaux <devaux.br@gmail.com>
* fix(thinking): handle None thinking param in is_thinking_enabled (#28598) Squash-merged by litellm-agent from Terrajlz's PR. * feat(helm): support tpl rendering in podAnnotations (#28609) Squash-merged by litellm-agent from devauxbr's PR. * Add 1-hour cache write pricing for us-gov Haiku 4.5 AWS Bedrock GovCloud applies a +20% premium over global Anthropic rates. Global Haiku 4.5 5m/1h cache write is $1.25 / $2.00 per MTok; us-gov is therefore $1.50 / $2.40 per MTok (the 5m rate was already correct in litellm; the 1h field was missing). Adds `cache_creation_input_token_cost_above_1hr: 2.4e-06` to the two us-gov Haiku 4.5 entries in both pricing JSON files: - bedrock/us-gov-east-1/anthropic.claude-haiku-4-5-20251001-v1:0 - bedrock/us-gov-west-1/anthropic.claude-haiku-4-5-20251001-v1:0 New parametrized regression test tests/test_litellm/test_bedrock_usgov_haiku_1hr_cache.py pins both entries and enforces the 1.6x 5m-to-1h ratio invariant matching the pattern used by the existing Bedrock and Vertex 1h-cache tests. Companion to the us-gov Sonnet 4.5 pricing fix. * chore: trigger shin-agent re-eval on retargeted staging base * chore: trigger shin-agent re-eval against updated Greptile state --------- Co-authored-by: Terrajlz <info@jouleselectrictech.com> Co-authored-by: Bruno Devaux <devaux.br@gmail.com> Co-authored-by: Sameer Kankute <sameer@berri.ai>
* fix(thinking): handle None thinking param in is_thinking_enabled (#28598) Squash-merged by litellm-agent from Terrajlz's PR. * feat(helm): support tpl rendering in podAnnotations (#28609) Squash-merged by litellm-agent from devauxbr's PR. * Fix us-gov-{east,west}-1 Bedrock Sonnet 4.5 pricing (Fixes #27120) AWS GovCloud Bedrock pricing carries a +20% premium over the global Anthropic rates, not the +10% premium that applies to commercial US regions. Litellm's five us-gov Claude Sonnet 4.5 entries mirror the commercial-US prices (3.3e-06 input, 1.65e-05 output, 4.125e-06 cache write), undercharging customers by ~9%. Corrects all five affected entries in both model_prices_and_context_window.json and the bundled model_prices_and_context_window_backup.json to match AWS's published GovCloud rates (per https://aws.amazon.com/bedrock/pricing/): - input_cost_per_token 3.3e-06 -> 3.6e-06 ($3.60/MTok) - output_cost_per_token 1.65e-05 -> 1.8e-05 ($18.00/MTok) - cache_creation_input_token_cost 4.125e-06 -> 4.5e-06 ($4.50/MTok) - cache_read_input_token_cost 3.3e-07 -> 3.6e-07 ($0.36/MTok) - cache_creation_input_token_cost_above_1hr (added) ($7.20/MTok) Affected entries: - bedrock/us-gov-east-1/anthropic.claude-sonnet-4-5-20250929-v1:0 - bedrock/us-gov-west-1/anthropic.claude-sonnet-4-5-20250929-v1:0 - bedrock/us-gov-east-1/claude-sonnet-4-5-20250929-v1:0 - bedrock/us-gov-west-1/claude-sonnet-4-5-20250929-v1:0 - us-gov.anthropic.claude-sonnet-4-5-20250929-v1:0 New regression test tests/test_litellm/test_bedrock_usgov_pricing.py pins the corrected rates across all five entries and asserts the 1.2x global-to-us-gov ratio invariant, matching the pattern used by the existing Vertex and EU/AU/JP Bedrock pricing tests. * chore: trigger shin-agent re-eval on retargeted staging base * Apply us-gov 200k-tier GovCloud premium to cross-region sonnet-4-5 The us-gov.anthropic.claude-sonnet-4-5-20250929-v1:0 cross-region inference profile carried _above_200k_tokens fields at the +10% commercial-US rate while the base tier was correctly at +20%. AWS GovCloud pricing applies the same +20% uplift across all tiers, so long-context requests through this profile were undercharging by ~9%. Updates four existing fields (input/output/cache_creation/cache_read _above_200k_tokens) and adds cache_creation_input_token_cost_above_1hr _above_200k_tokens for parity with the us. cross-region entry. Extends test_bedrock_usgov_pricing.py with five parametrized field assertions plus a 1.2x ratio invariant across all 200k-tier fields, so any future regression on either dimension is caught. * chore: trigger shin-agent re-eval against updated Greptile state --------- Co-authored-by: Terrajlz <info@jouleselectrictech.com> Co-authored-by: Bruno Devaux <devaux.br@gmail.com> Co-authored-by: Sameer Kankute <sameer@berri.ai>
…28572) * fix(thinking): handle None thinking param in is_thinking_enabled (#28598) Squash-merged by litellm-agent from Terrajlz's PR. * feat(helm): support tpl rendering in podAnnotations (#28609) Squash-merged by litellm-agent from devauxbr's PR. * Forward custom_llm_provider through the Responses API bridge (Fixes #28505) (#28575) * Forward custom_llm_provider through the Responses API bridge (Fixes #28505) When a Chat Completions request to a GPT-5.4+ model contains both `tools` and `reasoning_effort`, `completion()` auto-routes through `responses_api_bridge`. The bridge handler called `litellm.responses()` / `litellm.aresponses()` without forwarding the already-resolved `custom_llm_provider`, so the downstream call re-invoked `get_llm_provider()` with `custom_llm_provider=None` and stripped a second provider prefix from a `provider/provider/model` deployment string. For a deployment configured as `openai/openai/openai/gpt-5.5`, the bridge flow sent `openai/gpt-5.5` to the upstream API instead of the correct `openai/openai/gpt-5.5`. Upstream APIs that enforce model-name allow-lists rejected this as `key_model_access_denied`. Fix: pass the locally-resolved `custom_llm_provider` into both the sync `responses()` and async `aresponses()` calls so the downstream `_resolve_model_provider_for_responses` sees an explicit provider and skips the second prefix-strip. New regression test `tests/test_litellm/completion_extras/test_responses_bridge_provider_propagation.py` pins both call sites: each must forward `custom_llm_provider`. * fix(28505): set custom_llm_provider on request_data instead of as duplicate kwarg Greptile flagged that the previous patch passed custom_llm_provider as an explicit kwarg to responses()/aresponses() while request_data already carried it via the spread of sanitized_litellm_params, which would raise TypeError: got multiple values for keyword argument on every real bridge call. Switches to assigning request_data['custom_llm_provider'] before the call so the resolved provider wins over whatever sanitized_litellm_params spread in, without duplicating the kwarg. Updates the regression test to seed request_data with a sentinel custom_llm_provider so it actually exercises the overwrite path (the previous test mocked transform_request with a minimal dict and never hit the conflict). * chore: trigger shin-agent re-eval on retargeted staging base * chore: trigger shin-agent re-eval against updated Greptile state * Add 1-hour cache write pricing for EU/AU/JP Bedrock Anthropic models The 1-hour prompt-cache write tier (`cache_creation_input_token_cost_above_1hr`) was added to the us./global. variants of the Claude 4.5/4.6/4.7 family on Bedrock, but the eu./au./jp. cross-region inference profiles were left without it. AWS Bedrock pricing applies the same +10% regional premium across all geo profiles, so eu./au./jp. should carry the same 1-hour rates as us. (1.6x the 5-minute regional rate). Without these fields, cost tracking on EU/AU/JP Bedrock 1-hour-TTL prompt caching falls back to the 5-minute write rate and undercounts spend by ~60% for European, Australian, and Japanese tenants. Adds the 1-hour tier (and Sonnet 4.5's long-context >200K tier where AWS publishes one) to 14 regional Bedrock entries in both `model_prices_and_context_window.json` and the bundled `model_prices_and_context_window_backup.json`: - eu./au. Opus 4.6 ($11.00 / MTok) - eu./au. Opus 4.7 ($11.00 / MTok) - eu./au./jp. Sonnet 4.6 ($6.60 / MTok) - eu./au./jp. Sonnet 4.5 ($6.60 / MTok regular, $13.20 / MTok LC) - eu./au./jp. Haiku 4.5 ($2.20 / MTok) Also extends `tests/test_litellm/test_bedrock_anthropic_1hr_cache_pricing.py` with a `REGIONAL_EXPECTED` parametrized block covering all 13 new entries plus the existing 1.6x ratio invariant. Note: `eu.anthropic.claude-opus-4-5-20251101-v1:0` carries the wrong 5m rate today (base 6.25e-06 instead of regional 6.875e-06), which would break the 1.6x ratio check. It is intentionally left out of this PR so the scope stays "1-hour cache tier addition" — a separate follow-up should correct the EU 5m rates for Opus 4.5. --------- Co-authored-by: Terrajlz <info@jouleselectrictech.com> Co-authored-by: Bruno Devaux <devaux.br@gmail.com> Co-authored-by: Sameer Kankute <sameer@berri.ai>
…28569) * fix(thinking): handle None thinking param in is_thinking_enabled (#28598) Squash-merged by litellm-agent from Terrajlz's PR. * feat(helm): support tpl rendering in podAnnotations (#28609) Squash-merged by litellm-agent from devauxbr's PR. * Forward custom_llm_provider through the Responses API bridge (Fixes #28505) (#28575) * Forward custom_llm_provider through the Responses API bridge (Fixes #28505) When a Chat Completions request to a GPT-5.4+ model contains both `tools` and `reasoning_effort`, `completion()` auto-routes through `responses_api_bridge`. The bridge handler called `litellm.responses()` / `litellm.aresponses()` without forwarding the already-resolved `custom_llm_provider`, so the downstream call re-invoked `get_llm_provider()` with `custom_llm_provider=None` and stripped a second provider prefix from a `provider/provider/model` deployment string. For a deployment configured as `openai/openai/openai/gpt-5.5`, the bridge flow sent `openai/gpt-5.5` to the upstream API instead of the correct `openai/openai/gpt-5.5`. Upstream APIs that enforce model-name allow-lists rejected this as `key_model_access_denied`. Fix: pass the locally-resolved `custom_llm_provider` into both the sync `responses()` and async `aresponses()` calls so the downstream `_resolve_model_provider_for_responses` sees an explicit provider and skips the second prefix-strip. New regression test `tests/test_litellm/completion_extras/test_responses_bridge_provider_propagation.py` pins both call sites: each must forward `custom_llm_provider`. * fix(28505): set custom_llm_provider on request_data instead of as duplicate kwarg Greptile flagged that the previous patch passed custom_llm_provider as an explicit kwarg to responses()/aresponses() while request_data already carried it via the spread of sanitized_litellm_params, which would raise TypeError: got multiple values for keyword argument on every real bridge call. Switches to assigning request_data['custom_llm_provider'] before the call so the resolved provider wins over whatever sanitized_litellm_params spread in, without duplicating the kwarg. Updates the regression test to seed request_data with a sentinel custom_llm_provider so it actually exercises the overwrite path (the previous test mocked transform_request with a minimal dict and never hit the conflict). * chore: trigger shin-agent re-eval on retargeted staging base * chore: trigger shin-agent re-eval against updated Greptile state * Add 1-hour cache write pricing tier for Vertex AI Anthropic models GCP Vertex AI publishes a separate 1-hour cache write column for the Claude family (1.6x the 5-minute write rate, matching the documented Bedrock ratio). LiteLLM's Vertex AI Anthropic entries only carry the 5-minute tier, so any request that uses `cache_control: {"ttl": "1h"}` on Vertex AI Claude is undercounted in cost tracking by ~60%. The runtime side already supports the 1-hour tier — `VertexAIAnthropicConfig` extends `AnthropicConfig`, populating `ephemeral_1h_input_tokens`, and `_calculate_cache_creation_cost` reads `cache_creation_input_token_cost_above_1hr`. Only the price registry was missing data. Adds the field to 19 vertex_ai/claude-* entries across both `model_prices_and_context_window.json` and the bundled `model_prices_and_context_window_backup.json`: - Haiku 4.5 ($1.25 -> $2.00 / MTok) - Sonnet 3.7 / 4 / 4.5 / 4.6 ($3.75 -> $6.00 / MTok) - Opus 4.5 / 4.6 / 4.7 ($6.25 -> $10.00 / MTok) - Opus 4 / 4.1 ($18.75 -> $30.00 / MTok) Adds `tests/test_litellm/test_vertex_anthropic_1hr_cache_pricing.py` mirroring the Bedrock equivalent — pins each (5m, 1h) pair per model and asserts the 1.6x ratio across the family. Fixes #27781. --------- Co-authored-by: Terrajlz <info@jouleselectrictech.com> Co-authored-by: Bruno Devaux <devaux.br@gmail.com> Co-authored-by: Sameer Kankute <sameer@berri.ai>
* Fix incorrect agent API request example payload structure (#29556) * fix(otel): add litellm_metadata fallback in _get_span_context and _end_proxy_span_from_kwargs (#29427) * fix(otel): add litellm_metadata fallback in _get_span_context and _end_proxy_span_from_kwargs On /v1/messages and other LITELLM_METADATA_ROUTES, the parent OTel span is stored in litellm_params['litellm_metadata'] instead of litellm_params['metadata']. When the request body contains a native 'metadata' field (e.g. Anthropic's {"user_id": "..."}), litellm_params['metadata'] gets overwritten and the parent span is lost, producing orphan root spans with a different trace_id. Add fallback checks to litellm_metadata in: - _get_span_context(): so child spans find the correct parent - _end_proxy_span_from_kwargs(): so the proxy span gets closed Fixes: #27934 * test(otel): tighten assertions per Greptile review - test_span_context_metadata_takes_priority: assert litellm_metadata span is never accessed, proving metadata takes priority - test_span_context_no_parent_when_neither_has_span: assert both ctx and detected_span are None --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Aneesh-Fiddler <aneeshfiddler@gmail.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix: remove premature end-user budget check from get_end_user_object (#29420) * fix(proxy): remove premature end-user budget check from get_end_user_object Problem: - `_check_end_user_budget()` was called inside `get_end_user_object()` - This caused budget checks to run BEFORE `skip_budget_checks` could be evaluated - Zero-cost models (e.g., local vLLM) were incorrectly blocked when end-users exceeded their budget, even though they should bypass budget checks Solution: - Remove `_check_end_user_budget()` calls from `get_end_user_object()` - Budget enforcement now happens exclusively in `common_checks()` where `skip_budget_checks` context is available - `get_end_user_object()` keeps `route` as optional in function parameter for backwards compatibility and future implementation. * refactor(tests): update budget enforcement tests to reflect changes in get_end_user_object - test_get_end_user_object() verifies data fetching - test_check_end_user_budget() verifies enforcement - test_budget_enforcement_blocks_over_budget_users() integrates _check_end_user_budget() - test_resolve_end_user_reraises_budget_exceeded() is now test_resolve_end_user since no budget exceeded is thrown in get_end_user_object() * Gemini /images/generate and /images/edits billing fixes + add support for size and aspect ratio params (#29534) * Fix Gemini image config mapping * Address Gemini image config review * Format Gemini image generation transform * Fix Gemini image token usage logging * Share Gemini image request helpers * Fix Gemini Imagen model routing * Fixes as per self code review * Fixes per internal code review * Stop gating Imagen imageSize forwarding * Document Gemini image size mapping source * chore: retrigger lint * Clarify Gemini candidate count precedence * Add Inception provider (#29522) * add inception as provider (chat, fim) * linting * seperate test suite for chat and fim * fix test coverage * fix: model hub custom pricing model info (#29293) * Opik user auth key metadata extractors (#28397) * fix: enhance Opik metadata extraction to include user API key auth context fixed after refactoring to extractor logic * test: add unit tests for OPik metadata extraction logic * fix: enhance extract_opik_metadata function to prioritize metadata sources for improved accuracy * fix(ci): clarified comments and edited unit tests * test: add unit tests for OPik metadata extraction with auth and requester overrides * fix(ui): replace fixed favicon.ico with current api get /get_favicon (#29532) Signed-off-by: José Luis Di Biase <josx@interorganic.com.ar> * fix(vertex/gemini): keep tool_call reference when a text-only assistant message follows (#29561) `_gemini_convert_messages_with_history` tracks `last_message_with_tool_calls` so a following tool result can be matched back to its tool call. The assignment was inside a branch guarded by `assistant_msg.get("tool_calls", []) is not None`, which is also True for a text-only assistant message (an empty list is not None). As a result, an assistant message with no tool calls that appears between a tool call and its tool result overwrote the reference, and conversion failed with: Exception: Missing corresponding tool call for tool response message. This shape is common: a model emits a short narration/assistant message after a tool call before the tool result is appended. Only update `last_message_with_tool_calls` when the assistant message actually carries tool_calls (or a function_call). Adds a regression test. Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * Add 1-hour cache write pricing for EU/AU/JP Bedrock Anthropic models (#28572) * fix(thinking): handle None thinking param in is_thinking_enabled (#28598) Squash-merged by litellm-agent from Terrajlz's PR. * feat(helm): support tpl rendering in podAnnotations (#28609) Squash-merged by litellm-agent from devauxbr's PR. * Forward custom_llm_provider through the Responses API bridge (Fixes #28505) (#28575) * Forward custom_llm_provider through the Responses API bridge (Fixes #28505) When a Chat Completions request to a GPT-5.4+ model contains both `tools` and `reasoning_effort`, `completion()` auto-routes through `responses_api_bridge`. The bridge handler called `litellm.responses()` / `litellm.aresponses()` without forwarding the already-resolved `custom_llm_provider`, so the downstream call re-invoked `get_llm_provider()` with `custom_llm_provider=None` and stripped a second provider prefix from a `provider/provider/model` deployment string. For a deployment configured as `openai/openai/openai/gpt-5.5`, the bridge flow sent `openai/gpt-5.5` to the upstream API instead of the correct `openai/openai/gpt-5.5`. Upstream APIs that enforce model-name allow-lists rejected this as `key_model_access_denied`. Fix: pass the locally-resolved `custom_llm_provider` into both the sync `responses()` and async `aresponses()` calls so the downstream `_resolve_model_provider_for_responses` sees an explicit provider and skips the second prefix-strip. New regression test `tests/test_litellm/completion_extras/test_responses_bridge_provider_propagation.py` pins both call sites: each must forward `custom_llm_provider`. * fix(28505): set custom_llm_provider on request_data instead of as duplicate kwarg Greptile flagged that the previous patch passed custom_llm_provider as an explicit kwarg to responses()/aresponses() while request_data already carried it via the spread of sanitized_litellm_params, which would raise TypeError: got multiple values for keyword argument on every real bridge call. Switches to assigning request_data['custom_llm_provider'] before the call so the resolved provider wins over whatever sanitized_litellm_params spread in, without duplicating the kwarg. Updates the regression test to seed request_data with a sentinel custom_llm_provider so it actually exercises the overwrite path (the previous test mocked transform_request with a minimal dict and never hit the conflict). * chore: trigger shin-agent re-eval on retargeted staging base * chore: trigger shin-agent re-eval against updated Greptile state * Add 1-hour cache write pricing for EU/AU/JP Bedrock Anthropic models The 1-hour prompt-cache write tier (`cache_creation_input_token_cost_above_1hr`) was added to the us./global. variants of the Claude 4.5/4.6/4.7 family on Bedrock, but the eu./au./jp. cross-region inference profiles were left without it. AWS Bedrock pricing applies the same +10% regional premium across all geo profiles, so eu./au./jp. should carry the same 1-hour rates as us. (1.6x the 5-minute regional rate). Without these fields, cost tracking on EU/AU/JP Bedrock 1-hour-TTL prompt caching falls back to the 5-minute write rate and undercounts spend by ~60% for European, Australian, and Japanese tenants. Adds the 1-hour tier (and Sonnet 4.5's long-context >200K tier where AWS publishes one) to 14 regional Bedrock entries in both `model_prices_and_context_window.json` and the bundled `model_prices_and_context_window_backup.json`: - eu./au. Opus 4.6 ($11.00 / MTok) - eu./au. Opus 4.7 ($11.00 / MTok) - eu./au./jp. Sonnet 4.6 ($6.60 / MTok) - eu./au./jp. Sonnet 4.5 ($6.60 / MTok regular, $13.20 / MTok LC) - eu./au./jp. Haiku 4.5 ($2.20 / MTok) Also extends `tests/test_litellm/test_bedrock_anthropic_1hr_cache_pricing.py` with a `REGIONAL_EXPECTED` parametrized block covering all 13 new entries plus the existing 1.6x ratio invariant. Note: `eu.anthropic.claude-opus-4-5-20251101-v1:0` carries the wrong 5m rate today (base 6.25e-06 instead of regional 6.875e-06), which would break the 1.6x ratio check. It is intentionally left out of this PR so the scope stays "1-hour cache tier addition" — a separate follow-up should correct the EU 5m rates for Opus 4.5. --------- Co-authored-by: Terrajlz <info@jouleselectrictech.com> Co-authored-by: Bruno Devaux <devaux.br@gmail.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> * Add 1-hour cache write pricing tier for Vertex AI Anthropic models (#28569) * fix(thinking): handle None thinking param in is_thinking_enabled (#28598) Squash-merged by litellm-agent from Terrajlz's PR. * feat(helm): support tpl rendering in podAnnotations (#28609) Squash-merged by litellm-agent from devauxbr's PR. * Forward custom_llm_provider through the Responses API bridge (Fixes #28505) (#28575) * Forward custom_llm_provider through the Responses API bridge (Fixes #28505) When a Chat Completions request to a GPT-5.4+ model contains both `tools` and `reasoning_effort`, `completion()` auto-routes through `responses_api_bridge`. The bridge handler called `litellm.responses()` / `litellm.aresponses()` without forwarding the already-resolved `custom_llm_provider`, so the downstream call re-invoked `get_llm_provider()` with `custom_llm_provider=None` and stripped a second provider prefix from a `provider/provider/model` deployment string. For a deployment configured as `openai/openai/openai/gpt-5.5`, the bridge flow sent `openai/gpt-5.5` to the upstream API instead of the correct `openai/openai/gpt-5.5`. Upstream APIs that enforce model-name allow-lists rejected this as `key_model_access_denied`. Fix: pass the locally-resolved `custom_llm_provider` into both the sync `responses()` and async `aresponses()` calls so the downstream `_resolve_model_provider_for_responses` sees an explicit provider and skips the second prefix-strip. New regression test `tests/test_litellm/completion_extras/test_responses_bridge_provider_propagation.py` pins both call sites: each must forward `custom_llm_provider`. * fix(28505): set custom_llm_provider on request_data instead of as duplicate kwarg Greptile flagged that the previous patch passed custom_llm_provider as an explicit kwarg to responses()/aresponses() while request_data already carried it via the spread of sanitized_litellm_params, which would raise TypeError: got multiple values for keyword argument on every real bridge call. Switches to assigning request_data['custom_llm_provider'] before the call so the resolved provider wins over whatever sanitized_litellm_params spread in, without duplicating the kwarg. Updates the regression test to seed request_data with a sentinel custom_llm_provider so it actually exercises the overwrite path (the previous test mocked transform_request with a minimal dict and never hit the conflict). * chore: trigger shin-agent re-eval on retargeted staging base * chore: trigger shin-agent re-eval against updated Greptile state * Add 1-hour cache write pricing tier for Vertex AI Anthropic models GCP Vertex AI publishes a separate 1-hour cache write column for the Claude family (1.6x the 5-minute write rate, matching the documented Bedrock ratio). LiteLLM's Vertex AI Anthropic entries only carry the 5-minute tier, so any request that uses `cache_control: {"ttl": "1h"}` on Vertex AI Claude is undercounted in cost tracking by ~60%. The runtime side already supports the 1-hour tier — `VertexAIAnthropicConfig` extends `AnthropicConfig`, populating `ephemeral_1h_input_tokens`, and `_calculate_cache_creation_cost` reads `cache_creation_input_token_cost_above_1hr`. Only the price registry was missing data. Adds the field to 19 vertex_ai/claude-* entries across both `model_prices_and_context_window.json` and the bundled `model_prices_and_context_window_backup.json`: - Haiku 4.5 ($1.25 -> $2.00 / MTok) - Sonnet 3.7 / 4 / 4.5 / 4.6 ($3.75 -> $6.00 / MTok) - Opus 4.5 / 4.6 / 4.7 ($6.25 -> $10.00 / MTok) - Opus 4 / 4.1 ($18.75 -> $30.00 / MTok) Adds `tests/test_litellm/test_vertex_anthropic_1hr_cache_pricing.py` mirroring the Bedrock equivalent — pins each (5m, 1h) pair per model and asserts the 1.6x ratio across the family. Fixes #27781. --------- Co-authored-by: Terrajlz <info@jouleselectrictech.com> Co-authored-by: Bruno Devaux <devaux.br@gmail.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> * Fix Gemini multimodal function responses (#29325) Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * address greptile review: add _transform_image_usage method and model-map supports_image_size flag - Add _transform_image_usage instance method to GoogleImageGenConfig that delegates to transform_gemini_image_usage, fixing the regression test - Replace hardcoded "2.5-flash" string check in supports_gemini_image_size with a get_model_info lookup on supports_image_size (default true) - Add supports_image_size: false to all gemini-2.5-flash model entries in model_prices_and_context_window.json so capability is controlled via the model map rather than embedded in code * fix test failures: schema validation, mypy type, model info plumbing, pricing test - Add supports_image_size to ModelInfoBase TypedDict so get_model_info surfaces it - Pass supports_image_size through _get_model_info_helper constructor call - Fix supports_gemini_image_size to use value is not False (None means unset, defaults to True) - Add supports_image_size to JSON schema in test_aaamodel_prices_and_context_window_json_is_valid - Correct gemini-3.1-flash-lite pricing assertions in test to match JSON values * Add Azure AI Kimi K2.6 metadata (#27052) * Add Azure AI Kimi K2.6 metadata * Scope Kimi metadata test cost map setup * fall back to substring check for models not in model_prices_and_context_window.json Models like gemini-2.5-flash-image-preview are not in the pricing JSON, so get_model_info raises. Fall back to "2.5-flash" not in model when the JSON has no explicit supports_image_size entry for the model. * fix(inception): don't forward global litellm.api_key to Inception FIM Match the Inception chat config: resolve only an Inception-specific key (param, litellm.inception_key, or INCEPTION_API_KEY) for the text-completion FIM path. The global litellm.api_key (often an OpenAI key) was both leaking to api.inceptionlabs.ai and taking precedence over the configured Inception key when set. * fix(auth): enforce end-user budget on custom-auth path that skips common_checks get_end_user_object() no longer raises BudgetExceededError, so custom-auth deployments with custom_auth_run_common_checks unset (which skip the centralized common_checks gate) stopped enforcing the end-user budget, letting an over-budget end user keep making requests. Re-enforce the budget in _run_post_custom_auth_checks on that path. --------- Signed-off-by: José Luis Di Biase <josx@interorganic.com.ar> Co-authored-by: Isha <72744901+IshaMeera@users.noreply.github.com> Co-authored-by: aneeshsangvikar <aneeshsangvikar@fiddler.ai> Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Aneesh-Fiddler <aneeshfiddler@gmail.com> Co-authored-by: Suleiman Elkhoury <108065141+suleimanelkhoury@users.noreply.github.com> Co-authored-by: Dmitriy Alergant <93501479+DmitriyAlergant@users.noreply.github.com> Co-authored-by: Yanis Miraoui <yanis.miraoui19@imperial.ac.uk> Co-authored-by: Lovro Seder <vrovro@gmail.com> Co-authored-by: Thomas Mildner <12685945+Thomas-Mildner@users.noreply.github.com> Co-authored-by: José Luis Di Biase <josx@interorganic.com.ar> Co-authored-by: Lai Quang Huy <64073540+1qh@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Terrajlz <info@jouleselectrictech.com> Co-authored-by: Bruno Devaux <devaux.br@gmail.com> Co-authored-by: ZHONG Ziwen <67355585+zzw-math@users.noreply.github.com> Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
…28567) * fix(thinking): handle None thinking param in is_thinking_enabled (#28598) Squash-merged by litellm-agent from Terrajlz's PR. * feat(helm): support tpl rendering in podAnnotations (#28609) Squash-merged by litellm-agent from devauxbr's PR. * Forward custom_llm_provider through the Responses API bridge (Fixes #28505) (#28575) * Forward custom_llm_provider through the Responses API bridge (Fixes #28505) When a Chat Completions request to a GPT-5.4+ model contains both `tools` and `reasoning_effort`, `completion()` auto-routes through `responses_api_bridge`. The bridge handler called `litellm.responses()` / `litellm.aresponses()` without forwarding the already-resolved `custom_llm_provider`, so the downstream call re-invoked `get_llm_provider()` with `custom_llm_provider=None` and stripped a second provider prefix from a `provider/provider/model` deployment string. For a deployment configured as `openai/openai/openai/gpt-5.5`, the bridge flow sent `openai/gpt-5.5` to the upstream API instead of the correct `openai/openai/gpt-5.5`. Upstream APIs that enforce model-name allow-lists rejected this as `key_model_access_denied`. Fix: pass the locally-resolved `custom_llm_provider` into both the sync `responses()` and async `aresponses()` calls so the downstream `_resolve_model_provider_for_responses` sees an explicit provider and skips the second prefix-strip. New regression test `tests/test_litellm/completion_extras/test_responses_bridge_provider_propagation.py` pins both call sites: each must forward `custom_llm_provider`. * fix(28505): set custom_llm_provider on request_data instead of as duplicate kwarg Greptile flagged that the previous patch passed custom_llm_provider as an explicit kwarg to responses()/aresponses() while request_data already carried it via the spread of sanitized_litellm_params, which would raise TypeError: got multiple values for keyword argument on every real bridge call. Switches to assigning request_data['custom_llm_provider'] before the call so the resolved provider wins over whatever sanitized_litellm_params spread in, without duplicating the kwarg. Updates the regression test to seed request_data with a sentinel custom_llm_provider so it actually exercises the overwrite path (the previous test mocked transform_request with a minimal dict and never hit the conflict). * chore: trigger shin-agent re-eval on retargeted staging base * chore: trigger shin-agent re-eval against updated Greptile state * Add jp. Bedrock cross-region inference profile for claude-opus-4-7 AWS Bedrock documents jp.anthropic.claude-opus-4-7 alongside the existing us./eu./au./global. profiles for Claude Opus 4.7 (ap-northeast-1 Tokyo / ap-northeast-3 Osaka), but the entry is missing from model_prices_and_context_window.json. Tokyo-region users currently get an "unknown model" error when routing through the JP geo profile. Adds the entry to both the canonical file and the bundled backup, mirroring the recent pattern for sonnet-4-6 (#27831). Pricing matches the other regional profiles (10% premium over base/global). Regression test pins all six documented profiles (base, global, us, eu, au, jp) and asserts pricing parity between jp. and au. variants. Source: https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-anthropic-claude-opus-4-7.html --------- Co-authored-by: Terrajlz <info@jouleselectrictech.com> Co-authored-by: Bruno Devaux <devaux.br@gmail.com> Co-authored-by: Sameer Kankute <sameer@berri.ai>
…28567) * fix(thinking): handle None thinking param in is_thinking_enabled (#28598) Squash-merged by litellm-agent from Terrajlz's PR. * feat(helm): support tpl rendering in podAnnotations (#28609) Squash-merged by litellm-agent from devauxbr's PR. * Forward custom_llm_provider through the Responses API bridge (Fixes #28505) (#28575) * Forward custom_llm_provider through the Responses API bridge (Fixes #28505) When a Chat Completions request to a GPT-5.4+ model contains both `tools` and `reasoning_effort`, `completion()` auto-routes through `responses_api_bridge`. The bridge handler called `litellm.responses()` / `litellm.aresponses()` without forwarding the already-resolved `custom_llm_provider`, so the downstream call re-invoked `get_llm_provider()` with `custom_llm_provider=None` and stripped a second provider prefix from a `provider/provider/model` deployment string. For a deployment configured as `openai/openai/openai/gpt-5.5`, the bridge flow sent `openai/gpt-5.5` to the upstream API instead of the correct `openai/openai/gpt-5.5`. Upstream APIs that enforce model-name allow-lists rejected this as `key_model_access_denied`. Fix: pass the locally-resolved `custom_llm_provider` into both the sync `responses()` and async `aresponses()` calls so the downstream `_resolve_model_provider_for_responses` sees an explicit provider and skips the second prefix-strip. New regression test `tests/test_litellm/completion_extras/test_responses_bridge_provider_propagation.py` pins both call sites: each must forward `custom_llm_provider`. * fix(28505): set custom_llm_provider on request_data instead of as duplicate kwarg Greptile flagged that the previous patch passed custom_llm_provider as an explicit kwarg to responses()/aresponses() while request_data already carried it via the spread of sanitized_litellm_params, which would raise TypeError: got multiple values for keyword argument on every real bridge call. Switches to assigning request_data['custom_llm_provider'] before the call so the resolved provider wins over whatever sanitized_litellm_params spread in, without duplicating the kwarg. Updates the regression test to seed request_data with a sentinel custom_llm_provider so it actually exercises the overwrite path (the previous test mocked transform_request with a minimal dict and never hit the conflict). * chore: trigger shin-agent re-eval on retargeted staging base * chore: trigger shin-agent re-eval against updated Greptile state * Add jp. Bedrock cross-region inference profile for claude-opus-4-7 AWS Bedrock documents jp.anthropic.claude-opus-4-7 alongside the existing us./eu./au./global. profiles for Claude Opus 4.7 (ap-northeast-1 Tokyo / ap-northeast-3 Osaka), but the entry is missing from model_prices_and_context_window.json. Tokyo-region users currently get an "unknown model" error when routing through the JP geo profile. Adds the entry to both the canonical file and the bundled backup, mirroring the recent pattern for sonnet-4-6 (#27831). Pricing matches the other regional profiles (10% premium over base/global). Regression test pins all six documented profiles (base, global, us, eu, au, jp) and asserts pricing parity between jp. and au. variants. Source: https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-anthropic-claude-opus-4-7.html --------- Co-authored-by: Terrajlz <info@jouleselectrictech.com> Co-authored-by: Bruno Devaux <devaux.br@gmail.com> Co-authored-by: Sameer Kankute <sameer@berri.ai>
* Mark xAI models retiring on 2026-05-15 (#28788) Per https://docs.x.ai/developers/migration/may-15-retirement, xAI is retiring the following slugs on 2026-05-15 (auto-redirect to grok-4.3 with various reasoning efforts; callers continuing to use the old slugs will be billed at grok-4.3 pricing): grok-4-1-fast-reasoning{,-latest} -> grok-4.3 (low effort) grok-4-1-fast-non-reasoning{,-latest} -> grok-4.3 (none) grok-4-fast-reasoning -> grok-4.3 (low effort) grok-4-fast-non-reasoning -> grok-4.3 (none) grok-4-0709 -> grok-4.3 (low effort) grok-code-fast-1{,-0825} -> grok-build-0.1 grok-3 -> grok-4.3 (none) Only the direct xai/ slugs are tagged; third-party hosts (azure_ai, oci, vercel_ai_gateway, perplexity/xai) run their own schedules. The grok-3 retirement list explicitly names only the base grok-3 slug — the -mini / -fast / -beta / -latest variants are not listed, so they remain untouched. * feat(moonshot): advertise json_schema response support on live models (#29683) litellm.responses() already routes Moonshot through the responses->chat-completions bridge, and Moonshot honors response_format json_schema on chat completions. The cost-map entries left supports_response_schema unset, so discovery layers that gate on that flag dropped Moonshot from structured-output / responses listings even though the capability works end to end. Set supports_response_schema on the nine models currently live on api.moonshot.ai: kimi-k2.5, kimi-k2.6, the moonshot-v1 8k/32k/128k text and vision-preview variants, and moonshot-v1-auto. Verified against the live API that each honors json_schema and that litellm.responses() returns schema-valid structured output through the bridge. * chore(moonshot): mark models retired from api.moonshot.ai as deprecated (#29685) Thirteen Moonshot/Kimi models in the cost map no longer resolve on api.moonshot.ai (all return 404). Stamp each with its deprecation_date from platform.kimi.ai/docs/models rather than deleting the entries, so historical cost calculation keeps resolving the names while tooling can surface the retirement. Dates: kimi-thinking-preview 2025-11-11; kimi-latest and its 8k/32k/128k context variants 2026-01-28; the kimi-k2 preview/turbo/thinking series 2026-05-25; the moonshot-v1 -0430 snapshots use their own 2024-04-30 snapshot date (Moonshot publishes no discontinuation date for them). * fix(moonshot): drop temperature for reasoning models (kimi-k2.5/k2.6) (#29687) Kimi reasoning models reject every temperature except 1; a request with temperature=0.2 returns "invalid temperature: only 1 is allowed for this model". litellm only clamped temperature into [0.3, 1], so any value below 1 still 400'd. Drop the temperature param entirely for reasoning models (gated on supports_reasoning, the same signal transform_request already uses) so the model default is used; the non-reasoning moonshot-v1 models keep the existing clamp. Co-authored-by: Sameer Kankute <sameer@berri.ai> * feat(mcp): add per-server timeout configuration (#29672) * feat(mcp): add per-server timeout configuration * fix(mcp): address timeout field review comments - use is not None guard instead of or for 0.0 edge case - copy timeout in both LiteLLM_MCPServerTable constructions (health check path + _build_mcp_server_table) - add timeout Float? column to all three schema.prisma files - extend round-trip test to cover _build_mcp_server_table direction - add test for zero timeout not treated as falsy * fix(mcp): forward timeout in _build_temporary_mcp_server_record * fix(mcp): return 504 instead of 500 when per-server timeout fires * test(mcp): add 504 timeout regression test; fix black formatting * Add jp. Bedrock cross-region inference profile for claude-opus-4-7 (#28567) * fix(thinking): handle None thinking param in is_thinking_enabled (#28598) Squash-merged by litellm-agent from Terrajlz's PR. * feat(helm): support tpl rendering in podAnnotations (#28609) Squash-merged by litellm-agent from devauxbr's PR. * Forward custom_llm_provider through the Responses API bridge (Fixes #28505) (#28575) * Forward custom_llm_provider through the Responses API bridge (Fixes #28505) When a Chat Completions request to a GPT-5.4+ model contains both `tools` and `reasoning_effort`, `completion()` auto-routes through `responses_api_bridge`. The bridge handler called `litellm.responses()` / `litellm.aresponses()` without forwarding the already-resolved `custom_llm_provider`, so the downstream call re-invoked `get_llm_provider()` with `custom_llm_provider=None` and stripped a second provider prefix from a `provider/provider/model` deployment string. For a deployment configured as `openai/openai/openai/gpt-5.5`, the bridge flow sent `openai/gpt-5.5` to the upstream API instead of the correct `openai/openai/gpt-5.5`. Upstream APIs that enforce model-name allow-lists rejected this as `key_model_access_denied`. Fix: pass the locally-resolved `custom_llm_provider` into both the sync `responses()` and async `aresponses()` calls so the downstream `_resolve_model_provider_for_responses` sees an explicit provider and skips the second prefix-strip. New regression test `tests/test_litellm/completion_extras/test_responses_bridge_provider_propagation.py` pins both call sites: each must forward `custom_llm_provider`. * fix(28505): set custom_llm_provider on request_data instead of as duplicate kwarg Greptile flagged that the previous patch passed custom_llm_provider as an explicit kwarg to responses()/aresponses() while request_data already carried it via the spread of sanitized_litellm_params, which would raise TypeError: got multiple values for keyword argument on every real bridge call. Switches to assigning request_data['custom_llm_provider'] before the call so the resolved provider wins over whatever sanitized_litellm_params spread in, without duplicating the kwarg. Updates the regression test to seed request_data with a sentinel custom_llm_provider so it actually exercises the overwrite path (the previous test mocked transform_request with a minimal dict and never hit the conflict). * chore: trigger shin-agent re-eval on retargeted staging base * chore: trigger shin-agent re-eval against updated Greptile state * Add jp. Bedrock cross-region inference profile for claude-opus-4-7 AWS Bedrock documents jp.anthropic.claude-opus-4-7 alongside the existing us./eu./au./global. profiles for Claude Opus 4.7 (ap-northeast-1 Tokyo / ap-northeast-3 Osaka), but the entry is missing from model_prices_and_context_window.json. Tokyo-region users currently get an "unknown model" error when routing through the JP geo profile. Adds the entry to both the canonical file and the bundled backup, mirroring the recent pattern for sonnet-4-6 (#27831). Pricing matches the other regional profiles (10% premium over base/global). Regression test pins all six documented profiles (base, global, us, eu, au, jp) and asserts pricing parity between jp. and au. variants. Source: https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-anthropic-claude-opus-4-7.html --------- Co-authored-by: Terrajlz <info@jouleselectrictech.com> Co-authored-by: Bruno Devaux <devaux.br@gmail.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> * feat(soniox): add soniox audio transcription integration (#29508) * feat(openmeter): add OPENMETER_TRUST_REQUEST_USER to prevent forged attribution (#29650) The OpenMeter callback resolves the CloudEvent subject from kwargs["user"] first, then falls back to the key-bound user_api_key_user_id. For multi-tenant proxy deployments, a client can set `"user": "..."` in the request body and cause their usage to be attributed to that arbitrary string — a billing-attribution forgery risk. Adds OPENMETER_TRUST_REQUEST_USER env var (default "true" for backward compatibility). When set to "false", the request-supplied `user` field is ignored and the subject is resolved solely from user_api_key_user_id. Matches the existing env-var-driven config pattern in this file (OPENMETER_API_KEY, OPENMETER_API_ENDPOINT, OPENMETER_EVENT_TYPE). * feat(search): add you_com as a search provider (#28370) * feat(search): add you_com as a search provider Registers You.com Search API as a first-class `search_provider` in the `search_tools` registry, alongside Tavily, Exa, Perplexity, etc. - New adapter: litellm/llms/you_com/search/transformation.py - POSTs to https://ydc-index.io/v1/search - Auth: X-API-Key from YOUCOM_API_KEY (or explicit api_key) - Maps Perplexity unified spec: max_results -> count, search_domain_filter -> include_domains, country -> country - Flattens results.web + results.news into a single SearchResult list; snippet prefers snippets[0], falls back to description; page_age -> date - Registry: SearchProviders.YOU_COM in litellm/types/utils.py and wired into ProviderConfigManager.get_provider_search_config() - Pricing entry: model_prices_and_context_window.json (placeholder $0.0; happy to adjust to maintainers' preferred public number) - Docs: example router config snippet and example proxy yaml updated - Tests: tests/search_tests/test_you_com_search.py - 5 mocked tests (payload shape, domain filter mapping, snippet fallback, news flattening, missing-api-key error) Refs upstream expansion signal: #15942 * review fixups: normalize api_base, lowercase country, scope env-var to test Addresses Greptile inline review comments on #28370: - get_complete_url: strip trailing slashes from api_base *before* the endswith("/v1/search") check, so a custom base like ".../v1/search/" doesn't become ".../v1/search/v1/search". - transform_search_request: .lower() country before sending, matching Tavily's convention so callers using the unified spec form ("US") get consistent behavior across providers. - Tests: replace direct os.environ writes with an autouse monkeypatch fixture so YOUCOM_API_KEY is set per-test and removed afterwards. The missing-key test now uses monkeypatch.delenv. New test asserts the trailing-slash normalization above. Reverts the ARCHITECTURE.md / example yaml edits per the reviewer note that documentation changes belong in the litellm-docs repo. * support keyless free tier (api.you.com/v1/agents/search) as default You.com offers an IP-throttled keyless endpoint that returns the same response shape as the keyed one (~100 queries/day, no signup). This is a significant onboarding lever - mirrors the keyless DuckDuckGo/SearXNG providers already in the search_tools registry. Behavior: - YOUCOM_API_KEY set -> keyed: POST https://ydc-index.io/v1/search (X-API-Key header) - no key -> free: POST https://api.you.com/v1/agents/search (no auth) - YOUCOM_API_BASE override -> honored as-is Tests: - New: test_you_com_search_keyless_free_tier - asserts URL + absence of X-API-Key when no key is configured. - New: test_you_com_search_validate_environment_keyless - asserts the config no longer raises when the key is absent. - Removed: test_you_com_search_raises_without_api_key (the precondition no longer holds). - Existing payload/domain-filter/etc tests still cover keyed mode via the autouse YOUCOM_API_KEY fixture. Verified both endpoints accept POST + return identical JSON shape: results.web[] / results.news[] with title, url, snippets, description, page_age. * register you_com in provider_endpoints_support.json Adding `litellm/llms/you_com/` requires a corresponding entry in provider_endpoints_support.json or the code-quality/check_provider_folders_documented CI check fails. Follows the compact tavily/serper pattern - endpoints: { search: true }. Local run of the check now reports "All 114 provider folders are documented". * move tests under tests/test_litellm/llms/ so CI exercises them The litellm CI workflows scope unit tests to `tests/test_litellm/...` (see test-unit-llm-providers.yml: `tests/test_litellm/llms` path), so tests living under `tests/search_tests/` are never run in CI - which is why codecov reports 0% patch coverage for the new adapter even though the unit tests exist and pass locally. Move test_you_com_search.py into `tests/test_litellm/llms/you_com/` so the test-unit-llm-providers job picks it up. 7/7 tests still pass at the new location. (Sibling search-only providers - tavily, exa_ai, brave, etc. - still live only in `tests/search_tests/` and would benefit from the same move, but that is out of scope for this PR.) * fix(you_com): pin Accept-Encoding: identity to dodge keyless gzip bug The keyless free-tier endpoint (api.you.com/v1/agents/search) advertises Content-Encoding: gzip but returns a body that httpx's decoder rejects with `zlib.error: Error -3 while decompressing data: incorrect header check`, surfacing as litellm.APIConnectionError in user code. curl works because it doesn't request compression by default. Pin Accept-Encoding: identity in validate_environment so the upstream server skips compression entirely. Harmless on the keyed endpoint (ydc-index.io/v1/search) which negotiates content-encoding correctly. The header uses setdefault so a caller-supplied Accept-Encoding still takes precedence. (Server-side bug has been flagged to the You.com team separately - once fixed there, this workaround can be removed.) New unit test: test_you_com_search_pins_identity_accept_encoding. --------- Co-authored-by: Sameer Kankute <sameer@berri.ai> * docs: fix README typo (#29419) Correct clear spelling mistakes in documentation without changing behavior. Confidence: high Scope-risk: narrow Tested: git diff --check; uvx codespell on changed files Not-tested: Full docs build not run; text-only changes * Fix(langfuse): pass httpx_client to Langfuse in langfuse_prompt_management to respect SSL_VERIFY (#29480) * fix(langfuse): pass ssl_verify to Langfuse httpx client * fix_langfuse_ * add unit tests * addressed comments --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * feat(models): add minimax/MiniMax-M3 to model cost map (#29412) Add MiniMax's new flagship MiniMax-M3 to the native minimax provider: 512K context, 128K max output, native multimodal (supports_vision), reasoning, prompt caching. Pricing (USD/M tokens): input 0.6 / output 2.4 / cache read 0.12. M3 has no active prompt-cache-write tier, so cache_creation_input_token_cost is omitted. Updated both the root model_prices_and_context_window.json (remote source) and the bundled litellm/model_prices_and_context_window_backup.json (local fallback), keeping them in sync. * fix(logging): handle ResponseCompletedEvent in anthropic_messages streaming spend log (#29394) * fix(logging): handle ResponseCompletedEvent in anthropic_messages streaming spend log * fix(logging): extend terminal event handling to ResponseIncompleteEvent and ResponseFailedEvent; fix return type annotation * feat(provider): Add Neosantara provider as OpenAI Compatible (#29646) * Add Neosantara provider * Register Neosantara provider enum * Address Neosantara provider review feedback * Add Neosantara packaged endpoint support --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * fix: address greptile and veria review feedback - langfuse: guard httpx_client injection behind version check (>= 2.7.3) - soniox: propagate audio_transcription_duration in _hidden_params for spend tracking - soniox: give SONIOX_API_BASE env var priority over caller-supplied api_base - mcp: replace CancelledError catch with asyncio.wait_for + TimeoutError * chore(mcp): add migration for per-server timeout column * fix(test): add tool_use_system_prompt_tokens to model prices schema validator * fix: mcp timeout test uses real asyncio.wait_for timeout; you_com get_complete_url respects resolved api_key * fix: forward resolved api_key into you_com endpoint selection and apply timeout to soniox polling GETs The search flow resolves api_key in validate_environment but never passed it into get_complete_url, so a programmatic api_key (with no YOUCOM_API_KEY in the env) set the X-API-Key header yet still selected the keyless free-tier endpoint. Forward api_key through both the search entrypoint and the http handler so the keyed endpoint is chosen. HTTPHandler.get/AsyncHTTPHandler.get had no timeout parameter, so the Soniox poll and transcript-fetch GETs silently used the client global default instead of the caller timeout. Add a per-request timeout to get() and forward the configured timeout from the Soniox handler. * fix(soniox): price stt-async-v4 per second so transcriptions are billed The handler stores audio_transcription_duration in _hidden_params, but the model carried only token cost fields and the response has no token usage, so the transcription cost path fell through to cost_per_second and returned $0. An authenticated caller could transcribe Soniox audio without decrementing their budget. Switch the entry to output_cost_per_second at Soniox's published $0.10/hour async rate so the stored duration produces a real charge. * fix(langfuse): use a dedicated httpx client for the SDK injection The httpx_client handed to the Langfuse SDK came from _get_httpx_client(), which returns LiteLLM's globally cached HTTPHandler. If Langfuse closed that client on teardown it would invalidate the shared client used by every other LiteLLM HTTP call. Build a dedicated httpx.Client instead, still resolving SSL verification and client certificate from LiteLLM's configuration. * fix(soniox): prefer caller-supplied api_base over SONIOX_API_BASE env var * fix(cohere): support max_completion_tokens on cohere v2 chat (default route) (#29779) * fix(cohere): support max_completion_tokens on cohere v2 chat The default cohere_chat route resolves to CohereV2ChatConfig, which did not list or map max_completion_tokens, so get_optional_params raised UnsupportedParamsError for the standard OpenAI parameter (the modern replacement for the deprecated max_tokens). The v1 config already maps it to cohere's max_tokens; mirror that in v2 and add v2 regression tests. * fix(cohere): make max_completion_tokens take precedence over max_tokens on v2 When both max_tokens and max_completion_tokens are supplied, prefer max_completion_tokens explicitly rather than relying on dict iteration order, and cover both orderings with a regression test. --------- Co-authored-by: Daniel Yudelevich <4537920+yudelevi@users.noreply.github.com> Co-authored-by: hectorc98 <hector.chamorroalvarez@adyen.com> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Terrajlz <info@jouleselectrictech.com> Co-authored-by: Bruno Devaux <devaux.br@gmail.com> Co-authored-by: Dan Lemon <dan@danlemon.com> Co-authored-by: Saswat <saswatds@users.noreply.github.com> Co-authored-by: Brian Sparker <brainsparker@users.noreply.github.com> Co-authored-by: Zhao73 <156770117+Zhao73@users.noreply.github.com> Co-authored-by: Urain Ahmad Shah <60431964+urainshah@users.noreply.github.com> Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: kape <168134658+kapelame@users.noreply.github.com> Co-authored-by: danisalvaa <159898202+danisalvaa@users.noreply.github.com> Co-authored-by: Just R <remixingmagelang@gmail.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: abhay23-AI <abhaytrivedi22@gmail.com>
* fix(key_generate): allow team members to create keys on org-scoped teams (#29310)
* fix(key_generate): allow team members to create keys on org-scoped teams
When a virtual key is created for a team, enterprise logic inherits the
team's organization_id onto the key (add_team_organization_id). Since the
VERIA-55 org-IDOR fix, /key/generate then required the caller to be an
explicit LiteLLM_OrganizationMembership member of that org, returning
403 "Caller is not a member of organization_id=<uuid>". Admins normally
only add users to teams (not orgs), so self-serve key creation regressed
for any user on an org-scoped team (regression since v1.84.0-rc.1).
Skip the org-membership check when organization_id was inherited from the
key's team (organization_id == team_table.organization_id). Team-level
authorization already gates this path, so team membership is sufficient.
The membership check still runs when a caller assigns an organization_id
that did not come from the key's team, preserving the IDOR protection.
Adds regression tests covering both the team-inherited (allowed) and
foreign-org (still blocked) cases.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(key_generate): cover mismatched team org IDOR path on generate
Add test_generate_key_foreign_org_with_mismatched_team_still_enforces_membership
for the case where a team is present but request organization_id differs from
team_table.organization_id. Enterprise inheritance is no-op'd in the test so
the guard is exercised directly; membership validation must still run.
Addresses Greptile review on #29310.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(pass-through): move Gemini pass-through tests to gemini-3.1-flash-lite (#29595)
* test(pass-through): move Gemini pass-through tests to gemini-3.1-flash-lite
gemini-2.5-flash-lite is a generation behind and is slated for discontinuation on Vertex AI no earlier than October 16, 2026, so the pass-through suite was exercising an aging model. Every reference now points at gemini-3.1-flash-lite, which is GA and already priced in the cost map so the spend-logging assertions still compute a real cost
test_vertex.test.js also gains jest.retryTimes(3) to match the sibling spend tests. The CI failures were intermittent 429 RESOURCE_EXHAUSTED from Vertex quota pressure, and that file was the only one without a retry, so a single rate-limited request was failing the whole job
* test(pass-through): point Vertex tests at the global endpoint for gemini-3.1-flash-lite
gemini-3.1-flash-lite is not served on the Vertex us-central1 regional endpoint for the CI project, so the Vertex pass-through tests were returning a deterministic 404 "Publisher Model ... was not found or your project does not have access to it" while the Gemini API tests passed. Move the Vertex clients to the global location, which the pass-through router maps to aiplatform.googleapis.com, where the 3.1 family is served
* Litellm oss staging 030626 (#29578)
* Fix incorrect agent API request example payload structure (#29556)
* fix(otel): add litellm_metadata fallback in _get_span_context and _end_proxy_span_from_kwargs (#29427)
* fix(otel): add litellm_metadata fallback in _get_span_context and _end_proxy_span_from_kwargs
On /v1/messages and other LITELLM_METADATA_ROUTES, the parent OTel span
is stored in litellm_params['litellm_metadata'] instead of
litellm_params['metadata']. When the request body contains a native
'metadata' field (e.g. Anthropic's {"user_id": "..."}),
litellm_params['metadata'] gets overwritten and the parent span is lost,
producing orphan root spans with a different trace_id.
Add fallback checks to litellm_metadata in:
- _get_span_context(): so child spans find the correct parent
- _end_proxy_span_from_kwargs(): so the proxy span gets closed
Fixes: https://github.com/BerriAI/litellm/issues/27934
* test(otel): tighten assertions per Greptile review
- test_span_context_metadata_takes_priority: assert litellm_metadata
span is never accessed, proving metadata takes priority
- test_span_context_no_parent_when_neither_has_span: assert both ctx
and detected_span are None
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Aneesh-Fiddler <aneeshfiddler@gmail.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* fix: remove premature end-user budget check from get_end_user_object (#29420)
* fix(proxy): remove premature end-user budget check from get_end_user_object
Problem:
- `_check_end_user_budget()` was called inside `get_end_user_object()`
- This caused budget checks to run BEFORE `skip_budget_checks` could be evaluated
- Zero-cost models (e.g., local vLLM) were incorrectly blocked when
end-users exceeded their budget, even though they should bypass budget checks
Solution:
- Remove `_check_end_user_budget()` calls from `get_end_user_object()`
- Budget enforcement now happens exclusively in `common_checks()` where
`skip_budget_checks` context is available
- `get_end_user_object()` keeps `route` as optional in function parameter for backwards compatibility and future implementation.
* refactor(tests): update budget enforcement tests to reflect changes in get_end_user_object
- test_get_end_user_object() verifies data fetching
- test_check_end_user_budget() verifies enforcement
- test_budget_enforcement_blocks_over_budget_users() integrates _check_end_user_budget()
- test_resolve_end_user_reraises_budget_exceeded() is now test_resolve_end_user since no budget exceeded is thrown in get_end_user_object()
* Gemini /images/generate and /images/edits billing fixes + add support for size and aspect ratio params (#29534)
* Fix Gemini image config mapping
* Address Gemini image config review
* Format Gemini image generation transform
* Fix Gemini image token usage logging
* Share Gemini image request helpers
* Fix Gemini Imagen model routing
* Fixes as per self code review
* Fixes per internal code review
* Stop gating Imagen imageSize forwarding
* Document Gemini image size mapping source
* chore: retrigger lint
* Clarify Gemini candidate count precedence
* Add Inception provider (#29522)
* add inception as provider (chat, fim)
* linting
* seperate test suite for chat and fim
* fix test coverage
* fix: model hub custom pricing model info (#29293)
* Opik user auth key metadata extractors (#28397)
* fix: enhance Opik metadata extraction to include user API key auth context fixed after refactoring to extractor logic
* test: add unit tests for OPik metadata extraction logic
* fix: enhance extract_opik_metadata function to prioritize metadata sources for improved accuracy
* fix(ci): clarified comments and edited unit tests
* test: add unit tests for OPik metadata extraction with auth and requester overrides
* fix(ui): replace fixed favicon.ico with current api get /get_favicon (#29532)
Signed-off-by: José Luis Di Biase <josx@interorganic.com.ar>
* fix(vertex/gemini): keep tool_call reference when a text-only assistant message follows (#29561)
`_gemini_convert_messages_with_history` tracks `last_message_with_tool_calls`
so a following tool result can be matched back to its tool call. The assignment
was inside a branch guarded by
`assistant_msg.get("tool_calls", []) is not None`, which is also True for a
text-only assistant message (an empty list is not None). As a result, an
assistant message with no tool calls that appears between a tool call and its
tool result overwrote the reference, and conversion failed with:
Exception: Missing corresponding tool call for tool response message.
This shape is common: a model emits a short narration/assistant message after a
tool call before the tool result is appended.
Only update `last_message_with_tool_calls` when the assistant message actually
carries tool_calls (or a function_call). Adds a regression test.
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Add 1-hour cache write pricing for EU/AU/JP Bedrock Anthropic models (#28572)
* fix(thinking): handle None thinking param in is_thinking_enabled (#28598)
Squash-merged by litellm-agent from Terrajlz's PR.
* feat(helm): support tpl rendering in podAnnotations (#28609)
Squash-merged by litellm-agent from devauxbr's PR.
* Forward custom_llm_provider through the Responses API bridge (Fixes #28505) (#28575)
* Forward custom_llm_provider through the Responses API bridge (Fixes #28505)
When a Chat Completions request to a GPT-5.4+ model contains both
`tools` and `reasoning_effort`, `completion()` auto-routes through
`responses_api_bridge`. The bridge handler called
`litellm.responses()` / `litellm.aresponses()` without forwarding the
already-resolved `custom_llm_provider`, so the downstream call
re-invoked `get_llm_provider()` with `custom_llm_provider=None` and
stripped a second provider prefix from a `provider/provider/model`
deployment string.
For a deployment configured as `openai/openai/openai/gpt-5.5`,
the bridge flow sent `openai/gpt-5.5` to the upstream API instead of
the correct `openai/openai/gpt-5.5`. Upstream APIs that enforce
model-name allow-lists rejected this as `key_model_access_denied`.
Fix: pass the locally-resolved `custom_llm_provider` into both the
sync `responses()` and async `aresponses()` calls so the downstream
`_resolve_model_provider_for_responses` sees an explicit provider
and skips the second prefix-strip.
New regression test
`tests/test_litellm/completion_extras/test_responses_bridge_provider_propagation.py`
pins both call sites: each must forward `custom_llm_provider`.
* fix(28505): set custom_llm_provider on request_data instead of as duplicate kwarg
Greptile flagged that the previous patch passed custom_llm_provider as an
explicit kwarg to responses()/aresponses() while request_data already
carried it via the spread of sanitized_litellm_params, which would raise
TypeError: got multiple values for keyword argument on every real bridge
call.
Switches to assigning request_data['custom_llm_provider'] before the call
so the resolved provider wins over whatever sanitized_litellm_params spread
in, without duplicating the kwarg.
Updates the regression test to seed request_data with a sentinel
custom_llm_provider so it actually exercises the overwrite path (the
previous test mocked transform_request with a minimal dict and never hit
the conflict).
* chore: trigger shin-agent re-eval on retargeted staging base
* chore: trigger shin-agent re-eval against updated Greptile state
* Add 1-hour cache write pricing for EU/AU/JP Bedrock Anthropic models
The 1-hour prompt-cache write tier
(`cache_creation_input_token_cost_above_1hr`) was added to the
us./global. variants of the Claude 4.5/4.6/4.7 family on Bedrock, but
the eu./au./jp. cross-region inference profiles were left without it.
AWS Bedrock pricing applies the same +10% regional premium across all
geo profiles, so eu./au./jp. should carry the same 1-hour rates as
us. (1.6x the 5-minute regional rate).
Without these fields, cost tracking on EU/AU/JP Bedrock 1-hour-TTL
prompt caching falls back to the 5-minute write rate and undercounts
spend by ~60% for European, Australian, and Japanese tenants.
Adds the 1-hour tier (and Sonnet 4.5's long-context >200K tier where
AWS publishes one) to 14 regional Bedrock entries in both
`model_prices_and_context_window.json` and the bundled
`model_prices_and_context_window_backup.json`:
- eu./au. Opus 4.6 ($11.00 / MTok)
- eu./au. Opus 4.7 ($11.00 / MTok)
- eu./au./jp. Sonnet 4.6 ($6.60 / MTok)
- eu./au./jp. Sonnet 4.5 ($6.60 / MTok regular, $13.20 / MTok LC)
- eu./au./jp. Haiku 4.5 ($2.20 / MTok)
Also extends `tests/test_litellm/test_bedrock_anthropic_1hr_cache_pricing.py`
with a `REGIONAL_EXPECTED` parametrized block covering all 13 new
entries plus the existing 1.6x ratio invariant.
Note: `eu.anthropic.claude-opus-4-5-20251101-v1:0` carries the
wrong 5m rate today (base 6.25e-06 instead of regional 6.875e-06),
which would break the 1.6x ratio check. It is intentionally left out
of this PR so the scope stays "1-hour cache tier addition" — a
separate follow-up should correct the EU 5m rates for Opus 4.5.
---------
Co-authored-by: Terrajlz <info@jouleselectrictech.com>
Co-authored-by: Bruno Devaux <devaux.br@gmail.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* Add 1-hour cache write pricing tier for Vertex AI Anthropic models (#28569)
* fix(thinking): handle None thinking param in is_thinking_enabled (#28598)
Squash-merged by litellm-agent from Terrajlz's PR.
* feat(helm): support tpl rendering in podAnnotations (#28609)
Squash-merged by litellm-agent from devauxbr's PR.
* Forward custom_llm_provider through the Responses API bridge (Fixes #28505) (#28575)
* Forward custom_llm_provider through the Responses API bridge (Fixes #28505)
When a Chat Completions request to a GPT-5.4+ model contains both
`tools` and `reasoning_effort`, `completion()` auto-routes through
`responses_api_bridge`. The bridge handler called
`litellm.responses()` / `litellm.aresponses()` without forwarding the
already-resolved `custom_llm_provider`, so the downstream call
re-invoked `get_llm_provider()` with `custom_llm_provider=None` and
stripped a second provider prefix from a `provider/provider/model`
deployment string.
For a deployment configured as `openai/openai/openai/gpt-5.5`,
the bridge flow sent `openai/gpt-5.5` to the upstream API instead of
the correct `openai/openai/gpt-5.5`. Upstream APIs that enforce
model-name allow-lists rejected this as `key_model_access_denied`.
Fix: pass the locally-resolved `custom_llm_provider` into both the
sync `responses()` and async `aresponses()` calls so the downstream
`_resolve_model_provider_for_responses` sees an explicit provider
and skips the second prefix-strip.
New regression test
`tests/test_litellm/completion_extras/test_responses_bridge_provider_propagation.py`
pins both call sites: each must forward `custom_llm_provider`.
* fix(28505): set custom_llm_provider on request_data instead of as duplicate kwarg
Greptile flagged that the previous patch passed custom_llm_provider as an
explicit kwarg to responses()/aresponses() while request_data already
carried it via the spread of sanitized_litellm_params, which would raise
TypeError: got multiple values for keyword argument on every real bridge
call.
Switches to assigning request_data['custom_llm_provider'] before the call
so the resolved provider wins over whatever sanitized_litellm_params spread
in, without duplicating the kwarg.
Updates the regression test to seed request_data with a sentinel
custom_llm_provider so it actually exercises the overwrite path (the
previous test mocked transform_request with a minimal dict and never hit
the conflict).
* chore: trigger shin-agent re-eval on retargeted staging base
* chore: trigger shin-agent re-eval against updated Greptile state
* Add 1-hour cache write pricing tier for Vertex AI Anthropic models
GCP Vertex AI publishes a separate 1-hour cache write column for the
Claude family (1.6x the 5-minute write rate, matching the documented
Bedrock ratio). LiteLLM's Vertex AI Anthropic entries only carry the
5-minute tier, so any request that uses `cache_control: {"ttl": "1h"}`
on Vertex AI Claude is undercounted in cost tracking by ~60%.
The runtime side already supports the 1-hour tier — `VertexAIAnthropicConfig`
extends `AnthropicConfig`, populating `ephemeral_1h_input_tokens`, and
`_calculate_cache_creation_cost` reads `cache_creation_input_token_cost_above_1hr`.
Only the price registry was missing data.
Adds the field to 19 vertex_ai/claude-* entries across both
`model_prices_and_context_window.json` and the bundled
`model_prices_and_context_window_backup.json`:
- Haiku 4.5 ($1.25 -> $2.00 / MTok)
- Sonnet 3.7 / 4 / 4.5 / 4.6 ($3.75 -> $6.00 / MTok)
- Opus 4.5 / 4.6 / 4.7 ($6.25 -> $10.00 / MTok)
- Opus 4 / 4.1 ($18.75 -> $30.00 / MTok)
Adds `tests/test_litellm/test_vertex_anthropic_1hr_cache_pricing.py`
mirroring the Bedrock equivalent — pins each (5m, 1h) pair per model
and asserts the 1.6x ratio across the family.
Fixes #27781.
---------
Co-authored-by: Terrajlz <info@jouleselectrictech.com>
Co-authored-by: Bruno Devaux <devaux.br@gmail.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* Fix Gemini multimodal function responses (#29325)
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* address greptile review: add _transform_image_usage method and model-map supports_image_size flag
- Add _transform_image_usage instance method to GoogleImageGenConfig that
delegates to transform_gemini_image_usage, fixing the regression test
- Replace hardcoded "2.5-flash" string check in supports_gemini_image_size
with a get_model_info lookup on supports_image_size (default true)
- Add supports_image_size: false to all gemini-2.5-flash model entries in
model_prices_and_context_window.json so capability is controlled via the
model map rather than embedded in code
* fix test failures: schema validation, mypy type, model info plumbing, pricing test
- Add supports_image_size to ModelInfoBase TypedDict so get_model_info surfaces it
- Pass supports_image_size through _get_model_info_helper constructor call
- Fix supports_gemini_image_size to use value is not False (None means unset, defaults to True)
- Add supports_image_size to JSON schema in test_aaamodel_prices_and_context_window_json_is_valid
- Correct gemini-3.1-flash-lite pricing assertions in test to match JSON values
* Add Azure AI Kimi K2.6 metadata (#27052)
* Add Azure AI Kimi K2.6 metadata
* Scope Kimi metadata test cost map setup
* fall back to substring check for models not in model_prices_and_context_window.json
Models like gemini-2.5-flash-image-preview are not in the pricing JSON,
so get_model_info raises. Fall back to "2.5-flash" not in model when the
JSON has no explicit supports_image_size entry for the model.
* fix(inception): don't forward global litellm.api_key to Inception FIM
Match the Inception chat config: resolve only an Inception-specific key
(param, litellm.inception_key, or INCEPTION_API_KEY) for the text-completion
FIM path. The global litellm.api_key (often an OpenAI key) was both leaking
to api.inceptionlabs.ai and taking precedence over the configured Inception
key when set.
* fix(auth): enforce end-user budget on custom-auth path that skips common_checks
get_end_user_object() no longer raises BudgetExceededError, so custom-auth
deployments with custom_auth_run_common_checks unset (which skip the
centralized common_checks gate) stopped enforcing the end-user budget,
letting an over-budget end user keep making requests. Re-enforce the
budget in _run_post_custom_auth_checks on that path.
---------
Signed-off-by: José Luis Di Biase <josx@interorganic.com.ar>
Co-authored-by: Isha <72744901+IshaMeera@users.noreply.github.com>
Co-authored-by: aneeshsangvikar <aneeshsangvikar@fiddler.ai>
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Aneesh-Fiddler <aneeshfiddler@gmail.com>
Co-authored-by: Suleiman Elkhoury <108065141+suleimanelkhoury@users.noreply.github.com>
Co-authored-by: Dmitriy Alergant <93501479+DmitriyAlergant@users.noreply.github.com>
Co-authored-by: Yanis Miraoui <yanis.miraoui19@imperial.ac.uk>
Co-authored-by: Lovro Seder <vrovro@gmail.com>
Co-authored-by: Thomas Mildner <12685945+Thomas-Mildner@users.noreply.github.com>
Co-authored-by: José Luis Di Biase <josx@interorganic.com.ar>
Co-authored-by: Lai Quang Huy <64073540+1qh@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com>
Co-authored-by: Terrajlz <info@jouleselectrictech.com>
Co-authored-by: Bruno Devaux <devaux.br@gmail.com>
Co-authored-by: ZHONG Ziwen <67355585+zzw-math@users.noreply.github.com>
Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* Fix : a2a bugs 030626 (#29566)
* Fix error code and context id injection bug
* Add support for all A2A methods
* Add logging
* address greptile review: relay upstream JSON-RPC errors, move _PASCAL_TO_WIRE to module level, add error path tests
* fix(a2a): run pre_call_hook for tasks/resubscribe SSE path to enforce guardrails
tasks/resubscribe was returning the raw SSE stream without calling proxy_logging_obj.pre_call_hook, silently bypassing any guardrails configured on the agent. This patch calls pre_call_hook before streaming begins and wires post_call_failure_hook into the SSE generator so errors are logged. Adds a regression test verifying the hook is called.
* fix(a2a): use get_async_httpx_client instead of creating httpx clients per request
Creating httpx.AsyncClient instances per-request adds ~500ms latency. Switch _forward_jsonrpc and _forward_jsonrpc_sse to use the shared client from get_async_httpx_client(httpxSpecialProvider.A2A).
* fix(a2a): forward caller identity headers on task ops; validate push notification URL
Two security fixes for task management methods:
1. All task operations (tasks/get, tasks/list, tasks/cancel, tasks/resubscribe, push notification config methods) now forward X-LiteLLM-User-Id and X-LiteLLM-Team-Id headers to the upstream agent, so the agent can scope task access to the authenticated caller.
2. tasks/pushNotificationConfig/set validates the callback URL before forwarding: requires HTTPS and rejects private/loopback/reserved IP ranges and localhost hostnames to prevent SSRF.
* Fix A2A task hook and push URL handling
* fix(a2a): fix mypy type errors for request_id and header_name dict key types
* Fix A2A request id and params forwarding
* Forward trace IDs for A2A task calls
* fix(a2a): strip client-forwarded X-LiteLLM-* headers before applying authenticated identity
A client could send x-a2a-<agent>-x-litellm-user-id in their request and have it forwarded to the upstream agent as an authenticated identity header. Fix: sanitize any X-LiteLLM-* headers from agent_extra_headers before merging, then apply the authenticated identity headers last so they always override client-supplied values.
* Fix A2A SSE fallback JSON-RPC error code
* Fix A2A SSE error id backfill
* fix(a2a): validate both push notification url fields to close SSRF bypass
* fix(a2a): widen request_id annotation to match JSON-RPC id call sites
* fix(a2a): run post-call streaming hook for tasks/resubscribe so agent guardrails apply
tasks/resubscribe returned the raw upstream SSE stream without routing events
through the post-call streaming hook, so output guardrails configured on the
agent were silently skipped for streaming task subscriptions while every other
task method and message/stream applied them. Parse upstream JSON-RPC SSE events
and feed them through async_streaming_data_generator, matching message/stream,
so guardrails inspect the streamed task content. Adds a regression test that
fails when the streamed events bypass the guardrail hook.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic/adapter): emit thinking block for reasoning_content-only streaming chunks (#29600)
* fix(anthropic/adapter): open thinking block for reasoning_content-only streaming chunks
The /v1/messages streaming content-block classifier (_translate_streaming_openai_chunk_to_anthropic_content_block) only recognized thinking_blocks. OpenAI-compatible reasoning backends (vLLM/SGLang reasoning parsers: DeepSeek-R1, Qwen3, gpt-oss, ...) populate reasoning_content with thinking_blocks=None, so the classifier fell through to a text block. The delta translator already emits thinking_delta for reasoning_content, so those deltas landed inside a text block and Anthropic streaming clients (Claude Code, SDK .stream()) silently dropped the chain-of-thought.
Mirror the reasoning_content fallback already present in the non-stream translator and the streaming delta translator so the classifier opens a thinking block. Adds a focused regression test.
* fix(anthropic/adapter): reach reasoning_content branch when thinking_blocks attr is absent
Delta deletes the thinking_blocks attribute when unset, so the prior nested check was unreachable for reasoning-only chunks (vLLM/SGLang). Make it a sibling elif so the content block is classified as thinking.
* test(proxy): stop component-allowlist test leaking DATABASE_URL into xdist peers
The component-allowlist test pins throwaway DATABASE_URL/LITELLM_MASTER_KEY
values at import time via os.environ so importing proxy_server doesn't need a
live database. Those values persisted for the whole pytest-xdist worker, so a
sibling test sharing the worker (test_key_rotation_e2e's DB-backed E2E case)
saw the leaked sqlite DATABASE_URL, treated it as an available database instead
of skipping, and the Prisma engine rejected the non-postgres URL (P1012 ->
httpx.ConnectError). Restore the prior environment after the import so the
throwaway values never escape the module.
---------
Co-authored-by: Tai An <antai12232931@outlook.com>
* ci: reproduce default-Windows wheel install to guard MAX_PATH (#29597)
* ci: reproduce default-Windows wheel install to guard MAX_PATH
The existing using_litellm_on_windows job installs the project with
`uv sync`, an editable source install that never copies package files
into a deep site-packages path, so it cannot see the 260-char MAX_PATH
overflow that breaks `pip install litellm` on default Windows. The
content-filter benchmark fixtures have hit that limit three times
(#21941, #22039, #29536), each caught only after release.
This adds a guard to the same job that builds the wheel and installs it
the way an end user would: into a venv whose site-packages prefix is
padded to a realistic worst-case Windows length (~100 chars), then
asserts the install completes and litellm imports. Any packaged path
long enough to bust MAX_PATH at that prefix is reported up front, so the
check is deterministic regardless of the runner's long-path setting,
while the real install also covers failure modes a length heuristic
cannot (half-unpacked packages, reserved names, case collisions).
This commit is the guard only; on the current tree it correctly fails
because nine fixtures still exceed the limit. The rename that brings
them back under it follows on this branch.
* fix(packaging): shorten content-filter benchmark fixtures under MAX_PATH
The 10 content-filter benchmark result fixtures used the legacy
block_{topic}_-_contentfilter_({yaml}).json naming, up to 176 chars
inside the wheel, which busts the Windows 260-char MAX_PATH limit once
extracted under a realistic site-packages prefix and aborts
`pip install litellm` on default Windows.
Rename them to the short {topic}_cf.json scheme that
_save_confusion_results already emits today (it splits the label on the
em-dash and writes f"{topic}_cf"), matching the insults_cf.json and
investment_cf.json files fixed earlier. Re-running the eval suite now
regenerates these same short names rather than recreating the long ones.
This drops the longest packaged path from 176 to 128, so the guard added
in the previous commit goes from red to green with a 32-char margin.
* test(windows): tidy MAX_PATH guard per review
Close the wheel zip via a context manager rather than leaning on
refcount collection, and select the wheel under dist/ by newest mtime so
a stale artifact from an earlier build cannot be tested instead of the
one just produced. Also pin down the venv-depth formula with a short
note: the +2 is the separator joining the venv root to "Lib" plus the
trailing separator before the entry, which lands the simulated
site-packages prefix at exactly 100 chars.
* fix(vertex): strip output_config.effort for Vertex Claude models that reject it (Haiku 4.5) (#29585)
* fix(vertex): strip output_config.effort for models that reject it
Haiku 4.5 on Vertex AI does not support output_config.effort and 400s with
"output_config.effort: Extra inputs are not permitted". PR #27074 emptied
VERTEX_UNSUPPORTED_OUTPUT_CONFIG_KEYS so effort would forward for Opus/Sonnet
4.6+, but that made the strip unconditional across every Vertex Anthropic
model, including ones that don't support it. Claude Code injects effort into
its default Messages payload, so `claude --model claude-haiku-4.5` started
failing.
Make the sanitizer model-aware: drop output_config.effort for models that
don't advertise output_config support (or any reasoning effort level) while
forwarding it for those that do. The fix covers both the chat-completion and
Messages pass-through transformation paths since they share the helper.
* chore(vertex): log at debug when dropping unsupported output_config.effort
Operators pointing an unregistered Vertex Claude alias that does support
effort would otherwise see it stripped with no signal. Debug level keeps it
out of normal logs since Claude Code sends effort on every request.
* Litellm websocket improvements (#29563)
* Add support for websocket via codex
* Add model alias and creds support
* fix: skip cost tracking for WS session wrapper call types
The @client decorator on _aresponses_websocket fires async_success_handler
with result=None after the session ends. This triggered cost tracking errors
because standard_logging_object is never built for None results.
Per-turn costs are correctly tracked by individual litellm.aresponses calls
inside the session. The outer session-level logging obj should not attempt
cost tracking.
Fix: skip _aresponses_websocket and _arealtime call types in deployment_callback_on_success,
RouterBudgetLimiting.async_log_success_event, and _PROXY_track_cost_callback.
* fix: address Greptile review comments
Fix JSON injection: use json.dumps instead of f-string interpolation for model name in WS body.
Add 30s timeout for first WS frame to prevent unbounded connection resource tie-up.
Restore per-event model override in streaming_iterator; fall back to connection-level model when event omits it.
Strengthen regression test: inject alias into kwargs via _update_kwargs_with_deployment mock so the test would fail on un-fixed code.
* fix: handle nested response.create format in first-frame model extraction
When ?model= is omitted, the first WS frame can carry the model in either flat
format (first_event["model"]) or nested format (first_event["response"]["model"]).
The flat-only check would silently reject clients using the nested wire format.
Mirrors the same two-format logic in _build_base_call_kwargs.
* fix: don't force connection-level custom_llm_provider on per-event model overrides
If a client sends a different model per response.create turn, litellm needs to
re-resolve the provider from that model string. Forcing the connection-level
custom_llm_provider would silently route the request to the wrong backend.
Only inject custom_llm_provider when the per-event model matches the
connection-level model.
* refactor: extract WS model extraction into testable function
Pull the flat/nested model extraction into _extract_model_from_first_ws_event
so tests import and exercise the real function rather than a copy.
* fix: compare providers not full model strings in _inject_credentials
The model == self.model guard was too strict: same-provider model variants
(e.g., vertex_ai/gemini-2.0 -> vertex_ai/gemini-1.5 on one connection) would
lose custom_llm_provider, breaking routing when a custom api_base is in use.
Compare the provider extracted by get_llm_provider instead, so same-provider
variants still inherit the connection-level provider while cross-provider
overrides let litellm re-resolve.
* style: black formatting
* refactor: extract first-frame model resolution to fix PLR0915 (too many statements)
* Fix responses WebSocket first-frame validation
* fix: classify WS first-frame read errors and clarify cost-skip log
Distinguish client disconnects from server errors when reading the
responses WebSocket first frame, make the cost-tracking skip log message
accurate for session wrappers (which do carry a model), and resolve the
connection-level provider once per session instead of on every
response.create event.
* test: cover WS first-frame read errors and same-provider credential injection
Adds regression tests for the still-uncovered responses WebSocket paths:
the timeout, invalid-JSON and missing-model branches of
_read_ws_model_from_first_frame, plus the provider comparison in
ManagedResponsesWebSocketHandler._same_provider and _inject_credentials
(same-provider model variants keep the connection provider; cross-provider
models re-resolve).
* fix(responses-ws): fall back to explicit custom_llm_provider when connection model is unresolvable
When a WebSocket session is opened with a custom deployment alias that litellm
cannot resolve to a provider, _connection_provider was None, so _same_provider
returned False for every resolvable per-event model and the connection-level
custom_llm_provider was dropped. Use the explicitly-set custom_llm_provider as
the connection provider in that case so same-provider per-event models still
inherit it while genuinely cross-provider models continue to re-resolve.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(arize/phoenix): OpenInference rendering parity — tool_calls, cost, passthrough I/O, session/user, multimodal, cache tokens (#28800)
* feat(arize): enrich OpenInference attributes for better span rendering
Pure rendering enhancements to the Arize / Arize Phoenix integration. No
existing attribute keys or values are removed or overwritten; every new
emit is independently try/except-wrapped and fires only when its source
data is present so existing behavior is preserved.
What this adds
- Coerce non-dict response objects (e.g. httpx.Response from passthrough
routes) via JSON decode so id/model/usage extraction stops crashing
with "'Response' object has no attribute 'get'". Dicts and Pydantic
objects with .get pass through unchanged.
- Set OPENINFERENCE_SPAN_KIND defensively early so a downstream failure
can't blank the kind; the original late write (incl. TOOL upgrade) is
preserved.
- Add "passthrough" keyword to _infer_open_inference_span_kind so
allm_passthrough_route / llm_passthrough_route resolve to LLM instead
of UNKNOWN.
- Emit cache token breakdown: LLM_TOKEN_COUNT_PROMPT_DETAILS_CACHE_READ /
_CACHE_WRITE / _AUDIO. Sources covered: OpenAI prompt_tokens_details
and Anthropic / Bedrock cache_{read,creation}_input_tokens.
- Render assistant tool_calls on both input and output messages via
MESSAGE_TOOL_CALLS.* (Pydantic-aware, handles ModelResponse choices).
Tool-result input messages also get MESSAGE_TOOL_CALL_ID and
MESSAGE_NAME.
- Render multimodal list-shaped content via MESSAGE_CONTENTS.* (OpenAI
image_url, Anthropic source.{media_type,data} as data: URI). Legacy
MESSAGE_CONTENT write is unchanged.
- Emit SESSION_ID (end_user_id / trace_id), USER_ID (only when not
already set by optional_params.user or model_params.user), and
litellm.{team_id,team_alias,key_alias} from StandardLoggingPayload
metadata.
- Emit llm.response.cost as float from StandardLoggingPayload.response_cost.
- Bedrock / Anthropic passthrough normalization: extract input from
additional_args.complete_input_dict and output from the coerced
provider response so INPUT_VALUE / OUTPUT_VALUE / LLM_INPUT_MESSAGES /
LLM_OUTPUT_MESSAGES are populated. Only runs when call_type contains
"passthrough" / "pass_through".
Tests
- 15 new unit tests covering each addition plus explicit regression
guards (USER_ID overwrite protection, passthrough normalizer scope,
coerce identity for dicts/.get-bearing objects, no spurious cache
emits).
- Existing test_arize_set_attributes count bumped from 26 to 27 to
account for the additional defensive span.kind write (same value,
written twice).
- tests/test_litellm/integrations/arize/: 70 passed (55 baseline + 15
new). tests/test_litellm/integrations/test_opentelemetry.py: 221
passed.
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(arize): collapse additive try/except blocks into _safe_emit helper
The additive attribute emitters all share the same shape: run a callable,
swallow any exception to debug log so it cannot blank the span. Hoisting
that pattern into a single _safe_emit(label, fn, *args, **kwargs) helper
removes 5 repeated try/except blocks. Behavior unchanged; arize test
suite still passes (70/70).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(arize): emit cost under canonical llm.cost.total key
Arize's "Total Cost" column reads the OpenInference-standard
`llm.cost.total` attribute. The previous custom `llm.response.cost`
key never surfaced in the trace list. Now emits both keys (canonical +
legacy) so renderers + any existing consumers both work.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(arize): keep span.kind=LLM for tool-using completions + render tool_calls in Output
A chat completion that passes `tools=[...]` or returns `tool_calls` is still
an LLM call per the OpenInference spec — TOOL is reserved for actual tool
execution. The previous override demoted these to TOOL, breaking Arize's
LLM-scoped dashboards/evals and skewing token/cost analytics for any
tool-using traffic.
Additionally, when an assistant response had no text content but did
request tool calls, `output.value` was set to the empty string so Arize's
"Output" pane rendered blank. Now serializes the tool_calls into a compact
JSON summary in `output.value` (the structured `MESSAGE_TOOL_CALLS.*`
attributes are still emitted unchanged).
Cleanups:
- extract `_get_tool_calls` and `_normalize_tool_call` helpers,
deduplicating the dict-vs-Pydantic + function-dict logic across
`_set_choice_outputs`, `_emit_message_tool_calls`, and the new
`_summarize_tool_calls_for_output`.
- drop redundant late `OPENINFERENCE_SPAN_KIND` write — the defensive
early write is now the single source of truth.
- remove a dead local re-import of `MessageAttributes`/`SpanAttributes`.
Tests: 73 pass (added regression guard asserting span.kind stays LLM for
completions that pass tools AND return tool_calls; existing call_count
assertion restored to 26).
Co-authored-by: Cursor <cursoragent@cursor.com>
* chore(arize): tighten cleanup — fold _get_tool_calls into _safe_get
Two tiny cleanups, no behavior change:
- collapse `_get_tool_calls` to use `_safe_get`, removing a 7-line
hand-rolled dict-vs-attribute fallback that duplicated existing logic.
- trim the `_set_choice_outputs` tool-call summary comment from 4 lines
to 2 (was over-explaining).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(arize): address Greptile review — drop session_id=trace_id fallback, remove dead code, fix Black
Three Greptile-flagged issues + the Black formatting CI failure.
1. SESSION_ID no longer falls back to trace_id. Previously every span
without an explicit `user_api_key_end_user_id` would have its
session.id set to the per-request trace_id, which creates one
distinct "session" per request and breaks Arize's Session-grouping
analytics. Now SESSION_ID is emitted only when an explicit end-user
identifier exists, and the trace_id is emitted under its own
`litellm.trace_id` key so spans remain filterable by trace.
2. Removed dead `ArizeOTELAttributes.set_response_output_messages`
override. Confirmed zero callers in the entire repo (the live path
is `_set_choice_outputs` via `_set_response_attributes`). The
override was preexisting dead code, but the expansion of
`_set_choice_outputs` in this PR made the divergence misleading.
3. Removed permanently-dead first branch in cache_write detection.
`_safe_get(prompt_token_details, "cache_creation_tokens")` looks
for a key that neither OpenAI's `prompt_tokens_details` nor
Anthropic's payload ever exposes. Now reads straight off `usage`
for `cache_creation_input_tokens`.
4. Reformatted both files under Black 26.3.1 (the version CI uses
via `uv sync --frozen`). Local previously used 24.10.0.
Tests: 74/74 pass in the arize suite (added
`test_arize_does_not_use_trace_id_as_session_id_fallback`).
Combined arize + opentelemetry suite: 295/295 pass.
End-to-end verified live: tool-call still emits `span.kind=LLM` and
JSON tool_calls in `output.value`; `session.id` is now correctly
unset when no end_user_id is provided; `litellm.trace_id` is
populated; Bedrock passthrough input/output unchanged.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(arize): gate passthrough prompt export on message redaction
- Skip the complete_input_dict bridge in _maybe_normalize_passthrough when
should_redact_message_logging() is true, so enabling redaction no longer
leaks raw passthrough prompts into Arize (Veria security finding).
- Split passthrough input/output rendering into helpers to satisfy PLR0915.
- Remove dead call_type assignment (F841).
Validated live against a Bedrock passthrough proxy exporting to Arize:
non-redacted renders the real prompt on litellm_request; global
turn_off_message_logging yields input.value=redacted-by-litellm with the
raw_gen_ai_request child span suppressed and no SSN/marker leakage.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: passthrough endpoints duplicate logs (#29598)
* fix duplicate cost callbacks for anthropic streaming pass-through
Two bugs caused _PROXY_track_cost_callback to see stream=True +
complete_streaming_response=None on every streaming pass-through request,
making the dedup guard in dispatch_success_handlers permanently inactive:
1. pass_through_endpoints.py created the Logging object with stream=False
for all requests. _is_assembled_stream_success short-circuits on
self.stream is not True, so has_dispatched_final_stream_success was
never set and any second dispatch went through unchecked.
Fix: set logging_obj.stream = True after stream detection.
2. _create_anthropic_response_logging_payload set complete_streaming_response
inside the try block after litellm.completion_cost(), so a pricing error
caused an early return without setting it on model_call_details.
Fix: set complete_streaming_response before the try block.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix stream
* add stream to logging obj
* test(pass_through): give mock logging object a real model_call_details dict
The anthropic passthrough logging payload now records the assembled
response on model_call_details before cost calculation, which requires
model_call_details to support item assignment. In production it is always
a dict; the existing unit test stubbed the logging object with a bare Mock
whose attribute is not subscriptable, so the new assignment raised
TypeError. Use a real dict to match the production logging object.
* test(pass_through): cover streaming logging-obj stream flag
The streaming branch of pass_through_request that marks the logging object
as streaming (logging_obj.stream and model_call_details["stream"]) had no
unit coverage, so the patch coverage gate flagged it. Add a regression test
that drives a streaming pass-through request through pass_through_request and
asserts the logging object is flagged as a stream before dispatch.
* test(pass_through): cover SSE-response stream flag fallback branch
The auto-detected streaming branch of pass_through_request (when a request
that was not flagged as streaming returns a text/event-stream response) sets
logging_obj.stream and model_call_details["stream"] but had no unit coverage,
so the codecov patch gate failed at 60%. Drive a non-streaming pass-through
request whose upstream response is SSE through pass_through_request and assert
the logging object is flagged as a stream before dispatch.
* fix(pass_through): gate complete_streaming_response on stream flag
perform_redaction only scrubs complete_streaming_response when
model_call_details["stream"] is True. Setting it unconditionally for
non-streaming Anthropic pass-through responses left the assembled
response unredacted in model_call_details, which is handed to logging
callbacks as kwargs when message logging is disabled. Only record it for
actual streaming responses so redaction always applies.
---------
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(ci): keep coverage rename green when a parallel node runs no tests (#29608)
* fix(ci): keep coverage rename green when a parallel node runs no tests
local_testing_part1 and local_testing_part2 run with parallelism 4. When
CircleCI reruns only the failed tests, the failed test lands on a single
node and the other nodes receive an empty bucket, so pytest never writes
coverage.xml or .coverage. The unguarded "mv coverage.xml ..." then exits
1 and turns the whole job red even though the rerun passed; the next
persist_to_workspace step would fail the same way on the missing paths.
Guard the rename so a node with no coverage emits empty placeholders
instead. coverage combine tolerates the empty files, so the downstream
upload-coverage job keeps the real nodes' data intact.
* fix(ci): pre-create test-results in litellm_router_testing for empty-bucket reruns
litellm_router_testing also runs with parallelism 4. On a rerun of only the
failed tests, a node can receive no tests, so the test command never creates
test-results and the final store_test_results step can fail on the missing
path. Pre-create the directory up front, matching what local_testing_part1
and part2 already do and CircleCI's own guidance for parallel reruns.
* test(openai): retry wildcard chat completion on transient OpenAI 500
build_and_test reddened on test_openai_wildcard_chat_completion when the
real gpt-3.5-turbo-0125 call returned an OpenAI 500 ("The server had an
error while processing your request"). The base branch passed the same
call concurrently, so the 500 is an intermittent OpenAI server error, not
a regression. Add the same pytest-retry marker the sibling real-call tests
in this file already use so a transient upstream 500 no longer fails CI.
* test(vcr): close out the remaining VCR live-call leaks (#29603)
* Fix remaining VCR live-call leaks
* test(vcr): dedupe live-test helpers and drop spurious kwargs
Extract the duplicated isVertexQuotaError/runVertexRequestOrSkip Vertex
quota-skip helpers into tests/pass_through_tests/vertex_test_helpers.js and the
duplicated _skip_live_prompt_caching_test guard into tests/_live_test_helpers.py
so each lives in one place. In test_aarun_thread_litellm, build a separate
message_data carrying role/content for add_message and a thread_data without
them for run_thread/run_thread_stream/get_messages, which no longer receive the
spurious message fields.
* test(overhead): assert mock transport is exercised in non-streaming and stream tests
* fix(key_generate): exempt UI/CLI session tokens from the budget ceiling for team keys (#29612)
Non-admin users creating a team key through the UI were rejected with
"max_budget cannot exceed the caller's own max_budget (0.25)". The request is
authenticated by a UI/CLI session token whose max_budget is the per-session chat
spend cap (max_ui_session_budget, default $0.25), and the delegated-authority
budget ceiling (GHSA-q775-qw9r-2r4g) treated that cap as a delegation limit.
Skip the ceiling only when a session token creates a team key (data.team_id set);
that key's spend is bounded by the team budget at request time. Personal keys and
every other non-admin caller keep the ceiling, so a session token cannot mint an
arbitrary-budget personal key.
* fix(realtime): allow null transcripts in stream logging payloads (#29625)
Allow realtime event transcript fields to be nullable so GA conversation.item payloads with transcript=null don't fail logging normalization and suppress success callbacks.
Co-authored-by: Cursor <cursoragent@cursor.com>
* build(ui): migrate eslint to flat config and bump eslint-config-next to 16 (#29626)
ESLint 9 defaults to flat config and eslint-config-next was pinned at 15
while Next is on 16, so eslint only ran with ESLINT_USE_FLAT_CONFIG=false
and next lint is gone on Next 16. Replace .eslintrc.json with a native
flat eslint.config.mjs (config-next 16 ships flat configs, so no
FlatCompat shim is needed), bump eslint-config-next to 16.2.6, add
@eslint/js and typescript-eslint as explicit devDeps for the recommended
rule sets, and point the lint script at eslint directly.
This only makes eslint runnable on modern tooling; it does not wire it
into CI. The same rules carry over (next/core-web-vitals, eslint and
typescript-eslint recommended, prettier, unused-imports)
* fix(key_generate): scope session-token team-key budget exemption to caller-supplied team_id (#29641)
#29612 exempts UI/CLI session tokens from the key budget ceiling when they
create a team key, keyed on data.team_id. That value is read after the
default_key_generate_params loop can populate team_id, so on deployments that
set default_key_generate_params.team_id a request the caller did not scope to a
team is treated as a team key and skips the ceiling. Capture _requested_team_id
before defaults run and key the exemption off it, mirroring how
_requested_max_budget is already captured. Requests the caller did not scope to a
team keep the ceiling.
* fix(proxy): disable proxy buffering on streaming SSE responses (#29557)
Streaming responses from the proxy (/chat/completions, /v1/messages,
/v1/responses, assistants) all return through create_response() but never
sent the headers that tell an intermediary reverse proxy not to buffer the
SSE stream. nginx with the default proxy_buffering, k8s ingress-nginx, and
Envoy/Istio sidecars therefore hold the whole stream and release it in one
batch, which looks like a broken/buffered stream to the client even though
litellm is yielding chunks incrementally.
Add Cache-Control: no-cache and X-Accel-Buffering: no to every
StreamingResponse create_response() returns, matching what the proxy already
does for its own usage/policy SSE endpoints. Fixes #28384.
* fix(mcp): gate /public/mcp_hub strictly on litellm.public_mcp_servers (#27764)
* fix(mcp): gate /public/mcp_hub strictly on litellm.public_mcp_servers
* fix(mcp): add public_mcp_hub_strict_whitelist flag (default True) for migration
* ci(ui): frontend-lint job enforcing prettier + eslint on changed files (#29633)
* ci(ui): add frontend-lint job enforcing prettier and eslint on changed files
Lints only the files a PR adds or modifies under ui/litellm-dashboard,
so new and touched code must be prettier-clean and eslint-clean while the
existing tree is grandfathered. Skips cleanly when a PR touches no
lintable UI files. This lets us adopt the formatters incrementally
without a repo-wide reformat
* ci(ui): write frontend-lint file lists to $RUNNER_TEMP
Keep the prettier/eslint changed-file lists out of the checkout dir so
they cannot collide with a future source file of the same name
* lint(ui): baseline existing eslint findings so only new ones block
Capture the current error-level eslint findings (318 across 183 files)
in a committed suppressions baseline via eslint --suppress-all. Every
rule stays at its error severity, so any newly introduced violation
fails the frontend-lint gate, while the existing tree is grandfathered;
touching a legacy file never forces fixing its pre-existing issues. CI
runs eslint with --pass-on-unpruned-suppressions so that fixing a
baselined issue does not fail on a now-stale suppression, and the
generated baseline is prettier-ignored since eslint owns its format.
Burn the baseline down over time with eslint --prune-suppressions
* lint(ui): enforce a count budget for explicit any
Make @typescript-eslint/no-explicit-any a warning and cap the total
instead of hard-blocking each new one. A frontend-lint step counts the
repo-wide explicit any and fails only when it exceeds the committed
budget in eslint-any-budget.json. max starts at 2031, ten above the
current 2021, so the next ten land as warnings and the build fails once
that headroom is gone. Lower max over time toward target to ratchet the
count down. New anys still surface as warnings on changed files via the
normal eslint step
* lint(ui): enable zero-cost rules no-var, no-self-assign, react/no-danger
These have no existing violations, so they need no baseline; turning them
on purely blocks new instances. react/no-danger guards against new
dangerouslySetInnerHTML (XSS), no-var enforces let/const, and
no-self-assign catches self-assignment typos. no-debugger is already
enforced by the recommended preset
* lint(ui): add baselined complexity rules
Enable complexity:20, max-depth:4, max-params:4, max-nested-callbacks:4,
with thresholds set near the codebase p99 so only genuine outliers are
flagged. The 272 existing over-threshold functions are grandfathered in
the suppressions baseline; new over-threshold functions block. Lower the
thresholds over time to ratchet complexity down. max-lines-per-function
is intentionally left off since React components are legitimately long
* lint(ui): ban new raw fetch, standardize on React Query
Add a no-restricted-syntax rule flagging bare fetch() calls, pointing
contributors at React Query (@tanstack/react-query). The rule is not
exempted anywhere, including the already-bloated networking.tsx, so all
331 existing fetch calls are grandfathered but no new ones can be added
there or elsewhere. New data access goes through React Query, and the
networking layer can be migrated out and pruned from the baseline over
time
* lint(ui): ban new @tremor/react imports
Add a no-restricted-imports rule flagging imports from @tremor/react so
tremor is phased out rather than spread further. The 232 existing tremor
imports are grandfathered in the baseline; new ones block and point at
antd. Migrate components off tremor and prune the baseline over time
* lint(ui): widen explicit-any budget headroom to 2040
Raise max from 2031 to 2040, giving ~19 of slack over the current 2021
instead of 10
* style(ui): prettier-format eslint.config.mjs
The frontend-lint gate flagged its own config file. Format it so the
prettier check on this PR's changed files passes
* lint(ui): soften complexity and max-depth to warnings
These two are smell metrics with arbitrary thresholds where a legit new
function can trip them, so make them advisory rather than hard-blocking.
They drop out of the baseline (now 963). max-params, max-nested-callbacks,
and the react-hooks rules stay strict since those are clear-cut
* lint(ui): move complexity and max-depth to the count-budget pattern
Generalize the explicit-any budget into a shared lint-budget mechanism:
eslint-budgets.json maps a rule to {max, target} and check-lint-budgets.mjs
counts each across the repo and fails when a count exceeds its max.
complexity (129, max 140) and max-depth (61, max 70) now use the same
slack-plus-counter model as explicit-any (2021, max 2040): they warn
per-file and the build only fails if the repo-wide total crosses the
ceiling. Lower each max toward its target over time
* docs(ui): note pruning the eslint suppressions baseline when fixing lint debt
* fix(gemini): googleSearch + server-side tools and googleMaps JSON schema (#29582)
* fix(gemini): keep googleSearch with server-side tools and googleMaps JSON schema
Wire include_server_side_tool_invocations through completion() so mixed
google_search and function tools are not dropped on Gemini 3+. Rewrite
generationConfig to responseFormat when googleMaps is used with JSON schema.
Fixes #27479
Fixes #29451
Co-authored-by: Cursor <cursoragent@cursor.com>
* address greptile review feedback (greploop iteration 1)
* style: fix black formatting in main.py for py312 compat
* Fix Gemini Google Maps extra_body JSON rewrite
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): passthrough 404 when SERVER_ROOT_PATH is set (#29658)
* fix(proxy): match passthrough registry routes bare-to-bare with SERVER_ROOT_PATH
After #28547, get_request_route strips the deployment prefix while registry
lookup still re-inflated stored paths via SERVER_ROOT_PATH, causing 404s
under paths like /llmproxy/ml. Compare normalized bare routes in both
is_registered_pass_through_route and get_registered_pass_through_route.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(proxy): patch utils.get_server_root_path in passthrough auth tests
After removing get_server_root_path from pass_through_endpoints, route
and JWT tests must mock litellm.proxy.utils where normalization reads it.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(gemini-realtime): use GA event names for Pipecat 1.3.x compatibility (#29662)
* fix(gemini-realtime): use GA event names for Pipecat 1.3.x compatibility
Pipecat v1.3.0 adopted the OpenAI Realtime API GA event naming:
response.audio.delta -> response.output_audio.delta
response.text.delta -> response.output_text.delta
response.audio.done -> response.output_audio.done
response.text.done -> response.output_text.done
The proxy was still emitting the old beta names; Pipecat's
`parse_server_event` raises "Unimplemented server event type" for any
unknown type, which killed the receive task handler and broke audio
playback and tool-call delivery.
Also:
- conversation.item.created -> conversation.item.added (already handled)
- client audio is buffered until backend setupComplete in deferred mode
- call_id fallback UUID when Gemini returns empty id
- status_details / token detail fields added to Pydantic-strict events
The _GA_TO_BETA_EVENT_TYPES map in RealTimeStreaming already translates
GA names back to beta for clients that opt in with the openai-beta
header, so legacy clients are unaffected.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(gemini-realtime): address greptile review comments
- emit outputTranscription as response.output_audio_transcript.delta
instead of suppressing it; GA_TO_BETA map handles translation for
legacy clients
- cap pre-setup audio buffer at 200 frames to prevent memory exhaustion;
log a warning when the limit is hit and additional frames are dropped
- log remaining dropped message count on flush error
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(gemini-realtime): address veria review comments
- remove unused OpenAIRealtimeConversationItemCreated import
- fix guardrail bypass: semantic_vad early-return now preserves
create_response when set so a guardrail-injected create_response:false
is not silently dropped
- add per-connection 10 MB byte cap alongside the 200-frame count cap
for the pre-setup audio buffer to prevent memory exhaustion
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(gemini-realtime): fix mypy arg-type on _finalize_gemini_live_setup
setup parameter typed as BidiGenerateContentSetup to match the TypedDict
passed at both call sites; was dict which mypy rejected.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(gemini-realtime): widen _finalize_gemini_live_setup to Dict[str, Any]
BidiGenerateContentSetup (TypedDict) is a subtype of Dict[str,Any] so
both call sites (one passing a plain dict, one passing the TypedDict)
satisfy mypy.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(gemini-realtime): cast BidiGenerateContentSetup to Dict at _finalize call site
mypy rejects TypedDict as dict[str, Any] argument; cast at the call site
where follow_up_setup is BidiGenerateContentSetup to satisfy the checker.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix Gemini realtime beta compatibility
* Fix deferred Gemini setup audio ordering
* fix: preserve Gemini audio transcript ids
* fix(realtime): cap pre-setup client buffer on all append paths
Route every append to the deferred-setup pending buffer through the
per-connection message/byte caps. Previously only the audio-buffer
fast path enforced the caps; once one frame was buffered, a client
that withheld session.update could stream arbitrary frames into
_pending_messages_until_setup unbounded and exhaust proxy memory.
* style(gemini-realtime): apply black formatting to transformation.py
* fix(gemini-realtime): log beta-translation fallback and name native-audio marker
Surface the previously swallowed exception in _send_event_to_client so a
failed GA->beta translation is observable instead of silently forwarding the
untranslated event. Extract the native-audio model substring used by
_finalize_gemini_live_setup into a named constant documenting why speechConfig
is dropped on those setups.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* Litellm oss staging 040626 (#29671)
* fix(azure): apply api_version fallback chain to image edit URL
`AzureImageEditConfig.get_complete_url` only read `api_version` from
`litellm_params`. When callers configured it via `litellm.api_version`
or `AZURE_API_VERSION`, the constructed URL had no `?api-version=` and
Azure responded `404 Resource not found`.
Apply the same fallback chain the Azure chat path already uses in
`common_utils.py`:
litellm_params > litellm.api_version > AZURE_API_VERSION env >
litellm.AZURE_DEFAULT_API_VERSION
Adds 5 unit tests pinning each layer of the chain plus a regression
guard for `api_base` that already carries `?api-version=`.
* feat(mcp): core sampling and elicitation flow with security hardening
- Add sampling_handler.py: full MCP sampling/createMessage flow with
model selection (hint-based + priority-based), auth enforcement,
budget checks, route restriction gates, and tag policy pre-auth
- Add elicitation_handler.py: MCP elicitation/create relay with
downstream client capability detection
- Wire sampling/elicitation callbacks in mcp_server_manager.py
gated behind allow_sampling/allow_elicitation config flags
- Add allow_sampling/allow_elicitation fields to MCPServer type
- Fix session lock deadlock: skip lock for JSON-RPC response POSTs
(elicitation/sampling replies) with truncated-body heuristic
- Extend client.py with sampling_callback and elicitation_callback
- Security: RouteChecks gate, tag-budget bypass fix, x-forwarded-for
spoofing fix, Latin-1 header encoding guard
- Add 4 new test modules (model access, priority selection, request
builder, tool conversion) + update existing MCP tests
* fix(security): run pre-call guardrails before MCP sampling acompletion
Without this, an upstream MCP server with allow_sampling enabled could
send prompts that bypass every guardrail (content filtering, PII
redaction, prompt-injection detection) configured on /chat/completions.
- Call proxy_logging_obj.pre_call_hook(call_type='acompletion') before
llm_router.acompletion so guardrails fire for sampling sub-calls
- Add HTTPException to the re-raise list so guardrail rejections
propagate correctly instead of being swallowed as generic errors
* feat(bedrock_mantle): add Responses API support (/openai/v1/responses) (#29490)
* feat(bedrock_mantle): add Responses API transformation config
* test(bedrock_mantle): cover trailing-slash api_base normalization
* feat(bedrock_mantle): export BedrockMantleResponsesAPIConfig
* feat(bedrock_mantle): register gpt-5.x Responses config (gpt-oss unchanged)
* feat(bedrock_mantle): add gpt-5.5/gpt-5.4 Responses price-map entries
* refactor(bedrock_mantle): exclude gpt-oss instead of allow-listing gpt-5 for Responses routing
Frontier OpenAI models on Bedrock Mantle are Responses-only on /openai/v1/responses;
gpt-oss is the legacy family that also speaks chat-completions. Gate by excluding
gpt-oss (which keeps its chat-completions emulation) and defaulting everything else
to the native Responses config, so future frontier models (gpt-6, etc.) route
correctly without a code change. Verified against the live us-east-2 Mantle endpoint:
gpt-oss 400s on /openai/v1/responses while gpt-5.5 400s on both standard paths.
* test(bedrock_mantle): cover supports_native_websocket opt-out
Closes the one uncovered line flagged by codecov on the Responses config.
The assertion documents that Mantle Responses has no realtime/websocket
transport, so realtime routing must not attempt a socket it cannot serve.
* fix(bedrock_mantle): route file_search through emulation instead of forwarding to Mantle
BedrockMantleResponsesAPIConfig inherited supports_native_file_search()
-> True from OpenAIResponsesAPIConfig but never overrode it. Mantle has no
OpenAI vector stores, so a forwarded file_search tool is rejected with a
400 (verified upstream: Tool type 'file_search' is not supported). Opting
out, like the existing supports_native_websocket override, routes the tool
through LiteLLM's file_search emulation instead.
* fix(bedrock_mantle): only route ope…
* fix(mcp): handle OAuth IdP error responses in /callback (LIT-2750) Per RFC 6749 section 4.1.2.1, when the IdP rejects an OAuth authorization request it redirects back to the client with ?error=...&error_description=... and no code. The MCP /callback handler declared code and state as required query params, so FastAPI rejected such error responses with a 422 before the handler ran -- stranding the MCP client waiting on the loopback. This change: - Makes code and state optional and accepts the RFC-defined error, error_description, and error_uri params. - When state decodes to a trusted client redirect_uri, propagates the error params back to that URI with the client's original (un-wrapped) state preserved, so the client's OAuth library can surface the failure. - When state is missing/undecryptable or the encoded redirect_uri is no longer trusted, renders a 400 HTML page with the (HTML-escaped) error details instead of leaking to an attacker-controlled redirect. - Preserves the existing success path (code + state -> 302 to validated client redirect_uri with original state). Fixes LIT-2750. * test(mcp): regression tests for /callback handling IdP error responses (LIT-2750) Adds a new test module covering the LIT-2750 fix: the MCP OAuth /callback endpoint must accept IdP error responses (e.g. ?error=access_denied) per RFC 6749 section 4.1.2.1 instead of returning a 422 because ``code`` is missing. Coverage: - IdP error with no state -> 400 HTML page surfacing the error. - HTML escaping of user-controlled error / error_description fields. - IdP error with a trusted (loopback) state -> 302 propagating error / error_description / original client state to the client. - IdP error with an untrusted redirect_uri encoded in state -> 400 inline (no open-redirect to attacker-controlled origin). - IdP error with an undecryptable state -> 400 HTML fallback. - Bare GET /callback with no params -> 400 HTML (not Pydantic 422). - Success path (code + state) still 302 to validated client redirect_uri with the original (un-wrapped) state preserved. * refactor(mcp): drop unused _OAUTH_ERROR_PARAMS constant (Greptile P2) The tuple was leftover scaffolding from an earlier draft of the LIT-2750 fix; nothing references it. The explanatory RFC 6749 §4.1.2.1 comment block above the callback handler covers the same intent. * fix(mcp/oauth): preserve empty original_state and clarify missing-param error in /callback Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(thinking): handle None thinking param in is_thinking_enabled (BerriAI#28598) Squash-merged by litellm-agent from Terrajlz's PR. * feat(helm): support tpl rendering in podAnnotations (BerriAI#28609) Squash-merged by litellm-agent from devauxbr's PR. * fix: apply black formatting to base_llm chat transformation Fix CI black --check failure on is_thinking_enabled return formatting. Co-authored-by: Cursor <cursoragent@cursor.com> * merge main (BerriAI#28836) * fix(proxy): Bedrock Knowledge Base pass-through: preserve SigV4 headers and signed request body (BerriAI#27526) * Fix Bedrock KB pass-through SigV4 headers and signed body Coerce botocore HeadersDict to a dict for pass-through routes. When forward_headers is true, drop request headers that collide case-insensitively with signed headers so client Bearer auth does not shadow AWS SigV4. Send prepped.body as raw content so the outbound payload matches the signature after logging hooks mutate the parsed dict. Co-authored-by: Cursor <cursoragent@cursor.com> * Simplify pass-through raw body handling Read the SigV4-signed bytes directly from request.state inside pass_through_request instead of threading a custom_raw_body argument through three functions. Helper methods are restored to their original signatures, and the new branch lives in one place at each httpx call site. Co-authored-by: Cursor <cursoragent@cursor.com> * Harden pass-through raw body read from request.state Guard missing request.state (test fixtures) and ignore non-bytes/str values so MagicMock does not trigger the SigV4 raw-body path. Co-authored-by: Cursor <cursoragent@cursor.com> * Test pass_through_request state_raw_body uses httpx content= Cover non-streaming (async_client.request) and streaming (build_request) paths so SigV4 bytes on request.state are not replaced by json= of a hook-mutated dict. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> * chore(tests): migrate Bedrock CI to AWS account 941277531214 (BerriAI#28728) * chore(tests): migrate Bedrock CI from AWS account 888602223428 to 941277531214 The original account (888602223428) was put under a security restriction by AWS after a root access key leaked in a PR comment. While that account works its way through the AWS Support unlock process, Bedrock-touching CI tests have been migrated to a fresh account (941277531214). Changes: - Replace 26 hardcoded references to 888602223428 with 941277531214 across 8 files (provisioned-model ARNs, imported-model ARNs, AgentCore runtime ARNs, batch execution role ARN, and example proxy config). - The provisioned-model and imported-model ARNs are referenced only from mocked unit tests — no AWS resources to recreate. - The batch execution IAM role has been recreated in the new account with the same name and equivalent permissions. - The two AgentCore runtimes (hosted_agent_r9jvp-3ySZuRHjLC, hosted_agent_13sf6-cALnp38iZD) are being recreated in the new account under the same names — see tools/agentcore-deploy/ in a follow-up. CircleCI env vars AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_REGION_NAME were updated separately via the CircleCI API to point at the new account. Smoke-tested locally against the new account: aws bedrock-runtime converse --region us-west-2 \ --model-id us.anthropic.claude-sonnet-4-5-20250929-v1:0 \ --messages '[{"role":"user","content":[{"text":"ping"}]}]' → 200, model returned 'pong' * chore(tests): refresh AgentCore ARN suffixes to match newly-deployed runtimes The first migration commit replaced just the account ID, but AgentCore auto-assigns a random 10-char suffix to every runtime on creation — we can't reuse the original suffixes (`3ySZuRHjLC`, `cALnp38iZD`) in the new account. Updated the AgentCore-runtime ARNs in the three files that reference real runtime IDs (not the mock-based unit-test ARNs). Deployed runtimes: arn:aws:bedrock-agentcore:us-west-2:941277531214:runtime/hosted_agent_r9jvp-Rq79QFC2fp arn:aws:bedrock-agentcore:us-west-2:941277531214:runtime/hosted_agent_13sf6-4046UzHSwy Both runtimes are status=READY and pass a smoke invoke: $ aws bedrock-agentcore invoke-agent-runtime --agent-runtime-arn ... --payload '{"prompt":"ping"}' → 200, {"result": "echo: ping"} The agent is a minimal echo (see /tmp/agentcore_deploy/agent.py for the deploy artifacts). Tests that only verify the SDK wiring will pass; if any test asserts on agent output content, swap the echo for the real agent. * chore(tests): point Bedrock batch tests at new-account S3 bucket The account migration (888602223428 -> 941277531214) was a flat account-ID swap, which only rewrites ARNs that embed the account number. S3 bucket names carry no account ID, so the live Bedrock batch tests still uploaded to `litellm-proxy` — a bucket that lives in the old account. S3 names are globally unique, and the old account still holds that name, so it can't be recreated in the new account. Rename to `litellm-proxy-941277531214` (account-ID suffix guarantees global uniqueness). The bucket must be created in 941277531214 and the batch execution role granted s3:GetObject/PutObject/ListBucket on it before this job is run in CI. * chore(tests): point live S3 logging test at new-account bucket Same account-ID-free blind spot as the batch bucket: `load-testing-oct` lives in the old account and its name can't be reused globally. The `logging_testing` CI job is wired into the workflow and runs test_basic_s3_logging, which uploads to this bucket with the CI env creds, then lists and deletes objects — a live dependency. Rename to `load-testing-oct-941277531214`. The bucket must exist in the new account with the CI IAM principal granted s3:PutObject/GetObject/ListBucket/DeleteObject before this job runs. * chore(tests): repoint Bedrock guardrail IDs to new-account guardrails The migration left guardrail IDs untouched (no account ID in them), so all live guardrail tests failed with "guardrail identifier or version does not exist" against 941277531214. Recreated both guardrails in the new account and updated the hardcoded IDs: - wf0hkdb5x07f -> zgkmukebruil (PII mask: PHONE + CREDIT_DEBIT_CARD, with explicit inputAction=ANONYMIZE so masking applies to INPUT, which is the source litellm's moderation hook sends) - ff6ujrregl1q -> 4w3d1di3snt5 (blocks "coffee"; blocked message set to the exact string the tests assert on) Updated test_bedrock_guardrails.py, otel_test_config.yaml, and the guardrailConfig in test_bedrock_completion.py. Verified locally: the 5 previously-failing guardrail tests now pass. * test(bedrock): migrate legacy models to current inference profiles The new CI account (941277531214) cannot invoke legacy Bedrock models (AWS gates them: "marked by provider as Legacy... not actively using in the last 30 days"). Migrated the live-call tests: - anthropic.claude-3-sonnet-20240229 -> us.anthropic.claude-sonnet-4-5-20250929-v1:0 - anthropic.claude-3-haiku-20240307 -> us.anthropic.claude-haiku-4-5-20251001-v1:0 Current Claude models on Bedrock require the us. inference-profile prefix (bare on-demand ids are rejected). cohere.command-r-plus has no working replacement (all Cohere is legacy- gated in the new account): swapped to claude-haiku-4-5 in provider- agnostic param lists. amazon.titan-image-generator skipped (no working replacement). Mocked/transformation/cost tests that reference the legacy strings are intentionally left unchanged. Verified live against the new account. * test(bedrock): repoint SageMaker + Knowledge Base to new-account resources These referenced account-scoped resources by hardcoded id that only existed in the old account, so the migration's account-ID swap missed them. Recreated in 941277531214 and repointed: - SageMaker endpoint jumpstart-dft-hf-textgeneration1-mp-20240815-185614 -> litellm-ci-textgen (gpt2 on a TGI container, ml.g5.xlarge) - Bedrock Knowledge Base T37J8R4WTM -> LCYXFBR2TU (OpenSearch Serverless vector store + titan-embed-text-v2, seeded with a LiteLLM doc) Verified live: test_sagemaker.py (12 passed) and test_bedrock_knowledgebase_hook.py (12 passed). * test(reasoning_effort_grid): skip bedrock claude-opus-4-7 cells (not entitled on 941277531214) claude-opus-4-7 is listed in the new Bedrock CI account's foundation models but invoke is denied (AccessDeniedException: "not available for this account"). Bedrock access to the flagship Opus requires an AWS Sales request, not the self-serve model-access toggle, so it can't be enabled inline with the rest of the account migration. Add an optional `skip_reason` to ModelEntry and set it on the bedrock-claude-opus-4-7 entry; the grid test honors it via pytest.skip. Cell count (231) and route coverage are unchanged, so the structural asserts still pass. Restore coverage by deleting the one skip_reason line once access is granted. * test(bedrock): swap/skip legacy-gated models unavailable on new CI account The migrated AWS account (941277531214) cannot access several models that the old account could, so the remaining red CI jobs were hitting real Bedrock "Access denied / Legacy" and "account not authorized" errors: - image_gen: skip both Nova Canvas test classes (amazon.nova-canvas-v1:0 is legacy-gated), matching the existing titan skip. - batches: skip test_async_file_and_batch (Bedrock batch inference is not authorized on the new account; requires an AWS support case). - litellm_overhead: swap legacy claude-3-5-haiku for the active us.anthropic.claude-haiku-4-5 inference profile. - test_completion_claude_3_function_call: swap legacy claude-3-sonnet for the active us.anthropic.claude-sonnet-4-5 inference profile. https://claude.ai/code/session_01Y7zgHYu9GX29YRwV4yiWAa * test(bedrock): fix remaining e2e legacy-model + batch failures on new CI account - e2e_openai_endpoints: skip test_bedrock_batches_api (Bedrock batch inference is not authorized on account 941277531214) and migrate the missed s3_bucket_name in oai_misc_config.yaml to litellm-proxy-941277531214. - build_and_test: swap legacy bedrock claude-3-sonnet for the active us.anthropic.claude-sonnet-4-5 inference profile in the proxy structured output e2e test. https://claude.ai/code/session_01Y7zgHYu9GX29YRwV4yiWAa * test(bedrock): make opus-4-7 + batch cells fail loudly and mock image-gen (BerriAI#28791) Replace the silent skips added for the new CI account with noisier behavior: - reasoning-effort grid: opus-4-7 cells now fail (when AWS creds are present) instead of skipping, so the missing entitlement stays visible in CI; they still skip when AWS creds are absent (local dev) - Bedrock batch inference tests: drop the skip so they run and fail until batch access is granted - Titan + Nova Canvas image-gen tests: mock the Bedrock HTTP call so the transform + cost-tracking path stays under test without live model access https://claude.ai/code/session_01MT7SWDnXUjv6e6EPG7BDjT * test(bedrock): use pytest.xfail for known-failing opus-4-7 cells Replace pytest.fail with pytest.xfail when a model has a fail_reason, so known-broken cells stay visible as XFAIL without keeping CI red. Co-authored-by: Yassin Kortam <yassin@berri.ai> --------- Co-authored-by: Mateo <mateo@Mateos-MacBook-Pro.local> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(otel): export SERVER span on management-endpoint success without http_request (BerriAI#28794) Co-authored-by: Yassin Kortam <yassinkortam@Yassins-MacBook-Pro.local> * chore(ci): merge dev branch (BerriAI#28801) * chore(proxy): route path-dependent call sites through get_request_route Replace direct ``request.url.path`` reads in auth, ACL, routing, and audit-log decisions with ``get_request_route(request)`` — the helper already added in ``auth/auth_utils.py`` that returns the ASGI ``scope["path"]`` with ``root_path`` stripped. Starlette reconstructs ``url.path`` from the Host header; ``scope["path"]`` is uvicorn's parse of the request line and matches what FastAPI dispatches on, so it's the authoritative route for any decision that should agree with the actual handler. Sites: - _experimental/mcp_server/auth/user_api_key_auth_mcp.py - management_endpoints/mcp_management_endpoints.py - vector_store_endpoints/utils.py - pass_through_endpoints/pass_through_endpoints.py - auth/route_checks.py - litellm_pre_call_utils.py - spend_tracking/spend_management_endpoints.py - common_utils/http_parsing_utils.py - management_helpers/utils.py - health_endpoints/_health_endpoints.py Adds regression tests in tests/proxy_unit_tests/test_proxy_routes.py that construct a Request with scope["path"] set to a benign route and the Host header crafted so url.path would resolve differently; each site's decision is asserted against scope["path"]. * chore(proxy): make get_request_route imports lazy at call sites Move the ``from litellm.proxy.auth.auth_utils import get_request_route`` imports added in the prior commit back to the function bodies that use them. The module-level form participates in a long-standing import cycle through ``auth_utils -> _types -> ...`` and was flagged by CodeQL on the PR; the lazy form matches the pattern the proxy already uses for ``user_api_key_auth`` and related helpers elsewhere in these files. Also drop the ``RouteChecks._is_assistants_api_request`` delegation in ``_get_metadata_variable_name`` introduced in the prior commit — the delegation pulled ``RouteChecks`` into the same cycle, and the call site reuses the resolved route for its other branches, so inlining the substring check is both cycle-free and avoids a redundant second ``get_request_route`` call. Comment in test_proxy_routes.py acknowledges that the two MCP table entries exercise ``get_request_route`` directly rather than the full production handler (which needs ASGI scope + MCP state to invoke). --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: user <70670632+stuxf@users.noreply.github.com> * chore(ci): merge dev branch (BerriAI#28657) * feat(dashboard): navbar hierarchy + Agent Platform notifications (BerriAI#27543) * feat(dashboard): refine navbar zones and Agent Platform notice Restructure the admin navbar for production users: clear product vs community vs personal columns with vertical dividers, icon-only Slack/GitHub in a shared chip, and Docs/Blog typography aligned on an 8px rhythm. Add a notifications bell with popover linking to the LiteLLM Agent Platform repo and optional mark-as-read persistence. Promote the account control with initials avatar, single-line display name, and navDisplayName mapping for placeholder user ids (e.g. default_user_id). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(dashboard): address PR review — AntD buttons, public page guard, dedupe regex - Replace raw <button> with AntD Button in BlogDropdown, NotificationsBell, UserDropdown, and test mock - Guard NotificationsBell + container behind !isPublicPage to avoid rendering on public pages - Remove redundant equality checks in navDisplayName (regex already covers them) - Remove unused `lower` variable after simplification --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * fix(dashboard): drop dead useHealthReadiness import in navbar The module was removed in BerriAI#27896 (replaced by useHealthReadinessDetails), but the import survived the rebase. The symbol is unused — only useHealthReadinessDetails is consumed in the file. Removing the dead import unblocks the UI TypeScript build. * fix(dashboard): align CommunityEngagementButtons test with icon-only aria-labels The component was refactored to an icon-only chip with aria-label='LiteLLM on GitHub' (squash BerriAI#27543), but the test still asserted /star us on github/i. Update the query to match the rendered accessible name. * refactor(dashboard): drop unused props from NavbarProps The navbar refactor moved user identity + dark-mode state to internal hooks (useAuthorized, useWorker), but the NavbarProps interface still declared userID, userEmail, userRole, premiumUser, isDarkMode, and toggleDarkMode as required, forcing every caller to thread them through. Drop them from the interface and all four call sites (page.tsx, (dashboard)/layout.tsx, public_model_hub.tsx, navbar.test.tsx). Also shrinks the destructure in layout.tsx so the now-unused locals stop being pulled out of useAuthorized(). * refactor(dashboard): use useSyncExternalStore for NotificationsBell dismiss flag Reads/writes of the litellmHideAgentPlatformBanner key were done directly inside NotificationsBell via a useEffect + useState pair. Every other localStorage-backed flag in the dashboard (Disable ShowPrompts, DisableBouncingIcon, DisableShowNewBadge, DisableUsageIndicator, DisableBlogPosts) is wrapped in a useSyncExternalStore hook over localStorageUtils so all mounted components stay in sync. Extract useHideAgentPlatformBanner to follow the same shape, swap NotificationsBell to consume it, and add a regression test that two sibling bells stay in sync without a remount when one is dismissed. * refactor: mask credential fields in proxy settings GET responses (BerriAI#28682) * refactor: mask credential fields in proxy settings GET responses Brings SSO settings, cache settings, and the email/Slack alerting view in /get/config/callbacks in line with the HashiCorp Vault config-override pattern, so persisted credentials are not transported back to the UI in plaintext. * refactor: harden short-value masking and hoist alerting var constant Closes two review observations: - mask_sensitive_keys now replaces short values (below the visible prefix+suffix length) with an all-mask string instead of returning them unchanged, so a 1-7 character credential is no longer round-tripped verbatim. - _ALERTING_SENSITIVE_VARS is moved out of get_config() to a module-level constant, matching the analogous _SSO_SENSITIVE_FIELDS and _CACHE_SENSITIVE_FIELDS in the SSO and cache endpoint files. --------- Co-authored-by: Krrish Dholakia <krrish+github@berri.ai> Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ui): show 2-decimal precision for max_budget on key overview (BerriAI#28809) The Key Info Overview tab's Spend card truncated sub-dollar budgets to "$0" because formatNumberWithCommas defaults to 0 decimals. The Settings tab passes 2; align the overview so a $0.10 budget renders as "$0.10". Resolves LIT-2845 * feat(proxy): allow `llm_api_routes` virtual keys to list MCP servers (BerriAI#28442) * feat(proxy): allow llm_api_routes virtual keys to list MCP servers Add a new `mcp_discovery_routes` group (GET /v1/mcp/server and GET /v1/mcp/server/{server_id}) and include it in `llm_api_routes` so that virtual keys configured with `allowed_routes=["llm_api_routes"]` can discover the MCP servers they have access to. Previously these calls failed with 'Virtual key is not allowed to call this route. Only allowed to call routes: [llm_api_routes]'. The GET handlers already sanitize the response for restricted virtual keys via `_sanitize_mcp_server_list_for_virtual_key`, stripping credential-bearing fields (url, headers, env). Write methods (POST/PUT/DELETE) on the same paths remain gated by the existing handler-level admin role checks. The new discovery list is intentionally kept OUT of `mcp_inference_routes`, so `is_llm_api_route()` still returns False for these paths — this preserves the existing contract that DISABLE_LLM_API_ENDPOINTS must not block the Admin UI from listing MCP servers. Co-authored-by: ryan-crabbe-berri <ryan-crabbe-berri@users.noreply.github.com> * refactor(proxy): make MCP discovery carve-out method-aware Replace the `mcp_discovery_routes` group in `llm_api_routes` with a method-aware special case inside `is_virtual_key_allowed_to_call_route`. Virtual keys with allowed_routes=["llm_api_routes"] are now permitted to call only GET /v1/mcp/server and GET /v1/mcp/server/{server_id} — non-GET methods and multi-segment admin sub-paths fall through to the existing 403. This keeps the general llm_api_routes list free of management paths and avoids accidentally exposing POST/PUT/DELETE writes through the route-check layer. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: ryan-crabbe-berri <ryan-crabbe-berri@users.noreply.github.com> * chore(ci): merge dev branch (BerriAI#28807) * chore(proxy): route path-dependent call sites through get_request_route Replace direct ``request.url.path`` reads in auth, ACL, routing, and audit-log decisions with ``get_request_route(request)`` — the helper already added in ``auth/auth_utils.py`` that returns the ASGI ``scope["path"]`` with ``root_path`` stripped. Starlette reconstructs ``url.path`` from the Host header; ``scope["path"]`` is uvicorn's parse of the request line and matches what FastAPI dispatches on, so it's the authoritative route for any decision that should agree with the actual handler. Sites: - _experimental/mcp_server/auth/user_api_key_auth_mcp.py - management_endpoints/mcp_management_endpoints.py - vector_store_endpoints/utils.py - pass_through_endpoints/pass_through_endpoints.py - auth/route_checks.py - litellm_pre_call_utils.py - spend_tracking/spend_management_endpoints.py - common_utils/http_parsing_utils.py - management_helpers/utils.py - health_endpoints/_health_endpoints.py Adds regression tests in tests/proxy_unit_tests/test_proxy_routes.py that construct a Request with scope["path"] set to a benign route and the Host header crafted so url.path would resolve differently; each site's decision is asserted against scope["path"]. * chore(proxy): make get_request_route imports lazy at call sites Move the ``from litellm.proxy.auth.auth_utils import get_request_route`` imports added in the prior commit back to the function bodies that use them. The module-level form participates in a long-standing import cycle through ``auth_utils -> _types -> ...`` and was flagged by CodeQL on the PR; the lazy form matches the pattern the proxy already uses for ``user_api_key_auth`` and related helpers elsewhere in these files. Also drop the ``RouteChecks._is_assistants_api_request`` delegation in ``_get_metadata_variable_name`` introduced in the prior commit — the delegation pulled ``RouteChecks`` into the same cycle, and the call site reuses the resolved route for its other branches, so inlining the substring check is both cycle-free and avoids a redundant second ``get_request_route`` call. Comment in test_proxy_routes.py acknowledges that the two MCP table entries exercise ``get_request_route`` directly rather than the full production handler (which needs ASGI scope + MCP state to invoke). --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: user <70670632+stuxf@users.noreply.github.com> * fix(team): keep team_alias cache in sync on _cache_team_object writes (BerriAI#28737) * fix(team): keep team_alias cache in sync on _cache_team_object writes _cache_team_object wrote only to the team_id:<id> cache key, but the JWT auth path that uses team_alias_jwt_field reads from a separate team_alias:<alias> key (get_team_object_by_alias caches under both keys on miss, but reads only the alias-keyed one). After any team-mutation endpoint (team_model_add, team_model_delete, update_team, the two access-group writes) the team_id cache was refreshed but the team_alias cache stayed stale until TTL — JWT callers using team_alias_jwt_field kept seeing the pre-mutation team for the full cache window. Mirror the write under the alias key inside _cache_team_object so every existing caller stays in sync without further changes. Skip the alias write when team_alias is None/empty so we don't collide across alias-less teams. Surfaced testing the LIT-3244 cherry-pick on patch/1.86.0: the LIT-3244 fix correctly invalidated the team_id cache but the customer's JWT used team_alias_jwt_field, so they kept hitting the stale alias-keyed entry. * fix(team): delete (not overwrite) team_alias cache on _cache_team_object The prior shape of this PR wrote both team_id:<id> AND team_alias:<alias> from _cache_team_object. team_alias is NOT unique in the schema (no @unique on LiteLLM_TeamTable.team_alias), and get_team_object_by_alias enforces uniqueness on its own DB-fetch path (len(teams) > 1 raises). Writing the alias-keyed cache from the generic refresh path bypassed that check: a team admin renaming their team to collide with another team's alias could silently overwrite the cached team for JWT-by-alias auth, swapping the resolved team under that alias for the cache window. Switch the alias-keyed operation from a write to a delete (mirroring the dual-cache delete pattern in _delete_cache_key_object). After every team write, the next JWT-by-alias reader cache-misses and falls through to get_team_object_by_alias, which (a) re-fetches the fresh team from DB, closing the LIT-3244 staleness gap that motivated this PR, and (b) enforces alias uniqueness before populating either cache key. team_id:<id> writes are unchanged — team_id is the table PK and is guaranteed unique. Surfaced in veria-ai review on BerriAI#28739. * fix(managed-files): anchor model_id regex so it doesn't match llm_output_file_model_id extract_model_id_from_unified_id used `re.search(r"model_id,([^;]+)", ...)` which substring-matches the `model_id,` inside the file-ID encoding's `llm_output_file_model_id,<deployment_uuid>` field. parse_unified_id then fed that deployment UUID back into the auth path as a model candidate via _extract_models_from_managed_resource_id, and every team-BYOK file attach 403'd with: team not allowed to access model. This team can only access models=['openai/*']. Tried to access <deployment-uuid> The team's models list correctly contains the public name (`openai/*`) that target_model_names matches, but the bogus UUID candidate fails the wildcard check first. Anchor the regex to a field boundary (`(?:^|;)model_id,`) so it matches the legitimate top-level `model_id,<value>` field on vector_store unified IDs and skips substring matches inside other fields. File-IDs (which have no top-level `model_id` field) now return None and contribute no spurious UUID candidate. Surfaced reproducing LIT-3244 on patch/1.86.0 with the customer's exact flow: team with openai/* BYOK deployment, JWT-scoped user, POST /v1/vector_stores/{id}/files attaching a file uploaded with target_model_names=openai/gpt-4o. * fix(proxy): hydrate wildcard discovery credentials (BerriAI#28284) (BerriAI#28822) * fix(proxy): hydrate wildcard discovery credentials * fix(proxy): constrain wildcard credential hydration Co-authored-by: Dibyo Mukherjee <dibyo@adobe.com> * ci: add daily oss-agent-shin branch creation workflow (BerriAI#28829) Creates litellm_oss_agent_shin_MM_DD_YYYY from main every day at 00:00 UTC. Lets us retarget oss-agent-shin fork PRs onto a canonical branch so CircleCI runs with secrets, without granting the agent write access. Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com> * test(proxy): add harness for proxy_server.py behavior-pinning (BerriAI#28827) * test(proxy): add harness for proxy_server.py behavior-pinning Creates tests/test_litellm/proxy/proxy_server/ with: - conftest.py: 11 shared fixtures (app, client, mock_prisma, auth_as, mock_router with parametrized response builders, normalize, etc.) - _coverage_check.py: per-PR coverage gate (line + branch) against a baseline, self-selects target by inspecting which placeholder files have been filled - _pin_check.py: AST-based gate that verifies every pin-list item has >=1 happy + >=1 error test with a real assertion (no status-only) - test_harness_smoke.py: 19 smoke tests covering every fixture + both scripts end-to-end - 26 placeholder test files (one docstring each) reserved for follow-up PRs per the directory ownership in the Notion plan - .coverage_baseline pinned at 0% so future PRs measure deltas against new-tests-only and aren't entangled with the broader scattered test suite Adds a dedicated proxy-server job to test-unit-proxy-endpoints.yml so this directory's runtime + coverage are tracked independently. Plan: https://www.notion.so/36c43b8acdab81ee845fd5365128a2fc * ci(proxy-endpoints): allow workflow_dispatch Lets the workflow be triggered manually on a branch via `gh workflow run`, which is needed for the verify-first flow on workflow changes before opening a PR. * test(proxy): address review feedback on proxy_server harness - conftest.py: anchor sys.path insert to __file__ (Path(__file__).resolve().parents[4]) instead of CWD-relative os.path.abspath("../../../../") which resolved to the wrong directory when pytest is launched from the repo root. - _coverage_check.py: actually read .coverage_baseline and use it as the floor (line_min = max(target, baseline)). Closes the gap between the PR description's "delta semantics" and what the script was doing. With baseline=0.0 today this is a no-op; future PRs that update the baseline cause regressions (test deletions etc.) to trip the gate even if the static PR target is still met. - _pin_check.py: drop unreachable startswith("_") guard (test_*.py glob never yields underscore-prefixed names) and read each test file once instead of twice. * feat(openai): apply regional-processing cost uplift for EU/US data residency (BerriAI#28626) * feat(openai): apply regional-processing cost uplift for EU/US data residency OpenAI charges a 10% uplift on the latest GPT models when requests are served from a regionalized hostname (eu./us.api.openai.com). Infer the region from `api_base`, expose it on `kwargs["litellm_params"]["data_residency"]`, and multiply the computed cost by a per-model `regional_processing_uplift_multiplier_<region>` field. https://claude.ai/code/session_012ebH44s7ohYxjoix5CXzTW * test: allow regional_processing_uplift_multiplier_{eu,us} in model_prices schema * fix(cost): tighten data_residency inference and restore model_cost in tests - Only infer OpenAI data_residency when custom_llm_provider == "openai"; drop the implicit None fallback so non-OpenAI callers can't accidentally pick up a regional tag from a stray OpenAI hostname. - _local_model_cost_map fixture now snapshots and restores litellm.model_cost and LITELLM_LOCAL_MODEL_COST_MAP so tests don't leak state across the session. * refactor(openai): move data_residency helper under llms/openai * fix: thread data_residency through realtime stream cost calculation Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(cost): thread data_residency through batch_cost_calculator Apply the OpenAI regional-processing uplift multiplier to retrieve_batch cost paths so Batch API requests served via eu./us.api.openai.com are priced at the same uplifted token rates as completions/transcriptions. * refactor(openai): encapsulate provider check inside infer_openai_data_residency Move the custom_llm_provider == "openai" guard from get_litellm_params into the helper itself so the core utility no longer carries provider-specific dispatch logic. Callers pass through the provider unconditionally; the helper returns None for any non-OpenAI provider. * fix(responses): thread data_residency through Responses logging params The Responses API paths build their logging litellm_params dict after provider resolution but did not include data_residency, so cost calc saw None even when the effective api_base was a regional OpenAI host. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> --------- Co-authored-by: milan-berri <milan@berri.ai> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Mateo <mateo@Mateos-MacBook-Pro.local> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: Yassin Kortam <yassinkortam@Yassins-MacBook-Pro.local> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: user <70670632+stuxf@users.noreply.github.com> Co-authored-by: Krrish Dholakia <krrish+github@berri.ai> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> Co-authored-by: ryan-crabbe-berri <ryan-crabbe-berri@users.noreply.github.com> Co-authored-by: Dibyo Mukherjee <dibyo@adobe.com> Co-authored-by: ishaan-berri <155045088+ishaan-berri@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com> * fix: preserve OTEL response payload and remove duplicate constant - _emit_management_endpoint_otel_span now passes result as response on success - remove duplicate _CREDENTIAL_LITELLM_PARAM_FIELDS assignment in model_checks Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix: address bug detection findings - pass_through_endpoints: use request.method instead of hardcoded POST in streaming SigV4-signed request path for consistency with the non-streaming branch - llm_cost_calc/utils: hoist DataResidency value set to a module-level frozenset to avoid rebuilding it on every cost calculation - example_config_yaml/oai_misc_config: replace real-looking AWS account ID with placeholder 123456789012 in example bucket and role ARN Co-authored-by: Yassin Kortam <yassin@berri.ai> * chore(github_copilot): refresh model catalog from upstream /models API (BerriAI#28055) Aligns the github_copilot catalog with values returned by Copilot's public /models endpoint (capabilities.limits + capabilities.supports + model.supported_endpoints). - Adds 10 new model entries: claude-opus-4.7, claude-sonnet-4.6, gemini-3-flash-preview, gemini-3.1-pro-preview, gpt-4-0125-preview, gpt-5.2-codex, gpt-5.4, gpt-5.4-mini, gpt-5.5, oswe-vscode-prime. - Updates max_input_tokens for existing entries to reflect each model's true context window (e.g. gpt-4o-mini 64000 -> 128000, gpt-5-mini 128000 -> 264000, gpt-5.3-codex 128000 -> 400000, claude-haiku-4.5 128000 -> 200000). - Adds supports_reasoning, supports_response_schema, supports_function_calling, supports_parallel_function_calling, supports_vision based on capabilities.supports. - Declares supported_endpoints for entries missing it (e.g. gpt-3.5-turbo, gpt-4o, embeddings). - For responses-only models (gpt-5.2-codex, gpt-5.4, gpt-5.4-mini, gpt-5.5), sets mode to 'responses'. - gpt-41-copilot.mode changes from 'completion' to 'chat' because Copilot reports capabilities.type = 'chat'. Revertible on request. Pricing fields and other manually-curated values are preserved. * feat(datadog): emit litellm.overhead.latency as a standalone Datadog metric (BerriAI#28831) Adds a new `litellm.overhead.latency` gauge metric to `DatadogMetricsLogger` (the `/api/v2/series` path). The value is sourced from `hidden_params["litellm_overhead_time_ms"]` already computed in `ResponseMetadata` and exposed in `StandardLoggingPayload`. Matches the Prometheus integration which exposes the same value via `litellm_overhead_latency_metric`. Emitted in seconds (ms ÷ 1000) for consistency with the other latency series. Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Shin <shin@litellm.ai> Co-authored-by: ishaan-berri <155045088+ishaan-berri@users.noreply.github.com> * feat(arize): route Phoenix traces via per-project TracerProviders (BerriAI#28876) Use LRU-cached TracerProviders with project-scoped OTEL Resources so team/key metadata routes traces correctly. On the proxy, project selection is limited to server-controlled user_api_key_auth_metadata; client metadata fields stay banned. * fix(arize_phoenix): skip _emit_semantic_logs on failure path Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(arize_phoenix): skip raw request logging and metrics on failure path Restores pre-refactor behavior: _handle_failure no longer emits raw-request sub-spans or records OTEL metrics, matching the original _handle_failure that did not call these helpers. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(security): close two medium telemetry trust-boundary issues Issue 1 (arize_phoenix.py — caller-controlled telemetry routing): - _is_proxy_request no longer detects proxy mode by checking user_api_key_auth_metadata in request metadata. That field is user-supplied, so an authenticated caller could fake proxy-mode detection and have _project_from_metadata_dict read their own dict for project selection, routing telemetry to arbitrary Arize/Phoenix projects. Proxy mode is now determined solely by the server-set proxy_server_request field in litellm_params. - auth_utils.py adds user_api_key_auth_metadata to the banned request body params list so the proxy rejects any attempt to supply the field at the HTTP layer. The field is server-reserved: it is written exclusively by add_user_api_key_auth_to_request_metadata from the authenticated key's database record after the ban check runs. Issue 2 (management_helpers/utils.py — API key in OTEL span): - _emit_management_endpoint_otel_span stripped plaintext credential fields (key, token, api_key, secret, …) from the response dict before passing it to the OTEL success hook. dict(result) on a Pydantic GenerateKeyResponse includes the freshly-generated key field, which would previously be written as a span attribute to every configured OTEL collector/backend. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: Terrajlz <info@jouleselectrictech.com> Co-authored-by: Bruno Devaux <devaux.br@gmail.com> Co-authored-by: milan-berri <milan@berri.ai> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Mateo <mateo@Mateos-MacBook-Pro.local> Co-authored-by: Yassin Kortam <yassinkortam@Yassins-MacBook-Pro.local> Co-authored-by: user <70670632+stuxf@users.noreply.github.com> Co-authored-by: Krrish Dholakia <krrish+github@berri.ai> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> Co-authored-by: ryan-crabbe-berri <ryan-crabbe-berri@users.noreply.github.com> Co-authored-by: Dibyo Mukherjee <dibyo@adobe.com> Co-authored-by: ishaan-berri <155045088+ishaan-berri@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com> Co-authored-by: rinto <54238243+ririnto@users.noreply.github.com> Co-authored-by: Shin <shin@litellm.ai> Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
* Fix incorrect agent API request example payload structure (BerriAI#29556) * fix(otel): add litellm_metadata fallback in _get_span_context and _end_proxy_span_from_kwargs (BerriAI#29427) * fix(otel): add litellm_metadata fallback in _get_span_context and _end_proxy_span_from_kwargs On /v1/messages and other LITELLM_METADATA_ROUTES, the parent OTel span is stored in litellm_params['litellm_metadata'] instead of litellm_params['metadata']. When the request body contains a native 'metadata' field (e.g. Anthropic's {"user_id": "..."}), litellm_params['metadata'] gets overwritten and the parent span is lost, producing orphan root spans with a different trace_id. Add fallback checks to litellm_metadata in: - _get_span_context(): so child spans find the correct parent - _end_proxy_span_from_kwargs(): so the proxy span gets closed Fixes: BerriAI#27934 * test(otel): tighten assertions per Greptile review - test_span_context_metadata_takes_priority: assert litellm_metadata span is never accessed, proving metadata takes priority - test_span_context_no_parent_when_neither_has_span: assert both ctx and detected_span are None --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Aneesh-Fiddler <aneeshfiddler@gmail.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix: remove premature end-user budget check from get_end_user_object (BerriAI#29420) * fix(proxy): remove premature end-user budget check from get_end_user_object Problem: - `_check_end_user_budget()` was called inside `get_end_user_object()` - This caused budget checks to run BEFORE `skip_budget_checks` could be evaluated - Zero-cost models (e.g., local vLLM) were incorrectly blocked when end-users exceeded their budget, even though they should bypass budget checks Solution: - Remove `_check_end_user_budget()` calls from `get_end_user_object()` - Budget enforcement now happens exclusively in `common_checks()` where `skip_budget_checks` context is available - `get_end_user_object()` keeps `route` as optional in function parameter for backwards compatibility and future implementation. * refactor(tests): update budget enforcement tests to reflect changes in get_end_user_object - test_get_end_user_object() verifies data fetching - test_check_end_user_budget() verifies enforcement - test_budget_enforcement_blocks_over_budget_users() integrates _check_end_user_budget() - test_resolve_end_user_reraises_budget_exceeded() is now test_resolve_end_user since no budget exceeded is thrown in get_end_user_object() * Gemini /images/generate and /images/edits billing fixes + add support for size and aspect ratio params (BerriAI#29534) * Fix Gemini image config mapping * Address Gemini image config review * Format Gemini image generation transform * Fix Gemini image token usage logging * Share Gemini image request helpers * Fix Gemini Imagen model routing * Fixes as per self code review * Fixes per internal code review * Stop gating Imagen imageSize forwarding * Document Gemini image size mapping source * chore: retrigger lint * Clarify Gemini candidate count precedence * Add Inception provider (BerriAI#29522) * add inception as provider (chat, fim) * linting * seperate test suite for chat and fim * fix test coverage * fix: model hub custom pricing model info (BerriAI#29293) * Opik user auth key metadata extractors (BerriAI#28397) * fix: enhance Opik metadata extraction to include user API key auth context fixed after refactoring to extractor logic * test: add unit tests for OPik metadata extraction logic * fix: enhance extract_opik_metadata function to prioritize metadata sources for improved accuracy * fix(ci): clarified comments and edited unit tests * test: add unit tests for OPik metadata extraction with auth and requester overrides * fix(ui): replace fixed favicon.ico with current api get /get_favicon (BerriAI#29532) Signed-off-by: José Luis Di Biase <josx@interorganic.com.ar> * fix(vertex/gemini): keep tool_call reference when a text-only assistant message follows (BerriAI#29561) `_gemini_convert_messages_with_history` tracks `last_message_with_tool_calls` so a following tool result can be matched back to its tool call. The assignment was inside a branch guarded by `assistant_msg.get("tool_calls", []) is not None`, which is also True for a text-only assistant message (an empty list is not None). As a result, an assistant message with no tool calls that appears between a tool call and its tool result overwrote the reference, and conversion failed with: Exception: Missing corresponding tool call for tool response message. This shape is common: a model emits a short narration/assistant message after a tool call before the tool result is appended. Only update `last_message_with_tool_calls` when the assistant message actually carries tool_calls (or a function_call). Adds a regression test. Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * Add 1-hour cache write pricing for EU/AU/JP Bedrock Anthropic models (BerriAI#28572) * fix(thinking): handle None thinking param in is_thinking_enabled (BerriAI#28598) Squash-merged by litellm-agent from Terrajlz's PR. * feat(helm): support tpl rendering in podAnnotations (BerriAI#28609) Squash-merged by litellm-agent from devauxbr's PR. * Forward custom_llm_provider through the Responses API bridge (Fixes BerriAI#28505) (BerriAI#28575) * Forward custom_llm_provider through the Responses API bridge (Fixes BerriAI#28505) When a Chat Completions request to a GPT-5.4+ model contains both `tools` and `reasoning_effort`, `completion()` auto-routes through `responses_api_bridge`. The bridge handler called `litellm.responses()` / `litellm.aresponses()` without forwarding the already-resolved `custom_llm_provider`, so the downstream call re-invoked `get_llm_provider()` with `custom_llm_provider=None` and stripped a second provider prefix from a `provider/provider/model` deployment string. For a deployment configured as `openai/openai/openai/gpt-5.5`, the bridge flow sent `openai/gpt-5.5` to the upstream API instead of the correct `openai/openai/gpt-5.5`. Upstream APIs that enforce model-name allow-lists rejected this as `key_model_access_denied`. Fix: pass the locally-resolved `custom_llm_provider` into both the sync `responses()` and async `aresponses()` calls so the downstream `_resolve_model_provider_for_responses` sees an explicit provider and skips the second prefix-strip. New regression test `tests/test_litellm/completion_extras/test_responses_bridge_provider_propagation.py` pins both call sites: each must forward `custom_llm_provider`. * fix(28505): set custom_llm_provider on request_data instead of as duplicate kwarg Greptile flagged that the previous patch passed custom_llm_provider as an explicit kwarg to responses()/aresponses() while request_data already carried it via the spread of sanitized_litellm_params, which would raise TypeError: got multiple values for keyword argument on every real bridge call. Switches to assigning request_data['custom_llm_provider'] before the call so the resolved provider wins over whatever sanitized_litellm_params spread in, without duplicating the kwarg. Updates the regression test to seed request_data with a sentinel custom_llm_provider so it actually exercises the overwrite path (the previous test mocked transform_request with a minimal dict and never hit the conflict). * chore: trigger shin-agent re-eval on retargeted staging base * chore: trigger shin-agent re-eval against updated Greptile state * Add 1-hour cache write pricing for EU/AU/JP Bedrock Anthropic models The 1-hour prompt-cache write tier (`cache_creation_input_token_cost_above_1hr`) was added to the us./global. variants of the Claude 4.5/4.6/4.7 family on Bedrock, but the eu./au./jp. cross-region inference profiles were left without it. AWS Bedrock pricing applies the same +10% regional premium across all geo profiles, so eu./au./jp. should carry the same 1-hour rates as us. (1.6x the 5-minute regional rate). Without these fields, cost tracking on EU/AU/JP Bedrock 1-hour-TTL prompt caching falls back to the 5-minute write rate and undercounts spend by ~60% for European, Australian, and Japanese tenants. Adds the 1-hour tier (and Sonnet 4.5's long-context >200K tier where AWS publishes one) to 14 regional Bedrock entries in both `model_prices_and_context_window.json` and the bundled `model_prices_and_context_window_backup.json`: - eu./au. Opus 4.6 ($11.00 / MTok) - eu./au. Opus 4.7 ($11.00 / MTok) - eu./au./jp. Sonnet 4.6 ($6.60 / MTok) - eu./au./jp. Sonnet 4.5 ($6.60 / MTok regular, $13.20 / MTok LC) - eu./au./jp. Haiku 4.5 ($2.20 / MTok) Also extends `tests/test_litellm/test_bedrock_anthropic_1hr_cache_pricing.py` with a `REGIONAL_EXPECTED` parametrized block covering all 13 new entries plus the existing 1.6x ratio invariant. Note: `eu.anthropic.claude-opus-4-5-20251101-v1:0` carries the wrong 5m rate today (base 6.25e-06 instead of regional 6.875e-06), which would break the 1.6x ratio check. It is intentionally left out of this PR so the scope stays "1-hour cache tier addition" — a separate follow-up should correct the EU 5m rates for Opus 4.5. --------- Co-authored-by: Terrajlz <info@jouleselectrictech.com> Co-authored-by: Bruno Devaux <devaux.br@gmail.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> * Add 1-hour cache write pricing tier for Vertex AI Anthropic models (BerriAI#28569) * fix(thinking): handle None thinking param in is_thinking_enabled (BerriAI#28598) Squash-merged by litellm-agent from Terrajlz's PR. * feat(helm): support tpl rendering in podAnnotations (BerriAI#28609) Squash-merged by litellm-agent from devauxbr's PR. * Forward custom_llm_provider through the Responses API bridge (Fixes BerriAI#28505) (BerriAI#28575) * Forward custom_llm_provider through the Responses API bridge (Fixes BerriAI#28505) When a Chat Completions request to a GPT-5.4+ model contains both `tools` and `reasoning_effort`, `completion()` auto-routes through `responses_api_bridge`. The bridge handler called `litellm.responses()` / `litellm.aresponses()` without forwarding the already-resolved `custom_llm_provider`, so the downstream call re-invoked `get_llm_provider()` with `custom_llm_provider=None` and stripped a second provider prefix from a `provider/provider/model` deployment string. For a deployment configured as `openai/openai/openai/gpt-5.5`, the bridge flow sent `openai/gpt-5.5` to the upstream API instead of the correct `openai/openai/gpt-5.5`. Upstream APIs that enforce model-name allow-lists rejected this as `key_model_access_denied`. Fix: pass the locally-resolved `custom_llm_provider` into both the sync `responses()` and async `aresponses()` calls so the downstream `_resolve_model_provider_for_responses` sees an explicit provider and skips the second prefix-strip. New regression test `tests/test_litellm/completion_extras/test_responses_bridge_provider_propagation.py` pins both call sites: each must forward `custom_llm_provider`. * fix(28505): set custom_llm_provider on request_data instead of as duplicate kwarg Greptile flagged that the previous patch passed custom_llm_provider as an explicit kwarg to responses()/aresponses() while request_data already carried it via the spread of sanitized_litellm_params, which would raise TypeError: got multiple values for keyword argument on every real bridge call. Switches to assigning request_data['custom_llm_provider'] before the call so the resolved provider wins over whatever sanitized_litellm_params spread in, without duplicating the kwarg. Updates the regression test to seed request_data with a sentinel custom_llm_provider so it actually exercises the overwrite path (the previous test mocked transform_request with a minimal dict and never hit the conflict). * chore: trigger shin-agent re-eval on retargeted staging base * chore: trigger shin-agent re-eval against updated Greptile state * Add 1-hour cache write pricing tier for Vertex AI Anthropic models GCP Vertex AI publishes a separate 1-hour cache write column for the Claude family (1.6x the 5-minute write rate, matching the documented Bedrock ratio). LiteLLM's Vertex AI Anthropic entries only carry the 5-minute tier, so any request that uses `cache_control: {"ttl": "1h"}` on Vertex AI Claude is undercounted in cost tracking by ~60%. The runtime side already supports the 1-hour tier — `VertexAIAnthropicConfig` extends `AnthropicConfig`, populating `ephemeral_1h_input_tokens`, and `_calculate_cache_creation_cost` reads `cache_creation_input_token_cost_above_1hr`. Only the price registry was missing data. Adds the field to 19 vertex_ai/claude-* entries across both `model_prices_and_context_window.json` and the bundled `model_prices_and_context_window_backup.json`: - Haiku 4.5 ($1.25 -> $2.00 / MTok) - Sonnet 3.7 / 4 / 4.5 / 4.6 ($3.75 -> $6.00 / MTok) - Opus 4.5 / 4.6 / 4.7 ($6.25 -> $10.00 / MTok) - Opus 4 / 4.1 ($18.75 -> $30.00 / MTok) Adds `tests/test_litellm/test_vertex_anthropic_1hr_cache_pricing.py` mirroring the Bedrock equivalent — pins each (5m, 1h) pair per model and asserts the 1.6x ratio across the family. Fixes BerriAI#27781. --------- Co-authored-by: Terrajlz <info@jouleselectrictech.com> Co-authored-by: Bruno Devaux <devaux.br@gmail.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> * Fix Gemini multimodal function responses (BerriAI#29325) Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * address greptile review: add _transform_image_usage method and model-map supports_image_size flag - Add _transform_image_usage instance method to GoogleImageGenConfig that delegates to transform_gemini_image_usage, fixing the regression test - Replace hardcoded "2.5-flash" string check in supports_gemini_image_size with a get_model_info lookup on supports_image_size (default true) - Add supports_image_size: false to all gemini-2.5-flash model entries in model_prices_and_context_window.json so capability is controlled via the model map rather than embedded in code * fix test failures: schema validation, mypy type, model info plumbing, pricing test - Add supports_image_size to ModelInfoBase TypedDict so get_model_info surfaces it - Pass supports_image_size through _get_model_info_helper constructor call - Fix supports_gemini_image_size to use value is not False (None means unset, defaults to True) - Add supports_image_size to JSON schema in test_aaamodel_prices_and_context_window_json_is_valid - Correct gemini-3.1-flash-lite pricing assertions in test to match JSON values * Add Azure AI Kimi K2.6 metadata (BerriAI#27052) * Add Azure AI Kimi K2.6 metadata * Scope Kimi metadata test cost map setup * fall back to substring check for models not in model_prices_and_context_window.json Models like gemini-2.5-flash-image-preview are not in the pricing JSON, so get_model_info raises. Fall back to "2.5-flash" not in model when the JSON has no explicit supports_image_size entry for the model. * fix(inception): don't forward global litellm.api_key to Inception FIM Match the Inception chat config: resolve only an Inception-specific key (param, litellm.inception_key, or INCEPTION_API_KEY) for the text-completion FIM path. The global litellm.api_key (often an OpenAI key) was both leaking to api.inceptionlabs.ai and taking precedence over the configured Inception key when set. * fix(auth): enforce end-user budget on custom-auth path that skips common_checks get_end_user_object() no longer raises BudgetExceededError, so custom-auth deployments with custom_auth_run_common_checks unset (which skip the centralized common_checks gate) stopped enforcing the end-user budget, letting an over-budget end user keep making requests. Re-enforce the budget in _run_post_custom_auth_checks on that path. --------- Signed-off-by: José Luis Di Biase <josx@interorganic.com.ar> Co-authored-by: Isha <72744901+IshaMeera@users.noreply.github.com> Co-authored-by: aneeshsangvikar <aneeshsangvikar@fiddler.ai> Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Aneesh-Fiddler <aneeshfiddler@gmail.com> Co-authored-by: Suleiman Elkhoury <108065141+suleimanelkhoury@users.noreply.github.com> Co-authored-by: Dmitriy Alergant <93501479+DmitriyAlergant@users.noreply.github.com> Co-authored-by: Yanis Miraoui <yanis.miraoui19@imperial.ac.uk> Co-authored-by: Lovro Seder <vrovro@gmail.com> Co-authored-by: Thomas Mildner <12685945+Thomas-Mildner@users.noreply.github.com> Co-authored-by: José Luis Di Biase <josx@interorganic.com.ar> Co-authored-by: Lai Quang Huy <64073540+1qh@users.noreply.github.com> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Terrajlz <info@jouleselectrictech.com> Co-authored-by: Bruno Devaux <devaux.br@gmail.com> Co-authored-by: ZHONG Ziwen <67355585+zzw-math@users.noreply.github.com> Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* Mark xAI models retiring on 2026-05-15 (BerriAI#28788) Per https://docs.x.ai/developers/migration/may-15-retirement, xAI is retiring the following slugs on 2026-05-15 (auto-redirect to grok-4.3 with various reasoning efforts; callers continuing to use the old slugs will be billed at grok-4.3 pricing): grok-4-1-fast-reasoning{,-latest} -> grok-4.3 (low effort) grok-4-1-fast-non-reasoning{,-latest} -> grok-4.3 (none) grok-4-fast-reasoning -> grok-4.3 (low effort) grok-4-fast-non-reasoning -> grok-4.3 (none) grok-4-0709 -> grok-4.3 (low effort) grok-code-fast-1{,-0825} -> grok-build-0.1 grok-3 -> grok-4.3 (none) Only the direct xai/ slugs are tagged; third-party hosts (azure_ai, oci, vercel_ai_gateway, perplexity/xai) run their own schedules. The grok-3 retirement list explicitly names only the base grok-3 slug — the -mini / -fast / -beta / -latest variants are not listed, so they remain untouched. * feat(moonshot): advertise json_schema response support on live models (BerriAI#29683) litellm.responses() already routes Moonshot through the responses->chat-completions bridge, and Moonshot honors response_format json_schema on chat completions. The cost-map entries left supports_response_schema unset, so discovery layers that gate on that flag dropped Moonshot from structured-output / responses listings even though the capability works end to end. Set supports_response_schema on the nine models currently live on api.moonshot.ai: kimi-k2.5, kimi-k2.6, the moonshot-v1 8k/32k/128k text and vision-preview variants, and moonshot-v1-auto. Verified against the live API that each honors json_schema and that litellm.responses() returns schema-valid structured output through the bridge. * chore(moonshot): mark models retired from api.moonshot.ai as deprecated (BerriAI#29685) Thirteen Moonshot/Kimi models in the cost map no longer resolve on api.moonshot.ai (all return 404). Stamp each with its deprecation_date from platform.kimi.ai/docs/models rather than deleting the entries, so historical cost calculation keeps resolving the names while tooling can surface the retirement. Dates: kimi-thinking-preview 2025-11-11; kimi-latest and its 8k/32k/128k context variants 2026-01-28; the kimi-k2 preview/turbo/thinking series 2026-05-25; the moonshot-v1 -0430 snapshots use their own 2024-04-30 snapshot date (Moonshot publishes no discontinuation date for them). * fix(moonshot): drop temperature for reasoning models (kimi-k2.5/k2.6) (BerriAI#29687) Kimi reasoning models reject every temperature except 1; a request with temperature=0.2 returns "invalid temperature: only 1 is allowed for this model". litellm only clamped temperature into [0.3, 1], so any value below 1 still 400'd. Drop the temperature param entirely for reasoning models (gated on supports_reasoning, the same signal transform_request already uses) so the model default is used; the non-reasoning moonshot-v1 models keep the existing clamp. Co-authored-by: Sameer Kankute <sameer@berri.ai> * feat(mcp): add per-server timeout configuration (BerriAI#29672) * feat(mcp): add per-server timeout configuration * fix(mcp): address timeout field review comments - use is not None guard instead of or for 0.0 edge case - copy timeout in both LiteLLM_MCPServerTable constructions (health check path + _build_mcp_server_table) - add timeout Float? column to all three schema.prisma files - extend round-trip test to cover _build_mcp_server_table direction - add test for zero timeout not treated as falsy * fix(mcp): forward timeout in _build_temporary_mcp_server_record * fix(mcp): return 504 instead of 500 when per-server timeout fires * test(mcp): add 504 timeout regression test; fix black formatting * Add jp. Bedrock cross-region inference profile for claude-opus-4-7 (BerriAI#28567) * fix(thinking): handle None thinking param in is_thinking_enabled (BerriAI#28598) Squash-merged by litellm-agent from Terrajlz's PR. * feat(helm): support tpl rendering in podAnnotations (BerriAI#28609) Squash-merged by litellm-agent from devauxbr's PR. * Forward custom_llm_provider through the Responses API bridge (Fixes BerriAI#28505) (BerriAI#28575) * Forward custom_llm_provider through the Responses API bridge (Fixes BerriAI#28505) When a Chat Completions request to a GPT-5.4+ model contains both `tools` and `reasoning_effort`, `completion()` auto-routes through `responses_api_bridge`. The bridge handler called `litellm.responses()` / `litellm.aresponses()` without forwarding the already-resolved `custom_llm_provider`, so the downstream call re-invoked `get_llm_provider()` with `custom_llm_provider=None` and stripped a second provider prefix from a `provider/provider/model` deployment string. For a deployment configured as `openai/openai/openai/gpt-5.5`, the bridge flow sent `openai/gpt-5.5` to the upstream API instead of the correct `openai/openai/gpt-5.5`. Upstream APIs that enforce model-name allow-lists rejected this as `key_model_access_denied`. Fix: pass the locally-resolved `custom_llm_provider` into both the sync `responses()` and async `aresponses()` calls so the downstream `_resolve_model_provider_for_responses` sees an explicit provider and skips the second prefix-strip. New regression test `tests/test_litellm/completion_extras/test_responses_bridge_provider_propagation.py` pins both call sites: each must forward `custom_llm_provider`. * fix(28505): set custom_llm_provider on request_data instead of as duplicate kwarg Greptile flagged that the previous patch passed custom_llm_provider as an explicit kwarg to responses()/aresponses() while request_data already carried it via the spread of sanitized_litellm_params, which would raise TypeError: got multiple values for keyword argument on every real bridge call. Switches to assigning request_data['custom_llm_provider'] before the call so the resolved provider wins over whatever sanitized_litellm_params spread in, without duplicating the kwarg. Updates the regression test to seed request_data with a sentinel custom_llm_provider so it actually exercises the overwrite path (the previous test mocked transform_request with a minimal dict and never hit the conflict). * chore: trigger shin-agent re-eval on retargeted staging base * chore: trigger shin-agent re-eval against updated Greptile state * Add jp. Bedrock cross-region inference profile for claude-opus-4-7 AWS Bedrock documents jp.anthropic.claude-opus-4-7 alongside the existing us./eu./au./global. profiles for Claude Opus 4.7 (ap-northeast-1 Tokyo / ap-northeast-3 Osaka), but the entry is missing from model_prices_and_context_window.json. Tokyo-region users currently get an "unknown model" error when routing through the JP geo profile. Adds the entry to both the canonical file and the bundled backup, mirroring the recent pattern for sonnet-4-6 (BerriAI#27831). Pricing matches the other regional profiles (10% premium over base/global). Regression test pins all six documented profiles (base, global, us, eu, au, jp) and asserts pricing parity between jp. and au. variants. Source: https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-anthropic-claude-opus-4-7.html --------- Co-authored-by: Terrajlz <info@jouleselectrictech.com> Co-authored-by: Bruno Devaux <devaux.br@gmail.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> * feat(soniox): add soniox audio transcription integration (BerriAI#29508) * feat(openmeter): add OPENMETER_TRUST_REQUEST_USER to prevent forged attribution (BerriAI#29650) The OpenMeter callback resolves the CloudEvent subject from kwargs["user"] first, then falls back to the key-bound user_api_key_user_id. For multi-tenant proxy deployments, a client can set `"user": "..."` in the request body and cause their usage to be attributed to that arbitrary string — a billing-attribution forgery risk. Adds OPENMETER_TRUST_REQUEST_USER env var (default "true" for backward compatibility). When set to "false", the request-supplied `user` field is ignored and the subject is resolved solely from user_api_key_user_id. Matches the existing env-var-driven config pattern in this file (OPENMETER_API_KEY, OPENMETER_API_ENDPOINT, OPENMETER_EVENT_TYPE). * feat(search): add you_com as a search provider (BerriAI#28370) * feat(search): add you_com as a search provider Registers You.com Search API as a first-class `search_provider` in the `search_tools` registry, alongside Tavily, Exa, Perplexity, etc. - New adapter: litellm/llms/you_com/search/transformation.py - POSTs to https://ydc-index.io/v1/search - Auth: X-API-Key from YOUCOM_API_KEY (or explicit api_key) - Maps Perplexity unified spec: max_results -> count, search_domain_filter -> include_domains, country -> country - Flattens results.web + results.news into a single SearchResult list; snippet prefers snippets[0], falls back to description; page_age -> date - Registry: SearchProviders.YOU_COM in litellm/types/utils.py and wired into ProviderConfigManager.get_provider_search_config() - Pricing entry: model_prices_and_context_window.json (placeholder $0.0; happy to adjust to maintainers' preferred public number) - Docs: example router config snippet and example proxy yaml updated - Tests: tests/search_tests/test_you_com_search.py - 5 mocked tests (payload shape, domain filter mapping, snippet fallback, news flattening, missing-api-key error) Refs upstream expansion signal: BerriAI#15942 * review fixups: normalize api_base, lowercase country, scope env-var to test Addresses Greptile inline review comments on BerriAI#28370: - get_complete_url: strip trailing slashes from api_base *before* the endswith("/v1/search") check, so a custom base like ".../v1/search/" doesn't become ".../v1/search/v1/search". - transform_search_request: .lower() country before sending, matching Tavily's convention so callers using the unified spec form ("US") get consistent behavior across providers. - Tests: replace direct os.environ writes with an autouse monkeypatch fixture so YOUCOM_API_KEY is set per-test and removed afterwards. The missing-key test now uses monkeypatch.delenv. New test asserts the trailing-slash normalization above. Reverts the ARCHITECTURE.md / example yaml edits per the reviewer note that documentation changes belong in the litellm-docs repo. * support keyless free tier (api.you.com/v1/agents/search) as default You.com offers an IP-throttled keyless endpoint that returns the same response shape as the keyed one (~100 queries/day, no signup). This is a significant onboarding lever - mirrors the keyless DuckDuckGo/SearXNG providers already in the search_tools registry. Behavior: - YOUCOM_API_KEY set -> keyed: POST https://ydc-index.io/v1/search (X-API-Key header) - no key -> free: POST https://api.you.com/v1/agents/search (no auth) - YOUCOM_API_BASE override -> honored as-is Tests: - New: test_you_com_search_keyless_free_tier - asserts URL + absence of X-API-Key when no key is configured. - New: test_you_com_search_validate_environment_keyless - asserts the config no longer raises when the key is absent. - Removed: test_you_com_search_raises_without_api_key (the precondition no longer holds). - Existing payload/domain-filter/etc tests still cover keyed mode via the autouse YOUCOM_API_KEY fixture. Verified both endpoints accept POST + return identical JSON shape: results.web[] / results.news[] with title, url, snippets, description, page_age. * register you_com in provider_endpoints_support.json Adding `litellm/llms/you_com/` requires a corresponding entry in provider_endpoints_support.json or the code-quality/check_provider_folders_documented CI check fails. Follows the compact tavily/serper pattern - endpoints: { search: true }. Local run of the check now reports "All 114 provider folders are documented". * move tests under tests/test_litellm/llms/ so CI exercises them The litellm CI workflows scope unit tests to `tests/test_litellm/...` (see test-unit-llm-providers.yml: `tests/test_litellm/llms` path), so tests living under `tests/search_tests/` are never run in CI - which is why codecov reports 0% patch coverage for the new adapter even though the unit tests exist and pass locally. Move test_you_com_search.py into `tests/test_litellm/llms/you_com/` so the test-unit-llm-providers job picks it up. 7/7 tests still pass at the new location. (Sibling search-only providers - tavily, exa_ai, brave, etc. - still live only in `tests/search_tests/` and would benefit from the same move, but that is out of scope for this PR.) * fix(you_com): pin Accept-Encoding: identity to dodge keyless gzip bug The keyless free-tier endpoint (api.you.com/v1/agents/search) advertises Content-Encoding: gzip but returns a body that httpx's decoder rejects with `zlib.error: Error -3 while decompressing data: incorrect header check`, surfacing as litellm.APIConnectionError in user code. curl works because it doesn't request compression by default. Pin Accept-Encoding: identity in validate_environment so the upstream server skips compression entirely. Harmless on the keyed endpoint (ydc-index.io/v1/search) which negotiates content-encoding correctly. The header uses setdefault so a caller-supplied Accept-Encoding still takes precedence. (Server-side bug has been flagged to the You.com team separately - once fixed there, this workaround can be removed.) New unit test: test_you_com_search_pins_identity_accept_encoding. --------- Co-authored-by: Sameer Kankute <sameer@berri.ai> * docs: fix README typo (BerriAI#29419) Correct clear spelling mistakes in documentation without changing behavior. Confidence: high Scope-risk: narrow Tested: git diff --check; uvx codespell on changed files Not-tested: Full docs build not run; text-only changes * Fix(langfuse): pass httpx_client to Langfuse in langfuse_prompt_management to respect SSL_VERIFY (BerriAI#29480) * fix(langfuse): pass ssl_verify to Langfuse httpx client * fix_langfuse_ * add unit tests * addressed comments --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * feat(models): add minimax/MiniMax-M3 to model cost map (BerriAI#29412) Add MiniMax's new flagship MiniMax-M3 to the native minimax provider: 512K context, 128K max output, native multimodal (supports_vision), reasoning, prompt caching. Pricing (USD/M tokens): input 0.6 / output 2.4 / cache read 0.12. M3 has no active prompt-cache-write tier, so cache_creation_input_token_cost is omitted. Updated both the root model_prices_and_context_window.json (remote source) and the bundled litellm/model_prices_and_context_window_backup.json (local fallback), keeping them in sync. * fix(logging): handle ResponseCompletedEvent in anthropic_messages streaming spend log (BerriAI#29394) * fix(logging): handle ResponseCompletedEvent in anthropic_messages streaming spend log * fix(logging): extend terminal event handling to ResponseIncompleteEvent and ResponseFailedEvent; fix return type annotation * feat(provider): Add Neosantara provider as OpenAI Compatible (BerriAI#29646) * Add Neosantara provider * Register Neosantara provider enum * Address Neosantara provider review feedback * Add Neosantara packaged endpoint support --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * fix: address greptile and veria review feedback - langfuse: guard httpx_client injection behind version check (>= 2.7.3) - soniox: propagate audio_transcription_duration in _hidden_params for spend tracking - soniox: give SONIOX_API_BASE env var priority over caller-supplied api_base - mcp: replace CancelledError catch with asyncio.wait_for + TimeoutError * chore(mcp): add migration for per-server timeout column * fix(test): add tool_use_system_prompt_tokens to model prices schema validator * fix: mcp timeout test uses real asyncio.wait_for timeout; you_com get_complete_url respects resolved api_key * fix: forward resolved api_key into you_com endpoint selection and apply timeout to soniox polling GETs The search flow resolves api_key in validate_environment but never passed it into get_complete_url, so a programmatic api_key (with no YOUCOM_API_KEY in the env) set the X-API-Key header yet still selected the keyless free-tier endpoint. Forward api_key through both the search entrypoint and the http handler so the keyed endpoint is chosen. HTTPHandler.get/AsyncHTTPHandler.get had no timeout parameter, so the Soniox poll and transcript-fetch GETs silently used the client global default instead of the caller timeout. Add a per-request timeout to get() and forward the configured timeout from the Soniox handler. * fix(soniox): price stt-async-v4 per second so transcriptions are billed The handler stores audio_transcription_duration in _hidden_params, but the model carried only token cost fields and the response has no token usage, so the transcription cost path fell through to cost_per_second and returned $0. An authenticated caller could transcribe Soniox audio without decrementing their budget. Switch the entry to output_cost_per_second at Soniox's published $0.10/hour async rate so the stored duration produces a real charge. * fix(langfuse): use a dedicated httpx client for the SDK injection The httpx_client handed to the Langfuse SDK came from _get_httpx_client(), which returns LiteLLM's globally cached HTTPHandler. If Langfuse closed that client on teardown it would invalidate the shared client used by every other LiteLLM HTTP call. Build a dedicated httpx.Client instead, still resolving SSL verification and client certificate from LiteLLM's configuration. * fix(soniox): prefer caller-supplied api_base over SONIOX_API_BASE env var * fix(cohere): support max_completion_tokens on cohere v2 chat (default route) (BerriAI#29779) * fix(cohere): support max_completion_tokens on cohere v2 chat The default cohere_chat route resolves to CohereV2ChatConfig, which did not list or map max_completion_tokens, so get_optional_params raised UnsupportedParamsError for the standard OpenAI parameter (the modern replacement for the deprecated max_tokens). The v1 config already maps it to cohere's max_tokens; mirror that in v2 and add v2 regression tests. * fix(cohere): make max_completion_tokens take precedence over max_tokens on v2 When both max_tokens and max_completion_tokens are supplied, prefer max_completion_tokens explicitly rather than relying on dict iteration order, and cover both orderings with a regression test. --------- Co-authored-by: Daniel Yudelevich <4537920+yudelevi@users.noreply.github.com> Co-authored-by: hectorc98 <hector.chamorroalvarez@adyen.com> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Terrajlz <info@jouleselectrictech.com> Co-authored-by: Bruno Devaux <devaux.br@gmail.com> Co-authored-by: Dan Lemon <dan@danlemon.com> Co-authored-by: Saswat <saswatds@users.noreply.github.com> Co-authored-by: Brian Sparker <brainsparker@users.noreply.github.com> Co-authored-by: Zhao73 <156770117+Zhao73@users.noreply.github.com> Co-authored-by: Urain Ahmad Shah <60431964+urainshah@users.noreply.github.com> Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: kape <168134658+kapelame@users.noreply.github.com> Co-authored-by: danisalvaa <159898202+danisalvaa@users.noreply.github.com> Co-authored-by: Just R <remixingmagelang@gmail.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: abhay23-AI <abhaytrivedi22@gmail.com>
Closes #28332
Summary
Render
Values.podAnnotationsthroughtpl ... $indeploy/charts/litellm-helm/templates/deployment.yaml, so users can include template expressions (e.g. a sha256 of their custom ConfigMap) inside pod annotations.{{- with .Values.podAnnotations }} -{{- toYaml . | nindent 8 }} +{{- tpl (toYaml .) $ | nindent 8 }} {{- end }}Motivation
When users disable
proxyConfigMap.createto provide their own ConfigMap, the chart's built-inchecksum/configannotation is also disabled, so changes to a user-managed ConfigMap no longer roll the deployment. AllowingtplinpodAnnotationslets users re-implement that annotation themselves, e.g.:Consistency with existing chart code
This matches the
tpl (toYaml .) $pattern already used elsewhere in the same chart:deploy/charts/litellm-helm/templates/deployment.yaml:50—extraInitContainersdeploy/charts/litellm-helm/templates/deployment.yaml:215—extraContainersdeploy/charts/litellm-helm/templates/migrations-job.yaml:40,99— migrations jobDirect precedent: commit
87d7e86479("feat(helm): add tpl support to extraContainers and extraInitContainers", April 2026).Tests
Added one new
helm unittestcase todeploy/charts/litellm-helm/tests/deployment_tests.yaml:should support tpl in podAnnotations— setsproxyConfigMap.create: falseto mirror the real-world scenario, then asserts that a templated annotation referencing.Values.image.tagresolves correctly (proves$is wired), a second key referencing.Values.image.repositoryresolves correctly, and a plain-string annotation passes through unchanged (backward-compat canary).Run locally with
make test-unit-helm— all 54 tests pass.Also manually verified with
helm templatethat.Release.Nameand| sha256sumpipelines resolve correctly inside annotations.Backward compatibility
tplpasses plain strings (no{{ }}) through unchanged, so existing chart users see no behavior change. Users with a literal{{inside an annotation value would now see it rendered as a template — that caveat already applies to every othertplusage in this chart, and to the precedent PR above.Checklist
make test-unit-helmpasseshelm lint deploy/charts/litellm-helmpasses