merge main - #27658
Conversation
…onds fields OVHCloud is deprecating two response fields on 2026-05-11: - reasoning_content replaced by reasoning (LLM reasoning models) - duration replaced by seconds (Speech-to-Text models) Adds backward-compatible support for both field names during the transition window, preferring the new field when present and falling back to the legacy field. Fixes #26586
Replaces falsy or with explicit is not None check so that a valid seconds=0.0 value is not silently dropped during field migration. Addresses Greptile review feedback on #26595
…ions for Responses API Fixes #25840 The OTel integration's set_attributes() method never populates gen_ai.output.messages, gen_ai.system_instructions, or gen_ai.response.finish_reasons for /v1/responses calls because ResponsesAPIResponse uses 'output' instead of 'choices' and the system prompt arrives as 'instructions' instead of 'system_instructions'. Changes: - Add elif branch for response_obj.get('output') to extract response text from Responses API output items (type='message'/output_text) and tool calls (type='function_call') - Coalesce system_instructions/instructions/system kwargs so the system prompt is captured for Responses API, Anthropic Messages API, and Vertex AI Gemini paths - Handle plain-string system prompts without unnecessary wrapping - Extract response_obj.get('status') as finish reason for Responses API - Add _transform_responses_api_output_to_otel() method
…uctions, and finish reasons Add 21 tests covering the new Responses API OTel attribute handling: TestOpenTelemetryResponsesAPI (13 tests): - gen_ai.output.messages from output items (text, function_call, mixed, multi-part) - gen_ai.response.finish_reasons from ResponsesAPIResponse.status - gen_ai.system_instructions from instructions/system/system_instructions kwargs - Precedence and absence edge cases - Regression test for existing choices-based responses TestTransformResponsesAPIOutput (8 tests): - Message with output_text, function_call items, unknown types - Edge cases: empty output, empty text, missing call_id, default role, non-dict items
Build the tool_call part dict separately with an explicit type annotation so mypy can track the type, avoiding the 'Unsupported target for indexed assignment' error on tool_call["parts"][0]["id"].
…r-tool-call attrs - Replace isinstance(item, dict) with hasattr(item, 'get') so Pydantic model instances (ResponseOutputMessage, ResponseFunctionToolCall) are accepted alongside plain dicts (P1) - Use 'is not None' guards instead of or-chain for system_instructions coalescing to prevent falsy values (e.g. []) falling through to the wrong kwarg (P2) - Emit per-tool-call span attributes (gen_ai.completion.N.function_call.*) for Responses API function_call items, matching the choices branch parity with _tool_calls_kv_pair (P2) - Add 4 new tests: Pydantic-like objects, falsy fallthrough guard, per-tool-call attribute emission, multiple tool call indexing
…onses Adds transform_response to OVHCloudChatConfig to normalise the new easoning field to easoning_content in non-streaming responses, matching the existing streaming fix in chunk_parser. Addresses maintainer feedback on #26595
The parent OpenAIGPTConfig already handles reasoning->reasoning_content for non-streaming via _extract_reasoning_content. The override was dead code giving false confidence. Streaming fix in chunk_parser is the only change needed for chat completions. Addresses Agent Shin review feedback on #26595
…y handles non-streaming via _extract_reasoning_content
…mation The openai SDK returns ResponseOutputMessage and ResponseOutputText as raw Pydantic v2 models that lack .get() (unlike LiteLLM's own wrapper objects). Add a _to_dict() helper that normalizes plain dicts, BaseLiteLLMOpenAIResponseObject (has .get()), and raw Pydantic models (has .model_dump()) into a consistent dict interface.
Iterate user/key/team/team_member/org/end_user/tag spend dicts in sorted order inside each Prisma transaction so concurrent pods acquire row locks in the same order, avoiding PostgreSQL deadlocks under load.
…nto OpenAPI schema
Co-authored-by: ishaan-berri <ishaan-berri@users.noreply.github.com>
Co-authored-by: ishaan-berri <ishaan-berri@users.noreply.github.com>
Squash-merged by litellm-agent from oss-pr-review-agent-shin[bot]'s PR.
…26325) Squash-merged by litellm-agent from milan-berri's PR.
Squash-merged by litellm-agent from Michael-RZ-Berri's PR.
Squash-merged by litellm-agent from noahnistler's PR.
…n /v1/messages (#27534) Squash-merged by litellm-agent from Anai-Guo's PR.
…27531) Squash-merged by litellm-agent from krisxia0506's PR.
…27521) Squash-merged by litellm-agent from Bytechoreographer's PR.
Squash-merged by litellm-agent from shivamrawat1's PR.
Squash-merged by litellm-agent from oss-agent-shin's PR.
…6_2026 [litellm-agent] Staging → litellm_internal_staging (5/6/2026)
Reject fnmatch wildcards on non-scope claims when the claim string contains whitespace so malformed iss values cannot match patterns like trusted.*. Merge every entry when team_id_jwt_field resolves to a list instead of keeping only the first element. Co-authored-by: Cursor <cursoragent@cursor.com>
[litellm-agent] Staging → litellm_internal_staging (5/7/2026)
[litellm-agent] Staging → litellm_internal_staging (5/9/2026)
Co-authored-by: Cursor <cursoragent@cursor.com>
79618b1
into
litellm_reasoning_summary_chat_bridge
|
Michael Riad Zaky seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 3 potential issues.
Bugbot Autofix prepared fixes for all 3 issues found in the latest run.
- ✅ Fixed: Non-dict context_management escapes filter without being removed
- Non-dict context_management is now removed before returning from the Bedrock invoke filter.
- ✅ Fixed: Duplicate AZURE_REDIS_SCOPE constant across two modules
- The Redis module now imports AZURE_REDIS_SCOPE from the credential provider instead of redefining it.
- ✅ Fixed: Model name replace strips all "responses/" occurrences not just prefix
- Responses aliases now strip only a leading responses/ prefix via removeprefix.
Preview (4c1d91d96f)
diff --git a/litellm/_redis.py b/litellm/_redis.py
--- a/litellm/_redis.py
+++ b/litellm/_redis.py
@@ -19,6 +19,7 @@
from litellm import get_secret, get_secret_str
from litellm._redis_credential_provider import (
+ AZURE_REDIS_SCOPE,
AzureADCredentialProvider,
GCPIAMCredentialProvider,
_generate_gcp_iam_access_token,
@@ -28,9 +29,7 @@
from ._logging import verbose_logger
-AZURE_REDIS_SCOPE = "https://redis.azure.com/.default"
-
def _get_redis_kwargs():
arg_spec = inspect.getfullargspec(redis.Redis)
diff --git a/litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py b/litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py
--- a/litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py
+++ b/litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py
@@ -429,6 +429,7 @@
"""
cm = anthropic_messages_request.get("context_management")
if not isinstance(cm, dict):
+ anthropic_messages_request.pop("context_management", None)
return
edits = cm.get("edits")
if not isinstance(edits, list):
diff --git a/litellm/router.py b/litellm/router.py
--- a/litellm/router.py
+++ b/litellm/router.py
@@ -7077,8 +7077,8 @@
k: v for k, v in _model_info.items() if k not in _custom_pricing_fields
}
_backend_alias_cost = {_model_name: _shared_model_info}
- if "responses/" in _model_name:
- _stripped_model_name = _model_name.replace("responses/", "")
+ if _model_name.startswith("responses/"):
+ _stripped_model_name = _model_name.removeprefix("responses/")
_backend_alias_cost[_stripped_model_name] = _shared_model_info
litellm.register_model(model_cost=_backend_alias_cost)
@@ -7785,8 +7785,8 @@
k: v for k, v in _model_info_dict.items() if k not in _custom_pricing_fields
}
_backend_alias_cost = {_model_name: _shared_model_info}
- if "responses/" in _model_name:
- _stripped_model_name = _model_name.replace("responses/", "")
+ if _model_name.startswith("responses/"):
+ _stripped_model_name = _model_name.removeprefix("responses/")
_backend_alias_cost[_stripped_model_name] = _shared_model_info
litellm.register_model(model_cost=_backend_alias_cost)You can send follow-ups to the cloud agent here.
Reviewed by Cursor Bugbot for commit 9bc90f6. Configure here.
| """ | ||
| cm = anthropic_messages_request.get("context_management") | ||
| if not isinstance(cm, dict): | ||
| return |
There was a problem hiding this comment.
Non-dict context_management escapes filter without being removed
Low Severity
When context_management is present but not a dict (e.g., a string or list passed by a non-standard caller), _filter_context_management_for_bedrock_invoke returns early at line 431–432 without removing it from the request. Since context_management was added to BedrockInvokeAnthropicMessagesRequest (and thus BEDROCK_INVOKE_ALLOWED_TOP_LEVEL_FIELDS), this malformed value survives the downstream safety-net allowlist filter and reaches Bedrock, which would 400. The other two exit paths in this function correctly pop the key; this early-return path is missing the same cleanup.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 9bc90f6. Configure here.
|
|
||
| from ._logging import verbose_logger | ||
|
|
||
| AZURE_REDIS_SCOPE = "https://redis.azure.com/.default" |
There was a problem hiding this comment.
Duplicate AZURE_REDIS_SCOPE constant across two modules
Low Severity
AZURE_REDIS_SCOPE is defined identically in both litellm/_redis.py and litellm/_redis_credential_provider.py. If the scope URI ever needs updating, the change must be made in two places or they'll silently diverge — one module would authenticate against the wrong scope. One of the definitions could import from the other.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 9bc90f6. Configure here.
| ) | ||
| _backend_alias_cost = {_model_name: _shared_model_info} | ||
| if "responses/" in _model_name: | ||
| _stripped_model_name = _model_name.replace("responses/", "") |
There was a problem hiding this comment.
Model name replace strips all "responses/" occurrences not just prefix
Low Severity
_model_name.replace("responses/", "") removes every occurrence of "responses/" in the string, not just a leading prefix. The _model_name is constructed as custom_llm_provider + "/" + model, so a model like "myhost/responses/v2" with provider "responses" would produce "responses/myhost/responses/v2" and be stripped to "myhost/v2" instead of "myhost/responses/v2". Using removeprefix (or splitting on the first occurrence) would be more precise.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 9bc90f6. Configure here.
Greptile SummaryThis is a large merge from
Confidence Score: 4/5Safe to merge with minor follow-up; the core changes are well-structured, but the polling wait cap in shared health checks and the duplicated Azure scope constant are worth addressing. The broad set of changes is individually well-reasoned and tested. The health-check polling loop now waits up to a full lock TTL (potentially 60 s) before falling back to a local check, which could noticeably slow /health responses under contention. The AZURE_REDIS_SCOPE constant duplication is a minor DRY gap. Everything else — deadlock fix, new bulk-update endpoint, JWT scope routing, Bedrock context-management filtering, OTel Responses API support — looks correct and purposeful. litellm/_redis.py (duplicated constant), litellm/proxy/health_check_utils/shared_health_check_manager.py (polling wait cap), litellm/proxy/management_endpoints/key_management_endpoints.py (all_keys_in_team token flow — verify _hash_token_if_needed idempotency assumption)
|
| Filename | Overview |
|---|---|
| litellm/_redis.py | Adds Azure AD authentication for Redis (sync and async paths); duplicates AZURE_REDIS_SCOPE constant already defined in _redis_credential_provider.py |
| litellm/_redis_credential_provider.py | Adds AzureADCredentialProvider class mirroring GCPIAMCredentialProvider pattern; correctly wraps sync get_token in asyncio.to_thread for the async path |
| litellm/proxy/management_endpoints/key_management_endpoints.py | Adds /team/key/bulk_update endpoint with RBAC checks; refactors _process_single_key_update to accept pre-fetched key rows; the all_keys_in_team token handling relies on _hash_token_if_needed idempotency |
| litellm/types/proxy/management_endpoints/key_management_endpoints.py | Adds KeyUpdateFields (extra=forbid Pydantic model) and BulkUpdateTeamKeysRequest with appropriate validators; allowlist design correctly blocks RBAC/ownership field mutations |
| litellm/proxy/db/db_spend_update_writer.py | Sorts all transaction dictionaries before iteration to enforce consistent lock-acquisition order across pods, preventing deadlocks in batched DB writes |
| litellm/proxy/auth/user_api_key_auth.py | Adds scope selector to JWTRoutingOverride and wildcard matching (fnmatch) to _routing_selector_matches_claim; whitespace guard correctly prevents wildcards from spanning multi-value iss/aud claims |
| litellm/proxy/_experimental/mcp_server/oauth_utils.py | Adds validate_trusted_redirect_uri accepting same-origin OR loopback; moves get_request_base_url here from discoverable_endpoints; exception fallback to loopback-only is appropriately conservative |
| litellm/proxy/_experimental/mcp_server/discoverable_endpoints.py | Switches from loopback-only to validate_trusted_redirect_uri across all redirect-URI validation points; callback endpoint now correctly receives Request to pass to the validator |
| litellm/proxy/health_check_utils/shared_health_check_manager.py | Replaces 2-second fixed sleep with a polling loop up to lock_ttl; correctly handles Redis-unavailable path separately; max wait duration may be too long for health-check use cases |
| litellm/proxy/guardrails/guardrail_registry.py | Adds source provenance tracking (db vs config) and reconcile_db_guardrails for cross-pod consistency; clean implementation with correct cleanup on delete |
| litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py | Adds _filter_context_management_for_bedrock_invoke to strip unsupported context_management edit types before Bedrock InvokeModel, resolving 400 errors from Claude Code clear_thinking edits |
| litellm/integrations/opentelemetry.py | Adds Responses API output tracing support and coalesces system_instructions/instructions/system kwargs; _to_dict helper handles dict/duck-type/Pydantic v2 forms correctly |
| litellm/router.py | Registers backend alias cost entries for responses/ prefix models; moves _model_info_dict population before the model_id guard so dynamically-added deployments also register correct pricing |
| litellm/proxy/common_utils/reset_budget_job.py | Adds cache invalidation for tags in _cascade_reset_spend_for_budget_link so stale spend values do not block tags post-reset; correctly optional via cache_key_fn callback |
| litellm/proxy/litellm_pre_call_utils.py | Fixes header-value coercion for httpx (dict/list JSON-encoded, other types str-cast); adds provider resolution from deployment when model name lacks a prefix |
| litellm/proxy/_types.py | Adds TEAM_KEY_BULK_UPDATE route to LiteLLMRoutes; expands internal_user_view_only_routes to include compliance_check_routes, granting read-only users access to compliance endpoints |
| litellm/proxy/management_endpoints/tag_management_endpoints.py | Adds optional date-range filtering to /tag/list with validation helper; correctly applies the date filter only to the dynamic-tag query, not to stored tags |
| litellm/llms/ovhcloud/chat/transformation.py | Normalises OVHCloud reasoning_content/reasoning field names during streaming; correctly checks legacy field before overwriting to avoid clobbering explicit legacy values |
| litellm/llms/ovhcloud/audio_transcription/transformation.py | Adds seconds to duration field migration for OVHCloud STT responses; migration deadline comment is dated today (2026-05-11), follow-up cleanup should be tracked |
Reviews (1): Last reviewed commit: "chore: remove accidental .evidence scree..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
|
|
||
| from ._logging import verbose_logger | ||
|
|
||
| AZURE_REDIS_SCOPE = "https://redis.azure.com/.default" |
There was a problem hiding this comment.
AZURE_REDIS_SCOPE constant
AZURE_REDIS_SCOPE is defined identically in both litellm/_redis.py and litellm/_redis_credential_provider.py. _redis_credential_provider.py is imported into _redis.py, so there is no need for a second definition. If the scope ever changes, only the owning module's constant will be updated and the other will silently diverge, causing token-fetch failures at the wrong scope.
| # OVHCloud field migration (deadline: 2026-05-11): | ||
| # `duration` is replaced by `seconds` in STT responses. | ||
| # Prefer `seconds`, fall back to `duration`, normalize to `duration` | ||
| # so downstream consumers see a consistent key. | ||
| duration = ( | ||
| response_json["seconds"] | ||
| if "seconds" in response_json and response_json["seconds"] is not None | ||
| else response_json.get("duration") | ||
| ) |
There was a problem hiding this comment.
The comment says # OVHCloud field migration (deadline: 2026-05-11) and the current date is 2026-05-11. The fallback to the duration field was presumably meant to be removed once OVHCloud retired the old key, but the PR is only adding this compatibility shim now, on the deadline date. Make sure there is a tracked follow-up to remove this shim once the transition is confirmed complete.
|
|
||
| # Wait a bit for the other pod to complete | ||
| await asyncio.sleep(2) | ||
| poll_interval = 5 # seconds between cache checks | ||
| max_wait = self.lock_ttl # wait at most as long as the lock can live | ||
| elapsed = 0 | ||
|
|
||
| # Try to get cached results again | ||
| cached_results = await self.get_cached_health_check_results() | ||
| if cached_results is not None: | ||
| return ( | ||
| cached_results.get("healthy_endpoints", []), | ||
| cached_results.get("unhealthy_endpoints", []), | ||
| {}, | ||
| ) | ||
| while elapsed < max_wait: | ||
| await asyncio.sleep(poll_interval) | ||
| elapsed += poll_interval | ||
|
|
||
| cached_results = await self.get_cached_health_check_results() | ||
| if cached_results is not None: | ||
| verbose_proxy_logger.info( | ||
| "Pod %s using cached health check results after waiting %ds", | ||
| self.pod_id, | ||
| elapsed, | ||
| ) | ||
| return ( | ||
| cached_results.get("healthy_endpoints", []), | ||
| cached_results.get("unhealthy_endpoints", []), | ||
| {}, | ||
| ) | ||
|
|
||
| # Still no cache, fall back to local health check | ||
| # Check if the lock is still held — if it was released without | ||
| # caching (e.g. the holder crashed), stop waiting early. | ||
| try: | ||
| lock_key = self.get_health_check_lock_key() | ||
| current_owner = await self.redis_cache.async_get_cache(lock_key) | ||
| if current_owner is None: | ||
| verbose_proxy_logger.debug( | ||
| "Pod %s detected lock released without cache, stopping wait", | ||
| self.pod_id, | ||
| ) | ||
| break | ||
| except Exception: | ||
| # Redis hiccup — continue polling rather than crashing out | ||
| pass | ||
|
|
||
| # Exhausted wait — fall back to local health check | ||
| verbose_proxy_logger.warning( | ||
| "Pod %s falling back to local health check (no cache available)", | ||
| "Pod %s falling back to local health check after waiting %ds (no cache available)", |
There was a problem hiding this comment.
lock_ttl seconds on /health
The old code waited a fixed 2 seconds before falling back; the new loop polls every 5 seconds for up to self.lock_ttl seconds (e.g., 60 s by default). A pod that cannot acquire the lock will block its caller for up to one full lock TTL before doing its own health check. For a /health endpoint hit by load balancers on tight intervals, a 60-second hang is far worse than a redundant check. The early-exit on a released lock helps but only fires 5 seconds after the lock disappears. Consider capping max_wait to a fraction of lock_ttl (e.g., 20 s) and letting the local fallback run sooner.
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
| ) | ||
|
|
||
| if deployment_by_id is not None: | ||
| config_litellm_params = deployment_by_id.litellm_params.model_dump( |
There was a problem hiding this comment.
Medium: Deployment id is not bound to the authorization check
model_info.id is caller-controlled, and the code loads that deployment's litellm_params before authorizing against the caller-supplied model_info. A team admin can submit another deployment id while setting model_info.team_id to their own team, causing the health check to run with credentials for a deployment they are not allowed to manage. Use the resolved deployment's model_info for can_user_make_model_call, or reject the request when the supplied id's actual team/model does not match the caller-authorized model info.
Medium: Test-connection deployment lookup is not authorization-boundThis PR lets Status: 1 new · 1 open |
16 participants
Relevant issues
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
tests/test_litellm/directory, Adding at least 1 test is a hard requirement - see detailsmake test-unit@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewDelays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
CI (LiteLLM team)
Branch creation CI run
Link:
CI run for the last commit
Link:
Merge / cherry-pick CI run
Links:
Screenshots / Proof of Fix
Type
🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test
Changes
Note
Medium Risk
Touches authentication/authorization flows (JWT routing overrides, OAuth redirect validation, Azure AD Redis auth) and key-management endpoints, so misconfigurations could affect access control or connectivity; changes are covered by targeted tests but span multiple subsystems.
Overview
Adds Azure AD authentication for Redis (sync + async/cluster) using long-lived
azure-identitycredentials and a newAzureADCredentialProvider, with env/config knobs (azure_*) and safeguards against simultaneous GCP IAM config.Extends OpenTelemetry to support OpenAI Responses API payloads (
output,status,instructions) and improves system-prompt attribute coalescing; adds extensive unit tests.Hardens/extends proxy behavior: adds
/team/key/bulk_updatewith a broadcast update payload + new request types/validation and route/RBAC wiring; enhances JWT routing overrides withscopematching and wildcard selector support; fixes header forwarding by coercing non-string values; improves health-check lock waiting when Redis is enabled; and tightens MCP OAuth redirect validation to allow same-origin UI callbacks while still blocking open redirects.Fixes several provider/runtime edge cases: Bedrock Invoke now filters
context_managementto Bedrock-supported compaction edits (and injects the required beta), OVHCloud normalizes migrated response fields (seconds→duration,reasoning→reasoning_content), spend/budget reset now invalidates staleuser_api_key_cachefor tags, spend-update writers sort batch updates to reduce DB deadlocks, and in-memory guardrails now trackdbvsconfigprovenance and reconcile stale DB-deleted entries across pods.Reviewed by Cursor Bugbot for commit 9bc90f6. Bugbot is set up for automated code reviews on this repo. Configure here.