merge main - #26984
Merged
Sameerlite merged 519 commits intoMay 1, 2026
Merged
Conversation
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
[Infra] Bump Versions
[Infra] Promote Internal Staging to main
…26441) * fix(redis): cache GCP IAM token to prevent async event loop blocking ## Problem GCPIAMCredentialProvider.get_credentials() calls _generate_gcp_iam_access_token on every Redis connection establishment. This function performs synchronous HTTP and gRPC calls (google-auth + google-cloud-iam) which block Python's asyncio event loop while running. Under concurrent load (e.g. connection pool warm-up, parallel health checks), multiple connections are established simultaneously, each triggering an independent blocking IAM token refresh. These refreshes serialise behind each other inside the single-threaded event loop, causing individual Redis spans to take 20-25 seconds instead of milliseconds. Observed in production via Datadog APM: a single INCRBYFLOAT Redis span took 25.6 seconds (90% of a 28.4s trace), with GCP metadata + GenerateAccessToken gRPC calls visible inside the span. This cascaded into aiohttp SocketTimeoutError on upstream LLM API calls — not because the upstream was slow, but because the event loop was frozen and the 30-second sock_read timer fired on a connection that was never given CPU time. ## Fix Add a module-level token cache (dict keyed by service account, value is (token, expiry_monotonic)). _get_cached_gcp_iam_token() returns the cached token on cache hit (no I/O), and refreshes only when expired using double-checked locking so only one thread performs the network round-trip. GCP IAM tokens are valid for 1 hour; the cache TTL is set to 55 minutes (_GCP_IAM_TOKEN_TTL_SECONDS = 3300) to refresh safely before expiry. The cache is shared across all GCPIAMCredentialProvider instances for the same service account, so N concurrent Redis connections on the same pod share a single token and avoid N concurrent blocking refreshes. get_credentials_async() already used asyncio.to_thread (non-blocking), and is updated to call _get_cached_gcp_iam_token so it also benefits from caching. ## Tests - Updated existing test that expected a fresh token on every call to reflect the new caching behaviour. - Added tests for: cache hit (no redundant I/O), cache expiry and refresh, and cache sharing across multiple provider instances. - Added autouse fixture to clear the module-level cache between tests. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * refactor(redis): remove unused Optional import from _redis_credential_provider.py * refactor(redis): improve documentation for GCPIAMCredentialProvider class Updated the docstring for the GCPIAMCredentialProvider class to clarify its purpose and the caching mechanism for GCP IAM tokens. The changes enhance readability and maintainability by providing a more concise explanation of the token caching strategy and its benefits for Redis authentication. * refactor(redis): improve documentation for GCPIAMCredentialProvider class Updated the docstring for the GCPIAMCredentialProvider class to clarify its purpose and the caching mechanism for GCP IAM tokens. The changes enhance readability and maintainability by providing a more concise explanation of the token caching strategy and its benefits for Redis authentication. --------- Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…5855) Bedrock enforces non-increasing TTL ordering across cache_control blocks (tools → system → messages). The tool cache_control TTL was being unconditionally dropped to the default 5m, while system blocks preserved the user-specified TTL for Claude 4.5+ models. This mismatch caused "a ttl='1h' block must not come after a ttl='5m' block" errors when users set ttl='1h' on both tools and system. Converse path: add_cache_point_tool_block() now accepts a model param and preserves TTL for Claude 4.5+, matching _get_cache_point_block(). Invoke path: _remove_ttl_from_cache_control() now also processes tools (was only processing system and messages). Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…onses (#20270) (#26262) * fix(proxy): invoke post-call guardrails on pass-through endpoint responses (#20270) Wire post_call_success_hook into non-streaming pass-through response path, gated on explicit guardrail config (opt-in only, no backwards-compat break). - Call post_call_success_hook after reading non-streaming response body - Build enriched hook_data with guardrails metadata and litellm_logging_obj at call site (avoids mutation of _parsed_body which is shared by logging) - Handle ModifyResponseException with provider-agnostic error envelope, post_call_failure_hook, and defensive try/except - Strip stale content-length when guardrail modifies response body - Move ModifyResponseException to litellm.exceptions to break cyclic import; re-export from custom_guardrail for backwards compat - Add call_type fallback in UnifiedLLMGuardrails for pass-through endpoints using CallTypes.pass_through.value enum * test: add unit tests for pass-through post-call guardrails 5 tests covering the post-call guardrail invocation on pass-through endpoints: - post_call_success_hook fires when guardrails configured - post_call_success_hook skipped when no guardrails (backwards compat) - ModifyResponseException returns 200 with provider-agnostic error - UnifiedLLMGuardrails resolves call_type from logging_obj for pass-through - ModifyResponseException re-export from custom_guardrail stays in sync
…n fallback path (#25888)
…#26122) tool_calls on assistant messages were translated to OllamaToolCall format but never copied into the outgoing OllamaChatCompletionMessage, so Ollama received {role: assistant, content: ''} with no tool_calls. The model then had no record of having made a tool call, causing it to re-issue the identical call on every turn (infinite loop). Similarly, tool_call_id on role:tool messages was silently dropped. Ollama uses this field to resolve the tool name from conversation history. Also add tool_call_id to OllamaChatCompletionMessage TypedDict. Fixes #26094
litellm oss branch
* Use auth key name if there are no app id in in headers or in extra_data * use key alias instead of key name * Fix * last priority key alias * Fix * Add tests * [Feat] Day-0 support for GPT-5.5 and GPT-5.5 Pro (#26449) * feat(openai): day-0 support for GPT-5.5 and GPT-5.5 Pro Add pricing + capability entries for the new GPT-5.5 family launched by OpenAI on 2026-04-24: - gpt-5.5 / gpt-5.5-2026-04-23 (chat): $5/$30/$0.50 per 1M input/output/cached input - gpt-5.5-pro / gpt-5.5-pro-2026-04-23 (responses-only): $60/$360/$6 per 1M input/output/cached input Other fees (long-context >272k, flex, batches, priority, cache discounts) follow the same ratios as GPT-5.4, with context window retained at 1.05M input / 128K output. No transformation / classifier code changes are required: OpenAIGPT5Config.is_model_gpt_5_4_plus_model() already matches 5.5+ via numeric version parsing, and model registration is driven from the JSON. The existing responses-API bridge for tools + reasoning_effort (litellm/main.py:970) already covers gpt-5.5-pro. Tests: - GPT5_MODELS regression list now covers gpt-5.5-pro and dated variants - New test_generic_cost_per_token_gpt55_pro cost-calc test - Updated test_generic_cost_per_token_gpt55 for long-context fields * fix(openai): mirror reasoning_effort flags onto gpt-5.5 dated variants gpt-5.5-2026-04-23 and gpt-5.5-pro-2026-04-23 were missing the supports_none_reasoning_effort, supports_xhigh_reasoning_effort, and supports_minimal_reasoning_effort flags that their non-dated counterparts define. Reasoning-effort routing in OpenAIGPT5Config is fully capability-driven from these JSON flags — since an absent flag is treated as False for opt-in levels (xhigh), users pinning to a dated snapshot would silently lose xhigh support and diverge from the base alias on logprobs + flexible temperature handling. Copy the flags onto both dated variants so every dated snapshot inherits the base model's reasoning-effort capability profile. Adds a parametrized regression test that asserts supports_{none,minimal,xhigh}_reasoning_effort parity between each dated variant and its non-dated counterpart, preventing future drift when new snapshots are added. * [Feat] Add azure/gpt-5.5 + azure/gpt-5.5-pro entries (+ dated variants) (#26361) * feat(azure): add azure/gpt-5.5 + azure/gpt-5.5-pro entries (+ dated variants) Azure variants of OpenAI's GPT-5.5 family. Microsoft has not yet shipped GPT-5.5 on Azure OpenAI (latest GA on the Foundry models page is GPT-5.4 as of 2026-04-24), but adding the entries day-0 mirrors the established precedent for azure/gpt-5.4* (which were in the cost map before the Azure rollout) so cost tracking and capability flags work the moment customers deploy. Schema follows the existing azure/gpt-5.4* shape: - Same base/long-context pricing as openai/gpt-5.5*: $5/$30 chat, $60/$360 pro per 1M, with priority tier 2x base - Azure variants drop the flex/batches keys (Azure has no flex tier) but keep priority pricing, matching gpt-5.4* precedent - mode=chat for the thinking model, mode=responses for pro reasoning_effort capability flags mirror the OpenAI variants exactly since Azure proxies the same API contract: minimal rejection on both chat and pro, low/none rejection on pro. Once #26456 (which sets supports_low_reasoning_effort + minimal=false on openai/gpt-5.5*) lands, OpenAI and Azure flag profiles align. Tests pin entry presence + pricing for all four Azure variants and verify the live-API-derived reasoning_effort flags. * test: register supports_low_reasoning_effort in cost-map JSON schema azure/gpt-5.5-pro and azure/gpt-5.5-pro-2026-04-23 added in this branch carry supports_low_reasoning_effort=false. The strict 'additionalProperties: false' schema in test_aaamodel_prices_and_context_window_json_is_valid rejected the new key. Register it alongside the other supports_*_reasoning_effort entries. Note: the runtime side of this flag (code that reads it) lands in #26456. Until that PR merges the flag is inert for both Azure and OpenAI pro entries, but having the schema accept it lets cost-map tests pass on either merge order. * Use sanitize deep copy style to replace deepcopy usage * Added test checking error is not happening anymore * Added warning log when json copy failed * Reduce to one change * Fix spaces --------- Co-authored-by: Ido Lavi <ido@noma.security> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: TomAlon <tom@noma.security>
…tch-422 fix(ui): use stored-credentials endpoint for tools fetch on MCP edit page
… triage Adds a CLI flag (`--timeout_worker_healthcheck`, env `TIMEOUT_WORKER_HEALTHCHECK`) that forwards to uvicorn's `timeout_worker_healthcheck` Config kwarg (added in uvicorn 0.37.0). Lets operators raise the supervisor's worker-ping timeout above the default 5s when triaging workers being killed and respawned under load. The helper introspects `uvicorn.Config.__init__` and only sets the kwarg if supported, otherwise prints a warning - so the existing uvicorn>=0.32.1,<1.0.0 floor pin is unaffected. Gunicorn and Hypercorn paths are unchanged (the uvicorn supervisor isn't running there); the value is also not passed to the helper at all on those paths so the "uvicorn too old" warning never fires spuriously.
…itellm_fix-logging-settings-admin-only
…lthcheck-flag feat(proxy): add --timeout_worker_healthcheck flag for uvicorn worker triage
fix(ci): support CircleCI rerun failed tests for local_testing jobs
…backs Switch the spend-logs save flow from mutateAsync + try/catch to mutate + callbacks. Errors now surface through a single onError path (no more double toast on failure), and the delete-then-update sequencing runs through onSettled instead of awaited promises. handleFormSubmit is no longer async. Tighten the corresponding test to assert exactly one error toast fires.
Previously, useStoreRequestInSpendLogs and useDeleteProxyConfigField did not refresh the proxyConfig cache on success, so the Logging Settings form continued to render the pre-save values until React Query refetched on its own. Wire both hooks to invalidate proxyConfigKeys on success so any active observer (currently the Logging Settings page) repulls fresh data. Export proxyConfigKeys for cross-hook reuse.
chore(mcp): encrypt user-scoped MCP credentials at rest
chore(mcp): SSRF guard on OAuth metadata discovery follow-up fetches
[Fix] Replace subprocess startup-import diff with static source scan
The proxy's ingress hardening (commit 842eea0) now strips client-supplied `mock_response` from the request body unless the calling key or team has the `allow_client_mock_response: true` admin-metadata flag set. The e2e model access tests rely on `mock_response` to short-circuit the LLM call, so without the flag they hit real backends — the bedrock wildcard route fakes out to a shared example endpoint that now 404s on unsupported paths, causing `test_model_access_patterns[key_models2-bedrock/anthropic.claude-3-True]` (and the bedrock/anthropic.* row that pytest -x never reaches) to fail. Set `allow_client_mock_response: true` on every key and team this test file provisions so `mock_response` is preserved end-to-end.
chore(passthrough): default auth=True and drop enterprise gate on the safe option
chore(proxy): contain UI_LOGO_PATH / LITELLM_FAVICON_URL on unauthenticated asset endpoints
chore(cli): tighten CLI SSO session flow
[Test] Proxy E2E: Opt In To Client Mock Response For Model Access Tests
The async/sync delete_response_api_handler always passed json=data into
httpx.delete, where data is {} from the transformer. httpx serializes that
to a 2-byte body. The Azure Responses DELETE endpoint now rejects any
request body with code: unexpected_body, breaking
test_basic_openai_responses_delete_endpoint on the llm_responses_api_testing
job. Build the kwargs dict and only set json= when data is truthy.
Add unit tests that patch httpx.delete and assert json/data are not in the
captured kwargs for the Azure DELETE path (sync and async).
…-19bdeb [Fix] Responses API: Omit Empty Body On DELETE
Run pre_call_hook on Google generateContent endpoints
[Fix] Refresh Redis TTL on counter writes, skip stale in-memory in Redis
Add pagination controls to model health status
…/litellm into litellm_auth_bypass_tag_based_routing
…a_labels feat(vertex_ai): propagate metadata labels to embedding, Imagen, rerank
…nstreaming-mixed-tools fix(anthropic): json response_format + user tools non-streaming
[Infra] Bump Versions
…routing add test(tag-routing): prevent header regex bypass for strict plain t…
Sameerlite
merged commit May 1, 2026
a94ae62
into
litellm_azure-container-file-routing-fix
159 of 165 checks passed
Contributor
|
Too many files changed for review. ( |
|
Michael Riad Zaky seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
fzowl
pushed a commit
to fzowl/litellm
that referenced
this pull request
Jun 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relevant issues
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
tests/test_litellm/directory, Adding at least 1 test is a hard requirement - see detailsmake test-unit@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewDelays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
CI (LiteLLM team)
Branch creation CI run
Link:
CI run for the last commit
Link:
Merge / cherry-pick CI run
Links:
Screenshots / Proof of Fix
Type
🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test
Changes