merge main - #27037
Merged
Sameerlite merged 554 commits intoMay 2, 2026
Merged
Conversation
The async/sync delete_response_api_handler always passed json=data into
httpx.delete, where data is {} from the transformer. httpx serializes that
to a 2-byte body. The Azure Responses DELETE endpoint now rejects any
request body with code: unexpected_body, breaking
test_basic_openai_responses_delete_endpoint on the llm_responses_api_testing
job. Build the kwargs dict and only set json= when data is truthy.
Add unit tests that patch httpx.delete and assert json/data are not in the
captured kwargs for the Azure DELETE path (sync and async).
litellm's default LiteLLMAiohttpTransport routes requests through aiohttp, which sits below httpx and is invisible to vcrpy's httpx-stub interception. Under vcrpy + aiohttp, requests reach the real network but responses come back through the stubbed httpx transport as empty 200s, surfacing as 'Unable to get json response - Expecting value: line 1 column 1 (char 0)' in providers like Anthropic, Gemini, and any other path that exercises the aiohttp transport. Disabling the aiohttp transport when the VCR persister is registered forces all calls through pure httpx, which vcrpy can record and replay correctly.
…-19bdeb [Fix] Responses API: Omit Empty Body On DELETE
Run pre_call_hook on Google generateContent endpoints
Azure OpenAI's responses-API DELETE endpoint rejects requests that carry
a JSON body with: "Unexpected body with size 2. This API method does
not accept a request body.". The default LiteLLMAiohttpTransport silently
elides empty-dict bodies on DELETE so this was masked, but the pure-httpx
transport (used when DISABLE_AIOHTTP_TRANSPORT=True or under vcrpy/respx
patching) sends literal '{}' (2 bytes), which Azure rejects.
Only attach json= when the provider's transform actually returned a
non-empty dict; otherwise issue a bodyless DELETE.
…itellm_vcr-cassette-llm-tests-af37 # Conflicts: # litellm/llms/custom_httpx/llm_http_handler.py
…dict When model_info.id equals model_name (common for batch models), the router resolves via has_model_id and returns one deployment dict instead of a list. The dict branch incorrectly iterated deployment keys (model_name, litellm_params, model_info), producing non-string values that broke LiteLLM_ManagedFileTable validation on managed file upload. Normalize list vs dict by wrapping single deployments and extracting model_info.id for each response pair. Add regression tests including the batch model id == model_name case. Made-with: Cursor
[Fix] Refresh Redis TTL on counter writes, skip stale in-memory in Redis
…odex/budget-race-enforcement-greptile-fix # Conflicts: # litellm/proxy/db/spend_counter_reseed.py # litellm/proxy/proxy_server.py
The Anthropic replay tests hardcoded specific token counts and content
strings ('Hello! How can I help you today?', prompt_tokens == 12). On a
fresh CI Redis those values must match a pre-recorded cassette that
doesn't exist, so the first run hits the live API and gets different
real bytes back.
Assert on shape instead: non-empty content, positive token counts,
finish_reason in the known set, and (for streaming) more than one chunk.
The tests still exercise the full transformation pipeline end-to-end and
catch shape regressions; drift in the exact text/token counts is
expected and now tolerated.
…transport vcrpy's aiohttp stub captures response bodies via 'await response.read()', which drains aiohttp's StreamReader. Downstream consumers of the same ClientResponse (litellm's AiohttpResponseStream, which iterates response.content.iter_chunked) then see an empty body and surface as JSON 'Expecting value: line 1 column 1 (char 0)' errors on every record-path call. The previous workaround set litellm.disable_aiohttp_transport=True for the whole VCR-active session, which made the tests exercise pure httpx instead of the production aiohttp transport. That hid the production transport from coverage and surfaced its own bugs (e.g. the Azure DELETE-with-empty-body case fixed in upstream staging). Replace the workaround with a targeted monkey-patch that re-feeds the captured body into the StreamReader via unread_data after vcrpy records it. Tests now run through the same transport customers do, both on first record and on replay, for both unary and streaming endpoints. Verified locally against api.anthropic.com with the production LiteLLMAiohttpTransport: record path passes (real network, 4.2s), replay path passes (Redis cache, 1.8s).
Add pagination controls to model health status
The batch rate limiter (`_check_and_increment_batch_counters`) and the dynamic rate limiter (`_check_rate_limits`) implemented rate limiting in two disjoint awaits: a `should_rate_limit(read_only=True)` check followed by a separate increment. Concurrent requests could all observe the same pre-increment state, all pass enforcement, and all then increment — multiplying the effective quota by the concurrency level. Demonstrated bypass (see new test): - Batch: 5 concurrent batches of 40 tokens each against TPM=100 consumed 200 tokens (100% over). - Dynamic: 5 concurrent priority="high" requests against RPM=2 all passed Phase 1 + Phase 3. Wrap both critical sections in a per-instance asyncio.Lock so the read and increment execute atomically within a process. Multi-replica deployments still rely on Redis Lua atomicity for cross-process safety; that is a follow-up. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…/litellm into litellm_auth_bypass_tag_based_routing
…afety The previous fix for the TOCTOU bypass relied on a per-instance asyncio.Lock, which closed the window only within a single proxy worker. Multi-replica deployments still raced across processes — A and B both read counter=99, both passed validation, both incremented to 100/100 → effective limit doubled. Add `CHECK_AND_INCREMENT_BY_N_SCRIPT` Lua script that processes any number of (window_key, counter_key, limit, increment, ttl) descriptors atomically with all-or-nothing semantics: if any descriptor would exceed its limit, no counter is modified and the script returns OVER_LIMIT with the offending descriptor's state. When Redis isn't configured, the in-memory fallback uses the existing asyncio.Lock for single-process atomicity. Expose this as `_PROXY_MaxParallelRequestsHandler_v3.atomic_check_and_increment_by_n` and rewire both call sites: - batch_rate_limiter._check_and_increment_batch_counters: replace the read_only=True check + separate async_increment_tokens_with_ttl_preservation with a single atomic call passing the batch's (request_count, total_tokens) as the increment. - dynamic_rate_limiter_v3._check_rate_limits: bundle model_saturation_check (always enforced) and priority_model (enforced only when saturated) into one atomic call. When priority is unenforced, increment its counter via the existing should_rate_limit(read_only=False) path for tracking only. Update structural regression tests to assert the new atomic path is used rather than the legacy two-phase pattern. Tests: 4/4 TOCTOU tests pass, 59 existing rate-limiter tests pass, no regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…a_labels feat(vertex_ai): propagate metadata labels to embedding, Imagen, rerank
…nstreaming-mixed-tools fix(anthropic): json response_format + user tools non-streaming
Fix runtime policy attachment initialization
Conflict resolution for #26968 dropped the `Iterator` typing import (NameError at module load), left a dead `fallback_models = cast(...)` block, and the new tests called `_enforce_key_and_fallback_model_access` without the now-required `request` kwarg.
fix(prometheus): escape api_key for PromQL string literal (VERIA-53)
[Fix] Isolate dual OTEL handlers
…jack fix(proxy): close project hijacking and key org IDOR (VERIA-55)
…te And Regenerate
Mirrors the membership rule on /key/update so that /key/generate and
/key/{key}/regenerate apply the same `_validate_caller_can_assign_key_org`
gate when the caller specifies an `organization_id`. Proxy admins bypass.
The check no-ops when `organization_id` is not being set.
chore(auth): require trusted proxy for header identity auth
chore(sso): bind generic SSO state to a session cookie
Three live-API tests pinned to claude-4-sonnet-20250514, which is a non-canonical alias of claude-sonnet-4-20250514. Anthropic's main API no longer resolves the legacy form under freshly issued keys, so the tests fail with not_found_error. The token counter test pinned to claude-sonnet-4-20250514 itself (deprecation_date 2026-05-14, two weeks out) was on borrowed time too. Bump all four to claude-haiku-4-5-20251001 — capability superset for what these tests exercise (streaming, parallel tool calling, extended thinking, token counting), no upcoming deprecation, cheaper per-token.
This file is a regenerable UI build artifact that should not be tracked in source. Removing so the merge into litellm_internal_staging stays clean.
…aiku 4.5 test_anthropic_messages_streaming_cost_injection hits the proxy's /v1/messages route, which routes via the anthropic/* wildcard to api.anthropic.com. The 404 surfaced in the test was Anthropic's own not_found_error propagated back through the proxy (visible from the x-litellm-model-id hash on the response — the proxy did route). Same root cause as the prior commit: the legacy claude-4-sonnet-20250514 alias is no longer recognized by Anthropic's main API under the new key. Swap to claude-haiku-4-5-20251001 — same routing path, canonical model.
…release_detection [Fix] Release Workflow: Detect SemVer-Style Pre-Release Dev Tags
… Alias base_anthropic_messages_test.test_anthropic_messages_with_thinking and test_anthropic_streaming_with_thinking still pinned to claude-4-sonnet-20250514 — the same legacy alias Anthropic no longer recognizes under freshly issued keys. The other four tests in this base class already use claude-sonnet-4-5-20250929; these two were missed. Bump to claude-haiku-4-5-20251001 (supports_reasoning=true, no upcoming deprecation). Subclasses including TestAnthropicPassthroughBasic inherit these methods.
[Infra] Merge dev branch
[Test] Anthropic: Replace Legacy Claude-4-Sonnet Alias With Haiku 4.5
…fication fix(auth): support JWT issuer verification + warn when unscoped
fix(router): constrain same-name deployment routing by access groups
…t_thinking fix(gemini): follow provider defaults for Gemini 3 thinking
feat(mcp): enforce org-level MCP server and toolset permissions
Newer Cloudflare Workers AI models (e.g. Nemotron) emit 'response_text' instead of 'response' on streamed chunks. The non-streaming path was already updated to fall back to 'response_text' (#26385), but the streaming chunk parser still only read 'response', which caused streaming requests against those models to silently produce empty content. Mirror the non-streaming fallback in CloudflareChatResponseIterator.chunk_parser and add a streaming test for the response_text shape. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
…itellm_oss_staging_04_25_2026 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
…eam-passthrough fix(guardrails): preserve responses event streams in presidio output masking
chore(staging): roll oss_staging_04_25_2026 into internal staging (output_config fix + 4 upstream sync fixes)
Contributor
|
Too many files changed for review. ( |
Sameerlite
merged commit May 2, 2026
6d13264
into
litellm_fix_stateful_statless_mcp
106 checks passed
|
Michael Riad Zaky seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
19 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relevant issues
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
tests/test_litellm/directory, Adding at least 1 test is a hard requirement - see detailsmake test-unit@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewDelays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
CI (LiteLLM team)
Branch creation CI run
Link:
CI run for the last commit
Link:
Merge / cherry-pick CI run
Links:
Screenshots / Proof of Fix
Type
🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test
Changes