merge main - #27036
Merged
Sameerlite merged 538 commits intoMay 2, 2026
Merged
Conversation
``variant`` is user-controlled (passed through from ``litellm.video_content(variant=...)``) and was interpolated raw into the URL query string. A value like ``thumbnail&extra=1`` would inject additional query parameters into the upstream request — the same class of issue this PR's path-segment encoding addresses. Wrap the value in ``quote(value, safe="")`` so ``&`` / ``=`` / ``#`` cannot terminate the ``variant`` value or open a new parameter. Adds a regression test asserting that a malicious ``thumbnail&extra=1`` ends up percent-encoded in the URL, and that the legitimate ``thumbnail`` value still round-trips cleanly.
[Test] Proxy E2E: Opt In To Client Mock Response For Model Access Tests
Two Greptile review findings addressed: 1. (P1, security) The ``litellm_oauth_state`` cookie is the sole guard against Login-CSRF in the PKCE flow but was set without the ``Secure`` attribute, so a network observer on plain HTTP could read and replay it — bypassing the protection this PR adds. Thread the originating ``Request`` down through ``get_sso_login_redirect`` and ``get_generic_sso_redirect_response`` and set ``Secure`` based on ``request.url.scheme == "https"``. When no request is supplied (programmatic callers / tests) default to ``Secure=True`` — production-safe. Local HTTP dev still works because the request scheme is observed at runtime. 2. (P2) The cookie was set unconditionally, but the callback only validates it inside the PKCE branch. Two concurrent SSO sessions (one PKCE, one plain) could overwrite each other's state cookie and produce spurious 400s for the plain-flow user. Move the ``set_cookie`` call inside the existing ``if code_verifier and "state" in redirect_params`` block so the cookie is only written when PKCE is active and the validation will actually fire. Tests cover both paths: PKCE-on (cookie set with Secure default), PKCE-off (cookie not set), and HTTP dev request (Secure dropped so the browser will actually attach the cookie on the callback hop).
The async/sync delete_response_api_handler always passed json=data into
httpx.delete, where data is {} from the transformer. httpx serializes that
to a 2-byte body. The Azure Responses DELETE endpoint now rejects any
request body with code: unexpected_body, breaking
test_basic_openai_responses_delete_endpoint on the llm_responses_api_testing
job. Build the kwargs dict and only set json= when data is truthy.
Add unit tests that patch httpx.delete and assert json/data are not in the
captured kwargs for the Azure DELETE path (sync and async).
litellm's default LiteLLMAiohttpTransport routes requests through aiohttp, which sits below httpx and is invisible to vcrpy's httpx-stub interception. Under vcrpy + aiohttp, requests reach the real network but responses come back through the stubbed httpx transport as empty 200s, surfacing as 'Unable to get json response - Expecting value: line 1 column 1 (char 0)' in providers like Anthropic, Gemini, and any other path that exercises the aiohttp transport. Disabling the aiohttp transport when the VCR persister is registered forces all calls through pure httpx, which vcrpy can record and replay correctly.
…-19bdeb [Fix] Responses API: Omit Empty Body On DELETE
Run pre_call_hook on Google generateContent endpoints
Azure OpenAI's responses-API DELETE endpoint rejects requests that carry
a JSON body with: "Unexpected body with size 2. This API method does
not accept a request body.". The default LiteLLMAiohttpTransport silently
elides empty-dict bodies on DELETE so this was masked, but the pure-httpx
transport (used when DISABLE_AIOHTTP_TRANSPORT=True or under vcrpy/respx
patching) sends literal '{}' (2 bytes), which Azure rejects.
Only attach json= when the provider's transform actually returned a
non-empty dict; otherwise issue a bodyless DELETE.
…itellm_vcr-cassette-llm-tests-af37 # Conflicts: # litellm/llms/custom_httpx/llm_http_handler.py
…dict When model_info.id equals model_name (common for batch models), the router resolves via has_model_id and returns one deployment dict instead of a list. The dict branch incorrectly iterated deployment keys (model_name, litellm_params, model_info), producing non-string values that broke LiteLLM_ManagedFileTable validation on managed file upload. Normalize list vs dict by wrapping single deployments and extracting model_info.id for each response pair. Add regression tests including the batch model id == model_name case. Made-with: Cursor
[Fix] Refresh Redis TTL on counter writes, skip stale in-memory in Redis
…odex/budget-race-enforcement-greptile-fix # Conflicts: # litellm/proxy/db/spend_counter_reseed.py # litellm/proxy/proxy_server.py
The Anthropic replay tests hardcoded specific token counts and content
strings ('Hello! How can I help you today?', prompt_tokens == 12). On a
fresh CI Redis those values must match a pre-recorded cassette that
doesn't exist, so the first run hits the live API and gets different
real bytes back.
Assert on shape instead: non-empty content, positive token counts,
finish_reason in the known set, and (for streaming) more than one chunk.
The tests still exercise the full transformation pipeline end-to-end and
catch shape regressions; drift in the exact text/token counts is
expected and now tolerated.
…transport vcrpy's aiohttp stub captures response bodies via 'await response.read()', which drains aiohttp's StreamReader. Downstream consumers of the same ClientResponse (litellm's AiohttpResponseStream, which iterates response.content.iter_chunked) then see an empty body and surface as JSON 'Expecting value: line 1 column 1 (char 0)' errors on every record-path call. The previous workaround set litellm.disable_aiohttp_transport=True for the whole VCR-active session, which made the tests exercise pure httpx instead of the production aiohttp transport. That hid the production transport from coverage and surfaced its own bugs (e.g. the Azure DELETE-with-empty-body case fixed in upstream staging). Replace the workaround with a targeted monkey-patch that re-feeds the captured body into the StreamReader via unread_data after vcrpy records it. Tests now run through the same transport customers do, both on first record and on replay, for both unary and streaming endpoints. Verified locally against api.anthropic.com with the production LiteLLMAiohttpTransport: record path passes (real network, 4.2s), replay path passes (Redis cache, 1.8s).
Add pagination controls to model health status
The batch rate limiter (`_check_and_increment_batch_counters`) and the dynamic rate limiter (`_check_rate_limits`) implemented rate limiting in two disjoint awaits: a `should_rate_limit(read_only=True)` check followed by a separate increment. Concurrent requests could all observe the same pre-increment state, all pass enforcement, and all then increment — multiplying the effective quota by the concurrency level. Demonstrated bypass (see new test): - Batch: 5 concurrent batches of 40 tokens each against TPM=100 consumed 200 tokens (100% over). - Dynamic: 5 concurrent priority="high" requests against RPM=2 all passed Phase 1 + Phase 3. Wrap both critical sections in a per-instance asyncio.Lock so the read and increment execute atomically within a process. Multi-replica deployments still rely on Redis Lua atomicity for cross-process safety; that is a follow-up. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…/litellm into litellm_auth_bypass_tag_based_routing
…afety The previous fix for the TOCTOU bypass relied on a per-instance asyncio.Lock, which closed the window only within a single proxy worker. Multi-replica deployments still raced across processes — A and B both read counter=99, both passed validation, both incremented to 100/100 → effective limit doubled. Add `CHECK_AND_INCREMENT_BY_N_SCRIPT` Lua script that processes any number of (window_key, counter_key, limit, increment, ttl) descriptors atomically with all-or-nothing semantics: if any descriptor would exceed its limit, no counter is modified and the script returns OVER_LIMIT with the offending descriptor's state. When Redis isn't configured, the in-memory fallback uses the existing asyncio.Lock for single-process atomicity. Expose this as `_PROXY_MaxParallelRequestsHandler_v3.atomic_check_and_increment_by_n` and rewire both call sites: - batch_rate_limiter._check_and_increment_batch_counters: replace the read_only=True check + separate async_increment_tokens_with_ttl_preservation with a single atomic call passing the batch's (request_count, total_tokens) as the increment. - dynamic_rate_limiter_v3._check_rate_limits: bundle model_saturation_check (always enforced) and priority_model (enforced only when saturated) into one atomic call. When priority is unenforced, increment its counter via the existing should_rate_limit(read_only=False) path for tracking only. Update structural regression tests to assert the new atomic path is used rather than the legacy two-phase pattern. Tests: 4/4 TOCTOU tests pass, 59 existing rate-limiter tests pass, no regressions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
chore(proxy): tighten router-settings-override and mock-testing trust
… (VERIA-39) (#27015) * fix(batches): count non-chat tokens and validate every model in batch file Two security control bypasses on POST /v1/batches: 1. `_get_batch_job_input_file_usage` only summed tokens for `body.messages` (chat completions). Embedding (`input`) and text completion (`prompt`) batches reported zero, letting massive non-chat workloads slip past TPM rate limits. Extend the counter to handle string and list shapes for both fields. 2. The batch input file was forwarded to the upstream provider without inspecting the models named inside the JSONL — only the outer `model` query parameter was checked against the caller's allowlist. A caller restricted to gpt-3.5 could submit a batch targeting gpt-4o and the upstream would execute it under the proxy's shared API key. Add `_get_models_from_batch_input_file_content` (returns the distinct `body.model` values) and call it from `_enforce_batch_file_model_access` in the pre-call hook, which runs each model through `can_key_call_model` so the same allowlist semantics (wildcards, access groups, all-proxy-models, team aliases) the proxy enforces on `/chat/completions` apply here too. Any unauthorized model raises a 403 before the file is forwarded. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(batches): count pre-tokenized prompt/input shapes, classify 403 logs Two follow-ups from the Greptile review on the batch validation PR: 1. P1 TPM bypass via integer token arrays. The OpenAI batch schema accepts ``prompt`` and ``input`` as ``list[int]`` (a single pre-tokenized prompt) or ``list[list[int]]`` (multiple) in addition to the string and ``list[str]`` shapes. Pre-fix only the string shapes were counted, so a caller could submit a batch with hundreds of millions of pre-tokenized tokens and the rate limiter would record zero. Extract the per-field logic into ``_count_prompt_or_input_tokens`` and count each int as one token. 2. P2 access-denial logs were indistinguishable from I/O failures. ``count_input_file_usage`` caught every exception under a generic "Error counting input file usage" message, so an intentional 403 from ``_enforce_batch_file_model_access`` looked the same in the logs as a missing file or a Prisma timeout. Catch ``HTTPException`` separately and log 403s at WARNING level with a security-relevant message before re-raising. Tests cover the new shapes: single ``list[int]``, ``list[list[int]]`` (the worst-case bypass vector), and embeddings ``input`` with pre-tokenized arrays. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
) * fix(proxy): re-validate user_id ownership after /user/info re-parses query The route-level access check in `RouteChecks.non_proxy_admin_allowed_routes_check` reads `request.query_params.get("user_id")`, which decodes literal `+` to spaces. The endpoint then re-parses the raw query string with `urllib.unquote` in `get_user_id_from_request` to preserve `+` characters (so plus-addressed emails work as user_ids). Those two paths produce different ids: a caller who registered a user_id containing a literal space could pass the route check and then read another user's row by sending the encoded `+` form. Add `_enforce_user_info_access` and call it after `_normalize_user_info_user_id` returns the final id. Proxy admin / view-only admin still bypass; everyone else must match the resolved user_id (or have no user_id, which falls back to the caller's own id later in the handler). Tests cover the admin bypass, owner-match path, and the cross-user lookup that this change blocks. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(proxy): apply user_info ownership check to PROXY_ADMIN_VIEW_ONLY `_enforce_user_info_access` was bypassing both PROXY_ADMIN and PROXY_ADMIN_VIEW_ONLY, but the upstream route check in `RouteChecks.non_proxy_admin_allowed_routes_check` only treats PROXY_ADMIN as a true admin for the `/user/info` route — view-only admins go through the `user_id == valid_token.user_id` enforcement along with regular users. Mirroring that asymmetry left the same encoded-`+` bypass open for view-only admins whose user_id contains a literal space. Drop the PROXY_ADMIN_VIEW_ONLY exemption so the post-decode re-check matches the upstream rule. Update tests: a view-only admin must now be blocked from cross-user lookups but still allowed to read their own row. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fix(mcp): run pre_call_tool_check on OpenAPI/local-registry path (VERIA-7)
Fix runtime policy attachment initialization
Conflict resolution for #26968 dropped the `Iterator` typing import (NameError at module load), left a dead `fallback_models = cast(...)` block, and the new tests called `_enforce_key_and_fallback_model_access` without the now-required `request` kwarg.
fix(prometheus): escape api_key for PromQL string literal (VERIA-53)
[Fix] Isolate dual OTEL handlers
…jack fix(proxy): close project hijacking and key org IDOR (VERIA-55)
…te And Regenerate
Mirrors the membership rule on /key/update so that /key/generate and
/key/{key}/regenerate apply the same `_validate_caller_can_assign_key_org`
gate when the caller specifies an `organization_id`. Proxy admins bypass.
The check no-ops when `organization_id` is not being set.
chore(auth): require trusted proxy for header identity auth
chore(sso): bind generic SSO state to a session cookie
Three live-API tests pinned to claude-4-sonnet-20250514, which is a non-canonical alias of claude-sonnet-4-20250514. Anthropic's main API no longer resolves the legacy form under freshly issued keys, so the tests fail with not_found_error. The token counter test pinned to claude-sonnet-4-20250514 itself (deprecation_date 2026-05-14, two weeks out) was on borrowed time too. Bump all four to claude-haiku-4-5-20251001 — capability superset for what these tests exercise (streaming, parallel tool calling, extended thinking, token counting), no upcoming deprecation, cheaper per-token.
This file is a regenerable UI build artifact that should not be tracked in source. Removing so the merge into litellm_internal_staging stays clean.
…aiku 4.5 test_anthropic_messages_streaming_cost_injection hits the proxy's /v1/messages route, which routes via the anthropic/* wildcard to api.anthropic.com. The 404 surfaced in the test was Anthropic's own not_found_error propagated back through the proxy (visible from the x-litellm-model-id hash on the response — the proxy did route). Same root cause as the prior commit: the legacy claude-4-sonnet-20250514 alias is no longer recognized by Anthropic's main API under the new key. Swap to claude-haiku-4-5-20251001 — same routing path, canonical model.
…release_detection [Fix] Release Workflow: Detect SemVer-Style Pre-Release Dev Tags
… Alias base_anthropic_messages_test.test_anthropic_messages_with_thinking and test_anthropic_streaming_with_thinking still pinned to claude-4-sonnet-20250514 — the same legacy alias Anthropic no longer recognizes under freshly issued keys. The other four tests in this base class already use claude-sonnet-4-5-20250929; these two were missed. Bump to claude-haiku-4-5-20251001 (supports_reasoning=true, no upcoming deprecation). Subclasses including TestAnthropicPassthroughBasic inherit these methods.
[Infra] Merge dev branch
[Test] Anthropic: Replace Legacy Claude-4-Sonnet Alias With Haiku 4.5
…fication fix(auth): support JWT issuer verification + warn when unscoped
fix(router): constrain same-name deployment routing by access groups
…t_thinking fix(gemini): follow provider defaults for Gemini 3 thinking
feat(mcp): enforce org-level MCP server and toolset permissions
Contributor
|
Too many files changed for review. ( |
Sameerlite
merged commit May 2, 2026
a5a8f39
into
litellm_presidio-responses-stream-passthrough
34 of 56 checks passed
|
Michael Riad Zaky seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
fzowl
pushed a commit
to fzowl/litellm
that referenced
this pull request
Jun 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
19 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relevant issues
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
tests/test_litellm/directory, Adding at least 1 test is a hard requirement - see detailsmake test-unit@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewDelays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
CI (LiteLLM team)
Branch creation CI run
Link:
CI run for the last commit
Link:
Merge / cherry-pick CI run
Links:
Screenshots / Proof of Fix
Type
🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test
Changes