chore(ci): promote internal staging to main - #36286
Merged
Merged
Conversation
…ead of IndexError 500
…ing chunks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…nd delete Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Bedrock managed-batch file upload read `messages` unconditionally, so a JSONL record shaped for /v1/completions (`prompt`) or /v1/responses (`input`) reached the per-provider transform with an empty message list. Anthropic and Nova rejected it at POST /v1/files, and the passthrough providers shipped an empty conversation to AWS. Classify each record by its OpenAI batch `url`, then normalize the non-embedding shapes to chat completions before the Bedrock transforms: `prompt` wraps into user messages the way litellm.text_completion does in real time, and `input` goes through the existing Responses-to-Chat bridge. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
list_user_batches parsed each stored batch blob and returned it as-is, so any
row whose blob still carried raw provider file ids (for example a batch that
reached a terminal state through the cost poller, or rows written before
output registration existed) leaked raw output_file_id and error_file_id
values that clients cannot fetch through the proxy. The list path now runs
each row through ensure_batch_response_managed_file_ids, which swaps in
existing managed ids and registers missing ones under the batch owner's
identity, matching what GET /batches/{id} already does
…itellm_vertex_batch_create_error_propagation
…raw file ids in one query
…itellm_list_batches_resolves_unified_ids # Conflicts: # enterprise/litellm_enterprise/proxy/hooks/managed_files.py
…_choices Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…itellm_wt_35148_merge
The staging merge tightened the LIT002 ceiling, so the three mutable dict literals this branch added now breach it. Build the S3 request headers as MappingProxyType and resolve the encryption key from a tuple of sources.
… against the calling key An intercepted web search called litellm.asearch() with only the search tool's litellm_params, so the search request carried no owner. The proxy's spend hook skips any call with no key, user or team attached, so the search's provider cost never reached SpendLogs; it was missing from the Logs page and never counted against the caller's budget. The same path never ran the rate limiter either, so an intercepted search was free of the key's RPM/TPM limits. The search now carries the originating key's attribution metadata (key hash, alias, user, team, org, plus model_group set to the resolved search tool) and runs the caller's rate limit checks before hitting the provider, matching what a direct /v1/search request gets. SDK calls with no proxy auth context are unchanged. Resolves LIT-5033 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
test_get_deployment_credentials_with_provider_bedrock_batch_fields already covers s3_encryption_key_id on the base branch, and the new test passes with every production file in this branch reverted, so it guards nothing.
… and audience (#36137) The Azure Sentinel logger hardcoded the commercial Entra authority and the commercial Azure Monitor audience, so Log Analytics ingestion could not work in Azure Government even when the ingestion endpoint pointed at a sovereign Data Collection Endpoint. Resolve the authority from AZURE_AUTHORITY_HOST and derive the matching Logs Ingestion audience from it. Moving only the token URL is not enough: sovereign Entra would then be asked for a token scoped to the commercial audience, which the sovereign endpoint rejects.
fix(bedrock): pass SSE-KMS key through to the batch input-file S3 upload
…ices fix(anthropic adapter): stop indexing choices[0] on choiceless streaming chunks
…_records fix(bedrock): normalize /v1/completions and /v1/responses batch records
…ejected (#36166) * fix(proxy): return the real status code when a credential update is rejected update_credential ended its except clause with 'return handle_exception_on_proxy(e)'. Returning the exception makes it the response body, so FastAPI answers 200 and every rejection on this route reads as a successful write to any caller that checks the status; the admin dashboard's API client is one. Patching a name that does not exist answered 200 with the real 404 buried in the body. The sibling handlers in this file already raise. The route had no test coverage, which is why it survived. * Update tests/test_litellm/proxy/credential_endpoints/test_endpoints.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
…lete (#36028) * fix(proxy): invalidate cached project object on /project/update and /project/delete The auth path reads projects cache-first via get_project_object with a 60s TTL and no freshness check, but no project write endpoint ever evicted the project_id:{id} cache entry. A project cached before /project/update added a model allowlist kept an empty models list in cache, so _run_project_checks skipped can_project_access_model and project-bound keys could call team models outside the project allowlist until the TTL expired. The same staleness applied to blocked status and budget fields, and /project/delete left the deleted project enforceable from cache. Evict the cache entry after the DB write in update_project and delete_project via a shared delete_cached_project_object helper, with the cache key derivation shared with get_project_object. * fix(proxy): broadcast project cache invalidation to all workers and make eviction best-effort Single-worker eviction leaves every other worker serving its in-memory copy of the mutated project until the 60s TTL expires, so a project allowlist change was still bypassable on multi-worker deployments. Add a coordination Redis pub/sub channel (litellm_proxy.auth_cache_invalidation): project eviction publishes the cache key and a per-worker subscriber deletes the local in-memory entry, with the next auth read refetching from the DB. Subscriber starts on any deployment with a coordination Redis and falls back to the TTL when none is configured. Also wrap the eviction in a best-effort catch: the DB write has already committed when eviction runs, so a cache backend error must not turn a successful update into a 500 or abort the remaining ids in /project/delete. * fix(lint): sort auth cache invalidation import and suppress best-effort shutdown catch The strict-budget gate flagged the new import block as un-sorted (I001) and the broad except in stop_auth_cache_invalidation_subscriber (BLE001); the catch is intentional since a failing stop must not break proxy shutdown, so it carries a named suppression instead of counting against the budget.
Contributor
PR overviewThis pull request promotes internal staging changes to main, including updates to Bedrock guardrail content processing and dependency-scanner configuration. Three security issues remain open, with none addressed so far. An authenticated caller can amplify a large prompt into many sequential Bedrock requests, consuming shared guardrail quota and tying up request tasks; oversized inputs can also bypass phrase detection at chunk boundaries. A known vulnerable PDF dependency remains allowed despite a reachable ingestion path that can be abused for CPU or memory exhaustion. Open issues (3)
Fixed/addressed: 0 · PR risk: 6/10 |
chore(typing): clear 1.4k basedpyright Any errors across 21 hotspot files
|
|
10 tasks
Since #35491, every Router joins the module-global _live_routers weak set at construction, and every model cost map swap replays the deployments of every member on top of the freshly adopted map. #36039 isolated the register_model ledger half of that replay but not this half: under pytest-xdist, a Router created by an earlier test in the same worker that was still referenced (or simply not yet garbage collected) re-registered its deployments during TestPriceDataReloadIntegration::test_distributed_reload_check_function, and register_model hydrated the sparse mocked gpt-3.5-turbo entry into a full ModelInfo dict, failing the exact-equality assert (reruns cannot help since the polluting router survives in the worker process) The autouse isolate_litellm_state fixture now snapshots _live_routers before each test and restores its membership on teardown, so a test's routers stop contributing to cost map rebuilds once the test ends. A canary pair in test_conftest_isolation.py asserts the rollback
…l_staging chore(ci): sync main into internal staging
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
| @@ -1,3 +1,13 @@ | |||
| [[IgnoredVulns]] | |||
| id = "GHSA-fwg2-594c-jp42" | |||
Contributor
There was a problem hiding this comment.
Low: Reachable pypdf denial of service is ignored
The lock resolves pypdf 6.14.2, which is affected by this advisory and GHSA-fp3f-mc75-235c; both are fixed in 6.15.0. An authenticated caller can upload a crafted PDF to /rag/ingest, where its bytes are passed to PdfReader and page.extract_text(), causing excessive CPU or memory consumption while these ignores prevent the dependency scanner from blocking the release. Upgrade to 6.15.0 before suppressing the scan, or defer this change until the lockfile's exclusion window permits that version.
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
…evin_ai_lit_5033_websearch_interception_spend # Conflicts: # litellm/integrations/websearch_interception/handler.py
build(lint): rename make pre-commit to make check with a working-tree fallback
* fix(ui): show team BYOK models in team fallback settings Team router settings loaded fallback options from /model_group/info, which resolves models without a team, so a team's own BYOK deployments were never selectable in its own fallback config. Load the team-scoped listing when a team id is present. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): ignore stale team model responses in router settings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(ui): use react-query for fallback model listing in router settings accordion --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
| self.chunk_budget_chars, | ||
| ) | ||
| batch_results: Final = [ # mutable-ok: await needs a list comprehension; frozen to a tuple below | ||
| await self._apply_guardrail_content_with_chunking( |
Contributor
There was a problem hiding this comment.
Medium: Unbounded guardrail request amplification
There is no limit on the number of batches or recursive splits processed for one request, and each resulting chunk can also receive multiple throttle retries. An authenticated caller can submit a sufficiently large prompt that fans out into hundreds or thousands of sequential Bedrock calls, consuming the shared guardrail quota and tying up request tasks. Cap the total chunks and attempts per logical request, and reject content that would exceed that limit.
some users do not use make pre-commit as it is a multi-minute process. I personally use it but I want users themselves to decide whether to pre-commit before each commit or not, based on what works best for them
* fix(otel): mark v2 server spans as failed for pre-call errors (LIT-4780) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): authenticate malformed-body requests before rejecting them (LIT-4780) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(auth): cover malformed-body rejection when auth error is recovered Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(auth): skip authorization for a request whose body never parsed Deferring the parse failure ran the full auth phase, including budget reservation, whose reserved amount is only released by the endpoint's post call path; the endpoint never runs, so malformed requests leaked reservations and locked a budgeted key out. Authorization now runs only when the body parsed, and a parse failure with a rejected key keeps returning the 400 it returned before. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: shivam <shivam@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…terception_spend fix(websearch_interception): bill intercepted searches to the calling key
chore: remove pre-commit rule
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…mportance docs: clarify guideline priority ordering in CLAUDE.md
…dated_tests test: repair stale CircleCI contracts
yucheng-berri
approved these changes
Aug 8, 2026
shin-berri
approved these changes
Aug 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
11 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
Problem this solves:
How it solves it:
User Flow
Relevant issues
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Type
🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test
Changes
QA runbook
Final Attestation