merge main - #26258
Merged
Sameerlite merged 96 commits intoApr 22, 2026
Merged
Conversation
Opt-in CLI flag, off by default, no env var. Only affects the uvicorn run path; gunicorn/hypercorn paths and prod (which doesn't pass the flag) are unaffected.
…itellm_feat-add-reload-flag-proxy
allow model routing to improve based on conversation signals ensures router is picking best model for task
[Infra] Promote staging to main
Adds a CI job that rebuilds the admin UI from source and fails if the committed static export at litellm/proxy/_experimental/out/ has drifted from what npm run build produces. This prevents silently shipping stale UI bytes and is a prerequisite for the non_root Dockerfile streamlining work, which will stage the UI from _experimental/out/ directly instead of rebuilding it inside the image. Also regenerates litellm/proxy/_experimental/out/ to match a fresh npm run build (Node 20.20.2) — the committed tree had drifted from source prior to this commit. Co-authored-by: yuneng-jiang <yuneng-berri@users.noreply.github.com>
The checked-in Next.js static export at litellm/proxy/_experimental/out/ is kept fresh by the UI Drift Guard CI workflow. Stage it directly instead of re-running npm ci + npm run build inside the image. This removes: nvm install, node 20.20.2 install, npm ci (801 pkgs), next build, and the resulting intermediate node_modules/out tree. Build time: ~6m25s -> ~2m (fuse-overlayfs DinD); image 6.57GB -> 5.0GB. Behavior parity verified: API endpoints, UI screenshots (all 10 routes pixel-perfect), and Trivy HIGH/CRITICAL CVE count (6 -> 5, one npm GHSA removed) all match or improve over baseline. Co-authored-by: yuneng-jiang <yuneng-berri@users.noreply.github.com>
npm was installed in the runtime only to globally install vulnerability patched versions of tar/glob/brace-expansion/minimatch/diff and to in-place rewrite npm's own bundled package.json. Both were to silence CVE scanners against modules that ship with npm itself. Since we no longer run npm anywhere in the runtime (Prisma uses the node binary directly for migrate deploy and generate), we can just skip installing npm in the first place. This eliminates both the ~25-line CVE-patch shuffle AND the underlying CVE surface. Kept: nodejs (needed by prisma-python's CLI and migrate deploy). Removed: npm apk package, all 'npm install -g', all find+sed patching, the redundant 'apk upgrade --no-cache nodejs' (already covered by the preceding 'apk upgrade'). Image: 4.97GB (opt-1) -> 4.97GB (opt-2); the real win is that two CVEs (CVE-2026-33671 and GHSA-q4gf-8mx6-v5v3) drop off the Trivy HIGH/CRITICAL list. No new CVEs introduced. API parity and UI visual regression both match baseline. Co-authored-by: yuneng-jiang <yuneng-berri@users.noreply.github.com>
After Task 2.1 removed the in-image Next.js build, the builder stage no longer needs a full C/C++ + Clang toolchain. Keep gcc + python3-dev (required to compile ml-dtypes 0.4.1 from source — no wheel published for Python 3.13 yet). Drop everything else. Removed from apk: clang, llvm, lld, linux-headers, build-base, openssl-dev, npm. Removed NVM_DIR env and /root/.nvm from PATH (no nvm-based Node install anymore). Kept: python3, python3-dev, gcc, bash, coreutils, curl, openssl, libsndfile, nodejs. gcc (15.2) serves both C and C++; the separate g++ package doesn't exist in Wolfi. Image size unchanged (builder stage doesn't end up in the runtime); cold builds slightly slower due to ml-dtypes source compile, but that will be recovered in the next task via a BuildKit uv cache mount. API parity and UI visual regression both match baseline, Trivy HIGH/CRITICAL CVE count unchanged from opt-2 (4 CVEs, none new). Co-authored-by: yuneng-jiang <yuneng-berri@users.noreply.github.com>
Mount /app/.cache/uv as a BuildKit type=cache on both 'uv sync' steps. The cache persists across builds on the same builder (and, when used with type=gha in CI, across CI runs) so repeat builds don't re-download every wheel. Side-effect: because the cache lives outside the image layer, the ~742MB of downloaded wheel archives that were previously baked into /app/.cache/uv drop out of the final image. Compressed image size goes from ~5.0GB to ~3.7GB, and the 'USER nobody' prisma-generate layer is 1.7GB vs 2.4GB. Warm-build timing: a uv-sync-invalidating edit now takes ~1m30s vs ~2m39s without the cache mount, on this dev VM. API parity and UI visual regression continue to match baseline. Trivy HIGH/CRITICAL: 6 at baseline -> 2 now, no new CVEs. Co-authored-by: yuneng-jiang <yuneng-berri@users.noreply.github.com>
Five small, individually-verified cleanups collected into one commit: - Drop 'prisma migrate diff --from-empty ... > /dev/null 2>&1 || true' from the builder. Stdout/stderr/exit-status all discarded; nothing reads the output. Dead line. - Drop 'mkdir -p /app/.cache/npm' from the same RUN. npm is gone. - Drop the runtime's redundant 'sed -i' + 'chmod +x' on the entrypoint scripts. The builder already does the same three lines, and the runtime copies /app from the builder via COPY --from=builder, so the normalized files (and exec bits, which buildkit preserves) are already in place. - Drop NPM_CONFIG_CACHE and NPM_CONFIG_PREFER_OFFLINE from the runtime ENV — nothing reads them after Task 2.2 removed npm. - Drop '/.npm' and '/tmp/.npm' from the runtime's mkdir + chown. These directories only existed as npm's writable dirs for the non-root user; npm is gone. .dockerignore: add 'ui/'. After Task 2.1 the non_root image sources its UI bytes from litellm/proxy/_experimental/out/, so the whole ui/litellm-dashboard/ source tree is dead weight when the blanket 'COPY . .' pulls it into /app. Verified (with ripgrep) that no Python code under litellm/ opens any file under ui/. All string references to 'ui/...' are URL paths, not filesystem paths. Final image size: 6.57GB baseline -> 1.96GB. API parity and UI visual regression match baseline across all 12 API scenarios and 10 UI routes. Trivy HIGH/CRITICAL: 6 -> 2, no new CVEs introduced. Co-authored-by: yuneng-jiang <yuneng-berri@users.noreply.github.com>
Remove _experimental/out/ changes from this PR — these are auto-generated Next.js build outputs, not part of the adaptive router feature. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Unrelated timestamp and version drift was showing in the PR diff. This PR adds no new deps — keep uv.lock identical to main. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Router coverage check flagged this method as untested. Adds two cases: - initializes AdaptiveRouter from model_list and is idempotent on re-entry - no-op when no adaptive deployments are configured Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…roxy feat(proxy): add --reload flag for uvicorn hot reload (dev only)
Revert the .dockerignore ui/ exclusion and remove the UI Drift Guard workflow. _experimental/out/ refresh is already handled by the release runbook; the global .dockerignore change also broke Dockerfile.custom_ui (explicit COPY ./ui/litellm-dashboard) and the enterprise-colors inline rebuild path in Dockerfile, Dockerfile.database, and Dockerfile.dev. Dockerfile.non_root itself is unchanged functionally — still stages the UI from the checked-in _experimental/out/. Only the companion workflow and global dockerignore exclusion are dropped.
prisma --version invokes the Schema Engine, which has no binary for the Wolfi base image (only debian). In the baseline this was silenced by a trailing || true wrapping the whole prisma chain; removing that wrapper uncovered the failure on arm64 builds. The main Dockerfile does not call prisma --version at all, so drop it here to match — prisma generate is sufficient to validate the toolchain.
* feat(scaleway): add SCALEWAY to LlmProviders enum * feat(scaleway): add audio transcription config and dispatch wiring Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test(scaleway): add behavior tests for audio transcription config Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore(scaleway): advertise audio_transcriptions in endpoint-support JSON * docs(scaleway): document audio transcription support * fix(scaleway): address PR review — plain-text response_format + missing-key fail-fast Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test(scaleway): cover new response paths, drop gettysburg.wav coupling Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
- Use LoggingCallbackManager.add_litellm_callback instead of litellm.callbacks.append (required by callback_manager_test) - init_adaptive_router_deployment now uses model_name_to_deployment_indices for O(k) lookup instead of scanning model_list - Rephrase comment in set_model_list to avoid the 'in self.model_list' substring that the linear-scan test greps for - Whitelist _finalize_adaptive_router_if_configured in test_no_linear_scans_in_router — prefix match on 'auto_router/adaptive_router' has no supporting index; runs once at init Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: new agentic loop event hook simplifies how to create logic for tool based multi llm calls * fix: compress - make it work on anthropic input as well * fix(compress.py): working prompt compression for claude code ensures claude code messages can run through proxy easily * docs: add agentic loop hook guide * docs: add agentic_loop_hook to sidebar * fix: fix multiple arguments error * fix: fix tool call loop for compression on streaming /v1/messages * fix: fix linting errors * fix: fix ci/cd errors * feat(litellm_pre_call_utils.py): use claude code session for litellm session id allows claude code logs to be stitched together, making it easy to know they were all part of the same conversation * fix: suppress incorrect mypy warning rE: module * revert: drop PR's changes to litellm/proxy/_experimental/out/ Restores the 34 HTML files under _experimental/out/ to their pre-PR paths (X/index.html -> X.html). All renames are R100 (content unchanged); no other files are touched. * fix: address greptile review comments on PR #25729 - Skip ``kwargs["tools"] = []`` injection when compression is a no-op — Anthropic Messages rejects empty tool arrays on requests that did not originally declare tools. - Move agentic-loop safety guards (fingerprint cycle / max depth) out of the per-callback try/except so they propagate instead of being swallowed by the generic exception handler. Extracted _check_agentic_loop_safety. - Gate generic ``x-<vendor>-session-id`` capture behind the LITELLM_CAPTURE_VENDOR_SESSION_HEADERS env var (off by default) to preserve backwards compatibility; explicit x-litellm-* headers are unaffected. - Fix monkeypatch target in pre-call-hook test to patch the actual module-level binding (litellm.integrations.compression_interception.handler.compress). - Add regression tests for empty-tools skip and opt-in session capture. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * revert: drop LITELLM_CAPTURE_VENDOR_SESSION_HEADERS flag Generic x-<vendor>-session-id header capture is a new feature and only runs *after* the explicit x-litellm-trace-id / x-litellm-session-id checks, so it does not change behavior for any existing caller that was already using the LiteLLM headers — no backwards-incompatibility to gate. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(compress): replace input_type with CallTypes call_type Drop the bespoke ``CompressionInputType`` literal and use the existing ``litellm.types.utils.CallTypes`` enum instead. ``litellm.compress()`` now takes ``call_type: Union[CallTypes, str]`` (default ``CallTypes.completion``) — no new concept to learn, and the enum is already the way the rest of the codebase talks about request shapes. Supported values: ``completion`` / ``acompletion`` (OpenAI chat-completions shape) and ``anthropic_messages`` (Anthropic structured content blocks). Updated: compress(), the compression_interception handler, tests, docs, and the two eval scripts. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Use 'auto_router/adaptive_router' prefix in example yaml, docs, and README — the old 'adaptive_router/...' and 'openai/gpt-4o-mini' values silently skipped adaptive-router init because detection requires the 'auto_router/adaptive_router' prefix. - Read x-litellm-min-quality-tier from request headers (and the 'min_quality_tier' metadata key as fallback) in async_pre_routing_hook. Previously the documented header was defined but never extracted, so the quality-floor feature was inert. - Evict expired entries from _session_states. The cache grew without bound — added a parallel expiry map (same TTL as _owner_cache) and an opportunistic bulk sweep when the cache crosses a size threshold. - Align adaptive-router migration SQL with Prisma schema: all count columns and the 'clean_credit_awarded' / 'last_processed_turn' fields are NOT NULL in the data model, so the migration now declares them NOT NULL. Fixes test_aaaasschema_migration_check. Tests: 8 new covering header/metadata/precedence/invalid-value paths for min_quality_tier and TTL-based eviction of _session_states. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add supported providers to prompt caching doc * Move Z.ai / GLM to cache_control marker list * Mark xAI models as supporting prompt caching * Narrow xAI prompt caching flag to models with documented cache pricing * Add prompt caching flag to grok-4, grok-4-0709, grok-4-latest --------- Co-authored-by: Michael Riad Zaky <michaelr@Michaels-MacBook-Air.local>
…ires
The post-call hook was hardcoding tool_results=[] on every Turn, so the
failure detector never saw tool errors and the bandit only learned from
satisfaction — never from negative tool outcomes.
Added _recent_tool_results(messages): walks the request messages from the
tail and collects the contiguous run of role=='tool' entries — those are
the results from the most recent assistant tool_calls round. Normalizes
each to {content, is_error}, the only fields signals._detect_failure /
_detect_exhaustion read.
Tests: 6 new covering empty input, trailing-run extraction, is_error
propagation, boundary at first non-tool message, no-trailing-tool case,
and the end-to-end path from hook -> Turn.tool_results.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: /health/readiness returns 503 when DB is unreachable due to handle_db_exception re-raising handle_db_exception() re-raises the Prisma exception inside _db_health_readiness_check's except block, which propagates out to health_readiness() and gets wrapped in a 503. The health endpoint never reached the reconnect path and the service never recovered. Fix: - Remove handle_db_exception() call from _db_health_readiness_check — that helper is for API request handlers (allow_requests_on_db_unavailable flag), not health checks - Replace raw disconnect()+connect() with attempt_db_reconnect(), which uses the proper lock, cooldown, escalation, and heavy-reconnect (recreate_prisma_client) machinery * test: update health readiness tests for handle_db_exception removal - Remove tests that expected handle_db_exception to re-raise (old buggy behaviour) - Remove tests asserting disconnect()/connect() calls (replaced by attempt_db_reconnect) - Add regression tests covering the 503 loop fix: - transport errors never raise (ClientNotConnectedError, httpx.ConnectError, etc.) - reconnect success path returns 'connected' - reconnect failure path returns 'disconnected' without raising - non-transport errors return 'disconnected', skip reconnect --------- Co-authored-by: yuneng-jiang <yuneng@berri.ai>
…nbook The UI rebuild bundled into this PR is not needed for the Dockerfile change — the image simply copies whatever _experimental/out/ is in the tree. Regenerating here conflicts with the release-time refresh policy and adds ~100 files of review noise / merge-conflict risk for any concurrent UI PR.
…1 in anthropic messages passthrough test Anthropic retired claude-3-haiku-20240307 on 2026-04-20, causing the test_anthropic_messages_litellm_router_non_streaming_with_logging test to 404. Update the model references in this file to the current pinned haiku version.
[Feature] Proxy: opt-in v2 migration resolver
P1: start the adaptive-router flusher loop unconditionally at proxy boot
instead of gating on 'adaptive_routers is non-empty'. Adaptive routers
added via /config/reload after boot now have their queues drained.
State is lazy-loaded per router on first flush tick (new _state_loaded
flag on AdaptiveRouter) so hot-reloaded routers still get their
persisted priors.
P2: _finalize_adaptive_router_if_configured now prunes stale
AdaptiveRouterPostCallHook callbacks from every litellm callback list
before registering new ones. Without this, every Router replacement
left the old hooks wired up in litellm.callbacks and double-fired
signal recording for every request. Uses
logging_callback_manager.remove_callbacks_by_type (same pattern as the
semantic tool filter).
CI fixes:
- black --check failure: reformatted litellm/router.py
- schema migration diff: aligned @@index with the explicit index name
('idx_adaptive_router_session_activity') from the original migration
by adding 'map:' to all three schema.prisma copies. No new migration
needed.
Tests: 1 new covering the prune-on-hot-reload path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fix(bedrock_guardrails): use Bedrock OUTPUT source for apply_guardrail when scanning model responses
The prevent_key_leaks_in_exceptions CI check forbids '{args}' in
f-strings because it's a common shape for accidental API key leaks
in exception messages. _signature() uses an entirely local variable
named 'args' for tool-call arguments (loop-detection signatures, no
exception path), but the grep is substring-based. Rename to 'call_args'.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…he team default Previously, members added to a team without an explicit per-member budget were all linked to the same `litellm_budgettable` row referenced by the team's `metadata.team_member_budget_id`. Updating one member's budget via `/team/member_update` mutated the shared row and silently changed every other member's budget too. Now both write paths produce a private, per-member budget: - `add_new_member` clones the team's default budget into a fresh row when a member is added without `max_budget_in_team`/`allowed_models`. If no team default exists, the membership is created with no budget. - `_upsert_budget_and_membership` detects when an existing membership still points at the team's default budget id and clones-on-write, relinking the membership to the new private budget before applying the update. - `team_member_update` reads `team_member_budget_id` from team metadata and passes it through so the helper can make this distinction. Adds unit tests for clone-on-write, in-place update of a private budget, and the no-default-no-budget add path. Made-with: Cursor
P1 review: adaptive_router.py had a top-level import of
AdaptiveRouterUpdateQueue from litellm.proxy.db, which broke the
SDK/proxy boundary that every other router strategy respects. No
other router_strategy module imports from litellm.proxy at module
level.
The queue only depends on litellm._logging — it never needed to
live under litellm.proxy. Moved:
litellm/proxy/db/db_transaction_queue/adaptive_router_update_queue.py
→ litellm/router_strategy/adaptive_router/update_queue.py
tests/test_litellm/proxy/db/db_transaction_queue/
test_adaptive_router_update_queue.py
→ tests/test_litellm/router_strategy/adaptive_router/test_update_queue.py
Also switched the queue's logger from verbose_proxy_logger to
verbose_router_logger to match the new module's ownership.
P2 review: drop unused constant STAGNATION_JACCARD_EXACT from
config.py — it was defined but never referenced.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
[Infra] Bump version
…-budgets Litellm individual team member budgets
* add moonshot/kimi-k2.6 to model registry * add moonshot/kimi-k2.6 to backup model registry * add tests for moonshot/kimi-k2.6 model registry * fix moonshot/kimi-k2.6 pricing and add reasoning support * fix moonshot/kimi-k2.6 pricing and add reasoning support in backup * update kimi-k2.6 tests: fix pricing, add tool_choice and reasoning checks * fix: load kimi-k2.6 registry tests from local backup instead of remote cost map
* fix(otel): preserve Splunk Observability Cloud trace OTLP endpoint (#26183) * fix(otel): preserve Splunk Observability Cloud trace OTLP URL Splunk ingest uses /v2/trace/otlp; _normalize_otel_endpoint must not append /v1/traces. - Return trace endpoints unchanged when they match Splunk OTLP path patterns - Add unit tests for observability.splunkcloud.com, signalfx.com, and /trace/otlp suffix - Set OTEL_EXPORTER_OTLP_PROTOCOL in protocol selection tests (from_env precedence over OTEL_EXPORTER) Made-with: Cursor * test(otel): use parameterized.expand for Splunk OTLP URL cases Made-with: Cursor * fix(otel): narrow Splunk trace URL guard to /v2/trace/otlp only Made-with: Cursor * test(otel): cover OTEL_EXPORTER fallback when OTLP protocol env unset Made-with: Cursor * Add Openrouter Opus 4.7 Entry (#26130) --------- Co-authored-by: milan-berri <milan@berri.ai> Co-authored-by: Matt Greathouse <matt5316@gmail.com>
Litellm adaptive routing
Contributor
|
Too many files changed for review. ( |
Sameerlite
merged commit Apr 22, 2026
221dc2f
into
litellm_vertex_image_edit_credentials_fix
92 of 93 checks passed
|
|
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
yuneng-berri
temporarily deployed
to
integration-postgres
April 22, 2026 18:17 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
yuneng-berri
temporarily deployed
to
integration-postgres
April 22, 2026 18:17 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
yuneng-berri
temporarily deployed
to
integration-postgres
April 22, 2026 18:17 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
yuneng-berri
temporarily deployed
to
integration-postgres
April 22, 2026 18:17 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
yuneng-berri
temporarily deployed
to
integration-postgres
April 22, 2026 18:17 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
yuneng-berri
temporarily deployed
to
integration-postgres
April 22, 2026 20:47 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
yuneng-berri
temporarily deployed
to
integration-postgres
April 22, 2026 20:47 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
yuneng-berri
temporarily deployed
to
integration-postgres
April 22, 2026 20:47 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
yuneng-berri
temporarily deployed
to
integration-postgres
April 22, 2026 20:47 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
yuneng-berri
temporarily deployed
to
integration-postgres
April 22, 2026 20:47 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
fzowl
pushed a commit
to fzowl/litellm
that referenced
this pull request
Jun 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
13 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relevant issues
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
tests/test_litellm/directory, Adding at least 1 test is a hard requirement - see detailsmake test-unit@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewDelays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
CI (LiteLLM team)
Branch creation CI run
Link:
CI run for the last commit
Link:
Merge / cherry-pick CI run
Links:
Screenshots / Proof of Fix
Type
🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test
Changes