merge main - #26379
Merged
Sameerlite merged 149 commits intoApr 24, 2026
Merged
Conversation
…th team-scoped deployments When vector store endpoints (POST/GET /v1/vector_stores) are called, model=None is passed to the router. map_team_model(None, team_id) was returning None unchanged after the team model routing fix in #25148, so the router never found the team's BYOK deployment and forwarded requests without the API key. Fix: when team_model_name is None, return the matched deployment's team_public_model_name (or model_name fallback) so the router can route to it and inject the BYOK credentials. Does not affect the sibling-deployment load-balancing fix since that only applies when a non-None model is passed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… litellm_vector-store-team-byok-model-none
…transformation logic and tests
Use wildcard-aware deployment lookup when building order-based fallback levels so requests like openai/gpt-4.1-mini can advance from order=1 to order=2, and add a regression test for wildcard routing. Made-with: Cursor
allow model routing to improve based on conversation signals ensures router is picking best model for task
Remove _experimental/out/ changes from this PR — these are auto-generated Next.js build outputs, not part of the adaptive router feature. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Unrelated timestamp and version drift was showing in the PR diff. This PR adds no new deps — keep uv.lock identical to main. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Router coverage check flagged this method as untested. Adds two cases: - initializes AdaptiveRouter from model_list and is idempotent on re-entry - no-op when no adaptive deployments are configured Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Use LoggingCallbackManager.add_litellm_callback instead of litellm.callbacks.append (required by callback_manager_test) - init_adaptive_router_deployment now uses model_name_to_deployment_indices for O(k) lookup instead of scanning model_list - Rephrase comment in set_model_list to avoid the 'in self.model_list' substring that the linear-scan test greps for - Whitelist _finalize_adaptive_router_if_configured in test_no_linear_scans_in_router — prefix match on 'auto_router/adaptive_router' has no supporting index; runs once at init Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Use 'auto_router/adaptive_router' prefix in example yaml, docs, and README — the old 'adaptive_router/...' and 'openai/gpt-4o-mini' values silently skipped adaptive-router init because detection requires the 'auto_router/adaptive_router' prefix. - Read x-litellm-min-quality-tier from request headers (and the 'min_quality_tier' metadata key as fallback) in async_pre_routing_hook. Previously the documented header was defined but never extracted, so the quality-floor feature was inert. - Evict expired entries from _session_states. The cache grew without bound — added a parallel expiry map (same TTL as _owner_cache) and an opportunistic bulk sweep when the cache crosses a size threshold. - Align adaptive-router migration SQL with Prisma schema: all count columns and the 'clean_credit_awarded' / 'last_processed_turn' fields are NOT NULL in the data model, so the migration now declares them NOT NULL. Fixes test_aaaasschema_migration_check. Tests: 8 new covering header/metadata/precedence/invalid-value paths for min_quality_tier and TTL-based eviction of _session_states. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ires
The post-call hook was hardcoding tool_results=[] on every Turn, so the
failure detector never saw tool errors and the bandit only learned from
satisfaction — never from negative tool outcomes.
Added _recent_tool_results(messages): walks the request messages from the
tail and collects the contiguous run of role=='tool' entries — those are
the results from the most recent assistant tool_calls round. Normalizes
each to {content, is_error}, the only fields signals._detect_failure /
_detect_exhaustion read.
Tests: 6 new covering empty input, trailing-run extraction, is_error
propagation, boundary at first non-tool message, no-trailing-tool case,
and the end-to-end path from hook -> Turn.tool_results.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Six tests in test_hooks.py were written against an older API and had been failing in CI. Updated: - test_resolve_session_key_* (4 tests): _resolve_session_key now requires at least SIGNAL_GATE_MIN_MESSAGES messages before deriving a hash (it returns None on shorter convos to match the signal-processing gate). Switched the tests to use _long_messages() so they hit the hash path. - test_post_call_success_hook_* (2 tests): the hook was migrated from async_post_call_success_hook (mutates response._hidden_params) to async_post_call_response_headers_hook (returns a headers dict) because the former fires too late for streaming responses. Rewrote the tests against the new API; added a metadata-not-dict noop case. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…empty tool output - SessionState now carries clean_credit_awarded + last_processed_turn (matching the DB schema). Satisfaction only fires once per session AND only after MIN_TURNS_FOR_CLEAN_CREDIT turns of context — early "thanks" no longer inflates alpha. - _detect_failure no longer treats empty content as failure. Many tools legitimately return empty output (zero-result searches, silent bash); penalizing those corrupted the bandit posterior. Only is_error fires now. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…redact PII - _owner_cache now opportunistically sweeps expired entries past _OWNER_CACHE_SWEEP_THRESHOLD live entries. Previously sessions that never came back piled up forever. - flush_session_to_db strips session_id/router_name/model_name from the update payload. Prisma rejects writes to @@id fields. - record_turn no longer persists last_user_content / last_assistant_content / tool_call_history / pending_tool_calls. Those are needed only in-memory for the next turn's signal detection; writing user prompts and tool payloads to the DB would store PII for every conversation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ertex AI credentials When aimage_edit or image_edit was called with Vertex AI Gemini/Imagen models via YAML-style config (vertex_project / vertex_credentials in proxy YAML), the credentials were dropped during handler-to-config plumbing, causing fallback to Application Default Credentials and DefaultCredentialsError. Root cause: image_edit_handler and async_image_edit_handler did not pass litellm_params to validate_environment, unlike image_generation_handler. Fixes: 1. Widen BaseImageEditConfig.validate_environment signature to accept litellm_params and api_base (optional kwargs). 2. Forward dict(litellm_params) and litellm_params.api_base from both sync and async image_edit handlers to validate_environment. 3. Update VertexAIImagenImageEditConfig.validate_environment to read vertex_ai_project/vertex_ai_credentials from litellm_params first, matching Gemini config pattern (secondary latent bug fix). 4. Widen all image-edit config override signatures to match base. Made-with: Cursor
Adds three test cases to prevent regression of the Vertex AI image_edit credentials bug: 1. test_validate_environment_signature_includes_litellm_params: ensures all image-edit configs accept litellm_params (contract for the handler) 2. test_vertex_gemini_image_edit_reads_credentials_from_litellm_params: verifies Gemini config reads from litellm_params first 3. test_vertex_imagen_image_edit_reads_credentials_from_litellm_params: verifies Imagen config reads from litellm_params first These tests catch if the fix is accidentally reverted or if new image-edit configs are added without the litellm_params parameter. Made-with: Cursor
…Imagen get_complete_url VertexAIImagenImageEditConfig.get_complete_url was resolving vertex_project and vertex_location only from env vars and global settings, ignoring litellm_params. Users supplying project/location exclusively via YAML config would get a ValueError or wrong URL even after auth headers were fixed. Mirrors the pattern already used by VertexAIGeminiImageEditConfig and image_generation counterpart (safe_get_vertex_ai_project/location). Also fixes api_key type hint in MockImageEditConfig (str -> Optional[str]) and adds a test covering get_complete_url credential resolution. Made-with: Cursor
Adds total_spend column to LiteLLM_TeamMembership that accumulates continuously and is not zeroed by the budget cycle reset job. This enables UI surfaces to distinguish current-cycle spend (the existing spend column, which resets) from lifetime spend per team member. Also exposes budget_reset_at on LiteLLM_BudgetTable so /team/info callers can see when a member's budget window next resets. The field was already stored in the DB but stripped by the response Pydantic model. Includes regression tests that: - Guard the reset job against ever writing total_spend: 0 - Verify the spend writer increments both spend and total_spend in one UPDATE statement.
Made-with: Cursor
The uv migration added PRISMA_BINARY_CACHE_DIR=/app/.cache/... and XDG_CACHE_HOME=/app/.cache to the runtime stages of Dockerfile and Dockerfile.database. BINARY_PATHS in the generated prisma client was baked to point into /app/.cache, so any deployment that mounts a volume there (common with securityContext.readOnlyRootFilesystem: true and an emptyDir/tmpfs for a writable cache) wipes the pre-downloaded query engine at pod startup, producing BinaryNotFoundError during connect(). Before the uv migration, prisma-python defaulted to $HOME/.cache = /root/.cache (runtime stage runs as root), which was unaffected by any /app/* volume mounts. Restore that behaviour: drop the env vars from the runtime stage, re-run prisma generate there so the query engine AND the baked BINARY_PATHS both land in /root/.cache, and remove the stale builder-stage /app/.cache (~800 MB). Dockerfile.non_root is intentionally left alone — its /app/.cache location is by design for the hardened offline-install flow.
LiteLLM_BudgetTable is documented as "user-controllable params" and its model_fields.keys() is used as the allowlist for extracting budget fields from incoming API request bodies (management_helpers/utils.py:88, organization_endpoints.py:112/255/537/549, project_endpoints.py:197/245/632, customer_endpoints.py:598). Request models like NewOrganizationRequest inherit from LiteLLM_BudgetTable, so anything on the base class becomes user-settable — a caller could set budget_reset_at far in the future and evade budget cycling. Move budget_reset_at from the base class to LiteLLM_BudgetTableFull so it appears on API responses without becoming writable, and type LiteLLM_TeamMembership.litellm_budget_table as Union[Full, Base] so Pydantic picks Full when the data has server-managed fields (/team/info reads Prisma rows that include budget_reset_at and created_at) and Base when callers construct with only user-settable fields (existing auth tests and caches).
Follow-up on review feedback: the previous commit had the builder download the query engine into /app/.cache, then threw it away in the runtime stage and re-downloaded into /root/.cache. That doubled the build-time network fetch. Remove PRISMA_BINARY_CACHE_DIR and XDG_CACHE_HOME from the builder stage as well, so its prisma generate lands in /root/.cache with the correct path layout on its own. Drop the runtime-stage prisma generate and instead COPY --from=builder /root/.cache /root/.cache. Single download, smaller image.
Previous run (13.8m total) was bottlenecked by shards with 9-12m wall-clock. Setup + xdist spawn + coverage teardown is ~3m per shard, so each shard's pytest runtime must stay under ~4m to fit inside 7m total. Observed per-shard pytest times (before split): db-and-spend 9:08 (170s outlier: test_aaaasschema_migration_check) proxy-server 7:15 logging-and-callbacks 6:45 guardrails-budget-hooks 6:37 proxy-utils 6:23 auth-and-jwt 6:54 Split 6 shards into 12, keeping key-generation and endpoints-and-responses (already <7m). Adds a `keyword` input to _test-unit-services-base.yml so test_proxy_utils.py can be split by -k expression (same file, two runners). New matrix entries: auth-and-jwt -> auth-checks + jwt-and-keys proxy-server -> proxy-server-core + proxy-runtime logging-and-callbacks -> custom-logging + logging-misc db-and-spend -> schema-migration (isolated 170s test) + db-and-spend guardrails-budget-hooks-> guardrails-hooks + budgets proxy-utils -> proxy-utils-a-h + proxy-utils-i-z (-k split) The -k expression split is verified to cover every one of the 64 test functions in test_proxy_utils.py exactly once. The assert-shard-coverage guard still catches any file not in any shard.
Default GHA matrix job names join every matrix field, producing unreadable
check labels like:
'proxy-db (logging-misc, tests/proxy_unit_tests/test_proxy_reject_logging.py
tests/proxy_unit_tests/test_audit_logs_proxy.py ..., 8, loadscope, "", 15)'
Set the job's display name to '${{ matrix.test-group }}' so each check
shows just 'logging-misc', 'proxy-utils-a-h', etc.
[Fix] Tests - drain logging worker in test_router_caching_ttl to fix flakiness
…-k split Two changes: 1. workers: 8 -> 4 on every non-serial proxy-db shard. ubuntu-latest is a 4-core runner; -n 8 oversubscribes 2x and workers block each other during their cold-start imports (pytest-cov instruments every litellm module per worker). Measured ~441% CPU locally with -n 8 on 8 cores (i.e. ~55% effective). Matching -n to physical cores should give ~2x faster worker startup, which is where most of the ~9m wall-clock per shard goes (7+ minutes is plugin load + xdist imports before any test runs). 2. Revert the -k split on test_proxy_utils.py. It was split into proxy-utils-a-h / proxy-utils-i-z as a semantic-adjacent hack; merge back to a single proxy-utils shard. Still uses --dist=worksteal so xdist can balance the 188 parametrized cases across workers. Also drops the now-unused `keyword` input from _test-unit-services-base.yml and its matching matrix field across all proxy-db entries. Shard count: 14 -> 13 (+ the assert-shard-coverage guard).
Relative labels ("today", "in 2 days", "on May 12, 2026") mixed three
shapes in one column, breaking scannability. Always render MMM D, YYYY
for consistency and easier at-a-glance comparison across members.
…ion locations (#26281) Vertex multi-region endpoints (e.g. us, eu) use the rep host pattern, not {geo}-aiplatform.googleapis.com. Regional IDs still contain a hyphen. common_utils.get_vertex_base_url centralizes the rule for SDK/API URL building. Proxy pass-through duplicates the same branching in a local get_vertex_base_url (with trailing slashes) to avoid importing from common_utils there; live WebSocket passthrough uses the same multi-region host logic for wss://. Tests cover us/eu for the common_utils helper. Made-with: Cursor
[Fix] Infra: grant contents:write to create-release-branch caller job
[Fix] Deflake spend tracking tests
…is (#26162) (#26318) Temporary MCP OAuth sessions were kept in process-local memory, so on multi-instance/LB proxy deployments a session created on instance A could not be found when the follow-up /server/oauth/{server_id}/... request landed on instance B. Persist temporary session records to Redis (encrypted with the existing proxy encryption helpers) as a best-effort L2 cache alongside the current in-memory L1. Convert get_cached_temporary_mcp_server to async and await it from the authorize/token/register OAuth endpoints. Made-with: Cursor
test_db_schema_migration.py has exactly one test, and that test is mostly waiting on prisma subprocesses (~170s: prisma migrate deploy + prisma migrate diff). No CPU-bound Python work inside the test body, and only one test in the file means xdist's parallelism is unused regardless. Previous run on commit 5df9f39: 10.0m wall-clock for the shard, of which 4:56 was silence between step start and pytest banner — the cost of 4 xdist workers each cold-starting (pytest plugin load + litellm import + pytest-cov instrumentation) so that exactly one of them could pick up the single test. Switching to workers: 0 takes the serial pytest branch in the base workflow, which already handles this case correctly (no -n, no --dist). Single-process startup instead of 4. Expected wall-clock: ~6m.
The `_test-unit-services-base.yml` reusable workflow attached every job to the `integration-postgres` GHA environment to read three "secrets": DATABASE_URL, POSTGRES_USER, POSTGRES_PASSWORD. These are not secrets — the postgres service container is spawned per-job on localhost and destroyed with the job, so the user/password are bootstrap values for a throwaway container and the URL is always `postgresql://…@localhost:…`. Each environment attachment produces a "temporarily deployed to integration-postgres" deployment record, which the PR timeline renders as a message per matrix shard per push. With 14 proxy-db shards that's ~14 notifications per push, drowning the PR conversation. Changes: * Hardcode POSTGRES_USER/POSTGRES_PASSWORD/POSTGRES_DB and the derived DATABASE_URL in `_test-unit-services-base.yml`. * Delete the `environment: integration-postgres` attachment. * Delete the `secrets:` declarations on the reusable workflow and on the two callers (test-unit-proxy-db.yml, test-unit-security.yml). * The `services:` container still starts a fresh postgres per job; the connection string now matches what the container boots up with. Security review: no regression. The environment wasn't gating anything real — no protection rules configured, no approval gates, and the branch restriction is already enforced by `on: push: branches: [...]` on both caller workflows. Zizmor pedantic-mode findings are identical before and after (same 6 pre-existing findings, zero new ones). The `integration-postgres` environment and its three "secrets" in repo settings are now unreferenced and can be deleted from repo admin.
[Fix] Reset budget windows failing due to Prisma Json? null filter
…itellm_team_member_total_spend_frontend
Out of scope for the members-tab feature and regressed legacy teams whose budget_reset_at is null (duration was previously shown as a fallback).
Members tab column reads this field; dropping it from the type in the previous revert broke the type check without affecting the reverted render logic.
…d_frontend Surface per-member budget cycle in Teams > Members tab
[Infra] Bump version 1.83.12 → 1.83.13
…m-byok-model-none
…-model-none fix(router): restore BYOK key injection for vector store endpoints with team-scoped deployments
[Infra] Remove CCI/GHA test duplication and semantically shard proxy DB tests
Contributor
|
Too many files changed for review. ( |
Sameerlite
merged commit Apr 24, 2026
9d58e6e
into
litellm_anthropic-json-mode-nonstreaming-mixed-tools
80 of 84 checks passed
|
Michael Riad Zaky seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
Contributor
Low: Security hardening across multiple subsystemsThis large merge PR adds an adaptive router feature, several security hardening measures, and bug fixes. The security-relevant changes are net-positive:
Status: 0 open Posted by Veria AI · 2026-04-24T03:41:27.203Z |
|
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
|---|---|---|---|---|---|
| 31539530 | Triggered | Generic Password | 66bf890 | .github/workflows/_test-unit-services-base.yml | View secret |
| 31539530 | Triggered | Generic Password | 8e652d1 | .github/workflows/_test-unit-services-base.yml | View secret |
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secrets safely. Learn here the best practices.
- Revoke and rotate these secrets.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
fzowl
pushed a commit
to fzowl/litellm
that referenced
this pull request
Jun 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relevant issues
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
tests/test_litellm/directory, Adding at least 1 test is a hard requirement - see detailsmake test-unit@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewDelays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
CI (LiteLLM team)
Branch creation CI run
Link:
CI run for the last commit
Link:
Merge / cherry-pick CI run
Links:
Screenshots / Proof of Fix
Type
🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test
Changes