chore(release): backport #33244, #33592, #33554, #33853 to stable/1.92.x and cut 1.92.1 - #33892
Conversation
…ading adaptive thinking for pre-4.6 models (#33244) * fix(anthropic/passthrough): drop temperature and cap thinking budget when downgrading adaptive thinking for pre-4.6 models * test(anthropic/passthrough): use sufficient max_tokens for reasoning_effort thinking mapping * fix(anthropic/passthrough): drop incompatible temperature when downgrading adaptive thinking for pre-4.6 models Narrow the fix to the temperature reconciliation; the reasoning_effort budget cap is reverted because the live translation grid relies on budget_tokens >= max_tokens to reject unsupported effort tiers (xhigh/max) on budget-mode models, so capping turned those 400s into 200s. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> (cherry picked from commit 71dffc1)
#33592) * fix(docker): restore litellm-proxy-extras source dir in runtime images #30243 narrowed the runtime stage to an allowlist COPY, which dropped /app/litellm-proxy-extras from the published images. Downstream migration jobs point prisma migrate deploy at that path; with the schema gone (or a schema with no adjacent migrations dir, where prisma exits 0 without applying anything) those jobs went green while never migrating the database. Restore the folder in all three runtime stages and assert in image-scan that the schema and a non-empty migrations dir ship at the source path * chore(ci): drop image-scan migration-assets assertion (cherry picked from commit 111d447)
…attachments and remove the attachment count cap (#33554) * fix(model_armor): add skip_unscannable_attachments to allow reference-only attachments through * fix(model_armor): wire skip_unscannable_attachments through guardrail config * fix(model_armor): make max_file_attachments configurable and scan overflow instead of dropping * fix(model_armor): remove the per-request attachment count cap and scan all attachments --------- Co-authored-by: yucheng <yucheng@berri.ai> (cherry picked from commit 0d7b0f7)
|
|
|
Converting to draft: the deep-verification pass found that The same coercion exists on |
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
| metadata["_model_armor_status"] = "blocked" | ||
| raise self._unscannable_block_error(reason) | ||
|
|
||
| if len(attachments) > MAX_FILE_ATTACHMENTS_PER_REQUEST: |
There was a problem hiding this comment.
Medium: Unbounded guardrail request amplification
An authenticated user can include hundreds of small inline attachments in one request and cause a separate Model Armor API call for each attachment. The proxy accounts for this as one client request, so removing this bound permits guardrail quota exhaustion and prolonged worker occupancy; retain a configurable per-request cap or enforce an equivalent fan-out budget before scanning.
PR overviewThis PR backports several changes to the stable/1.92.x branch and prepares the 1.92.1 release. The affected proxy guardrail code includes Model Armor handling for scanning request content and inline attachments. There is one open security concern in the Model Armor guardrail path. An authenticated user can submit many small inline attachments in a single request, causing unbounded downstream Model Armor scan calls while the proxy accounts for it as only one client request. This creates a practical quota-exhaustion and worker-occupancy risk until a per-request cap or equivalent fan-out budget is enforced. Open issues (1)
Fixed/addressed: 0 · PR risk: 6/10 |
Greptile SummaryThis is a backport release (v1.92.1) cherry-picking four targeted fixes from staging onto
Confidence Score: 5/5All four cherry-picks are narrowly scoped bug fixes with matching tests; no new logic paths affect the auth layer, DB, or routing hot path. The temperature-drop logic is correctly placed after all thinking-translation steps, handles all model variants (adaptive pass-through, legacy enabled, Opus 4.5 effort), and is covered by six unit tests with no network calls. The Prisma baking adds build-time assertions that would catch a broken image before it ships. The skip_unscannable_attachments change introduces a secure-by-default opt-in flag with proper kwarg plumbing and full test coverage. No pre-existing guards were removed without replacement, and no auth, DB, or routing paths were touched. No files require special attention.
|
| Filename | Overview |
|---|---|
| litellm/llms/anthropic/experimental_pass_through/messages/transformation.py | Adds _drop_incompatible_temperature_for_thinking staticmethod that removes non-1 temperature values when non-adaptive models have extended thinking enabled, fixing Anthropic 400 errors for clients like Claude Code that pin temperature. |
| litellm/proxy/guardrails/guardrail_hooks/model_armor/model_armor.py | Removes the MAX_FILE_ATTACHMENTS_PER_REQUEST=10 cap and introduces skip_unscannable_attachments opt-in param that lets reference-only attachments pass through without blocking, while API errors still respect fail_on_error. |
| litellm/proxy/guardrails/guardrail_hooks/model_armor/file_scanning.py | Removes the MAX_FILE_ATTACHMENTS_PER_REQUEST constant; all other scanning logic unchanged. |
| litellm/proxy/guardrails/guardrail_hooks/model_armor/init.py | Forwards skip_unscannable_attachments from LitellmParams to ModelArmorGuardrail constructor; straightforward plumbing change. |
| litellm/types/guardrails.py | Adds skip_unscannable_attachments: Optional[bool] field to BaseLitellmParams with a secure default of False. |
| Dockerfile | Bakes Prisma CLI and engines under /opt/prisma with fixed env vars, adds build-time assertions, and restores the litellm-proxy-extras copy; drops the brittle /root/.cache copy approach. |
| docker/Dockerfile.database | Same Prisma baking and litellm-proxy-extras fix as the main Dockerfile, applied to the database-focused image. |
| docker/Dockerfile.non_root | Adds the missing litellm-proxy-extras copy to the non-root image; Prisma setup remains via its existing /app/.cache path. |
| tests/test_litellm/llms/anthropic/experimental_pass_through/messages/test_anthropic_messages_effort.py | Adds six new pure-transform unit tests covering temperature-drop scenarios: haiku downgrade, temperature=1 preservation, non-reasoning model passthrough, adaptive model passthrough, opus-4-5 effort, and reasoning_effort alias. |
| tests/test_litellm/proxy/guardrails/guardrail_hooks/test_model_armor.py | Replaces the attachment-count-cap test with a no-cap test; adds six new tests covering skip_unscannable_attachments behavior for file_id, gs:// URIs, API error passthrough, config forwarding, and default value. |
| pyproject.toml | Version bump 1.92.0 -> 1.92.1. |
Reviews (2): Last reviewed commit: "chore: refresh uv.lock for 1.92.1" | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| deployment = self.get_deployment_by_model_group_name(model_group_name=model_name) | ||
| if deployment is None: | ||
| return (None, None) | ||
|
|
||
| def _as_int(value: object) -> "int | None": | ||
| if value is None or isinstance(value, bool): | ||
| return None | ||
| try: | ||
| return int(value) | ||
| except (TypeError, ValueError): | ||
| return None | ||
|
|
||
| model_info = deployment.model_info | ||
| return ( | ||
| _as_int(model_info.get("max_input_tokens")), | ||
| _as_int(model_info.get("max_output_tokens")), | ||
| ) |
There was a problem hiding this comment.
get_deployment_by_model_group_name always returns indices[0], so when a model group has two deployments with different model_info.max_input_tokens values, /v1/models will silently report whichever deployment happens to be first in the list. For example, if one deployment is a 128k context replica and another is an 8k replica under the same group name, clients that rely on the reported limit for sizing requests could receive a limit that is too large for the deployment they're actually routed to. A comment or docstring clarifying that this returns the first deployment's limits (not the minimum or representative value across the group) would prevent operators from being surprised by this.
…migrations work for any uid offline (#33853) * fix(docker): bake prisma CLI and engines at a fixed path so fresh-DB migrations work for any uid offline The runtime image shipped the prisma CLI and engines under /root/.cache, the default HOME-derived prisma-python cache location. Any deployment whose runtime HOME is not /root (kubernetes runAsUser, docker --user, HOME overrides) missed that cache on a fresh database, fell back to a nodeenv Node download that crashes on Wolfi (libatomic.so.1), and started the proxy with zero tables while every DB-backed endpoint returned 500 The bake now lives at /opt/prisma, a path no HOME resolution or cache volume mount can shadow. The builder records the engine paths there at generate time, and the runtime stage pins PRISMA_BINARY_CACHE_DIR, PRISMA_CLI_PATH, PRISMA_CLI_QUERY_ENGINE_TYPE=binary and PRISMA_OFFLINE_MODE so both litellm-proxy-extras and prisma-python resolve the baked CLI and engines directly. prisma migrate deploy on a fresh database now needs no npm and no network access for any runtime uid, including readOnlyRootFilesystem deployments Verified against live containers: fresh and existing databases as root, uid 12345, HOME overridden, on an internal-only docker network, and with a read-only root filesystem all migrate and serve /team/new successfully Fixes #33650, #24554 * chore(docker): fail the image build if the baked prisma CLI layout drifts Asserts the baked CLI shim is executable and its entrypoint exists in the runtime stage after the COPY and chmod, so a layout change in a future prisma-python release breaks the image build loudly instead of silently degrading the migration path at container startup (cherry picked from commit 567ebcb)
df77de0 to
e7d3f93
Compare
yuneng-berri
changed the title
|
Dropped #33721 and #33864 from this backport and rebased the branch: the deep-verification replay showed The four remaining picks do not touch that code path. On the rebuilt branch the live replay shows |
3 participants
Relevant issues
Backports four staging fixes onto
stable/1.92.xand cuts 1.92.1 (v1.92.0 is already published on DockerHub and GHCR, so the tip version must move). Two of the picks repair the shipped Docker images: #33592 restores the/app/litellm-proxy-extrassource dir that downstream migration jobs pointprisma migrate deployat (without it those jobs exit green while never migrating), and #33853 bakes the prisma CLI and engines at/opt/prismaso fresh-database migrations work for any runtime uid, offline, including readOnlyRootFilesystem deployments (the failure modes behind #33650 and #24554). #33244 drops a pinnedtemperaturewhen adaptive thinking is downgraded for pre-4.6 Anthropic models on the/v1/messagespassthrough, so clients like Claude Code stop receiving 400s. #33554 addsskip_unscannable_attachmentsto the Model Armor guardrail and removes the per-request attachment count capThe original set also carried #33721 and #33864 (the /v1/models token-limit rework). Deep verification found the listing regressed on this branch for deployments carrying a malformed configured token limit, through the registered cost-map path (details in the PR comments), so that pair was dropped and returns in a later patch once the upstream hardening lands. The live replay on the rebuilt branch confirms the listing behaves exactly as 1.92.0 does today
On top of the picks: routine dependency maintenance (lock-only bumps of
mcpto 1.28.1 andsoupsieveto 2.8.4; each lock regeneration moved exactly the target package and nothing else), then the version bump and lock refreshLinear ticket
What is included
Cherry-picks in staging merge order, each with a
-xfooter pointing at its staging squash commit:Then four tool-generated commits:
chore(deps): bump mcp to 1.28.1,chore(deps): bump soupsieve to 2.8.4(both confined touv.lock),bump: version 1.92.0 -> 1.92.1(pyproject only), andchore: refresh uv.lock for 1.92.1(moves only litellm's own entry)Adaptation notes (one pick is not byte-identical to its staging commit)
_drop_incompatible_temperature_for_thinkingstaticmethod inserts at a spot where this line still holds_cap_thinking_budget_to_max_tokens(carried onto 1.92.0 by the rc2 backport of fix(anthropic): translate adaptive thinking/effort to pre-4.6 model support #32867; staging later relocated that method in a commit outside this set). Resolution keeps both methods side by side, byte-identical to how the 1.93.0 cut resolved the same conflict (f7f8ebe)Known noise on this line
Pre-existing on the untouched
stable/1.92.xtip, unrelated to the picks, so reviewers can discount them in CI:tests/test_litellm/proxy/test_proxy_utils.py::test_get_custom_urlfails at the baseline (before any pick) and identically aftertests/test_litellm/proxy/client/test_models.py,tests/test_litellm/realtime_api/test_main.py)Pre-Submission checklist
Screenshots / Proof of Fix
Targeted test set (the five test files the original picks touch, run on this line): baseline on the untouched tip was 269 passed, 1 failed (the known-noise failure above); on the final branch it is 280 passed and the same single pre-existing failure. Zero new failures; every test the remaining picks add passes on this line.
ruff checkandruff format --checkare clean on every touched moduleLive proxy on this branch (config includes a deployment with
model_info: {max_input_tokens: "128,000"}to pin the listing contract):Reproducer replay for #33244 (
claude-sonnet-4-5,thinkingenabled,temperature: 0, real Anthropic call). Before the pick, on the untouched tip:After the pick, same request:
Docker picks, verified by building both changed images from this branch and running them (throwaway local tags, removed after):
Dependency maintenance evidence:
uv lock --upgrade-packagemoved exactlymcp 1.26.0 -> 1.28.1andsoupsieve 2.8.3 -> 2.8.4(nothing else), and the synced environment resolves both at their targetsRemaining deep verification (adversarial behavioral gauntlet and the full-suite regression delta against a pristine baseline) is in progress; this body will be updated with the results
Type
🐛 Bug Fix
🚄 Infrastructure
Changes
Four cherry-picks onto
stable/1.92.x(listed above), two lock-only dependency maintenance commits, and the 1.92.0 -> 1.92.1 version bump with its lock refreshQA runbook
uv sync --frozen --all-groups --all-extrasuv run pytest tests/test_litellm/proxy/guardrails/guardrail_hooks/test_model_armor.py tests/test_litellm/llms/anthropic/experimental_pass_through/messages/test_anthropic_messages_effort.py -q; expect all greenclaude-sonnet-4-5) andPOST /v1/messageswith{"model": "claude-sonnet-4-5", "max_tokens": 1100, "thinking": {"type": "enabled", "budget_tokens": 1024}, "temperature": 0, ...}: expect 200 (it 400s before this PR)docker build -f Dockerfile -t litellm-192x-qa:main .thendocker run --rm --user 12345:12345 -e HOME=/tmp --entrypoint sh litellm-192x-qa:main -c 'ls /app/litellm-proxy-extras/litellm_proxy_extras/schema.prisma && test -x /opt/prisma/binaries/node_modules/.bin/prisma && echo OK'Final Attestation