chore(release): backport #30480, #30787, #30788, #31035, #31133, #31122 to stable/1.85.x and cut 1.85.7 - #31173
Merged
Merged
Conversation
…30480) * fix(integrations): cap Anthropic cache_control injection at 4 blocks Respect Anthropic's 4 cache_control breakpoint limit by counting client-supplied blocks, skipping messages that already carry cache_control, and stopping further auto-injection once the limit is reached. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(integrations): reserve cache slot for tool_config and short-circuit cap Address review feedback on the cache_control cap: break out of the injection loop before resolving target indices once the limit is reached, and reserve one of the four breakpoint slots when a tool_config injection point is present so the cachePoint appended by the Bedrock transform does not push the total past Anthropic's limit. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> (cherry picked from commit fc9d789)
…treams (#30788) A streaming request that breaks mid-flight, for example on a mid-stream read timeout, still bills the provider for the chunks already delivered, yet the proxy recorded that interrupted request as a zero-spend failure. An earlier revision logged the recovered partial usage through the success path, which mislabeled a failed request as a success and produced a misleading spend row This recovers the partial usage where the failure is actually logged. The streaming handler assembles the usage from the chunks seen so far and stashes it, with its cost, on the logging object before firing the failure handlers. The proxy failure hook lifts that usage and cost onto request_data before the non-serialisable logging object is popped, and the spend-log writer records the real partial spend on the failure row instead of a hardcoded zero; get_logging_payload honors the recovered usage for the token columns and _failure_handler_helper_fn preserves the recovered cost so the non-DB failure loggers stay consistent A request that recovers via a successful fallback is unaffected: the failure hook only fires when the whole request fails, so the fallback's combined-usage success row stays the single source of truth and there is no double counting Resolves LIT-3825 Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> (cherry picked from commit 4847fa5)
Prerequisite for #31035 on this line, and a latent-bug fix in its own right. #31035's usage-only fallback builds server_tool_use as a dict and prices it via AnthropicConfig.calculate_usage, whose Usage(**model_dump()) round-trip drops it back to a plain dict; without this Usage.__init__ coercion the recovered-cost path does attribute access on a dict and raises, so #31035's web-search/server-tool cost recovery is dead on arrival here. The same round-trip already affected the pre-existing ChunkProcessor.calculate_usage path: every production consumer on this line (litellm/llms/anthropic/cost_calculation.py, litellm/litellm_core_utils/llm_cost_calc/tool_call_cost_tracking.py) reads usage.server_tool_use.web_search_requests by attribute, so a dict there is a latent AttributeError on streaming web-search cost. The coercion makes the value a ServerToolUse, which all consumers expect. Also updates the one test that pinned the old dict-subscript shape (test_stream_chunk_builder_anthropic_web_search) to assert the ServerToolUse type and attribute access, matching staging. Content-verified present on litellm_internal_staging via aggregator f49707b (fix(otel) #30257), which carries both the coercion and the test assertion update; this restores only those, not the rest of that aggregator. The coercion also shipped to stable/1.89.x as 24e30b5. (cherry picked from commit 24e30b5)
…nthropic streams (#31035) Streaming and pass-through requests could be logged with $0 cost or dropped from SpendLogs entirely while the upstream provider still billed every token. This closes the leak paths not already covered by #30160, #30787 and #30788. - Catch a stream_chunk_builder raise in the core CustomStreamWrapper (sync and async). Large agentic tool-use / thinking streams can make assembly re-raise as APIError from inside the except-StopIteration handler, where the sibling except does not catch it, so it escaped __next__/__anext__ and dropped the request; recover best-effort usage from the raw chunks instead - Add a usage-only fallback for Anthropic streaming pass-through: when stream_chunk_builder returns None or raises, rebuild usage from the message_start / message_delta SSE events via AnthropicConfig.calculate_usage so cache, web-search and geo tokens are priced instead of left at $0 - Decode buffered pass-through bytes with errors="replace" so a stream cut mid-multibyte-sequence still logs the usage events already received - Record response_cost into model_call_details on the pass-through success path (it is read from there, not from kwargs), matching the gemini/cohere/openai handlers - Name the key (alias + masked key) in the virtual-key BudgetExceededError so operators don't have to reverse-map spend back to a key (cherry picked from commit b24b964)
…31133) Re-pins LITELLM_BUILD_IMAGE and LITELLM_RUNTIME_IMAGE across all 6 Dockerfiles from the prior digests (openssl 3.6.2-r3) to the current chainguard wolfi-base digest c61ac691 (openssl 3.6.3-r2, >= the fixed 3.6.3-r0). The runtime stage is the shipped image, so the runtime digest is what actually resolves the customer-facing CVE; the build image is bumped too for hygiene. Two Dockerfiles tracked a second equally-stale digest; both are unified onto the patched one. (cherry picked from commit fda08dd)
|
|
shin-berri
approved these changes
Jun 24, 2026
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Scoped to the customer-shipped dependencies on this line: cryptography 48.0.1, python-multipart 0.0.32, pypdf 6.13.3, and semantic-router >=0.1.15,<1.0 (the pinned 0.1.12 is yanked, CVE-2026-42208); mlflow is loosened to >=3.11.1,<4.0 so cryptography can move, and the dashboard js-yaml 4.2.0 and ws 8.21.0 overrides are bumped. uv.lock and package-lock.json are regenerated on this line. The osv-scanner.toml and osv-scan.yml hunks are dropped (absent on 1.85.x), and the non-shipping CI/test stack (langchain, langgraph, vcrpy, aiohttp) is left out. semantic-router >=0.1.15,<1.0 is content-verified present on litellm_internal_staging (via aggregator f49707b) and matches the 1.84.x backport of this PR (1a56faa). (cherry picked from commit a8a1472)
Contributor
Greptile SummaryThis is a backport release (1.85.7) that cherry-picks six already-merged staging fixes onto
Confidence Score: 4/5The backport is safe to merge; the one issue found is an edge case in the tool_config block reservation that only matters when multiple non-message cache-control injection points are configured simultaneously. The multi-layer spend-recovery chain is logically consistent and well-tested with mocked unit tests covering each seam (stash → lift → failure hook → spend log). The ServerToolUse coercion, UTF-8 error-replace, and digest bumps are low-risk. The only concern is the reserved_blocks = 1 constant in _apply_message_injections: it prevents the most common single-tool_config-point overflow but would still allow a request with two tool_config injection points to hit 5 blocks. litellm/integrations/anthropic_cache_control_hook.py — the tool_config block reservation logic; litellm/litellm_core_utils/streaming_handler.py and litellm/proxy/hooks/proxy_track_cost_callback.py for the coordinated spend-recovery chain.
|
| Filename | Overview |
|---|---|
| litellm/integrations/anthropic_cache_control_hook.py | Refactored _process_message_injection into _apply_message_injections + _resolve_target_indices to enforce the 4-block Anthropic limit; reserves a slot for tool_config but only reserves 1 regardless of tool_config point count |
| litellm/litellm_core_utils/litellm_logging.py | Preserves recovered spend on interrupted streams by only zeroing response_cost when no combined_usage_object is set by the streaming handler |
| litellm/litellm_core_utils/streaming_handler.py | Adds _record_partial_usage_for_failure to stash partial usage+cost on the logging object before failure handlers run; wraps stream_chunk_builder in try/except at end-of-stream to recover best-effort usage from raw chunks |
| litellm/proxy/pass_through_endpoints/llm_provider_handlers/anthropic_passthrough_logging_handler.py | Adds interrupted-stream detection, re-tokenization of output text, usage-only fallback response builder, and stream assembly exception handling to recover cost on interrupted/agentic Anthropic passthrough streams |
| litellm/proxy/hooks/proxy_track_cost_callback.py | Failure hook now reads combined_usage_object from request_data (lifted by proxy_utils) and passes the recovered response_cost to update_database instead of the hardcoded 0.0 |
| litellm/proxy/utils.py | Lifts combined_usage_object and response_cost from litellm_logging_obj.model_call_details onto request_data before popping the logging obj, ensuring failure callbacks can read the recovered partial spend |
| litellm/proxy/spend_tracking/spend_tracking_utils.py | Adds fallback to combined_usage_object when response_obj carries no usable usage, so partial spend is recorded in spend logs for interrupted streams |
| litellm/types/utils.py | Coerces server_tool_use from dict to ServerToolUse in Usage.init, fixing a latent AttributeError on the streaming web-search cost path |
| litellm/proxy/pass_through_endpoints/streaming_handler.py | Switches bytes decode to errors="replace" so a client disconnect cutting a multibyte sequence still delivers already-received usage events to SpendLogs |
| pyproject.toml | Bumps cryptography to 48.0.1, python-multipart to 0.0.32, pypdf to 6.13.3, loosens mlflow to >=3.11.1,<4.0 to clear CVEs; version bumped to 1.85.7 |
Reviews (1): Last reviewed commit: "chore: refresh uv.lock for 1.85.7" | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+77
to
+85
| # Non-message points (currently Bedrock tool_config) are handled in the | ||
| # provider transform, where each tool_config point appends at most one | ||
| # cachePoint to the tools. That block also counts toward Anthropic's | ||
| # limit, so reserve a slot for it here to leave room. | ||
| reserved_blocks = ( | ||
| 1 | ||
| if any(p.get("location") == "tool_config" for p in remaining_points) | ||
| else 0 | ||
| ) |
Contributor
There was a problem hiding this comment.
reserved_blocks is fixed at 1 regardless of how many tool_config injection points exist. If two tool_config points each inject a cache block, the message-level budget would be MAX_CACHE_CONTROL_BLOCKS - 1 = 3, yet the total would reach 5 (3 message + 2 tool_config), still exceeding Anthropic's hard limit. Counting the actual number of remaining non-message points matches the intent of the reservation.
Suggested change
| # Non-message points (currently Bedrock tool_config) are handled in the | |
| # provider transform, where each tool_config point appends at most one | |
| # cachePoint to the tools. That block also counts toward Anthropic's | |
| # limit, so reserve a slot for it here to leave room. | |
| reserved_blocks = ( | |
| 1 | |
| if any(p.get("location") == "tool_config" for p in remaining_points) | |
| else 0 | |
| ) | |
| # Non-message points (currently Bedrock tool_config) are handled in the | |
| # provider transform, where each tool_config point appends at most one | |
| # cachePoint to the tools. That block also counts toward Anthropic's | |
| # limit, so reserve a slot per non-message point to leave room. | |
| reserved_blocks = sum( | |
| 1 | |
| for p in remaining_points | |
| if p.get("location") == "tool_config" | |
| ) |
yuneng-berri
force-pushed
the
litellm_backport_1_85_x_0623
branch
from
June 24, 2026 04:51
5194c6a to
4f8f540
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relevant issues
Backports six already-merged staging fixes onto stable/1.85.x and cuts 1.85.7. The
theme is interrupted/agentic Anthropic streaming cost accuracy plus two security
dependency bumps. #30480 stops the Anthropic cache_control hook from injecting a
5th breakpoint past Anthropic's hard limit of 4 (which 400s the request). #30787,
#30788 and #31035 recover token usage and spend on streams that break mid-flight
or assemble large agentic tool-use payloads, so interrupted requests are no longer
logged at $0 while the provider still billed them. #31133 re-pins the wolfi-base
image digest to pick up the patched openssl. #31122 is brought over scoped to the
customer-shipped dependencies that clear CVEs on this line.
Linear ticket
n/a
What is included
In merge order:
prerequisite for fix(passthrough,streaming): recover cost on interrupted and agentic Anthropic streams #31035 (and a latent-bug fix in its own right) that is
content-verified present on internal staging via aggregator f49707b and
already shipped to stable/1.89.x as 24e30b5
the shipped runtime deps on this line (cryptography 48.0.1, python-multipart
0.0.32, pypdf 6.13.3, semantic-router >=0.1.15,<1.0 since the pinned 0.1.12 is
yanked, mlflow loosened to >=3.11.1,<4.0 so cryptography can move, plus the
dashboard js-yaml 4.2.0 and ws 8.21.0 overrides)
Adaptation notes
The 1.85.x tree predates a fair amount of staging, so several picks were adapted
rather than applied verbatim; each is content-equivalent to its staging source.
unpicked staging PRs into the conflicts; only each PR's own added tests were
kept. fix(proxy): record partial spend on the failure row for interrupted streams #30788's failure hook carries only its combined_usage_object/response_cost
lift, not the unrelated first_api_call_start_time lift that the line does not
have. fix(passthrough,streaming): recover cost on interrupted and agentic Anthropic streams #31035's auth_checks change applies its BudgetExceededError key-naming onto
the line's budget guard (the line does not carry the separate math.isfinite NaN
guard, and math is not imported there).
usage.server_tool_use.web_search_requests by attribute, so the dict that
Usage(**model_dump()) produced was a latent AttributeError on the streaming
web-search cost path. The coercion makes the value a ServerToolUse, matching all
consumers; the one test that pinned the old dict-subscript shape is updated to
isinstance + attribute access, exactly as staging did.
patched digest is applied to the three present (Dockerfile,
docker/Dockerfile.database, docker/Dockerfile.non_root) and the three absent
ones are dropped.
ruff rule (staging does not select PLR0915), so each carries an inline
noqa: PLR0915, matching this file's existing convention.
Known noise on this line
The targeted suite has one pre-existing failure unrelated to these picks:
tests/test_litellm/proxy/test_proxy_utils.py::test_get_custom_url asserts
http://0.0.0.0:4000/... but resolves http://localhost:4000/..., a host-resolution
quirk on the build host. It fails identically on the line before any pick.
Type
🐛 Bug Fix
🚄 Infrastructure
Changes
See "What is included" above.
Screenshots / Proof of Fix
Live proxy on the picked branch (real Anthropic calls), #30480 reproduced before
and after on a no-fallback model so the error surfaces directly.
Before (line tip, cache_control hook injects a 5th breakpoint):
After (same request, same branch with the picks):
Proxy sanity on the dependency-bumped tree (cryptography 48.0.1, mlflow 3.14.0):
liveliness returns "I'm alive!" and a real claude-haiku-4-5 completion returns
"deps-ok", so the runtime dep bump does not break the auth/serving path.
Targeted-test delta against the line baseline: 540 passed, 1 failed, where the one
failure is the pre-existing test_get_custom_url host quirk noted above; every test
that the picked PRs added or modified passes (for example #31035's
TestAnthropicUsageOnlyFallback and the recovered-spend tests, all green).
Behavioral verification: an adversarial multi-agent gauntlet over the full pick set
returned SURVIVED on all three sub-claims (identifier/import resolution; each pick
delivers its claim with no new test failure; no pre-existing caller of a
pick-modified function is broken). An earlier run caught exactly one issue, the
stale server_tool_use dict-subscript test, which is fixed here by the companion
assertion update described in the adaptation notes.