chore(release): backport #30787, #30788, #31035, #31036, #31122, #31133 to stable/1.89.x (litellm-enterprise 0.1.42.post2) - #31259
Merged
yuneng-berri merged 9 commits intoJun 25, 2026
Conversation
…treams (#30788) A streaming request that breaks mid-flight, for example on a mid-stream read timeout, still bills the provider for the chunks already delivered, yet the proxy recorded that interrupted request as a zero-spend failure. An earlier revision logged the recovered partial usage through the success path, which mislabeled a failed request as a success and produced a misleading spend row This recovers the partial usage where the failure is actually logged. The streaming handler assembles the usage from the chunks seen so far and stashes it, with its cost, on the logging object before firing the failure handlers. The proxy failure hook lifts that usage and cost onto request_data before the non-serialisable logging object is popped, and the spend-log writer records the real partial spend on the failure row instead of a hardcoded zero; get_logging_payload honors the recovered usage for the token columns and _failure_handler_helper_fn preserves the recovered cost so the non-DB failure loggers stay consistent A request that recovers via a successful fallback is unaffected: the failure hook only fires when the whole request fails, so the fallback's combined-usage success row stays the single source of truth and there is no double counting Resolves LIT-3825 Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> (cherry picked from commit 4847fa5)
…nthropic streams (#31035) Streaming and pass-through requests could be logged with $0 cost or dropped from SpendLogs entirely while the upstream provider still billed every token. This closes the leak paths not already covered by #30160, #30787 and #30788. - Catch a stream_chunk_builder raise in the core CustomStreamWrapper (sync and async). Large agentic tool-use / thinking streams can make assembly re-raise as APIError from inside the except-StopIteration handler, where the sibling except does not catch it, so it escaped __next__/__anext__ and dropped the request; recover best-effort usage from the raw chunks instead - Add a usage-only fallback for Anthropic streaming pass-through: when stream_chunk_builder returns None or raises, rebuild usage from the message_start / message_delta SSE events via AnthropicConfig.calculate_usage so cache, web-search and geo tokens are priced instead of left at $0 - Decode buffered pass-through bytes with errors="replace" so a stream cut mid-multibyte-sequence still logs the usage events already received - Record response_cost into model_call_details on the pass-through success path (it is read from there, not from kwargs), matching the gemini/cohere/openai handlers - Name the key (alias + masked key) in the virtual-key BudgetExceededError so operators don't have to reverse-map spend back to a key (cherry picked from commit b24b964)
…31133) Re-pins LITELLM_BUILD_IMAGE and LITELLM_RUNTIME_IMAGE across all 6 Dockerfiles from the prior digests (openssl 3.6.2-r3) to the current chainguard wolfi-base digest c61ac691 (openssl 3.6.3-r2, >= the fixed 3.6.3-r0). The runtime stage is the shipped image, so the runtime digest is what actually resolves the customer-facing CVE; the build image is bumped too for hygiene. Two Dockerfiles tracked a second equally-stale digest; both are unified onto the patched one. (cherry picked from commit fda08dd)
* fix(vertex/files): stream OpenAI->Vertex batch JSONL uploads to fix OOM on large files
Large (1GB+) batch JSONL uploads to Vertex AI / GCS caused OOM or killed the worker
because the request body was buffered and multiplied 2-3x in size. The create-file
path is now streaming end-to-end: transform_create_file_request returns a
ResumableChunkedUploadConfig carrying a lazy _OpenAIToVertexBatchUploadStream, and the
HTTP handler opens a GCS resumable session and PUTs the body in bounded 8 MiB chunks
(Content-Range, 308 between chunks) so the transformed payload is never held in full.
The proxy /v1/files endpoint streams from Starlette's spooled upload handle instead of
reading the whole body, and batch rate limiting counts tokens and models in a single
streaming pass.
Only gcs_bucket_name is supported for the GCS target; the legacy bucket_name key is
intentionally not read.
Also removes the unreachable VertexAIFilesHandler create path and everything only it
kept alive (VertexAIJsonlFilesTransformation, _stream_openai_jsonl_to_vertex, the legacy
transform helpers), plus the orphaned batch_utils helpers the streaming rewrite replaced.
* fix(batches): return original JSONL on unparseable row to avoid silent batch truncation
The streaming rewrite of replace_model_in_jsonl accumulated physical lines and
skipped a row on JSONDecodeError to support multi-line objects, but a genuinely
malformed or truncated row never completes: it poisons the buffer, swallows every
following row, and the function still returned the partial rewrite (the rows before
the bad one, already model-rewritten) as if the batch were complete. That turned the
pre-rewrite behavior of returning the original file unchanged (so the provider rejects
the bad batch loudly) into a silent partial submission.
Restore the original-content fallback: when an unparseable remainder is left after the
loop, return the original file_content (rewinding a consumed seekable source) instead of
the truncated output. The multi-line happy path is unchanged.
* test(batches): mock resumable GCS upload in vertex batch prediction test
The vertex batch file-create path now streams to a GCS resumable session via
_aresumable_chunked_upload (httpx send) instead of AsyncHTTPHandler.post, so the
existing test's post mock no longer intercepted the upload and a real request hit
GCS (401). Mock _aresumable_chunked_upload to return the GCS object response; the
resumable protocol itself is covered in test_vertex_ai_files_streaming.py.
* fix(batches): resilient per-row token accounting; no hard-block on count failure
The batch input-file pass iterated a generator whose json.loads raised on a
malformed line; the outer except caught it and stopped the loop, so any body.model
on rows after a bad line was never collected and the model allowlist check ran
against a partial set. It also hard-blocked the batch with a 400 whenever token
counting raised, a backwards-incompatible change from the prior swallow-and-proceed
behavior that breaks legitimate rows the token counter cannot measure (e.g. some
multimodal content).
Iterate the JSONL line-by-line and account each row independently. A malformed line
is skipped (its request cannot run upstream anyway) and a row the counter cannot
measure falls back to a conservative size-based estimate. The loop never aborts, so
the allowlist check always sees every parseable model, and the token total is never
zeroed, so a crafted uncountable row still cannot evade the TPM limit, without
hard-rejecting a legitimate batch.
* perf(vertex/files): unblock async upload; drop empty finalize; widen batch MIME types
Three review follow-ups on the resumable batch upload:
- _aresumable_chunked_upload pulled chunks from a synchronous generator that runs
the per-row transform inline on the event loop thread, blocking other requests
between PUTs on large uploads. Each chunk is now produced via asyncio.to_thread.
- _iter_resumable_chunks no longer yields a trailing empty chunk, so an exactly
chunk-aligned upload finalizes on its last data chunk instead of an extra
zero-byte PUT; a 0-byte stream still finalizes via the caller's empty request.
- valid_content_type now accepts the MIME types clients label .jsonl batch uploads
with (text/plain, application/json, ndjson, ...), so such a batch file no longer
silently bypasses the streaming path into the buffered media upload.
* fix(vertex/files): keep legacy bucket_name as GCS bucket fallback
The rename to gcs_bucket_name dropped the legacy bucket_name key entirely, so an SDK caller passing bucket_name to a Vertex AI file create/retrieve/content call with GCS_BUCKET_NAME unset got ValueError("GCS bucket_name is required") where it previously resolved the bucket. _get_configured_bucket_name now reads gcs_bucket_name, then bucket_name, then the env var, and bucket_name is restored to OPTIONAL_KWARGS_KEYS so it survives get_litellm_params on the retrieve and content paths. gcs_bucket_name keeps precedence when both are present
* style: sort imports in llm_http_handler to satisfy I001 budget
---------
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
(cherry picked from commit 5682592)
Bumps the 12 packages osv-scanner flags on litellm_internal_staging, taking the scan from 24 known vulnerabilities to zero. vcrpy goes to 8.2.1 first so aiohttp can move to 3.14.1 (vcrpy <= 8.1.1 cannot import aiohttp 3.14), then the two aiohttp ignore entries are dropped from osv-scanner.toml. The langchain stack moves together since langchain 1.3.9 requires langgraph 1.2.x. Runtime deps cryptography (48.0.1), starlette (1.3.1), python-multipart (0.0.32), pydantic-settings (2.14.2) and pypdf (6.13.3) are bumped via relock, and the dashboard's js-yaml, ws and form-data overrides are bumped too. Also removes the paths filter on the OSV workflow so it runs on every PR rather than only when a lockfile changes, which is why it never showed up on recent code-only PRs (cherry picked from commit a8a1472)
…1.89.x The #31122 cherry-pick bumps the deps it pins (cryptography 48.0.1, aiohttp 3.14.1, vcrpy 8.2.1, the langchain/langgraph stack) and relocks, but this line's lock baseline kept several ranged runtime deps at versions the ranges still allow, so osv-scanner still flagged them. Pull them to their fixed releases so the scan is clean: - starlette 1.1.0 -> 1.3.1 (GHSA-82w8-qh3p-5jfq, GHSA-jp82-jpqv-5vv3) - python-multipart 0.0.27 -> 0.0.32 (GHSA-5rvq-cxj2-64vf and three others) - pydantic-settings 2.14.1 -> 2.14.2 (GHSA-4xgf-cpjx-pc3j) - pypdf 6.13.2 -> 6.13.3 (GHSA-jm82-fx9c-mx94) - pyjwt 2.12.0 -> 2.13.0 (PYSEC-2026-175/177/178/179) - langsmith 0.8.3 -> 0.8.18 (GHSA-f4xh-w4cj-qxq8) Dashboard build deps (not in the shipped bundle; lockfile hygiene, no UI rebuild): - vite 7.3.2 -> 7.3.5 (GHSA-fx2h-pf6j-xcff, GHSA-v6wh-96g9-6wx3) - esbuild override 0.28.1 (GHSA-g7r4-m6w7-qqqr) - form-data override 4.0.6 (GHSA-hmw2-7cc7-3qxx) After this the only osv-scanner finding is diskcache GHSA-w8v5-vhqr-4h9v, which has no fixed release and is the entry staging's osv-scanner.toml already ignores until a fix lands.
|
|
Contributor
Greptile SummaryBackports six staged fixes onto
Confidence Score: 5/5Safe to merge; all changes are additive bug fixes and the production code is byte-identical to already-merged staging commits modulo import-sort resolutions and two ruff suppressions. The backport is a faithful cherry-pick of six already-reviewed and merged staging commits. Every helper referenced by the picks resolves on this line, the 79 new tests all pass, and the two nits found (buffer-reset ordering in replace_model_in_jsonl and cost-calculation ordering in the interrupted-stream path) are minor accuracy gaps that do not cause incorrect behavior for the normal case. No files require special attention beyond the two flagged nits in litellm/router_utils/batch_utils.py and litellm/proxy/pass_through_endpoints/llm_provider_handlers/anthropic_passthrough_logging_handler.py.
|
| Filename | Overview |
|---|---|
| litellm/proxy/pass_through_endpoints/llm_provider_handlers/anthropic_passthrough_logging_handler.py | Adds interrupted-stream detection, output-token recovery via re-tokenization, a usage-only fallback for large agentic streams when stream_chunk_builder fails, and ensures model_call_details response_cost is written so the pass-through success path logs correct spend. |
| litellm/llms/custom_httpx/llm_http_handler.py | Adds sync/async resumable GCS chunked-upload support to BaseLLMHTTPHandler; large JSONL bodies are streamed in 8 MiB chunks instead of being materialised in full. |
| litellm/llms/vertex_ai/files/transformation.py | Complete rewrite of the Vertex JSONL upload path with lazy iterators and a resumable GCS session so large uploads stay memory-bounded; response retrieval is similarly streaming. |
| litellm/litellm_core_utils/streaming_handler.py | Adds _record_partial_usage_for_failure to stash partial usage/cost on the logging object before failure callbacks run; wraps stream_chunk_builder in try/except at end-of-stream. |
| litellm/proxy/hooks/proxy_track_cost_callback.py | async_post_call_failure_hook now lifts recovered partial spend from request_data onto the failure row instead of always writing 0.0. |
| litellm/router_utils/batch_utils.py | replace_model_in_jsonl rewritten to iterate line-by-line instead of reading the whole file into memory. |
| litellm/batches/batch_utils.py | Adds streaming JSONL helpers _iter_batch_input_lines, _count_entry_tokens, _estimate_batch_entry_tokens; removes list-based equivalents. |
| litellm/proxy/openai_files_endpoints/files_endpoints.py | For purpose=batch uploads, exposes file.file (Starlette-spooled handle) instead of await file.read(); get_first_json_object generalised to accept BinaryIO. |
| litellm/files/utils.py | Expands valid_content_type to cover the full set of MIME types a batch JSONL upload can carry; adds is_batch_jsonl_request. |
| Dockerfile | Wolfi-base digest bumped to patch the openssl CVE; no logic changes. |
Reviews (2): Last reviewed commit: "chore: refresh uv.lock for litellm-enter..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+197
to
229
| if not isinstance(response, ModelResponse): | ||
| return | ||
| if not AnthropicPassthroughLoggingHandler._stream_was_interrupted(all_chunks): | ||
| return | ||
| usage = getattr(response, "usage", None) | ||
| if usage is None: | ||
| return | ||
| output_text = get_content_from_model_response(response) | ||
| if not output_text: | ||
| return | ||
| try: | ||
| recovered_output_tokens = litellm.token_counter( | ||
| model=model, text=output_text, count_response_tokens=True | ||
| ) | ||
| except Exception: | ||
| verbose_proxy_logger.warning( | ||
| "Could not re-tokenize interrupted stream output; " | ||
| "keeping placeholder completion token count." | ||
| ) | ||
| return | ||
| if recovered_output_tokens <= (usage.completion_tokens or 0): | ||
| return | ||
| usage.completion_tokens = recovered_output_tokens | ||
| usage.total_tokens = (usage.prompt_tokens or 0) + recovered_output_tokens | ||
| # Anthropic costing reads completion_tokens_details.text_tokens, so the | ||
| # stale message_start placeholder there must be corrected too or spend | ||
| # stays undercounted even after completion_tokens is fixed. | ||
| details = getattr(usage, "completion_tokens_details", None) | ||
| if details is not None and getattr(details, "text_tokens", None) is not None: | ||
| details.text_tokens = recovered_output_tokens | ||
|
|
||
| @staticmethod | ||
| def _create_anthropic_response_logging_payload( |
Contributor
There was a problem hiding this comment.
When _build_complete_streaming_response fails and _build_usage_only_response_from_chunks produces the response instead, the resulting ModelResponse has content="" (empty message body). _recover_interrupted_stream_output_tokens then calls get_content_from_model_response → empty string → if not output_text: return, so re-tokenization is silently skipped. The usage-only path extracts output_tokens from message_start/message_delta SSE events, so for interrupted agentic streams where both assembly and message_delta arrival fail, the reported output_tokens stays at the message_start placeholder (~1–3 tokens) rather than the actual delivered text tokens.
Comment on lines
+3545
to
+3565
| init_resp.read() | ||
| if init_resp.status_code not in (200, 201): | ||
| init_resp.raise_for_status() | ||
| session_url = init_resp.headers.get(session_url_header) | ||
| if not session_url: | ||
| raise ValueError( | ||
| f"resumable upload: no session URL in '{session_url_header}' header" | ||
| ) | ||
|
|
||
| offset = 0 | ||
| pending: Optional[bytes] = None | ||
| for chunk in self._iter_resumable_chunks(stream.iter_bytes(), chunk_size): | ||
| if pending is not None: | ||
| self._send_resumable_chunk( | ||
| httpx_client, | ||
| session_url, | ||
| base_headers, | ||
| pending, | ||
| offset, | ||
| is_final=False, | ||
| timeout=timeout, |
Contributor
There was a problem hiding this comment.
_send_resumable_chunk calls resp.raise_for_status() before raise ValueError(...). For a non-final chunk that unexpectedly receives a 200 or 201, raise_for_status() is a no-op (those are success codes), so control falls through to raise ValueError(...). Runtime behaviour is correct, but the comment "4xx/5xx raise here" is only accurate for error codes; a developer reading a ValueError: unexpected status 200 in logs may not expect this path. Minor clarity issue.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Collaborator
Author
tin-berri
approved these changes
Jun 25, 2026
hbjydev
pushed a commit
to hbjydev/phoebe
that referenced
this pull request
Jun 25, 2026
….4) (#196) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/berriai/litellm](https://images.chainguard.dev/directory/image/wolfi-base/overview) ([source](https://github.com/BerriAI/litellm)) | patch | `v1.89.3` → `v1.89.4` | --- ### Release Notes <details> <summary>BerriAI/litellm (ghcr.io/berriai/litellm)</summary> ### [`v1.89.4`](https://github.com/BerriAI/litellm/releases/tag/v1.89.4) [Compare Source](BerriAI/litellm@v1.89.4...v1.89.4) ##### Verify Docker Image Signature All LiteLLM Docker images are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](BerriAI/litellm@0112e53). **Verify using the pinned commit hash (recommended):** A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \ ghcr.io/berriai/litellm:v1.89.4 ``` **Verify using the release tag (convenience):** Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/v1.89.4/cosign.pub \ ghcr.io/berriai/litellm:v1.89.4 ``` Expected output: ``` The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key ``` *** ##### What's Changed - chore(release): backport [#​31029](BerriAI/litellm#31029) to stable/1.89.x and cut 1.89.4 by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​31168](BerriAI/litellm#31168) - chore(ui): rebuild dashboard artifacts for stable/1.89.x by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​31170](BerriAI/litellm#31170) - chore(release): backport [#​30787](BerriAI/litellm#30787), [#​30788](BerriAI/litellm#30788), [#​31035](BerriAI/litellm#31035), [#​31036](BerriAI/litellm#31036), [#​31122](BerriAI/litellm#31122), [#​31133](BerriAI/litellm#31133) to stable/1.89.x (litellm-enterprise 0.1.42.post2) by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​31259](BerriAI/litellm#31259) **Full Changelog**: <BerriAI/litellm@v1.89.3...v1.89.4> ### [`v1.89.4`](https://github.com/BerriAI/litellm/releases/tag/v1.89.4) [Compare Source](BerriAI/litellm@v1.89.3...v1.89.4) ##### Verify Docker Image Signature All LiteLLM Docker images are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](BerriAI/litellm@0112e53). **Verify using the pinned commit hash (recommended):** A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \ ghcr.io/berriai/litellm:v1.89.4 ``` **Verify using the release tag (convenience):** Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/v1.89.4/cosign.pub \ ghcr.io/berriai/litellm:v1.89.4 ``` Expected output: ``` The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key ``` *** ##### What's Changed - chore(release): backport [#​31029](BerriAI/litellm#31029) to stable/1.89.x and cut 1.89.4 by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​31168](BerriAI/litellm#31168) - chore(ui): rebuild dashboard artifacts for stable/1.89.x by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​31170](BerriAI/litellm#31170) - chore(release): backport [#​30787](BerriAI/litellm#30787), [#​30788](BerriAI/litellm#30788), [#​31035](BerriAI/litellm#31035), [#​31036](BerriAI/litellm#31036), [#​31122](BerriAI/litellm#31122), [#​31133](BerriAI/litellm#31133) to stable/1.89.x (litellm-enterprise 0.1.42.post2) by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​31259](BerriAI/litellm#31259) **Full Changelog**: <BerriAI/litellm@v1.89.3...v1.89.4> </details> --- ### Configuration 📅 **Schedule**: (in timezone Europe/London) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yMzQuMiIsInVwZGF0ZWRJblZlciI6IjQzLjIzNC4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJyZW5vdmF0ZS9jb250YWluZXIiLCJ0eXBlL3BhdGNoIl19--> Reviewed-on: https://forgejo.hayden.moe/hayden/phoebe/pulls/196
doonga
pushed a commit
to greyrock-labs/home-ops
that referenced
this pull request
Jun 25, 2026
….4) (#386) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/berriai/litellm](https://images.chainguard.dev/directory/image/wolfi-base/overview) ([source](https://github.com/BerriAI/litellm)) | patch | `v1.89.3` → `v1.89.4` | --- ### Release Notes <details> <summary>BerriAI/litellm (ghcr.io/berriai/litellm)</summary> ### [`v1.89.4`](https://github.com/BerriAI/litellm/releases/tag/v1.89.4) [Compare Source](BerriAI/litellm@v1.89.4...v1.89.4) ##### Verify Docker Image Signature All LiteLLM Docker images are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](BerriAI/litellm@0112e53). **Verify using the pinned commit hash (recommended):** A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \ ghcr.io/berriai/litellm:v1.89.4 ``` **Verify using the release tag (convenience):** Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/v1.89.4/cosign.pub \ ghcr.io/berriai/litellm:v1.89.4 ``` Expected output: ``` The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key ``` *** ##### What's Changed - chore(release): backport [#​31029](BerriAI/litellm#31029) to stable/1.89.x and cut 1.89.4 by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​31168](BerriAI/litellm#31168) - chore(ui): rebuild dashboard artifacts for stable/1.89.x by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​31170](BerriAI/litellm#31170) - chore(release): backport [#​30787](BerriAI/litellm#30787), [#​30788](BerriAI/litellm#30788), [#​31035](BerriAI/litellm#31035), [#​31036](BerriAI/litellm#31036), [#​31122](BerriAI/litellm#31122), [#​31133](BerriAI/litellm#31133) to stable/1.89.x (litellm-enterprise 0.1.42.post2) by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​31259](BerriAI/litellm#31259) **Full Changelog**: <BerriAI/litellm@v1.89.3...v1.89.4> ### [`v1.89.4`](https://github.com/BerriAI/litellm/releases/tag/v1.89.4) [Compare Source](BerriAI/litellm@v1.89.3...v1.89.4) ##### Verify Docker Image Signature All LiteLLM Docker images are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](BerriAI/litellm@0112e53). **Verify using the pinned commit hash (recommended):** A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \ ghcr.io/berriai/litellm:v1.89.4 ``` **Verify using the release tag (convenience):** Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/v1.89.4/cosign.pub \ ghcr.io/berriai/litellm:v1.89.4 ``` Expected output: ``` The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key ``` *** ##### What's Changed - chore(release): backport [#​31029](BerriAI/litellm#31029) to stable/1.89.x and cut 1.89.4 by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​31168](BerriAI/litellm#31168) - chore(ui): rebuild dashboard artifacts for stable/1.89.x by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​31170](BerriAI/litellm#31170) - chore(release): backport [#​30787](BerriAI/litellm#30787), [#​30788](BerriAI/litellm#30788), [#​31035](BerriAI/litellm#31035), [#​31036](BerriAI/litellm#31036), [#​31122](BerriAI/litellm#31122), [#​31133](BerriAI/litellm#31133) to stable/1.89.x (litellm-enterprise 0.1.42.post2) by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​31259](BerriAI/litellm#31259) **Full Changelog**: <BerriAI/litellm@v1.89.3...v1.89.4> </details> --- ### Configuration 📅 **Schedule**: (in timezone America/New_York) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yMzQuMSIsInVwZGF0ZWRJblZlciI6IjQzLjIzNC4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJyZW5vdmF0ZS9jb250YWluZXIiLCJ0eXBlL3BhdGNoIl19--> Reviewed-on: https://git.greyrock.io/greyrock-labs/home-ops/pulls/386
blake-hamm
added a commit
to blake-hamm/bhamm-lab
that referenced
this pull request
Jun 27, 2026
…to v1.89.4 (#228) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [https://github.com/BerriAI/litellm.git](https://github.com/BerriAI/litellm) | patch | `v1.89.3` → `v1.89.4` | --- ### Release Notes <details> <summary>BerriAI/litellm (https://github.com/BerriAI/litellm.git)</summary> ### [`v1.89.4`](https://github.com/BerriAI/litellm/releases/tag/v1.89.4) [Compare Source](BerriAI/litellm@v1.89.3...v1.89.4) #### Verify Docker Image Signature All LiteLLM Docker images are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](BerriAI/litellm@0112e53). **Verify using the pinned commit hash (recommended):** A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \ ghcr.io/berriai/litellm:v1.89.4 ``` **Verify using the release tag (convenience):** Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/v1.89.4/cosign.pub \ ghcr.io/berriai/litellm:v1.89.4 ``` Expected output: ``` The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key ``` *** #### What's Changed - chore(release): backport [#​31029](BerriAI/litellm#31029) to stable/1.89.x and cut 1.89.4 by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​31168](BerriAI/litellm#31168) - chore(ui): rebuild dashboard artifacts for stable/1.89.x by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​31170](BerriAI/litellm#31170) - chore(release): backport [#​30787](BerriAI/litellm#30787), [#​30788](BerriAI/litellm#30788), [#​31035](BerriAI/litellm#31035), [#​31036](BerriAI/litellm#31036), [#​31122](BerriAI/litellm#31122), [#​31133](BerriAI/litellm#31133) to stable/1.89.x (litellm-enterprise 0.1.42.post2) by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​31259](BerriAI/litellm#31259) **Full Changelog**: <BerriAI/litellm@v1.89.3...v1.89.4> </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yMjAuMCIsInVwZGF0ZWRJblZlciI6IjQzLjIyMC4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJkZXBlbmRlbmNpZXMiXX0=--> Co-authored-by: Renovate Bot <renovate@bhamm-lab.com> Reviewed-on: https://codeberg.org/blake-hamm/bhamm-lab/pulls/228
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relevant issues
Backports six already-merged staging fixes onto
stable/1.89.xand relocks the runtimedependencies so the line clears its known CVEs. Three of the picks close cost-tracking leaks
on interrupted and agentic Anthropic pass-through streams, one streams large Vertex batch
JSONL uploads to remove an out-of-memory failure, one re-pins the wolfi-base image digest to
patch the openssl CVE in the shipped image, and one moves the osv-flagged dependencies
forward. There is no
litellmversion bump; the line tip is already at1.89.4and thatversion has not been released yet (no
release/v1.89.4branch orv1.89.4tag), so thesepicks ride the pending
1.89.4. Because #31036 modifies an enterprise source file(
enterprise/litellm_enterprise/proxy/hooks/managed_files.py), this also cutslitellm-enterprise0.1.42.post2Pre-Submission checklist
What is included
In cherry-pick (merge) order:
The three stream fixes are one stack; #31035 builds on #30787 and #30788, so they are applied
in that order. Every helper they reference (calculate_total_usage, _split_sse_chunk_into_events,
_build_complete_streaming_response, AnthropicConfig.calculate_usage, map_finish_reason,
get_content_from_model_response) resolves on 1.89.x
0.1.42.post2is the correct enterprise increment:0.1.42and0.1.42.post1(1.88.x's) and0.1.43(staging's) are all already published to PyPI, so0.1.42.post2is the next freepost-release and sorts
0.1.42 < .post1 < .post2 < 0.1.43Adaptation notes
The production code of the six picks is byte-identical to staging except for the divergences
below, which are import-sort resolutions, a test import name, and two lint suppressions this
line's stricter ruff config requires
fix(proxy): record partial spend on the failure row for interrupted streams #30788 adds
# noqa: PLR0915toasync_post_call_failure_hook(52 statements) and fix(passthrough,streaming): recover cost on interrupted and agentic Anthropic streams #31035adds it to the new
_build_usage_only_response_from_chunks(72 statements) while preservingthe line's existing suppression on
batch_creation_handler, because this line's ruff enforcesPLR0915 and staging's does not. Each pick's new tests are appended at this line's end of file
because the line orders neighbor test classes differently than staging; no class is duplicated
router and batch utilities, the types and the enterprise managed-files hook) and all test
files, including the new streaming suite, are byte-identical to staging. Two source files
diverge:
llm_http_handler.pykept the streaming body verbatim and dropped three importsunused on this line (
functools.lru_cache,asyncify.run_async_function,types.realtime.RealtimeQueryParams) plus a duplicateurllib.parseimport the mergeintroduced;
batch_rate_limiter.pykept the line's typing-import form plus the one name thepick uses (
Iterable) and droppedNoReturn. One new test imported the publicOPTIONAL_KWARGS_KEYS; on this line that frozenset is named_OPTIONAL_KWARGS_KEYS(thepublic alias came from an unrelated later commit), so the test points at the line's name; same
object, same assertion
edits do not exist on any stable line, so they are dropped;
uv.lockand the dashboardpackage-lock.jsonwere regenerated on the line rather than carried from staging, since thestaging lockfiles encode staging's full dependency graph. The follow-on chore(deps) commit
pulls the remaining ranged runtime deps (starlette, python-multipart, pydantic-settings, pypdf,
pyjwt, langsmith) and dashboard build deps (vite, esbuild, form-data) to their fixed releases,
because this line's lock baseline kept them at versions the ranges still allowed
Known noise on this line
The targeted-test baseline captured on the line tip before any pick has one pre-existing failure,
tests/test_litellm/proxy/test_proxy_utils.py::test_get_custom_url. It asserts a host stringequals
http://0.0.0.0:4000/...but the environment resolves it tohttp://localhost:4000/....None of the picks touch
get_custom_url; this failure is present with and without the backportand should be discounted
Screenshots / Proof of Fix
Dependency CVE scan (local osv-scanner 2.3.8) against the branch lockfiles, after the relock:
The one remaining finding has no fixed release and is the entry staging's osv-scanner.toml
already ignores until a fix lands
Openssl image CVE (the wolfi-base digest in #31133). Built the non-root image from this branch
and scanned it with grype 0.114.0:
The only grype findings on the image are three Medium CVEs in the python-3.13 apk shipped by
wolfi-base (CVE-2025-15366, CVE-2025-15367, CVE-2026-12003), unrelated to openssl and not
addressed by the digest bump
Live proxy on the line (per-worktree proxy, real provider keys), before and after the picks:
The new dependencies (cryptography 48.0.1, starlette 1.3.1 and the rest of the relock) boot the
proxy and serve completions without regression. The interrupted-stream cost recovery and the
1GB+ Vertex upload need a live interrupted stream and live GCS/Vertex credentials respectively,
neither available here, so they are covered by the unit tests below rather than a live replay
Targeted tests, judged as a delta:
The targeted set is the picks' own test files plus the mirrored tests for every touched source.
Each pick's new tests pass explicitly, including the new 696-line Vertex GCS resumable-upload
suite, the interrupted-stream recovery tests, the partial-spend failure-row tests and the
batch-file-validation tests
A behavioral verification pass with adversarial subagents (deep, five lenses) confirmed the
backport is a faithful port with no introduced regression: every identifier the picked diffs
reference resolves on this line, each pick's own tests pass as a clean delta against the lone
baseline failure, and no existing caller of a modified function is broken on 1.89.x. The
divergences from the upstream commits are limited to the import-sort resolutions, the test
import name and the two PLR0915 suppressions documented above
Type
🐛 Bug Fix
🚄 Infrastructure
Changes
Six cherry-picks onto stable/1.89.x (interrupted-stream cost recovery, Vertex batch JSONL
streaming, the wolfi-base openssl digest, and the osv dependency bumps), a runtime-dependency
relock that clears the line's remaining osv findings, and the
litellm-enterprise0.1.42.post2cut required because a pick touches an enterprise source file. No
litellmversion bump