fix(proxy): release budget reservation when a request is cancelled mid-flight by Bytechoreographer · Pull Request #30522 · BerriAI/litellm

Bytechoreographer · 2026-06-16T09:02:17Z

Relevant issues

Intermittent 429 Budget has been exceeded! Key=... Current cost: 805.8, Max budget: 800.0 on keys whose real spend is far below budget (~$280 of an $800 budget), self-healing after a pause and recurring under load.

Pre-Submission checklist

I have added meaningful tests (mutation-verified)
My PR passes all unit tests on make test-unit
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Type

🐛 Bug Fix

Changes

The pre-call budget reservation (reserve_budget_for_request) increments the cross-pod spend counter by a request's worst-case cost up front, then reconciles it to the actual cost on success (cost callback) or on error (async_post_call_failure_hook). Neither runs when a streaming request is cancelled mid-flight: a client disconnect or timeout surfaces as asyncio.CancelledError (a BaseException, so except Exception misses it) or GeneratorExit in the streaming generator, so the reservation leaks. Under an agent retry storm the leaked reservations accumulate, pin the counter above the key's real spend, and 429 subsequent requests; the counter only recovers when its TTL lapses, which is why it is intermittent and self-healing.

The fix releases the reservation in async_streaming_data_generator (which async_sse_data_generator delegates to, so it covers OpenAI-style chat, Anthropic /v1/messages, and Google streaming) on the (CancelledError, GeneratorExit) path. release_budget_reservation_on_cancel wraps release_budget_reservation in asyncio.shield so it finishes despite the in-progress cancellation, short-circuits on the reservation's finalized flag, and is best-effort (a failing release is swallowed rather than replacing the in-flight cancellation).

Refund is gated on whether any chunk reached the client. On cancellation the stream wrapper logs no cost (CancelledError / GeneratorExit skip both the success and failure handlers in CustomStreamWrapper.__anext__), so a partial stream is never billed to the DB or the counter; refunding a stream the caller already consumed would let them read partial output then disconnect to dodge the charge. The reservation is therefore released only when no chunk was produced.

Non-streaming cancellation is intentionally not special-cased. A completed non-streaming response is reconciled to its real cost by the success cost callback, which fires independently of the client connection, so the reservation is charged correctly without any release on cancel. Reducing how aggressive the reservation estimate is (worst-case max_tokens) is a separate policy question and is left unchanged.

Tests in tests/test_litellm/proxy/test_budget_reservation.py: cancel-release gives the counter back and is idempotent; a finalized reservation is left untouched; a failing release is swallowed; a stream cancelled before any chunk is refunded while a stream cancelled after a chunk keeps its hold (removing the gate flips that test red).

Screenshots / Proof of Fix

Live proxy run to be added.

codecov · 2026-06-16T09:06:19Z

Codecov Report

❌ Patch coverage is 78.84615% with 11 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/common_request_processing.py	61.90%	8 Missing ⚠️
litellm/proxy/spend_tracking/budget_reservation.py	90.32%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!

greptile-apps · 2026-06-16T09:07:40Z

Greptile Summary

Fixes intermittent 429s caused by budget reservations leaking when streaming requests are cancelled mid-flight (CancelledError / GeneratorExit bypass the except Exception handlers that normally reconcile the pre-call reservation). The fix adds a delivered_chunk sentinel and release_budget_reservation_on_cancel to async_streaming_data_generator; reservations cancelled before any output is delivered are reconciled to the estimated input-token cost rather than zero, and asyncio.shield keeps the reconcile running after the outer task is cancelled.

budget_reservation.py: adds estimate_request_input_cost (in-memory, no I/O) called inside reserve_budget_for_request to store an input_cost floor, and release_budget_reservation_on_cancel that reconciles to that floor via a shielded coroutine; the finalized flag makes it idempotent with the normal success/failure paths.
common_request_processing.py: restructures the fast-path vs slow-path chunk processing into a single if not fast_path: guard, sets delivered_chunk = True before each yield (so a GeneratorExit raised at the suspended yield still records delivery), and calls release_budget_reservation_on_cancel from the cancel handler only when no chunk has been delivered.
test_budget_reservation.py: adds eight new mock-only tests covering cancel-before-chunk, cancel-after-chunk, idempotency, error swallowing, slow-path cancel, and aclose() scenarios.

Confidence Score: 5/5

Safe to merge — the change is well-scoped to the cancellation path and the finalized flag prevents it from interfering with the normal success/failure reconciliation flow.

The core mechanism is correct: delivered_chunk is set before yield so GeneratorExit at the suspend point is handled properly; asyncio.shield plus the swallow-all except block ensure the reconcile never disrupts Starlette's disconnect handling; the idempotency guard prevents double-reconciliation with the normal cost callbacks. The two observations raised are about a known best-effort limitation (shutdown races) and a clarifying question about non-streaming paths — neither represents a defect in the changed code paths.

No files require special attention; the logic in budget_reservation.py and common_request_processing.py is consistent, and the test coverage is thorough for the new cancellation scenarios.

Important Files Changed

Filename	Overview
litellm/proxy/common_request_processing.py	Adds `delivered_chunk` flag (set before `yield`, never after) to gate budget release on cancellation; restructures fast-path/slow-path logic into a single `if not fast_path:` block. Flag placement is correct for both `CancelledError` (during hook await) and `GeneratorExit` (at the yield point). The `getattr(..., "budget_reservation", None)` default gracefully handles requests that never called `reserve_budget_for_request`.
litellm/proxy/spend_tracking/budget_reservation.py	Adds `estimate_request_input_cost` / `_estimate_request_input_cost_for_model` (in-memory, no I/O) and `release_budget_reservation_on_cancel`, which reconciles to input-token cost via `asyncio.shield` so the inner coroutine survives outer-task cancellation. The `finalized` idempotency guard is correct; `except (asyncio.CancelledError, Exception): pass` swallows all swallowable errors. Shutdown-race is a known best-effort limitation.
tests/test_litellm/proxy/test_budget_reservation.py	New tests cover: cancel before any chunk (reconciles to input cost), cancel after chunk (reservation kept), finalized reservation is a no-op, release errors are swallowed, slow-path cancellation before yield, `aclose()` after a chunk, and a successful slow-path yield. All use mocks, no real network calls — consistent with existing test style in this file.

_{Reviews (5): Last reviewed commit: "fix(proxy): reconcile a cancelled reserv..." | Re-trigger Greptile}

veria-ai · 2026-06-16T09:09:12Z

PR overview

This PR updates proxy request handling so budget reservations are released or reconciled when an in-flight request is cancelled. The touched streaming request path adjusts cancellation cleanup around partial stream progress and cost reservation handling.

There is one open issue remaining after four have been addressed. The remaining concern is that streaming cancellations can still undercount budget usage if provider output has been received but not yet delivered to the client during a slow hook or callback. That leaves a path for repeated cancelled streams to consume provider output outside the configured budget, so the PR is improved but not fully closed from a security perspective.

Open issues (1)

Medium: Streaming budget undercount — litellm/proxy/common_request_processing.py:2597

Fixed/addressed: 4 · PR risk: 6/10

Bytechoreographer · 2026-06-16T09:26:14Z

Addressed the cancelled-stream bypass in 6d6ebca. Confirmed that on cancellation the stream wrapper logs nothing (CancelledError and GeneratorExit are BaseException, so both the success cost callback and the failure handler are skipped in CustomStreamWrapper.anext), so a partial stream's cost is never billed to the DB or the counter. Refunding such a stream would let a caller read partial output then disconnect to avoid the charge.

The refund is now gated on whether any chunk reached the client. A non-streaming cancellation returns nothing to the caller and is still refunded. For streaming, the reservation is released only when no chunk was produced; once the upstream yields a chunk the hold is kept. Regression tests cover both directions (no-chunk refunds, post-chunk keeps the hold) and the post-chunk test fails if the gate is removed.

Bytechoreographer · 2026-06-16T09:48:55Z

Addressed in 88dbd1f. The non-streaming cancel handler was removed entirely. It sat inside the post-call try block (after response = responses[1]), so it only ever ran once the model response had already been obtained, which is exactly the completed-call-refund you flagged. A completed non-streaming response is reconciled to its actual cost by the success cost callback, which fires independently of the client connection, so no release is needed on that path and removing it closes the refund bypass. Cancellation during the upstream call itself (cancel_on_disconnect) raises before any response exists, outside this try, and is out of scope for this PR.

Bytechoreographer · 2026-06-16T09:58:46Z

@greptileai

Bytechoreographer · 2026-06-16T12:36:56Z

Fixed the 4/5 finding in 68651aa. You were right: delivered_chunk was set at loop entry, before the slow-path async_post_call_streaming_hook await, so a cancellation at that await on the first chunk saw the flag already true and skipped the refund on the non-fast path.

The slow-path processing is now folded under a single if not fast_path branch with one shared yield, and delivered_chunk is set only after that yield, so it is true only once a chunk has actually reached the client. Added test_streaming_cancel_in_slow_path_before_yield_refunds, which forces the slow path (include_cost_in_streaming_usage) and cancels at the hook await before any yield; it asserts the reservation is refunded and fails if the flag is moved back to loop entry.

@greptileai

greptile-apps · 2026-06-16T12:37:43Z

The fix is correct. The restructured code now has a single shared yield at the bottom of the loop body, and delivered_chunk = True is set only after that yield — meaning a cancellation anywhere in the slow-path hook await leaves the flag False and the refund fires as expected. The fast path falls through to the same yield naturally without any separate branch.

The new test test_streaming_cancel_in_slow_path_before_yield_refunds pins this precisely: async_post_call_streaming_hook raises CancelledError, received == [], counter returns to 0, and finalized is set. If someone moves delivered_chunk = True back above the hook call, that test will catch it.

The original concern is fully addressed — this looks good.

Tip: You can customize Greptile's behavior for this repo with .greptile/rules.md and .greptile/config.json.

Bytechoreographer · 2026-06-16T12:50:47Z

Fixed in 44353a9. You're right about the async-generator suspension: with the flag set after the yield, a GeneratorExit at the suspended yield (client consumed a chunk then disconnected) skipped the assignment and refunded a delivered stream.

Moved delivered_chunk to immediately before the yield, after the slow-path hook await. That covers both directions: a cancellation during the slow-path await (before the flag) still refunds since nothing was sent, and a disconnect at the suspended yield (after the flag) keeps the hold since a chunk was delivered. Added test_streaming_disconnect_after_consuming_chunk_keeps_reservation, which consumes one chunk then calls aclose() and asserts the reservation is kept; it fails if the flag is moved back after the yield.

@greptileai

greptile-apps · 2026-06-16T12:51:36Z

The fix is correct. The semantics are exactly right:

delivered_chunk = True is set after the slow-path hook await but before yield — so a CancelledError during the hook (nothing sent) leaves it False, and a GeneratorExit at the suspended yield (client consumed the chunk) sees it as True.
The new regression test test_streaming_disconnect_after_consuming_chunk_keeps_reservation directly exercises the failure mode: consume one chunk, aclose() the generator, assert the reservation is not refunded.
The restructured if not fast_path: block cleanly groups the slow-path work, making the flag placement unambiguous.

LGTM.

Bytechoreographer · 2026-06-16T12:52:29Z

@greptileai

greptile-apps · 2026-06-16T12:58:48Z

Want your agent to iterate on Greptile's feedback? Try greploops.

Sameerlite · 2026-06-17T03:57:43Z

Thanks for your contribution! A few things to get this ready:

Merge conflicts — could you rebase/merge to resolve the conflicts with the base branch?

We're also triggering a Greptile code review:

@greptileai

…livered The pre-call budget reservation increments the cross-pod spend counter by a request's worst-case cost, then reconciles it on success (cost callback) or error (failure hook). A client disconnect or timeout cancels the request and surfaces as CancelledError / GeneratorExit, which neither path catches, so the reservation leaks. Under a retry storm the leaked holds accumulate, pin the counter above real spend, and return spurious 429 "Budget has been exceeded" to keys whose spend is far below budget; the counter only recovers when its TTL lapses, so the failure is intermittent and self-healing. Release the reservation in async_streaming_data_generator (which the Anthropic and Google SSE generators delegate to) on the (CancelledError, GeneratorExit) path, alongside the existing max_parallel_requests release. release_budget_ reservation_on_cancel runs under asyncio.shield so it completes despite the in-progress cancellation, is guarded by the reservation's finalized flag, and swallows a failing release so it cannot replace the in-flight cancellation. The refund is gated on whether a chunk reached the client. The flag is set immediately before the yield, after the slow-path hook await: an async generator suspends at the yield, so a GeneratorExit on disconnect after a delivered chunk sees it True (keep the hold), while a cancellation during the slow-path await leaves it False (refund, nothing sent). A non-streaming cancellation delivers nothing and a completed non-streaming response is reconciled by the success callback, so neither needs a release here. Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

Bytechoreographer · 2026-06-17T04:22:10Z

Thanks @Sameerlite. Rebased onto the latest litellm_internal_staging and the conflicts are resolved; it is now a single clean commit and the PR shows mergeable again.

The only overlap was in async_streaming_data_generator, which staging recently refactored to add the stream_completed / client_disconnected flags and the _finalize_streaming_generator_cleanup finally. I integrated the budget-reservation release into that structure rather than reverting any of it: on the (CancelledError, GeneratorExit) path it now also calls release_budget_reservation_on_cancel, gated on a delivered_chunk flag set immediately before the yield. That way a disconnect after a chunk was delivered keeps the hold, while a cancellation before anything reached the client refunds it. budget_reservation.py and the tests did not conflict. Greptile's earlier except-broadening suggestion is preserved (co-authored on the commit).

40 unit tests pass locally and the new CI run is green so far. Ready for another look when you have a chance.

Bytechoreographer · 2026-06-17T05:01:48Z

Addressed in 01983c7. You're right that by the time this generator is consuming the response the provider call was already dispatched, so the input tokens were billed even with no chunk delivered, and refunding to zero let a caller abort pre-token to dodge that charge.

The cancelled reservation now reconciles to the request's input-token cost instead of zero. The input cost is computed at reservation time (estimate_request_input_cost) and stored on the reservation; on a mid-flight cancel the worst-case output portion is released (so a legitimate disconnect no longer pins the counter and 429s the key) while the input the provider already processed is charged. A delivered-output stream still keeps its full hold via the delivered_chunk gate. Regression tests assert the counter reconciles to the input cost on both the fast-path and slow-path pre-token cancellations, and reconciling to zero fails them.

A streaming request cancelled before the first chunk previously reconciled its reservation to zero and finalized it. But by the time the generator is consuming the response the provider call was already dispatched, so the input tokens were billed even though no chunk reached the client, and the success/failure cost callbacks are skipped on cancellation. Refunding to zero let a caller send an expensive request and abort pre-token to dodge the input charge. Compute the request's input-token cost at reservation time and reconcile the cancelled reservation to it instead of zero. The worst-case output portion of the reservation is still released (so a legitimate mid-flight cancellation no longer pins the counter and 429s the key), while the input the provider already processed is charged.

veria-ai · 2026-06-17T05:26:20Z

                    user_api_key_dict
                )
                client_disconnected = True
+            if not delivered_chunk:


Medium: Streaming budget undercount

delivered_chunk stays false while async_post_call_streaming_hook is awaited, but at that point the upstream iterator has already yielded a provider chunk. A caller can disconnect during a slow guardrail/custom callback after provider output has been generated, and this branch reconciles the reservation to input_cost only, letting repeated cancelled streams consume provider output outside the configured budget. Track whether any upstream chunk was received, not only whether it was delivered to the client, before applying the input-only refund.

Bytechoreographer · 2026-06-17T05:27:48Z

Claude is shit

Bytechoreographer · 2026-06-17T06:46:03Z

Thanks. This undercount is real but I've deliberately not closed it by charging for received-but-undelivered output, because the only ways to do that both regress the bug this PR fixes or go well beyond its scope:

Tracking received (not delivered) and then keeping the reservation means keeping the worst-case max_tokens hold for any stream cancelled mid-flight. That is exactly the leak this PR exists to fix: those worst-case holds accumulating on the cross-pod counter are what pinned a key at Current cost: 805.8 while its real spend was ~$280, returning spurious 429s to a key far under budget. Re-keeping them reintroduces that production incident.

The fully correct alternative is to reconcile to the actual generated-output cost (input + output produced so far), not the worst-case reservation. That requires computing partial output cost in the cancellation path, and more fundamentally litellm does not bill a cancelled stream at all today (on CancelledError/GeneratorExit neither the success nor failure cost callback runs, so the DB spend is never charged either). So even a perfectly accurate counter would diverge from the DB and be lost on the next reseed. Billing cancelled partial streams is the real fix and is separate, larger work.

Given that, the reconcile-to-input-cost here is an intentional floor: it charges the input the provider definitely received, releases the worst-case output reservation so legitimate mid-flight cancellations stop 429ing under-budget keys, and the only under-charge is the output produced during a slow post-call hook before the first yield (typically a single chunk). I'd propose tracking the precise cancelled-stream cost as a follow-up rather than regressing the 429 fix here, but happy to go whichever way the maintainers prefer.

jingyu-lin · 2026-06-17T09:34:41Z

Adding a related observation: this PR fixes the budget leak, but there's another half of the problem — when a client aborts a streaming request mid-flight, the already-generated tokens are never written to SpendLogs at all.

We discovered this while investigating billing reconciliation discrepancies in our production environment (v1.87.0):

In async_streaming_data_generator, _fire_deferred_stream_logging() is only called after the async for loop completes normally
asyncio.CancelledError from client disconnect is a BaseException, which does NOT enter the except Exception branch
Result: tokens already received from upstream are never accounted for in SpendLogs (total_tokens=0, spend=0)
But the upstream providers (Azure/OpenAI/Google) still charge us for them

Impact on reconciliation:

5% abort rate → 5% billing discrepancy, fully absorbed by the platform
This discrepancy is invisible when looking only at SpendLogs; you only find it when you cross-reference against provider invoices

PR #30522 fixes the budget reservation release, which is great. However, the actual billing logging problem still exists.

Suggested fix direction:

In the finally block, check if _deferred_stream_complete_args was ever set
If not set (meaning abnormal termination), use stream_chunk_builder to assemble partial usage from already-received chunks
Shield the logging operation with asyncio.shield so it completes despite the cancellation

This is a critical issue for all users doing proxy-based billing — we've had thousands of dollars in unreconciled charges for two consecutive months because of this.

补充一个相关发现：这个 PR 修复了预算泄漏，但还有一半的问题——客户端 abort 场景下，已生成的 token 完全不会写入 SpendLogs。

我们在生产环境（v1.87.0）排查对账差额时发现：

async_streaming_data_generator 中，_fire_deferred_stream_logging() 只在 async for 循环正常结束后调用
客户端断开导致的 asyncio.CancelledError 是 BaseException，不进 except Exception 分支
结果是：已收到的上游 token 完全不计入 SpendLogs（total_tokens=0, spend=0）
但云商（Azure/OpenAI/Google）那边照样收我们的钱

对差额的影响：

5% abort 率 → 5% 对账差额，完全由平台承担
这个差额是隐形的，查 SpendLogs 看不出来，只有和云商账单对才发现

PR #30522 修了预算预扣的释放，这很好。但实际计费落库的问题还在。

建议修复方向：

在 finally 块里检查 _deferred_stream_complete_args 是否被设置
如果没设置（说明是异常中止），用 stream_chunk_builder 拼已收到的 chunk 计算 partial usage
用 asyncio.shield 保护落库过程

Bytechoreographer · 2026-06-17T09:57:09Z

@jingyu-lin thank you, this is a great find and it lines up exactly with what came up in review here. It is the other half of the same root cause: on a mid-stream client disconnect, CancelledError / GeneratorExit are BaseException, so neither the success nor the failure logging path runs. This PR addresses the budget-reservation side (the pre-call hold leaks and pins the cross-pod counter, producing spurious 429s on under-budget keys). What you are describing is the SpendLogs side: the tokens already produced upstream are never written, so the platform eats the provider cost.

The two are linked. Once a cancelled stream logs its partial usage the way you describe (assemble partial usage from received chunks via stream_chunk_builder in the finally, shielded), the existing cost callback fires on cancel and reconciles the reservation to the real partial cost as a side effect. That also resolves the reviewer concern on this PR about the cancel path reconciling to input cost only, since it would then reconcile to actual generated output instead.

@Sameerlite to keep this PR focused and mergeable I would prefer to land the reservation-release fix here (it stops the 429s on its own), and do the cancelled-stream SpendLogs logging as a focused follow-up rather than expand this one. The reconcile-to-input-cost in this PR is an intentional floor in the meantime. Does that scoping work for you? @jingyu-lin if you are open to it I am happy to collaborate on the follow-up since you have already root-caused it in production.

Bytechoreographer · 2026-06-17T10:11:35Z

Opened the SpendLogs follow-up as #30630 (@jingyu-lin's billing-side fix). This PR stays scoped to the budget-reservation release; #30630 records the partial spend for cancelled streams. cc @Sameerlite

@jingyu-lin

On a mid-stream client disconnect the stream never reaches normal completion: CancelledError / GeneratorExit are BaseException, so neither the success nor the failure logging path runs, and the assembled-response success logging that writes SpendLogs never fires. The tokens already produced upstream are billed by the provider but never recorded on the proxy, so spend undercounts by roughly the abort rate; the gap is invisible in SpendLogs and only shows up against provider invoices. In the shielded streaming cleanup, when a client disconnect is recorded, assemble the partial usage from the chunks received so far via stream_chunk_builder and dispatch success logging for it. dispatch_success_handlers de-dupes via has_dispatched_final_stream_success, so it is a no-op when normal completion already logged, and it only runs on the cancellation path (the exception path sets stream_completed and already emits a failure log). Reported and root-caused in production by @jingyu-lin. Complements BerriAI#30522, which releases the budget reservation on the same cancellation path.

@jingyu-lin

On a mid-stream client disconnect the stream never reaches normal completion: CancelledError / GeneratorExit are BaseException, so neither the success nor the failure logging path runs, and the assembled-response success logging that writes SpendLogs never fires. The tokens already produced upstream are billed by the provider but never recorded on the proxy, so spend undercounts by roughly the abort rate; the gap is invisible in SpendLogs and only shows up against provider invoices. In the shielded streaming cleanup, when a client disconnect is recorded, assemble the partial usage from the chunks received so far via stream_chunk_builder and dispatch success logging for it. dispatch_success_handlers de-dupes via has_dispatched_final_stream_success, so it is a no-op when normal completion already logged, and it only runs on the cancellation path (the exception path sets stream_completed and already emits a failure log). Reported and root-caused in production by @jingyu-lin. Complements BerriAI#30522, which releases the budget reservation on the same cancellation path.

jingyu-lin · 2026-06-17T14:44:29Z

@Bytechoreographer Thank you for the incredibly fast and high-quality fix!

This was a silent killer in production — the discrepancy was invisible in SpendLogs and only showed up against provider invoices.

Just to give you an update from our side:

We've already cherry-picked this commit onto our v1.87 staging branch for pre-release validation
On our production v1.87 deployment, we're seeing ~2,000 disconnected streaming requests on GPT-5.5 alone
We're doing manual reconciliation on those first, and will report back immediately once the stable release is out and we've run our abort regression suite.

Looking forward to the official release!

Sameerlite · 2026-06-18T03:53:31Z

Thanks for the PR! A couple of things to get this over the finish line:

Could you add proof of the change working (screenshots, test output, or a sample request/response)? Even a quick curl before/after really speeds up the review.

Triggering Greptile for a code review in the meantime:

@greptileai

@jingyu-lin

On a mid-stream client disconnect the stream never reaches normal completion: CancelledError / GeneratorExit are BaseException, so neither the success nor the failure logging path runs, and the assembled-response success logging that writes SpendLogs never fires. The tokens already produced upstream are billed by the provider but never recorded on the proxy, so spend undercounts by roughly the abort rate; the gap is invisible in SpendLogs and only shows up against provider invoices. In the shielded streaming cleanup, when a client disconnect is recorded, assemble the partial usage from the chunks received so far via stream_chunk_builder and dispatch success logging for it. dispatch_success_handlers de-dupes via has_dispatched_final_stream_success, so it is a no-op when normal completion already logged, and it only runs on the cancellation path (the exception path sets stream_completed and already emits a failure log). Reported and root-caused in production by @jingyu-lin. Complements BerriAI#30522, which releases the budget reservation on the same cancellation path.

@staticmethod

* fix(proxy): bump health-check max_tokens default to 16 for GPT-5 compatibility (#30708) OpenAI GPT-5 models require max_completion_tokens >= 16. Health checks were using 5 (proxy/health_check.py) and 10 (health_check_helpers.py), causing failures on GPT-5 models. Fixes #23836 * fix: increase health check max_tokens from 5 to 16 (#23836) (#26610) GPT-5 models enforce a minimum of 16 for max_output_tokens. The current default of 5 still causes health checks to fail for these models. Bump the non-wildcard default to 16 — the smallest value that satisfies all known provider minimums while keeping health checks lightweight. Also tightens the wildcard test assertion from a weak disjunctive check to strict key-absence. Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix: ensure checks show gemini-3-flash-preview supports responseJsonS… (#30696) * fix: ensure checks show gemini-3-flash-preview supports responseJsonSchema. * fix: remove async keyword from test. * fix: make Bedrock Mantle Responses routing data-driven per model (#30700) * Make Bedrock Mantle Responses routing data-driven per model Route Bedrock Mantle models to the native Responses API based on each model's price-map capability signal instead of a hardcoded model-name heuristic, and derive the OpenAI-compatible base path segment per model. Responses dispatch now selects the native config when the model advertises responses support (/v1/responses in supported_endpoints, or mode=responses), both overridable via register_model and proxy model_info. This enables native Responses for gpt-oss-120b/20b and the gemma-4 family while keeping chat-only models (gpt-oss safeguard, nvidia, mistral, ...) on the existing chat-completions emulation. Capability is per-model, so gpt-oss-120b routes natively while gpt-oss-safeguard-120b does not despite sharing the gpt-oss substring. The wire path is a separate concern, driven by the existing use_openai_responses_path flag rather than a model-name match: gpt-5.x and gemma-4-* on /openai/v1, everything else (incl. gpt-oss) on /v1. The chat config now derives its base from the same flag, fixing gemma-4 chat-completions requests that previously went to /v1 instead of /openai/v1. Cost maps: add supported_endpoints to the gpt-oss entries (responses for the non-safeguard variants, chat-only for safeguard) and supported_endpoints + use_openai_responses_path to all three gemma-4 entries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Address review: move capability helper into bedrock_mantle package Move the Responses capability check out of utils.py into litellm/llms/bedrock_mantle/common_utils.py as mantle_supports_responses, alongside its companion wire-path helper mantle_base_segment. Both are now pure functions of (model, model_cost): the price-map mode/supported_endpoints read replaces the get_model_info call, so the rules are unit-testable without patching global state and the Bedrock Mantle package is self-contained. Use str | None instead of Optional[str] on the new signatures to satisfy the ruff UP045 strict-rule gate. Add direct unit tests for both helpers. Fix test_register_model_restore_undoes_existing_key_overwrite: gpt-oss-120b now legitimately supports Responses, so it can no longer be the "None after restore" vehicle; use the chat-only safeguard variant, which isolates the register/restore effect from the model's own capability. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup (#30366) * fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect. Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure. Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme. Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string. Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection. * fix: resolve CI failures and proxy DB URL typing issue * fix(dashscope): treat an explicit 0.0 tier cost as a real price, not missing (#30653) The tiered cost calculator resolved a tier's per-token cost with `tier.get(cost_key) or tier.get(fallback_cost_key, 0)`. Because `or` short-circuits on any falsy value, a tier that legitimately prices a component at 0.0 (e.g. a free-cache-read tier with cache_read_input_token_cost: 0.0, or a free-reasoning tier) is treated as missing and silently billed at the full fallback rate (input_cost_per_token / output_cost_per_token). The flat-pricing path in the same module already handles this correctly with an `is None` guard. Resolve tier costs through a small helper that mirrors it, so 0.0 is honored at both the in-range and overflow sites. No shipped model currently has a 0.0 tier cost, so this is a latent defect; the fix makes the tiered path consistent with the flat path and prevents over-charging the first time such a tier appears. Adds unit tests covering the in-range and overflow paths, and drops an unused import flagged by ruff in the touched test file. * feat(proxy): show session-aggregate cost and duration in request logs (#25708) (#30507) * fix(anthropic): don't leak tool 'type' into OpenAI function parameters schema (#30618) In the messages->chat/completions bridge, translate_anthropic_tools_to_openai merged every non-mapped tool key into the function parameters dict. The Anthropic tool 'type' (e.g. 'custom') thus overwrote parameters.type ('object' -> 'custom'), and providers reject it ('custom' is not a valid JSON-Schema type). Exclude 'type' from the passthrough. Fixes #30557. * fix(proxy): stop IAM-refresh engine restart from cascading reconnects (#29176) (#30183) An RDS IAM token refresh recreates the Prisma client, which SIGKILLs the running query-engine and spawns a new one. That planned kill was indistinguishable from a crash, and three reconnect paths used two uncoordinated locks, so a single refresh triggered a cascade of engine kill/respawn cycles: 1. `_safe_refresh_token` (holds `_reconnection_lock`) -> recreate -> kill old engine, spawn new one. 2. The engine-death watcher sees that kill, assumes a crash, and calls `attempt_db_reconnect(force=True)` (a different lock, `_db_reconnect_lock`) -> recreate again -> kills the fresh engine. 3. In-flight queries failing during the swap are classified as transport errors and trigger their own `attempt_db_reconnect` -> recreate again. Fix coordinates planned restarts across the wrapper and the watcher: - PrismaWrapper records the old engine PID in `_expected_engine_deaths` before killing it; all four watcher death-detectors (waitpid thread, pidfd, already-dead probe, os.kill poll) consume that PID and skip the reconnect instead of treating it as a crash. - `recreate_prisma_client` now serializes through `_reconnection_lock` and bumps a monotonic `_engine_generation`. Callers pass `expected_generation` as an optimistic-lock token, so racing/cascading recreates collapse into a single restart (losers no-op). This closes the two-lock gap. - The direct reconnect path probes the writer with SELECT 1 before recreating; a healthy connection (e.g. engine already replaced by a refresh) skips the recreate entirely. - `_safe_refresh_token` coalesces: it skips when the current token still has more than the refresh buffer of runway, so stacked triggers (proactive loop + __getattr__ fallback) don't each restart the engine. An `on_engine_replaced` hook re-arms the watcher on the new PID. RoutingPrismaWrapper forwards `expected_generation` and skips recreating the reader when the writer recreate was skipped. * feat(bedrock): support file content retrieval for batch output files (#30595) Implements transform_file_content_request and transform_file_content_response in BedrockFilesConfig so GET /v1/files/{id}/content works for Bedrock batch files. The request transform resolves the file id (direct s3:// URI or base64 unified id) to its S3 object, validates bucket and key prefix against the server-configured bucket, and SigV4-signs an S3 GetObject using the same credential and region resolution as the existing upload path. The credential and region params are validated into a typed model at the boundary, so the only untyped values left are the botocore signing primitives. Also fixes the proxy managed-files path: CredentialLiteLLMParams now carries s3_bucket_name (previously dropped when building deployment credentials) and the managed-files hook passes the deployment credential snapshot when routing afile_content, so unified-id content retrieval works with per-model bucket config instead of only the AWS_S3_BUCKET_NAME env var. Preserves managed-file access control: the proxy file-content endpoint now rejects raw cloud-storage ids (s3://, gs://), which would otherwise skip the owner/team check that only runs for unified ids and let a caller read another tenant's batch output by its object key. Managed outputs are reachable only through their unified file id. The afile_content "not found" error now reports the caller's unified id rather than the resolved internal S3 URI. Fixes #16186, #15563 * fix(oci): make Cohere {{trace}} judges work (tool param types + agentic tool-calling continuation) (#30646) * fix(oci): map Cohere tool array/object params to lowercase builtins OCI's Cohere backend returns HTTP 500 on a tool parameter typed as a bare "List", which is what OCI_JSON_TO_PYTHON_TYPES produced for JSON-schema arrays. MLflow {{trace}} judges trip this: their tools (get_root_span, get_span) take an attributes_to_fetch array. The lowercase builtins list/dict are accepted; only the bare "List" 500s ("Dict" happens to be tolerated, but both are lowercased for consistency). Verified live against us-chicago-1 (cohere.command-a-03-2025 and command-latest). Adds a unit regression on the transformed parameterDefinitions plus a gated integration test exercising an array-param tool end to end. * fix(oci): make Cohere agentic tool-calling continuation work Two bugs broke the OCI Cohere tool-calling loop that MLflow {{trace}} judges drive once a tool has been executed and its result is fed back. Request side: litellm pulled the last user message into the top-level `message` and emitted the tool result as a TOOL entry in chatHistory. OCI rejects that ("cannot specify message if the last entry in chat history contains tool results"), and an empty message alone is rejected too ("message must be at least 1 token long or tool results must be specified"). OCI carries the current turn's results in a dedicated top-level `toolResults` field. The Cohere transform now sends an empty message, keeps the user turn in chatHistory, and puts the results in `toolResults`, matching the langchain-oracle reference. Tool results are no longer represented as chatHistory entries. Response side: tool-grounded answers come back with citations carrying `documentIds` (camelCase) and no `document_ids`, which made the required `CohereCitation.document_ids` field fail validation and sink the whole response parse. Those citations are never surfaced, so the field (and CohereSearchQuery's generation_id) is now optional. Verified live against us-chicago-1 (cohere.command-a-03-2025 and command-latest), single and multi-round tool loops. Adds unit regressions on the transformed request shape and on citation parsing, plus gated integration tests for the continuation. * feat: integrate Repelloai Argus guardrail (#30673) * feat(guardrails): add RepelloAI Argus guardrail integration (#1) * feat(guardrails): add RepelloAI Argus guardrail integration Add a new guardrail hook backed by RepelloAI Argus, with dashboard-managed asset policies enforced via an asset_id and X-API-Key auth. * fix(guardrails): harden RepelloAI Argus guardrail - scan streaming responses on output (was bypassing the guardrail) - log blocked verdicts as guardrail_intervened instead of success - treat auth/config errors (401/403/404/422) as misconfiguration that always blocks, not a fail-open-able unreachable error - default unreachable_fallback to fail_closed and read it directly; block on unknown/malformed verdicts so an API change can't silently disable enforcement - type unreachable_fallback as a Literal, drop the duplicate config model, expose unreachable_fallback in the config schema, and stop leaking the raw provider response / exception strings to the client * fix(guardrails): address RepelloAI Argus review feedback - support ARGUS_API_KEY (with REPELLOAI_API_KEY fallback) - make asset_id required in the config model - normalize unreachable_fallback so only fail_open opens; block on 400 misconfig - correct the shared unreachable_fallback field description * docs(guardrails): add RepelloAI Argus docs page and dashboard listing - add docs page covering config, env vars, modes, verdicts, failure semantics - list RepelloAI Argus in the Guardrail Garden with provider/logo mappings - add a regression test for the provider logo and display-name resolution * fix(guardrails): keep RepelloAI asset_id optional in config model A required asset_id leaked onto the shared LitellmParams (which inherits RepelloAIGuardrailConfigModel), breaking validation for every other guardrail. Keep it optional like sibling models; the guardrail __init__ still raises when asset_id is missing, which is the real enforcement. * Add comment for last user turn scanning * feat(guardrails): harden repelloai scanning * feat(guardrails): expand repelloai scanning to include tool definitions Add extraction of tool definitions and tool call arguments to the RepelloAI guardrail scanning. Improves detection coverage by including function schemas and parameters in the prompt sent to the guardrail service. Also captures detailed error responses in logs and adds guardrail header to streaming responses. * refactor(guardrails): fix and harden repelloai schema text extraction - Fix duplicate text in _iter_schema_text: previously all dict values were re-queued onto the stack even after scalar/list keys were already extracted explicitly, causing names/descriptions to appear twice in the scanned prompt - Extract schema key frozensets to module-level constants so they are not reconstructed on every call - Change _iter_schema_text from @classmethod to @staticmethod (cls unused) - Narrow _call_analyze stage param from str to Literal["prompt", "response"] - Add HttpxResponse type annotation to _raise_for_config_error - Add LLMResponseTypes annotation to async_post_call_success_hook response param * fix(guardrails): resolve pyright type errors in repelloai guardrail - Narrow async_handler.post return from Response|None to Response with explicit None guard before calling raise_for_status/json - Fix list comprehension returning str|None by switching to explicit loop with isinstance guard so pyright tracks the narrowing - Cast model_dump() result to Dict since hasattr does not narrow object type in pyright * fix(guardrails/repello): include Responses API instructions field in prompt scan The /v1/responses top-level `instructions` field was not included in _extract_prompt_text, allowing a caller to bypass guardrail policy checks by putting blocked content in `instructions` while keeping `input` benign. * feat: add api_key to config model and read prompt from data dict * fix(guardrails/repello): plug input_text and tool-call response bypass gaps Responses API input content parts with type 'input_text' were silently dropped by build_inspection_messages (which only handles type='text'), allowing callers to send blocked content via that path without triggering the pre-call scan. Fix: add _extract_input_text_parts to RepelloAIGuardrail and call it when walking the Responses API input messages. Post-call scanning skipped responses whose choices contained only tool_calls or function_call (message.content=None), letting models put blocked output in function arguments undetected. Fix: _extract_chat_completion_text now calls _extract_tool_call_args_from_message on each choice message. Also replace typing.Dict/List with builtin dict/list to clear TID251 strict ruff violations introduced by this file. * fix(guardrails/repello): scan Responses API function_call output arguments Output items with type 'function_call' in a /v1/responses response were skipped by _extract_responses_api_text; only 'message' items were walked. A model could return blocked content in function_call.arguments undetected. Now extract arguments from function_call output items before scanning. * refactor(guardrails/repello): clean up typing and remove lint-any workarounds - Replace Optional[X]/Union[X,Y] with X|None/X|Y union syntax throughout - Use dict[str, object] instead of bare dict in all signatures - Remove **kwargs from __init__; declare guardrail_name, event_hook, default_on explicitly - Replace getattr(litellm_params, ...) with direct attribute access now that LitellmParams inherits RepelloAIGuardrailConfigModel - Add _event_hook_from_mode() to convert str|list[str]|Mode to typed GuardrailEventHooks - Use TypeAdapter.validate_json() instead of response.json() + manual dict construction - Add _is_object_dict/_is_object_list TypeGuard helpers to narrow object types without Any - Remove cast() workarounds and typed intermediate variables that existed only for the now-removed lint-any CI check - Drop _AddLiteLLMCallback Protocol; budget has sufficient slack for the one reportUnknownMemberType - Fix GuardrailConfigModel missing type arg: GuardrailConfigModel[BaseModel] * fix(guardrails/repello): suppress LIT007 on TypeGuard helpers and add streaming scan-skip warning - Add guard-ok suppressions to _is_object_dict and _is_object_list to satisfy the LIT007 hard-zero budget gate - Emit verbose_proxy_logger.warning when the streaming hook finds no inspectable text after assembly, matching observability of pre/post hooks * refactor: modifications for lint check * feat: add Pinstripes as an OpenAI-compatible provider (#30567) * feat: add Pinstripes as an OpenAI-compatible provider Pinstripes (https://pinstripes.io) is an OpenAI-compatible inference provider serving open-source models (GLM-4.5-Air, Qwen3, DeepSeek, etc.) with per-token pricing and no subscriptions. Changes: - `litellm/llms/openai_like/providers.json`: register pinstripes with base_url, api_key_env, and max_completion_tokens→max_tokens mapping - `litellm/types/utils.py`: add `PINSTRIPES = "pinstripes"` to LlmProviders - `litellm/constants.py`: add to openai_compatible_providers and openai_compatible_endpoints lists - `litellm/litellm_core_utils/get_llm_provider_logic.py`: auto-detect provider when api_base is "https://pinstripes.io/v1" - `provider_endpoints_support.json`: document supported endpoints - `tests/`: 7 unit tests covering provider registration, resolution, URL auto-detection, api_base override, and Router config Usage: import litellm response = litellm.completion( model="pinstripes/ps/glm-4.5-air", messages=[{"role": "user", "content": "Hello"}], api_key=os.environ["PINSTRIPES_API_KEY"], ) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pinstripes): resolve Greptile P1 review comments - Add api_base_env: PINSTRIPES_API_BASE to providers.json so env var override works - Set responses: false in provider_endpoints_support.json — not actually wired up - Remove docs/my-website/docs/providers/pinstripes.md — belongs in litellm-docs repo Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pinstripes): add api_base_env and correct responses capability - Add api_base_env: PINSTRIPES_API_BASE to providers.json - Set responses: false in provider_endpoints_support.json Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pinstripes): wire up Responses API — add supported_endpoints Adds supported_endpoints: ["/v1/chat/completions", "/v1/responses"] so JSONProviderRegistry.supports_responses_api returns true correctly, matching what provider_endpoints_support.json advertises. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(pinstripes): enable embeddings endpoint Pinstripes serves nomic-embed-text-v1.5 and bge-m3 via /v1/embeddings. Add /v1/embeddings to supported_endpoints and set embeddings: true. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pinstripes): use 4-space indentation in model_prices_and_context_window.json Matches the file's existing convention. Flagged by Greptile review. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pinstripes): set a2a: false — A2A protocol not implemented All comparable JSON-configured providers (tensormesh, parasail, empiriolabs, libertai, neosantara) have a2a: false. Pinstripes does not implement the Google A2A protocol, so this should be false to match. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: inference_provider <max@redactedlab.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(rag): attach existing OpenAI file ids (#30628) * fix(rag): attach existing OpenAI file ids * chore: use modern typing in rag ingest fix * chore: retrigger ci * fix(anthropic-messages): apply cache_control_injection_points on /v1/messages path (#30341) cache_control_injection_points was only consumed by the chat/completions prompt-management hook; on the native Anthropic /v1/messages path it was forwarded unused, so deployment-level cache injection was silently dropped (cache_creation_input_tokens stayed 0 for Anthropic-native clients). Add AnthropicCacheControlHook.apply_to_anthropic_messages_request to inject cache_control at block level for system / tools / message locations (the only forms /v1/messages accepts), wire it into the native anthropic_messages handler, and pop the param so it does not leak upstream as an unknown field. A {location: message, role: system} config is redirected to the top-level system prompt so the same YAML works on both endpoints. Injection respects Anthropic's 4-block cache_control limit shared across system, tools, and messages: client-supplied markers count toward the cap and are never overwritten, a slot is reserved per Bedrock tool_config point, and injection stops once the budget is exhausted. Locations this path cannot represent (tool_config) are forwarded downstream instead of being silently consumed, mirroring get_chat_completion_prompt's remaining_points pass-through. Built on litellm_internal_staging. Refs #30293 * fix(proxy): release budget reservation when a request is cancelled mid-flight (#30522) * fix(proxy): release budget reservation on cancel when no chunk was delivered The pre-call budget reservation increments the cross-pod spend counter by a request's worst-case cost, then reconciles it on success (cost callback) or error (failure hook). A client disconnect or timeout cancels the request and surfaces as CancelledError / GeneratorExit, which neither path catches, so the reservation leaks. Under a retry storm the leaked holds accumulate, pin the counter above real spend, and return spurious 429 "Budget has been exceeded" to keys whose spend is far below budget; the counter only recovers when its TTL lapses, so the failure is intermittent and self-healing. Release the reservation in async_streaming_data_generator (which the Anthropic and Google SSE generators delegate to) on the (CancelledError, GeneratorExit) path, alongside the existing max_parallel_requests release. release_budget_ reservation_on_cancel runs under asyncio.shield so it completes despite the in-progress cancellation, is guarded by the reservation's finalized flag, and swallows a failing release so it cannot replace the in-flight cancellation. The refund is gated on whether a chunk reached the client. The flag is set immediately before the yield, after the slow-path hook await: an async generator suspends at the yield, so a GeneratorExit on disconnect after a delivered chunk sees it True (keep the hold), while a cancellation during the slow-path await leaves it False (refund, nothing sent). A non-streaming cancellation delivers nothing and a completed non-streaming response is reconciled by the success callback, so neither needs a release here. Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(proxy): reconcile a cancelled reservation to input cost, not zero A streaming request cancelled before the first chunk previously reconciled its reservation to zero and finalized it. But by the time the generator is consuming the response the provider call was already dispatched, so the input tokens were billed even though no chunk reached the client, and the success/failure cost callbacks are skipped on cancellation. Refunding to zero let a caller send an expensive request and abort pre-token to dodge the input charge. Compute the request's input-token cost at reservation time and reconcile the cancelled reservation to it instead of zero. The worst-case output portion of the reservation is still released (so a legitimate mid-flight cancellation no longer pins the counter and 429s the key), while the input the provider already processed is charged. --------- Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(caching): encode object name in GCS cache GET path (#30378) GCS cache reads always missed when gcs_path was set. The GET methods interpolated the object name directly into the URL path, while the GCS JSON API requires it to be URL-encoded (a "/" must be sent as %2F). With gcs_path configured the object name is "<prefix>/<sha256>", so the raw slash produced a malformed object path and GCS returned 404. httpx does not raise on 4xx, so the status_code == 200 check fell through and get/async_get returned None, silently missing on every read. Without gcs_path the key has no slash, which is why this went unnoticed. Wrap the object name with urllib.parse.quote(..., safe="") in get_cache and async_get_cache. Apply the same encoding to the name= query parameter in set_cache and async_set_cache so the key written matches the key read back. Adds regression tests asserting the GET path and SET query are encoded (%2F) when gcs_path is set, for both sync and async paths; these fail on the unpatched code. Fixes #30377 * chore: add soniox stt-async-v5 model (#30672) * fix(proxy): include model group aliases in v1 model info (#30626) * Include model group aliases in v1 model info * Fix model info alias implementation * removed extra blank line * chore: rerun CI * fix(lint): remove redundant noqa directive in proxy_cli.py * fix: address greptile review - restore bedrock_mantle auth symbols, guard OCI empty message list, validate DIRECT_URL scheme * Revert "fix: address greptile review - restore bedrock_mantle auth symbols, guard OCI empty message list, validate DIRECT_URL scheme" This reverts commit 52c7a07. * Revert "fix(anthropic-messages): apply cache_control_injection_points on /v1/messages path (#30341)" This reverts commit c9e8a17. * Revert "fix(proxy): stop IAM-refresh engine restart from cascading reconnects (#29176) (#30183)" This reverts commit 85828da. * fix(proxy): stop IAM-refresh engine restart from cascading reconnects (#29176) (#30183) An RDS IAM token refresh recreates the Prisma client, which SIGKILLs the running query-engine and spawns a new one. That planned kill was indistinguishable from a crash, and three reconnect paths used two uncoordinated locks, so a single refresh triggered a cascade of engine kill/respawn cycles: 1. `_safe_refresh_token` (holds `_reconnection_lock`) -> recreate -> kill old engine, spawn new one. 2. The engine-death watcher sees that kill, assumes a crash, and calls `attempt_db_reconnect(force=True)` (a different lock, `_db_reconnect_lock`) -> recreate again -> kills the fresh engine. 3. In-flight queries failing during the swap are classified as transport errors and trigger their own `attempt_db_reconnect` -> recreate again. Fix coordinates planned restarts across the wrapper and the watcher: - PrismaWrapper records the old engine PID in `_expected_engine_deaths` before killing it; all four watcher death-detectors (waitpid thread, pidfd, already-dead probe, os.kill poll) consume that PID and skip the reconnect instead of treating it as a crash. - `recreate_prisma_client` now serializes through `_reconnection_lock` and bumps a monotonic `_engine_generation`. Callers pass `expected_generation` as an optimistic-lock token, so racing/cascading recreates collapse into a single restart (losers no-op). This closes the two-lock gap. - The direct reconnect path probes the writer with SELECT 1 before recreating; a healthy connection (e.g. engine already replaced by a refresh) skips the recreate entirely. - `_safe_refresh_token` coalesces: it skips when the current token still has more than the refresh buffer of runway, so stacked triggers (proactive loop + __getattr__ fallback) don't each restart the engine. An `on_engine_replaced` hook re-arms the watcher on the new PID. RoutingPrismaWrapper forwards `expected_generation` and skips recreating the reader when the writer recreate was skipped. * fix(lint): modernize type annotations in IAM-refresh prisma client files (UP006/UP045) * Revert "feat(proxy): show session-aggregate cost and duration in request logs (#25708) (#30507)" This reverts commit f530b22. * Revert "fix(dashscope): treat an explicit 0.0 tier cost as a real price, not missing (#30653)" This reverts commit 4f58bd0. * Revert "fix(oci): make Cohere {{trace}} judges work (tool param types + agentic tool-calling continuation) (#30646)" This reverts commit 50f34e0. * Revert "fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup (#30366)" This reverts commit 0544eed. * fix(bedrock_mantle): restore BedrockMantleAuthMixin and constants removed by routing rewrite * fix(key management): restore exact /key/list user_id & key_alias matching by default (#30593) Before substring search was added (commit 33bd570), /key/list matched user_id and key_alias exactly. That change made admin-authenticated calls substring-match by default, breaking the prior contract: a caller passing an exact user_id as an access filter (e.g. an integration scoping to one user with an admin key) then received other users' keys -- user_id="alice" also returned "alice2", "alice-test", etc. This is a cross-user key disclosure. Make substring matching opt-in via a new admin-only substring_matching=true query param; default to exact, restoring the prior behavior. The dashboard search box (keyListCall) passes the flag so partial search still works. Non-admins remain exact and scoped to their own keys. Updates the proxy-behavior key_alias test to opt in and adds an exact-by-default guard; adds list_keys unit coverage for the opt-in gate. --------- Co-authored-by: perseus <51974392+tcconnally@users.noreply.github.com> Co-authored-by: Hannah Smith <64043506+hannahmadison@users.noreply.github.com> Co-authored-by: Charlie Patterson <Pattersoncharlesl@gmail.com> Co-authored-by: Matthew Lapointe <mlapointe@alpha-sense.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: KRISH SONI <67964054+krishvsoni@users.noreply.github.com> Co-authored-by: Yash Raj Pandey <55940078+devYRPauli@users.noreply.github.com> Co-authored-by: Nitish Agarwal <1592163+nitishagar@users.noreply.github.com> Co-authored-by: hcl <chenglunhu@gmail.com> Co-authored-by: tushar8408 <32977767+tushar8408@users.noreply.github.com> Co-authored-by: AD Mohanraj <admohanraj@gmail.com> Co-authored-by: Fede Kamelhar <federico.kamelhar@oracle.com> Co-authored-by: Lavish Bansal <lavish.bansal619@gmail.com> Co-authored-by: max-amos <gruffulom@gmail.com> Co-authored-by: inference_provider <max@redactedlab.com> Co-authored-by: NK <93352237+Nithish-Yenaganti@users.noreply.github.com> Co-authored-by: 安妮的心动录 <74543653+anneheartrecord@users.noreply.github.com> Co-authored-by: Rick <26716961+Bytechoreographer@users.noreply.github.com> Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Burak Ömür <burak.omur.1998@gmail.com> Co-authored-by: Dan Lemon <daniel.lemon@amazee.io> Co-authored-by: Vanika Dangi <166420943+vanika02@users.noreply.github.com> Co-authored-by: Jay Gowdy <130084966+jgowdy-godaddy@users.noreply.github.com>

@jingyu-lin

On a mid-stream client disconnect the stream never reaches normal completion: CancelledError / GeneratorExit are BaseException, so neither the success nor the failure logging path runs, and the assembled-response success logging that writes SpendLogs never fires. The tokens already produced upstream are billed by the provider but never recorded on the proxy, so spend undercounts by roughly the abort rate; the gap is invisible in SpendLogs and only shows up against provider invoices. In the shielded streaming cleanup, when a client disconnect is recorded, assemble the partial usage from the chunks received so far via stream_chunk_builder and dispatch success logging for it. dispatch_success_handlers de-dupes via has_dispatched_final_stream_success, so it is a no-op when normal completion already logged, and it only runs on the cancellation path (the exception path sets stream_completed and already emits a failure log). Reported and root-caused in production by @jingyu-lin. Complements BerriAI#30522, which releases the budget reservation on the same cancellation path.

@jingyu-lin

…30630) * fix(proxy): bill partial usage when a streaming request is cancelled On a mid-stream client disconnect the stream never reaches normal completion: CancelledError / GeneratorExit are BaseException, so neither the success nor the failure logging path runs, and the assembled-response success logging that writes SpendLogs never fires. The tokens already produced upstream are billed by the provider but never recorded on the proxy, so spend undercounts by roughly the abort rate; the gap is invisible in SpendLogs and only shows up against provider invoices. In the shielded streaming cleanup, when a client disconnect is recorded, assemble the partial usage from the chunks received so far via stream_chunk_builder and dispatch success logging for it. dispatch_success_handlers de-dupes via has_dispatched_final_stream_success, so it is a no-op when normal completion already logged, and it only runs on the cancellation path (the exception path sets stream_completed and already emits a failure log). Reported and root-caused in production by @jingyu-lin. Complements #30522, which releases the budget reservation on the same cancellation path. * fix(proxy): close upstream stream before billing partial usage on disconnect Release the provider connection before running partial-usage success logging on a client disconnect, so slow or external success callbacks can no longer keep the upstream stream held open. aclose() only closes completion_stream and leaves response.chunks intact, so the partial billing still assembles usage from the chunks already received. --------- Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com>

@jingyu-lin

On a mid-stream client disconnect the stream never reaches normal completion: CancelledError / GeneratorExit are BaseException, so neither the success nor the failure logging path runs, and the assembled-response success logging that writes SpendLogs never fires. The tokens already produced upstream are billed by the provider but never recorded on the proxy, so spend undercounts by roughly the abort rate; the gap is invisible in SpendLogs and only shows up against provider invoices. In the shielded streaming cleanup, when a client disconnect is recorded, assemble the partial usage from the chunks received so far via stream_chunk_builder and dispatch success logging for it. dispatch_success_handlers de-dupes via has_dispatched_final_stream_success, so it is a no-op when normal completion already logged, and it only runs on the cancellation path (the exception path sets stream_completed and already emits a failure log). Reported and root-caused in production by @jingyu-lin. Complements BerriAI#30522, which releases the budget reservation on the same cancellation path.

@jingyu-lin

…30630) * fix(proxy): bill partial usage when a streaming request is cancelled On a mid-stream client disconnect the stream never reaches normal completion: CancelledError / GeneratorExit are BaseException, so neither the success nor the failure logging path runs, and the assembled-response success logging that writes SpendLogs never fires. The tokens already produced upstream are billed by the provider but never recorded on the proxy, so spend undercounts by roughly the abort rate; the gap is invisible in SpendLogs and only shows up against provider invoices. In the shielded streaming cleanup, when a client disconnect is recorded, assemble the partial usage from the chunks received so far via stream_chunk_builder and dispatch success logging for it. dispatch_success_handlers de-dupes via has_dispatched_final_stream_success, so it is a no-op when normal completion already logged, and it only runs on the cancellation path (the exception path sets stream_completed and already emits a failure log). Reported and root-caused in production by @jingyu-lin. Complements #30522, which releases the budget reservation on the same cancellation path. * fix(proxy): close upstream stream before billing partial usage on disconnect Release the provider connection before running partial-usage success logging on a client disconnect, so slow or external success callbacks can no longer keep the upstream stream held open. aclose() only closes completion_stream and leaves response.chunks intact, so the partial billing still assembles usage from the chunks already received. --------- Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com>

@staticmethod

* fix(proxy): bump health-check max_tokens default to 16 for GPT-5 compatibility (BerriAI#30708) OpenAI GPT-5 models require max_completion_tokens >= 16. Health checks were using 5 (proxy/health_check.py) and 10 (health_check_helpers.py), causing failures on GPT-5 models. Fixes BerriAI#23836 * fix: increase health check max_tokens from 5 to 16 (BerriAI#23836) (BerriAI#26610) GPT-5 models enforce a minimum of 16 for max_output_tokens. The current default of 5 still causes health checks to fail for these models. Bump the non-wildcard default to 16 — the smallest value that satisfies all known provider minimums while keeping health checks lightweight. Also tightens the wildcard test assertion from a weak disjunctive check to strict key-absence. Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix: ensure checks show gemini-3-flash-preview supports responseJsonS… (BerriAI#30696) * fix: ensure checks show gemini-3-flash-preview supports responseJsonSchema. * fix: remove async keyword from test. * fix: make Bedrock Mantle Responses routing data-driven per model (BerriAI#30700) * Make Bedrock Mantle Responses routing data-driven per model Route Bedrock Mantle models to the native Responses API based on each model's price-map capability signal instead of a hardcoded model-name heuristic, and derive the OpenAI-compatible base path segment per model. Responses dispatch now selects the native config when the model advertises responses support (/v1/responses in supported_endpoints, or mode=responses), both overridable via register_model and proxy model_info. This enables native Responses for gpt-oss-120b/20b and the gemma-4 family while keeping chat-only models (gpt-oss safeguard, nvidia, mistral, ...) on the existing chat-completions emulation. Capability is per-model, so gpt-oss-120b routes natively while gpt-oss-safeguard-120b does not despite sharing the gpt-oss substring. The wire path is a separate concern, driven by the existing use_openai_responses_path flag rather than a model-name match: gpt-5.x and gemma-4-* on /openai/v1, everything else (incl. gpt-oss) on /v1. The chat config now derives its base from the same flag, fixing gemma-4 chat-completions requests that previously went to /v1 instead of /openai/v1. Cost maps: add supported_endpoints to the gpt-oss entries (responses for the non-safeguard variants, chat-only for safeguard) and supported_endpoints + use_openai_responses_path to all three gemma-4 entries. * Address review: move capability helper into bedrock_mantle package Move the Responses capability check out of utils.py into litellm/llms/bedrock_mantle/common_utils.py as mantle_supports_responses, alongside its companion wire-path helper mantle_base_segment. Both are now pure functions of (model, model_cost): the price-map mode/supported_endpoints read replaces the get_model_info call, so the rules are unit-testable without patching global state and the Bedrock Mantle package is self-contained. Use str | None instead of Optional[str] on the new signatures to satisfy the ruff UP045 strict-rule gate. Add direct unit tests for both helpers. Fix test_register_model_restore_undoes_existing_key_overwrite: gpt-oss-120b now legitimately supports Responses, so it can no longer be the "None after restore" vehicle; use the chat-only safeguard variant, which isolates the register/restore effect from the model's own capability. --------- Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup (BerriAI#30366) * fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect. Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure. Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme. Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string. Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection. * fix: resolve CI failures and proxy DB URL typing issue * fix(dashscope): treat an explicit 0.0 tier cost as a real price, not missing (BerriAI#30653) The tiered cost calculator resolved a tier's per-token cost with `tier.get(cost_key) or tier.get(fallback_cost_key, 0)`. Because `or` short-circuits on any falsy value, a tier that legitimately prices a component at 0.0 (e.g. a free-cache-read tier with cache_read_input_token_cost: 0.0, or a free-reasoning tier) is treated as missing and silently billed at the full fallback rate (input_cost_per_token / output_cost_per_token). The flat-pricing path in the same module already handles this correctly with an `is None` guard. Resolve tier costs through a small helper that mirrors it, so 0.0 is honored at both the in-range and overflow sites. No shipped model currently has a 0.0 tier cost, so this is a latent defect; the fix makes the tiered path consistent with the flat path and prevents over-charging the first time such a tier appears. Adds unit tests covering the in-range and overflow paths, and drops an unused import flagged by ruff in the touched test file. * feat(proxy): show session-aggregate cost and duration in request logs (BerriAI#25708) (BerriAI#30507) * fix(anthropic): don't leak tool 'type' into OpenAI function parameters schema (BerriAI#30618) In the messages->chat/completions bridge, translate_anthropic_tools_to_openai merged every non-mapped tool key into the function parameters dict. The Anthropic tool 'type' (e.g. 'custom') thus overwrote parameters.type ('object' -> 'custom'), and providers reject it ('custom' is not a valid JSON-Schema type). Exclude 'type' from the passthrough. Fixes BerriAI#30557. * fix(proxy): stop IAM-refresh engine restart from cascading reconnects (BerriAI#29176) (BerriAI#30183) An RDS IAM token refresh recreates the Prisma client, which SIGKILLs the running query-engine and spawns a new one. That planned kill was indistinguishable from a crash, and three reconnect paths used two uncoordinated locks, so a single refresh triggered a cascade of engine kill/respawn cycles: 1. `_safe_refresh_token` (holds `_reconnection_lock`) -> recreate -> kill old engine, spawn new one. 2. The engine-death watcher sees that kill, assumes a crash, and calls `attempt_db_reconnect(force=True)` (a different lock, `_db_reconnect_lock`) -> recreate again -> kills the fresh engine. 3. In-flight queries failing during the swap are classified as transport errors and trigger their own `attempt_db_reconnect` -> recreate again. Fix coordinates planned restarts across the wrapper and the watcher: - PrismaWrapper records the old engine PID in `_expected_engine_deaths` before killing it; all four watcher death-detectors (waitpid thread, pidfd, already-dead probe, os.kill poll) consume that PID and skip the reconnect instead of treating it as a crash. - `recreate_prisma_client` now serializes through `_reconnection_lock` and bumps a monotonic `_engine_generation`. Callers pass `expected_generation` as an optimistic-lock token, so racing/cascading recreates collapse into a single restart (losers no-op). This closes the two-lock gap. - The direct reconnect path probes the writer with SELECT 1 before recreating; a healthy connection (e.g. engine already replaced by a refresh) skips the recreate entirely. - `_safe_refresh_token` coalesces: it skips when the current token still has more than the refresh buffer of runway, so stacked triggers (proactive loop + __getattr__ fallback) don't each restart the engine. An `on_engine_replaced` hook re-arms the watcher on the new PID. RoutingPrismaWrapper forwards `expected_generation` and skips recreating the reader when the writer recreate was skipped. * feat(bedrock): support file content retrieval for batch output files (BerriAI#30595) Implements transform_file_content_request and transform_file_content_response in BedrockFilesConfig so GET /v1/files/{id}/content works for Bedrock batch files. The request transform resolves the file id (direct s3:// URI or base64 unified id) to its S3 object, validates bucket and key prefix against the server-configured bucket, and SigV4-signs an S3 GetObject using the same credential and region resolution as the existing upload path. The credential and region params are validated into a typed model at the boundary, so the only untyped values left are the botocore signing primitives. Also fixes the proxy managed-files path: CredentialLiteLLMParams now carries s3_bucket_name (previously dropped when building deployment credentials) and the managed-files hook passes the deployment credential snapshot when routing afile_content, so unified-id content retrieval works with per-model bucket config instead of only the AWS_S3_BUCKET_NAME env var. Preserves managed-file access control: the proxy file-content endpoint now rejects raw cloud-storage ids (s3://, gs://), which would otherwise skip the owner/team check that only runs for unified ids and let a caller read another tenant's batch output by its object key. Managed outputs are reachable only through their unified file id. The afile_content "not found" error now reports the caller's unified id rather than the resolved internal S3 URI. Fixes BerriAI#16186, BerriAI#15563 * fix(oci): make Cohere {{trace}} judges work (tool param types + agentic tool-calling continuation) (BerriAI#30646) * fix(oci): map Cohere tool array/object params to lowercase builtins OCI's Cohere backend returns HTTP 500 on a tool parameter typed as a bare "List", which is what OCI_JSON_TO_PYTHON_TYPES produced for JSON-schema arrays. MLflow {{trace}} judges trip this: their tools (get_root_span, get_span) take an attributes_to_fetch array. The lowercase builtins list/dict are accepted; only the bare "List" 500s ("Dict" happens to be tolerated, but both are lowercased for consistency). Verified live against us-chicago-1 (cohere.command-a-03-2025 and command-latest). Adds a unit regression on the transformed parameterDefinitions plus a gated integration test exercising an array-param tool end to end. * fix(oci): make Cohere agentic tool-calling continuation work Two bugs broke the OCI Cohere tool-calling loop that MLflow {{trace}} judges drive once a tool has been executed and its result is fed back. Request side: litellm pulled the last user message into the top-level `message` and emitted the tool result as a TOOL entry in chatHistory. OCI rejects that ("cannot specify message if the last entry in chat history contains tool results"), and an empty message alone is rejected too ("message must be at least 1 token long or tool results must be specified"). OCI carries the current turn's results in a dedicated top-level `toolResults` field. The Cohere transform now sends an empty message, keeps the user turn in chatHistory, and puts the results in `toolResults`, matching the langchain-oracle reference. Tool results are no longer represented as chatHistory entries. Response side: tool-grounded answers come back with citations carrying `documentIds` (camelCase) and no `document_ids`, which made the required `CohereCitation.document_ids` field fail validation and sink the whole response parse. Those citations are never surfaced, so the field (and CohereSearchQuery's generation_id) is now optional. Verified live against us-chicago-1 (cohere.command-a-03-2025 and command-latest), single and multi-round tool loops. Adds unit regressions on the transformed request shape and on citation parsing, plus gated integration tests for the continuation. * feat: integrate Repelloai Argus guardrail (BerriAI#30673) * feat(guardrails): add RepelloAI Argus guardrail integration (#1) * feat(guardrails): add RepelloAI Argus guardrail integration Add a new guardrail hook backed by RepelloAI Argus, with dashboard-managed asset policies enforced via an asset_id and X-API-Key auth. * fix(guardrails): harden RepelloAI Argus guardrail - scan streaming responses on output (was bypassing the guardrail) - log blocked verdicts as guardrail_intervened instead of success - treat auth/config errors (401/403/404/422) as misconfiguration that always blocks, not a fail-open-able unreachable error - default unreachable_fallback to fail_closed and read it directly; block on unknown/malformed verdicts so an API change can't silently disable enforcement - type unreachable_fallback as a Literal, drop the duplicate config model, expose unreachable_fallback in the config schema, and stop leaking the raw provider response / exception strings to the client * fix(guardrails): address RepelloAI Argus review feedback - support ARGUS_API_KEY (with REPELLOAI_API_KEY fallback) - make asset_id required in the config model - normalize unreachable_fallback so only fail_open opens; block on 400 misconfig - correct the shared unreachable_fallback field description * docs(guardrails): add RepelloAI Argus docs page and dashboard listing - add docs page covering config, env vars, modes, verdicts, failure semantics - list RepelloAI Argus in the Guardrail Garden with provider/logo mappings - add a regression test for the provider logo and display-name resolution * fix(guardrails): keep RepelloAI asset_id optional in config model A required asset_id leaked onto the shared LitellmParams (which inherits RepelloAIGuardrailConfigModel), breaking validation for every other guardrail. Keep it optional like sibling models; the guardrail __init__ still raises when asset_id is missing, which is the real enforcement. * Add comment for last user turn scanning * feat(guardrails): harden repelloai scanning * feat(guardrails): expand repelloai scanning to include tool definitions Add extraction of tool definitions and tool call arguments to the RepelloAI guardrail scanning. Improves detection coverage by including function schemas and parameters in the prompt sent to the guardrail service. Also captures detailed error responses in logs and adds guardrail header to streaming responses. * refactor(guardrails): fix and harden repelloai schema text extraction - Fix duplicate text in _iter_schema_text: previously all dict values were re-queued onto the stack even after scalar/list keys were already extracted explicitly, causing names/descriptions to appear twice in the scanned prompt - Extract schema key frozensets to module-level constants so they are not reconstructed on every call - Change _iter_schema_text from @classmethod to @staticmethod (cls unused) - Narrow _call_analyze stage param from str to Literal["prompt", "response"] - Add HttpxResponse type annotation to _raise_for_config_error - Add LLMResponseTypes annotation to async_post_call_success_hook response param * fix(guardrails): resolve pyright type errors in repelloai guardrail - Narrow async_handler.post return from Response|None to Response with explicit None guard before calling raise_for_status/json - Fix list comprehension returning str|None by switching to explicit loop with isinstance guard so pyright tracks the narrowing - Cast model_dump() result to Dict since hasattr does not narrow object type in pyright * fix(guardrails/repello): include Responses API instructions field in prompt scan The /v1/responses top-level `instructions` field was not included in _extract_prompt_text, allowing a caller to bypass guardrail policy checks by putting blocked content in `instructions` while keeping `input` benign. * feat: add api_key to config model and read prompt from data dict * fix(guardrails/repello): plug input_text and tool-call response bypass gaps Responses API input content parts with type 'input_text' were silently dropped by build_inspection_messages (which only handles type='text'), allowing callers to send blocked content via that path without triggering the pre-call scan. Fix: add _extract_input_text_parts to RepelloAIGuardrail and call it when walking the Responses API input messages. Post-call scanning skipped responses whose choices contained only tool_calls or function_call (message.content=None), letting models put blocked output in function arguments undetected. Fix: _extract_chat_completion_text now calls _extract_tool_call_args_from_message on each choice message. Also replace typing.Dict/List with builtin dict/list to clear TID251 strict ruff violations introduced by this file. * fix(guardrails/repello): scan Responses API function_call output arguments Output items with type 'function_call' in a /v1/responses response were skipped by _extract_responses_api_text; only 'message' items were walked. A model could return blocked content in function_call.arguments undetected. Now extract arguments from function_call output items before scanning. * refactor(guardrails/repello): clean up typing and remove lint-any workarounds - Replace Optional[X]/Union[X,Y] with X|None/X|Y union syntax throughout - Use dict[str, object] instead of bare dict in all signatures - Remove **kwargs from __init__; declare guardrail_name, event_hook, default_on explicitly - Replace getattr(litellm_params, ...) with direct attribute access now that LitellmParams inherits RepelloAIGuardrailConfigModel - Add _event_hook_from_mode() to convert str|list[str]|Mode to typed GuardrailEventHooks - Use TypeAdapter.validate_json() instead of response.json() + manual dict construction - Add _is_object_dict/_is_object_list TypeGuard helpers to narrow object types without Any - Remove cast() workarounds and typed intermediate variables that existed only for the now-removed lint-any CI check - Drop _AddLiteLLMCallback Protocol; budget has sufficient slack for the one reportUnknownMemberType - Fix GuardrailConfigModel missing type arg: GuardrailConfigModel[BaseModel] * fix(guardrails/repello): suppress LIT007 on TypeGuard helpers and add streaming scan-skip warning - Add guard-ok suppressions to _is_object_dict and _is_object_list to satisfy the LIT007 hard-zero budget gate - Emit verbose_proxy_logger.warning when the streaming hook finds no inspectable text after assembly, matching observability of pre/post hooks * refactor: modifications for lint check * feat: add Pinstripes as an OpenAI-compatible provider (BerriAI#30567) * feat: add Pinstripes as an OpenAI-compatible provider Pinstripes (https://pinstripes.io) is an OpenAI-compatible inference provider serving open-source models (GLM-4.5-Air, Qwen3, DeepSeek, etc.) with per-token pricing and no subscriptions. Changes: - `litellm/llms/openai_like/providers.json`: register pinstripes with base_url, api_key_env, and max_completion_tokens→max_tokens mapping - `litellm/types/utils.py`: add `PINSTRIPES = "pinstripes"` to LlmProviders - `litellm/constants.py`: add to openai_compatible_providers and openai_compatible_endpoints lists - `litellm/litellm_core_utils/get_llm_provider_logic.py`: auto-detect provider when api_base is "https://pinstripes.io/v1" - `provider_endpoints_support.json`: document supported endpoints - `tests/`: 7 unit tests covering provider registration, resolution, URL auto-detection, api_base override, and Router config Usage: import litellm response = litellm.completion( model="pinstripes/ps/glm-4.5-air", messages=[{"role": "user", "content": "Hello"}], api_key=os.environ["PINSTRIPES_API_KEY"], ) * fix(pinstripes): resolve Greptile P1 review comments - Add api_base_env: PINSTRIPES_API_BASE to providers.json so env var override works - Set responses: false in provider_endpoints_support.json — not actually wired up - Remove docs/my-website/docs/providers/pinstripes.md — belongs in litellm-docs repo * fix(pinstripes): add api_base_env and correct responses capability - Add api_base_env: PINSTRIPES_API_BASE to providers.json - Set responses: false in provider_endpoints_support.json * fix(pinstripes): wire up Responses API — add supported_endpoints Adds supported_endpoints: ["/v1/chat/completions", "/v1/responses"] so JSONProviderRegistry.supports_responses_api returns true correctly, matching what provider_endpoints_support.json advertises. * feat(pinstripes): enable embeddings endpoint Pinstripes serves nomic-embed-text-v1.5 and bge-m3 via /v1/embeddings. Add /v1/embeddings to supported_endpoints and set embeddings: true. * fix(pinstripes): use 4-space indentation in model_prices_and_context_window.json Matches the file's existing convention. Flagged by Greptile review. * fix(pinstripes): set a2a: false — A2A protocol not implemented All comparable JSON-configured providers (tensormesh, parasail, empiriolabs, libertai, neosantara) have a2a: false. Pinstripes does not implement the Google A2A protocol, so this should be false to match. --------- Co-authored-by: inference_provider <max@redactedlab.com> * fix(rag): attach existing OpenAI file ids (BerriAI#30628) * fix(rag): attach existing OpenAI file ids * chore: use modern typing in rag ingest fix * chore: retrigger ci * fix(anthropic-messages): apply cache_control_injection_points on /v1/messages path (BerriAI#30341) cache_control_injection_points was only consumed by the chat/completions prompt-management hook; on the native Anthropic /v1/messages path it was forwarded unused, so deployment-level cache injection was silently dropped (cache_creation_input_tokens stayed 0 for Anthropic-native clients). Add AnthropicCacheControlHook.apply_to_anthropic_messages_request to inject cache_control at block level for system / tools / message locations (the only forms /v1/messages accepts), wire it into the native anthropic_messages handler, and pop the param so it does not leak upstream as an unknown field. A {location: message, role: system} config is redirected to the top-level system prompt so the same YAML works on both endpoints. Injection respects Anthropic's 4-block cache_control limit shared across system, tools, and messages: client-supplied markers count toward the cap and are never overwritten, a slot is reserved per Bedrock tool_config point, and injection stops once the budget is exhausted. Locations this path cannot represent (tool_config) are forwarded downstream instead of being silently consumed, mirroring get_chat_completion_prompt's remaining_points pass-through. Built on litellm_internal_staging. Refs BerriAI#30293 * fix(proxy): release budget reservation when a request is cancelled mid-flight (BerriAI#30522) * fix(proxy): release budget reservation on cancel when no chunk was delivered The pre-call budget reservation increments the cross-pod spend counter by a request's worst-case cost, then reconciles it on success (cost callback) or error (failure hook). A client disconnect or timeout cancels the request and surfaces as CancelledError / GeneratorExit, which neither path catches, so the reservation leaks. Under a retry storm the leaked holds accumulate, pin the counter above real spend, and return spurious 429 "Budget has been exceeded" to keys whose spend is far below budget; the counter only recovers when its TTL lapses, so the failure is intermittent and self-healing. Release the reservation in async_streaming_data_generator (which the Anthropic and Google SSE generators delegate to) on the (CancelledError, GeneratorExit) path, alongside the existing max_parallel_requests release. release_budget_ reservation_on_cancel runs under asyncio.shield so it completes despite the in-progress cancellation, is guarded by the reservation's finalized flag, and swallows a failing release so it cannot replace the in-flight cancellation. The refund is gated on whether a chunk reached the client. The flag is set immediately before the yield, after the slow-path hook await: an async generator suspends at the yield, so a GeneratorExit on disconnect after a delivered chunk sees it True (keep the hold), while a cancellation during the slow-path await leaves it False (refund, nothing sent). A non-streaming cancellation delivers nothing and a completed non-streaming response is reconciled by the success callback, so neither needs a release here. Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(proxy): reconcile a cancelled reservation to input cost, not zero A streaming request cancelled before the first chunk previously reconciled its reservation to zero and finalized it. But by the time the generator is consuming the response the provider call was already dispatched, so the input tokens were billed even though no chunk reached the client, and the success/failure cost callbacks are skipped on cancellation. Refunding to zero let a caller send an expensive request and abort pre-token to dodge the input charge. Compute the request's input-token cost at reservation time and reconcile the cancelled reservation to it instead of zero. The worst-case output portion of the reservation is still released (so a legitimate mid-flight cancellation no longer pins the counter and 429s the key), while the input the provider already processed is charged. --------- Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(caching): encode object name in GCS cache GET path (BerriAI#30378) GCS cache reads always missed when gcs_path was set. The GET methods interpolated the object name directly into the URL path, while the GCS JSON API requires it to be URL-encoded (a "/" must be sent as %2F). With gcs_path configured the object name is "<prefix>/<sha256>", so the raw slash produced a malformed object path and GCS returned 404. httpx does not raise on 4xx, so the status_code == 200 check fell through and get/async_get returned None, silently missing on every read. Without gcs_path the key has no slash, which is why this went unnoticed. Wrap the object name with urllib.parse.quote(..., safe="") in get_cache and async_get_cache. Apply the same encoding to the name= query parameter in set_cache and async_set_cache so the key written matches the key read back. Adds regression tests asserting the GET path and SET query are encoded (%2F) when gcs_path is set, for both sync and async paths; these fail on the unpatched code. Fixes BerriAI#30377 * chore: add soniox stt-async-v5 model (BerriAI#30672) * fix(proxy): include model group aliases in v1 model info (BerriAI#30626) * Include model group aliases in v1 model info * Fix model info alias implementation * removed extra blank line * chore: rerun CI * fix(lint): remove redundant noqa directive in proxy_cli.py * fix: address greptile review - restore bedrock_mantle auth symbols, guard OCI empty message list, validate DIRECT_URL scheme * Revert "fix: address greptile review - restore bedrock_mantle auth symbols, guard OCI empty message list, validate DIRECT_URL scheme" This reverts commit 52c7a07. * Revert "fix(anthropic-messages): apply cache_control_injection_points on /v1/messages path (BerriAI#30341)" This reverts commit c9e8a17. * Revert "fix(proxy): stop IAM-refresh engine restart from cascading reconnects (BerriAI#29176) (BerriAI#30183)" This reverts commit 85828da. * fix(proxy): stop IAM-refresh engine restart from cascading reconnects (BerriAI#29176) (BerriAI#30183) An RDS IAM token refresh recreates the Prisma client, which SIGKILLs the running query-engine and spawns a new one. That planned kill was indistinguishable from a crash, and three reconnect paths used two uncoordinated locks, so a single refresh triggered a cascade of engine kill/respawn cycles: 1. `_safe_refresh_token` (holds `_reconnection_lock`) -> recreate -> kill old engine, spawn new one. 2. The engine-death watcher sees that kill, assumes a crash, and calls `attempt_db_reconnect(force=True)` (a different lock, `_db_reconnect_lock`) -> recreate again -> kills the fresh engine. 3. In-flight queries failing during the swap are classified as transport errors and trigger their own `attempt_db_reconnect` -> recreate again. Fix coordinates planned restarts across the wrapper and the watcher: - PrismaWrapper records the old engine PID in `_expected_engine_deaths` before killing it; all four watcher death-detectors (waitpid thread, pidfd, already-dead probe, os.kill poll) consume that PID and skip the reconnect instead of treating it as a crash. - `recreate_prisma_client` now serializes through `_reconnection_lock` and bumps a monotonic `_engine_generation`. Callers pass `expected_generation` as an optimistic-lock token, so racing/cascading recreates collapse into a single restart (losers no-op). This closes the two-lock gap. - The direct reconnect path probes the writer with SELECT 1 before recreating; a healthy connection (e.g. engine already replaced by a refresh) skips the recreate entirely. - `_safe_refresh_token` coalesces: it skips when the current token still has more than the refresh buffer of runway, so stacked triggers (proactive loop + __getattr__ fallback) don't each restart the engine. An `on_engine_replaced` hook re-arms the watcher on the new PID. RoutingPrismaWrapper forwards `expected_generation` and skips recreating the reader when the writer recreate was skipped. * fix(lint): modernize type annotations in IAM-refresh prisma client files (UP006/UP045) * Revert "feat(proxy): show session-aggregate cost and duration in request logs (BerriAI#25708) (BerriAI#30507)" This reverts commit f530b22. * Revert "fix(dashscope): treat an explicit 0.0 tier cost as a real price, not missing (BerriAI#30653)" This reverts commit 4f58bd0. * Revert "fix(oci): make Cohere {{trace}} judges work (tool param types + agentic tool-calling continuation) (BerriAI#30646)" This reverts commit 50f34e0. * Revert "fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup (BerriAI#30366)" This reverts commit 0544eed. * fix(bedrock_mantle): restore BedrockMantleAuthMixin and constants removed by routing rewrite * fix(key management): restore exact /key/list user_id & key_alias matching by default (BerriAI#30593) Before substring search was added (commit 33bd570), /key/list matched user_id and key_alias exactly. That change made admin-authenticated calls substring-match by default, breaking the prior contract: a caller passing an exact user_id as an access filter (e.g. an integration scoping to one user with an admin key) then received other users' keys -- user_id="alice" also returned "alice2", "alice-test", etc. This is a cross-user key disclosure. Make substring matching opt-in via a new admin-only substring_matching=true query param; default to exact, restoring the prior behavior. The dashboard search box (keyListCall) passes the flag so partial search still works. Non-admins remain exact and scoped to their own keys. Updates the proxy-behavior key_alias test to opt in and adds an exact-by-default guard; adds list_keys unit coverage for the opt-in gate. --------- Co-authored-by: perseus <51974392+tcconnally@users.noreply.github.com> Co-authored-by: Hannah Smith <64043506+hannahmadison@users.noreply.github.com> Co-authored-by: Charlie Patterson <Pattersoncharlesl@gmail.com> Co-authored-by: Matthew Lapointe <mlapointe@alpha-sense.com> Co-authored-by: KRISH SONI <67964054+krishvsoni@users.noreply.github.com> Co-authored-by: Yash Raj Pandey <55940078+devYRPauli@users.noreply.github.com> Co-authored-by: Nitish Agarwal <1592163+nitishagar@users.noreply.github.com> Co-authored-by: hcl <chenglunhu@gmail.com> Co-authored-by: tushar8408 <32977767+tushar8408@users.noreply.github.com> Co-authored-by: AD Mohanraj <admohanraj@gmail.com> Co-authored-by: Fede Kamelhar <federico.kamelhar@oracle.com> Co-authored-by: Lavish Bansal <lavish.bansal619@gmail.com> Co-authored-by: max-amos <gruffulom@gmail.com> Co-authored-by: inference_provider <max@redactedlab.com> Co-authored-by: NK <93352237+Nithish-Yenaganti@users.noreply.github.com> Co-authored-by: 安妮的心动录 <74543653+anneheartrecord@users.noreply.github.com> Co-authored-by: Rick <26716961+Bytechoreographer@users.noreply.github.com> Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Burak Ömür <burak.omur.1998@gmail.com> Co-authored-by: Dan Lemon <daniel.lemon@amazee.io> Co-authored-by: Vanika Dangi <166420943+vanika02@users.noreply.github.com> Co-authored-by: Jay Gowdy <130084966+jgowdy-godaddy@users.noreply.github.com>

@jingyu-lin

…30630) * fix(proxy): bill partial usage when a streaming request is cancelled On a mid-stream client disconnect the stream never reaches normal completion: CancelledError / GeneratorExit are BaseException, so neither the success nor the failure logging path runs, and the assembled-response success logging that writes SpendLogs never fires. The tokens already produced upstream are billed by the provider but never recorded on the proxy, so spend undercounts by roughly the abort rate; the gap is invisible in SpendLogs and only shows up against provider invoices. In the shielded streaming cleanup, when a client disconnect is recorded, assemble the partial usage from the chunks received so far via stream_chunk_builder and dispatch success logging for it. dispatch_success_handlers de-dupes via has_dispatched_final_stream_success, so it is a no-op when normal completion already logged, and it only runs on the cancellation path (the exception path sets stream_completed and already emits a failure log). Reported and root-caused in production by @jingyu-lin. Complements #30522, which releases the budget reservation on the same cancellation path. * fix(proxy): close upstream stream before billing partial usage on disconnect Release the provider connection before running partial-usage success logging on a client disconnect, so slow or external success callbacks can no longer keep the upstream stream held open. aclose() only closes completion_stream and leaves response.chunks intact, so the partial billing still assembles usage from the chunks already received. --------- Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com>

Bytechoreographer mentioned this pull request Jun 16, 2026

fix(proxy): release budget reservation when a request is cancelled mid-flight #30519

Closed

4 tasks

greptile-apps Bot reviewed Jun 16, 2026

View reviewed changes

Comment thread litellm/proxy/spend_tracking/budget_reservation.py

veria-ai Bot reviewed Jun 16, 2026

View reviewed changes

Comment thread litellm/proxy/common_request_processing.py Outdated

veria-ai Bot reviewed Jun 16, 2026

View reviewed changes

Comment thread litellm/proxy/common_request_processing.py Outdated

veria-ai Bot reviewed Jun 16, 2026

View reviewed changes

Comment thread litellm/proxy/common_request_processing.py

Bytechoreographer force-pushed the litellm_budget_reservation_cancel_release branch from 12f71b1 to 228fafe Compare June 17, 2026 04:19

veria-ai Bot reviewed Jun 17, 2026

View reviewed changes

Comment thread litellm/proxy/common_request_processing.py

Bytechoreographer force-pushed the litellm_budget_reservation_cancel_release branch from 01983c7 to f2393bf Compare June 17, 2026 05:17

veria-ai Bot reviewed Jun 17, 2026

View reviewed changes

Bytechoreographer mentioned this pull request Jun 17, 2026

fix(proxy): bill partial usage when a streaming request is cancelled #30630

Merged

4 tasks

Sameerlite changed the base branch from litellm_internal_staging to litellm_oss_staging June 18, 2026 12:56

Sameerlite approved these changes Jun 18, 2026

View reviewed changes

Sameerlite merged commit 9f0cfc1 into BerriAI:litellm_oss_staging Jun 18, 2026
77 checks passed

Bytechoreographer mentioned this pull request Jun 25, 2026

fix(proxy): load DB models in a deterministic order #31186

Merged

4 tasks

Uh oh!

Uh oh!

Conversation

Bytechoreographer commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Relevant issues

Pre-Submission checklist

Type

Changes

Screenshots / Proof of Fix

Uh oh!

codecov Bot commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

greptile-apps Bot commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

Uh oh!

Uh oh!

veria-ai Bot commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR overview

Open issues (1)

Uh oh!

Bytechoreographer commented Jun 16, 2026

Uh oh!

Uh oh!

Bytechoreographer commented Jun 16, 2026

Uh oh!

Bytechoreographer commented Jun 16, 2026

Uh oh!

Bytechoreographer commented Jun 16, 2026

Uh oh!

greptile-apps Bot commented Jun 16, 2026

Uh oh!

Uh oh!

Bytechoreographer commented Jun 16, 2026

Uh oh!

greptile-apps Bot commented Jun 16, 2026

Uh oh!

Bytechoreographer commented Jun 16, 2026

Uh oh!

greptile-apps Bot commented Jun 16, 2026

Uh oh!

Sameerlite commented Jun 17, 2026

Uh oh!

Bytechoreographer commented Jun 17, 2026

Uh oh!

Uh oh!

Bytechoreographer commented Jun 17, 2026

Uh oh!

veria-ai Bot Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

Bytechoreographer commented Jun 17, 2026

Uh oh!

Bytechoreographer commented Jun 17, 2026

Uh oh!

jingyu-lin commented Jun 17, 2026

Uh oh!

Bytechoreographer commented Jun 17, 2026

Uh oh!

Bytechoreographer commented Jun 17, 2026

Uh oh!

jingyu-lin commented Jun 17, 2026

Uh oh!

Sameerlite commented Jun 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

Bytechoreographer commented Jun 16, 2026 •

edited

Loading

codecov Bot commented Jun 16, 2026 •

edited

Loading

greptile-apps Bot commented Jun 16, 2026 •

edited

Loading

veria-ai Bot commented Jun 16, 2026 •

edited

Loading