chore: promote staging to main - #35328
Merged
Merged
Conversation
Fireworks publishes a 262144-token context window for the Kimi K2.5, K2.6 and K2.7 models but caps generation well below that. Every fireworks_ai Kimi K2.5/K2.6/K2.7 alias had max_output_tokens/max_tokens flattened to 262144 (equal to the context window), so the pre-call context-window check admitted requests asking for a full 262144-token completion that Fireworks rejects. Correct max_output_tokens/max_tokens to 32768 while keeping max_input_tokens at 262144, and add a regression test pinning the limits for all ten aliases. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…p fix Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
… without a DB and make their ids stable
…ified guardrail translation
…nd expose them via list APIs
…to OpenAI's latest article (#35258) Adjust the price of gpt-5.6-terra to 80% of its original rate (2/12), and gpt-5.6-luna to 20% of its original rate (0.2/1.2). References: https://openai.com/index/advancing-the-price-performance-frontier-with-gpt-5-6/ https://developers.openai.com/api/docs/pricing
…OpenAI's cut AWS rolled out the 2026-07-30 GPT-5.6 price cut the same day, but the bedrock_mantle entries still carried values derived from the pre-cut OpenAI base, so Terra billed 1.25x and Luna 5x over the published rate. Re-derive both from the AWS Bedrock pricing page, which prices in-region inference at parity with OpenAI's data residency tier (1.1x base). Sol was not cut and is unchanged. Also drop tests/test_litellm/test_gpt_5_6_model_metadata.py; its Azure and openai pricing assertions are covered by test_llm_cost_calc_utils.py.
…est metadata onto a ContextVar The v3 parallel-request limiter stashed its per-request bookkeeping (TPM reservation, descriptors, parallel slot, rate-limit response snapshot, released flag) in the request body's metadata channels. On routes where metadata is a provider request parameter (Responses API and the other LITELLM_METADATA_ROUTES) that leaked internal keys upstream and produced HTTP 400s, and it required denylist stripping plus dual-channel writes to contain. The stash now lives on an asyncio ContextVar holding a single typed RequestRateLimiterStash per request. The pre-call hook writes it, and the success/failure callbacks, disconnect release, and post-call hooks read and clear the same shared instance, which keeps the refund and slot release idempotent across sibling callbacks. The request body is never touched, so the stash-key stripping, the metadata mirror writes, and the all_litellm_params denylist entries are removed
…context rate OpenAI publishes a long-context column on the Flex tier, at half the standard long-context rate. We had no field for it, so a >272k flex request fell through to the standard long-context price and billed 2x: Terra $4/$18 instead of $2/$9, Luna $0.40/$1.80 instead of $0.20/$0.90, Sol $10/$45 instead of $5/$22.50. Adding the values to the cost map alone does nothing, because get_model_info builds ModelInfoBase from an explicit kwargs list and silently drops any key not named there. Declare the four *_above_272k_tokens_flex fields and wire them through, then add the values for sol, terra, luna, and the gpt-5.6 alias. That same gap was already swallowing cache_creation_input_token_cost_flex, _priority, and _above_272k_tokens, which were present in the cost map but never reached the calculator; they are wired through here too. Fast mode (ex-Priority) publishes no long-context column, so nothing is added there rather than deriving a rate by analogy.
… so nested calls cannot release it
…itellm_v3_limiter_contextvar_stash # Conflicts: # litellm/proxy/hooks/parallel_request_limiter_v3.py # tests/test_litellm/proxy/hooks/test_parallel_request_limiter_v3.py
…s record provider Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…miter raise-branch test
…sking guardrails on their own stream hook
…rsions in policies list
…ovider fix(caching): stamp provider on embedding cache-hit spend logs
…m client streams Streamed chat completions that did not opt into stream_options.include_usage were logged with tiktoken estimates over the visible text, so hidden reasoning tokens (billed as output by OpenAI-compatible providers) were never counted and SpendLogs could undercount output tokens by 90%+ on reasoning models. The proxy now injects include_usage upstream for /v1/chat/completions streams by default and strips the injection artifacts (the final usage chunk and the empty prompt-filter chunk) from the client-facing SSE stream, so accounting uses provider-billed usage while the client-visible stream stays byte-identical to today. always_include_stream_usage keeps its existing semantics: true forwards the usage chunk to clients as before, and an explicit false now acts as a kill switch that disables the upstream injection for OpenAI-compatible backends that reject stream_options.
…t_call_guardrails fix(proxy): run post_call guardrails on /v1/messages streaming via unified guardrail translation
…nd neutralize client-sent strip marker Bytez and OCI param maps raise on stream_options when drop_params is unset, so the default injection would have broken every streamed chat completion routed to them. Injection now only happens when every router deployment behind the requested model (wildcards and aliases included) declares stream_options in its supported OpenAI params; providers that do not declare it either reject the param or already stream usage natively, so skipping them keeps old behavior instead of erroring. _litellm_strip_stream_usage arriving in the client request body is now overwritten at ingress (and popped in the experimental queue endpoint), so a client can no longer suppress the usage chunk it explicitly requested by planting the internal marker.
…put_limits fix(fireworks_ai): correct Kimi K2.5/K2.6/K2.7 max output token limits
The single-element tuple loop that bound the extracted deployment model inside the comprehension read poorly; an assignment expression in the filter clause does the same call-once-and-filter in one line.
… guardrail info reaches spend logs
…ths (#34856) The keyless flow (gateway as authorization server, no virtual key) worked only at the aggregate /mcp scope: the session-bearer admission arm was gated on _is_aggregate_mcp_scope, the 401 fallback only challenged at aggregate scope, and per-server protected-resource metadata for plain oauth2 servers pointed clients at the per-server relay, whose flow returns the raw upstream token that ingress can never accept keylessly (401 "LiteLLM Virtual Key expected. Received=gho_****"). Per-server spellings now join the same gateway flow for gateway-managed oauth2 servers (auth_type oauth2 without delegate_auth_to_upstream, new MCPServer.is_gateway_managed_oauth2 owner): - the session-bearer arm admits at any MCP scope; downstream grant resolution already intersects the admitted subject's servers with the path or header targets fail-closed, so a narrower scope never broadens - the 401 challenge is scope-aware: a single gateway-managed oauth2 path target gets the per-server resource_metadata in the spelling the request used, everything else gets the aggregate document; unknown names, CSV multi-target paths, and every client-forwarded or delegated mode keep their existing behavior - per-server PRM for explicitly named gateway-managed oauth2 servers advertises the gateway AS ({base}/mcp); delegate, passthrough, bridge, OBO, and the root-resolved unnamed shape are byte-identical - the preemptive 401 for an admitted keyless subject with no vaulted token challenges with resource_metadata (re-entering the gateway flow, whose authorize interlude vaults the upstream token) instead of the relay authorization_uri, which cannot vault without a litellm key The per-server challenge URL builder moved from server.py to oauth_utils.py (shared with the auth module) and now inserts the SERVER_ROOT_PATH segment exactly as the discovery routes do. Resolves LIT-4864
Team-scoped models store an internal model_name_{team_id}_{uuid} name
with the public alias only in team_public_model_name, so resolving them
through get_model_list without team_id returned no deployments and the
gate skipped injection, leaving those streams on tiktoken estimates.
Thread user_api_key_dict.team_id through the gate.
fix(proxy): request stream usage upstream by default and strip it from client streams
…stash refactor(rate-limits): move the v3 limiter per-request stash off request metadata onto a ContextVar
#35301) The batch rate limiter counts input tokens by awaiting litellm.afile_content with no timeout, so a slow Files API holds POST /v1/batches open past any client deadline; stage saw 63.6s against the harness's 60s read timeout. The test times out before reaching the unattributed-spend-row assertion it exists to guard, so it reports an infrastructure hang rather than the contract. Skipping keeps the signal honest until the fetch is bounded.
…t on (#35294) The Headroom guardrail sent every message to /v1/compress, including the system prompt and the user's current instruction. On an agentic /v1/messages request the live turn is the largest compressible blob, so it came back as a hash marker; the model then called headroom_retrieve and got its own instruction returned in a tool_result block, which reads as data it fetched rather than a request to act on, so it described the content instead of doing the work. litellm already owns the policy for what a compressor may never rewrite: get_protected_indices covers the system rows, the last user row and the last assistant row, and compress() expands it over whole tool exchanges. Headroom now consults it (promoted from a private name and given tests) and expands it the same way, so the trailing tool result cannot come back as a marker standing in for the result of the call the model just made. Protected rows are withheld from the payload rather than pinned afterwards, so their tokens are not reported as savings that are never applied; the write-back discards a compressed system prompt outright, so that saving never existed. The cost is that a query-aware service no longer sees the newest user message. A response whose row count differs from what was sent can no longer be interleaved with the withheld rows, so it goes through the configured fail policy instead of being adopted. Fail-open now returns the caller's own inputs object: translation handlers detect a rewrite by identity, so a rebuilt copy sent an unchanged request through the Anthropic write-back for nothing. That write-back rebuilt the request with one anthropic_messages_pt call, which merges every run of consecutive user/tool rows, so a tool_result turn and the user turn after it arrived fused. Converting a row at a time would separate them but breaks tool pairing: with modify_params on, an assistant row whose results are converted separately reads as an orphaned tool call and the sanitizer answers it with a synthetic "tool execution skipped" result while dropping the real one. Conversion is now grouped by tool_call_id ownership, which satisfies both, and the same grouping decides which rows headroom protects, so the two agree by construction. The CCR follow-up also dropped any text the model wrote alongside its tool call, and echoed tool calls it had no results for. Both are fixed by reusing compresr's extraction helper, now shared instead of duplicated. Resolves LIT-5018
…35302) The Default User Settings form on Internal Users, the org settings form and the org create dialog all rendered their money fields as `<input type="number" step={0.01}>` inside a form that never opted out of native constraint validation. Any value with more than two decimals, such as a 0.001 max budget, failed the browser's step check, so Chrome vetoed the submit before react-hook-form ran. No request went out, no field error was shown, and the read view kept displaying the old value; it looked like the budget silently refused to stick. Money fields now use `step="any"`, and the three react-hook-form forms carry `noValidate` so zod stays the only validator and a DOM-level constraint can never swallow a submit again.
…lookup fix(guardrails): serve config guardrails from list and info endpoints without a DB and make their ids stable
* fix(anthropic): split mixed reasoning stream chunks * style: use builtin generic annotation * fix(anthropic): split mixed stream chunks by payload kind The mixed-chunk split cleared only the fields it knew about on each deep-copied piece, so any other payload riding the chunk survived on both pieces: tool_calls were emitted as two tool_use blocks with the same id, thinking_blocks on the text piece emitted duplicated thinking into a text block while dropping the answer text, and chunks whose reasoning arrived only as thinking_blocks never split at all Rebuild each piece's delta from scratch with exactly one payload kind (reasoning, text, tool calls), ordered to match native Anthropic block order. Fresh Delta construction keeps unset attributes deleted, which matters because the translators branch on hasattr, and prevents future Delta fields from riding along on every piece * fix(anthropic): keep continuation and multi-choice chunks unsplit, emit signature-less thinking once Adversarial verification against the merge-base found three shapes where the payload-kind split changed behavior beyond its target: a mixed chunk carrying a tool argument continuation was torn into a truncated block plus a fabricated one, a multi-choice chunk lost its secondary choices' payload, and a signature-less thinking_blocks piece inherited the non-empty block start body so accumulators collected the thinking twice Continuation and multi-choice chunks now pass through the splitter untouched, matching the merge-base byte for byte, and signature-less thinking_blocks pieces are normalized to reasoning_content so the block start opens empty and the thinking text is emitted exactly once --------- Co-authored-by: Napuh <naamanynadiemas@gmail.com>
…rride is removed and keep same-named DB drafts reachable in the UI
…1) (#35185) The ComplexityRouter's LLM classifier saw only the last user message, so on a multi-turn conversation it classified whatever happened to be last rather than what the human actually asked, and a near-constant classifier input pinned a whole session to one tier. The blindness turned out to be narrower than first diagnosed, and the fix is correspondingly smaller. Tool output was never the problem: on the Messages surface it rides a user turn as tool_result content blocks, which are not text parts, so flattening to `type == "text"` already dropped those turns; on chat completions it arrives on a `tool` role the extractor never read. Both surfaces were already handled before this change. What actually leaked through was the harness `<system-reminder>` block, which arrives as ordinary text, survives flattening, and became the current ask on any turn that carried one. So reminders are stripped rather than used to reject the turn, because a harness injects them alongside the live ask and not as a turn of their own; rejecting the turn would lose the ask, and keeping the block would feed the classifier the near-constant boilerplate that flattens tier selection in the first place. An earlier revision of this change also pattern-matched serialized tool_result payloads. That check only ever fired on a hand-serialized string neither request surface produces, it was where every review finding in this PR lived, and it is deleted here; the tests now pin the real shapes instead of the synthetic one they were built on. The classifier call is split into a system role carrying the rubric plus the caller's own system prompt, which stays byte-stable across a session so a provider can prompt-cache it, and a user role carrying the variable context: a bounded window of prior user turns, a conversation-depth signal, and the current ask. The caller's system prompt rides every turn, so task constraints are never dropped. The depth signal measures content-parts messages too, since counting only string content reported ~0 tokens for exactly the deep Messages-surface conversations that most need an expensive tier, and it is omitted entirely on the prompt-only path rather than asserting a false zero. Prior turns are excluded by matching the current ask rather than by dropping the newest turn positionally, because `aclassify` takes `prompt` and `messages` separately and a caller may classify something other than the newest turn. Truncated turns carry a marker so the classifier can tell a turn was clipped. Only the LLM classifier's input changes. The heuristic scorer, keyword overrides, escalation matching and semantic embedding still read the extracted current ask, which is why that extraction has to yield one clean human-authored string: those are substring and vector matchers, and an escalation keyword sitting inside a reminder blob would otherwise trip a tier jump on its own. Defaults keep single-turn classification equivalent to before. The prior-turn window is on by default so existing LLM-classifier deployments actually get the fix; the config field documents that those turns reach the classifier model, which may be a different provider than the routed completion model, and that the call already carries the current ask and the caller's system prompt in full. Scoped to the ComplexityRouter; the semantic AutoRouter is not touched.
…n classifier call (#35300) The complexity router's classifier sub-call copies the parent request's metadata verbatim, so its spend log row carries the caller's key, team and user and is indistinguishable from traffic the caller actually sent. Nothing on the row says otherwise: call_type is "acompletion" either way, model_group is overwritten to the classifier's own model group so the row never looks auto-routed, and routing_decision is absent exactly as it is on an ordinary request. Record the fact the system already knows at call time. internal_call_origin is declared on SpendLogsMetadata, which is the allowlist _get_spend_logs_metadata projects onto, and stamped in _classifier_call_metadata; both classifier paths already route through that one function and it feeds the metadata and litellm_metadata buckets alike, so every request surface is covered at one site. The key is reserved rather than caller-supplied, so it joins routing_decision in the untrusted-metadata strip and a caller cannot label their own traffic as router overhead. The classifier call also inherited no session identity, so the router minted a fresh trace id and the row landed in a session of its own. Forwarding the parent's session puts it in the trace of the request that triggered it, which is where an operator looks for what the routing cost.
…hen all DB versions are deleted
…rail_info fix(logging): bind litellm_metadata by reference in function_setup so guardrail info reaches spend logs
… order-dependent flake
…35291) * feat(s3): support SSE-KMS encryption params on both S3 logging paths * fix(s3): ignore non-string SSE config values instead of crashing logger init * Update litellm/integrations/s3.py Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(s3): invalidate only the mistyped SSE field instead of dropping both --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…e_db_sync fix(policy_engine): preserve config-defined policies across DB sync and expose them via list APIs
…ail_flake test: fix order-dependent flake in passthrough guardrail call-type test
…itellm_responses_error_code_map
fix(pricing): correct gpt-5.6 prices for openai, bedrock, and flex long context
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
fix(cost): bill the fast service tier at the priority rate
fix(responses): map all documented in-stream error codes to real HTTP statuses
tin-berri
approved these changes
Jul 31, 2026
test(pricing): cover gpt-5.6 cache-cost plumbing and bedrock_mantle responses billing
shin-berri
approved these changes
Jul 31, 2026
Contributor
Greptile SummaryThis promotion updates proxy request processing, MCP behavior, guardrails, policy management, rate limiting, logging, pricing metadata, compression, caching, and corresponding dashboard surfaces and tests.
Confidence Score: 4/5The PR should not merge until nested limiter acquisitions are isolated and policy registry updates remain consistent with successful database operations. Nested calls can lose a parent concurrency acquisition for up to an hour, updated policies can continue executing stale definitions by UUID, and a failed post-delete synchronization can leave a removed attachment actively enforced. Files Needing Attention: litellm/proxy/hooks/parallel_request_limiter_v3.py; litellm/proxy/policy_engine/policy_registry.py; litellm/proxy/policy_engine/attachment_registry.py
|
| Filename | Overview |
|---|---|
| litellm/proxy/hooks/parallel_request_limiter_v3.py | Introduces shared request-stash ownership for limiter bookkeeping, but nested calls can overwrite and leak a parent acquisition. |
| litellm/proxy/policy_engine/policy_registry.py | Adds policy version lifecycle support, but successful non-production updates leave explicit ID-based resolution stale. |
| litellm/proxy/policy_engine/attachment_registry.py | Adds persisted policy attachment management, but a failed post-delete synchronization leaves the removed attachment active in memory. |
| litellm/proxy/_experimental/mcp_server/auth/user_api_key_auth_mcp.py | Refines MCP admission for delegated credentials, passthrough requests, bridge envelopes, and gateway sessions without an accepted actionable defect. |
| litellm/proxy/_experimental/mcp_server/server.py | Adds stale-session and stateful-session handling with ownership and synchronization protections. |
| litellm/integrations/s3.py | Adds validated server-side encryption and KMS-key parameters to legacy S3 logging. |
| litellm/integrations/s3_v2.py | Extends batched S3 logging with matching KMS encryption configuration. |
| litellm/compression/compress.py | Exposes a generalized protected-message index helper while preserving system and latest-turn protection. |
| ui/litellm-dashboard/src/app/(dashboard)/policies/_components/PolicyTable.tsx | Updates the policy management interface for the new backend lifecycle and attachment behavior. |
Comments Outside Diff (2)
-
litellm/proxy/policy_engine/policy_registry.py, line 495 (link)Policy ID cache stays stale
When an administrator updates a draft or published policy, the database write succeeds without refreshing
_policies_by_id. Explicitpolicy_<uuid>requests therefore continue executing the previous guardrail definition until a later full synchronization.Context Used: CLAUDE.md (source)
-
litellm/proxy/policy_engine/attachment_registry.py, line 352 (link)When the database deletion succeeds but the subsequent full attachment synchronization fails, the endpoint returns an error while the deleted attachment remains in the in-memory registry. New requests continue enforcing a policy attachment that no longer exists in the database until another synchronization completes.
Context Used: CLAUDE.md (source)
Reviews (1): Last reviewed commit: "Merge pull request #35307 from BerriAI/l..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+393
to
+398
| def claim_request_stash_for_data(data: dict) -> RequestRateLimiterStash: | ||
| stash = get_or_create_request_stash() | ||
| owner_call_id = data.get("litellm_call_id") | ||
| if isinstance(owner_call_id, str): | ||
| stash.owner_litellm_call_id = owner_call_id | ||
| return stash |
Contributor
There was a problem hiding this comment.
When a rate-limited request invokes a nested LiteLLM call, the nested pre-call hook overwrites the shared stash's owner and parallel slot. The parent callback then cannot recover its acquisition, leaving it counted for up to the one-hour TTL and eventually causing false rate-limit rejections.
Context Used: CLAUDE.md (source)
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.