chore: sync upstream 2026-07-17 - #133
Merged
shudonglin merged 42 commits intoJul 17, 2026
Merged
Conversation
…single-server REST statuses
The aggregate MCP tools/list absorbed every per-server failure (upstream 401/403/5xx, timeouts,
network errors) into that server contributing zero tools, making a broken upstream indistinguishable
from a healthy server with no tools; the single-server REST list masked the same failures as
{"tools": [], "error": null, "message": "Successfully retrieved tools"}
Phase 2 of the MCP error-handling framework (LIT-4419): the manager fetch hops now raise a
classified MCPServerListError (faults/list_outcomes.py: total classifier, frozen outcome values)
instead of returning [], and each boundary applies the relay-vs-absorb policy matrix. The aggregate
keeps serving the healthy subset but records each server's outcome, surfaced on the tools/list
result _meta under litellm.ai/server_outcomes (the SDK passes a ListToolsResult through unwrapped)
and in spend logs as per_server_list_outcomes. Single-server REST requests relay truthful statuses
(unreachable/upstream_error 502, timeout 504, internal 500) and access denials now surface as real
403s instead of 200 unexpected_error bodies; upstream 403s surface through MCPUpstreamAuthError
like 401s. Outcome wire values carry category and status code only, never upstream prose
Resolves LIT-4421
…a healthy empty server A cancelled fetch absorbed to [] made that server contribute ServerListOk(tool_count=0), the exact healthy-but-empty impostor this change removes. Cancellation stays suppressed (the pre-existing choice); it now carries an internal fault so outcomes stay truthful
…ding the upstream response
…m listing failures Both review findings shared one root cause: two exception-tree walkers with drifted semantics. _extract_upstream_auth_failure walked the incidental __context__ chain before explicit causes, so a 403 raised while handling the causal 401 could shadow it; and the generic _get_tools_from_server arm classified without extracting the challenge, so a nested 401 at client-build time surfaced without the WWW-Authenticate the client needs. upstream_auth_challenge and raise_classified_list_failure in faults/list_outcomes.py are now the single traversal and the single choice-point; both fetch arms and _extract_upstream_auth_failure (also serving tool calls and the connect-time probe) delegate to them, with dcr_bridge challenge suppression as a parameter so it holds on every path. The stale _fetch_tools_with_timeout docstring describing the pre-change 403 absorb is rewritten to the actual contract: 403 relays with its own status, an upstream-sent challenge relays verbatim per RFC 6750 insufficient_scope, and a challenge is only ever fabricated for a challenge-less 401
…che_control injection Anthropic only caches a prompt when the request carries explicit cache_control breakpoints, unlike OpenAI where prompt caching is automatic and needs no configuration. Today litellm can inject those breakpoints server-side, but only when an admin hand-writes cache_control_injection_points into a model's litellm_params (or router_settings.default_litellm_params). Clients such as Claude Code and Claude Desktop never set cache_control themselves, and the admin recipe is easy to miss, so Anthropic traffic through the proxy silently pays full price on every repeated prefix. This adds an opt-in litellm_settings flag, enable_anthropic_prompt_caching. When it is on and the request has no injection points configured and no client-supplied cache_control, litellm synthesizes a default pair of breakpoints (the system prompt and the trailing turn) so the stable prefix is cached while the breakpoint advances with the conversation. It is wired into both surfaces: /chat/completions seeds the points before the existing prompt-management gate, and /v1/messages resolves them in maybe_inject_cache_control, so the existing AnthropicCacheControlHook applies them unchanged and keeps its four-block cap and its refusal to overwrite client breakpoints. The default is off, so no existing deployment changes behavior. Injection is gated to providers that actually consume cache_control markers (anthropic and bedrock) and to models the cost map flags as supporting prompt caching; note that supports_prompt_caching alone is not a sufficient gate, since OpenAI, Azure and Gemini models report it as well but never take cache_control markers. The default ttl is Anthropic's 5 minute ephemeral cache, with an optional anthropic_prompt_caching_ttl of "5m" or "1h"; ttl is also added to ChatCompletionCachedContent, which the bedrock and anthropic transforms already read at runtime but the type never declared Resolves LIT-4478
Both enable_anthropic_prompt_caching and anthropic_prompt_caching_ttl are now read from LITELLM_ENABLE_ANTHROPIC_PROMPT_CACHING and LITELLM_ANTHROPIC_PROMPT_CACHING_TTL at import, so the flag can be turned on without a config file. An unsupported ttl falls back to the provider default rather than reaching the provider verbatim
…ols/call honors them
_request_has_cache_control only looked at messages and system, so a client that marks cache_control on tools alone did not suppress auto-injection. Tool breakpoints count toward the provider's four-block limit, so three of them plus the two injected here is five, which Anthropic rejects. Thread tools through both entry points and treat a client-marked tool as the stand-down signal it already is for messages and system.
The consolidation regressed the pre-existing walker semantics: _extract_upstream_auth_failure used to keep scanning until it found a 401/403, while the consolidated helper took the first response of any status and then tested it, so a causal 401 sitting behind an unrelated 5xx (retry attempts, multi-stream task groups) was misclassified as upstream_error and its challenge lost on the listing, tool-call, and probe paths. The traversal is now an iterator in deliberate order and each consumer applies its predicate over the stream: the auth scan takes the first 401/403 even behind non-auth responses, generic classification takes the first response, and classify_list_exception derives its auth arm from the same scan so the carrier choice and the classification can never disagree
…l_name (BerriAI#33691) * fix(router): tag-aware pre-routing strategy selection for shared model_name Complexity/auto/adaptive/quality router registries were keyed by model_name alone, so a second deployment sharing a model_name but carrying different tags was rejected and every request used the first config. This made tag-based routing to distinct provider configs behind one alias impossible, surfacing as 401 'Not allowed to access model due to tags configuration' for the second tag. Each registry now holds a list of tag-scoped strategies and async_pre_routing_hook selects the entry whose tags match the request before classification, falling back to a default-tagged then first-registered entry. A repeat of the same (model_name, tags) pair is still rejected. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): cover tag-scoped pre-routing strategy registry helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore: re-trigger CI Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…auge (BerriAI#32441) * fix(proxy): enforce max_parallel_requests as a per-slot concurrency gauge The v3 rate limiter tracked max_parallel_requests with the same sliding-window machinery as RPM/TPM. A concurrency gauge cannot live on a windowed counter: every window roll reset the counter to 1 while requests were still in flight, the completion decrements for those forgotten requests then drove the counter negative, and rejected requests left stranded increments that nothing released. Under sustained load a key with max_parallel_requests=5 let backend concurrency climb to the full client concurrency (observed 60 on a live proxy) while the proxy kept returning 429s for everyone else Replace the windowed counter with a per-slot registry (Redis sorted set of slot ids scored by acquire time, with an asyncio-locked in-memory fallback): admission atomically prunes expired slots and registers a new slot id only when in_flight + 1 <= limit, so rejected requests never occupy a slot; success, failure, and client-disconnect paths release exactly the slot id this request acquired (stashed in the request metadata channels), so a release without a matching acquire or a double-fired callback can never free another request's slot; and a slot leaked by a crashed worker is pruned individually after its TTL even under continuous traffic Resolves LIT-4259 Fixes BerriAI#16011 * fix(proxy): release every acquired gauge and respect mirrored counts in the in-memory fallback Address review findings on the slot-registry gauge: the acquisition stash now carries the gauge counter keys alongside the slot id, so the release paths free the slot from every gauge it was registered under instead of hardcoding the api_key scope, and the disconnect release keys off the stashed acquisition instead of the key object's current max_parallel_requests configuration (which can change mid-request). The in-memory fallback now treats a cached integer (the count mirrored from the last successful Redis script call) as real occupancy, carrying it forward as a floored counter during a Redis outage instead of restarting from an empty registry * fix(proxy): release the parallel slot on proxy-level rejections async_post_call_failure_hook is the only callback that fires when a downstream hook (guardrail, budget check) rejects a request after the rate limiter's pre-call hook acquired a slot; async_log_failure_event is a completion-level callback and never runs for proxy-side rejections. Release the stashed acquisition at the top of the hook, before the TPM reservation guard, so those slots do not linger for the full slot TTL and wedge the key at its limit under moderate rejection rates. Clearing the acquisition marker keeps the release idempotent when a later failure callback runs in the same flow * test(proxy): cover success release, read-only count, Redis release mirror, and TPM rejection release Four behaviors of the slot-registry gauge had no direct test: a successful completion releasing exactly its acquired slot, read_only callers counting in-flight slots through the count script (and degrading to the local mirror when the script fails) without acquiring, the Redis release script mirroring returned counts into the local cache, and the TPM reservation rejection releasing the already-acquired slot before raising * style(proxy): use builtin generics and union syntax in new rate limiter annotations The slot-gauge code added Tuple/List/Dict and Optional[...] annotations, pushing the UP006 and UP045 strict-rule totals past their ceilings in ruff-strict-budget.json. Convert only the annotations this branch introduces to builtin generics and PEP 604 unions, leaving the rest of the module untouched.
…l on auth-enforced pass-through routes (BerriAI#33710) * fix(proxy): stop treating upstream model body field as a LiteLLM model on auth-enforced pass-through routes An auth: true user-defined pass-through endpoint runs full virtual-key auth, and get_model_from_request unconditionally extracted the request body model field, so key/team/user/project model allowlist checks rejected requests whose model only exists upstream (key_model_access_denied), even when the key was explicitly granted the route via allowed_passthrough_routes. The pass-through route registry moves to a leaf module (route_registry.py) that the auth layer can import without re-entering the pass_through_endpoints -> user_api_key_auth -> auth_utils import cycle. get_model_from_request now returns None for routes registered as user-defined pass-through endpoints (exact and subpath), which skips model allowlist and per-model budget enforcement on those routes while key auth, allowed_passthrough_routes, and spend/budget checks stay intact. Built-in provider passthrough routes (/vertex_ai, /gemini, ...) keep model enforcement. Resolves LIT-4299 * fix(proxy): key pass-through model-access skip on the dispatched endpoint, not the request path Addresses a model-authorization bypass: the first version decided whether to skip model-allowlist extraction by matching the request path against the pass-through route registry. That ignored the HTTP method and, more importantly, whether the request was actually dispatched to a pass-through handler. A custom pass-through whose path collides with a built-in route (e.g. /v1/chat/completions, or an include_subpath prefix of one) still writes a registry entry even though FastAPI serves the built-in handler, so a normal request to that route had its model checks skipped and could reach a model outside the key/team/user/project allowlist. The skip is now keyed off the FastAPI-resolved endpoint. create_pass_through_route tags its handler with LITELLM_PASS_THROUGH_ENDPOINT_MARKER, and get_model_from_request returns None only when request.scope["endpoint"] carries that marker. Because routing runs before auth dependencies, this reflects the handler that actually serves the request: on a collision the built-in handler is dispatched and carries no marker, so model enforcement stays on. This also removes the need for the separate route_registry module, so that extraction is reverted. Regression tests cover a pass-through-dispatched request (model suppressed), a built-in-dispatched request on the same path (model still enforced), and the no-request budget path. Resolves LIT-4299
…_call_grants fix(mcp): expand toolset grants in shared permission primitives so tools/call honors them
…omes Append-append conflict at the end of test_mcp_server.py between this branch's aggregate-outcome tests and the mode-aware preemptive-401 tests from staging; both kept
…33656) * feat(complexity-router): user-triggered escalation keywords Add an escalation_keywords config option to the complexity router so a user can force a bump to the next-higher complexity tier by including a phrase in their message (a stronger model, but not one they get to choose). Defaults to ['LITELLM ESCALATE'] when unset, case-sensitive so it only fires on the deliberate shouted form; admins can override the list or set [] to disable. Escalation applies across every routing path: heuristic/LLM classification, literal and semantic keyword_tier_rules overrides, adaptive routing, and session affinity (where it bumps relative to the pinned model and persists the higher tier for the rest of the session). Capped at the highest configured tier and skips unconfigured intermediate tiers. Expose it in the Auto-Router v2 UI as an Escalation Keywords field wired into the complexity_router_config payload. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(complexity-router): validate escalation keywords and pin at tier ceiling Strip blank/whitespace escalation keywords so an empty phrase can't match every message and escalate all traffic. Keep the exact pinned model when a session escalates at the highest configured tier instead of randomly hopping to a peer in a multi-model pool. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…#33714) Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…BerriAI#33728) The gpt-realtime family (OpenAI and Azure) only serves /v1/realtime and is rejected by /v1/chat/completions with "This is not a chat model", but the cost map tagged them mode=chat. Retag them mode=realtime (a value already used by gemini-live and handled by the health-check realtime handler) and add realtime to the ModelInfoBase mode literal. Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ic_auto_cache feat(anthropic): add enable_anthropic_prompt_caching for automatic cache_control injection
…edrock/Vertex (BerriAI#33719) * fix(anthropic): self-heal on missing thinking-signature errors from Bedrock/Vertex Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): narrow thinking signature error marker Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): stabilize prompt caching fixture size Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore: re-trigger CI Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…gins from installed packages (BerriAI#33644) Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…every team key (BerriAI#33632) * test(e2e): assert bare-key budget refusal is 429 and /key/info spend reaches the cap * test(e2e): keep the bare-key budget assertion to the 429 refusal shape * test(e2e): assert a team's max_budget blocks every key on the team * test(e2e): focus the team budget case on the 429 blocking behavior
BerriAI#33638) * test(e2e): assert bare-key budget refusal is 429 and /key/info spend reaches the cap * test(e2e): keep the bare-key budget assertion to the 429 refusal shape * test(e2e): assert a team's max_budget blocks every key on the team * test(e2e): focus the team budget case on the 429 blocking behavior * test(e2e): assert an org budget block is a 429 naming the organization
…omes Conflict in _list_mcp_tools: staging (BerriAI#33612) moved toolset-grant expansion into the shared permission primitives and removed the _merge_toolset_permissions call; resolution applies that removal to this branch's AggregateToolListing structure
…id-stream (BerriAI#33736) * fix(proxy): bill partial streamed spend when the client disconnects mid-stream * fix(router): guard FallbackStreamWrapper chunks alias for non-CSW streams * fix(proxy): await disconnect billing dispatch instead of unrooted create_task * fix(proxy): make disconnect slot release single-owner to avoid double release * fix(proxy): use union syntax for disconnect cleanup params (UP045 budget)
tests/e2e/grafana/status_history_panels.md was prose describing Loki/Grafana status-history panels and LogQL queries. Nothing in the tree imports, reads, or links to it; the e2e suite only emits the E2E_RESULT lines those panels consume (tests/e2e/conftest.py, tests/e2e/e2e_result_reporter.py) and never depends on this file. Dashboards drift when versioned as prose in the repo, so remove it; if we want them versioned it should be dashboard-as-code in the observability repo, not markdown here.
… and scope the no-unit-tests rule (BerriAI#33755) The e2e docs claimed `e2e`-marked tests skip when no proxy answers the liveness probe, but the harness has always hard-failed: conftest.py's pytest_runtest_setup calls pytest.fail, its module docstring states "hard failures only ... never skip", and logging/conftest.py forbids skipping outright. Align the docs to the code so the single most important contract reads the same everywhere; a dead proxy turns a run red instead of being silently skipped and mistaken for a pass. The per-suite conftest docstrings that described the shared hook as a "proxy liveness skip" are corrected to "liveness gate" for the same reason. Also scope the no-unit-tests hard rule to what it means: never substitute a unit test for e2e feature coverage, while explicitly allowing tests that cover the harness itself (e.g. coverage_registry/test_collector.py), which carry no e2e marker and run whether or not a proxy is up. No product code and no harness logic changed. Resolves LIT-4554
…fix, never canonical names Outcome keys in the tools/list _meta, the spend-log outcome and count maps, and the REST error messages now all use get_server_prefix (alias, or the short prefix when that mode is enabled), the same naming the caller already sees on tool names. Keying them by canonical server_name let an authenticated caller enumerate internal server names and their health or auth state that the alias and short-prefix schemes deliberately hide (Veria finding). One helper decides the key for every surface; exception messages reaching the multi-server REST error list are mapped to their fault tag with the display prefix instead of relaying exception text carrying canonical names. Server-side logs keep the real names
…port (BerriAI#33758) * refactor(e2e): replace bespoke result reporter with standard JUnit report tests/e2e/e2e_result_reporter.py hand-rolled a per-test logfmt emitter that reimplemented outcome mapping, logfmt escaping, and node-id parsing to print one E2E_RESULT line per finished test. Outcome, duration, and node id are all things a standard pytest reporter already produces, so the only genuinely custom data is the covers marker ids and the normalized package label Delete the module and emit a standard pytest JUnit XML report (--junitxml) instead, carrying the two custom signals as user_properties (JUnit <property> entries) attached at collection time in pytest_collection_modifyitems, so they land on every test on every outcome including skips and setup errors. The small package/covers extraction lives in junit_properties.py and is unit tested plus checked end to end against a real JUnit artifact in test_junit_properties.py Shipping the JUnit report to Loki is a thin infra-side transform, documented in grafana/status_history_panels.md * chore(e2e): remove grafana status history panels doc and junit properties e2e test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…na alert (provisioned outside the repo) (BerriAI#33749)
…utcomes feat(mcp): per-server outcomes for aggregate tools/list and truthful single-server REST statuses
Add an e2e suite at tests/e2e/mcp/ that proves MCP authorization over the api_key auth family. An admin registers an upstream MCP server through the management API (POST /v1/mcp/server, persisted in the DB and picked up without a restart) and queues its deletion. Two keys are created against that one server: one granted access through object_permission.mcp_servers and one with no MCP grant. The permitted key is a live control proving the upstream is reachable and the tool is callable, so a denial on the ungranted key is an authorization decision rather than a dead server. The denied key then sees none of the server's tools on tools/list and is refused a tools/call with a 403 access_denied. A deterministic self-hosted FastMCP upstream (add/multiply over streamable-http) is added to the e2e compose stack so the suite runs offline with a known tool set. KeyGenerateBody gains an optional typed object_permission so the shared gateway can create a key with an MCP grant.
* fix(embeddings): accept encoding_format='float' for vertex_ai/gemini embeddings (BerriAI#33617) OpenAI SDKs (and litellm's own client since ~1.84) send encoding_format='float' by default, but the vertex embedding config only supports ['dimensions'], so get_optional_params_embeddings raised UnsupportedParamsError at the provider default value. Any OpenAI-compatible client talking to a litellm proxy with vertex embedding models got a 400 unless the operator set proxy-wide drop_params: true. Float lists are exactly what the vertex API returns, so the param is a no-op: pop it before validation. Other values (e.g. 'base64') keep the existing unsupported-param behavior (dropped with drop_params, raise otherwise). Fixes BerriAI#33173 Co-authored-by: Mihidum Hettiyahandi <55163074+mihidumh@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(guardrails): add Singulr guardrail integration for LiteLLM gateway (BerriAI#31302) * singulr guardrail support for litellm gateway * Update litellm/proxy/guardrails/guardrail_hooks/singulr/singulr.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix comments * improvement * fix: resolve review comments and implement requested improvements * fix:Guardrail bypass through uninspected messages * fix:tool text scanning * fix: Legacy function definitions bypass scanning by adding indirect message scaning * chore: remove unintended basedpyright budget file * fix:Response schema bypasses guardrail scanning (response_format.json_schema) * chore: restore basedpyright-code-budget.json and update lint baselines Restores the file deleted in c698b88 to match upstream litellm_internal_staging. Regenerates basedpyright and ruff-strict budget baselines via make lint-budget-update. * fix: scan system messages as indirect prompt injection in Singulr guardrail * chore: restore lint budget files to upstream baseline * fix: resolve ruff UP006 and I001 violations in singulr guardrail * Update litellm/proxy/guardrails/guardrail_hooks/singulr/singulr.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * resolve review comments on Singulr guardrail * fix: scan tool call results as indirect prompt injection in Singulr guardrail * Apply suggestion from @greptile-apps[bot] Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * minor * formating fix * refactor: shift extraction logic to singulr side * refactor:keep precall hook only * fix:formatting * fix:linting * improve config description * Trigger CI * fix * fix:field description * fix:errors due to change in field names * style: apply ruff line-wrap formatting to singulr guardrail * fix:exception * fix:formatting * fix playground * improved * Update litellm/proxy/guardrails/guardrail_hooks/singulr/singulr.py Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> * Update litellm/proxy/guardrails/guardrail_hooks/singulr/singulr.py Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> * fix * fix ci issues * remove uv.lock from pr * fix * fix:resolved comments * chore: trigger CI * remove uv.lock * fix * fix linting * fix linting * fix linting * remove doc strings * remove test fixes * chore: retrigger CI * change in singulr api contract * remove some ut * send litellm call_id to singulr --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: aniket-kardile <aniket.kardile@singulr.ai> Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> * Fix non-conformant UUIDv7 generation in native Opik integration (BerriAI#31294) create_uuid7() encoded the timestamp in units of 16 seconds instead of milliseconds, so the top 48 bits came out ~4096x the real unix-ms. Opik's backend validates the embedded UUIDv7 timestamp on ingestion (OPIK-7067); the bad encoding decoded to ~year 2201 and every trace/span batch was rejected with HTTP 400. Rewrite create_uuid7() to be RFC 9562 conformant (top 48 bits = unix-ms), using the standard library only so no new dependency is added. Add unit tests covering UUIDv7 validity and millisecond timestamp encoding. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(proxy): expose uvicorn concurrency limit (BerriAI#33077) Expose uvicorn's limit_concurrency as a --limit_concurrency CLI flag and LIMIT_CONCURRENCY environment variable. Uvicorn counts both active tasks and accepted connections and returns HTTP 503 once the configured limit is reached. Reject non-positive limits at CLI parse time and only add the setting to the uvicorn startup arguments. Because idle connections also consume capacity, deployments should use upstream connection/header timeouts and per-client connection limits. * test: reorder test_utils tail to keep the daily merge conflict-free (BerriAI#33788) The daily OSS branch and litellm_internal_staging each appended an independent test block at the very end of tests/test_litellm/test_utils.py, so merging the two collides on that shared end-of-file position even though the additions are unrelated (this branch adds the vertex embedding encoding-format tests; staging adds the per-model prompt-cache-minimum tests). Moving this branch's new TestVertexEmbeddingEncodingFormat class above test_gemini_image_models_do_not_support_reasoning, which both branches share, gives the two additions different anchors, so git applies both without a conflict and without pulling staging into this branch. Pure reorder; no test bodies change --------- Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Mihidum Hettiyahandi <55163074+mihidumh@users.noreply.github.com> Co-authored-by: madan-singulr <150280287+madan-singulr@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: aniket-kardile <aniket.kardile@singulr.ai> Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> Co-authored-by: Aliaksandr Kuzmik <98702584+alexkuzmik@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Salva Madrid <50212436+salvamadrid@users.noreply.github.com>
…ncel tests Upstream BerriAI#33736 moved the max_parallel_requests slot release into an async proxy_logging_obj._arelease_max_parallel_requests_on_disconnect() call but didn't update this sibling test file's MagicMock stand-ins, so awaiting it raised TypeError. Reproduces on a clean upstream checkout too.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relevant issues
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Ran
uv run pytest tests/test_litellm/proxy/test_budget_reservation.py -qlocally against the merged tree: 51 passed (previously 4 failed withTypeError: object MagicMock can't be used in 'await' expression).All fork patches listed in
.github/fork-patches.txtwere verified present against the merged tree with the patch-verification pattern for each one, confirming none were silently dropped by the merge.Type
🚄 Infrastructure
Changes
Full
-X theirssync of BerriAI/litellmlitellm_internal_staging(41 commits), picking up the drift that landed after this morning's #129 sync. No merge conflicts.Upstream commit ae92e51 (BerriAI#33736) moved the
max_parallel_requestsslot release into an asyncproxy_logging_obj._arelease_max_parallel_requests_on_disconnect()call insideasync_streaming_data_generator's disconnect cleanup path, and added coverage for it in a newtests/test_litellm/proxy/test_common_request_processing.py, but never updated the pre-existing siblingtests/test_litellm/proxy/test_budget_reservation.py, whose bareMagicMock()stand-ins forproxy_logging_objdon't have that attribute configured. Awaiting an unconfiguredMagicMockattribute call raisedTypeError: object MagicMock can't be used in 'await' expressionin 4 tests. Verified this reproduces against a clean upstream checkout (both files are byte-identical toupstream/litellm_internal_staging), so it's upstream's own test suite gap, not a fork artifact. Fixed by adding_arelease_max_parallel_requests_on_disconnect = AsyncMock()to the two mock construction sites, documented in.github/fork-patches.txt.QA runbook
N/A, infra sync.
Final Attestation