fix(ci): rebase diarized_json transcription onto current litellm_oss_staging - #1
Draft
benlangfeld wants to merge 265 commits into
Conversation
The App Router migration moved pages to deeper path segments and the proxy can be mounted under a sub-path (e.g. /litellm behind a reverse proxy). Local logo asset paths were emitted without the server root prefix, so they resolved off the origin root and 404'd. Route every local logo src through a single resolver that prefixes the live server root path and leaves external URLs untouched, fixing provider, guardrail, vector store, callback, MCP and audit-log logos at any route depth and root path.
…c_rust feat: make rust OCR async-first
…custom-pricing-leak bugs (BerriAI#31249) Both tests were xfail(strict=True) for known proxy bugs: /team/new writing budget_limits as a raw list (Prisma 500) and custom per-token pricing leaking into the shared cost map for sibling deployments. Both are fixed, so the tests pass and strict mode reports the unexpected pass as a failure. Remove the markers (as their reasons instructed) so they run as plain regression guards; docstrings updated to describe the regression each now pins.
…et_config (BerriAI#31117) * fix(proxy): stop double-decrypting email/slack alerting env vars in get_config proxy_config.get_config() already returns environment_variables decrypted (the DB overlay decrypts them in _update_config_fields, and YAML values are plaintext), so the /get/config/callbacks slack and email blocks were running decrypt_value_helper() a second time on plaintext. That second decrypt always failed and the helper swallowed the error and returned None, so every SMTP_* field came back blank when the Admin UI reloaded the email settings, and the proxy logged a misleading "Did your master_key/salt key change recently?" error even when nothing changed. Consume the already-decrypted values directly, matching process_callback's handling of the same dict for langfuse/datadog/etc. Sensitive-value masking is preserved. Fixes BerriAI#19221 * fix(proxy): preserve a cleared slack webhook instead of falling back to OS env Use an explicit is-not-None guard rather than truthiness when deciding whether to fall back to os.getenv for SLACK_WEBHOOK_URL. With `or`, a webhook the admin cleared (stored as "") is falsy and would surface a stale SLACK_WEBHOOK_URL from the OS environment; only a truly absent key should trigger the OS lookup. No decryption is reintroduced.
…control (BerriAI#31264) * fix(mcp): correct misleading no-trusted-proxy warning for XFF access control * test(mcp): assert the no-trusted-ranges warning was logged instead of relying on StopIteration
…rded_for is off (BerriAI#31266) * fix(mcp): warn loudly when X-Forwarded-For is present but use_x_forwarded_for is off When a request carries an X-Forwarded-For header but use_x_forwarded_for is unset, get_mcp_client_ip silently falls back to the direct peer's IP (the load balancer / reverse proxy). That peer almost always sits inside mcp_internal_ip_ranges, so the 'Internal network only' (available_on_public_internet: false) restriction trusts every external caller as internal and effectively exposes those servers. Emit a one-shot loud error pointing the operator at use_x_forwarded_for instead of hard-failing: on a deployment with no load balancer, a crafted X-Forwarded-For header must not be able to take the service down, and a one-shot log keeps a flood of crafted headers from spamming the logs. * fix(mcp): re-arm XFF-disabled warning on config change and harden test assertion Address PR review: tie the one-shot warning flag to the observed use_x_forwarded_for value so it re-arms whenever the setting is seen enabled, restoring the diagnostic on a later rollback to disabled. Also assert against str(call_args) so the test survives a positional-to-keyword logger refactor.
…#31254) * fix(mcp): resolve toolset tools by the server's known prefix Toolsets store {server_id, bare tool_name} and reconcile that against the live prefixed tool name at list time. The reconciliation chopped the live name at the first MCP_TOOL_PREFIX_SEPARATOR with no server context, so a server whose prefix contains the separator (a hyphenated alias, or the UUID server_id used as the prefix when a server has no alias) had its tools silently dropped from /toolset/<name>/mcp while listing fine everywhere else. Strip the exact known prefix for the tool's server_id instead of guessing the boundary, on both the resolve and filter sides Also render toolset tools as {server-prefix}-{tool} in the dashboard picker result and chips; this is display only, the persisted record stays {server_id, bare tool_name} Resolves LIT-3419 * test(mcp): add focused unit tests for strip_known_server_prefix Cover the LIT-3419 cases directly on the helper with real MCPServer objects: clean prefix round-trip, hyphenated alias, UUID server_id fallback, unprefixed passthrough, and the server=None legacy fallback
…e_metadata (BerriAI#31255) An oauth2 MCP server with delegate_auth_to_upstream=true never prompted the user to sign in. On an unauthenticated initialize the gateway answered locally (200, no tools) and emitted no WWW-Authenticate, so clients like Claude Desktop either connected empty or hit "OAuth probe timeout after 10000ms". BerriAI#30124 added a bare `continue` in _raise_preemptive_401_for_unauthenticated_servers to stop sending LiteLLM's gateway authorization_uri challenge for delegate-auth servers, expecting the upstream to emit its own challenge. On initialize the gateway never probes upstream, so no challenge ever reached the client. Replace the `continue` with a preemptive 401 carrying the proxied resource_metadata (RFC 9728) challenge, the same form passthrough servers and MCPUpstreamAuthError already use. This keeps BerriAI#29770 fixed (still no authorization_uri) while restoring the upstream PKCE sign-in prompt.
…iAI#31270) * fix(ci): point OSS contributor workflows to litellm_oss_staging Workflow triggers and guard error messages incorrectly referenced litellm_oss_branch; update them to the branch we actually use for external contributions. * fix(ci): include test-rust.yml in litellm_oss_staging rename Missed test-rust.yml when updating OSS contributor target branch references. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…erriAI#30216) * fix(streaming): word-sliced cache replay for stream=true cache hits * fix(streaming): align mypy and replay happy-path test with word-sliced cache replay * fix(streaming): short-circuit whitespace-only content in cache replay splitter * fix(streaming): emit tool_calls/function_call only on first replay slice * refactor(streaming): drop dead delattr guard in cache replay A non-None usage on the replay base object always lives in __pydantic_extra__ (it is attached via setattr earlier in the same function), so delattr can never raise here; the try/except AttributeError that silently swallowed a failure was dead defensive code that could only ever hide a real regression, so it is removed in both the async and sync generators. Also switches the new replay annotations from typing.List to the builtin list to satisfy the strict ruff UP006 gate and drops the unused PLR0915 noqa directives (the rule is not enabled in this repo's ruff config, so RUF100 flagged them). * fix(streaming): drop carried-over metadata from later cache replay slices The word-sliced cache replay deep-copies the full ModelResponseStream per slice, so reasoning_content, thinking_blocks, logprobs, enhancements, annotations and the rest of the per-message metadata rode on every slice, not just the first. Downstream handlers that accumulate streamed deltas would collect each one once per slice, e.g. duplicating a cached reasoning trace N times on a stream=true cache hit. Later slices are now rebuilt as a content-only delta with choice-level logprobs and enhancements stripped, so the whole metadata class stays on the first slice. Adds async (logprobs) and sync (reasoning_content/thinking_blocks/logprobs/ enhancements, plus annotations) regression tests --------- Co-authored-by: Mateo <277851410+mateo-berri@users.noreply.github.com>
…ent IP resolution (BerriAI#31257) * feat(mcp): add mcp_xff_num_trusted_hops to harden XFF client IP resolution MCP per-server IP access control reads the client IP from X-Forwarded-For and trusts the leftmost entry. Behind an append-style proxy or load balancer (AWS ALB, nginx with $proxy_add_x_forwarded_for, HAProxy, Envoy, Cloudflare), a client can prepend an arbitrary value to the header, so the leftmost entry is attacker-controllable even when the direct peer is a trusted proxy. An attacker can therefore spoof an internal IP and reach servers marked available_on_public_internet=false. This adds an optional mcp_xff_num_trusted_hops general setting modelled on Envoy's xff_num_trusted_hops. When set to N, the client IP is read N entries from the right of the chain (where N is the number of trusted appending proxies in front of the gateway) instead of the leftmost value, so any entries a client prepends are ignored. It composes with mcp_trusted_proxy_ranges, which still validates the direct peer, and only takes effect once that check passes; without a validated direct peer the gateway keeps failing closed, so hop counting cannot be abused by a direct-to-pod attacker. The chain must contain at least N valid entries or resolution fails closed. Default is unset, preserving existing behaviour. * chore(ui): regenerate dashboard schema for mcp_xff_num_trusted_hops * fix(mcp): warn when mcp_xff_num_trusted_hops is below the minimum A 0 or negative value is silently treated as disabled, which could leave an operator believing they enabled append-style X-Forwarded-For hardening while client IP resolution stays on the spoofable leftmost value. Emit a warning, consistent with how the module already surfaces invalid CIDR config, so the misconfiguration is visible in logs. * fix(mcp): reject mcp_xff_num_trusted_hops < 1 at config-parse time Add a ge=1 bound to the ConfigGeneralSettings field so the update_config_general_settings path rejects 0 and negative values with a clear validation error instead of accepting them, and self-documents the valid range. The runtime warning stays as defense-in-depth for raw-dict config that bypasses model validation. * style(mcp): black-format ip_address_utils.py * fix(mcp): fail closed when mcp_xff_num_trusted_hops is set but invalid A present-but-invalid mcp_xff_num_trusted_hops (non-integer, or below 1) previously made _resolve_num_trusted_hops return None, which the caller treated identically to "unset" and silently fell back to the legacy leftmost X-Forwarded-For value. An operator who set the value to harden client IP resolution but typo'd it would get weaker security than before, with no fail-closed signal. Model the setting as a tagged union (_HopCountUnset, _HopCountInvalid, _HopCount) so the three states are distinct: unset keeps the legacy path, a valid count drives hop-counting, and an invalid value fails closed (returns "") instead of reverting to the spoofable leftmost address. The caller matches on the union exhaustively. Add a parametrized regression test asserting get_mcp_client_ip returns "" for 0, -1, "abc", and 1.5 even with a spoofed internal leftmost entry, and update the resolver unit tests for the new return type.
…erriAI#31262) * fix(otel): hashable scope for _emit_once when guardrail_mode is list `_emit_once` keys `spans_logged` by `(class, id, *scope)`. When a guardrail entry's `guardrail_mode` arrives as a `List[GuardrailEventHooks]` (the shape Presidio expands to with `output_parse_pii: true`, and the shape `event_hook` carries for any `mode: [...]` in config), the tuple contains a list and `spans_logged.get(dedupe_key)` raises `TypeError: unhashable type: 'list'`. On the post-call path this fires inside the logging callback and is swallowed; the request returns 200 but the OTEL `guardrail` span is silently dropped. On the blocking path the same error surfaces as HTTP 500. Adds `_freeze_for_dedupe`, a small recursive normalizer that turns lists and tuples into tuples, sets into frozensets, dicts into frozensets of `(key, value)` pairs, and falls back to `repr` for arbitrary unhashables. Applied inside `_emit_once` before the dict lookup, so all three callsites are protected without touching the guardrail-specific callsite. Helper assumes acyclic input; `guardrail_mode` values are built fresh from config (str enums, lists of str enums, TypedDict of str/list-of-str), so no cycle can arise in practice. Regression tests in `TestOpenTelemetrySpanDedupe` cover the list crash, distinct-list-scope collision, dict and set scope parts, and an end-to-end `_create_guardrail_span` exercise that confirms exactly one `guardrail` span is emitted across repeated lifecycle entrypoints. Each new test fails on a reverted helper (4/4 mutation kill) * fix(otel): cap _freeze_for_dedupe recursion depth and ignore in recursive detector CI's recursive_detector blocks new recursive functions in litellm/ unless they are in the allowlist with a documented bound. Cap the helper at 16 levels and return repr(value) past the cap; this is well past the realistic depth of guardrail_mode (1-3 levels) and means a future caller passing a cyclic container can no longer push the proxy logging path into a RecursionError. Add a regression test that exercises the cycle path. * refactor(otel): annotate _freeze_for_dedupe return as a HashableScope union Per review feedback from @mateo-berri: replace the loose `-> object` annotation with a recursive `HashableScope` union (str | int | float | bool | bytes | None | Tuple[HashableScope, ...] | FrozenSet[HashableScope]) so the helper's contract is visible at the signature. Replace the `try/except hash(value); return value` passthrough with an explicit isinstance check over the hashable-scalar types so the type checker can narrow without requiring `cast(Hashable, value)` on the return. Symmetric: dict keys also flow through the freezer (a TypedDict key is already a string in practice, so behaviorally identical). All 16 regression tests still pass; mutation kill behavior preserved * fix: avoid explicit casting --------- Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
* feat: package rust ocr bridge in litellm wheel * Install Rust in Windows CircleCI job * Address Rust wheel review feedback * Pin Windows rustup installer hash
Ignore the compiled, platform-specific Rust extension output (litellm/rust_bridge/_native*.so/.pyd) and the litellm-rust/target/ build dir so local maturin/cargo builds don't show up as untracked files. Also drop the two stale self-referential .gitignore entries; .gitignore is tracked, so ignoring it did nothing except add confusion. Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
… the strict gate (BerriAI#31335) * chore(lint): widen ruff budget slack to 10% of baseline for high-volume ANN rules and PLR0913 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * chore(lint): drop PLR0913 from strict gate to roll out rules gradually Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(lint): ratchet-guard rising baselines even when slack is cut to mask them Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* feat: port OCR providers to Rust gateway * chore(deps): update langgraph checkpoint lock * ci: scope ruff format check to changed files * ci: fix OCR lint and patch coverage * fix(ocr): block mapped IPv6 fetch targets * test(ocr): include rust bridge coverage in OCR shard * ci: rerun responses shard
The namespace configured under cache_params was only applied to get/set/
increment paths. Operations that take keys through other code paths (the Lua
scripts registered via async_register_script, delete, scan_iter, rpush, lpop,
get_ttl, and the sync increment_cache) hit raw keys. With a namespace set, the
rate limiter ({key}:tokens/requests/window), pod-lock release, and budget
limiters wrote keys outside the configured prefix, breaking multi-tenant key
isolation and leaving those operations reading keys the namespaced writes never
created.
check_and_fix_namespace is now applied uniformly across every key-taking
RedisCache operation. It is a no-op when no namespace is configured, so
deployments without a namespace are unaffected. The prefix is prepended ahead of
any {hash-tag}, so Redis Cluster slotting is preserved.
Resolves LIT-3374
…rriAI#30022) Fixes BerriAI#29794. Adds bare, gemini/, and vertex_ai/ entries copied from preview models so proxy cost tracking works for GA model names. Co-authored-by: Cursor <cursoragent@cursor.com>
…riAI#31933) Knip flagged remark-gfm as unused and date-fns as imported-but-undeclared, so drop remark-gfm (which prunes its transitive markdown subtree from the lockfile) and declare date-fns, which keyExpiryUtils.ts imports but only received transitively. Also delete the dead memory/components/index.tsx barrel, since nothing imports it once the page pulls MemoryView from its module directly Knip itself could not run: its Playwright plugin imports every config referenced by a --config flag in package.json scripts, and migration.serverRootPath.config.ts threw at import time when SERVER_ROOT_PATH was unset. Move that guard into a config-specific globalSetup so importing the config is side-effect-free; the check still fires loudly before any test runs when the prefix is missing
…AI#31897) Moves the user-management component tree (view_users plus BulkEditUsers, edit_user, DefaultUserSettings, user_edit_view, and the view_users table/columns/info-view) out of the shared src/components dump into the users route segment under _components, now that the app router owns the route. The page imports from a trimmed ./_components barrel UserInfo moves into networking.tsx beside UserListResponse, its real owner: networking defines the user API response shapes that embed it, and previously reached up into a view folder (components/view_users/types) to import the type. Defining it in networking removes that backwards data-layer-to-view dependency and drains the view_users/ folder entirely. CreateUserButton and onboarding_link stay in components/ since the create-key flow also consumes them Relative imports in the moved files are rewritten to @/components/* absolute paths, and the eight pre-existing eslint-suppressions entries are re-keyed to the new paths so the move stays behavior and lint neutral Verified: the moved suites pass with the same 75 assertions as before the move, tsc and eslint are clean, and next build compiles the /users route
BerriAI#31937) Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…on (BerriAI#31811) * feat(vertex_ai): pass full imageConfig dict for Gemini image generation Support all ImageConfig fields (aspectRatio, imageSize, personGeneration, imageOutputOptions) when calling Vertex AI Gemini image generation endpoints. Previously only aspectRatio and imageSize were extracted; other fields were silently dropped. Co-authored-by: Cursor <cursoragent@cursor.com> * style: ruff format vertex_gemini_transformation Co-authored-by: Cursor <cursoragent@cursor.com> * fix(vertex_ai): warn on non-dict imageConfig instead of silently dropping Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…de Sonnet 4 (BerriAI#31943) * fix(bedrock): drop strict/additionalProperties from toolSpec for Claude Sonnet 4 Claude Sonnet 4 on Bedrock Converse rejects toolSpec.strict and additionalProperties the same way Opus 4.7/4.8 do. Add bedrock_converse_supports_strict_tools: false to all Sonnet 4 regional variants so those fields are suppressed before the request is sent. Co-authored-by: Cursor <cursoragent@cursor.com> * test(bedrock): assert additionalProperties dropped for strict-unsupported models Rename the regression test to reflect Opus 4.7/4.8 and Sonnet 4 coverage, and assert both strict and additionalProperties are stripped from toolSpec. Co-authored-by: Cursor <cursoragent@cursor.com> * test(fireworks): skip embeddings live test when provider account is suspended --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude <noreply@anthropic.com>
…riAI#31899) Co-authored-by: Yassin Kortam <yassin@berri.ai>
…riAI#31809) * fix(ui): show info message when MCP tool preview returns 403 Internal users submitting MCP servers hit an admin-only preview endpoint; replace the red connection error with a clear review notice while leaving other failures unchanged. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): let BYOM submitters see their approved servers Approved user-submitted MCP servers defaulted to no access groups and allow_all_keys=false, so submitters could not see them after admin approval. Grant creator visibility for active submissions in get_allowed_mcp_servers. Co-authored-by: Cursor <cursoragent@cursor.com> * Improve dialogue box * fix(security): restrict MCP semantic filter settings to proxy admins Add an explicit PROXY_ADMIN check on PATCH /update/mcp_semantic_filter_settings and hide Semantic Filter and Network Settings tabs from non-admin users in the MCP Servers UI. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lint): use list[str] instead of List[str] to satisfy UP006 budget Co-authored-by: Cursor <cursoragent@cursor.com> * perf(mcp): cache BYOM submitter server lookup with 60s TTL Co-authored-by: Cursor <cursoragent@cursor.com> * style: fix ruff format and prettier formatting Co-authored-by: Cursor <cursoragent@cursor.com> * fix: preserve approved BYOM server visibility * fix(mcp): keep no-mcp-servers opt-out absolute and gate BYOM union by key scope The autofix in 94fd2bf made the no-mcp-servers sentinel return the caller's submitted BYOM servers, which weakened an explicit key-level opt-out into a soft preference. Restore the absolute opt-out and additionally skip the BYOM union for keys with an explicit object_permission.mcp_servers list and for toolset-scoped requests, mirroring how allow_all_keys servers are handled. Add unit tests for the sentinel, explicit scoping, toolset scope, the cache invalidation helper, the cache-miss DB path, and the db.py query helper. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude <noreply@anthropic.com>
…ios (BerriAI#30958) * tests: add e2e tests for spend, budgets and llms * style: make chained comparison of status_code clearer Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * remove e2e_tests folder * test: add spend tracking tests * fix: p0 issues, added types and shared functions for each test suite * style: carry clearer status_code comparison into renamed e2e dir * refactor: migrate to gateway client * fix: add new tests, split gateway * test(e2e): add live batches suite across providers and routing scenarios * test(batches): cover real cost tracking on completed batch retrieve * test(e2e): assert managed vs raw file and batch id shapes per routing scenario * test(e2e): assert full response shape of each batches and files endpoint * test(e2e): only accept transitional statuses for a freshly created batch * test(prompt-factory): make test_convert_url deterministic with a data URL picsum.photos is down (HTTP 522), so test_convert_url failed on every run. Swap the live external image for an inline data: URL and assert the round-trip through convert_url_to_base64 genuinely. A data URL is already inline base64 image data, so convert_url_to_base64 now short-circuits it instead of attempting an impossible HTTP fetch; add a regression for that branch in the mapped image_handling test * fix: pass through async image data urls * fix(image-handling): short-circuit data URLs in async path too Bugbot flagged that convert_url_to_base64 returns data: base64 URLs unchanged but async_convert_url_to_base64 still tried to fetch them, so async OCR flows (Bedrock, Azure) would reject inline images the sync path accepts. Add the same guard to the async function and a regression test that asserts the async path returns the data URL without touching the HTTP client * Fix: openai batches lifecycle * Fix: add e2e azure openai tests * Fix e2e for vertex ai * Add all models for testing * test(managed-files): assert idempotent upsert in store_unified_file_id store_unified_file_id switched from create to upsert to avoid UniqueViolationError when re-storing the same unified_file_id (e.g. batch output files stored before metadata is available). Update the unit test to assert the upsert call and its create payload instead of the removed create call. * test(batches): reconcile vertex_ai native batch-id comment with fallback guard * fix(test-config): keep rust-ocr models in model_list by moving files_settings after it * fix(test-config): move batch models after OCR block to keep merge with internal_staging clean * fix(batches): use '24hrs' completion window and allow managed-files listing with provider filter Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style: ruff format transformation.py and endpoints.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(e2e/batches): set Azure raw_model to gpt-4.1-mini-batch to match deployed model Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(vertex-ai/batches): correct completion_window to 24h per Literal type definition * test(vertex-ai/batches): align completion_window assertion to 24h * fix: update managed file metadata on upsert --------- Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…AI#31940) * fix(logging): resolve model_map_value for proxy custom pricing Use deployment model for standard logging cost-map lookup when the router overrides response.model to a group alias, and flush stdout when printing the payload. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(logging): add comment and test for deployment fallback in standard logging payload Address review: explain why the metadata["deployment"] fallback is unconditional, and add a test covering the get_standard_logging_object_payload code path. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(test): update model_map_key assertion for provider-prefixed keys Co-authored-by: Cursor <cursoragent@cursor.com> * fix(logging): scope base_model to model param only under custom_pricing Passing model=base_model unconditionally caused _get_provider_for_cost_calc to infer and prepend a provider prefix on all non-custom-pricing calls, changing model_map_key for existing deployments. Scope it to custom_pricing=True where the fix is actually needed. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
…AI#31576) * fix(mcp): roll up MCP tool spend to user counters and usage UI Direct REST MCP tool calls now fire success logging so spend_logs and user/team rollups include configured mcp_server_cost_info charges. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): gate key-info enrichment to requests missing user_id; fix import order - Only call _enrich_failure_metadata_with_key_info when user_api_key_user_id is absent, avoiding a cache/DB lookup on every normal LLM request. - Move LiteLLMProxyRequestSetup import to correct alphabetical position (I001). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): scope MCP spend aggregate by api_key to prevent cross-tenant disclosure Add api_key = ANY($2) to the MCP session aggregate query so it is bounded by the same ownership already applied to the main page query. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix spend logs for call and list mcp tools * Add tags in mcp logging * Fix ruff * fix(lint): replace List/Dict with list/dict in new annotations (UP006) Replace the 8 new UP006 violations introduced by the mcp-tags changes: - Optional[List[str]] → Optional[list[str]] for request_tags params - List[str] return type → list[str] in _get_parent_request_tags - Dict[str, Dict[...]] → dict[str, dict[...]] for mcp_spend_map annotation Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lint): keep call_tool_rest_api within complexity budget and narrow MCP spend enrichment except to PrismaError * fix(mcp): keep final streaming chunk when draining inner stream fails * fix: handle MCP logging edge cases * fix: propagate MCP logging cancellation --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
…1442) * feat(proxy): track cost for unmanaged Vertex AI batch jobs CheckBatchCost previously skipped Vertex batches created via the raw GCS input_file_id path, since their unified_object_id is a raw provider job id that fails the base64 managed-id check. Behind the opt-in general_settings flag track_unmanaged_vertex_batch_cost, the poller now derives the model from the gs:// input_file_id, maps it to a configured vertex_ai deployment, polls the batch, computes cost, and marks batch_processed=True. * Update tracking for failed", "expired", "cancelled" * fix(proxy): apply ruff format to proxy_server.py * address greptile review feedback (greploop iteration 1) Filter unmanaged Vertex batch deployments by vertex_ai provider so a shared model group name can't route to a wrong-provider deployment. Move gs:// URI parsing into VertexAIBatchTransformation. Add test coverage for the failed/expired/cancelled terminal-status DB update. * fix: route unmanaged vertex batches to matching deployment --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
…out (BerriAI#31632) * fix(prometheus): bound per-request budget metric emission with a timeout Wrap the per-request budget-metric gather in asyncio.wait_for so a slow Redis or DB lookup cannot consume the whole LoggingWorker watchdog and get the success-logging event cancelled. On timeout the emission is skipped in isolation; budget gauges are still refreshed by the periodic cron. The timeout is configurable via PROMETHEUS_BUDGET_METRICS_PER_REQUEST_TIMEOUT and defaults to 5.0 seconds, falling back to the default on an invalid value instead of raising * fix(prometheus): reject non-finite and non-positive budget-metrics timeout env float() accepts 0, negatives, nan and inf, which bypass the fallback: a value <= 0 makes asyncio.wait_for time out immediately and skip every per-request emission, and inf reintroduces the unbounded wait the timeout was meant to bound. Validate the parsed value is finite and greater than zero before using it, otherwise fall back to the default
When a guardrail blocks a post-call response, the synthetic violation response reported hard-coded zero usage, discarding the token usage the upstream call had already consumed. Fix the root cause rather than re-counting tokens: - Add an optional `original_response` field to ModifyResponseException. - The unified guardrail's post-call success hook attaches the blocked LLM response to the exception. - The /v1/messages and OpenAI-format (/v1/chat/completions, /v1/completions) block handlers report `original_response.usage` directly. Pre-call blocks never invoked the LLM, so usage is zero. Mock-based tests cover the helper (returns original usage / zero), the success hook attaching original_response, and the endpoint reporting it end-to-end. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ock (BerriAI#31389) Streaming moderation improvements for the unified guardrail post-call streaming iterator hook: - streaming_buffer_until_moderated: withhold all chunks until end-of-stream moderation passes, then release the original response (clean) or only the block message (blocked) -- the original content is never delivered on a block. Snapshot chunks with a shallow list() copy (end-of-stream builds a separate assembled response; chunks aren't mutated in place). - Clean Anthropic SSE on block: synthesize a well-formed termination sequence instead of a bare data: {"error": ...} blob that truncates the stream. Provider-specific synthesis lives in AnthropicMessagesHandler via build_block_sse_chunks (format-agnostic routing stays in the hook). - Mid-stream blocks continue the in-progress message (close open content block, append block message, terminate) rather than emitting a second message_start, which clients reject. Standalone envelope only when no chunks were sent (buffered path). - ModifyResponseException imported under TYPE_CHECKING + locally at runtime to avoid a module-level cyclic import. Adds regression tests for buffering (content withheld on block) and mid-stream continuation (single message_start). Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… content-rewriting guardrails - _standalone_block_chunks and _block_continuation_chunks now read real token usage from ModifyResponseException.original_response instead of hardcoding zero, matching the non-streaming _blocked_response_usage path. Shared helper moved to guardrail_translation/utils.py. - streaming_buffer_until_moderated is now forced off when the guardrail has mask_response_content=True, since buffered replay releases the withheld original chunks verbatim -- unsafe for a guardrail that rewrites content (e.g. PII masking). - Fix inverted streaming-flag precedence comment.
…-of-stream detection _check_streaming_has_ended assumed responses_so_far held ModelResponse objects with .choices, but for the Responses API the accumulated chunks are raw SSE event dicts, causing an AttributeError on every call
…emitting a broken stream When the initial LLM call inside MCPEnhancedStreamingIterator fails (e.g. an invalid previous_response_id -> provider 400 'No tool output found for function call ...'), the proxy returned HTTP 200 and the stream emitted the pre-generated mcp_list_tools discovery events with no response.created before them. That violates the Responses API streaming contract and crashes SDK stream accumulators (openai-node: "expected 'response.created' event, got response.mcp_list_tools.in_progress"). - aresponses_api_with_mcp now makes the initial call eagerly, before any SSE bytes are written, and re-raises the stashed failure so the client gets a real 4xx/5xx with the provider error body. - If a creation failure still surfaces during iteration, the stream emits a single terminal 'error' event instead of discovery events. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…low-up failures When tool execution failed as a batch, the stream proceeded to a follow-up call carrying function_call items with no outputs — rejected by the provider with 'No tool output found for function call ...' — and when the follow-up call itself failed, the stream simply ended with no terminal event. In both cases the client received HTTP 200 and a stream that looks like a truncated success: tool events, then silence. - Stash tool-execution and follow-up failures on the iterator. - Skip the doomed follow-up call entirely after a tool-execution failure. - Emit a single terminal OpenAI-style 'error' stream event carrying the mapped failure instead of ending silently. Builds on the initial-call failure handling from the previous commit (shares the _stream_error stash and _make_stream_error_event helper). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A request that explicitly asks for MCP tools via server_url litellm_proxy/... but resolves none of them (the API key/team has no access to the MCP server via allow_all_keys=false and no object-permission grant, the server name does not exist, or allowed_tools matches nothing) was silently sent to the model with no tools. The model then hallucinates, and the only trace is a list_mcp_tools spend log with status success and an empty response — the request looks healthy end to end while being completely broken. Raise a 400 BadRequestError naming the requested server URLs and the likely causes instead. Guard scope: - Mixed requests are exempt: with other (function) tools present, the request proceeds using those tools, matching the previous fallback behaviour. - Opt-out via litellm.reject_empty_mcp_resolved_tools = False (default True, per maintainer guidance). The auth-header pass-through test in tests/mcp_tests now resolves a dummy tool, since its purpose is header propagation, not zero-tool behaviour. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rop orphaned tool events on batch failure Review feedback (Greptile on BerriAI#32579): - The terminal error event was numbered sequence_number=1, out of order after tool-execution events. __anext__ now tracks the highest sequence_number that passed through the stream and the error event is numbered after it. - A batch tool-execution failure queued mcp_call.in_progress events that never received a terminal per-item event. Those queued events are now dropped; the terminal error event carries the failure instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Drop test_initial_call_success_does_not_emit_error_event: the tool-call happy path (test_tool_call_happy_path_emits_no_error_event) already guards against false-positive error events and exercises more of the changed code (tool-exec + follow-up success paths). - Drop the stream=True parametrization on the zero-resolved-tools guard: the guard runs before the stream/non-stream branch in aresponses_api_with_mcp, so both cases hit identical code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Removed MCP server configuration for deepwiki.
…cp_gateway_failure_handling fix(responses): fail loudly on MCP gateway failures (initial call, mid-stream, zero resolved tools)
ElevenLabs speech-to-text only accepted language and temperature, so a client asking for response_format=diarized_json either errored with UnsupportedParamsError or, with drop_params on, got it silently dropped. Even when diarization was enabled via the diarize passthrough, the response transformer flattened everything to text plus word timings and discarded the per-word speaker_id, so speaker information never reached the caller. This maps response_format=diarized_json to ElevenLabs diarize=true and shapes the response into OpenAI's diarized_json form: segments carrying a speaker label, plus usage and duration. The shape is chosen from what the provider returns rather than from request flags, since the response transformer only receives the raw response: a transcripts[] body (use_multi_channel=true) or words with a non-null speaker_id (diarize=true) produce segments, and a plain response keeps the existing flat words output untouched. For multi-channel audio, e.g. a stereo call with one person per side, the channel is used as the speaker so the two sides don't collide on the speaker ids ElevenLabs assigns independently per channel. The untyped JSON is validated through Pydantic models before use instead of being indexed as dicts, which also drops the file's basedpyright error count.
…on parse failure Address Greptile review feedback. _words_from_channels now falls back to the transcript's list position when channel_index is absent, so multi-channel speakers never collapse onto a single "speaker_None" label. The response transform again wraps parsing and validation in one try/except that re-raises ValueError, keeping the prior error contract instead of leaking pydantic ValidationError on malformed responses.
…count ElevenLabs rejects diarize and use_multi_channel together, so a caller asking for diarized_json could not blindly send both. The request transform now reads the channel count from the audio header (soundfile, already a dependency): a diarized_json request on multi-channel audio switches to use_multi_channel with combined output and drops diarize, while mono keeps acoustic diarization. An explicit use_multi_channel from the caller still wins, and an unreadable header falls back to mono, so one client call shape (response_format=diarized_json) covers both mono and stereo without the caller knowing which it sent.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
BerriAI#31324 was failing 3 CI checks (
code-quality,osv-scan,integrations / Run tests) and showing asCONFLICTINGagainst its base branch. None of those failures were caused by this PR's own changes — the branch had simply fallen behindlitellm_oss_staging, which has since:Router._reset_custom_routing_strategy(thecode-qualityrouter-coverage check was failing on 0.40% untested)langgraph-checkpointto4.1.1, fixingGHSA-fjqc-hq36-qh5p(theosv-scanfailure)test_export_window_passes_max_rows_as_limitintest_mavvrik_destination.py(theTypeError: object MagicMock can't be used in 'await' expressionfailure)Comparing
gh pr difffor BerriAI#31324 against currentlitellm_oss_stagingconfirmed the only genuinely new work is 3 files (~525 lines): the ElevenLabs transcription transformation, its new Pydantic types, and its test suite. Everything else in the stale diff (router.py, rerank transformations, UI dashboard, etc.) was base-branch drift.Changes
This branch takes the 4 real commits from
neimaravila:litellm_elevenlabs_diarized_json:feat(elevenlabs): diarized_json transcription and multi-channel speakersfix(elevenlabs): guard missing channel_index and preserve ValueError on parse failurestyle(elevenlabs): use builtin generics to stay within UP006 budgetfeat(elevenlabs): auto-select diarization mechanism by audio channel countand replays them on top of the current
litellm_oss_stagingtip, resolving one small conflict inget_supported_openai_params(adding"response_format"to the existing supported-params list).Verification
Run locally against the rebased tree:
pytest tests/test_litellm/llms/elevenlabs/— 18 passedpython ./tests/code_coverage_tests/router_code_coverage.py—untested_perc: 0.0(was failing before rebase)pytest tests/test_litellm/integrations/focus/test_mavvrik_destination.py— 35 passeduv.lockalready carrieslanggraph-checkpoint==4.1.1ruff checkon the touched files — cleanType
🧹 CI/merge fix (no functional change beyond the rebase)