Repository navigation
Conversation
…ck/mantle/... (BerriAI#31141) The bedrock/mantle chat and messages paths hardcoded the public Mantle host and ignored api_base, so private VPC/VPCE/GovCloud endpoints could not be used. Route URL construction through a shared helper that prefers api_base and aws_bedrock_runtime_endpoint before falling back to the regional public host. Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com>
…erriAI#31133) Re-pins LITELLM_BUILD_IMAGE and LITELLM_RUNTIME_IMAGE across all 6 Dockerfiles from the prior digests (openssl 3.6.2-r3) to the current chainguard wolfi-base digest c61ac691 (openssl 3.6.3-r2, >= the fixed 3.6.3-r0). The runtime stage is the shipped image, so the runtime digest is what actually resolves the customer-facing CVE; the build image is bumped too for hygiene. Two Dockerfiles tracked a second equally-stale digest; both are unified onto the patched one.
…me (2/2) (BerriAI#31135) * add CoreError::Routing variant for deployment selection failures * add minimal Rust Router (simple-shuffle) mirroring router.py spec * add litellm-router crate manifest * add ai-gateway POST /v1/realtime handler calling router.realtime * add ai-gateway health routes * wire ai-gateway routes into the axum app * add ai-gateway AppState holding the shared router * add ai-gateway axum server entrypoint * add litellm-ai-gateway binary crate manifest * docs: add ai-gateway folder-architecture AGENTS.md * register router + ai-gateway crates and axum/rand deps in workspace * update Cargo.lock for router + ai-gateway crates * split router: extract model_list types into deployment module * split router: extract routing policy into strategy module * split router: move Router orchestration into router module * router lib: wire submodules and re-export public API * add read_model_list helper reusing ProxyConfig env/secret resolution * add GIL-activity tracker (records acquisitions, 30s window) * add GET /health/gil endpoint for polling GIL activity * add pyo3 load_router_from_config bridge (feature-gated, load-time only) * register /health/gil route in ai-gateway * wire build_router: load from python config when feature enabled * add optional pyo3 dep + python-config feature to ai-gateway * update Cargo.lock for optional pyo3 dependency * fix: satisfy strict ruff budget (FA100) in read_model_list * test: cover read_model_list env resolution + empty config * ai-gateway: bind localhost by default, warn on bad PORT/missing keys, wire gateway key * ai-gateway: add gateway_key to AppState for realtime auth * ai-gateway: require bearer auth + map unknown model to 404 on /v1/realtime * ai-gateway: move python interop into python/ with load-time-only AGENTS.md * ai-gateway: document auth, gil, and python folder in AGENTS.md * core: add router module (model_list types + simple-shuffle selection) * ai-gateway: dispatch realtime via core router + providers (drop router crate dep) * update Cargo.lock: fold router into core * workspace: drop crates/router member and litellm-router dep * read_model_list: reuse ProxyConfig.get_config (includes + os.environ + DB) instead of thin yaml read * ai-gateway: constant-time bearer compare + 500 (not 503) for unconfigured key * ai-gateway: trim stored gateway key to match trimmed bearer token * ai-gateway: add subtle dep for constant-time comparison * workspace: add subtle dependency * update Cargo.lock for subtle * core router: make strategy a folder (one module per strategy, simple_shuffle) * providers: make realtime() a streaming splice (client stream <-> OpenAI) instead of collect * providers: add futures-channel dev-dep for the streaming live test * ai-gateway: make /v1/realtime a WebSocket (auth before upgrade, splice typed events) * ai-gateway: dispatch realtime as a stream splice * ai-gateway: route /v1/realtime via GET (WebSocket), drop POST * ai-gateway: enable axum ws feature + futures-util * update Cargo.lock for ws feature + futures-channel * core router: add has_deployment() for pre-flight model checks * ai-gateway: extract auth into auth/ module (single master key, LITELLM_MASTER_KEY) * ai-gateway routes: adopt router()-per-module template + merge in app() * ai-gateway: document auth/ + routes template in AGENTS.md * ai-gateway: realtime route as thin handler + service + transport * ai-gateway: auth as a RequireMasterKey extractor (idiomatic axum FromRequestParts) * ai-gateway: docs for auth extractor + simplified route template * ai-gateway: collapse realtime route to mod.rs + service.rs; docs for extractor/template * providers realtime: enforce idle timeout around the splice (reap stalled sessions) * ai-gateway: rename realtime service timeout param to idle_timeout --------- Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
…#31154) * build: make rust ai-gateway Dockerfile config.yaml-based (repo-root context) * build: add dockerignore to shrink repo-root context for ai-gateway * build: add sample realtime config.yaml for rust ai-gateway * build: point ai-gateway render blueprint at config.yaml + repo-root context * docs: document config.yaml as primary path for rust ai-gateway * build: run rust ai-gateway container as non-root user * ci: re-trigger flaky otel fake-openai-endpoint cooldown
…nv-var endpoints (BerriAI#31171) The per-user BYOK, OAuth (OBO), and env-var management endpoints resolved the target MCP server through a DB-only lookup (get_mcp_server / get_all_mcp_servers_for_user). A server defined in config.yaml lives only in the in-memory registry and never gets a row in LiteLLM_MCPServerTable, so those endpoints raised 404 "MCP Server <id> not found" (or 403 for non-admins) before any credential could be stored, leaving config-server users unable to connect and forced to re-authorize forever. Route all three through a single registry-aware resolver: DB first, then the in-memory registry (built into LiteLLM_MCPServerTable via _build_mcp_server_table, the same fallback fetch_mcp_server already uses), then the canonical get_allowed_mcp_servers authorization the MCP gateway enforces on tool calls. Admins get a 404 for an unknown id; non-admins get 403 for a missing-or-forbidden server so server ids stay non-enumerable. This also closes a gap where the two store endpoints performed no per-server authorization at all.
…lishment latency (BerriAI#31163) * docs: realtime pre-warmed connection pool design + raw-passthrough follow-up * docs: realtime pool benchmark, repro steps, and deploy guidance * feat: expose Router::deployments() for host-side upstream enumeration * refactor: split realtime dial/splice and add warm-handoff entry point * feat: pre-warmed upstream realtime connection pool with fresh-dial fallback * feat: add realtime pool handle to gateway AppState * feat: try warm pooled upstream before fresh-dial in realtime service * feat: thread realtime pool through the realtime route bridge * feat: build and pre-warm the realtime pool at gateway startup * build: lean Dockerfile for the realtime gateway (default features, env stand-in) Minimal multi-stage image for load-testing the realtime pool: builds the gateway with default features (no python-config, no libpython), runs on a debian-slim base (~157MB), and reads model_list from the OPENAI_REALTIME_MODEL env stand-in. No config.yaml or pip install needed. Build context is the repo root; only litellm-rust/ is included via the sidecar .dockerignore. * docs: add generic ai-gateway benchmarking skill Teaches an agent how to benchmark any ai-gateway endpoint: deploy the gateway, run one load generator against both provider-direct and the gateway with the same protocol, phase-decompose latency (dial/session/first-token/total), compare at scale, and report success% + p50/p95. Documents the benchmarks/<endpoint>/ layout and the hard no-committed-keys rule. * docs: drop per-endpoint realtime benchmark README The measured results table lives in the PR description (numbers go stale in a committed README). The benchmarks/realtime/ dir now holds only the sanitized load-gen harness; the generic method is in benchmarks/SKILL.md. * test: add sanitized realtime WS load-gen harness for gateway benchmarks Copies the ws-bench Go load generator (main.go, go.mod, go.sum, Dockerfile, run.sh) into benchmarks/realtime/. Measures dial/session/first-audio/total per WebSocket connection against both OpenAI-direct and the gateway. No keys are hardcoded — the bearer token comes from -key / $OPENAI_API_KEY; default host is api.openai.com. * test: add hosted-runner serve.sh wrapper for the realtime harness Render one-off jobs don't surface stdout via the Logs API, so on a hosted runner the long-lived web service runs the leg and PUBLISHES the result: serve.sh decodes the base64 flag list, runs wsbench teeing output to /tmp/web/result.txt, then serves it over HTTP so the result is fetchable at /result.txt. No secrets are written to the served file (the -key is only in wsbench's argv). The Dockerfile now copies both run.sh and serve.sh. * test: make serve.sh publish result atomically and clear stale output Remove any prior result.txt at startup and write the new run to a .partial file that's atomically moved into place only once complete. Prevents a fetcher from reading a previous run's numbers while the current run is still in flight. * perf: refill the realtime pool concurrently so warm supply keeps up at scale The replenisher dialed missing warm sockets sequentially, so a full refill cost needed x handshake (~needed x 350ms). Under high connect rates the pool drained faster than it refilled and ~85% of connects missed (measured: only ~15% pool hits at 5000/500). Firing the dials together with join_all refills in ~one handshake window, keeping warm supply close to peak concurrent connects so the sub-millisecond warm handoff becomes the median rather than the lucky-hit tail. Each warm_one dial is independent (no shared state until the final push under the lock), so concurrent refill is safe. Pool unit tests unchanged and passing. * build: drop Dockerfile.lean The lean load-test image isn't worth carrying in the repo; deploy the gateway however you normally do and set the pool env vars. * docs: slim benchmarks/realtime to a README (harness moved to its own repo) Drop the Go load-gen, Dockerfile, run.sh, serve.sh, and the generic SKILL.md from the repo. The harness now lives at github.com/ishaan-berri/litellm-realtime-bench; benchmarks/realtime/README.md carries the results table and links there for repro. * docs: add realtime route README with pooling design + diagram Pooling is now documented as a section in src/routes/realtime/README.md next to the code (handoff diagram, sizing rule, config, notes) instead of the standalone REALTIME_POOL_DESIGN.md RFC. Update the realtime_pool.rs doc pointer to it. * fix: satisfy clippy manual_flatten on concurrent pool refill Use .into_iter().flatten() instead of an if-let-Ok in the for loop over the join_all results, and let rustfmt wrap it. Clears the CI clippy -D warnings failure; fmt + clippy + cargo test all green locally. --------- Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
I notice Claude applies the rule irregularly. This aims to fix that
…I#31152) * fix(anthropic): drop unsupported speed param with drop_params Anthropic fast mode (speed) is Opus 4.6/4.7/4.8 on the direct API only. Strip speed when the model map lacks supports_speed and drop_params is set, for both chat completions and /v1/messages passthrough. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ci): allow supports_speed in model map schema The new supports_speed flag on Opus entries must pass JSON schema validation in test_aaamodel_prices_and_context_window_json_is_valid. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(review): raise on unsupported speed without drop_params Passthrough /v1/messages now raises UnsupportedParamsError when speed is unsupported and drop_params is false. Emit drop warning from map_openai_params when speed is silently skipped. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(anthropic): gate speed param by routed provider, not just model id Vertex, Azure, and Bedrock reuse the shared Anthropic transform and strip their provider prefix first, so a bare `claude-opus-4-8` resolved to the direct-API model-map entry (`supports_speed: true`) and forwarded `speed` upstream, producing the same 400 that drop_params is meant to prevent. Gate fast mode on `custom_llm_provider == "anthropic"` so it stays on the direct Anthropic API across both the chat completions and `/v1/messages` passthrough paths, and collapse the duplicated drop/raise logic in map_openai_params into the shared `_maybe_drop_speed_param` helper. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
…erriAI#31153) A user provisioned with "All Proxy Models" stores the literal "all-proxy-models" sentinel in user.models. get_direct_access_models looked that string up as a real model_name via get_model_list, which matched no deployment, so /v2/model/info marked every model direct_access=false and the Models + Endpoints page rendered empty for such users when they have no teams. The model dropdown / Playground worked because get_key_models already expands the sentinel to the full proxy model list, hence the inconsistency in the report. Expand the sentinel to all non-team deployment ids via get_model_ids(exclude_team_models=True), the same call the PROXY_ADMIN branch in the caller already uses. This fixes both /v1/model/info and /v2/model/info since they share _populate_team_access_on_models. Empty user.models stays "no direct access" to match get_key_models semantics. Fixes BerriAI#22791
Deleting every budget window from a virtual key looked like it saved but reverted on reload, while editing a window persisted. The key edit form set budget_limits to undefined once the window list was emptied, and JSON.stringify drops undefined keys, so /key/update received no budget_limits field at all and model_dump(exclude_unset=True) skipped the existing clear-on-empty branch. Sending [] instead lets the backend store JSON null and clear the stored windows, matching how it already treats an explicit empty list Resolves LIT-3742
…#31157) * feat(ui): track frontend lint counts in a committed snapshot Persist the eslint budget-rule counts (no-explicit-any, complexity, max-depth) to eslint-metrics.json so the trend is queryable straight from git history and can later feed a dashboard. A CI drift check regenerated from the same lint report keeps the snapshot honest, so a PR that shifts a count has to run npm run lint:metrics and commit it * fix(ui): harden lint-metrics drift check and eslint failure handling Make the drift comparison symmetric over the union of committed and actual keys so a phantom rule left in eslint-metrics.json (for example after a rule is dropped from eslint-budgets.json) is caught instead of silently passing. Only swallow eslint's lint-errors exit code in the generator and rethrow anything else, so a fatal eslint failure surfaces its real output rather than a confusing ENOENT on the missing report
…rict 3-crate layers) (BerriAI#31218) * refactor(litellm-rust): move provider transforms into litellm-core + crate allowlist test * feat(litellm-rust): ai-gateway absorbs route I/O (io/) with lib+server feature split * refactor(litellm-rust): point python-bridge at litellm-ai-gateway * build(litellm-rust): macOS pyo3 dynamic_lookup linker flag for cdylib builds * docs(litellm-rust): 3-crate map in README/AGENTS + refresh CLAUDE boundary * refactor(litellm-rust): update workspace members to the three crates
…#31036) * fix(vertex/files): stream OpenAI->Vertex batch JSONL uploads to fix OOM on large files Large (1GB+) batch JSONL uploads to Vertex AI / GCS caused OOM or killed the worker because the request body was buffered and multiplied 2-3x in size. The create-file path is now streaming end-to-end: transform_create_file_request returns a ResumableChunkedUploadConfig carrying a lazy _OpenAIToVertexBatchUploadStream, and the HTTP handler opens a GCS resumable session and PUTs the body in bounded 8 MiB chunks (Content-Range, 308 between chunks) so the transformed payload is never held in full. The proxy /v1/files endpoint streams from Starlette's spooled upload handle instead of reading the whole body, and batch rate limiting counts tokens and models in a single streaming pass. Only gcs_bucket_name is supported for the GCS target; the legacy bucket_name key is intentionally not read. Also removes the unreachable VertexAIFilesHandler create path and everything only it kept alive (VertexAIJsonlFilesTransformation, _stream_openai_jsonl_to_vertex, the legacy transform helpers), plus the orphaned batch_utils helpers the streaming rewrite replaced. * fix(batches): return original JSONL on unparseable row to avoid silent batch truncation The streaming rewrite of replace_model_in_jsonl accumulated physical lines and skipped a row on JSONDecodeError to support multi-line objects, but a genuinely malformed or truncated row never completes: it poisons the buffer, swallows every following row, and the function still returned the partial rewrite (the rows before the bad one, already model-rewritten) as if the batch were complete. That turned the pre-rewrite behavior of returning the original file unchanged (so the provider rejects the bad batch loudly) into a silent partial submission. Restore the original-content fallback: when an unparseable remainder is left after the loop, return the original file_content (rewinding a consumed seekable source) instead of the truncated output. The multi-line happy path is unchanged. * test(batches): mock resumable GCS upload in vertex batch prediction test The vertex batch file-create path now streams to a GCS resumable session via _aresumable_chunked_upload (httpx send) instead of AsyncHTTPHandler.post, so the existing test's post mock no longer intercepted the upload and a real request hit GCS (401). Mock _aresumable_chunked_upload to return the GCS object response; the resumable protocol itself is covered in test_vertex_ai_files_streaming.py. * fix(batches): resilient per-row token accounting; no hard-block on count failure The batch input-file pass iterated a generator whose json.loads raised on a malformed line; the outer except caught it and stopped the loop, so any body.model on rows after a bad line was never collected and the model allowlist check ran against a partial set. It also hard-blocked the batch with a 400 whenever token counting raised, a backwards-incompatible change from the prior swallow-and-proceed behavior that breaks legitimate rows the token counter cannot measure (e.g. some multimodal content). Iterate the JSONL line-by-line and account each row independently. A malformed line is skipped (its request cannot run upstream anyway) and a row the counter cannot measure falls back to a conservative size-based estimate. The loop never aborts, so the allowlist check always sees every parseable model, and the token total is never zeroed, so a crafted uncountable row still cannot evade the TPM limit, without hard-rejecting a legitimate batch. * perf(vertex/files): unblock async upload; drop empty finalize; widen batch MIME types Three review follow-ups on the resumable batch upload: - _aresumable_chunked_upload pulled chunks from a synchronous generator that runs the per-row transform inline on the event loop thread, blocking other requests between PUTs on large uploads. Each chunk is now produced via asyncio.to_thread. - _iter_resumable_chunks no longer yields a trailing empty chunk, so an exactly chunk-aligned upload finalizes on its last data chunk instead of an extra zero-byte PUT; a 0-byte stream still finalizes via the caller's empty request. - valid_content_type now accepts the MIME types clients label .jsonl batch uploads with (text/plain, application/json, ndjson, ...), so such a batch file no longer silently bypasses the streaming path into the buffered media upload. * fix(vertex/files): keep legacy bucket_name as GCS bucket fallback The rename to gcs_bucket_name dropped the legacy bucket_name key entirely, so an SDK caller passing bucket_name to a Vertex AI file create/retrieve/content call with GCS_BUCKET_NAME unset got ValueError("GCS bucket_name is required") where it previously resolved the bucket. _get_configured_bucket_name now reads gcs_bucket_name, then bucket_name, then the env var, and bucket_name is restored to OPTIONAL_KWARGS_KEYS so it survives get_litellm_params on the retrieve and content paths. gcs_bucket_name keeps precedence when both are present * style: sort imports in llm_http_handler to satisfy I001 budget --------- Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
…AI#31126) Creating or updating a key with a specific (non-allow_all_keys) MCP server or access group failed with a 403 when the key had no team: Key is not in a team. Only globally available (allow_all_keys) MCP servers can be assigned validate_key_mcp_servers_against_team computed the allowed set as team servers + allow_all_keys servers. For a teamless key the team set is empty, so the allowed set collapsed to just allow_all_keys servers and any explicitly-picked server or access group was rejected. This was asymmetric with runtime: get_allowed_mcp_servers honors a teamless key's own object_permission.mcp_servers verbatim, with no team gate and no allow_all_keys filter. So the create/update path refused to persist a grant the run path would have served. Thread is_proxy_admin into the validator from both call sites (/key/generate and /key/update). When a key has no team and the caller is a proxy admin, the requested servers and access groups are folded into the allowed set so the existing subset checks pass. A proxy admin can already reach every MCP server, so there is nothing to escalate. Non-admins and every team-scoped key are unchanged. Resolves LIT-3815
…static family) (BerriAI#31058) * feat(mcp): add v1 bridge + none/api_key resolver arms (unwired) PR4a of the MCP v2 outbound-credential migration, stacked on the resolver skeleton. Builds the bridge for the first live modes without wiring it onto the request path: - resolver.py: the none arm (NoOpAuth) and the api_key shared-key arm (StaticHeaderAuth from the config); the BYOK source and the other five arms stay not_implemented. - adapter.py: the v1 <-> v2 edge (to_subject, to_server_spec, raise_public, should_defer). to_server_spec maps only none + the static-header family and returns None to defer every other mode to v1. Imports v1, kept out of the package __init__ so the resolver core stays v1-free. - MCPClient gains an optional resolved_auth that feeds the factory's auth= slot, taking precedence over the SigV4 aws_auth; default None keeps current behavior. Nothing calls these from _create_mcp_client yet, so production behavior is unchanged; the graft lands in PR4b. Unit tests cover the two arms, the full mapping table, and the auth plumbing. * feat(mcp): graft v2 resolver onto _create_mcp_client for migrated modes Wire the none + api_key static-family resolver arms from PR4a onto v1's live request path. In _create_mcp_client's HTTP/SSE branch, to_server_spec decides per mode: a migrated mode resolves through the injected UpstreamCredentialProvider and feeds the resulting httpx.Auth into the new resolved_auth slot; every other mode returns None and falls through to the unchanged v1 construction. resolve_mcp_auth now runs only when the mode defers, so a migrated server skips the v1 token-exchange / M2M I/O. stdio is untouched: auth_type/auth_value never reach the upstream on the stdio path (_get_auth_headers is HTTP/SSE only), so there is nothing to graft there. No v1 code is deleted yet; resolve_mcp_auth's static return still backs stdio and the not-yet-migrated modes until later PRs retire it. * test(mcp): cover the v2-resolver graft in _create_mcp_client Regression tests for the PR4 graft. Migrated HTTP modes resolve through the provider into resolved_auth: none -> NoOpAuth, and the static api_key family emits the right header per scheme (X-API-Key, Bearer, token, raw authorization, base64 basic). Deferred modes (oauth2) and a missing static token fall back to v1's auth_value. A stdio server with a migrated auth_type still defers to v1, since httpx.Auth never reaches the subprocess. A resolver Error is mapped to the public HTTP contract (401) via an injected provider, exercising the DI seam. * fix(mcp): defer to v1 when an inbound credential would be overridden The graft attaches the resolved static credential as an httpx.Auth, whose auth flow writes its header after extra_headers. That silently overrode an inbound Authorization: a per-request mcp_auth_header override, or a header supplied via a guardrail hook / static_headers / forwarded caller header. v1 lets those win, so the graft had inverted the credential precedence for the migrated static modes. Mirror the v2 egress credential-isolation invariant: defer the request to v1 when mcp_auth_header is set, or when the header the resolved credential would write is already present in extra_headers. none writes no header, so it never defers. * test(mcp): cover the credential-isolation defer guard Regression tests for the precedence fix. A per-request mcp_auth_header override and an Authorization already present in extra_headers (guardrail hook like the JWT signer, static_headers, or a forwarded caller header) both defer a migrated static server to v1 so the inbound credential wins; none stays on v2 and does not clobber an inbound Authorization since NoOpAuth writes nothing. The deferred cases assert resolved_auth is None, which fails if the guard is removed. * refactor(mcp): resolve inbound-header conflict on v2 instead of deferring For an Authorization already supplied via extra_headers (a guardrail hook such as the JWT signer, static_headers, or a forwarded caller header), keep the request on the v2 path and skip resolved_auth rather than deferring to v1. The inbound header still wins since nothing overwrites it, but hooks no longer pin a v1 fallback, which is what lets resolve_mcp_auth be retired once the remaining modes migrate. The mcp_auth_header per-request override still defers to v1, since that value becomes the upstream credential rather than sitting in extra_headers; that defer falls away once the per-user modes stop writing mcp_auth_header. * fix(mcp): clear UP037 lint gate and fix allowed-servers test under the graft adapter.py uses `from __future__ import annotations`, so the quoted "UserAPIKeyAuth" / "MCPServer" annotations in to_subject/to_server_spec/_shared_key_spec were unnecessary and pushed UP037 over the strict-rule budget; drop the quotes. test_list_tools_only_returns_allowed_servers passed a MagicMock as user_api_key_auth. The graft now builds a Subject from the principal, and the MagicMock's non-string org_id/user_id fail Subject validation, so the listing came back empty. Use a real UserAPIKeyAuth instead (MagicMock for an injected dependency was the anti-pattern here). * test(mcp): assert config token via resolved_auth, not the headers dict test_mcp_server_config_auth_value_header_used inspected _get_auth_headers(), but the graft now carries the static credential on the client's httpx.Auth (resolved_auth) and writes the header at send time, so that dict is empty. Assert the header the StaticHeaderAuth emits onto the request instead. Both config keys (authentication_token, auth_value) stay covered. * chore(typecheck): set reportMatchNotExhaustive slack to 0 The previous slack of 3 put the ceiling at baseline + slack = 4, so a newly non-exhaustive match (for instance dropping an Error arm off a Result match) could land without tripping the gate. Setting slack to 0 pins the ceiling at the current baseline of 1, so any added non-exhaustive match now fails CI while the one pre-existing violation in router.py stays within budget
* tests: add e2e tests for spend, budgets and llms * style: make chained comparison of status_code clearer Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * remove e2e_tests folder * test: add spend tracking tests * test: multi-window budgets coverage * fix: p0 issues, added types and shared functions for each test suite * chore: add config.yml * test: passthrough endpoints stream/non-stream e2e * style: carry clearer status_code comparison into renamed e2e dir * fix: rename cost breakdown function * fix: pydantic validation for budget info, dont allow explicit type cast * refactor: migrate to gateway client * test: add custom pricing tests * chore: change master key * test(e2e): address greptile review feedback Remove the duplicate cache/cache_params block in the gateway config so the two can't silently diverge under future edits. Reorder the soft-budget test to assert the call isn't a budget block before require_successful_call, since that helper hard-fails any non-2xx and left the budget-block check unreachable; the misleading "skip" comment is corrected. Add a deferred delete in test_budget_delete_removes_it so a failed delete doesn't leak a budget on the shared proxy. Scope the spend_tracking sys.path insertion in pytest_sessionfinish to just the cleanup import so a broader "pytest tests/" run isn't left with a mutated path. * test(e2e): drop misleading skip comment on require_successful_call require_successful_call fails hard, it does not skip; the trailing comment was factually wrong. The function name already states intent, so the comment is removed in both per-model and tag budget helpers. * test(e2e): assert budget-isolation invariant before success check On the should-still-succeed path of the per-model and tag isolation tests, check is_budget_block before require_successful_call. If the isolation bug fires the unaffected model/tag is blocked, so asserting the specific 'blocked by X' invariant first yields the diagnostic message instead of a generic upstream-failure. Matches the ordering in test_soft_budget_e2e.py. * fix(e2e): guard spend-log truncate on skip and stop returning unrelated priced rows * fix(e2e): run case init() inside try so partial-init failures tear down run_case called case.init() outside the try/finally that runs teardown(), so a case that registers cleanups progressively (create team, then user, then key) and then fails partway through init() would leak the already-created entities on the long-lived shared proxy. Move init() inside the try so teardown always runs. Add a regression test that registers a cleanup then raises mid-init and asserts the resource is still released. * test(e2e): mark known pricing-leak isolation test xfail(strict) test_custom_pricing_is_isolated_from_sibling_deployment documents a real proxy gap (a deployment's custom per-token pricing leaks into the shared cost map for sibling deployments of the same underlying model) and was left unconditionally failing, which pollutes the suite's pass/fail signal. Mark it xfail(strict=True) so the suite stays green while the leak persists and turns into a failure the moment isolation is fixed, prompting the marker's removal. * refactor(e2e): make suite pass its shipped strict basedpyright config The suite ships tests/pyrightconfig.json (strict, no Any), but basedpyright --project tests reported four errors in it: three reportAny on the parametrize ids=lambda c: c.__name__, and one reportUnusedFunction on the underscore-prefixed autouse fixture _require_live_proxy. Replace the untyped lambda with a typed _case_id(case_cls: Type[_BudgetCase]) -> str so the ids are no longer Any, and rename the fixture to require_live_proxy so basedpyright no longer treats it as an unused private function (it is referenced only by pytest's autouse machinery). basedpyright --project tests now reports zero errors. * fix(tests/e2e): gate spend-log truncate on e2e marker, not test directory * test(e2e): run harness unit tests without a live proxy The autouse session fixture skipped the whole tests/e2e session when no proxy answered, which also skipped test_lifecycle.py, a pure unit test of run_case that never touches the proxy. A regression test that silently skips gives no signal, so the skip now lives in pytest_runtest_setup gated on the same e2e marker the spend-log truncate guard already uses: live tests skip when no proxy is up while harness unit coverage always runs. The liveness probe is cached with lru_cache so it still runs once per session * test(e2e): clean up gateway config comment debris Fix the typo on the header comment and drop the orphaned namespace/ttl comment remnants left indented under cache_params; the active values are already set above. Flagged by greptile review. * fix: add new tests, split gateway * test(e2e): type the redis spend-counter probe for strict basedpyright The new cold-counter reseed test drove its redis client untyped, so the strict tests/pyrightconfig.json (reportUnknown*, reportAny) flagged ten errors once the file landed: scan_iter/get came back unknown and the pool.map lambda had an untyped parameter. Annotate the client as redis.Redis[str] via a TYPE_CHECKING import (the runtime import stays lazy so the suite still skips, not errors, when redis is absent), which resolves scan_iter to Iterator[str] and get to str | None, and replace the lambda with a typed inner function mirroring _burst. basedpyright --project tests is back to zero errors. * test(e2e): xfail the known team multi-window failure and isolate member teardown Greptile flagged two issues in the mirrored split-gateway commit. The team multi-window budget test documents a real /team/new write bug (budget_limits go straight to the Json? column and Prisma 500s, unlike the json.dumps'd key and /team/update paths) and was left as an unconditional hard failure, which would turn any live-proxy CI run red; mark it xfail(strict=True) like the custom-pricing isolation test so the suite stays green while the bug persists and flips to a failure the moment the write is fixed and the marker should go. The class-scoped member fixture in test_team_member_budget_e2e.py tore down its key, user, and team sequentially with no exception isolation, so a failed delete_key would strand the user and team on the long-lived shared proxy. Route cleanup through a ResourceManager: register each delete progressively and run them LIFO best-effort in a finally, so a partial-setup failure still releases what came before and one failed delete never blocks the rest. * test(e2e): set fast budget-reset cadence in gateway config so staging windows reset within e2e timeouts * test(e2e): surface real /spend/tags errors instead of masking them as missing tags The spend-tracking e2e client swallowed every non-200 from /spend/tags into an empty list, so a real server error or a response-shape mismatch showed up only as the generic "tag never appeared in /spend/tags" with no diagnostics. That masking is what made the original cluster failure undiagnosable. spend_by_tags now raises SpendTagsError carrying the actual HTTP status and body for any non-Success result, and poll_tag_spend fails fast on a hard server error rather than polling it into a timeout; eventual consistency only manifests as a 200 whose payload does not yet carry the tag, so only that case waits. The tag test now reports the last observed status and asserts the endpoint returned 200 at least once, with no weakened assertions. Hardening surfaced the real defect in the test itself: /spend/tags returns a top-level JSON array (List[LiteLLM_SpendLogs]), but the client validated against a SpendTagsResponse dict wrapper that never matched, so every call fell through to the empty-list mask. Wired spend_by_tags to the existing TagSpends RootModel and removed the dead SpendTagsResponse model. Verified against the real Postgres that request_tags is stored as proper JSONB arrays and /spend/tags aggregates them correctly, so there is no encoding bug to fix here. * test(e2e): drop flaky test_tag_spend_matches_sum_of_tagged_logs The test wrote tagged requests and polled /spend/tags expecting read-after-write consistency. /spend/tags itself is fine; verified live that request_tags is stored as a JSON array and the endpoint reflects a fresh tag within seconds, so the failures were a timing flake under full-suite load rather than a real defect. Coverage is retained by test_request_tags_round_trip (tags persist onto the row) and the /spend/tags route probe in test_spend_routes.py. Also remove the now-dead tag-spend scaffolding this test was the only user of: poll_tag_spend, spend_by_tags, TagSpendPoll, SpendTagsError, the TagSpend/TagSpends models, and their imports. * test(e2e): widen budget-reset wait windows to de-flake wall-clock-aligned resets The short-window reset tests asserted the reset landed within WINDOW_SECONDS + 45 (~75s), but the 30s budget window is wall-clock-aligned, so the reset can land up to a full window after start, then the rescheduler (~15-20s) zeroes the spend, plus poll and DB lag. A real run measured 84s, just over the 75s bound, and which of the short-window siblings tripped flipped run to run. Widen the wait loops to 150s and the elapsed assertions to WINDOW_SECONDS + 90 (120s for the key test). A genuinely stuck rescheduler is still caught by the wait-loop timeout, so this only removes the timing flake, not the regression signal. * test(e2e): let the spend-counter reseed test reach a cluster-mode TLS redis The test's _redis() built a standalone, non-TLS client on the docker-compose defaults (localhost:6380), so against the EKS serverless ElastiCache (cluster-mode + TLS) it could never connect and the test skipped. Honor E2E_REDIS_SSL and E2E_REDIS_CLUSTER so it builds a TLS RedisCluster client when the deploy provides them, and E2E_REDIS_NAMESPACE so the counter is read with a direct GET (cluster-safe) rather than a keyspace scan that can't span shards. The local standalone path and the graceful skip-on-unreachable behavior are unchanged. * test(e2e): take the direct-GET spend-counter path on E2E_REDIS_CLUSTER The gateway's cache sets no namespace, so the counter key is the bare spend:key:<hash>. Trigger the cluster-safe direct GET on E2E_REDIS_CLUSTER (not only on E2E_REDIS_NAMESPACE) so the cluster deploy need not set a namespace it does not use; the namespaced key is still tried first when a namespace is given. * test(e2e): use REDIS_HOST/REDIS_PORT and drop the unused redis knobs The runner is a standalone test pod, so the proxy's own REDIS_HOST/REDIS_PORT names are unambiguous - no E2E_ prefix needed. The only deployed redis it talks to is the serverless ElastiCache (always TLS + cluster), so that is inferred from REDIS_HOST being set rather than carried as ssl/cluster knobs. Stage sets no cache namespace (bare counter key, read directly on the cluster) and is passwordless, so the namespace and password env are gone; the local namespace is still handled by the standalone SCAN. * test(e2e): replace the vacuous failure-row test with per-model attribution test_failure_call_writes_failure_status_row had two skip hatches (the call did not fail, or no failure row landed) and never asserted anything on this proxy - gemini accepts an empty message (HTTP 200), and live failure-row logging is non-deterministic across providers. Replace it with a deterministic check: one key calling gemini-2.5-flash and claude-haiku-4-5 gets one spend row per call, each carrying its own model and a nonzero cost, under distinct request_ids that match the call's response id. Verified live on stage (gemini/gemini-2.5-flash $0.00053, anthropic/claude-haiku-4-5 $0.000038, distinct ids matching the responses). Failure-status row construction stays covered by the unit suite. * test(e2e): assert /spend/logs returns the key's spend without 5xx Regression for the intermittent 500s on /spend/logs (DB query / serialization errors under load). The existing spend_logs() helper swallows non-success responses into an empty list, so a 500 looks identical to 'rows not flushed yet'. This test queries the endpoint directly and asserts a Success response on every poll, failing loudly on any 5xx, then requires the call's nonzero spend to surface. --------- Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com>
…hrough callbacks (BerriAI#31134) * feat(proxy): add logging_endpoints package init * feat(proxy): add POST /v1/callbacks/logs to replay logging payloads through the success/failure callback fan-out * feat(proxy): register callback_logs_router * test(proxy): add logging_endpoints test package init * test(proxy): cover /v1/callbacks/logs replay, admin guard, and partial-failure handling * refactor(proxy): move callback-logs request/response models to litellm/types/proxy * refactor(proxy): wrap callback-logs replay in CallbackLogsReplayer class with payload logging * test(proxy): update callback-logs tests for class-based replayer and separated types * fix(proxy): cover /v1/callbacks/ in backend component allowlist The new /v1/callbacks/logs route was dropped by both component allowlists, failing test_gateway_plus_backend_covers_full_app. It's an admin-only spend-logging route, so it belongs on the backend (control plane) alongside the existing /callbacks family. * refactor(proxy): use builtin dict/list generics in callback-logs endpoint Switch Dict/List from typing to builtin dict/list to satisfy the ruff strict-rule budget (UP006). * refactor(proxy): use builtin dict/list generics in callback-logs types UP006: builtin generics over typing.Dict/List. * chore(ui): regenerate schema.d.ts for /v1/callbacks/logs Run npm run gen:api to add the CallbackLogRecord/CallbackLogsRequest/ CallbackLogsResponse types and the /v1/callbacks/logs path, keeping the dashboard types in sync with the proxy OpenAPI spec. * fix(proxy): force stream=False when replaying callback logs A replayed StandardLoggingPayload is a terminal, fully-aggregated event — the producer (e.g. the rust realtime gateway) already collected the whole session before POSTing. Marking the rebuilt Logging object as streaming made async_success_handler wait for a complete_streaming_response that never arrives, so the spend log was never written. Realtime sessions now land in LiteLLM_SpendLogs. * feat(litellm-rust): CustomLogger callback layer posting to /v1/callbacks/logs integrations/ mirrors litellm/integrations/: a sync, typed CustomLogger trait (base contract), a typed StandardLoggingPayload, and LiteLLMPythonProxyAPILogger — the first concrete logger, owning a bounded channel + background worker that batches and POSTs to the Python proxy's /v1/callbacks/logs. * feat(litellm-rust): RealTimeStreaming per-session log collector 1:1 with Python's RealTimeStreaming: observe() accumulates O(1) usage/model/id per event (never buffers frames); log_messages() builds one StandardLoggingPayload on session close and fans out to the CustomLogger callbacks. request_id == the OpenAI realtime session id (sess_…), with the gateway id as fallback. * feat(litellm-rust): wire realtime logging into the splice (lock-free observe) The collector is owned on the splice task and observed via a synchronous &mut callback threaded through providers::realtime::realtime() — no Arc/Mutex/atomic on the per-frame hot path. On session close the bridge flushes one payload. AppState carries the registered loggers; main spawns the proxy logger. * docs(litellm-rust): ai-gateway realtime logging architecture * docs(litellm-rust): document request-log egress to the LiteLLM control plane Add a 'Request logging' guide to the ai-gateway README: how to point the gateway at a LiteLLM proxy via LITELLM_PROXY_BASE_URL (+ LITELLM_MASTER_KEY for the admin-only /v1/callbacks/logs POST), and the non-blocking / one-payload-per-session behavior. * feat(litellm-rust): make log-egress tunables env-overridable Channel capacity, batch size, and flush interval now read from LITELLM_LOG_CHANNEL_CAPACITY / LITELLM_LOG_BATCH_SIZE / LITELLM_LOG_FLUSH_INTERVAL_MS, falling back to the DEFAULT_* consts on missing/invalid/non-positive values. Grouped behind an EgressTunables::from_env() read once at logger construction. * docs(litellm-rust): document log-egress tuning env vars * docs(litellm-rust): require constants in a crate-level constants.rs Mirror of Python's litellm/constants.py rule — magic numbers and fixed strings go in src/constants.rs, not inline in feature modules; env-overridable tunables keep their DEFAULT_* value there. * refactor(litellm-rust): move ai-gateway constants into constants.rs Per the new rule: the log-egress defaults (proxy base, ingest path, channel capacity, batch size, flush interval) and the realtime provider default move to crates/ai-gateway/src/constants.rs; modules import from it. * ci: run logging_endpoints tests in the proxy-infra coverage shard tests/test_litellm/proxy/logging_endpoints wasn't in any coverage-uploading job, so callback_logs_endpoints.py showed only import-level coverage (~35%) on codecov/patch despite being ~98% covered locally. Add it to proxy-infra's test-path so the test is exercised under --cov. * fix(litellm-rust): hash the master key before logging — never send the raw credential Greptile/Veria P1: user_api_key_hash was the plaintext LITELLM_MASTER_KEY, which fans out to spend logs and every callback (Langfuse/Datadog) and could be recovered from logs. SHA-256 it (auth::hash_token, matching the proxy's hash_token); the field is named *_hash and the proxy stores it verbatim when it isn't sk-prefixed, so the DB value is identical with zero plaintext exposure. * fix(litellm-rust): observe realtime logging on upstream events only Greptile P1: observe ran on the client->upstream arm too, so an authenticated client could send a fabricated response.done and inflate its own spend log. session.created/response.done are server->client events; observe the upstream arm only. * feat(proxy): bound callback-logs batch + return per-record failures Greptile P2: cap /v1/callbacks/logs at MAX_CALLBACK_LOG_RECORDS (default 1000, env-overridable) so one POST can't trigger an unbounded callback/DB fan-out; and return per-record {index, error} failures so a caller (the rust gateway) can distinguish a transient callback error from a structurally bad payload. * chore(ui): regenerate schema.d.ts for CallbackLogFailure / failures field * fix(constants): make MAX_CALLBACK_LOG_RECORDS a plain constant It doesn't need to be env-configurable (only the rust egress tunables are). As an os.getenv var it tripped tests/documentation_tests/test_env_keys.py, which requires every env key to be documented in the (separate-repo) config_settings.md. Plain constant → not scanned → code-quality + documentation checks pass. * docs(litellm-rust): trim ai-gateway ARCHITECTURE.md to one diagram + notes * docs(litellm-rust): tighten the README request-logging section * docs(litellm-rust): ARCHITECTURE.md is just the diagram (gateway = inference, spend = callback) * docs(litellm-rust): drop em-dashes from the request-logging section --------- Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
The App Router migration moved pages to deeper path segments and the proxy can be mounted under a sub-path (e.g. /litellm behind a reverse proxy). Local logo asset paths were emitted without the server root prefix, so they resolved off the origin root and 404'd. Route every local logo src through a single resolver that prefixes the live server root path and leaves external URLs untouched, fixing provider, guardrail, vector store, callback, MCP and audit-log logos at any route depth and root path.
…custom-pricing-leak bugs (BerriAI#31249) Both tests were xfail(strict=True) for known proxy bugs: /team/new writing budget_limits as a raw list (Prisma 500) and custom per-token pricing leaking into the shared cost map for sibling deployments. Both are fixed, so the tests pass and strict mode reports the unexpected pass as a failure. Remove the markers (as their reasons instructed) so they run as plain regression guards; docstrings updated to describe the regression each now pins.
…et_config (BerriAI#31117) * fix(proxy): stop double-decrypting email/slack alerting env vars in get_config proxy_config.get_config() already returns environment_variables decrypted (the DB overlay decrypts them in _update_config_fields, and YAML values are plaintext), so the /get/config/callbacks slack and email blocks were running decrypt_value_helper() a second time on plaintext. That second decrypt always failed and the helper swallowed the error and returned None, so every SMTP_* field came back blank when the Admin UI reloaded the email settings, and the proxy logged a misleading "Did your master_key/salt key change recently?" error even when nothing changed. Consume the already-decrypted values directly, matching process_callback's handling of the same dict for langfuse/datadog/etc. Sensitive-value masking is preserved. Fixes BerriAI#19221 * fix(proxy): preserve a cleared slack webhook instead of falling back to OS env Use an explicit is-not-None guard rather than truthiness when deciding whether to fall back to os.getenv for SLACK_WEBHOOK_URL. With `or`, a webhook the admin cleared (stored as "") is falsy and would surface a stale SLACK_WEBHOOK_URL from the OS environment; only a truly absent key should trigger the OS lookup. No decryption is reintroduced.
…control (BerriAI#31264) * fix(mcp): correct misleading no-trusted-proxy warning for XFF access control * test(mcp): assert the no-trusted-ranges warning was logged instead of relying on StopIteration
…rded_for is off (BerriAI#31266) * fix(mcp): warn loudly when X-Forwarded-For is present but use_x_forwarded_for is off When a request carries an X-Forwarded-For header but use_x_forwarded_for is unset, get_mcp_client_ip silently falls back to the direct peer's IP (the load balancer / reverse proxy). That peer almost always sits inside mcp_internal_ip_ranges, so the 'Internal network only' (available_on_public_internet: false) restriction trusts every external caller as internal and effectively exposes those servers. Emit a one-shot loud error pointing the operator at use_x_forwarded_for instead of hard-failing: on a deployment with no load balancer, a crafted X-Forwarded-For header must not be able to take the service down, and a one-shot log keeps a flood of crafted headers from spamming the logs. * fix(mcp): re-arm XFF-disabled warning on config change and harden test assertion Address PR review: tie the one-shot warning flag to the observed use_x_forwarded_for value so it re-arms whenever the setting is seen enabled, restoring the diagnostic on a later rollback to disabled. Also assert against str(call_args) so the test survives a positional-to-keyword logger refactor.
…#31254) * fix(mcp): resolve toolset tools by the server's known prefix Toolsets store {server_id, bare tool_name} and reconcile that against the live prefixed tool name at list time. The reconciliation chopped the live name at the first MCP_TOOL_PREFIX_SEPARATOR with no server context, so a server whose prefix contains the separator (a hyphenated alias, or the UUID server_id used as the prefix when a server has no alias) had its tools silently dropped from /toolset/<name>/mcp while listing fine everywhere else. Strip the exact known prefix for the tool's server_id instead of guessing the boundary, on both the resolve and filter sides Also render toolset tools as {server-prefix}-{tool} in the dashboard picker result and chips; this is display only, the persisted record stays {server_id, bare tool_name} Resolves LIT-3419 * test(mcp): add focused unit tests for strip_known_server_prefix Cover the LIT-3419 cases directly on the helper with real MCPServer objects: clean prefix round-trip, hyphenated alias, UUID server_id fallback, unprefixed passthrough, and the server=None legacy fallback
…e_metadata (BerriAI#31255) An oauth2 MCP server with delegate_auth_to_upstream=true never prompted the user to sign in. On an unauthenticated initialize the gateway answered locally (200, no tools) and emitted no WWW-Authenticate, so clients like Claude Desktop either connected empty or hit "OAuth probe timeout after 10000ms". BerriAI#30124 added a bare `continue` in _raise_preemptive_401_for_unauthenticated_servers to stop sending LiteLLM's gateway authorization_uri challenge for delegate-auth servers, expecting the upstream to emit its own challenge. On initialize the gateway never probes upstream, so no challenge ever reached the client. Replace the `continue` with a preemptive 401 carrying the proxied resource_metadata (RFC 9728) challenge, the same form passthrough servers and MCPUpstreamAuthError already use. This keeps BerriAI#29770 fixed (still no authorization_uri) while restoring the upstream PKCE sign-in prompt.
…iAI#31270) * fix(ci): point OSS contributor workflows to litellm_oss_staging Workflow triggers and guard error messages incorrectly referenced litellm_oss_branch; update them to the branch we actually use for external contributions. * fix(ci): include test-rust.yml in litellm_oss_staging rename Missed test-rust.yml when updating OSS contributor target branch references. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…erriAI#30216) * fix(streaming): word-sliced cache replay for stream=true cache hits * fix(streaming): align mypy and replay happy-path test with word-sliced cache replay * fix(streaming): short-circuit whitespace-only content in cache replay splitter * fix(streaming): emit tool_calls/function_call only on first replay slice * refactor(streaming): drop dead delattr guard in cache replay A non-None usage on the replay base object always lives in __pydantic_extra__ (it is attached via setattr earlier in the same function), so delattr can never raise here; the try/except AttributeError that silently swallowed a failure was dead defensive code that could only ever hide a real regression, so it is removed in both the async and sync generators. Also switches the new replay annotations from typing.List to the builtin list to satisfy the strict ruff UP006 gate and drops the unused PLR0915 noqa directives (the rule is not enabled in this repo's ruff config, so RUF100 flagged them). * fix(streaming): drop carried-over metadata from later cache replay slices The word-sliced cache replay deep-copies the full ModelResponseStream per slice, so reasoning_content, thinking_blocks, logprobs, enhancements, annotations and the rest of the per-message metadata rode on every slice, not just the first. Downstream handlers that accumulate streamed deltas would collect each one once per slice, e.g. duplicating a cached reasoning trace N times on a stream=true cache hit. Later slices are now rebuilt as a content-only delta with choice-level logprobs and enhancements stripped, so the whole metadata class stays on the first slice. Adds async (logprobs) and sync (reasoning_content/thinking_blocks/logprobs/ enhancements, plus annotations) regression tests --------- Co-authored-by: Mateo <277851410+mateo-berri@users.noreply.github.com>
…ent IP resolution (BerriAI#31257) * feat(mcp): add mcp_xff_num_trusted_hops to harden XFF client IP resolution MCP per-server IP access control reads the client IP from X-Forwarded-For and trusts the leftmost entry. Behind an append-style proxy or load balancer (AWS ALB, nginx with $proxy_add_x_forwarded_for, HAProxy, Envoy, Cloudflare), a client can prepend an arbitrary value to the header, so the leftmost entry is attacker-controllable even when the direct peer is a trusted proxy. An attacker can therefore spoof an internal IP and reach servers marked available_on_public_internet=false. This adds an optional mcp_xff_num_trusted_hops general setting modelled on Envoy's xff_num_trusted_hops. When set to N, the client IP is read N entries from the right of the chain (where N is the number of trusted appending proxies in front of the gateway) instead of the leftmost value, so any entries a client prepends are ignored. It composes with mcp_trusted_proxy_ranges, which still validates the direct peer, and only takes effect once that check passes; without a validated direct peer the gateway keeps failing closed, so hop counting cannot be abused by a direct-to-pod attacker. The chain must contain at least N valid entries or resolution fails closed. Default is unset, preserving existing behaviour. * chore(ui): regenerate dashboard schema for mcp_xff_num_trusted_hops * fix(mcp): warn when mcp_xff_num_trusted_hops is below the minimum A 0 or negative value is silently treated as disabled, which could leave an operator believing they enabled append-style X-Forwarded-For hardening while client IP resolution stays on the spoofable leftmost value. Emit a warning, consistent with how the module already surfaces invalid CIDR config, so the misconfiguration is visible in logs. * fix(mcp): reject mcp_xff_num_trusted_hops < 1 at config-parse time Add a ge=1 bound to the ConfigGeneralSettings field so the update_config_general_settings path rejects 0 and negative values with a clear validation error instead of accepting them, and self-documents the valid range. The runtime warning stays as defense-in-depth for raw-dict config that bypasses model validation. * style(mcp): black-format ip_address_utils.py * fix(mcp): fail closed when mcp_xff_num_trusted_hops is set but invalid A present-but-invalid mcp_xff_num_trusted_hops (non-integer, or below 1) previously made _resolve_num_trusted_hops return None, which the caller treated identically to "unset" and silently fell back to the legacy leftmost X-Forwarded-For value. An operator who set the value to harden client IP resolution but typo'd it would get weaker security than before, with no fail-closed signal. Model the setting as a tagged union (_HopCountUnset, _HopCountInvalid, _HopCount) so the three states are distinct: unset keeps the legacy path, a valid count drives hop-counting, and an invalid value fails closed (returns "") instead of reverting to the spoofable leftmost address. The caller matches on the union exhaustively. Add a parametrized regression test asserting get_mcp_client_ip returns "" for 0, -1, "abc", and 1.5 even with a spoofed internal leftmost entry, and update the resolver unit tests for the new return type.
…erriAI#31262) * fix(otel): hashable scope for _emit_once when guardrail_mode is list `_emit_once` keys `spans_logged` by `(class, id, *scope)`. When a guardrail entry's `guardrail_mode` arrives as a `List[GuardrailEventHooks]` (the shape Presidio expands to with `output_parse_pii: true`, and the shape `event_hook` carries for any `mode: [...]` in config), the tuple contains a list and `spans_logged.get(dedupe_key)` raises `TypeError: unhashable type: 'list'`. On the post-call path this fires inside the logging callback and is swallowed; the request returns 200 but the OTEL `guardrail` span is silently dropped. On the blocking path the same error surfaces as HTTP 500. Adds `_freeze_for_dedupe`, a small recursive normalizer that turns lists and tuples into tuples, sets into frozensets, dicts into frozensets of `(key, value)` pairs, and falls back to `repr` for arbitrary unhashables. Applied inside `_emit_once` before the dict lookup, so all three callsites are protected without touching the guardrail-specific callsite. Helper assumes acyclic input; `guardrail_mode` values are built fresh from config (str enums, lists of str enums, TypedDict of str/list-of-str), so no cycle can arise in practice. Regression tests in `TestOpenTelemetrySpanDedupe` cover the list crash, distinct-list-scope collision, dict and set scope parts, and an end-to-end `_create_guardrail_span` exercise that confirms exactly one `guardrail` span is emitted across repeated lifecycle entrypoints. Each new test fails on a reverted helper (4/4 mutation kill) * fix(otel): cap _freeze_for_dedupe recursion depth and ignore in recursive detector CI's recursive_detector blocks new recursive functions in litellm/ unless they are in the allowlist with a documented bound. Cap the helper at 16 levels and return repr(value) past the cap; this is well past the realistic depth of guardrail_mode (1-3 levels) and means a future caller passing a cyclic container can no longer push the proxy logging path into a RecursionError. Add a regression test that exercises the cycle path. * refactor(otel): annotate _freeze_for_dedupe return as a HashableScope union Per review feedback from @mateo-berri: replace the loose `-> object` annotation with a recursive `HashableScope` union (str | int | float | bool | bytes | None | Tuple[HashableScope, ...] | FrozenSet[HashableScope]) so the helper's contract is visible at the signature. Replace the `try/except hash(value); return value` passthrough with an explicit isinstance check over the hashable-scalar types so the type checker can narrow without requiring `cast(Hashable, value)` on the return. Symmetric: dict keys also flow through the freezer (a TypedDict key is already a string in practice, so behaviorally identical). All 16 regression tests still pass; mutation kill behavior preserved * fix: avoid explicit casting --------- Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
* feat: package rust ocr bridge in litellm wheel * Install Rust in Windows CircleCI job * Address Rust wheel review feedback * Pin Windows rustup installer hash
Ignore the compiled, platform-specific Rust extension output (litellm/rust_bridge/_native*.so/.pyd) and the litellm-rust/target/ build dir so local maturin/cargo builds don't show up as untracked files. Also drop the two stale self-referential .gitignore entries; .gitignore is tracked, so ignoring it did nothing except add confusion. Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
… the strict gate (BerriAI#31335) * chore(lint): widen ruff budget slack to 10% of baseline for high-volume ANN rules and PLR0913 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * chore(lint): drop PLR0913 from strict gate to roll out rules gradually Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(lint): ratchet-guard rising baselines even when slack is cut to mask them Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* feat: port OCR providers to Rust gateway * chore(deps): update langgraph checkpoint lock * ci: scope ruff format check to changed files * ci: fix OCR lint and patch coverage * fix(ocr): block mapped IPv6 fetch targets * test(ocr): include rust bridge coverage in OCR shard * ci: rerun responses shard
The namespace configured under cache_params was only applied to get/set/
increment paths. Operations that take keys through other code paths (the Lua
scripts registered via async_register_script, delete, scan_iter, rpush, lpop,
get_ttl, and the sync increment_cache) hit raw keys. With a namespace set, the
rate limiter ({key}:tokens/requests/window), pod-lock release, and budget
limiters wrote keys outside the configured prefix, breaking multi-tenant key
isolation and leaving those operations reading keys the namespaced writes never
created.
check_and_fix_namespace is now applied uniformly across every key-taking
RedisCache operation. It is a no-op when no namespace is configured, so
deployments without a namespace are unaffected. The prefix is prepended ahead of
any {hash-tag}, so Redis Cluster slotting is preserved.
Resolves LIT-3374
…rriAI#30022) Fixes BerriAI#29794. Adds bare, gemini/, and vertex_ai/ entries copied from preview models so proxy cost tracking works for GA model names. Co-authored-by: Cursor <cursoragent@cursor.com>
Add search_context_cost_per_query to gpt-5 and gpt-4o model entries in the cost map. Count web_search_call output items from Responses API responses using dict-safe type extraction, and multiply the per-query OpenAI rate by the number of completed searches. Fixes BerriAI#31316
Contributor
|
Too many files changed for review. ( |
|
|
6 tasks done
Contributor
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
15 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add missing
search_context_cost_per_querypricing for GPT-5 and GPT-4o chat models, and charge perweb_search_calloutput item returned by the OpenAI Responses API.Motivation
LiteLLM reported $0 web search tool cost for GPT-5 models used via
/responseswith the hostedweb_searchtool, even when the response contained multiple completedweb_search_callitems. The cost map lacked web-search pricing for GPT-5 entries, and the cost tracker only applied a single per-request fee.Fixes #31316
Changes
search_context_cost_per_query($0.01/query) andweb_search_billing_unit: per_queryto 39 GPT-5 models and 5 GPT-4o chat variants in bothmodel_prices_and_context_window.jsonand the bundled backup fileresponse_objectinto web search cost handlingweb_search_calloutput itemsTests
test_gpt5_mini_has_web_search_pricingtest_responses_api_web_search_cost_counts_each_call(4 calls × $0.01 = $0.04)Notes
web_searchrate as GPT-5 reasoning models, per current OpenAI pricing ($10/1k calls)