fix(mavvrik_focus): carry prompt/completion tokens in FOCUS Tags - #24
Closed
pghuge-cloudwiz wants to merge 475 commits into
Closed
fix(mavvrik_focus): carry prompt/completion tokens in FOCUS Tags#24pghuge-cloudwiz wants to merge 475 commits into
pghuge-cloudwiz wants to merge 475 commits into
Conversation
…iAI#31691) Docs live in BerriAI/litellm-docs; these four files were swept into the repo by unrelated PRs. The Crusoe provider and XecGuard guardrail pages are migrated to litellm-docs (BerriAI/litellm-docs#438); plugin_architecture.md is already covered there by docs/proxy/plugins.md, and the orphaned image was referenced by no doc.
* chore: remove _experimental/out * fix(ci): recreate _experimental/out before copying UI build output The build scripts cp the Next.js output into litellm/proxy/_experimental/out, which was removed from git. cp failed because the target directory no longer existed; mkdir -p recreates it before the copy. * fix(proxy): make UI serving resilient to a missing _experimental/out Removing the committed UI export means the source/test tree no longer ships litellm/proxy/_experimental/out. Three things assumed it was always present and broke once it was gone: - get_favicon hard-coded the built favicon path and 404'd without it; it now falls back to the bundled swagger/favicon.ico - the /_next and /ui static mounts raised at construction when the export was absent, so the whole UI-setup block was swallowed and no mounts registered; they now use check_dir=False - _restructure_ui_html_files was a nested function only exposed as a module attribute when that block happened to succeed; it is now a real module-level function test_admin_ui_export_serves_nested_extensionless_routes validated the committed artifact, whose premise this PR removes; it now drives the same MCP OAuth callback restructure guarantee through a synthetic export. * chore(greptile): ignore generated _experimental/out so review fits the file limit * Revert "chore(greptile): ignore generated _experimental/out so review fits the file limit" ignorePatterns is applied after Greptile counts the files changed, so it does not bring the diff under the file limit; the config had no effect.
…head + guardrails) (BerriAI#31593) litellm_overhead_latency_metric only covers the SDK wrapper window and excludes proxy guardrails. Add a histogram that sums SDK overhead plus pre/post-call guardrail durations (during-call excluded since it runs concurrently with the LLM call, alongside logging_only and MCP modes that never block the response), recorded next to the existing overhead metric with the same labels and buckets. No existing metric's value is changed.
….md rule (BerriAI#31544) * chore: shift CI lint left with a pre-commit hook and CLAUDE.md rule Add an opt-in pre-commit hook (.githooks/pre-commit, active after make install-hooks) that runs the CI-equivalent checks against staged files: make lint for Python, prettier plus eslint for the dashboard, and a gen:api drift check for the proxy OpenAPI types. Document the same expectation in CLAUDE.md so reds surface locally instead of in CI. * fix: make `make lint` isomorphic to the CI lint job `make lint` diverged from test-linting.yml in ways that produced both false reds and false greens: its format-check ran over the whole repo (CI scopes it to changed files vs the base), its ruff-strict budget ran in absolute mode (CI runs it as a delta vs base), and it omitted the type-discipline gate entirely. Recompose `lint` to replay CI's exact sequence: diff-scoped ruff format check, whole-tree ruff check, the strict / type-discipline / basedpyright budgets as a delta resolved the same way CI resolves it (merge-base with origin/litellm_internal_staging), then circular-import and import-safety. Factor the repeated base fetch into one shared prerequisite so the chain hits the network once. Align the pre-commit hook's eslint invocation with the CI frontend-lint job (`--pass-on-unpruned-suppressions`) and fix the CLAUDE.md guidance to point at the diff-scoped frontend commands instead of the whole-folder npm scripts, which are broader than CI. * fix(githooks): make pre-commit 1:1 with CI frontend-lint, lint, and type-gen The shift-left pre-commit hook diverged from the CI jobs it claims to mirror, so a clean commit did not actually mean a green CI lint. The dashboard block only ran prettier and eslint over js/jsx/ts/tsx/mjs/cjs, but CI's frontend-lint runs prettier over a wider set (also json, css, scss, md, mdx, yml, yaml, html) and additionally gates the whole-folder eslint lint budgets via scripts/check-lint-budgets.mjs. The hook now mirrors that split and runs the budget check, so a dashboard commit that passes locally passes the job. The API-types block ran npm run gen:api without LITELLM_PYTHON, so it shelled out to the system python3 which has no litellm installed and always failed with a false 'could not regenerate API types' red. It now passes LITELLM_PYTHON="uv run --no-sync python" the way check-ui-api-types.yml does. make lint format-checks the files in origin/base...HEAD, which at pre-commit time predates the staged change, so a brand-new commit's formatting went unchecked. The Python block now also runs ruff format --check over the staged litellm files directly to cover that case, and its trigger is scoped to staged litellm/ files (the only tree CI's lint job inspects) so a tests-only or scripts-only commit skips the slow make lint instead of wasting time on a run that could not catch anything. CLAUDE.md's shift-left rule was cut off mid-sentence and understated the frontend checks; it now describes all three gates accurately and points agents at make install-hooks to run them automatically before each commit. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(githooks): scope the API-types check to all of check-ui-api-types.yml's triggers spec_files was filtered from the staged Python files, so the gen:api drift check only fired for .py changes under litellm/proxy or litellm/types. CI's check-ui-api-types.yml triggers on any file under those directories (Prisma schema, configs) plus the generator script and the dashboard package files, so a non-Python proxy/types change could pass the hook and still fail CI. Match the workflow's full trigger set instead. * fix(pre-commit): run prisma generate before gen:api to mirror CI * refactor(githooks): run shift-left lint via on-demand make pre-commit, not an auto-firing hook The pre-commit hook ran make lint plus the dashboard eslint budgets, which are minutes of work (basedpyright over litellm/, a whole-folder eslint . pass at ~40s). Wiring that into core.hooksPath via make install-hooks meant every human commit, not just an agent's, paid that cost, which is real friction for interactive committers. Move the staged-file checks out of .githooks/ into scripts/pre_commit_lint.sh and expose them as make pre-commit, and keep .githooks/ to only the fast Conventional Commits / Branches hooks so make install-hooks no longer makes commits slow. Agents run make pre-commit right before each commit (CLAUDE.md instructs this), so the slow gates fire only for the commits an agent is making and never auto-fire for a human typing git commit. The script stays hook-compatible for anyone who still wants it to fire automatically via a symlink. Preferred this over sniffing an agent env var to auto-fire only for agents: that is fragile (misses agents when the var is unset, fires on humans when it leaks into their shell, and silently no-ops a hook a human deliberately installed), whereas an on-demand command achieves the same humans-never, agents-per-commit outcome deterministically. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(pre-commit): run make lint last so it can't prune the proxy deps gen:api needs make lint's install-dev prerequisite runs uv sync --frozen, which prunes the proxy extras (prisma, websockets, ...) from the venv. With the Python block running first, the subsequent API-types block then failed: gen:api imports litellm.proxy.proxy_server, which needs those deps, so every litellm/proxy change (the main trigger for the API-types check) hit a false 'could not regenerate API types' red. Run the dashboard and API-types blocks before the Python block so gen:api sees an intact env; CI is unaffected because there the lint and check-ui-api-types jobs run in separate environments. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix: make CLAUDE.md more concise * fix(makefile): give make lint the CI lint env and stop it pruning the venv make lint diverged from test-linting.yml's lint job in two ways: it never generated the Prisma client (so basedpyright resolved the DB wrappers as Unknown, drifting from CI's counts), and its bare uv sync --frozen pruned the proxy extras (prisma, websockets, ...) out of the venv on every run, which broke the gen:api step that imports litellm.proxy.proxy_server and left a dev unable to run the proxy until re-syncing. Add a lint-install target that mirrors the job's environment (the proxy-dev group plus prisma generate) and runs before the checks, and make both it and install-dev use uv sync --inexact so they top up the venv instead of tearing packages out. CI is unaffected since it installs its own env per job. Because make lint no longer prunes, the pre-commit reorder that ran it last (to dodge the prune) is no longer needed, so restore the original block order. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(makefile): drop lint-install so make lint matches CI's slimmer env test-linting.yml's lint job installs deps with a bare uv sync --frozen (default dev group only, no proxy-dev, no prisma generate), but the lint-install target chained into make lint pulled in --group proxy-dev and ran prisma generate. Because the basedpyright budget step compares head and base counts against fixed thresholds, the extra symbols and Prisma client locally resolved can shift error counts away from CI's, producing false greens or false reds on the type-check gate. Remove the lint-install target and its slot in lint. The remaining sub-targets already chain install-dev, which now uses uv sync --inexact --frozen, so the venv still isn't pruned but the installed set stays aligned with what CI sees. * ci(linting): install proxy-dev and generate prisma in lint job, matching make lint make lint now installs the proxy-dev group and generates the Prisma client so basedpyright resolves the DB wrappers; the lint job here still installed only the base env, so a local pre-commit could pass while the required CI lint failed (or vice versa). Bring this job in line, which is the same environment litellm_internal_staging's lint job already uses. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(makefile): keep make lint on the proxy-dev + prisma env to match CI A concurrent change dropped lint-install to match what looked like CI's slim env, but test-linting.yml's lint job (and the merge ref this PR's CI actually runs) installs --group proxy-dev and generates the Prisma client. With make lint slim and CI fat, basedpyright resolves fewer symbols locally than CI, so a prisma-typed error can stay Unknown locally (green) while CI catches it (red). Restore lint-install so make lint installs the same env CI does; the previous commit also brought this PR's test-linting.yml in line with that env, so the two now match. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
…rage (BerriAI#31043) * feat(proxy): type Customer Management response_model for OpenAPI coverage Add response_model to the five remaining untyped /customer operations (block, unblock, new, update, delete) so the generated OpenAPI schema documents a concrete response body. new/update reuse the canonical LiteLLM_EndUserTable (matching info/list); block, unblock, and delete get small dedicated models in litellm/types/proxy/management_endpoints/customer_endpoints.py. Together with the already-typed info/list/daily-activity routes this brings the Customer Management group to full response_model coverage. Regression tests assert each public /customer/* route declares the expected response_model and that /customer/new surfaces a typed schema in app.openapi(), so dropping a response_model fails CI. * fix(proxy): keep budget_id in typed customer responses Address review feedback on the Customer Management response_model typing. Greptile flagged that response_model=LiteLLM_EndUserTable on /customer/new and /customer/update silently drops fields the raw Prisma model_dump() echoed. Checking the schema, budget_id is the only such scalar column that was missing from the Pydantic model (created_at/updated_at/tpm_limit do not exist on litellm_endusertable), so add budget_id to LiteLLM_EndUserTable. This restores budget_id on new/update and also fixes the pre-existing gap where /customer/info and /customer/list (already typed) dropped it, which the UI Customer type expects. A regression test pins budget_id surviving the response_model filter on /customer/update. Also document UnblockUsersResponse.blocked_users via a Field description: it holds the users that remain blocked after the call. The key name predates this PR and is kept to avoid a backwards-incompatible rename on a beta route. * fix(proxy): keep nested budget fields in customer responses response_model=LiteLLM_EndUserTable nests the budget as the narrow write allowlist LiteLLM_BudgetTable, which silently drops the server-managed fields the customer endpoints used to return (budget_reset_at, created_at). Introduce CustomerResponse, a thin response model that nests LiteLLM_BudgetTableFull (the repo's budget response model), and apply it on /customer/new, /customer/update, /customer/info and /customer/list. list also builds CustomerResponse so its budget isn't narrowed at construction time. created_by/updated_at/updated_by remain omitted, matching how budgets are returned elsewhere. The shared LiteLLM_EndUserTable is left untouched: it's constructed in many places that pass narrow budget instances, and pydantic v2 won't coerce a budget instance into a wider nested model. Typing only at the response boundary (where the handler hands FastAPI a dict) sidesteps that. A regression test pins budget_reset_at + created_at through the filter and asserts the internal audit fields stay out. * test(proxy): add golden-master characterization tests for customer responses Lock the exact JSON body each customer-object endpoint (info/list/new/update) and delete return today, so the upcoming type-safety refactor of the handlers is only allowed to land if it reproduces these byte for byte. Pins null-field inclusion, the nested budget shape (server fields kept, audit fields dropped), and object_permission reverse-relation stripping. Green against current code. * refactor(proxy): make the customer response flow type-safe Replace the untyped dict + bolt-on response_model pattern on the customer object endpoints with explicit typed construction. A single mapper, _to_customer_response, validates a DB row into CustomerResponse at one Any -> typed seam; new/update/info/list now return it (or a list of it) and carry real -> CustomerResponse / -> List[CustomerResponse] return annotations, and delete returns DeleteCustomersResponse. basedpyright now verifies the handlers' return shapes instead of a runtime filter doing it silently. This also deletes the four copy-pasted object_permission reverse-relation cleanup loops: pydantic's extra=ignore drops those undeclared fields during validation, so the loops were dead code (proven by the golden-master tests, which stay byte-for-byte green). basedpyright errors on the file drop from 140 to 116, all from removed dict plumbing. CustomerResponse stays a thin subclass of LiteLLM_EndUserTable so it inherits the existing validators/config unchanged (behavior preservation); only the nested budget type is widened. * refactor(proxy): annotate customer response mapper param as BaseModel Address review nit: the mapper's untyped `record` added an ANN001 violation. The incoming rows are pydantic v2 models, so type the param as BaseModel rather than object (object has no model_dump, which would just move the problem to basedpyright). This clears the ANN001 and also drops three basedpyright unknown-type violations the untyped param was adding. * style(test): ruff format customer endpoint tests * test(proxy): give customer budget test update mocks a valid model_dump The type-safe response refactor validates the update result via _to_customer_response (CustomerResponse.model_validate(record.model_dump())). These budget tests mocked the end-user update to return a bare MagicMock, so model_dump() yielded a MagicMock that fails validation. Give each update mock a minimal valid dict; the tests assert on the prisma calls, not the body. * chore(ui): regenerate API types from proxy OpenAPI spec * fix(ui): make generated API types stable across Python versions Python 3.13 strips a docstring's common leading indentation at compile time while 3.12 keeps it, so app.openapi() emits differently-indented description strings depending on the interpreter. The dashboard type generator ran locally on 3.13 and in CI on 3.12, so schema.d.ts drifted and the "Verify schema.d.ts matches the proxy OpenAPI spec" check failed Normalize every description through inspect.cleandoc in the spec dump so the output is identical regardless of interpreter, then regenerate
…n-21b5d0 fix(ui): stop Request Logs page from overflowing horizontally and size its columns
…rriAI#31707) The proxy auth path calls phase_span() and seed_request_identity() in litellm/integrations/otel/runtime.py on every request, each doing a try/except lazy import of litellm.integrations.otel.logger. When the OpenTelemetry SDK is not installed (the default), that import raises, and CPython never caches a failed import, so every request re-scanned sys.path and contended on the import lock. At 750 concurrent users this cost about 12% throughput versus v1.85.0. Resolve the hooks once and cache the outcome, absence included, with functools.cache, so the import is attempted a single time instead of per request. Throughput returns to the v1.85.0 baseline.
…l_v2 (BerriAI#31525) * feat(otel): emit a tools/list CLIENT span for MCP discovery under otel_v2 Under otel_v2 an MCP tools/call already produced a dedicated CLIENT span, but tools/list produced none. The discovery call surfaced only as the bare POST /{mcp_server_name}/mcp server span with no MCP attributes, indistinguishable from initialize and impossible to query by method The list success event already reaches the v2 logger with call_type list_mcp_tools, but _emit_mcp_tool_call only matched call_mcp_tool, so listing fell through to the LLM-call path and emitted nothing. This adds a dedicated MCP_LIST_TOOLS span role with its own MCPListToolsSpanData, emitted from a sibling _emit_mcp_list_tools branch that mirrors the tools/call path Per the OTel GenAI MCP semantic conventions the span is named tools/list (the method name alone, since there is no low-cardinality target), is a CLIENT span parented to the request span, and carries mcp.method.name plus the call id. It deliberately omits gen_ai.operation.name and gen_ai.tool.name, which the convention reserves for tool executions, since listing runs no tool * fix(otel): anchor MCP spans to params._meta trace context, not the transport span MCP streamable-HTTP multiplexes many JSON-RPC messages over one session, so the request-root anchor captured on initialize persisted and every later message's span (tools/call, tools/list) nested under it. A tools/list run 44s after the initialize rendered 44s to the right of its parent with a clock-skew warning, because the MCP message and the HTTP transport are independent lifecycles Following the OTel GenAI MCP semantic conventions, an MCP span now parents to the W3C trace context the client propagated in the request's params._meta (a remote parent, per SEP-414), records the transport/session span as a span link rather than the parent, and starts its own root trace when nothing was propagated. The MCP gateway captures traceparent/tracestate/baggage from each message's params._meta into a per-message contextvar that the otel_v2 emitter reads; opentelemetry stays an optional dependency via guarded lazy imports This applies to tools/call as well as the new tools/list span, since both shared the same transport-anchoring bug * fix(otel): drop client baggage from MCP params._meta to prevent identity spoofing The MCP trace propagation added a W3CBaggagePropagator, so resolve_mcp_span_context extracted the client's W3C Baggage from params._meta into the span's parent context. The LiteLLMBaggageSpanProcessor then stamps allowlisted baggage keys onto the span, and the list-tools/tool-call mappers don't set those identity keys, so nothing overwrites them. A malicious MCP client could send params._meta.baggage: litellm.team.id=...,litellm.metadata.user_api_key_user_id=... and have those identity attributes attributed to its spans. Extract trace context only (traceparent/tracestate) in the propagator, and stop collecting the baggage key at the source in _mcp_meta_trace_carrier. Parenting to the client's trace context, the actual goal, needs only trace context; remote baggage had no legitimate consumer here. Regression tests at both layers assert a spoofed params._meta.baggage never lands as a span identity attribute. * style(mcp): clear ruff strict-budget breach in otel trace-carrier helpers The otel MCP trace-carrier helpers added in this branch pushed the BLE001 and UP006 strict-rule totals past their ceilings. Use PEP 585 `dict[str, str]` instead of `Dict`, and narrow the optional-import guards to `except ImportError` (the only failure these can hit, matching the "when otel_v2 is unavailable" intent) instead of a blind `except Exception`. * fix(otel): stamp authenticated identity baggage onto MCP spans Parenting MCP spans to the client's params._meta trace context over an empty Context() meant the tool-call and tools/list spans carried no team/key/metadata identity at all, so they couldn't be attributed or filtered by team in a traces backend. The LLM-call span already re-seeds identity from the parsed, authenticated StandardLoggingPayload rather than trusting ambient/remote context; extract that into a shared _seed_identity_baggage helper and run both MCP emitters through it. Identity comes only from the authenticated payload, never the client carrier, so this keeps the earlier spoofing fix intact while restoring attribution. Regression tests assert the authenticated team lands on both MCP spans and that a spoofed params._meta.baggage value can't override it. * refactor(otel): model MCP spans as roots that link the transport in SPAN_REGISTRY
…hot paths (BerriAI#31716) Guard the per-request CPU cost of the chat completion, MCP tool and A2A message transforms against regressions on every commit. All benchmarks are pure in-process work with no network I/O so they stay deterministic under CodSpeed's simulation mode, and they import under the base dependency set the benchmark job installs. Inference covers the full SDK overhead via mock_response (simple, multi-turn, tools, streaming) plus convert_to_model_response_object as a deterministic anchor. MCP covers the client-side tool translation and the proxy server-side tool-name prefix round-trip. A2A covers the client request/response transforms and the proxy server-ingress message conversion. Adds the mcp and a2a-sdk packages to the benchmark run since those transform modules need them, and broadens the workflow triggers to litellm_internal_staging so the internal branch flow is benchmarked too.
…I#31652) * fix(ui): allow any git host on the skills add form (LIT-4053) The skills add form only accepted GitHub URLs: its URL parser bailed on any host that did not start with github.com, so GitLab, Bitbucket, and self-hosted repos (and any repo subfolder on them) were rejected before a request was ever sent. The backend already accepts arbitrary git hosts via its url and git-subdir sources, with no host allowlist, so this was a client-side restriction only. Generalize the parser into an exported, host-agnostic parseSkillSource: GitHub URLs keep their github / git-subdir shorthand, every other host is treated as a raw repo url, and an optional Subfolder path field turns any repo into a git-subdir source (url + path). When a pasted GitHub tree/blob URL already encodes a subfolder, the field is cleared and disabled so a contradictory source can never be submitted. The parser is hardened to match the backend contract: query strings and fragments are stripped, the host match is case-insensitive and drops a leading www., the extracted and field-entered subfolder paths are both validated against the same regex the server uses, a real file-extension allowlist (not "any dot") decides whether a trailing blob segment is a file, a branch-only tree URL falls back to the repo, non-GitHub URLs require at least an org/repo, and the suggested skill name is kebab-cased so it satisfies the name field's own rule. The git-subdir source is now handled in the display helpers (getSourceDisplayText, getSourceLink, formatInstallCommand), which previously showed it as "Unknown source" with no link. The submit path is fully typed (RegisterPluginRequest plus an AddPluginFormValues interface), removing the two prior any usages; as a result an author with an email but no name is dropped rather than sent, since the backend requires the author name. No backend changes. Tests cover the full host/subfolder matrix at the parser level plus form-submit assertions on the exact source payload. * refactor(ui): sync skill register types to the generated OpenAPI schema, surface backend errors Replace the hand-maintained, already-drifted API types for the skills add flow with the generated ones from schema.d.ts: PluginAuthor now aliases components["schemas"]["PluginAuthor"], the registration payload is a new SkillRegisterRequest (the generated RegisterPluginRequest envelope with source narrowed to our PluginSource union, since the backend types source as a loose string map, and version kept optional since the backend defaults it), and the dead, mismatched RegisterPluginResponse is deleted. registerClaudeCodePlugin's inline payload type (which was missing the git-subdir path field entirely) is replaced with SkillRegisterRequest, so the networking layer and the form can no longer drift from the backend. Error handling: the add-skill form swallowed the real failure and always showed "Failed to register skill". registerClaudeCodePlugin already derives the backend message and throws it, so the form now surfaces it ("Failed to register skill: <reason>"), and the networking helper falls back to the raw body / status when the error response is not JSON instead of throwing a JSON parse error. A regression test asserts the backend message reaches the user. * fix(ui): reject credentialed git URLs on the skills form A repo URL with embedded user-info (user:token@host) passed the raw-host parser and was stored verbatim as the skill source, which is served on the unauthenticated /public/skill_hub and marketplace.json feeds, leaking the credentials. Reject any host segment containing '@'. * fix(ui): validate skill repo URLs through one WHATWG URL gate Replace the ad-hoc string parsing (stripScheme / splitHost / manual scheme, @, ?# checks) with a single parseRepoUrl gate built on the URL parser, so every malformed/unsafe class is handled in one place and the URL stored on the public skill feeds is always canonical. It enforces https (rejecting http/ssh/git/file/javascript/data and protocol-relative //host), rejects embedded credentials (user:token@host, including userinfo-confusion like github.com@evil.com), rejects IP-literal hosts (loopback/private/metadata and obfuscated/IPv6 forms), and rebuilds the stored url from origin+pathname so query strings, fragments, and trailing slashes can never be published. The GitHub org/repo shorthand is now charset-validated like the other paths, so junk can't reach the stored repo. Closes both Veria findings (credentialed and http sources) plus the adversarial-review follow-ups, with regression tests for each class.
The existing comment rule is not strict enough
…rriAI#31730) * feat(guardrails): expose streaming knobs on generic_guardrail_api Wire streaming_end_of_stream_only and streaming_sampling_rate through optional params, initialize_guardrail, and get_config_model so the generic guardrail API participates in UnifiedLLMGuardrails streaming checks with configurable cadence and end-of-stream-only mode. * fix(guardrails): use builtin type[] in get_config_model return Avoids a new UP006 violation that tripped the ruff strict-rule budget gate on the PR lint job. * fix(guardrails): default optional streaming knobs to None Non-None Pydantic defaults on GenericGuardrailAPIOptionalParams made _get_config_value treat unset nested fields as explicit values, which shadowed top-level litellm_params streaming flags whenever any other optional_params key was present. Real defaults stay in the constructor. * fix(guardrails): address review nits on generic_guardrail_api streaming Validate streaming_sampling_rate >= 1 in the constructor and Pydantic optional_params (ge=1), and add /v1/responses streaming coverage through the unified post-call hook so Responses API usage is exercised alongside chat completions. * fix(guardrails): read nested streaming config from dict optional_params Guardrail API/UI delivers optional_params as a plain dict, so getattr was silently ignoring streaming_sampling_rate and streaming_end_of_stream_only. Handle both dict and model shapes in _get_config_value with regression tests. * fix(guardrails): clear ruff findings in generic_guardrail_api tests/types * style(guardrails): ruff format generic_guardrail_api modules --------- Co-authored-by: Marton Schneider <marton@schneider.co.nl>
Adds `!` prefix negation to tag-based routing so callers can exclude deployments by exact tag value without enumerating every allowed alternative. `!provider:anthropic` removes all deployments tagged exactly `provider:anthropic` before routing, and positive and negation tags compose. Matching is exact literal membership (frozenset intersection), so there is no regex or ReDoS surface for client-supplied tags. Ban-only requests that carry only negation tags stay within the default pool, mirroring untagged-request semantics so callers can't use negation to escape it. Fallback chains keep working because get_deployments_for_tag runs on each routing hop Copy of BerriAI#31680; implementation credit to @deepanshululla Co-authored-by: deepanshululla <15312873+deepanshululla@users.noreply.github.com>
The Loadbalancing tab rendered routing_groups as a generic text input and sent its array value back as the JSON string "[]", which fails Pydantic list validation on POST /config/update and returns 422. routing_groups has its own dedicated Routing Groups tab, so this tab must neither render nor write it; exclude it the same way retry_policy and model_group_retry_policy are excluded for the Model Retry Settings tab. The save was also fire-and-forget: setCallbacksCall was not awaited, so the rejected promise escaped the try/catch and the success toast fired unconditionally, showing success even when the backend rejected the change. Await the call, gate the success toast on resolution, and surface the error.
Address Greptile P2: the routing_groups test read setCallbacksCall.mock.calls[0][1] immediately after the now-async save handler, so any latency in the mock would throw an opaque TypeError instead of a clean assertion failure. Assert through toHaveBeenCalledWith inside waitFor with expect.not.objectContaining, dropping the index access and the cast. Also drop the ticket id from the test names.
Register claude-sonnet-5 across the Anthropic, Bedrock (base + global/us/eu/au/jp cross-region inference profiles), Vertex AI, and Azure AI cost-map entries in both the root and bundled-backup model maps, plus BEDROCK_CONVERSE_MODELS and the setup-wizard provider list. Sonnet 5 ships with the gen-5 adaptive-thinking profile (adaptive thinking always on, no extended thinking, effort defaults to high), so the entries mirror the Fable 5 / Opus 4.8 sampling-param and prefill restrictions rather than the older Sonnet 4.6 behavior: supports_sampling_params and supports_assistant_prefill are false while supports_adaptive_thinking, supports_xhigh_reasoning_effort, and supports_max_reasoning_effort are true. Pricing follows standard Sonnet rates ($3 / $15 per MTok) with the 10% regional premium on the us/eu/au/jp profiles. Add a reasoning-effort grid entry for the Anthropic direct route and a regression test pinning pricing, capabilities, regional premiums, backup parity, and bare-name provider resolution. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
…057) Drives the real save flow against a live proxy: seeds a present routing_groups array (the LIT-4057 trigger) via the typed /config/update contract, changes num_retries on the Loadbalancing tab, and asserts the POST returns 200 instead of 422, the success toast appears, and the value still shows after a reload (the ticket's "refresh shows old values" symptom). The round-trip is typed against the OpenAPI-generated backend schema (ConfigYAML write, RouterSettingsResponse read) through a type-only import, so a backend contract drift fails the type check.
… tools/list (BerriAI#31684) * fix(mcp): stop one unauthenticated server from emptying the aggregate tools/list On the aggregate MCP route (/mcp), the gateway fans out to every server the caller can access and flattens their tools. _fetch_and_filter_server_tools re-raises MCPUpstreamAuthError unconditionally (added with the OAuth passthrough feature in BerriAI#28356) so it surfaces a 401 on single-server routes, but on the aggregate route that exception propagates through the asyncio.gather fan-out and the outer handler turns it into an empty list. The result: a single delegate/passthrough OAuth server the user has not authenticated (e.g. a delegate-auth server) zeroes the tools of every other server, including the ones that resolve fine, so the client connects and sees no tools. Surface the upstream auth error only when a single server was explicitly targeted (so that route still drives the upstream OAuth flow); across the aggregate, absorb it to [] for that one server so the rest still list their tools. This restores the graceful per-server degradation that predated BerriAI#28356. Adds regression tests: the aggregate keeps a healthy server's tools when a sibling raises MCPUpstreamAuthError, and a single-server listing still surfaces it. * fix(mcp): decide aggregate vs single-server listing by route scope, not server count Addresses review: keying the surface-vs-absorb decision off the server count (len(allowed_mcp_servers), and even len(mcp_servers)) misclassifies an aggregate /mcp request from a key that can access exactly one server as a targeted single-server listing, so that one server's MCPUpstreamAuthError re-raises and empties the aggregate again for one-server permission sets. Use the path-derived single-server scope instead: _mcp_gateway_server_name, set by _gateway_initialize_instructions_request_scope only when the request path names exactly one upstream server (/<server>/mcp) and never from client headers, is None on the aggregate route (/mcp) regardless of how many servers the key can access. Single-server routes still surface the upstream-auth challenge; the aggregate absorbs it per server. Adds a regression test that an aggregate request with a single accessible server still absorbs, plus renames the single-server test to drive the route scope explicitly. The new test fails on the count-based logic. * fixing aggregation error * style(mcp): collapse single-line debug log to satisfy ruff format
…net-5 The Sonnet 5 grid entry raised the Anthropic direct route to 30 model combos, so test_grid_cell_count now expects 330 cells instead of 319.
Address an adversarial review of the Loadbalancing e2e: - The "typed against the backend schema" claim was hollow: nothing type-checked e2e_tests (the root tsconfig excludes it and no CI step runs tsc), so a contract drift would compile and run unchanged. Add e2e_tests/tsconfig.json, a typecheck:e2e script, and a CircleCI step so the schema typing actually gates. - The two describe blocks both mutate the proxy's shared router_settings, and the Loadbalancing save echoes the whole settings object, so they could clobber each other under local fullyParallel. Run the file serially. - patchRouterSettings swallowed a failed seed, which surfaced later as a misleading UI timeout. Assert the write succeeded, and rely on the server-side merge instead of echoing the whole settings object back (drops a cast and a GET). - Empty routing_groups already reproduces the bug, so drop the non-empty seed and its model coupling.
…orted_endpoints (BerriAI#31685) * feat(messages): passthrough /v1/messages to native endpoints via supported_endpoints The unified /v1/messages proxy endpoint always translated inbound Anthropic requests down to /v1/chat/completions (or the Responses API for openai) when the deployment's provider lacked a native Anthropic-messages config, dropping Anthropic-only features like cache_control and thinking. Some customers run OpenAI-compatible servers (self-hosted vLLM, DeepSeek's Anthropic endpoint, etc.) that also natively expose /v1/messages and want the raw Anthropic payload forwarded untranslated, while keeping provider openai so /v1/chat/completions to the same deployment stays native. Opt in per deployment via model_info.supported_endpoints containing /v1/messages. When present, the gate routes to a generic, provider-agnostic OpenAILikeAnthropicMessagesConfig that POSTs the Anthropic payload to {api_base}/v1/messages with Bearer auth, instead of translating. Default behavior is unchanged. Generalizes and supersedes the hosted_vllm-only, env-var-toggled PR BerriAI#28745. * fix(messages): preserve standard-cased caller headers in native passthrough The OpenAI-like Anthropic passthrough config only checked for lowercase header names before injecting Bearer auth, anthropic-version, and content-type defaults. A caller sending standard-cased Authorization, Anthropic-Version, or Content-Type was treated as missing those headers, so LiteLLM added duplicate lowercase variants and overwrote the caller's credential/version at the HTTP layer. Header presence is now checked case-insensitively and the merge no longer mutates the caller dict. Also moves the feature docs out of the main repo (docs live in litellm-docs). * fix(openai_like/messages): delegate to parent transform and inject anthropic-beta headers The passthrough config bypassed the parent transform and skipped header beta injection. Both gaps cause native /v1/messages features (context management, advisor tool, fast mode, structured outputs, reasoning_effort, advisor stripping) to silently degrade on opted-in deployments. Reuse the parent's pipeline and call _update_headers_with_anthropic_beta after merging defaults * fix: normalize anthropic-beta header key case before beta injection * style: collapse anthropic-beta header normalization to single line ruff format --check requires the comprehension on one line (it fits within the 120 char limit); fixes the lint job failure on the bugbot autofix commit * fix(messages): forward anthropic-beta to native passthrough upstream The shared anthropic_messages HTTP handler ran update_headers_with_filtered_beta with the deployment's custom_llm_provider after validate. For the native /v1/messages passthrough that provider is openai, which has no beta-header mapping, so every anthropic-beta value (caller-supplied or feature-derived for speed/context_management/etc.) was stripped to empty before the upstream request, breaking beta passthrough to the Anthropic-compatible endpoint. Beta filtering only makes sense on cross-provider translation paths where the upstream cannot understand Anthropic betas. Gate it on a new should_filter_anthropic_beta_headers() that defaults to True (bedrock, vertex_ai, native anthropic unchanged) and is overridden to False by OpenAILikeAnthropicMessagesConfig, whose upstream is a native Anthropic endpoint, so betas pass through verbatim. * chore: remove accidentally committed local QA logs and config --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
…re Sonnet 5 The Vertex AI and Azure AI Sonnet 5 entries carried supports_output_config: true, which the gen-5 siblings (vertex_ai/claude-opus-4-8, azure_ai/claude-fable-5, etc.) do not. The flag only feeds AnthropicConfig._model_supports_effort_param, which already returns true for these entries via supports_xhigh/max_reasoning_effort, so output_config.effort still forwards on both routes. Removing it is behavior neutral and matches the existing per-platform convention for gen-5 Claude.
…p the whole batch (BerriAI#31705) update_spend_logs flushes the queue with a single create_many per batch, so one row carrying bytes Postgres refuses (a residual NUL byte is the canonical case) fails the entire insert and drops every good spend log alongside it. PR BerriAI#29515 strips NUL bytes from the JSON columns, but the scalar string columns (end_user, model, session_id, ...) still flow through unsanitized, so a poisoned row can still reach the write and take a batch of up to 1000 good rows down with it. On a genuine data-layer rejection the batch is now bisected so the good rows still persist and only the offending row is dropped and logged with its request_id. The classification lives in PrismaDBExceptionHandler.is_prisma_data_error (matched by exact type so systemic subclasses like a missing table are not mistaken for a single poison row), which keeps prisma an in-function import and litellm.proxy.utils importable without the proxy extra. Transport failures, including the "can't reach database server" outage that prisma mislabels as a DataError, are re-raised unchanged so the existing connection-retry path still runs and a transient outage never turns into silent per-row data loss. The bisection carries a per-batch isolation budget so an authenticated caller flooding poisoned rows cannot amplify one failed bulk insert into ~2N failed inserts and N log lines; once the budget is spent the still-failing remainder is dropped wholesale under a single log line. Resolves LIT-4103
…rt (BerriAI#31577) When redis_startup_nodes is set the async cluster client was built with no health check and no TCP keepalive, so a connection silently dropped by a cluster restart (e.g. ElastiCache Serverless maintenance) stayed in the pool and got reused while dead; the first command after the restart stalled in re-initialization until the LoggingWorker timeout cancelled it, surfacing as CancelledError then TimeoutError on the spend-counter path Build the async cluster client with a 25s health_check_interval and socket_keepalive so an idle connection is PING-validated and reconnected before reuse, and expose both through the cluster kwarg allow-list so an explicit value from config still wins Resolves LIT-4083
…he editor The e2e runs against the real proxy, so a contract drift already fails the test at runtime; tsc only checks the spec against schema.d.ts, a generated snapshot, so a backend change with a stale snapshot would pass tsc while the live test still catches it. The dedicated tsconfig + script + CI step were circular ceremony for that. Keep the zero-runtime-cost type-only import, which still catches mistakes in the editor, and make its comment honest about what enforces the contract.
…er (BerriAI#31733) RealTimeStreaming.log_messages dispatched the success handler with a bare asyncio.create_task, bypassing GLOBAL_LOGGING_WORKER (which gives a per-coroutine timeout and a concurrency cap). On a long-lived realtime websocket a slow logging callback left one suspended task per logged turn, each pinning that turn's assembled response, accumulating without bound (~12-15k in-flight under load in a repro) until OOM. Route realtime success logging through the bounded worker so in-flight logging is capped and a hung callback is cancelled at the worker timeout. The chat and responses streaming success-logging paths are intentionally left unchanged: their success callbacks must complete within the call's event-loop run (the non-streaming path pairs the worker with a synchronous callback; the streaming path has no such companion), so deferring them through the worker would drop logs for one-shot SDK calls and breaks test_async_custom_handler_stream. Bounding those paths needs a load-shedding approach and is left to a follow-up.
…he whole response (BerriAI#31503) The Presidio streaming post-call hooks (_stream_apply_output_masking for apply_to_output and _stream_pii_unmasking for output_parse_pii) collected every upstream chunk, reassembled the full completion with stream_chunk_builder at end-of-stream, ran Presidio over it, then emitted one reconstructed SSE chunk. Time-to-first-token collapsed to the total generation time and token-by-token streaming was lost whenever Presidio output handling was enabled. With the default presidio_filter_scope both, an apply_to_output masking instance is always created, so even the unmask configuration buffered the stream. Both paths now transform and forward chunks as they arrive. The unmask path replaces placeholder tokens per chunk, holding back only the trailing run that could still grow into a token so a placeholder split across SSE chunks (<PER + SON_1>) is still rewritten atomically. The mask path emits a prefix only when masking it in isolation matches the corresponding prefix of masking the whole buffer, with a lookahead margin still buffered past the cut, so an entity straddling the cut is detected and held until complete; past _PRESIDIO_STREAM_MAX_BUFFER the run is bounded without splitting an entity. Tool-call and legacy function-call argument fragments are accumulated per choice and transformed once the choice closes, content is buffered independently per choice index for correct n>1 streaming, raw Anthropic SSE bytes and /v1/responses events pass through with any held content flushed first so events never reorder, and a masking error redacts only the affected chunk (fail closed, keeping finish_reason) while the stream continues. Resolves LIT-3222
…routes (BerriAI#32267) * fix(e2e): route model management to the control plane and restore Gateway.create_model The split-transport routing table listed only /model/info as a control-plane prefix, so /model/new and /model/delete were sent to the data-plane gateway, which does not serve management routes and 404s them. Every suite that registers deployments at runtime (llm_translation, batches, access_control) failed on the split stage deployment because of this. Widen the prefix to /model/ so all model-management routes reach the control plane while /models stays on the data plane. Separately, batch_client.py and several llm_translation tests call gateway.create_model, but Gateway never had that method, so all 17 batch tests errored at fixture setup with AttributeError. Add create_model/delete_model to Gateway (with the optional mode that batches needs) and make EndpointsClient delegate to it instead of carrying its own copy. Regression tests cover both: the routing predicate for management vs LLM paths and the Gateway model-management surface via a typed fake Transport. Both fail on the previous code * test(e2e): make the fake transport payload depend on response_type The recording fake always answered with {"model_id": ...} even when the caller asked for NoBody, which only validated because pydantic ignores extra fields by default. Return an empty payload for response types that carry no fields so a future extra="forbid" on NoBody cannot turn the delete test into a ValidationError inside the fake * test(e2e): probe the full spend read surface including schema-hidden routes The curated spend-route list missed twelve read endpoints, most of them include_in_schema=False and therefore invisible to the schema-discovery test: /spend/logs/v2, /spend/logs/session/ui, /global/all_end_users, /global/activity/exceptions/deployment, and the per-entity daily activity family (user, user aggregated, team, organization, customer, end_user, tag). Add them all, verified responsive against the live split stage deployment. /end_user was missing from CONTROL_PLANE_PREFIXES, so /end_user/daily/activity would have been routed to the data plane and 404ed like /model/new used to; add the prefix and pin it plus the daily-activity routes in the transport routing test. /provider/budgets stays excluded with a documented reason: it returns 500 whenever router_settings.provider_budget_config is absent, so probing it on a proxy without provider budget routing configured can never be green
…ring (BerriAI#32270) * fix(azure): build responses input_items url with path before query string * chore(azure): drop stale inline comment in responses url helper
Refresh the pinned cgr.dev/chainguard/wolfi-base digest from c61ac6 to 42df77a9 (current wolfi-base:latest, a multi-arch index covering amd64 and arm64). This advances the glibc family from 2.43-r8 to 2.43-r10, with libcrypto3 and libssl3 from 3.6.3-r2 to r3 and libgcc from 16.1.0-r2 to r4; no packages are added or removed. The image scan reports CVE-2026-6791 against glibc 2.43-r8 (fixed in r10). The glibc subpackages are exact-version pinned, so the in-Dockerfile apk upgrade cannot advance them past the base's baked revision, which is why refreshing the digest is required. Same six Dockerfiles as BerriAI#31133
…itellm_/elated-noyce-6fc150
…fc150 fix(docker): bump wolfi-base digest for glibc 2.43-r10
…client registration (BerriAI#32283) Only the gateway-managed interactive flow reaches this persist (the public /register routes never pass persist_credentials), so the row it writes is authorization_code by definition. It was not recorded, which left the row as client creds + token_url with no persisted authorization_url and a null oauth2_flow: exactly the shape the legacy M2M inference in _resolve_oauth2_flow matches. The row normally survives because endpoint discovery backfills authorization_url in memory before the inference runs, but on any transient discovery failure at registry build the server flips to client_credentials for that load, routing per-user traffic to the M2M path Stamping the flow at the write site makes the classification explicit and permanent, so a DCR-registered interactive server no longer depends on discovery succeeding to classify correctly. First step of persisting oauth2_flow at every write site so the legacy inference can eventually be deleted
…licate success callbacks (BerriAI#32265) * fix(logging): classify allm_passthrough_route as async to prevent duplicate success callbacks Async passthrough requests set kwargs["allm_passthrough_route"]=True but that flag is never propagated into litellm_params, and _is_sync_litellm_request only checks acompletion/aresponses/aembedding/aimage_generation/atranscription. Every async passthrough is misclassified as sync, which trips the CustomLogger sync branch in success_handler and fires log_success_event in addition to the async worker's async_log_success_event, causing 2-3 duplicate LangSmith runs per Bedrock passthrough request Propagate allm_passthrough_route through get_litellm_params and teach the classifier about it. /chat/completions and other non-passthrough paths are untouched * test(passthrough): assert allm_passthrough_route flag propagates end-to-end Integration-level guard on top of the unit tests in test_litellm_logging.py: verifies that when kwargs["allm_passthrough_route"]=True enters llm_passthrough_route, the flag survives get_litellm_params(**kwargs), lands in the logging object's litellm_params, and _is_sync_litellm_request reads the request as async --------- Co-authored-by: yucheng <yucheng@yuchengs-MBP.localdomain>
BerriAI#32146) * fix(mcp): forward short OAuth state upstream, keep session in a cookie Some upstream authorization servers reject the OAuth authorize request with "state parameter too long" because LiteLLM replaced the client's short state with its own long encrypted session blob (base_url, original state, PKCE, client redirect_uri) and sent that upstream as state. Forward a short random handle as the upstream state instead, and carry the encrypted session in a per-flow HttpOnly, SameSite=lax cookie bound to that handle. The browser replays the cookie on /callback, so the session is recovered without any server-side store and the client still gets its own original state back. /callback falls back to decoding state directly when no cookie is present, so flows in flight across a deploy keep working. Resolves LIT-4197 * test(mcp): cover /callback error path cookie read and clear The happy-path regression test already asserts the short-handle -> cookie round trip. Add a focused test for the IdP-error branch of /callback: it must recover the client's original state from the per-flow cookie (not the short handle), propagate the error to the client's redirect_uri, and expire the one-time cookie. Fails if the error path stops reading or clearing the cookie.
* fix(bedrock): honor AWS auth params in realtime handler * fix(bedrock): raise clear auth error when no AWS credentials resolve for realtime
…ext_datazone_pricing feat(pricing): add azure data-zone and long-context pricing for gpt-5.4/5.5
…all modes (BerriAI#32296) ModelArmorGuardrail.async_pre_call_hook and async_moderation_hook hardcoded their inner should_run_guardrail event type to pre_call / during_call. The central dispatcher already remaps call_mcp_tool -> pre_mcp_call/during_mcp_call and passes the outer gate, but Model Armor's redundant inner gate then rejected MCP calls for a guardrail configured with mode pre_mcp_call/during_mcp_call, so tool-call content was silently skipped. Remap call_mcp_tool -> pre_mcp_call/during_mcp_call in both hooks, matching the existing behavior of the noma and cisco guardrails. Adds regression tests covering both hooks (scan runs on MCP calls, still skipped for chat traffic). Generated with AI Co-Authored-By: Claude Code Co-authored-by: eugene-yao-zocdoc <eugene.yao@zocdoc.com>
…ms (BerriAI#31356) * feat(jwt): fall back to DB team memberships when JWT has no team claims * style(jwt): use PEP 585/604 annotations in DB team fallback to clear strict gate * fix(jwt): preserve DB teams on no-claim sync, model-gate DB fallback, stop team-id leak When fallback_to_db_teams is enabled and a JWT carries no team claims, sync_user_role_and_teams previously computed teams_to_remove as every existing DB membership and wiped the user out of all their teams on each request, which also left the DB fallback nothing to resolve. Skip team removal in that case so memberships survive and the fallback can attribute usage. Apply the same per-team model-access check the claim-based path enforces when selecting a DB fallback team, so a team's models restriction is no longer bypassed; a team that cannot serve the requested model is skipped in favor of one that can. Drop the user's team-id list from the x-litellm-team-id membership 403 detail so a valid-JWT caller can no longer enumerate team IDs. * fix(jwt): load team membership on DB fallback; scope header check to provisional teams The DB-team fallback resolved a team but never loaded its team membership row, so per-team membership budget limits were silently skipped on that path. _resolve_db_team_fallback now fetches the resolved team's membership when a user_id is known and returns it, matching the claim-based path so downstream LiteLLM_TeamMembership budget enforcement works there too. The provisional x-litellm-team-id validation also fired on any non-None team_id, including an RBAC role-derived one, which 403'd RBAC team flows when the asserted team was not also a DB membership. It now runs only when team_id actually came from the header (team_id == header_team_id). * fix(jwt): surface DB-fallback membership lookup failures at warning level A transient get_team_membership failure on the DB team fallback path is recoverable: the team is still resolved and the request proceeds, just without per-team membership budget enforcement for that request. Logging that at debug hid a silent budget-enforcement gap from operators, so it now logs at warning and states that enforcement was skipped. Behavior is otherwise unchanged: the resolved team is returned with a None membership rather than failing the request, covered by test_resolve_db_team_fallback_survives_membership_lookup_error. * fix(jwt-auth): tighten db-team fallback gating and passthrough enforcement Resolves four issues in the fallback_to_db_teams path: - _resolve_db_team_fallback now surfaces a model-access denial when memberships exist but none can access the requested model, instead of always returning the no-membership message - auth_builder gates the fallback on real JWT team claims via get_all_jwt_team_ids so a configured team_id_default does not silently route claimless tokens to the default team - A team selected only via _resolve_db_team_fallback is re-validated against the team's allowed_passthrough_routes; the earlier gate ran while team_id was still None - sync_user_role_and_teams considers both plural and singular team claim shapes when reconciling DB memberships so singular-only tokens (Okta/Auth0 defaults) no longer leave stale teams behind * fix(jwt): don't upsert a provisional x-litellm-team-id before membership check When fallback_to_db_teams is on and the JWT carries no team claims, an x-litellm-team-id header is accepted provisionally and only validated against the user's DB memberships later in auth_builder. With team_id_upsert also enabled, get_team_object ran the upsert on that unvalidated header team first, so an attacker-supplied header could create an orphaned team row before the 403 membership check. Suppress the upsert whenever the team is provisional (db_team_fallback), since a genuine membership team already exists and an invalid one must not be created. Regression: test_auth_builder_provisional_header_team_is_not_upserted. * fix(jwt): pin RBAC-asserted team against db-team-fallback header override When a JWT carries an RBAC team role but no group claims, auth_builder already sets team_id from the RBAC object_id. db_team_fallback still evaluated true there, so the provisional x-litellm-team-id path accepted a header team and silently overrode the RBAC-asserted team with any team the caller belonged to. Gate db_team_fallback on team_id being unset, and drive the header's provisional acceptance off db_team_fallback rather than the raw flag, so an RBAC token plus a non-claim header team is rejected with 403 instead of substituting the team. Regression: test_auth_builder_header_cannot_override_rbac_team_under_db_fallback. * fix(jwt): scope dual-claim membership sync to fallback_to_db_teams The membership sync read both plural and singular JWT team claims via get_all_jwt_team_ids unconditionally, which silently changed reconciliation for every deployment using sync_user_role_and_teams, not just those opting into fallback_to_db_teams: a singular-only IdP token that previously stripped all DB teams would now be recognized. Gate the dual-claim read on fallback_to_db_teams so flag-off deployments keep the upstream plural-only behavior, honoring the PR's contract that existing deployments are unchanged. Regression: test_sync_user_role_and_teams_singular_claim_only_recognized_under_flag. * fix(jwt): drop user team IDs from db-fallback model-access 403 detail The model-access-denied 403 in _resolve_db_team_fallback echoed the user's full DB team-id list in its detail. It is only the caller's own memberships, but it is inconsistent with the membership-validation 403 in the same feature that was deliberately scrubbed of team IDs. Replace the enumerated list with a generic "no team you are a member of has access" message. Regression extends test_resolve_db_team_fallback_distinguishes_no_membership_vs_model_denied to assert the team id is absent from the detail. * fix(jwt): keep db-team fallback off for alias-only tokens * test(jwt): cover alias-only token skipping db-team fallback The autofix in ed21199 added a get_team_alias clause to the db_team_fallback gate so an alias-only JWT (team_alias_jwt_field set, no team-id claims) resolves its alias via find_and_validate_specific_team_id instead of being mis-attributed to the user's first DB team, but it shipped without a regression test. This drives auth_builder with an alias-only token whose alias resolves to a different team than the user's DB membership and asserts the result is the alias-resolved team; reverting the get_team_alias clause flips the result to the DB-membership team, so the test fails without the fix * fix(jwt): prefer alias resolution over team_id_default When the JWT only carries an alias claim and the operator configures team_id_default, JWTHandler.get_team_id silently substitutes the default into find_and_validate_specific_team_id. That made the helper return the default team without ever attempting alias resolution, so spend and access attached to the default team even though the token identified a different team via its alias. Use get_all_jwt_team_ids (which ignores team_id_default) to detect when the resolved team_id is only the default and clear it so alias resolution runs first; the default remains the fallback when no alias claim is present. * fix(jwt): enforce team_allowed_routes in db-team fallback resolution The claim-based path runs allowed_routes_check when selecting a team, but _resolve_db_team_fallback selected a team purely on model access, so a DB-resolved team could reach routes excluded by team_allowed_routes with no downstream backstop. This mirrors the claim path's route gate in the fallback, exempting auth-enforced passthrough routes that are gated separately by allowed_passthrough_routes at the call site * fix(jwt): enforce team_allowed_routes on header-team db fallback path The auto-pick DB-team fallback already gates against team_allowed_routes, but a claimless JWT presenting x-litellm-team-id under fallback_to_db_teams set team_id directly from the header and only re-validated DB membership afterwards, skipping the route gate. A caller could reach management/info routes that the JWT config narrowed for team-role callers by supplying the header even though the auto-pick path on the same route returns no team. * refactor(jwt): narrow db-team fallback except clauses to actual failure types * fix(jwt): collapse provisional header team lookup failure into membership denial A caller holding a valid claimless JWT under fallback_to_db_teams could distinguish nonexistent teams (404 from get_team_object) from existing teams they do not belong to (membership 403) by varying x-litellm-team-id, giving an authenticated team-id existence oracle. The provisional header path now rewrites the lookup failure into the exact 403 the membership check raises, while claim-backed header teams keep the upstream 404. Also drop the unreachable falsy-team guard in _resolve_db_team_fallback (get_team_object returns a team or raises, never None) and stop codecov carryforward for three dead flags whose stale sessions were measured against old file revisions and sank patch coverage with phantom executable lines --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
…ion for Codex CLI (BerriAI#32258) * fix(responses-bridge): custom tool round-trip and allowlist preservation for Codex CLI Convert Responses API custom tools to Chat Completions function tools and map function_call responses back to custom_tool_call output items so Codex CLI gets the apply_patch round-trip it expects. Preserve and validate allowed_callers during the custom->function conversion so the Anthropic adapter's caller allowlist is not silently dropped, which would let a tool meant to be callable only by another tool be invoked directly by the model. Use modern type annotations (list/dict/set/X | None) throughout to keep the ruff strict budget within its ratcheted ceilings. * fix(responses-bridge): address review feedback on custom tool bridge Type convert_custom_tool_to_function_tool against Mapping/ChatCompletionToolParam and validate allowed_callers with a strict TypeAdapter so the two new cast() calls that tripped the LIT006 ceiling are gone. Warn when dropping Responses-only tool types (computer_use, image_generation, namespace, shell) instead of discarding them silently. Return output items as Pydantic models instead of model_dump()ing every item to a dict, matching the declared return type. Apply the same None-safe metadata pattern to the request_data paths that still used setdefault, and drop the unused build_custom_tool_call_item helper. * fix(responses-bridge): recover custom tool input when arguments is empty * fix(auth): extract custom tool names for allowlist enforcement on responses route The Responses guardrail translation handler only extracted function and mcp tool names, so a key or team restricted by metadata.allowed_tools could invoke a disallowed tool by declaring it with type custom now that the bridge converts custom tools into callable Chat Completions function tools. Extract custom tool names through the same path so check_tools_allowlist rejects them. * fix(responses-bridge): scope input payload recovery to custom_tool_call items Recovering tool arguments from the input field on any falsy arguments value made plain function_call input items with empty arguments and a stray input key get rewritten into a {"content": ...} envelope, corrupting multi-turn replay for normal function tools. Gate the recovery on the item type so it only applies to custom_tool_call items, which are the ones that store their payload in input. * fix(responses-bridge): default missing function_call arguments to empty string With input recovery scoped to custom_tool_call items, a plain function_call input item without an arguments key left raw_arguments as None and the downstream str() turned it into the literal string None. Coerce to an empty string instead, matching the pre-bridge behavior. --------- Co-authored-by: duanhongyi <duanhongyi@doopai.com>
…ing it at read time (BerriAI#32288) * feat(mcp): persist oauth2_flow explicitly on create instead of inferring it at read time The UI create payload never carried oauth2_flow, so every UI-created oauth2 server persisted a null flow and relied on _resolve_oauth2_flow's field-shape inference at registry build. That inference cannot tell a DCR-registered interactive server (client creds + token_url, no persisted authorization_url) from an M2M server unless endpoint discovery succeeds first, and the dashboard cannot reproduce it at all because credentials are redacted in responses The create form now persists the selected flow for oauth2 servers: authorization_code for Interactive (PKCE), client_credentials for M2M. The REST create endpoints stamp an omitted oauth2_flow server-side with the same discriminator the legacy inference uses, run at write time where the payload carries plaintext credentials, so the decision is made once with full information and stored. Applied to the admin create, the BYOM submission, and the temporary session-server endpoints The edit form derives its flow display from oauth2_flow instead of token_url presence (token_url is present on authorization_code servers too, so it cannot distinguish M2M) and deliberately never writes oauth2_flow: it has no flow selector, so a write from edit could only erase an explicit value, including the authorization_code stamp the DCR flow persists. Regression tests pin all of this down Second step of persisting oauth2_flow at every write site so the legacy inference can eventually be deleted; the backfill for existing null rows lands next * refactor(mcp): name the create-time flow stamp for its fallback-only contract stamp_omitted_oauth2_flow with a dedicated explicit-value early return and the shape check renamed to has_m2m_shape, so the precedence (caller's oauth2_flow always wins, inference only fills an omitted field) reads directly off the code
…table within a request (BerriAI#32299)
…n support (BerriAI#32274) * fix(llm_http_handler): send dict transcription request data as a JSON body httpx form-encodes dicts passed via data= and silently ignores json=, so the generic audio transcription path never actually sent a JSON body. No provider hit this before; JSON-body speech APIs need it. * feat(vertex_ai): add Google Cloud Speech-to-Text Chirp 3 transcription support Adds a VertexAIAudioTranscriptionConfig wired through ProviderConfigManager so vertex_ai/chirp_3 works on /v1/audio/transcriptions (sync and async) via the Speech-to-Text v2 recognize API. Auth reuses the standard Vertex credential resolution (vertex_project/vertex_location/vertex_credentials or ADC); the location defaults to the us multi-region since chirp_3 is only served from the us and eu multi-regions, and non-global locations use the regional <location>-speech.googleapis.com host. Maps language to languageCodes (auto language detection by default), joins all result alternatives into the transcript, and tracks cost from totalBilledDuration with a vertex_ai/chirp_3 price entry at Google's published $0.016/min. * fix(vertex_ai): map bare ISO-639-1 language codes to BCP-47 for Speech-to-Text OpenAI clients send language codes like "en", which Google rejects with 400 ("not supported by the model chirp_3 in the location us"); Speech-to-Text wants region-qualified BCP-47 like "en-US". Adds a shared normalize_transcription_language_to_bcp47 helper in audio_utils (NVIDIA Riva's transcription config already hand-rolled the same table privately) that maps common bare codes and passes region-qualified ones through, and applies it in the Vertex transcription request. Also narrows the response JSON parse guard to ValueError. * fix(vertex_ai): drop zero output_cost_per_second so chirp_3 cost tracking works cost_per_second prefers output_cost_per_second whenever it is not None, so the 0.0 in the chirp_3 entry priced every transcription at $0.00 instead of using input_cost_per_second. Remove it from both cost maps and pin the behavior with a regression test computing 18s of chirp_3 audio to ~$0.0048. * fix(vertex_ai): validate client-controllable location to prevent SSRF in Speech-to-Text get_complete_url interpolated vertex_location straight into the request host, and vertex_location is client-controllable on the proxy (it flows from the request body and is not on the request-body blocklist). An authenticated caller could send vertex_location="attacker.example/" to point the host at their own server, so the proxy would POST the audio plus its admin-minted Google bearer token and x-goog-user-project header to the attacker, exfiltrating a cloud-platform-scoped OAuth token minted from the admin's credentials. Factor the location validation the rest of vertex_ai already applied in get_vertex_base_url (^[a-z][a-z0-9-]*$ plus the global allowance) into a shared validate_vertex_location helper in common_utils and call it from both the chat host builder and the new speech host builder. Invalid locations now raise a 400 VertexAIError instead of building a host. Also reject vertex_project values that carry URL-structural characters, since it lands in the URL path. Regression tests assert on the parsed netloc so the security property is pinned: valid locations always resolve to a *speech.googleapis.com host and injection inputs are rejected. * fix(vertex_ai): reject unsupported transcription response_format values instead of silently ignoring
…lback Drive the real parallel_request_limiter through _pre_call_with_fallbacks for the LIT-3890 customer scenario: a key-level model_tpm_limit raises ProxyRateLimitError from the pre-call hook and the configured gateway fallback serves the request instead of returning a 429. Unlike the existing tests, this exercises the actual limiter rather than a hand-built error. Also switch the new _pre_call_with_fallbacks return annotation to builtin tuple to stay within the ruff UP006 strict-rule budget.
…erriAI#32290) * feat(mcp): startup backfill stamping oauth2_flow on legacy null rows Rows created before the write-side stamps carry a null oauth2_flow and rely on read-time field-shape inference, which cannot tell a DCR-registered interactive server (client creds + token_url, no persisted authorization_url) from an M2M server unless endpoint discovery succeeds first; on a transient discovery failure those servers flip to client_credentials for that registry load The backfill classifies each null oauth2 row once, at rest, ordered by signal strength: per-user token rows (only the interactive flow mints them, so this is definitive and catches the DCR-trap cohort), then a persisted authorization_url, then a persisted registration_url (DCR implies interactive; this covers registered-but-never-signed-in rows), then the M2M credential shape mirroring the legacy inference, else the interactive default that matches how needs_user_oauth_token treats a null flow. Every stamp is logged with the rule that fired and written with updated_by=oauth2_flow_backfill for auditability Runs in _init_mcp_servers_in_db before the registry load so the first build of the boot classifies from the column, is isolated so a failure cannot block server loading, and is idempotent: a healed fleet exits after one indexed query. This unblocks deleting the read-time inference for DB rows in the follow-up Third step of the oauth2_flow persistence sequence, after BerriAI#32283 and BerriAI#32288 * fix(mcp): backfill leaves the ambiguous M2M shape unstamped instead of guessing client_credentials The credential shape (client_id + client_secret + token_url, no interactive signal) is shared by real M2M servers and DCR-registered interactive servers nobody has signed into: the DCR persist writes creds and token_url but not authorization_url or registration_url. Stamping client_credentials from that shape permanently mislabeled the interactive cohort, and once explicit the value is authoritative, so per-user traffic would run on the proxy's stored client credential with no discovery rescue and no backstop (it only guards null rows) The backfill now stamps only what it can prove. Interactive signals keep stamping authorization_code; the ambiguous shape is left null with an actionable warning naming the server and the fix (set oauth2_flow via the dashboard or PUT /v1/mcp/server). A true M2M row keeps working per-request through the security backstop while the warning nags; an interactive row keeps its Authorize button (null renders interactive), and one completed sign-in creates the per-user token that stamps it authorization_code at the next boot. Mirrors the config-level rule: M2M is asserted by a human, never guessed Raised by review on the PR * perf(mcp): batch the backfill stamps into one update_many per flow value The per-row update loop issued one DB round-trip per legacy row at startup; rows sharing a stamped value now go out as a single update_many, so the DB cost is constant in fleet size. Per-row logging keeps the rule that fired for each server Raised by review on the PR * fix(mcp): backfill stamps only rows still null at write time and counts only real OAuth token rows as sign-in proof Two review findings. The batched update_many matched on server_id alone, so an explicit oauth2_flow set between the backfill's read and its write (an admin PUT or a sign-in's DCR stamp landing in the boot window) would be overwritten with the inferred value; the where clause now also requires oauth2_flow to still be null, so an explicit value can never be clobbered under any interleaving And the per_user_tokens rule counted any LiteLLM_MCPUserCredentials row as proof of an interactive sign-in, but that table doubles as BYOK storage for user-supplied API keys; a BYOK-flavored row would have stamped an M2M-shaped server authorization_code. The rule now counts only rows whose payload decodes as a type oauth2 token via the existing _decode_oauth_payload discriminator, so bare keys, undecodable rows, and stale leftovers from a BYOK-to-oauth2 auth switch prove nothing Raised by review on the PR
…hes in proof of fix (BerriAI#32303)
…and route permissions (BerriAI#32300) * test(e2e): add management suite covering key/team/user/org lifecycle and route permissions * test(e2e): decouple the enforcement-flip assertion from upstream health Polling for a 200 on the newly-allowed model required it to be a routable, healthy upstream, which is not the contract under test; poll until the key_model_access_denied 403 lifts instead, excluding 401 so a revoked key cannot read as success. Also document that the delete test's deferred teardown firing on an already-deleted key is deliberate: cleanup must survive the test failing before the in-body delete, and the repeat delete is a warn-free no-op (the proxy answers 404 No keys found) * test(e2e): inline the management suite's model and tpm literals * test(e2e): drop the models_mgmt suite line from the folder list * test(e2e): write the tpm limit as a plain integer literal
…ssages and /v1/responses (BerriAI#32284) Streaming pass-through for native Anthropic /v1/messages and the /v1/responses streaming iterator never set logging_obj.completion_start_time, so _success_handler_helper_fn fell back to completion_start_time = end_time. Downstream TTFT consumers (Prometheus, OTEL, Langfuse, Admin UI, spend logs completionStartTime) then reported time-to-first-token equal to total request duration. Stamp completion_start_time on the first chunk in PassThroughStreamingHandler. chunk_processor and BaseResponsesAPIStreamingIterator._process_chunk, mirroring CustomStreamWrapper for /chat/completions. Resolves LIT-4185 Co-authored-by: yucheng <yucheng@yuchengs-MBP.localdomain>
…ader (BerriAI#32282) The x-litellm-semantic-filter-tools response header was sliced mid-name at MAX_MCP_SEMANTIC_FILTER_TOOLS_HEADER_LENGTH with a trailing "...", so the admin UI test panel rendered the last selected tool name chopped. Truncate the CSV at a tool name boundary instead so the header only ever carries complete names, and note in the test panel how many selected tools did not fit in the header
…n batch cost job (BerriAI#32307) * fix(batches): price anthropic passthrough message batches correctly in batch cost job Anthropic message batches created via the /anthropic passthrough were never cost tracked. The CheckBatchCost job fetched batch results from the Files API (POST /v1/files/msgbatch_.../content), which Anthropic rejects with "File id must have file_ prefix"; the error response was silently wrapped as file content, parsed as zero successful rows, logged as a $0 aretrieve_batch spend row, and the job was marked batch_processed=true so the $0 was permanent. Route msgbatch_ file ids to GET /v1/messages/batches/{id}/results in the anthropic files transformation, raise on HTTP error status in retrieve_file_content instead of returning the error body as content, parse Anthropic's results JSONL shape (result.type == "succeeded", result.message.usage with cache creation/read tokens) in batch_utils, price cache creation tokens at cache_creation_input_token_cost in the batch cost fallback (50% batch discount preserved for base input, cache reads, cache writes, and output), and leave the managed object row unprocessed when cost tracking fails so a later poll retries instead of permanently recording $0. * fix(batches): carry cache token details into aggregated anthropic batch usage
…t-fallbacks fix(proxy): trigger gateway fallbacks on local rate limit errors
FOCUS v1.2 has no standard column for LLM token counts, and the shared FocusTransformer used by every destination (Mavvrik, Vantage, CloudZero) drops prompt_tokens/completion_tokens even though the source query already selects them. Merge the two counts into the existing Tags JSON column, which is the spec's own escape hatch for non-standard fields, inside the Mavvrik-only export path so no shared transformer changes.
Author
|
Closing in favor of a PR targeting BerriAI/litellm:litellm_oss_staging directly, matching the convention used for the earlier mavvrik-metrics-marker-v3 fix (PR BerriAI#31068). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
14 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relevant issues
Mavvrik FOCUS exports do not carry LLM token usage (
prompt_tokens,completion_tokens), even though LiteLLM's own DB and UI have this data. FOCUS v1.2 has no standard column for token counts, so the sharedFocusTransformer(used by every FOCUS destination: Mavvrik, Vantage, CloudZero) drops these two columns duringtransform(), even thoughdatabase.py's query already selects them fromLiteLLM_DailyUserSpend.Pre-Submission checklist
Type
Bug Fix
Changes
FocusTransformer.transform()(shared core code, not touched by this PR) builds the final FOCUS-shaped frame via an explicit columnselect(...)that enumerates ~35 output columns.prompt_tokens/completion_tokensare not in that list, so they are silently dropped between the raw query result and the exported CSV. This is true for every destination built on this transformer, not just Mavvrik.Rather than changing the shared transformer (which would affect Vantage and CloudZero too), this PR merges the two token counts into the existing
TagsJSON column, entirely insideMavvrikFocusLogger._export_window(), a Mavvrik-only file.Tagsis FOCUS v1.2's own escape hatch for non-standard fields, and core already uses it to carryteam_id,model,custom_llm_provider, etc._export_window()holds both the pre-transform frame (data, still has the token columns) and the post-transform frame (normalized, tokens already dropped) at the same point, right before serialization. The new_with_token_tags()helper zips the two frames by row position (transform()only adds/renames columns and never filters or reorders rows, so a 1:1 row correspondence is guaranteed) and addsprompt_tokens/completion_tokensas extra string keys into each row's existingTagsJSON.No changes to
litellm/integrations/focus/database.pyorlitellm/integrations/focus/transformer.py.Screenshots / Proof of Fix
E2E verified on the QA VM using a native LiteLLM proxy (not Docker) against a dedicated Postgres database (
litellm_tokentest), with real Azuregpt-4o-minicompletions and a real Mavvrik sandbox connection.FOCUS_CRON_OFFSETwas used to fire the daily export job a few minutes after startup; test rows were backdated one day in Postgres so the daily window (which only exports strictly-past dates) would pick them up.Before fix, commit
9076c3334760d4c4d6be4b2555c874e9d49c2733(branch base, unpatchedmavvrik_focus_logger.py)3 real completions produced this Postgres row:
Export log:
Resulting
Tagsvalue in the exported CSV, no token counts despite Postgres having them:After fix, commit
f3e2aba6abad405aba263a9904334a214646e821(this branch's HEAD)3 new real completions produced this Postgres row:
Export log:
Resulting
Tagsvalue in the exported CSV, downloaded from GCS and decompressed. Token counts now present and matching Postgres exactly:All other FOCUS columns (
BilledCost,ConsumedQuantity,ChargePeriodStart/End, etc.) are unchanged between the two runs, confirming the fix is additive toTagsonly.