Sync with upstream BerriAI/litellm; reassess fork patches - #6
Merged
Conversation
…ion_url is set manually
…shared DataTable Rewrites the three Batch A tables from hand-rolled TanStack + tremor renderers into thin DataTable consumers with a separate ColumnDef module each, matching the Guardrails and Tags migrations. Row actions move into a per-row overflow menu (edit/copy/delete for vector stores; copy for everyone plus admin-gated delete for prompts and skills). The vector stores parent gains an isLoading flag resolved on every exit path so the table shows the shared skeleton instead of flashing the empty state. Table files are renamed to PascalCase and stale eslint bulk-suppressions for the rewritten files are pruned.
Only the name cell and the overflow menu act on a row, matching the unified table pattern; the previous table navigated on any row click
…single-server REST statuses
The aggregate MCP tools/list absorbed every per-server failure (upstream 401/403/5xx, timeouts,
network errors) into that server contributing zero tools, making a broken upstream indistinguishable
from a healthy server with no tools; the single-server REST list masked the same failures as
{"tools": [], "error": null, "message": "Successfully retrieved tools"}
Phase 2 of the MCP error-handling framework (LIT-4419): the manager fetch hops now raise a
classified MCPServerListError (faults/list_outcomes.py: total classifier, frozen outcome values)
instead of returning [], and each boundary applies the relay-vs-absorb policy matrix. The aggregate
keeps serving the healthy subset but records each server's outcome, surfaced on the tools/list
result _meta under litellm.ai/server_outcomes (the SDK passes a ListToolsResult through unwrapped)
and in spend logs as per_server_list_outcomes. Single-server REST requests relay truthful statuses
(unreachable/upstream_error 502, timeout 504, internal 500) and access denials now surface as real
403s instead of 200 unexpected_error bodies; upstream 403s surface through MCPUpstreamAuthError
like 401s. Outcome wire values carry category and status code only, never upstream prose
Resolves LIT-4421
…a healthy empty server A cancelled fetch absorbed to [] made that server contribute ServerListOk(tool_count=0), the exact healthy-but-empty impostor this change removes. Cancellation stays suppressed (the pre-existing choice); it now carries an internal fault so outcomes stay truthful
…ding the upstream response
…m listing failures Both review findings shared one root cause: two exception-tree walkers with drifted semantics. _extract_upstream_auth_failure walked the incidental __context__ chain before explicit causes, so a 403 raised while handling the causal 401 could shadow it; and the generic _get_tools_from_server arm classified without extracting the challenge, so a nested 401 at client-build time surfaced without the WWW-Authenticate the client needs. upstream_auth_challenge and raise_classified_list_failure in faults/list_outcomes.py are now the single traversal and the single choice-point; both fetch arms and _extract_upstream_auth_failure (also serving tool calls and the connect-time probe) delegate to them, with dcr_bridge challenge suppression as a parameter so it holds on every path. The stale _fetch_tools_with_timeout docstring describing the pre-change 403 absorb is rewritten to the actual contract: 403 relays with its own status, an upstream-sent challenge relays verbatim per RFC 6750 insufficient_scope, and a challenge is only ever fabricated for a challenge-less 401
…esponses (LIT-3787) (BerriAI#33234)
token_storage_ttl_seconds previously won outright over the token's expires_in, so a TTL longer than the token's lifetime kept the Redis fast path serving an expired bearer until eviction, while the stored refresh_token sat unused because refresh only runs on the DB read-through The configured TTL is now capped at expires_in minus the expiry buffer. Shorter TTLs and servers without the field behave exactly as before, and the TTL still applies verbatim when the upstream reports no expires_in. The dashboard tooltips on the create and edit forms are updated to describe the capped behavior
…fig (BerriAI#33251) * feat(router): resolve auto-router routing plugins from proxy YAML config Router(plugins=[...]) was Python-SDK constructor only, so proxy/YAML users had no way to configure it, and the merged pipeline narrowed candidates from the outer model alias rather than the auto-router's actual tier pool, making it a no-op for auto_router deployments. Add complexity_router_config.plugins (dotted-path strings resolved via get_instance_fn, the same convention litellm_settings.callbacks uses) and run the resolved plugins against ComplexityRouter's tier pool at every model-pick site, so a policy plugin narrows what get_model_for_tier actually returns instead of the outer alias list. adaptive=True with plugins set now raises at config validation instead of silently ignoring the plugins, since the bandit selector doesn't consume narrowed pools yet. Also fixes a latent bug in Router._generate_model_id: it json.dumps every litellm_params dict value to build a deployment hash id, which crashed once a live plugin object could land inside complexity_router_config. * fix(router): use stable class name, not object repr, in model-id json fallback json.dumps(v, default=str) on a litellm_params dict containing a live RoutingPlugin instance fell back to object.__repr__'s default <module.Class object at 0x...>, embedding the instance's memory address. _generate_model_id's hash (and therefore the deployment id) changed on every process restart/hot-reload for any deployment with complexity_router_config.plugins configured, defeating the function's own "consistently generate the same id" contract and orphaning anything keyed on that id across restarts (e.g. Redis-backed per-deployment state). Use the plugin's fully-qualified class name instead, which is stable across restarts. * test(router): cover _json_default_stable_id for router_code_coverage gate router_code_coverage.py's AST scanner requires every router.py function be called by name somewhere in tests/, and flagged the new _json_default_stable_id helper from the previous commit. * fix(router): close two routing-plugin policy-bypass gaps flagged by Veria AI Session-affinity pin shortcut: async_pre_routing_hook returned a session's first-turn pinned model on every later turn without ever re-running it through the plugin pipeline, so a policy plugin (e.g. a budget cap crossed mid-session) was only enforced on turn one. Now the pin shortcut is disabled whenever plugins are configured, so every turn re-runs _classify_and_route (and therefore the plugins). Plugin resolution validation: get_instance_fn accepts any dotted path and returns whatever object it finds there, so a misconfigured complexity_router_config.plugins entry passed proxy startup silently and only surfaced as a confusing AttributeError on the first request that reached the plugin pipeline. Extracted the resolution logic into resolve_complexity_router_plugins() and added an isinstance(..., RoutingPlugin) check that fails proxy startup immediately with a clear error instead. * fix(router): raise instead of falling back to default_model on empty plugin-narrowed tier default_model was never checked against the configured plugins, so it functioned as an unconditional escape hatch around whatever policy a plugin enforces -- a tenant/budget plugin narrowing a tier to zero candidates could still be bypassed by the fallback. Drop the fallback entirely for this path; a plugin narrowing to zero is a policy decision, not something to route around, matching the fail-closed behavior the Router-level plugin pipeline already uses for the same situation. Flagged by Veria AI on PR BerriAI#33251. * style: ruff format complexity_router.py * style(proxy): use modern str | None instead of Optional[str] in resolve_complexity_router_plugins * fix(router): stop default_model short-circuit from skipping plugins on no-user-message path self.config.default_model or await self._pick_model_for_tier(...) -- Python's `or` short-circuits on a truthy default_model, so _pick_model_for_tier (and therefore the plugin pipeline) never ran at all for the no-user-message path whenever default_model was configured. A tenant/budget plugin's decision was silently bypassable this way even after the other two policy-bypass fixes, since this call site had a different shape from the other three pick sites. Removed the short-circuit; falls through to _pick_model_for_tier -> get_model_for_tier, which already checks the MEDIUM tier before default_model -- the same priority every other call site uses. Flagged by Veria AI on PR BerriAI#33251. * fix(router): address Greptile findings on the plugin-bypass fixes Preserve default_model-first priority in the no-user-message path when no plugins are configured, instead of unconditionally flipping to the MEDIUM tier -- the plugin-bypass fix must not silently change model selection for the (much larger) population of users who don't use plugins at all. Gated on self.config.plugins, matching the pattern already used elsewhere in this PR, per CLAUDE.md's guidance against backwards-compat flags when a plain conditional does the job. Also close a gap in the plugin validation added earlier: @runtime_checkable only checks that `run` exists as an attribute, not that it's a coroutine function, so a synchronous `def run(self, context)` passed isinstance(resolved_plugin, RoutingPlugin) at startup and only failed at request time with a confusing TypeError. Added an inspect.iscoroutinefunction check. Both flagged by Greptile on PR BerriAI#33251.
…sage and status (LIT-4179) (BerriAI#33304) * test(e2e): failed request error span carries the full untruncated message and status Covers logging.otel.failure.exports_metric on chat_completions: a request that fails at the provider (invalid upstream key deployment) must export one complete trace whose gen-AI span carries the LIT-4179 error contract, declared as one reviewable payload (EXPECTED_ERROR_SPAN_ATTRIBUTES) plus an untruncated error.message proven by parsing the embedded provider error JSON back out of the attribute. The root SERVER span must record the 401 the client received. Adds STORE_MODEL_IN_DB to the compose stack so /model/new works locally, which the suite's model-registering tests already assume * test(e2e): clean failure diagnostics on the error-span contract per review A truncated error.message with missing braces now fails with a readable assertion instead of an unhandled ValueError, an unparseable embedded JSON fails via pytest.fail with the truncation context, and the retry loop now asserts the upstream provider failure was actually observed so a fresh-key propagation deadline cannot masquerade as a trace-export failure * test(e2e): pin the full error attribute set including the litellm.provider.error keys The LIT-4179 fix restored error.message/code/stack_trace/llm_provider; a later refactor (BerriAI#32591) moved the litellm-specific keys under litellm.provider.error.*, which the initial contract missed. The payload now pins error, error.type, otel.status_code, litellm.provider.error.code=401, and litellm.provider.error.llm_provider=anthropic exactly, plus non-empty litellm.provider.error.stack_trace and the untruncated error.message * test(e2e): author the error-span test docstring
…itellm_fix_stream_reset_empty_200
…-table-tags-d2e4f0 refactor(ui): migrate tags table onto shared DataTable
…ns skew (BerriAI#33309) * fix(proxy): tell outdated litellm CLIs to upgrade when CLI SSO login id is legacy sk- format * fix(cli): surface server error detail when SSO login polling fails and stop on permanent 4xx * fix(cli): exhaustive, actionable error handling across the CLI SSO login flow
…ual authorization_url Discovery is rooted at the MCP resource, so a compromised upstream can advertise an attacker-run authorization server. When authorization_url is manually configured and another field is blank, the per-field merge would combine the trusted authorize endpoint with the advertised token_url, and the gateway would redeem authorization codes (with the stored client secret and PKCE verifier) at that endpoint, then persist it. Discovered token_url and registration_url are now accepted only when the same metadata document advertises an authorization_endpoint matching the configured value (scheme+host+path). Scope backfill is unaffected. Applies to both the DB and config build paths.
… elide default port The corroboration check belongs to adopting a token_url from any non-manual source, not to discovery alone. Carry-forward is the other such source: it copied a prior registry entry's token_url/registration_url onto a rebuild whose authorization_url had been re-pointed to a different server, reviving an uncorroborated token endpoint the discovery gate would reject. Both sites now share one predicate, _endpoints_corroborate_authorization_url: previous endpoints carry forward only when the previous authorization_url corroborates the authorize endpoint the build will use (absent -> the previous one is adopted too, a consistent group; else it must match). Endpoint comparison now elides the default port so :443 and formatting-only differences still match.
Register bedrock_mantle/openai.gpt-5.6-{sol,terra,luna} with
mode=responses, /v1/responses in supported_endpoints, and
use_openai_responses_path so the data-driven gate routes them through
BedrockMantleResponsesAPIConfig on the openai/v1 Mantle base path.
Without these entries the models fall through to chat-completions
emulation, which the Mantle endpoint rejects.
Pricing and context window sourced from the AWS Bedrock pricing page
and the GPT-5.6 model cards (272K context, OpenAI first-party rates
with the 1.1x in-region US uplift, 90% cached-input discount, 1.25x
cache write).
…tion server, scopes included Provenance is a property of the whole discovered metadata document, not per field. Waving scopes through while gating endpoints left a second inflation vector: a compromised upstream advertises broad scopes via the resource metadata (RFC 9728 / WWW-Authenticate), the gateway requests them from the trusted authorization server, and the resulting token flows back to the upstream. Both that and the token-endpoint mix-up are now one rule: when authorization_url is admin-pinned, discovered token_url/registration_url are kept only if the document corroborates the pin, and scopes come from the authorization server's own scopes_supported (a new authorization_server_scopes field, trusted tier) rather than the resource-advertised scopes. A document that does not corroborate backfills nothing. Blank (empty-string) authorization_url is treated as unpinned so the merge and the gate agree. Carry-forward, the other non-manual source, drops the same three across an authorization_url change.
…y points A whitespace-only authorization_url was truthy to the row/config merges and has_all check but blank to the corroboration gate, so discovery and carry-forward adopted token_url/registration_url/scopes as if unpinned while the broken whitespace value was still used for redirects. Rather than add another strip() at each site, the pinned authorization_url/token_url/ registration_url are normalized once per build path (DB and config) via _blank_to_none, so the merge, has_all gate, discovery gate, persist hook, and carry-forward all see a single notion of blank. Empty and whitespace pins now behave identically to an omitted field.
…gpt_5_6 feat(bedrock_mantle): add GPT-5.6 sol/terra/luna to model cost map
…backend-deps-180b66 chore(codeowners): exempt generated schema.d.ts from UI ownership
… deployments (BerriAI#31592) * feat(proxy): push-based OTLP billable-request metering for enterprise deployments Adds opt-in, license-gated metering that counts 2xx HTTP requests to LLM inference, MCP, and A2A endpoints and exports them over mutual TLS to a global OpenTelemetry Collector for request-based billing. A pure ASGI middleware (BillableRequestMetricsMiddleware) classifies each request by route and records one count per 2xx response via an injected recorder. The recorder (BillingMetricsRecorder) owns a dedicated OTEL meter provider and an OTLP/gRPC exporter authenticated with client certificates, kept isolated from the global meter provider so a customer's own OTEL metrics are untouched. The recorder is built only when a valid LITELLM_LICENSE is present and the cert material is configured; otherwise the middleware is a transparent pass-through. Deployment identity rides on the mTLS client certificate rather than the payload, so the secret license key is never sent as an attribute or header; only the license org id travels as a resource attribute for cross-checking. Resolves LIT-4089 * fix(proxy): align billable-request metering with the global collector - switch the exporter to OTLP/HTTP with a TLS client certificate. The collector front end terminates mutual TLS and validates the client cert against our CA; server verification uses the system trust store, so the CA env var is now an optional override for private collectors - resolve the metrics recorder on the first request via a factory instead of at import time, so deployments that provide the license and cert env vars through the YAML config's environment_variables export correctly - close the metering bypass: classify /images/edits, /images/variations, /v1/messages, /v1/videos, video remix, /v1/ocr and Gemini generateContent as billable, and gate LLM routes to POST so GET reads (list videos, fetch a response) do not bill. Verified live: the collector count matches the UI usage page successful_requests exactly, with failures excluded on both sides * fix(proxy): wrap enterprise billing import in try-except per code-quality gate The check_unsafe_enterprise_import gate requires every import from an enterprise-pathed module to be guarded. Annotate the factory with the middleware's BillingRecorder protocol so no enterprise type import is needed at type-check time * chore: satisfy strict lint gates in billing modules - builtin generics per UP006 (dict/tuple instead of typing.Dict/Tuple) - noqa the deliberate blind catch that keeps metering from breaking startup - sort proxy_server import blocks split by the guarded enterprise import * fix(proxy): bill provider passthrough, search, and rag routes Route-inventory audit against LiteLLMRoutes.llm_api_routes found more SpendLogs-producing surfaces the classifier missed: provider passthrough (/bedrock, /vertex-ai, /cohere and the rest of mapped_pass_through_routes), /v1/search and vector-store search, and the rag ingest/query routes. All are counted by the dashboard usage page, so missing them undercounts billing. The passthrough prefix list is read from LiteLLMRoutes so new providers are picked up without touching this module. /langfuse is excluded: it forwards observability traffic and writes no SpendLogs row. Known limitation recorded in the PR: /v1/realtime is a websocket flow the HTTP middleware does not see * fix(proxy): bill MCP and A2A requests by protocol transport routes only The billable-request classifier matched the whole /v1/mcp prefix, so management and discovery reads such as GET /v1/mcp/tools and GET /v1/mcp/server counted as billable MCP requests, while real MCP tool calls on the /{server}/mcp and /toolset/{name}/mcp aliases were missed because their route handlers rewrite the ASGI scope only after this middleware has already classified the original path. Classify MCP by the concrete transport surface (the /mcp streamable-HTTP and SSE sub-app plus the single-segment server and toolset aliases) and exclude the /v1/mcp management API. Apply the same shape to A2A, which had the identical issue: only the /message/send invoke route bills, not /v1/a2a/discover or the .well-known agent-card reads. * fix(proxy): harden billable-request classification and recorder lifecycle Exact-match Anthropic /v1/messages so OpenAI Assistants thread-message routes no longer bill, add Google Interactions create routes, guard recorder.record() so a broken exporter can never fail a served request, lock lazy recorder resolution against concurrent first requests, and disable metering on empty-string env config instead of accepting a blank endpoint * chore(ui): regenerate eslint metrics after staging merge * docs(proxy): state the lower-bound billing contract in middleware comments * fix(proxy): bill mcp-rest tool calls and bare a2a agent invokes POST /mcp-rest/tools/call executes a tool and fires the same MCP spend logging as the /mcp transport, and POST /a2a/{agent_id} is the JSON-RPC invoke route whose method (message/send or message/stream) travels in the body; both returned 2xx without being recorded * fix(proxy): flush billable-request counts on proxy shutdown PeriodicExportingMetricReader buffers up to one export interval of counts; without a final flush every restart silently dropped them. The factory registers the recorder it builds and proxy_shutdown_event pops and flushes it, bounded by a 5s timeout so a dead collector cannot stall shutdown * fix(proxy): stop billing bare a2a task RPCs and close the shutdown race POST /a2a/{agent_id} multiplexes JSON-RPC methods off the request body. Only message/send and message/stream write a SpendLogs row; tasks/get, tasks/cancel and the pushNotificationConfig RPCs are forwarded upstream and write none. Classifying the bare path as billable counted those task RPCs and pushed the metric above the dashboard's successful-request count. Since a path-only classifier cannot read the body, the bare route no longer bills; the explicit /message/send routes still do. Missing a bare-path invoke undercounts, which is the only direction this metric is allowed to drift. The /mcp transport keeps billing every method because its list path logs a SpendLogs row too. The billing middleware also sat outside InFlightRequestsMiddleware, and it records after the inner app returns. A request could therefore be counted as drained while its record() had not yet run, letting proxy_shutdown_event flush and stop the exporter underneath it. Registering it before the in-flight tracker nests it inside, so wait_for_drain covers the record * test(proxy): stub the OTLP exporter in the recorder-build test test_premium_with_full_config_builds_recorder built a real MeterProvider, so the shutdown flush resolved collector.example and opened a TLS connection from a unit test. The exporter is now stubbed, and a getaddrinfo spy asserts nothing resolves the collector host so the stub cannot be quietly dropped later * fix(helm): truncate the helm.sh/chart label to 63 bytes Kubernetes caps a label value at 63 bytes and .Chart.Version is unbounded. CI publishes branch builds as 0.0.0-branch-<branch>-<sha>, so helm.sh/chart rendered as a 64 byte value and the API server rejected every labeled resource with "must be no more than 63 bytes", including the migrations Job. The litellm-helm chart already guards this through a litellm.chart helper; this adds the same helper here. Swept the rest of the chart for label and name values built from unbounded input. .Chart.Version appeared only in this label. The remaining candidates all derive from .Release.Name, which helm itself caps at 53 characters, so they cannot overflow; three of them are selector labels feeding immutable Deployment matchLabels, where adding trunc would risk churn for no gain. They are left alone deliberately. Verified with a new helm-unittest suite, tests/chart_label_tests.yaml, which overrides chart.version per test: helm unittest -f 'tests/*.yaml' helm/litellm # 13 passed helm unittest -f 'tests/*.yaml' helm/litellm-helm # 54 passed The truncation cases fail against the previous helper. Reproduced the original overflow by rendering with the real branch version and measuring the label: helm template rel helm/litellm -f helm/litellm/tests/values/required.yaml \ | grep helm.sh/chart # 64 bytes before, 63 after * feat(proxy): accept inline PEM for the billing-metrics mTLS credentials LITELLM_BILLING_METRICS_CLIENT_CERT, _CLIENT_KEY and _CA_CERT took a filesystem path. ECS injects Secrets Manager values as environment content and cannot mount them as files, so a licensed deployment there could not turn metering on. Each variable now takes either a path or the PEM itself. Inline PEM, detected by the "-----BEGIN" prefix, is written once when the recorder is built into a 0700 temp dir as a 0600 file, and the config points at that path. The OTLP exporter still only ever sees paths. A write failure disables metering through the existing failure-as-None path rather than raising, and path-valued variables are passed through untouched, so nothing changes for deployments that mount files. The mixed case works too: mount the CA, inject the client credentials * feat(helm): add first-class billingMetrics values to the componentized chart Turning enterprise billable-request metering on meant hand-rolling the env vars and the cert volume through gateway.extraEnv and gateway.volumes. This adds a top-level billingMetrics block, off by default, consumed only by the gateway since that is the component serving billable traffic. When enabled it renders LITELLM_BILLING_METRICS_ENDPOINT plus the two cert paths and mounts secretName read-only at /etc/litellm/billing-mtls. caSecretName is optional and only needed for private collectors whose server certificate is not on the public web PKI; when set it mounts at /etc/litellm/billing-mtls-ca and adds the CA env var. exportIntervalMs is passed through only when set. Enabling without secretName or with an empty endpoint fails the render with a named message rather than producing a gateway that silently never exports. The generic gateway.volumes, gateway.volumeMounts and gateway.extraEnv paths are untouched and still compose with this, so existing overlays keep working. The chart has no values.schema.json and no README, so there is nothing further to update. Verified with a new helm-unittest suite: helm unittest -f 'tests/*.yaml' helm/litellm # 23 passed helm unittest -f 'tests/*.yaml' helm/litellm-helm # 54 passed * feat(terraform): billing-metrics variables for the aws and gcp templates * feat(helm): add billingMetrics values to the classic chart The componentized chart just gained a first-class billingMetrics block; this mirrors it in litellm-helm so enabling enterprise billable-request metering no longer means hand-rolling the env vars and the cert volume through envVars and volumes. When enabled the proxy Deployment renders LITELLM_BILLING_METRICS_ENDPOINT plus the two cert paths, and mounts secretName read-only at /etc/litellm/billing-mtls. secretName defaults to litellm-billing-metrics-mtls, the conventional name, so enabling the block is enough once that Secret exists. caSecretName is optional and only needed for private collectors whose server certificate is not on the public web PKI; when set it mounts at /etc/litellm/billing-mtls-ca and adds the CA env var. exportIntervalMs is passed through only when set. The env entries render after envVars and extraEnvVars, so a user-supplied LITELLM_BILLING_METRICS_ENDPOINT cannot silently redirect the export under Kubernetes last-wins duplicate-env semantics; this is the same ordering the migrations Job relies on for DISABLE_SCHEMA_UPDATE. Enabling with an emptied secretName or endpoint fails the render with a named message rather than producing a proxy that silently never exports. The generic volumes, volumeMounts, envVars and extraEnvVars paths are untouched and still compose with this, so existing overlays keep working. The chart has no values.schema.json; README parameters and a setup section are updated. helm unittest -f 'tests/*.yaml' helm/litellm-helm # 68 passed (54 + 14 new) helm lint helm/litellm-helm # 0 failed * test(helm): pin that the migrations job never mounts the billing cert The componentized chart's suite asserts the backend Deployment stays clear of the billing wiring, since only the gateway serves billable traffic. The classic chart has no backend, but it does have a second pod: the migrations Job, which renders its own env from envVars and extraEnvVars. Nothing today wires the billing include into it, and nothing stopped a future edit from doing so. Asserts absence of the env, and that the Job grows no volumes or volumeMounts at all. Both are notExists rather than notContains because the Job renders neither key by default, so a notContains would fail on an unknown path instead of checking the absence it looks like it is checking. * fix(helm): meter the backend too, it serves the MCP transport Scoping billingMetrics to the gateway was wrong. Applying each component's own route allowlist to the proxy app shows the split is 75 billable routes on the gateway and one on the backend: /{mcp_server_name}/mcp, the named-server MCP transport, which writes a SpendLogs row on success. Metering only the gateway would have silently dropped every MCP transport call from the counter, an undercount proportional to a customer's MCP traffic. The backend deployment now renders the same env and mounts the same read-only cert secret. The migrations job still gets neither; it runs prisma and serves no traffic, and a test pins that. helm unittest -f 'tests/*.yaml' helm/litellm # 25 passed helm unittest -f 'tests/*.yaml' helm/litellm-helm # 69 passed This also aligns the chart with the terraform templates, which inject the credentials into both components. * fix(proxy): never log billing credential values when they fail to resolve Accepting inline PEM turned the cert env vars into secret-bearing values, but the disable warning still echoed them. A value that is neither a readable path nor `-----BEGIN`-prefixed PEM, for example a key with a preamble or a malformed secret, fell through to the path branch and was written to the proxy logs verbatim, exposing the client certificate or private key to anyone who can read them. The warning now names the offending environment variables and tells the operator what a valid value looks like, without ever printing one * Revert "fix(helm): truncate the helm.sh/chart label to 63 bytes" This reverts commit 4f7f706. Version hygiene belongs to the pipeline that mints chart versions, not to the chart. The build workflow now caps the version slug so litellm-<version> fits the 63 byte label budget, which removes the overflow at the source rather than silently truncating a value operators use to identify the build. Drops the litellm.chart helper, restores the direct helm.sh/chart printf, and removes tests/chart_label_tests.yaml. Both chart suites stay green: helm unittest -f 'tests/*.yaml' helm/litellm # 20 passed helm unittest -f 'tests/*.yaml' helm/litellm-helm # 69 passed * feat(helm): default billingMetrics.secretName to the conventional name The componentized chart required an explicit secretName while the classic chart defaults to litellm-billing-metrics-mtls. Both now default to it, so the common path is to create that Secret with tls.crt and tls.key and set enabled: true. The required() guard stays, and with a default it now only fires when someone explicitly blanks the override, which the tests pin from both sides * feat(proxy): log once when billing metrics are actually enabled build_billing_metrics_recorder returned None silently when the deployment was not licensed, while every other disable path logged a warning. An operator reading logs could not tell "metering active" from "metering off because this component never saw the license", and a component can carry the cert mount and the billing env and still meter nothing. That is the undercount direction the metric is not allowed to drift in. A successful build now emits one info line naming the collector endpoint and the export interval; neither the certificate contents nor the license appear. The unlicensed path logs at debug rather than warning, because unlicensed is the common case and a warning there would be noise on every OSS proxy * fix(terraform): fail the plan on a partial billing-metrics config Each PEM secret is created only when its own variable is non-empty, so setting billing_metrics_endpoint with a certificate but no key applied cleanly and left the proxy logging "missing config" and never exporting. Silent non-export is the undercount direction this metric must not drift in, and every other surface fails fast on a half-configured metering block. Both templates now carry a lifecycle precondition requiring the client certificate and its key together whenever the endpoint is set. It lives on the gateway task definition (aws) and the gateway Cloud Run service (gcp) rather than on the secret resources, because those are themselves count-gated on the PEM being present and would never evaluate in the failing case. Cross-variable `validation` blocks would need terraform 1.9; versions.tf pins >= 1.6, and preconditions work there. ca_cert_pem stays optional, so an empty value still falls back to the system trust store. endpoint cert key result "" any any metering off, no secrets created set set set metering on set missing either plan fails Verified each row with `terraform console` against the condition, and reran `terraform fmt -check` and `terraform validate` in both directories * docs(terraform): record why the billing guard sits on the gateway resource The precondition cannot live on the cert secret, which is count-gated on the cert itself and so has zero instances in exactly the case the guard must catch. That makes the guard's correctness depend on this resource staying unconditional, which nothing else records and no test enforces * fix(terraform): guard the backend against a partial billing config too The precondition only sat on the gateway, but the backend receives the billing endpoint as well, because it serves the named-server MCP transport and meters it. A targeted apply of just the backend task or service would therefore skip the guard entirely and provision a component holding a billing endpoint with no credentials to use it, which is the silent never-export failure the guard exists to prevent. Both templates now carry the same precondition on the backend resource. The condition and truth table are unchanged; ca_cert_pem stays optional. terraform fmt -check and terraform validate clean in both directories * docs(team): document mcp_rpm_limit in update_team docstring * chore(ui): regenerate schema.d.ts for update_team docstring change
…ge_ttl_cap fix(mcp): cap per-user OAuth token cache TTL at the token's own lifetime
…I#33432) Co-authored-by: ryan <ryan@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ialModal (BerriAI#32572) * refactor(ui): consolidate Add/Edit credential modals into one CredentialModal AddCredentialModal and EditCredentialModal were ~90% identical: the same provider select, ProviderSpecificFields, and submit/filter logic, differing only in title, button text, edit-mode prefill, and the disabled credential name. Replace both with a single CredentialModal driven by a mode: 'add' | 'edit' prop, and point the two call sites in credentials.tsx at it. Removes ~120 lines of duplication and drops the no-explicit-any and no-restricted-imports baselines. The two per-file tests merge into one CredentialModal.test.tsx covering both modes (add: editable empty name; edit: prefilled, disabled name; provider fields render). * refactor(ui): derive credential name disabled state from mode, not data The disabled flag on the credential name field was tied to whether existingCredential?.credential_name is truthy, an artifact of the old EditCredentialModal. Drive it from the isEdit flag like the rest of the component so mode='add' with a stray existingCredential can't disable the field and mode='edit' with an empty name can't leave it editable. Behavior is unchanged for real call sites; adds a regression test for the edit-with- empty-name case. * refactor(ui): prefill credential form declaratively instead of via useEffect The edit-mode form was seeded with an imperative form.setFieldsValue inside a useEffect that also set React state (setSelectedProvider), an antd anti- pattern carried over from the old EditCredentialModal. Both call sites mount the modal fresh with existingCredential already present (conditional && plus destroyOnHidden), so there is no 'prop arrives after mount' case to handle. Replace it with antd's declarative initialValues on the Form and a lazy useState initializer for the provider. Removes the effect, its react-hooks/set-state-in-effect suppression and exhaustive-deps warning, and one any cast; behavior is unchanged (edit now shows the real provider on first paint instead of flashing the default). Existing tests cover prefill and the disabled name field.
Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # litellm/router.py
…itellm_list_vs_fil
…t for MCP egress (BerriAI#31516) * feat(mcp): add ID-JAG egress auth as a v2 outbound-credentials arm Adds the oauth2_id_jag MCP egress auth mode (draft-ietf-oauth-identity-assertion-authz-grant, shipped by Okta as "AI agent token exchange") as a first-class arm of the v2 outbound_credentials resolver rather than a standalone v1 handler. ID-JAG is a two-leg flow: an RFC 8693 token exchange swaps the caller's id_token for an ID-JAG assertion at the IdP org authorization server, then an RFC 7523 jwt-bearer grant presents that assertion to the MCP's resource authorization server for the access token used to call the upstream. The gateway authenticates to both endpoints with a private-key JWT client_assertion, falling back to client_secret when no key is configured. The mode is modeled as IdJagConfig in the AuthConfig discriminated union, with client auth as a ClientAuth tagged union (private_key_jwt or client_secret) so required fields are enforced at construction and illegal states are unrepresentable. A new token_endpoint collaborator performs the authenticated OAuth token-endpoint call and caches the result with per-key single-flight; the resolver's _id_jag arm runs the two legs and returns an httpx.Auth or a typed CredError. A missing caller identity token fails closed (precondition_required), so an ID-JAG server never falls back to a static credential. The v1->v2 adapter maps oauth2_id_jag servers onto IdJagConfig and the existing live v2 path resolves them, so no standalone handler, has_id_jag_config flag, or resolve_mcp_auth precedence branch is needed. The ID-JAG client_private_key is encrypted at rest alongside client_secret. * fix(mcp): sort token_endpoint imports to satisfy the I001 budget gate * fix(mcp): give token_endpoint pyright suppressions reasons for the LIT004 budget The freshly-merged base ratcheted the LIT004 ceiling down, so the six unexplained pyright suppressions in token_endpoint.py went over budget. Annotate each with why the boundary is untyped (litellm http handler and InMemoryCache are untyped; response.json() is validated by _TokenEndpointResponse in fetch) so the gate counts them as explained. * fix(mcp): enforce ID-JAG exchange over caller auth overrides and redact token endpoint from client errors For oauth2_id_jag servers the v2 resolver mints the upstream assertion from the caller's identity token; a caller-supplied x-mcp-auth / x-mcp-<alias>-authorization override or a conflicting injected Authorization must not disable that exchange and forward an arbitrary bearer, so IdJagConfig now joins authorization_code and token_exchange as a resolver-owned mode that keeps the v2 spec and ignores the override. The token endpoint error branches previously returned the configured endpoint URL in the client-visible 503 detail. The endpoint now stays in server-side logs and clients get a generic token-exchange failure. * fix(mcp): bind the ID-JAG token cache to the exchange config and map token endpoint network errors to typed CredErrors * fix(mcp): fail closed when an oauth2_id_jag server is half-configured instead of deferring to v1 static credentials * fix(mcp): evict the cached ID-JAG bearer on an upstream 401 so the retry re-exchanges * fix(mcp): map an unsignable client assertion to a typed misconfigured error instead of an unhandled 500 * fix(mcp): redact credential fields from the server-registry debug dump
…erriAI#33827) * refactor(ui): migrate policy attachments table onto shared DataTable * refactor(ui): pass a specific success message to the attachment copy action
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
…d kwarg Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…coped fixture (BerriAI#33750) The shared proxy wrapper in tests/e2e/e2e_gateway.py was misnamed: Gateway is not a gateway server, it is the client every suite uses to talk to the proxy (keys, models, chat/embed/ocr, spend read-backs, poll helpers). Rename the module to proxy_client.py and the class to ProxyClient, with build_gateway becoming build_proxy_client and the GatewayProvider protocol becoming ProxyClientProvider. The .gateway attribute suites held is now .proxy. Only identifiers changed; prose and string literals that use the word gateway for the proxy-server concept were left alone. Each suite previously built its own instance through a per-suite build_client() that called build_gateway() inside, duplicating the proxy wiring across suites. There is now one session-scoped proxy fixture in tests/e2e/conftest.py; every suite's client fixture depends on it and injects it, so the wiring lives in one place. claude_code keeps building its own client directly since it has its own harness and does not use the shared fixtures. Behavior is unchanged: shared transport, data-plane/control-plane split routing, poll budget, typed request/response models, and resource cleanup all go through the same object.
…ust:true (BerriAI#33616) * feat(messages): route Azure Anthropic /messages through Rust behind rust:true Adds an opt-in Rust path for non-streaming Azure Anthropic Messages. A deployment sets rust: true in litellm_params to route litellm.messages() and the proxy /v1/messages endpoint through the native Rust bridge; a missing flag or rust: false keeps the existing Python path, and non-Azure providers, streaming, an unavailable bridge, or a None result all fall back to Python. Rust-backed responses carry an x-litellm-rust: true response header so callers can see which path served the request. Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * test(docs): exclude LITELLM_USE_RUST_MESSAGES rollout flag from env-doc check Mirrors the existing LITELLM_USE_RUST_OCR entry; the flag is an internal rollout toggle that is intentionally not in the public environment settings docs yet. Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(rust_bridge): isolate OCR enable flag and drop dead messages global toggle use_litellm_rust only mutates the OCR enabled flag when configuring OCR (or called with no bridge kwargs, preserving the legacy contract), so configuring only the messages bridge no longer flips OCR state. Remove the vestigial global enabled/env state from the messages bridge. Routing is controlled per deployment by rust:true in the shared handler gate, so the messages module never consulted the global toggle; drop it rather than leave a no-op switch. Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * refactor(rust/messages): split Anthropic config into its own provider file and type the request/response contract Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * feat(messages): route eligible Azure Anthropic streaming through Rust via buffered fake-stream Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(messages): fold system-role messages for Azure Anthropic and fall back to Python on Rust bridge errors Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(rust_bridge): use Python::attach for amessages after pyo3 bump Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * test(proxy): mock get_configured_token_limits in model_info tests Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * ci: run rust_bridge unit tests in misc shard Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * Revert "ci: run rust_bridge unit tests in misc shard" This reverts commit c86d861. * test(anthropic): move rust messages bridge tests into misc-shard dir Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
) CodSpeed benchmarks the SDK with no IO, so it can't catch regressions that only appear under real concurrent load through the full proxy stack (auth, routing, logging, spend, Postgres, Redis). This adds a Locust load test under tests/e2e/load that drives concurrent POST /chat/completions traffic against a mock deployment (litellm_params.mock_response), so the measured throughput reflects proxy overhead rather than a provider's latency, and asserts an aggregate RPS SLO with a failure-ratio guard. The test is marked load and the parent conftest sorts load-marked items last so it never perturbs latency-sensitive suites. Covers reliability.perf.throughput.under_slo.
fix(proxy): resolve team wildcard credentials for vector store files
…ds (BerriAI#33760) * refactor(e2e): fold claude_code HTTP probes onto shared Gateway methods Migrate tests/e2e/claude_code/http_probe.py off its own httpx client onto the shared transport, and promote count_tokens and native anthropic messages to first-class Gateway methods (Gateway.count_tokens / Gateway.messages) with typed request/response models in the shared models.py so other suites reuse them. The probes now take an injected Gateway and issue their request through the shared count_tokens/messages methods, reusing the split control/data-plane routing, timeout, and typed Result handling the rest of tests/e2e uses. The wire shape is preserved: the pydantic bodies serialize byte-for-byte to what the old httpx probes sent, and the anthropic-version header is carried by a small AnthropicHeaders model. httpx is gone from the module. * test(e2e): drop unit-level probe harness test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): harden stage flakes for batches, UI, and MCP Unique batch model names avoid load-balancing onto stale azure-batch deployments that still pointed at the retired gpt-4.1-mini-batch, which only the managed/unified path was hitting. Retry batch retrieve on 500 and /ui/api-keys navigation on ERR_ABORTED. Skip the MCP key-access suite when the compose-only mcp-upstream is unreachable on stage k8s * test(e2e): cover Datadog remote MCP via search_datadog_logs Register the regional Datadog MCP endpoint with DD-API-KEY / DD-APPLICATION-KEY static headers (CI-safe header auth; browser OAuth is not headless-automatable). Seed a chat completion marked e2e-datadog-mcp-*, assert the proxy shipped it, list tools, call search_datadog_logs for the marker, and delete the server on teardown. Math-upstream key-access tests only skip when that compose service is unreachable * test(e2e): drop compose math MCP upstream; use Datadog only Key-access denial and happy-path MCP e2e both register the real regional Datadog remote MCP server with DD-API-KEY / DD-APPLICATION-KEY headers. Remove the mcp-upstream compose service and FastMCP add/multiply fixture * docs(e2e): require real Datadog MCP for all mcp suite tests Document that tests/e2e/mcp must register via datadog_mcp helpers against mcp.<site>/v1/mcp and must not introduce compose or fake MCP upstreams * chore: restore mcp_e2e_upstream_server.py Keep the FastMCP fixture file; e2e no longer wires it in compose, but the module itself is not part of the Datadog-only cleanup * fix(e2e): load tests/e2e/.env and fix datadog_reader importlib load pytest on the host never inherited compose env_file keys, so DD_API_KEY stayed empty. load_dotenv tests/e2e/.env in e2e_config. Register the dynamically loaded datadog_reader module in sys.modules so dataclasses do not crash under Python 3.12 * test(e2e/batches): harden azure/vertex unified lifecycle flakes Put the provider deployment name in every JSONL body so Azure does not depend on a perfect model rewrite. Retry create/retrieve/cancel on transient statuses with backoff. Drop cancel assertions for azure and vertex (registry only has a shared basic cell; create+retrieve prove routing, cancel stays best-effort cleanup) * test(e2e/ui): treat api-keys shell as success after SPA ERR_ABORTED Post-login client redirects abort the first /ui/api-keys/ goto on stage. Wait off /ui/login after cookie set, then accept the page once Create New Key is visible even if goto raised ERR_ABORTED * test(e2e): drop flaky key models dropdown Playwright suite API management e2e already covers key generate/update persistence. The UI Models-dropdown sentinel cases only added SPA ERR_ABORTED noise and no unique product signal. Remove the suite and unused browser fixtures
…#33839) * test(e2e): harden stage flakes for batches, UI, and MCP Unique batch model names avoid load-balancing onto stale azure-batch deployments that still pointed at the retired gpt-4.1-mini-batch, which only the managed/unified path was hitting. Retry batch retrieve on 500 and /ui/api-keys navigation on ERR_ABORTED. Skip the MCP key-access suite when the compose-only mcp-upstream is unreachable on stage k8s * test(e2e): cover Datadog remote MCP via search_datadog_logs Register the regional Datadog MCP endpoint with DD-API-KEY / DD-APPLICATION-KEY static headers (CI-safe header auth; browser OAuth is not headless-automatable). Seed a chat completion marked e2e-datadog-mcp-*, assert the proxy shipped it, list tools, call search_datadog_logs for the marker, and delete the server on teardown. Math-upstream key-access tests only skip when that compose service is unreachable * test(e2e): drop compose math MCP upstream; use Datadog only Key-access denial and happy-path MCP e2e both register the real regional Datadog remote MCP server with DD-API-KEY / DD-APPLICATION-KEY headers. Remove the mcp-upstream compose service and FastMCP add/multiply fixture * docs(e2e): require real Datadog MCP for all mcp suite tests Document that tests/e2e/mcp must register via datadog_mcp helpers against mcp.<site>/v1/mcp and must not introduce compose or fake MCP upstreams * chore: restore mcp_e2e_upstream_server.py Keep the FastMCP fixture file; e2e no longer wires it in compose, but the module itself is not part of the Datadog-only cleanup * fix(e2e): load tests/e2e/.env and fix datadog_reader importlib load pytest on the host never inherited compose env_file keys, so DD_API_KEY stayed empty. load_dotenv tests/e2e/.env in e2e_config. Register the dynamically loaded datadog_reader module in sys.modules so dataclasses do not crash under Python 3.12 * test(e2e/batches): harden azure/vertex unified lifecycle flakes Put the provider deployment name in every JSONL body so Azure does not depend on a perfect model rewrite. Retry create/retrieve/cancel on transient statuses with backoff. Drop cancel assertions for azure and vertex (registry only has a shared basic cell; create+retrieve prove routing, cancel stays best-effort cleanup) * test(e2e/ui): treat api-keys shell as success after SPA ERR_ABORTED Post-login client redirects abort the first /ui/api-keys/ goto on stage. Wait off /ui/login after cookie set, then accept the page once Create New Key is visible even if goto raised ERR_ABORTED * test(e2e): drop flaky key models dropdown Playwright suite API management e2e already covers key generate/update persistence. The UI Models-dropdown sentinel cases only added SPA ERR_ABORTED noise and no unique product signal. Remove the suite and unused browser fixtures * test(e2e/batches): fail clearly when OPENAI/AZURE provider is missing Replace bare next() over PROVIDERS with _model_for that raises ValueError naming the missing provider and the known list, instead of StopIteration * fix(e2e): migrate load suite from e2e_gateway to ProxyClient Stage collection failed with ModuleNotFoundError: e2e_gateway after the Gateway rename. Wire load/conftest and LoadClient to the shared ProxyClient fixture like every other suite * fix(e2e): drop duplicate datadog_mcp_url and CLAUDE section after merge
Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…riAI#33830) * test(e2e): cover /v1/responses openai basic nonstream and stream Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * test(e2e): assert responses stream ends on final raw completed event Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * test(e2e): centralize responses stream event models Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
…AI#33835) * test(e2e): cover /v1/responses openai basic nonstream and stream Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * test(e2e): assert responses stream ends on final raw completed event Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * test(e2e): cover /v1/responses openai cost_logged and tool_use Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * test(e2e): centralize responses stream event models Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
…riAI#33838) Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
…dge (BerriAI#33753) Add a live spend-tracking e2e that drives a streaming anthropic-format /v1/messages request through litellm's anthropic-messages -> OpenAI Responses adapter and asserts the consumed stream writes exactly one SpendLogs row with nonzero cost and token counts, attributed to the calling key under custom_llm_provider openai and the /v1/messages call_type. The deployment is a Responses-only OpenAI model (gpt-5.3-codex), so a served, costed row proves the Responses path was taken; the chat-completions bridge would have failed at OpenAI on an endpoint the model does not expose. Adds a streaming /v1/messages method to the shared Gateway and the suite client, the model to the inline compose config and driver-model registration, a coverage registry row (quota_management.spend_tracking.messages_bridge.logs_cost), and the matching variant vocab entry. The _summarize spend-row detail also gains call_type and custom_llm_provider so a failed assertion prints the fields it asserts on. Resolves LIT-4546
…migrations work for any uid offline (BerriAI#33853) * fix(docker): bake prisma CLI and engines at a fixed path so fresh-DB migrations work for any uid offline The runtime image shipped the prisma CLI and engines under /root/.cache, the default HOME-derived prisma-python cache location. Any deployment whose runtime HOME is not /root (kubernetes runAsUser, docker --user, HOME overrides) missed that cache on a fresh database, fell back to a nodeenv Node download that crashes on Wolfi (libatomic.so.1), and started the proxy with zero tables while every DB-backed endpoint returned 500 The bake now lives at /opt/prisma, a path no HOME resolution or cache volume mount can shadow. The builder records the engine paths there at generate time, and the runtime stage pins PRISMA_BINARY_CACHE_DIR, PRISMA_CLI_PATH, PRISMA_CLI_QUERY_ENGINE_TYPE=binary and PRISMA_OFFLINE_MODE so both litellm-proxy-extras and prisma-python resolve the baked CLI and engines directly. prisma migrate deploy on a fresh database now needs no npm and no network access for any runtime uid, including readOnlyRootFilesystem deployments Verified against live containers: fresh and existing databases as root, uid 12345, HOME overridden, on an internal-only docker network, and with a read-only root filesystem all migrate and serve /team/new successfully Fixes BerriAI#33650, BerriAI#24554 * chore(docker): fail the image build if the baked prisma CLI layout drifts Asserts the baked CLI shim is executable and its entrypoint exists in the runtime stage after the COPY and chmod, so a layout change in a future prisma-python release breaks the image build loudly instead of silently degrading the migration path at container startup
…riAI#33829) * feat(chat-ui): add personal Logs view scoped to the current user Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(chat-ui): show request payload from proxy_server_request in logs detail Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(chat-ui): address logs panel review feedback (stable detail key, error state) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…les (BerriAI#33867) Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
…/models (BerriAI#33864) A deployment whose model_info carried a non-numeric max_input_tokens or max_output_tokens (for example "128,000" or an empty string) made the bare int() in get_configured_token_limits raise inside the per-model /v1/models loop, so one misconfigured deployment turned the entire listing into a 500. Coerce each configured limit safely and treat malformed values as absent, matching the graceful degradation the listing had before the cost-map switch
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
chore(ci): promote internal staging to main
Sync 1715 upstream commits. Fork patch disposition: - PR #4 (streaming reasoning blocks + first-delta re-queue): DROPPED; upstream absorbed equivalent handling (reasoning_content thinking blocks, generalized _delta_has_content re-queue, empty-delta suppression). Fork's streaming_iterator.py taken from upstream wholesale. - PR #2 openrouter cache_control (QWEN enum entry): KEPT; upstream's CacheControlSupportedModels still lacks QWEN. - PR #2 adapter qwen cache gate: KEPT but NARROWED. Upstream now uses is_anthropic_claude_model for thinking-param passthrough too, so the blunt qwen widening would wrongly pass Anthropic thinking params to qwen. Split into supports_cache_control_passthrough(), OR-ed into the cache_control gate only; thinking translation keeps claude-only semantics (qwen thinking converts to reasoning_effort). - PR #2 adapter billing-header filter: KEPT; upstream still filters x-anthropic-billing-header in the messages path but not the adapter path. - Host streaming cache_read fix: NOT re-applied; upstream's _get_cache_read_input_tokens now falls back to prompt_tokens_details.cached_tokens. New fix on top: the thinking_blocks branch of _translate_streaming_openai_chunk_to_anthropic_content_block embedded the first chunk's thinking/signature in the content_block_start while the iterator also re-queues the trigger delta, duplicating the first thinking fragment for clients that concatenate block_start body + deltas (caught by the retained fork regression test). Block starts are now always empty per the Anthropic protocol; the two unit tests that asserted the embedding are updated. Tests: adapters + openrouter + messages suites, 263 passed; the 22 failures in messages/test_streaming_iterator.py and context_management/test_compact.py are identical on pristine upstream/main (missing proxy test deps), not introduced here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Merges 1715 upstream commits (BerriAI/litellm main @ 5d4c4d0) and re-evaluates every fork patch against current upstream.
Fork patch disposition
_delta_has_contentre-queue, empty-delta suppression).streaming_iterator.pyis now upstream verbatim.QWENinCacheControlSupportedModels(openrouter)cache_controlblocks.is_anthropic_claude_modelfor thinking-param passthrough too; blunt widening would pass Anthropicthinkingparams to qwen. Split intosupports_cache_control_passthrough(), OR-ed into the cache_control gate only. Qwen thinking still converts toreasoning_effort.x-anthropic-billing-headerin the messages path but still not in the adapter path; without it the per-request hash busts the prefix cache every turn.cache_readfix (uncommitted)_get_cache_read_input_tokensnow falls back toprompt_tokens_details.cached_tokens, fixing Claude Code context tracking on auto-caching OpenRouter routes.New fix on top
The
thinking_blocksbranch of_translate_streaming_openai_chunk_to_anthropic_content_blockembedded the first chunk's thinking/signature in thecontent_block_startwhile the iterator also re-queues the trigger delta, duplicating the first thinking fragment for clients that concatenate block_start body + deltas. Caught by the retained fork regression test (test_async_native_thinking_blocks_not_duplicated_on_transition). Block starts are now always empty per the Anthropic protocol; two upstream unit tests that asserted the embedding are updated.Tests
adapters/+openrouter/+messages/suites: 263 passed. The 22 failures inmessages/test_streaming_iterator.pyandcontext_management/test_compact.pyreproduce identically on pristine upstream/main in this environment (missing proxy test deps); not introduced here.Follow-up
cllm-setupLITELLM_FORK_COMMITwill be bumped to the merge commit of this PR.