chore(ci): promote internal staging to main - #34200
Merged
Merged
Conversation
MCP tool calling worked on /v1/chat/completions and /v1/responses but not on /v1/messages. Those are the only two surfaces with an MCP gateway entry point, so a litellm_proxy MCP reference reached Anthropic verbatim inside tools and the API rejected the request with "Input tag 'mcp' found using 'type' does not match any of the expected tags". The playground never surfaced this because it dropped the reference before sending, and disabled the MCP selector for the endpoint. Add the third entry point in anthropic_messages_handler, ahead of the provider branch so it covers the native path and both bridges from one place. The gateway expands the reference against the caller's own credentials and access control, which is the whole point of routing it through litellm rather than handing the url to the provider. /v1/messages needs Anthropic's own tool shape, so transform_mcp_tool_to_anthropic_tool joins the OpenAI chat and Responses transforms alongside it. The tool loop speaks tool_use and tool_result rather than OpenAI tool_calls, and reuses the existing FakeAnthropicMessagesStreamIterator to re-stream the result, the same pattern the websearch interception already uses on this route. Argument extraction moves into the shared extractor: an Anthropic tool_use block carries its arguments under `input`, and reading only `arguments` failed silently, executing the tool with every argument dropped. On the frontend the request builder declared selectedMCPTools and never read it, so no tools key was ever sent. Wire it through a shared block builder and add the endpoint to MCP_SUPPORTED_ENDPOINTS, which is what greys the selector out. Resolves LIT-4517 Resolves LIT-4518
…face The /v1/messages handler resolved only the auth object and the trace id, so tool listing and tool execution ran without the caller's MCP auth headers. That fails quietly rather than loudly: the tool still executes, just with no credentials, so every server behind interactive OAuth, a bearer token or per-user env vars returns nothing while the model reports it has no access. Only a no-auth server looks healthy, which is exactly what the first proof used. Threading the missing arguments would have left the real problem in place. Each gateway surface rebuilds the same context by hand (responses/main.py twice, chat_completions_handler, mcp_streaming_iterator), which is why a new surface drops fields; this adds a fifth that dropped six of eight. Resolve it once into a frozen MCPRequestContext and have the handlers take that, so a field cannot be forgotten at a call site. chat_completions_handler now uses it too, and the resolver reads user_api_key_auth from both metadata keys because LITELLM_METADATA_ROUTES carry it in litellm_metadata while chat uses metadata. Also stop the loop when every tool call was skipped. tool_results is empty then, and the tool_result message built from it has empty content, which Anthropic rejects; the caller saw a 400 from mid-loop instead of the model's own answer. Tests pin both: dropping the headers from either listing or execution fails, and so does removing the empty-results guard.
…ames
Two review findings, both a chat-vs-messages divergence.
transform_mcp_tool_to_anthropic_tool sent the MCP inputSchema to Anthropic almost
as-is, while the chat path (_map_tool_helper) coerces the type to object, inlines
legacy definitions with unpack_legacy_defs, and allow-lists keys to
AnthropicInputSchema. So a tool whose schema carried $schema, legacy definitions
or oneOf worked on /chat/completions and 400d on /v1/messages; a clean-schema
server hid it. Both paths now run the same sanitize_input_schema_for_anthropic,
extracted next to unpack_legacy_defs so they cannot drift again, and the chat
path is refactored onto it rather than keeping its own copy.
buildMcpToolBlocks percent-encoded the server and toolset names inside
litellm_proxy/mcp/... urls, but the gateway resolves the name with a raw
server_url.split("/")[-1] and never url-decodes, so a name with a space failed
lookup. The already-working chat path does not encode; the shared builder now
matches it.
Tests pin both: reverting the transform to the unfiltered schema fails, and
re-adding encodeURIComponent fails the builder test.
_should_auto_execute_tools returned True as soon as any MCP reference set require_approval="never", so a request that mixed a "never" reference with an "always" or "manual" one auto-executed every tool call the model produced, including the approval-gated ones. A prompt could name the approval-required tool and have it run with no approval. Make the gate fail closed: auto-execute only when every reference opts in with "never". A single approval-required reference (including the object form or an unset value) returns the model's tool calls to the caller instead of running them, so an approval-gated tool can never be auto-invoked. This is the shared decision behind /chat/completions, /responses, the streaming iterator and the new /v1/messages path, so all four fail closed from one change. The common case, every reference "never", is unchanged. The alternative, executing the "never" calls and returning only the approval-required ones, needs partial execution that the Anthropic tool loop cannot express without fabricating tool_result blocks for the calls it withheld, so the whole-request fail-closed gate is the safe minimum. A future change can add per-call partial execution if a caller needs it. Test covers the mixed and manual cases; reverting to "any never" fails it.
The team settings guardrails dropdown always rendered the Global and Other headers, so a proxy with no global guardrails showed an empty Global heading above the list.
…regates (#33810) * feat(spend): track prompt compression saved tokens in daily spend aggregates Native compression interception now records tokens_before/after/saved into the request litellm_metadata so savings land in the SpendLog metadata JSON under a typed compression_savings key. A single normalizer (extract_compression_saved_tokens) sums that key with Headroom guardrail tokens_saved; the two writers are disjoint and run at different stages, so summing never double-counts. The spend-log redactor now preserves purely numeric compression stats inside guardrail_response so Headroom savings survive the store_prompts_in_spend_logs=false default. compression_saved_tokens is threaded through BaseDailySpendTransaction, queue aggregation, the daily upsert blocks, a new BigInt column on all six daily spend tables, and the daily activity read path (SpendMetrics, DailySpendMetadata, raw-SQL rollups) * fix(spend): normalize legacy guardrail shapes and float token stats in compression savings reader * feat(spend): aggregate compression and prompt caching dollar savings in daily rollups Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(spend): update daily spend aggregation fixtures for savings columns Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(ui): add Cost Optimization dashboard page New left-nav Cost Optimization page under Observability that surfaces money saved by prompt compression and prompt caching. It reads the daily activity rollup (userDailyActivityCall / get_daily_activity) and never scans SpendLogs, so it stays fast at 1M+ rows. Renders a Total saved card, per-driver Compression and Prompt caching cards, a savings-over-time area chart, and a savings-by-driver donut, all aggregated in memory from the per-day metrics.compression_savings_spend and metrics.prompt_caching_savings_spend fields. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…rt window resets (#33832) * test(e2e): assert the long budget window keeps blocking after the short window resets The multi-window budget tests proved the tight window blocks and self-heals but never asserted the other direction: a long (1d) window whose cap the accumulated spend already crossed must keep refusing calls even inside a fresh short window. Adds one test per file (key and team) that drives spend to a block, waits for the short window's reset_at to strictly advance (the reset job zeroes that window's counter in the same pass), then polls until the refusal is attributed to the 1d window ("over 1d budget"), failing immediately if any call succeeds or a non-budget error leaks. Harness gains per-window reset_at readback: BudgetWindowState in models.py and key_window_reset_at / team_window_reset_at on BudgetClient. * refactor(e2e): hoist shared budget-suite helpers into budget_client drive_to_block and as_datetime existed as five and four per-file copies in the budgets suite; both move to budget_client with each file keeping a thin delegating wrapper so call sites and per-file pacing stay unchanged. The three /team/info readers in budget_client now share a private _team_info. Also guard the long-window reset_at snapshots with explicit non-None asserts so the midnight-roll diagnostic cannot misreport when the window is missing from the info response (greptile P2s). * docs(e2e): tighten the multi-window module docstrings * refactor(e2e): type window reset_at as datetime and expose plain window readers BudgetWindowState.reset_at becomes a pydantic-parsed datetime, so the multi-window tests compare real datetimes instead of hand-parsing strings. The duration-keyed accessors are replaced by two plain readers, key_budget_windows and team_budget_windows, with the pure window_reset_at lookup exported; the client no longer encodes one test's access pattern. * test(e2e): name the tiny short-window cap and comment the wait loops * test(e2e): surface the 429 budget-block assert in the multi-window tests drive_to_block now returns the blocking response so a test body can assert on its shape; the two long-window tests assert status 429 explicitly, which also pins the multi-window enforcement path's HTTP mapping (the enforcement suite only covers the single-budget path). Other callers ignore the return and are unchanged. * refactor(e2e): scope this PR to the multi-window test, drop the cross-suite hoist The helper hoist rewrote four unrelated budget test files (reset, reset_advances, team_member_reset, user_across_keys) to pull drive_to_block and as_datetime out of budget_client, which is refactor churn beyond this PR's multi-window scope. This restores those four to their pre-PR state and gives the two multi-window tests their own inline drive-to-block loop again, so the PR touches only the multi-window feature: its two tests plus the budget_client window readers and the reset_at datetime typing they actually use. The suite-wide helper dedup can land on its own PR * docs(e2e): number the long-window key test steps inline * docs(e2e): number the long-window team test steps inline
…LITELLM_RUST env var (#33848) * feat(messages): route native Anthropic /messages through Rust behind RUST env var Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * test(docs): exclude RUST rollout flag from env-key documentation check Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * refactor(messages): rename RUST rollout env var to LITELLM_RUST Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(e2e): cover credential-backed /v1/messages request Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * test(e2e): use runtime Anthropic credential Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
…e axum gateway (#33880) * feat(rust): expose anthropic messages route Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(rust): use provider model for messages upstream Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * feat(rust): stream Anthropic Messages SSE on POST /v1/messages Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * test(rust): prove alias is substituted with provider model on /v1/messages Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(rust): make anthropic messages provider constant available without server feature Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
…e mcp_gateway_dcr flag The flag guarded no breaking change: the aggregate discovery lives at new /mcp-suffixed routes, the challenge only fires at aggregate scope, and the authorize/token/register/admission arms self-gate on the llm_dcrc_/llm_session_ prefixes. Bare-origin and per-server discovery are left exactly as they were, and a server literally named mcp keeps its own discovery via disambiguation, so turning it on for everyone changes nothing about existing flows.
…y challenges
Two RFC 9728 / 8414 discovery fixes on the aggregate front door, both raised by Bugbot on this PR
The aggregate authorization-server document at /.well-known/oauth-authorization-server/mcp used to
defer to a per-server row literally named "mcp", serving issuer {base} while the aggregate
protected-resource document advertises {base}/mcp as its authorization server. A spec client
following that chain fails the RFC 8414 issuer check and cannot sign in. The single segment /mcp is
now reserved for the aggregate so the issuer stays {base}/mcp and matches the protected-resource
document; a server named "mcp" keeps its standard two-segment discovery at
/.well-known/oauth-authorization-server/mcp/mcp
The 401 challenges built the resource_metadata URL as {base}/.well-known/oauth-protected-resource/mcp
with no SERVER_ROOT_PATH segment, but the routes are registered with the path-inserted root segment,
so a proxy mounted under a sub-path pointed DCR clients at a URL that 404s. Both the aggregate
challenge and the pre-existing per-server pass-through challenge now derive the path from one
well_known_root_suffix helper that the route registrations also use, so the advertised URL cannot
drift from the served route
tests/test_litellm/proxy/test_custom_proxy.py sets SERVER_ROOT_PATH at import time (its app mounts under a custom path) and never restores it, so in a shared shard the value leaks into the process. The discovery routes and the 401 challenges now read SERVER_ROOT_PATH to path-insert it where they previously ignored it, so a leaked value rewrites every resource_metadata URL and the exact-URL assertions in the delegate, pass-through, and aggregate challenge tests fail depending on shard order An autouse fixture clears SERVER_ROOT_PATH for the MCP discovery tests so they deterministically exercise the default root-mounted deployment; the tests that assert a sub-path deployment set the value explicitly within their own body. No assertion changed; the leak was invisible before only because the code ignored the variable
… only The SERVER_ROOT_PATH fix for the per-server pass-through challenge belongs with its sibling in exceptions.py (both fabricate a per-server resource_metadata URL and both omit the root segment), and both are pre-existing paths unrelated to the aggregate discovery this PR adds. Reverting the server.py change keeps this PR to the aggregate front door and avoids leaving the two per-server challenge builders inconsistent; the per-server root-path fix lands as its own change covering both sites.
…st (#33849) * feat(rust): add OpenAI Responses WebSocket gateway Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * test(rust): cover Responses WebSocket gateway behavior Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(rust): align Responses WebSocket parity Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * feat(rust): expose Responses WebSockets through bridge Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(rust): reject non-openai responses deployments early Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(rust): align Responses WebSocket bridge semantics Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * refactor(rust): move Responses instrumentation into core Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(rust): preserve Responses WebSocket callback dispatch Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * build(deps): authorize vcrpy and locust licenses in liccheck Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
…dels create_model_info_response cast cost-map max_input_tokens / max_output_tokens with unguarded int(). The surrounding try/except covers only the get_model_info lookup, so a deployment whose model_info carries a non-numeric limit (e.g. "128,000" or an empty string) raised inside the per-model listing loop and failed the entire GET /v1/models and /models response with a 500, taking healthy deployments down with it. A deployment's model_info is registered into litellm.model_cost verbatim, so the malformed value reaches the cost map and not just the router index. Router.get_configured_token_limits already coerced this safely for the deployment path; the cost-map path was missed, so the two together still regressed. Both now share coerce_token_limit in litellm_core_utils, which returns None for a malformed value so the listing omits that one limit instead of failing, matching the graceful degradation the endpoint had before the cost-map switch.
fix(proxy): treat malformed cost-map token limits as absent on /v1/models
…itellm-core as a base provider (#33888) * feat(rust): add feature-gated Bedrock AWS auth Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * refactor(rust): move Bedrock auth into core Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(rust): fall through caller identity lookup errors Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * test(rust): add live Bedrock proof and CI coverage Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * refactor(rust): share in-memory cache with Bedrock auth Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(rust): preserve web identity credential expiry Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
…e model field (#33875) * feat(complexity-router): optionally return raw model name Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): restore asyncio import Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(tests): preserve staging asyncio import Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): drop unused local asyncio import Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(dashboard): add complexity router raw model toggle Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(complexity-router): move metadata key constant to constants.py Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(proxy-tests): preserve module spacing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
feat(mcp): always-on aggregate gateway DCR discovery front door
* fix(langfuse): send v4 ingestion header for otel callback * refactor(langfuse): inline otel ingestion header literals * test(langfuse): assert v4 ingestion header on dynamic key config paths * style: apply ruff format to langfuse otel header changes * chore(langfuse): drop stale development annotation on json import --------- Co-authored-by: Hassieb Pakzad <68423100+hassiebp@users.noreply.github.com>
* feat(ui): add configuration tabs to Cost Optimization page Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(ui): reuse AutoRouter v2 and Router Settings prompt-caching panel in Cost Optimization; clarify Headroom compression Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(ui): add experimental dashboard banner with feedback discussion link to Cost Optimization Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(ui): add savings methodology note and per-key/team compression enterprise callout to Cost Optimization Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(ui): assert active tab state in Cost Optimization tab-switch test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Contributor
|
Too many files changed for review. ( |
|
|
… for all stream modes (#34126)
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
…thropic (#34159) httpbin.org is an external dependency prone to transient 503s (caused the stage failure); its echo-body assertion also doesn't exercise a real LLM provider. Point the custom pass-through endpoint at the real Anthropic Messages API instead. Anthropic doesn't echo headers back, but it gates real behavior on two of them, which is enough to prove forwarding: a static x-api-key configured on the endpoint (never supplied by the caller) must reach upstream or the call 401s, and an invalid x-pass-anthropic-version sent by the caller must reach upstream with the prefix stripped, which Anthropic echoes verbatim in its 400 body. Verified live against a local proxy and the real Anthropic API: valid version returns a real completion, invalid version returns the exact marker in the 400 body.
* fix(e2e): reference client.proxy in mid-conversation native providers test EndpointsClient exposes the shared ProxyClient as .proxy and has never had a .gateway attribute, so these two calls raised AttributeError at runtime and failed the tests/e2e basedpyright zero-error gate for any PR touching e2e files. Introduced in 23b5b7d. * test(e2e): cover 12 non-core LLM coverage registry cells Raises Non-Core LLMs registry coverage from 24/50 to 36/50 (overall 51.9% to 54.8%). Four cells were already asserted by existing tests and only gain their covers marker (openai embeddings, openai image generation, openai TTS, cohere rerank); one is dual-marked onto the existing spend-tracking embeddings test rather than duplicated. New tests: bedrock and vertex embeddings, streaming TTS (asserts chunked transfer encoding so a buffered body cannot pass), audio transcriptions via the realtime suite's wav fixture, moderations flag/pass pair, and files list/retrieve in the batches suite. Harness: e2e_http.upload generalized to any form model with a file_content_type override (batches path unchanged), new stream_binary primitive + BinaryStream for binary chunked responses, transcribe and moderations client methods, file retrieve/list client methods. * fix(e2e): close streamed TTS response on error paths and surface the error body With stream=True a non-2xx response returned with the body unread, keeping the socket checked out until garbage collection; the sibling _streaming_outcome already consumes resp.text on error. The response now closes on every path and BinaryStream carries a bounded error_body so a failed stream call is triageable. * test(e2e): assert streamed TTS response carries no content-length
…xture (#34204) The spend-log metadata schema gained a compression_savings key, so the gcs pubsub v1 payload now carries it. The golden fixture was never updated, and the comparator flags any key present in the payload but absent from the fixture, so test_async_gcs_pub_sub_v1 failed on every run. Pin the key as null rather than adding it to ignored_keys; the value is deterministic on this path, so ignoring it would leave the assertion blind to the field entirely.
docs: add TLDR section to PR template
…ination handling (#34189)
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
|
|
||
| def _should_return_raw_model_name(request_data: dict[str, object]) -> bool: | ||
| return any( | ||
| isinstance(metadata, dict) and metadata.get(RETURN_RAW_MODEL_NAME_METADATA_KEY) is True |
Contributor
There was a problem hiding this comment.
Low: Client-controlled disclosure of raw model names
An authenticated caller can set _complexity_router_return_raw_model_name: true in request metadata and prevent response restamping, exposing provider-prefixed or otherwise internal model names. Treat this as server-owned state, or strip the key from client metadata before routing.
| response.raise_for_status() | ||
| response_json = response.json() | ||
|
|
||
| verbose_proxy_logger.debug("DeepKeep guardrail response: %s", response_json) |
Contributor
There was a problem hiding this comment.
Low: Guarded content written to debug logs
A caller can cause prompt text, structured messages, or tool arguments returned by DeepKeep to be copied into proxy debug logs because the complete response is logged here. Log only non-content fields such as the action and status.
Suggested change
| verbose_proxy_logger.debug("DeepKeep guardrail response: %s", response_json) | |
| verbose_proxy_logger.debug("DeepKeep guardrail response action=%s", response_json.get("action")) |
Contributor
PR overviewThis PR promotes the internal staging branch to main, carrying updates that touch proxy request processing and the DeepKeep guardrail integration. The changes appear centered on CI/release promotion rather than a discrete user-facing feature. There are three low-severity security issues still open. The remaining concerns are limited information disclosure paths for authenticated callers, including exposure of internal model names, DeepKeep endpoint details in error handling, and guarded content being written to debug logs. No issues have been fixed or addressed yet, so the PR still needs cleanup before the security posture is positive. Open issues (3)
Fixed/addressed: 0 · PR risk: 4/10 |
…grations (#34206) Delete Key moved into the key info page's overflow dropdown (#34116) and the credentials table's row actions moved into a shared DataTable overflow menu, so both specs were clicking a button that no longer exists. Point them at the menu items instead. Add a CredentialsPanel unit test asserting the update payload drops the masked api key and keeps the edited api base, so that guard is not held up solely by an e2e a table migration can silently disarm.
test(e2e): add live A2A agent e2e suite
| **({"http_status_code": http_status_code} if http_status_code else {}), | ||
| ) | ||
| verbose_proxy_logger.error("DeepKeep guardrail API error: %s", str(error)) | ||
| raise DeepKeepGuardrailAPIError(f"DeepKeep guardrail API failed: {str(error)}") |
Contributor
There was a problem hiding this comment.
Low: Guardrail URL disclosure
An authenticated caller can trigger a DeepKeep non-2xx response and receive the raw httpx exception through the proxy's generic exception mapper. HTTPStatusError includes the request URL, which may expose the DeepKeep hostname and path; keep the detailed exception in server logs and return a generic message.
Suggested change
| raise DeepKeepGuardrailAPIError(f"DeepKeep guardrail API failed: {str(error)}") | |
| raise DeepKeepGuardrailAPIError("DeepKeep guardrail API failed") |
… the stored password (#34160) The Cache Settings page read only the database row, so a response cache pointed at Redis purely through REDIS_* env vars showed a blank page while the cache worked. It also masked credentials on read with a partial-reveal string and re-persisted whatever the form submitted, so an admin who edited an unrelated field and pressed Save wrote the mask string over the real Redis password, breaking auth. GET /cache/settings now overlays the same REDIS_* kwargs the runtime resolves from when the stored config leaves a field unset, and redacts credentials with a fixed marker. POST /cache/settings restores the stored secret behind any credential echoed back as the marker or omitted, and drops an env-sourced marker rather than persisting it; the response no longer echoes plaintext credentials. The connection test resolves a redacted credential back to the stored value the same way. The dashboard never prefills a credential and drops the marker from the save payload, mirroring the Coordination Redis tab. Resolves LIT-4315
…34156) The UI theme and logging-callback read endpoints reported only stored config while the features resolve their values from the process environment, so a gateway configured purely through env vars showed blank settings pages even though branding rendered and callbacks fired. /get/ui_theme_settings read only litellm_settings.ui_theme_config; logo_url and favicon_url now fall back to UI_LOGO_PATH and LITELLM_FAVICON_URL when the stored config leaves them blank. process_callback (the logging-callbacks block of /get/config/callbacks) reported every callback env var as unset unless it lived in the config environment_variables overlay; it now falls back to os.getenv, matching the slack block. Secret values stay redacted for non-admins via the existing callback role gate. Stored values keep winning over the environment, so the UI-driven flow is unchanged. Resolves LIT-4667
…ths (#34164) * test(e2e): cover customer chat/messages cost + streaming paths Fills five uncovered P0 registry cells matching the customer's confirmed stack (OpenAI SDK, Bedrock, /v1/messages) and their per-request cost dependency: - /v1/messages logs cost that matches the x-litellm-response-cost header (LIT-4076) - OpenAI /chat/completions streams real content, and a non-streamed call is costed - Bedrock Converse /chat/completions returns real content non-streamed and streamed The streaming checks aggregate delta content and parse every chunk as JSON, so a clean-but-empty stream or a truncated chunk fails instead of passing on a bare 200. * test(e2e): add tool-use coverage for openai, bedrock converse, anthropic responses Function-calling regression guards on the paths the customer's agentic SDK usage exercises: OpenAI and Bedrock Converse /chat/completions, and Anthropic /v1/responses. The model is forced to call a weather tool and the test asserts the returned tool call names the function and carries JSON-parseable arguments with the expected field, so a dropped tool_call or malformed argument JSON fails instead of passing on a bare 200. Adds a minimal tool_calls field to the response OutMessage. * test(e2e): cover bedrock converse responses + thinking Adds llm.responses.bedrock_converse.basic/tool_use and llm.chat_completions.bedrock_converse.thinking. The thinking test enables extended thinking and requires reasoning_content plus a real answer, so a path that drops the reasoning block fails rather than passing. * test(e2e): cover bedrock embeddings + openai structured output and reasoning Bedrock Titan embeddings return a real vector; OpenAI structured output must yield schema-conforming JSON with the correct extracted values (age==42, not just valid JSON); an OpenAI reasoning call must report reasoning tokens, so a non-reasoning fallback fails. Adds response_format to ChatBody and reasoning-token details to Usage. * test(e2e): cover vision + streaming tool calls on openai and bedrock converse Vision on both providers must describe the image (not just 200); the streamed OpenAI tool call is reassembled from its fragments and its argument JSON parsed, so a stream that never completes the call or splits its JSON fails. Extends ChatMessage content to a typed text/image union. * test(e2e): cover openai prompt caching hit on repeated large prefix A repeated large-prefix prompt must report cached prompt tokens on the second call, so a cache regression that stops reusing the prefix (and silently re-bills full input) fails here. * test(e2e): cover openai audio speech + bedrock rerank and image generation Marks the OpenAI TTS cell and adds Bedrock Titan rerank (top_n honored, scored) and Bedrock Titan image generation (returns b64/url), the customer's non-chat AWS surfaces. * test(e2e): cover end-user (customer) create persistence mgmt.end_user.new.happy_path: create an end-user via /customer/new and confirm /customer/info reports it, the end-user-identity surface the customer relies on for per-customer controls. Adds customer models + management-client methods. * test(e2e): enforce key model allow-list on the passthrough route other.auth.passthrough.model_allowlist_enforced: a key scoped to gemini must be denied a claude call through the anthropic passthrough route (403), so custom-auth scoping is not bypassable by going through passthrough instead of /chat/completions. * test(e2e): address Greptile - assert stream data events, correlate messages spend by key - streaming: assert len(stream_events) > 1 instead of chunks > 1, since chunks counts the terminal data: [DONE] marker and would pass a single content event - messages cost: correlate the spend row by the unique scoped key rather than the Anthropic response id, which need not equal the proxy spend-log request_id
yuneng-berri
enabled auto-merge
July 22, 2026 02:00
shin-berri
approved these changes
Jul 22, 2026
yucheng-berri
approved these changes
Jul 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
12 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relevant issues
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Type
🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test
Changes
QA runbook
Final Attestation