fix(e2e): poll for both spend rows before asserting the cache-hit contract - #34968
Merged
shin-berri merged 1 commit intoJul 28, 2026
Merged
Conversation
…tract The cache-hit and paid rows for the two driver calls flush from different pods on independent update_spend timers, so waiting only for the cache-hit row can return a half-arrived result set where the paid-row assertion then fails on an empty list. Requiring both row kinds in the poll predicate lets the existing deadline absorb the slower flush without weakening any assertion
ryan-crabbe-berri
enabled auto-merge (squash)
July 28, 2026 17:43
mubashir1osmani
approved these changes
Jul 28, 2026
Contributor
Greptile SummaryStrengthens the cache-hit spend-tracking end-to-end test by polling until both cache-hit and non-cache-hit rows are available before asserting billing behavior. Confidence Score: 5/5The PR appears safe to merge because the revised predicate waits for the complete result state required by the existing assertions. The test uses a freshly scoped key for two intended calls, and the changed predicate correctly prevents the cache-hit row alone from ending polling before the paid row arrives.
|
| Filename | Overview |
|---|---|
| tests/e2e/quota_management/spend_tracking/test_spend_tracking_e2e.py | Updates the polling predicate to wait for both expected spend-row kinds, preventing assertions against a partially flushed result set without weakening existing billing and cache-hit checks. |
Reviews (1): Last reviewed commit: "fix(e2e): poll for both spend rows befor..." | Re-trigger Greptile
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
shin-berri
disabled auto-merge
July 28, 2026 18:21
blackflame007
added a commit
to nolgiainc/litellm
that referenced
this pull request
Aug 3, 2026
…~372 commits) (#36) * fix(ui): restore the wide Add MCP Server dialog The shadcn migration carried the antd modal's 1000px width over as an unprefixed max-w-[1000px], which tailwind-merge keeps alongside the DialogContent base class sm:max-w-md; the responsive variant wins from 640px up, so the dialog rendered at 448px. Prefix the override so the merge drops the base clamp * style(ui): match MCP Servers tabs to the dashboard's line tab pattern The MCP Servers page was the only page-level tab bar using the segmented (pill) TabsList stretched with w-full, which rendered a full-width grey bar with a lone pill on the left. Every other page-level tab bar (budgets, vector stores, access groups, organizations, routing groups, API reference) uses the underlined line variant, so use that here too. * fix(vertex): decide rawPredict passthrough streaming from the request body Vertex passthrough classified any target URL containing "stream" as a streaming request. `:streamRawPredict` carries that substring, so a unary Claude-on-Vertex call whose body omits `stream` was routed through the streaming logging path. That path never consults the response content-type, so a complete `"type": "message"` JSON body was handed to the Anthropic SSE chunk parser, which recognises none of it; the spend log recorded 0 prompt tokens, 0 completion tokens and zero cost Streaming for the rawPredict family now comes from the request body, which is what the Anthropic Messages contract uses for those endpoints. The generateContent family keeps its URL signal because the Gemini REST body has no `stream` field, and `?alt=sse` is still appended for every request that is classified as streaming, so Gemini framing and its usage parsing are unchanged Both passthrough streaming predicates read `.get("stream")` off a body that is only annotated as a dict; `_read_request_body` returns whatever the JSON parser produced, so an array body raised AttributeError. The two predicates are now one owner that answers False for any non-object body, which covers the vertex, mistral, anthropic, vllm and azure passthrough routes * fix(ui): stop the custom-server action colliding with the dialog close button DialogContent's close button is absolutely positioned 16px from the right edge at 32px wide, so it overlays the rightmost 24px of the p-6 content box. The justify-between header pins "+ Custom Server" to that same edge and, being out of flow, the close button reserves nothing. Give the action a right margin that clears it; keeping the margin on the button rather than the row leaves the header rule full-bleed * fix(ui): center vertical toolbar dividers The shadcn separator primitive ships `data-vertical:self-stretch` so a bare vertical divider fills its row, but every call site overrides the height with `h-5`. A definite cross size makes `align-self: stretch` behave as `flex-start`, so the dividers rendered flush with the top of their flex line instead of centered: 0px above and 18px below in the dashboard header, 0px above and 12px below in the models table toolbar Routes the three vertical dividers through a ToolbarSeparator that pairs the fixed height with a same-variant `data-vertical:self-center`. Matching the variant is what matters; tailwind-merge then drops the conflicting class outright, whereas a plain `self-center` ties on specificity (the variant is defined with `:where()`) and loses on utility order. The CLI-managed primitive is left untouched * fix(guardrails): resolve judge_model credentials via lazy Router lookup in llm_as_a_judge (#34509) * fix(guardrails): resolve judge_model credentials via Router in llm_as_a_judge Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): wire llm_router into DB-backed judge guardrail init paths Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): assert patch endpoint forwards llm_router to sync Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(guardrails): resolve judge Router lazily and fix wildcard/alias dispatch Resolve the proxy Router at judge-call time via an injected provider instead of capturing it at construction, so a DB-backed judge guardrail created before the Router exists no longer captures None permanently. Select the Router path with router.get_model_list(model_name=judge_model) so wildcard routes and model_group_alias keys resolve, not just literal deployment names. Isolate the judge call from user-traffic routing with num_retries=0 and fallbacks=[]. Revert the llm_router threading through the DB sync/reinit/create/approve/patch paths since the lazy provider makes it unnecessary. Replace mocked-Router tests with real Router coverage for plain deployments, model_group_alias, and wildcard routes, plus lazy per-call resolution. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): harden judge verdict parsing and guard proxy import Strip markdown fences and surrounding prose before json.loads so fencing-prone judge models evaluate instead of failing open, guard the proxy_server import in _default_router_provider so an unimportable proxy falls back to the SDK, and snapshot/restore global callback lists in the DB-path judge registry tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): reject non-object judge verdicts instead of failing open as success * fix(guardrails): route hidden model_group_alias judge models through the Router --------- Co-authored-by: milan <milan@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <yucheng@berri.ai> * fix(proxy): roll up tool spend daily instead of scanning SpendLogs GET /v1/tool/spend served the Cost Optimization card with two raw queries over LiteLLM_SpendLogToolIndex x LiteLLM_SpendLogs on every dashboard load; the totals query's driving scan was all of SpendLogs in the window. Both per-request tables reach 1M+ rows at customer scale, so the card cost O(traffic) per view and had to be capped at 30 days. The index writer also mined proxy_server_request.tools, i.e. tools DECLARED in the request body, attributing each request's full spend to tools that never ran; and all non-MCP mining ran against payload fields that are '{}' unless store_prompts_in_spend_logs is enabled, so non-MCP coverage silently depended on a privacy setting. Now the spend writer builds a ToolUsageTransaction at request time from invoked tools only, resolved by the shared get_tool_calls_from_response normalizer so every response surface (chat completions, Responses API, Anthropic Messages) is covered; the tool registry's response arm delegates to the same owner. Transactions queue beside the spend-log queue and the flush job writes index rows plus a new LiteLLM_DailyToolSpend rollup (date, tool_name PK) in one transaction, retrying connection errors with backoff (a failed batch commits nothing, so the retry cannot double-count) and dropping the batch with an error log on anything else. The endpoint aggregates in SQL: by_tool is the top TOOL_SPEND_TOP_TOOLS tools by spend via group_by and daily covers only those tools, so the response is bounded by days x TOOL_SPEND_TOP_TOOLS regardless of range or tool-name cardinality; the 30-day clamp is gone. total_spend is dropped from the response; it was never rendered and its deduplicated semantics are not computable from a rollup. Spend-log retention deliberately does not touch the rollup, so tool spend history outlives per-request rows. * fix(ui): keep the spend-by-tool legend from overlapping the charts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): color spend-by-tool charts with an ordered ramp Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): truncate long team names in the models table team dropdown The Team dropdown popup is pinned to the trigger width via w-(--anchor-width) and clips its overflow, while Base UI's ItemText wrapper is flex-1 shrink-0 with min-width: auto, so it sizes itself to the full nowrap label and simply overflows the popup. Teams without a team_alias render their 36-char id, so those options were sliced mid-character with no ellipsis. Clears min-width: auto off the text wrapper and truncates the label at the call site. The underlying gap is in the shared Select primitive, which any long-labelled select in the dashboard will hit; that is left for a separate change. * fix(ui): use a single muted blue ramp for the tool charts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(management): move the logs end-user filter onto /management/v1 `/customer/aliases` shipped two days ago and has not been in a release, so its wire contract is still free to change. This lands it on the control-plane contract before that stops being true, since after a release the path, the param names and the envelope would all need a permanent legacy adapter The endpoint becomes `GET /management/v1/spend_logs/end_users`. It is a facet, the distinct values one column takes over a filtered query on a resource, not an entity collection; naming it after `customers` implied it listed the end-user table when it actually reads spend logs, which is a different row set. Serving it under the parent resource means its filters are the parent's filters, so the dropdown offers exactly the values the logs table can show without two endpoints having to keep agreeing on that Contract changes: `size` becomes `page_size`, `search` becomes `q`, the window moves from flat `start_date` / `end_date` to `filter[startTime][gte]` / `[lte]`, and the body becomes `{data, meta, links}`. Unknown query params are now a 400 rather than being silently dropped, because an ignored filter over-returns data. Errors are RFC 9457 problem documents on this prefix only; every other route keeps the shape its callers already parse `links` is what makes the rest deferrable. The dashboard hook follows the server's `links.next` instead of computing `page + 1`, so moving this to cursor pagination later changes the links and nothing the client does. That matters because the inner scan is a sliding window, so offset paging can currently skip or repeat an end user across pages; the fix is a follow-up, and the hypermedia means it will not be a breaking one Cursor mode, `sort`, `include`, ETag / `If-None-Match` and the generic `ListSpec` framework are all deliberately out of scope here. They are additive or internal, so none of them needs to beat the release * fix(management): stop emitting a dead docs link in problem documents The RFC 9457 `type` was `https://docs.litellm.ai/errors/<slug>`, copied from the standard's own error example. That path is a 404 and there is no docs section behind it, so every error body shipped a broken link RFC 9457 only requires `type` to identify the problem type; it encourages, but does not require, that dereferencing it yield documentation. An https URI makes a promise we are not keeping, so use `urn:litellm:error:<slug>` instead, which carries the same machine-readable identity with nothing to resolve. Switching to an https base later is a contract change for anyone matching on `type`, so that should wait for pages that actually exist A test pins the identifier against regressing to an https docs URL, since the existing assertion built the expected value from the same constant and would have stayed green whatever it held * fix(proxy): close the adversarial-review findings on the tool spend rollup Three fixes from an adversarial review of this branch, each at the owning seam rather than the report site. The flush retried DB_CONNECTION_ERROR_TYPES, which includes ReadTimeout. A ReadTimeout is the committed-but-unacked case: the review reproduced the engine abandoning the transaction open on the pooled connection, the retry stacking its statements into it, and one commit applying both increment sets while the flush reports success. The retry now covers only ConnectError, the one failure that proves the statements never reached the database; post-send failures drop the batch with an error log. The docstring no longer claims an idempotency the pattern does not have. The same hazard exists in the untouched daily spend writer and is left for its own change. get_tool_calls_from_response read choices[0] only, so a tool invoked in a later choice of an n>1 response earned spend but never reached the rollup, the index, or the registry. Choice scope is now an explicit parameter: accounting passes include_all_choices=True because every choice costs money; guardrails keep the primary-choice default because they rebuild the primary assistant message. First multi-choice fixtures in the suite pin both scopes. maxBarSize=64 had been added to the shared BarChart unconditionally, resizing every existing consumer. It is now a prop; only the tool spend charts opt in. The legend flex-wrap changes stay global because clipping overflow was a defect, not a preference. * chore(typing): clear 2.7k basedpyright Any errors across 15 hotspot files Replace Any-typed seams with real types in the files carrying the highest reportAny/reportExplicitAny density: typed Prisma read helpers in the MCP db layer and verification token repository, TypedDicts for OAuth credential payloads and aggregated spend rows, a DailySpendRecord protocol for the daily activity endpoints, and concrete request/response types in the volcengine, openai evals, azure batches, azure_ai count_tokens, and ocr transformation modules. Modernize touched annotations to PEP 604/585 forms. No casts, no type: ignore, no noqa, no new Any annotations, no behavior changes. Whole-tree basedpyright: reportAny 27,005 -> 24,427, reportExplicitAny 7,439 -> 7,280, no rule increased anywhere. Budgets ratcheted: basedpyright -2,869, ruff-strict -1,505, type-discipline -167. * fix(install): pass an explicit Python version request to uv tool install uv selects an interpreter before resolving dependencies, so with no --python request the stock macOS /usr/bin/python3 (3.9.6) satisfies the unconstrained request and resolution then fails against litellm's requires-python (>=3.10,<3.15) instead of downloading a managed Python. Request the requires-python range explicitly in install-cli.sh and install.sh so uv reuses a compatible system interpreter when present and downloads a managed one otherwise. The manual-fallback hint in the die message carries the same flag so it no longer reproduces the failure. * fix(management): cover the new control plane route in CI's two guards Both failures are from this branch, not pre-existing The component allowlist test asserts the gateway and backend route sets union to the whole app, so any route on neither is a 404 on both pods. Allowlist the `/management/v1/` prefix on the backend, next to the other control plane entries, so every resource that moves under it later is covered without a per-resource edit The otel handler test builds its request as a SimpleNamespace carrying only `state`. The validation handler now reads `request.url.path` to decide whether the caller is on a surface with its own error contract, so the fake needs a url; a real Request always has one, which is why the handler does not guard for it The control plane branch returns early, and nothing covered that it still closes the dangling SERVER span first, so those requests would have leaked a span apiece. Added a case that pins it; removing the close call fails it * test(proxy): pin both branches of the validation exception handler Same cause as the otel handler test: this file builds its request as a SimpleNamespace carrying only `state`, and the validation handler now reads `request.url.path` to pick an error contract, so the fake needs a url While here, cover what the two existing tests do not. They only exercise the proxy-wide 422, and the control plane's 400 problem document was reachable only through the route test, which registers its own copy of the handler in a local app rather than the real one. Two cases now pin the real handler directly: a `/management/v1` path returns problem+json with a `detail` string, and paths that merely resemble the prefix (`/management`, `/v1/management/foo`) keep the 422 shape their callers parse * chore(deps): bump gitpython to 3.1.55 and brace-expansion to 5.0.8 gitpython arrives transitively through mlflow-skinny; re-resolved with uv so the lock moves that one package only. brace-expansion is a dev-only transitive dep already pinned in the dashboard 'overrides' block, so the pin is bumped alongside the lockfile to keep the change durable across reinstalls. 5.0.8 narrows its engines range from '18 || 20 || >=22' to '20 || >=22'; the dashboard already requires node >=20.9.0 and every CI job pins node 20, so nothing loses support. * fix(e2e/ui): resolve dashboard base URL from env instead of hardcoding localhost (#34739) * fix(guardrails): compress content-parts messages in headroom guardrail Anthropic-format requests translate to messages whose content is a list of part dicts, which the headroom compression service's transforms silently skip (they only rewrite string content), so compression never applied to Anthropic client traffic while the guardrail still reported itself as applied. Flatten all-text part lists to plain strings for /v1/compress and restore the original shapes from the response: untouched rows keep their exact original parts, a rewritten row collapses to one part carrying the last declared cache_control breakpoint (a breakpoint caches the prefix ending at its part, so the last one and its TTL still describe the merged row). Rows with any non-text part are never flattened, since merging text across a non-text part would move a later breakpoint to the other side of it; they pass through the service untouched, matching its own behavior for non-string content. Flattening and write-back use the shared content_text helpers that compresr's breakpoint fix also uses. Resolves LIT-4795 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(proxy): stop retrying post-send ambiguous DB errors in every spend writer Resolves LIT-4823. An adversarial review reproduced against real Postgres that a batched increment upsert stalling past the prisma engine timeout leaves its transaction open on the pooled connection; the retry draws the same connection, its statements stack into the still-open transaction, and one commit applies both increment sets while the writer reports success. httpx.ReadTimeout is exactly that post-send case and every spend writer retried it. DB_RETRY_SAFE_ERROR_TYPES (ConnectError only, the failure that proves the statements never reached the database) is now the single owner of what a non-idempotent writer may retry. All seven entity and daily spend writer retry arms and the tool usage flush consume it. DB_CONNECTION_ERROR_TYPES is unchanged for the idempotent spend-log writer, whose create_many with skip_duplicates may safely retry the full tuple. The corruption was reproduced on update_daily_user_spend (seeded 10|100|1, expected 11|110|2, observed 12|120|3); the new policy tests pin that a ReadTimeout drops the batch loudly on the first attempt and a ConnectError still retries. * ci: publish a generated JSON schema for model_prices_and_context_window.json * fix: match exact class in callback dedup so a custom subclass does not block a built-in logger (#34804) * fix(prometheus): populate cache write token metrics for OpenAI-style usage (#34803) litellm_provider_cache_creation_input_tokens_metric only read the Anthropic-style top-level usage.cache_creation_input_tokens and had no prompt_tokens_details fallback, unlike its cache-read twin. OpenAI models that bill prompt cache writes report them only in prompt_tokens_details.cache_write_tokens, so the counter never fired for them. Resolve provider cache read/write tokens through a shared helper that falls back to prompt_tokens_details.cache_write_tokens (canonical) then cache_creation_tokens when the explicit top-level field is absent, and give litellm_input_cache_creation_tokens_metric the same fallback for raw usage dicts that only carry cache_write_tokens * fix(db_scripts): pin the tool spend backfill session to UTC The backfill compares the naive start_time column against a timestamptz cutover, and that coercion follows the session time zone, so a non-UTC session shifts the cutover boundary by the offset. Pinning the session makes the whole script timezone-independent. The date bucketing itself was already safe: to_char on a timestamp without time zone ignores the session time zone and the stored values are UTC * ci(lint): raise node heap for the basedpyright budget check basedpyright's inference load now exceeds node's ~4GB default heap cap on ubuntu-latest once the Any hotspots carry real types; the node process died with a JS heap OOM, emitted nothing, and the gate refused the vacuous run. 12GB leaves headroom on the 16GB runner. * test: cover volcengine responses and openai evals transformations Exercises the streaming field-fill heuristics, model_construct fallbacks, and the get/cancel/delete/list request and response transforms that had no tests. * test(e2e): stop racing control-plane writes across the mcp, a2a, guardrail and passthrough suites (#34833) * test(e2e): wait for MCP tool discovery instead of racing it /v1/mcp/server returns as soon as the DB row is written, but the gateway runs the initialize + tools/list handshake against the upstream lazily, on the first request that needs it. Every MCP test read tools/list immediately after registering, so it raced that handshake. The gateway reports a server it has not discovered yet exactly like a dead one: it catches the per-server handshake exception and returns an empty tool list. The tests asserted on a single read, so the race surfaced as "granted key never saw search_datadog_logs; tools=frozenset()" while a sibling test against the same upstream in the same run passed. Add McpClient.await_tool, which polls tools/list to the suite's existing poll_timeout and returns the qualified tool name, and route the four discovery sites through it. An unreachable upstream or an unapplied grant still fails, and the failure now names the last tools/list result. Refs LIT-4821 * test(e2e): wait for a2a agents to reach the data plane after registration POST /v1/agents is a control-plane write; the /a2a/{agent_id} routes that serve the card and run message/send are data plane and only see the agent after the next DB reload. Every test registered an agent and immediately read its card or sent it a message, so the first data-plane touch could 404 on the agent it had just created. register_agent now waits for the card to become servable before returning, the same way ProxyClient.create_model waits for a new model, so callers do not each have to poll. Registration failures skip the wait, leaving the two rejection tests unchanged. A genuine propagation failure now fails naming the agent id and the last card read rather than as a bare 404 on whichever /a2a call ran first. Refs LIT-4821 * test(e2e): wait for presidio guardrails to sync before asserting masking Registering a guardrail is a control-plane write; the data-plane worker that serves /chat/completions only picks it up on its next periodic DB sync (~30s), so the first call after the create ran against a worker with no guardrail and passed the raw email straight through. The tests asserted on that first call, so they read in-flight propagation as a PII leak. Confirmed directly against a live proxy: the same call is unmasked at t=0s and masked at t=8s, and the presidio analyzer itself correctly returns EMAIL_ADDRESS with score 1.0 the whole time. The MCP guardrail suite already documents and waits out this exact sync delay; presidio never got the same treatment. Poll the call until the placeholder replaces the PII, so the assertions judge the synced state. A guardrail that never masks still fails, on the last unmasked content. pre_call and post_call now pass repeatably. Refs LIT-4821 * test(e2e): drop the presidio logging_only check pending LIT-4841 pre_call and post_call masking both pass once the guardrail-sync wait is in place, but logging_only left the raw email in the OTEL span's gen_ai.input.messages on every attempt across a full poll deadline. Keeping an assertion against known-failing behavior just turns every run red, so the cell is tracked in LIT-4841 instead. The registry row stays, so guardrail.presidio.logging_only.masks now reports as an uncovered gap rather than silently disappearing. Refs LIT-4821, LIT-4841 * test(e2e): wait for guardrail sync in bedrock, moderation and block-code checks All three asserted on the first call after registering a guardrail, so they were served by a data-plane worker that had not synced it yet (~30s DB poll) and read in-flight propagation as a guardrail that failed to block. Verified directly: the openai_moderation guardrail lets a flagged prompt through at t=0s and returns "Violated OpenAI moderation policy" at t=8s. The reasoning-only responses noted in triage (content=None with reasoning_tokens set) were a symptom of the same thing, not the cause; these are pre_call guardrails, so a synced guardrail rejects the request before the model runs. Add poll_until_blocked to guardrails_client for the two that surface a non-success status, and poll on the block marker in the block_code_execution check, which replaces the reply rather than erroring. All eight guardrail tests now pass. Refs LIT-4821 * test(e2e): drop the openai prompt-cache check pending LIT-4841 Prompt caching never engages through the proxy: cached_tokens is 0 on every repeat, while the identical payload sent straight to OpenAI reports 3615 cached tokens on the second call. Pinning prompt_cache_key on the proxy request restores caching (3328 tokens), so something varying per request is defeating OpenAI's automatic prefix cache. That is a product bug with a direct billing cost, tracked in LIT-4841. The registry row stays, so llm.chat_completions.openai.prompt_cache_5m.nonstream.works now reports as an uncovered gap instead of failing every run. Refs LIT-4821, LIT-4841 * test(e2e): drop the responses metadata redis-ttl check It failed on a Redis read timeout against the stage serverless cache (berrie-litellm-stage-ieib2i.serverless.use1.cache.amazonaws.com:6379), a reachability problem this suite has hit before rather than a proxy defect the assertion can pin down. The file held only this test. Its other cell, llm.responses.openai.basic.nonstream.works, is still covered by test_responses_e2e.py; other.config.responses.metadata_redis_ttl_bounded becomes an uncovered registry row, taking headline coverage 314/431 -> 312/431. Refs LIT-4821 * test(e2e): fix passthrough header propagation and openai body, drop the cost check Three separate problems behind the two passthrough failures. The header test 404'd because POST /config/pass_through_endpoint is a control-plane write and the worker serving the route only registers it on its next config reload; measured at ~18s on a live proxy. Wait for the route to stop 404ing before calling it. The readiness probe reuses the master key and omits anthropic-version so polling does not bill a completion per attempt. The openai passthrough body sent max_tokens, which the gpt-5 family rejects outright ("Unsupported parameter: 'max_tokens' is not supported with this model"). Confirmed against OpenAI directly: max_tokens 400s, max_completion_tokens 200s. Passthrough forwards the body untouched by design, so the body was simply wrong. test_openai_passthrough_nonstreaming_logs_cost still finds no SpendLogs row for its call_id after the fix, so it is removed rather than left red; the gemini and anthropic passthrough cost checks still cover that path. Passthrough suite is 8/8 green. Refs LIT-4821 * fix(router): release the pre-routing strategy slot when a deployment is replaced or deleted Auto-router-family deployments live in two structures: the model_list, and a pre-routing strategy registry keyed by (model_name, tags). Removing a deployment dropped it from the model_list without releasing its registry slot, so the re-add that follows hit the "already exists" guard in _register_pre_routing_strategy and ignore_invalid_deployments swallowed it. The deployment came out and never went back, while the DB row and the endpoint response both looked fine. Only a restart healed it, and under multiple replicas each pod diverged into holding a different subset of routers. Removal now releases the (model_name, tags) slot from every strategy registry, in both upsert_deployment and delete_deployment, guarded on the auto_router/ prefix so removing a regular deployment cannot evict a router that merely shares its model_name. Releasing from every registry rather than the first match is what makes this correct for hybrids: registration is one-to-many, since a complexity router configured with adaptive is also registered in adaptive_routers under the same key by the deferred finalize pass. Releasing only the first match left that adaptive strategy live, so a deleted or replaced alias stayed routable through it. Adaptive post-call hooks are rebuilt whenever the adaptive registry changes, not only at the end of set_model_list. The hook set is defined as exactly one hook per registered adaptive router, so a released router stops recording turns instead of holding a hook bound to a strategy nothing points at any more. The swallowed upsert failure is logged at warning instead of debug, which is below the default log level and left this failure with no observable signal anywhere. delete_deployment resolves the outgoing deployment before popping it, and a resolution failure no longer aborts the removal; previously an entry that failed validation would have been left in the model_list permanently. delete_model drops its blanket pop across all four registries. That predates this change and over-evicts: it removes every tag variant registered under the name while only one is being deleted, and nothing reloads on that path to restore the survivors. delete_deployment now handles it correctly and tag-scoped, so the endpoint-level eviction and its helper are removed rather than left to mask it. * fix(responses-bridge): return CustomStreamWrapper from the completed-response stream helper * fix(router): rebuild the adaptive companion when an upserted complexity router participates in adaptive routing The finalize re-run in upsert_deployment keyed off the auto_router/adaptive_router prefix only, so editing a complexity router with adaptive enabled released its adaptive_routers entry (and post-call hook) without rebuilding it: complexity routing kept serving while bandit recording, DB persistence and /adaptive_router/state went silently dark until the next full reload. Gate the re-run on a participation predicate that mirrors both arms of the finalize pass, drop the import that pass no longer uses, and pin the registry helpers with direct contract tests * fix(ui): validate default team values in Default User Settings (#34815) * fix(ui): validate default team values in Default User Settings The Default User Settings form accepted any free-text team id, and the proxy persisted it without checking the team exists. New users were then silently never added to the default team because the consume-time 404 from team_member_add was swallowed at debug level. Backend: PATCH /update/internal_user_settings now rejects unknown and duplicate team ids with a 400 naming them, before any persistence or team budget side effects. Team-add failures in _add_user_to_team now log at ERROR with user and team ids. UI: DefaultUserSettings rewritten as a shadcn + react-hook-form + zod form following the org-settings pattern. The team id free-text input is replaced with a searchable server-backed team picker, so only existing teams can be selected; zod blocks empty and duplicate rows. The shared deriveErrorMessage helper now unwraps the HTTPException detail.error shape so backend validation errors surface readably in toasts. * fix(ui): restore read-only view with Edit Settings toggle on default user settings Parity with the pre-migration form: the tab renders a read-only summary of the saved defaults, Edit Settings opens the RHF form, Cancel discards pending edits and returns to the summary, and a successful save returns to the summary showing the new values. Model sentinel labels in the summary are derived from ModelSelect's now-exported special values instead of duplicating the strings. * refactor(ui): rename MODEL_SELECT_SPECIAL_VALUES_ARRAY to MODEL_SENTINEL_OPTIONS * fix(ui): move Edit Settings into the card header action slot * fix(router): repair deployment indices before releasing strategies on delete delete_deployment resolved the outgoing deployment through get_deployment before popping it, and ran the strategy release before repairing the index maps. Both halves of that ordering could leave the router inconsistent. A resolution failure meant the entry left the model_list with its registry slots still held, so the alias stayed routable and the name could not be reused; a failure inside the release meant the outer handler returned None with the entry already popped and model_id_to_deployment_index_map never repaired, breaking every later lookup and delete until a restart. upsert_deployment already had this right: it pops, repairs the caches and indices, and only then releases the slot. delete_deployment now follows the same sequence and resolves the deployment from the item it just popped rather than through a lookup that can fail. Releasing the slot is secondary to structural integrity, so it runs last and a failure there is logged instead of abandoning a removal that has already happened. * fix(vertex_ai): source managed-file read bucket + credentials from per-model litellm_params Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): classify all 4xx HTTPException guardrail blocks as intervened (#33821) * fix(guardrails): classify all 4xx HTTPException guardrail blocks as intervened * fix(guardrails): narrow HTTPException block classification to 400/403/422 --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(jwt_auth): grant only /v1/messages routes to JWT teams by default, not all anthropic_routes * chore: update Next.js build artifacts (2026-07-28 00:06 UTC, node v20.20.2) (#34859) * test: unstale the reasoning-effort grid count and the responses bridge test (#34868) * test(reasoning-effort-grid): bump cell-count assertion for claude-opus-5 The claude-opus-5 grid entry added in ae81625ee6 raised the Anthropic direct route to 31 model combos, but test_grid_cell_count still expected 30, so the suite went red on the tripwire rather than on any behavior change. * test(openai): swap the retired deep-research model out of the bridge test OpenAI shut down o3-deep-research and o4-mini-deep-research on 2026-07-23, so the live call in this test now comes back as a 400 'Model not found'. The test was never about deep research specifically; the bridge fires on any model whose cost-map mode is "responses", so it now uses gpt-5.5-pro, the newest responses-only OpenAI model, and is renamed to say that. gpt-5.5-pro was confirmed present on the CI account with an authenticated GET /v1/models before being picked. * test(e2e): realign Admin UI specs with the MCP dialog and keyless landing (#34870) Both specs assert against UI that has since moved, so they fail on selectors rather than on behavior. The MCP discovery modal became a shadcn/Base UI dialog when mcp-servers migrated off antd, so `.ant-modal` no longer matches it; locate it by its dialog role instead. The create form below it is still an antd Modal and keeps its existing locator. The no-team internal user has no keys, and a keyless non-admin is now sent to /ui/connect on the post-login landing, which has no sidebar. Wait for that redirect to settle, then navigate to the keys page explicitly; the redirect is gated on the ?login=success marker that the fresh navigation drops, so the dashboard sticks and the rest of the test is unchanged. * test(e2e): unblock the ui suite, fix the mcp registration race, park two known product bugs (#34853) * test(e2e): let the ui suite run from a read-only cwd The playwright suite never executed on stage. It died in globalSetup before a single test ran, and the reported error was a red herring. /app/e2e/ui is a read-only filesystem in the packaged e2e image (the image runner already redirects playwright's own artifacts to TMPDIR for this reason), but the suite wrote three things relative to cwd: the per-role storageState files, the failure-screenshot directory, and the html report. Reproduced in the pod: storageState raises EROFS, mkdir test-results raises ENOENT. Worse, the catch block that exists to capture a screenshot threw its own ENOENT while handling a failure, so the real login error was replaced by a filesystem error. That is why the run looked like a missing directory rather than whatever actually went wrong. Route every artifact through ARTIFACT_DIR (E2E_UI_ARTIFACT_DIR, default "." to keep run_e2e.sh behavior unchanged), make the diagnostic screenshot best-effort so it can never mask the underlying failure, and point playwright's reporter and outputDir at the same place so a bare `npx playwright test` works there too. fixtures/users.ts had its own copy of the five storageState filenames; it now re-exports the ones from constants so the paths have a single definition. Verified in the read-only pod: both writes fail before, both succeed after. 85 tests enumerate and tsc --noEmit is clean. Refs LIT-4821 * fix(e2e): create the ui artifact root before writing into it storageState() does not create missing parents, and nothing created ARTIFACT_DIR itself. Pointing E2E_UI_ARTIFACT_DIR at a writable path that did not exist yet therefore failed with ENOENT on the very first role's snapshot, before any UI test ran; the same class of failure the artifact-dir change was meant to remove, just moved one level up. Reproduced: writing admin.storageState.json into a missing directory raises ENOENT. My earlier pod verification masked this because the probe called mkdirSync itself, which the real code path never did. mkdir the root once at the top of globalSetup, before the login loop. recursive makes it idempotent, handles nested paths, and keeps the default "." a no-op. Playwright creates its own outputDir lazily, so globalSetup is the only place that needs this, and migration.serverRootPath.globalSetup delegates here so it is covered too. * test(e2e): skip the mid-conversation cache checks pending LIT-4873 A mid-conversation role="system" reminder invalidates the prompt cache on the vertex_ai, azure_ai and bedrock_invoke Messages paths. Measured on the reminder turn, same conversation shape throughout: direct to api.anthropic.com 7013 read cache preserved litellm -> anthropic/claude-opus-4-8 7013 read cache preserved litellm -> vertex_ai/claude-opus-4-8 0 read cache destroyed and the Vertex control with the same added assistant/user turns but no reminder reads 7013, so it is the reminder on the non-first-party paths and not the extra turns. Anthropic keeping the cache rules out provider behavior; litellm's first-party anthropic path keeping it rules out the shared Messages transform. That makes these assertions correct and the failure a real billing bug, so the tests are skipped rather than weakened; the bodies stay intact and must be restored unchanged with the fix. Registry rows are left in place, so the three mid_conversation_system.nonstream.cache_hit cells report as uncovered gaps. Skips are decorators rather than a pytest.skip() inside the shared helper: a mid-function skip fires only after setup has already registered a real deployment via /model/new and left the rest of the body unreachable. Only Vertex was measured end to end. Azure Foundry and Bedrock Invoke are inferred from matching nightly failures and should be confirmed with the fix. Refs LIT-4821, LIT-4873 * test(e2e): make MCP and prometheus e2e tests robust to data-plane sync lag (#34854) * test(e2e): harden harness and tests against data-plane pod churn A stage autoscaler scale-down produced a 2s window of ALB 502s that killed six budget tests on their first management call, and a freshly scaled-up pod that had not run its 30s DB object sync yet failed two MCP tests and one prometheus cardinality test. Retry transient gateway errors (502/503/504, connection errors) once at the shared e2e_http dispatch seam, poll MCP server registration to the poll deadline instead of asserting a single-shot listing, anchor the MCP guardrail full-sync wait to the later of the guardrail and server writes, and turn the prometheus alias poll into a drive-and-scrape convergence loop that re-sends traffic for missing aliases and unions results across scrapes * test(e2e): drain request body in retry stub handler so keep-alive reuse cannot misparse leftovers as requests * revert(e2e): drop the transient-502 retry seam A raw 502 during a pod scale-down is what a real client sees, so the suite retrying past it hides an availability gap instead of flagging it. The gateway-side fix is graceful drain on the deployment; until then the failures are signal * test(e2e): cap per-alias driver re-drives in the prometheus cardinality poll Bounds worst-case provider spend to 4 completions per alias while scrapes keep polling to the deadline; counters persist on whichever pod served them, so the cap costs no convergence unless that pod dies * test(e2e): drop driver re-drives from the prometheus cardinality poll The per-key cardinality contract is process-local and counters persist on whichever pod served the driver call, so unioning aliases across free scrape polls converges without re-sending billable traffic. The residual gap, a pod dying inside the poll window, is deferred to direct per-pod scraping * fix(auth): resolve managed batch/file deployment model_id to model name for team access checks * test(auth): cover managed batch/file team access denial end to end Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Fix cache leakage card layout to keep date picker on right (#34885) * Fix cache leakage card layout to keep date picker on right and prevent content overlap Removes flex-wrap and mt-3 to ensure date picker stays pinned to the right side of the card header regardless of zoom level, preventing it from covering card content below * Remove overflow-hidden from Card to allow dropdowns and overlays to display fully Fixes date picker dropdown being clipped when opened in cards like the Cache Leakage Card. By removing overflow-hidden from the Card container, popovers, dropdowns, and other overflow content can now display properly without being clipped by the card boundaries. * Make cache leakage card descriptions consistent with line clamping Adds line-clamp-2 to ensure both 'by model' and 'by virtual key' cards maintain consistent height. Removes conditional anthropic-specific text that caused height variations between dimensions. * fix(gateway): route /a2a through the gateway component (#34958) * fix(gateway): route /a2a through the gateway component A2A message-send runs the completion bridge, an outbound LLM call, but the ingress only listed /v1/a2a so the serving routes at /a2a/{agent_id} fell to the backend catch-all. Backend pods hold no provider credentials, so every invocation died with a missing-provider-key auth error while the same call succeeds on the gateway fleet. Adds /a2a to the ingress gateway prefixes and the gateway route allowlist, plus a parity test so an ingress prefix that the gateway trims can never reappear * revert(test): drop the allowlist parity tests --------- Co-authored-by: yuneng-jiang <yuneng@berri.ai> * fix(e2e): poll for both spend rows before asserting the cache-hit contract (#34968) The cache-hit and paid rows for the two driver calls flush from different pods on independent update_spend timers, so waiting only for the cache-hit row can return a half-arrived result set where the paid-row assertion then fails on an empty list. Requiring both row kinds in the poll predicate lets the existing deadline absorb the slower flush without weakening any assertion * feat(mcp): manual authorization-code delivery for headless MCP clients The aggregate gateway DCR flow ends in a 303 to the client's loopback redirect_uri. When the MCP client runs on a browserless machine (EC2, SSH box, container) the user authorizes from a browser on another machine, so the 303 dereferences the wrong loopback and the code never reaches the client. The connect banner now offers manual delivery for loopback clients: the finish form posts delivery=manual and /authorize/complete renders the callback URL on a no-store page instead of redirecting. The user pastes it into the client (Claude Code v2.1.191+ accepts a pasted callback URL) or fetches it from the client machine's terminal. Manual codes keep the same sealing, PKCE binding, and single-use guard, with a 5 minute expiry instead of 2 to survive the copy-paste hop; the used-code marker TTL derives from the code's own remaining lifetime so the single-use property holds for the full 5 minutes. The default redirect path is unchanged. Resolves LIT-4863 * test(e2e): skip passthrough headers test until stage can route custom paths to provider creds (#34980) * feat(ui): mark Cost Optimization as beta in the left nav (#34984) * fix(proxy): skip team model aliases that point at deleted deployments A team's model_aliases can map a public name like gpt-4 to the internal routing key (model_name_{team_id}_{uuid}) of a team deployment that has since been deleted, e.g. after replacing per-team duplicates with one gateway-level model. The pre-call rewrite then sent every request to a name the router cannot serve, failing with "no healthy deployments for model_name_..." even though the requested name still resolves at the gateway level. The rewrite is now skipped when the alias target has no live deployment in the router delete_model also skipped the team alias scan for internal-shaped names on the assumption they can never be alias values, which is exactly the shape legacy team model aliases have, so deleting a legacy team model left the stale alias behind. The scan now always runs, and a public name that still resolves to a live router deployment (e.g. a shared gateway-level model group) stays in team.models so the delete does not revoke the team's access to it * fix(proxy): keep team model aliases while a surviving replica serves the deleted name Scrub aliases on delete only when the deleted deployment's model_name no longer resolves in the router. A legacy load-balanced team model can have several deployment rows sharing one internal name; deleting one replica must not remove aliases that still route to the survivors, in any team * fix(proxy): avoid DB outage during planned RDS IAM rotation (#34749) * fix(proxy): warm rotate Prisma client for IAM refresh * fix(proxy): drain Prisma operations during IAM rotation * fix(proxy): bound the drain wait when retiring a replaced prisma engine A replaced engine waited indefinitely for its drain tracker to empty. Hung queries self-release via prisma's 30s default HTTP timeout, but a transaction whose owner is hard-cancelled before commit/rollback leaks its drain count forever, keeping the retired engine and its DB connection pool alive indefinitely; at one rotation per 12 minutes such engines accumulate. Cap the wait at 90 seconds, which exceeds every legitimate operation bound (30s HTTP timeout, 60s max interactive transaction timeout in this codebase), then kill the engine anyway. Work killed at the deadline degrades to the pre-drain behavior and is retried by the existing reconnect/backoff layers. --------- Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> * ci: tighten deprecation_date pattern to reject impossible months and days * ci: enforce format assertions so calendar-impossible deprecation dates fail validation * feat(prometheus): add service_tier label to latency and spend metrics (#34966) * fix(aiohttp): keep keep-alive connector config when a session is rebuilt (#34962) * fix(helm): pin bundled postgres and redis to the bitnamilegacy images (#34963) Bitnami retired the versioned tags under docker.io/bitnami and republished the archived builds under docker.io/bitnamilegacy, so every install and upgrade of the chart with the bundled database fails to pull docker.io/bitnami/postgresql:16.2.0-debian-12-r6. Repoint the subchart images at the bitnamilegacy copies of the exact builds those subchart versions shipped with, so the on-disk data directory layout is unchanged for existing installs. Pin the subchart dependency ranges to the versions already in Chart.lock. The current bitnami postgresql chart defaults to `tag: latest`, which is PostgreSQL 18 today, so an open-ended range turns a dependency refresh into a major-version jump on an existing volume. Refuse to render when postgresql.image.tag is empty or `latest` while the bundled database is deployed. Starting a different PostgreSQL major against an existing data directory leaves the server unable to boot with no in-place way back, which is how the reported install lost its data. Resolves LIT-4708 * test(e2e): bound the post-/model/new servable wait at 40s _await_model_servable used poll_timeout (120s), the spend/log read-back budget. A stuck model reload therefore stalled every suite that creates a deployment for two minutes before failing Give create_model a fixed harness middle ground: model_servable_timeout=40s, polled every 2s, with each /v1/models call capped at 5s and clamped to the remaining deadline so one slow GET cannot overrun the wait. Happy path still returns on the first listing. Not derived from proxy general_settings or env Transport.get accepts an optional per-call timeout for that clamp. Unit tests cover the deadline arithmetic and clamp without a live proxy (cherry picked from commit c082a0e6488f50978bf5255f6b5298ba7e8fd8da) * fix(e2e): wait one default DB reload interval of continuous listing create_model returned after the first /v1/models hit that listed the model, so chat could still land on a cold gateway worker (numWorkers>1 / peer pod) and 400 Invalid model name. Require continuous listing for the product default add_deployment interval (30s) after first sight so every worker has synced from the DB; first listing still bounded at 40s (cherry picked from commit 7d1ee2ff861b970f6de3f6759ff015947af9d2a1) * test(e2e): drop proxy_client model-servable unit tests Keep the create_model DB-sync wait in the harness; the pure-function unit file is not needed for this PR (cherry picked from commit 89204651d1a4537c6f21550c5ab85448ae0923f8) * fix(e2e): never skip the final deadline-clamped model-servable poll When less than one full poll interval remained in the first-listing budget, the pre-sleep check returned NotServable without another /v1/models call. Sleep only min(interval, time left) so a model that becomes listable in the last seconds of the timeout still gets a clamped final poll (cherry picked from commit 8439195922c913d118cb146409c75cc081d23e6e) * fix(e2e): reject first listing that returns after the 40s deadline A poll may start with remaining budget and still return after started+timeout if the transport overruns its clamp. Recheck the first-listing deadline after the response so a late listing does not open the continuous DB-sync phase (cherry picked from commit 7ff2bcbf1498ee82f7dbe0b4330c1ab48927ed01) * fix(proxy): report when a model write does not survive the post-write reload Every model-write endpoint returned 200 off the DB write alone; a model the reload dropped (ignore_invalid_deployments, or a wholesale reload failure) stayed invisible on every channel at once, which is how the registry-leak defect went undiagnosed for three weeks. ProxyConfig.add_deployment and clear_cache now return whether the reload pass completed, and each write endpoint verifies the rows it wrote are live in this pod's router afterwards, distinguishing a deliberately environment-inactive model via the same predicate the Router's own gate uses. The access-group writers return the mutated id set instead of discarding it * fix(proxy): resolve named credentials on provider-only batch and files calls * test(proxy): cover stale-alias warning dedup and key-cache eviction * fix(proxy): reject model writes that corrupt an auto-router pseudo-model An auto-router deployment's litellm_params.model (auto_router/...) is the discriminator the router loads it by, but the model management endpoints accepted any client-supplied value verbatim; a doubled or stripped prefix made router init fail on the next load and ignore_invalid_deployments silently dropped the deployment. Validate writes that supply litellm_params.model at all three endpoints against the merged params and reject incoherent values with an actionable 400. Classification is extracted to router_utils/auto_router_model_naming.py so the Router predicates and the validation share one source * fix(router): never resolve another team's deployment credentials for shared model names * fix(proxy): honor key-level model allowlist in provider-only credential resolution * test(router): directly cover team-ownership credential filter helpers * fix(anthropic-adapter): open the first content block with the real upstream type so reasoning-first streams start with thinking (#34433) * fix(anthropic-adapter): open first content block with the real upstream type * fix(anthropic): defer blank leading stream deltas * test(e2e): poll MCP tools across multi-worker lag (#35047) * fix(mcp): resolve call_tool by registry without requiring tool map Multi-worker reloads put MCP servers in the registry from the DB but do not re-run tools/list on every process. Gating call_tool on tool_name_to_mcp_server_name_mapping made cold workers 500 with Tool not found after another worker had already listed the tool. Treat a registry match on server id/name/alias as enough; upstream rejects unknown tools * test(e2e): poll MCP register, tools/list, and tools/call across multi-worker lag Stage multi-worker gateways only load MCP servers and tool maps on the process that handled the request. Poll until the server is listed, the tool appears on tools/list, and tools/call is not a cold-worker 500 so key-access and Datadog MCP e2e stop racing the LB * Revert "fix(mcp): resolve call_tool by registry without requiring tool map" This reverts commit 8b56e51e39b876d13d1112efa4130554ddf5f173. * test(e2e): tighten MCP multi-worker lag classifier Only retry tools/call on gateway shapes Tool <name> not found and server_not_found, not any 500 that mentions tool/server not found, so upstream failures are not retried until the poll deadline * test(e2e): drop unit file for MCP lag classifier The live await_call_tool polls already cover multi-worker lag; a separate string-match unit module is not worth keeping * test(e2e): poll MCP tools across multi-worker lag (#35047) * fix(mcp): resolve call_tool by registry without requiring tool map Multi-worker reloads put MCP servers in the registry from the DB but do not re-run tools/list on every process. Gating call_tool on tool_name_to_mcp_server_name_mapping made cold workers 500 with Tool not found after another worker had already listed the tool. Treat a registry match on server id/name/alias as enough; upstream rejects unknown tools * test(e2e): poll MCP register, tools/list, and tools/call across multi-worker lag Stage multi-worker gateways only load MCP servers and tool maps on the process that handled the request. Poll until the server is listed, the tool appears on tools/list, and tools/call is not a cold-worker 500 so key-access and Datadog MCP e2e stop racing the LB * Revert "fix(mcp): resolve call_tool by registry without requiring tool map" This reverts commit 8b56e51e39b876d13d1112efa4130554ddf5f173. * test(e2e): tighten MCP multi-worker lag classifier Only retry tools/call on gateway shapes Tool <name> not found and server_not_found, not any 500 that mentions tool/server not found, so upstream failures are not retried until the poll deadline * test(e2e): drop unit file for MCP lag classifier The live await_call_tool polls already cover multi-worker lag; a separate string-match unit module is not worth keeping (cherry picked from commit c274cf321c5c35c629220a89bb497d15b56f870f) * chore(typing): clear basedpyright Any errors in proxy management endpoints Convert pydantic table-model construction from Cls(**row.model_dump()) kwargs-unpacking to Cls.model_validate(...) across the management endpoint hotspot files (team, key, internal user, scim, model management, spend tracking, auth checks, proxy_server). Unpacking an untyped dict reports one Any-typed argument per matched model field, so each converted site clears 10-35 diagnostics while running the exact same pydantic validation. Conversions were limited to models verified to use pydantic's default __init__; UserAPIKeyAuth and LiteLLM_VerificationTokenView keep their custom kwargs-rewriting __init__ and are untouched. Two locally-verified helper params move from Any to object. Whole-tree basedpyright, measured against the branch point in the same environment: reportAny 24,431 -> 22,741 (-1,690), reportArgumentType 2,189 -> 2,136 (-53), reportUnknownArgumentType 34,370 -> 34,067 (-303), reportExplicitAny 7,285 -> 7,283 (-2); total 154,882 -> 152,834 (-2,048) with no rule increasing anywhere and no per-file increases. No casts, no suppressions, no behavior changes. Budgets ratcheted: basedpyright -2,048 across 4 rules, ruff ANN401 -2. * test: cover the model_validate conversion sites flagged by codecov Add regression tests for the db-fetch paths whose converted construction lines were uncovered: the auth_checks getters (default end user budget, end user, team membership, access group, team by alias, org by alias, object permission, managed vector stores, project), get_all_team_memberships and list_available_teams in team_endpoints, and the proxy admin user info helper. Each test feeds a mocked prisma row through the real function and asserts the validated model's fields, so a bad model_validate conversion on any of these paths now fails a test instead of only dropping coverage. * fix(scim): stop provisioning nested group ids as internal users (#34997) * fix(scim): stop provisioning nested group ids as internal users POST/PUT/PATCH /scim/v2/Groups treated every member.value as a user id, so with the default scim_upsert_user=true an unknown id was auto-created as an internal user. Entra sends nested groups as members carrying "type": "Group", which meant every nested group produced a phantom internal user whose id and email were the group GUID, and those users counted toward licensed seats. Group members are now classified before they are used: members typed "Group" are skipped without a database hit, an id that names an existing team is skipped too (Okta sends untyped ids through filtered paths, so the type alone is not enough), and only ids that resolve to a user, or that resolve to nothing at all, keep today's behavior. The user lookup runs before the team lookup so a user whose id collides with a team id keeps syncing. The type was previously dropped at parse time on POST/PUT because SCIMMember had no such field, and on PATCH because the raw member dicts were reduced to bare ids; both paths now share one resolver and one parser that preserves it. Member removals no longer upsert: a remove of an id we do not know is an idempotent no-op rather than a reason to create a user and immediately drop it, and strict mode (scim_upsert_user=false) no longer rejects it. Removal of an id that is on the roster but has no user row still cleans up membership. Responses now state members are of type "User" instead of emitting a null, and the advertised Group schema documents the members.type sub-attribute. * fix(scim): harden group member classification after adversarial review Removals now bypass classification and drop exactly the ids they name, restoring cleanup of roster entries the old bug left behind. The team-id fallback only applies to untyped members, so an explicit User type always provisions even when the id collides with a team. Member types are normalized before matching; a type other than User or Group only skips when the id is not an existing user. Non-string type values are tolerated as absent on every verb instead of failing validation. Admitted member ids are deduped order-preserving, which also closes a pre-existing duplicate-row hazard on group creation. * fix(scim): only treat scim-managed teams as nested groups A PR reviewer flagged that an untyped SCIM member whose id collides with an admin-created team was silently skipped, suppressing that user's provisioning. SCIM group writes (POST, PUT, and every PATCH) now stamp the team with scim_managed metadata, and the typeless team-id skip only applies to teams carrying that marker or the scim_data blob older PUTs already wrote. Admin-created teams stay unmarked, so a colliding untyped member provisions the user in permissive mode and returns the standard unknown-user 400 in strict mode. Teams SCIM touched before this change adopt the marker on their next group write. * fix(ui): show public model names in usage breakdowns * fix(ui): size object permissions card grid by container width (#35019) The card variant used viewport breakpoints (md:grid-cols-2 lg:grid-cols-3) but every card usage sits in a one-third-width grid cell, so on desktop the narrow card still rendered three internal columns of roughly 100px each and the text spilled out of its boxes. Switch to Tailwind container queries so the internal column count follows the card's own width * fix(proxy): allow /key/update to identify the key by key_alias (#34851) * fix(proxy): allow /key/update to identify the key by key_alias * fix(ui): drop machine-dependent union-order churn from generated schema.d.ts * feat(ui): split failed requests into their own series on the cache dashboard (#34862) * feat(ui): chart failed requests as their own series on the cache dashboard Spend logs for failed requests are stored with an empty call_type, so the Cache Hits vs API Requests chart lumped them into an Unknown bar that read as normal LLM API traffic. The activity query now also returns a per-group failed_rows count (status = 'failure') and the dashboard charts it as a third stacked series, so failures are visibly separate from successful requests and cache hits. The chart data transform moves into a pure summarizeCacheActivity helper with unit tests; header stats keep their existing semantics (cache hit ratio still counts failures in the denominator). * refactor(ui): move cache dashboard aggregation server-side with a typed response The /global/activity/cache_hits endpoint previously returned raw per (key, call_type, model) spend-log aggregates typed as LiteLLM_SpendLogs (wrong), and the dashboard reduced them in the browser: grouping by call_type, relabeling empty call_type as Unknown, and computing the stat card totals. All of that now happens server-side. The SQL groups per call_type and splits cache hits vs successful vs failed requests, a new cache_activity module validates rows into Pydantic models and computes totals plus the key-alias/model filter options, and the endpoint declares a real response_model so schema.d.ts types it correctly. The dashboard consumes it through a typed $api react-query hook (filters ride the query key and are applied in SQL instead of the browser), the hand-rolled summarizeCacheActivity transform and the adminGlobalCacheActivity fetch helper are deleted, and the refresh button now actually refetches. The endpoint is UI-internal (hidden from the public swagger), so the response reshape is not a public API break. * fix(proxy): fall back on empty-string model_group in aggregated usage SQL * feat(ui): shareable log links via log_id query param on the logs page (#34879) * feat(ui): shareable log links via log_id query param on the logs page Clicking a log row now writes ?log_id=<request_id> to the URL, closing the drawer removes it, and loading the logs page with ?log_id= opens the drawer for that log. When the log is not in the loaded page, it is fetched by request_id (the backend already drops the date window for id lookups), so links keep working for logs of any age. Drawer open state derives from the URL, mirroring the models page ?model= pattern. * fix(ui): close the log drawer on brow…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
Problem this solves:
How it solves it:
Relevant issues
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Before (stage e2e run of 2026-07-28 14:28 UTC): the test failed with
AssertionError: the non-cached call should still be chargedand the failure output shows exactly one spend-log row, the zero-cost_cache_hitrow; the paid row for the first call had not flushed yet. The two driver calls round-robin onto different gateway pods and each pod batches SpendLogs writes on its ownupdate_spendinterval timer, so the two rows land in the DB on independent clocks. The poll's predicate was satisfied by the cache-hit row alone, so whenever the hit-serving pod's timer fired first the test asserted against a half-arrived result setAfter (this branch): the poll keeps waiting until the result set contains both a cache-hit row and a non-cache-hit row, so the existing 120s deadline absorbs whichever pod flushes slower. No assertion is weakened; a cache hit that gets charged, a missing
_cache_hitsuffix, or a paid row that never lands (now a poll timeout instead of an instant false negative) all still fail. The next scheduled stage e2e run is the live proof; the 14:28 UTC run is the failing baselineType
🐛 Bug Fix
Changes
tests/e2e/quota_management/spend_tracking/test_spend_tracking_e2e.py: the
poll_logs_for_keypredicate for test_cache_hit_is_zero_cost_and_suffixed now requiresany(cache_hit == "True")andany(cache_hit != "True")instead of the cache-hit row aloneQA runbook
Final Attestation