chore: sync upstream 2026-07-17 - #129
Merged
shudonglin merged 101 commits intoJul 17, 2026
Merged
Conversation
Port of BerriAI#16162 by @dhruvyad onto litellm_internal_staging. OpenRouter sends a usage chunk (including a provider-reported cost field) after the finish_reason chunk. Previously the stream handler raised StopIteration on the first post-finish chunk, so that usage/cost never reached the assembled response and cost tracking fell back to token-based estimates. Carry usage.cost through chunk accumulation, preserve stripped usage in _hidden_params, and propagate the provider cost into _hidden_params["additional_headers"]["llm_provider-x-litellm-response-cost"] so the cost calculator uses it.
The /guardrails/usage/{overview,detail,logs} endpoints resolved guardrails only
from the litellm_guardrailstable Prisma table, so guardrails defined in
config.yaml (which live only in IN_MEMORY_GUARDRAIL_HANDLER) were invisible:
detail 404'd, overview omitted them or rendered them as Custom/Guardrail
orphans, and logs missed their logical-name alias.
Add config-owned accessors (list_config_guardrails, get_config_guardrail_by_id)
to the in-memory handler and use them in the usage endpoints, mirroring the
union/fallback already used by list_guardrails_v2 and get_guardrail_info. Also
preserve guardrail_info when storing a config guardrail (type/description were
dropped at initialize time) and read the Prisma-row / dict / LitellmParams
shapes uniformly.
Resolves LIT-2529
Addresses PR review: _to_dict special-cased LitellmParams and returned an empty dict for any other pydantic model. Switch to isinstance(value, BaseModel) so it coerces any pydantic model uniformly (BaseModel is already imported for the response models), which also drops the now-unused LitellmParams import. Behavior is unchanged for the current call sites.
…ilter startup index The semantic tool filter builds its index by listing every MCP server without per-user credentials, so servers needing per-user auth (interactive OAuth tokens, user-scoped env vars) reject the anonymous tools/list and contribute zero routes. Request-time expansion resolves that auth, so filter_tools received tools the router could never select: an empty index failed open N->N (past 128 tools OpenAI rejects the request outright) and a partial index matched only unavailable tools, stripping every tool from the request. filter_tools now syncs missing tools into the router before matching (building the router when absent) behind an asyncio lock so each tool embeds once, and falls back to the full tool list when matches map to no available tool, consistent with the zero-match fallback. Context-window overflows keep failing closed.
…dict allocation in the fast path
…equesting call Match candidates are restricted to the request's own tool names via route_filter, so routes learned from other principals' listings cannot displace the caller's tools from top_k. Lazy indexing now uses the async aadd flow exclusively; an embedding failure, including a context-window overflow on an oversized description, raises for the requesting call only and never writes the shared context_window_error, so one request cannot poison the filter for every user on the worker. The router is also sized to the configured top_k, which the semantic-router index layer otherwise silently caps at its default of 5.
…shared DataTable Rewrites the three Batch A tables from hand-rolled TanStack + tremor renderers into thin DataTable consumers with a separate ColumnDef module each, matching the Guardrails and Tags migrations. Row actions move into a per-row overflow menu (edit/copy/delete for vector stores; copy for everyone plus admin-gated delete for prompts and skills). The vector stores parent gains an isLoading flag resolved on every exit path so the table shows the shared skeleton instead of flashing the empty state. Table files are renamed to PascalCase and stale eslint bulk-suppressions for the rewritten files are pruned.
Only the name cell and the overflow menu act on a row, matching the unified table pattern; the previous table navigated on any row click
…itellm_fix_stream_reset_empty_200
…r OpenAI, Azure OpenAI, and Bedrock Mantle
…st anchor Adds an admin-configured issuer to MCP servers. When set, OAuth metadata is fetched from the issuer's own origin and adopted only when the document self-attests that same issuer (RFC 8414 §3.3), making token_endpoint, registration_endpoint, and scopes authoritative for the pinned issuer instead of a document the MCP resource server chose. This closes the mix-up where a compromised resource echoes a pinned authorization_url to smuggle its own token endpoint and inflated scopes past the corroboration gate. Discovery is same-authority against the issuer origin, fails closed on a §3.3 mismatch, and does not fall back to resource-rooted discovery. Rows without an issuer keep the existing corroboration-gate behavior unchanged. Backend + schema only; UI field and live-proxy proof follow.
…ear on url/auth_type change + UI Discover the issuer from the upstream and persist it trust-on-first-use (fill-empty-only, frozen thereafter), so admins do not have to type it; an admin-configured issuer always wins and is never overwritten. Re-pointing the server url now clears the discovered issuer and endpoints (matching the existing auth_type-change clearing) so a new upstream re-discovers instead of anchoring on the previous upstream's issuer. Adds the issuer to the per-user OAuth token identity so re-pointing it purges stale tokens. Surfaces the issuer as an optional, auto-discovered, overridable field in the create and edit MCP server forms. C901 gate shows +1 vs staging; that is inherited from the BerriAI#33317 stack base (delta 0 against
The issuer anchor is for the token/registration endpoints only (the RFC 9700 mix-up). Scope selection stays resource-driven per the MCP authorization spec Scope Selection Strategy: _fetch_issuer_anchored_oauth_metadata now validates the issuer document (RFC 8414 §3.3) for the endpoints and separately fetches the resource's advertised scopes (WWW-Authenticate challenge, else RFC 9728 scopes_supported) for the scope value, instead of using the issuer document's own scopes_supported. The resource can influence only the requested scope, which the authorization server and user consent bound (RFC 6749 §3.3), never the token endpoint.
…site When an admin pins an issuer, RFC 8414 section 3.3 makes that issuer the sole authoritative source of the authorization and token endpoints, so a compromised or misconfigured upstream cannot smuggle a token endpoint by echoing the pinned authorize URL. The first cut enforced that only on the database build path; the carry-forward, persistence, config-load, serialization and sanitization paths could still restore or emit upstream-derived endpoints for an issuer-anchored server, which is the class of gap the review flagged. Every site now routes through one predicate. _endpoints_yield_to_issuer returns all-None whenever the issuer is the anchor, so both build paths, has_all_upstream_oauth_fields, needs_discovery and the endpoint merge defer to the issuer. _carry_forward_resolved_oauth_endpoints carries only scopes for an issuer-anchored server and fails closed on endpoints. _persist_discovered_oauth_endpoints skips endpoint writes under the anchor. The two table serializers round-trip the issuer and both non-admin sanitizers redact it. Scope selection stays resource-driven per the MCP authorization spec: _fetch_issuer_anchored_oauth_metadata takes endpoints from the issuer document and scopes from the resource document. The OAuth metadata resolution and corroboration gating for the database build path move into _resolve_table_oauth_metadata so build_mcp_server_from_table stays within the cyclomatic-complexity budget without changing behavior. Regression tests pin the invariant at each site: the issuer overrides stored endpoints even when they are populated, carry-forward does not restore endpoints under the anchor, persistence does not write endpoints under the anchor, a url or auth_type change clears stale issuer-scoped fields even when resubmitted unchanged, the Azure heuristic stays reachable under a required issuer, and anchored metadata takes endpoints from the issuer while scopes come from the resource
…covered Two lifecycle gaps let the issuer trust anchor drift out of sync with the endpoints it governs. Changing or clearing a previously pinned issuer left the authorization_url and token_url that were resolved under the old issuer in the row, so clearing the anchor could revive stale, possibly untrusted endpoints instead of re-discovering. And a build that discovered an issuer trust-on-first-use persisted it to the row while the returned in-memory server kept the issuer unset, so the registry and the row disagreed and the per-user OAuth token identity, which includes the issuer, differed between that build and the next rebuild and forced a spurious re-auth. update_mcp_server now treats a change to a previously pinned issuer the same as a url or auth_type change and clears the auth-flow-scoped endpoint fields that were resolved under it. The trigger fires only when an issuer was already pinned and is now changed or cleared, so establishing one for the first time, including the trust-on-first-use discovery write-back, does not wipe the fields it just resolved. Both build paths, build_mcp_server_from_table and load_servers_from_config, now construct the server with effective_issuer = manual_issuer or the discovered issuer, skipping an origin-fallback guess exactly as the persistence does, so the in-memory object always reflects what the row will hold. Regression tests pin each case: clearing and re-pointing a pinned issuer clear the stale endpoints, a first-time establish preserves the discovered fields, and a build reflects the discovered issuer while an origin-fallback guess is not reflected
…eps endpoints Making the in-memory issuer reflect a trust-on-first-use discovered value fixed the registry/row token-identity drift, but it overloaded a single field: the carry-forward gate keyed on issuer truthiness as a proxy for "endpoints are anchored to a pinned issuer, fail-closed". A discovered issuer is truthy yet not anchored, so a resource-rooted server that had learned its issuer would drop its last-known-good endpoints on a transient discovery blip instead of carrying them forward. Anchoring is now a first-class property rather than a proxy. MCPServer carries issuer_is_anchored, set at both build paths from the single _uses_issuer_anchor definition (a pinned issuer on a discovery auth type). issuer stays the identity value used by the token-identity tuple and the serializers; issuer_is_anchored is the provenance value the carry-forward gate reads to decide fail-closed. The two properties can no longer be conflated, so a discovered issuer keeps its resource-rooted endpoints carrying forward while a pinned issuer still fails closed. Regression tests pin both directions: a discovered-but-not-anchored server restores its endpoints on a discovery blip, an anchored server does not, and the build sets issuer_is_anchored true only when the issuer is pinned
…ing, and thinking+signature streaming (BerriAI#33507)
test(ocr): use mistral-document-ai-2512 in azure_ai OCR tests
…tor-details-fix-8845d6 fix(guardrails): show YAML-defined guardrails in the Guardrail Monitor
…er_chat_tool_format fix(mcp): keep the MCP reference intact when the semantic filter narrows tools
…, gate only bedrock_mantle The openai and azure_openai columns have working credentials in every suite runner, so only the bedrock_mantle column still needs an opt-in flag while the AWS account waits on the Mantle allowlist; COMPAT_GPT_CELLS becomes COMPAT_MANTLE_CELLS. The six tool_use cells now resolve the proxy through claude_code._env like the basic_messaging cells instead of hardcoding LITELLM_PROXY_BASE_URL/LITELLM_PROXY_API_KEY.
The preemptive-401 gate for auth_type=oauth2 MCP servers keyed the challenge on whether an Authorization header was present (not oauth2_headers). Because the header parser classifies any Authorization bearer as an OAuth token before the target server is resolved, a LiteLLM virtual key presented as Authorization: Bearer sk-... suppressed the challenge on a gateway-managed authorization_code server; the session then opened with no upstream token and tools/list masked the failure as 200 with an empty tool list. The same gate also wrongly challenged client_credentials (M2M) servers, which the gateway authenticates by minting its own token at egress. The decision is per oauth2 sub-mode, not per header. Gateway-managed modes never receive a client-supplied upstream token: client_credentials mints at egress so it is never challenged, and gateway-managed interactive (authorization_code, non-delegate) is challenged whenever no stored per-user token exists, regardless of any bearer. Only the delegate/upstream-PKCE mode, where a present bearer genuinely is the upstream token, keeps keying on the Authorization header. oauth2_headers itself is left untouched so the delegate/passthrough egress paths that forward the client bearer are unchanged.
…e-401 gate Grafted from PR BerriAI#33582 (closing as superseded by this PR): drives handle_streamable_http_mcp with real MCPServer objects, parametrized over a stamped client_credentials row and a legacy unstamped M2M-shape row; both must reach the session manager without the per-user token store being consulted
…rriAI#33312) A `lite login` token 429'd with "Budget has been exceeded! Max budget: 0.25" even when no budget was configured anywhere. cli_poll_key stamped the minted CLI session token with litellm.max_ui_session_budget ($0.25) as a fallback whenever the user and team had no budget of their own. That cap was designed for the Admin UI "Test Key" chat pane; the CLI reused the same session-token machinery, so it inherited a playground-sized budget baked into the encrypted token at login (unchangeable without re-login), which trips fast under real CLI/agent use. The cap is also redundant: the token already carries user_id and team_id, so the real user/team budgets are enforced independently at request time. Pass max_budget=None so the CLI token is governed only by those real budgets, and drop the now-dead user/team budget lookups. The UI login token's guard (get_experimental_ui_login_jwt_auth_token) is untouched.
* test(e2e): read datadog log delivery back from the real datadog api (BerriAI#33604) * test(e2e): read datadog log delivery back from the real datadog api * test(e2e): compare datadog-read cost with math.isclose, not bit-equality The response_cost now round-trips through DataDog's attribute indexing pipeline, whose float serialization is not guaranteed to preserve the exact bit pattern the proxy shipped. rel_tol=1e-9 (equal to 9 significant digits) still fails on any real cost discrepancy while tolerating representation drift. Addresses the Greptile P2 on this PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): widen the duplicate-settle window to 30s for real DataDog Against the local sink one poll interval (5s) after the first hit was enough to catch a same-call duplicate, because both events arrived in the same flush batch. Against real DataDog, ingestion jitter can make one call's two events searchable tens of seconds apart, so a 5s settle could let the LIT-4447 duplicate slip past the exactly-one assertion. The reader now keeps re-reading for DD_SETTLE_SECONDS (default 30s, env-overridable via E2E_DD_SETTLE_SECONDS) after the first event appears, returning early only when a duplicate is already visible - more waiting cannot clear it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(e2e): point UI tests at dashboard service; register complexity router Stage gateway 404s /ui; the Next.js dashboard is litellm-ui:3000. Drive playwright against E2E_UI_BASE_URL and wait on login placeholders after client render. Register complexity-smart-router via /model/new when the proxy does not already list it so stage matches compose config * docs(e2e): clarify E2E_UI_BASE_URL should be ALB when ingress splits UI * docs(e2e): prefer single path-routing host for control plane and UI CONTROL_PLANE and UI already default to PROXY_BASE_URL; clarify that stage should set one ALB host rather than three endpoints * fix(e2e): always capture complexity router model_id for teardown Split /model/new from the data-plane wait so a propagation timeout still deletes the control-plane registration (greptile orphan-model concern) * fix(e2e): click exact Login button so SSO control is not matched Playwright strict mode matched both Login and Login with SSO * fix(router): score complexity by difficulty not request length The LLM classifier prompt treated short wording as SIMPLE, so probes like "Is P equal to NP?" stayed on the SIMPLE backend even though the classifier ran. Judge intellectual difficulty so short hard questions route higher * fix(e2e): open key edit via Key ID and wait for team models Key Alias text is not the row open control on the virtual keys table; KeyInfoView opens from the Key ID button in that row. Also wait for a real team model in the edit Models dropdown so we do not race the async availableModels fetch that only has All Team Models on first paint * fix(e2e): keep settled DD events on empty search; bump mcp for OSV Do not let a transient empty DataDog search wipe events already seen in the settle window (Greptile P1). Make the logs-search from window env-overridable via E2E_DD_SEARCH_FROM (Greptile P2). Prefer the mono Key ID button when opening key edit. Bump mcp 1.26.0 -> 1.28.1 so OSV clears the three high GHSA findings on the staging PR * revert: drop mcp lock bump from e2e staging PR OSV mcp upgrade is unrelated to the e2e fixes; leave the dep pin alone --------- Co-authored-by: yucheng-berri <yucheng@berri.ai> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…_gpt_big3 test(e2e/claude_code): add GPT-5.6 Sol/Terra/Luna columns for OpenAI, Azure OpenAI, and Bedrock Mantle
…enge_mode_aware fix(mcp): make the preemptive-401 OAuth challenge decision mode-aware
… 1024 MINIMUM_PROMPT_CACHE_TOKEN_COUNT was a flat 1024 described as "minimum number of tokens to cache a prompt by Anthropic". Anthropic's minimum cacheable prefix is per-model and ranges from 512 to 4096, and it can differ per platform for the same model, so one constant is wrong in both directions is_prompt_caching_valid_prompt gates PromptCachingDeploymentCheck, which is what optional_pre_call_checks: ["prompt_caching"] turns on. When it believes a prompt is cacheable, async_filter_deployments pins routing to whichever deployment previously served that prefix. For a prompt between 1024 and 4096 tokens on Opus 4.6, Opus 4.5 or Haiku 4.5, litellm judged it cacheable and constrained routing while the provider never cached it, so the pin cost load balancing for nothing. In the other direction Fable 5 caches from 512 tokens, so a 512 to 1024 token prefix was refused a pin it had earned The minimum now resolves from prompt_cache_min_tokens in the model cost map, which keeps it current with new models and lets the Bedrock override for Fable 5 fall out of the existing per-entry keys with no special casing. MINIMUM_PROMPT_CACHE_TOKEN_COUNT stays as a global escape hatch when explicitly set, and as the fallback for models the cost map has no entry for async_filter_deployments only ever receives the model group alias, never a model name, so it resolves the threshold from healthy_deployments instead. A group may mix models with different minimums, so it takes the max: a prompt is only treated as cacheable when it clears every member's minimum, because an unnecessary pin is the defect being fixed while a missed pin only forfeits an optimization Gemini context caching shares this gate and has the same defect; its entries are left unset so they keep today's behavior, tracked separately in LIT-4525
…#33628) The policy attachment form fetched /team/list with the caller's own user_id, which the backend treats as a membership filter even for proxy admins. Admins only saw teams they were personally a member of, and the scope validation added in BerriAI#32131 then rejected every other valid team alias as nonexistent. Drop the user_id filter; the policies page is admin-only and /team/list without user_id returns all teams for admin roles. Fixes LIT-4199
get_model_info is lru_cached, so swapping litellm.model_cost is not enough on its own. An earlier test that resolved these models against the remote map, which does not carry prompt_cache_min_tokens yet, leaves cached entries without it, and the stale hit resolves to the default. The assertions would then pass for the wrong reason or fail depending on execution order Clear on teardown as well, so entries these tests warm against the local map do not leak into later tests, matching the fixture already used in test_utils.py Also pin that a wildcard route resolves the underlying model's minimum. That works only because pattern_match_deployments substitutes the real model name into litellm_params before the deployment reaches the check; without the assertion that claim is unpinned and the threshold would silently fall back to the default
…o shared DataTable (BerriAI#33629) * refactor(ui): migrate AI Hub, public hub, and MCP Toolsets tables onto shared DataTable * test(ui): stub skillHubPublicCall in the public model hub networking mock
…he real datadog api (BerriAI#33566) * fix(e2e): make the datadog read-back find what DataDog actually indexes Live verification of the merged BerriAI#33604 against real DataDog (us5) exposed three read-back defects that the local-sink tests could never see; all three fixes are verified against the real API: - Marker search: DataDog consumes the shipped JSON message into the event's attributes and leaves the indexed message EMPTY, so the full-text '"marker"' query matched nothing and every test failed with zero events. The query is now '*:*marker*', which scans all attributes (the marker sits in messages.content); verified to return exactly the event for the call. - Rate limit: the Logs Search API budget is 2 requests per 10s org-wide (x-ratelimit-name logs_public_search_api). Polling at POLL_INTERVAL=5s sat exactly at the limit and the reader hard-failed on the first 429. Searches now pace at DD_SEARCH_INTERVAL (10s default) and a 429 backs off and retries up to 5 times; only non-429 failures stay hard fails. - Envelope status: DataDog re-derives the indexed event status from the parsed payload's status attribute ('success') and normalizes it to its OK severity, so the assertion expects 'ok', not the shipped 'info'. Live run: chat_completions and responses pass every assertion including the exact response-cost cross-check; messages red-pins the LIT-4447 duplicate for real (one call -> two sync-sweep copies + one async batch copy, same request id, confirmed in proxy debug logs). The duplicate is race-dependent, so the pin flickers until BerriAI#33589 lands. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(e2e): datadog log delivery for streamed chat, messages, and responses Rewritten from the dd-sink version (original BerriAI#33566) to judge delivery on what real DataDog ingested, matching the merged BerriAI#33604 conversion: the dd_logs reader searches events back through the Logs Search API and the assertions validate the indexed envelope (source:litellm tag, ok status) and the StandardLoggingPayload fields under the event's attributes. Each streamed test drives one STREAMED call per route, asserts the stream actually streamed (event-stream content type, >0 chunks, no upstream error event), then pins exactly one DataDog event whose payload records stream=true, the aggregated token count, and a response_cost equal to the /spend/logs row for the call - a stream's headers ship before its cost exists, so the spend row is the cross-check anchor, and the spend row and DataDog event must also agree on total_tokens. Coverage registry: adds logging.datadog.stream.exports_metric exercised on chat_completions, messages, and responses. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Update test_datadog_log_e2e.py --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…sage (BerriAI#33533) Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ghest The read gate cannot cause a wrong pin. A deployment is only pinned when the cache already holds an entry for the prefix, and async_log_success_event writes entries against the deployment's real model rather than the group alias, so a model that will not cache a prefix never records one and there is nothing to pin it to That makes this gate purely a cheap short-circuit deciding whether the cache lookup is worth doing, so the threshold must be the lowest minimum in the group. Taking the highest skipped the lookup for a prefix a lower-minimum member had genuinely cached, losing a hit it earned, and protected against nothing. It also broke the Fable 5 direction this ticket is meant to fix: its real minimum is 512, so a group gate stuck at a higher value would skip the lookup for a prefix Fable 5 had actually cached
…t) (BerriAI#33634) * test(e2e): harness fixes for long_context, complexity router, UI, and unit coverage Point long_context_1m at 1M-capable models, harden complexity-smart-router registration and spend-log assertions, fix key models dropdown selectors, and add gateway/lifecycle/transport and claude_code unit tests * test(e2e): harden remaining stage failures in harness Register complexity-smart-router via create_model + callable probe, fix create-key UI navigation race, retry management writes and budget ALB 502s, mark Vertex count_tokens N/A when unsupported, and tighten tool_search model lists for Azure/Bedrock capability gaps * test(e2e): drop claude_code and harness unit tests from this PR Keep management, router, budget, and shared conftest harness fixes only * test(e2e): restore E2E_RESULT pytest_runtest_makereport hook Accidentally dropped in an earlier harness commit; Grafana status history depends on these structured log lines * test(e2e): drop management control-plane write retries Transient 500 retries do not fix the underlying control plane failures * test(e2e): skip stage-red claude_code cells; fix multi-window budget latency Mark the twelve failing claude_code matrix cells skip until product/config lands. Multi-window budget polls gpt-5.5 with max_tokens=1 instead of Claude so the reset wait stays under ALB target idle timeout rather than masking awselb 502s * test(e2e): require exactly one LLM-tier spend row for complexity router Keep alias membership for compose vs stage model names, but assert len(served) == 1 so a leaked classifier sub-call cannot pass. Also pin LIT-4521 skip and align LIT-4522/23/24 skip reasons * test(e2e): harden router callable probe and multi-window budget exhaustion _router_is_callable treated any non-success chat whose body lacked "Invalid model name" as callable, so an unpropagated probe key (401), a generic 502, or a connection reset let the session proceed and hit real "Invalid model name" failures inside the tests. Require a Success outcome instead; the reload-race 400 and every infra/auth error now correctly read as not-callable. The multi-window budget test capped the tight window at 3e-6, which gpt-5.5 exhausts on the first call but a cheaper CHEAP_OPENAI_MODEL might not within the 20-call loop, turning a reset test into a spurious "window never enforced" failure. Drop the tight cap to 1e-9 so the first billed call exhausts it regardless of model price; the roomy 1m window stays at 1.0 and never blocks. * test(e2e): use a tradeoff-decision prompt for the complexity router classifier "Is P equal to NP?" reads to the LLM classifier as a short yes/no question, so gpt-5.5 classified it SIMPLE and the request routed to the openai backend, which made the test fail even though the classifier was running. The tier definitions key on what the request demands, not how hard the answer is, and a short direct question maps to SIMPLE regardless of subject. Swap in "Should I pay off my mortgage early or invest the extra money instead?". It carries none of the heuristic scorer's reasoning/technical/code keywords and stays short, so heuristic scoring still lands SIMPLE (openai), but the LLM reads it as a decision that has to weigh tradeoffs and lands it above SIMPLE, which the config routes to anthropic. Any non-SIMPLE tier serves anthropic, so the classifier only has to avoid SIMPLE for the test to distinguish a real classifier run from the heuristic fallback.
…in_tokens fix(router): resolve prompt cache minimum per model instead of a flat 1024
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
# Conflicts: # ui/litellm-dashboard/src/components/model_hub_table_columns.tsx
The upstream sync merge added Optional[X]/List[X] usages beyond the ruff-strict-budget.json ceiling (UP045/UP006), which only ratchets down. Converts the specific lines the strict gate flagged as newly over budget to X | None / list[X], with no behavior change.
# Conflicts: # uv.lock
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Full -X theirs sync of BerriAI/litellm litellm_internal_staging (99 commits).
Upstream's AI Hub refactor (BerriAI#33629, shared DataTable migration) deleted
ui/litellm-dashboard/src/components/model_hub_table_columns.tsxand moved its columns toui/litellm-dashboard/src/components/AIHub/ModelHubTableColumns.tsx, which dropped our fork-only provider display-name enhancement along the way (same drop-only-the-import pattern already documented in fork-patches.txt for other files). This PR reapplies it at the new location plus the sibling Providers-modal usage inAIHub/ModelHubTable.tsx, updates the corresponding test, and updates fork-patches.txt to track the new file paths.The sync also pulled in enough new
Optional/Listusages to push the ruff-strict-budget ratchet (UP006/UP045) over its ceiling; those specific lines are modernized toX | None/list[X], no behavior change.All other documented fork patches (Dockerfile digest pins, CodeQL config excludes, alert-bridge workflow, anthropic legacy-thinking translation, auth_checks regex escape, content_filter path containment, dependabot.yml, ws override, etc.) were verified present after the merge.
Relevant issues
Linear ticket
Pre-Submission checklist
Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Ran
npx vitest run src/components/AIHub/ModelHubTableColumns.test.tsx src/components/AIHub/ModelHubTable.test.tsxlocally against the merged tree: 2 files, 17 tests, all passing.Ran
python3 scripts/ruff_strict_gate.py --base origin/litellm_internal_staginglocally: OK, every strict rule within its codebase ceiling.Type
🚄 Infrastructure
Changes
Routine upstream sync merge (
-X theirs) plus restoration of the fork-only provider display-name enhancement that upstream's AI Hub table refactor silently dropped, plus the type-hint modernization needed to clear the strict-rule budget ratchet.QA runbook
N/A, infra sync.
Final Attestation