fix: correct max_output_tokens for Claude Haiku 4.5 (vertex_ai 8192→64000, openrouter 200000→64000) - #32197
Open
Code-weaver1 wants to merge 119 commits into
Open
Conversation
Realtime cost calculation computed totals but never populated logging_obj.cost_breakdown, so spend logs and the UI Metrics/Cost Breakdown showed no input/output cost details. Co-authored-by: Cursor <cursoragent@cursor.com>
git diff --name-only includes deleted paths, so a PR that removes a litellm/**/*.py file feeds the gone path to ruff format --check, which exits 123 with 'No such file or directory'. Add --diff-filter=ACMR so only added/copied/modified/renamed files are checked, matching the pattern already used in test-litellm-ui-build.yml.
chore(ci): promote internal staging to main
chore(ci): promote internal staging to main
chore(ci): promote internal staging to main
chore(ci): promote internal staging to main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…BerriAI#31932) Route non-full-admin callers through _sanitize_mcp_server_list_for_non_admin, matching the pattern the fetch and list handlers adopted. Replace the two regression tests that pinned the old partial-blank behavior with a sanitize/full-admin pair mirroring the fetch/list coverage. Resolves LIT-3929
…oauth2 (BerriAI#31736) * fix(mcp): gate OAuth authorize/token/register/discovery on auth_type=oauth2 A non-oauth2 MCP server (notably auth_type=none, access-group gated) has no client_id and no authorization URL, yet the gateway OAuth endpoints did not check auth_type. authorize() raised "client_id is required" before the auth_type was ever examined, and the .well-known discovery builders always advertised authorization_servers / authorization_endpoint / token_endpoint / registration_endpoint, so spec-compliant MCP clients were pointed at an OAuth flow that can never succeed. Add an auth_type != oauth2 guard to the authorize, token, register, protected-resource and authorization-server paths (covering the internal UI OAuth endpoints too). The discovery guard sits after the OAuth pass-through branch so genuine pass-through servers keep proxying their upstream metadata. oauth2 servers are unaffected. * fix(mcp): accurate non-oauth2 message; 404 unknown discovery names to close enumeration oracle Address review feedback on the auth_type gate. The 400 message no longer claims access is governed by access groups, which is only true for auth_type=none; it now states that the gateway runs the OAuth client_id/authorize/token/register flow only for oauth2 servers and that the server is reached using its configured auth_type, which is accurate for every non-oauth2 type (api_key, oauth2_token_exchange, etc.). The discovery gate previously 404'd a named non-oauth2 server but still returned 200 metadata for an unknown name, which both serves a broken document for a typo and lets an unauthenticated caller enumerate non-OAuth server names by comparing 404 vs 200. A named discovery request now returns 200 only when it resolves to an oauth2 server; unknown (or hidden) and non-oauth2 names return the same 404. Root discovery and pass-through servers are unaffected. * Apply suggestions from code review Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
…erriAI#31939) * refactor(ui): drive cache settings form from a typed frontend schema The Cache Settings form was dynamically generated from field metadata shipped by the backend, and read its values back out of the DOM with document.querySelector. That loses type safety and makes client-side validation awkward, which is a poor fit for a form whose shape only changes when a developer edits code. Move the field definitions (name, label, type, default, help text, which redis type they apply to, section, and validation rules) into a typed frontend module and render them through antd Form with controlled state. The GET /cache/settings endpoint is still used to populate current values, and the save/test payload shape sent to POST /cache/settings and /cache/settings/test is unchanged. Per-field validation now lives on each field's antd rules, so an inline error can surface before and on submit; this is where the upcoming Redis URL validation will slot in. The backend's fields output in GET /cache/settings is no longer consumed by the UI, but is left in place since removing it is a separate backend change. * refactor(ui): validate list-field JSON inline so bad input blocks save sentinel_nodes and redis_startup_nodes had no validation rule, so malformed JSON passed validateFields, was caught while building the save payload, and the field was silently omitted; the user's cluster/sentinel config was discarded with no feedback. Add a jsonListRule (same shape as portRule) to both list fields so an invalid value surfaces inline and blocks save. * fix(ui): show valid-JSON examples for cache list fields and clarify the error The Startup Nodes and Sentinel Nodes help text showed Python-style single-quoted examples (e.g. [{'host': '127.0.0.1', 'port': '7001'}]), which the JSON validator correctly rejects, so pasting the example we display failed. Switch both examples to valid JSON with double quotes and change the parse-error message to "Must be a valid JSON array (use double quotes)" so the hint points at the fix. Also add a regression test asserting a numeric field (Database Index) is included in the save payload. * fix(ui): validate numeric cache fields as text so bad input blocks save Numeric fields (Database Index, TTL, Max Connections, Similarity Threshold) rendered as antd InputNumber, which silently coerces non-numeric input to empty. Because the fields are optional, an invalid entry like a full connection URL pasted into Database Index passed validation and was silently dropped from the save payload. Render numeric fields as text inputs with a validation rule (non-negative integer for Database Index and Max Connections, number for TTL and Similarity Threshold), mirroring how Port already works, so invalid input is preserved, flagged inline, and blocks submit instead of vanishing. The save payload still coerces these to real numbers. Adds a regression test for a non-numeric value entered into a numeric field.
…BerriAI#31793) For LITELLM_METADATA_ROUTES (responses, /v1/messages, batches, etc.), the proxy stores admin metadata under data["litellm_metadata"] while user-supplied metadata lives in data["metadata"]. Tags placed in metadata.tags by the caller were never merged into litellm_metadata.tags, causing SpendLogs.request_tags to be empty on these routes Closes BerriAI#31584 Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ging override (LIT-3587) (BerriAI#31905) The security fix in 34e9be1 removed turn_off_message_logging from _supported_callback_params to stop callers bypassing global redaction via the request body. That also killed the documented admin-only per-key or per-team override because both flows resolve through the same allowlist in initialize_standard_callback_dynamic_params. Put turn_off_message_logging back in _supported_callback_params so an admin-configured metadata.logging[].callback_vars.turn_off_message_logging survives into StandardCallbackDynamicParams and can override the global setting for that key or team, as documented at docs/proxy/team_logging#disableenable-message-redaction. Consolidate the metadata traversal so the extractor and the proxy strip walk the same set of client-controllable slots. iter_client_callback_metadata_dicts in litellm_core_utils/initialize_dynamic_callback_params.py is the single source of truth for metadata, litellm_metadata, and litellm_params.metadata; _strip_client_message_redaction_opt_out imports it so a future addition to one side automatically reaches the other. The extractor iterates the helper in reversed order so litellm_params.metadata keeps overriding metadata, matching the pre-refactor merge precedence. Client bypass stays blocked. Restoring the field re-enrolls it in the auth layer's _BANNED_REQUEST_BODY_PARAMS (derived from _supported_callback_params via _build_banned_observability_params), so client submissions at the top level, inside metadata, or inside a JSON-string litellm_metadata all 401 at ingress. is_request_body_safe also now descends into litellm_params.metadata for the same 401 defense against the nested-body attack vector, matching how the metadata and litellm_metadata slots are handled. _strip_client_message_redaction_opt_out runs after the litellm_metadata JSON parse and before the admin callback_vars unpack, so admin values survive while any leftover client-supplied opt-out is dropped when global redaction is on and the key or team lacks allow_client_message_redaction_opt_out. Flip the two dynamic-param e2e tests added by the security fix to reflect the restored override behavior, keeping the invariant that proxy client bypass is stopped by the auth layer 401 above. Co-authored-by: yucheng <yucheng@yuchengs-MBP.attlocal.net> Co-authored-by: Cursor Agent <cursoragent@cursor.com>
…AI#31641) Add an optional per-server max_concurrent_requests that caps how many tool calls LiteLLM sends to one MCP server at once, so batch-processing backends are not overwhelmed by unbounded parallel dispatch. Excess calls queue on a per-server asyncio.Semaphore instead of being rejected. Unset or non-positive means unlimited, preserving existing behavior. Resolves LIT-2749
BerriAI#31986) * fix(release): create tag before release and set make_latest post-publish The Create Release workflow failed for every stable maintenance release while pre-releases succeeded. Two independent bugs were behind that. createRelease was minting the tag from target_commitish, and that path returns "Resource not accessible by integration" (403) to the Actions token, or 404 to a user token, for certain commits (cli/cli#9773). The stable-line tips tripped it; the dev/rc commits happened not to. Create the tag up front with git.createRef and drop target_commitish so the release attaches to the existing tag instead of minting one. A 422 from createRef (tag already exists) is tolerated so re-runs are idempotent. make_latest is silently ignored during the draft-to-published transition (cli/cli#8201), so a backport that published would seize the repo "latest" badge from a newer line. Publish first, then set make_latest in a separate call, and only for non-prereleases. Each operation here is already runtime-proven: git.createRef under the workflow token by prior release-branch jobs, the no-target createRelease and non-prerelease publish and separate make_latest PATCH by a manual 1.89.5 cut. * fix(release): pin tag_name on publish so the draft binding can't reset Pre-creating the tag means the draft is edited while a tag ref already exists, and a draft PATCH that omits tag_name can reset it to the untagged placeholder. Send tag_name explicitly on both updateRelease calls so publish always attaches to the intended tag. * fix(release): fail loudly when the tag exists at a different commit The createRef 422 swallow kept re-runs idempotent but also masked a tag that already exists at the wrong SHA, which would publish the release against the wrong commit silently. On 422, compare the existing tag ref to the intended commit and error on a mismatch, keeping idempotency only for a genuine same-SHA re-run.
…AI#31929) * fix(bedrock): honor ttl for tool_config cache injection points Pass cache_control_injection_points control.ttl through to Bedrock toolConfig cachePoint blocks, matching message/system cache behavior. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(bedrock): drive Claude 4.5+ ttl support from pricing JSON, not regex is_claude_4_5_on_bedrock hardcoded a model-name pattern list that needed a manual update for every new Claude release (it already silently missed Sonnet 5 and Fable 5). Replace it with a lookup against cache_creation_input_token_cost_above_1hr in model_prices_and_context_window.json, which AWS docs confirm tracks the same 1h-TTL-capable model set. Also fixes two bedrock Claude 3.5 Sonnet entries that incorrectly carried that pricing field (their own regional variants didn't have it), which would have made the JSON-driven check wrongly grant them 1h TTL support. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tests): use real Claude Sonnet 4.5 release id in ttl cache-point tests test_add_cache_point_tool_block_passes_ttl_for_claude_4_5 and test_bedrock_tools_pt_passes_ttl_for_claude_4_5 used a fabricated model id (...-20250514-v1:0) that never shipped. This passed under the old regex-based is_claude_4_5_on_bedrock, which matched on substring alone, but fails now that it looks up cache_creation_input_token_cost_above_1hr in litellm.model_cost, since the fake id has no pricing entry. Also force the bundled local cost map in both tests so ttl eligibility reads this branch's pricing data instead of the network-fetched main copy, which lacks the fix until merge. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(bedrock): restore cache and tool config compatibility * fix(bedrock): preserve Sonnet 5 parallel tool config * fix(bedrock): decouple parallel tool support from cache ttl * refactor(bedrock): drive parallel tool use config from JSON, not hardcoded patterns Replace the hardcoded _CLAUDE_BEDROCK_PARALLEL_TOOL_USE_PATTERNS tuple and bedrock_converse_supports_strict_tool_schemas (dead code) with a supports_parallel_tool_use_config key in model_prices_and_context_window.json, matching how is_claude_4_5_on_bedrock already reads cache_creation_input_token_cost_above_1hr from the pricing JSON. New models pick up parallel tool use support automatically when their pricing entry ships with the key set, with no code change required Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(tests): use real model id in parallel-tool-use-without-ttl-pricing test anthropic.claude-opus-4-7-unlisted-v1:0 has no entry in model_prices_and_context_window.json, so bedrock_converse_supports_parallel_tool_use_config returned False and the test died with KeyError on additionalModelRequestFields. Use jp.anthropic.claude-opus-4-7, a real entry that carries supports_parallel_tool_use_config without 1h-TTL cache pricing, which is exactly the decoupling this test exists to cover * test(utils): allow supports_parallel_tool_use_config in pricing schema The misc unit test job validates model_prices_and_context_window.json against the INTENDED_SCHEMA allowlist in test_utils.py, which rejects unknown keys. Add the supports_parallel_tool_use_config key this PR introduced so test_aaamodel_prices_and_context_window_json_is_valid passes again * fix(bedrock): preserve ttl for regional claude models * fix(bedrock): fall back to base model entry when regional pricing lacks capability fields Regional model_cost entries like jp.anthropic.claude-opus-4-7 that omit cache_creation_input_token_cost_above_1hr shadowed the base entry that has it, so is_claude_4_5_on_bedrock returned False and requested cache ttl values were dropped for those deployments. Both capability lookups now consult the full model id and the region-stripped base entry, matching the coverage of the old name-pattern list. Also restores ToolBlock keyword construction for the tool_config cachePoint; PEP 589 TypedDict keyword instantiation works on every supported Python version --------- Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(e2e): add vertex_ai passthrough spend-log coverage Port the de-flake of the SDK-based vertex spend test (BerriAI#31689) into the tests/e2e/llm_translation harness. The vertexai SDK intermittently ignored the proxy api_endpoint override and billed Vertex directly, so the request never reached LiteLLM and no spend was logged; driving native generateContent over the shared transport always reaches the proxy, which the harness already guarantees. The vertex deployment is added at runtime through /model/new with use_in_pass_through rather than declared in the gateway config, and deleted on teardown. That registers the deployment's service account for the /vertex_ai route, so the passthrough call sends only its litellm virtual key in x-litellm-api-key and no upstream bearer, and the proxy mints the Vertex token itself. The credential is the one the proxy already holds, read from the same VERTEXAI_CREDENTIALS/VERTEXAI_PROJECT env; the test never mints a token. Asserts both that the forward succeeds and that a costed SpendLogs row lands (vertex_ai provider, a gemini model, spend > 0, call_type pass_through_endpoint), correlated by the x-litellm-call-id header. * Update tests/e2e/llm_translation/test_vertex_passthrough_e2e.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Update tests/e2e/llm_translation/test_vertex_passthrough_e2e.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
…le-server routes (BerriAI#31921) A 401 while listing tools (a missing or expired per-user OAuth token, or an upstream 401 for any auth_type) was swallowed to an empty tool list, so a single-server client got a 200 with no tools and no WWW-Authenticate challenge instead of a 401 it could re-authenticate against. Only oauth pass-through and delegate-to-upstream oauth2 servers surfaced it; every other auth_type, and the missing-token case for all of them, masked it. The surface-vs-absorb decision now keys on the route, not the auth_type. An upstream 401 in _fetch_tools_with_timeout becomes an MCPUpstreamAuthError regardless of auth_type, and the per-user OAuth challenge raised during client creation (a bare HTTPException 401 carrying a WWW-Authenticate header) is converted to the same type in _get_tools_from_server. The challenge is scoped to 401: a 403 (authenticated but forbidden, e.g. insufficient scope) is not a re-auth signal and degrades to an empty list like any other non-auth error, and the stdio-allowlist 403 (no challenge header) stays absorbed. The existing routing then does the right thing: single-server routes turn the error into a 401 + WWW-Authenticate, while the multi-server aggregator absorbs it to an empty list so one unauthenticated server does not fail the whole listing. On the UI tools page, an OBO (per-user authorization_code) server now shows the Authorize gate when the list call returns 401, not only when no credential row exists. The backend already refreshes a still-refreshable token on the list call, so a 401 means there is no valid token and none could be minted (expired with no usable refresh token), which is exactly when the user must reauthorize.
* feat(tencent): add Tencent TokenHub as a provider Tencent TokenHub is OpenAI- and Anthropic-compatible. This registers it as a new provider: TencentChatConfig routes /v1/chat/completions and gates the thinking/reasoning_effort params behind supports_reasoning, and TencentAnthropicMessagesConfig routes the Anthropic-compatible Messages API. Adds cost tracking, the deepseek-v4-pro/flash model entries, and provider endpoint support metadata. * test(tencent): add unit tests for Tencent TokenHub provider Covers TencentChatConfig (chat completions) and TencentAnthropicMessagesConfig (messages API) across transformation, param mapping, URL building, and header validation, plus get_optional_params routing. Tests mock supports_reasoning to stay independent of remote model cost data. * fix(tencent): correct max_output_tokens and reuse parent messages env validation Raise max_output_tokens/max_tokens for tencent/deepseek-v4-pro and tencent/deepseek-v4-flash from 8192 to 384000, matching Tencent TokenHub's published DeepSeek-V4 output limit; the 8192 value mirrored the native DeepSeek default and would have rejected valid larger requests before they reached Tencent Delegate validate_anthropic_messages_environment to the parent via super() so the Tencent messages endpoint keeps content-type and anthropic-beta header injection instead of dropping them, keeping only the TENCENT_API_KEY resolution overridden Add regression tests covering beta-header injection, the cost-calculator delegation, provider-info secret resolution, and validate_environment key handling * fix(tencent): normalize messages URL when TENCENT_API_BASE has chat completions suffix * fix(tencent): register tencent in models_by_provider The provider was added to the LlmProviders enum and cost map but not to the models_by_provider lookup, so test_models_by_provider (which asserts every litellm_provider present in the cost map is registered) failed once the tencent models were loaded. Add the tencent_models set, populate it from the cost map, and expose it under the tencent key, mirroring deepseek. * fix(tencent): import generic_cost_per_token from its canonical module Import generic_cost_per_token from litellm.litellm_core_utils.llm_cost_calc.utils instead of the top-level litellm.cost_calculator dispatcher, which imports the tencent cost module at load time. Removing the back-reference avoids the circular import and matches how deepseek and the other providers source the helper. --------- Co-authored-by: Felipe Rodrigues Gare Carnielli <felipe.gare@hotmail.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com>
…erriAI#31985) * fix(bedrock): map guardrailConfig to InvokeModel guardrail headers The InvokeModel API takes the guardrail identifier, version and trace as X-Amzn-Bedrock-* request headers, unlike Converse which takes them in the request body. The invoke transformer never set these headers, so guardrailConfig was silently dropped (or leaked into the request body) and Bedrock guardrails never ran on invoke-route models. Pop guardrailConfig in AmazonInvokeConfig.validate_environment, validate it, and set the headers before SigV4 signing; explicitly passed headers keep winning over guardrailConfig so existing workarounds are unaffected * fix(bedrock): reject guardrailConfig missing guardrailIdentifier A guardrailConfig without guardrailIdentifier (e.g. an empty dict) would validate, produce no guardrail headers, and let the request proceed with guardrails silently not applied; that silent skip is the exact failure mode this fix exists to remove, so fail fast with a 400 instead
… cascade fix (BerriAI#31995) * feat(ui): shadcn migration foundation: Tailwind v4, shadcn init, antd cascade fix Upgrade the dashboard from Tailwind v3 to v4 with CSS-first config: the official upgrade codemod renamed utilities across 151 files, and tailwind.config.js (plus the dead tailwind.config.ts) is replaced by @theme tokens, @source globs, and @plugin directives in globals.css. The Tremor safelist becomes @source inline patterns and the legacy tremor theme tokens carry over verbatim. ui_colors.json was build-time only and fed the dying Tremor palette, so its brand values are inlined and the file removed; runtime theming replaces that path next. shadcn is initialized with a hand-authored components.json (rsc, cssVariables, baseColor gray) pointing utils at the existing lib/cva.config.ts, which now exports cn (cva beta cx + twMerge) instead of adding class-variance-authority as a second variant library. The two ad-hoc cn helpers fold into it. Button lands as the canary primitive, adapted to cva beta and React 18 forwardRef, with tests covering the variant, twMerge, asChild, and ref seams. --radius is 0.5rem so the shadcn radius scale reproduces Tailwind defaults and legacy rounded-* classes render unchanged. antd v5 emits unlayered CSS-in-JS that would beat every layered v4 utility, so AntdGlobalProvider now wraps the app in StyleProvider layer and ConfigProvider cssVar, and globals.css declares @layer theme, base, antd, components, utilities. antd wins over preflight but yields to utilities, which is what lets migrated shadcn pages coexist with legacy antd pages. Preflight stays global with the three v3 behaviors pinned (default border color, button cursor, placeholder color). * fix(ui): restore tremor opacity tints removed by tailwind v4 Tailwind v4 removed the *-opacity-* utilities, but the precompiled @tremor/react dist still composes them with shade-500 palette classes (bg-opacity-10 over bg-<color>-500 etc.), so Badge, BadgeDelta, Callout, light Icon and Button, BarList, and ProgressBar lost their tints and rendered solid 500-shade fills. Adversarial review caught it; the original smoke pages only exercised antd Tags. tremor-v3-compat.css restores exactly the pairs tremor emits: for each of the 22 safelisted colors, bg-opacity-{10,20,40}, hover/group-hover bg-opacity-{20,30}, and ring-opacity-{20,40} against the -500 shade, via color-mix into the utilities layer. Tremor's colorPalette maps both background and iconRing to 500, so the -500 pairing covers every composition in the dist; dark: variants are inert until dark mode ships. The shim dies with @tremor/react at the end of the migration. The upgrade codemod also missed two hand-rolled modal scrims using bg-black bg-opacity-{30,50} (solid black under v4); now bg-black/30 and bg-black/50. Removed the docker/build_admin_ui.sh copy of enterprise_colors.json into the deleted ui_colors.json; that build-time rebrand path is retired and its runtime replacement lands with the theming phase. * fix(ui): pair ring-opacity-40 with shade 300 in tremor compat shim Tremor's colorPalette maps ring to shade 300, and the only consumer of ring-opacity-40 (Icon variant outlined) composes it with that shade, so the shade-500 rows were dead and outlined icon rings would render at full opacity. Latent today (no dashboard usage of the outlined variant); caught by adversarial review. ring-opacity-20 stays at 500 (iconRing), matching Badge and BadgeDelta.
…rriAI#31914) * chore(e2e): untrack gateway config and document e2e test location Stop tracking tests/e2e/gateway/litellm-config.yml so the local proxy config stays on the machine Add a note to CLAUDE.md that new e2e tests belong in tests/e2e/ and must follow that directory's conventions * chore(e2e): add self-contained docker compose stack for local runs Ship a docker-compose.yml that starts the proxy with a throwaway Postgres and Redis and inlines the proxy config with example models, so contributors can bring up a local gateway with nothing but a .env. Update CONTRIBUTING.md to match the inline-config flow * chore(e2e): drop the second gemini deployment; one key is enough locally * docs(e2e): make pre-commit steps ordered and require flagging internally found issues
…kip redundant prisma generate (BerriAI#32000) * perf(lint): skip and cache base gate passes, parallelize make lint, skip redundant prisma generate make pre-commit paid for a full second basedpyright pass over a merge-base worktree on every run even when no rule was over its ceiling, re-generated an unchanged Prisma client, and ran seven independent checks sequentially. The basedpyright and ruff strict gates now skip the base pass when head is within every limit (the same early-out type_discipline_gate already had), the basedpyright base counts are cached under the git common dir keyed by merge-base commit, pyrightconfig.json, and uv.lock, prisma generate only runs when the schema or prisma version changed, and make lint fans its checks out through a parallel sub-make after a single setup phase * fix(lint): keep the base-cache scratch file out of the prune glob The tmp+rename scratch in store_counts was named basedpyright-base-<hash>.json.tmp, which the stale-entry prune glob (basedpyright-base-*) also matches, so a concurrent lint run from another worktree sharing the same git common dir could unlink it between write_text and replace and crash the gate with FileNotFoundError. The scratch is now dot-prefixed so the glob can never see it, pid-suffixed so concurrent writers of the same entry never share a scratch, and the prune glob is restricted to committed *.json entries
… path (BerriAI#31979) * fix(a2a): record agent cost_per_query and input tokens on native send path * test(a2a): add __init__.py to avoid test_utils.py module collision
* fix(prometheus): bound per-request budget metric emission with a timeout (BerriAI#31632) * fix(prometheus): bound per-request budget metric emission with a timeout Wrap the per-request budget-metric gather in asyncio.wait_for so a slow Redis or DB lookup cannot consume the whole LoggingWorker watchdog and get the success-logging event cancelled. On timeout the emission is skipped in isolation; budget gauges are still refreshed by the periodic cron. The timeout is configurable via PROMETHEUS_BUDGET_METRICS_PER_REQUEST_TIMEOUT and defaults to 5.0 seconds, falling back to the default on an invalid value instead of raising * fix(prometheus): reject non-finite and non-positive budget-metrics timeout env float() accepts 0, negatives, nan and inf, which bypass the fallback: a value <= 0 makes asyncio.wait_for time out immediately and skip every per-request emission, and inf reintroduces the unbounded wait the timeout was meant to bound. Validate the parsed value is finite and greater than zero before using it, otherwise fall back to the default * fix: report the blocked LLM response's real token usage (BerriAI#31217) When a guardrail blocks a post-call response, the synthetic violation response reported hard-coded zero usage, discarding the token usage the upstream call had already consumed. Fix the root cause rather than re-counting tokens: - Add an optional `original_response` field to ModifyResponseException. - The unified guardrail's post-call success hook attaches the blocked LLM response to the exception. - The /v1/messages and OpenAI-format (/v1/chat/completions, /v1/completions) block handlers report `original_response.usage` directly. Pre-call blocks never invoked the LLM, so usage is zero. Mock-based tests cover the helper (returns original usage / zero), the success hook attaching original_response, and the endpoint reporting it end-to-end. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(guardrails): buffer + cleanly terminate streamed responses on block (BerriAI#31389) Streaming moderation improvements for the unified guardrail post-call streaming iterator hook: - streaming_buffer_until_moderated: withhold all chunks until end-of-stream moderation passes, then release the original response (clean) or only the block message (blocked) -- the original content is never delivered on a block. Snapshot chunks with a shallow list() copy (end-of-stream builds a separate assembled response; chunks aren't mutated in place). - Clean Anthropic SSE on block: synthesize a well-formed termination sequence instead of a bare data: {"error": ...} blob that truncates the stream. Provider-specific synthesis lives in AnthropicMessagesHandler via build_block_sse_chunks (format-agnostic routing stays in the hook). - Mid-stream blocks continue the in-progress message (close open content block, append block message, terminate) rather than emitting a second message_start, which clients reject. Standalone envelope only when no chunks were sent (buffered path). - ModifyResponseException imported under TYPE_CHECKING + locally at runtime to avoid a module-level cyclic import. Adds regression tests for buffering (content withheld on block) and mid-stream continuation (single message_start). Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: report real usage on streaming blocks, disable buffered mode for content-rewriting guardrails - _standalone_block_chunks and _block_continuation_chunks now read real token usage from ModifyResponseException.original_response instead of hardcoding zero, matching the non-streaming _blocked_response_usage path. Shared helper moved to guardrail_translation/utils.py. - streaming_buffer_until_moderated is now forced off when the guardrail has mask_response_content=True, since buffered replay releases the withheld original chunks verbatim -- unsafe for a guardrail that rewrites content (e.g. PII masking). - Fix inverted streaming-flag precedence comment. * style: ruff format after greploop fixes * fix: handle Anthropic streaming guardrail blocks * fix(responses): check terminal event type for streaming guardrail end-of-stream detection _check_streaming_has_ended assumed responses_so_far held ModelResponse objects with .choices, but for the Responses API the accumulated chunks are raw SSE event dicts, causing an AttributeError on every call * fix: preserve Anthropic blocked stream usage --------- Co-authored-by: FERNANDO IZAR <fizar@me.com> Co-authored-by: Joseph Barker <156112794+seph-barker@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com>
…act 18 Input didn't wrap its function component in React.forwardRef, so the ref ConversationList passes for rename auto-focus/select silently never attached under React 18 (function components need forwardRef to receive a ref; that requirement is dropped in React 19, but this app is on 18.3.1).
…al_user_access fix(proxy): route realtime HTTP endpoints through router for credenti…
feat(ui): migrate chat UI from antd to shadcn/ui + add key management and usage panels
…3b07c fix(ci): exclude deleted files from ruff format check
Generic pass-through endpoints called raise_for_status() on upstream 4xx/5xx responses and re-raised as HTTPException, which the outer handler reshaped into a ProxyException with the upstream body stringified into error.message. Success responses were already forwarded as-is, so failures were the only case where passthrough wasn't actually transparent. Removes the raise_for_status() calls for both streaming and non-streaming passthrough so upstream status, body, and headers reach the client unchanged, while keeping guardrails/managed-id rewriting scoped to successful responses and leaving internal proxy failures (auth, config, network errors before any upstream response) on the existing ProxyException path. Co-authored-by: Cursor <cursoragent@cursor.com>
…_redaction fix(proxy): stop leaking master_key and database_url in startup DEBUG logs
…h upstream errors Follow-up to 8c98780: returning upstream 4xx/5xx bodies unchanged also skipped post_call_failure_hook entirely, so spend-tracking and alerting callbacks never fired for upstream errors, and response_body was hardcoded to None in the log payload so the actual upstream error body never reached logging integrations. Adds a small helper that calls post_call_failure_hook for upstream errors without altering the client-facing response, and parses response_body unconditionally for logging while still scoping guardrails and managed-id rewriting to status_code < 400. Co-authored-by: Cursor <cursoragent@cursor.com>
…tachment create (BerriAI#32131) * fix(policies): reject non-existent team/key/model scope entries on attachment create Creating a policy attachment accepted arbitrary team, key, and model values with no validation, so a typo'd or non-existent team was silently persisted (LIT-4199). The create endpoint now rejects a concrete (non-wildcard) team, key, or model that does not resolve to a real entity, wiring the previously-dead PolicyValidator existence checks and reusing RouteChecks._is_wildcard_pattern so validation agrees with request-time matching, where only a trailing "*" is a wildcard. Wildcard patterns are still allowed through since they may match zero entities today and more later, and tags stay free-form. The Admin UI's Teams field validates the same rule for immediate feedback when its team list has loaded, deferring to the backend otherwise. * style(policies): use builtin list generics and | None in scope validator Keeps the new find_invalid_scope_entries signature off the UP006/UP045 strict ruff budgets instead of copying the surrounding legacy typing.List/Optional idiom. * fix(policies): separate multiple attachment scope errors with ' | ' Addresses Greptile review: joining per-entry validation messages with a bare space read as one run-on sentence; ' | ' makes the multi-error 400 detail easier to parse for users and programmatically.
…advisor tool (BerriAI#32093) * fix(anthropic): require caller api_key and SSRF-validate api_base in advisor tool The advisor_20260301 interceptor honored a caller-supplied api_base once allow_client_side_credentials was enabled, even without a caller-supplied api_key. AnthropicModelInfo.get_auth_header() then fell back to the proxy's own ANTHROPIC_API_KEY/ANTHROPIC_AUTH_TOKEN, so the server's real credentials plus the conversation history got sent to a caller-chosen destination _resolve_advisor_credentials() now only honors api_base alongside a non-empty caller-supplied api_key, requires the https scheme, and validates api_base via validate_url() before use, mirroring check_complete_credentials in auth_utils.py. https is required because validate_url only DNS-pins the connection for http; for https with TLS verification on it returns the URL unchanged and relies on certificate validation to block DNS rebinding * fix(anthropic): also reject advisor api_base when ssl_verify is disabled validate_url only DNS-pins the connection for http, or for https with litellm.ssl_verify disabled; the previous https-only check missed the ssl_verify=False case, where validate_url's rewritten URL was still being discarded, per Greptile's review of this PR. Reject api_base outright when ssl_verify is False so the discarded rewrite can no longer matter
…itellm_realtime_cost_metrics
Pass transcription_cost through additional_costs so cost_breakdown's input_cost + output_cost + additional_costs sums to total_cost instead of silently folding it into total_cost only. Co-authored-by: Cursor <cursoragent@cursor.com>
…tream errors Two bugs from the upstream-error fixes: the success handler has no status-code awareness, so removing raise_for_status() left it firing for every upstream 4xx/5xx too, meaning the new failure hook and the existing success handler both logged the same request (corrupting SpendLogs/cost tracking). Separately, the failure hook was passed the raw httpx.HTTPStatusError, which ProxyLogging's alerting only excludes HTTPException/ProxyException from, so a normal upstream 403 would trigger a "High" severity llm_exceptions alert. Gates the success handler (both non-streaming and end-of-stream) to status_code < 400, and reports upstream failures to post_call_failure_hook as an HTTPException instead of the raw httpx error, matching how auth/rate-limit errors are already excluded from alerting. Co-authored-by: Cursor <cursoragent@cursor.com>
… DB is down at startup (BerriAI#31951) * fix(proxy): keep serving reads from the read replica when the primary DB is down at startup RoutingPrismaWrapper.connect() connected the writer first and let a writer failure propagate, so a proxy that started during a primary outage ended up with no Prisma client at all (startup swallows the error under allow_requests_on_db_unavailable): DB-stored models never loaded and every inference request failed with 400 Invalid model name, even with a healthy DATABASE_URL_READ_REPLICA. Workers recycled via MAX_REQUESTS_BEFORE_RESTART hit this mid-outage and stayed broken for the rest of the outage. connect() now degrades on a writer-only failure: reads (key auth, DB-stored model loads) are served by the reader, writes fail at call time, and the DB health watchdog keeps retrying the writer reconnect, which clears the degraded flag once the primary recovers. A full outage (both sides down) still raises as before. Resolves LIT-4159 * fix(proxy): clear degraded-writer flag when the reconnect probe finds the writer already healthy The direct-reconnect path returns early when the writer probe succeeds (engine already reconnected by another path, e.g. an IAM token refresh), skipping recreate_prisma_client, which was the only runtime path clearing _writer_unavailable. The stale flag made the watchdog fire reconnect attempts against a healthy writer on every cooldown cycle until restart. Clear the flag in the early-return branch and cover it with a regression test that fails without the change
…292bcc build: restore maturin backend to bundle the Rust bridge in the wheel
chunk_processor now reads response.status_code to gate end-of-stream success logging. These mocks used AsyncMock(spec=httpx.Response), which spec's against the class and doesn't expose status_code since it's an instance attribute, not a class attribute, so accessing it raised AttributeError. Sets status_code=200 explicitly on the success-path mocks. Co-authored-by: Cursor <cursoragent@cursor.com>
…etrics fix(cost): store cost breakdown for /v1/realtime sessions
…or_normalisation fix(proxy): return upstream error bodies unchanged in passthrough
The CircleCI proxy containers passed --detailed_debug, and litellm's log level defaults to DEBUG when LITELLM_LOG is unset, so CI produced very verbose debug output for no reason. Drop --detailed_debug and set LITELLM_LOG=ERROR on the proxy containers so real failures still surface without the debug noise
…6c3e bump: litellm-enterprise 0.1.46 -> 0.1.47
chore(ci): build ui for release
…nd cache metrics (BerriAI#32126) * feat(prometheus): add api_provider label to token, latency, request and cache metrics The token (input/output/total), latency (llm_api, time_to_first_token, request_total, request_queue_time), proxy request (total/failed) and cache metrics were emitted from the same call sites as litellm_spend_metric and litellm_requests_metric, which already carry api_provider, yet these were missing it. That left no way to break tokens, latency, request counts or cache hits down by upstream provider even though the provider is already on the payload as custom_llm_provider. Add api_provider to each metric's label allow-list. The success path already populates enum_values.api_provider from standard_logging_payload, so those metrics emit it with no further plumbing. The cache label is added to the shared _cache_metric_labels list, so alongside litellm_cache_hits_metric and litellm_cache_misses_metric it also covers litellm_cached_tokens_metric and the provider prompt-cache read/creation token metrics; the label-presence test asserts all of them. For the client-side failure path, where a deployment may not have been resolved, derive it best-effort from litellm_params.custom_llm_provider, a partial standard_logging_object, or inference from the requested model name via litellm.get_llm_provider, falling back to empty rather than guessing. Resolves LIT-4178 * fix(prometheus): satisfy ruff BLE001 budget and update enterprise label assertions - Suppress the strict-rule BLE001 budget breach with a justified noqa; the broad except in the failure-path provider extraction is intentional defense-in-depth (covered by test_extract_api_provider_swallows_unknown_model_but_logs_unexpected_errors), not dead code to delete - Update tests/enterprise assertions for litellm_tokens_metric, litellm_input_tokens_metric, litellm_output_tokens_metric, the three latency metrics, and the proxy request counters to expect the new api_provider label, matching what litellm_mapped_enterprise_tests caught in CI --------- Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
chore(ci): promote internal staging to main
vertex_ai/claude-haiku-4-5 and @20251001: 8192 -> 64000 (stale Claude 3.5 Haiku cap). openrouter/anthropic/claude-haiku-4.5: 200000 -> 64000 (context window pasted into output fields). Matches the already-correct anthropic/bedrock/azure_ai entries. Fixes BerriAI#32184
|
|
Contributor
Greptile SummaryThis PR corrects stale
Confidence Score: 5/5Catalog-only data correction; no code paths are touched and the new values are confirmed by official Vertex AI docs and the OpenRouter API. All six changed lines update numeric values that were demonstrably wrong (an old output cap and a copy of the context-window size). The replacement value of 64 000 is consistent with every other Claude Haiku 4.5 entry already in the file, is backed by the Vertex AI documentation, and matches the OpenRouter API response. Both JSON files are kept in sync with identical edits, and no logic, tests, or interfaces are affected. No files require special attention.
|
| Filename | Overview |
|---|---|
| model_prices_and_context_window.json | Corrects max_output_tokens and max_tokens for openrouter/anthropic/claude-haiku-4.5 (200000→64000) and both vertex_ai/claude-haiku-4-5 entries (8192→64000), consistent with all other Claude Haiku 4.5 entries in the file |
| litellm/model_prices_and_context_window_backup.json | Mirror of the main JSON fix — same three entries corrected identically; both files kept in sync |
Reviews (1): Last reviewed commit: "fix: correct max_output_tokens for Claud..." | Re-trigger Greptile
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
15 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relevant issues
Fixes #32184
Type
🐛 Bug Fix (model catalog data only — no code changes)
Changes
Corrects
max_tokens/max_output_tokensfor three Claude Haiku 4.5 entries, in bothmodel_prices_and_context_window.jsonandlitellm/model_prices_and_context_window_backup.json(6 lines each, nothing else touched):
vertex_ai/claude-haiku-4-5vertex_ai/claude-haiku-4-5@20251001openrouter/anthropic/claude-haiku-4.5The 8192 appears carried over from Claude 3.5 Haiku's old output cap; the 200000 is the
context window pasted into the output fields.
max_input_tokens(200000) is left as is.Evidence
https://docs.cloud.google.com/vertex-ai/generative-ai/docs/partner-models/claude/haiku-4-5
GET https://openrouter.ai/api/v1/models, no auth):anthropic/claude-haiku-4.5→top_provider.max_completion_tokens: 64000,context_length: 200000claude-haiku-4-5(anthropic),azure_ai/claude-haiku-4-5,and all
*.anthropic.claude-haiku-4-5-*bedrock entries already say 64000.Testing
Catalog-only change. Verified locally:
json.load)litellm.get_model_info("vertex_ai/claude-haiku-4-5")["max_output_tokens"]→ 64000litellm.get_model_info("openrouter/anthropic/claude-haiku-4.5")["max_output_tokens"]→ 64000