🔄 Upstream Sync: LiteLLM v1.92.0 - #121
Merged
Merged
Conversation
chore(ci): promote internal staging to main
…lose guardrail bypass (BerriAI#31519) * fix(realtime): stop sending a second Gemini Live setup on follow-up session.update Gemini Live (BidiGenerateContent) accepts setup as the first-and-only client message; a second setup closes the socket with 1007 Request contains an invalid argument. The AI Studio Gemini path forwarded every client session.update after the first as a follow-up setup, and GA clients (pipecat) send several while configuring the session, so the second one tore the session down before the first turn. Callers saw silence after the first response, exponential per-turn latency from reconnect/retry churn, and intermittent 1011 errors. Drop subsequent session.updates instead of resending setup, matching what the Vertex subclass already does. Tools and instructions must ride on the first session.update before any conversation content. Adds regression tests covering the plain follow-up, a follow-up that adds tools (the case the previous identical-only dedup still forwarded), and the guardrail create_response=False warning path. * fix(realtime): retry the backend open handshake instead of failing with 1011 The upstream Live API open handshake (e.g. Gemini Live) intermittently hangs; waiting longer never recovers a hung attempt, but a fresh attempt almost always connects in ~1s. The proxy opened the backend websocket once with the default open_timeout and no retry, so a single slow handshake surfaced to the caller as a fatal 1011 internal error and dropped the call. Bound each open attempt with a short open_timeout and retry; a bounded attempt that already timed out spaces out the next try, so no backoff is needed. Deterministic handshake-status rejections (auth/4xx) are not retried, and the retry only ever wraps the open, never a live session. Adds tests for retry-then-succeed, raise-after-max-attempts, and no-retry-on-auth-failure. * fix(realtime): close guardrail bypass + surface handshake status; drop obsolete tests Three review fixes on the Gemini Live realtime path. Transcription-guardrail bypass: Gemini Live rejects a second setup (1007), so once the initial setup is sent the guardrail's automaticActivityDetection.disabled=true can no longer be delivered as a follow-up session.update. With that follow-up now dropped, the model's auto-response stayed enabled and a realtime_input_transcription guardrail was bypassed (the model answered before the proxy could gate the turn). Fold the disable into the one-and-only setup instead: the handler injects it into the auto-sent setup (gemini_live_defer_setup false) and _send_to_backend injects it into the deferred first setup. OpenAI sessions accept follow-up updates and are left untouched. Backend handshake status: the open-retry treated only InvalidStatusCode as deterministic; websockets>=15 raises InvalidStatus for a rejected client handshake, so a 401/403 fell into the broad WebSocketException branch and was retried before the caller closed the client with 1011 instead of the upstream status. Treat both as non-retryable. Obsolete tests: the four tests asserting a follow-up session.update is merged and re-sent as a second setup asserted behavior that crashes Gemini Live with 1007 (verified directly against the API). Removed; the drop is covered by new regression tests. * style(realtime): reformat changed files to ruff line-length 120 Post-merge with litellm_internal_staging, which unified ruff format width to 120 (BerriAI#31518). The realtime change set was formatted at 88, so the changed lines tripped the whole-file ruff format check. Reformat with ruff 0.15.3 at the repo's 120 width; no logic changes.
…I#29718) * feat: declarative fallback generalizations for unknown models Unknown or newly-released models previously degraded (missed cost lookups, wrong supports_* flags, broken provider routing) and were patched with one-off hardcoded regexes scattered across Python. This adds a single data-driven source of truth: a fallback_generalizations block in model_prices_and_context_window.json holding ordered, case-insensitive regex rules that map a model name to the metadata to apply when it has no exact entry. A new fallback_generalizations module owns the rules and a compiled-regex cache that is built once and invalidated on reload, so the O(n) scan runs only on a cache miss. get_llm_provider now routes an otherwise-unknown model via the first matching rule's litellm_provider, replacing the hardcoded _CLAUDE_PATTERN and _matches_claude_model_pattern. _get_model_info_helper falls back to a matching rule's model_info after the exact lookups miss, so get_model_info and the supports_* helpers resolve unknown models from the same rule. get_model_cost_map extracts the block out of the returned map, and the integrity check now counts real model entries (excluding reserved meta keys) so the new key cannot mask a genuinely shrunk upstream file. The top level of the file stays a flat map of models so existing litellm releases that fetch the live file keep working and keep receiving updates; the block ships in both the root file and the bundled backup. An anthropic-claude rule reproduces the old future-claude routing and additionally supplies capability flags and a context window https://claude.ai/code/session_01G8Jro8dPLktwnaaSJwVDpo * refactor(anthropic): derive adaptive-thinking from a version threshold; harden generalizations Replace the per-minor-version _is_claude_4_6_model / _is_claude_4_7_model substring matchers with a single _claude_version_at_least predicate that parses the Claude family version from the model name and compares against 4.6. This covers 4.8/4.9/5.x without a code change (the old matchers missed 4.8 entirely) while keeping an explicit supports_adaptive_thinking flag authoritative when present, so there is one source of truth. The two direct call sites in the chat transformation now route through _is_adaptive_thinking_model instead of the deleted matchers. Also address review feedback on the generalizations module: return a copy of the matched model_info so a future caller cannot mutate the compiled-rule cache, document that patterns are matched with re.search and must anchor with ^ and $, and reindent the fallback_generalizations block to the file's 2-space style in both JSON files. https://claude.ai/code/session_01G8Jro8dPLktwnaaSJwVDpo * fix(anthropic): surface adaptive-thinking from the cost map; fix date misparse supports_adaptive_thinking shipped in the model cost map but was never declared on ModelInfo nor copied during construction, so get_model_info (and the supports_* factory) silently dropped it for every provider-prefixed or generalized name; only a bare base entry resolved. Wire it through ModelInfo like the other capability flags and backfill the flag onto the genuine Claude 4.6/4.7/4.8 entries across providers so the data, not code, declares the capability. The anthropic-claude fallback rule also carries the flag (and now accepts a dotted minor, e.g. 4.6) so an unmapped future Claude degrades to adaptive thinking without a code change. Tighten the Claude version parser so an eight-digit date suffix (claude-opus-4-20250514, the non-adaptive Opus 4.0) is no longer read as minor 4.20250514. The cost map stays authoritative; the version check is only a fallback for provider-prefixed names (bedrock/invoke routes, -v1-less ids) that resolve to no mapped entry and so cannot be reached by an exact lookup or the bare-name rule. https://claude.ai/code/session_01G8Jro8dPLktwnaaSJwVDpo * fix(anthropic): date-safe adaptive-thinking version fallback, conservative fallback pricing, ruff strict gate Reconcile adaptive-thinking detection after merging litellm_internal_staging. Keep the cost-map resolver (_supports_model_capability) as the source of truth and add a date-safe opus/sonnet/haiku >= 4.6 name version as a fallback for provider-prefixed ids the cost map cannot resolve (e.g. bedrock/invoke/us.anthropic.claude-opus-4-6). A two-digit cap on the minor keeps an eight-digit date suffix from being misread as a minor version, so the dated Claude 4.0 release stays non-adaptive Price the shipped anthropic-claude fallback rule at the Opus tier so an unknown or newly released Claude is over-costed rather than billed as free Drop the module-level global state in fallback_generalizations (PLW0603) in favor of a small registry object, and switch its annotations plus the new utils helper to builtin generics (UP006), bringing the ruff strict-rule totals back under ceiling * refactor(anthropic): drive adaptive-thinking version gate from a declarative rule Replace the bespoke _claude_version_at_least heuristic with a version-gated fallback_generalizations rule. Unmapped Claude ids now resolve adaptive thinking purely from the cost map: an explicit entry, or the new self-contained anthropic-claude-adaptive-thinking rule that matches opus/sonnet/haiku >= 4.6 (covering 5.x, 6.x and beyond with no code change). New families ship via Price Data Reload instead of a code edit The rule carries the same Opus-tier pricing as the broad anthropic-claude rule plus supports_adaptive_thinking, and is matched first; the broad rule stays version-neutral, so an unmapped >= 4.6 Claude resolves to full pricing and the adaptive flag from one rule, while a sub-4.6 alias such as claude-opus-4-0 is still priced yet stays non-adaptive. The regex caps the minor at two digits so a dated 4.0 id (...-4-20250514) is never read as a >= 4.6 minor * refactor(anthropic): dedupe adaptive-thinking rule via declarative extends The version-gated anthropic-claude-adaptive-thinking rule duplicated the broad anthropic-claude rule's entire Opus-tier price block because rules do not merge: first match wins and returns one rule's whole model_info, so the adaptive rule had to be self-contained. Add a declarative extends field to fallback_generalizations: a rule names a parent and inherits its model_info, with its own keys overriding. Inheritance is resolved once at install time against each rule's raw model_info, so the adaptive rule now carries only its delta (supports_adaptive_thinking) and inherits pricing from the broad rule. Runtime matching, provider routing and gating are unchanged; the broad rule stays anchored and first-match-wins still holds. * docs(anthropic): add ignored description key documenting each generalization regex * fix(anthropic): drop fabricated pricing from the anthropic-claude fallback rule Per review feedback, the base rule no longer carries input/output/cache costs, and the adaptive-thinking rule that extends it inherits that no-pricing model_info. Pricing an unmapped model at a guessed tier reports a confidently-wrong cost without the caller knowing; dropping it keeps the standard unpriced behavior (zero, not a fabricated number) so a missing price stays visible. The rules still supply provider routing, context window, and capability flags, so a brand-new Claude can still be called and its capabilities (including adaptive thinking for >= 4.6) resolved. Description and tests updated to match
The error string is already produced by the f-string interpolation; the trailing .format() call on it was redundant. Add a regression test that the message renders the model name verbatim.
get_model_list always returns a list, never None, so the is-None branch could not execute. Collapse to the single reachable message.
…erriAI#30631) Keep only video test files and CI workflow entries; drop unrelated production code and non-video test changes from this branch. Co-authored-by: Cursor <cursoragent@cursor.com>
…BerriAI#30529) * test(batches): add 1:1 test file scaffold for batches component paths Co-authored-by: Cursor <cursoragent@cursor.com> * Add harness test for create batch endpoint * Add retrieve endpoint harness tests * Add list endpoint harness tests * Add cancel endpoint harness tests * Add cancel endpoint harness tests * Add test for litellm/batches/main.py * Add test for litellm/tests/test_litellm/batches/test_batch_utils.py * Add handler and transformation tests for all providers * Fix: run batches tests in cicd * fix(tests): remove azure/__init__.py that shadowed azure namespace package Adding __init__.py to tests/test_litellm/llms/azure/ caused pytest to insert tests/test_litellm/llms/ into sys.path[0], making our empty azure/ dir shadow the real azure-identity namespace package. Any test that patched azure.identity.* would then fail with AttributeError. * style(tests): apply ruff format to test_batch_utils.py Base migrated the formatter from black to ruff format (BerriAI#31317); reformat the batches scaffold test file to match. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
…erriAI#30950) * feat(a2a): support a2a-sdk 1.x proxy routing for 0.3 and 1.0 agents Bump a2a-sdk to 1.x and wire send/stream through compat conversions so the proxy accepts A2A 1.0 JSON-RPC while preserving 0.3 wire clients. Co-authored-by: Cursor <cursoragent@cursor.com> * Add user controlled protocol version in agents * Fix exeception mapping * Fix a2a base url * Add e2e test for a2a * Fix lint * Fix lint * fix(a2a): harden card version detection and header isolation coverage Use protocolVersion when inferring agent card wire format, assert distinct httpx cache keys in the header-isolation test, and suppress targeted basedpyright errors for optional SDK imports. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(a2a): suppress reportArgumentType for SDK compat types and fix streaming trace ID - Add pyright: ignore[reportArgumentType] to SendMessageSuccessResponse id= and result= args in _send_message, and SendStreamingMessageResponse root= in _stream_messages, where a2a-sdk compat types diverge from basedpyright's inferred signature, reducing the reportArgumentType count back within budget. - Fix streaming trace ID in astream_a2a_message to use str(request.id) when available instead of always generating a new uuid4(), restoring JSON-RPC request-ID correlation for observability. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style(a2a): expand SendStreamingMessageResponse for black formatting Move pyright: ignore comment to the root= argument line so Black accepts the expanded multi-line form. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(a2a): fix 2 reportArgumentType errors without suppression - main.py: narrow logging_obj from object|None to Optional[Logging] via isinstance check before A2AStreamingIterator call, fixing the "Logging | object" argument type mismatch at line 699. - a2a_endpoints.py: extract response_dict with explicit isinstance(dict) guard before passing to normalize_jsonrpc_response, fixing the "LLMResponseTypes | dict[str, Any]" type mismatch at line 835. - Remove spurious pyright: ignore comments added in previous commits that were not suppressing the actual errors. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(a2a): rewrite upstream URL for 1.0 agent cards in getAuthenticatedExtendedCard 1.0 upstream agent cards store the endpoint URL in supportedInterfaces[0].url rather than a top-level url field. The previous guard only rewrote url when it existed at the top level, so after normalize_agent_card lowered a 1.0 card to 0.3 the upstream internal address leaked into the url field of the 0.3 response. Fix: rewrite both url and supportedInterfaces[0].url to the proxy address before calling normalize_agent_card, ensuring the upstream address is never visible to downstream clients regardless of the upstream card's wire format. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: extend _served_version to all PascalCase methods; add direct httpx-client isolation proof - _served_version now checks `_PASCAL_TO_WIRE` membership instead of two hardcoded names, so GetTask/CancelTask/etc. are promoted to 1.0 wire format alongside SendMessage — prevents mixed wire formats mid-session - test_create_a2a_client_uses_fresh_httpx_client now asserts a2a_client_a._litellm_httpx_client is not a2a_client_b._litellm_httpx_client (direct proof that header bleed cannot occur), in addition to the cache-key inequality check Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: id:0 silently dropped in version_convert; explicit continue in stream retry - version_convert.py: replace `request_id or ""` with `str(request_id) if request_id is not None else ""` in both _send_result_to and _stream_result_to; id=0 is valid JSON-RPC and must not be coerced to "" which breaks response correlation - main.py: add explicit `continue` after the A2ALocalhostURLError retry in _execute_a2a_stream_with_retry so the control flow (retry → next iteration → stream_succeeded guard) is unambiguous Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: preserve a2a retry and discovery card urls * Fix black * Fix test * fix(a2a): avoid KeyError in discovery log after 0.3→1.0 card normalization When a 0.3-style agent card is normalized to 1.0, the top-level url key is replaced by supportedInterfaces; log the already-computed proxy_url instead. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(a2a): preserve taskId when lowering push notification config set params Flatten 1.x create envelope fields before parsing into TaskPushNotificationConfig so 1.0 clients forwarding to 0.3 upstream keep taskId and config. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(a2a): ignore unknown fields in message/send proto fallback ParseDict in _build_message_send_params now matches other inbound paths so 1.0 clients with extra proto fields are not rejected with -32602. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(a2a): normalize tasks/list params and response across protocol versions Convert list task entries on the response path and lower ListTasksRequest params including status filters when forwarding 1.0 clients to 0.3 upstream. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(a2a): avoid reportArgumentType in _lower_list_tasks_params; use local var instead of _parse return Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(a2a): drop private SDK symbol in tasks/list status lowering _lower_list_tasks_params imported _CORE_TO_COMPAT_TASK_STATE, a private a2a-sdk symbol that could disappear on a patch release and silently break status-filter lowering. Derive the 0.3 wire string from the public protobuf enum name instead (TASK_STATE_<NAME> maps to the 0.3 value once the prefix is dropped and underscores become dashes) and validate the result against the 0.3 TaskState enum's own values via a fully-typed pure helper. Behavior is unchanged for every state; unspecified or unrecognized states still drop the filter. Adds parametrized regression tests covering dashed wire values (input-required, auth-required) and the unspecified drop. * fix(a2a): drop redundant push-notification envelope key; unify MessageToDict import _flatten_create_push_notification_params used `config or pushNotificationConfig`, which short-circuits so a co-present pushNotificationConfig key was never popped and leaked into the flattened params. Pop both keys unconditionally and prefer config when present. Adds a regression test on the helper that fails on the old leak. Also import MessageToDict from a2a.compat.v0_3.conversions in _lower_list_tasks_params to match every other conversion helper in the module instead of pulling it straight from google.protobuf.json_format. * fix(a2a): reject invalid message/stream params early with -32602 _handle_stream_message built MessageSendParams lazily inside the stream_response() generator, so malformed 1.0 params surfaced as a generic -32603 after the 200 status line was already committed. The non-streaming path validates up front and returns -32602 (Invalid params). Validate eagerly before returning the StreamingResponse and emit -32602 on failure so both paths reject malformed params identically. Adds a regression test asserting the streamed error code is -32602. * fix(a2a): raise clear error when non-streaming send ends on an update event _send_message fed the SDK iterator's last event straight into SendMessageSuccessResponse, whose result only accepts Message or Task. A non-standard upstream whose final event is a TaskStatusUpdateEvent or TaskArtifactUpdateEvent made the response construction raise an opaque pydantic ValidationError. Guard the converted result and raise a clear RuntimeError instead, consistent with the no-response guard above it. Adds regression tests for the Message happy path and the update-event rejection via an injected fake client. * test(a2a): lock in clean merged agent-card URL without PROXY_BASE_URL Regression coverage proving _build_merged_agent_card produces no double slash in supportedInterfaces[0].url when PROXY_BASE_URL is unset and request.base_url carries a trailing slash. get_custom_url routes through join_paths, which rstrips the base, so the f-string join stays clean. * style(a2a): modernize type annotations to satisfy strict ruff budget After merging the black->ruff-format migration from base, the A2A files owned by this PR still used Optional[X]/quoted annotations that pushed UP037/UP045 over their lowered ceilings. Convert to X | None, drop the now-unnecessary quoted local annotation in _send_message, and remove the imports left unused by the rewrite. Type semantics are unchanged. * style(a2a): type a2a_endpoints dict params as dict[str, Any] The merge with the formatter-migration baseline tightened the reportUnknownArgumentType ceiling; bare dict annotations made every value Unknown and pushed the codebase total over cap. Annotate the JSON-RPC params, body, metadata, and litellm_params dicts as dict[str, Any] so their values are typed, dropping the unknown-argument count back under the ceiling. No behavior change. * fix(a2a): guard localhost retry against a missing agent card handle_a2a_localhost_retry rewrote the card URL and called create_client with whatever agent_card it received. The caller resolves the card from the SDK client (Optional), so a None card reached set_agent_card_url and create_client, surfacing an opaque SDK error instead of a clear one. Add an early RuntimeError guard mirroring the httpx-client check, drop the now always-true card None-check on the stash line, and cover it with a regression test. * style(a2a): disable reportUnknownArgumentType in a2a-sdk boundary modules The lint env type-checks without the optional a2a-sdk/protobuf installed, so every call into the protobuf-generated compat conversions counts as an Unknown-typed argument and the new A2A code pushed the codebase reportUnknownArgumentType total over its ceiling. These three modules are the A2A SDK boundary; turn the rule off file-wide with a documented reason instead of scattering dozens of per-line ignores across every SDK call. * fix(a2a): tolerate unknown fields when lowering 1.0->0.3; align streaming trace id Two issues greptile flagged: version_convert: the 1.0->0.3 lowering paths (_send_result_to, _task_to, _stream_result_to) called ParseDict without ignore_unknown_fields=True, so a 1.0 upstream response carrying vendor extensions raised and best-effort fell back to passing the un-lowered 1.0 shape to a 0.3 client. Set the flag to match the agent-card path and every inbound path; unknown fields are now dropped and the result is correctly lowered. main.py: asend_message_streaming derived X-LiteLLM-Trace-Id from the JSON-RPC request id, unlike asend_message which uses the logging object's litellm_trace_id. Prefer the logging trace id (then request id, then a uuid) so streamed and non-streamed calls correlate under the same trace. Adds regression tests for both, including the stream-event lowering path. * style(a2a): apply ruff format to a2a protocol and proxy modules Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
…oned format and re-encryption migration (BerriAI#31215) * feat(proxy): add AES-256-GCM at-rest credential encryption with versioned format and re-encryption migration * test(proxy): add behavior scenarios for credential migration endpoints * fix(proxy): scan covered tables in encryption check, fix CI lint and route types * fix(proxy): migrate callback_settings credentials, clear CI lint/recursion gates, add encryption endpoint+CLI tests * fix(proxy): correct dry-run/real-run migrated vs residual-legacy counters in config and SSO walkers * fix(proxy): make callback-vars residual detection gate-independent in encryption check
…del_error_cleanup chore(router): simplify unknown-model error message construction
chore(ci): re-issue 31566 for full CI run
…0) (BerriAI#31533) * fix(ui): keep virtual-keys filters across delete and refresh (LIT-4080) Filtering virtual keys by User ID and then deleting a key reset the filter to show all keys, and re-clicking Fetch did not re-apply it. The page ran two competing fetch paths: useKeys (React Query) fetched the page unfiltered while a separate useFilterLogic hook held its own filteredKeys list and, on any refresh, only re-applied Team and Organization client-side, silently dropping the User ID and Key Alias filters. Delete refreshed through the unfiltered useKeys path, so the filtered view collapsed back to everything VirtualKeysTable now owns its filter state and feeds every filter (team, organization, key alias, user id, key hash) straight into the useKeys options, so the filters are part of the React Query key. Any refetch or invalidation re-runs the same filtered query, which makes the reset-on-delete bug structurally impossible. Free-text inputs are debounced with @tanstack/react-pacer, sorting and pagination are server-side, and changing a filter or sort resets to page 1 Delete now invalidates keyKeys.lists() from key_info_view, matching the create path, instead of prop-drilling a refetch; the window "storage" refetch effect is removed. The dual-path useFilterLogic hook (and its test) are deleted Regression coverage: VirtualKeysTable threads an active User ID filter into the useKeys query and clears it on reset, useKeys encodes filter options in its query key so a filter change refetches, and key_info_view invalidates the keys list on delete * refactor(ui): simplify virtual-keys table data flow VirtualKeysTable now fetches its own teams and organizations via useOrganizations and the existing all-teams query instead of taking them as props, so the prop-drill through UserDashboard and the two page callers (page.tsx, ApiKeysDashboard) is gone along with their redundant organization state and fetch Filter state collapses from a useState plus a useDebouncedState mirror into a single source whose debounced copy is derived with useDebouncedValue, and one typed toKeyListFilters adapter maps it to the key/list query options. Behavior is unchanged; same 300ms debounce and the same reset timing The unused onSortChange/currentSort props and their sync effect are removed since no caller passed them, leaving sorting fully internal Adds a created_by_user alias-over-email regression test that fails if the display precedence is swapped * test(ui): add required last_active to useKeys mock fixtures The KeyResponse type requires last_active, so the typed mockKeys fixtures were missing it. Add it so the file type-checks cleanly. * chore(ui): ratchet lint budgets after virtual-keys refactor Deleting filter_logic.tsx and simplifying VirtualKeysTable lowered the no-explicit-any (2026 to 2016) and complexity (128 to 127) counts, so the eslint-metrics.json baseline was stale and failed the frontend-lint budget gate. Regenerate it, and drop the now-dead filter_logic.tsx suppression entry for the file this PR removed. * fix(ui): show a loading state for data-backed filter dropdowns The Team ID and Organization ID filters source their options from async hooks (teams / organizations). While that data was still loading the dropdowns rendered 'No results found', so they looked empty rather than loading. Add an opt-in loading flag to FilterOption that the searchable select surfaces as a spinner and a 'Loading...' empty state, and wire it from the teams and organizations query loading states. While loading, the filter no longer caches an empty initial-options list, so the real options appear once the data arrives.
…rriAI#31638) * perf(ui): load virtual-keys team filter from the fast v2 endpoint The virtual-keys table sourced all teams through fetchAllTeams, which hits the unpaginated /team/list. On a proxy with 125 teams that call takes ~9.5s, so the Team ID filter and the team-alias/budget columns sat empty for that whole window. The key list itself does not carry team_alias or team_max_budget, so the table genuinely needs a team lookup and cannot just drop the fetch. Add useAllTeams, which pages the fast /v2/team/list to completion (~0.6s per 100-team page, so ~1.2s for 125 vs ~9.5s), and point VirtualKeysTable at it instead of fetchAllTeams. The allTeams shape, the filter searchFn, the column lookups, and the loading indicator are all unchanged; only the source endpoint changes. fetchAllTeams stays for its other callers. * test(ui): tighten team-filter test readability and robustness Address adversarial review of the added tests. Rename the single-page useAllTeams test to match what it asserts (one request for a one-page result) rather than implying it guards the early-return, and drop the unread, misleading total: 125 from the mock page response. Scope the created_by alias-over-email assertion to the key's table row so it checks the visible cell value; the hover popover that also holds the email is portaled out of the row, so the previous document-wide negative assertion was relying on antd's lazy popover mounting. * fix(ui): scope useAllTeams cache by access token The previous /team/list query keyed on accessToken, so a user switch in the same SPA session produced a distinct cache entry. useAllTeams dropped that, so team IDs and aliases could be briefly reused across users until the staleTime expired. Put accessToken back in the query key to restore per-identity isolation, and add a regression test that a token switch triggers a refetch rather than serving the cached list.
…ws tool_calls (BerriAI#31633) * fix(databricks): split parallel tool calls so each tool message follows tool_calls Databricks OpenAI-compatible serving (e.g. GPT models) 400s with "messages with role 'tool' must be a response to a preceeding message with 'tool_calls'" when an assistant turn makes parallel tool calls. LiteLLM faithfully sends one assistant message holding all tool_calls followed by one 'tool' message per result, so every result after the first is preceded by another 'tool' message rather than the assistant tool_calls message, which Databricks rejects. Re-emit each result immediately after an assistant message that carries only its matching tool_call, turning assistant(tool_calls=[A, B]), tool(A), tool(B) into assistant(tool_calls=[A]), tool(A), assistant(tool_calls=[B]), tool(B). The rewrite is a no-op when the turn is already valid (single call), the group is incomplete, or ids don't line up, so no tool call is ever dropped. Scoped to non-Claude models, matching the existing OpenAI-shaped transformation path. * style(databricks): use builtin list generics in parallel tool-call split Switch the List[...] annotations introduced by _split_parallel_tool_calls to lowercase list[...] so the UP006 strict-rule budget stays within its ceiling.
…s (VERIA-392) (BerriAI#31469) /key/generate validated the caller's delegation ceiling against data.max_budget only. The per-window entries in data.budget_limits bypassed the check, so a non-admin caller could mint a key whose 1-day window vastly exceeded their own max_budget. The data.permissions dict also went unvalidated for non-admin callers, so they could self-grant capabilities like allow_pii_controls (and on Enterprise, get_spend_routes). Both gates now live in _common_key_generation_helper, covering /key/generate and /key/service-account/generate. The existing empty {} default on permissions still passes for non-admin callers.
…riAI#31604) increment_spend_counters was parallelized in BerriAI#31578, but the dominant per-request cost under high concurrency is the pre-call budget enforcement in common_checks, which still ran a Redis-first get_current_spend per scope (team, team windows, key windows, org, tag, user, team member, end user) one sequential await after another inside the auth span. The per-scope reads target distinct counter keys with no cross-scope ordering dependency, so they now run concurrently under asyncio.gather. Key metadata.tags injection still runs before the gather so the tag budget check sees it, and every scope settles before the first error in scope-priority order propagates, preserving the previous rejection semantics. Resolves LIT-4090
…erriAI#31630) Enforce that every `budget_limits[*].max_budget` is a finite number; applies to every caller including proxy admin and runs before the role / ceiling checks. Six parametrized regression tests cover NaN / +inf / -inf for both non-admin and admin callers.
…erriAI#31631) Mirror the scalar `max_budget` guard in `_common_key_generation_helper` for the per-window check: a CLI session token caller (carrying `max_budget=None`) cannot set `budget_limits` on a personal key. Pass `team_table` into the helper so it can detect the personal-key shape; reject before the `delegation_ceiling is None` early return. Four new regression tests cover the personal-key reject, the team-key happy path, the team-key over-team-budget path, and the proxy-admin exemption.
…reuse Defines the TypedDict mirror of LiteLLM_ObjectPermissionBase under litellm/types/object_permission.py so SDK-side modules can adopt the type without violating the SDK-must-not-import-from-proxy layering rule. litellm/proxy/_types.py re-exports it for existing callers. Supports the validator-surface retypes in the preceding proxy refactor commit.
… on large uploads (BerriAI#31653) * fix(vertex_ai/files): upload batch files in a single media request to fix 499s on large uploads PR BerriAI#31036 switched the vertex batch file upload from a single GCS media upload to a chunked resumable session. The resumable path sends the body as many sequential PUTs, each waiting a full round-trip to GCS before the next, so a multi-GB upload accumulates hundreds of round-trips and overruns the client/load-balancer request timeout, surfacing as 499s (client closed connection) on files as small as 500MB. This was a regression from the last-known-good commit, where the upload completed as one continuous request. Revert the batch upload to a single uploadType=media request, but stage the transformed payload to a temp file first so peak memory stays bounded (the goal of the resumable rewrite) without the per-chunk round-trips. The temp file is closed deterministically (TemporaryFile unlinks on close), not left to the GC. The now-unused resumable chunked-upload plumbing is removed. Also swap the per-row transform's stdlib json for orjson (parse + serialize), which is ~4x faster on this hot path; the streaming body now emits compact orjson bytes. The request stays synchronous, so the returned file object is real and POST /v1/batches keeps working immediately against the uploaded object. Tests: single media request carries the whole payload with a real Content-Length (no chunked transfer-encoding); failed upload raises; the staged temp file is closed deterministically; byte-for-byte transform parity. * test(vertex_ai/files): mock single media upload POST instead of removed resumable method test_avertex_batch_prediction patched BaseLLMHTTPHandler._aresumable_chunked_upload, which was removed when the batch jsonl upload moved from a chunked resumable GCS session to a single uploadType=media request. Patch the raw httpx.AsyncClient.post that _astage_and_upload_media issues so the real staging, upload and response transform run while the GCS object response is mocked, and assert the media URL and Content-Type. * fix(vertex_ai/files): forward request timeout to media upload, drop orjson, sort imports Forward the per-request timeout through _stage_and_upload_media / _astage_and_upload_media to the GCS POST. Every other upload branch forwards it; the new media path was dropping it, so a caller-provided timeout was silently ignored (the files path passes 600s by default, but a custom request_timeout would not have reached this upload). Regression test asserts the resolved timeout reaches the request (mutation-verified). Revert the orjson swap in the batch transform: importing orjson at module load in this core-path file broke `import litellm` on environments without orjson (the Windows import test). Back to stdlib json; the upload leg dominates large uploads anyway, so the transform-side win was marginal. Fix import ordering in llm_http_handler.py (I001) introduced by the new imports. * fix(vertex_ai/files): stream batch upload to GCS instead of staging to a temp file Addresses a disk-exhaustion concern: staging the full transformed batch body to a local temp file before the GCS request meant an authenticated user could fill the proxy's temp volume with large concurrent uploads (on top of Starlette's input spool). GCS's simple/media upload accepts chunked transfer-encoding, so stream the transform straight to the single media request instead. Each block is produced on a worker thread (the transform never runs on the event loop) and sent chunked, so the body is neither buffered in memory nor written to disk, and the upload is still one continuous request (no per-chunk round-trips, no 499). Drops the temp-file staging, the tempfile/IO imports, and Content-Length computation. Regression test asserts the upload streams (chunked transfer-encoding, no Content-Length) and creates no temp file; mutation-verified that reintroducing staging fails it.
…I#31227) * fix(proxy): count only active users toward license seat limit SCIM-deactivated users (metadata.scim_active == false) are kept in LiteLLM_UserTable for audit and reactivation, but they were still counted toward the per-user license limit, so deactivating a user never freed a seat. Okta never sends a SCIM DELETE and Entra only hard-deletes well after deactivation, so deactivation has to be what frees the seat Add UserRepository.count_billable_users(), which counts every row except those where metadata.scim_active is false (absent, null, and true all count), and route the user-create license gate, the free-SSO 5-user cap, and the enterprise /user/available_users display through it. A separate litellm_active_users Prometheus gauge reports the billable count while litellm_total_users keeps its original meaning so existing dashboards are unaffected * fix(proxy): floor billable user count at zero count_billable_users() runs two separate count queries (total, then deactivated). Under a burst of deactivations between them, the deactivated count can momentarily exceed the earlier total and produce a negative result, which would flow into is_over_limit as a negative and show a negative seat count in the display and gauge. Clamp the result to zero so a transient race can never yield a nonsensical negative; the value self-corrects on the next call Addresses Greptile P1 on the PR * refactor(proxy): count teams via TeamRepository in available_users * style: ruff format changed files at line-length 120
…n endpoint (BerriAI#31657) * fix(mcp): resolve per-user OAuth identity authoritatively at the token endpoint The OAuth token endpoint stored a user's per-server token under the identity returned by _extract_user_id_from_request, which read only the Authorization header and did getattr(cached, "user_id") on a raw user_api_key_cache lookup with no model_type rehydration and no DB fallback. That silently returned None in two common cases on a multi-replica gateway: the LiteLLM key arrives on x-litellm-api-key (what MCP clients such as Claude Desktop and Claude Code send) rather than Authorization, and a cross-replica cache hit deserializes to a plain dict rather than a UserAPIKeyAuth, so getattr finds no attribute. When it returned None the token was not persisted. This was survivable until the authorization_code v2 migration began stripping the caller's Authorization for migrated per-user OAuth servers and routing the preemptive 401 existence check through the stored token, so a persist miss now hard-fails: the egress challenges with 401 on every reconnect (the client sees "rejected them on reconnect" or a successful connect with zero tools). Resolve identity through get_key_object, the canonical resolver that reads the cache with model_type and falls back to the DB, and accept the key from x-litellm-api-key as well as Authorization. The silent persist skip is now a warning. The caller-Authorization stripping stays as is, since reinstating it would reopen the cross-user credential override it was added to prevent. * fix(mcp): reject blocked or expired keys when resolving the token-endpoint identity The OAuth token endpoint is unauthenticated, and get_key_object resolves a key row without the blocked/expiry checks the main user_api_key_auth pipeline runs (that pipeline is bypassed here). So a holder of a revoked or expired LiteLLM key could POST a valid upstream authorization code with that key in x-litellm-api-key/Authorization and write or overwrite the stored per-user OAuth token for that key's user. The cache-only resolver this replaced incidentally dropped blocked keys (blocking purges the cache entry), so moving to the authoritative cache-then-DB resolution removed that accidental shield. Validate the resolved key before trusting its identity: return None when blocked or expired, so the upsert is skipped. Deleted keys are already rejected, since get_key_object raises on a missing row. Regression tests cover the blocked and expired cases and fail without the guard.
…nal_key_metadata fix(proxy): reject team-scoped object_permission on personal keys for non-admins
…es (BerriAI#31645) * fix(passthrough): drop top-level additional_drop_params on /v1/messages On the Anthropic Messages pass-through path, additional_drop_params only stripped nested dotted paths, so plain top-level keys like `thinking` and `context_management` were forwarded to the provider. Bedrock rejects these with "Extra inputs are not permitted", returning a 400 to Claude App/CLI even when the user configured `additional_drop_params: ["thinking"]`. delete_nested_value already handles plain top-level fields, so route every drop param through it and remove the nested-only filter. Fixes BerriAI#25931. * fix(passthrough): drop thinking for bedrock inference-profile ARNs on /v1/messages Opaque Bedrock Application Inference Profile ARNs contain neither "anthropic" nor "claude", so is_anthropic_claude_model returned False and the thinking param was rewritten to reasoning_effort before additional_drop_params ran. That made additional_drop_params: ["thinking"] a no-op for the converse-ARN form, and the Bedrock Converse transform re-expanded reasoning_effort back into additionalModelRequestFields.thinking, so the request 400'd. Extend the thinking-translation gates to also accept bedrock ARNs via the existing is_bedrock_arn_model helper, mirroring the cache_control path, so thinking is preserved as thinking and additional_drop_params can drop it.
…erriAI#31655) The Model Armor guardrail only sent text extracted from user messages to sanitizeUserPrompt, so harmful content inside attached PDFs, Office docs, and CSVs reached the LLM unscanned. A file-only message had no extractable text, so the pre-call and moderation hooks returned early and the document was never submitted to Model Armor at all. Wire inline document/file scanning into async_pre_call_hook and async_moderation_hook. extract_file_attachments walks message content blocks (OpenAI type:file file_data and Anthropic type:document source), decodes the base64 bytes, maps the MIME type to a Model Armor byteDataType, and skips remote URLs, bare file_id references, oversize files past the 4 MB limit, and unsupported types. Each attachment is sent through the byte API and a MATCH_FOUND blocks the request before it reaches the LLM. Resolves LIT-4084
…31668) * fix: skip health check for semantic auto_router deployments auto_router/<name> deployments are semantic meta-routers that select among real LLM deployments at request time. They have no LLM endpoint to probe. The health check was passing model=auto_router/router_1 to get_llm_provider(), which raised BadRequestError: "Unmapped LLM provider for this endpoint" because auto_router is not a real LLM provider, causing these deployments to always appear unhealthy and curl requests to hang. Detect semantic auto_router deployments in _run_model_health_check and return {} (healthy) without calling litellm.ahealth_check. Sub-strategies (complexity_router, adaptive_router, quality_router) are excluded from this fast path and continue to be health-checked normally. * ci: trigger circleci
…request (LIT-3858) (BerriAI#31663) * fix(bedrock): drop unmappable Responses tools instead of failing the request (LIT-3858) When an OpenAI Responses request is routed to a Bedrock Converse Anthropic model, litellm translates the tools array into Bedrock toolConfig. Responses built-in tool types beyond function (web_search, image_generation, namespace, tool_search, custom) have no Bedrock equivalent, and previously caused two failures. A web_search tool is derived into a web_search_options param. Bedrock Anthropic models do not list web_search_options in get_supported_openai_params, so the request raised UnsupportedParamsError (HTTP 400) even though it never needed web search. The derived param is now dropped on the Bedrock chat-completion bridge for models that do not support it, scoped to Bedrock so other providers are untouched and without requiring drop_params. Nova still keeps it since it maps to a nova_grounding systemTool. The remaining non-function tools reached _bedrock_tools_pt and were emitted as junk litellm_unnamed_tool_N toolSpecs with empty schemas, polluting toolConfig with tools the model could hallucinate calls to. They are now dropped because they carry neither an OpenAI function nor an Anthropic input_schema, while mappable function and input_schema tools survive untouched. * refactor(responses): drop derived web_search_options via provider config Greptile flagged that the LIT-3858 fix put Bedrock-specific logic in the generic Responses->Chat Completion bridge: it imported AmazonConverseConfig and branched on custom_llm_provider.startswith("bedrock"). Read web_search_options support from each provider's own get_supported_openai_params instead, so the bridge stays provider-agnostic and Bedrock capability knowledge lives in the Bedrock config that already owns it. Behavior is unchanged for the cases the PR targeted (Bedrock Anthropic drops, Bedrock Nova and OpenAI keep) and now generalizes correctly to any provider whose config does not support the derived param. Add a Cohere regression test proving the drop is provider-agnostic; it fails under the old bedrock-only check and passes now. * fix(responses): drop derived web_search_options for bedrock_converse alias Greptile/T-Rex caught that the provider-agnostic drop regressed the bedrock_converse route: get_supported_openai_params did not map the bedrock_converse alias (only "bedrock"), so it returned None (unmapped) and the derived web_search_options was forwarded for model="bedrock/converse/us.anthropic.claude-sonnet-4-6", custom_llm_provider="bedrock_converse" instead of being dropped. The previous startswith("bedrock") check happened to match the alias. Map bedrock_converse through AmazonConverseConfig in get_supported_openai_params, mirroring the existing ["bedrock", "bedrock_converse"] pairing in _strip_model_name. Add regression tests at both levels: the alias now drops the derived param for Anthropic Converse models, still keeps it for Nova, and the helper resolves identically to "bedrock".
…ession storage The PR #121 conflict resolution took upstream's streaming_iterator.py __init__ (which has no litellm_completion_request param) while git auto-merged handler.py cleanly, keeping CARTO's call sites that pass litellm_completion_request= (PR #16, Redis session storage). Result: every streaming Responses API request raised TypeError: LiteLLMCompletionStreamingIterator.__init__() got an unexpected keyword argument 'litellm_completion_request' (3,079 occurrences in the cloud-native integration run for PR BerriAI#26437; 36 AI integration tests failed). The method body _store_session_in_redis also reads self.litellm_completion_request, which was never assigned. Restore the param and the attribute assignment exactly as on carto/main before the sync (7244577), with CARTO markers so future syncs see it.
tests/litellm/responses/test_streaming_iterator_tool_calls.py (from PR #68, OCI Gemini tool-call UUID fallback) tested iterator internals that upstream v1.92.0 removed: _handle_tool_call_delta and _emit_function_call_done_events no longer exist, so all 8 tests fail with AttributeError. The behavior itself is preserved upstream with a better implementation: litellm/llms/oci/chat/generic.py now synthesizes deterministic content-derived tool-call ids (_synthesize_oci_tool_call_id) whenever OCI omits them, and the iterator routes id-less deltas via index-to-id mapping (_queue_tool_call_delta_events). Covered by tests/test_litellm/llms/oci/ and tests/test_litellm/responses/litellm_completion_transformation/ test_tool_call_streaming_transformation.py (375 tests passing). The carto-features.yml manifest was already updated to the new patterns during the PR #121 conflict resolution.
Collaborator
|
Integration-test failure root cause + fix (from cloud-native PR #26437 AI test failures): The conflict resolution took upstream's Pushed two commits:
Audit of all 11 manifest features against the branch: all verification patterns pass, and the targeted unit suites (responses transformation, chunk builder, snowflake, databricks, oci, azure) pass locally: 971 passed, 0 failed. The Redis wiring break was the only production-code regression. |
Collaborator
Author
📊 CARTO Feature Analysis StartedMode: Analysis + Auto-fix if issues found Analyzing how each CARTO customization was handled during conflict resolution.
|
Collaborator
Author
✅ CARTO Feature Analysis Complete
Overall Assessment: PASS 📋 Full details in PR description above. |
4 tasks
…andler Second dropped wiring from the PR #121 conflict resolution, subtler than the first: async_responses_api_session_handler lost the CARTO PATCH that reads sessions from Redis before the DB-backed store (PR #16). Both Redis helpers survived the merge, so manifest pattern greps passed, but _patch_get_session_from_redis was left orphaned - defined with zero call sites. Session writes (restored in 1c1f2a0) landed in Redis while reads went to the batch-delayed SpendLogs store, so immediate follow-up turns lost their conversation history. Observed blast radius in cloud-native PR BerriAI#26437 integration rerun: conversations answered with other conversations' content, ai-api agent loops spun (4,333 /responses calls in 13 minutes, ~25/min is normal), and the CI org burned through its 10M AI quota mid-run. Restore the Redis-first read exactly as on carto/main pre-sync, and add regression tests pinning the read-before-DB wiring so an orphaned helper can't silently pass future syncs.
Collaborator
Author
📊 CARTO Feature Analysis StartedMode: Analysis + Auto-fix if issues found Analyzing how each CARTO customization was handled during conflict resolution.
|
Collaborator
Author
✅ CARTO Feature Analysis Complete
Overall Assessment: PASS 📋 Full details in PR description above. |
…e Redis store keys Two changes closing the remaining session-context regression from the v1.92.0 sync (integration rerun on cloud-native PR BerriAI#26437 still showed multi-turn context loss: "assistant failed to recall the previous instruction", agent loops, 3,259 /responses calls): 1. Replace the earlier condensed restoration of the Redis-first session lookup with carto/main's exact block, byte-identical including its verbose_logger.debug lines (verified with diff against 7244577). Verbatim over rewrite - the debug lines also make CI diagnosable. 2. Key Redis session stores by the DECODED response id. This bug cannot be fixed by copying carto/main code because it does not exist at v1.83.14: upstream v1.92.0 introduced b64-encoded response ids (response.completed carries the encoded id) while previous_response_id is decoded in responses/utils.py before reaching the session handler. carto/main's code keyed stores by response.id, which at v1.83.14 was the raw provider id (store == lookup); on v1.92.0 the same code keys by the encoded id and the decoded lookup misses 100% of the time. Verified empirically by driving the streaming iterator on both versions. Decode at the two store call sites; the CARTO Redis helpers themselves remain byte-identical to carto/main. Adds a regression test driving the real streaming iterator and asserting the stored key equals the decoded form of the client-visible response id.
Collaborator
Author
📊 CARTO Feature Analysis StartedMode: Analysis + Auto-fix if issues found Analyzing how each CARTO customization was handled during conflict resolution.
|
Collaborator
Author
✅ CARTO Feature Analysis Complete
Overall Assessment: PASS 📋 Full details in PR description above. |
…hunks SnowflakeStreamingHandler._parse_anthropic_chunk set function["name"] to the cached tool name on every input_json_delta continuation chunk, not just the content_block_start chunk. OpenAI-compatible clients accumulate tool_call name by index across chunks, so the repeated name concatenates into e.g. "set_layer_styleset_layer_style..." and the tool is no longer recognized by the caller.
Collaborator
Author
📊 CARTO Feature Analysis StartedMode: Analysis + Auto-fix if issues found Analyzing how each CARTO customization was handled during conflict resolution.
|
Collaborator
Author
✅ CARTO Feature Analysis Complete
Overall Assessment: PASS 📋 Full details in PR description above. |
handle_generic_stream_chunk never set an index on tool-call deltas, so every concurrent tool call in one agent turn defaulted to index 0. litellm's stream_chunk_builder accumulates streaming tool calls by index, so a second tool call in the same turn silently overwrote the first in the final reconstructed message instead of appearing alongside it. Assign the index by first-seen tool-call id, persisted across chunks on OCIStreamWrapper the same way the existing Cohere dedup state is threaded through.
Collaborator
Author
📊 CARTO Feature Analysis StartedMode: Analysis + Auto-fix if issues found Analyzing how each CARTO customization was handled during conflict resolution.
|
Collaborator
Author
✅ CARTO Feature Analysis Complete
Overall Assessment: PASS 📋 Full details in PR description above. |
… turns Cortex's Anthropic-compatible /messages endpoint rejects unknown fields inside text content blocks: messages.N.content.0.text.annotations: Extra inputs are not permitted The OpenAI Agents SDK attaches an `annotations` field to text blocks when replaying a prior assistant turn, and that field rides straight through every follow-up-turn request. This is the same class of gap the v1.92.0 rewrite introduced in the two prior fixes: a CARTO customization for this exact problem (_strip_openai_annotations) still existed in the file but was never wired into the new Anthropic-format message builder. Restored it as a content-level, non-mutating helper and call it from both plain message pass-through branches in _extract_system_and_messages.
…ll site The prior pattern set (def _content_to_text_blocks, def _strip_openai_ annotations, content_blocks) only proved those symbols existed in the file, not that they were reachable. The resolver weakened the original text_prelude_blocks pattern to content_blocks -- a name that also appears in unrelated new code -- specifically to make this check pass after the v1.92.0 rewrite silently dropped the annotation-stripping call site. That's why CI stayed green through the regression this PR fixed. Verification now greps for the actual call site. Also narrows the description to what this entry actually verifies today (annotation stripping only) rather than the original PR's full scope, since the consecutive-assistant-turn merge is not currently implemented against the new message builder.
Collaborator
Author
📊 CARTO Feature Analysis StartedMode: Analysis + Auto-fix if issues found Analyzing how each CARTO customization was handled during conflict resolution.
|
Collaborator
Author
|
| Decision | Count |
|---|---|
| Upstream Substitutes | 2 |
| Customized Upstream | 4 |
| Preserved CARTO | 6 |
| Incorrectly Dropped | 0 |
Overall Assessment: PASS with one item requiring verification. All 12 CARTO features have either been preserved, adapted to the new architecture, or superseded by equivalent upstream functionality. The _content_to_text_string function needs verification to confirm the /messages endpoint accepts array content
📋 Full details in PR description above.
…lls order OCI GENERIC validates tool results positionally against the assistant message's toolCalls, rejecting out-of-order results with "Invalid parameter: 'toolCallId' of '<id>' not found in 'toolCalls' of previous message" even though the id is present in that message. Models that issue parallel tool calls (xAI Grok) get their results back in completion order, which need not match the emission order, so every multi-call turn 400s. Confirmed against a captured outbound request: assistant toolCalls [id-0, id-1] followed by TOOL(id-1), TOOL(id-0) is rejected on id-1. Reorder each run of consecutive TOOL messages to the preceding assistant's toolCalls order in the GENERIC adapter; results are keyed by toolCallId everywhere else, so the reorder is semantically neutral. Unknown ids keep arrival order after the known ones.
Collaborator
Author
📊 CARTO Feature Analysis StartedMode: Analysis + Auto-fix if issues found Analyzing how each CARTO customization was handled during conflict resolution.
|
Collaborator
Author
✅ CARTO Feature Analysis Complete
Overall Assessment: PASS 📋 Full details in PR description above. |
mateo-di
approved these changes
Aug 4, 2026
mateo-di
left a comment
mateo-di
left a comment
Collaborator
There was a problem hiding this comment.
QA Result: success. Merging and creating new release.
3 tasks
This was referenced Aug 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
16 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🔄 Upstream Sync: LiteLLM v1.92.0
Syncs CARTO's LiteLLM fork with upstream stable release v1.92.0.
1.83.14→v1.92.0Caution
Use "Create a merge commit" only. Squashing destroys upstream history and breaks future syncs.
🧪 Pre-Merge Checklist
pyproject.tomlversion matches upstream📊 Release Information (click to expand)
v1.92.01.83.14🔀 Branch Flow (click to expand)
BerriAI/litellm:mainmerged intoCartoDB/litellm:mainupstream-sync/v1.92.0upstream-sync/v1.92.0→carto/main📝 CARTO-Specific File Guidelines (click to expand)
When reviewing or resolving conflicts:
✅ Keep CARTO Versions (Ours)
.github/workflows/carto_*.yaml- CARTO workflows.github/workflows/carto-*.yml- CARTO workflowsCARTO_*.md,docs/CARTO_*.md- CARTO documentation🔄 Accept Upstream (Theirs)
pyproject.toml- Version fieldlitellm/- Core library codetests/- Upstream testsrequirements.txt- DependenciesDockerfile,docker/Dockerfile.non_root- CARTO customizationsMakefile- Check# CARTO:sections🔧 Conflict Resolution (click to expand)
If this PR has conflicts:
Option 1: Automated (Recommended)
The carto-upstream-sync-resolver workflow triggers automatically.
What it does:
carto/main→ ✏️ Resolves conflicts → 🧪 Runs tests → 📌 Pushes to this PRYou just need to: Wait for resolution commits, verify CARTO customizations, merge.
Option 2: Manual Resolution
📚 Documentation Links (click to expand)
🤖 This PR was automatically created by the carto-upstream-sync workflow.
🔧 CARTO Feature Fixes Applied
Status: ✅ Fixed
Features Restored: 1
PR #118: fix(docker): fetch arm64 prisma schema-engine for non_root image
Fixed: 2026-07-14 02:47:30 UTC
Workflow Run: #31
CARTO Customizations Analysis
Overall Assessment: ✅ PASS
CARTO Feature Preservation Analysis
Summary
Overall Assessment: PASS
All 12 CARTO features from the manifest have been correctly handled during the upstream sync resolution. No features were incorrectly dropped. The verification patterns for all features exist in the resolved codebase.
Feature Details
Upstream Substitutes (2)
These features are now provided by upstream LiteLLM, so CARTO-specific code was correctly not duplicated:
setdefaultfor tool call ID generation, achieving the same functionalityCustomized Upstream (2)
These features use upstream as the base while adding CARTO-specific enhancements:
select_azure_base_url_or_endpointwas enhanced with CARTO'sre.sublogic to strip operation suffixes that the SDK would append again (+22 lines)Preserved CARTO (8)
These are CARTO-only features not present in upstream, all correctly preserved:
_validate_and_repair_tool_argumentsrepairs malformed JSON in streaming_store_session_in_redisand_patch_store_session_in_redisfor session persistence_strip_openai_annotationsremoves OpenAI-only fields Cortex rejects_content_to_text_stringflattens array-form content to plain stringsIssues Found
None. All CARTO features are present and verified.
Feature-by-Feature Breakdown
PR #68: OCI Gemini Tool Call UUIDs
PR #121: OCI Parallel Tool Result Reordering
PR BerriAI#17159: OCI Inline PEM Key Normalization
PR #[38,58]: Snowflake Streaming + Tool Calling
PR #[]: Snowflake Full URL Passthrough
PR #70: Azure URL Suffix Stripping
PR #54: JSON Repair for Streaming Tool Calls
PR #16: Redis Session Storage
PR #111: Snowflake Cortex Claude Function-Calling Follow-up Turns
PR #112: Snowflake Cortex Array Content Flattening
PR #[109,110]: Databricks Empty Tool Call Arguments Normalization
PR #110: Databricks Strip OpenAI Annotations
Analyzed: 2026-08-03 17:15:24 UTC
Workflow Run: #40
Analysis Artifacts: Download JSON/MD
Method: Claude Code (Opus 4.5) post-resolution semantic analysis