chore(release): backport 11 staging PRs onto patch-1.92.0rc2 for the 1.92.0 stable cut - #32959
Merged
Merged
Conversation
…ricing in get_model_info (LIT-4056) (#32389) * fix(utils): resolve bedrock regional inference profiles to regional pricing in get_model_info (LIT-4056) * test(register_model): use a triple provider prefix as the unresolvable-key fixture get_model_info now resolves bedrock/bedrock/... like a routing prefix, so the double-prefix fixture stopped exercising the register_model fallback path. Lock the new double-prefix resolution in as a model-info regression test (cherry picked from commit 734fd29)
…elpers (#32542) * fix(guardrails): walk Responses-API text taxonomy in shared content helpers Every guardrail sharing litellm/proxy/guardrails/_content_utils.py silently drops all text on the /v1/responses path. AIM turns it into a loud 422 ( {"error":"No messages in the request"}); every other guardrail (Lakera v2, Cato, Lasso, Repello, IBM, Azure Content Safety, enterprise secret detection) scans an empty payload and lets the request through unscanned. Three defects, all in _content_utils.py: 1. _iter_text_parts_in_content recognised only part.type == "text", but the Responses API uses input_text (request) and output_text (assistant). 2. _coerce_input_to_messages gated on "every item has a role key"; any Responses input list containing a function_call or function_call_output item failed the check and was wrapped as one opaque blob. 3. build_inspection_messages forwarded any role through, including a bare tool role missing tool_call_id, which validators like AIM's /fw/v1/analyze reject with a schema error. Fix walks the actual Responses item taxonomy (message, function_call, function_call_output, bare content parts and strings), recognises {text, input_text, output_text} everywhere, and coerces any role outside {system, user, assistant} to user in the outbound inspection payload. * style: ruff-format changed guardrail files * test(guardrails): cover function_call_output string form; drop em-dash in new docstring * fix(guardrails): map function_call_output straight to user role Avoids ever materialising a schema-invalid bare tool message. The downstream role-safety coercion in build_inspection_messages still guards genuinely caller-supplied non-standard roles (developer, function, custom values); add a regression test covering that path so the coercion has real coverage after this simplification. * test(guardrails): pin chat-completions tool-role coercion in build_inspection_messages * docs(test): soften AIM-specific claims in LIT-4294 test docstrings Ryan's review flagged that several test docstrings assert AIM's /fw/v1/analyze validates + rejects specific schema violations. That behavior is customer-reported in the LIT-4294 writeup, not directly verified by us. Rephrase to attribute the AIM 422 to the customer's writeup and describe the underlying constraint as the OpenAI chat schema; any downstream API that validates against that schema rejects the same shape. * refactor(guardrails): move unsupported-role coercion into AIM only The generic coercion in build_inspection_messages collapsed any role outside {system, user, assistant} to user for every caller of the helper. Combined with the pre-existing apply_redacted_messages_back write-back behavior in Lakera/AIM/Cato, that turned a loud OpenAI 400 on chat-completions tool-message masking into a silent semantic corruption of the outbound request (role tool with tool_call_id got rewritten to bare role user, dropping the assistant + tool_calls sibling). AIM specifically requires the coercion because its /fw/v1/analyze validates the payload against the OpenAI chat schema; other guardrails either do not validate roles or do their own reconstruction. Move the coercion to AimGuardrail._build_aim_inspection_messages so the shared helper keeps caller roles intact and no new cross-guardrail role corruption is introduced. The pre-existing apply_redacted_messages_back structural flatten remains as separate follow-up work. function_call_output items still synthesise role user in the shared helper because they have no natural role field, which is a different concern from coercing a caller-supplied role. * refactor(guardrails): preserve role fidelity in shared _content_utils Shared inspection helpers should extract text and preserve semantic role signals; role coercion for third-party schema safety stays inside the guardrail that needs it (AIM). Three shared-helper changes: - Bare content-part dicts (input_text/output_text) with an explicit role keep it; only role-less parts default to user. - Responses message items already had their role preserved; the behavior is now covered by an explicit test. - function_call_output items default to role tool (semantic equivalent of the chat-completions tool message shape) instead of role user, so Responses and chat completions produce symmetric inspection payloads. A caller-supplied role on the item is still preserved. AIM's schema-safe coercion in _build_aim_inspection_messages already handles the resulting role tool: it collapses to user before the POST to /fw/v1/analyze so AIM's OpenAI-schema validator does not reject the bare tool message (no tool_call_id can survive the flatten). Added a regression test in test_aim.py covering that path. (cherry picked from commit e84a19a)
(cherry picked from commit d82645d)
…de Invoke (#32578) Hoisting every role system entry into the top-level system field mutates the cache prefix whenever a client such as Claude Code appends a new mid-conversation system message, invalidating the prompt cache for the entire message history on Bedrock Invoke. Bedrock only rejects a system entry at messages.0, so hoist just the leading run and forward the rest in place (cherry picked from commit cc36d54)
…d LLM calls (#32655) * feat(otel): emit the gen_ai.client.operation.exception event on failed LLM calls The GenAI semantic conventions record failures of a GenAI client operation as a log-based event named gen_ai.client.operation.exception, carrying the exception.type / exception.message / exception.stacktrace trio at severity WARN and correlated to the failed span. OTel v2 never emitted it: a failed LLM call produced only the deprecated error.* span attributes, a generic exception span event without a stacktrace, and the stacktrace under the vendor key litellm.provider.error.stack_trace. Build the logs pipeline (LoggerProvider + console/OTLP log exporters mirroring the metrics plumbing) and record the event behind the enable_events flag, which until now was defined but consumed nowhere. An operator-configured LoggerProvider global is reused so the events ride their existing logs pipeline; an explicit NoOpLoggerProvider global is honored as an opt-out and builds no recorder at all. The existing span-side error surface (error.type, error.message, the exception span event, and the litellm.provider.error.* detail keys) is untouched for backwards compatibility. * fix(otel): always ride the semconv-required exception pair on the GenAI event Filtering the event attributes on truthiness conflated "absent" with "empty", so an empty exception.type or exception.message would have been dropped, leaving an event with neither semconv-required field. Build the attributes so the pair is unconditional and only the recommended stacktrace is omitted when the payload carries none. * docs(otel): document the events plumbing module in the package README * test(otel): cover the log exporter selection and logs endpoint normalization The new logs plumbing had no coverage for exporter-kind selection, the console fallback for an unrecognized kind, the /v1/logs signal-path rewriting that lets one OTEL_ENDPOINT serve every signal, or the simple-vs-batch processor split. (cherry picked from commit 99b4c5e)
…upport (#32867) * fix(anthropic): translate adaptive thinking/effort to pre-4.6 model support AnthropicMessagesConfig now reshapes the 4.6+ adaptive-thinking interface (thinking:{type:adaptive} + output_config:{effort:...}) to whatever the routed model supports. Thinking-capable non-adaptive models (e.g. Haiku 4.5, Sonnet 4.5) get the effort translated to a legacy thinking budget_tokens. Models with no reasoning support have thinking/effort dropped under drop_params. And because adaptive thinking carries no budget while the legacy form must satisfy Anthropic's max_tokens > budget_tokens rule, the translated budget is capped below max_tokens, dropping thinking when max_tokens can't fit the minimum budget. 4.6+ models pass through untouched. This matters because clients like Claude Code speak native Anthropic /v1/messages and send the adaptive interface unconditionally, regardless of the routed model. The native passthrough previously only capability-gated the OpenAI-style reasoning_effort alias and forwarded native output_config/adaptive thinking raw, so a pre-4.6 model rejected it with "This model does not support the effort parameter" and the request failed. Claude Code already gets drop_params auto-set, so its requests now succeed. * test(anthropic): gate undersized-max_tokens thinking drop on drop_params; add edge tests Addresses review feedback on the max_tokens-too-small branch. Previously a thinking-capable model whose max_tokens could not fit the minimum thinking budget had thinking silently dropped regardless of drop_params, while a residual output_config field in the same call still raised when drop_params was off. Gate both consistently on drop_params: raise a clear error (naming max_tokens for the undersized case) when drop_params is off, drop otherwise. Claude Code gets drop_params auto-set, so it still succeeds. Adds tests for the undersized-max_tokens raise, the residual output_config raise, and the no-adaptive-interface passthrough on a non-adaptive model. * fix(anthropic): make adaptive-effort translation silent to avoid breaking provider strip contracts The previous raise-when-not-drop_params behavior broke existing bedrock and vertex messages tests: those providers already silently strip unsupported output_config for pre-4.6 models (issue #22797) with no drop_params required, and the shared parent transform raising pre-empted that. It also conflicted with the goal of keeping requests working rather than failing them. Make the reshape silent: translate effort to legacy thinking for thinking-capable models, drop thinking for non-reasoning models, and remove only the consumed effort key from output_config, leaving any residual (e.g. format) for provider subclasses (bedrock/vertex) to handle. No raise, no drop_params gating. This also resolves the review note about inconsistent drop_params handling by making every path uniform. Updates the tests to assert the silent behavior and residual output_config preservation. * fix(anthropic): handle output_config-capable but non-adaptive models (Opus 4.5) Greptile caught a real bug: the early-return guard treated supports_output_config as equivalent to supporting adaptive thinking. Claude Opus 4.5 advertises supports_output_config (it accepts output_config.effort) but is not adaptive, so it rejects thinking:{type:adaptive} with "adaptive thinking is not supported on this model". The guard early-returned for Opus 4.5 and forwarded the adaptive thinking block raw, reproducing the exact failure the fix is meant to prevent. thinking:{type:adaptive} and output_config.effort are independent capabilities. Only early-return for adaptive-thinking models. For a model that supports output_config.effort but is not adaptive, keep the native effort and drop only the unsupported adaptive thinking block. Verified live against Opus 4.5: the Claude Code payload now returns 200 instead of 400. Adds regression tests for Opus 4.5 with and without adaptive thinking. * fix(anthropic): translate adaptive thinking for effort-capable pre-4.6 models Claude Opus 4.5 advertises supports_output_config but not adaptive thinking, so the early-return guard forwarded thinking.type=adaptive raw and Anthropic rejected it. The guard now only skips true adaptive models; effort-only requests on effort-capable models still pass through untouched. The _map_reasoning_effort call is wrapped to surface unrecognized effort values as a clean 400, matching _translate_reasoning_effort_to_anthropic * fix(anthropic): fall back to legacy thinking when effort level unsupported Opus 4.5 accepts output_config.effort but only low/medium/high; Claude Code defaults to xhigh on newer models, so preserving that level raw gets rejected by Anthropic. Gate the native-effort passthrough on _validate_effort_for_model and fall through to the budget translation for unsupported levels * fix(anthropic): keep effort-only requests untouched for provider normalization The xhigh fall-through consumed effort-only requests on effort-capable models, breaking bedrock invoke's own normalization which clamps xhigh to the model's ceiling after the base transform runs (test_bedrock_messages_normalizes_output_config_effort_for_opus). Restrict the fall-through to requests that carry adaptive thinking; effort-only requests pass through so provider subclasses keep owning level clamping --------- Co-authored-by: Abhimanyu Kapur <38531241+akapur99@users.noreply.github.com> (cherry picked from commit 3a62e54)
…rsation_system (#32882) Exact cost-map hits resolve before fallback-generalization rules, so the mapped Sonnet 5, Fable 5 and jp Opus 4.8 Bedrock entries bypassed the bedrock-anthropic-claude-mid-conversation-system rule and hoisted mid-conversation system messages, invalidating the prompt cache. (cherry picked from commit c15891f)
…_split refactor(fallback-generalizations): split rules into routing and provider-neutral capability kinds (cherry picked from commit 45d3644)
…lity_probes fix(anthropic): thread real provider through capability probes instead of pinning anthropic (cherry picked from commit ead7ad3)
(cherry picked from commit bf02a4a)
|
|
Contributor
Greptile SummaryThis release backport cherry-picks eleven already-merged staging PRs onto
Confidence Score: 4/5Safe to merge with one small fix: the The production logic across all eleven cherry-picks is well-reasoned and thoroughly tested (1163 tests, 0 failures, determinism sweep). The capability-provider threading, Bedrock mid-conversation system support, fallback generalisation split, and OTel exception event plumbing are all correctly implemented. The single defect is the bare
|
| Filename | Overview |
|---|---|
| litellm/proxy/guardrails/guardrail_hooks/aim/aim.py | Adds _build_aim_inspection_messages to normalize non-standard roles before sending to AIM; uses m["role"] (direct key access) which can KeyError on malformed messages. |
| litellm/litellm_core_utils/fallback_generalizations.py | Refactors generalizations into distinct routing and capability rule kinds; install-time compilation replaces lazy caching; adds legacy-schema shim for remote-fetched JSON. Well-structured and tested. |
| litellm/llms/anthropic/common_utils.py | Threads custom_llm_provider through all capability probes so cost-map lookups use the caller's actual provider instead of always pinning to "anthropic". |
| litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py | Adds supports_mid_conversation_system awareness; for flagged models (Opus 4.8+) only hoists leading system-role messages, preserving prompt-cache validity. |
| litellm/utils.py | Switches split_model lookup priority, adds Bedrock routing-prefix strip, and fixes _get_model_info_from_generalization to skip rules when any candidate has an exact cost-map entry. |
| litellm/proxy/guardrails/_content_utils.py | Extends text-part type recognition to Responses-API types; replaces bulk-message heuristic with per-item dispatch in _coerce_input_to_messages. |
| litellm/integrations/otel/plumbing/providers.py | Adds LoggerProvider/EventLoggerProvider plumbing for the new GenAI exception event signal, mirroring the existing meter-provider resolution pattern. |
| litellm/integrations/otel/plumbing/events.py | New module owning GenAIEventRecorder; emits gen_ai.client.operation.exception as a WARN-severity OTel log event with trace/span correlation. |
| litellm/llms/anthropic/experimental_pass_through/messages/transformation.py | Adds _translate_adaptive_effort_for_non_adaptive_model to downgrade the 4.6+ adaptive-thinking interface for older models. |
| litellm/llms/openai_like/messages/transformation.py | Adds JSONProviderAnthropicMessagesConfig for JSON-configured providers advertising /v1/messages. |
| litellm/llms/anthropic/chat/transformation.py | Adds _resolved_provider property and threads it through all capability method call sites. |
| model_prices_and_context_window.json | Adds muse-spark-1.1 (Meta), au/eu/us.anthropic.claude-opus-4-8 regional entries with supports_mid_conversation_system: true, and the meta fallback routing rule. |
Reviews (1): Last reviewed commit: "test: add /v1/messages to supported_endp..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+124
to
+125
| safe_roles = {"system", "user", "assistant"} | ||
| return [{**m, "role": "user"} if m["role"] not in safe_roles else m for m in build_inspection_messages(data)] |
Contributor
There was a problem hiding this comment.
m["role"] will raise KeyError for any message dict that lacks a "role" key. While build_inspection_messages currently guarantees a "role" on every message, this is an implementation detail of the callee — a future change to that function or a malformed intermediate message would silently break the entire guardrail pre-check with an unhandled exception. m.get("role") is the defensive form and matches the style of the surrounding code.
Suggested change
| safe_roles = {"system", "user", "assistant"} | |
| return [{**m, "role": "user"} if m["role"] not in safe_roles else m for m in build_inspection_messages(data)] | |
| safe_roles = {"system", "user", "assistant"} | |
| return [{**m, "role": "user"} if m.get("role") not in safe_roles else m for m in build_inspection_messages(data)] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relevant issues
This backports eleven already-merged
litellm_internal_stagingPRs ontopatch-1.92.0rc2(the branch cut at thev1.92.0-rc.2tag) so they ride the pending1.92.0stable release. Seven were requested for the release; the other four are prerequisites those seven need to compile and pass their own tests on this older line, without which the requested picks would be dead on arrival.No version bump: the branch's
pyproject.tomlalready holds the final1.92.0, andv1.92.0has not shipped anywhere yet (no DockerHub/GHCR image, no git tag, norelease/v1.92.0branch), so the picks ride the pending version rather than bumping it. This matches the rc1 -> rc2 backport, which was also bump-less.Linear ticket
What is included (staging merge order)
Requested for the release
Prerequisites (required for the above to work on this line; each is itself a merged staging PR)
Every commit is a
git cherry-pick -xof the corresponding staging squash/merge; there is no hand-written code and no version-bump, lock, or UI-artifact commit (no production.tsxis touched).Adaptation notes
Four picks diverge from their staging source; the rest are byte-for-byte (patch-id verbatim).
test_meta_provider.pyshipped without a local-cost-map fixture, so its four muse-spark model-info tests passed only when an earlier test happened to leave the bundled cost map inlitellm.model_cost; run in isolation they failed becausemuse-spark-1.1is a day-0 model not yet on the remotemainmap. Added the samelocal_model_cost_mapfixture that sibling day-0 model tests use, pinning the bundled map for those four tests. This is a determinism fix, not a behavior change; the provider code itself is unchanged and correct.jp.anthropic.claude-opus-4-8(added to staging by a PR outside this set), so that one entry is skipped and the other ten are flagged. Behavior is preserved because fix(bedrock): gate in-place system role messages on model support for Claude Invoke #32831'sbedrock-anthropic-claude-mid-conversation-systemfallback rule still flags unmapped Claude 4.8+ ids, including the jp variant. Its parametrized tests that namejp.anthropic.claude-opus-4-8resolve through that rule and pass.basedpyright-code-budget.json/type-discipline-budget.jsonratchets from the staging diff are dropped; this line keeps its own ceilings. Staging's lowered limits were computed against staging's full tree, not this subset, so importing them could produce a too-tight gate. The functional refactor (utils.py, fallback_generalizations.py, model-cost JSON) is applied verbatim.TestDefaultSuffixAdaptiveThinking, the LIT-3393 beta-headers fixture). Those were excluded; only fix(anthropic): thread real provider through capability probes instead of pinning anthropic #32874's own additions were kept (TestCapabilityProbeUsesCallerProvider,test_bedrock_messages_thinking_shape_follows_exact_bedrock_entry_flag) plus the required_is_adaptive_thinking_model(model)->(model, "anthropic")signature migration on the seven existing call sites this line does have. Every source caller of the changed signatures is updated, so no existing caller breaks.Known noise on this line
litellm/proxy/management_endpoints/credential_migration.pyis not formatted to the line'sruff formatgate onpatch-1.92.0rc2itself; no pick touches it, so a redformat-checkon that file is pre-existing and not introduced here.proxy_config.yamlpoints caching at a placeholderredis.example.com, so the proxy log shows a Redis connection error at boot. Unrelated to these changes.Pre-Submission checklist
@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewScreenshots / Proof of Fix
Live proxy on the picked code (per-worktree proxy at
localhost:4001), real Anthropic API call, no mocks:The proxy boots on the picked code (40 models registered, including the #32701 provider path and #32874's rewritten anthropic transformation), then serves a real completion end to end.
Targeted-test delta: the pre-pick baseline was clean (591 passed, 0 failures across the pre-existing touched test files); after the picks the full targeted set is green at 1163 passed, 0 failures. Each pick's own tests pass, so each delivers its claim on this line, and there are zero new failures versus the baseline.
Behavioral gauntlet (universal, standard depth): the two regression-relevant sub-claims held, symbol closure (every identifier the picks reference resolves) SURVIVED and no-existing-caller-broken SURVIVED. The run flagged that #32701's day-0 muse-spark tests passed only with the bundled cost map in place; the strongest dissent confirmed the provider code is correct and the gap was a missing test fixture. That fixture was added (see adaptation notes), after which every touched test file passes in isolation with no env var forced, a full per-file fragility sweep run to confirm nothing else was order-dependent.
Type
🐛 Bug Fix
Changes
Eleven cherry-picks onto
patch-1.92.0rc2in staging merge order, no version bump, three picks adapted as noted above, everything else verbatim.