fix(bedrock): flag mapped Claude 4.8+ entries with supports_mid_conversation_system - #32882
Merged
mateo-berri merged 1 commit intoJul 11, 2026
Conversation
…rsation_system Exact cost-map hits resolve before fallback-generalization rules, so the mapped Sonnet 5, Fable 5 and jp Opus 4.8 Bedrock entries bypassed the bedrock-anthropic-claude-mid-conversation-system rule and hoisted mid-conversation system messages, invalidating the prompt cache.
mateo-berri
added a commit
that referenced
this pull request
Jul 11, 2026
…rsation_system (#32882) Backport of #32882 to stable/1.91.x. Cherry-picked from litellm_internal_staging (pending merge there). Adapted for this line: only the four anthropic.claude-fable-5 cost map entries exist on stable/1.91.x (no sonnet-5 or jp opus-4-8 entries), and the fallback-generalizations invariant test is omitted because the feature is absent here.
5 tasks
Contributor
Greptile SummaryThis PR fixes a Bedrock prompt-cache invalidation bug where mid-conversation
Confidence Score: 5/5Safe to merge — changes are additive JSON flag additions to two cost map files plus expanded test coverage, with no logic changes to production code paths. All 11 entries the PR description identifies are present and consistent across both JSON files. The invariant test independently verifies the fix by re-running the same pattern match at test time, so future model additions cannot silently re-introduce the bug. The parametrized regression test adds representative coverage across all three newly-flagged model families. No files require special attention.
|
| Filename | Overview |
|---|---|
| model_prices_and_context_window.json | Adds supports_mid_conversation_system: true to 6 Bedrock Sonnet-5 entries and 4 Fable-5 entries — all previously matched by the fallback generalization rule but bypassed by exact-key lookup precedence. |
| litellm/model_prices_and_context_window_backup.json | Identical 11-entry flag additions as the primary cost map file, including jp.anthropic.claude-opus-4-8 which was the sole missing Opus-4-8 variant. Both files remain in sync. |
| tests/test_litellm/llms/bedrock/messages/invoke_transformations/test_anthropic_claude3_transformation.py | Converts the in-place regression test to a parametrized case covering representative models from each newly-flagged family, and adds a forward-looking invariant test that reads the backup JSON and asserts every cost-map entry matching the fallback rule's pattern carries the flag explicitly. |
Reviews (1): Last reviewed commit: "fix(bedrock): flag mapped Claude 4.8+ en..." | Re-trigger Greptile
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
shin-berri
self-requested a review
July 11, 2026 05:50
shin-berri
approved these changes
Jul 11, 2026
mateo-berri
enabled auto-merge (squash)
July 11, 2026 05:51
mateo-berri
deleted the
litellm_bedrock_mid_conversation_system_sonnet5_fable5
branch
July 11, 2026 05:51
Contributor
Merged
4 tasks
yuneng-berri
added a commit
that referenced
this pull request
Jul 11, 2026
…1.92.0 stable cut (#32959) * fix(utils): resolve bedrock regional inference profiles to regional pricing in get_model_info (LIT-4056) (#32389) * fix(utils): resolve bedrock regional inference profiles to regional pricing in get_model_info (LIT-4056) * test(register_model): use a triple provider prefix as the unresolvable-key fixture get_model_info now resolves bedrock/bedrock/... like a routing prefix, so the double-prefix fixture stopped exercising the register_model fallback path. Lock the new double-prefix resolution in as a model-info regression test (cherry picked from commit 734fd29) * fix(guardrails): walk Responses-API text taxonomy in shared content helpers (#32542) * fix(guardrails): walk Responses-API text taxonomy in shared content helpers Every guardrail sharing litellm/proxy/guardrails/_content_utils.py silently drops all text on the /v1/responses path. AIM turns it into a loud 422 ( {"error":"No messages in the request"}); every other guardrail (Lakera v2, Cato, Lasso, Repello, IBM, Azure Content Safety, enterprise secret detection) scans an empty payload and lets the request through unscanned. Three defects, all in _content_utils.py: 1. _iter_text_parts_in_content recognised only part.type == "text", but the Responses API uses input_text (request) and output_text (assistant). 2. _coerce_input_to_messages gated on "every item has a role key"; any Responses input list containing a function_call or function_call_output item failed the check and was wrapped as one opaque blob. 3. build_inspection_messages forwarded any role through, including a bare tool role missing tool_call_id, which validators like AIM's /fw/v1/analyze reject with a schema error. Fix walks the actual Responses item taxonomy (message, function_call, function_call_output, bare content parts and strings), recognises {text, input_text, output_text} everywhere, and coerces any role outside {system, user, assistant} to user in the outbound inspection payload. * style: ruff-format changed guardrail files * test(guardrails): cover function_call_output string form; drop em-dash in new docstring * fix(guardrails): map function_call_output straight to user role Avoids ever materialising a schema-invalid bare tool message. The downstream role-safety coercion in build_inspection_messages still guards genuinely caller-supplied non-standard roles (developer, function, custom values); add a regression test covering that path so the coercion has real coverage after this simplification. * test(guardrails): pin chat-completions tool-role coercion in build_inspection_messages * docs(test): soften AIM-specific claims in LIT-4294 test docstrings Ryan's review flagged that several test docstrings assert AIM's /fw/v1/analyze validates + rejects specific schema violations. That behavior is customer-reported in the LIT-4294 writeup, not directly verified by us. Rephrase to attribute the AIM 422 to the customer's writeup and describe the underlying constraint as the OpenAI chat schema; any downstream API that validates against that schema rejects the same shape. * refactor(guardrails): move unsupported-role coercion into AIM only The generic coercion in build_inspection_messages collapsed any role outside {system, user, assistant} to user for every caller of the helper. Combined with the pre-existing apply_redacted_messages_back write-back behavior in Lakera/AIM/Cato, that turned a loud OpenAI 400 on chat-completions tool-message masking into a silent semantic corruption of the outbound request (role tool with tool_call_id got rewritten to bare role user, dropping the assistant + tool_calls sibling). AIM specifically requires the coercion because its /fw/v1/analyze validates the payload against the OpenAI chat schema; other guardrails either do not validate roles or do their own reconstruction. Move the coercion to AimGuardrail._build_aim_inspection_messages so the shared helper keeps caller roles intact and no new cross-guardrail role corruption is introduced. The pre-existing apply_redacted_messages_back structural flatten remains as separate follow-up work. function_call_output items still synthesise role user in the shared helper because they have no natural role field, which is a different concern from coercing a caller-supplied role. * refactor(guardrails): preserve role fidelity in shared _content_utils Shared inspection helpers should extract text and preserve semantic role signals; role coercion for third-party schema safety stays inside the guardrail that needs it (AIM). Three shared-helper changes: - Bare content-part dicts (input_text/output_text) with an explicit role keep it; only role-less parts default to user. - Responses message items already had their role preserved; the behavior is now covered by an explicit test. - function_call_output items default to role tool (semantic equivalent of the chat-completions tool message shape) instead of role user, so Responses and chat completions produce symmetric inspection payloads. A caller-supplied role on the item is still preserved. AIM's schema-safe coercion in _build_aim_inspection_messages already handles the resulting role tool: it collapses to user before the POST to /fw/v1/analyze so AIM's OpenAI-schema validator does not reject the bare tool message (no tool_call_id can survive the flatten). Added a regression test in test_aim.py covering that path. (cherry picked from commit e84a19a) * feat: add Meta Model API provider and muse-spark-1.1 (day-0) (#32701) (cherry picked from commit d82645d) * fix(bedrock): keep mid-conversation system messages in place for Claude Invoke (#32578) Hoisting every role system entry into the top-level system field mutates the cache prefix whenever a client such as Claude Code appends a new mid-conversation system message, invalidating the prompt cache for the entire message history on Bedrock Invoke. Bedrock only rejects a system entry at messages.0, so hoist just the leading run and forward the rest in place (cherry picked from commit cc36d54) * feat(otel): emit the gen_ai.client.operation.exception event on failed LLM calls (#32655) * feat(otel): emit the gen_ai.client.operation.exception event on failed LLM calls The GenAI semantic conventions record failures of a GenAI client operation as a log-based event named gen_ai.client.operation.exception, carrying the exception.type / exception.message / exception.stacktrace trio at severity WARN and correlated to the failed span. OTel v2 never emitted it: a failed LLM call produced only the deprecated error.* span attributes, a generic exception span event without a stacktrace, and the stacktrace under the vendor key litellm.provider.error.stack_trace. Build the logs pipeline (LoggerProvider + console/OTLP log exporters mirroring the metrics plumbing) and record the event behind the enable_events flag, which until now was defined but consumed nowhere. An operator-configured LoggerProvider global is reused so the events ride their existing logs pipeline; an explicit NoOpLoggerProvider global is honored as an opt-out and builds no recorder at all. The existing span-side error surface (error.type, error.message, the exception span event, and the litellm.provider.error.* detail keys) is untouched for backwards compatibility. * fix(otel): always ride the semconv-required exception pair on the GenAI event Filtering the event attributes on truthiness conflated "absent" with "empty", so an empty exception.type or exception.message would have been dropped, leaving an event with neither semconv-required field. Build the attributes so the pair is unconditional and only the recommended stacktrace is omitted when the payload carries none. * docs(otel): document the events plumbing module in the package README * test(otel): cover the log exporter selection and logs endpoint normalization The new logs plumbing had no coverage for exporter-kind selection, the console fallback for an unrecognized kind, the /v1/logs signal-path rewriting that lets one OTEL_ENDPOINT serve every signal, or the simple-vs-batch processor split. (cherry picked from commit 99b4c5e) * fix(bedrock): gate in-place system role messages on model support for Claude Invoke (#32831) * fix(bedrock): gate in-place system role messages on model support for Claude Invoke * feat(bedrock): default unmapped Claude 4.8+ to in-place system role handling via fallback rule (cherry picked from commit 5e23a5a) * fix(anthropic): translate adaptive thinking/effort to pre-4.6 model support (#32867) * fix(anthropic): translate adaptive thinking/effort to pre-4.6 model support AnthropicMessagesConfig now reshapes the 4.6+ adaptive-thinking interface (thinking:{type:adaptive} + output_config:{effort:...}) to whatever the routed model supports. Thinking-capable non-adaptive models (e.g. Haiku 4.5, Sonnet 4.5) get the effort translated to a legacy thinking budget_tokens. Models with no reasoning support have thinking/effort dropped under drop_params. And because adaptive thinking carries no budget while the legacy form must satisfy Anthropic's max_tokens > budget_tokens rule, the translated budget is capped below max_tokens, dropping thinking when max_tokens can't fit the minimum budget. 4.6+ models pass through untouched. This matters because clients like Claude Code speak native Anthropic /v1/messages and send the adaptive interface unconditionally, regardless of the routed model. The native passthrough previously only capability-gated the OpenAI-style reasoning_effort alias and forwarded native output_config/adaptive thinking raw, so a pre-4.6 model rejected it with "This model does not support the effort parameter" and the request failed. Claude Code already gets drop_params auto-set, so its requests now succeed. * test(anthropic): gate undersized-max_tokens thinking drop on drop_params; add edge tests Addresses review feedback on the max_tokens-too-small branch. Previously a thinking-capable model whose max_tokens could not fit the minimum thinking budget had thinking silently dropped regardless of drop_params, while a residual output_config field in the same call still raised when drop_params was off. Gate both consistently on drop_params: raise a clear error (naming max_tokens for the undersized case) when drop_params is off, drop otherwise. Claude Code gets drop_params auto-set, so it still succeeds. Adds tests for the undersized-max_tokens raise, the residual output_config raise, and the no-adaptive-interface passthrough on a non-adaptive model. * fix(anthropic): make adaptive-effort translation silent to avoid breaking provider strip contracts The previous raise-when-not-drop_params behavior broke existing bedrock and vertex messages tests: those providers already silently strip unsupported output_config for pre-4.6 models (issue #22797) with no drop_params required, and the shared parent transform raising pre-empted that. It also conflicted with the goal of keeping requests working rather than failing them. Make the reshape silent: translate effort to legacy thinking for thinking-capable models, drop thinking for non-reasoning models, and remove only the consumed effort key from output_config, leaving any residual (e.g. format) for provider subclasses (bedrock/vertex) to handle. No raise, no drop_params gating. This also resolves the review note about inconsistent drop_params handling by making every path uniform. Updates the tests to assert the silent behavior and residual output_config preservation. * fix(anthropic): handle output_config-capable but non-adaptive models (Opus 4.5) Greptile caught a real bug: the early-return guard treated supports_output_config as equivalent to supporting adaptive thinking. Claude Opus 4.5 advertises supports_output_config (it accepts output_config.effort) but is not adaptive, so it rejects thinking:{type:adaptive} with "adaptive thinking is not supported on this model". The guard early-returned for Opus 4.5 and forwarded the adaptive thinking block raw, reproducing the exact failure the fix is meant to prevent. thinking:{type:adaptive} and output_config.effort are independent capabilities. Only early-return for adaptive-thinking models. For a model that supports output_config.effort but is not adaptive, keep the native effort and drop only the unsupported adaptive thinking block. Verified live against Opus 4.5: the Claude Code payload now returns 200 instead of 400. Adds regression tests for Opus 4.5 with and without adaptive thinking. * fix(anthropic): translate adaptive thinking for effort-capable pre-4.6 models Claude Opus 4.5 advertises supports_output_config but not adaptive thinking, so the early-return guard forwarded thinking.type=adaptive raw and Anthropic rejected it. The guard now only skips true adaptive models; effort-only requests on effort-capable models still pass through untouched. The _map_reasoning_effort call is wrapped to surface unrecognized effort values as a clean 400, matching _translate_reasoning_effort_to_anthropic * fix(anthropic): fall back to legacy thinking when effort level unsupported Opus 4.5 accepts output_config.effort but only low/medium/high; Claude Code defaults to xhigh on newer models, so preserving that level raw gets rejected by Anthropic. Gate the native-effort passthrough on _validate_effort_for_model and fall through to the budget translation for unsupported levels * fix(anthropic): keep effort-only requests untouched for provider normalization The xhigh fall-through consumed effort-only requests on effort-capable models, breaking bedrock invoke's own normalization which clamps xhigh to the model's ceiling after the base transform runs (test_bedrock_messages_normalizes_output_config_effort_for_opus). Restrict the fall-through to requests that carry adaptive thinking; effort-only requests pass through so provider subclasses keep owning level clamping --------- Co-authored-by: Abhimanyu Kapur <38531241+akapur99@users.noreply.github.com> (cherry picked from commit 3a62e54) * fix(bedrock): flag mapped Claude 4.8+ entries with supports_mid_conversation_system (#32882) Exact cost-map hits resolve before fallback-generalization rules, so the mapped Sonnet 5, Fable 5 and jp Opus 4.8 Bedrock entries bypassed the bedrock-anthropic-claude-mid-conversation-system rule and hoisted mid-conversation system messages, invalidating the prompt cache. (cherry picked from commit c15891f) * Merge pull request #32873 from BerriAI/litellm_fallback_rules_routing_split refactor(fallback-generalizations): split rules into routing and provider-neutral capability kinds (cherry picked from commit 45d3644) * Merge pull request #32874 from BerriAI/litellm_thread_provider_capability_probes fix(anthropic): thread real provider through capability probes instead of pinning anthropic (cherry picked from commit ead7ad3) * test: add /v1/messages to supported_endpoints schema enum (#32739) (cherry picked from commit bf02a4a) --------- Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: yucheng-berri <yucheng@berri.ai> Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: Abhimanyu Kapur <38531241+akapur99@users.noreply.github.com> Co-authored-by: tin-berri <tin@berri.ai>
This was referenced Jul 12, 2026
Merged
shin-berri
pushed a commit
that referenced
this pull request
Jul 15, 2026
* feat(router): add LLM-based classifier option to complexity router (#32169)
* feat(router): add LLM-based classifier option to complexity router
Adds classifier_type: "heuristic" | "llm" to complexity_router_config.
When set to "llm", the router calls a configured model (e.g. a small
model like haiku) via structured output to pick the complexity tier,
falling back to the existing regex/keyword scorer on any error, empty
response, or unparseable output.
* feat(ui): add classifier_type option to complexity router UI, fix edit flow
Adds an "Advanced: Classification Method" section to ComplexityRouterConfig
with a heuristic/LLM toggle, revealing a classifier model picker and timeout
when LLM is selected.
Also fixes the auto router edit modal, which never rendered the complexity
router UI at all (it only handled the semantic router), and the "Edit Auto
Router" button visibility check, which was gated on auto_router_config and
never matched complexity router deployments.
* fix(router): attribute classifier calls to caller, raise default timeout
Forwards the original request's litellm_metadata into the classifier's
acompletion call. Without it, the proxy's cost-tracking gate sees no
user_api_key/team_id/user_id and silently drops spend logging and budget
accounting for every classifier call, letting an authenticated user rack
up unaccounted provider spend via repeated requests.
Also raises the default classifier timeout from 400ms to 3000ms (400ms
undershoots real LLM latency and would silently degrade to the heuristic
scorer on most requests) and corrects the module/class docstrings, which
still claimed zero external API calls after the llm classifier path was
added.
* fix(ci): resolve ruff strict-budget and frontend-lint failures
- Use PEP 585 generics (dict/tuple/list) in the new aclassify/_classify_with_llm
signatures instead of typing.Dict/Tuple/List, and suppress BLE001 on the
intentionally broad except in aclassify's fallback path with a reason.
- Fix prettier formatting in ComplexityRouterConfig.tsx.
- Regenerate eslint-metrics.json (was stale after the classifier UI changes).
* fix(ci): regenerate stale eslint-metrics.json
* fix(router): strip parent budget reservation from classifier metadata
The classifier's internal acompletion call previously forwarded the
parent request's full litellm_metadata, including its budget
reservation (user_api_key_budget_reservation / user_api_key_auth).
That reservation belongs to the routed completion the classifier is
deciding on, not to the classifier call itself, so it's now stripped
while key/team attribution fields are still forwarded for spend
logging.
* fix(bedrock): add jp.anthropic.claude-opus-4-8 to model cost map (#32840)
* fix(bedrock): add jp.anthropic.claude-opus-4-8 to model cost map
* test: use apac regional profile for cost-map fallback test since jp now has an entry
* fix(responses): preserve reasoning_tokens through chat->responses usage translation (#32837)
* fix(responses): preserve reasoning_tokens through chat->responses usage translation
Remove the unconditional else-branch that wrote reasoning_tokens=0 whenever
completion_tokens_details.reasoning_tokens was None or absent. Also change
OutputTokensDetails.reasoning_tokens from int=0 to Optional[int]=None so that
re-instantiation without explicit reasoning_tokens no longer silently zeroes out
the field, and remove the same hardcoded zero from the mock_responses_api_response
initializer.
* test(responses): update assertions to match Optional[int] reasoning_tokens default
* fix(responses): preserve explicit reasoning_tokens=0 in usage translation
Align the reasoning_tokens guard with the is-not-None guards used for
text_tokens and image_tokens: a provider-reported zero passes through
while an absent value stays omitted.
---------
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
* fix(bedrock): gate in-place system role messages on model support for Claude Invoke (#32831)
* fix(bedrock): gate in-place system role messages on model support for Claude Invoke
* feat(bedrock): default unmapped Claude 4.8+ to in-place system role handling via fallback rule
* fix(responses-api): raise APIError on in-stream error events; widen ErrorEventError.param to accept dict (#32835)
* fix(responses-api): raise APIError on in-stream error events; widen ErrorEventError.param
- BaseResponsesAPIStreamingIterator._maybe_raise_for_error_event inspects each
chunk and raises litellm.APIError for type=error and type=response.failed events
so callers see an exception instead of a benign stream chunk
- rate_limit* codes map to 429; client error codes (invalid_request_error,
context_length_exceeded, etc.) map to 400; all other codes default to 500;
raw integer codes are never used as-is as HTTP status codes
- ErrorEventError.param widened from Optional[str] to Optional[Union[str, Dict]]
to prevent Pydantic ValidationError on dict-typed param payloads silently
dropping error events before any type inspection
* test(responses-api): add streaming iterator error event tests to CI-covered path
* test(responses-api): cover response.failed, dict-error, null-error, and sync iterator paths
* test(responses-api): set completion_start_time on mock logging objects for internal staging _process_chunk
* fix(responses-api): map insufficient_quota to 429, derive failed-response log status from error code, and record failed-stream usage for spend accounting
insufficient_quota moves out of the 400 bucket; OpenAI returns HTTP 429 for it and the non-streaming exception mapping treats 429 as RateLimitError, so the in-stream mapping now agrees
_handle_logging_failed_response previously hardcoded APIError(status_code=500), so a rate-limited response.failed was logged to integrations as 500 while the caller saw 429; it now shares the same error-code-to-status mapping via _error_event_fields and _status_code_for_error_code
usage carried on a response.failed event is now stashed as combined_usage_object with its computed cost on the logging object before failure handlers run, reusing the mid-stream-interruption spend recovery path (_failure_handler_helper_fn, proxy post_call_failure_hook, _ProxyDBLogger), so failed streams count their billed tokens instead of logging zero cost
dedupe: TestMaybeRaiseForErrorEvent in tests/llm_responses_api_testing duplicated tests/test_litellm/responses/test_streaming_iterator_error_events.py, which is the canonical mirrored location and CI-covered via test-unit-responses-caching-types; the duplicate class is removed
* fix(responses-api): wrap retriable in-stream errors in MidStreamFallbackError and map error type field to status
Mirror chat streaming semantics from _handle_stream_fallback_error: 429 and
5xx in-stream error events now raise MidStreamFallbackError carrying the
mapped APIError so the router's FallbackResponsesStreamWrapper triggers
mid-stream fallback and cooldown; non-retriable 4xx still raise APIError
directly. Status mapping now reads both the OpenAI error type and code
fields, so type-classified client errors (e.g. invalid_request_error with
code invalid_prompt) map to 400 instead of falling through to 500.
* fix(responses-api): accumulate streamed output text so mid-stream fallback continues instead of restarting
MidStreamFallbackError was always raised with generated_content="", so the
router's stream_with_fallbacks treated every mid-stream error as pre-first-chunk
and retried with the original input, streaming duplicated content to clients
that had already received partial output. The iterators now accumulate
response.output_text.delta text (mirroring chat's response_uptil_now) and pass
it as generated_content, letting the router build a continuation input via
_build_responses_continuation_input.
* test(responses-api): pin in-stream token limit error to raised APIError
---------
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
* fix(prometheus): skip budget metric DB lookups when gauges are NoOpMetric (#32834)
adds a top-level guard in _increment_remaining_budget_metrics that returns early
when all four budget gauges are NoOpMetric (excluded from prometheus_metrics_config),
and per-entity guards in each _set_*_budget_metrics_after_api_request helper for
partial disabling. eliminates four async DB/cache round-trips per successful LLM
request when budget metrics are disabled.
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
* fix(anthropic): strip @version suffix in _model_map_lookup_candidates (#32833)
vertex_ai/claude-opus-4-8@default (and sibling @default models) were
misclassified as non-adaptive because _model_map_lookup_candidates only
stripped provider prefixes but never the @<suffix> portion. The lookup
produced candidates like ["vertex_ai/claude-opus-4-8@default",
"claude-opus-4-8@default"], neither of which exists in model_cost, so
_is_adaptive_thinking_model returned False. LiteLLM then sent
thinking.type=enabled to a @default Vertex AI endpoint that requires
thinking.type=adaptive, resulting in a 400.
_strip_version_suffix now removes @<suffix> from each candidate,
adding the bare model name (e.g. "claude-opus-4-8") to the lookup
chain. Also adds supports_adaptive_thinking: true to the three
@default model_cost entries that were missing it as belt-and-suspenders.
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
* fix(datadog): split log batches proactively under intake payload limits (#32860)
* fix(datadog): split log batches proactively under intake payload limits
* fix(datadog): size intake chunks with exact wire serialization
* fix(anthropic): translate adaptive thinking/effort to pre-4.6 model support (#32867)
* fix(anthropic): translate adaptive thinking/effort to pre-4.6 model support
AnthropicMessagesConfig now reshapes the 4.6+ adaptive-thinking interface
(thinking:{type:adaptive} + output_config:{effort:...}) to whatever the routed
model supports. Thinking-capable non-adaptive models (e.g. Haiku 4.5, Sonnet 4.5)
get the effort translated to a legacy thinking budget_tokens. Models with no
reasoning support have thinking/effort dropped under drop_params. And because
adaptive thinking carries no budget while the legacy form must satisfy Anthropic's
max_tokens > budget_tokens rule, the translated budget is capped below max_tokens,
dropping thinking when max_tokens can't fit the minimum budget. 4.6+ models pass
through untouched.
This matters because clients like Claude Code speak native Anthropic /v1/messages
and send the adaptive interface unconditionally, regardless of the routed model.
The native passthrough previously only capability-gated the OpenAI-style
reasoning_effort alias and forwarded native output_config/adaptive thinking raw, so
a pre-4.6 model rejected it with "This model does not support the effort parameter"
and the request failed. Claude Code already gets drop_params auto-set, so its
requests now succeed.
* test(anthropic): gate undersized-max_tokens thinking drop on drop_params; add edge tests
Addresses review feedback on the max_tokens-too-small branch. Previously a
thinking-capable model whose max_tokens could not fit the minimum thinking budget
had thinking silently dropped regardless of drop_params, while a residual
output_config field in the same call still raised when drop_params was off. Gate
both consistently on drop_params: raise a clear error (naming max_tokens for the
undersized case) when drop_params is off, drop otherwise. Claude Code gets
drop_params auto-set, so it still succeeds.
Adds tests for the undersized-max_tokens raise, the residual output_config raise,
and the no-adaptive-interface passthrough on a non-adaptive model.
* fix(anthropic): make adaptive-effort translation silent to avoid breaking provider strip contracts
The previous raise-when-not-drop_params behavior broke existing bedrock and vertex
messages tests: those providers already silently strip unsupported output_config
for pre-4.6 models (issue #22797) with no drop_params required, and the shared
parent transform raising pre-empted that. It also conflicted with the goal of
keeping requests working rather than failing them.
Make the reshape silent: translate effort to legacy thinking for thinking-capable
models, drop thinking for non-reasoning models, and remove only the consumed effort
key from output_config, leaving any residual (e.g. format) for provider subclasses
(bedrock/vertex) to handle. No raise, no drop_params gating. This also resolves the
review note about inconsistent drop_params handling by making every path uniform.
Updates the tests to assert the silent behavior and residual output_config
preservation.
* fix(anthropic): handle output_config-capable but non-adaptive models (Opus 4.5)
Greptile caught a real bug: the early-return guard treated supports_output_config
as equivalent to supporting adaptive thinking. Claude Opus 4.5 advertises
supports_output_config (it accepts output_config.effort) but is not adaptive, so it
rejects thinking:{type:adaptive} with "adaptive thinking is not supported on this
model". The guard early-returned for Opus 4.5 and forwarded the adaptive thinking
block raw, reproducing the exact failure the fix is meant to prevent.
thinking:{type:adaptive} and output_config.effort are independent capabilities.
Only early-return for adaptive-thinking models. For a model that supports
output_config.effort but is not adaptive, keep the native effort and drop only the
unsupported adaptive thinking block. Verified live against Opus 4.5: the Claude Code
payload now returns 200 instead of 400.
Adds regression tests for Opus 4.5 with and without adaptive thinking.
* fix(anthropic): translate adaptive thinking for effort-capable pre-4.6 models
Claude Opus 4.5 advertises supports_output_config but not adaptive thinking,
so the early-return guard forwarded thinking.type=adaptive raw and Anthropic
rejected it. The guard now only skips true adaptive models; effort-only
requests on effort-capable models still pass through untouched. The
_map_reasoning_effort call is wrapped to surface unrecognized effort values
as a clean 400, matching _translate_reasoning_effort_to_anthropic
* fix(anthropic): fall back to legacy thinking when effort level unsupported
Opus 4.5 accepts output_config.effort but only low/medium/high; Claude Code
defaults to xhigh on newer models, so preserving that level raw gets rejected
by Anthropic. Gate the native-effort passthrough on _validate_effort_for_model
and fall through to the budget translation for unsupported levels
* fix(anthropic): keep effort-only requests untouched for provider normalization
The xhigh fall-through consumed effort-only requests on effort-capable
models, breaking bedrock invoke's own normalization which clamps xhigh to
the model's ceiling after the base transform runs
(test_bedrock_messages_normalizes_output_config_effort_for_opus). Restrict
the fall-through to requests that carry adaptive thinking; effort-only
requests pass through so provider subclasses keep owning level clamping
---------
Co-authored-by: Abhimanyu Kapur <38531241+akapur99@users.noreply.github.com>
* test(models): assert capability fields on regional Azure gpt-5.6 entries (#32875)
* feat(auto_router): keyword tier overrides and semantic keyword matching for the complexity router
Add deterministic keyword-to-tier overrides and optional embedding-based
(semantic) keyword matching to the complexity router, and surface both in the
Add Auto Router UI behind a Router Type selector: "Auto-Router v2 [Recommended]"
(complexity tiers + keyword overrides + semantic matching, the default) and
"Semantic Router [to be deprecated]" (the existing utterance-based router,
unchanged). Keyword-to-tier overrides resolve to the highest tier matched
rather than the first keyword matched, so match order no longer affects the
routing decision.
Backend:
- config: KeywordTierRule model plus keyword_tier_rules, semantic_keyword_matching,
embedding_model, and match_threshold on ComplexityRouterConfig, with a validator
requiring an embedding model and rules when semantic matching is on
- complexity_router: evaluate keyword rules before scoring; lexical matches escalate
to the most-severe matched tier (order-independent), and semantic mode reuses
LiteLLMRouterEncoder + SemanticRouter to match paraphrases by cosine similarity,
falling back to the scorer when nothing matches
- model management: clear complexity_routers on cache reload so config edits take effect
Frontend:
- Add Auto Router tab restores the Router Type radio (Auto-Router v2 recommended
by default, Semantic Router still available) and sends keyword_tier_rules plus
the semantic settings on the recommended path, instead of flattening keywords
into custom_technical_keywords
- client-side guard blocks submit when semantic matching is enabled without an
embedding model or without any keyword tier rules, mirroring the backend validator
- moved the "How Classification Works" explainer below Custom Technical Keywords
and above Keyword Tier Overrides
- remove the Test Connection action from the recommended flow, which can't build a
valid pre-save payload for a router (leaves a TODO for a JSON preview / config
test follow-up)
Tests cover lexical escalation, semantic matching via the real library with injected
embeddings, the semantic config guard, config validation, the reload-clear
regression, and the frontend payload builder
* fix(bedrock): flag mapped Claude 4.8+ entries with supports_mid_conversation_system (#32882)
Exact cost-map hits resolve before fallback-generalization rules, so the
mapped Sonnet 5, Fable 5 and jp Opus 4.8 Bedrock entries bypassed the
bedrock-anthropic-claude-mid-conversation-system rule and hoisted
mid-conversation system messages, invalidating the prompt cache.
* feat(team): let org admins reach PATCH /team/{team_id} like POST /team/update
Wire the coarse route gate so PATCH /team/{team_id} is reachable by exactly the roles that can call POST /team/update: proxy admins, org admins of the team's own organization, and JWT admins. Regular internal users and view-only proxy admins stay blocked, matching the existing endpoint
Because the team id lives in the path rather than the body, the org-context resolver now also reads it from path_team_id for the bare /team/{team_id} route, so an org admin's organization is resolved and injected the same way it already is for POST /team/update. /team/{team_id} is added to management_routes rather than the role-agnostic self_managed_routes; the latter would have opened POST /team/new to any authenticated user through the shared /team/{team_id} path pattern
* feat(ui): root the gateway breadcrumb in the AI Gateway selector
The AI Gateway select (ViewSwitcher) now sits at the root of the DashboardHeader breadcrumb instead of on the right, so the top bar reads [AI Gateway select] > Page to match the redesign. It keeps the same dropdown, including the Chat / Chat UI options.
When no plugins are registered and Chat UI is disabled there is nothing to switch between, so the breadcrumb falls back to the static section crumb rather than rendering a dangling leading separator
* fix(complexity_router): build semantic route index once under concurrent cold-start
Concurrent first requests each hit asyncio.to_thread to build the SemanticRouter
index, firing duplicate embedding calls for the static route utterances. Guard the
lazy build with a per-router asyncio.Lock (double-checked) so the index is
constructed exactly once regardless of how many callers race in cold.
Adds a regression test asserting ten simultaneous cold-start requests build the
index the same number of times as a single request, and reworks the fake embedding
router to count builds by how often a route utterance is embedded (robust to which
embedding path the library uses) while still recording sync-call thread ids for the
off-event-loop assertion.
* feat(ui): always show the gateway selector with a discoverable Chat entry
The AI Gateway selector now always renders at the breadcrumb root, even with no plugins and Chat UI disabled, so the Chat feature stays discoverable. The Chat entry is always listed: clickable when enabled, and disabled with an "Admins can enable in Settings" hint when it is off.
Since the selector is now unconditional, the useViewSwitcherVisible hook and the section-crumb fallback added in the previous commit are removed
* fix(proxy): guard delete_model router eviction on auto_router/ prefix
delete_model popped the auto_routers/complexity_routers registries by the deleted
deployment's model_name without checking it was actually an auto_router/* deployment.
Deleting a regular DB model that merely shares a name with a config-defined router
therefore evicted that router, which add_deployment never restores, leaving it
unroutable until a proxy restart. This is the same cross-tenant DoS clear_cache was
hardened against; mirror its auto_router/ prefix guard here.
Extracts _deployment_name_and_model to read model_name and litellm_params.model from
the deployment (delete_deployment returns the raw model_list dict at runtime despite
its Deployment annotation), and adds a regression test asserting a same-named config
router survives deletion of an unrelated regular model.
* refactor(fallback-generalizations): split rules into routing and provider-neutral capability kinds
* feat(fallback-generalizations): widen adaptive-thinking gate to any claude family at major 5+
* fix(fallback-generalizations): tolerate legacy remote rule schema and keep register_model cache-pricing inheritance
* fix(fallback-generalizations): let exact cost-map entries beat capability rules across lookup-candidate ladders
* fix(team): bound json merge patch recursion depth
apply_json_merge_patch recurses into nested objects, which the repo's recursive_detector code-quality check flags because unbounded recursion over caller-supplied JSON has caused CPU/stack issues before. Cap the recursion at a depth far above any realistic team-metadata shape and reject deeper patches with a ValueError so a pathologically nested body fails closed instead of overflowing the stack, then register the function in the detector's ignore list alongside the other depth-bounded JSON walkers
* fix(auth): tolerate request objects without path_params in common_checks
The PATCH /team/{team_id} org-context wiring reads request.path_params to
resolve the team id from the path. A real Starlette Request always exposes
path_params, but common_checks is exercised with lightweight request doubles
that don't, which raised AttributeError. Read it defensively so a missing or
null path_params falls back to no path team id, matching the "not a bare team
route" outcome; real requests are unaffected
* fix(mcp): re-register DCR client when proxy origin no longer matches its registered redirect_uri
A dynamically registered (RFC 7591) OAuth client persisted onto the MCP server row is bound to the redirect_uri it was first registered with, but that binding was never recorded. After the proxy's public origin changed, every authorize paired the reused client with the new callback and the IdP rejected it permanently.
The DCR persist now records redirect_uris alongside the client identity. The admin register path treats a positive mismatch between the recording and the current callback as stale and re-registers a replacement client; rows without a recording (pre-existing installs and admin-configured clients) are grandfathered so upgrades never re-mint client_ids or orphan refresh tokens. The persist also writes client_secret and token_endpoint_auth_method explicitly as None when absent so the credential blob merge cannot pair a re-registered public client with the previous client's secret. Public register routes and non-admin callers keep existing behavior.
Closes #32473
* fix(mcp): emit one operator warning per DCR re-registration event
The stale-redirect path logged three warnings for a single re-registration: the staleness probe plus the reuse skip in both register_client_with_server and the persist race guard. The reuse-skip message is a mechanical consequence of the probe's decision, so it now logs at debug; the actionable warning that names both bindings and the re-authentication impact is emitted once by _persisted_dcr_redirect_uri_is_stale
* ci: gate tests/e2e on zero basedpyright errors in pre-commit and lint CI
* refactor(ui): use TanStack Pacer debounce for the team keys search
Replace lodash/debounce in TeamVirtualKeysTable with useDebouncedValue from
@tanstack/react-pacer, matching the sibling VirtualKeysTable and
PaginatedKeyAliasSelect which already debounce their key-alias search that way.
Pacer is already a dependency, so this drops the odd-one-out lodash usage and
keeps the search-debounce pattern consistent across the key tables.
* fix(fallback-generalizations): cover bare Claude majors in baseline and routing, require claude- prefix in adaptive gate
* fix(mcp): strip scheme default port from get_request_base_url netloc
* feat(ui): typed openapi-fetch foundation (fetchClient) + first typed caller (useCustomers) (#29884)
* feat(ui): add the typed openapi-fetch client (fetchClient) as the dashboard fetch foundation
Introduces fetchClient (openapi-fetch) bound to schema.d.ts, used inside ordinary TanStack Query hooks so path/query/body types come from the proxy's OpenAPI spec. A small runtime registry feeds the client the base URL and auth header name (registered by networking) and the session token (published by AuthContext), so call sites carry no token plumbing; auth-header injection and ApiError mapping live in openapi-fetch middleware reusing deriveErrorMessage/ApiError from client.ts, and non-2xx maps to a thrown ApiError so query functions just read .data.
The base URL default resolves from NEXT_PUBLIC_BASE_URL so a request still targets the right origin if it fires before networking registers its getter. AuthContext clears accessToken alongside the token on logout so no query fires unauthenticated after the session ends.
Foundation only; callers migrate one at a time, each fully typed, in follow-up changes.
* feat(ui): migrate useCustomers to the typed fetchClient
Converts useCustomers from allEndUsersCall to fetchClient.GET("/customer/list"); the response is typed as LiteLLM_EndUserTable[] from the schema, so the hand-written Customer/CustomersResponse types are deleted. They were also inaccurate (allowed_model_region was string but is "eu"|"us", and a budget_id the table has no field for). No cast; the schema type flows to the one consumer. First caller on the new pattern.
* fix(ui): route typed-client errors through the session-expiry handler
The typed fetchClient middleware threw ApiError without invoking the
handleError side effect that the legacy createApiClient wires via
onError, so a migrated caller hitting an expired key no longer triggered
the auto-logout. Add an error-handler seam to runtime.ts, register
handleError from networking.tsx alongside the base-url/header getters,
and call it in the middleware before throwing so both clients behave the
same. Regression test asserts the handler fires with the derived message
on non-2xx and stays silent on success
* fix(ui): point the customers EndUser type at CustomerResponse
The /customer/list response model was renamed to CustomerResponse on
staging; the merged branch still aliased EndUser to LiteLLM_EndUserTable,
so the exported type and its test mock had drifted from what the schema
actually returns. CustomerResponse is also the accurate shape (it types
allowed_model_region as 'eu' | 'us' and carries budget_id)
* chore(ui): refresh eslint-metrics baseline after staging merge
The recorded baseline predated the litellm_internal_staging merge, so its
no-explicit-any and no-large-inline-object-arg counts were higher than the
merged tree actually has. Regenerate via npm run lint:metrics so the gate
reflects current reality
* refactor(ui): source the typed client token from the session cookie, not AuthContext
The typed client read its bearer from a runtime value that AuthContext pushed
via setAuthToken, but migrated hooks gate enabled on useAuthorized, which
decodes the cookie directly. Two independent derivations of the same cookie with
different timing: on first load the query fires (useAuthorized sees the token)
before AuthContext's async effect publishes it, so the first request goes out
unauthenticated and only succeeds on a React Query retry.
Make the token a registered getter like the base-url and header-name getters,
reading the same cookie useAuthorized decodes, so the client's token and the
gate can't diverge. Revert the AuthContext changes entirely; nothing is pushed
from React state anymore.
* test(e2e): cover Langfuse logging.yaml P0 logs_spend cells (#32857)
* test(e2e): cover Langfuse logging.yaml P0 logs_spend cells
Team, user/key, and org-scoped dynamic Langfuse callbacks drive real chat
traffic and assert calculatedTotalCost matches StandardLogging response_cost
and proxy spend. Also assert tool calls and applied guardrails land on the
trace. Missing env or proxy is a hard failure, never a skip
* test(e2e): use langfuse_otel callback for Langfuse spend coverage
Team and key dynamic logging attach callback_name=langfuse_otel (OTLP to
Langfuse) instead of the classic langfuse SDK. Match generations named
litellm_request by prompt marker and user_api_key_alias
* test(e2e): require Langfuse spend assert; drop AGENTS.md
Guardrail path no longer soft-gates logs_spend. Non-stream responses must
return positive x-litellm-response-cost; remove tests/e2e/AGENTS.md
* test(e2e): fail when Langfuse spend is missing on guardrail path
Always run logs_spend assertions for tool_permission; require positive
x-litellm-response-cost on non-stream and positive /spend/logs spend
* test(e2e): do not fall back to unmatched spend log rows
poll_proxy_spend_for_key returns None when response_id or positive-spend
filters match nothing, instead of silently using rows[0]
* fix(complexity_router): use max aggregation for semantic keyword route scoring
SemanticRouter defaults to mean aggregation across a route's utterances. Since
each tier's route holds one utterance per configured keyword, a real semantic
match on one keyword was averaged together with the tier's other, unrelated
keywords and dragged below match_threshold — e.g. a MEDIUM tier with keywords
[beep, boop, new york] never fired for a genuine "new york" paraphrase, because
mean(sim_to_beep, sim_to_boop, sim_to_new_york) landed well under the threshold
even though sim_to_new_york alone cleared it. Pass aggregation="max" so a tier
matches if the query is close enough to any one of its keywords, not the
average of all of them.
Verified against live Voyage embeddings: raw cosine similarity for "new york"
vs a paraphrase was 0.54 (above a 0.5 threshold), but the route scored 0.28
under mean aggregation and never matched; max aggregation fixes it.
Adds a regression test with a tier holding one matching and two unrelated
keywords, asserting the tier still fires; fails without aggregation="max".
* refactor(auth): resolve PATCH team org-context from the route template
Replace the request.path_params read (and its defensive getattr guard) with
the route template. A real Starlette request always exposes path_params, but
common_checks runs on lightweight request doubles that don't, so reading it
directly forced a getattr workaround that only existed to tolerate those
doubles.
Instead, match the route template (/team/{team_id}) to identify the RESTful
update route and take the team id from the last path segment. This drops the
path_params dependency entirely, and because the template distinguishes the
PATCH route from its single-segment siblings (/team/new, /team/list, ...), it
also avoids a spurious team lookup those routes would otherwise trigger if we
matched the resolved path shape alone.
* chore(ui): remove eslint-metrics.json lint-count snapshot
The eslint-metrics.json snapshot duplicated the violation counts already
enforced by eslint-budgets.json. Keeping it current added a CI drift check,
a pre-commit regenerate-and-flag step, and a standalone npm run lint:metrics
script, none of which caught anything the budget gate did not, yet all of
which failed noisily whenever the snapshot went stale. This drops the file
and that machinery while leaving eslint-budgets.json as the actual ratchet
gate
* fix(complexity_router): preserve user_api_key_auth in sub-call metadata
Removing user_api_key_auth entirely from classifier/embedding sub-call
metadata (as _BUDGET_RESERVATION_METADATA_KEYS previously did) prevented
_filter_deployments_by_model_access_groups from scoping those sub-calls to
the caller's authorized access groups. An access-group-scoped caller could
therefore reach embedding/classifier deployments outside their group.
Only strip user_api_key_budget_reservation, which is the actual budget-
reservation state that must not reach sub-calls. user_api_key_auth is now
kept so access-group filtering works correctly for both the embedding path
and the LLM classifier path.
* test(e2e): drop vertex from pipecat tool smoke (#32925)
Exclude vertex_ai from pipecat tool smoke; raw-ws tool_call_round_trip
remains the Vertex source of truth. Also remove the Playwright key models
dropdown suite so stage is not blocked by that UI harness
* fix(complexity_router): sanitize budget reservation inside forwarded user_api_key_auth
* fix(complexity_router): review hardening - blank keywords, router registry eviction, edit-modal controls
- config: KeywordTierRule now strips and drops blank/whitespace keywords (a stray
"" makes _keyword_matches match every prompt, silently forcing that tier for all
traffic); still requires at least one real keyword to remain
- frontend build_complexity_router_config: trim keywords and drop rules left empty so
an unfilled "Add keyword rule" row no longer ships a rule the backend rejects with a
400 in the heuristic (non-semantic) flow, where the client-side semantic guard doesn't run
- proxy clear_cache / delete_model: the auto_router/ prefix also covers quality_router/
and adaptive_router/, so pop the model_name from all four router registries (no-op
where absent) instead of only auto/complexity; otherwise a DB quality_router's stale
entry made reload raise "already exists" and abort, and adaptive left a leak
- frontend ComplexityRouterConfig: only render the Keyword Tier Overrides and Semantic
keyword matching sections when their change handlers are provided, so the edit-auto-
router modal (which omits them) no longer shows interactive-but-dead controls
* fix(anthropic): thread real provider through capability probes instead of pinning anthropic
* docs(anthropic): note the two provider params' roles in _map_reasoning_effort
* fix(anthropic): override custom_llm_provider in provider config subclasses so capability probes use the right namespace
* feat(mcp): admit dcr_bridge oauth_delegate clients via a single envelope bearer
* fix(mcp): admit dcr_bridge envelopes under the live key, not a frozen identity
The bridge envelope sealed only user_id/server_id, and admission fabricated a
UserAPIKeyAuth(user_id=...) with no object_permission, team_id, org_id, or key
identity. Downstream MCP permission checks read the missing restrictions as
unrestricted, so a caller holding a valid envelope for a restricted key could
reach tools and servers that key was never granted, and a revoked key kept
working until the envelope expired.
Bind the hashed authorizing key into the envelope identity and reload the live
UserAPIKeyAuth by it at admission via get_key_object, failing closed with a 401
when the key is missing, blocked, or expired. Authorization is resolved fresh
per request instead of frozen at mint time, so current key/team/org and tool
restrictions plus revocation are enforced.
* fix(mcp): enforce team block and alias-priority token injection on bridge admission
Two follow-ups on the envelope admission arm flagged in review.
Team revocation bypass: _reload_admitted_key checked only the key's own
blocked/expires, so blocking a key's team left every envelope minted under it
live until expiry. Reload the team and reject a blocked team, mirroring
common_checks, so a team block revokes its envelopes immediately.
Caller-overridable upstream token: egress resolves the per-server auth header
alias-first, but injection keyed under server_name, so for a server with a
distinct alias a caller-forwarded x-mcp-{alias}-authorization sat at the
higher-priority slot and paired the admitted identity with an attacker's
upstream credential. Inject under alias-first so the sealed token owns the slot
egress resolves.
* fix(mcp): route bridge admission through the centralized policy gate and mirror the SCIM owner check
* fix(mcp): import assert_never from typing_extensions for Python 3.10
* fix(mcp): surface real status from bridge admission policy gate instead of flattening to 401
Over-budget rendered 401 (should be 429), model-access and other typed
failures collapsed to 401, and a transient DB outage was masked as an auth
error. Mirror UserAPIKeyAuthExceptionHandler: budget maps to 429, a sub-check's
own HTTPException/ProxyException keeps its status, a DB outage is a retryable
503, and only a genuinely unresolvable failure stays the fail-closed 401.
* fix(rate-limit-v3): populate x-ratelimit-* remaining/limit values in standard_logging_object for streaming (LIT-4333) (#32711)
Streaming requests return from common_request_processing before
async_post_call_success_hook runs, so response._hidden_params.additional_headers
never gets the v3 x-ratelimit-{descriptor_key}-{remaining|limit}-{rate_limit_type}
entries. Prometheus / logging callbacks that read those values from
standard_logging_object.hidden_params.additional_headers then see nothing;
combined with the pre-existing gap that Prometheus reads from that same slot
(LIT-2577 / PR #28816), per-key remaining RPM/TPM cannot be monitored for
streaming traffic at all.
Fix in three parts:
- Stash the pre-call RateLimitResponse in the metadata channels the async
success-logging callback inherits, alongside the existing top-level entry
the non-streaming path reads.
- Add async_logging_hook to the v3 handler. It fires in a distinct earlier
loop inside async_success_handler (all callbacks' async_logging_hook
complete before any async_log_success_event starts), so mirroring the
pre-call snapshot into standard_logging_object.hidden_params.additional_headers
and response._hidden_params.additional_headers here guarantees every
downstream success callback sees the values regardless of registration
order. Non-streaming keeps the existing async_post_call_success_hook write
and this hook re-populates the same values idempotently.
- Extract the shared `_merge_ratelimit_statuses_into_additional_headers`
helper the non-streaming path already had inlined so both callsites emit
the identical key shape.
* fix(proxy): skip None model_name in clear_cache router eviction set
* fix(mcp): map a DB outage during bridge key reload to a retryable 503
get_key_object's raw transport error propagated uncaught out of
_reload_admitted_key as an opaque 500; classify it via the shared
_raise_503_if_db_unavailable helper (also used by the live-policy gate) so a
database outage is a retryable 503, while a key-not-found ProxyException stays
the fail-closed 401.
* test: remove live OpenAI fine-tuning job-creation test blocked by platform wind-down (#32933)
OpenAI is winding down self-serve fine-tuning and the org can no longer
create fine-tuning jobs (403 training_not_available; the CI key surfaces
it as a 500 server_error), so test_create_fine_tune_jobs_async fails on
every batches_testing run since 2026-07-11 and reruns never clear it.
The request contract stays covered by the mocked create/list/cancel/
retrieve tests in the same file, and the deleted test's unique
standard_logging_object assertions now run inside
test_mock_openai_create_fine_tune_job.
* refactor(anthropic): consolidate the provider fallback into a _resolved_provider property
* feat: add lite auth print-token for Claude Code apiKeyHelper support (#32846)
* feat: add silent CLI token refresh for apiKeyHelper support
lite auth print-token prints a valid proxy credential for use as Claude
Code's apiKeyHelper, transparently refreshing it first if the cached JWT
is stale. This unblocks MDM-managed apiKeyHelper deployments (managed
via `lite auth print-token`) that need silent mid-session credential
rotation without restarting the client.
Refresh capability is backed by a virtual key minted with an empty model
list and cli_refresh metadata, kept strictly separate from the actual
(short-lived, real-model-scoped) call credential -- so a leak of the
credential that flows through every LLM request and subprocess env var
can't also self-renew. The refresh flow is single-use: /sso/cli/refresh
mints a fresh JWT + refresh token pair and blocks the presented refresh
token immediately, so a replay can't mint a second pair from it.
Server: /sso/cli/refresh (rotate) and /sso/cli/logout (revoke) endpoints.
lite login now also stores a refresh token; lite logout revokes it
server-side instead of only clearing the local file.
* fix: allow non-admin users to hit CLI refresh routes; resolve apiKeyHelper base_url from token.json
Found via a live end-to-end test against a real proxy + real Claude Code
session: /sso/cli/refresh and /sso/cli/logout were unreachable for any
non-proxy-admin caller, since Depends(user_api_key_auth) pulls in a
route-RBAC gate that 403s any route not on an explicit allowlist. That
made the feature unusable for actual end users, who authenticate as
internal_user. Add both routes to internal_user_routes; the handlers
already do their own fine-grained check (metadata.cli_refresh) same as
/key/block does today.
Also: `lite auth print-token` required an explicit --base-url/
LITELLM_PROXY_URL matching the stored token's origin, defaulting to
localhost:4000 otherwise. But apiKeyHelper is configured bare (no
flags), so this always mismatched a real deployment. Track whether
--base-url was explicitly passed (via click's ParameterSource) and, if
not, resolve the server from token.json directly instead of the CLI
default.
* test: mock refresh-token minting in test_cli_poll_key_tolerates_missing_user_row
Landed on litellm_internal_staging after this branch's refresh-key minting
change; needs the same mock as the other cli_poll_key tests since minting
now runs unconditionally whenever a JWT is generated.
* fix(ci): update test_cli_auth.py for refresh_token contract, regenerate schema.d.ts
_poll_for_authentication now always includes "refresh_token" in its
returned dict, and _handle_team_selection_during_polling returns a dict
instead of a bare JWT string -- test_cli_auth.py predates this branch's
refresh-token work and still asserted the old shapes.
schema.d.ts regenerated via `npm run gen:api` to pick up the new
/sso/cli/refresh and /sso/cli/logout routes (plus unrelated drift from
other PRs merged since it was last generated).
* fix(ci): apply CI's own schema.d.ts diff (enterprise routes I can't generate locally)
Local `npm run gen:api` only sees OSS routes -- this machine's
litellm_enterprise editable install points at a now-deleted temp
directory, so it silently drops enterprise-only routes from the spec.
Applied the exact diff CI's own generation produced instead of
re-running the generator locally.
* fix: close refresh-token race, fail closed on DB down, fix logout base_url
Addresses Greptile review findings on the CLI refresh-token PR:
- cli_refresh_token minted a new JWT + refresh token BEFORE blocking the
presented one. Two concurrent requests bearing the same refresh token
could both pass auth and both mint fresh pairs, yielding four live
credentials from one consumed token. Now the presented token is
consumed atomically first via update_many (only succeeding if it flips
blocked from False/None to True); the loser gets count=0 and is
rejected before anything is minted.
- When prisma_client is None, refresh silently returned a new JWT
without ever being able to mark the presented token consumed, leaving
it valid indefinitely. Now fails closed with a 500 instead.
- `lite logout` sent its revocation POST to ctx.obj["base_url"], which
defaults to localhost:4000 when --base-url isn't passed -- the same
bug print_token had before the base_url_explicit fix, just missed
here. Now resolves the same way: trust the stored token's origin
unless the caller explicitly overrode --base-url.
* fix(ci): satisfy ruff format and narrow token_data type in logout
* fix(security): never trust refresh-token metadata for authorization
Addresses a real privilege-escalation path Veria flagged: cli_refresh_token
read team_id, team_alias, and max_budget straight off the presented
token's own metadata and used them to authorize the new JWT. Since any
authenticated user can self-mint a virtual key with arbitrary metadata
via the ordinary /key/generate endpoint, a self-forged key with
{"cli_refresh": true, "team_id": "<any-team>", "max_budget": 999999999}
would sail through _require_cli_refresh_token's only check
(metadata.cli_refresh == True) and get a JWT scoped to a team the
caller never belonged to, with a budget it never had -- full
cross-team / budget bypass, and a removed team member could keep
refreshing team-scoped sessions indefinitely.
Metadata's team_id is now treated as an untrusted UX hint only: honored
solely if the CALLER (identified by the authenticated key's own
user_id, not client input) is a current member per a fresh
get_user_object lookup. team_alias and max_budget are never read back
from metadata at all -- team_alias comes from a live get_team_object
lookup and max_budget is recomputed with the exact same capping logic
the initial SSO login poll uses. _mint_cli_refresh_token no longer
accepts or stores team_alias/max_budget, only the team_id hint.
Added regression tests proving: a forged/stale team_id is dropped
(falls back to no team, not silently honored), and a forged max_budget
in metadata never reaches the issued JWT.
* fix(ci): catch HTTPException specifically instead of bare Exception (BLE001)
* fix: un-consume refresh token if minting the replacement fails
Greptile flagged a real reliability gap: cli_refresh_token blocks the
presented token atomically, then does several more DB calls before
returning a replacement (user lookup, team lookup, JWT mint, new
refresh-key mint). Since this endpoint exists specifically for fully
unattended apiKeyHelper operation, a single transient failure in that
window (DB hiccup, etc.) permanently stranded the user: their old
token was already dead and no new one was issued, with no recovery
path short of a full interactive browser re-login.
Wrap that window in try/except; on any failure, best-effort revert the
consumed token back to usable (blocked=False) before re-raising, so a
retry can succeed. Standard compensating-action pattern since
generate_key_helper_fn doesn't take an injectable transaction, so
wrapping the whole thing in a real DB transaction isn't practical here.
* fix(security): refresh key had unrestricted model access, not none
Critical bug: _mint_cli_refresh_token used models=[] intending "no LLM
access", but that's backwards in this codebase. Per
_check_model_access_helper: `len(filtered_models) == 0 and len(models)
== 0` -> all_model_access = True. An empty models list on a key with no
team_id means UNRESTRICTED access to every model, not zero access. The
CLI refresh token -- meant to be usable for nothing but silently
exchanging itself for a new JWT -- was actually a fully unrestricted
API key for its entire 90-day lifetime, completely undermining the
whole point of keeping it separate from the short-lived call
credential.
Fixed with two independent layers: allowed_routes hard-restricts the
key to exactly /sso/cli/refresh and /sso/cli/logout (the real enforced
boundary, checked in the shared user_api_key_auth dependency for every
route); models is set to an unmatchable sentinel string as
defense-in-depth in case any code path only consults the models field.
Added an end-to-end regression test that exercises the actual
model-access-control function against a key shaped like the minted
refresh token, rather than only asserting on what arguments were passed
to the key-generation call -- the latter kind of test is exactly what
let the original bug ship, since asserting `models == []` is equally
consistent with "no access" and "unrestricted access" without checking
what the access-control code actually does with that shape.
Also: the compensating-rollback added for reliability un-blocked a
consumed refresh token even when the underlying user no longer exists.
That's a permanent, intentional rejection, not a transient failure --
un-blocking it would let a stale refresh token become valid again for a
different account if the user_id is ever reused/re-registered. Moved
the user-existence check outside the rollback-on-failure block so it
stays permanently blocked.
* refactor: rotate CLI refresh tokens via regenerate_key_fn instead of hand-rolled consume/rollback
The refresh token is already a plain litellm virtual key, so rotation can
delegate to the same atomic DB update /key/regenerate uses instead of a
bespoke update_many + compensating-rollback dance. This makes silent CLI
refresh an Enterprise feature, same as regular key regeneration.
* refactor: replace CLI stateless JWT + refresh-key pair with one self-rotating virtual key
The CLI previously minted two credentials on login: a stateless self-signed
JWT for LLM calls, and a separate DB-backed refresh-only key (scoped away
from ever calling an LLM) just to authorize minting a new JWT. Collapse
this into a single real virtual key, used directly as the LLM bearer token
and re-presented to /sso/cli/refresh to rotate its own secret in place.
This also means the CLI session key now shows up in the Admin UI's Keys
page and can be revoked/regenerated like any other key, rather than being
an invisible, unmanageable stateless token.
* refactor: drop silent CLI refresh, key just expires and requires re-login
/sso/cli/refresh only ever benefited Enterprise deployments (regenerate_key_fn's
gate), while everyone else already fell through to "re-run lite login" on
failure. Cut the endpoint, the rotation logic, and the client-side refresh
path entirely; print-token now just prints the cached key until it hits its
LITELLM_CLI_JWT_EXPIRATION_HOURS duration, then fails fast telling the user
to log in again. Session key itself is unaffected: still a real, revocable
virtual key visible in the Keys UI, `lite logout` still revokes it directly.
* fix(ci): regenerate schema.d.ts after removing /sso/cli/refresh route
* revert: go back to stateless JWT, keep only lite auth print-token
The virtual-key redesign (revocable, Keys-UI-visible credential) wasn't
needed just to support print-token, and cost real server-side surface
(a mint path, a logout-revoke endpoint, migrated tests/docs) for a property
this repo doesn't need yet. Reverting cli_poll_key/_types.py/schema.d.ts
back to the original stateless-JWT design; the only durable addition from
this whole effort is `lite auth print-token` (reads the cached credential,
prints it while fresh, fails with a clear message once it's past
LITELLM_CLI_JWT_EXPIRATION_HOURS) plus the base_url_explicit plumbing it
needs. `lite logout` goes back to clearing the local file only, since a
stateless JWT can't be revoked server-side.
* refactor: move CLI token freshness check to cli_token_utils, drop unnecessary renames
Addresses review: the freshness check is a pure token-shape/timestamp
util, not command logic, so it belongs alongside the other SDK-level
CLI token helpers (load_cli_token, get_litellm_gateway_api_key) rather
than in commands/auth.py. Also reverted a few incidental jwt_token/
session_key variable and string renames that weren't load-bearing.
* fix(mcp): run the route gate on bridge admission so allowed_routes are enforced
The envelope arm reloaded the identity and ran _run_centralized_common_checks
but skipped RouteChecks.should_call_route, which the standard pipeline runs
between the builder and common_checks. Because the centralized checks treat MCP
as an inference route and never re-check allowed_routes, a key barred from MCP
routes could mint an envelope at the token endpoint (not itself an MCP route)
and replay it against MCP. Run the route gate before admitting, and clear the
request-scoped budget_reservation, matching the wrapper's sequence; a disallowed
route now surfaces the gate's own 403.
* fix(proxy): reserve budget for tiered pricing
Ensure tier-only models reserve their estimated request cost so concurrent requests cannot bypass exhausted budgets.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): bill tier-only deployments instead of $0
Route cost calculation to the deployment's router_model_id entry when it carries tiered_pricing but no flat per-token rate, so models like dashscope/qwen3.7-plus are billed via their tier table rather than the pricing-stripped shared alias.
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(ui): convert activity metrics charts to shadcn/recharts (#32726)
* refactor(ui): convert activity metrics charts to shadcn/recharts
Swap the seven tremor AreaChart/BarChart sites in activity_metrics.tsx to
the shared shadcn/recharts wrappers and switch CustomLegend/CustomTooltip
to the ported versions in shared/charts. Chart props, colors, formatters,
and legend behavior are unchanged; tests now assert on real recharts SVG
output instead of tremor mocks.
* fix(ui): restore tremor No data placeholder for empty AreaChart data
* test(ui): scope activity metrics chart assertions to card titles instead of render order
* feat(ui): extend topnav border across the sidebar header (#32920)
Pin the sidebar header to the same 56px height as the dashboard topnav and
give it a matching bottom border, so the two borders sit flush and read as one
continuous line. Revert to auto height when the rail is collapsed so the
stacked logo and toggle are not clipped.
* fix(cost): coerce string tiered-pricing costs and share tier helper
YAML-parsed tier costs can arrive as strings (e.g. "4e-07"), which broke
arithmetic in the graduated tiered-pricing calculation. Coerce per-token
costs to float in both the in-range and remaining-tokens paths.
Move the tiered-cost helper out of the Dashscope module into a
provider-neutral home so the proxy budget reservation no longer depends on
a provider-specific module.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(anthropic): clarify the Opus 4.5 branch in adaptive-effort translation
Add an inline comment explaining that the effort-capable non-adaptive branch in
_translate_adaptive_effort_for_non_adaptive_model exists for models like Claude Opus
4.5 that accept output_config.effort but reject adaptive thinking, and why effort-only
requests pass through while adaptive requests with an unsupported effort level fall
through to the legacy translation.
* fix(anthropic): translate raw adaptive thinking for chat completions on pre-4.6 models
Clients that pass thinking={"type": "adaptive"} directly (not via the
reasoning_effort alias) on the /chat/completions interface had it forwarded
unmodified to pre-4.6 Anthropic models, which reject the shape. Mirrors the
translation already applied on the native /v1/messages passthrough (#32867):
translate to legacy thinking={type: enabled, budget_tokens}, capped below
max_tokens, dropping thinking when max_tokens can't fit even the minimum
budget. Hoists the shared budget-capping helper onto AnthropicConfig so both
paths use one implementation.
* fix(proxy): reserve tiered budget all-or-nothing across all deployments
Alibaba Model Studio (Dashscope) tiered pricing is all-or-nothing: the tier is
selected by a request's total input tokens and every token, input and output, is
billed at that one tier's rate. The reservation path used graduated slicing and,
worse, picked the output tier from the output-token count, so a long-context
request with a large output allowance reserved far less than the provider charges
and could slip past a depleted budget. Select the tier from input tokens and apply
its rates to all input and output tokens.
Reservation also read tiered pricing from only the first deployment in a model
group. A caller could hit an alias whose cheaper deployment was listed first and
exceed the budget once routed to a costlier sibling. Estimate against every
eligible deployment's pricing and reserve the maximum.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(complexity_router): log the cause of each routing decision
The complexity router's info log didn't say what drove a routing decision.
Literal and semantic keyword matches logged an identical "keyword rule fired"
line (no way to tell which mechanism fired), and the scorer's line carried no
consistent marker tying it to the same question.
Emit one greppable line per decision naming the cause: literal_keyword_match,
semantic_keyword_match, or complexity_scorer. The hook already knows which ran
(the config's semantic_keyword_matching flag distinguishes lexical from
semantic; the override-vs-scorer branch distinguishes keyword match from
scorer), so this is label-only: no behavior change, no new types, no added
latency.
Adds regression tests asserting each decision path logs its cause; they fail if
a label is swapped or the cause= marker is dropped.
* feat(ui): add redesigned sidebar account menu (#32931)
* feat(ui): add redesigned sidebar account menu
Introduce SidebarAccountMenu, a sidebar-only account/logout menu built on
shadcn Popover/Switch/Badge/Separator/Button, and wire it into leftnav in
place of the shared UserDropdown. The panel has a LiteLLM header with the
bouncing moon and a clickable version tag, Tier/Role/Email/User ID rows
with copy actions, the five display toggles, and Logout.
UserDropdown is left untouched so the control-plane / chat navbar keeps
its existing menu. The version tag links to the same release notes page
as the navbar tag, and the bouncing icon reuses the existing header
animation gated by the Hide Bouncing Icon toggle.
* test(ui): point account-menu e2e specs at the migrated sidebar menu
The sidebar account menu moved from an antd Dropdown to a Base UI popover
(SidebarAccountMenu), so the login, logout, proxy-logout-url, and internal
user identity specs were still waiting on antd-era locators
(.ant-dropdown, the popupRender wrapper class, the user-dropdown-panel test
id, and a menuitem-role Logout). Point them at the new panel test id
(sidebar-account-menu-panel) and the button-role Logout instead. The logout
behavior is unchanged since both menus call the same useLogout handler.
* fix(bedrock-converse): translate adaptive thinking for pre-4.6 models
Follow-up to #32867 (native /v1/messages) and the /chat/completions
commit earlier on this branch, extending the same adaptive-thinking
translation to the Bedrock Converse path.
Clients like Claude Code send thinking={type: "adaptive"} on every
request. When routed via Bedrock Converse to pre-4.6 models
(claude-haiku-4-5, claude-sonnet-4-5), this was forwarded as-is and
rejected by the model. Mirrors the translation already applied on the
/chat/completions and /v1/messages paths: map to legacy
thinking={type: enabled, budget_tokens}, capped below max_tokens.
Also fixes the missing custom_llm_provider arg in the chat completions
path's call to AnthropicConfig._map_reasoning_effort.
* fix(mcp): run proxy-wide pre-DB gates on bridge envelope admission
The envelope arm bypasses user_api_key_auth, so it never ran
pre_db_read_auth_checks (request-size and body-safety limits, the IP allowlist,
and the general_settings route allowlist) that the normal MCP admission path
runs before any key lookup. A caller blocked by IP or a disallowed proxy route
could be admitted through an envelope where the same principal on the normal
path is rejected. Run those gates before the envelope crypto, mirroring the
pipeline's pre-DB ordering; a blocked IP or route surfaces its own 403.
* fix(anthropic): pass resolved provider to adaptive-thinking check
The rebase onto staging changed _is_adaptive_thinking_model to require
custom_llm_provider (no default), so the one-arg call in the raw adaptive
thinking branch raised TypeError at runtime for any /chat/completions
caller sending thinking={type: adaptive}. Use self._resolved_provider,
matching the reasoning_effort branch just below. Caught by Greptile.
* test(bedrock-converse): cover adaptive-thinking drop when max_tokens too small
Adds the regression test for the warning-drop branch in the Converse
adaptive-thinking translation, mirroring the chat completions path's
test_raw_adaptive_thinking_dropped_when_max_tokens_too_small.
* fix(guardrails): filter Add-Guardrail mode dropdown per provider (#32712)
* fix(guardrails): filter Add-Guardrail mode dropdown per provider
The GET /guardrails/ui/add_guardrail_settings endpoint returned every
GuardrailEventHooks value in one flat supported_modes list, so the Admin
UI rendered pre_mcp_call as a selectable Mode for every guardrail. Saving
Content Filter or Tool Permission with pre_mcp_call then failed with a
400 because those guardrails' server-side supported_event_hooks list
excludes it.
Expose each guardrail's supported hooks as a get_supported_event_hooks
classmethod on CustomGuardrail (mirrors the existing get_config_model
pattern) and have the endpoint iterate guardrail_class_registry to build
a supported_modes_by_provider map. The UI Mode dropdown filters by that
map when the selected provider is known and falls back to the global
list otherwise. __init__ now sources its own supported_event_hooks list
from the classmethod so the two sides can't drift.
Also register BedrockGuardrail, ToolPermissionGuardrail, lakera,
lakera_v2, and presidio in guardrail_class_registry so they participate
in the map (they were previously only in guardrail_initializer_registry
and had no class-registry entry).
Behavior change: guardrails that previously had no supported_event_hooks
declared (aim, javelin, azure/text_moderation, cato_networks,
crowdstrike_aidr, headroom, hiddenlayer, lasso, noma, onyx,
prompt_security, qualifire, repelloai, zscaler_ai_guard, aporia_ai,
lakera_ai, lakera_ai_v2, mcp_jwt_signer, model_armor, presidio) now
validate the configured mode at instantiation. Existing configs where
the mode was silently a no-op will fail at proxy startup with a clear
validation error rather than running as a broken guardrail.
Resolves LIT-4226
* fix(guardrails): add LITELLM_STRICT_GUARDRAIL_MODES escape hatch, preserve current mode in edit form
Address Greptile P1 (startup break) and P2 (edit form UX):
LITELLM_STRICT_GUARDRAIL_MODES defaults to true (raise on unsupported
event_hook, unchanged behavior for the guardrails validated pre-PR).
Setting it to false logs a warning and continues, giving deployments an
opt-out while they fix configs that now surface as errors instead of
silently no-op'ing. Regression test covers both modes.
Edit form now surfaces the currently-saved mode even when it is not in
the filtered per-provider list, so a legacy row (e.g. content_filter
saved with pre_mcp_call before this fix) no longer disappears from the
dropdown; the option renders with a 'not supported by <provider>' note
so the user knows to pick another.
* fix(guardrails): correct audited hook lists, prune stale modes on provider switch, clean form lint
Audited every get_supported_event_hooks classmethod against the hooks
each guardrail's own tests exercise and its handler methods. Five were
too narrow and their tests caught it in CI: rubrik gains pre_call,
presidio gains during_call and pre_mcp_call, prompt_security, onyx and
qualifire gain during_call. The remaining classes match either their
original __init__ declarations or their exercised modes exactly.
Cursor review fixes: the Add form now drops selected modes the new
provider does not support when the user switches providers, so a
pre_mcp_call selection cannot ride along into a provider that rejects
it at save; the edit form handles list-shaped stored modes instead of
treating mode as always a string.
Extracted shared toModeArray and getSupportedModesForProvider helpers
into guardrail_info_helpers so both forms use one implementation, typed
the remaining any usages in both forms, removed nested ternaries, and
committed the ratcheted-down eslint metrics and pruned suppressions
* fix(proxy): reserve tiered output at the higher reasoning rate
Some tiered Dashscope models price reasoning output above standard output
(output_cost_per_reasoning_token > output_cost_per_token). The reservation charged
all output at the standard rate, so a reasoning-heavy request reserved too little
and concurrent calls could exceed the budget before reconciliation. The reasoning
share is unknown before the reques…
16 tasks
mateo-berri
added a commit
that referenced
this pull request
Jul 20, 2026
…/messages
Azure AI Foundry and Vertex AI serve Claude on the first-party Anthropic
Messages contract, which was verified live to be byte-identical to
api.anthropic.com: a leading role:"system" entry in messages is rejected on
every model ("messages.0: use the top-level 'system' parameter"), and a
mid-conversation role:"system" reminder is accepted in place on Claude 4.8+/5
but 400s on Claude 4.7 and older ("role 'system' is not supported on this
model"). This is the same contract Bedrock Invoke already handles model-aware
(PRs #32578/#32831/#32882); Vertex and Azure did no hoisting at all, so a Claude
Code session on an older Vertex/Azure Claude model hard-400s on its reminder
turns, and the only thing sparing 4.8+/5 was that nothing was hoisted
Extract Bedrock's model-gated normalization into the shared
AnthropicMessagesConfig base as _normalize_system_role_messages and call it from
the Vertex and Azure messages configs. Flagged models (4.8+/5) hoist only the
leading run of system entries and keep mid-conversation reminders in place so
the top-level system prefix stays byte-identical and the prompt cache is
preserved; unflagged models hoist every system entry so the request returns a
completion instead of a 400
Add supports_mid_conversation_system to the azure_ai and vertex_ai Claude 4.8+/5
cost-map entries. Exact cost-map hits win over the claude-mid-conversation-system
fallback rule, so without the explicit flag those models would be treated as
unsupported and hoist every reminder, collapsing the prompt cache (the exact
customer regression). A per-provider test guards this so future 4.8+/5 entries
cannot silently miss the flag
Closes the Vertex/Azure gap from the customer RCA
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relevant issues
Follow-up to #32831
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Live two-turn conversation against Bedrock through a local proxy, with a large cached system prompt and a mid-conversation
role: "system"reminder in turn two. Proxy config and calls:Before (735a5bf): the reminder is hoisted into the top-level
systemfield, the prefix changes, and the entire cache is re-writtenAfter (cfe9193): the reminder stays in place in
messages, the cached prefix survives, and turn two reads the full cacheOne live observation worth noting for reviewers: Bedrock Invoke enforces a position rule for in-place system messages on Sonnet 5 ("role 'system' must follow a 'user' message or an 'assistant' message ending in a server tool result"), which the hoist-all behavior used to mask. Claude Code's mid-conversation reminders follow user turns, so they satisfy the rule
Type
🐛 Bug Fix
Changes
#32831 gates in-place handling of mid-conversation
role: "system"messages on thesupports_mid_conversation_systemcost-map flag, and added the flag to five Opus 4.8 Bedrock entries plus a fallback-generalization rule for unmapped Claude 4.8+ ids. Exact cost-map hits resolve before fallback rules, so the mapped Sonnet 5 and Fable 5 entries (andjp.anthropic.claude-opus-4-8, added after #32831) never reached the rule and still hoisted every system message, invalidating the prompt cache for the whole message history on every reminderThis adds
supports_mid_conversation_system: trueto the eleven mapped Bedrock entries the rule's own pattern matches but that were missing the flag: all sixanthropic.claude-sonnet-5variants, all fouranthropic.claude-fable-5variants, andjp.anthropic.claude-opus-4-8, in both cost map filesTests: the in-place regression test is parametrized over the newly flagged ids, and a new invariant test asserts that every mapped Bedrock entry matching the
bedrock-anthropic-claude-mid-conversation-systemrule pattern carries the flag explicitly, so future 4.8+ model additions cannot silently reintroduce the hoist-all behavior. Without the cost-map changes the parametrized cases and the invariant test fail; with them the file's 86 tests pass