feat(messages): route Azure Anthropic /messages through Rust behind rust:true

[devin-ai-integration]

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

End-to-end against a live proxy on localhost:4000 hitting the real Azure Anthropic API (no mocks), captured at commit ee027e9c5b

Two deployments of the same model differ only in the rust flag

model_list:
  - model_name: claude-rust
    litellm_params:
      model: azure_ai/claude-sonnet-4-5
      api_base: os.environ/AZURE_AI_API_BASE_ANTHROPIC
      api_key: os.environ/AZURE_AI_API_KEY
      rust: true
  - model_name: claude-python
    litellm_params:
      model: azure_ai/claude-sonnet-4-5
      api_base: os.environ/AZURE_AI_API_BASE_ANTHROPIC
      api_key: os.environ/AZURE_AI_API_KEY
      rust: false

The rust: true deployment returns a real 200 and carries the x-litellm-rust: true marker header

$ curl -sS -D - -o /dev/null -X POST http://localhost:4000/v1/messages \
    -H "Authorization: Bearer sk-1234" -H "content-type: application/json" \
    -d '{"model":"claude-rust","max_tokens":32,"messages":[{"role":"user","content":"hi"}]}'
HTTP/1.1 200 OK
x-litellm-rust: true

The rust: false twin returns a real 200 with no marker header, i.e. it stayed on the existing Python path

$ curl -sS -D - -o /dev/null -X POST http://localhost:4000/v1/messages \
    -H "Authorization: Bearer sk-1234" -H "content-type: application/json" \
    -d '{"model":"claude-python","max_tokens":32,"messages":[{"role":"user","content":"hi"}]}'
HTTP/1.1 200 OK

Beyond curl, Claude Code was pointed at the gateway and made real code edits through the Rust route across two models (sonnet-4-5-rust fixed a bug, sonnet-4-6-rust added a function). The recorded walkthrough and a written test report with screenshots are shared with the requester rather than embedded here

Type

🆕 New Feature

Changes

This adds an opt-in Rust path for Azure Anthropic Messages, mirroring the existing OCR Rust bridge as closely as possible

Rollout is per-deployment and off by default. A request routes through Rust only when the provider is azure_ai and the deployment sets rust: true in litellm_params. Every other case falls back to the existing Python implementation: a missing rust flag, rust: false, a non-Azure provider, a native bridge that is not built, or a bridge that returns None. The gate also wraps the bridge call so that any Rust bridge exception falls back to Python, which keeps the public status and error body identical to the Python path (previously a Rust upstream error surfaced as a 500 with an OCR-flavored message). Because the gate lives inside the shared anthropic_messages handler and only shortcuts the outbound network call, all Python hooks, interceptors, provider resolution, logging, and mock/short-circuit behavior are preserved

When Rust serves a request the response carries an x-litellm-rust: true header (set through _hidden_params.additional_headers) so a caller can tell which path handled it

Streaming now also routes through Rust for eligible requests. Python prepares the normal Anthropic request, Rust performs the upstream call with stream removed, and Python re-emits the full response as Anthropic SSE through FakeAnthropicMessagesStreamIterator while still exposing x-litellm-rust: true. This is buffered Rust-handled streaming, not incremental Rust SSE transport, and it keeps the existing event shape (message_start, content_block_*, message_delta, message_stop)

The work follows the three-crate Rust layout. litellm-core holds the pure transforms, now split so the generic Anthropic Messages config lives in its own provider file (providers/anthropic/messages) and the Azure config composes it (providers/azure_ai/messages) adding /anthropic/v1/messages URL normalization, x-api-key auth, and cache_control scope stripping. The request and response contract is typed end to end (AnthropicMessagesRequest / AnthropicMessagesResponse with typed system, messages, and content blocks) instead of a bare serde_json::Value, covering the full documented Anthropic Messages parameter set. litellm-ai-gateway owns the buffered network execution with a reused reqwest client, connect and full-request timeouts, and bounded, sanitized upstream error bodies that never log payloads or secrets. litellm-python-bridge exposes thin PyO3 messages and amessages adapters

The Azure transform also folds a role: "system" chat message into the top-level system field. Some agent clients (e.g. Claude Code) send an extra role: "system" message that Anthropic and Azure both reject; folding it into system is spec-compliant and lets those clients complete real turns through the gateway

Changes

Files of note: the Python gate and buffered-stream helper in litellm/llms/custom_httpx/llm_http_handler.py, the bridge wrapper in litellm/rust_bridge/messages.py, the core transforms under litellm-rust/crates/core/src/providers/{anthropic,azure_ai}/messages, and the gateway I/O under litellm-rust/crates/ai-gateway/src/messages

Tests cover both layers. cargo test --workspace exercises the core transforms (supported-parameter coverage, request shape, response normalization, cache-control scope stripping, and the system-role fold), the gateway request/auth/error handling, and the bridge marshaling. tests/test_litellm/rust_bridge/test_messages.py covers the gate on and off, the Azure-only restriction, sync and async argument forwarding, timeout conversion, the header metadata, buffered SSE events, and each fallback including the new bridge-exception fallback. The existing OCR bridge tests still pass

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/f57af0b79ee8472287d328e6dd984593
Requested by: @ishaan-berri

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

This PR adds an opt-in Rust path for Azure Anthropic /messages requests, mirroring the existing OCR Rust bridge. Routing is per-deployment via rust: true in litellm_params; every other case (missing flag, non-azure_ai provider, bridge absent, or bridge exception) falls back to the existing Python path with no change in public behavior.

• The Python gate (_maybe_rust_anthropic_messages) is inserted after pre_call logging and correctly consults _has_agentic_completion_hook before marking a streaming request eligible for the buffered Rust path; non-eligible streaming and all non-Azure requests bypass Rust entirely.
• The Rust-side transform splits out typed AnthropicMessagesRequest/AnthropicMessagesResponse structs, adds Azure-specific URL normalization, x-api-key auth, cache_control.scope stripping, and the system-role-message fold that makes Claude Code clients work through Azure's strict validation.
• All previously flagged review concerns are addressed: the _rust_ocr_enabled side-effect in use_litellm_rust is fixed by the configuring_ocr or not configuring_messages guard, and the messages bridge no longer exposes a dead global-toggle path — routing is purely per-deployment as documented.

Confidence Score: 5/5

Safe to merge: the Rust path is fully opt-in per deployment, any bridge failure falls back to the existing Python implementation, and all previously flagged issues are resolved.

The change introduces no new default behavior — every request not explicitly flagged with rust: true on an azure_ai deployment is unaffected. The only finding is a URL normalization edge case that would only trigger if an Azure resource were named anthropic; even then, the Python fallback ensures no user-visible breakage.

litellm-rust/crates/core/src/providers/azure_ai/messages/transformation.rs — the split_once("/anthropic") URL normalization deserves a follow-up to restrict the search to the path portion of the URL.

Important Files Changed

Filename	Overview
litellm/llms/custom_httpx/llm_http_handler.py	Inserts Rust gate and fake-stream helper before the existing Python HTTP call; extracts _finalize_anthropic_messages_response to avoid duplicating the post-call logic across the Rust and Python paths. The placement relative to pre_call logging and agentic-hook detection is correct.
litellm/rust_bridge/messages.py	Thin Python wrapper for the Rust messages bridge; previous reviewer concerns about dead rust_messages_enabled() and a missing enabled field are fully resolved — the gate is now purely per-deployment via litellm_params.get("rust"), matching the PR's opt-in design.
litellm/rust_bridge/ocr.py	Fixed the OCR-enabled side-effect flagged in the previous review: _rust_ocr_enabled is now only updated when configuring_ocr or not configuring_messages, so callers who only set the messages bridge no longer inadvertently toggle OCR.
litellm-rust/crates/core/src/providers/azure_ai/messages/transformation.rs	Implements Azure-specific transforms: URL normalization, x-api-key auth, cache_control.scope stripping, and system-role message folding. URL normalization uses split_once("/anthropic") which can spuriously match the second // in the URL scheme when a resource hostname starts with "anthropic".
litellm-rust/crates/ai-gateway/src/messages/handler.rs	Buffered HTTP executor: applies per-request timeout override on top of the shared OnceLock reqwest client and truncates upstream error bodies to MESSAGES_ERROR_BODY_MAX_CHARS before crossing the host boundary. Logic is straightforward and correct.
litellm-rust/crates/ai-gateway/src/messages/prepare.rs	Resolves provider config, injects auth only when the header is absent (correctly avoids duplicating Python-set headers), sets default headers idempotently, then deserializes and transforms the typed request. Correct and safe.
litellm-rust/crates/core/src/messages/types.rs	Well-typed end-to-end request/response structs with #[serde(flatten)] extra for forward-compatibility with new Anthropic parameters. Named fields cover the full documented parameter set; unknown fields are preserved and round-tripped.
litellm-rust/crates/python-bridge/src/lib.rs	Adds PyO3 messages and amessages adapters mirroring the existing ocr/aocr pattern; sync path releases the GIL, async path uses future_into_py. Both validate that body is a JSON object before dispatch.
tests/test_litellm/rust_bridge/test_messages.py	All tests use injected mock bridges; no real network calls. Covers gate on/off, Azure-only restriction, argument forwarding, timeout conversion, bridge-exception fallback, streaming eligibility, SSE event shape, and bridge unavailability.
litellm/rust_bridge/timeouts.py	Extracts _timeout_to_seconds from ocr.py into a shared utility; OCR callers updated to import from here. Clean refactor with no logic changes.

_{Reviews (2): Last reviewed commit: "fix(messages): fold system-role messages..." | Re-trigger Greptile}

[greptile-apps]

rust_messages_enabled() is never consulted in the routing gate

rust_messages_enabled() returns _STATE.enabled, which is initialized from the LITELLM_USE_RUST_MESSAGES environment variable and updated by use_litellm_rust. However, _maybe_rust_anthropic_messages only checks litellm_params.get("rust") is not True — it never calls rust_messages_enabled(). This means setting LITELLM_USE_RUST_MESSAGES=0 or calling use_litellm_rust(False) has no effect on the messages path, contrary to what the OCR path does (rust_ocr_enabled() is checked in ocr/main.py before every call). The function is effectively dead code and the global disable toggle is silently broken for messages.

[greptile-apps]

load_rust_messages / load_rust_amessages ignore _STATE.enabled

Both loader functions return a bridge implementation without checking _STATE.enabled. If a caller injects a bridge via set_rust_messages(enabled=False, messages=bridge), load_rust_messages() will still return bridge because only the _STATE.messages is not None guard is evaluated. The enabled field is stored and exported as rust_messages_enabled() but never enforced in the load path — the OCR bridge has the same architectural shape but the OCR load path is gated externally in ocr/main.py. The messages path has no equivalent external gate, so the enabled field is a no-op in the current design.

[codecov]

Codecov Report

❌ Patch coverage is 93.33333% with 7 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/rust_bridge/messages.py	92.00%	4 Missing ⚠️
litellm/llms/custom_httpx/llm_http_handler.py	90.62%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!

[devin-ai-integration]

Demo: live gateway proof

Ran the proxy locally on :4000 with four Azure Anthropic deployments (same resource, rust:true on three, rust:false on one) and hit real POST /v1/messages (real provider calls).

Results

• sonnet-4-5-rust (rust:true) → HTTP 200, x-litellm-rust: true, real Claude reply
• sonnet-4-5-python (rust:false, same model) → HTTP 200, no x-litellm-rust header (Python)
• haiku-4-5-rust + sonnet-4-6-rust → HTTP 200, x-litellm-rust: true each
• stream:true → HTTP 200, Anthropic SSE events, no Rust header (Python fallback — Rust SSE not implemented yet)

rust:true vs rust:false (same model) — screenshots

Automated: cargo test --workspace 96 passed / 0 failed / 1 ignored; rust-bridge + OCR pytest 47 passed; make pre-commit clean.

Devin session

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_rust_messages_azure (ef38697) with litellm_internal_staging (010b200)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (6f4f4f6) during the generation of this report, so 010b200 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[devin-ai-integration]

Updated demo: Claude Code making real edits through the gateway (Rust route)

This supersedes the earlier curl-only demo. Claude Code is pointed at the local gateway (ANTHROPIC_BASE_URL=http://localhost:4000) and makes real code edits routed through the Azure Anthropic Rust path, across two models. Captured at commit ee027e9c5b

What the recording shows

The config has sonnet-4-5-rust (rust: true) and a sonnet-4-5-python twin (rust: false). The same request to the rust deployment returns HTTP 200 with x-litellm-rust: true, while the python twin returns 200 with no header. Then Claude Code on sonnet-4-5-rust fixes a bug in calc.py (return a - b -> return a + b, output goes from 2 + 3 = -1 to 2 + 3 = 5), and after switching to sonnet-4-6-rust it adds a subtract() function (output 10 - 4 = 6). The gateway log shows the Claude Code turns returning 200

One honest caveat: Claude Code also fires a small-fast-model title-generation call that uses structured outputs, which this Azure workspace is not entitled for, so that single side call returns 400. It does not affect the coding turns or the edits, and both the Python and Rust deployments reject that exact request identically

[devin-ai-integration]

@greptileai please re-review. Since the last review this PR split the Anthropic Messages config into its own provider file, typed the request/response contract end to end (no bare serde_json::Value in the messages path), added Rust-handled streaming, added a Python fallback on any Rust bridge exception, and folds role:"system" chat messages into the top-level system for Azure.