feat(messages): route native Anthropic /messages through Rust behind LITELLM_USE_RUST_MESSAGES

[devin-ai-integration]

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Captured against a live proxy on localhost:4000 hitting the real Anthropic API (no mocks), so the critical AI call runs in Rust. Proof is being captured at the head of this branch; a recorded Claude Code walkthrough (Opus 4.8, real code edits through the Rust route) plus the raw curl runs are shared with the requester rather than embedded here, matching the team preference and PR #33616

The proxy is started with the rollout flag on

LITELLM_USE_RUST_MESSAGES=true python litellm/proxy/proxy_cli.py --config litellm/proxy/dev_config.yaml --detailed_debug

with a native Anthropic deployment

model_list:
  - model_name: claude-opus
    litellm_params:
      model: anthropic/claude-opus-4-8
      api_key: os.environ/ANTHROPIC_API_KEY

A real 200 comes back carrying the x-litellm-rust: true marker header, i.e. Rust served the upstream call

$ curl -sS -D - -o /dev/null -X POST http://localhost:4000/v1/messages \
    -H "Authorization: Bearer sk-1234" -H "content-type: application/json" \
    -d '{"model":"claude-opus","max_tokens":32,"messages":[{"role":"user","content":"hi"}]}'
HTTP/1.1 200 OK
x-litellm-rust: true

With LITELLM_USE_RUST_MESSAGES unset the same request returns a real 200 with no marker header, i.e. it stayed on the existing Python path

Type

🆕 New Feature

Changes

This builds on PR #33616 (Azure Anthropic /messages through Rust) and does two things

First, it extends the Rust /messages path to the native anthropic provider, not just azure_ai. The pure transform for native Anthropic already existed in litellm-core (providers/anthropic/messages), it just was not reachable from the gateway; the only wiring needed was an anthropic arm in messages_provider_config so the native config (x-api-key auth, https://api.anthropic.com/v1/messages, default anthropic-version/content-type headers, identity request transform) resolves. Native Anthropic deliberately does not strip cache_control.scope or fold role: "system" chat messages, matching the Python native path; those transforms stay Azure only

Second, it switches messages enablement to the LITELLM_USE_RUST_MESSAGES env var, mirroring the OCR bridge's LITELLM_USE_RUST_OCR and rust_ocr_enabled(). The gate now shortcuts to Rust when rust_messages_enabled() is true instead of requiring a per-deployment rust: true flag. use_litellm_rust(enabled, ...) toggles the messages flag under the same guard OCR uses, so programmatic control stays symmetric. When the flag is on, eligible providers (native anthropic, azure_ai) route through Rust and every other provider falls back to Python because the Rust provider lookup returns nothing and the bridge yields None. All Python hooks, provider resolution, logging, and the streaming re-emit path are unchanged, and any Rust bridge exception still falls back to the Python path

Streaming stays buffered as in #33616: Python prepares the request, Rust performs the upstream call with stream removed, and Python re-emits the buffered result as Anthropic SSE while exposing x-litellm-rust: true. This is buffered Rust-handled streaming, not incremental Rust SSE transport

Tests cover both layers. cargo test --workspace exercises native-Anthropic provider resolution end to end through prepare_messages_call (URL, x-api-key auth, default headers, scope left intact) and that an unknown provider is rejected. tests/test_litellm/rust_bridge/test_messages.py covers the env gate on and off, native-anthropic routing and argument forwarding, the x-litellm-rust header, and fallback for an unsupported provider

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/1a5f6d0494a145e3a864c45bcc741dc8
Requested by: @ishaan-berri

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

This PR extends the Rust-backed /messages path (introduced for azure_ai in #33616) to the native anthropic provider, and replaces the per-deployment rust: true litellm_params flag with a global LITELLM_USE_RUST_MESSAGES environment variable that mirrors the existing LITELLM_USE_RUST_OCR pattern.

• Rust-side: A single "anthropic" arm is added to messages_provider_config in common_utils.rs, wiring native Anthropic requests to the pre-existing ANTHROPIC_MESSAGES_CONFIG (which intentionally preserves cache_control.scope and system messages, unlike the Azure transform). New cargo tests verify URL construction, x-api-key auth, default headers, scope preservation, and rejection of unknown providers.
• Python-side: rust_bridge/messages.py gains an enabled field on _RustMessagesState initialized from the env var at import time; rust_bridge/ocr.py propagates enabled to set_rust_messages under all call combinations; llm_http_handler.py replaces the custom_llm_provider == "azure_ai" and litellm_params.rust == True guard with a single rust_messages_enabled() call, delegating provider eligibility entirely to the Rust bridge.
• Side effect: When LITELLM_USE_RUST_MESSAGES=true, requests to providers not supported by the bridge (e.g., openai) now incur a bridge call that always returns None or raises, before falling back to Python — where the old code short-circuited immediately.

Confidence Score: 4/5

Safe to merge — the change is well-scoped, both the Rust and Python layers have tests, and the fallback to Python on any bridge error is preserved throughout.

The core routing logic and Rust bridge extension are correct. The two minor observations are: ineligible-provider requests now pay an unnecessary bridge round-trip before falling back (where the old code short-circuited in Python), and two test cases became functionally identical after the per-deployment rust flag was removed from the gate — their names now imply a distinction that no longer exists in the code.

litellm/llms/custom_httpx/llm_http_handler.py — the removed Python-level provider guard means every non-eligible provider invokes the bridge when the env var is on; tests/test_litellm/rust_bridge/test_messages.py — the two 'flag absent / flag false' tests now exercise the same condition.

Important Files Changed

Filename	Overview
litellm/llms/custom_httpx/llm_http_handler.py	Gate logic simplified: per-deployment rust: true flag and provider check removed; now only rust_messages_enabled() controls routing. Bridge is invoked for all providers when enabled, with fallback on None or exception.
litellm/rust_bridge/messages.py	Adds enabled field to _RustMessagesState, reads LITELLM_USE_RUST_MESSAGES env var at module load, and exposes rust_messages_enabled(). Mirrors the OCR pattern cleanly.
litellm/rust_bridge/ocr.py	Refactors use_litellm_rust to propagate enabled to messages state and handles all four call combinations (both, messages-only, ocr-only, bare) correctly. Logic is symmetric with OCR gating.
tests/test_litellm/rust_bridge/test_messages.py	Adds native Anthropic gate test and NoneAsyncMessages helper. Two tests (flag_absent, flag_false) now exercise identical conditions after the per-deployment rust flag was removed from the gate; names are misleading.
litellm-rust/crates/ai-gateway/src/messages/common_utils.rs	Adds "anthropic" arm to messages_provider_config, wiring native Anthropic requests to the existing ANTHROPIC_MESSAGES_CONFIG. Minimal, correct change.
litellm-rust/crates/ai-gateway/src/messages/tests.rs	Updates Rust tests: adds prepare_messages_call unit tests for native Anthropic (URL, auth header, scope preservation) and unknown-provider rejection; corrects messages_rejects_unsupported_provider to use "openai" now that "anthropic" is valid.

Comments Outside Diff (2)

1. tests/test_litellm/rust_bridge/test_messages.py, line 296-315 (link)

   Tests now redundantly check the same condition

   test_gate_skips_rust_when_flag_absent and test_gate_skips_rust_when_flag_false have been updated to both call litellm.use_litellm_rust(False, amessages=bridge). Since the gate now only checks rust_messages_enabled() and ignores litellm_params.rust entirely, the rust=False kwarg in the second test's litellm_params no longer has any effect. Both tests are now identical in what they actually exercise: rust_messages_enabled() == False → skip. The names imply two distinct scenarios (absent vs. explicitly-false per-deployment flag), but the code path they exercise is the same. The tests remain correct but could be consolidated or the rust=False kwarg removed to avoid misleading future readers.

   Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

2. litellm/llms/custom_httpx/llm_http_handler.py, line 2264-2291 (link)

   Unsupported providers now always invoke the bridge when the env var is set

   Previously, the gate short-circuited before calling the bridge for any provider that wasn't azure_ai. With the new logic, when LITELLM_USE_RUST_MESSAGES=true, every request — regardless of provider — invokes the Rust bridge, which then raises CoreError::InvalidProvider and falls back to Python via the except Exception block. For deployments that mix eligible providers (native Anthropic, Azure AI) with ineligible ones (OpenAI, Gemini, etc.), every ineligible request pays an extra round-trip to the bridge on the hot path before falling back. The test test_gate_skips_rust_for_non_listed_provider explicitly captures this: bridge.calls == 1 rather than 0. Adding a fast-path provider allowlist check in Python before calling amessages would avoid this overhead.

_{Reviews (1): Last reviewed commit: "feat(messages): route native Anthropic /..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing devin/1784399610-rust-anthropic-messages (93be28e) with litellm_internal_staging (fdf380d)}