refactor(native): separate request data from execution context

[yujonglee-berri]

TLDR

Problem this solves:

• Provider inputs mixed with LiteLLM execution context
• The Python/Rust boundary repeated request-field extraction

How it solves it:

• Separates provider data, connection options, and execution context
• Shares typed request extraction across native routes

User Flow

Before: developers use the existing endpoint request and response formats

1. Send POST https://litellm-domain/v1/chat/completions, https://litellm-domain/v1/messages, or https://litellm-domain/v1/ocr
2. Receive the endpoint's existing response format

After: the same requests retain their existing formats

1. Send the same request to the same endpoint
2. Receive the endpoint's existing response format

Implementation

Separates provider request data from execution context. Dispatch plumbing is reviewed in #39913, error/fallback fixes in #39923, and endpoint-adapter migration in #39914

All seven Python contracts use native(request, *, context). Route dataclasses contain provider inputs and shared connection options. LiteLLM metadata, forwarding configuration, call identity, and attribution live in execution context. Responses API metadata remains provider data

PyO3 extracts dataclass attributes into bridge-local types and converts them into Python-independent Rust types. Core and gateway entrypoints receive request data and execution context separately. Tests cover field ownership, provider payloads, lifecycle identity, and malformed boundary inputs

Typed native selectors, callable inference, and exception-class validation remain shared with #39913. Rust enablement remains process-wide as established in #39928. Type-contract and configuration tests persist through every later layer

Validation: 146 focused Python tests pass. Local make check passes. Rust source is unchanged by this restack; previous Rust validation was not rerun

Relevant issues

Stack order: #39928, #39913, #39923, #39914, #39766, then #39484 for callback/readiness work

Linear ticket

Pre-Submission checklist

• Meaningful boundary tests accompany the changes
• Focused tests and local quality gates pass
• All required CI checks pass
• Scope is request data and execution context
• Greptile confidence is at least 4/5

Screenshots / Proof of Fix

No live provider proof is claimed. This refactor preserves public endpoint schemas

Type

Refactoring
Test

Caveats (if any)

Medium

• Native implementations remain incomplete
• Callback readiness requires refactor(rust): establish callback contracts and readiness #39484 before rollout

Final Attestation

• Tests cover extraction, metadata ownership, and lifecycle identity

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_ocr_dispatch_contract (d03b9a1) with main (7d5b645)¹}

Footnotes

1. No successful run was found on codex/native-endpoint-adapters (1f85bd6) during the generation of this report, so main (7d5b645) was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[greptile-apps]

Greptile Summary

This PR centralizes native-versus-Python selection behind typed endpoint dispatch contracts and makes fallback conservative after native work may have begun.

• Gates native routes through explicit ready_endpoints metadata and callable validation.
• Keeps native request preparation lazy until the native implementation is selected.
• Permits Python fallback only for unavailable, disabled, ineligible, or explicitly declined native operations.
• Adds common callback-handle contracts and endpoint-specific dispatch wrappers.
• Leaves packaged native routes unregistered while callback and lifecycle support is incomplete.
• Reclassifies ambiguous connection failures as terminal upstream failures to avoid duplicate provider requests.

Confidence Score: 5/5

The PR appears safe to merge because no new actionable failure remains and the packaged readiness registry prevents incomplete native routes from becoming active.

The centralized dispatch consistently falls back only before native commitment, while upstream and unknown native failures remain terminal to prevent duplicate provider operations. Lazy request preparation and explicit readiness metadata preserve existing Python behavior for optional surfaces. The previous OCR finding was manually resolved and the current implementation permits explicit native declines to return to Python.

Important Files Changed

Filename	Overview
litellm/rust_bridge/runtime.py	Defines the shared sync, async, and required-native dispatch policy, including conservative exception classification.
litellm/rust_bridge/bindings.py	Resolves typed native callables only when readiness metadata declares the corresponding route.
litellm/rust_bridge/chat_completions.py	Routes eligible non-streaming Anthropic and Bedrock chat requests through the centralized dispatch contract.
litellm/main.py	Moves the complete Python provider switch behind a single chat-completion fallback selected by EndpointDispatch.
litellm/rust_bridge/messages.py	Adds centralized sync and async Messages dispatch with lazy request construction and safe-decline fallback.
litellm/ocr/main.py	Integrates OCR with readiness-gated lazy native preparation while preserving Python fallback.
litellm/responses/main.py	Adds one-shot Responses HTTP dispatch while retaining Python handling for streams, custom clients, and unavailable native routes.
litellm/rust_bridge/responses_websocket.py	Wraps native Responses WebSocket sessions in the shared readiness and fallback policy.
litellm/rust_bridge/transcription.py	Expresses transcription as an intentionally required-native endpoint with explicit unavailable errors.
litellm-rust/crates/python-bridge/src/errors.rs	Treats connection and other ambiguous post-selection failures as terminal upstream errors rather than fallback signals.
litellm-rust/crates/python-bridge/src/lib.rs	Publishes an empty packaged readiness registry so incomplete native stubs remain inactive.

_{Reviews (2): Last reviewed commit: "fix(python): avoid runtime generic const..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 93.72197% with 42 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/main.py	80.50%	31 Missing ⚠️
litellm/rust_bridge/runtime.py	97.80%	4 Missing ⚠️
litellm/llms/custom_httpx/llm_http_handler.py	81.81%	2 Missing ⚠️
...opic/experimental_pass_through/messages/handler.py	94.11%	1 Missing ⚠️
litellm/ocr/main.py	94.11%	1 Missing ⚠️
litellm/responses/main.py	94.11%	1 Missing ⚠️
litellm/rust_bridge/chat_completions.py	96.87%	1 Missing ⚠️
litellm/rust_bridge/provider_count_tokens.py	96.29%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[CLAassistant]

All committers have signed the CLA.

[yujonglee-berri]

@greptile-apps review

[yujonglee-berri]

The request/context layer is restored for review in #39941 after GitHub automatically marked this PR merged during branch reordering