refactor(rust): move failure classification and request ownership into core, drop the unreachable bridge routes by yujonglee-berri · Pull Request #41745 · BerriAI/litellm

yujonglee-berri · 2026-09-18T03:20:55Z

TLDR

Problem this solves:

The bridge re-derived error meaning from every core error variant
Core request structs borrowed strings, forcing owned twins in callers
Native exceptions carried data through ad hoc attributes, decoded twice
The OCR route host imported package modules by path at call time
Two native routes were unreachable from Python

How it solves it:

Every core route error answers phase(): rejected, upstream, or after dispatch
Request structs own their fields; the timeout rule lives in core
Native exceptions carry (message, rejection) and (status, body, headers) in args
Python decodes a native failure once, in litellm.rust_bridge.failures
The OCR route host receives its Python helpers as an argument
The messages and chat completions bridge routes are removed

Stacked on #41733; the base is that PR's branch so the diff here is only these four commits

User Flow

Before: a developer whose app transcribes audio through Bedrock on the Rust route gets a bare RuntimeError when the provider throttles them, with no status to branch on

They call litellm.transcription(model="bedrock/mistral.voxtral-mini-3b-2507", file=audio) with the Rust route enabled
Bedrock answers 429 with a throttling body
Their code receives RuntimeError: upstream request failed with status 429: {...}, no status_code, and no litellm exception type to catch

After: the same throttle comes back as the litellm exception every other route raises

They call litellm.transcription(model="bedrock/mistral.voxtral-mini-3b-2507", file=audio) with the Rust route enabled
Bedrock answers 429 with a throttling body
Their code receives litellm.APIError with status_code == 429 and the provider body in the message

OCR callers see no change: the same public exceptions come back from the same provider answers, which the second proof case shows

Relevant issues

Affected release

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Shared setup: the native extension is built from the commit under test (cargo build --release -p litellm-python-bridge --features extension-module, copied to litellm/rust_bridge/_native.abi3.so). AZURE_AI_API_KEY and AZURE_AI_API_BASE come from .env. The OCR script is the end user's own code from #41733:

# sync_ocr_with_custom_logger.py
import base64, os, sys, urllib.request
import litellm
from litellm.integrations.custom_logger import CustomLogger

litellm.callbacks.append(CustomLogger())
pdf = urllib.request.urlopen("https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf").read()
response = litellm.ocr(
    model=sys.argv[1],
    document={"type": "document_url", "document_url": "data:application/pdf;base64," + base64.b64encode(pdf).decode()},
    api_key=os.environ["AZURE_AI_API_KEY"],
    api_base=os.environ["AZURE_AI_API_BASE"],
)
print(type(response).__name__, "pages:", len(response.pages))

No provider returns a throttle on demand, so the transcription case calls the native route directly against the wheel harness's recording server, which answers 429 for the x-test-outcome: 429 header. It uses the harness from tests/test_litellm/rust_bridge/native_route_wheel_test.py:

# transcription_429.py
import sys, threading
from http.server import ThreadingHTTPServer
from pathlib import Path
import native_route_wheel_test as harness

native = harness.load_native(Path(sys.argv[1]))
server = ThreadingHTTPServer(("127.0.0.1", 0), harness.NativeRouteHandler)
threading.Thread(target=server.serve_forever, daemon=True).start()
api_base = f"http://127.0.0.1:{server.server_address[1]}"
try:
    native.transcription(**harness.route_kwargs(api_base, "429"))
except Exception as error:
    print(f"{type(error).__name__}: args={error.args!r}")

Before (`e3d46f0`)

OCR against Azure AI keeps its public exception

LITELLM_RUST=1 python sync_ocr_with_custom_logger.py azure_ai/mistral-document-ai-2512
Output: the request reaches Azure AI, whose resource has no Document AI deployment, and the provider's 404 comes back. The trailing RuntimeError is the pre-existing sync-path bug listed under Caveats in refactor(rust): route-neutral callback contract, legacy Logging confined to callbacks-legacy #41733

_native.RustUpstreamError: (404, '{"error":{"code":"404","message": "Resource not found"}}')
  ...
litellm.exceptions.APIConnectionError: litellm.APIConnectionError: (404, '{"error":{"code":"404","message": "Resource not found"}}')
  ...
    raise NotFoundError(
  ...
RuntimeError: The request instance has not been set on this response.

Transcription on a provider 429

python transcription_429.py litellm/rust_bridge/_native.abi3.so
Output: a bare RuntimeError with the status inside the message string

RuntimeError: args=('upstream request failed with status 429: {"error":"native-rate-limit"}',)

After (`dd7b1ba`)

OCR against Azure AI keeps its public exception

LITELLM_RUST=1 python sync_ocr_with_custom_logger.py azure_ai/mistral-document-ai-2512
Output: the same request, the same 404 from Azure AI, and the same public path; the native exception now carries the provider headers and its message is the provider body rather than a tuple

_native.RustUpstreamError: (404, '{"error":{"code":"404","message": "Resource not found"}}', [('content-type', 'application/json'), ('date', ...), ('content-length', '56'), ('apim-request-id', ...), ...])
  ...
litellm.exceptions.APIConnectionError: litellm.APIConnectionError: {"error":{"code":"404","message": "Resource not found"}}
  ...
    raise NotFoundError(
  ...
RuntimeError: The request instance has not been set on this response.

Transcription on a provider 429

python transcription_429.py litellm/rust_bridge/_native.abi3.so
Output: the typed upstream exception with status, body and headers, which the Python runtime turns into litellm.APIError(status_code=429) for litellm.transcription callers

RustUpstreamError: args=(429, '{"error":"native-rate-limit"}', [])

Type

🧹 Refactoring

Caveats (if any)

Medium

Transcription now raises RustBridgeDeclined and RustUpstreamError instead of ValueError and RuntimeError. The Python runtime already handled both types for this route, so litellm.transcription callers get litellm exceptions where they previously got bare ones
OCR rejections named unsupported still map to APIConnectionError. Letting them fall back to the Python OCR path is a one-line change in map_failure, left out on purpose so this PR changes no public OCR outcome

Low

Exception classes with typed fields need Python 3.12 under abi3 (pyo3 passes an empty tuple to a native base), so the payload travels in args with the layout documented on each class and decoded once in litellm.rust_bridge.failures
The transcription proof runs against the harness's recording server because no provider throttles on demand; the OCR case is the real provider call
The OCR After run ends at the provider's 404 like the parent PR, and the trailing sync-path RuntimeError is the pre-existing bug noted there
The wheel harness's 32-way concurrency burst is flaky on macOS with socketserver's default listen backlog; CI runs it on Linux

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

🤖 Generated with Claude Code

…idge routes The Python side binds anthropic_messages, completion and their async twins, names the native module never exported, and the catalog only routes OCR and transcription to Rust. The flat-keyword messages and chat_completions functions in the bridge were therefore reachable only from the wheel smoke test, and if those routes return to Rust they return in the machine-driven shape, not this one. The wheel harness now exercises transcription for the sync, async, concurrency and SIGINT cases, and the route contract tests cover the same validation ordering through transcription's arguments. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…imeout rule MessagesRequest, ChatCompletionsRequest and AudioTranscriptionRequest borrowed their strings, so every async caller had to keep an owned twin alive across the await and destructure it back into borrows. The bridge's RouteOptions and per-route execute glue existed only for that. The request types now own their fields, and the rule that a non-positive or non-finite timeout means no timeout lives in litellm_core::timeout instead of the bridge. The transcription route now builds the core request inline from its parameters, and the websocket connection uses the same timeout rule. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…e typed pair of native exceptions Every route error in core now answers phase(): rejected before any provider I/O (with a named rejection), an upstream status with body and headers, or a failure after dispatch. The match is exhaustive per enum, so adding a variant forces the decision where the variant lives. The umbrella litellm_core::Error existed only so the bridge could re-derive this classification variant by variant; it is gone, along with is_request, is_response and the bridge's copies of the MissingApiKey rule. The bridge maps every route through the same function: file errors keep their Python types, other rejections raise RustBridgeDeclined with (message, rejection), and anything at or after the provider raises RustUpstreamError with (status, body, headers), status 0 when no response arrived. Subclassing a native exception with typed fields needs Python 3.12 under abi3, so the payload travels in args and Python decodes it once, in litellm.rust_bridge.failures. The OCR route host no longer sets status_code, message, headers or a format flag on exceptions and Python no longer reads them back; it builds the mapper's input from the decoded failure instead. Transcription now raises the same declined and upstream exceptions the Python runtime already handled for it, instead of ValueError and RuntimeError that fell through to the caller. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

… importing the package The OCR route host looked up litellm.rust_bridge.ocr.route_host and litellm.rust_bridge.timeouts by module path at call time, a back-edge from the extension into package internals that unit tests had to stub through sys.modules. Python now passes an OcrHostHelpers value (response, map_failure, timeout_to_seconds) as the hook's fourth argument, the host keeps that handle for the call and traverses it for GC, and the bridge imports nothing from the package. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

codspeed · 2026-09-18T03:23:46Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_bridge_policy_into_core (dd7b1ba) with main (db37977)¹}

No successful run was found on litellm_extract_callback_contracts (9d8c9b6) during the generation of this report, so main (db37977) was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

greptile-apps · 2026-09-18T03:25:42Z

Confidence Score: 3/5

The PR is not safe to merge until transport and OCR pre-dispatch failures preserve the correct public exception classifications

Findings

Transport Failures Lose Type ▶
Redirect Failures Misclassified ▶
Public Rust API Breaks ▶

Summary

This PR centralizes Rust route failure phases and timeout conversion, gives core request types ownership of their fields, injects OCR’s Python helpers, standardizes native exception payloads, and removes unused bridge exports

Native failures now carry structured rejection or upstream data
OCR response, timeout, and exception helpers are injected into the native host
Transcription provider failures now reach Python’s public exception mapping
Core request structs own strings instead of borrowing them
Unreachable messages and chat-completions extension exports are removed

_{Reviews (1) · Last reviewed commit: "refactor(rust): hand the OCR route host ..."}

greptile-apps · 2026-09-18T03:25:45Z

+                body,
+                headers: &[],
+            },
+            Self::Network(_) => Phase::AfterProvider,


Transport failures lose type Connection failures and timeouts become statusless upstream errors, so callers receive APIError(500) instead of normalized exceptions

Knowledge Base Used:

Model request execution

Fallbacks, retries, and failure handling

greptile-apps · 2026-09-18T03:25:45Z

+            | Self::MissingRedirectLocation
+            | Self::InvalidRedirect


Redirect failures misclassified These errors occur before provider dispatch, but this classification maps them as connection failures instead of request failures

Knowledge Base Used: Model request execution

greptile-apps · 2026-09-18T03:25:45Z

+pub struct AudioTranscriptionRequest {
+    pub model: String,
    pub audio: Value,
-    pub api_key: Option<&'a str>,
-    pub api_base: Option<&'a str>,
-    pub custom_llm_provider: Option<&'a str>,
+    pub api_key: Option<String>,
+    pub api_base: Option<String>,
+    pub custom_llm_provider: Option<String>,
    pub extra_headers: Option<Map<String, Value>>,
    pub optional_params: Map<String, Value>,
    pub timeout: Option<Duration>,


Public Rust API breaks Owned request fields and the removed Error export break callers, violating the requirement for user-controlled compatibility flags

Rule Used: What: avoid backwards-incompatible changes without user-controlled flags Why: This breaks current behaviour for users using existing functionality Example of BAD: this PR (#22164) introduced run_post_custom... (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

devin-ai-integration

Devin Review found 1 potential issue.

3 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

devin-ai-integration · 2026-09-18T03:25:54Z

audio handler

+        }
+    }


🟡 Connection failures block safe fallback

When connection setup fails, several handlers create Network, which reports AfterProvider. Hosts then treat an unsent request as potentially billed and suppress safe fallback.

Learn more

Phase::BeforeProvider means no provider I/O occurred, so another implementation can safely run the request. Reqwest reports failed DNS, TCP, and TLS connection setup through errors where is_connect() is true. The audio handler, messages handler, and OCR transport adapter convert those errors directly to Network. The new phase() implementation therefore loses dispatch certainty even though the Connect variant exists for this case.

Example: A Bedrock hostname fails DNS resolution. The request never leaves the process, but the handler creates Network, and phase() returns AfterProvider instead of BeforeProvider(Unreachable).

Recommended fix: Route pre-dispatch reqwest failures through Error::from_reqwest_before_dispatch in every HTTP handler. Keep response-body read failures and timeouts as Network, because those requests can already have reached the provider.

Was this helpful? React with 👍 or 👎 to provide feedback.

codecov · 2026-09-18T03:26:20Z

Codecov Report

❌ Patch coverage is 98.63014% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/rust_bridge/failures.py	96.15%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!


+def _mapper_input(error: Exception, failure: failures.Declined | failures.Upstream | None) -> Exception:
+    match failure:
+        case failures.Upstream(status=0, body=body):



+def _mapper_input(error: Exception, failure: failures.Declined | failures.Upstream | None) -> Exception:
+    match failure:
+        case failures.Upstream(status=0, body=body):


    return request.kwargs


+def _mapper_input(error: Exception, failure: failures.Declined | failures.Upstream | None) -> Exception:


yujonglee-berri · 2026-09-18T03:47:30Z

Folded into #41733 as one PR: its four commits are cherry-picked there, so this stacked branch is no longer needed.

@client

… a legacy Logging adapter Extracted from BerriAI#41733 without the router loop, the cache machine layer, streaming, or the error, timeout and route-pruning work that moved to BerriAI#41745 litellm-callbacks holds the contract a native call and its host share: Machine, HostOp, CallEvent, the in-process run loop, and Passthrough, which is built only by comparing the caller's inputs with the body the route sends, so a route can never mark a key it rewrote. litellm-host-python (formerly python-interop) owns the CPython driver and the Execution handle, and litellm-callbacks-legacy is the @client wrapper as the native call sees it: function_setup, the deployment hooks, pre_call and post_call, the success and failure fan-out and the deferred proxy release. OCR is the one route on it, and the old core and bridge lifecycles are gone The passthrough rule is the structural fix for the bug BerriAI#41719 patched in core and BerriAI#41716 reworks: an inlined remote document no longer counts as the caller's value, so the legacy adapter never hands the caller's URL back into the body. core/tests/ocr/passthrough.rs pins it for every route and document source, including that unchanged values stay passthrough, and callbacks-legacy/tests/payload.rs pins the adapter side with a real pre_call callback Python OCR integration tests that only exercised core behavior now live as Rust tests, so tests/test_litellm_rust keeps the cases that need the full Python stack

yujonglee-berri and others added 4 commits September 17, 2026 20:01

yujonglee-berri added this pull request to stack #41746 September 18, 2026 03:21

greptile-apps Bot reviewed Sep 18, 2026

View reviewed changes

devin-ai-integration Bot reviewed Sep 18, 2026

View reviewed changes

github-advanced-security Bot found potential problems Sep 18, 2026

View reviewed changes

yujonglee-berri closed this Sep 18, 2026

yujonglee-berri mentioned this pull request Sep 18, 2026

refactor(rust): route-neutral callback contract, legacy Logging confined to callbacks-legacy #41733

Closed

3 of 6 tasks

Navigation Menu

Repository navigation

refactor(rust): move failure classification and request ownership into core, drop the unreachable bridge routes - #41745

refactor(rust): move failure classification and request ownership into core, drop the unreachable bridge routes#41745
yujonglee-berri wants to merge 4 commits into
litellm_extract_callback_contractsfrom
litellm_bridge_policy_into_core

yujonglee-berri commented Sep 18, 2026 •

edited by devin-ai-integration Bot

Loading

Uh oh!

codspeed Bot commented Sep 18, 2026

Uh oh!

greptile-apps Bot commented Sep 18, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot Sep 18, 2026

Uh oh!

greptile-apps Bot Sep 18, 2026

Uh oh!

greptile-apps Bot Sep 18, 2026

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

devin-ai-integration Bot Sep 18, 2026

Uh oh!

codecov Bot commented Sep 18, 2026 •

edited

Loading

Uh oh!

yujonglee-berri commented Sep 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		return request.kwargs


		def _mapper_input(error: Exception, failure: failures.Declined \| failures.Upstream \| None) -> Exception:

Conversation

yujonglee-berri commented Sep 18, 2026 • edited by devin-ai-integration Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Relevant issues

Affected release

Linear ticket

Pre-Submission checklist

Delays in PR merge?

Screenshots / Proof of Fix

Before (e3d46f0)

OCR against Azure AI keeps its public exception

Transcription on a provider 429

After (dd7b1ba)

OCR against Azure AI keeps its public exception

Transcription on a provider 429

Type

Caveats (if any)

Medium

Low

Final Attestation

Uh oh!

codspeed Bot commented Sep 18, 2026

Merging this PR will not alter performance

Footnotes

Uh oh!

greptile-apps Bot commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 3/5

Findings

Summary

Uh oh!

greptile-apps Bot Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Sep 18, 2026

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

yujonglee-berri commented Sep 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yujonglee-berri commented Sep 18, 2026 •

edited by devin-ai-integration Bot

Loading

Before (`e3d46f0`)

After (`dd7b1ba`)

greptile-apps Bot commented Sep 18, 2026 •

edited

Loading

codecov Bot commented Sep 18, 2026 •

edited

Loading