refactor(native): discover providers at shared dispatch boundaries by yujonglee-berri · Pull Request #39939 · BerriAI/litellm

yujonglee-berri · 2026-09-05T20:00:13Z

TLDR

Problem this solves:

Generic admission rejected supported native OCR formats

How it solves it:

Make private admission pure and provider-aware

User Flow

Before: the SDK request can violate this layer contract.

They request Azure Document Intelligence native output.
A generic format rule declines the request.

After: the same request follows one predictable contract.

They send the same provider-specific request.
Pure admission accepts the selected provider native format.

Relevant issues

Stack root: refactor(rust): remove per-request enablement arguments #39928

Linear ticket

Pre-Submission checklist

I have added meaningful tests
Focused tests covering this change pass locally
All required CI/CD checks pass
This PR is isolated to one stack responsibility
Greptile Confidence Score is at least 4/5

Stack

39928 -> 39913 -> 39923 -> 39941 -> 39939 -> 39484 -> 39986 -> 40054 -> 40055 -> 40056 -> 39765

Layer 5 of 11.

Screenshots / Proof of Fix

After (`e5060f8838df1aa6ca2a54473e58cda191cebec2`)

Rust admission tests and strict Clippy: passed
UV_PYTHON=3.12 make check: passed
Final stack Rust workspace and strict Clippy: passed

No paid-provider calls were used for deterministic validation.

Type

Bug Fix

Caveats (if any)

Medium

Provider execution remains in later stack layers.

Final Attestation

Tests cover this layer intended contracts and edge cases

codspeed · 2026-09-05T20:34:56Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_native_chat_eligibility (35f3894) with litellm_native_request_context (905c5cb)¹}

No successful run was found on litellm_native_request_context (12c02dd) during the generation of this report, so c6c29bc was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

github-actions · 2026-09-05T20:36:13Z

LiteLLM Rust workflow

Workflow completed successfully for 792d8da10fd86cbfc7cc73c297d546e76329cbd1

View workflow run

codecov · 2026-09-05T20:51:57Z

Codecov Report

❌ Patch coverage is 94.82121% with 42 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/rust_bridge/runtime.py	94.11%	12 Missing ⚠️
litellm/main.py	86.25%	11 Missing ⚠️
litellm/llms/bedrock/chat/converse_handler.py	77.41%	7 Missing ⚠️
litellm/rust_bridge/transcription.py	93.75%	6 Missing ⚠️
...tellm/llms/bedrock/audio_transcription/__init__.py	81.81%	2 Missing ⚠️
litellm/llms/anthropic/chat/handler.py	91.66%	1 Missing ⚠️
...opic/experimental_pass_through/messages/handler.py	96.66%	1 Missing ⚠️
litellm/ocr/main.py	98.76%	1 Missing ⚠️
litellm/rust_bridge/responses_websocket.py	95.83%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

greptile-apps · 2026-09-06T14:23:24Z

Greptile Summary

This PR centralizes native-route admission and fallback dispatch for chat completions, messages, OCR, transcription, and Responses WebSockets

Adds provider-aware Rust preflight checks and shared Python dispatch primitives
Moves native preparation and fallback ownership to shared endpoint boundaries
Preserves Azure Document Intelligence native OCR data within the normalized response
Introduces two test-name regressions and repository-guideline violations in changed Python code

Confidence Score: 4/5

The dispatch refactor appears behaviorally sound, but explicit repository requirements and accidentally uncollected tests must be fixed before merging

No blocking runtime failure was established, but two tests silently stop running and changed provider handlers violate the repository’s source-comment and immutability requirements

Files Needing Attention: tests/test_litellm/llms/custom_httpx/test_llm_http_handler.py, litellm/llms/anthropic/chat/handler.py, litellm/llms/bedrock/chat/converse_handler.py

Important Files Changed

Filename	Overview
litellm/rust_bridge/runtime.py	Introduces shared endpoint selection, preflight, fallback, and upstream-error mapping
litellm/rust_bridge/chat_completions.py	Moves native chat discovery and request preparation to the shared completion boundary
litellm/rust_bridge/messages.py	Adds public Messages dispatch with provider-aware preflight and Python fallback
litellm/ocr/main.py	Moves OCR native preparation into the public dispatch path while supporting provider-native response preservation
litellm/rust_bridge/transcription.py	Adds shared transcription preparation and required-native handling for Bedrock
tests/test_litellm/llms/custom_httpx/test_llm_http_handler.py	Two renamed test functions no longer match pytest collection conventions
litellm/llms/anthropic/chat/handler.py	Removes local native dispatch but adds prohibited commentary and parameter rebinding
litellm/llms/bedrock/chat/converse_handler.py	Removes local native dispatch while introducing prohibited comments, parameter rebinding, and mutation

_{Reviews (1): Last reviewed commit: 7899a93 | Re-trigger Greptile}

greptile-apps · 2026-09-06T14:23:27Z



-def test_has_agentic_completion_hook_detection(monkeypatch):
+def testhas_agentic_completion_hook_detection(monkeypatch):


Tests are no longer collected

Both renamed functions miss pytest's test_ prefix, violating the requirement that modified tests preserve coverage and requiring correction before merge

Rule Used: What: Flag any modifications to existing tests and... (source)

greptile-apps · 2026-09-06T14:23:27Z

+        def build_request() -> tuple[dict, dict]:  # mutable-ok: rewritten in place downstream
+            """Translate the request the Python way, returning `(headers, data)`.
+
+            The pair stays mutable because the streaming path rewrites it in
+            place (`data["stream"] = True`) before sending.
+            """


Disallowed source commentary

The new request-building docstring and headings explain ordinary code, violating the repository's source-comment policy and requiring removal before merge

Context Used: CLAUDE.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

greptile-apps · 2026-09-06T14:23:27Z

-                python_client: Final = (
-                    client if isinstance(client, HTTPHandler) else _get_httpx_client(params={"timeout": timeout})
-                )
+                if client is None or not isinstance(client, HTTPHandler):


Parameters are rebound

The changed handlers reassign client and timeout and mutate _params, violating the repository's immutability requirement and requiring correction before merge

Context Used: CLAUDE.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

veria-ai · 2026-09-06T15:30:43Z

+            else initial_headers
+        )
+        log_details: Final[NativePreCallDetails] = {
+            "complete_input_dict": {"model": ctx.model, "messages": ctx.messages, **params},


High: Bedrock credentials are copied into pre-call logs

For Bedrock deployments, params contains transport fields such as aws_access_key_id, aws_secret_access_key, aws_session_token, and aws_web_identity_token. Expanding it into complete_input_dict sends those values verbatim to pre-call callbacks, so an authenticated user can trigger disclosure of the deployment's AWS credentials into configured observability systems. Build the logging payload from the provider request body or redact all transport-only fields first; the same issue occurs in litellm/rust_bridge/transcription.py:253.

veria-ai · 2026-09-06T15:30:53Z

PR overview

This PR refactors native request handling so providers are discovered at shared dispatch boundaries, including the Rust bridge paths for chat completions and transcription.

A significant issue remains: authenticated Bedrock requests can cause AWS credentials and session tokens to be copied into pre-call callback payloads and configured observability systems. No reported issues have yet been addressed, so the credential-handling problem remains unresolved.

Open issues (1)

High: Bedrock credentials are copied into pre-call logs — litellm/rust_bridge/chat_completions.py:446

Fixed/addressed: 0 · PR risk: 7/10

yujonglee-berri · 2026-09-16T18:04:51Z

Closing; stack is being cleaned up.

yujonglee-berri force-pushed the litellm_native_chat_eligibility branch from 9433cae to c979a20 Compare September 5, 2026 20:21

yujonglee-berri changed the base branch from codex/native-endpoint-adapters to litellm_native_request_context September 5, 2026 20:22

yujonglee-berri force-pushed the litellm_native_chat_eligibility branch from c979a20 to 7a101d4 Compare September 5, 2026 20:31

yujonglee-berri force-pushed the litellm_native_chat_eligibility branch from 7a101d4 to 405deac Compare September 5, 2026 21:19

yujonglee-berri changed the title ~~refactor(native): move chat eligibility checks into Rust~~ refactor(native): discover providers at shared dispatch boundaries Sep 5, 2026

yujonglee-berri mentioned this pull request Sep 5, 2026

refactor(native): unify bridge execution under a shared harness #39913

Closed

4 of 6 tasks

yujonglee-berri force-pushed the litellm_native_chat_eligibility branch from 405deac to 660722e Compare September 6, 2026 00:37

yujonglee-berri force-pushed the litellm_native_chat_eligibility branch from 660722e to 019080e Compare September 6, 2026 01:08

yujonglee-berri force-pushed the litellm_native_chat_eligibility branch from 019080e to 27a51f0 Compare September 6, 2026 01:49

yujonglee-berri force-pushed the litellm_native_chat_eligibility branch from 27a51f0 to 9e0832c Compare September 6, 2026 04:41

yujonglee-berri force-pushed the litellm_native_chat_eligibility branch from 9e0832c to 0ad7ae3 Compare September 6, 2026 06:19

yujonglee-berri force-pushed the litellm_native_chat_eligibility branch from 0ad7ae3 to 840963b Compare September 6, 2026 06:35

yujonglee-berri force-pushed the litellm_native_chat_eligibility branch from 840963b to 152d041 Compare September 6, 2026 07:55

yujonglee-berri force-pushed the litellm_native_chat_eligibility branch 2 times, most recently from d43fcc4 to 7899a93 Compare September 6, 2026 08:33

yujonglee-berri marked this pull request as ready for review September 6, 2026 14:14

yujonglee-berri force-pushed the litellm_native_chat_eligibility branch from 7899a93 to ae2e895 Compare September 6, 2026 14:21

yujonglee-berri requested a review from a team September 6, 2026 14:21

greptile-apps Bot reviewed Sep 6, 2026

View reviewed changes

yujonglee-berri force-pushed the litellm_native_chat_eligibility branch from ae2e895 to e5060f8 Compare September 6, 2026 14:32

yujonglee-berri force-pushed the litellm_native_chat_eligibility branch from e5060f8 to 792d8da Compare September 6, 2026 15:23

veria-ai Bot reviewed Sep 6, 2026

View reviewed changes

yujonglee-berri added 7 commits September 7, 2026 10:43

refactor(native): move chat eligibility checks into Rust

a278ee0

feat(rust): discover providers at shared SDK dispatch boundaries

f177246

refactor(native): centralize typed route eligibility

3d359ba

fix(native): preserve opaque metadata eligibility

242ab68

test(bedrock): use separated native options

3dd53c0

fix(native): restack shared eligibility contracts

7e8613c

fix(native): make OCR admission provider aware

35f3894

yujonglee-berri force-pushed the litellm_native_chat_eligibility branch from 792d8da to 35f3894 Compare September 7, 2026 17:43

yujonglee-berri closed this Sep 16, 2026

yujonglee-berri deleted the litellm_native_chat_eligibility branch September 16, 2026 18:05

Navigation Menu

Repository navigation

refactor(native): discover providers at shared dispatch boundaries - #39939

refactor(native): discover providers at shared dispatch boundaries#39939
yujonglee-berri wants to merge 7 commits into
litellm_native_request_contextfrom
litellm_native_chat_eligibility

yujonglee-berri commented Sep 5, 2026 •

edited

Loading

Uh oh!

codspeed Bot commented Sep 5, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 5, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Sep 5, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Sep 6, 2026

Important Files Changed

Uh oh!

greptile-apps Bot Sep 6, 2026

Uh oh!

greptile-apps Bot Sep 6, 2026

Uh oh!

greptile-apps Bot Sep 6, 2026

Uh oh!

veria-ai Bot Sep 6, 2026

Uh oh!

veria-ai Bot commented Sep 6, 2026

Uh oh!

yujonglee-berri commented Sep 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant



		def test_has_agentic_completion_hook_detection(monkeypatch):
		def testhas_agentic_completion_hook_detection(monkeypatch):

Conversation

yujonglee-berri commented Sep 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Relevant issues

Linear ticket

Pre-Submission checklist

Stack

Screenshots / Proof of Fix

After (e5060f8838df1aa6ca2a54473e58cda191cebec2)

Type

Caveats (if any)

Medium

Final Attestation

Uh oh!

codspeed Bot commented Sep 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

github-actions Bot commented Sep 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

LiteLLM Rust workflow

Uh oh!

codecov Bot commented Sep 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

greptile-apps Bot commented Sep 6, 2026

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Uh oh!

greptile-apps Bot Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

veria-ai Bot Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

veria-ai Bot commented Sep 6, 2026

PR overview

Open issues (1)

Uh oh!

yujonglee-berri commented Sep 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

yujonglee-berri commented Sep 5, 2026 •

edited

Loading

After (`e5060f8838df1aa6ca2a54473e58cda191cebec2`)

codspeed Bot commented Sep 5, 2026 •

edited

Loading

github-actions Bot commented Sep 5, 2026 •

edited

Loading

codecov Bot commented Sep 5, 2026 •

edited

Loading