feat: Async Rust OCR Bridge and MCP OAuth UI Restore - #31453
Conversation
[Infra] Promote internal staging to main
[Infra] Promote internal staging to main
chore(ci): promote internal staging to main
chore(ci): promote internal staging to main
chore(ci): promote internal staging to main
chore(ci): promote internal staging to main
chore(ci): promote internal staging to main
chore(ci): promote internal staging to main
chore(ci): promote internal staging to main
Adds a quickstart for the two published Terraform modules on the public registry (BerriAI/litellm/aws and BerriAI/litellm/google). Copy-paste main.tf for each cloud, the one-time GCP Artifact Registry remote-repo command, and pointers to the registry pages for the full input surface. Sits inside the Get Started section, between the gateway/SDK table and Run in Developer Mode -- where someone scanning the README for "how do I deploy this" will land. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
GCP gets the real 1-click: Open in Cloud Shell badge that clones the repo and walks through `terraform apply` via the existing DeployStack tutorial (already shipped at terraform/litellm/gcp/examples/default/ TUTORIAL.md). User just picks a project. AWS gets a soft 1-click: a Launch in AWS CloudShell badge that opens an in-browser, already-authenticated shell. User runs four commands (clone + cd + cp tfvars + terraform apply) once inside. There's no native AWS deeplink that pre-clones a repo + runs a tutorial -- CFN "Launch Stack" + CodeBuild would be needed for that, and that's a separate piece of work. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…on font GitHub rewrites an image's height attribute to "height: auto; max-height: Npx", which only caps and never stretches, so each image renders at its intrinsic height. The AWS/GCP shields badges are intrinsically 28px while the Render/Railway buttons are 40px, leaving the row uneven regardless of the height="48" we set. Replace the two shields badges with committed 40px PNGs so all four header buttons render at the same 40px. Also swap the Cloud Shell button from open-btn.svg to open-btn.png. The SVG renders its label as live text with font-family "Roboto, Sans" and no generic fallback; since neither font exists in GitHub's render environment, the text fell back to a serif (Times New Roman). The PNG bakes in the correct typeface.
The Railway button wrapped its img across indented lines, so the anchor contained leading and trailing whitespace. GitHub underlines link content, rendering that whitespace as a small blue underline beside the button. Put the anchor on one line like the other three buttons so there is no inner whitespace to underline.
Backports only the model map changes from BerriAI#30064 so deployments on released litellm versions pick up Fable 5 pricing, context window, and the adaptive thinking flag through the hosted cost map fetch without upgrading. Includes the supports_sampling_params flag on the 28 Fable 5 / Opus 4.7 / Opus 4.8 entries (ignored by released code, read by the gating that ships with the next release) and the matching one-line schema declaration so the map validation test passes. https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm
…cost_map Add Claude Fable 5 cost map entries (data-only hotfix for the hosted map)
chore(ci): promote internal staging to main
chore(ci): promote internal staging to main
…c_rust feat: make rust OCR async-first
|
|
Greptile SummaryThis PR refactors the LiteLLM OCR infrastructure to support an async Rust bridge path alongside the existing sync one, and introduces compile-time provider code generation from
Confidence Score: 3/5Do not merge until the PR description and title are reconciled with the actual changes — it currently claims to fix an MCP OAuth UI issue but contains only OCR and Rust bridge changes. The OCR refactoring itself is well-structured with shared request preparation, proper async Rust bridge, defensive getattr in the async loader, and comprehensive mock tests. The main blocker is that the PR metadata points to a completely different feature, so the claimed issue #31253 has no evidence of being resolved and the wrong issue may be closed on merge. litellm/ocr/main.py (PR description mismatch, core logic change); litellm/ocr/rust_bridge.py (minor inconsistency in load_rust_ocr attribute access); litellm-rust/crates/core/src/providers/README.md (new doc file per repo rules).
|
| Filename | Overview |
|---|---|
| litellm/ocr/main.py | Major refactor: extracted shared _prepare_ocr_request helper used by both ocr() and aocr(); aocr now has its own async implementation instead of delegating to the sync path via thread-pool; PR description does not match these changes. |
| litellm/ocr/rust_bridge.py | Added RustAocr protocol, load_rust_aocr(), and aocr param on use_litellm_rust(); load_rust_ocr() inconsistently uses direct attribute access instead of the defensive getattr pattern used by load_rust_aocr(). |
| litellm-rust/crates/providers/src/ocr.rs | Migrated from blocking to async reqwest client; added multi-provider dispatch via ocr_config_for; extra_headers support with case-insensitive Authorization de-duplication; Mistral URL/key resolution is still hardcoded with a TODO for future generalization. |
| litellm-rust/crates/python-bridge/src/lib.rs | Added aocr Python-callable async function alongside existing sync ocr; marshalling helpers extracted; block_on used for sync path via pyo3_async_runtimes::tokio::get_runtime(). |
| litellm-rust/crates/core/build.rs | New build script that generates the LlmProvider enum and static metadata from provider_endpoints_support.json at compile time; validates uniqueness and non-empty display names/URLs; includes self-consistency tests in generated code. |
| tests/test_litellm/ocr/test_rust_bridge.py | Significantly expanded with RecordingAsyncBridge, RaisingBridge/RaisingAsyncBridge, build_prepared_request factory, and new async test cases; all tests use mocks with no real network calls. |
| provider_endpoints_support.json | Added default_creds section with Mistral's default API base; added missing url fields for apertis, chutes, nanogpt, poe, and synthetic providers needed by the new build.rs. |
| litellm/llms/base_llm/ocr/transformation.py | Added get_api_key_env_var() hook to base class returning None; allows provider subclasses to expose their key env var name for use by the Rust bridge resolver. |
| litellm/llms/mistral/ocr/transformation.py | Introduced MISTRAL_OCR_API_KEY_ENV_VAR constant and overrides get_api_key_env_var() to return it; refactors inline string literal to use the constant. |
| litellm/proxy/ocr_endpoints/endpoints.py | Converted top-level import of convert_file_document_to_url_document / get_mime_type to a lazy local import inside _build_document_from_upload, breaking a potential import cycle. |
Comments Outside Diff (1)
-
litellm/ocr/main.py, line 694 (link)PR description does not match the code changes
The PR title is "feat(ui): restore mcp oauth credentials tab layout" and the description references closing feat: make rust OCR async-first #31253 (a UI issue about
MCPCredentialsTab.tsx). However, none of the 23 changed files contain any frontend or UI code — every change is OCR infrastructure: async Rust bridge,_prepare_ocr_requestextraction, provider code generation frombuild.rs, and related tests. There is noMCPCredentialsTab.tsxchange anywhere in the diff.Per the team rule, a PR claiming to fix a specific issue must include evidence that the issue is resolved. As written, this PR either has the wrong description or the wrong code changes attached to it.
Rule Used: What: Ensure that any PR claiming to fix an issue ... (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Reviews (1): Last reviewed commit: "fix(deps): bump langgraph-checkpoint to ..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| # Rust Provider Metadata | ||
|
|
||
| The repo-root `provider_endpoints_support.json` is the shared source of truth | ||
| for provider identity and docs metadata in `litellm-rust`. `crates/core/build.rs` | ||
| reads it at compile time and generates the typed `LlmProvider` enum plus static | ||
| provider metadata. Runtime code does not parse this JSON. | ||
|
|
||
| To add a provider: | ||
|
|
||
| - Add a `provider_endpoints_support.json` provider entry using the LiteLLM | ||
| provider slug, display name, docs URL, and endpoint support flags. | ||
| - Add optional defaults under the top-level `default_creds` map only when there | ||
| is a stable provider-level base URL. Keep route-specific key env var names in | ||
| the provider transform/config so auth resolution has one owner. | ||
| - Put request/response logic under | ||
| `crates/providers/src/<provider>/<route>/transformation.rs`; do not put | ||
| transforms, signing logic, or secrets in provider metadata. | ||
| - Run `cargo test -p litellm-core --locked`; it verifies the Rust registry stays | ||
| in parity with `provider_endpoints_support.json`. |
There was a problem hiding this comment.
crates/core/src/providers/README.md is a newly added documentation file. The team rule requires documentation to live in the litellm-docs repo rather than this one. If this is intended as developer-only contributing guidance for the Rust crate, please clarify — otherwise it should be moved to the docs repo or removed.
Rule Used: Prevent documentation from being added - needs to ... (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
| @@ -72,3 +98,14 @@ def load_rust_ocr() -> RustOcr | None: | |||
| except ImportError: | |||
| return None | |||
| return cast(RustOcr, litellm_python_bridge.ocr) | |||
There was a problem hiding this comment.
load_rust_ocr() accesses litellm_python_bridge.ocr as a direct attribute, which raises AttributeError if the compiled extension was built without that symbol. The newer load_rust_aocr() is defensive and uses getattr(..., "aocr", None). Applying the same pattern to load_rust_ocr() keeps both loaders consistent and prevents hard failures if the native extension is partially built or from an older version.
| return cast(RustOcr, litellm_python_bridge.ocr) | |
| return cast(RustOcr, getattr(litellm_python_bridge, "ocr", None)) |
ArjunPakhan
changed the title
mateo-berri
closed this pull request by merging all changes
into
BerriAI:litellm_internal_staging
in
41c3133
9 participants
Overview
This PR combines frontend layout restoration for the MCP OAuth credentials view alongside core backend updates transitioning the Rust OCR infrastructure to an async architecture.
Changes
MCPCredentialsTab.tsxlayout and stitched component states insidemcp_servers.tsx.aocr()as a native async function and extracted shared request preparation logic via_prepare_ocr_request.reqwest, added compile-time provider code generation incrates/core/build.rs, and introduced a defensivegetattrloader pattern to prevent attribute failures.Pre-Submission checklist
load_rust_ocrsafety.