Repository navigation
feat: add LiteLLM Rust workspace with Mistral OCR bridge #31033
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
e58a561
a72414a
79b4578
35f6961
06f6cfc
a021a5b
5be0797
fc7538f
8f41fdb
ccd8c41
260f5f1
a505734
d0ff175
daba177
1a18184
8ba452c
2c56b4d
65890b7
97f067b
99e1a76
7cf1b26
9608dd5
7f57a7a
343e453
dcf1b44
70afb75
6a13510
fea5204
6b83353
b86d316
74b1e6d
ba8c541
055bef8
0ce378d
9240cad
8feefec
c697154
b370c1c
d7ce63f
d2339a9
100861f
c6aa060
6fee78a
8325d52
54577a6
3188f93
88a9e6e
a4b4965
d6b9928
fe1cbe5
bbbb9ed
c9b712c
718b1d5
17791a2
ca904f5
53f059a
a21a807
097bab9
657c062
39d132f
b126fbb
b87dcb2
f99f0cd
7d6f037
8a40dcd
e70e3d6
8f30534
9b52a03
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| name: LiteLLM Rust | ||
|
|
||
| on: | ||
| push: | ||
| paths: | ||
| - "litellm-rust/**" | ||
| - ".github/workflows/test-rust.yml" | ||
| pull_request: | ||
| branches: | ||
| - main | ||
| - litellm_internal_staging | ||
| - litellm_oss_branch | ||
| - "litellm_**" | ||
| paths: | ||
| - "litellm-rust/**" | ||
| - ".github/workflows/test-rust.yml" | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }} | ||
| cancel-in-progress: true | ||
|
|
||
| jobs: | ||
| rust-checks: | ||
| name: rustfmt, clippy, test | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 10 | ||
| defaults: | ||
| run: | ||
| working-directory: litellm-rust | ||
| env: | ||
| CARGO_TERM_COLOR: always | ||
|
|
||
| steps: | ||
| - name: Checkout repository | ||
| uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0 | ||
| with: | ||
| persist-credentials: false | ||
|
|
||
| - name: Set up Rust | ||
| run: | | ||
| rustup toolchain install stable --profile minimal --component clippy,rustfmt | ||
| rustup default stable | ||
|
|
||
| - name: Cache Cargo registry and target | ||
| uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 | ||
| with: | ||
| path: | | ||
| ~/.cargo/registry | ||
| ~/.cargo/git | ||
| litellm-rust/target | ||
| key: ${{ runner.os }}-cargo-${{ hashFiles('litellm-rust/Cargo.lock') }} | ||
| restore-keys: | | ||
| ${{ runner.os }}-cargo- | ||
|
|
||
| - name: Check Rust formatting | ||
| run: cargo fmt --check | ||
|
Comment on lines
+54
to
+59
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
The cache step hashes |
||
|
|
||
| - name: Run Clippy | ||
| run: cargo clippy --workspace --all-targets --locked -- -D warnings | ||
|
|
||
| - name: Run Rust tests | ||
| run: cargo test --workspace --locked | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| /target/ |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,88 @@ | ||
| # CLAUDE.md | ||
|
|
||
| This file defines the rules for Rust work in LiteLLM. | ||
|
|
||
| ## Core Boundary | ||
|
|
||
| The `core` and `providers` crates describe work; hosts execute work. | ||
|
|
||
| Route-level Rust structure mirrors LiteLLM's Python responsibilities: | ||
| - `core/src/<route>/` owns the route contract, shared types, and provider | ||
| template traits. For OCR, this means `core/src/ocr`. | ||
| - `providers/src/<provider>/<route>/transformation.rs` owns the | ||
| provider-specific transform. For Mistral OCR, this means | ||
| `providers/src/mistral/ocr/transformation.rs`. | ||
| - Future network execution belongs in a host/transport layer such as | ||
| `llm_http_handler`, not inside `core` or `providers`. | ||
|
|
||
| Allowed in `core` and `providers`: | ||
| - Pure request transforms | ||
| - Pure response transforms | ||
| - Pure stream chunk normalization | ||
| - Shared data types and validation errors | ||
| - Deterministic token/cost helper logic | ||
|
|
||
| Not allowed in `core` or `providers`: | ||
| - Network calls | ||
| - Environment variable or secret reads | ||
| - Filesystem access | ||
| - Database or cache access | ||
| - Provider SDK signing or auth flows | ||
| - Logging callbacks, spend writes, or custom callbacks | ||
| - Global mutable runtime state | ||
|
|
||
| Python owns rollout state and fallback while Rust is being introduced. Rust | ||
| paths must be off by default until parity tests prove equivalence with Python. | ||
|
|
||
| ## Production Bar | ||
|
|
||
| Rust code in this workspace is held to a strict parity and robustness bar from | ||
| the first PR: | ||
|
|
||
| - Correctness parity is proven with tests. Do not rely on README claims or | ||
| manual inspection for a port that mirrors Python behavior. | ||
| - Every provider transform must have unit tests for supported-parameter | ||
| filtering, request body shape, response normalization, missing/null fields, | ||
| and bad-input errors. | ||
| - When Rust is exposed through Python, add Python tests that prove disabled, | ||
| enabled, and unavailable-bridge fallback behavior. | ||
| - Avoid panics on user/provider input. Return typed errors and let the host map | ||
| them to Python exceptions or HTTP responses. | ||
| - OCR handles documents that often contain personal data. Do not log document | ||
| contents, base64 payloads, provider response bodies, or secrets. | ||
| - Error messages must be useful but data-minimized. Truncate or sanitize any | ||
| upstream body before it crosses a host boundary. | ||
| - Treat empty or whitespace-only credentials, URLs, and config values as absent | ||
| at the host/config resolution layer. | ||
| - Preserve Python output shape intentionally. If a field is always serialized as | ||
| `null` for Python parity, leave a short comment explaining that parity choice. | ||
|
|
||
| ## Host I/O Rules | ||
|
|
||
| These rules apply when adding future crates or modules that execute network I/O, | ||
| such as `ai-gateway`, router hosts, or standalone servers: | ||
|
|
||
| - Set connect and full-request timeouts. No unbounded waits. | ||
| - Reuse HTTP clients; do not construct clients per request. | ||
| - Prefer rustls TLS for portable Python wheels and Linux images unless there is | ||
| a documented reason not to. | ||
| - Add request IDs and structured tracing at the host layer, without logging OCR | ||
| document contents or secrets. | ||
| - Do not echo raw upstream response bodies to callers. Sanitize and bound them. | ||
| - Avoid `expect`/`unwrap` in server startup and request paths unless the panic is | ||
| impossible by construction and documented. | ||
|
|
||
| ## Checks | ||
|
|
||
| Run these before pushing Rust changes. The same checks run in GitHub Actions | ||
| for changes under `litellm-rust/`. | ||
|
|
||
| ```bash | ||
| cd litellm-rust | ||
| cargo fmt --check | ||
| cargo clippy --workspace --all-targets -- -D warnings | ||
| cargo test --workspace | ||
| ``` | ||
|
|
||
| When a Rust path is exposed through Python, add Python parity tests that compare | ||
| the existing Python output with the Rust-backed output. |
Uh oh!
There was an error while loading. Please reload this page.