test(e2e): move rust OCR e2e into llm_translation on the shared harness - #31488
Conversation
The rust OCR smoke lived under tests/e2e/gateway and spoke raw httpx with ad-hoc dataclasses, diverging from the rest of tests/e2e. Move it to tests/e2e/llm_translation and rebuild it on the shared harness: typed pydantic bodies in models.py (OcrDocument/OcrBody/OcrPage/OcrResponse), a Gateway.ocr() route through the shared transport, Result/unwrap for outcomes, the e2e marker, and the client/scoped_key fixtures. No test touches httpx or requests directly now. Behavior preserved: the config-presence check still reads gateway/litellm-config.yml without a proxy, /model/info confirms the proxy loaded every rust-ocr deployment, and each provider case asserts a well-formed OCR document over /v1/ocr. Also add tests/e2e/CONTRIBUTING.md documenting the end-to-end testing flow so new features land with coverage that walks the feature like production does.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Greptile SummaryThis PR moves Rust OCR e2e coverage onto the shared
Confidence Score: 5/5The changes are isolated to test harness code and documentation, with no product runtime behavior changed. The moved OCR coverage preserves the existing scenarios while aligning them with the shared typed transport patterns, and the surrounding model and gateway additions are narrowly scoped to the test suite.
What T-Rex did
Reviews (2): Last reviewed commit: "docs(e2e): add CLAUDE.md harness convent..." | Re-trigger Greptile |
| @@ -0,0 +1,94 @@ | |||
| # Contributing e2e tests | |||
There was a problem hiding this comment.
Move docs out
This PR adds repository documentation under tests/e2e/CONTRIBUTING.md, but the repo rule says new documentation belongs in the litellm-docs repository instead. Keeping this guide here creates a second docs surface that will drift from the published contributor guidance; please move the e2e testing guide to litellm-docs or remove it from this PR.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
tests/e2e/CLAUDE.md captures the harness code-style rules (suite-as-a-class, shared transport, typed pydantic models, Result/unwrap, markers, typing) and the coverage-registry naming grammar; CONTRIBUTING.md gets the Contributors Guide intro and a Setup section
mateo-berri
left a comment
There was a problem hiding this comment.
A few nits but otherwise LGTM. Thanks!
| @@ -0,0 +1,132 @@ | |||
| # e2e harness conventions | |||
There was a problem hiding this comment.
thanks for the CLAUDE.md! A handwritten one is better but at least I like that you've specified rules for which folders go where. Making a meta rule of "if you add a new folder, you must add a new rule here to describe what kind of tests go there" is cool but this is quite good
| ## Setup | ||
|
|
||
| 1. Use litellm-config.yml in gateway/ if your tests require adding new models | ||
| 2. Deploy a docker image in docker-compose.yml to run your e2e tests |
There was a problem hiding this comment.
nit: can you say "How to run locally" with the exact commands to use in a concise ordered list? Not just prose?
Relevant issues
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewScreenshots / Proof of Fix
The rust OCR path is not built into the standard image (the Dockerfile installs the rust toolchain only to compile Python deps; the wheel build is uv_build and never compiles litellm-rust into
litellm.rust_bridge._native), so a live curl proof requires a rust-enabled stage deployment where all five provider credentials exist. I verified the moved test locally as far as the environment allows:ruff format+ruff checkclean,basedpyrightclean (0 errors) on the changed filescargo build -p litellm-python-bridge --release) and wiring it in madeget_native_bridge()load withocr/aocr, and a reallitellm.ocrcall reached the rust vertex pathRunning the full parametrized suite over a live proxy is left for the rust-enabled stage deployment that carries Mistral, Azure, Azure Document Intelligence, and Vertex credentials
Type
🧹 Refactoring
✅ Test
Changes
The rust OCR smoke test lived under
tests/e2e/gatewayand spoke rawhttpxwith ad-hoc dataclasses, diverging from the rest oftests/e2e. This moves it totests/e2e/llm_translationand rebuilds it on the shared harness so it reads like the other suitesThe OCR route now goes through typed pydantic bodies in
models.py(OcrDocument,OcrBody,OcrPage,OcrResponse) and a newGateway.ocr()method that calls the sharedtransport, so no test toucheshttpxorrequestsdirectly. Outcomes come back as theResulttagged union handled withunwrap, proxy-backed cases carry thee2emarker, and the cases live in aTestRustOcrGatewayclass using theclientandscoped_keyfixturesBehavior is preserved. The config-presence check still reads
gateway/litellm-config.ymlwithout a proxy,/model/infoconfirms the proxy loaded every rust-ocr deployment, and each provider case asserts a well-formed OCR document over/v1/ocrThis also adds
tests/e2e/CONTRIBUTING.mddocumenting the end-to-end testing flow (the seven-step create -> configure -> act -> settle -> assert state -> assert enforcement -> teardown contract, the keep-cases-in-a-class style, and the shared-transport rules) so new features land with coverage that walks the feature like production does