Skip to content

test(e2e): move rust OCR e2e into llm_translation on the shared harness - #31488

Merged
mubashir1osmani merged 4 commits into
litellm_internal_stagingfrom
litellm_rust_ocr_e2e_llm_translation
Jun 28, 2026
Merged

test(e2e): move rust OCR e2e into llm_translation on the shared harness#31488
mubashir1osmani merged 4 commits into
litellm_internal_stagingfrom
litellm_rust_ocr_e2e_llm_translation

Conversation

@mubashir1osmani

Copy link
Copy Markdown
Collaborator

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • My PR passes all CI/CD checks (e.g., lint, format, unit tests)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

The rust OCR path is not built into the standard image (the Dockerfile installs the rust toolchain only to compile Python deps; the wheel build is uv_build and never compiles litellm-rust into litellm.rust_bridge._native), so a live curl proof requires a rust-enabled stage deployment where all five provider credentials exist. I verified the moved test locally as far as the environment allows:

  • ruff format + ruff check clean, basedpyright clean (0 errors) on the changed files
  • the suite collects all 7 cases and the markerless config-presence test passes without a proxy
  • building the bridge (cargo build -p litellm-python-bridge --release) and wiring it in made get_native_bridge() load with ocr/aocr, and a real litellm.ocr call reached the rust vertex path

Running the full parametrized suite over a live proxy is left for the rust-enabled stage deployment that carries Mistral, Azure, Azure Document Intelligence, and Vertex credentials

Type

🧹 Refactoring
✅ Test

Changes

The rust OCR smoke test lived under tests/e2e/gateway and spoke raw httpx with ad-hoc dataclasses, diverging from the rest of tests/e2e. This moves it to tests/e2e/llm_translation and rebuilds it on the shared harness so it reads like the other suites

The OCR route now goes through typed pydantic bodies in models.py (OcrDocument, OcrBody, OcrPage, OcrResponse) and a new Gateway.ocr() method that calls the shared transport, so no test touches httpx or requests directly. Outcomes come back as the Result tagged union handled with unwrap, proxy-backed cases carry the e2e marker, and the cases live in a TestRustOcrGateway class using the client and scoped_key fixtures

Behavior is preserved. The config-presence check still reads gateway/litellm-config.yml without a proxy, /model/info confirms the proxy loaded every rust-ocr deployment, and each provider case asserts a well-formed OCR document over /v1/ocr

This also adds tests/e2e/CONTRIBUTING.md documenting the end-to-end testing flow (the seven-step create -> configure -> act -> settle -> assert state -> assert enforcement -> teardown contract, the keep-cases-in-a-class style, and the shared-transport rules) so new features land with coverage that walks the feature like production does

The rust OCR smoke lived under tests/e2e/gateway and spoke raw httpx with
ad-hoc dataclasses, diverging from the rest of tests/e2e. Move it to
tests/e2e/llm_translation and rebuild it on the shared harness: typed pydantic
bodies in models.py (OcrDocument/OcrBody/OcrPage/OcrResponse), a Gateway.ocr()
route through the shared transport, Result/unwrap for outcomes, the e2e marker,
and the client/scoped_key fixtures. No test touches httpx or requests directly
now.

Behavior preserved: the config-presence check still reads gateway/litellm-config.yml
without a proxy, /model/info confirms the proxy loaded every rust-ocr deployment,
and each provider case asserts a well-formed OCR document over /v1/ocr.

Also add tests/e2e/CONTRIBUTING.md documenting the end-to-end testing flow so new
features land with coverage that walks the feature like production does.
@mubashir1osmani
mubashir1osmani marked this pull request as draft June 27, 2026 02:35
@codecov

codecov Bot commented Jun 27, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@greptile-apps

greptile-apps Bot commented Jun 27, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR moves Rust OCR e2e coverage onto the shared llm_translation harness. The main changes are:

  • Adds typed OCR request and response models in tests/e2e/models.py
  • Adds Gateway.ocr() so OCR calls use the shared transport
  • Replaces the old ad-hoc gateway OCR test with a class-based llm_translation suite
  • Adds e2e contributor guidance under tests/e2e/CONTRIBUTING.md

Confidence Score: 5/5

The changes are isolated to test harness code and documentation, with no product runtime behavior changed.

The moved OCR coverage preserves the existing scenarios while aligning them with the shared typed transport patterns, and the surrounding model and gateway additions are narrowly scoped to the test suite.

T-Rex T-Rex Logs

What T-Rex did

  • Performed the OCR harness migration pre-check, which failed due to missing pytest and showed initial gateway attributes such as base_test_exists_in_gateway: True and imports_httpx: True, along with ad-hoc dataclass usage and the absence of OcrBody, OcrDocument, and OcrResponse.
  • Performed the OCR harness migration after-check, where pytest collection/config-test commands still failed due to missing pytest, but fallback checks reported simulated_collected_test_cases: 7, five parametrized OCR cases, five matching config models, TestRustOcrGateway_e2e_marked: True, typed body imports present, gateway_ocr_call_present: True, and no_direct_httpx_or_requests: True.
  • Notes that script artifacts were included to enable reproducibility of the before/after command captures.
  • The pre-checker ran in /tmp/e2e-base and failed only on a missing file.
  • The after-checker ran in /home/user/repo and every required documentation claim passed.
  • The after-run script used for traceability was saved as an artifact.

View all artifacts

T-Rex Ran code and verified through T-Rex

Reviews (2): Last reviewed commit: "docs(e2e): add CLAUDE.md harness convent..." | Re-trigger Greptile

Comment thread tests/e2e/CONTRIBUTING.md Outdated
@@ -0,0 +1,94 @@
# Contributing e2e tests

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Move docs out
This PR adds repository documentation under tests/e2e/CONTRIBUTING.md, but the repo rule says new documentation belongs in the litellm-docs repository instead. Keeping this guide here creates a second docs surface that will drift from the published contributor guidance; please move the e2e testing guide to litellm-docs or remove it from this PR.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Smd

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah this is silly

tests/e2e/CLAUDE.md captures the harness code-style rules (suite-as-a-class, shared transport, typed pydantic models, Result/unwrap, markers, typing) and the coverage-registry naming grammar; CONTRIBUTING.md gets the Contributors Guide intro and a Setup section
@mubashir1osmani
mubashir1osmani marked this pull request as ready for review June 28, 2026 01:00

@mateo-berri mateo-berri left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few nits but otherwise LGTM. Thanks!

Comment thread tests/e2e/CLAUDE.md
@@ -0,0 +1,132 @@
# e2e harness conventions

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the CLAUDE.md! A handwritten one is better but at least I like that you've specified rules for which folders go where. Making a meta rule of "if you add a new folder, you must add a new rule here to describe what kind of tests go there" is cool but this is quite good

Comment thread tests/e2e/CONTRIBUTING.md Outdated
Comment on lines +10 to +13
## Setup

1. Use litellm-config.yml in gateway/ if your tests require adding new models
2. Deploy a docker image in docker-compose.yml to run your e2e tests

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can you say "How to run locally" with the exact commands to use in a concise ordered list? Not just prose?

@mubashir1osmani
mubashir1osmani merged commit b443037 into litellm_internal_staging Jun 28, 2026
122 checks passed
@mubashir1osmani
mubashir1osmani deleted the litellm_rust_ocr_e2e_llm_translation branch June 28, 2026 15:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants