test(harness): add OCR parity with migration strategy runners

[yujonglee-berri]

TLDR

Problem this solves:

• OCR parity needs repeatable provider recordings
• Harness runners differ from the agreed migration spec

How it solves it:

• Add recorded OCR SDK parity and shared comparison tools
• Follow docs: define Rust Python harness structure #39456's strategy and shared-library layout
• Run strategies independently and combine their reports
• Validate unit mappings, backend selection, and native test results
• Preserve staging ledger validation across all strategy runners

User Flow

Before: a developer cannot select the agreed migration strategies

1. Run uv run python -m tests.rust-python-harness --list
2. Observe the older strategy names alongside existing_e2e_test_sdk
3. Try running the E2E strategy independently and observe a missing-module error

After: a developer can select and run each strategy independently

1. Run uv run python -m tests.rust-python-harness --list
2. Observe e2e_parity, trace_parity, unit_tests, and existing_e2e_test_sdk
3. Run uv run python -m tests.rust-python-harness.strategies.e2e_parity.runner --function ocr --plain and inspect the parity report

Relevant issues

Implements the harness organization and runner responsibilities documented in #39456. First PR in the two-PR stack, based on litellm_internal_staging; #39425 adds the expanded provider corpus and recording improvements

OCR execution and all 31 original cassettes live under strategies/e2e_parity/sdk/ocr/. Shared parity, tracing, and reporting tools live under shared/. E2E and trace runners accept SDK and gateway cases through the same orchestration. Existing Python tests remain at their original locations, and native Rust tests remain beside their implementation

The unit strategy combines collected Python/Rust test mapping, separate Python processes with Rust disabled and enabled, backend probes, outcome comparison, and Cargo execution. Missing or ambiguous counterparts, backend mismatches, failing tests, and skipped native tests cannot count as passing evidence. Trace comparison checks mapped operations, counts, and required execution order

The staging OCR ledger is retained and synchronized with this branch: 12 of 146 Python tests have Rust mappings, and 32 Rust tests are tracked separately. Ledger validation remains available through --validate-ledger, including from each independent runner

Trace instrumentation, gateway cases, and complete API unit mappings remain unconfigured and visibly planned. Existing Messages, Responses, Chat Completions, and Transcription bridge coverage remains partial. No assertion or expected-failure marker was weakened to hide OCR mismatches

The existing live SDK test row from #39463 is preserved under strategies/existing_e2e_test_sdk/, with its own runner. Chat Completions and Transcription selectors and recursive directory matching are preserved

The harness runs locally only. The existing CI allowlist covers it; no CI execution was added. Production Python remains unchanged relative to staging. Existing Rust OCR changes preserve upstream errors and validation details

Linear ticket

Resolves LIT-6523

Pre-Submission checklist

• I have added meaningful tests
• All affected parity tests pass locally
• My PR passes all required CI/CD checks
• My PR's scope is isolated to the migration harness
• I have received a current Greptile Confidence Score of at least 4/5

Validation

Merged staging at bcd3e2d94d, including #39463. The resolution preserves the new live SDK test row and six SDK function columns while retaining independent strategy execution and the upstream ledger audit

All 110 focused harness, ledger, Python/Cargo runner, reporting, trace, and CrowdStrike tests pass locally. The directory-selector regression exercises both parity and existing-SDK runners in separate processes. Ruff test checks and whitespace validation pass

The previous proxy-endpoints failure remains covered by staging's #39467. New CI checks are running for this merge. The previous Docker image jobs failed on the UI import of autorouter_presets.json; this merge does not change the Docker build setup

All 31 original cassette blobs are unchanged. The expanded OCR parity suite was not rerun for this merge. Earlier native-extension testing reported eight OCR parity failures and 26 existing expected failures; this update does not claim those provider mismatches are fixed

Screenshots / Proof of Fix

Run these commands from the repository root using the project Python environment

Before (bcd3e2d)

1. Run python -m tests.rust-python-harness --list
2. Observe e2e_fuzz_tests, unit_tests_rust, validate_sub_methods, and existing_e2e_test_sdk
3. Run python -m tests.rust-python-harness.strategies.e2e_parity.runner --help
4. Observe ModuleNotFoundError: No module named 'tests.rust-python-harness.strategies.e2e_parity'

After (0e4232b)

1. Run python -m tests.rust-python-harness --list
2. Observe e2e_parity, trace_parity, unit_tests, and existing_e2e_test_sdk, including Chat Completions and Transcription
3. Run python -m tests.rust-python-harness.strategies.e2e_parity.runner --help
4. Observe the independent runner options, including --surface and --validate-ledger

Type

Test

Refactoring

Caveats (if any)

Medium

• Existing OCR validation and provider-error parity mismatches remain
• Reducto retains its existing expected failures
• Native extension reuse limits implementation-level validation

Low

• Trace instrumentation and gateway cases remain planned
• Complete API unit-test mappings remain planned
• Native OCR stays disabled by default

Final Attestation

• Every provider case is proven equivalent end to end

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_ocr_sdk_parity_tests (92b52ae) with litellm_internal_staging (4990f06)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (bd9de39) during the generation of this report, so 4990f06 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[greptile-apps]

Greptile Summary

Adds a reusable recorded-fixture parity harness for comparing Python and accelerated OCR request and response behavior

• Records and replays provider responses through local test servers
• Runs sync and async OCR fixtures through persistent Python and accelerated workers
• Adds deterministic fixture generation, validation, caching, and CI coverage

Confidence Score: 5/5

The PR appears safe to merge

No blocking failure remains

Important Files Changed

Filename	Overview
tests/test_litellm/ocr/test_sdk_parity.py	Exercises both sync and async OCR routes through isolated workers and validates request and normalized-response parity
tests/route_parity/compare.py	Verifies accelerated execution did not use the Python HTTP fallback before comparing requests and responses
tests/route_parity/runner.py	Manages persistent parity worker subprocesses with bounded execution and cleanup
tests/route_parity/fixture_recorder.py	Generates deterministic inputs and records versioned provider-response fixtures through a local proxy
tests/route_parity/replay.py	Replays recorded HTTP responses while capturing provider requests for parity assertions

_{Reviews (2): Last reviewed commit: "test(ocr): preserve explicit model fixtu..." | Re-trigger Greptile}

[ishaan-berri]

nits

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 3 · PR risk: 0/10

[ishaan-berri]

• So follow up here: the tests should not run on CI. Make a folder called Rust Python harness tests.
• Beyond this, you should have a base class for each testing strategy that we're using, and then for each provider or endpoint, we just reuse that same base class.

[CLAassistant]

All committers have signed the CLA.