fix(ocr): keep inlined remote documents when the host restores retained fields

[devin-ai-integration]

TLDR

Problem this solves:

• Azure AI OCR rejects every remote document_url since refactor(ocr): mirror Python provider layout and preserve tests #41550 ("invalid OCR document data URI")
• Vertex Mistral OCR and Azure cohere_parse remote documents break the same way
• CI's ocr_testing job has been red on every branch since dc81cf5

How it solves it:

• The Rust OCR path hands the caller's document back only when the body still sends it unchanged
  • Alternatives: a per-adapter "inlines documents" flag (drift-prone, which is how this regressed) and full JSON equality (the body decoder drops extra document fields, so Mistral documents with extra keys would stop being retained). Source-only comparison is the one that means "the provider got what the caller sent"
• Documents Azure AI, Vertex Mistral, and Azure cohere_parse inlined now reach the provider as data URIs
• Facade regression tests for all three inlining adapters, plus one proving Mistral still gets the caller's document
• A body document that no longer decodes is not retained (main retained on the bare key); no adapter produces one

User Flow

Before: a developer running OCR through the proxy against an Azure AI Foundry Mistral Document AI deployment sends a public PDF link and gets a 400 back, with no way to make the link work

1. The proxy admin adds azure_ai/mistral-document-ai-2512 to the config as azure-ocr with the Foundry api_base and key, and starts the proxy
2. The developer sends POST https://litellm-domain/v1/ocr with {"model":"azure-ocr","document":{"type":"document_url","document_url":"https://cdn.jsdelivr.net/gh/BerriAI/litellm@d769e81c/tests/llm_translation/fixtures/dummy.pdf"}}
3. They get HTTP 400 {"error":{"message":"litellm.BadRequestError: Azure_aiException - invalid OCR document data URI. Received Model Group=azure-ocr ...","type":"invalid_request_error","code":"400"}} back in about a second, and nothing in the error says what a valid document would look like
4. The same call with the PDF pasted in as data:application/pdf;base64,... returns 200, so the only workaround is downloading and base64-encoding every document client-side

After: the same PDF link is accepted and the developer gets the OCR pages back

1. The proxy admin adds azure_ai/mistral-document-ai-2512 to the config as azure-ocr with the Foundry api_base and key, and starts the proxy
2. The developer sends POST https://litellm-domain/v1/ocr with {"model":"azure-ocr","document":{"type":"document_url","document_url":"https://cdn.jsdelivr.net/gh/BerriAI/litellm@d769e81c/tests/llm_translation/fixtures/dummy.pdf"}}
3. They get HTTP 200 with "pages":[{"index":0,"markdown":"Test PDF File",...}], "usage_info":{"pages_processed":1,"doc_size_bytes":589} and "object":"ocr", the same response the data URI call returned
4. The data URI call keeps working as before, so nothing changes for clients that already inline documents themselves

Relevant issues

Regression from #41550

Affected release

Linear ticket

Resolves LIT-8058

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

/qa run at PR tip 75ca8af (merge base 4c70cb4). Every checkpoint below booted its own worktree venv the way CI's ocr_testing job builds it (uv sync --frozen --all-groups --all-extras --python 3.12, so the Rust OCR path is on, native_bridge_available() printed True at every checkpoint), started a no-DB proxy with --num_workers 2 on a random free port, and hit the real Azure AI Foundry mistral-document-ai-2512 deployment (real $$$). This path holds no per-process state, so a single two-worker instance per checkpoint is the topology. run_test.py loads the .env (AZURE_API_KEY, AZURE_API_BASE) and calls pytest.main with the rest of argv. Proxy config at every checkpoint:

model_list:
  - model_name: azure-ocr
    litellm_params:
      model: azure_ai/mistral-document-ai-2512
      api_key: os.environ/AZURE_FOUNDRY_API_KEY
      api_base: os.environ/AZURE_FOUNDRY_API_BASE

general_settings:
  master_key: sk-mat574-repro

Commit 6a9f6ff landed after this run and only rewrites four Rust tests plus their test-support builder (every hunk sits under #[cfg(test)]), so it cannot change behavior and the proof keeps the hash it ran at.

Regression table

Case 1 is POST /v1/ocr with a public PDF URL, case 2 the same PDF inlined as a data URI (control), case 3 CI's own tests/ocr_tests/test_ocr_azure_ai.py::TestAzureAIOCR::test_basic_ocr_with_url (both params).

Checkpoint	Commit	Port	Expected	Case 1 URL	Case 2 data URI	Case 3 CI test	Verdict
Before the regression	1e7b03a (first parent of #41550)	63455	URL accepted	200	200	2 passed	PASS
Regression	dc81cf5 (#41550 merge)	50375	URL rejected	400	200	2 failed	PASS
Before the fix	4c70cb4 (merge base)	63900	URL rejected	400, 400	200	2 failed	PASS
After the fix	75ca8af (PR tip)	50844	URL accepted	200, 200	200	2 passed	PASS

Before (4c70cb4, port 63900)

Case 1, run twice, byte-identical output both times:

curl -s -X POST http://localhost:63900/v1/ocr -H 'Authorization: Bearer sk-mat574-repro' -H 'Content-Type: application/json' -d '{"model":"azure-ocr","document":{"type":"document_url","document_url":"https://cdn.jsdelivr.net/gh/BerriAI/litellm@d769e81c90d453240c61fc572cdb27fae06a89d0/tests/llm_translation/fixtures/dummy.pdf"}}' -w '\nHTTP %{http_code}\n'

{"error":{"message":"litellm.BadRequestError: Azure_aiException - invalid OCR document data URI. Received Model Group=azure-ocr\nAvailable Model Group Fallbacks=None","type":"invalid_request_error","param":null,"code":"400"}}
HTTP 400

Case 2, the data URI control, so the deployment itself is healthy:

B64=$(base64 -i dummy.pdf | tr -d '\n')
curl -s -X POST http://localhost:63900/v1/ocr -H 'Authorization: Bearer sk-mat574-repro' -H 'Content-Type: application/json' -d "{\"model\":\"azure-ocr\",\"document\":{\"type\":\"document_url\",\"document_url\":\"data:application/pdf;base64,${B64}\"}}" -w '\nHTTP %{http_code}\n'

{"pages":[{"index":0,"markdown":"Test PDF File","images":[],"dimensions":{"dpi":93,"height":1023,"width":791},"footer":null,"header":null,"hyperlinks":[],"tables":[]}],"model":"azure-ocr","document_annotation":null,"usage_info":{"pages_processed":1,"pages_processed_annotation":0,"credits":null,"doc_size_bytes":589},"content":null,"tables":null,"keyValuePairs":null,"object":"ocr","content_filter_results":null}
HTTP 200

Case 3, CI's own test from inside the worktree:

.venv/bin/python run_test.py .env 'tests/ocr_tests/test_ocr_azure_ai.py::TestAzureAIOCR::test_basic_ocr_with_url' -p no:cacheprovider -vv

tests/ocr_tests/test_ocr_azure_ai.py::TestAzureAIOCR::test_basic_ocr_with_url[True] <- tests/ocr_tests/base_ocr_unit_tests.py FAILED [ 50%]
tests/ocr_tests/test_ocr_azure_ai.py::TestAzureAIOCR::test_basic_ocr_with_url[False] <- tests/ocr_tests/base_ocr_unit_tests.py FAILED [100%]
FAILED tests/ocr_tests/test_ocr_azure_ai.py::TestAzureAIOCR::test_basic_ocr_with_url[True] - Failed: OCR call failed: litellm.BadRequestError: Azure_aiException - invalid OCR document data URI
FAILED tests/ocr_tests/test_ocr_azure_ai.py::TestAzureAIOCR::test_basic_ocr_with_url[False] - Failed: OCR call failed: litellm.BadRequestError: Azure_aiException - invalid OCR document data URI
============================== 2 failed in 5.28s ===============================

After (75ca8af, port 50844)

Case 1, same command against port 50844, run twice, byte-identical output both times:

curl -s -X POST http://localhost:50844/v1/ocr -H 'Authorization: Bearer sk-mat574-repro' -H 'Content-Type: application/json' -d '{"model":"azure-ocr","document":{"type":"document_url","document_url":"https://cdn.jsdelivr.net/gh/BerriAI/litellm@d769e81c90d453240c61fc572cdb27fae06a89d0/tests/llm_translation/fixtures/dummy.pdf"}}' -w '\nHTTP %{http_code}\n'

{"pages":[{"index":0,"markdown":"Test PDF File","images":[],"dimensions":{"dpi":93,"height":1023,"width":791},"footer":null,"header":null,"hyperlinks":[],"tables":[]}],"model":"azure-ocr","document_annotation":null,"usage_info":{"pages_processed":1,"pages_processed_annotation":0,"credits":null,"doc_size_bytes":589},"content":null,"tables":null,"keyValuePairs":null,"object":"ocr","content_filter_results":null}
HTTP 200

usage_info.doc_size_bytes 589 matches the fetched dummy.pdf, so the provider received the inlined document, not the URL.

Case 2, the data URI control, same command against port 50844:

{"pages":[{"index":0,"markdown":"Test PDF File","images":[],"dimensions":{"dpi":93,"height":1023,"width":791},"footer":null,"header":null,"hyperlinks":[],"tables":[]}],"model":"azure-ocr","document_annotation":null,"usage_info":{"pages_processed":1,"pages_processed_annotation":0,"credits":null,"doc_size_bytes":589},"content":null,"tables":null,"keyValuePairs":null,"object":"ocr","content_filter_results":null}
HTTP 200

Case 3, CI's own test, same command:

tests/ocr_tests/test_ocr_azure_ai.py::TestAzureAIOCR::test_basic_ocr_with_url[True] <- tests/ocr_tests/base_ocr_unit_tests.py PASSED [ 50%]
tests/ocr_tests/test_ocr_azure_ai.py::TestAzureAIOCR::test_basic_ocr_with_url[False] <- tests/ocr_tests/base_ocr_unit_tests.py PASSED [100%]
============================== 2 passed in 4.53s ===============================

The two regression-side checkpoints (1e7b03a and dc81cf5) ran the same three cases with the same commands and produced the table rows above: the first parent of #41550 accepted the URL (200, "Test PDF File", 2 passed) and the #41550 merge rejected it with the same invalid OCR document data URI 400 the merge base shows, with the data URI control at 200 on both.

Surprises

• base_ocr_unit_tests.py:47 prints the api_key into captured stdout; PR leaves it alone
• Uvicorn access log carries no worker pid; PR leaves it alone
• Both proxies warn "Enterprise billing metrics disabled" on boot; PR leaves it alone
• Rust Vertex auth fails without project_id in credentials; identical both sides, pre-existing
• Three CircleCI jobs red on main's own pipeline too; PR leaves them alone

Type

🐛 Bug Fix

Caveats (if any)

Low

• Vertex Mistral OCR and Azure cohere_parse ride the same changed predicate but were proven through facade tests (real Rust adapters over HTTP to a local mock provider), not a live provider call. The predicate is provider-agnostic and the inlining those adapters do is untouched here, so the live legs would only re-prove unchanged provider behavior; closing them needs a Vertex project with mistral-ocr enabled, a fix for the pre-existing Rust Vertex auth gap (a credentials file without project_id fails before any call, identical on base and head, filed separately), and a cohere parse Foundry deployment, which outweighs what the gap costs
• pre_ocr and post_ocr logging payloads now carry the inlined data URI for remote documents on inlining providers, matching the wire body and the pre-refactor(ocr): mirror Python provider layout and preserve tests #41550 Python-path behavior; the URL is still what the caller sent and what non-inlining providers (Mistral) log. Verified live with a payload-recording callback on base, head, and head with LITELLM_RUST=0
• Azure cohere_parse already dropped extra document fields on the wire before this PR; unchanged here
• ci/circleci: litellm_utils_testing, llm_translation_testing, and pass_through_unit_testing are red on this PR and fail the same way on main's own scheduled pipeline; none of them run OCR code. Not required checks

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge, with the OCR retention fix covered across affected adapters and no outstanding actionable findings.

Summary

This PR corrects OCR document retention so host hooks restore the caller’s document only when the provider body still sends the same document source.

• Prevents original remote URLs from replacing data URIs generated by Azure AI, Azure Cohere Parse, and Vertex AI adapters.
• Preserves caller-supplied document fields when the document source is unchanged.
• Adds facade regression coverage for inlined remote documents and retained fields.
• Replaces direct test-request hook mutation with a construction helper, addressing the previous resolved review thread.

_{Reviews (2) · Last reviewed commit: "test(ocr): set retained-fields hooks at ..."}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 6a9f6ff. Configure here.}

[yujonglee-berri]

probably dupe of #41719, and also handled in #41752