[Model] Support Unlimited OCR

[gty111]

Purpose

Support https://huggingface.co/baidu/Unlimited-OCR

Test

OmniDocBench

Metric	FA4	FlexAttention	Paper v1.6
Overall ↑	92.12	92.38	93.92
Text Edit ↓	0.089	0.087	0.042
Formula CDM ↑	95.34	95.53	95.79
Formula Edit ↓	0.108	0.105	—
Table TEDS ↑	89.90	90.33	90.16
Table TEDS-S ↑	93.27	93.60	93.32
Read-order Edit ↓	0.143	0.143	0.129

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

[mergify]

Documentation preview: https://vllm--46564.org.readthedocs.build/en/46564/

[MurphyYin]

Hi, I’m from the Unlimited-OCR team. Could you clarify whether, in the OmniDocBench v1.5 evaluation, the coordinate information between <|det|> and <|\det|> was kept and the full output was used directly for evaluation?

[gty111]

Hi, I’m from the Unlimited-OCR team. Could you clarify whether, in the OmniDocBench v1.5 evaluation, the coordinate information between <|det|> and <|\det|> was kept and the full output was used directly for evaluation?

Hi, thanks for reaching out!

Short answer: No, the coordinates were not kept. We don't feed the raw output into the evaluator. Each prediction first goes through a to_markdown step (same as the official Unlimited-OCR / DeepSeek-OCR post-processing) that strips all <|ref|> / <|det|>…<|/det|> markers and their coordinates, keeping only the recognized text. The cleaned markdown is what OmniDocBench v1.5 scores.

Example — raw output:

<|det|>title [120, 45, 480, 78]<|/det|>Annual Financial Report 2025
<|ref|>image<|/ref|><|det|>[100, 150, 400, 350]<|/det|>
<|det|>text [88, 370, 510, 405]<|/det|>Revenue grew by 12% year over year.

After to_markdown (this is what gets evaluated):

Annual Financial Report 2025
![](images/0.jpg)
Revenue grew by 12% year over year.

Notes:

• Bboxes are only used for marker removal, never scored.
• Image regions → ![](images/N.jpg); reading order is taken as emitted, not re-sorted by coordinates.

If your v1.5 protocol expects coordinates to be retained or a different post-process, please share it and we'll re-run to match.

[gty111]

Thanks for the PR! I made a quick pass and I have one quick question on the KV cache management part: The current implementation reuses FullAttentionSpec and its manager, while I feel RSWA looks more similar to SlidingWindowSpec. So I was wondering if we should make it a subclass of SlidingWindowSpec/Manager instead. Would like to understand the high-level idea better.

@ivanium
We chose FullAttentionSpec as the base because R-SWA is full attention over the prefix plus a sliding window over the generated tokens — the prefix (image + prompt) stays globally visible to every decode token, and only the decode region is windowed. FullAttentionManager's defaults already match this (prefix is never evicted, pool sizing uses the full-attention worst-case bound), so RSWAManager just adds remove_gap_blocks to keep the runtime held set bounded at prefix + window.

Inheriting from SlidingWindowSpec wouldn't actually reuse anything, because the pattern is the opposite: it windows the whole sequence and evicts the head, whereas R-SWA must keep the head and only window the decode region. We'd end up overriding all of its core (max_memory_usage_bytes, remove_skipped_blocks, find_longest_cache_hit), so there's nothing left to gain.

[ivanium]

Thanks for the PR! I made a quick pass and I have one quick question on the KV cache management part: The current implementation reuses FullAttentionSpec and its manager, while I feel RSWA looks more similar to SlidingWindowSpec. So I was wondering if we should make it a subclass of SlidingWindowSpec/Manager instead. Would like to understand the high-level idea better.

@ivanium We chose FullAttentionSpec as the base because R-SWA is full attention over the prefix plus a sliding window over the generated tokens — the prefix (image + prompt) stays globally visible to every decode token, and only the decode region is windowed. FullAttentionManager's defaults already match this (prefix is never evicted, pool sizing uses the full-attention worst-case bound), so RSWAManager just adds remove_gap_blocks to keep the runtime held set bounded at prefix + window.

Inheriting from SlidingWindowSpec wouldn't actually reuse anything, because the pattern is the opposite: it windows the whole sequence and evicts the head, whereas R-SWA must keep the head and only window the decode region. We'd end up overriding all of its core (max_memory_usage_bytes, remove_skipped_blocks, find_longest_cache_hit), so there's nothing left to gain.

Thanks for the detailed reply! I will make a pass now.

[ivanium]

Thanks for the reply and updates! I made another pass with a few questions on the kv cache manager part

[mergify]

Hi @gty111, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

[njhill]

Thanks @gty111. Model runner changes look fine to me but could we ensure that it also works with Model Runner V2? We're hoping to deprecate Model Runner V1 soon.

[gty111]

Thanks @gty111. Model runner changes look fine to me but could we ensure that it also works with Model Runner V2? We're hoping to deprecate Model Runner V1 soon.

Thanks for the review. R-SWA is now wired through Model Runner V2 as well: V2 passes per-request prompt_len as rswa_prefix_lens into the shared FlashAttention metadata builders (same as V1’s num_prompt_tokens path).

One caveat: Unlimited-OCR still needs NGramPerReqLogitsProcessor for good long-document OCR, and V2 does not support custom logits processors yet — production is recommended to use V1 until that lands. We’ve smoke-tested V2 on the 14-page PDF merge without the ngram processor; R-SWA itself works on V2.

[ywang96]

@gty111 Thanks for the great work and the collaboration with Baidu team @MurphyYin 🤝

[AndreasKaratzas]

I think that this PR broke:

• https://buildkite.com/vllm/amd-ci/builds/10179/list?tab=output&jid=019f103d-8d39-4bd3-bd98-318bae39113c#L2645

Will be pushing a fix soon.

PS
Test group name is weird cause it belongs in a custom branch, but I expect the failure to appear in our nightly as well.