mtmd: Add DeepSeekOCR 2 Support

[sfallah]

Overview

This PR adds support for DeepSeek-OCR-2 deepseek-ai/DeepSeek-OCR-2.

GGUF model files for testing this PR are available at sabafallah/DeepSeek-OCR-2-GGUF.

The implementation adds the DeepSeek-OCR-2 vision path that reuses SAM encoder (from DeepSeek-OCR v1), a new Qwen2-based vision encoder, and multi-tile dynamic-resolution image preprocessing.

The preprocessing includes multi-tile dynamic-resolution, with a 1024 global view and grid of tiles with 768 tile views.
The image resize/padding behavior is Pillow-based to match the original deepseek-ai implementation.

Main changes

• added DeepSeek-OCR-2 model support in mtmd
• reused the SAM implementation from DeepSeek-OCR where possible
• added the Qwen2 vision encoder
• added multi-tile dynamic-resolution preprocessing
• implemented the DeepSeek-OCR-2 attention mask handling for the Qwen2 encoder
• updated mtmd_encode to handle multi-tile and global view token counts diff
• added converter support for DeepSeek-OCR-2 vision weights
• extended tools/mtmd/tests/test-deepseek-ocr.py to cover both DeepSeek-OCR and DeepSeek-OCR-2

Implementation notes

• The SAM encoder is shared with DeepSeek-OCR through the existing build_sam.
• The Qwen2 attention mask is prepared CPU-side.
• The dynamic-resolution preprocessing follows the InternVL-style behavior used by the reference implementation.

Testing

• Extended tools/mtmd/tests/test-deepseek-ocr.py to cover both v1 and v2.
• DeepSeek-OCR-2 result: CER 0.6944 / chrF 23.01
• DeepSeek-OCR-2 HF reference: CER 0.7761 / chrF 28.70
• Manually verified with llama-mtmd-cli.
• Manually verified with llama-server.

Caveats

• The DeepSeek-OCR-2 test gate is intentionally loose (cer_tol = 0.12). The current test image is low quality, and the HF reference itself has a high CER on this sample.

• The DRY sampler is used as an approximation for the HF no_repeat_ngram_size behavior.

• For DeepSeek-OCR-2, llama-server currently requires --chat-template deepseek-ocr --no-jinja. This is due to the template handling for DeepSeek-OCR-2.

• Running llama-server:

  build/bin/llama-server \
  -m gguf_models/deepseek-ai/deepseek-ocr-2-bf16.gguf \
  --mmproj gguf_models/deepseek-ai/mmproj-deepseek-ocr-2-bf16.gguf \
  --chat-template deepseek-ocr --no-jinja \
  --temp 0 \
  --flash-attn off \
  --no-warmup \
  -n 2048 \
  --dry-multiplier 0.8 --dry-base 1.75 --dry-allowed-length 2 \
  --dry-penalty-last-n -1 --dry-sequence-breaker none

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - I used AI assistance for code review, debugging, implementation checks, and testing. I have reviewed the submitted changes and take responsibility for the full contents of this PR.

[ngxson]

you may also need to enable mtmd_decode_use_non_causal for this model, so that text is causal while image tokens are non-causal

[ngxson]

hmm, please ignore what I said earlier, it seems like their paper is poorly written and the v2 is not easier than v1.

basically it's a 3-step pipeline: SAM encode the image into embeddings, Qwen take embedding in and generate yet another embeddings, then finally they are fed into the main deepseek model

if my understanding is correct, that means the prompt will now be processed by vision encoder, which make things a whole lot more complicated

[ngxson]

small correction, seems like what they refers to "query" on the diagram is not the text prompts, but simply a set of pre-trained tokens (fixed size), so that won't be too complicated to implement

the only thing that need to be provided correctly to the cgraph is the attention mask that will correctly mask the image tokens as non-causal and the query as causal, it should not be complicated

and finally, because SAM is the same between 2 models it is recommended that you extract SAM as a function and inherit it in DS-OCR v2, example:

struct clip_graph_deepseekocr : clip_graph {
    clip_graph_deepseekocr(clip_ctx * ctx, const clip_image_f32 & img) : clip_graph(ctx, img) {}
    ggml_cgraph * build() override;
    ggml_cgraph * build_sam(); // build the SAM model
};

struct clip_graph_deepseekocr2 : clip_graph_deepseekocr { // inherit
    clip_graph_deepseekocr2(clip_ctx * ctx, const clip_image_f32 & img) : clip_graph(ctx, img) {}
    ggml_cgraph * build() override; // can directly reuse build_sam() from base class, no need to duplicate the code
};

[ngxson]

@sfallah can you check locally if test-llama-archs passes? the CI fails on that test but I'm not sure if it has something to do with this PR, since the test only runs the text model

[sfallah]

@sfallah can you check locally if test-llama-archs passes? the CI fails on that test but I'm not sure if it has something to do with this PR, since the test only runs the text model

@ngxson

All tests pass locally.
As far as I can tell, this test is also failing on master.
DSOCR-2 is currently excluded from test-llama-archs. In fact, I have already prepared a small follow-up PR that adds DeepSeek-OCR, since you have already approved this PR.