Skip to content

mtmd: refactor llava-uhd overview image handling (always use ov_img_first) - #24769

Merged
ngxson merged 5 commits into
ggml-org:masterfrom
ngxson:xsn/refactor_ov_uhd
Jun 18, 2026
Merged

mtmd: refactor llava-uhd overview image handling (always use ov_img_first)#24769
ngxson merged 5 commits into
ggml-org:masterfrom
ngxson:xsn/refactor_ov_uhd

Conversation

@ngxson

@ngxson ngxson commented Jun 18, 2026

Copy link
Copy Markdown
Collaborator

Overview

Important

if you get the error "no image tokens produced by preprocessor", comment below the model you are using (link to GGUF file) as well as the input image size that you are using

llava-uhd overview image logic is a bit messy, because some models hard-coding the handling inside preprocessor, while the other rely on ov_img_first

With this PR, ov_img_first is now the single source of truth.

Test results:

[vision] OK:   ggml-org/SmolVLM-500M-Instruct-GGUF:Q8_0
[vision] OK:   ggml-org/SmolVLM2-2.2B-Instruct-GGUF:Q4_K_M
[vision] OK:   ggml-org/SmolVLM2-500M-Video-Instruct-GGUF:Q8_0
[vision] OK:   ggml-org/gemma-3-4b-it-GGUF:Q4_K_M
[vision] OK:   THUDM/glm-edge-v-5b-gguf:Q4_K_M
[vision] OK:   second-state/Llava-v1.5-7B-GGUF:Q2_K
[vision] OK:   cjpais/llava-1.6-mistral-7b-gguf:Q3_K_M
[vision] OK:   ibm-research/granite-vision-3.2-2b-GGUF:Q4_K_M
[vision] OK:   second-state/MiniCPM-Llama3-V-2_5-GGUF:Q2_K
[vision] OK:   openbmb/MiniCPM-V-2_6-gguf:Q2_K
[vision] OK:   openbmb/MiniCPM-o-2_6-gguf:Q4_0
[vision] OK:   bartowski/Qwen2-VL-2B-Instruct-GGUF:Q4_K_M
[vision] OK:   ggml-org/Qwen2.5-VL-3B-Instruct-GGUF:Q4_K_M
[vision] OK:   ggml-org/InternVL2_5-1B-GGUF:Q8_0
[vision] OK:   ggml-org/InternVL3-1B-Instruct-GGUF:Q8_0
[vision] OK:   ggml-org/Qwen2.5-Omni-3B-GGUF:Q4_K_M
[vision] OK:   ggml-org/LFM2-VL-450M-GGUF:Q8_0
[vision] OK:   ggml-org/granite-docling-258M-GGUF:Q8_0
[vision] OK:   ggml-org/LightOnOCR-1B-1025-GGUF:Q8_0
[vision] OK:   ggml-org/DeepSeek-OCR-GGUF:Q8_0
[vision] OK:   ggml-org/dots.ocr-GGUF:Q8_0
[vision] OK:   ggml-org/HunyuanOCR-GGUF:Q8_0
[vision] OK:   ggml-org/gemma-4-E2B-it-GGUF:Q8_0
[audio]  OK:   ggml-org/ultravox-v0_5-llama-3_2-1b-GGUF:Q8_0
[audio]  OK:   ggml-org/Qwen2.5-Omni-3B-GGUF:Q4_K_M
[audio]  OK:   ggml-org/Voxtral-Mini-3B-2507-GGUF:Q4_K_M
[audio]  OK:   ggml-org/LFM2-Audio-1.5B-GGUF:Q8_0
[audio]  OK:   ggml-org/gemma-4-E2B-it-GGUF:Q8_0
[audio]  OK:   ggml-org/Qwen3-ASR-0.6B-GGUF:Q8_0

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: minor corrections provided by glm-5.1

@ngxson
ngxson requested a review from a team as a code owner June 18, 2026 16:07
@ngxson

ngxson commented Jun 18, 2026

Copy link
Copy Markdown
Collaborator Author

@sfallah this should be merged before #24717 , please take a look

@ngxson

ngxson commented Jun 18, 2026

Copy link
Copy Markdown
Collaborator Author

nvm, this seems good, I'm merging it now to unblock the other PR

=== Summary ===
  DeepSeek-OCR -- single-view scan                 PASS
  DeepSeek-OCR-2 -- single-view scan               PASS
Overall: PASS

@ngxson ngxson changed the title mtmd: refactor llava-uhd overview image handling mtmd: refactor llava-uhd overview image handling (always use ov_img_first) Jun 18, 2026
@ngxson
ngxson merged commit 060ce1b into ggml-org:master Jun 18, 2026
24 of 25 checks passed
adrianhoehne pushed a commit to adrianhoehne/llama.cpp that referenced this pull request Jul 5, 2026
…irst) (ggml-org#24769)

* add dedicated "overview" for mtmd_image_preproc_out

* corrections

* correct (again)

* nits

* nits (2)
zommiommy pushed a commit to zommiommy/llama.cpp that referenced this pull request Aug 18, 2026
…irst) (ggml-org#24769)

* add dedicated "overview" for mtmd_image_preproc_out

* corrections

* correct (again)

* nits

* nits (2)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant