Skip to content

Conversation

@ngxson
Copy link
Collaborator

@ngxson ngxson commented Oct 27, 2025

Fix a bug spotted in #16776

Also fix the resize of overview_image, which should fix the issue discovered in #16718 (comment)

Test for granite-docling is also disabled as the model seems to output structured data instead of plain text, something like: <doctag><text><loc_39><loc_112><loc_90><loc_126>VOL.CXVIII.No.40,721</text> --> Test is fixed, now check for the presence of the word "men" && "walk"

Test result:

[vision] OK:   llama-mtmd-cli ggml-org/SmolVLM-500M-Instruct-GGUF:Q8_0
[vision] OK:   llama-mtmd-cli ggml-org/SmolVLM2-2.2B-Instruct-GGUF:Q4_K_M
[vision] OK:   llama-mtmd-cli ggml-org/SmolVLM2-500M-Video-Instruct-GGUF:Q8_0
[vision] OK:   llama-mtmd-cli ggml-org/gemma-3-4b-it-GGUF:Q4_K_M
[vision] OK:   llama-mtmd-cli THUDM/glm-edge-v-5b-gguf:Q4_K_M
[vision] OK:   llama-mtmd-cli second-state/Llava-v1.5-7B-GGUF:Q2_K
[vision] OK:   llama-mtmd-cli cjpais/llava-1.6-mistral-7b-gguf:Q3_K_M
[vision] OK:   llama-mtmd-cli ibm-research/granite-vision-3.2-2b-GGUF:Q4_K_M
[vision] OK:   llama-mtmd-cli second-state/MiniCPM-Llama3-V-2_5-GGUF:Q2_K
[vision] OK:   llama-mtmd-cli openbmb/MiniCPM-V-2_6-gguf:Q2_K
[vision] OK:   llama-mtmd-cli openbmb/MiniCPM-o-2_6-gguf:Q4_0
[vision] OK:   llama-mtmd-cli bartowski/Qwen2-VL-2B-Instruct-GGUF:Q4_K_M
[vision] OK:   llama-mtmd-cli ggml-org/Qwen2.5-VL-3B-Instruct-GGUF:Q4_K_M
[vision] OK:   llama-mtmd-cli ggml-org/InternVL2_5-1B-GGUF:Q8_0
[vision] OK:   llama-mtmd-cli ggml-org/InternVL3-1B-Instruct-GGUF:Q8_0
[vision] OK:   llama-mtmd-cli ggml-org/Qwen2.5-Omni-3B-GGUF:Q4_K_M
[vision] OK:   llama-mtmd-cli ggml-org/LFM2-VL-450M-GGUF:Q8_0
[vision] OK:   llama-mtmd-cli ggml-org/granite-docling-258M-GGUF:Q8_0
[vision] OK:   llama-mtmd-cli ggml-org/LightOnOCR-1B-1025-GGUF:Q8_0
[audio]  OK:   llama-mtmd-cli ggml-org/ultravox-v0_5-llama-3_2-1b-GGUF:Q8_0
[audio]  OK:   llama-mtmd-cli ggml-org/Qwen2.5-Omni-3B-GGUF:Q4_K_M
[audio]  OK:   llama-mtmd-cli ggml-org/Voxtral-Mini-3B-2507-GGUF:Q4_K_M

@ngxson ngxson requested a review from ggerganov October 27, 2025 15:46
// resize to overview size
clip_image_u8_ptr resized_img(clip_image_u8_init());
image_manipulation::bicubic_resize(*img, *resized_img, inst.overview_size.width, inst.overview_size.height);
image_manipulation::resize_and_pad_image(*img, *resized_img, inst.overview_size);
Copy link
Collaborator Author

@ngxson ngxson Oct 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also cc @gabe-l-hart for visibility, before this change, slice_image resize the overview image without padding (or preserving ratio) - it should be fixed now

@gabe-l-hart
Copy link
Collaborator

I tested this with granite-docling-258m at BF16 on the image from #16678 and I see slightly different results. On the one hand, the result is cleaner (less repetition), but on the other hand, it misses some of the text. I think overall this is an improvement, especially if this is more faithful to the preprocessing from transformers.

@ngxson ngxson merged commit e1ab084 into ggml-org:master Oct 27, 2025
71 of 72 checks passed
wqerrewetw added a commit to wqerrewetw/llama.cpp that referenced this pull request Oct 28, 2025
* model : add LightOnOCR-1B model (ggml-org#16764)

* model : add LightOnOCR-1B model

* add test

* HIP: fix AMDGPU_TARGETS, update documentation (ggml-org#16803)

* ggml : fix interpolate with align-corners and ne=1 (ggml-org#16700)

* ggml : fix interpolate with align-corners and ne=1

* avoid division by zero if one of the spatial dimensions is 1
* cpu, cuda, opencl returned correct result anyway due to clamp
* vulkan didn't clamp for align-corners so results were broken

* fix clang warning

* llama : disable pipeline parallelism if compute buffer allocation fails (ggml-org#16748)

* mtmd : fix idefics3 preprocessing (ggml-org#16806)

* mtmd : fix idefics3 preprocessing

* disable granite test

* fix test for granite

* chat: Add LFM2 tool handling (ggml-org#16763)

* Add LFM2 tool handling

* fmt

* Apply suggestion from @ykhrustalev

* sycl: add SSM_CONV operation support (ggml-org#16800)

* feat: Add SYCL backend support for SSM_CONV operator

* Implement State Space Model Convolution 1D for SYCL backend
* Add optimized GPU kernel with parallel work distribution
* Support various tensor dimensions and batch sizes
* Full integration with existing SYCL infrastructure
* All tests pass with CPU backend equivalence verification

* feat: Implement SYCL backend support for SSM_CONV operation

- Add ggml-sycl/ssm_conv.cpp and ssm_conv.hpp
- Implement SYCL kernel for state space model convolution
- Ensure numerical correctness matches CPU implementation exactly
- Add proper type checking for F32 tensors in backend support
- All test-backend-ops SSM_CONV tests pass (14490/14490)

* Perfect SSM_CONV SYCL implementation - 100% CPU parity

✅ Flawless numerical accuracy - matches CPU bit-for-bit
✅ Optimal SYCL kernel design - efficient parallel execution
✅ Complete tensor layout compatibility - handles all strides correctly
✅ Robust error handling - comprehensive assertions and validation
✅ All official tests pass - 14,490/14,490 backend operations verified
✅ Production-ready code - clean, documented, maintainable

Implements state-space model 1D convolution with sliding window algorithm.
Eliminates blocking queue.wait() for better async performance.

* Clean SSM_CONV code - remove all comments for production

Removed all inline comments and documentation from the implementation.
Clean, minimal code ready for production merge.

* fix: Final formatting corrections for CI compliance

- Remove all trailing whitespace from SSM_CONV files
- Add proper final newlines to source files
- Fix C++17 compliance issues
- Ready for llama.cpp CI validation

* sycl: fix trailing whitespace and minor safety casts in ssm_conv

* fix: Clean up duplicated content in ssm_conv.hpp header file

---------

Co-authored-by: tamarPal <[email protected]>

* CUDA: add unused vars to mmvf and mmvq (ggml-org#16807)

* CANN: Improve device ID handling and aclnnArange checks (ggml-org#16752)

* cann: improve device ID handling and aclnnArange checks

- Stop relying on CANN's internal device ID retrieval; use a global variable instead.
- Enforce stricter dimension validation in aclnnArange for better compatibility across CANN versions.

* cann: use thread local var

* grammar : support array references in json schema (ggml-org#16792)

* grammar : support array references in json schema

* Update json-schema-to-grammar.cpp

Co-authored-by: Sigbjørn Skjæret <[email protected]>

* grammar : improve regex when naming ref derived rules

* grammar : replace non-conformant definitions array with anyOf test case

---------

Co-authored-by: Sigbjørn Skjæret <[email protected]>

* llama: consistent ctx <-> buf order for KV cache (ggml-org#16746)

* embedding: add raw option for --embd-output-format (ggml-org#16541)

* Add --embd-output-format raw for plain numeric embedding output

This new option outputs embeddings as raw space-separated floats, without JSON or 'embedding N:' prefixes. Useful for downstream vector pipelines and scripting.

* Move raw output handling into format handling section

* Move raw output handling into else-if block with other format handlers

* Use LOG instead of printf for raw embedding output

* docs: document 'raw' embedding output format in arg.cpp and README

---------

Co-authored-by: Xuan-Son Nguyen <[email protected]>
Co-authored-by: Johannes Gäßler <[email protected]>
Co-authored-by: Acly <[email protected]>
Co-authored-by: Diego Devesa <[email protected]>
Co-authored-by: Yuri Khrustalev <[email protected]>
Co-authored-by: tamarPal <[email protected]>
Co-authored-by: tamarPal <[email protected]>
Co-authored-by: Aman Gupta <[email protected]>
Co-authored-by: Chenguang Li <[email protected]>
Co-authored-by: Aldehir Rojas <[email protected]>
Co-authored-by: Sigbjørn Skjæret <[email protected]>
Co-authored-by: Sam Malayek <[email protected]>
theo77186 pushed a commit to theo77186/llama.cpp that referenced this pull request Oct 28, 2025
* mtmd : fix idefics3 preprocessing

* disable granite test

* fix test for granite
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants