Add OpenVINO backend#15307

Merged

ggerganov merged 320 commits intoggml-org:masterfrom

ravi9:dev_backend_openvino

Mar 14, 2026

This pull request is big! We're only showing the most recent 250 commits

Commits on Jan 15, 2026

Update build doc

wine99
authored and
cavusmustafa
committed
Add cgraph tensor output name to OV op name

wine99
authored and
cavusmustafa
committed
Update openvino build instructions

ravi9
authored and
cavusmustafa
committed
Add initial NPU support

wine99
authored and
cavusmustafa
committed
draft NPU support version 2: prefill + kvcache

wine99
authored and
cavusmustafa
committed
NPU support version 2: prefill + kvcache

wine99
authored and
cavusmustafa
committed
Change due to ggml cgraph changes, not correct yet

wine99
authored and
cavusmustafa
committed
Change due to ggml cgraph changes, llama-3.2 CPU work

wine99
authored and
cavusmustafa
committed
Add AMD64 to CMakeLists

wine99
authored and
cavusmustafa
committed
Change due to ggml cgraph changes, all device work

wine99
authored and
cavusmustafa
committed
Refactor: clean, fix warning

wine99
authored and
cavusmustafa
committed
Update clang-format

wine99
authored and
cavusmustafa
committed
Statful transformation for CPU GPU

wine99
authored and
cavusmustafa
committed
Add SwiGLU

wine99
authored and
cavusmustafa
committed
Fuse to SDPA

wine99
authored and
cavusmustafa
committed
Replace Concat with Broadcast in MulMat for GQA

wine99
authored and
cavusmustafa
committed
Pull out indices creation for kv cache update

wine99
authored and
cavusmustafa
committed
Refactor: remove past_token_len from extra_inputs

wine99
authored and
cavusmustafa
committed
Fix Phi3 SwiGLU and SoftMax

wine99
authored and
cavusmustafa
committed
Pull out sin cos from rope

wine99
authored and
cavusmustafa
committed
Reduce memory: free ov weights node after graph conversion

wine99
authored and
cavusmustafa
committed
Fix CPY due to cgraph change

wine99
authored and
cavusmustafa
committed
Added OpenVINO CI/CD. Updated docs

ravi9
authored and
cavusmustafa
committed
Fix llama-cli

wine99
authored and
cavusmustafa
committed
Fix Phi3 ROPE; Add test-backend-ops

wine99
authored and
cavusmustafa
committed
Fix NPU

wine99
authored and
cavusmustafa
committed
Fix llama-bench; Clang-format

wine99
authored and
cavusmustafa
committed
Fix llama-perplexity

wine99
authored and
cavusmustafa
committed
temp. changes for mark decomp
cavusmustafa
committed
matmul in fp32

wine99
authored and
cavusmustafa
committed
mulmat input conversion fix
cavusmustafa
committed
mulmat type conversion update
cavusmustafa
committed
add mark decomp pass
cavusmustafa
committed
Revert changes in fuse_to_sdpa

wine99
authored and
cavusmustafa
committed
Update build.md

ravi9
authored and
cavusmustafa
committed
Fix test-backend-ops

wine99
authored and
cavusmustafa
committed
Skip test-thread-safety; Run ctest only in ci/run.sh

wine99
authored and
cavusmustafa
committed
Use CiD for NPU

wine99
authored and
cavusmustafa
committed
Optimize tensor conversion, improve TTFT

wine99
authored and
cavusmustafa
committed
Support op SET_ROWS

wine99
authored and
cavusmustafa
committed
Fix NPU

wine99
authored and
cavusmustafa
committed
Remove CPY

wine99
authored and
cavusmustafa
committed
Fix test-backend-ops

wine99
authored and
cavusmustafa
committed
Minor updates for raising PR

wine99
authored and
cavusmustafa
committed
Perf: RMS fused to OV internal RMS op

wine99
authored and
cavusmustafa
committed
Fix after rebasing

wine99
authored and
cavusmustafa
committed
Change openvino device_type to GPU; Enable flash_attn

wine99
authored and
cavusmustafa
committed
Update supports_buft and supports_op for quantized models

wine99
authored and
cavusmustafa
committed
Add quant weight conversion functions from genai gguf reader

wine99
authored and
cavusmustafa
committed
Quant models run with accuracy issue

wine99
authored and
cavusmustafa
committed
Fix accuracy: disable cpu_repack

wine99
authored and
cavusmustafa
committed
Fix CI; Disable test-backend-ops

wine99
authored and
cavusmustafa
committed
Fix Q4_1

wine99
authored and
cavusmustafa
committed
Fix test-backend-ops: Treat quantized tensors as weights

wine99
authored and
cavusmustafa
committed
Add NPU Q4_0 support

wine99
authored and
cavusmustafa
committed
NPU perf: eliminate zp

wine99
authored and
cavusmustafa
committed
Dequantize q4_1 q4_k q6_k for NPU

wine99
authored and
cavusmustafa
committed
Add custom quant type: q8_1_c, q4_0_128

wine99
authored and
cavusmustafa
committed
Set m_is_static=false as default in decoder

wine99
authored and
cavusmustafa
committed
Simpilfy translation of get_rows

wine99
authored and
cavusmustafa
committed
Fix after rebasing

wine99
authored and
cavusmustafa
committed
Improve debug util; Eliminate nop ReshapeReshape

wine99
authored and
cavusmustafa
committed
STYLE: make get_types_to_requant a function

wine99
authored and
cavusmustafa
committed
Support BF16 model

wine99
authored and
cavusmustafa
committed
Fix NPU compile

wine99
authored and
cavusmustafa
committed
WA for npu 1st token acc issue

wine99
authored and
cavusmustafa
committed
Apply EliminateZP only for npu

wine99
authored and
cavusmustafa
committed
Add GeGLU

wine99
authored and
cavusmustafa
committed
Fix Hunyuan

wine99
authored and
cavusmustafa
committed
Support iSWA

wine99
authored and
cavusmustafa
committed
Fix NPU accuracy

wine99
authored and
cavusmustafa
committed
Fix ROPE accuracy when freq_scale != 1

wine99
authored and
cavusmustafa
committed
Minor: not add attention_size_swa for non-swa model

wine99
authored and
cavusmustafa
committed
Minor refactor

wine99
authored and
cavusmustafa
committed
Add Q5_K to support phi-3-q4_k_m

wine99
authored and
cavusmustafa
committed
Requantize Q6_K (gs16) to gs32 on GPU

wine99
authored and
cavusmustafa
committed
Fix after rebasing

wine99
authored and
cavusmustafa
committed
Always apply Eliminate_ZP to fix GPU compile issue on some platforms

wine99
authored and
cavusmustafa
committed
kvcachefusion support
cavusmustafa
committed
env variable GGML_OPENVINO_DISABLE_SDPA_OPTIMIZATION added
cavusmustafa
committed
Fix for Phi3
cavusmustafa
committed
Fix llama-cli (need to run with --no-warmup)

wine99
authored and
cavusmustafa
committed
Fix add_sliced_mask; Revert mulmat, softmax; Remove input attention_size, iSWA model not working

wine99
authored and
cavusmustafa
committed
fix after rebasing

wine99
authored and
cavusmustafa
committed
Fix llama-3-8b and phi3-mini q4_0 NPU

wine99
authored and
cavusmustafa
committed
Update to OV-2025.3 and CMakeLists.txt

ravi9
authored and
cavusmustafa
committed
Add OV CI cache

wine99
authored and
cavusmustafa
committed
Apply CISC review and update CI to OV2025.3

ravi9
authored and
cavusmustafa
committed
Update CI to run OV dep install before build

ravi9
authored and
cavusmustafa
committed
Update OV dockerfile to use OV2025.3 and update build docs

ravi9
authored and
cavusmustafa
committed
Style: use switch in supports_ops

wine99
authored and
cavusmustafa
committed
Style: middle ptr and ref align, omit optional struct keyword

wine99
authored and
cavusmustafa
committed
NPU Unify PD (#14)

wine99
authored and
cavusmustafa
committed
Clean placeholders in ggml-openvino.cpp

wine99
authored and
cavusmustafa
committed
NPU unify PD (handled internally)

wine99
authored and
cavusmustafa
committed
change graph to 4d, support multi sequences

wine99
authored and
cavusmustafa
committed
Fix llama-bench

wine99
authored and
cavusmustafa
committed
Fix NPU

wine99
authored and
cavusmustafa
committed
Update ggml-decoder.cpp

I-N-T-E-L
authored and
cavusmustafa
committed
Update ggml-decoder.cpp

I-N-T-E-L
authored and
cavusmustafa
committed
Update ggml-decoder.cpp

I-N-T-E-L
authored and
cavusmustafa
committed
Update ggml-decoder.cpp

I-N-T-E-L
authored and
cavusmustafa
committed
Update ggml-decoder.cpp

I-N-T-E-L
authored and
cavusmustafa
committed
Update ggml-decoder.cpp

I-N-T-E-L
authored and
cavusmustafa
committed
Remove the second decoder for node. Moving the function into the model decoder

zhaixuejun1993
authored and
cavusmustafa
committed
Fix error for naive

zhaixuejun1993
authored and
cavusmustafa
committed
NPU prefill chunking

wine99
authored and
cavusmustafa
committed
NPU fix llama-bench

wine99
authored and
cavusmustafa
committed
fallback naive run with accuracy issue

wine99
authored and
cavusmustafa
committed
NPU support llma-perplexity -b 512 --no-warmup

wine99
authored and
cavusmustafa
committed
Refactor: split ov_graph_compute for dynamic and static

wine99
authored and
cavusmustafa
committed
remove unused API GgmlOvDecoder::get_output_stride(const std::string & name)

zhaixuejun1993
authored and
cavusmustafa
committed
minor update due to ov 2025.4

wine99
authored and
cavusmustafa
committed
remove unused API GgmlOvDecoder::get_output_names()

zhaixuejun1993
authored and
cavusmustafa
committed
remove unused API get_output_shape(const std::string & name)

zhaixuejun1993
authored and
cavusmustafa
committed
Modified API GgmlOvDecoder::get_output_type(const std::string & name)

zhaixuejun1993
authored and
cavusmustafa
committed
Removed API GgmlOvDecoder::get_output_op_params(const std::string & name)

zhaixuejun1993
authored and
cavusmustafa
committed
Removed API get_output_ggml_tensor(const std::string & name)

zhaixuejun1993
authored and
cavusmustafa
committed
Removed API m_outputs

zhaixuejun1993
authored and
cavusmustafa
committed
Removed m_output_names

zhaixuejun1993
authored and
cavusmustafa
committed
Removed API GgmlOvDecoder::get_input_names()

zhaixuejun1993
authored and
cavusmustafa
committed
Removed API GgmlOvDecoder::get_input_stride(const std::string& name)

zhaixuejun1993
authored and
cavusmustafa
committed
Removed API get_input_type

zhaixuejun1993
authored and
cavusmustafa
committed
Removed API get_input_type

zhaixuejun1993
authored and
cavusmustafa
committed
Removed API GgmlOvDecoder::get_input_shape(const std::string & name)

zhaixuejun1993
authored and
cavusmustafa
committed
Removed API GgmlOvDecoder::get_input_op_params(const std::string & name)

zhaixuejun1993
authored and
cavusmustafa
committed
Fix error for decoder cache

zhaixuejun1993
authored and
cavusmustafa
committed
Reuse cached decoder

wine99
authored and
cavusmustafa
committed
GPU remove Q6_K requantization

wine99
authored and
cavusmustafa
committed
NPU fix wrong model output shape

wine99
authored and
cavusmustafa
committed
NPU fix q4 perf regression

wine99
authored and
cavusmustafa
committed
Remove unused variable nodes

zhaixuejun1993
authored and
cavusmustafa
committed
Fix decoder can_reuse for llama-bench

wine99
authored and
cavusmustafa
committed
Update build.md for Windows

I-N-T-E-L
authored and
cavusmustafa
committed
backend buffer: allocate on host

wine99
authored and
cavusmustafa
committed
Use shared_buffer for GPU NPU; Refactor

wine99
authored and
cavusmustafa
committed
Add ov_backend_host_buffer; Use cached remote context

wine99
authored and
cavusmustafa
committed
Put kvcache on GPU

wine99
authored and
cavusmustafa
committed
Use ggml_aligned_malloc

wine99
authored and
cavusmustafa
committed
only use remote tensor for kvcache

wine99
authored and
cavusmustafa
committed
only use remote tensor for kvcache for GPU

wine99
authored and
cavusmustafa
committed
FIX: use remote tensor from singleton

wine99
authored and
cavusmustafa
committed
Update build.md to include OpenCL

wine99
authored and
cavusmustafa
committed
NPU always requant to q4_0_128

wine99
authored and
cavusmustafa
committed
Optimize symmetric quant weight extraction: use single zp

wine99
authored and
cavusmustafa
committed
Use Q8_0_C in token embd, lm_head, and for 5 and 6 bits quant

wine99
authored and
cavusmustafa
committed
Update build.md

wine99
authored and
cavusmustafa
committed
Support -ctk f32

wine99
authored and
cavusmustafa
committed
Initial stateful graph support
cavusmustafa
committed
Update ggml/src/ggml-openvino/ggml-decoder.cpp

cavusmustafa
and
ynimmaga
committed
code cleanup
cavusmustafa
committed
npu perf fix
cavusmustafa
committed
requant to f16 for Q6 embed on NPU
cavusmustafa
committed
Update ggml/src/ggml-openvino/ggml-decoder.cpp
cavusmustafa
committed
Update ggml/src/ggml-openvino/ggml-openvino-extra.cpp
cavusmustafa
committed
Create OPENVINO.md in llama.cpp backend docs

ynimmaga
authored and
cavusmustafa
committed
Update OPENVINO.md

ynimmaga
authored and
cavusmustafa
committed
Update OPENVINO.md

ynimmaga
authored and
cavusmustafa
committed
Update OPENVINO.md

ynimmaga
authored and
cavusmustafa
committed
Update build.md

ynimmaga
authored and
cavusmustafa
committed
Update OPENVINO.md

ynimmaga
authored and
cavusmustafa
committed
Update OPENVINO.md

ynimmaga
authored and
cavusmustafa
committed
Update OPENVINO.md

ynimmaga
authored and
cavusmustafa
committed
kq_mask naming fix
cavusmustafa
committed

Commits on Jan 16, 2026

Syntax correction for workflows build file
cavusmustafa
committed

Commits on Jan 21, 2026

Change ov backend buffer is_host to false
wine99
committed

Commits on Feb 11, 2026

Commits on Feb 12, 2026

Commits on Feb 14, 2026

Merge pull request #43 from cavusmustafa/additional_fixes_after_rebase
cavusmustafa
authored

Commits on Feb 18, 2026

Commits on Feb 21, 2026

updated model link to be GGUF model links
cavusmustafa
committed

Commits on Feb 25, 2026

Remove GGML_CPU_REPACK=OFF
wine99
committed

Commits on Feb 26, 2026

Commits on Feb 28, 2026

unify api calling

zhaixuejun1993
authored and
yangwang201911
committed
Update utils.cpp
zhaixuejun1993
authored

Commits on Mar 2, 2026

When the dim is dynamic, throw an error, need to is stastic forst

zhaixuejun1993
authored and
wine99
committed
Add interface compute_model_outputs(), which get the model output through computing the node use count & status in the cgraph to avoid the flag using

zhaixuejun1993
authored and
yangwang201911
committed
No need to return

zhaixuejun1993
authored and
yangwang201911
committed

Commits on Mar 3, 2026

Merge branch 'master' into dev_backend_openvino
wine99
committed
Fix test-backend-ops for OV GPU LNL
wine99
committed
Fix test-thread-safety
wine99
committed
use the shape from infer request of output tensor create to avoid issue

Xuejun Zhai
authored and
cavusmustafa
committed
fix dynamic output shape issue

Xuejun Zhai
authored and
cavusmustafa
committed
Merge pull request #49 from zhaixuejun1993/xuejun/unify-api-get_ov_output_tensor
wine99
authored
fix issue for the unused node in tests

Xuejun Zhai
authored and
cavusmustafa
committed

Commits on Mar 4, 2026

Commits on Mar 10, 2026

Commits on Mar 12, 2026

Update to OV ggml-ci to low-perf
ravi9
authored