Feature: Added LIGHTNING_INDEXER support for Deepseek V4 ops on Vulkan Backend by shenron0101 · Pull Request #27453 · ggml-org/llama.cpp

shenron0101 · 2026-08-20T18:13:44Z

Overview

Updated PR from the previous closed one.

LIGHTNING_INDEXER: Added support for F32, F16, BF16, and several quantized formats.

The Lightning Indexer has only been implemented in the CPU backend, CUDA backend but not in the Vulkan backend. This is an implementation of the Indexer for DeepSeek V4 on the Vulkan backend.

Assisted-by: OpenCode

Currently the Hyper-connection Operations for the Vulkan backend are being implemented by @kh0pper. The one missing operation in the PR was the lightning indexer.

##26585 — "vulkan: tiled transpose for 0<->2 permuted CONT"

The original PR which was closed:
##26548 — "vulkan: add DeepSeek-V4 hyper-connection fused ops (DSV4_HC_COMB/PRE/POST)".

Additional information

Implements GGML_OP_LIGHTNING_INDEXER for Vulkan. Previously CPU, CUDA, SYCL, Metal only.
K types: f32, f16, bf16, q8_0, q5_1, q5_0, q4_1, q4_0, iq4_nl.
Extracts the FA_TYPE_* values and helpers from flash_attn_base.glsl into a new shared fa_types.glsl, so the indexer reuses FA's dequant path rather than redeclaring it.
Two SPIR-V variants selected on subgroup_arithmetic && subgroup_require_full_support, same pattern as ssm_scan.

Numbers (RTX 3090, test-backend-ops perf, K=f16)

Shape	µs	GFLOPS
nh=32 kv=256	10.81	195.6
nh=64 kv=256	19.50	216.8
nh=32 kv=4096	81.55	414.7
nh=64 kv=4096	159.16	425.0

Correctness evidence

test-backend-ops: 180/180 LIGHTNING_INDEXER cases, 16931/16931 full Vulkan suite.
GGML_VULKAN_CHECK_RESULTS against CPU on live DeepSeek-V4-Flash shapes: avg_err 3.42e-10.

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: GPT-5.6 was used in the development.

…ane dot-product reduction moved from a shared-memory tree to subgroupAdd.

ggml-gh-bot · 2026-08-20T18:18:44Z

Hi @shenron0101, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

AI-generated content: While code is allowed to be generated by AI, please write the PR description and commit messages on your own without the help of AI.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

cdanis · 2026-08-21T00:33:01Z

Tried this out on a Strix Halo (AMD gfx1151). Used model -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-IQ2_XXS

180/180 passed, no problem on ./bin/test-backend-ops -b Vulkan0 -o LIGHTNING_INDEXER

No apparent change in performance or perplexity as compared to current master.

kh0pper · 2026-08-21T03:07:30Z

Possible explanation for the null result @cdanis measured on gfx1151: most of what indexer fusion can remove on this hardware already went away when the 0<->2 permuted CONT transpose landed (#26585, in master since Aug 19).

Same box (gfx1151 / RADV), DeepSeek-V4-Flash IQ3_XXS, 9040-token prompt (indexer n_kv 2304, -ub 512), before vs after that transpose fix:

	before	after
indexer chain	45.13% of prefill	5.17%
indexer glue (what fusion removes)	44.64%	4.15%
decode glue	0.66%	0.63%

In absolute terms the glue went 67.5 s -> 3.6 s of an 87.7 s prefill. That caps indexer fusion at roughly +4.3% prefill / +0.5% decode at this context length, and the fused kernel also has to absorb the indexer matmul (1.01% of prefill), which is a tuned MUL_MAT f16 today — so the realistic band includes outcomes that come out slightly negative.

Two caveats that cut the other way: this is n=1 at a single context length, and the indexer glue is O(n_kv) while the HC ops are O(1), so the indexer's share grows with depth. If this change is going to show up anywhere it is at depth — d24576 or a 32k prompt — rather than at the default. Worth measuring there before treating the null as final.

Minor: the HC ops referenced in the description are #26578 (reopened today), not #26548, which was superseded.

cdanis · 2026-08-21T08:37:19Z

I wasn't too surprised with the null result given the lack of fused HC ops. I'll try again combined with the re-opened PR, thanks :)

0cc4m · 2026-08-21T06:26:44Z

-
 layout (binding = 1) readonly buffer K_PACKED_BF16 { u16vec4 data[]; } k_packed_bf16;
+
+#ifndef FA_K_ONLY


I don't think this is needed, unused bindings make no difference and you'd just need something like #define FaTypeV 0 to avoid the compile issue.

jeffbolznv

The ggml-vulkan changes look mostly good to me, I just had a couple minor comments. I didn't do a detailed review of the test changes. I'm surprised such large changes were needed, but I didn't look closely.

jeffbolznv · 2026-08-21T15:47:04Z

+    barrier();
+
+    float score = 0.0;
+    for (uint h = 0; h < n_heads; ++h) {


Looks like each head is independent, so there's probably a lot of opportunity to optimize. But this version is clear and fine for the initial implementation.

Tested a version with optimizations that we can add if needed.

Shape Baseline Optimized Speedup

nh=32 kv=256 464.0 GFLOPS 486.2 GFLOPS 1.05x

nh=64 kv=256 469.0 GFLOPS 497.3 GFLOPS 1.06x

nh=32 kv=4096 468.9 GFLOPS 559.4 GFLOPS 1.19x

nh=64 kv=4096 473.8 GFLOPS 576.2 GFLOPS 1.22x

nh=32 kv=65536 461.3 GFLOPS 563.1 GFLOPS 1.22x

nh=64 kv=65536 467.3 GFLOPS 568.9 GFLOPS 1.22x

Shall I leave this as a seperate PR?

Yes, leave for a future change

jeffbolznv · 2026-08-22T15:08:53Z

 // F32 is fed as a vec4 "block" (4 floats), matching what dequant_funcs_cm2.glsl
 // does for F32 in the cm2 shader. FaBlockBytesK/V == 16 for F32.
 layout (binding = 1) readonly buffer K_PACKED_F32  { vec4 data[]; }                k_packed_f32;
-layout (binding = 2) readonly buffer V_PACKED_F32  { vec4 data[]; }                v_packed_f32;


these diffs can all be reverted now, right?

Yup reverted

I wasn't asking for the removal of FA_K_ONLY to be reverted. I was asking for all of the unnecessary reordering to be undone.

Sorry about that I fixed it in the new commit.

Please remove FA_K_ONLY.

Will we adding that in a later commit?

Also set #define FaTypeV FA_TYPE_F32 looks cleaner. I can change it if needed

This reverts commit fdcbdd9.

0cc4m · 2026-08-26T11:40:43Z

@ggml-org/ggml-cuda CUDA is failing these new tests. Not sure about the other CI issues.

am17an · 2026-08-26T12:08:16Z

cc @fairydreaming

jeffbolznv

I haven't reviewed the details of the test changes, but the ggml-vulkan code LGTM.

shenron0101 added 2 commits August 21, 2026 01:48

vulkan: add LIGHTNING_INDEXER op

9283284

vulkan: updated lightning_indexer.comp and ggml-vulkan.cpp with 128-l…

1065050

…ane dot-product reduction moved from a shared-memory tree to subgroupAdd.

shenron0101 requested review from a team and ggerganov as code owners August 20, 2026 18:13

github-actions Bot added testing Everything test related Vulkan Issues specific to the Vulkan backend ggml changes relating to the ggml tensor library for machine learning labels Aug 20, 2026

0cc4m reviewed Aug 21, 2026

View reviewed changes

jeffbolznv reviewed Aug 21, 2026

View reviewed changes

shenron0101 added 2 commits August 22, 2026 22:19

vulkan: cleanup; Skip bounds checks

00c191d

vulkan: cleanup FA_K_ONLY

fdcbdd9

jeffbolznv reviewed Aug 22, 2026

View reviewed changes

Revert "vulkan: cleanup FA_K_ONLY"

82c0ba2

This reverts commit fdcbdd9.

shenron0101 requested a review from jeffbolznv August 22, 2026 16:24

shenron0101 added 2 commits August 23, 2026 02:24

vulkan: restore interleaved K/V buffer ordering

b3d7873

vulkan: Remove FA_K_ONLY

00899e3

0cc4m reviewed Aug 24, 2026

View reviewed changes

Comment thread ggml/src/ggml-vulkan/vulkan-shaders/flash_attn_dequant.glsl Outdated

vulkan: Revert flash_attn_dequant

89e7f7e

0cc4m approved these changes Aug 26, 2026

View reviewed changes

jeffbolznv approved these changes Aug 26, 2026

View reviewed changes


		layout (binding = 1) readonly buffer K_PACKED_BF16 { u16vec4 data[]; } k_packed_bf16;

		#ifndef FA_K_ONLY

Shape	Baseline	Optimized	Speedup
nh=32 kv=256	464.0 GFLOPS	486.2 GFLOPS	1.05x
nh=64 kv=256	469.0 GFLOPS	497.3 GFLOPS	1.06x
nh=32 kv=4096	468.9 GFLOPS	559.4 GFLOPS	1.19x
nh=64 kv=4096	473.8 GFLOPS	576.2 GFLOPS	1.22x
nh=32 kv=65536	461.3 GFLOPS	563.1 GFLOPS	1.22x
nh=64 kv=65536	467.3 GFLOPS	568.9 GFLOPS	1.22x

Uh oh!

Conversation

shenron0101 commented Aug 20, 2026

Overview

Additional information

Requirements

Uh oh!

ggml-gh-bot Bot commented Aug 20, 2026

Uh oh!

cdanis commented Aug 21, 2026

Uh oh!

kh0pper commented Aug 21, 2026

Uh oh!

cdanis commented Aug 21, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeffbolznv left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

0cc4m commented Aug 26, 2026

Uh oh!

am17an commented Aug 26, 2026

Uh oh!

jeffbolznv left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants