vulkan: chunked parallel kernel for GATED_DELTA_NET#20377
vulkan: chunked parallel kernel for GATED_DELTA_NET#20377ProgenyAlpha wants to merge 14 commits intoggml-org:masterfrom
Conversation
github-actions
bot
added
model
|
Benchmarks, Strix Halo: master (e1a3999):
PR (795f15c):
That's 10-20% better PP performance, depending on the model. |
|
@lemmi Great numbers, thanks for testing. Updated the PR to actually enable the chunked Vulkan dispatch — it's now gated on shader core count (> 16 CUs) instead of being disabled. On my 890M (16 CUs) the 3-dispatch overhead makes chunked slower than autoregressive, so it stays off there. On your 8060S (32 CUs) it should activate automatically for n_tokens > 64 with d128 non-KDA configs. I can't validate the chunked dispatch path myself since I only have the integrated 890M. If you get a chance to test the latest push, that would tell us whether the chunked shaders actually help PP on discrete hardware or if they need more work (coopmat for the output kernel is the next step if so). |
|
Small clarification: the 8060s is the iGPU on Strix Halo (aka Ryzen AI MAX+ 395). The 8060s has 40CUs. Performance tanked with the latest patch:
After:
|
ProgenyAlpha
force-pushed
the
vulkan-gdn-chunked
branch
2 times, most recently
from
795f15c to
c0d0341
Compare
ProgenyAlpha
force-pushed
the
vulkan-gdn-chunked
branch
from
c0d0341 to
dbbe2a9
Compare
|
@0cc4m Rebased on master. Chunked kernels work but the scalar output kernel is too slow without coopmat, so the threshold is disabled for now. I've got a coopmat output kernel already in the works, but do you want me to add it here or keep this as infrastructure and open a separate PR for the coopmat or stop here? |
|
Do I understand correctly that to see a gain you need to merge this PR with another? What exact command line are you using where you see a 30% gain? I only see GDN taking about 5% of the time running |
|
@jeffbolznv Hey! This is noted in the PR description but the 30% PP gain comes from #20340's chunked op path on the graph side feeding my GDN vulkan autoregressive shader #20334 more efficiently, not from the Vulkan chunked shaders. Both #20334 and #20340 are already merged into master, so that improvement is already live. The Vulkan chunked dispatch in this PR is actually disabled ( So with this PR as-is, you'd see near identical performance to master since the chunked path doesn't activate. I was waiting to find out how 0cc4m would like to handle this or anyone in a position to give feedback. I can close out this PR until I've done more thorough validation and testing and reopen then, if preferred. |
|
I mostly want to understand what kind of use case/benchmark you're accelerating on so I can see how much theoretical upside there is. |
Three-dispatch chunked pipeline for prompt processing acceleration: intra-chunk WY decomposition, inter-chunk state propagation, output combination. Currently disabled (threshold=UINT32_MAX). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add gated_delta_net_chunk_output_cm1.comp — a cooperative matrix variant of the chunked output kernel that replaces the O(N²) scalar intra-chunk loop with an f16 coopmat GEMM: A_decayed[64×64] @ vnew[64×128]. Kernel structure: - Phase 1: Q@K^T via coopmat (unchanged from scalar variant) - Phase 2a: Build causal decay mask → sh_adecay (f16, clamped) - Phase 2b: Stage vnew into sh_kv (f16, pre-scaled by 1/√d) - Pass 1: Inter-chunk Q@S → dst (scalar, 128 threads) - Pass 2: Intra-chunk coopmat GEMM (full chunks) or scalar fallback (partial last chunk). 3 barriers total, 62.7KB shared memory. Pipeline registered but not yet dispatched (threshold remains disabled). Test tolerance bumped to 5e-3 for n_seq_tokens≥64 to account for f16 intermediate precision in the coopmat path. 16/16 backend tests pass.
Lower GDN_CHUNK_THRESHOLD from UINT32_MAX to 2 and prefer the coopmat output pipeline (cm1) when available, falling back to the scalar variant. PP-512: ~206 → ~210 t/s on Radeon 890M (RDNA3.5).
Comprehensive documentation for PR ggml-org#20377 covering architecture, benchmarks, PPL validation, per-kernel timing, and scaling analysis. Includes side-by-side autoregressive vs chunked comparison on 890M.
Merge the inter-chunk state propagation and output computation into a single dispatch, reducing the chunked pipeline from 3 dispatches to 2. State lives in registers across the sequential chunk loop. vnew is computed in-kernel and passed to the coopmat GEMM via shared memory (f16, packed with subgroup shuffles). This eliminates the VNew scratch buffer (wu_size) and H_snapshots buffer (h_size) — ~786KB/head/seq saved for PP-512. Architecture per chunk: Step 1: Load K, Q, gcum → shared (all 256 threads) Step 2: Q@K^T coopmat → sh_attn (all 256 threads) Step 3: Decay mask + O_inter = Q@state → dst (parallel) Step 4: vnew = U - W@state → sh_kv (128 threads + k_gated assist) Step 5: O_intra = A_decayed @ vnew coopmat GEMM → dst Step 6: state = exp(decay) * state + delta Shared memory: 63,744 / 65,536 bytes. 16/16 backend tests pass.
This reverts commit 08c355c01f3a298ef943216d4c55367a1c967286.
PR ggml-org#20443 removed redundant state transposes from the graph and updated the autoregressive shader to use col*S_V+i (coalesced) instead of i*S_V+col (strided). The chunked inter kernel was not updated, causing uncoalesced state reads and a ~8% PP regression. Fix state_in load and final_out write to match the new layout. h_snapshots (h_out/h_in) are internal scratch and keep their existing layout since inter and output kernels agree. PP-512: 202 → 218 t/s. 16/16 tests pass.
The autoregressive kernel dispatches one workgroup per attention head. For Qwen3-Next (n_head_kv=2), that's 2 workgroups per GDN layer. On my 890M (16 CU), GDN is ~8% of PP-512 time. Everything else (MLP, matmuls, norms) saturates all 16 CUs. Chunked doesn't show any improvement here because there's nothing for the iGPU to give. On a 7900 XTX (96 CU, ~960 GB/s), the non-GDN ops scale with both CU count and bandwidth. Roughly 10x faster than my shared DDR5. The GDN op also gets faster from bandwidth (~3x), but it doesn't scale with CU count, still 2 workgroups, 94 CUs idle. Dirty math (Amdahl's law, all estimates)
GDN's share grows from ~8% to ~25% of the pipeline. Chunked dispatches 16 workgroups instead of 2 for PP-512. If chunking allowed something like a ~4× improvement on the GDN portion, the rough Amdahl math would put total PP around ~9.75 vs ~12 (~19%). Obviously that depends heavily on whether the kernel actually scales that way. These are rough numbers. The bandwidth scaling on the GDN op, the actual compute vs memory bound split, the dispatch overhead of 3 stages, all of that needs real profiling data to pin down. Based on this rough Amdahl model, GDN's relative share could grow on larger GPUs where the rest of the pipeline scales with CU count but the autoregressive kernel remains limited to a small number of workgroups. I can't prove the exact crossover point locally on 16 CUs, but the theoretical upside on larger GPUs makes it seem worth exploring. |
Remove verbose algorithm comments, section dividers, stale inline constant annotations, and unused extensions. Match llama.cpp codebase style (minimal comments, no section decorators). No functional changes. 16/16 tests pass.
Load both s_w and s_kg before the first barrier instead of using separate barriers for each. Reduces per-token barriers from 3 to 2, eliminating 64 barriers per chunk. GDN per-op: 6818 → 5205 µs (-23.6%). 16/16 tests pass.
Remove unnecessary barrier after A-matrix dot product writes. Each thread writes only to its own row; s_A isn't read cross-thread until forward substitution. Cuts A-matrix barriers from 128 to 65 (one per broadcast + one before forward sub). Pad s_A stride from 64 to 65 to eliminate bank conflicts in the W/U accumulation phase where all active threads read A(tid, j) with the same j value. GDN per-op: 5205 → 5136 µs. Combined with inter fusion: 6818 → 5136 µs (-24.7%). 16/16 tests pass.
ProgenyAlpha
force-pushed
the
vulkan-gdn-chunked
branch
from
dbbe2a9 to
88396c3
Compare
github-actions
bot
added
the
documentation
Intra: - Strip all section/inline comments to match codebase style - Add [[unroll]] to fixed-bound loops (A-matrix zero, W/U tile init/write) - Guard chunk_len==0 underflow on s_decay[chunk_len-1] Inter: - Strip final comment No functional changes. 16/16 tests pass.
- Raise GDN_CHUNK_THRESHOLD from 2 to CHUNK_SIZE (64). Chunked path only activates when there's at least one full chunk. Below that, autoregressive is faster and the 3-dispatch overhead isn't justified. - Add maxStorageBufferRange guard on scratch allocation. Falls back to autoregressive if the scratch buffers would exceed device limits. - Fix inaccurate shared memory stride comment in cm1 output kernel. 16/16 tests pass.
|
Fresh data after several rounds of optimization. I spent the last couple of days trying to make the chunked kernel more efficient and reduce the per-op GDN. Today, I think I've finally broken even reducing it from 6818 µs to 5136 µs (-24.7%) after fusing inter kernel broadcasts and removing unnecessary barriers in the intra kernel. Chunked is now as efficent as autoregressive on 16 CU across both models tested. Per-kernel breakdown after optimization (890M 16 CU, Qwen3-Next Q4_K_M, PP-512)
Throughput benchmarks (back-to-back, same build)Qwen3-Next REAM Q4_K_M (36 GDN layers, n_head_kv=2):
Qwen3.5-35B-A3B Q4_K_M (60 GDN layers, n_head_kv=2):
PPL validation (back-to-back, Qwen3-Next, WikiText-2)
Lossless. What changed since last push
I think I'm going to reach out for testers on higher CU count hardware (8060S, 7900 XTX, 9070) to see how the coopmat GEMMs scale. |
|
Strix Halo (8060s, 40CU). TG is basically unaffected, PP is taking a measurable hit: pp2048:
|
|
@lemmi I'm working on finalizing a script to test, do you know which commit you ran with the above results? |
|
Benchmark results from AMD Strix Halo (Radeon 8060S, RDNA 3.5, 40 CUs, 128 GB unified LPDDR5X-8000). Baseline: pre-built toolbox binary (b8334, includes #20334 + #20340)
Qwen3.5-122B-A10B (MoE + GDN, Q3_K_XL, 53 GiB)
GPT-OSS-120B (standard MoE, no GDN — control, Q4_K_XL, 59 GiB)
Nemotron-3-Super-120B (Mamba-2 + MoE, Q4_K_XL, 78 GiB)
Summary
Happy to test with the chunked dispatch enabled or run |
|
Follow-up with native master build (89d0aec, b8357) for an apples-to-apples comparison. Same hardware (8060S, 40 CUs), same flags ( Qwen3.5-122B-A10B (MoE + GDN, Q3_K_XL, 53 GiB)
GPT-OSS-120B (standard MoE, no GDN, Q4_K_XL, 59 GiB)
Nemotron-3-Super-120B (Mamba-2 + MoE, Q4_K_XL, 78 GiB)
AnalysisWith a native-to-native comparison, PR #20377 shows a ~5% regression across all models on Strix Halo (40 CUs), including non-GDN models. This suggests the regression isn't GDN-specific — it might be in shared shader infrastructure or pipeline setup code. The earlier toolbox-vs-PR comparison masked this by attributing the gap to build environment differences, but now we can see it's a real regression on this hardware. Worth investigating whether it's specific to RDNA 3.5 / 40 CU configs. |
|
It suggests you ran the tests after each other and so the PR was throttling due to thermal constraints. |
5 participants
Follow-up to #20334. Adds the chunked parallel kernel infrastructure for Vulkan GATED_DELTA_NET, split out per @0cc4m's review feedback.
Depends on #20334 and #20340
Three new compute shaders implementing the chunked algorithm:
gated_delta_net_chunk_intra.comp— intra-chunk parallel computationgated_delta_net_chunk_inter.comp— inter-chunk state propagationgated_delta_net_chunk_output.comp— output reconstructionIncludes the
rq1→neq1broadcast fix to match #20340's interleaved Q/K layout (head_id % neq1instead ofhead_id / rq1).Chunked dispatch is currently disabled (
GDN_CHUNK_THRESHOLD = UINT32_MAX) — the autoregressive path handles all token counts. Enabling it will need cooperative matrix support for the output kernel to be competitive.16/16 backend-ops tests passing (includes chunked-specific test configs with n_seq_tokens=64/128).
890M benchmarks (Qwen3-Coder-Next REAM Q4_K_M):
The PP improvement comes from #20340's chunked op path feeding our autoregressive shader more efficiently. The Vulkan chunked dispatch itself isn't active yet — that's the next optimization pass.