sycl: parallelize the non-contiguous concat kernel by Titaniumtown · Pull Request #25852 · ggml-org/llama.cpp

Titaniumtown · 2026-07-18T04:37:09Z

Overview

The non-contiguous concat kernel launched a single-lane work-group (1, 1, 1), this PR changes it so that it will launch a (1, 1, SYCL_CONCAT_BLOCK_SIZE) one that can better utilize the hardware instead of having single-width iteration loops.

llama-bench (Arc Pro B70, Qwen3.6-27B-UD-Q4_K_XL, -fa on, q8_0 KV), on top of upstream master: pp2048 920 -> 1006 t/s (+9.4%)

Additional information

Should the 256 value that is shared among a bunch of macros in ggml/src/ggml-sycl/presets.hpp be dynamically set similar to #25205? I am wondering what the performance impact of setting that to 128 on Intel Alchemist hardware would be? May be faster due to 128 being the native work-group size.

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES, used Claude Opus 4.8 + Claude Fable 5 in a general auto-research loop that identified potential optimizations with access to an Intel Arc B70 and profiling ability. I then used Opus 4.8 to help me understand the codebase and the context of the code and debug issues.

arthw

For Qwen3.6-27B-UD-Q5_K_XL-MTP.gguf on B60.
I find the performance is reduced.
Through the LLM is not the target of this PR, it won't be impacted negative.

Test	fa	Base t/s	Primary t/s	Increase Rate (Primary vs Base)
pp512	0	44.87	41.28	-8.00%
pp512	1	41.29	38.04	-7.87%
tg128	0	2.75	2.49	-9.45%
tg128	1	2.77	2.50	-9.75%

Of course, I will test with Qwen3.6-27B-UD-Q4_K_XL.gguf.
Could you share the whole cmd to show the benefit of this PR?

Launch geometry only: the non-contiguous concat kernel launched a single-lane work-group (1, 1, 1), now it will launch a (1, 1, SYCL_CONCAT_BLOCK_SIZE) one. SYCL_CONCAT_BLOCK_SIZE is defined in `ggml/src/ggml-sycl/presets.hpp`. llama-bench (Arc Pro B70, Qwen3.6-27B-UD-Q4_K_XL, -fa on, q8_0 KV), on top of upstream master: pp2048 920 -> 1006 t/s (+9.4%)

Titaniumtown · 2026-07-19T19:37:36Z

@arthw It seems that you are offloading the model to system memory? Your tg128 is super low!

The command I used to get benchmarks was: llama-bench -m Qwen3.6-27B-UD-Q4_K_XL.gguf -fa 1 -ctk q8_0 -ctv q8_0 -p 512,2048 -n 128

I do not have a B60 to test, and I cannot repro the regression on my hardware. I believe it is due to oversubscribing the hardware when there isn't enough elements to fill up SYCL_CONCAT_BLOCK_SIZE. I just pushed a commit that may address your issue.

arthw · 2026-07-20T09:42:46Z

With the latest update in the PR and Qwen3.6-27B-UD-Q4_K_XL.gguf on B60.
I got the following perf data.

./build/bin/llama-bench -m ../models/Qwen3.6-27B-UD-Q4_K_XL.gguf -fa 0,1

Test	fa	Base t/s	Primary t/s	Increase Rate (Primary vs Base)
pp512	0	48.79	42.55	-12.79%
pp512	1	44.82	39.09	-12.78%
tg128	0	3.30	2.83	-14.24%
tg128	1	3.32	2.85	-14.16%

./build/bin/llama-bench -m ../models/Qwen3.6-27B-UD-Q4_K_XL.gguf -fa 1 -ctk q8_0 -ctv q8_0 -p 512,2048 -n 128
I got following perf data change on B60:
P512: +4%
P2048 +3.6%
tg128 +1.8%.

A good performance optimization shouldn't reduce other cases' perf when increase the target case.

Please check it!

Thank you!

…ONCAT_BLOCK_SIZE)

Titaniumtown · 2026-07-20T19:26:04Z

@arthw I still cannot reproduce the performance regression on my hardware. Could you try different values of this new environment variable I added: GGML_SYCL_CONCAT_BLOCK_SIZE, like 1, 128, 256 ? and see what those are? I have been tinkering and unable to reproduce anything you have shown.

arthw · 2026-07-21T02:18:51Z

@Titaniumtown
Yes, I will test and feedback.

What's your GPU and driver version?

lspci -nnk | grep -i vga -A3
dpkg -l | grep libze-intel-gpu1

Titaniumtown · 2026-07-21T03:08:09Z

I'm on kernel 7.0.14 on NixOS

GPU: Intel Arc Pro B70 Graphics: PCI 8086:E223 (subsys 8086:1701)
intel-compute-runtime (aka libze-intel-gpu1): 26.18.38308.1
IGC: 2.34.4

arthw · 2026-07-21T07:52:55Z

@Titaniumtown
OK, your GPU driver should be OK.

Here is the test result on B60 with Qwen3.6-27B-UD-Q4_K_XL.gguf
All cases show increase perf.

case 1

Base:

./build/bin/llama-bench -m ..//models/Qwen3.6-27B-UD-Q4_K_XL.gguf -fa 1 -ctk q8_0 -ctv q8_0 -p 512,2048 -n 128

model	size	params	backend	ngl	type_k	type_v	fa	test	t/s
qwen35 27B Q4_K - Medium	16.39 GiB	26.90 B	SYCL	-1	q8_0	q8_0	1	pp512	183.55 Â± 0.03
qwen35 27B Q4_K - Medium	16.39 GiB	26.90 B	SYCL	-1	q8_0	q8_0	1	pp2048	179.63 Â± 0.00
qwen35 27B Q4_K - Medium	16.39 GiB	26.90 B	SYCL	-1	q8_0	q8_0	1	tg128	16.64 Â± 0.01

PR:
export GGML_SYCL_CONCAT_BLOCK_SIZE=1

model	size	params	backend	ngl	type_k	type_v	fa	test	t/s
qwen35 27B Q4_K - Medium	16.39 GiB	26.90 B	SYCL	-1	q8_0	q8_0	1	pp512	188.80 Â± 0.05
qwen35 27B Q4_K - Medium	16.39 GiB	26.90 B	SYCL	-1	q8_0	q8_0	1	pp2048	184.70 Â± 0.02
qwen35 27B Q4_K - Medium	16.39 GiB	26.90 B	SYCL	-1	q8_0	q8_0	1	tg128	16.64 Â± 0.00

build: 2709909 (10068)

export GGML_SYCL_CONCAT_BLOCK_SIZE=128

model	size	params	backend	ngl	type_k	type_v	fa	test	t/s
qwen35 27B Q4_K - Medium	16.39 GiB	26.90 B	SYCL	-1	q8_0	q8_0	1	pp512	188.97 Â± 0.01
qwen35 27B Q4_K - Medium	16.39 GiB	26.90 B	SYCL	-1	q8_0	q8_0	1	pp2048	184.84 Â± 0.01
qwen35 27B Q4_K - Medium	16.39 GiB	26.90 B	SYCL	-1	q8_0	q8_0	1	tg128	16.64 Â± 0.00

build: 2709909 (10068)

export GGML_SYCL_CONCAT_BLOCK_SIZE=256

model	size	params	backend	ngl	type_k	type_v	fa	test	t/s
qwen35 27B Q4_K - Medium	16.39 GiB	26.90 B	SYCL	-1	q8_0	q8_0	1	pp512	188.94 Â± 0.04
qwen35 27B Q4_K - Medium	16.39 GiB	26.90 B	SYCL	-1	q8_0	q8_0	1	pp2048	184.78 Â± 0.01
qwen35 27B Q4_K - Medium	16.39 GiB	26.90 B	SYCL	-1	q8_0	q8_0	1	tg128	16.64 Â± 0.01

case 2

./build/bin/llama-bench -m ..//models/Qwen3.6-27B-UD-Q4_K_XL.gguf -fa 1

base

model	size	params	backend	ngl	fa	test	t/s
qwen35 27B Q4_K - Medium	16.39 GiB	26.90 B	SYCL	-1	1	pp512	187.08 Â± 0.02
qwen35 27B Q4_K - Medium	16.39 GiB	26.90 B	SYCL	-1	1	tg128	16.89 Â± 0.01

PR:
export GGML_SYCL_CONCAT_BLOCK_SIZE=1

model	size	params	backend	ngl	fa	test	t/s
qwen35 27B Q4_K - Medium	16.39 GiB	26.90 B	SYCL	-1	1	pp512	192.61 Â± 0.06
qwen35 27B Q4_K - Medium	16.39 GiB	26.90 B	SYCL	-1	1	tg128	16.89 Â± 0.01

build: 2709909 (10068)

export GGML_SYCL_CONCAT_BLOCK_SIZE=128

model	size	params	backend	ngl	fa	test	t/s
qwen35 27B Q4_K - Medium	16.39 GiB	26.90 B	SYCL	-1	1	pp512	192.74 Â± 0.06
qwen35 27B Q4_K - Medium	16.39 GiB	26.90 B	SYCL	-1	1	tg128	16.86 Â± 0.01

build: 2709909 (10068)

export GGML_SYCL_CONCAT_BLOCK_SIZE=256

model	size	params	backend	ngl	fa	test	t/s
qwen35 27B Q4_K - Medium	16.39 GiB	26.90 B	SYCL	-1	1	pp512	192.67 Â± 0.03
qwen35 27B Q4_K - Medium	16.39 GiB	26.90 B	SYCL	-1	1	tg128	16.89 Â± 0.01

Titaniumtown · 2026-07-22T03:35:25Z

@arthw so your performance regression went away? Interesting. Maybe was transient?

arthw · 2026-07-22T05:40:36Z

Yes, the latest code increase the performance in the test.

Titaniumtown · 2026-07-22T05:58:20Z

Glad to hear it! Let me know if there's anything else I can do for this PR!

…L_SYCL_CONCAT_BLOCK_SIZE)" This reverts commit 2709909.

Titaniumtown · 2026-07-30T06:26:19Z

Ready for review!

arthw

It's good job!

Thank you!

* sycl: parallelize the non-contiguous concat kernel Launch geometry only: the non-contiguous concat kernel launched a single-lane work-group (1, 1, 1), now it will launch a (1, 1, SYCL_CONCAT_BLOCK_SIZE) one. SYCL_CONCAT_BLOCK_SIZE is defined in `ggml/src/ggml-sycl/presets.hpp`. llama-bench (Arc Pro B70, Qwen3.6-27B-UD-Q4_K_XL, -fa on, q8_0 KV), on top of upstream master: pp2048 920 -> 1006 t/s (+9.4%) * sycl: cap non-contiguous concat block at ne0 * sycl: make non-contiguous concat block width env-tunable (GGML_SYCL_CONCAT_BLOCK_SIZE) * Revert "sycl: make non-contiguous concat block width env-tunable (GGML_SYCL_CONCAT_BLOCK_SIZE)" This reverts commit 2709909.

The non-contiguous concat launched a fixed work-group of SYCL_CONCAT_BLOCK_SIZE (256) regardless of ne0. Work-items past ne0 do zero loop iterations, but they are still dispatched and still hold sub-group and EU thread slots, which caps how many of the gridDim (ne1*ne2*ne3) work-groups run concurrently per Xe core. Clamp the block to GGML_PAD(ne0, WARP_SIZE), rounded up to a whole sub-group as the smallest useful unit. Scope, stated honestly: this is a NO-OP at the shape we deploy. Our production concat is dst ne0 = 2051, GGML_PAD(2051,16) = 2064, so the min() still picks 256 exactly as before. It only bites for ne0 < wg -- narrower conv/SSM states than ours -- and the occupancy win on those shapes is UNMEASURED here. Upstream reports no isolated number for it either; their +9.4% pp2048 is the launch fix as a whole. Taken for coverage of shapes and parts we do not run, not as a win for us. Deliberately NOT floored at WARP_SIZE, which is what the upstream PR does. That floor makes GGML_SYCL_CONCAT_WG=1 unreachable and so destroys the one-binary positive control for the 8.36x launch fix. Taking min() against the door value preserves it: at wg == 1 the expression evaluates to 1, which is exactly the pre-fix one-work-item-per-group launch, so both arms still come out of a single build. Adapted from llama.cpp PR ggml-org#25852 ("sycl: parallelize the non-contiguous concat kernel") by Simon Gardling (Titaniumtown), which introduced the shape-adaptive block width. Compile-verified with -fsyntax-only against the integration build flags (GGML_SYCL_WARP_SIZE=16); no new warnings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

74 upstream commits, 14 in our paths. Two textual conflicts, both expected, plus ONE silent-drift resolution that did NOT conflict -- which is the whole reason upstream-check.sh warns about auto-merges. ☠☠ fattn.cpp -- THE ONE THAT AUTO-MERGED CLEAN (upstream 66fa168 / ggml-org#25874) ggml-org#25874 extends oneDNN SDPA to non-F16 KV (Q4_0..Q8_0) and inserts its gate ABOVE the MKL gate -- which our 0801 sync had already deferred to MMA for exactly this reason. Its envelope (Q->ne[1] >= 32, K->ne[1] >= 1024, quantized KV now accepted) matches our deploy prefill exactly: q8_0 KV, ubatch 2048, any real depth. Verbatim it would have silently swapped the measured MMA kernel (2.87x served prefill at 43k) for an unmeasured one and staged the whole q8_0 KV cache to dense F16 first. ⇒ added the SAME conjunct, same wording, as the MKL gate below it: !ggml_sycl_fattn_mma_supported(dst). oneDNN keeps its full envelope everywhere MMA declines. A/B reachable both ways without a rebuild (GGML_SYCL_FATTN_MMA=0 / GGML_SYCL_FA_ONEDNN=0 -- the latter is checked inside _supported(), so it still governs the new early return). This is a precedence choice, not a revert, and it is UNMEASURED on this box. B59 is the finding that prices it. The three conflicts git DID flag in this file were cosmetic (debug kname lines + upstream re-calling get_best_fattn_kernel where we hoist it). Kept ours, added upstream's ONEDNN kname lines. concat.cpp -- upstream 6c8dcaa (ggml-org#25852) is a duplicate of our own fix Kept OURS: strict superset (same launch geometry + the GGML_SYCL_CONCAT_WG door + the o[dim] hoist + the i64 loop var). Ours deliberately omits their WARP_SIZE floor, which would make WG=1 unreachable and destroy the one-binary positive control for the 8.36x launch fix. Noted in the source so the next sync does not resolve toward upstream. Assessed, no action needed: 272700b (ggml-org#26105) iGPU classification -- NULL here, B70 is discrete and still reports TYPE_GPU. Relevant later for PVC/Max-1100 and B51. dbadb68 (ggml-org#22789) dynamic split-graph inputs -- mechanical, auto-merged beside our GGML_SCHED_HANDOFF_CENSUS. 596a579 (ggml-org#25784) DeepseekV4 MTP + DSpark -- ⚠ changes SHARED MTP plumbing (n_embd_out vs n_embd_inp in llama_context::decode, set_embeddings_layer_inp bounds, embeddings_layer_inp sized n_layer+1). We run MTP every step. NOT yet gated -- see the build/gate that follows. ☠ NOT YET BUILT, NOT YET GATED. Every absolute in f306 predates this merge.

* sycl: parallelize the non-contiguous concat kernel Launch geometry only: the non-contiguous concat kernel launched a single-lane work-group (1, 1, 1), now it will launch a (1, 1, SYCL_CONCAT_BLOCK_SIZE) one. SYCL_CONCAT_BLOCK_SIZE is defined in `ggml/src/ggml-sycl/presets.hpp`. llama-bench (Arc Pro B70, Qwen3.6-27B-UD-Q4_K_XL, -fa on, q8_0 KV), on top of upstream master: pp2048 920 -> 1006 t/s (+9.4%) * sycl: cap non-contiguous concat block at ne0 * sycl: make non-contiguous concat block width env-tunable (GGML_SYCL_CONCAT_BLOCK_SIZE) * Revert "sycl: make non-contiguous concat block width env-tunable (GGML_SYCL_CONCAT_BLOCK_SIZE)" This reverts commit 2709909.

Titaniumtown requested a review from a team as a code owner July 18, 2026 04:37

github-actions Bot added ggml changes relating to the ggml tensor library for machine learning SYCL https://en.wikipedia.org/wiki/SYCL - GPU programming language labels Jul 18, 2026

arthw reviewed Jul 19, 2026

View reviewed changes

Titaniumtown added 2 commits July 19, 2026 15:36

sycl: cap non-contiguous concat block at ne0

e792674

Titaniumtown force-pushed the pr/sycl-concat-parallelize branch from 0553bf6 to e792674 Compare July 19, 2026 19:37

Titaniumtown marked this pull request as draft July 20, 2026 13:03

sycl: make non-contiguous concat block width env-tunable (GGML_SYCL_C…

2709909

…ONCAT_BLOCK_SIZE)

arthw reviewed Jul 29, 2026

View reviewed changes

Comment thread ggml/src/ggml-sycl/concat.cpp Outdated

Revert "sycl: make non-contiguous concat block width env-tunable (GGM…

de1170a

…L_SYCL_CONCAT_BLOCK_SIZE)" This reverts commit 2709909.

Titaniumtown marked this pull request as ready for review July 30, 2026 06:26

arthw approved these changes Jul 30, 2026

View reviewed changes

arthw added the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Aug 1, 2026

CISC approved these changes Aug 1, 2026

View reviewed changes

arthw merged commit 6c8dcaa into ggml-org:master Aug 4, 2026
26 of 29 checks passed

Navigation Menu

Uh oh!

sycl: parallelize the non-contiguous concat kernel - #25852

sycl: parallelize the non-contiguous concat kernel#25852
arthw merged 4 commits into
ggml-org:masterfrom
Titaniumtown:pr/sycl-concat-parallelize

Titaniumtown commented Jul 18, 2026 •

edited

Loading

Uh oh!

arthw left a comment •

edited

Loading

Uh oh!

Titaniumtown commented Jul 19, 2026

Uh oh!

arthw commented Jul 20, 2026

Uh oh!

Titaniumtown commented Jul 20, 2026

Uh oh!

arthw commented Jul 21, 2026

Uh oh!

Titaniumtown commented Jul 21, 2026 •

edited

Loading

Uh oh!

arthw commented Jul 21, 2026

Uh oh!

Titaniumtown commented Jul 22, 2026

Uh oh!

arthw commented Jul 22, 2026

Uh oh!

Titaniumtown commented Jul 22, 2026

Uh oh!

Uh oh!

Titaniumtown commented Jul 30, 2026

Uh oh!

arthw left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Titaniumtown commented Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Additional information

Requirements

Uh oh!

arthw left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Titaniumtown commented Jul 19, 2026

Uh oh!

arthw commented Jul 20, 2026

Uh oh!

Titaniumtown commented Jul 20, 2026

Uh oh!

arthw commented Jul 21, 2026

Uh oh!

Titaniumtown commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

arthw commented Jul 21, 2026

case 1

case 2

Uh oh!

Titaniumtown commented Jul 22, 2026

Uh oh!

arthw commented Jul 22, 2026

Uh oh!

Titaniumtown commented Jul 22, 2026

Uh oh!

Uh oh!

Titaniumtown commented Jul 30, 2026

Uh oh!

arthw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Titaniumtown commented Jul 18, 2026 •

edited

Loading

arthw left a comment •

edited

Loading

Titaniumtown commented Jul 21, 2026 •

edited

Loading