metal : add CONV_2D_DW (depthwise convolution) support by Sou-ly · Pull Request #21565 · ggml-org/llama.cpp

Sou-ly · 2026-04-07T14:50:23Z

Summary

Addresses Feature Request: Implement missing ops from backends #14909
Adds Metal GPU acceleration for GGML_OP_CONV_2D_DW (depthwise 2D convolution), used by MobileNet/EfficientNet-style architectures
Supports both WHCN (contiguous) and CWHN (channels-first) memory layouts via byte-stride indexing
Supports f32 and f16 kernel weights with f32 input/output
Adds extra tests for performance benchmarking

Performance (Apple M5)

Shape	Layout	Kernel	Metal M5	CPU M5	Speedup
`[19,30,128,1]`	WHCN	`[7,7,1,128]`	88.01 us	4717.40 us	53.60x
`[19,30,128,1]`	CWHN	`[7,7,1,128]`	48.72 us	3795.78 us	77.91x
`[24,1,128,1]`	WHCN	`[15,1,1,128]`	8.59 us	2825.94 us	329.0x
`[24,1,256,1]`	WHCN	`[31,1,1,256]`	28.32 us	3398.27 us	120.0x
`[24,1,256,1]`	CWHN	`[31,1,1,256]`	15.66 us	2694.99 us	172.1x
`[512,512,256,1]`	WHCN	`[3,3,1,256]`	15265 us	86947 us	5.69x
`[512,512,256,1]`	CWHN	`[3,3,1,256]`	16445 us	41038 us	2.50x
`[112,112,32,1]`	WHCN	`[3,3,1,32]`	95.88 us	4590.27 us	47.88x
`[112,112,32,1]`	CWHN	`[3,3,1,32]`	98.48 us	2775.28 us	28.18x
`[56,56,128,1]`	WHCN	`[5,5,1,128]`	52.20 us	3828.11 us	73.34x
`[56,56,128,1]`	CWHN	`[5,5,1,128]`	42.93 us	4605.90 us	107.3x

Test plan

test-backend-ops test -o CONV_2D_DW — 4/4 tests pass on Metal (both WHCN and CWHN layouts)
Results verified against CPU reference implementation

AI usage disclosure

Yes: PR description (partial), code cleanup and template implementations for different data formats than FP32. Same as my implementation of ADD1 #21267

ggml-gh-bot · 2026-04-07T14:56:07Z

Hi @Sou-ly, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

AI-generated content: This project does not accept PRs, descriptions or commit messages that are fully or predominantly AI-generated. If you have used AI to assist you in writing code, please make sure to disclose that explicitly.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

QuintinShaw · 2026-06-10T11:12:25Z

FWIW, I tested this locally on an Apple M1 iMac at PR head 29477782f8da01a1a77bb032c511c604f6385f84.

The dedicated backend op test looks good here:

./build/bin/test-backend-ops test -o CONV_2D_DW
MTL0: 4/4 tests passed
3/3 backends passed

I also ran the full backend test suite on the PR head, and once more after a clean no-commit merge with current master (039e20a2db9e87b2477c76cc04905f3e1acad77f). Both runs finished with 3/3 backends passed.

For my downstream ASR path, I tried a few local-only depthwise-conv shapes that show up in encoder-style workloads:

Shape	Layout	Kernel	Metal M1	CPU	Speedup
`[19,30,128,1]`, kernel `[7,7,1,128]`	WHCN	F32	239.49 us	1520.58 us	6.35x
`[19,30,128,1]`, kernel `[7,7,1,128]`	CWHN	F32	323.92 us	468.19 us	1.45x
`[24,1,128,1]`, kernel `[15,1,1,128]`	WHCN	F32	10.29 us	44.56 us	4.33x
`[24,1,256,1]`, kernel `[31,1,1,256]`	WHCN	F32	16.11 us	144.93 us	9.00x
`[24,1,256,1]`, kernel `[31,1,1,256]`	CWHN	F32	16.62 us	203.10 us	12.22x

I also did a quick local check of the F16-weight Metal path by comparing Metal F16-kernel output against Metal F32-kernel output, using F16-representable weights. The 13 cases I tried all matched exactly (max_abs=0, NMSE=0). That included the ASR shapes above, dilation, prime channel counts, batch > 1, and a W=1 boundary case.

One caveat: the large synthetic [512,512,256,1] perf case was slower than CPU on this M1, so the win seems pretty shape-dependent. For the smaller ASR-style shapes I care about, though, this path was consistently faster.

Sou-ly · 2026-06-28T15:11:51Z

Thanks for the detailed testing @QuintinShaw! I reworked the Metal dispatch based on your feedback — 3D grid for the standard path, and a separate channel-tiled kernel for non-contiguous layouts (CWHN). Here are updated numbers on an M5:

Shape	Layout	Kernel	Metal M5	CPU M5	Speedup
`[19,30,128,1]`	WHCN	`[7,7,1,128]`	88.01 us	4717.40 us	53.60x
`[19,30,128,1]`	CWHN	`[7,7,1,128]`	48.72 us	3795.78 us	77.91x
`[24,1,128,1]`	WHCN	`[15,1,1,128]`	8.59 us	2825.94 us	329.0x
`[24,1,256,1]`	WHCN	`[31,1,1,256]`	28.32 us	3398.27 us	120.0x
`[24,1,256,1]`	CWHN	`[31,1,1,256]`	15.66 us	2694.99 us	172.1x
`[512,512,256,1]`	WHCN	`[3,3,1,256]`	15265 us	86947 us	5.69x
`[512,512,256,1]`	CWHN	`[3,3,1,256]`	16445 us	41038 us	2.50x
`[112,112,32,1]`	WHCN	`[3,3,1,32]`	95.88 us	4590.27 us	47.88x
`[112,112,32,1]`	CWHN	`[3,3,1,32]`	98.48 us	2775.28 us	28.18x
`[56,56,128,1]`	WHCN	`[5,5,1,128]`	52.20 us	3828.11 us	73.34x
`[56,56,128,1]`	CWHN	`[5,5,1,128]`	42.93 us	4605.90 us	107.3x

The large [512,512,256,1] case is now faster than CPU for both layouts. Metal wins across the board.

ggerganov · 2026-07-08T08:56:16Z

I added some F16 tests that are failing.

Sou-ly · 2026-07-08T14:11:28Z

@ggerganov thanks for the F16 tests, the CPU backend was reading F16 kernel weights as float (casting raw bytes), fixed in 365c8c1c0. Metal side was already handling it via the TK template. All 8 tests pass now.

ggerganov · 2026-07-09T13:03:08Z

@Sou-ly Could you take a look why these conv_2d_dw tests started failing: https://github.com/ggml-org/ggml/actions/runs/29016861329/job/86113894431?pr=1558#step:5:535

This is the branch: ggml-org/ggml#1558

* metal : add CONV_2D_DW (depthwise 2D convolution) support * test : add perf cases for CONV_2D_DW * metal : use 3D dispatch for CONV_2D_DW kernel * metal : add channel-tiled CONV_2D_DW kernel for non-contiguous layouts * metal : simplify CONV_2D_DW dispatch and trim comments * metal : merge duplicate CONV_2D_DW pipeline getters * tests : add F16 CONV2D_DW tests * cpu : fix F16 kernel support for CONV_2D_DW * tests : remove commented-out CONV_2D_DW test block --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* metal : per-op source split + parallel compile (#24021) * preliminary extract common header * op source split * split metallib into 8 libs && load in parallel * derive kernel->library routing from functionNames * x-macro lib list + underscore filenames, dedup QK_NL, MRC fixes * op source split 8 to 20 * improve robustness of source fallback * clean up * change bool -> atomic_bool * only prepend headers that source actually includes * no semaphore, use GCD global queue * dedup library compile path, fix NSError lifetime, rename gla * relocate upstream concat/rope_back/repeat kernel changes into split files * move ggml-common.h from common.h into dequantize.h to shrink binary size --------- Co-authored-by: lvyichen <lvyichen@stepfun.com> * metal: add col2im_1d op (f32/f16/bf16) (#25176) * metal : add set_rows with src0 f16 (#25434) * metal : add CONV_2D_DW (depthwise convolution) support (#21565) * metal : add Q2_0 support (#25419) * metal: fuse snake activation (mul, sin, sqr, mul, add) (#25459) * ggml-metal: FWHT kernel for metal backend (#25924) * metal : port new kernels into the split sources Move the kernels added on master after the split (lightning indexer, DSv4 hyper-connections, silu_back, f16 bin ops, TQ2_0, the flash-attn KV dequantization pass, rope offset/inplace, ssm_scan rollback, packed q8_0 dequantization and the tensor-API mat-mat K clamp) into the corresponding kernels/*.metal sources. Copied verbatim, no functional change. --------- Co-authored-by: lvyichen <lvyichen@stepfun.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* metal : per-op source split + parallel compile (ggml-org#24021) * preliminary extract common header * op source split * split metallib into 8 libs && load in parallel * derive kernel->library routing from functionNames * x-macro lib list + underscore filenames, dedup QK_NL, MRC fixes * op source split 8 to 20 * improve robustness of source fallback * clean up * change bool -> atomic_bool * only prepend headers that source actually includes * no semaphore, use GCD global queue * dedup library compile path, fix NSError lifetime, rename gla * relocate upstream concat/rope_back/repeat kernel changes into split files * move ggml-common.h from common.h into dequantize.h to shrink binary size --------- Co-authored-by: lvyichen <lvyichen@stepfun.com> * metal: add col2im_1d op (f32/f16/bf16) (ggml-org#25176) * metal : add set_rows with src0 f16 (ggml-org#25434) * metal : add CONV_2D_DW (depthwise convolution) support (ggml-org#21565) * metal : add Q2_0 support (ggml-org#25419) * metal: fuse snake activation (mul, sin, sqr, mul, add) (ggml-org#25459) * ggml-metal: FWHT kernel for metal backend (ggml-org#25924) * metal : port new kernels into the split sources Move the kernels added on master after the split (lightning indexer, DSv4 hyper-connections, silu_back, f16 bin ops, TQ2_0, the flash-attn KV dequantization pass, rope offset/inplace, ssm_scan rollback, packed q8_0 dequantization and the tensor-API mat-mat K clamp) into the corresponding kernels/*.metal sources. Copied verbatim, no functional change. --------- Co-authored-by: lvyichen <lvyichen@stepfun.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

Sou-ly requested a review from a team as a code owner April 7, 2026 14:50

Sou-ly requested a review from ggerganov as a code owner April 7, 2026 15:15

github-actions Bot added testing Everything test related ggml changes relating to the ggml tensor library for machine learning Apple Metal https://en.wikipedia.org/wiki/Metal_(API) labels Apr 7, 2026

Sou-ly force-pushed the metal-conv2d-dw branch from a8265cb to 2947778 Compare June 1, 2026 04:29

ggerganov self-assigned this Jun 27, 2026

Sou-ly force-pushed the metal-conv2d-dw branch from 2947778 to 8e6cb7b Compare June 28, 2026 15:11

Sou-ly force-pushed the metal-conv2d-dw branch from 5c8a3e2 to 365c8c1 Compare July 8, 2026 14:02

Sou-ly and others added 9 commits July 9, 2026 11:41

metal : add CONV_2D_DW (depthwise 2D convolution) support

69cb03d

test : add perf cases for CONV_2D_DW

dc964ac

metal : use 3D dispatch for CONV_2D_DW kernel

b897362

metal : add channel-tiled CONV_2D_DW kernel for non-contiguous layouts

309d67c

metal : simplify CONV_2D_DW dispatch and trim comments

12dbfda

metal : merge duplicate CONV_2D_DW pipeline getters

eb6456a

tests : add F16 CONV2D_DW tests

8ce0cb4

cpu : fix F16 kernel support for CONV_2D_DW

ce51dfc

tests : remove commented-out CONV_2D_DW test block

c8e6cb6

Sou-ly force-pushed the metal-conv2d-dw branch from 365c8c1 to c8e6cb6 Compare July 9, 2026 02:42

ggerganov approved these changes Jul 9, 2026

View reviewed changes

ggerganov merged commit 92b187c into ggml-org:master Jul 9, 2026
31 of 32 checks passed

ggerganov mentioned this pull request Jul 9, 2026

ggml : fix conv 2d dw #25490

Merged

bkutasi mentioned this pull request Jul 10, 2026

📊 AI CLI Tools Digest 2026-07-10 bkutasi/big_model_radar#6

Open

ggerganov added a commit that referenced this pull request Jul 10, 2026

metal : add CONV_2D_DW (depthwise convolution) support (#21565)

65dd913

ggerganov added a commit that referenced this pull request Jul 28, 2026

metal : add CONV_2D_DW (depthwise convolution) support (#21565)

ccb8445

forforever73 pushed a commit that referenced this pull request Aug 4, 2026

metal : add CONV_2D_DW (depthwise convolution) support (#21565)

5465ef1

abrisene pushed a commit to abrisene/llama.cpp that referenced this pull request Aug 19, 2026

metal : add CONV_2D_DW (depthwise convolution) support (ggml-org#21565)

7649856

forforever73 pushed a commit that referenced this pull request Aug 22, 2026

metal : add CONV_2D_DW (depthwise convolution) support (#21565)

f1397b9

Navigation Menu

Uh oh!

metal : add CONV_2D_DW (depthwise convolution) support - #21565

metal : add CONV_2D_DW (depthwise convolution) support#21565
ggerganov merged 9 commits into
ggml-org:masterfrom
Sou-ly:metal-conv2d-dw

Sou-ly commented Apr 7, 2026 •

edited

Loading

Uh oh!

ggml-gh-bot Bot commented Apr 7, 2026

Uh oh!

QuintinShaw commented Jun 10, 2026

Uh oh!

Sou-ly commented Jun 28, 2026

Uh oh!

ggerganov commented Jul 8, 2026

Uh oh!

Sou-ly commented Jul 8, 2026

Uh oh!

Uh oh!

ggerganov commented Jul 9, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Sou-ly commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Performance (Apple M5)

Test plan

AI usage disclosure

Uh oh!

ggml-gh-bot Bot commented Apr 7, 2026

Uh oh!

QuintinShaw commented Jun 10, 2026

Uh oh!

Sou-ly commented Jun 28, 2026

Uh oh!

ggerganov commented Jul 8, 2026

Uh oh!

Sou-ly commented Jul 8, 2026

Uh oh!

Uh oh!

ggerganov commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Sou-ly commented Apr 7, 2026 •

edited

Loading

ggerganov commented Jul 9, 2026 •

edited

Loading