metal: add fused Flux RoPE and direct conv2d kernels#9
Merged
gianni-cor merged 10 commits intoMay 21, 2026
Merged
Conversation
gianni-cor
pushed a commit
to gianni-cor/qvac
that referenced
this pull request
May 13, 2026
…E + conv2d) Adds a vcpkg overlay port for ggml that points to gianni-cor/ggml@feat/metal-conv2d-implicit-gemm (tetherto/qvac-ext-ggml PR tetherto#9). This overlay overrides the registry ggml port with the optimized version for testing. Changes in the ggml overlay: - Fused RoPE Metal kernel (GGML_OP_ROPE_FLUX): 36% faster Flux2 denoising on M4 - Fused V permute kernel (kernel_permute_cont_021) - Implicit GEMM conv2d (17% faster than im2col, saves ~1GB VRAM) - Flash attention NQPTG>8 query block fix Benchmarks: see tetherto/qvac-ext-ggml#9 Co-authored-by: Cursor <cursoragent@cursor.com>
gianni-cor
force-pushed
the
feat/metal-conv2d-implicit-gemm
branch
2 times, most recently
from
32f93bc to
fb25b0f
Compare
1. Fused RoPE (GGML_OP_ROPE_FLUX)
- New op: ggml_rope_flux(a, b) in ggml.h / ggml.c
- Metal kernel: kernel_rope_flux — applies interleaved rotary embedding
and permutes output layout in a single dispatch
- Metal kernel: kernel_permute_cont_021 — fused permute(0,2,1,3)+cont
for V tensor preparation (called via ggml_rope_flux(v, NULL))
- Dispatch: ggml-metal-ops.cpp selects kernel based on PE presence
- Supports_op: ggml-metal-device.m for F32 inputs
- Test: test-rope-flux.cpp — bit-exact correctness across 4 configs
2. Implicit GEMM conv2d (kernel_conv_2d)
- Rewrote naive per-pixel conv2d as implicit GEMM with simdgroup MMA
- 64x64 output tiles, half-precision loads, float accumulators
- 1x1 conv fast path, incremental k-decomposition
- Dispatch: ggml-metal-ops.cpp updated for 256 threads, threadgroup mem
- Pipeline: ggml-metal-device.cpp updated smem and bc_out condition
- Test: test-conv2d-direct.cpp — 14 configs, exact match vs im2col
3. Flash attention NQPTG>8 fix
- Added query block loops to QK and O accumulation sections
- Enables correct output for NQPTG>8 (though NQPTG=8 remains optimal)
Co-authored-by: Cursor <cursoragent@cursor.com>
gianni-cor
force-pushed
the
feat/metal-conv2d-implicit-gemm
branch
from
fb25b0f to
75c1904
Compare
Co-authored-by: Cursor <cursoragent@cursor.com>
Tighten fused RoPE validation and fallback behavior while covering the conv2d 1x1 edge case that could read out of bounds. Co-authored-by: Cursor <cursoragent@cursor.com>
Require simdgroup matrix multiply before advertising Metal conv2d support and add ROPE_FLUX to backend-op coverage. Co-authored-by: Cursor <cursoragent@cursor.com>
Keep Metal ROPE_FLUX support checks aligned with the int32 dispatch limit so oversized tensors fall back instead of asserting at runtime. Co-authored-by: Cursor <cursoragent@cursor.com>
gianni-cor
changed the title
aegioscy
previously approved these changes
May 20, 2026
jpgaribotti
reviewed
May 20, 2026
|
Tests don't cover non-contiguous source test-conv2d-direct.cpp has good shape coverage but only F16 weights with F32 input. The supports_op allows F32+F32 too — please add at least one F32 weight case. |
jpgaribotti
requested changes
May 20, 2026
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Keep backend correctness tests on executable IM2COL_3D type combinations and make CPU/Vulkan supports_op reject combinations that would otherwise assert during compute. Co-authored-by: Cursor <cursoragent@cursor.com>
jpgaribotti
approved these changes
May 21, 2026
Author
|
/review |
aegioscy
approved these changes
May 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
GGML_OP_ROPE_FLUX): applies rotary embedding + layout permute in a single Metal dispatch, replacing 13 ggml ops per Q/K. Eliminates 175 CONT memory copy operations per Flux denoising step.kernel_permute_cont_021): single dispatch for V tensor preparation in flash attention.Benchmarks (Flux2 Klein 4B, Q8_0)
Apple M4 (10 GPU cores, 16GB)
Apple M3 Ultra (60 GPU cores, 96GB)
sd.cpp optimized matches or beats iris.c on denoising at ≥1024×1024.
Test plan
test-rope-flux: bit-exact correctness across 4 configurations (small, medium, flux_klein, batch)test-conv2d-direct: 14 conv2d configurations with max_abs=0.0000GGML_ROPE_FLUX_DISABLE=1env var for fallback to original code pathFiles changed
include/ggml.h: addedGGML_OP_ROPE_FLUXsrc/ggml.c:ggml_rope_flux()functionsrc/ggml-metal/ggml-metal.metal:kernel_rope_flux,kernel_permute_cont_021,kernel_conv_2dimplicit GEMM, flash attention query block fixsrc/ggml-metal/ggml-metal-ops.cpp: dispatch logicsrc/ggml-metal/ggml-metal-impl.h: kargs structssrc/ggml-metal/ggml-metal-device.m: supports_op entriessrc/ggml-metal/ggml-metal-device.cpp: conv2d pipeline/smemtests/:test-rope-flux.cpp,test-conv2d-direct.cpp,test-mul-mat-bench.cppSPEED.md,SPEED-FLUX.md: optimization logs with benchmarks and images