metal: add TQ3_4S GPU support - #59
Merged
Merged
Conversation
TQ3_4S (ggml type 46) was CUDA-only; on Metal it aborted in the mul_mv
pipeline selector ("Asserting on type 46") and only ran on CPU at ~0.15
tok/s. This adds Metal support so TQ3_4S models run on the GPU.
TQ3_4S dequant is a per-32-block inverse randomized Hadamard transform
(8-level codebook lookup + E3M5 per-group scale, then a Walsh-Hadamard
butterfly). The block size QK_TQ3_0 == 32 maps to one 32-lane SIMD-group,
so the butterfly is done with simd_shuffle_xor, mirroring the CUDA kernel
in ggml-cuda/convert.cu.
- ggml-metal.metal: TQ3_0 centroid/sign constants, E3M5 scale decode and
3-bit unpack helpers, kernel_get_rows_tq3_4s (one SIMD-group per row),
and kernel_mul_mv_tq3_4s_f32. The mat-vec kernel exploits RHT
orthogonality to apply the butterfly once per block to the activation
(shared across rows) so weights only do a local codebook lookup.
- ggml-metal-impl.h: N_R0_TQ3_4S / N_SG_TQ3_4S tuning constants.
- ggml-metal-device.cpp: TQ3_4S case in the mul_mv pipeline selector.
- ggml-metal-ops.cpp: route TQ3_4S MUL_MAT through mat-vec (no mat-mat
GEMM yet) and dispatch get_rows as one 32-lane SIMD-group per row.
No mul_mm GEMM or MoE _id kernels yet (the qwen35 test model is dense);
those are the next optimizations for prefill / MoE.
Validated on M3 Pro with osmQwopus-3.6-27B-V2 TQ3_4S: Metal greedy output
is byte-identical to CPU; decode ~4 tok/s, prefill ~4.6 tok/s (~27x over
the CPU baseline). Other quant types are unaffected (only TQ3_4S-specific
cases/kernels were added).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace per-byte loads of the 16-byte block with a single aligned uint4 load plus register-side extraction. The 3-bit index of element j lives at bit 3*j of the 96-bit qs stream (groups pack 8x3 bits = 24 bits = 3 bytes with no cross-group gap), so the index/scale helpers work directly off the uint4. Cleaner than the switch-based unpack and reused by both kernels. Performance-neutral on decode (~4 tok/s) but better-formed memory access. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The mat-vec was bottlenecked by its memory access pattern, not the math:
the lane=element mapping (needed for the in-kernel inverse RHT butterfly)
forced all 32 lanes to broadcast-read the same 16-byte block, wasting
bandwidth (~33% of peak). A diagnostic capping the K-loop to one block
jumped decode from 4 to 48 tok/s, confirming weight reads dominate.
Fix: move the randomized Hadamard transform out of the mat-vec into a
dedicated pre-pass over the activation (RHT orthogonality:
dot(W_dequant, x) == dot(centroid*scale, RHT_fwd(x))). The mat-vec then
needs only a local codebook lookup on the weights, freeing the lane
mapping so each lane owns whole K-blocks and the 32 lanes read 32
consecutive blocks per step -> fully coalesced loads.
- ggml-metal.metal: kernel_tq3_4s_rht_f32 (forward RHT of activation) and a
rewritten kernel_mul_mv_tq3_4s_f32 (strided-K lanes, local dequant).
- ggml-metal-ops.cpp: dedicated ggml_metal_op_mul_mat_tq3_4s path
(pre-pass -> barrier -> mat-vec) so the generic mul_mat code is untouched
for other types; ggml_metal_op_mul_mat_extra_w1 reserves the scratch.
- ggml-metal.cpp: reserve the scratch via the alloc-size hook.
- ggml-metal-device.{h,cpp}: pipeline accessor for the pre-pass kernel.
- ggml-metal-impl.h: kargs_tq3_rht; retune N_R0/N_SG to 4/4.
Validated on M3 Pro (osmQwopus-3.6-27B-V2 TQ3_4S), Metal greedy output
byte-identical to CPU. Decode 4.06 -> 6.4 tok/s (~51% of bandwidth), prefill
4.7 -> 8.0 tok/s; ~43x over the CPU baseline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Batched evaluation (prefill, and speculative-decode verification) was stuck at mat-vec speed because the mat-vec re-reads the full weights once per column. This adds the simdgroup-matrix GEMM path. The RHT activation pre-pass (already used by the mat-vec) lets the weights use a purely local dequant, so a standard dequantize_tq3_4s plugs straight into the stock kernel_mul_mm template (element ordering matches dequantize_q8_0). The dedicated TQ3_4S mul_mat handler routes ne11>1 to mul_mm (reading the pre-transformed activation) and keeps the coalesced mat-vec for decode (ne11==1). - ggml-metal.metal: dequantize_tq3_4s (local, no butterfly) + kernel_mul_mm_tq3_4s_f32. - ggml-metal-ops.cpp: GEMM branch in ggml_metal_op_mul_mat_tq3_4s. Validated on M3 Pro (osmQwopus-3.6-27B-V2 TQ3_4S), greedy output byte-identical to CPU. Prefill 14 tok: 8 -> 32 tok/s; 217 tok: ~6.6 -> 85 tok/s (~13x). Decode unchanged at ~6.4 tok/s. This also makes speculative-decode verification cheap (the K-token batch now amortizes weight reads), the prerequisite for draft-model speculation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Covers the macOS build (incl. the util-linux uuid_string_t gotcha), running a TQ3_4S model on Metal (the nomtp_trunk_only override), which ops are accelerated, indicative M3 Pro throughput, and a tuned draft-model speculative command with the hybrid-target (checkpoint) caveat. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
github-actions
Bot
added
documentation
4 tasks
turbo-tan
added a commit
that referenced
this pull request
Aug 1, 2026
* metal: add TQ3_4S (TurboQuant 3-bit) GPU kernels for Apple Silicon
TQ3_4S (ggml type 46) was CUDA-only; on Metal it aborted in the mul_mv
pipeline selector ("Asserting on type 46") and only ran on CPU at ~0.15
tok/s. This adds Metal support so TQ3_4S models run on the GPU.
TQ3_4S dequant is a per-32-block inverse randomized Hadamard transform
(8-level codebook lookup + E3M5 per-group scale, then a Walsh-Hadamard
butterfly). The block size QK_TQ3_0 == 32 maps to one 32-lane SIMD-group,
so the butterfly is done with simd_shuffle_xor, mirroring the CUDA kernel
in ggml-cuda/convert.cu.
- ggml-metal.metal: TQ3_0 centroid/sign constants, E3M5 scale decode and
3-bit unpack helpers, kernel_get_rows_tq3_4s (one SIMD-group per row),
and kernel_mul_mv_tq3_4s_f32. The mat-vec kernel exploits RHT
orthogonality to apply the butterfly once per block to the activation
(shared across rows) so weights only do a local codebook lookup.
- ggml-metal-impl.h: N_R0_TQ3_4S / N_SG_TQ3_4S tuning constants.
- ggml-metal-device.cpp: TQ3_4S case in the mul_mv pipeline selector.
- ggml-metal-ops.cpp: route TQ3_4S MUL_MAT through mat-vec (no mat-mat
GEMM yet) and dispatch get_rows as one 32-lane SIMD-group per row.
No mul_mm GEMM or MoE _id kernels yet (the qwen35 test model is dense);
those are the next optimizations for prefill / MoE.
Validated on M3 Pro with osmQwopus-3.6-27B-V2 TQ3_4S: Metal greedy output
is byte-identical to CPU; decode ~4 tok/s, prefill ~4.6 tok/s (~27x over
the CPU baseline). Other quant types are unaffected (only TQ3_4S-specific
cases/kernels were added).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* metal: TQ3_4S use aligned uint4 block loads in get_rows/mul_mv
Replace per-byte loads of the 16-byte block with a single aligned uint4
load plus register-side extraction. The 3-bit index of element j lives at
bit 3*j of the 96-bit qs stream (groups pack 8x3 bits = 24 bits = 3 bytes
with no cross-group gap), so the index/scale helpers work directly off the
uint4. Cleaner than the switch-based unpack and reused by both kernels.
Performance-neutral on decode (~4 tok/s) but better-formed memory access.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* metal: TQ3_4S RHT activation pre-pass + coalesced mat-vec (decode +60%)
The mat-vec was bottlenecked by its memory access pattern, not the math:
the lane=element mapping (needed for the in-kernel inverse RHT butterfly)
forced all 32 lanes to broadcast-read the same 16-byte block, wasting
bandwidth (~33% of peak). A diagnostic capping the K-loop to one block
jumped decode from 4 to 48 tok/s, confirming weight reads dominate.
Fix: move the randomized Hadamard transform out of the mat-vec into a
dedicated pre-pass over the activation (RHT orthogonality:
dot(W_dequant, x) == dot(centroid*scale, RHT_fwd(x))). The mat-vec then
needs only a local codebook lookup on the weights, freeing the lane
mapping so each lane owns whole K-blocks and the 32 lanes read 32
consecutive blocks per step -> fully coalesced loads.
- ggml-metal.metal: kernel_tq3_4s_rht_f32 (forward RHT of activation) and a
rewritten kernel_mul_mv_tq3_4s_f32 (strided-K lanes, local dequant).
- ggml-metal-ops.cpp: dedicated ggml_metal_op_mul_mat_tq3_4s path
(pre-pass -> barrier -> mat-vec) so the generic mul_mat code is untouched
for other types; ggml_metal_op_mul_mat_extra_w1 reserves the scratch.
- ggml-metal.cpp: reserve the scratch via the alloc-size hook.
- ggml-metal-device.{h,cpp}: pipeline accessor for the pre-pass kernel.
- ggml-metal-impl.h: kargs_tq3_rht; retune N_R0/N_SG to 4/4.
Validated on M3 Pro (osmQwopus-3.6-27B-V2 TQ3_4S), Metal greedy output
byte-identical to CPU. Decode 4.06 -> 6.4 tok/s (~51% of bandwidth), prefill
4.7 -> 8.0 tok/s; ~43x over the CPU baseline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* metal: TQ3_4S GEMM (mul_mm) for batched eval — prefill up to ~13x
Batched evaluation (prefill, and speculative-decode verification) was stuck at
mat-vec speed because the mat-vec re-reads the full weights once per column.
This adds the simdgroup-matrix GEMM path.
The RHT activation pre-pass (already used by the mat-vec) lets the weights use a
purely local dequant, so a standard dequantize_tq3_4s plugs straight into the
stock kernel_mul_mm template (element ordering matches dequantize_q8_0). The
dedicated TQ3_4S mul_mat handler routes ne11>1 to mul_mm (reading the
pre-transformed activation) and keeps the coalesced mat-vec for decode (ne11==1).
- ggml-metal.metal: dequantize_tq3_4s (local, no butterfly) + kernel_mul_mm_tq3_4s_f32.
- ggml-metal-ops.cpp: GEMM branch in ggml_metal_op_mul_mat_tq3_4s.
Validated on M3 Pro (osmQwopus-3.6-27B-V2 TQ3_4S), greedy output byte-identical
to CPU. Prefill 14 tok: 8 -> 32 tok/s; 217 tok: ~6.6 -> 85 tok/s (~13x). Decode
unchanged at ~6.4 tok/s. This also makes speculative-decode verification cheap
(the K-token batch now amortizes weight reads), the prerequisite for draft-model
speculation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: how to run TQ3_4S on Metal (build, run, speculative)
Covers the macOS build (incl. the util-linux uuid_string_t gotcha), running a
TQ3_4S model on Metal (the nomtp_trunk_only override), which ops are accelerated,
indicative M3 Pro throughput, and a tuned draft-model speculative command with
the hybrid-target (checkpoint) caveat.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ggml-metal: guard SET_ROWS to supported index types
* ggml-metal: guard missing set_rows pipeline in set_rows op encoding
* ggml-metal: tighten backend-op CI coverage
* ggml-metal: use available error log macro
---------
Co-authored-by: Martin Romañuk <mromanuk@me.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: charpdev <charpdev@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Clean replacement for PR #39, rebased onto current main and reduced to the Metal TQ3_4S implementation only.
Scope:
Excluded from the old PR:
Validation run locally on Linux:
Metal runtime validation still needs Apple Silicon hardware.