CUDA: size routed MoE MMQ N-tiles from typical expert width on RDNA3 - #24546
Open
ravel7524 wants to merge 5 commits into
Open
CUDA: size routed MoE MMQ N-tiles from typical expert width on RDNA3#24546ravel7524 wants to merge 5 commits into
ravel7524 wants to merge 5 commits into
Conversation
github-actions
Bot
added
Nvidia GPU
ravel7524
marked this pull request as ready for review
June 12, 2026 21:04
Contributor
Author
|
Here is an additional sweep extending the prior Hardware/software: AMD Radeon Pro W7800 (
Across the tested 128 expert models, the gain persists at long context for smaller ubatches, and the larger ubatches are effectively neutral rather than regressing. I can do additional long context benchmarking for 256 expert models |
Contributor
Author
|
@jiachengjason would you be able to benchmark this PR on your RDNA4 GPU when you have time? Since you tested #23685 on RDNA4, having the same kind of validation here would be very helpful. If you know anyone who could also test on RDNA2 or CDNA, that would be great as well. No pressure. |
Contributor
Author
|
@IMbackK when you have time, could you sanity check whether this RDNA3 routed MoE MMQ selector direction looks reasonable? The PR is scoped to gfx1100/RDNA3 and uses measured W7800 results; I mainly want to know if the shape is acceptable or if you would prefer a different architecture gating/config style. |
Contributor
|
Do you expect this to work well on RDNA3.5 / Strix Halo as well? Happy to test! |
Contributor
Author
|
Yes, the PR’s RDNA3 predicate also includes RDNA3.5 / Strix Halo, so the routed-width selector should be active there. I expect the same general direction, although the performance magnitude may differ from gfx1100. Testing would be very helpful, ideally the same base vs PR prefill and decode comparison used in the PR. |
justinappler
added a commit
to justinappler/llama.cpp-strix-halo
that referenced
this pull request
Jul 17, 2026
…24546) Upstream PR ggml-org#24546 (ravel7524, open) sizes routed-MoE MMQ N-tiles from the typical expert width instead of the worst case. That is the same problem the static J_max=48 cap from Findings ggml-org#8/ggml-org#9 solves, approached from the other end: the cap clamps the search ceiling, the picker fixes the search objective. This commit swaps the cap for the picker so the two can be A/B'd on gfx1151. The PR does not apply as written - it patches mmq_x/mmq_x_max/ mmq_get_granularity_host in mul_mat_q_case, all deleted by PR ggml-org#24127, the same refactor that forced the Finding ggml-org#5 -> ggml-org#9 re-port. Re-authored onto mul_mat_q_switch_J's config-table search and scoped to RDNA3.5: gfx1151 is the only arch we can measure, and narrowing the predicate keeps the A/B single-variable. Upstream's predicate already covers RDNA3.5 via GGML_CUDA_CC_IS_RDNA3, and the author confirmed on 2026-07-04 that Strix Halo should benefit, but nobody has ever tested it there. On the routed-MoE path ncols_max is ne12 - the worst case of one expert receiving every token - so the search never reaches ntiles==1 and simply runs to whatever ceiling it is given. The picker instead tiles from ncols_dst/nchannels_x, the width a typical expert covers. launch_mul_mat_q still builds the grid from ncols_max, so worst-case coverage is unchanged. - mmq.cuh: replace the RDNA3.5 MoE J_max=48 clamp with the typical-width picker; J_max is now a uniform 128 and the search terminates on its own. - mmq-table-check.cpp: pick_J mirrors the selection loop, so update it in lockstep or the check silently becomes a rubber stamp. Adds the routed-MoE arg shape from mmq.cu and a crossover case. For Qwen 3.6 35B-A3B (256 experts, 8 active) at the production ub=2048 the typical width is 2048*8/256 = 64, so this is a 48 -> 64 change at our operating point - narrow enough that it may land in noise. Verified host-side without ROCm: MoE J=64, dense J=128, no aborts across all 21 types. Above ub=4096 the typical width reaches 128, the picker disengages and the search runs to J=128 where the cap held 48. Findings ggml-org#5/ggml-org#8 only ever measured the typical=64 regime, so that is untested on this chip; production ub=2048 cannot reach it. See strix-halo/mmq-moe-ncols-picker.md for the bench plan and the keep/revert criteria. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
justinappler
added a commit
to justinappler/llama.cpp-strix-halo
that referenced
this pull request
Jul 17, 2026
Benched b47bb31 against 05e837f on gfx1151 (ROCm 7.14.0, canonical Qwen 3.6 35B-A3B Q4_K_XL matrix). The picker is not distinguishable from the static cap at the production operating point: pp512 @ d=0 1428.13 -> 1388.93 (-2.7%) pp512 @ d=2048 1299.39 -> 1316.26 (+1.3%) pp512 @ d=8192 1135.42 -> 1142.23 (+0.6%) pp512 @ d=16384 971.25 -> 977.36 (+0.6%) Correctness was clean first (790/790 MUL_MAT_ID, 1134/1134 MUL_MAT), so the grid does still cover ncols_max while the tile is sized from the typical width. The tg128 control fell ~1.6% at every depth. This change cannot move tg - decode goes through MMVQ where ncols_dst==1, so the picker never runs - which makes that a session drift between two builds measured on different days, the same contamination Finding ggml-org#9 hit. The conclusion survives it either way: raw, pp is -2.7% to +1.3%; calibrated against the drift, -1.2% to +2.9%. Every depth is inside the host's ~2% noise floor, so the effect is smaller than this rig can resolve, and the d=0 -2.7% is no more real than the d=2048 +1.3%. So: at ub=2048, J=48 and J=64 are the same speed. The doc predicted this before the numbers existed ("narrow enough that it may well land in noise"), which is the only reason that reading is credible rather than post-hoc. Reverting to the static cap because it is the validated status quo, not because it won. Worth reporting on ggml-org#24546: the picker is neutral on RDNA3.5, on the exact model and ubatch its sweep claims +7.57% for on gfx1100. The PR is gated on GGML_CUDA_CC_IS_RDNA3, which covers both. gfx1151's table is I=64/ nthreads=128 vs gfx1100's I=128/nthreads=256 - our tiles are already half-width, so there is much less worst-case over-sizing left to recover. That is the data ravel7524 asked for on 2026-07-03. Doc kept with the numbers and the untested ub>=4096 crossover regime recorded. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
justinappler
added a commit
to justinappler/llama.cpp-strix-halo
that referenced
this pull request
Jul 22, 2026
…24546) Upstream PR ggml-org#24546 (ravel7524, open) sizes routed-MoE MMQ N-tiles from the typical expert width instead of the worst case. That is the same problem the static J_max=48 cap from Findings ggml-org#8/ggml-org#9 solves, approached from the other end: the cap clamps the search ceiling, the picker fixes the search objective. This commit swaps the cap for the picker so the two can be A/B'd on gfx1151. The PR does not apply as written - it patches mmq_x/mmq_x_max/ mmq_get_granularity_host in mul_mat_q_case, all deleted by PR ggml-org#24127, the same refactor that forced the Finding ggml-org#5 -> ggml-org#9 re-port. Re-authored onto mul_mat_q_switch_J's config-table search and scoped to RDNA3.5: gfx1151 is the only arch we can measure, and narrowing the predicate keeps the A/B single-variable. Upstream's predicate already covers RDNA3.5 via GGML_CUDA_CC_IS_RDNA3, and the author confirmed on 2026-07-04 that Strix Halo should benefit, but nobody has ever tested it there. On the routed-MoE path ncols_max is ne12 - the worst case of one expert receiving every token - so the search never reaches ntiles==1 and simply runs to whatever ceiling it is given. The picker instead tiles from ncols_dst/nchannels_x, the width a typical expert covers. launch_mul_mat_q still builds the grid from ncols_max, so worst-case coverage is unchanged. - mmq.cuh: replace the RDNA3.5 MoE J_max=48 clamp with the typical-width picker; J_max is now a uniform 128 and the search terminates on its own. - mmq-table-check.cpp: pick_J mirrors the selection loop, so update it in lockstep or the check silently becomes a rubber stamp. Adds the routed-MoE arg shape from mmq.cu and a crossover case. For Qwen 3.6 35B-A3B (256 experts, 8 active) at the production ub=2048 the typical width is 2048*8/256 = 64, so this is a 48 -> 64 change at our operating point - narrow enough that it may land in noise. Verified host-side without ROCm: MoE J=64, dense J=128, no aborts across all 21 types. Above ub=4096 the typical width reaches 128, the picker disengages and the search runs to J=128 where the cap held 48. Findings ggml-org#5/ggml-org#8 only ever measured the typical=64 regime, so that is untested on this chip; production ub=2048 cannot reach it. See strix-halo/mmq-moe-ncols-picker.md for the bench plan and the keep/revert criteria. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
justinappler
added a commit
to justinappler/llama.cpp-strix-halo
that referenced
this pull request
Jul 22, 2026
Benched b47bb31 against 05e837f on gfx1151 (ROCm 7.14.0, canonical Qwen 3.6 35B-A3B Q4_K_XL matrix). The picker is not distinguishable from the static cap at the production operating point: pp512 @ d=0 1428.13 -> 1388.93 (-2.7%) pp512 @ d=2048 1299.39 -> 1316.26 (+1.3%) pp512 @ d=8192 1135.42 -> 1142.23 (+0.6%) pp512 @ d=16384 971.25 -> 977.36 (+0.6%) Correctness was clean first (790/790 MUL_MAT_ID, 1134/1134 MUL_MAT), so the grid does still cover ncols_max while the tile is sized from the typical width. The tg128 control fell ~1.6% at every depth. This change cannot move tg - decode goes through MMVQ where ncols_dst==1, so the picker never runs - which makes that a session drift between two builds measured on different days, the same contamination Finding ggml-org#9 hit. The conclusion survives it either way: raw, pp is -2.7% to +1.3%; calibrated against the drift, -1.2% to +2.9%. Every depth is inside the host's ~2% noise floor, so the effect is smaller than this rig can resolve, and the d=0 -2.7% is no more real than the d=2048 +1.3%. So: at ub=2048, J=48 and J=64 are the same speed. The doc predicted this before the numbers existed ("narrow enough that it may well land in noise"), which is the only reason that reading is credible rather than post-hoc. Reverting to the static cap because it is the validated status quo, not because it won. Worth reporting on ggml-org#24546: the picker is neutral on RDNA3.5, on the exact model and ubatch its sweep claims +7.57% for on gfx1100. The PR is gated on GGML_CUDA_CC_IS_RDNA3, which covers both. gfx1151's table is I=64/ nthreads=128 vs gfx1100's I=128/nthreads=256 - our tiles are already half-width, so there is much less worst-case over-sizing left to recover. That is the data ravel7524 asked for on 2026-07-03. Doc kept with the numbers and the untested ub>=4096 crossover regime recorded. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
justinappler
added a commit
to justinappler/llama.cpp-strix-halo
that referenced
this pull request
Aug 3, 2026
…24546) Upstream PR ggml-org#24546 (ravel7524, open) sizes routed-MoE MMQ N-tiles from the typical expert width instead of the worst case. That is the same problem the static J_max=48 cap from Findings ggml-org#8/ggml-org#9 solves, approached from the other end: the cap clamps the search ceiling, the picker fixes the search objective. This commit swaps the cap for the picker so the two can be A/B'd on gfx1151. The PR does not apply as written - it patches mmq_x/mmq_x_max/ mmq_get_granularity_host in mul_mat_q_case, all deleted by PR ggml-org#24127, the same refactor that forced the Finding ggml-org#5 -> ggml-org#9 re-port. Re-authored onto mul_mat_q_switch_J's config-table search and scoped to RDNA3.5: gfx1151 is the only arch we can measure, and narrowing the predicate keeps the A/B single-variable. Upstream's predicate already covers RDNA3.5 via GGML_CUDA_CC_IS_RDNA3, and the author confirmed on 2026-07-04 that Strix Halo should benefit, but nobody has ever tested it there. On the routed-MoE path ncols_max is ne12 - the worst case of one expert receiving every token - so the search never reaches ntiles==1 and simply runs to whatever ceiling it is given. The picker instead tiles from ncols_dst/nchannels_x, the width a typical expert covers. launch_mul_mat_q still builds the grid from ncols_max, so worst-case coverage is unchanged. - mmq.cuh: replace the RDNA3.5 MoE J_max=48 clamp with the typical-width picker; J_max is now a uniform 128 and the search terminates on its own. - mmq-table-check.cpp: pick_J mirrors the selection loop, so update it in lockstep or the check silently becomes a rubber stamp. Adds the routed-MoE arg shape from mmq.cu and a crossover case. For Qwen 3.6 35B-A3B (256 experts, 8 active) at the production ub=2048 the typical width is 2048*8/256 = 64, so this is a 48 -> 64 change at our operating point - narrow enough that it may land in noise. Verified host-side without ROCm: MoE J=64, dense J=128, no aborts across all 21 types. Above ub=4096 the typical width reaches 128, the picker disengages and the search runs to J=128 where the cap held 48. Findings ggml-org#5/ggml-org#8 only ever measured the typical=64 regime, so that is untested on this chip; production ub=2048 cannot reach it. See strix-halo/mmq-moe-ncols-picker.md for the bench plan and the keep/revert criteria. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
justinappler
added a commit
to justinappler/llama.cpp-strix-halo
that referenced
this pull request
Aug 3, 2026
Benched b47bb31 against 05e837f on gfx1151 (ROCm 7.14.0, canonical Qwen 3.6 35B-A3B Q4_K_XL matrix). The picker is not distinguishable from the static cap at the production operating point: pp512 @ d=0 1428.13 -> 1388.93 (-2.7%) pp512 @ d=2048 1299.39 -> 1316.26 (+1.3%) pp512 @ d=8192 1135.42 -> 1142.23 (+0.6%) pp512 @ d=16384 971.25 -> 977.36 (+0.6%) Correctness was clean first (790/790 MUL_MAT_ID, 1134/1134 MUL_MAT), so the grid does still cover ncols_max while the tile is sized from the typical width. The tg128 control fell ~1.6% at every depth. This change cannot move tg - decode goes through MMVQ where ncols_dst==1, so the picker never runs - which makes that a session drift between two builds measured on different days, the same contamination Finding ggml-org#9 hit. The conclusion survives it either way: raw, pp is -2.7% to +1.3%; calibrated against the drift, -1.2% to +2.9%. Every depth is inside the host's ~2% noise floor, so the effect is smaller than this rig can resolve, and the d=0 -2.7% is no more real than the d=2048 +1.3%. So: at ub=2048, J=48 and J=64 are the same speed. The doc predicted this before the numbers existed ("narrow enough that it may well land in noise"), which is the only reason that reading is credible rather than post-hoc. Reverting to the static cap because it is the validated status quo, not because it won. Worth reporting on ggml-org#24546: the picker is neutral on RDNA3.5, on the exact model and ubatch its sweep claims +7.57% for on gfx1100. The PR is gated on GGML_CUDA_CC_IS_RDNA3, which covers both. gfx1151's table is I=64/ nthreads=128 vs gfx1100's I=128/nthreads=256 - our tiles are already half-width, so there is much less worst-case over-sizing left to recover. That is the data ravel7524 asked for on 2026-07-03. Doc kept with the numbers and the untested ub>=4096 crossover regime recorded. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
I think this is a reasonable idea, but it would have to be updated for the changes in #24127 |
ravel7524
force-pushed
the
mmqx-rdna3-routed-moe-tiling
branch
from
August 5, 2026 18:48
9ab2b07 to
3087cea
Compare
Contributor
Author
|
I adapted the implementation to the MMQ configuration changes from #24127 and I provisionally enabled the selector for NVIDIA Volta and newer so those architectures can be benchmarked. |
IMbackK
requested changes
Aug 5, 2026
| GGML_CUDA_CC_IS_RDNA2(cc) || | ||
| GGML_CUDA_CC_IS_RDNA3(cc) || | ||
| GGML_CUDA_CC_IS_RDNA4(cc); | ||
| } |
Contributor
There was a problem hiding this comment.
Instead of this, please enroll this tunable in the ggml_cuda_mmq_config table introduced in the linked pr
Contributor
Author
There was a problem hiding this comment.
Ok just Updated the PR is this how you want it ?
…ols_min_cc support
Contributor
|
I tested it on CDNA 1 and it seams that its not entirely a win in all cases.
|
IMbackK
previously approved these changes
Aug 6, 2026
IMbackK
left a comment
IMbackK
left a comment
Contributor
There was a problem hiding this comment.
Dont bother cleaning this up if its not uniformly performance positive at least on rdna3
IMbackK
self-requested a review
August 6, 2026 19:15
…configuration files
Contributor
Author
|
Benchmark Sweep, on newest changes RDNA3/ W7800:
Benchmark Sweep Detailed Results
|
Contributor
|
Thanks, ill give it another sweep myself on rdna, other than that i think this is solid now. |
baramofme
added a commit
to baramofme/llama-cpp-turboquant
that referenced
this pull request
Aug 15, 2026
- ggml/src/ggml-hip: Adding CDNA, RDNA2 and RDNA4 architecture support - ggml/src/ggml-hip: Indentation cleanup - ggml/src/ggml: adjust ncols_picker for routed MoE in mul_mat_q_case - server: fix reasoning budget WebUI precedence over model.ini Assisted-by: Sisyphus
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This PR improves routed MoE prefill performance on RDNA3 by changing how llama.cpp chooses MMQ N-tile sizes. Instead of using the worst case expert width, it estimates the typical routed expert width and uses that when it is smaller than the RDNA3 maximum tile width. The launch grid still uses the original worst case width, so output coverage and correctness are preserved. The change is limited to host side tile selection in
mul_mat_q_caseand does not modify structs, call sites, launch parameters, or device kernels. Benchmarks on a Radeon Pro W7800/gfx1100 with ROCm 7.2.3 show large prefill gains in many routed MoE cases, while decode and non RDNA3 architectures are mostly unchanged.If maintainers or other contributors have access to NVIDIA, CDNA, RDNA2, or RDNA4 hardware, the natural follow up would be to test the same routed width picker under the corresponding MMQ/launch policy and enable it per architecture where it is beneficial.
Benchmark Summary
Hardware/software:
-ngl 99GGML_CUDA_FA_ALL_QUANTS=ON-fa 1 -ctk q8_0 -ctv f16 -sm none -t 4 --poll 0 -r 5The improvement depends on the physical ubatch/chunk width that reaches routed MoE MMQ, not total context length:
So long prompts can still benefit when they are processed in smaller ubatch chunks.
Benchmark sweep
The 128 expert models show the expected crossover behavior at ubatch 2048:
pp2048and larger are neutral because the typical width has reachedmmq_x_max = 128. The 256-expert Qwen3.6 shape remains below crossover and still improves throughpp8192.Requirements