CUDA: size MMQ ids-path tail padding from the flattened row count, not ne11

[glennneuber]

Overview

The ids branch of ggml_cuda_mul_mat_q() sizes the src1_q8_1 allocation from two different row counts. The data term uses ne12*n_expert_used. The tail-padding term uses ne11. The correct value is already computed a few lines below as ne11_flat.

For MoE gate/up projections the activations are broadcast across experts, so ne11 == 1. ggml_cuda_mmq_get_J_max() with ne11 == 1 computes min(1,512) = 1, then 1 - 1%8 = 0, skips its loop and returns 0. The buffer gets no tail padding at all, while MMQ reads in tiles of up to J_max = 512 rows and runs past the end.

ffn_down is affected too but less: it passes ne11 = 8, so padding is sized for 8 rows instead of ne12*n_expert_used.

The !ids branch above is correct, because there ne11 really is the buffer's row count.

This patch uses ne12*n_expert_used for the padding term.

Additional information

Hit this on a MoE vision model (256 experts, 8 used, q4_K gate/up) on sm_120 / CUDA 13.0 when a whole image was submitted as one 2040-token ubatch:

decoding image batch 1/1, n_tokens_batch = 2040
find_slot: non-consecutive token position 4 after 3 for sequence 0 with 2040 new tokens
ggml-cuda.cu:106: CUDA error
ggml_cuda_compute_forward: MUL_MAT_ID failed
CUDA error: an illegal memory access was encountered
  current device: 0, in function ggml_cuda_compute_forward at ggml-cuda.cu:2374

The failing node is ffn_moe_gate, with ne2=2040 ne02=256 ne11=1 ne12=2040, type q4_K, taking the mmq branch. The find_slot line also shows up on runs that do not crash, so it marks the path rather than the fault.

Notes for testing:

• test-backend-ops does not catch it. The over-read lands in padding rows the kernel discards, so output is unchanged and NMSE is unaffected. compute-sanitizer --tool memcheck does flag it.
• A passing run does not prove much. The overrun is a fixed size past the end, so it only faults when it crosses an unmapped page. The same request crashes on a fresh pool and passes after the pool has served a larger allocation.
• Extra memory is at most 512 * sizeof(block_q8_1_mmq) = 72 KB per call, and does not grow with ne12.

Tested with 4 cold runs on the patched build (no fault) against 2 unpatched controls from the same tree (both fault).

May be related to #24399, #19705 and #18331. I have not reproduced those configurations, so this is a guess, but the GGML_CUDA_FORCE_CUBLAS=ON workaround in #24399 skips MMQ entirely and requantising changes the allocation size, which would both hide this.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - I used AI to investigate the crash, locate the cause and produce the patch. I have reviewed the change and can explain it.

[ggml-gh-bot]

Hi @glennneuber, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

• AI-generated content: While code is allowed to be generated by AI, please write the PR description and commit messages on your own without the help of AI.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[glennneuber]

This is a regression imho. I was able to pin it to a commit.

It was introduced by 6eddde0 ("CUDA: refactor MMQ kernel configuration", #24127), build b9992. That commit changed the padding term in both allocation branches:

-            get_mmq_x_max_host(cc)*sizeof(block_q8_1_mmq);
+            ggml_cuda_mmq_get_J_max(src0->type, fallback, cc, ne11) * sizeof(block_q8_1_mmq);
-        get_mmq_x_max_host(cc)*sizeof(block_q8_1_mmq);
+        ggml_cuda_mmq_get_J_max(src0->type, fallback, cc, ne11) * sizeof(block_q8_1_mmq);

That is correct for the !ids branch, where ne11 is the row count. In the ids branch the row count is ne12*n_expert_used, so passing ne11 is wrong there.

The difference matters because the two functions have different failure modes. get_mmq_x_max_host(cc) only looks at the architecture and returns 128 or 64, so the old code always allocated some padding no matter what the shape was. ggml_cuda_mmq_get_J_max() looks at ne11 as well, and for ne11 == 1 it computes min(1, 512) = 1, then 1 - 1 % 8 = 0, skips its loop and returns 0. So the broadcast case gets no padding at all rather than a bit too little.

Last clean build is b9990 (259ae1d), first affected is b9992. b9991 is not tagged and the only other commit in that range is Vulkan-only.

Bisected by source, then checked at runtime on both sides of the boundary. Same GPU, same request, cold server, first request each time, n_ubatch = 2048 and n_tokens_batch = 2040 in every log:

build	result
b9888 (ollama 0.32.1)	no fault, 3/3 cold runs
b10069 (ollama 0.32.2)	illegal memory access
b10353	illegal memory access

For anyone hitting this through ollama: v0.32.1 is the last release on a clean llama.cpp, v0.32.2 is the first affected.