Change sets 6-7: CDNA MMQ tile retune + SSD chunk size (pp16384 1775 → 2590, +45.9%)

[davetha]

Continues the CDNA2 prefill work. pp16384 2590 t/s vs 1775 upstream (+45.9%); gap to vLLM+AITER narrows from 2.29× to 1.57×.

build	pp4096	pp16384	vs base
upstream 67b9b0e	1366	1775	—
+ SSD on CDNA (4)	1624	2114	+19.1%
+ stream-k off, K-quants (5)	1684	2192	+23.5%
+ stream-k off, all types (5)	1834	2374	+33.7%
+ MMQ tile retune (6)	1962	2529	+42.5%
+ SSD chunk 128 (7)	2013	2590	+45.9%

Total GPU kernel time at pp4096: 7703 ms → 5331 ms (−31%).

5 widened — the K-quant scope was wrong

The three hottest MMQ kernels are types 12, 6, 8 = Q4_K, Q5_0, Q8_0, and the latter two are 27.4% of prefill on their own. They appear in a Q4_K_M model because llama-quant.cpp downgrades Q4_K→Q5_0 and Q6_K→Q8_0 for any tensor whose column count isn't divisible by 256 — which Nemotron's Mamba-2 projection widths frequently aren't. Covering all types: +13.0% / +12.3%.

6. MMQ tile retune (+6.5%)

mmq-config-cdna.cuh has never been tuned on CDNA hardware. PR #24127 transcribed pre-refactor blanket-AMD constants (nthreads=512 was tuned for gfx942; I=128 was the catch-all AMD value) and benchmarked only MI100, RX 6800 and a Radeon 8060S — no MI210/MI250/MI300.

nthreads 512→256, I 128→64: 1962 / 2529 t/s, and halves LDS per workgroup (48.2 → 28.2 KiB).

Two invariants that make this dangerous

• I == nthreads/4 on CDNA. rows_per_warp is hardcoded to 16 on the MFMA path with no loop over the row index.
• I must also divide 128 — the out-of-bounds fallback comes from a hardcoded args.nrows_x % 128 == 0, not from config.I. So 384/I=96 satisfies the first rule and still fails every quant type.

Violating either doesn't error. It computes a partial tile — and benchmarks faster for exactly that reason.

Four configs were faster and wrong

config	apparent gain	reality
I=64 alone	+29%	362 + 637 test failures
nthreads=256 alone	+23%	11 + 595 failures
384 / I=96	+2%	fails every quant type
MMQ_ITER_K=512	+25%	263 + 541 failures

The fastest number measured in the entire session was wrong. README documents all four, plus the detail that the harness prints failures as [MUL_MAT] ERR = ... : FAIL — a grep anchored to leading whitespace reports a clean run for a broken build.

7. SSD chunk size 256 → 128 (+2.5%)

Upstream value was tuned against cuBLAS. rocBLAS on gfx90a prefers half: 64→1932, 128→2013, 192→2005, 256→1964, 512→1867 (pp4096). All numerically correct; pure throughput choice.

Also rejected

• J=128/J=96: −26%/−22%, correct but slower — CDNA's J≤64 cap is right for gfx90a (LDS was never the constraint; J=128 fits in 37.5 KiB)
• occupancy: no-op — LLVM clamps a waves-per-EU request below the workgroup-derived default
• Forcing MMQ for the dense FP16 GEMMs, retested after stream-k removed the original objection: still −10.5%. rocBLAS genuinely wins those shapes.
• -ub 512/1024/4096, -b 2048/8192, -ctk f16: all already optimal or neutral

Verification

test-backend-ops MUL_MAT 1134/1134, MUL_MAT_ID 790/790, SSM_SCAN 7/7 per device, plus generated tokens at temperature 0. Patches verified to apply in sequence to upstream 67b9b0e and reproduce modified-files/ byte-for-byte.