perf(qwen-flash): accelerate prefill and MTP catch-up

NOTICE

@@ -16,6 +16,8 @@ Adapted code includes:
 - ccv ancestry through h3.c: Copyright (c) 2010 Liu Liu (BSD-3-Clause).
 - ROCm kernels/headers: AMD and the contributors identified in their notices.
 - hipCUB: Duane Merrill, NVIDIA CORPORATION and AMD (BSD-3-Clause).
+- Flash-Next GDN row reuse and SSM stride optimization: adapted from
+  mikealanni/gufo f450218743bc, Copyright (c) 2026 gufo contributors (MIT).
 
 Strix Halo optimization inspiration also came from these MIT-licensed projects:
 - LaurentZuijdwijk/llama.cpp: https://github.com/LaurentZuijdwijk/llama.cpp

README.md

@@ -22,7 +22,7 @@ All model documentation lives under [docs/models](docs/models/README.md):
 | Model | Inference modes | Hugging Face weights | Benchmarks | Quality |
 | --- | --- | --- | --- | --- |
 | [Qwen3.8 27B](docs/models/qwen3.8-27b/README.md) | Q4/Q8, images, AR, DFlash2 | Unsloth [Q4_K_XL](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/4ca720788d1e01f1bff70c033e0d0028fd02e502/Qwen3.8-27B-UD-Q4_K_XL.gguf) / [Q8_K_XL](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/4ca720788d1e01f1bff70c033e0d0028fd02e502/Qwen3.8-27B-UD-Q8_K_XL.gguf) · [DFlash2 Q4_K_M](https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2-GGUF/blob/2d9571f8ce46e151f61c6499c99dee6079e1d610/Qwen3.8-27B-DFlash2-Q4_K_M.gguf) | Q4: **656.33 tok/s pp**; up to **70.56 tok/s tg** single user and **123.00 aggregated tok/s** on 8 concurrent requests with DFlash2 · [Benchmarks](docs/models/qwen3.8-27b/BENCHMARKS.md) | [Quality](docs/models/qwen3.8-27b/QUALITY.md) |
-| [Qwen3.8 Flash-Next](docs/models/qwen3.8-flash-next/README.md) | Q4, images, AR, MTP | Unsloth [Q4_K_XL](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/38bb39ee97821de2c9009abb7e93950eec396e66/UD-Q4_K_XL) · [MTP Q8_0](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/blob/38bb39ee97821de2c9009abb7e93950eec396e66/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf) | **1,644.68 tok/s pp**; up to **59.02 tok/s tg** single user and **157.22 aggregated tok/s** on 8 concurrent requests with MTP · [Benchmarks](docs/models/qwen3.8-flash-next/BENCHMARKS.md) | [Quality](docs/models/qwen3.8-flash-next/QUALITY.md) |
+| [Qwen3.8 Flash-Next](docs/models/qwen3.8-flash-next/README.md) | Q4, images, AR, MTP | Unsloth [Q4_K_XL](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/38bb39ee97821de2c9009abb7e93950eec396e66/UD-Q4_K_XL) · [MTP Q8_0](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/blob/38bb39ee97821de2c9009abb7e93950eec396e66/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf) | **1,700.52 tok/s pp**; up to **60.39 tok/s tg** single user and **162.98 aggregated tok/s** on 8 concurrent requests with MTP · [Benchmarks](docs/models/qwen3.8-flash-next/BENCHMARKS.md) | [Quality](docs/models/qwen3.8-flash-next/QUALITY.md) |
 | [DeepSeek V4 Flash](docs/models/deepseek-v4-flash/README.md) | AR, DSpark | [antirez Flash 0731 IQ2XXS](https://huggingface.co/antirez/deepseek-v4-gguf/blob/1cd7b564460821938add0475a60b942c409295e0/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf) · [DSpark](https://huggingface.co/antirez/deepseek-v4-gguf/blob/e7f04037032990db0346398d249baf9fb9df1ccc/DeepSeek-V4-Flash-DSpark-support-0731.gguf) | **484.62 tok/s pp**; up to **26.62 tok/s tg** single user and **54.74 aggregated tok/s** on 8 concurrent requests with DSpark · [Benchmarks](docs/models/deepseek-v4-flash/BENCHMARKS.md) | [Quality](docs/models/deepseek-v4-flash/QUALITY.md) |
 | [Qwen3-ASR 1.7B](docs/models/qwen3-asr/README.md) | Speech recognition | [BF16](https://huggingface.co/Qwen/Qwen3-ASR-1.7B/tree/7278e1e70fe206f11671096ffdd38061171dd6e5) | **15.27× realtime** · [Benchmarks](docs/models/qwen3-asr/BENCHMARKS.md) | [Quality](docs/models/qwen3-asr/QUALITY.md) |
 | [Qwen3-TTS 1.7B](docs/models/qwen3-tts/README.md) | Speech synthesis and voice cloning | BF16 [CustomVoice](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice) / [VoiceDesign](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign) / [Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base) | Up to **2.54× realtime**; **201 ms** to first audio (CustomVoice) · [Benchmarks](docs/models/qwen3-tts/BENCHMARKS.md) | [Quality](docs/models/qwen3-tts/QUALITY.md) |

THIRD_PARTY_NOTICES.md

@@ -29,6 +29,7 @@ Non-Nix builds must retain the notices of the versions they actually distribute.
 | FFmpeg | Separate ffmpeg/ffprobe executables for media | `GPL-3.0-or-later` (Nix build with GPL/version3 components) | 8.1.2; flake.lock | [FFmpeg](https://github.com/FFmpeg/FFmpeg) |
 | GNU C/C++/OpenMP runtimes | System runtime libraries; no Gufo source import | `LGPL-2.1-or-later AND (GPL-3.0-or-later WITH GCC-exception-3.1)` | glibc/GCC packages in flake.lock | [GNU](https://www.gnu.org/software/) |
 | llama.cpp / ggml | Adapted quantization, attention and model-private HIP kernels | `MIT` | `5c0e9468378eba6bf3cc1989ff5d62fbbe4d9e3a`; attention `e9fa0781f1c25fc4fe8c86be1edc6970661ad6f0` | [llama.cpp](https://github.com/ggml-org/llama.cpp) |
+| mikealanni/gufo | Adapted Flash-Next GDN row reuse and SSM stride optimization | `MIT` | `f450218743bcaf77e3e1f8fab78f4cd4e4aafae3` | [gufo fork](https://github.com/mikealanni/gufo/tree/f450218743bcaf77e3e1f8fab78f4cd4e4aafae3) |
 | Qwen chat templates | Reference Jinja and adapted renderer behavior | `Apache-2.0` | `1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0` (27B), `de4b8e4d43b917e7706784d8bb445c9af86a3540` (Flash-Next) | [Qwen](https://huggingface.co/Qwen/Qwen3.8-27B) |
 | DS4 | Adapted loader, tokenizer, sessions and HIP kernels | `MIT` | `84cc882352757baf628a1776badf7cc54d584e28` | [antirez/ds4](https://github.com/antirez/ds4) |
 | DS4 GB10/GX10 fork | Adapted paired MoE launch code, now HIP | `MIT` | `910501e` | [xangel82/DS4](https://github.com/xangel82/DS4-GB10-GX10-DSpark-CUDA) |
@@ -47,6 +48,9 @@ and [H3 quality record](docs/models/minimax-h3/QUALITY.md).
 
 This software is based in part on the work of the Independent JPEG Group.
 
+The mikealanni/gufo fork retains the same Copyright (c) 2026 gufo contributors
+notice and MIT text as Gufo's root `LICENSE`.
+
 FFmpeg is invoked as a separate process, not linked into Gufo. The pinned Nix
 build enables GPL/version3 components; a system FFmpeg build can have different
 terms. Redistributors bundling FFmpeg or other copyleft dependencies must also

docs/models/qwen3.8-flash-next/BENCHMARKS.md

@@ -2,8 +2,8 @@
 
 AMD Strix Halo `gfx1151`, 128 GB unified memory. Unsloth `UD-Q4_K_XL` target and
 shared-Q8_0 MTP sidecar; Gufo uses adaptive MTP. HTTP, greedy, thinking off.
-Gufo single-user pp/tg: October 7, 2026 (`def2ed1e`), Performance power profile.
-Concurrency, loading, memory and reference results: September 22–23.
+Gufo pp/tg, concurrency and memory: October 8–9, 2026, production Nix build.
+Loading and reference results: September 22–23.
 llama.cpp uses `b11069` for AR and `6fcaa16f` for MTP.
 
 Positive gain favors Gufo.
@@ -17,14 +17,14 @@ Context capacities differ between engines; see the measurement details.
 <!-- bench:single-ar -->
 | Flash-Next Q4 AR<br>Depth (tokens) | Gufo pp (tok/s) | llama.cpp pp (tok/s) | Gain | Gufo tg (tok/s) | llama.cpp tg (tok/s) | Gain |
 | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
-| 0 | 1644.68 | 489.59 | +235.9% | 25.99 | 22.20 | +17.1% |
-| 4,096 | 1518.57 | 455.24 | +233.6% | 25.73 | 21.21 | +21.3% |
-| 8,192 | 1567.32 | 428.85 | +265.5% | 25.99 | 20.40 | +27.4% |
-| 12,288 | 1560.97 | 400.21 | +290.0% | 25.96 | 19.64 | +32.2% |
-| 16,384 | 1556.02 | 375.89 | +314.0% | 25.93 | 18.95 | +36.8% |
-| 32,768 | 1546.81 | 301.31 | +413.4% | 25.80 | 16.54 | +56.0% |
-| 65,536 | 1425.49 | 221.94 | +542.3% | 25.43 | 11.62 | +118.8% |
-| 131,072 | 1446.10 | 144.78 | +898.8% | 24.97 | 7.98 | +212.9% |
+| 0 | 1700.52 | 489.59 | +247.3% | 25.95 | 22.20 | +16.9% |
+| 4,096 | 1618.60 | 455.24 | +255.5% | 25.84 | 21.21 | +21.8% |
+| 8,192 | 1611.53 | 428.85 | +275.8% | 25.96 | 20.40 | +27.3% |
+| 12,288 | 1602.99 | 400.21 | +300.5% | 25.94 | 19.64 | +32.1% |
+| 16,384 | 1597.41 | 375.89 | +325.0% | 25.90 | 18.95 | +36.7% |
+| 32,768 | 1585.98 | 301.31 | +426.4% | 25.78 | 16.54 | +55.9% |
+| 65,536 | 1509.10 | 221.94 | +580.0% | 25.40 | 11.62 | +118.6% |
+| 131,072 | 1478.66 | 144.78 | +921.3% | 24.92 | 7.98 | +212.3% |
 <!-- /bench -->
 
 ![Single user, autoregressive](artifacts/charts/single-ar.svg)
@@ -37,14 +37,14 @@ text, including Gufo predictor catch-up.
 <!-- bench:single-mtp -->
 | Flash-Next Q4 MTP<br>Depth (tokens) | Gufo pp (tok/s) | llama.cpp pp (tok/s) | Gain pp | Gufo tg mixed (tok/s) | llama.cpp tg mixed (tok/s) | Gain mixed | Gufo tg repetitive (tok/s) | llama.cpp tg repetitive (tok/s) | Gain repetitive |
 | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
-| 0 | 1560.11 | 468.76 | +232.8% | 30.16 | 31.73 | -4.9% | 59.02 | 48.12 | +22.7% |
-| 4,096 | 1529.17 | 423.79 | +260.8% | 32.86 | 34.48 | -4.7% | 57.58 | 45.71 | +26.0% |
-| 8,192 | 1525.11 | 394.98 | +286.1% | 32.47 | 32.20 | +0.8% | 49.26 | 44.31 | +11.2% |
-| 12,288 | 1517.04 | 365.91 | +314.6% | 34.78 | 32.32 | +7.6% | 42.19 | 43.53 | -3.1% |
-| 16,384 | 1502.95 | 341.98 | +339.5% | 35.27 | 32.07 | +10.0% | 50.60 | 43.66 | +15.9% |
-| 32,768 | 1503.08 | 277.61 | +441.4% | 34.27 | 25.41 | +34.9% | 44.98 | 39.28 | +14.5% |
-| 65,536 | 1390.03 | 208.04 | +568.2% | 33.51 | 19.65 | +70.5% | 43.12 | 27.75 | +55.4% |
-| 131,072 | 1408.06 | 135.42 | +939.8% | 33.18 | 14.63 | +126.8% | 44.43 | 20.96 | +112.0% |
+| 0 | 1642.85 | 468.76 | +250.5% | 31.32 | 31.73 | -1.3% | 60.39 | 48.12 | +25.5% |
+| 4,096 | 1575.60 | 423.79 | +271.8% | 32.48 | 34.48 | -5.8% | 57.66 | 45.71 | +26.1% |
+| 8,192 | 1573.54 | 394.98 | +298.4% | 32.38 | 32.20 | +0.6% | 49.28 | 44.31 | +11.2% |
+| 12,288 | 1559.07 | 365.91 | +326.1% | 34.68 | 32.32 | +7.3% | 42.15 | 43.53 | -3.2% |
+| 16,384 | 1547.52 | 341.98 | +352.5% | 35.09 | 32.07 | +9.4% | 50.58 | 43.66 | +15.8% |
+| 32,768 | 1541.35 | 277.61 | +455.2% | 34.20 | 25.41 | +34.6% | 44.54 | 39.28 | +13.4% |
+| 65,536 | 1466.98 | 208.04 | +605.1% | 33.49 | 19.65 | +70.4% | 44.45 | 27.75 | +60.2% |
+| 131,072 | 1436.89 | 135.42 | +961.1% | 33.07 | 14.63 | +126.0% | 44.26 | 20.96 | +111.2% |
 <!-- /bench -->
 
 ![Single user, MTP](artifacts/charts/single-mtp.svg)
@@ -58,11 +58,11 @@ llama.cpp re-evaluates its four-token checkpoint tail.
 <!-- bench:multi-ar -->
 | Flash-Next Q4 AR<br>Users | Gufo AR (tok/s) | llama.cpp AR (tok/s) | Gain |
 | ---: | ---: | ---: | ---: |
-| 1 | 25.85 | 22.34 | +15.7% |
-| 2 | 45.70 | 37.08 | +23.2% |
-| 4 | 76.29 | 54.82 | +39.2% |
-| 6 | 95.66 | 65.34 | +46.4% |
-| 8 | 108.67 | 68.79 | +58.0% |
+| 1 | 25.84 | 22.34 | +15.7% |
+| 2 | 45.77 | 37.08 | +23.4% |
+| 4 | 76.33 | 54.82 | +39.2% |
+| 6 | 95.75 | 65.34 | +46.5% |
+| 8 | 108.68 | 68.79 | +58.0% |
 <!-- /bench -->
 
 ![Multiple users, autoregressive](artifacts/charts/multi-ar.svg)
@@ -76,11 +76,11 @@ request decode rates. C1 cross-checks the single-user table.
 <!-- bench:multi-mtp -->
 | Flash-Next Q4 MTP<br>Users | Gufo mixed (tok/s) | llama.cpp mixed (tok/s) | Gain | Gufo repetitive (tok/s) | llama.cpp repetitive (tok/s) | Gain |
 | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
-| 1 | 32.09 | 31.77 | +1.0% | 59.20 | 46.93 | +26.1% |
-| 2 | 51.68 | 42.79 | +20.8% | 93.16 | 52.57 | +77.2% |
-| 4 | 75.92 | 48.78 | +55.6% | 127.91 | 48.23 | +165.2% |
-| 6 | 89.18 | 52.59 | +69.6% | 142.14 | 48.51 | +193.0% |
-| 8 | 106.47 | 61.92 | +71.9% | 157.22 | 58.13 | +170.5% |
+| 1 | 31.28 | 31.77 | -1.5% | 60.10 | 46.93 | +28.1% |
+| 2 | 52.22 | 42.79 | +22.0% | 84.52 | 52.57 | +60.8% |
+| 4 | 76.16 | 48.78 | +56.1% | 131.10 | 48.23 | +171.8% |
+| 6 | 96.01 | 52.59 | +82.6% | 145.38 | 48.51 | +199.7% |
+| 8 | 107.77 | 61.92 | +74.0% | 162.98 | 58.13 | +180.4% |
 <!-- /bench -->
 
 ![Multiple users, MTP](artifacts/charts/multi-mtp.svg)
@@ -104,8 +104,8 @@ C1, context capacity 133121, AR. Peak memory reported by HIP.
 <!-- bench:memory -->
 | Flash-Next Q4 AR<br>Workload | Gufo GiB | llama.cpp GiB | Gain |
 | --- | ---: | ---: | ---: |
-| pp2048 + tg128 | 85.55 | 84.84 | -0.8% |
-| 16K prefix, pp4096 + tg128 | 86.27 | 85.31 | -1.1% |
+| pp2048 + tg128 | 85.56 | 84.84 | -0.8% |
+| 16K prefix, pp4096 + tg128 | 86.71 | 85.31 | -1.6% |
 <!-- /bench -->
 
 ![Memory occupation](artifacts/charts/memory.svg)

docs/models/qwen3.8-flash-next/EXPERIMENTS.md

@@ -2,6 +2,21 @@
 
 | Experiment | Decision / evidence |
 | --- | --- |
+| Cheaper expert selection, direct MTP normalization and partial SSM tiles | Retained: exact dequantization and live matrix operations, no added allocation. Mixed pp2048 d0/d32K: AR 1697.86/1534.50 tok/s; MTP 1650.29/1497.00. Separate d128K code continuation improves 1381.32 → 1400.23 tok/s AR and 1341.04 → 1361.96 MTP. Default sampled MTP PP improves 1543.99 → 1584.22 tok/s; TG 38.79 → 39.04. One sample per control; outputs, counts and acceptance match. [Evidence](artifacts/prefill-long-context.json). |
+| Four-phase expert loop and rolling boundary convolution | Rejected: exact microbenchmarks improved, but full HTTP did not establish a useful gain. Reverse d0 AR: four-phase 1690.95 versus simpler loop 1692.07 tok/s; deep AR/MTP were slightly slower. Both specializations removed. [Evidence](artifacts/prefill-long-context.json). |
+| Further barriers, selector scheduling and tail launches | Rejected: LDS-only barriers, code-cache rotations, smaller selector histograms, dot-product ILP groups and separate tail launches did not improve complete operations. Skipping ineffective tail checks also remained exact but slowed projection/convolution at 2040/2048/2060/4096 tokens. |
+| Skip unused SSM activation copies and routed-row clears | Retained: 96 fewer launches per pp2048. d0 AR 1675.26 → 1681.77 tok/s; MTP 1622.87 → 1629.90 tok/s against the preceding private increment. d32K stays within 0.6%, TG within 0.2%; no material depth gain. Full logits, sampled replay and all six greedy completions remain exact. [Evidence](artifacts/prefill-long-context.json). |
+| Fixed/split HC residual blocks | Rejected: the faster fixed-width variant changed F16 normalization bytes; exact variants and smaller blocks did not improve speed. |
+| Wider SSM projection launch groups | Rejected: groups 1/4/8/16/32 preserve output bytes but are 1.3–4.0% slower than the existing group of two at 2048/4096 rows. |
+| Canonical pp2048 HTTP control | Complete branch versus matched main: MTP d0 1581.01 → 1623.99 tok/s; d32K 1490.59 → 1524.10 tok/s. All 12 mixed/repetitive completions and draft counts match; TG stays within 0.5%. One sample per workload/depth through 32K. [Evidence](artifacts/prefill-long-context.json). |
+| Packed selector loads and grouped MTP Q8 projections | Retained: exact arithmetic, fewer narrow loads and projection launches, no added allocation. A separate default-sampling code continuation (2217 new tokens at 32K) gains 1.9% PP; d0/128K and TG are essentially unchanged. [Evidence](artifacts/prefill-long-context.json). |
+| MTP catch-up at the sparse boundary | Fixed: final-tile execution now uses the full predictor's dense/sparse split and query grouping. Exact stage/candidate/carry checks pass; target arithmetic and sampling policy are unchanged. |
+| First decode after cold 32K prefill | Timing remains inconclusive: initial 4.3 ms flags did not recur in the reverse-order pair (50.92 → 51.29 ms); unchanged baseline itself varied by 4.7 ms. GPU work and graph setup match. Retained separately from tg128 results. [Evidence](artifacts/prefill-long-context.json). |
+| Q8 vocabulary-head prefetch | Rejected: exact isolated kernels improved, but sampled tg128 stayed at 38.54 → 38.50 tok/s. No prefetch specialization retained. [Evidence](artifacts/prefill-long-context.json). |
+| Direct mapped n-gram input | Retained: GPU reads the existing coherent pinned buffer, removing one upload per target forward and 40 MiB of shared scratch at 4096-row capacity. AR/MTP, images, cancellation and C2/4/6/8 sampled replay stay exact. No material d0 TG gain. [Evidence](artifacts/prefill-long-context.json). |
+| Pinned weights and aligned Q8 head layouts | Rejected: coherent/noncoherent pinned weights and registered huge-page storage did not improve decode. A lossless planar head improved isolated kernels but only 0.8% in full MTP decoding while retaining another 644 MiB; no second head or private weight format retained. [Evidence](artifacts/prefill-long-context.json). |
+| GDN row reuse, padded SSM rows and full selector scratch utilization | Retained: pp2048 d0 1648.38 tok/s (+2.1%), MTP 1592.39 (+2.4%) in matched single-run HTTP controls. Cold 131K AR +1.3%; late AR prefill after 196K +2.0%. Cold 194K MTP +0.9%; its short continuation was 0.9% slower, so no deep-MTP continuation gain is established. No added model/session allocation; exact logits and state checks. [Evidence](artifacts/prefill-long-context.json). |
+| Cooperative FP32 selector scoring and other GDN layouts | Rejected: sharing score lanes or keys preserved bits but was slower. Two GDN state rows/four waves beat the other tested row/wave layouts. [Evidence](artifacts/prefill-long-context.json). |
 | Final-layer AR prefill pruning | Retained: pp2048 reaches 1501.84 tok/s at 128K (+1.8%) and 1713.11 at d0 (+1.2%) in matched single-run controls. Only unused output rows are skipped; target logits and cache replay remain exact. MTP needs the full hidden sequence. [Evidence](artifacts/prefill-final-rows.json). |
 | Further 128K selector/attention tuning | Rejected: FP32 key staging, lane/register scheduling, histogram aggregation and attention LDS/row tiling retained exact outputs but did not improve complete operations. [Profile](artifacts/prefill-final-rows.json). |
 | Bounded selector score scratch | Retained: exact deep-context logits; pp2048 gains about 1.4% AR / 1.2% MTP at 128K in single-run controls, with no measured shallow-context regression. Shared scratch falls from 128 to 20 MiB. [Evidence](artifacts/prefill-deep-context.json). |