perf(qwen-flash): 4096-token prefill chunks with an overlapped n-gram gather

.agents/skills/benchmark-model/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: benchmark-model
-description: Measure every retained number in docs/models/<model>/BENCHMARKS.md for one Gufo model on Strix Halo gfx1151 over HTTP, run the same workload against the reference project's OpenAI-compatible server (llama.cpp, audio.cpp, ...), and report the gain per cell.
+description: Measure every retained number in a Gufo model's BENCHMARKS.md on Strix Halo gfx1151 over HTTP, run the same workload against the reference project's OpenAI-compatible server (llama.cpp, audio.cpp, ...), and report the gain per cell.
 metadata:
   origin: gufo
 ---
@@ -21,6 +21,12 @@ paths remain valid. Prefer one warmed sample per point, run only the requested
 sweeps, and repeat only to investigate a concrete discrepancy. Prefix setup at
 64K/128K dominates runtime; avoid rebuilding it for unrelated checks.
 
+Record the power profile and starting CPU/GPU temperatures. Paired runs need
+the same profile and comparable thermal/warmup conditions. On this host,
+check `/sys/firmware/acpi/platform_profile`: Balanced and Performance can
+differ materially with identical binaries. Monitor the profile throughout
+long sweeps and retain their warmup/cache history.
+
 Start long runs in the background with output redirected to a log. Inspect
 progress and process status at short intervals while continuing independent
 work; do not leave a run unreviewed. Before starting, establish

README.md

@@ -22,7 +22,7 @@ All model documentation lives under [docs/models](docs/models/README.md):
 | Model | Inference modes | Hugging Face weights | Benchmarks | Quality |
 | --- | --- | --- | --- | --- |
 | [Qwen3.8 27B](docs/models/qwen3.8-27b/README.md) | Q4/Q8, images, AR, DFlash2 | Unsloth [Q4_K_XL](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/4ca720788d1e01f1bff70c033e0d0028fd02e502/Qwen3.8-27B-UD-Q4_K_XL.gguf) / [Q8_K_XL](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/4ca720788d1e01f1bff70c033e0d0028fd02e502/Qwen3.8-27B-UD-Q8_K_XL.gguf) · [DFlash2 Q4_K_M](https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2-GGUF/blob/2d9571f8ce46e151f61c6499c99dee6079e1d610/Qwen3.8-27B-DFlash2-Q4_K_M.gguf) | Q4: **656.33 tok/s pp**; up to **70.56 tok/s tg** single user and **123.00 aggregated tok/s** on 8 concurrent requests with DFlash2 · [Benchmarks](docs/models/qwen3.8-27b/BENCHMARKS.md) | [Quality](docs/models/qwen3.8-27b/QUALITY.md) |
-| [Qwen3.8 Flash-Next](docs/models/qwen3.8-flash-next/README.md) | Q4, images, AR, MTP | Unsloth [Q4_K_XL](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/38bb39ee97821de2c9009abb7e93950eec396e66/UD-Q4_K_XL) · [MTP Q8_0](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/blob/38bb39ee97821de2c9009abb7e93950eec396e66/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf) | **1,639.03 tok/s pp**; up to **60.36 tok/s tg** single user and **157.22 aggregated tok/s** on 8 concurrent requests with MTP · [Benchmarks](docs/models/qwen3.8-flash-next/BENCHMARKS.md) | [Quality](docs/models/qwen3.8-flash-next/QUALITY.md) |
+| [Qwen3.8 Flash-Next](docs/models/qwen3.8-flash-next/README.md) | Q4, images, AR, MTP | Unsloth [Q4_K_XL](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/38bb39ee97821de2c9009abb7e93950eec396e66/UD-Q4_K_XL) · [MTP Q8_0](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/blob/38bb39ee97821de2c9009abb7e93950eec396e66/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf) | **1,644.68 tok/s pp**; up to **59.02 tok/s tg** single user and **157.22 aggregated tok/s** on 8 concurrent requests with MTP · [Benchmarks](docs/models/qwen3.8-flash-next/BENCHMARKS.md) | [Quality](docs/models/qwen3.8-flash-next/QUALITY.md) |
 | [DeepSeek V4 Flash](docs/models/deepseek-v4-flash/README.md) | AR, DSpark | [antirez Flash 0731 IQ2XXS](https://huggingface.co/antirez/deepseek-v4-gguf/blob/1cd7b564460821938add0475a60b942c409295e0/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf) · [DSpark](https://huggingface.co/antirez/deepseek-v4-gguf/blob/e7f04037032990db0346398d249baf9fb9df1ccc/DeepSeek-V4-Flash-DSpark-support-0731.gguf) | **484.62 tok/s pp**; up to **26.62 tok/s tg** single user and **54.74 aggregated tok/s** on 8 concurrent requests with DSpark · [Benchmarks](docs/models/deepseek-v4-flash/BENCHMARKS.md) | [Quality](docs/models/deepseek-v4-flash/QUALITY.md) |
 | [Qwen3-ASR 1.7B](docs/models/qwen3-asr/README.md) | Speech recognition | [BF16](https://huggingface.co/Qwen/Qwen3-ASR-1.7B/tree/7278e1e70fe206f11671096ffdd38061171dd6e5) | **15.27× realtime** · [Benchmarks](docs/models/qwen3-asr/BENCHMARKS.md) | [Quality](docs/models/qwen3-asr/QUALITY.md) |
 | [Qwen3-TTS 1.7B](docs/models/qwen3-tts/README.md) | Speech synthesis and voice cloning | BF16 [CustomVoice](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice) / [VoiceDesign](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign) / [Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base) | Up to **2.54× realtime**; **201 ms** to first audio (CustomVoice) · [Benchmarks](docs/models/qwen3-tts/BENCHMARKS.md) | [Quality](docs/models/qwen3-tts/QUALITY.md) |

docs/BENCHMARKS.md

@@ -134,10 +134,10 @@ end-to-end model measurements. Collect profiler traces separately and use
 ## Reporting and artifact retention
 
 Record model/engine revisions, artifact hashes, hardware/software fingerprint,
-input identity, context, output count, sampling, concurrency, cache state,
-oracle identity, and the exact timed scope. A numerical mismatch or device
-failure is not a passing result. Add repetitions when timing noise or a
-suspected regression requires them.
+power profile and CPU/GPU temperatures, input identity, context, output count,
+sampling, concurrency, cache state, oracle identity, and the exact timed scope.
+A numerical mismatch or device failure is not a passing result. Add repetitions
+when timing noise or a suspected regression requires them.
 
 Checked-in JSON retains fixtures, model provenance, controller calibration,
 and bounded result summaries. Serving summaries preserve aggregate metrics,

docs/models/qwen3.8-flash-next/BENCHMARKS.md

@@ -2,8 +2,8 @@
 
 AMD Strix Halo `gfx1151`, 128 GB unified memory. Unsloth `UD-Q4_K_XL` target and
 shared-Q8_0 MTP sidecar; Gufo uses adaptive MTP. HTTP, greedy, thinking off.
-Gufo single-user pp/tg: October 6, 2026 (`4c5a00d2`); concurrency, loading,
-memory and reference results: September 22–23.
+Gufo single-user pp/tg: October 7, 2026 (`def2ed1e`), Performance power profile.
+Concurrency, loading, memory and reference results: September 22–23.
 llama.cpp uses `b11069` for AR and `6fcaa16f` for MTP.
 
 Positive gain favors Gufo.
@@ -17,14 +17,14 @@ Context capacities differ between engines; see the measurement details.
 <!-- bench:single-ar -->
 | Flash-Next Q4 AR<br>Depth (tokens) | Gufo pp (tok/s) | llama.cpp pp (tok/s) | Gain | Gufo tg (tok/s) | llama.cpp tg (tok/s) | Gain |
 | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
-| 0 | 1639.03 | 489.59 | +234.8% | 25.96 | 22.20 | +16.9% |
-| 4,096 | 1564.36 | 455.24 | +243.6% | 25.96 | 21.21 | +22.4% |
-| 8,192 | 1557.47 | 428.85 | +263.2% | 25.95 | 20.40 | +27.2% |
-| 12,288 | 1553.60 | 400.21 | +288.2% | 25.93 | 19.64 | +32.0% |
-| 16,384 | 1553.97 | 375.89 | +313.4% | 25.89 | 18.95 | +36.6% |
-| 32,768 | 1540.25 | 301.31 | +411.2% | 25.75 | 16.54 | +55.7% |
-| 65,536 | 1459.69 | 221.94 | +557.7% | 25.50 | 11.62 | +119.4% |
-| 131,072 | 1444.79 | 144.78 | +897.9% | 24.90 | 7.98 | +212.0% |
+| 0 | 1644.68 | 489.59 | +235.9% | 25.99 | 22.20 | +17.1% |
+| 4,096 | 1518.57 | 455.24 | +233.6% | 25.73 | 21.21 | +21.3% |
+| 8,192 | 1567.32 | 428.85 | +265.5% | 25.99 | 20.40 | +27.4% |
+| 12,288 | 1560.97 | 400.21 | +290.0% | 25.96 | 19.64 | +32.2% |
+| 16,384 | 1556.02 | 375.89 | +314.0% | 25.93 | 18.95 | +36.8% |
+| 32,768 | 1546.81 | 301.31 | +413.4% | 25.80 | 16.54 | +56.0% |
+| 65,536 | 1425.49 | 221.94 | +542.3% | 25.43 | 11.62 | +118.8% |
+| 131,072 | 1446.10 | 144.78 | +898.8% | 24.97 | 7.98 | +212.9% |
 <!-- /bench -->
 
 ![Single user, autoregressive](artifacts/charts/single-ar.svg)
@@ -37,14 +37,14 @@ text, including Gufo predictor catch-up.
 <!-- bench:single-mtp -->
 | Flash-Next Q4 MTP<br>Depth (tokens) | Gufo pp (tok/s) | llama.cpp pp (tok/s) | Gain pp | Gufo tg mixed (tok/s) | llama.cpp tg mixed (tok/s) | Gain mixed | Gufo tg repetitive (tok/s) | llama.cpp tg repetitive (tok/s) | Gain repetitive |
 | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
-| 0 | 1584.63 | 468.76 | +238.0% | 31.33 | 31.73 | -1.3% | 60.36 | 48.12 | +25.4% |
-| 4,096 | 1528.90 | 423.79 | +260.8% | 32.83 | 34.48 | -4.8% | 57.61 | 45.71 | +26.0% |
-| 8,192 | 1528.87 | 394.98 | +287.1% | 32.41 | 32.20 | +0.7% | 49.22 | 44.31 | +11.1% |
-| 12,288 | 1511.31 | 365.91 | +313.0% | 34.72 | 32.32 | +7.4% | 42.17 | 43.53 | -3.1% |
-| 16,384 | 1503.69 | 341.98 | +339.7% | 35.21 | 32.07 | +9.8% | 50.56 | 43.66 | +15.8% |
-| 32,768 | 1501.78 | 277.61 | +441.0% | 34.21 | 25.41 | +34.6% | 44.99 | 39.28 | +14.5% |
-| 65,536 | 1415.89 | 208.04 | +580.6% | 33.50 | 19.65 | +70.5% | 44.45 | 27.75 | +60.2% |
-| 131,072 | 1403.96 | 135.42 | +936.7% | 33.10 | 14.63 | +126.2% | 44.34 | 20.96 | +111.5% |
+| 0 | 1560.11 | 468.76 | +232.8% | 30.16 | 31.73 | -4.9% | 59.02 | 48.12 | +22.7% |
+| 4,096 | 1529.17 | 423.79 | +260.8% | 32.86 | 34.48 | -4.7% | 57.58 | 45.71 | +26.0% |
+| 8,192 | 1525.11 | 394.98 | +286.1% | 32.47 | 32.20 | +0.8% | 49.26 | 44.31 | +11.2% |
+| 12,288 | 1517.04 | 365.91 | +314.6% | 34.78 | 32.32 | +7.6% | 42.19 | 43.53 | -3.1% |
+| 16,384 | 1502.95 | 341.98 | +339.5% | 35.27 | 32.07 | +10.0% | 50.60 | 43.66 | +15.9% |
+| 32,768 | 1503.08 | 277.61 | +441.4% | 34.27 | 25.41 | +34.9% | 44.98 | 39.28 | +14.5% |
+| 65,536 | 1390.03 | 208.04 | +568.2% | 33.51 | 19.65 | +70.5% | 43.12 | 27.75 | +55.4% |
+| 131,072 | 1408.06 | 135.42 | +939.8% | 33.18 | 14.63 | +126.8% | 44.43 | 20.96 | +112.0% |
 <!-- /bench -->
 
 ![Single user, MTP](artifacts/charts/single-mtp.svg)

docs/models/qwen3.8-flash-next/EXPERIMENTS.md

@@ -59,6 +59,9 @@
 | Live final frontier and async prompt snapshots | Retained; immutable branch snapshot, worker capture, bounded persistence outside the lookup lock. |
 | Thread-local decode graph capture | Retained; independent snapshot workers no longer invalidate a peer's capture. Snapshot bytes and captured/replayed logits are exact; no request serialization added. |
 | Chunk-equivalent projections/attention | Retained with exact continued-image/cache/full-logit gates; one-token tails keep prefill arithmetic. |
+| Dense split at the indexer budget | Retained; pre-budget queries always take the dense tiles, so prefill logits no longer depend on the prefill batch. Before: batches 4096 and 8192 differed from 2048 on 4/4 repository documents; after: byte-identical decode dumps at 2048/4096/8192. |
+| 4096-token prefill chunks | Retained together with 128 readers and the next-chunk prefetch; exact given the dense split (byte-identical dumps on 4/4 repository documents, identical served greedy outputs). Served repository prompts on two gfx1151 machines, ABBA, 8+ samples per prompt: 24K–102K-token prompts +6–11% (median per prompt), 7K–8K prompts −0.1% to +0.6% in dedicated rechecks; pp4096 +5.3%/+5.4% at d0 and +5.8%/+5.9% at d32K; TG unchanged. Peak memory at 262K +1.3 GiB. Prefill beside an active decode stays at `--prefill-chunk` (512); two-session gaps unchanged. |
+| Next-chunk n-gram prefetch | Retained; a prefill chunk reads the next chunk's rows once its own have arrived, and a batch uses them only with identical tokens and history, so output is unchanged. With 4096-token chunks the gather then overlaps a whole chunk's forward instead of only layer 0. Adds one row buffer (≤ 40 MiB pageable, allocated on first use). |
 | More Q8 vocabulary rows/block | Rejected: no C1 improvement. |
 | Private shared-expert F16 rows | Retained; the shared expert's SwiGLU rows for its F16 down projection get their own buffer instead of overwriting the routed experts' narrowed token rows, which removes one full-batch narrowing pass per layer. Every GEMM reads identical bytes; greedy output, chunk-boundary logits and the C2–C8 batch checks are exact. Interleaved Nix A/B at d0: 1559.6 → 1568.1 tok/s (+0.6%); the profiled kernel time fell 1362.8 → 1333.1 ms. |
 | 256-row routed down-projection blocks | Rejected: exact output, Q5_1 unchanged and Q8_0 slower. The routed kernels already stream expert weights at roughly two thirds of DRAM bandwidth, so per-block prologue/epilogue and activation restaging were not the bound. |
@@ -70,7 +73,7 @@
 | Whole-tile carry across windows | Rejected: same output, but 16.5 KiB of LDS cost a resident block per CU and slowed d2K eight-row verification 8.5%. |
 | GPU greedy penalties and linear CPU anchor selection | Retained; exact FP64 penalties, unchanged proposals and snapshots. Short heat-pump tg400: 32.12 → 33.80 tok/s; C2/4/6/8 improve 7.6/8.5/16.3/14.2%. Unpenalized control unchanged. [Evidence](artifacts/penalty-verification.json). |
 | Prefill projection, attention and indexer kernels | Retained; bit-identical logit dumps and greedy hashes, plus a bitwise GEMM sweep against hipBLASLt for every routed n ≤ 4096. pp4096 +1.4% at d0 and +2.5–2.8% at 32K/115K (interleaved ABBA, `-r 6`, two machines); repository prompts 6.5K–102K +3.2% median; tg unchanged. The indexer projections use the own kernel only when hipBLASLt selects algorithm 4438 and otherwise stay on the library. |
-| 128 n-gram readers | Rejected: first gather 179 → 140 ms, but no end-to-end change on repository prompts on two different NVMe drives (ABBA medians within 1%). |
+| 128 n-gram readers | Retained with 4096-token chunks, which double each gather; readers block in `pread`, so the pool sets the queue depth. Readers are woken per read, so small gathers do not wake the pool; 96 more threads. |
 
 Separate d32K pp2048 profiling attributes 29.1% of kernel time to MoE, 34.9%
 to dense projections and 12.6% to attention/indexing. Final-tile catch-up

docs/models/qwen3.8-flash-next/QUALITY.md

@@ -69,8 +69,9 @@ weights. [Vision reproduction](../qwen3.8-27b/QUALITY.md#vision).
 
 ## Benchmark method
 
-Gufo single-user pp/tg refreshed October 6, 2026 (`4c5a00d2`); reference,
-concurrency, loading and memory measurements retain September 22–23 provenance.
+Gufo single-user pp/tg refreshed October 7, 2026 (`def2ed1e`), Performance power
+profile. Reference, concurrency, loading and memory retain September 22–23
+provenance.
 One warmed sample per point, greedy, thinking off, penalties disabled.
 Single-user uses pp2048/tg128; MTP pp is the maximum across mixed/repetitive
 workloads. Gufo capacity is 133760; reference capacity is 35456 through 32K,