[francescobozzo]

Summary

Serving prefill on Flash-Next has been reading PLE n-gram rows synchronously
since #573. #470 overlapped each chunk's gather with the previous chunk by
prefetching the next chunk's rows. #573 then made serving steps end on planned
cache checkpoints every 2048 tokens (src/cache/retention.cpp grid). The runner
only saw the prompt up to that frontier, so its prefetch hint was empty and
every chunk waited on a blocking read: 124–178 ms per 2048-token chunk on a
cold prompt (~11% of the chunk), with ~17 ms of overlap.

This PR:

1. Logs per-request n-gram row counters (prompted by the gfx1150 (Strix Point) field report: build support + six performance observations #634 field report).
   The completion line gains ngram_rows, ngram_dedup_rows,
   ngram_cache_hits and ngram_disk_reads, only for models with a PLE table.
   Rows are attributed to the session that gathered them: gathers carry their
   owner through the read pool, and the request reports what its state gathered
   since it was leased. Concurrent requests do not see each other's rows.
   Every row is exactly one of a dedup copy, a row-cache hit or a disk read:
   rows == dedup + hits + disk is checked in tests and held on every live
   request. No new log lines; four fields on the existing event=completed line.
2. Prefetches across cache frontiers. The pool passes the runner the prompt
   tokens between this step's frontier and the next planned one
   (TextRunnerState::prefill_lookahead), and Flash-Next hints those when a
   step ends on its frontier.
3. Claims prefetched rows by common prefix. A batch with the same history
   now uses prefetched rows when either token list is a prefix of the other,
   reading only the remainder. This covers a scheduler step shrunk to 512
   tokens while a peer captures a checkpoint, which previously discarded every
   such prefetch. Rows are a pure function of tokens and history, so outputs
   are unchanged.
4. Fixes qwen38_flash_next_session_test on main, broken by perf(qwen38_flash_next): price greedy MTP drafts at the decoding occupancy #664.
   CheckBatchedSessions, CheckLookupBatch and the draft-vocabulary
   lifecycle check compared batched cohorts with a serial reference priced at
   occupancy 1, while perf(qwen38_flash_next): price greedy MTP drafts at the decoding occupancy #664 intentionally prices unplanned greedy rows at the
   cohort width. Session::DecodeStep takes an optional occupancy (default 1,
   so production callers are unchanged), and the references pass the cohort
   width. Before perf(qwen38_flash_next): price greedy MTP drafts at the decoding occupancy #664 (8e8a23e) the test passes; at a7ca5f0 it fails with
   batched decode tokens, RNG or acceptance differ; with this PR it passes.

How the counters found it: two concurrent prefills reported ~2k tokens more
rows than their tokens; a trace showed the shrunk-step discards and, more
importantly, next=0 on every solo serving chunk.

Validation

All on gfx1151, pinned Flash-Next Q4_K_XL + Q8 MTP sidecar.

Exactness

• qwen38_flash_next_session_test (full): PASS. New CheckPrefetchHints
  compares exact, shorter, longer and diverging hints with a bulk prefill:
  logits bit-exact, and the row counters confirm each claim (no double reads).
• component_text_cache_test: new PrefillLookaheadReachesTheNextFrontier
  checks the lookahead starts at each frontier, ends at the next planned one and
  is cleared after the call.
• qwen38_flash_next.ngram: new owner/dedup/cache/disk attribution checks.
• text_model_runner_test, format check: PASS.
• Served greedy outputs, prompt sizes and cache reuse identical to main in
  every request of the timing matrix below.

Timing — release nix build of a7ca5f0 (base) and this branch, default
server flags (--model, --speculative mtp, --mtp-model), four arms
alternating base → PR → PR → base with 60 s cool-down, one fresh server per
arm. Content blocks are exactly 512/1024/2048/8192/16384/32768 tokens (sized
with count_tokens); depth cases add a new turn to a cached prefix. prompt_ms
from the response timings, per request; prefill tok/s = (prompt − cached) tokens / prompt_ms. Paired arm1 vs arm2 and arm4 vs arm3.
Verdict rule: slower/faster only if both pairs move > 5 % and > 3 ms.

Case	Prompt	Cached	Prefilled	Pair 1 base → PR	Pair 2 base → PR	Verdict
cold 512 #0	536	0	536	911 → 912 tok/s (588 → 588 ms, -0.1%)	912 → 904 tok/s (588 → 593 ms, +0.9%)	–
cold 512 #1	535	0	535	914 → 911 tok/s (585 → 587 ms, +0.3%)	917 → 914 tok/s (583 → 586 ms, +0.4%)	–
cold 1024 #0	1049	0	1049	1367 → 1378 tok/s (768 → 761 ms, -0.8%)	1376 → 1379 tok/s (762 → 761 ms, -0.2%)	–
cold 1024 #1	1049	0	1049	1391 → 1393 tok/s (754 → 753 ms, -0.1%)	1390 → 1387 tok/s (754 → 756 ms, +0.2%)	–
cold 2048 #0	2073	0	2073	1611 → 1614 tok/s (1287 → 1284 ms, -0.2%)	1612 → 1600 tok/s (1286 → 1296 ms, +0.8%)	–
cold 2048 #1	2073	0	2073	1608 → 1607 tok/s (1289 → 1290 ms, +0.1%)	1612 → 1602 tok/s (1286 → 1294 ms, +0.7%)	–
cold 8192 #0	8217	0	8217	1582 → 1650 tok/s (5194 → 4979 ms, -4.1%)	1590 → 1657 tok/s (5168 → 4958 ms, -4.1%)	–
cold 8192 #1	8217	0	8217	1573 → 1649 tok/s (5225 → 4983 ms, -4.6%)	1572 → 1642 tok/s (5228 → 5004 ms, -4.3%)	–
cold 16384 #0	16410	0	16410	1675 → 1753 tok/s (9799 → 9363 ms, -4.5%)	1679 → 1753 tok/s (9776 → 9360 ms, -4.2%)	–
cold 16384 #1	16410	0	16410	1670 → 1756 tok/s (9827 → 9346 ms, -4.9%)	1673 → 1750 tok/s (9809 → 9378 ms, -4.4%)	–
prefix 8192 (cold)	8215	0	8215	1574 → 1654 tok/s (5218 → 4967 ms, -4.8%)	1574 → 1653 tok/s (5218 → 4970 ms, -4.7%)	–
d8K + 512 #0	8762	8223	539	882 → 879 tok/s (611 → 613 ms, +0.4%)	878 → 880 tok/s (614 → 613 ms, -0.1%)	–
d8K + 512 #1	8762	8223	539	888 → 887 tok/s (607 → 607 ms, +0.1%)	887 → 885 tok/s (607 → 609 ms, +0.3%)	–
d8K + 1024 #0	9275	8223	1052	1352 → 1352 tok/s (778 → 778 ms, +0.0%)	1352 → 1349 tok/s (778 → 780 ms, +0.2%)	–
d8K + 1024 #1	9275	8223	1052	1338 → 1340 tok/s (786 → 785 ms, -0.2%)	1342 → 1340 tok/s (784 → 785 ms, +0.1%)	–
d8K + 2048 #0	10299	8223	2076	1611 → 1612 tok/s (1289 → 1287 ms, -0.1%)	1610 → 1612 tok/s (1290 → 1287 ms, -0.2%)	–
d8K + 2048 #1	10299	8223	2076	1592 → 1594 tok/s (1304 → 1302 ms, -0.1%)	1590 → 1590 tok/s (1306 → 1306 ms, -0.0%)	–
prefix 32768 (cold)	32793	0	32793	1731 → 1739 tok/s (18948 → 18856 ms, -0.5%)	1729 → 1737 tok/s (18963 → 18880 ms, -0.4%)	–
d32K + 512 #0	33339	32801	538	863 → 866 tok/s (623 → 621 ms, -0.3%)	861 → 863 tok/s (625 → 623 ms, -0.3%)	–
d32K + 512 #1	33340	32801	539	886 → 888 tok/s (609 → 607 ms, -0.3%)	885 → 887 tok/s (609 → 608 ms, -0.2%)	–
d32K + 1024 #0	33853	32801	1052	1296 → 1289 tok/s (812 → 816 ms, +0.6%)	1295 → 1294 tok/s (812 → 813 ms, +0.1%)	–
d32K + 1024 #1	33853	32801	1052	1311 → 1309 tok/s (803 → 804 ms, +0.1%)	1311 → 1306 tok/s (803 → 805 ms, +0.3%)	–
d32K + 2048 #0	34877	32801	2076	1540 → 1540 tok/s (1348 → 1348 ms, -0.0%)	1535 → 1539 tok/s (1352 → 1349 ms, -0.2%)	–
d32K + 2048 #1	34876	32801	2075	1537 → 1536 tok/s (1350 → 1351 ms, +0.0%)	1533 → 1531 tok/s (1354 → 1355 ms, +0.1%)	–
decode 128 (tg)	22	0	–	42.1 → 42.0 tok/s (-0.4%)	42.2 → 42.1 tok/s (-0.2%)	–

• No regression: every 512/1024/2048 cell, cold and at 8K/32K depth, stays
  within ±1 % (cold 512 ≈ 905–917 tok/s, cold 2048 ≈ 1600–1614 tok/s, a 2K
  turn at 32K ≈ 1531–1540 tok/s on both), and decode stays at 42.0–42.2 tok/s.
• Cold 8K–16K prompts are 4.1–4.9 % faster in both pairs: 8K
  1572–1590 → 1642–1657 tok/s, 16K 1670–1679 → 1750–1756 tok/s. Consistent,
  but under the 5 % rule, so not claimed as a qualified gain. The first chunk
  of a cold prompt cannot be prefetched, which bounds the gain.
• A cold 32K prompt moves −0.4/−0.5 % (1729–1731 → 1737–1739 tok/s): the grid
  plans only 4 checkpoints, so main already prefetched most of its chunks.

Functional (tests/functional/run.py --record-baseline, candidate release
binary, --sessions 2 --context 32768): prefill-scheduling, cache (all
RAM and disk cases), long-context, cache-depth, cache-concurrency and
cache-shared-prefix pass in both Flash-Next modes: mtp (64 drafts, 27
accepted) and off with the sidecar path kept (0 drafts). Every completion line
in those runs carries the counters, e.g. a cold and a repeated request:
ngram_rows=416 ngram_dedup_rows=8 ngram_cache_hits=0 ngram_disk_reads=408, then
ngram_rows=416 ngram_dedup_rows=8 ngram_cache_hits=407 ngram_disk_reads=1.

Not in this PR

• The gfx1150 (Strix Point) field report: build support + six performance observations #634 patch's other parts (draft-history retention, checkpoint identity,
  gfx1150 build, GET /) are separate decisions.

https://claude.ai/code/session_011rm7aPexQHBinHKAMkLup4

[github-actions]

✅ Devlab functional checks · PR tier: 4 configuration groups passed

	Mode · configuration	Passed	Failing suites
✅	MTP · standard	42/42
✅	MTP · single-session	3/3
✅	MTP · cache-stalls	1/1
✅	MTP · cache-pressure	1/1

Inference phases · merge base → PR

flowchart LR
  subgraph g0["Startup"]
    direction TB
    g0n0["Process init<br/>50 ms → 34 ms<br/>-32.1%"]:::noisy
    g0n1["Weights and model<br/>16.00 s → 16.00 s<br/>-0.0%"]:::same
    g0n0 --> g0n1
    g0n2["Vision sidecar<br/>525 ms → 538 ms<br/>+2.6%"]:::same
    g0n1 --> g0n2
    g0n3["Structured output<br/>32 ms → 33 ms<br/>+4.4%"]:::same
    g0n2 --> g0n3
    g0n4["HTTP ready<br/>11 ms → 11 ms<br/>+0.3%"]:::same
    g0n3 --> g0n4
  end
  subgraph g1["Short prompt · 1,549 tok"]
    direction TB
    g1n0["Prefill<br/>943 ms → 947 ms<br/>+0.4%"]:::same
    g1n1["Decode<br/>2.61 s → 2.61 s<br/>+0.0%"]:::same
    g1n0 --> g1n1
    g1n2["Cache snapshot<br/>3.9 ms → 3.3 ms<br/>-16.0%"]:::same
    g1n1 --> g1n2
  end
  subgraph g2["Long prompt · 12,370 tok"]
    direction TB
    g2n0["Prefill<br/>7.57 s → 7.48 s<br/>-1.1%"]:::same
    g2n1["Decode<br/>2.20 s → 2.21 s<br/>+0.1%"]:::same
    g2n0 --> g2n1
    g2n2["Cache snapshot<br/>15 ms → 16 ms<br/>+0.8%"]:::same
    g2n1 --> g2n2
  end
  subgraph g3["Follow-up · 1,964 tok"]
    direction TB
    g3n0["Cache restore<br/>3.0 ms → 2.2 ms<br/>-24.7%"]:::same
    g3n1["Prefill<br/>407 ms → 409 ms<br/>+0.6%"]:::same
    g3n0 --> g3n1
    g3n2["Decode<br/>2.14 s → 2.14 s<br/>-0.0%"]:::same
    g3n1 --> g3n2
    g3n3["Cache snapshot<br/>3.3 ms → 3.3 ms<br/>-0.5%"]:::same
    g3n2 --> g3n3
  end
  g0 --> g1 --> g2 --> g3
  classDef slower fill:#cf222e,stroke:#82071e,color:#fff
  classDef faster fill:#1a7f37,stroke:#044f1e,color:#fff
  classDef noisy fill:#bf8700,stroke:#7d4e00,color:#fff
  classDef same fill:#6e7781,stroke:#424a53,color:#fff
  classDef missing fill:#d0d7de,stroke:#8c959f,color:#24292f

Loading

_{Medians of 2 alternating base/PR arms after GPU cool-down. 🟥 slower and 🟩 faster in every repeat (>5% and >3 ms); 🟨 mixed repeats, inconclusive; ⬜ unchanged. Observations, not qualified timing regressions. Measured in 2.7 min.}

Every repeat

Phase	Base	PR	Verdict
Startup · Process init	67 ms, 33 ms	34 ms, 34 ms	noisy
Startup · Weights and model	16.06 s, 15.95 s	16.00 s, 15.99 s	same
Startup · Vision sidecar	524 ms, 525 ms	530 ms, 546 ms	same
Startup · Structured output	33 ms, 31 ms	34 ms, 32 ms	same
Startup · HTTP ready	11 ms, 11 ms	11 ms, 11 ms	same
Short prompt · 1,549 tok · Prefill	937 ms, 949 ms	945 ms, 948 ms	same
Short prompt · 1,549 tok · Decode	2.61 s, 2.61 s	2.60 s, 2.61 s	same
Short prompt · 1,549 tok · Cache snapshot	4.5 ms, 3.2 ms	3.3 ms, 3.3 ms	same
Long prompt · 12,370 tok · Prefill	7.57 s, 7.57 s	7.47 s, 7.49 s	same
Long prompt · 12,370 tok · Decode	2.21 s, 2.20 s	2.20 s, 2.21 s	same
Long prompt · 12,370 tok · Cache snapshot	16 ms, 15 ms	16 ms, 15 ms	same
Follow-up · 1,964 tok · Cache restore	3.7 ms, 2.2 ms	2.3 ms, 2.2 ms	same
Follow-up · 1,964 tok · Prefill	405 ms, 409 ms	409 ms, 410 ms	same
Follow-up · 1,964 tok · Decode	2.14 s, 2.14 s	2.14 s, 2.14 s	same
Follow-up · 1,964 tok · Cache snapshot	3.3 ms, 3.3 ms	3.3 ms, 3.3 ms	same

_{Workflow run · PR 3ae0a63 vs merge base a7ca5f0 · model flash-next-q4 · 4 deferred, 1 for the release run, 352 out of scope
devlab/functional is advisory on PRs; only the automatic release merge waits for it. Candidate correctness only; other-model, numerical, agent replay and timing qualification remain separate.}