[justinchuby]

docs(#750): mark the 2b-impl scoping complete and annotate what each increment corrected

All four increments landed (#913, #920, #924, #930), but the document still
read as an open plan, and 2c is being scoped against it right now.

The corrections are the more useful part of it now, because three of the
four returned one, so they are recorded both as a table up front and inline
at the exact line that was wrong -- a reader trips over the claim, not over
the summary.

• §3's "the pleasant surprise: KV growth stride math is already
  batch-general" holds only for head-major. A fixed full-context stride
  commit is batch-general in both layouts because the batch axis is
  outermost; a growing bucket is not, because seq-major's per-sequence
  stride IS the mutable capacity, so growth relocates every sequence b>0.
  feat(native-decode): make KV growth and commit geometry batch-aware, refuse seq-major relocation (2b-impl-2, #750) #920 refuses that case with a named bail! rather than computing a
  capacity-dependent stride, which would have passed every batch-1 test and
  corrupted the first batch-N run. The section conflated the two paths.
• §4 item 6 places live_prefix_ranges in onnx-genai-kv; it is in this crate
  at native_decode/kv_commit.rs:122, so no cross-crate coordination was
  needed. Line number checked against current main rather than copied from
  the PR, since it moved when the batch parameter was added.

Records the outcome the line was started for, reproduced by me rather than
quoted from the PRs: capture survives batch-N while weights stream --
enabled=true captures=1 invalidations=0 fallbacks=0 with
htod_bytes_per_token 1,234,271,232 at N=2 and 547,528,704 at N=4,
byte_hit_rate 68.64% -> 72.17%, solo with managed streaming forced. That
closes #891's flagged risk positively.

And records the near-miss, because it is the part most worth carrying into
2c: #930's first version reported capture and streaming as mutually
exclusive, from a real decline naming a real predicate, because its harness
called an entry point that passed cuda_offload_policy: None and stable_va
fell to a conservative default. The harness asserted pointer-instability;
the runtime was never asked.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com
Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.81%. Comparing base (c3a9831) to head (217a5a9).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #941      +/-   ##
==========================================
+ Coverage   77.59%   77.81%   +0.21%     
==========================================
  Files         342      354      +12     
  Lines      142769   148153    +5384     
  Branches   142769   148153    +5384     
==========================================
+ Hits       110787   115287    +4500     
- Misses      27674    28372     +698     
- Partials     4308     4494     +186     

Flag	Coverage Δ
cli-ort-linux	83.42% <ø> (?)
cli-ort-windows	82.92% <ø> (?)
mlas	80.96% <ø> (+0.04%)	⬆️
offline	77.55% <ø> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 17 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	595.11 µs	1.34 ms	+125.1%
🔴	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	67.58 µs	149.38 µs	+121.0%
🔴	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	59.41 µs	120.41 µs	+102.7%
🔴	sampling_latency/min_p_per_token	215.00 µs	298.76 µs	+39.0%
⚠️	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	3.74 ms	4.78 ms	+28.0%
⚠️	qwen3_sampling_processors/top_k_partial_selection	144.28 µs	178.27 µs	+23.6%
⚠️	tokenization/decode_tokens_per_second	6.34 ms	7.78 ms	+22.7%
⚠️	matmul/large_generic_f16_threads=8/32x1024x1024	88.81 µs	108.31 µs	+22.0%
⚠️	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	435.21 µs	521.42 µs	+19.8%
⚠️	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	5.82 ms	6.93 ms	+19.2%
⚠️	matmul/large_generic_bf16_threads=8/32x1024x1024	1.53 ms	1.79 ms	+17.3%
⚠️	sampling_latency/top_p_per_token	392.13 µs	459.92 µs	+17.3%
⚠️	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	80.05 µs	93.87 µs	+17.3%
⚠️	qwen3_sampling_processors/top_k_full_sort_baseline	2.19 ms	2.54 ms	+16.0%
✅	matmul/small_generic_bf16_threads=8/1x256x256	34.80 µs	39.97 µs	+14.9%
✅	tokenization/encode_tokens_per_second	394.90 µs	453.04 µs	+14.7%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	544.09 µs	617.53 µs	+13.5%
✅	kv_cache/alloc_dealloc_pages	39.48 µs	44.81 µs	+13.5%
✅	gather/large_f32_threads=1-internal/131072	38.30 µs	43.31 µs	+13.1%
✅	matmul/medium_generic_f16_threads=8/32x512x512	35.23 µs	39.67 µs	+12.6%
✅	matmul/large_generic_f32_threads=8/32x1024x1024	4.47 ms	5.00 ms	+11.9%
✅	gather/small_f16_threads=1-internal/4096	561.3 ns	628.0 ns	+11.9%
✅	sampling_latency/top_k_per_token	53.56 µs	59.37 µs	+10.9%
✅	matmul/large_generic_bf16_threads=1/32x1024x1024	1.99 ms	2.20 ms	+10.5%
✅	matmul/large_generic_f16_threads=1/32x1024x1024	79.50 µs	87.59 µs	+10.2%
✅	matmul/medium_generic_f16_threads=1/32x512x512	32.76 µs	36.06 µs	+10.1%
✅	matmul/small_generic_f16_threads=1/1x256x256	32.55 µs	35.63 µs	+9.5%
✅	gather/small_bf16_threads=1-internal/4096	698.9 ns	765.0 ns	+9.4%
✅	matmul/medium_generic_f32_threads=1/32x512x512	2.42 ms	2.62 ms	+8.5%
✅	grammar_masking/llguidance_compute_mask/32	79.83 µs	86.15 µs	+7.9%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	9.85 ms	10.57 ms	+7.3%
✅	matmul/small_generic_f16_threads=8/1x256x256	34.23 µs	36.67 µs	+7.1%
✅	qwen3_sampling_processors/top_k_top_p_fast	685.89 µs	730.53 µs	+6.5%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	553.96 µs	588.43 µs	+6.2%
✅	logit_processing/seven_processor_chain_per_step	329.27 µs	348.30 µs	+5.8%
✅	sampling_latency/greedy_per_token	3.31 µs	3.48 µs	+4.9%
✅	matmul/small_generic_bf16_threads=1/1x256x256	35.95 µs	37.03 µs	+3.0%
✅	matmul/medium_generic_f32_threads=8/32x512x512	1.20 ms	1.22 ms	+2.0%
✅	gather/medium_bf16_threads=1-internal/32768	2.75 µs	2.73 µs	-1.0%
✅	gather/medium_f16_threads=1-internal/32768	3.12 µs	2.99 µs	-4.2%
✅	gather/large_f16_threads=1-internal/131072	18.44 µs	17.59 µs	-4.6%
✅	matmul/small_generic_f32_threads=8/1x256x256	49.07 µs	46.75 µs	-4.7%
✅	gather/medium_f32_threads=1-internal/32768	7.12 µs	6.67 µs	-6.3%
✅	add/medium_bf16_threads=1-internal/262144	2.94 ms	2.69 ms	-8.6%
✅	matmul/small_generic_f32_threads=1/1x256x256	48.41 µs	44.16 µs	-8.8%
✅	reduce_mean/large_f32_threads=1-internal/262144	1.11 ms	1.01 ms	-9.0%
✅	reduce_mean/small_f32_threads=1-internal/4096	17.64 µs	15.49 µs	-12.2%
✅	add/large_f16_threads=1-internal/4194304	47.91 ms	42.00 ms	-12.3%
✅	add/medium_f32_threads=1-internal/262144	2.97 ms	2.59 ms	-12.6%
✅	add/large_f32_threads=1-internal/4194304	47.69 ms	41.47 ms	-13.0%
✅	add/small_bf16_threads=1-internal/1024	14.98 µs	12.99 µs	-13.2%
🟢	gather/small_f32_threads=1-internal/4096	847.0 ns	719.1 ns	-15.1%
🟢	add/medium_f16_threads=1-internal/262144	3.16 ms	2.66 ms	-15.6%
🟢	reduce_mean/medium_f32_threads=1-internal/65536	304.62 µs	254.95 µs	-16.3%
🟢	add/small_f16_threads=1-internal/1024	15.78 µs	13.13 µs	-16.8%
🟢	gather/large_bf16_threads=1-internal/131072	17.21 µs	13.93 µs	-19.0%
🟢	matmul/medium_generic_bf16_threads=8/32x512x512	591.32 µs	477.07 µs	-19.3%
🟢	add/large_bf16_threads=1-internal/4194304	53.86 ms	42.79 ms	-20.6%
🟢	add/small_f32_threads=1-internal/1024	293.4 ns	201.5 ns	-31.3%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 8.04 4.95 5.31 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)