[justinchuby]

Summary

Root-causes PR #932 finding 2 (M=K verify argmax ≠ M=1 greedy argmax). Verdict: near-tie floating-point noise in the eager M=K verify's in-block attention/KV numerics — not a bug, not a systematic offset. Adds a reusable bench-only diagnostic bin (verify_logits_probe); no engine change.

GPU: CUDA_VISIBLE_DEVICES=6 (verified idle). Model: qwen2.5-0.5b-int4.

Method (controlled)

Replay greedy M=1 decode on the native CUDA captured hot path, recording per-position logits; then re-run the same greedy continuation as the M=K draft through decode_verify, so each verify row has mathematically identical causal inputs to the corresponding M=1 forward. Any logit delta is therefore pure kernel numerics.

Evidence

First divergence (prose), output index 27, K-block P=24 row 3:

M=1 argmax=1817 (22.01366)   M=K argmax=6894 (21.85298)
M=1 top1-top2 gap=0.1697     M=K gap=0.0433
mean Δ=-0.0376  mean|Δ|=0.0827  max|Δ|=0.5066
M=1 top5: 1817 6894 279 264 1376
M=K top5: 6894 1817 279 264 1376   # same set, only top-2 swap

Discriminating K-sweep (prose, 64 tokens):

K	flips	first divergence
1	0	none
2	4	P=26, row 1
4	8	P=24, row 3
8	16	P=20, row 7

• K=1 → 0 flips: width-1 eager verify is argmax-identical to M=1. Eager-vs-captured dispatch is NOT the cause.
• Flips occur only at in-block rows ≥1 (first flip = last row of block), scale ~linearly with K, and only at near-ties (greedy top1-top2 gap ≤ ~0.17, min 0.014). Confident tokens never flip.

Classification: (1) near-tie fp noise

• Not systematic offset: mean Δ ≈ -0.01..-0.04 (negligible); K=1 has 0 flips (a real bias would flip row 0). Rejected.
• Not wild/wrong: top-5 sets identical, deltas 0.01–0.5, only near-tie top-2 swap; no indexing/masking bug (would corrupt row 0). Rejected.
• Near-tie fp noise: the M=K forward computes in-block draft K/V inside the batched GEMM + batched attention, vs the M=1 path reading those positions from persistent KV → fp non-associativity (~0.01–0.5 logit) flips argmax at near-ties. Localized to in-block attention/KV numerics in run_cuda_eager_rows.

Implication

Byte-lossless prompt-lookup is not achievable with a distinct M=K kernel: every committed position > base is decided by an in-block M=K row that isn't bit-exact. Routing just the last position through M=1 does not suffice (any row ≥1 can diverge; the bonus token also comes from an M=K row). Options:

1. Accept as approximate — greedy-quality, drop the "lossless" claim.
2. Near-tie guard — when an M=K row's top1-top2 gap < τ, re-decode that one position via the M=1 captured kernel. Near-ties are rare (~4–9% of rows), so cost is negligible and this restores exact greedy identity without serializing verify.

Not a one-line fix → no fix in this PR (design decision for the speculative owner). This PR lands only the diagnostic tool + analysis.

EAGLE-3 / MTP

They reuse the same eager M=K decode_verify, so they inherit the identical near-tie in-block divergence. Not a correctness bug for them (approximate by nature), but any "exact/lossless" guarantee would not hold, and the same near-tie guard would be needed for byte-exactness.

Notes / validation

• verify_logits_probe is bench-only (new [[bin]], required-features=["bench-native"]).
• Bare NativeDecodeSession::load_with_resolved_io needs ONNX_GENAI_CUDA_KV_MAX_LEN for model dirs with genai_config.json but no inference_metadata.yaml (else "full CUDA mask reservation size overflow").
• cargo fmt --all -- --check clean; cargo clippy -p onnx-genai-bench --features bench-native,cuda --bin verify_logits_probe clean.

Full analysis: .squad/decisions/inbox/deckard-verify-rootcause.md.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.81%. Comparing base (bc6bd61) to head (38693ca).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #935      +/-   ##
==========================================
- Coverage   78.51%   77.81%   -0.71%     
==========================================
  Files         352      354       +2     
  Lines      145983   148153    +2170     
  Branches   145983   148153    +2170     
==========================================
+ Hits       114622   115287     +665     
- Misses      26873    28372    +1499     
- Partials     4488     4494       +6     

Flag	Coverage Δ
cli-ort-linux	83.42% <ø> (ø)
cli-ort-windows	83.01% <ø> (ø)
mlas	80.96% <ø> (?)
offline	77.55% <ø> (-0.78%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 8 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	matmul/small_generic_f32_threads=1/1x256x256	36.70 µs	48.80 µs	+33.0%
⚠️	gather/small_f32_threads=1-internal/4096	909.1 ns	1.17 µs	+28.4%
⚠️	grammar_masking/llguidance_compute_mask/32	87.74 µs	108.07 µs	+23.2%
⚠️	matmul/small_generic_f16_threads=8/1x256x256	34.18 µs	41.47 µs	+21.4%
⚠️	matmul/large_generic_bf16_threads=8/32x1024x1024	1.78 ms	2.15 ms	+21.0%
⚠️	matmul/large_generic_bf16_threads=1/32x1024x1024	2.45 ms	2.96 ms	+20.6%
⚠️	matmul/large_generic_f16_threads=1/32x1024x1024	84.34 µs	101.16 µs	+19.9%
⚠️	add/medium_bf16_threads=1-internal/262144	2.70 ms	3.15 ms	+16.8%
⚠️	matmul/small_generic_f16_threads=1/1x256x256	31.36 µs	36.58 µs	+16.6%
✅	matmul/small_generic_f32_threads=8/1x256x256	44.11 µs	50.14 µs	+13.7%
✅	add/small_f16_threads=1-internal/1024	13.48 µs	14.57 µs	+8.1%
✅	qwen3_sampling_processors/top_k_top_p_fast	716.08 µs	766.36 µs	+7.0%
✅	add/medium_f16_threads=1-internal/262144	2.58 ms	2.76 ms	+6.9%
✅	add/small_bf16_threads=1-internal/1024	13.69 µs	14.48 µs	+5.8%
✅	gather/small_f16_threads=1-internal/4096	533.1 ns	562.2 ns	+5.5%
✅	matmul/medium_generic_bf16_threads=8/32x512x512	647.16 µs	674.28 µs	+4.2%
✅	add/medium_f32_threads=1-internal/262144	2.65 ms	2.72 ms	+2.9%
✅	matmul/large_generic_f16_threads=8/32x1024x1024	122.62 µs	125.11 µs	+2.0%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	607.62 µs	614.39 µs	+1.1%
✅	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	767.77 µs	774.30 µs	+0.9%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	9.93 ms	9.99 ms	+0.6%
✅	tokenization/decode_tokens_per_second	7.62 ms	7.59 ms	-0.5%
✅	logit_processing/seven_processor_chain_per_step	362.43 µs	355.44 µs	-1.9%
✅	matmul/medium_generic_f16_threads=1/32x512x512	38.27 µs	37.43 µs	-2.2%
✅	add/small_f32_threads=1-internal/1024	223.5 ns	216.5 ns	-3.2%
✅	kv_cache/alloc_dealloc_pages	40.26 µs	38.75 µs	-3.8%
✅	matmul/medium_generic_f16_threads=8/32x512x512	40.55 µs	38.71 µs	-4.6%
✅	gather/large_f32_threads=1-internal/131072	34.48 µs	32.90 µs	-4.6%
✅	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	81.41 µs	77.31 µs	-5.0%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	595.73 µs	565.21 µs	-5.1%
✅	sampling_latency/greedy_per_token	3.77 µs	3.54 µs	-6.2%
✅	reduce_mean/medium_f32_threads=1-internal/65536	312.09 µs	292.31 µs	-6.3%
✅	tokenization/encode_tokens_per_second	435.81 µs	403.76 µs	-7.4%
✅	sampling_latency/top_k_per_token	62.53 µs	57.90 µs	-7.4%
✅	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	6.57 ms	6.08 ms	-7.4%
✅	gather/medium_f16_threads=1-internal/32768	3.27 µs	3.03 µs	-7.4%
✅	gather/large_bf16_threads=1-internal/131072	17.92 µs	15.40 µs	-14.1%
🟢	gather/large_f16_threads=1-internal/131072	17.60 µs	14.92 µs	-15.2%
🟢	matmul/small_generic_bf16_threads=1/1x256x256	43.75 µs	37.05 µs	-15.3%
🟢	sampling_latency/min_p_per_token	279.63 µs	234.01 µs	-16.3%
🟢	matmul/large_generic_f32_threads=8/32x1024x1024	4.85 ms	4.03 ms	-17.0%
🟢	add/large_bf16_threads=1-internal/4194304	57.26 ms	47.24 ms	-17.5%
🟢	reduce_mean/large_f32_threads=1-internal/262144	1.47 ms	1.21 ms	-17.8%
🟢	gather/small_bf16_threads=1-internal/4096	653.8 ns	531.0 ns	-18.8%
🟢	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	603.41 µs	488.74 µs	-19.0%
🟢	add/large_f32_threads=1-internal/4194304	52.53 ms	42.42 ms	-19.2%
🟢	qwen3_sampling_processors/top_k_partial_selection	178.13 µs	141.05 µs	-20.8%
🟢	matmul/medium_generic_f32_threads=1/32x512x512	3.30 ms	2.58 ms	-21.8%
🟢	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	4.50 ms	3.50 ms	-22.3%
🟢	gather/medium_f32_threads=1-internal/32768	6.31 µs	4.84 µs	-23.2%
🟢	gather/medium_bf16_threads=1-internal/32768	3.42 µs	2.61 µs	-23.7%
🟢	matmul/small_generic_bf16_threads=8/1x256x256	48.98 µs	37.09 µs	-24.3%
🟢	add/large_f16_threads=1-internal/4194304	54.97 ms	41.56 ms	-24.4%
🟢	sampling_latency/top_p_per_token	578.79 µs	425.41 µs	-26.5%
🟢	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	112.52 µs	82.38 µs	-26.8%
🟢	reduce_mean/small_f32_threads=1-internal/4096	21.84 µs	15.51 µs	-29.0%
🟢	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	139.72 µs	93.67 µs	-33.0%
🟢	qwen3_sampling_processors/top_k_full_sort_baseline	3.27 ms	2.14 ms	-34.4%
🟢	matmul/medium_generic_f32_threads=8/32x512x512	1.96 ms	1.22 ms	-37.6%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 3.86 3.68 6.12 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)