[justinchuby]

Summary

• Adds the ORT 1.28 three-config fairness benchmark report for native CUDA EP, ORT backend under our engine, and ORT-GenAI direct.
• Confirms ORT 1.28.0/API 28 was loaded for Rust and Python/ORT-GenAI paths.
• Documents token-stream parity/diff results and capability probes for 35B-A3B QMoE, GLM-4-9B, DeepSeek-V2-Lite QMoE, and 27B.

Results headline

• Qwen2.5-0.5B: native 557.57 tok/s vs ORT-GenAI direct 203.22 tok/s (2.74x faster); ORT backend and ORT-GenAI tokens match, native diverges at token 68.
• DeepSeek-R1-Distill-Qwen-1.5B: native 599.49 tok/s vs ORT-GenAI direct 487.26 tok/s (1.23x faster); all streams eventually diverge.
• ORT 1.28 still does not load the 35B-A3B QMoE artifact via ORT backend due QMoE type binding; ORT-GenAI direct cannot run the available artifact because it lacks genai_config.json.

Validation

• Documentation-only change; no code tests required.
• Benchmarks/probes run on CUDA_VISIBLE_DEVICES=2 with ORT 1.28 pinned via ONNX_GENAI_ORT_LIB.

[justinchuby]

Independent review verdict: ✅ APPROVE (Harry)

Note: I could not post this as a formal --approve review because the only GitHub credential available in this environment is the PR author account (justinchuby), and GitHub blocks self-approval. A non-author reviewer will need to click Approve to unblock auto-merge. The substantive review below is complete.

I verified the fairness claims against the data in the doc itself; all hold up.

Token-parity honesty (verified against the printed arrays):

• Qwen2.5-0.5B: A(native) first diverges from B/C at index 68 (2213 vs 8794); B==C byte-identical across all 128 tokens. Matches the doc exactly.
• DeepSeek-R1-Distill-1.5B: A vs B first diverge at index 31 (22756 vs 21495); C diverges from both A and B at index 10 (25 vs 11). Matches the doc exactly. The doc honestly discloses that all three streams eventually diverge and that C (ORT-GenAI direct) diverges earliest — no cherry-picking or hidden divergences.

Headline honesty:

• 557.57/203.22 = 2.74x and 599.49/487.26 = 1.23x check out; all ms/tok values are internally consistent.
• The speedup is stated against the faster ORT path (C, the full-stack ceiling), not the slower B backend — a conservative, honest framing.
• No byte-identical claim is made. The headline explicitly caveats that native diverges from ORT (token 68 / token 10). These are benign near-tie greedy divergences between different EP implementations.

Methodology soundness:

• Shared host + 'report medians / treat small deltas as noisy' noted.
• ORT 1.28.0 (API 28) confirmed in-doc for both the Rust engine and the Python/ORT-GenAI paths.
• Greedy temp=0 stated (do_sample=False, temperature=0.0, top_p=1.0, top_k=0); steady-state decode window documented.
• Section 2 capability probes correctly labeled 'capability checks, not fair throughput numbers.'

No overstated or unsupported claims found. This is an honest data snapshot. Approving.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.01%. Comparing base (4dd5119) to head (d3a6df5).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #766      +/-   ##
==========================================
+ Coverage   79.33%   80.01%   +0.67%     
==========================================
  Files         329      331       +2     
  Lines      135292   137462    +2170     
  Branches   135292   137462    +2170     
==========================================
+ Hits       107337   109986    +2649     
+ Misses      23625    23132     -493     
- Partials     4330     4344      +14     

Flag	Coverage Δ
cli-ort-linux	83.41% <ø> (ø)
cli-ort-windows	82.90% <ø> (ø)
mlas	80.96% <ø> (?)
offline	79.85% <ø> (+0.68%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 8 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	matmul/small_generic_f32_threads=8/1x256x256	35.55 µs	55.29 µs	+55.5%
🔴	gather/medium_f32_threads=1-internal/32768	4.25 µs	5.99 µs	+41.1%
🔴	matmul/medium_generic_bf16_threads=8/32x512x512	423.20 µs	588.24 µs	+39.0%
🔴	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	47.38 µs	64.89 µs	+37.0%
🔴	tokenization/decode_tokens_per_second	6.23 ms	8.38 ms	+34.5%
🔴	matmul/medium_generic_f32_threads=8/32x512x512	1.25 ms	1.66 ms	+32.8%
🔴	add/medium_bf16_threads=1-internal/262144	2.62 ms	3.46 ms	+32.0%
⚠️	gather/small_f16_threads=1-internal/4096	475.7 ns	608.8 ns	+28.0%
⚠️	gather/small_bf16_threads=1-internal/4096	480.6 ns	610.4 ns	+27.0%
⚠️	matmul/medium_generic_f16_threads=1/32x512x512	37.57 µs	46.75 µs	+24.4%
⚠️	tokenization/encode_tokens_per_second	381.20 µs	474.12 µs	+24.4%
⚠️	reduce_mean/large_f32_threads=1-internal/262144	979.01 µs	1.20 ms	+23.0%
⚠️	gather/large_f16_threads=1-internal/131072	12.90 µs	15.79 µs	+22.4%
⚠️	matmul/small_generic_f32_threads=1/1x256x256	39.63 µs	48.34 µs	+22.0%
⚠️	add/large_f32_threads=1-internal/4194304	40.08 ms	48.70 ms	+21.5%
⚠️	gather/small_f32_threads=1-internal/4096	672.0 ns	812.2 ns	+20.9%
⚠️	gather/large_f32_threads=1-internal/131072	34.30 µs	40.00 µs	+16.6%
⚠️	add/large_f16_threads=1-internal/4194304	40.98 ms	47.59 ms	+16.1%
⚠️	reduce_mean/medium_f32_threads=1-internal/65536	244.36 µs	282.97 µs	+15.8%
✅	reduce_mean/small_f32_threads=1-internal/4096	15.28 µs	17.29 µs	+13.1%
✅	add/medium_f16_threads=1-internal/262144	2.58 ms	2.91 ms	+13.1%
✅	matmul/medium_generic_f16_threads=8/32x512x512	46.14 µs	51.97 µs	+12.6%
✅	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	549.93 µs	612.81 µs	+11.4%
✅	add/medium_f32_threads=1-internal/262144	2.55 ms	2.81 ms	+10.4%
✅	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	385.19 µs	416.87 µs	+8.2%
✅	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	51.99 µs	56.08 µs	+7.9%
✅	add/small_bf16_threads=1-internal/1024	13.36 µs	14.36 µs	+7.5%
✅	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	78.20 µs	83.33 µs	+6.6%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	584.43 µs	622.66 µs	+6.5%
✅	sampling_latency/greedy_per_token	3.25 µs	3.46 µs	+6.4%
✅	matmul/medium_generic_f32_threads=1/32x512x512	2.70 ms	2.85 ms	+5.5%
✅	sampling_latency/top_k_per_token	52.57 µs	54.58 µs	+3.8%
✅	add/large_bf16_threads=1-internal/4194304	48.29 ms	49.84 ms	+3.2%
✅	gather/large_bf16_threads=1-internal/131072	15.58 µs	16.05 µs	+3.0%
✅	gather/medium_f16_threads=1-internal/32768	3.08 µs	3.10 µs	+0.9%
✅	add/small_f32_threads=1-internal/1024	225.1 ns	224.2 ns	-0.4%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	580.78 µs	563.44 µs	-3.0%
✅	matmul/small_generic_f16_threads=1/1x256x256	41.27 µs	40.02 µs	-3.0%
✅	matmul/large_generic_bf16_threads=8/32x1024x1024	1.48 ms	1.43 ms	-3.4%
✅	matmul/small_generic_bf16_threads=1/1x256x256	42.89 µs	41.08 µs	-4.2%
✅	sampling_latency/top_p_per_token	410.17 µs	390.71 µs	-4.7%
✅	grammar_masking/llguidance_compute_mask/32	88.59 µs	84.35 µs	-4.8%
✅	matmul/small_generic_f16_threads=8/1x256x256	42.67 µs	40.53 µs	-5.0%
✅	matmul/large_generic_f16_threads=8/32x1024x1024	104.01 µs	98.73 µs	-5.1%
✅	matmul/large_generic_f16_threads=1/32x1024x1024	90.36 µs	85.15 µs	-5.8%
✅	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	3.97 ms	3.73 ms	-6.2%
✅	add/small_f16_threads=1-internal/1024	15.39 µs	14.36 µs	-6.7%
✅	qwen3_sampling_processors/top_k_top_p_fast	728.16 µs	676.28 µs	-7.1%
✅	qwen3_sampling_processors/top_k_partial_selection	160.24 µs	148.00 µs	-7.6%
✅	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	6.36 ms	5.80 ms	-8.8%
✅	matmul/large_generic_bf16_threads=1/32x1024x1024	2.32 ms	2.11 ms	-9.2%
✅	qwen3_sampling_processors/top_k_full_sort_baseline	2.38 ms	2.16 ms	-9.3%
✅	kv_cache/alloc_dealloc_pages	44.41 µs	39.54 µs	-11.0%
✅	logit_processing/seven_processor_chain_per_step	369.17 µs	325.85 µs	-11.7%
🟢	matmul/large_generic_f32_threads=1/32x1024x1024	11.45 ms	9.55 ms	-16.6%
🟢	sampling_latency/min_p_per_token	253.75 µs	209.80 µs	-17.3%
🟢	gather/medium_bf16_threads=1-internal/32768	3.30 µs	2.71 µs	-17.9%
🟢	matmul/small_generic_bf16_threads=8/1x256x256	50.09 µs	40.82 µs	-18.5%
🟢	matmul/large_generic_f32_threads=8/32x1024x1024	5.27 ms	4.04 ms	-23.3%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 4.17 3.42 6.14 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)