[justinchuby]

What this does

Comment-only follow-up to #1404. Corrects two landed doc comments in matmul_nbits.rs (the plain fp16_gate_up_swiglu_two_op_ulp_bound_sweep) that mischaracterized the default plain-M>1 path.

What was wrong

The comments said plain M>1 "dispatches Marlin split-K (default-on), which is documented non-byte-identical and possibly non-deterministic." Both halves are wrong. Verified against crates/onnx-runtime-ep-cuda/src/kernels/marlin_gemm.rs:

• Split-K is default OFF. It is gated by ONNX_GENAI_MARLIN_SPLITK (marlin_splitk_enabled()), separate from ONNX_GENAI_MARLIN_M_GT_1. maybe_launch_marlin_splitk elects it only when that flag is set and choose_split_k > 1; otherwise it calls the direct kernel. So split-K does not run under the default config.
• Split-K is deterministic where it does run: "a fixed-order fp32 partial reduction that is NOT byte-identical to the single-block kernel (it stays within the f64-oracle tolerance and is deterministic — greedy/argmax tokens remain byte-identical, validated e2e on glm-4-9b and qwen2.5-14b)."
• The actual default plain-M>1 path is the direct Marlin int4 tensor-core GEMM (launch_marlin_gemm), documented non-byte-identical to the tiled two-op reference by a different accumulation order (parity tests assert a 2e-2 * max_out tolerance, not equality).

What the corrected comments say

They now name the correct path (direct Marlin, not split-K), drop "possibly non-deterministic", and record that main does not hold "M>1 is bit-exact to the two-op reference" under the default config — fp16_gate_up_swiglu_is_bit_exact_to_two_op_path reds at M=5 with Marlin default-on and passes only under ONNX_GENAI_MARLIN_M_GT_1=0.

This aligns the code comments with the corrected framing already applied to PR #1404's body, issue #1405 (now titled "direct tensor-core GEMM, not split-K"), and the #1305 correction comment.

Why it's safe

Pure comment change — no code, no behavior change, no test change. The /// / // edits do not touch any executable line.

Follows-up #1404. Marlin write-up: #1405.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[justinchuby]

Comment-only correction; landing per the batching workstream convention (own-PR, CI not awaited). Aligns code comments with the corrected #1404 body / #1405 / #1305.

[github-actions]

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	gather/large_f32_threads=1-internal/131072	24.79 µs	37.05 µs	+49.4%
🔴	gather/small_f16_threads=1-internal/4096	440.1 ns	632.9 ns	+43.8%
🔴	reduce_mean/large_f32_threads=1-internal/262144	929.10 µs	1.27 ms	+36.2%
🔴	add/large_f32_threads=1-internal/4194304	557.96 µs	732.43 µs	+31.3%
⚠️	reduce_mean/small_f32_threads=1-internal/4096	13.88 µs	17.92 µs	+29.1%
⚠️	add/small_f32_threads=1-internal/1024	189.6 ns	242.1 ns	+27.7%
⚠️	reduce_mean/medium_f32_threads=1-internal/65536	227.42 µs	288.77 µs	+27.0%
⚠️	gather/small_f32_threads=1-internal/4096	616.3 ns	772.5 ns	+25.3%
⚠️	gather/medium_f32_threads=1-internal/32768	3.61 µs	4.39 µs	+21.9%
⚠️	add/small_f16_threads=1-internal/1024	428.4 ns	508.0 ns	+18.6%
⚠️	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	385.15 µs	451.69 µs	+17.3%
⚠️	gather/medium_f16_threads=1-internal/32768	2.22 µs	2.56 µs	+15.3%
✅	add/large_bf16_threads=1-internal/4194304	1.58 ms	1.80 ms	+13.9%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	11.21 ms	12.61 ms	+12.5%
✅	add/large_f16_threads=1-internal/4194304	1.53 ms	1.72 ms	+12.1%
✅	add/medium_f16_threads=1-internal/262144	97.00 µs	107.95 µs	+11.3%
✅	add/medium_f32_threads=1-internal/262144	23.32 µs	25.54 µs	+9.5%
✅	gather/small_bf16_threads=1-internal/4096	528.3 ns	572.8 ns	+8.4%
✅	add/small_bf16_threads=1-internal/1024	420.9 ns	452.2 ns	+7.4%
✅	add/medium_bf16_threads=1-internal/262144	101.07 µs	105.17 µs	+4.1%
✅	tokenization/decode_tokens_per_second	6.57 ms	6.84 ms	+4.0%
✅	matmul/medium_generic_bf16_threads=8/32x512x512	680.89 µs	685.82 µs	+0.7%
✅	sampling_latency/greedy_per_token	3.14 µs	3.13 µs	-0.4%
✅	gather/medium_bf16_threads=1-internal/32768	2.47 µs	2.45 µs	-0.6%
✅	sampling_latency/min_p_per_token	199.91 µs	196.95 µs	-1.5%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	638.17 µs	614.90 µs	-3.6%
✅	grammar_masking/llguidance_compute_mask/32	80.05 µs	76.97 µs	-3.9%
✅	sampling_latency/top_p_per_token	373.34 µs	354.38 µs	-5.1%
✅	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	3.59 ms	3.38 ms	-5.9%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	513.53 µs	483.39 µs	-5.9%
✅	sampling_latency/top_k_per_token	52.98 µs	49.08 µs	-7.4%
✅	qwen3_sampling_processors/top_k_full_sort_baseline	2.16 ms	1.99 ms	-8.0%
✅	qwen3_sampling_processors/top_k_top_p_fast	666.86 µs	612.52 µs	-8.1%
✅	matmul/large_generic_f16_threads=8/32x1024x1024	118.78 µs	107.37 µs	-9.6%
✅	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	5.84 ms	5.27 ms	-9.7%
✅	gather/large_bf16_threads=1-internal/131072	17.87 µs	16.01 µs	-10.4%
✅	qwen3_sampling_processors/top_k_partial_selection	145.40 µs	130.23 µs	-10.4%
✅	matmul/large_generic_f16_threads=1/32x1024x1024	105.21 µs	93.99 µs	-10.7%
✅	kv_cache/alloc_dealloc_pages	41.49 µs	36.39 µs	-12.3%
🟢	logit_processing/seven_processor_chain_per_step	351.15 µs	295.64 µs	-15.8%
🟢	matmul/large_generic_bf16_threads=1/32x1024x1024	2.80 ms	2.24 ms	-19.9%
🟢	matmul/small_generic_f32_threads=1/1x256x256	48.16 µs	38.51 µs	-20.0%
🟢	matmul/small_generic_f16_threads=1/1x256x256	36.76 µs	29.15 µs	-20.7%
🟢	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	105.24 µs	82.76 µs	-21.4%
🟢	matmul/small_generic_bf16_threads=1/1x256x256	37.55 µs	29.19 µs	-22.3%
🟢	gather/large_f16_threads=1-internal/131072	17.79 µs	13.68 µs	-23.1%
🟢	tokenization/encode_tokens_per_second	508.13 µs	388.48 µs	-23.5%
🟢	matmul/small_generic_bf16_threads=8/1x256x256	38.87 µs	28.81 µs	-25.9%
🟢	matmul/medium_generic_f16_threads=1/32x512x512	46.38 µs	34.02 µs	-26.6%
🟢	matmul/medium_generic_f32_threads=1/32x512x512	2.94 ms	2.14 ms	-27.2%
🟢	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	112.00 µs	79.53 µs	-29.0%
🟢	matmul/small_generic_f16_threads=8/1x256x256	39.81 µs	27.89 µs	-29.9%
🟢	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	83.53 µs	57.28 µs	-31.4%
🟢	matmul/small_generic_f32_threads=8/1x256x256	55.57 µs	36.02 µs	-35.2%
🟢	matmul/large_generic_f32_threads=8/32x1024x1024	7.60 ms	4.70 ms	-38.1%
🟢	matmul/large_generic_bf16_threads=8/32x1024x1024	2.34 ms	1.32 ms	-43.4%
🟢	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	1.21 ms	680.05 µs	-44.0%
🟢	matmul/medium_generic_f16_threads=8/32x512x512	58.01 µs	31.46 µs	-45.8%
🟢	matmul/medium_generic_f32_threads=8/32x512x512	1.84 ms	946.08 µs	-48.5%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 3.25 3.29 5.05 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)