[justinchuby]

Agents on this repo have repeatedly concluded no CUDA runtime on this machine and used that to accept GPU performance and correctness claims as unverified. It is false, and it degraded review quality on every CUDA PR.

The box has an RTX 4060 (driver 591.55, CUDA 13.1) and a complete CUDA 13 runtime under anaconda site-packages. It is simply not on PATH, and because this repo loads CUDA libraries dynamically by DLL name, adding two directories is enough. The recipe is in the decision note.

Two defects kept the belief alive and hid each other: the stale cudart name list (#1178, fixed) and the unprepared ::Attention workspace (#1179, open). Together they made native CUDA genuinely unrunnable here, and that unrunnability was reported upward as absence of hardware.

The lesson worth keeping past both bugs: distinguish absent from misconfigured before building on a negative result. A negative that excuses you from verification deserves more scrutiny than a positive, not less.

[github-actions]

✅ Benchmarks — No Regression

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
✅	qwen3_sampling_processors/top_k_full_sort_baseline	2.20 ms	2.22 ms	+0.6%
✅	grammar_masking/llguidance_compute_mask/32	87.35 µs	87.69 µs	+0.4%
✅	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	1.16 ms	1.15 ms	-1.2%
✅	kv_cache/alloc_dealloc_pages	39.21 µs	38.52 µs	-1.8%
✅	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	3.65 ms	3.53 ms	-3.3%
✅	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	5.91 ms	5.70 ms	-3.5%
✅	qwen3_sampling_processors/top_k_partial_selection	151.64 µs	145.06 µs	-4.3%
✅	qwen3_sampling_processors/top_k_top_p_fast	700.94 µs	665.68 µs	-5.0%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	552.41 µs	524.28 µs	-5.1%
✅	tokenization/encode_tokens_per_second	406.55 µs	378.77 µs	-6.8%
✅	sampling_latency/min_p_per_token	228.59 µs	206.54 µs	-9.6%
✅	add/medium_f32_threads=1-internal/262144	36.16 µs	32.57 µs	-9.9%
✅	logit_processing/seven_processor_chain_per_step	364.57 µs	319.15 µs	-12.5%
✅	matmul/medium_generic_f32_threads=1/32x512x512	2.68 ms	2.35 ms	-12.6%
✅	matmul/medium_generic_f16_threads=1/32x512x512	35.35 µs	30.67 µs	-13.2%
✅	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	199.34 µs	170.91 µs	-14.3%
🟢	sampling_latency/top_k_per_token	61.72 µs	51.92 µs	-15.9%
🟢	tokenization/decode_tokens_per_second	7.58 ms	6.38 ms	-15.9%
🟢	sampling_latency/top_p_per_token	456.64 µs	380.50 µs	-16.7%
🟢	sampling_latency/greedy_per_token	3.92 µs	3.21 µs	-18.2%
🟢	add/large_f16_threads=1-internal/4194304	2.61 ms	2.14 ms	-18.3%
🟢	gather/medium_bf16_threads=1-internal/32768	3.34 µs	2.72 µs	-18.7%
🟢	add/small_f32_threads=1-internal/1024	308.6 ns	247.5 ns	-19.8%
🟢	add/large_bf16_threads=1-internal/4194304	3.00 ms	2.33 ms	-22.5%
🟢	matmul/small_generic_bf16_threads=1/1x256x256	41.01 µs	31.77 µs	-22.5%
🟢	add/large_f32_threads=1-internal/4194304	1.09 ms	843.21 µs	-22.7%
🟢	add/small_f16_threads=1-internal/1024	771.4 ns	592.9 ns	-23.1%
🟢	reduce_mean/small_f32_threads=1-internal/4096	24.64 µs	18.83 µs	-23.6%
🟢	matmul/medium_generic_bf16_threads=1/32x512x512	683.46 µs	519.92 µs	-23.9%
🟢	reduce_mean/large_f32_threads=1-internal/262144	1.46 ms	1.11 ms	-24.0%
🟢	matmul/small_generic_bf16_threads=8/1x256x256	41.75 µs	31.70 µs	-24.1%
🟢	gather/medium_f16_threads=1-internal/32768	3.68 µs	2.80 µs	-24.1%
🟢	matmul/small_generic_f16_threads=1/1x256x256	41.62 µs	31.10 µs	-25.3%
🟢	matmul/large_generic_f16_threads=1/32x1024x1024	116.36 µs	85.18 µs	-26.8%
🟢	add/small_bf16_threads=1-internal/1024	737.9 ns	524.2 ns	-29.0%
🟢	matmul/small_generic_f16_threads=8/1x256x256	44.02 µs	30.74 µs	-30.2%
🟢	gather/small_f32_threads=1-internal/4096	1.09 µs	751.6 ns	-30.9%
🟢	gather/small_f16_threads=1-internal/4096	721.8 ns	487.1 ns	-32.5%
🟢	matmul/small_generic_f32_threads=1/1x256x256	59.91 µs	40.33 µs	-32.7%
🟢	matmul/large_generic_f32_threads=1/32x1024x1024	13.94 ms	9.29 ms	-33.3%
🟢	gather/small_bf16_threads=1-internal/4096	721.7 ns	479.6 ns	-33.6%
🟢	add/medium_bf16_threads=1-internal/262144	160.90 µs	105.61 µs	-34.4%
🟢	add/medium_f16_threads=1-internal/262144	160.98 µs	104.00 µs	-35.4%
🟢	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	737.95 µs	476.67 µs	-35.4%
🟢	reduce_mean/medium_f32_threads=1-internal/65536	431.93 µs	276.38 µs	-36.0%
🟢	matmul/medium_generic_f16_threads=8/32x512x512	46.34 µs	29.53 µs	-36.3%
🟢	matmul/large_generic_bf16_threads=8/32x1024x1024	2.24 ms	1.41 ms	-37.3%
🟢	matmul/large_generic_bf16_threads=1/32x1024x1024	3.22 ms	1.97 ms	-38.7%
🟢	gather/large_f16_threads=1-internal/131072	24.41 µs	14.73 µs	-39.7%
🟢	gather/medium_f32_threads=1-internal/32768	8.02 µs	4.62 µs	-42.4%
🟢	gather/large_f32_threads=1-internal/131072	63.05 µs	34.23 µs	-45.7%
🟢	matmul/large_generic_f32_threads=8/32x1024x1024	7.15 ms	3.78 ms	-47.1%
🟢	matmul/medium_generic_bf16_threads=8/32x512x512	777.88 µs	407.95 µs	-47.6%
🟢	matmul/medium_generic_f32_threads=8/32x512x512	1.82 ms	940.17 µs	-48.4%
🟢	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	111.26 µs	55.41 µs	-50.2%
🟢	gather/large_bf16_threads=1-internal/131072	26.84 µs	13.27 µs	-50.6%
🟢	matmul/small_generic_f32_threads=8/1x256x256	86.15 µs	36.28 µs	-57.9%
🟢	matmul/large_generic_f16_threads=8/32x1024x1024	209.57 µs	85.83 µs	-59.0%
🟢	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	149.36 µs	45.08 µs	-69.8%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 3.43 3.65 7.73 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.46%. Comparing base (508920d) to head (443b725).
⚠️ Report is 27 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##           main    #1188       +/-   ##
=========================================
+ Coverage      0   80.46%   +80.46%     
=========================================
  Files         0      359      +359     
  Lines         0   157760   +157760     
  Branches      0   157760   +157760     
=========================================
+ Hits          0   126938   +126938     
- Misses        0    26208    +26208     
- Partials      0     4614     +4614     

Flag	Coverage Δ
mlas	85.53% <ø> (?)
offline	80.36% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.
see 359 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.