[justinchuby]

Summary

• use native CUDA runtime KV hard max before metadata max_sequence_length for reservation
• documents GLM/DeepSeek native load-gap investigation

Validation

• CUDA_VISIBLE_DEVICES=2 cargo test -p onnx-genai-engine --features native-backend,cuda native_kv_reservation -- --nocapture
• ONNX_GENAI_CUDA_KV_MAX_LEN=4096 ONNX_GENAI_GLM4_9B_CUDA_DIR=/home/justinchu/glm-e2e-artifacts/glm-4-9b-int4-cuda CUDA_VISIBLE_DEVICES=2 cargo test -q -p onnx-genai-engine --features native-backend,cuda --test glm4_9b_decode_lock -- --ignored --nocapture

GLM native CUDA token stream: [11, 358, 1079, 264, 220, 18, 6498, 1042].

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.04%. Comparing base (1179f7b) to head (1b09e2b).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #770      +/-   ##
==========================================
+ Coverage   79.88%   80.04%   +0.16%     
==========================================
  Files         331      329       -2     
  Lines      137462   135292    -2170     
  Branches   137462   135292    -2170     
==========================================
- Hits       109812   108298    -1514     
+ Misses      23306    22662     -644     
+ Partials     4344     4332      -12     

Flag	Coverage Δ
cli-ort-linux	83.41% <ø> (ø)
cli-ort-windows	82.90% <ø> (ø)
mlas	?
offline	79.91% <ø> (+0.18%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 5 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

⚠️ Benchmark Change Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
⚠️	gather/large_bf16_threads=1-internal/131072	14.59 µs	16.90 µs	+15.8%
✅	gather/large_f16_threads=1-internal/131072	14.49 µs	16.54 µs	+14.2%
✅	gather/large_f32_threads=1-internal/131072	35.86 µs	40.68 µs	+13.4%
✅	matmul/large_generic_f16_threads=8/32x1024x1024	104.10 µs	109.84 µs	+5.5%
✅	matmul/large_generic_bf16_threads=1/32x1024x1024	2.03 ms	2.12 ms	+4.5%
✅	qwen3_sampling_processors/top_k_partial_selection	147.30 µs	152.52 µs	+3.5%
✅	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	3.55 ms	3.67 ms	+3.4%
✅	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	134.09 µs	138.26 µs	+3.1%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	9.89 ms	10.19 ms	+3.0%
✅	matmul/medium_generic_f32_threads=8/32x512x512	1.94 ms	1.98 ms	+2.5%
✅	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	1.31 ms	1.35 ms	+2.4%
✅	matmul/medium_generic_f32_threads=1/32x512x512	2.54 ms	2.58 ms	+1.7%
✅	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	109.39 µs	110.94 µs	+1.4%
✅	tokenization/decode_tokens_per_second	6.70 ms	6.79 ms	+1.4%
✅	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	5.86 ms	5.93 ms	+1.2%
✅	matmul/small_generic_bf16_threads=8/1x256x256	38.35 µs	38.79 µs	+1.1%
✅	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	77.71 µs	78.32 µs	+0.8%
✅	sampling_latency/min_p_per_token	214.50 µs	213.84 µs	-0.3%
✅	sampling_latency/top_k_per_token	54.29 µs	54.11 µs	-0.3%
✅	matmul/large_generic_f32_threads=8/32x1024x1024	7.58 ms	7.54 ms	-0.4%
✅	qwen3_sampling_processors/top_k_top_p_fast	673.86 µs	670.69 µs	-0.5%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	549.49 µs	545.78 µs	-0.7%
✅	gather/medium_bf16_threads=1-internal/32768	2.45 µs	2.43 µs	-1.0%
✅	matmul/small_generic_bf16_threads=1/1x256x256	38.66 µs	38.13 µs	-1.4%
✅	qwen3_sampling_processors/top_k_full_sort_baseline	2.19 ms	2.16 ms	-1.6%
✅	matmul/medium_generic_f16_threads=1/32x512x512	40.28 µs	39.59 µs	-1.7%
✅	matmul/small_generic_f16_threads=1/1x256x256	39.69 µs	38.99 µs	-1.7%
✅	matmul/large_generic_bf16_threads=8/32x1024x1024	2.45 ms	2.39 ms	-2.5%
✅	matmul/medium_generic_bf16_threads=8/32x512x512	660.97 µs	644.50 µs	-2.5%
✅	grammar_masking/llguidance_compute_mask/32	82.97 µs	80.76 µs	-2.7%
✅	gather/small_f32_threads=1-internal/4096	694.5 ns	675.9 ns	-2.7%
✅	kv_cache/alloc_dealloc_pages	42.48 µs	41.28 µs	-2.8%
✅	sampling_latency/greedy_per_token	3.39 µs	3.29 µs	-3.1%
✅	logit_processing/seven_processor_chain_per_step	339.10 µs	327.98 µs	-3.3%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	546.02 µs	527.95 µs	-3.3%
✅	matmul/medium_generic_f16_threads=8/32x512x512	43.76 µs	42.26 µs	-3.4%
✅	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	582.33 µs	560.54 µs	-3.7%
✅	matmul/small_generic_f16_threads=8/1x256x256	38.99 µs	37.32 µs	-4.3%
✅	matmul/large_generic_f16_threads=1/32x1024x1024	91.24 µs	86.99 µs	-4.7%
✅	sampling_latency/top_p_per_token	407.89 µs	381.07 µs	-6.6%
✅	matmul/small_generic_f32_threads=1/1x256x256	50.25 µs	46.90 µs	-6.7%
✅	matmul/small_generic_f32_threads=8/1x256x256	55.30 µs	50.13 µs	-9.3%
✅	add/medium_f16_threads=1-internal/262144	2.99 ms	2.58 ms	-13.5%
✅	add/medium_bf16_threads=1-internal/262144	3.06 ms	2.64 ms	-13.6%
✅	gather/medium_f32_threads=1-internal/32768	4.64 µs	3.95 µs	-14.8%
🟢	add/small_bf16_threads=1-internal/1024	15.33 µs	12.89 µs	-15.9%
🟢	gather/small_f16_threads=1-internal/4096	594.3 ns	499.1 ns	-16.0%
🟢	add/medium_f32_threads=1-internal/262144	3.07 ms	2.53 ms	-17.6%
🟢	add/large_f16_threads=1-internal/4194304	52.51 ms	42.90 ms	-18.3%
🟢	tokenization/encode_tokens_per_second	478.23 µs	381.62 µs	-20.2%
🟢	add/small_f32_threads=1-internal/1024	267.0 ns	212.1 ns	-20.6%
🟢	reduce_mean/small_f32_threads=1-internal/4096	19.37 µs	15.36 µs	-20.7%
🟢	add/small_f16_threads=1-internal/1024	16.98 µs	13.22 µs	-22.2%
🟢	add/large_f32_threads=1-internal/4194304	52.26 ms	40.65 ms	-22.2%
🟢	gather/small_bf16_threads=1-internal/4096	622.4 ns	483.2 ns	-22.4%
🟢	add/large_bf16_threads=1-internal/4194304	55.51 ms	42.80 ms	-22.9%
🟢	reduce_mean/large_f32_threads=1-internal/262144	1.33 ms	997.50 µs	-24.7%
🟢	gather/medium_f16_threads=1-internal/32768	3.33 µs	2.41 µs	-27.5%
🟢	reduce_mean/medium_f32_threads=1-internal/65536	363.12 µs	248.10 µs	-31.7%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 3.45 3.42 6.01 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)