perf(cuda-ep): optimize MatMulNBits int4 dequant GEMV on native decode (40→47 tok/s) by justinchuby · Pull Request #867 · justinchuby/onnx-genai

justinchuby · 2026-08-13T04:18:55Z

Summary

Native CUDA decode of Muse-Glimmer-30B was ORT-parity at 40.21 tok/s, with MatMulNBits the dominant op (~44% of eager decode). This PR pushes to a clear win over ORT: 47.31 tok/s (+17.7%), byte-exact.

Root cause: the BFloat16 activation path (MatMulNBitsKernel::run_bf16) re-staged every input BFloat16→Float16 into an ephemeral arena on every decode step — including the int4 block scales, which are immutable weights (N × ceil(K/32) elements per node). For Muse-Glimmer that is ~3.3 GB/token of pure-copy traffic (read+write, ≈25% of the 13.2 GB/token int4 weight traffic) plus 417 redundant cast launches, all reproducing an identical Float16 buffer.

Fix: a persistent per-kernel Bf16ConstCache that stages the constant scale slots once (general path input 2; gate/up SwiGLU fusion inputs 2 and 4) and reuses the f16 copies across steps. The per-step activation (input 0) and any per-token residual bound into the bias slot stay on the ephemeral arena — they are genuinely dynamic. Caching never keys on pointer identity for those dynamic slots (a reused activation buffer has a stable pointer but changing contents).

Numbers (H200, `CUDA_VISIBLE_DEVICES=0`, `ONNX_GENAI_CUDA_GRAPH=1`, `--pipeline --backend native --steady --warmups 1 --runs 3 --tokens 128`)

metric	before	after
tok/s (median)	40.21	47.31 (47.35 / 47.31 / 47.19)
MatMulNBits eager share	~44%	~31%
capture segments / seams	1 / 0	1 / 0
first-16 greedy ids	reference	match exactly

Reference ids: [24, 372, 1045, 10016, 328, 2885, 262, 5091, 8811, 511, 917, 4921, 768, 328, 2885, 262] — full 128-token sequence unchanged.

Correctness / numerics

Byte-exact — no numerics change, no Chew gate required. bf16→f16 conversion yields identical f16 bits whether cast once (cached) or per step, and the downstream fp16 GEMV reads identical scales, so decode output is bit-for-bit identical. The fp16 GEMV accumulation internals are untouched.

New unit test bf16_scale_cache_is_bit_exact_to_inline_staging proves the cached path is bit-identical to inline per-call staging and deterministic across steps. All 25 matmul_nbits CUDA tests pass; cargo fmt --all -- --check + clippy clean.

Capture-safety

The persistent buffer is allocated + populated on the pre-capture warmup call (cache miss → alloc + one cast per constant). Captured replays hit only cache lookups (no alloc, no cast), so the graph stays capture-stable — same lifecycle as the existing Bf16Scratch arena. Verified: 1 segment / 0 seams retained.

Follow-ups (deferred)

GroupQueryAttention is now the largest eager share (~41%). A fuller bf16-native GEMV (bf16 in/out, no f16 round-trip) or the accuracy_level=4 DP4A int8-activation path remain open levers, but both carry a numerics cost (Chew gate) and Muse-Glimmer's nodes declare no accuracy_level, so DP4A is not currently exercised.

…16 decode (40→47 tok/s) The BFloat16 activation path (`run_bf16`) re-staged every input to Float16 each decode step, including the int4 block scales — immutable weights totaling N×ceil(K/32) elements per node. For Muse-Glimmer-30B that is ~3.3 GB/token of pure-copy traffic (~25% of the int4 weight traffic) plus 417 redundant cast launches, all reproducing an identical Float16 buffer. Add a persistent per-kernel `Bf16ConstCache` that stages the constant scale slots once (general path input 2; gate/up SwiGLU inputs 2 and 4) and reuses the f16 copies across steps. The per-step activation and any per-token residual bound into the bias slot stay on the ephemeral arena; caching never keys on pointer identity for those dynamic slots. Byte-exact: bf16→f16 yields identical f16 bits whether cast once or per step, so decode output is bit-for-bit unchanged. New unit test `bf16_scale_cache_is_bit_exact_to_inline_staging` proves the cached path is bit-identical to inline staging and deterministic across steps. Capture-stable: the buffer is allocated + filled on the pre-capture warmup call; captured replays hit only cache lookups (no alloc, no cast). Measured on H200 (ONNX_GENAI_CUDA_GRAPH=1, --pipeline --backend native --steady --warmups 1 --runs 3 --tokens 128): 40.21 → 47.31 tok/s median (+17.7%), MatMulNBits eager share ~44% → ~31%, capture 1 segment / 0 seams, first-16 greedy ids match the reference exactly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…UDA 47 tok/s beats ORT - Merge sebastian-cuda-matmulnbits-gemv (#867) into decisions.md and delete inbox drop - Archive detailed 23→40 (#860) narrative to decisions-archive/2026-08.md; retain standing numerics rule - Append PR #867 milestone to sebastian/history.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…tok/s) (#868) ## Summary Docs-only. Records the MatMulNBits constant-scale cache lever (**#867**, merged, main `1002e360`) that took Muse-Glimmer-30B native CUDA decode from ORT-parity to a **clear win**: **40.21 → 47.25 tok/s** (H200 reconfirmed 47.25 median: 47.25/47.28/47.24), +18% over ORT's ~40. ## Changes to `docs/PROGRESS.md` - **2026-08-13 milestone**: retitled "reaches ORT parity (40 tok/s)" → "beats ORT (47 tok/s)"; added a sub-bullet for the `Bf16ConstCache` scale-caching win (root cause: ~3.3 GB/token of redundant bf16→f16 scale re-casts ≈25% of int4 weight traffic; fix caches the constant scale slots once). Records MatMulNBits eager share ~44% → ~31%, full 128-token greedy output byte-identical, capture **1 segment / 0 seams**, and flags `GroupQueryAttention` (~41%) as the next open lever. - **Top-of-file current-status snapshot**: "decodes at ORT parity … 40.21 tok/s (matches ORT's ~40)" → "decodes faster than ORT … 47.25 tok/s (clearly beats ORT's ~40, +18%)". - Bumped _Last updated_ timestamp and `origin/main` HEAD → `1002e360`. ## README Left unchanged after inspection: line 167 is a generic REPL live-stats illustration (613 in / 64 out), and lines 273-274 are `models/qwen2.5-0.5b` on the **CPU** EP — neither refers to Muse-Glimmer native CUDA decode, so per instructions they were not touched. Docs only — no cargo/clippy needed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

codecov · 2026-08-13T04:28:00Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.26%. Comparing base (bcef375) to head (abc61d8).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #867      +/-   ##
==========================================
- Coverage   78.30%   78.26%   -0.05%     
==========================================
  Files         353      351       -2     
  Lines      147642   145472    -2170     
  Branches   147642   145472    -2170     
==========================================
- Hits       115617   113858    -1759     
+ Misses      27529    27128     -401     
+ Partials     4496     4486      -10

Flag	Coverage Δ
cli-ort-linux	`83.42% <ø> (ø)`
cli-ort-windows	`82.92% <ø> (-0.10%)`	⬇️
mlas	`?`
offline	`78.07% <ø> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 3 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

github-actions · 2026-08-13T04:45:47Z

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	`gather/large_f32_threads=1-internal/131072`	21.34 µs	28.99 µs	+35.8%
✅	`gather/large_bf16_threads=1-internal/131072`	10.71 µs	11.58 µs	+8.1%
✅	`grammar_masking/llguidance_compute_mask/32`	69.15 µs	71.37 µs	+3.2%
✅	`qwen3_sampling_processors/top_k_top_p_fast`	610.89 µs	623.70 µs	+2.1%
✅	`logit_processing/seven_processor_chain_per_step`	296.67 µs	302.52 µs	+2.0%
✅	`qwen3_sampling_processors/top_k_partial_selection`	134.02 µs	136.03 µs	+1.5%
✅	`qwen3_sampling_processors/top_k_top_p_full_sort_baseline`	5.30 ms	5.33 ms	+0.6%
✅	`kv_cache/alloc_dealloc_pages`	36.60 µs	36.82 µs	+0.6%
✅	`matmul/small_generic_bf16_threads=8/1x256x256`	30.07 µs	29.86 µs	-0.7%
✅	`qwen3_sampling_processors/top_p_fast_after_top_k`	494.70 µs	489.14 µs	-1.1%
✅	`qwen3_sampling_processors/top_k_full_sort_baseline`	2.00 ms	1.97 ms	-1.7%
✅	`matmul/small_generic_f32_threads=8/1x256x256`	33.99 µs	32.96 µs	-3.0%
✅	`qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline`	3.48 ms	3.36 ms	-3.6%
✅	`matmul/small_generic_bf16_threads=1/1x256x256`	30.75 µs	29.48 µs	-4.1%
✅	`sampling_latency/top_p_per_token`	402.50 µs	385.12 µs	-4.3%
✅	`matmul/small_generic_f16_threads=8/1x256x256`	30.44 µs	28.97 µs	-4.9%
✅	`gather/large_f16_threads=1-internal/131072`	11.00 µs	10.38 µs	-5.6%
✅	`gather/medium_f32_threads=1-internal/32768`	3.82 µs	3.59 µs	-6.1%
✅	`gather/small_bf16_threads=1-internal/4096`	471.0 ns	438.6 ns	-6.9%
✅	`matmul/medium_generic_f32_threads=1/32x512x512`	2.34 ms	2.15 ms	-8.0%
✅	`sampling_latency/greedy_per_token`	3.28 µs	3.00 µs	-8.7%
✅	`sampling_latency/top_k_per_token`	54.47 µs	49.04 µs	-10.0%
✅	`matmul/small_generic_f32_threads=1/1x256x256`	38.87 µs	34.26 µs	-11.9%
✅	`matmul/large_generic_f32_threads=1/32x1024x1024`	9.98 ms	8.73 ms	-12.5%
✅	`add/medium_f16_threads=1-internal/262144`	2.76 ms	2.40 ms	-12.9%
✅	`block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1`	83.87 µs	72.43 µs	-13.6%
✅	`add/medium_f32_threads=1-internal/262144`	2.73 ms	2.36 ms	-13.8%
✅	`gather/small_f16_threads=1-internal/4096`	536.8 ns	461.1 ns	-14.1%
✅	`reduce_mean/medium_f32_threads=1-internal/65536`	264.29 µs	225.71 µs	-14.6%
✅	`tokenization/decode_tokens_per_second`	6.61 ms	5.63 ms	-14.9%
🟢	`sampling_latency/min_p_per_token`	233.00 µs	194.54 µs	-16.5%
🟢	`matmul/large_generic_bf16_threads=1/32x1024x1024`	2.21 ms	1.83 ms	-17.4%
🟢	`add/large_f32_threads=1-internal/4194304`	45.43 ms	37.49 ms	-17.5%
🟢	`tokenization/encode_tokens_per_second`	433.15 µs	356.25 µs	-17.8%
🟢	`gather/medium_bf16_threads=1-internal/32768`	2.74 µs	2.25 µs	-17.8%
🟢	`add/small_f32_threads=1-internal/1024`	220.1 ns	180.8 ns	-17.8%
🟢	`matmul/large_generic_f16_threads=1/32x1024x1024`	95.14 µs	77.37 µs	-18.7%
🟢	`matmul/medium_generic_bf16_threads=1/32x512x512`	601.66 µs	488.05 µs	-18.9%
🟢	`reduce_mean/large_f32_threads=1-internal/262144`	1.16 ms	930.46 µs	-19.7%
🟢	`add/large_f16_threads=1-internal/4194304`	48.44 ms	38.60 ms	-20.3%
🟢	`gather/small_f32_threads=1-internal/4096`	784.4 ns	620.0 ns	-21.0%
🟢	`add/medium_bf16_threads=1-internal/262144`	3.11 ms	2.45 ms	-21.0%
🟢	`matmul/small_generic_f16_threads=1/1x256x256`	36.41 µs	28.62 µs	-21.4%
🟢	`matmul/medium_generic_f16_threads=1/32x512x512`	35.47 µs	27.55 µs	-22.3%
🟢	`add/small_f16_threads=1-internal/1024`	15.27 µs	11.82 µs	-22.6%
🟢	`gather/medium_f16_threads=1-internal/32768`	2.98 µs	2.27 µs	-23.8%
🟢	`add/small_bf16_threads=1-internal/1024`	15.74 µs	11.96 µs	-24.0%
🟢	`matmul/large_generic_f16_threads=8/32x1024x1024`	103.71 µs	78.75 µs	-24.1%
🟢	`matmul/medium_generic_f16_threads=8/32x512x512`	41.36 µs	28.67 µs	-30.7%
🟢	`reduce_mean/small_f32_threads=1-internal/4096`	20.85 µs	13.91 µs	-33.3%
🟢	`block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1`	533.28 µs	353.04 µs	-33.8%
🟢	`add/large_bf16_threads=1-internal/4194304`	60.29 ms	39.06 ms	-35.2%
🟢	`matmul/medium_generic_f32_threads=8/32x512x512`	1.39 ms	892.95 µs	-35.9%
🟢	`matmul/large_generic_bf16_threads=8/32x1024x1024`	2.21 ms	1.30 ms	-41.3%
🟢	`matmul/large_generic_f32_threads=8/32x1024x1024`	6.47 ms	3.70 ms	-42.8%
🟢	`block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024`	88.70 µs	47.46 µs	-46.5%
🟢	`matmul/medium_generic_bf16_threads=8/32x512x512`	706.01 µs	374.56 µs	-46.9%
🟢	`block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024`	91.30 µs	46.08 µs	-49.5%
🟢	`block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024`	1.15 ms	550.19 µs	-52.2%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 3.06 3.14 5.92 }

What this cannot catch

Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
Sub-threshold regressions that compound over multiple PRs
Performance changes that only manifest under GPU execution
Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)

@Chew

…fe fusion (#871) ## Summary Muse-Glimmer-30B's decoder is **bfloat16** and expresses SwiGLU as decomposed graph ops (`Sigmoid → Mul(gate, sigmoid) → Mul(× up)`). The CUDA EP's decomposed-SiLU fusion only had a **Float16** kernel, so the standalone `SiluMulKernel` (and the decomposed `Silu` unary) hard-rejected BFloat16 with `"decomposed SiLU fusion requires float16"` — a portability defect that blocks the entire SwiGLU node-count reduction on bf16 models. This PR adds the missing bf16 kernels (the **kernel half** of the SwiGLU-fusion lever): - `decomposed_silu_mul_bf16` + `decomposed_silu_bf16` NVRTC kernels. - Admit `BFloat16` in the decomposed path of `SiluMulKernel` and `UnaryKernel`. - **Byte-exact**: Sigmoid and the silu product each round to bf16 via the same `__float2bfloat16_rn` the standalone `sigmoid_bf16`/`mul_bf16` ops use; **all intermediate math in fp32** (bf16 has ~8 mantissa bits). Reproduces the unfused two-op bf16 graph bit-for-bit. - Added an **f64/bf16 oracle test** asserting bit-equality vs the two-op reference across sign / rounding-boundary inputs. Non-decomposed bf16 already worked (`silu_mul_bf16`); this closes the decomposed gap. ## Context (important — read the decision drop) The coordinator framed SwiGLU-Mul folding as "self-contained in the kernel." **It isn't:** the fused `gate_up_swiglu` epilogue already folds `silu(gate)*up` (#867), and is already bf16-ready via `MatMulNBitsKernel::run_bf16` staging. The standalone `Mul` survives only because the graph-rewrite passes are **bf16-gated in `optimizer.rs`** (Batty's file): - `CudaSiluFusion` (optimizer.rs:129) rejects the bf16 Sigmoid input → **Batty one-liner** to admit bf16; composes with THIS PR to enable the safe `Sigmoid+Mul → silu_mul` fold. - `CudaGateUpSwiGluFusion::eligible_projection` (~1829) requires fp16 activation/scales → gates the FULL fold. ## Measured (H200, `ONNX_GENAI_CUDA_GRAPH=1`, staged Muse-Glimmer, capture 1seg/0seams) | config | nodes/token | tok/s | parity | |---|---|---|---| | baseline | 2568 | 47.20 | ref | | + gate #1 + this PR (safe silu_mul fold) | 2464 (−104) | 47.64 (flat, noise) | byte-exact | | + gate #2 (full gate_up fold) | 2256 (−312) | — | **non-deterministic** ⚠️ | **Honest result:** the safe fold removes the *cheapest* nodes → tok/s flat (confirms decode is dispatch-bound, per #870). The full gate_up bf16 fold reaches −312 nodes but is **non-deterministic** through the f16-staging path → needs a **bf16-native fused gate_up kernel** (scoped follow-up, Sebastian + Chew). See `.squad/decisions/inbox/sebastian-bf16-swiglu-fusion-contract.md`. ## Numerics fp32 accumulation; byte-exact oracle test. **Reviewer: @Chew** (bf16 dtype path). ## Validation `cargo fmt --all -- --check` ✓ · `cargo clippy -p onnx-runtime-ep-cuda --features cuda` clean ✓ · 5/5 silu tests pass (incl. new bf16 oracle) ✓ ⚠️ Perf-flat on its own; ships as the kernel prerequisite + a real bf16 portability fix. The tok/s win depends on Batty's gate #1 (safe) and the scoped bf16-native gate_up follow-up (full). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…decode ceiling Merge 5 inbox drops (chew-pr871-numerics, batty-fusion-contract, batty-qkv-contract, sebastian-bf16-swiglu-fusion-contract, sebastian-gqa-not-a-capture-lever) into the 'Fusion arc — 47.25 tok/s is the architectural ceiling' milestone (PRs #870/#871/#872/#873). Archive detailed #867 + capture-arc narratives to keep decisions.md under the 50 KB gate. Append #871/#872/#873 + ceiling conclusion to sebastian/chew/batty history. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…MLP measured (NO-GO for GEMV megakernel) (#898) ## The final go/no-go micro-bench before committing agents to the multi-week P2 P1.5 pinned the only viable architecture (persistent **multi-CTA cooperative**, grid.sync capturable, single-CTA ruled out at 926×). This PR **builds that exact kernel** for the MLP triple-GEMV block (`gate/up → SiLU·Mul → down`, grid sized to occupancy, `grid.sync` seams, activations in L2-resident global scratch) and measures per-layer time vs the identical-math per-op baseline. Throwaway `#[ignore]` probe `megakernel_multicta_mlp_probe` — not pipeline-wired. ### Measured (H200, median 200 iters, 3 repeats) | Variant | Launches | GPU time / layer-MLP | vs baseline | |---|---|---|---| | Per-op baseline (grid=N, full-device) | 4 | **0.656 ms** | 1× | | **Multi-CTA cooperative megakernel** | 1 | **0.676–0.680 ms** | **1.03× — ~3% SLOWER** | - **Recovered fraction: −3.2%** (reproducible −2.9%…−3.5%) — the megakernel is *slightly slower*. - **Occupancy** 8 blocks/SM → **1056 co-resident CTAs**; cooperative launch supported. - **`grid.sync` = 2.23 µs/barrier** (full 1056-CTA grid); a full layer pays ~6–8 → ~0.7–0.9 ms/token of pure barrier tax. - **Numerics: 0 ulp — byte-exact** (multi-CTA + grid.sync reorders nothing; no Chew gate). ### Why it doesn't win 1. **CUDA-graph replay already removes the per-launch overhead** the megakernel exists to recover (Phase A: eager 27.6 → captured 21.4 ms, ~6.1 ms already banked). 2. **The multi-CTA design must pay a `grid.sync` tax** the per-op path never pays; it cancels/exceeds the launch+round-trip savings. 3. The GEMVs are genuine **full-device weight-read work** (per-op already fans across all 132 SMs — that's why single-CTA was 926× worse). A megakernel does the *same* reads and can't accelerate them; the removed activation round-trips are already L2-resident (~80 KB). ### Verdict — **NO-GO on the whole-layer GEMV megakernel (P2)** Architecture is sound and capturable, but the measured per-layer payoff is **negative**. Do not fund the multi-week cooperative-megakernel integration + capture-safety/numerics gating. **Redirect the lever to graph-side glue node-collapse (Batty, `optimizer.rs`)** — the only decode component with recoverable overhead (Phase B: 85.6% of *glue* GPU time fusible), with no cooperative kernel, no `grid.sync` tax, no numerics reorder. Sebastian's kernel role stays limited to the already-landed fused epilogues (#867 SwiGLU-mul, #854 skip-RMSNorm). **Projected tok/s from a GEMV megakernel: ~0% (decode stays ~47 tok/s).** The earlier "~62 → ~100+ tok/s" projection assumed ~85% chain recovery; §7 shows it does not apply to the GEMVs. ### Caveats (stated, not papered over) Representative f32 int4 GEMV (not production f16 dp4a split-K) → *ratio* transfers, absolute ms doesn't; mechanism is kernel-speed-independent. Eager-timed baseline → under graph replay the megakernel's edge only worsens. MLP subset → attention adds more GEMVs + more barriers. ### One un-excluded future path A software-pipelined design overlapping **next-layer int4 weight prefetch with current compute** (Hazy-style) attacks the *GEMV time itself* (the real floor) rather than launch overhead — harder than node-collapse; scope only if glue collapse + GQA tuning are exhausted and still short. Docs: `docs/research/dense-decode-megakernel-feasibility.md` §7 (§5/§6 marked superseded). Decision drop: `.squad/decisions/inbox/sebastian-megakernel-multicta.md`. Validation: `cargo fmt --all -- --check` + clippy clean on the test target; probe passes; no production `src/` behavior change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…%, byte-exact) (#899) ## The kill-gate before staffing Batty on a multi-week optimizer.rs pass #898 killed the GEMV megakernel (NO-GO, −3.2%) and §7.4 redirected the decode lever to **graph-side glue node-collapse** (Batty, `optimizer.rs`). But that redirect had a self-referential risk: #898's own NO-GO mechanism was *"CUDA-graph replay already removes the per-launch overhead"* — and node-collapse targets that **same** overhead. The Phase B 85.6% glue recovery was measured **eager**. So before funding a multi-week `optimizer.rs` pass, this PR measures whether glue collapse recovers anything **under graph replay** (the real production decode path), not eager. New throwaway `#[ignore]` probe `glue_collapse_replay_gate_probe`: capture the ~22-op glue chain (22 graph nodes) and the fused single launch (1 node) **each into a CUDA graph**, and time **graph replay** (median ≥200 iters, CUDA events). Plus a launch-floor pair (22 trivial nodes vs 1) to isolate residual per-node cost. ### Measured (H200, median 200 iters, 4 repeats) | Path | per-op (22 nodes) | fused (1 node) | recovered | |---|---|---|---| | Eager (reference; reproduces Phase B) | 0.048–0.051 ms | 0.0077 ms | 84–85% | | **Under CUDA-graph replay** (production path) | **0.0280 ms** | **0.0069–0.0073 ms** | **74.0–75.5%** | - **~0.90 µs/node dispatch floor SURVIVES replay** (22 trivial graph nodes 0.0219 ms vs 1 node 0.0030 ms). - **Byte-exact, 0 ulp** (per-op-graph vs fused-graph). - **Projection (52 layers, 21.4 ms/token baseline):** saves ~1.08 ms/token → **46.7 → ~49.2 tok/s (+5.3%)** *(ceiling — glue nodes interleave with GEMVs; marked as projection).* ### Verdict — **GO on graph-side glue node-collapse** The #898 concern is **empirically disproven for glue**: graph replay amortizes per-node dispatch by only **~2.5×** (eager ~2.3 µs/op → ~0.9 µs/node), **not to zero**, and collapsing 22 nodes → 1 recovers 21 × ~0.9 µs ≈ 19 µs of the 28 µs per-op replay time. This is the **opposite** of the megakernel because glue ops are tiny **L2-resident, dispatch-bound** work (not irreducible full-device GEMV work), and collapse uses an ordinary fused launch — **no grid.sync tax (vs 2.23 µs/barrier in #898), no reduction reorder, no Chew gate**. | | #898 GEMV megakernel | this PR: glue node-collapse | |---|---|---| | Collapses | irreducible full-device GEMV weight-read work | tiny L2-resident dispatch overhead | | Recoverable under replay | ~none | **~0.9 µs/node floor that survives replay** | | Extra cost | grid.sync tax (2.23 µs/barrier) | **none** (ordinary fused launch) | | Verdict | **NO-GO** (−3.2%) | **GO** (+5.3% ceiling, byte-exact) | ### Costed plan for Batty (`optimizer.rs`) Collapse the elementwise+norm chain in the captured graph (target ~49 nodes/layer → a handful), reusing landed fused epilogues (#867 SwiGLU-mul, #854 skip-RMSNorm). No cooperative kernel, no grid.sync, no numerics reorder → no Chew gate. Bounded upside ~+5% decode; low risk. **Next validation:** measure collapse on the real captured decode graph to realize the ceiling. Docs: `docs/research/dense-decode-megakernel-feasibility.md` §8 (+ headline/status). Decision drop: `.squad/decisions/inbox/sebastian-glue-replay-gate.md`. Test target only, no production `src/` change; `cargo fmt --all -- --check` + clippy clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

….9% byte-exact (#899) (#900) Converts the measured GO from #899 (§8 "+5.3% ceiling" glue node-collapse GO, byte-exact under CUDA-graph replay) into a **realized, measured** number on the production Muse-Glimmer-30B native CUDA decode graph. ## What Extend `CudaSiluFusion` (`crates/onnx-runtime-ep-cuda/src/optimizer.rs`) to accept **BFloat16** activations — it was `Float16`-only, which left every SiLU/SwiGLU fusion **dormant** on Muse-Glimmer (its activation stream is bf16). With bf16 enabled, the standalone `Sigmoid(x)` + `Mul(x, sigmoid)` + `Mul(silu, up)` glue collapses through `CudaSiluFusion` → `CudaSwiGluFusion` into the tagged decomposed `Mul[_cuda_silu_mul]`, which the runtime lowers to the **already-landed** byte-exact `decomposed_silu_mul_bf16` epilogue (#867). - Deletes **104 standalone nodes** (2/layer × 52): `Sigmoid` 104→52, `Mul` 210→158; total decoder 2458→2354. - `CudaGateUpSwiGluFusion` requires an fp16 activation, so it stays dormant on bf16 → **the int4 MatMulNBits GEMVs are left untouched** (per #898). No cooperative kernel, no `grid.sync`, no reduction reorder → **no Chew gate**. ## Measured (H200, `CUDA_VISIBLE_DEVICES=0`, `ONNX_GENAI_CUDA_GRAPH=1`, `--steady --warmups 1 --runs 3 --tokens 128`, 3 interleaved A/B rounds on the same release binary) | metric | before | after | |---|---|---| | decode | 21.19 ms/token | 20.99 ms/token | | throughput | **47.20 tok/s** | **47.63 tok/s** (**+0.9%**) | | `Sigmoid` / forward | 104 | 52 | | `Mul` / forward | 210 | 158 | | decoder nodes | 2458 | 2354 (−104) | | greedy 24-tok stream | `[24, 372, 1045, 10016, …, 1740, 2885]` | **identical (byte/token-exact)** | ## Verdict: SHIP Small-but-real, byte-exact, zero-risk (activates a landed kernel the pass gate was hiding). Honest bound vs the +5.3% ceiling: only the 2 SiLU/SwiGLU-mul nodes/layer are byte-exactly collapsible here. The larger glue is blocked and documented in §8.5: - **6 norms/layer** — Gemma3 sandwich-norm residual is `x + norm(y)`; #854 `SkipSimplifiedLayerNormalization` (`norm(x+skip)`) applies across the layer seam, **but only f32/f16 skip kernels exist**, and the f32-template path bf16 would reach accumulates the RMS over the **unrounded** fp32 sum (vs standalone bf16-rounded) → not byte-exact → **flagged for Chew/Sebastian: needs a bf16 skip-RMSNorm kernel that rounds the residual sum**. - **~208 constant `gamma+1` Adds/layer** — #872 `CudaFoldConstantAdd` already MEASURED −2.8% regression; not re-attempted. - **4 reshapes/layer** — GQA head-split metadata coupled to the attention kernel, not free-standing deletions. ## Validation - `cargo fmt --all -- --check` clean; `cargo clippy -p onnx-runtime-ep-cuda --features cuda` clean. - `cargo test --release -p onnx-runtime-ep-cuda --lib --features cuda`: **366 passed / 0 failed** (incl. new `collapses_bf16_decomposed_swiglu_glue`, `leaves_fp32_decomposed_swiglu_glue_separate`). - CUDA EP cdylib `libonnx_runtime_ep_cuda_plugin.so` built; `ldd` confirms it does **not** link `libonnxruntime`. - Byte-exact greedy token stream verified before↔after. Docs: `docs/research/dense-decode-megakernel-feasibility.md` §8.5 (realized subsection). Decision drop: `.squad/decisions/inbox/batty-glue-node-collapse.md`. Refs #899. Do not self-merge. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

justinchuby merged commit 1002e36 into main Aug 13, 2026
10 of 13 checks passed

justinchuby deleted the squad/cuda-matmulnbits-gemv branch August 13, 2026 04:20

justinchuby mentioned this pull request Aug 13, 2026

docs: record MatMulNBits scale-cache win — native CUDA beats ORT (47 tok/s) #868

Merged

justinchuby mentioned this pull request Aug 13, 2026

perf(cuda-ep): bf16 decomposed SiLU/SwiGLU-mul kernels for capture-safe fusion #871

Merged

justinchuby mentioned this pull request Aug 13, 2026

perf(cuda-ep): dense megakernel P2-prototype — multi-CTA cooperative MLP measured (NO-GO for GEMV megakernel) #898

Merged

justinchuby mentioned this pull request Aug 13, 2026

perf(cuda-ep): glue node-collapse gate UNDER graph replay — GO (74–75%, byte-exact) #899

Merged

justinchuby mentioned this pull request Aug 13, 2026

perf(cuda-ep): collapse bf16 SiLU/SwiGLU-mul glue nodes — realized +0.9% byte-exact (#899) #900

Merged

Navigation Menu

Repository navigation

perf(cuda-ep): optimize MatMulNBits int4 dequant GEMV on native decode (40→47 tok/s) - #867

perf(cuda-ep): optimize MatMulNBits int4 dequant GEMV on native decode (40→47 tok/s)#867
justinchuby merged 1 commit into
mainfrom
squad/cuda-matmulnbits-gemv

justinchuby commented Aug 13, 2026

Uh oh!

Uh oh!

codecov Bot commented Aug 13, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 13, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

justinchuby commented Aug 13, 2026

Summary

Numbers (H200, CUDA_VISIBLE_DEVICES=0, ONNX_GENAI_CUDA_GRAPH=1, --pipeline --backend native --steady --warmups 1 --runs 3 --tokens 128)

Correctness / numerics

Capture-safety

Follow-ups (deferred)

Uh oh!

Uh oh!

codecov Bot commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Aug 13, 2026

🔴 Benchmark Regression Detected

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Numbers (H200, `CUDA_VISIBLE_DEVICES=0`, `ONNX_GENAI_CUDA_GRAPH=1`, `--pipeline --backend native --steady --warmups 1 --runs 3 --tokens 128`)

codecov Bot commented Aug 13, 2026 •

edited

Loading