perf(cuda-ep): collapse bf16 SiLU/SwiGLU-mul glue nodes — realized +0.9% byte-exact (#899) by justinchuby · Pull Request #900 · justinchuby/onnx-genai

justinchuby · 2026-08-13T17:58:50Z

Converts the measured GO from #899 (§8 "+5.3% ceiling" glue node-collapse GO, byte-exact under CUDA-graph replay) into a realized, measured number on the production Muse-Glimmer-30B native CUDA decode graph.

What

Extend CudaSiluFusion (crates/onnx-runtime-ep-cuda/src/optimizer.rs) to accept BFloat16 activations — it was Float16-only, which left every SiLU/SwiGLU fusion dormant on Muse-Glimmer (its activation stream is bf16). With bf16 enabled, the standalone Sigmoid(x) + Mul(x, sigmoid) + Mul(silu, up) glue collapses through CudaSiluFusion → CudaSwiGluFusion into the tagged decomposed Mul[_cuda_silu_mul], which the runtime lowers to the already-landed byte-exact decomposed_silu_mul_bf16 epilogue (#867).

Deletes 104 standalone nodes (2/layer × 52): Sigmoid 104→52, Mul 210→158; total decoder 2458→2354.
CudaGateUpSwiGluFusion requires an fp16 activation, so it stays dormant on bf16 → the int4 MatMulNBits GEMVs are left untouched (per perf(cuda-ep): dense megakernel P2-prototype — multi-CTA cooperative MLP measured (NO-GO for GEMV megakernel) #898). No cooperative kernel, no grid.sync, no reduction reorder → no Chew gate.

Measured (H200, `CUDA_VISIBLE_DEVICES=0`, `ONNX_GENAI_CUDA_GRAPH=1`, `--steady --warmups 1 --runs 3 --tokens 128`, 3 interleaved A/B rounds on the same release binary)

metric	before	after
decode	21.19 ms/token	20.99 ms/token
throughput	47.20 tok/s	47.63 tok/s (+0.9%)
`Sigmoid` / forward	104	52
`Mul` / forward	210	158
decoder nodes	2458	2354 (−104)
greedy 24-tok stream	`[24, 372, 1045, 10016, …, 1740, 2885]`	identical (byte/token-exact)

Verdict: SHIP

Small-but-real, byte-exact, zero-risk (activates a landed kernel the pass gate was hiding). Honest bound vs the +5.3% ceiling: only the 2 SiLU/SwiGLU-mul nodes/layer are byte-exactly collapsible here. The larger glue is blocked and documented in §8.5:

6 norms/layer — Gemma3 sandwich-norm residual is x + norm(y); perf(cuda-ep): fix bf16 skip-norm capture-safety flag → 0 seams (Muse-Glimmer +33%) #854 SkipSimplifiedLayerNormalization (norm(x+skip)) applies across the layer seam, but only f32/f16 skip kernels exist, and the f32-template path bf16 would reach accumulates the RMS over the unrounded fp32 sum (vs standalone bf16-rounded) → not byte-exact → flagged for Chew/Sebastian: needs a bf16 skip-RMSNorm kernel that rounds the residual sum.
~208 constant gamma+1 Adds/layer — perf(cuda-ep): fuse decode-graph nodes to cut captured node count (47->flat; cheap-node fusion is not a lever) #872 CudaFoldConstantAdd already MEASURED −2.8% regression; not re-attempted.
4 reshapes/layer — GQA head-split metadata coupled to the attention kernel, not free-standing deletions.

Validation

cargo fmt --all -- --check clean; cargo clippy -p onnx-runtime-ep-cuda --features cuda clean.
cargo test --release -p onnx-runtime-ep-cuda --lib --features cuda: 366 passed / 0 failed (incl. new collapses_bf16_decomposed_swiglu_glue, leaves_fp32_decomposed_swiglu_glue_separate).
CUDA EP cdylib libonnx_runtime_ep_cuda_plugin.so built; ldd confirms it does not link libonnxruntime.
Byte-exact greedy token stream verified before↔after.

Docs: docs/research/dense-decode-megakernel-feasibility.md §8.5 (realized subsection). Decision drop: .squad/decisions/inbox/batty-glue-node-collapse.md.

Refs #899. Do not self-merge.

….9% byte-exact (#899) Extend CudaSiluFusion to accept BFloat16 activations (was Float16-only), so the standalone Sigmoid(x) + Mul(x, sigmoid) + Mul(silu, up) glue chain on the bf16 Muse-Glimmer-30B decoder collapses through CudaSiluFusion -> CudaSwiGluFusion into the tagged decomposed Mul[_cuda_silu_mul], lowered by the already-landed byte-exact decomposed_silu_mul_bf16 epilogue (#867). Deletes 104 standalone nodes (2/layer x 52): Sigmoid 104->52, Mul 210->158. CudaGateUpSwiGluFusion requires an fp16 activation, so it stays dormant on bf16 -> the int4 GEMVs are left untouched (per #898). No cooperative kernel, no grid.sync, no reduction reorder -> no Chew gate. Measured (H200, CUDA_VISIBLE_DEVICES=0, ONNX_GENAI_CUDA_GRAPH=1, --steady --warmups 1 --runs 3 --tokens 128, 3 interleaved A/B rounds, same binary): decode 21.19 -> 20.99 ms/token, throughput 47.20 -> 47.63 tok/s (+0.9%); greedy token stream byte/token-identical before<->after. SHIP. Converts the §8 +5.3% ceiling (#899) into a realized number; §8.5 doc records why the larger glue is blocked (bf16 skip-RMSNorm kernel missing -> flagged for Chew; constant gamma+1 Add fold regressed in #872; reshapes are GQA-coupled). Tests: collapses_bf16_decomposed_swiglu_glue, leaves_fp32_decomposed_swiglu_glue_separate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

codecov · 2026-08-13T18:06:44Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.39%. Comparing base (38c9a7c) to head (d9011d6).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #900      +/-   ##
==========================================
- Coverage   78.39%   78.39%   -0.01%     
==========================================
  Files         354      354              
  Lines      148003   148003              
  Branches   148003   148003              
==========================================
- Hits       116023   116022       -1     
- Misses      27484    27485       +1     
  Partials     4496     4496

Flag	Coverage Δ
cli-ort-linux	`83.42% <ø> (ø)`
cli-ort-windows	`82.92% <ø> (-0.10%)`	⬇️
mlas	`80.87% <ø> (-0.05%)`	⬇️
offline	`78.16% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.
see 1 file with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

github-actions · 2026-08-13T18:32:39Z

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	`block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024`	41.96 µs	144.33 µs	+244.0%
🔴	`block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024`	40.64 µs	113.21 µs	+178.5%
🔴	`block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024`	515.16 µs	1.32 ms	+156.1%
🔴	`matmul/large_generic_f32_threads=8/32x1024x1024`	3.65 ms	5.37 ms	+47.2%
🔴	`matmul/medium_generic_bf16_threads=8/32x512x512`	371.89 µs	536.70 µs	+44.3%
🔴	`matmul/medium_generic_f16_threads=8/32x512x512`	28.97 µs	39.17 µs	+35.2%
🔴	`add/small_bf16_threads=1-internal/1024`	12.59 µs	16.60 µs	+31.8%
⚠️	`matmul/medium_generic_bf16_threads=1/32x512x512`	502.35 µs	617.68 µs	+23.0%
⚠️	`add/medium_f32_threads=1-internal/262144`	2.50 ms	3.07 ms	+22.7%
⚠️	`block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1`	351.23 µs	430.50 µs	+22.6%
⚠️	`matmul/medium_generic_f16_threads=1/32x512x512`	28.70 µs	34.66 µs	+20.8%
⚠️	`grammar_masking/llguidance_compute_mask/32`	73.06 µs	87.25 µs	+19.4%
⚠️	`add/small_f16_threads=1-internal/1024`	12.98 µs	15.29 µs	+17.8%
✅	`matmul/large_generic_f16_threads=8/32x1024x1024`	78.17 µs	89.84 µs	+14.9%
✅	`matmul/large_generic_bf16_threads=8/32x1024x1024`	1.60 ms	1.81 ms	+13.0%
✅	`qwen3_sampling_processors/top_k_top_p_fast`	674.03 µs	739.77 µs	+9.8%
✅	`kv_cache/alloc_dealloc_pages`	38.18 µs	41.52 µs	+8.7%
✅	`matmul/medium_generic_f32_threads=1/32x512x512`	2.15 ms	2.33 ms	+8.5%
✅	`matmul/large_generic_f32_threads=1/32x1024x1024`	8.95 ms	9.68 ms	+8.1%
✅	`matmul/medium_generic_f32_threads=8/32x512x512`	898.70 µs	969.71 µs	+7.9%
✅	`add/small_f32_threads=1-internal/1024`	202.8 ns	218.8 ns	+7.9%
✅	`matmul/large_generic_f16_threads=1/32x1024x1024`	73.96 µs	79.30 µs	+7.2%
✅	`logit_processing/seven_processor_chain_per_step`	329.53 µs	348.73 µs	+5.8%
✅	`qwen3_sampling_processors/top_k_top_p_full_sort_baseline`	5.76 ms	6.03 ms	+4.6%
✅	`add/medium_f16_threads=1-internal/262144`	2.64 ms	2.74 ms	+3.6%
✅	`block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1`	75.52 µs	77.95 µs	+3.2%
✅	`qwen3_sampling_processors/top_p_fast_after_top_k`	540.40 µs	554.45 µs	+2.6%
✅	`sampling_latency/top_p_per_token`	405.63 µs	410.39 µs	+1.2%
✅	`matmul/small_generic_f16_threads=1/1x256x256`	28.37 µs	28.49 µs	+0.4%
✅	`matmul/small_generic_bf16_threads=1/1x256x256`	28.40 µs	28.42 µs	+0.0%
✅	`matmul/small_generic_bf16_threads=8/1x256x256`	29.71 µs	29.55 µs	-0.5%
✅	`matmul/large_generic_bf16_threads=1/32x1024x1024`	1.96 ms	1.94 ms	-1.3%
✅	`matmul/small_generic_f16_threads=8/1x256x256`	28.00 µs	27.55 µs	-1.6%
✅	`add/large_f32_threads=1-internal/4194304`	43.82 ms	42.99 ms	-1.9%
✅	`gather/small_bf16_threads=1-internal/4096`	485.7 ns	464.3 ns	-4.4%
✅	`qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline`	4.68 ms	4.43 ms	-5.3%
✅	`matmul/small_generic_f32_threads=1/1x256x256`	36.02 µs	34.09 µs	-5.4%
✅	`qwen3_sampling_processors/top_k_partial_selection`	164.34 µs	152.54 µs	-7.2%
✅	`add/medium_bf16_threads=1-internal/262144`	3.08 ms	2.84 ms	-7.7%
✅	`qwen3_sampling_processors/top_k_full_sort_baseline`	2.32 ms	2.12 ms	-8.7%
✅	`sampling_latency/min_p_per_token`	228.95 µs	207.54 µs	-9.4%
✅	`sampling_latency/top_k_per_token`	55.21 µs	49.66 µs	-10.1%
✅	`matmul/small_generic_f32_threads=8/1x256x256`	37.27 µs	32.55 µs	-12.7%
✅	`tokenization/encode_tokens_per_second`	466.50 µs	401.51 µs	-13.9%
✅	`gather/small_f16_threads=1-internal/4096`	552.4 ns	474.4 ns	-14.1%
🟢	`gather/medium_f32_threads=1-internal/32768`	4.26 µs	3.61 µs	-15.4%
🟢	`reduce_mean/large_f32_threads=1-internal/262144`	1.22 ms	1.01 ms	-17.1%
🟢	`add/large_f16_threads=1-internal/4194304`	50.55 ms	39.86 ms	-21.1%
🟢	`gather/small_f32_threads=1-internal/4096`	893.2 ns	701.9 ns	-21.4%
🟢	`reduce_mean/small_f32_threads=1-internal/4096`	20.60 µs	15.58 µs	-24.4%
🟢	`add/large_bf16_threads=1-internal/4194304`	59.97 ms	44.71 ms	-25.4%
🟢	`gather/large_f16_threads=1-internal/131072`	13.00 µs	9.51 µs	-26.8%
🟢	`gather/medium_bf16_threads=1-internal/32768`	3.14 µs	2.23 µs	-28.9%
🟢	`gather/large_bf16_threads=1-internal/131072`	15.34 µs	10.80 µs	-29.6%
🟢	`gather/large_f32_threads=1-internal/131072`	38.65 µs	26.12 µs	-32.4%
🟢	`gather/medium_f16_threads=1-internal/32768`	3.45 µs	2.27 µs	-34.2%
🟢	`sampling_latency/greedy_per_token`	4.86 µs	3.16 µs	-35.1%
🟢	`reduce_mean/medium_f32_threads=1-internal/65536`	429.84 µs	254.07 µs	-40.9%
🟢	`tokenization/decode_tokens_per_second`	12.58 ms	6.69 ms	-46.9%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 4.68 3.69 5.28 }

What this cannot catch

Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
Sub-threshold regressions that compound over multiple PRs
Performance changes that only manifest under GPU execution
Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)

…ld opt-in (no-ship default) (#903) ## Summary Closes the §8.5 blocker from #900: adds a **byte-exact bf16 fused skip-RMSNorm kernel** and an optimizer fold that collapses the `Add → SimplifiedLayerNormalization` residual seam. Outcome: **SHIP the kernel** (proven 0-ulp), **NO-SHIP the fold as default** (measured −1.5% regression under graph replay) — the fold is retained opt-in behind an env flag, so the default binary is unchanged. Refs #900. **Do NOT self-merge — Chew gates numerics.** ## What 1. **`skip_rmsnorm_bf16` NVRTC kernel** (`kernels/normalization.rs`): `sum = __float2bfloat16_rn(f32(residual)+f32(x))` (bit-for-bit a standalone bf16 `Add`), stores the bf16-rounded sum as the next layer's residual, then the **identical** `rmsnorm_bf16` block-tree reduction (fp32 accumulate over the rounded sum, `NORM_BLOCK=256`). Guarded native dispatch with `run_bf16_via_f32` fallback for non-dense/bias/header-unavailable (Rule 11 portability). 2. **`CudaSkipRmsNormFusion`** optimizer pass: collapses the seam into one `com.microsoft::SkipSimplifiedLayerNormalization`, deleting the standalone `Add`+norm. bf16 (or f32-gamma-over-bf16) only; fp16 left to `CudaSkipRmsNormMatMulFusion`. **Opt-in** via `ONNX_GENAI_CUDA_ENABLE_SKIP_RMSNORM_FUSION` (default **OFF**). ## Numeric fidelity (Chew gate): BYTE-EXACT — 0-ulp - GPU unit tests `bf16_native_skip_rmsnorm_is_byte_exact_with_{bf16,f32}_gamma` pass (bit-identical vs standalone `Add(bf16)`→`rmsnorm_bf16` at H=6656). - Real-model greedy stream fold OFF vs ON: **bit-identical** (128/128 tokens). No reduction reordered. ## Perf A/B (H200, `ONNX_GENAI_CUDA_GRAPH=1`, `--steady --warmups 1 --runs 3 --tokens 128`, interleaved) | metric | fold OFF (default) | fold ON | |---|---|---| | `SkipSimplifiedLayerNormalization` nodes | 4 | **104** (2/layer × 52) | | decode | 20.93 ms/token | 21.25 ms/token | | throughput | **47.77 tok/s** | **47.06 tok/s (−1.5%)** | | greedy tokens | reference | **byte-identical** | **Mechanism (honest):** eager per-op timer shows the fused glue *is* faster (Add+norm+skip 355.6 → 349.0 ms — real launch saving), but **graph replay already amortizes launches** (~0.9 µs/node floor). At M=1 the RMS reduction is single-CTA; folding the residual add into it **serializes** work the standalone `Add` had spread across all 132 SMs. Same structural reason the multi-CTA GEMV megakernel was NO-GO (§7/#898) and glue collapse paid only +0.9% (§8.5/#900): decode is at its launch-amortized latency floor. ## Verdict - **Kernel: SHIP** — 0-ulp, portability-gated, no production change. Prerequisite for the only winning bf16 path: folding the norm into the neighbouring **multi-CTA int4 GEMV** prologue/epilogue (bf16 analogue of `CudaSkipRmsNormMatMulFusion`) — a larger GEMV job, not this fold. - **Fold: NO-SHIP as default** — opt-in only; default binary unchanged (47.77 tok/s == baseline). ## Validation - `cargo fmt --all -- --check` clean; clippy clean on ep-cuda (pre-existing optimizer.rs:3347 + standard_attention.rs warnings are not mine). - 61 optimizer + 20 normalization (incl. 2 new byte-exact) unit tests pass. - CUDA EP cdylib `ldd`-confirmed **not** linking `libonnxruntime`. - Doc `docs/research/dense-decode-megakernel-feasibility.md` §8.6 + decision drop `.squad/decisions/inbox/sebastian-bf16-skip-rmsnorm.md`. 🤖 Generated with Copilot. Co-authored-by: Copilot. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Merge 3 inbox drops into one "Decode latency-floor: node-collapse arc" section in decisions.md. Batch-1 native CUDA int4 decode is confirmed at its launch-amortized latency floor from three independent directions: megakernel NO-GO (#898), glue-collapse +0.9% ceiling (#899/#900), skip-RMSNorm fold -1.5% regression (#903). - #899 (Sebastian): glue node-collapse GO under graph replay (74-75%). - #900 (Batty): bf16 SiLU/SwiGLU-mul glue collapse SHIP (+0.9%, byte-exact, SiLU-fusion bf16 portability fix under Rule 11). - #903 (Sebastian): bf16 skip-RMSNorm KERNEL byte-exact SHIP; standalone FOLD -1.5% NO-SHIP (opt-in behind ONNX_GENAI_CUDA_ENABLE_SKIP_RMSNORM_FUSION, default OFF). Size gate: archived 2026-08-12 VMM/offload + #762 durable-lessons narratives to decisions-archive/2026-08.md to keep decisions.md under 50 KB (49,833 B). Appended cross-agent history for sebastian + batty; delete 3 processed drops. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

justinchuby merged commit e9c9fc4 into main Aug 13, 2026
12 of 13 checks passed

justinchuby deleted the squad/glue-node-collapse branch August 13, 2026 18:00

justinchuby mentioned this pull request Aug 13, 2026

perf(cuda): byte-exact bf16 skip-RMSNorm kernel (ship); standalone fold opt-in (no-ship default) #903

Merged

Navigation Menu

Repository navigation

perf(cuda-ep): collapse bf16 SiLU/SwiGLU-mul glue nodes — realized +0.9% byte-exact (#899) - #900

perf(cuda-ep): collapse bf16 SiLU/SwiGLU-mul glue nodes — realized +0.9% byte-exact (#899)#900
justinchuby merged 1 commit into
mainfrom
squad/glue-node-collapse

justinchuby commented Aug 13, 2026

Uh oh!

Uh oh!

codecov Bot commented Aug 13, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 13, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

justinchuby commented Aug 13, 2026

What

Measured (H200, CUDA_VISIBLE_DEVICES=0, ONNX_GENAI_CUDA_GRAPH=1, --steady --warmups 1 --runs 3 --tokens 128, 3 interleaved A/B rounds on the same release binary)

Verdict: SHIP

Validation

Uh oh!

Uh oh!

codecov Bot commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Aug 13, 2026

🔴 Benchmark Regression Detected

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Measured (H200, `CUDA_VISIBLE_DEVICES=0`, `ONNX_GENAI_CUDA_GRAPH=1`, `--steady --warmups 1 --runs 3 --tokens 128`, 3 interleaved A/B rounds on the same release binary)

codecov Bot commented Aug 13, 2026 •

edited

Loading