Fuse SSM/linear-attention reduction: drop fp32 cuDNN ReduceSum round-trip on native decode by justinchuby · Pull Request #1486 · justinchuby/onnx-genai

justinchuby · 2026-08-19T16:54:10Z

Summary

Route f16 ReduceSum/ReduceMean to the existing single-kernel NVRTC block reduction (fp16 IO, f32 register accumulation) instead of cuDNN. This eliminates the self-inflicted fp32 round-trip on the hybrid decode hot path.

The problem (profiling finding)

A faithful captured-graph node trace of qwen3.5-0.8b-hybrid (fp16io, keep_io_types=False) decode — agent Deckard, .squad/decisions/inbox/deckard-fair-hybrid-gap.md — localized native's #1 fixable inefficiency: the SSM / linear-attention path does its reduction as a cuDNN fp32 ReduceSum bracketed by fp16↔fp32 op_tensor cast wrappers, ~20.7% of decode GPU-kernel time (LinearAttention 9.9% + cuDNN ReduceSum 8.1% + op_tensor casts 2.7%), firing 18×/step.

cudnnReduceTensor rejects a half reduceTensorCompType and forces CUDNN_DATA_FLOAT, so an f16 reduce becomes a three-kernel round-trip (fp16→fp32 cast, fp32 ReduceSum, fp32→fp16 cast) through a full-size fp32 temporary.

The fix

The typed NVRTC block reduction already implements the exact ONNX accumulate-in-f32-then-cast rule in a single capture-safe kernel — no fp32 temporary, half the memory traffic. It already serves bf16 and every extended f16 reduction, so this is a general routing change (all f16 sum/mean, any shape/axis/dim), not a qwen-specific special case. f32 stays on cuDNN (native f32 comp type, byte-identical).

One-line behavioral change: the cuDNN reduce gate goes from Float32 | Float16 → Float32.

Before / after — native decode (H200, `--test-threads=1`, matched A/B on the same build)

	ms/tok	tok/s	captures	fallbacks
before (cuDNN f16 reduce)	4.664	214.41	4	0
after (fused NVRTC f16 reduce)	4.318	231.61	4	0

Reclaim: 0.346 ms/tok (+17.2 tok/s, ~7.4%) — within Deckard's predicted 0.3–0.4 ms.

Parity

generated_token_ids byte-identical before vs after (greedy decode).
GPU tests pass, tolerances unchanged: reduce_comptype_fp16_gpu, reduce_capture_gpu, nvrtc_reduce_capture_gpu, linear_attention_gpu.

Capture preserved

cuda_graph: enabled=true captures=4 replays=504 fallbacks=0 — no eager fallback; the fused f16 reduce folds into a single captured segment (verified by reduce_capture_gpu's single-segment assertion).

Generality

The NVRTC block reduction is shape/axis/dim/head agnostic (base/delta offset walk, one block per output element). No dims hardcoded; no model-shape assumptions. bf16 already used this path; f32 remains on cuDNN.

Refs profiling finding: .squad/decisions/inbox/deckard-fair-hybrid-gap.md.

…und-trip Route f16 ReduceSum/ReduceMean to the single-kernel NVRTC block reduction (fp16 IO, f32 register accumulation) instead of cuDNN. cuDNN rejects a half reduceTensorCompType and forces CUDNN_DATA_FLOAT, implementing an f16 reduce as an fp16->fp32 op_tensor cast + fp32 ReduceSum + fp32->fp16 cast: a three-kernel round-trip through a full-size fp32 temporary. A captured-graph node trace of qwen3.5-0.8b-hybrid decode (Deckard) localized that fragmentation at ~20.7% of decode GPU-kernel time on the SSM/linear-attention path (cudnn ReduceSum 8.1% + fp16<->fp32 op_tensor casts 2.7%, firing 18x/step). The NVRTC block reduction already does the exact ONNX accumulate-in-f32-then-cast rule in one capture-safe kernel with no fp32 temporary and half the memory traffic; it already serves bf16 and every extended f16 reduction, so this is a general routing change, not a per-model special case. f32 stays on cuDNN (native f32 comp type, byte-identical). Native qwen3.5-0.8b-hybrid fp16io decode (H200, --test-threads=1, matched A/B): before 4.664 ms/tok (214.41 tok/s) -> after 4.318 ms/tok (231.61 tok/s) reclaim 0.346 ms/tok (+17.2 tok/s). generated_token_ids byte-identical. cuda_graph captures=4 fallbacks=0 (capture preserved). Parity: reduce_comptype_fp16_gpu, reduce_capture_gpu, nvrtc_reduce_capture_gpu, linear_attention_gpu all pass (tolerances unchanged). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…kernel The SSM/linear-attention normalize-scale is exported as the primitive chain `Add(sumsq, eps) -> Sqrt -> Reciprocal -> Mul`, so `Sqrt` and `Reciprocal` run as two standalone per-layer kernels. A captured-graph node trace of qwen3.5-0.8b-hybrid decode measured `sqrt_f16` (1.2%) + `reciprocal_f16` (1.2%) = ~2.4% of decode GPU-kernel time, firing 2x per SSM layer (36/step). Add `CudaRsqrtFusion`: for every `Sqrt` whose single output is consumed only by one `Reciprocal` (and isn't a graph output), rewire the `Reciprocal` to read the pre-`Sqrt` value, tag it `_cuda_rsqrt`, and delete the `Sqrt`. The node stays a standard-domain `Reciprocal` (shape inference / registry dispatch unchanged); the CUDA unary-math factory reads the marker and launches a fused `rsqrt_{dtype}` kernel. The kernel reproduces the exact two-kernel rounding (Sqrt rounds `sqrtf(x)` to the storage dtype, then Reciprocal reads it back), so output is bit-identical to the unfused pair. General over f32/f16/bf16 and any shape; single-consumer/no-escape guard preserves semantics. Native qwen3.5-0.8b-hybrid fp16io decode (H200, --test-threads=1), incremental on top of the SSM-reduce fusion: before 4.318 ms/tok (231.61 tok/s) -> after 4.282 ms/tok (233.56 tok/s). generated_token_ids byte-identical. cuda_graph captures=4 fallbacks=0. nsys: sqrt_f16 + reciprocal_f16 both gone, replaced by one rsqrt_f16 (2.4% -> 1.3% of GPU-kernel time). Tests: optimizer unit tests (fuse + single-consumer guard) and a GPU byte-identity parity test (fused rsqrt == Reciprocal(Sqrt(x)) for f16/bf16). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

justinchuby · 2026-08-19T17:30:56Z

Follow-up (2nd commit): fuse SSM `Reciprocal(Sqrt(x))` → single `rsqrt` kernel

While warm on this decode path, profiled the remaining "glue tail" (nsys --cuda-graph-trace=node, same fp16io export). Highest-value clean fusion was the SSM normalize-scale glue Add(sumsq,eps)→Sqrt→Reciprocal→Mul: sqrt_f16 (1.2%) + reciprocal_f16 (1.2%) = ~2.4% of decode GPU-kernel time, firing 2×/SSM-layer.

CudaRsqrtFusion: retags a Reciprocal whose input is a single-consumer Sqrt with _cuda_rsqrt, rewires it to the pre-Sqrt value, deletes the Sqrt. The CUDA unary-math factory then launches a fused rsqrt_{dtype} kernel that reproduces the exact two-kernel rounding (Sqrt rounds sqrtf(x) to the storage dtype, Reciprocal reads it back) → bit-identical output. General over f32/f16/bf16 + any shape; single-consumer/no-escape guard preserves semantics; node stays standard-domain Reciprocal so a non-CUDA fallback (pre-pass graph restored) is unaffected.

Evidence (H200, matched, incremental on top of the reduce fusion)

	ms/tok	tok/s	captures	fallbacks
after reduce-fusion	4.318	231.61	4	0
+ rsqrt fusion	4.282	233.56	4	0

generated_token_ids byte-identical to baseline (greedy).
nsys: sqrt_f16 and reciprocal_f16 both gone, replaced by one rsqrt_f16 (2.4% → 1.3% of GPU-kernel time — the pair collapsed).
Tests: optimizer unit tests (fuse + single-consumer guard) + GPU byte-identity parity test (fused_rsqrt_is_byte_identical_to_sqrt_then_reciprocal, f16/bf16).

Combined result of this PR (both fusions)

Native qwen3.5-0.8b-hybrid fp16io decode: 4.664 → 4.282 ms/tok (214.4 → 233.6 tok/s, +8.9%), byte-identical greedy tokens, capture preserved.

Note: pre-existing unrelated failure binary_fixed_decode_shape_captures_replays_and_gates_other_paths (an Add binary-kernel capture-compat assertion) fails identically on the parent commit — not introduced here.

…verdict (#1489) Coordinator campaign-brain update (doc-only, .squad/identity/now.md). - Bank the honest fp16 fair-test reversal: qwen3.5 fp32 'moats' were export artifacts; fp16io same-graph is ~2.1x ORT-ahead (magnitude flagged unverified pending a real ORT captured harness). - Record PR #1486 native decode fusions (+8.9% qwen3.5-0.8b-hybrid, byte-identical, capture preserved). - Record #1474 memory-stack verification verdict (6 deterministic async-deferred-release GPU test failures; no decode regression). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ceSum off cuDNN to NVRTC block reduction (#1495) ## Summary Fuses the highest-ROI sub-lever of the native gated-delta **LinearAttention decode** path identified by Deckard's fair-gap kernel attribution: the **fp32 cuDNN `ReduceSum`**. The gated-delta linear-attention decode runs a per-layer q/k L2-norm SumSq as an f32 `ReduceSum` over `d_k` (16 outputs × 128 elements, **36×/decode step**). `ReduceKernel` previously sent *all* f32 sum/mean through `cudnnReduceTensor`, whose generic-reduce per-launch overhead dominates at that tiny decode shape — making it the **single largest decode op** (~11–13% of forward op time). This routes f32 sum/mean to the existing typed **NVRTC block reduction** (one capture-safe kernel, f32 register accumulation) whenever the reduction is well-parallelised — `out_count >= multiprocessor_count` **or** `reduce_count <= REDUCE_BLOCK (256)` — and keeps cuDNN only for the low-parallelism *"few outputs, huge group"* regime (e.g. global reduce-all) where a single serialising block would lose to cuDNN's multi-block reduce. It's the **fp32 analogue of the f16/bf16 routing from #1486**. ## Generality (RULES.md) The routing decision is a **general parallelism property** (SM count + per-output group size) — no head-dim, layer-count or qwen3.5 shape is hardcoded. Works for any gated-delta linear-attention config and any reduce shape/axis/rank. ## Results — qwen3.5-0.8b-hybrid **fp16io** export (H200, GPU 0, ORT 1.28 CUDA build) | | Baseline (cuDNN f32 reduce) | This PR | Δ | |---|---|---|---| | Decode throughput | **200.95 tok/s** | **209.71 tok/s** | **+4.36%** | | Decode latency | 4.976 ms/tok | 4.768 ms/tok | −0.208 ms | - Decode `generated_token_ids`: **byte-identical** before/after. - Golden lock `qwen35_0_8b_text_decode_lock`: **PASS** (byte-identical greedy stream). - Matches Deckard's ReduceSum-only prediction (~252 µs recovered ≈ +4.4%). `ONNX_GENAI_PROFILE_OPS` steady-state: `ReduceSum` share dropped from ~11–13% → ~7–10% of forward op dispatch. ## Validation - `qwen35_0_8b_text_decode_lock` (`--ignored`): **PASS** — byte-identical greedy tokens. - `reduce_capture_gpu`, `nvrtc_reduce_capture_gpu`, `reduce_comptype_fp16_gpu` (`--features cuda,gpu-tests`): **all pass**. - Added two **low-parallelism f32 capture tests** so the retained cuDNN path keeps coverage after the well-parallelised shapes moved to NVRTC. - Capture-safe: reuses the block-reduction path that already gates its trailing `synchronize()` on `!capturing` and marks the call capture-safe (folds into one graph segment). ## Honest ceiling — deferred sub-levers Deckard's #2 (fold the transpose/split/concat layout shuffle into kernel addressing) and #3 (fuse the recip/sqrt/sigmoid/softplus/mul gating chain into the `LinearAttention` epilogue) require a **graph-level fusion pass** rewriting the surrounding ONNX nodes into the kernel — a large change to graph optimisation + the kernel signature, with real byte-identity/generality risk. Out of clean surgical scope here. The ReduceSum lever alone delivered its full predicted ROI; Transpose (~2–3%), Sigmoid (~3%), Split (~2.4%) remain as the next graph-fusion targets. Working as Gaff (CUDA-kernel / SSM-fusion specialist). ⚠️ CI intentionally not awaited (async). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

… decay Softplus) into the decode kernel (#3) (#1496) ## Summary **Sub-lever #3** of the native-vs-ORT gated-delta `LinearAttention` decode gap (qwen3.5-0.8b hybrid, H200). Follows #1495 (fp32 ReduceSum → NVRTC block reduction, +4.36%) and #1486. The exported per-layer decode graph feeds the `com.microsoft::LinearAttention` kernel through standalone elementwise gate chains that ORT runs fused but native ran **decomposed**: - **beta = `Sigmoid(x)`** — delta-rule mixing gate - **decay = `exp(neg_exp_A · Softplus(a + dt_bias))`** — per-head decay gate (exported chain `Add(a, dt_bias) → Softplus → Mul(neg_exp_A, ·) → [Cast]`) New structural graph-fusion pass **`CudaLinearAttentionGatingFusion`** folds both chains into the kernel: rewires the beta slot onto the pre-`Sigmoid` value, the decay slot onto the pre-`Add` activation `a`, appends `dt_bias`/`neg_exp_A` as trailing kernel inputs, and marks two private attrs the CUDA factory consumes. Drops ~5 tiny elementwise nodes per layer (18 layers) from the captured decode graph. ## Byte-identity (non-negotiable) The kernel reproduces each folded op's device function bit-for-bit (`la_sigmoid`/`la_softplus` mirror `op_sigmoid`/`op_softplus`) **and** rounds every intermediate through the storage dtype (`round_store<T>`) exactly where the standalone `Add`/`Softplus`/`Mul`/`Sigmoid` kernels round — a no-op for the fp32 text export, an f16 round at each op boundary for the fp16-I/O hybrid. The fold is refused unless every folded operand shares the kernel's I/O dtype, so a dtype-changing `Cast` is never silently skipped. ## Generality (rules.md) Match is purely structural — op type, single-consumer / no-escape topology, which `Mul`/`Add` operand is a graph initializer, optional identity `Cast` tail. **No head-dim, layer-count, or qwen3.5 shape is baked in.** Fires for any head count / layer count; per-head and per-key-dim decay layouts both handled with element-count guards. Any gate whose chain escapes to another consumer is left exactly as exported. ## Capture-safety No host syncs, no per-step alloc, static launch config; the kernel just does more arithmetic per thread. Private attrs are compile-time constants (NVRTC key bumped `linear_attention_v1` → `v2`). ## Opt-out `ONNX_GENAI_CUDA_DISABLE_LINATTN_GATING_FUSION=1` restores the exact exported (unfused) chains. ## Verification - **Golden lock** (qwen3.5-0.8b text, fp32, native CUDA whole-graph capture): **PASS** — byte-identical, all 16 tokens. - **Pass unit tests** (3 new): full fold (both gates, ±trailing cast), beta-escape guard, env opt-out — all pass. - **Decode tok/s** (qwen3.5-0.8b fp16-I/O, H200, two-length method, best of 3 reps): | | rep1 | rep2 | rep3 | avg | |---|---|---|---|---| | fused (enabled) | 237.4 | 238.3 | 236.6 | **~237.4** | | unfused (disabled) | 232.7 | 229.3 | 226.3 | ~229.4 | **+3.4%** steady-state decode throughput. Honest, comparable to #1495's +4.36%. ## Sub-lever #2 (layout / L2-norm addressing) — DEFERRED, honest verdict **Not landed.** Folding the q/k L2-norm + transpose/split/concat layout shuffle into kernel addressing would require (1) reproducing ORT `ReduceSum`'s exact summation order (the one matched in #1495) *inside* the per-thread LA loop, redundantly across d_v threads — high byte-divergence risk against the golden lock; and (2) strided/permuted addressing in a kernel that currently declares `supports_strided_input = false` — a large tensor-plumbing change with a much bigger blast radius than #3. Neither can be guaranteed byte-identical + capture-safe within clean scope, so per honest-ceiling it is deferred rather than forced. A future pass can tackle it in isolation with its own dedicated byte lock. ## Files - `crates/onnx-runtime-ep-cuda/src/kernels/linear_attention.rs` - `crates/onnx-runtime-ep-cuda/src/optimizer.rs` _CI not awaited._ Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

#1519) Consolidation-only state PR (no code). Persists the campaign brain + decisions log to main after PR #1516 (int8 GEMV split-K, the 6th clean token-identical win) merged. ## What's in here - **`.squad/decisions.md`** — merges Deckard's corrected captured-replay attribution + Gaff's #1516 GEMV drop (22,963 → 25,954 bytes, under gate). - **`.squad/identity/now.md`** — new top verdict block: 6-win arc (#1486/#1495/#1496/#1501/#1503/#1516), fair gap 1.24× → **~1.057× ORT-ahead**, data-shuffle retired as a dead lever, honest ceiling ~260–270 tok/s reached → kernel-chasing stopped. - **`.squad/agents/{gaff,deckard}/history.md`** — appended. - **Removes** `.squad/decisions/inbox/gaff-gemv-bias-fusion.md` — it rode onto main via #1516's squash (tracked-inbox anti-pattern); already consolidated into decisions.md. ## Campaign standing Native qwen3.5-0.8b fp16io decode: **~263.8 tok/s** vs ORT eager ~279–290 = **~1.057× ORT-ahead** (from 1.24× at session start). Residual is a batch=1 launch/latency floor, not a buildable kernel lever. No manufactured levers remain. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-08-20T01:25:55Z

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	`block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024`	44.88 µs	125.76 µs	+180.2%
🔴	`block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1`	54.98 µs	109.93 µs	+100.0%
🔴	`block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024`	52.25 µs	92.17 µs	+76.4%
🔴	`matmul/large_generic_bf16_threads=8/32x1024x1024`	1.33 ms	2.32 ms	+75.2%
🔴	`matmul/large_generic_f32_threads=8/32x1024x1024`	3.68 ms	6.16 ms	+67.2%
🔴	`matmul/medium_generic_f32_threads=8/32x512x512`	1.02 ms	1.55 ms	+52.3%
🔴	`qwen3_sampling_processors/top_p_fast_after_top_k`	485.28 µs	738.51 µs	+52.2%
🔴	`block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1`	344.54 µs	497.15 µs	+44.3%
🔴	`tokenization/decode_tokens_per_second`	5.71 ms	8.08 ms	+41.6%
🔴	`matmul/large_generic_f16_threads=8/32x1024x1024`	78.12 µs	101.90 µs	+30.4%
⚠️	`qwen3_sampling_processors/top_k_partial_selection`	135.04 µs	175.14 µs	+29.7%
⚠️	`tokenization/encode_tokens_per_second`	354.50 µs	455.87 µs	+28.6%
⚠️	`qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline`	3.29 ms	4.22 ms	+28.3%
⚠️	`qwen3_sampling_processors/top_k_top_p_full_sort_baseline`	5.71 ms	7.31 ms	+28.1%
⚠️	`sampling_latency/greedy_per_token`	2.98 µs	3.79 µs	+27.1%
⚠️	`sampling_latency/top_p_per_token`	357.92 µs	445.34 µs	+24.4%
⚠️	`sampling_latency/min_p_per_token`	193.16 µs	238.57 µs	+23.5%
⚠️	`gather/small_f32_threads=1-internal/4096`	612.3 ns	743.1 ns	+21.4%
⚠️	`matmul/small_generic_f32_threads=8/1x256x256`	43.93 µs	51.92 µs	+18.2%
⚠️	`matmul/large_generic_f16_threads=1/32x1024x1024`	73.23 µs	85.11 µs	+16.2%
⚠️	`sampling_latency/top_k_per_token`	51.74 µs	59.53 µs	+15.1%
✅	`block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024`	529.63 µs	594.66 µs	+12.3%
✅	`matmul/small_generic_f16_threads=1/1x256x256`	31.86 µs	35.57 µs	+11.7%
✅	`reduce_mean/small_f32_threads=1-internal/4096`	13.91 µs	15.43 µs	+10.9%
✅	`qwen3_sampling_processors/top_k_full_sort_baseline`	2.23 ms	2.47 ms	+10.9%
✅	`matmul/large_generic_f32_threads=1/32x1024x1024`	8.86 ms	9.81 ms	+10.7%
✅	`matmul/small_generic_bf16_threads=1/1x256x256`	31.17 µs	34.27 µs	+10.0%
✅	`gather/small_f16_threads=1-internal/4096`	442.0 ns	484.8 ns	+9.7%
✅	`matmul/medium_generic_f16_threads=1/32x512x512`	31.07 µs	34.04 µs	+9.6%
✅	`reduce_mean/large_f32_threads=1-internal/262144`	910.29 µs	994.55 µs	+9.3%
✅	`add/large_f16_threads=1-internal/4194304`	1.75 ms	1.90 ms	+9.0%
✅	`matmul/large_generic_bf16_threads=1/32x1024x1024`	1.91 ms	2.08 ms	+8.7%
✅	`gather/small_bf16_threads=1-internal/4096`	448.5 ns	483.8 ns	+7.9%
✅	`gather/medium_f32_threads=1-internal/32768`	3.73 µs	4.02 µs	+7.7%
✅	`gather/large_f32_threads=1-internal/131072`	37.65 µs	40.55 µs	+7.7%
✅	`gather/medium_f16_threads=1-internal/32768`	2.30 µs	2.46 µs	+7.0%
✅	`matmul/small_generic_f32_threads=1/1x256x256`	40.61 µs	43.24 µs	+6.5%
✅	`matmul/small_generic_f16_threads=8/1x256x256`	34.18 µs	36.35 µs	+6.4%
✅	`reduce_mean/medium_f32_threads=1-internal/65536`	239.42 µs	254.65 µs	+6.4%
✅	`matmul/medium_generic_f16_threads=8/32x512x512`	37.50 µs	39.61 µs	+5.6%
✅	`matmul/small_generic_bf16_threads=8/1x256x256`	31.28 µs	32.62 µs	+4.3%
✅	`qwen3_sampling_processors/top_k_top_p_fast`	731.64 µs	756.51 µs	+3.4%
✅	`add/small_bf16_threads=1-internal/1024`	461.3 ns	475.7 ns	+3.1%
✅	`grammar_masking/llguidance_compute_mask/32`	79.56 µs	81.96 µs	+3.0%
✅	`add/large_f32_threads=1-internal/4194304`	689.15 µs	707.57 µs	+2.7%
✅	`add/large_bf16_threads=1-internal/4194304`	1.71 ms	1.74 ms	+1.5%
✅	`add/medium_bf16_threads=1-internal/262144`	110.67 µs	112.09 µs	+1.3%
✅	`logit_processing/seven_processor_chain_per_step`	343.17 µs	344.39 µs	+0.4%
✅	`add/medium_f32_threads=1-internal/262144`	29.52 µs	28.88 µs	-2.2%
✅	`matmul/medium_generic_bf16_threads=8/32x512x512`	404.30 µs	390.49 µs	-3.4%
✅	`add/small_f16_threads=1-internal/1024`	443.7 ns	426.5 ns	-3.9%
✅	`matmul/medium_generic_bf16_threads=1/32x512x512`	552.13 µs	527.14 µs	-4.5%
✅	`gather/medium_bf16_threads=1-internal/32768`	2.56 µs	2.42 µs	-5.2%
✅	`matmul/medium_generic_f32_threads=1/32x512x512`	2.43 ms	2.27 ms	-6.8%
✅	`kv_cache/alloc_dealloc_pages`	48.73 µs	45.40 µs	-6.8%
✅	`add/small_f32_threads=1-internal/1024`	209.9 ns	188.3 ns	-10.3%
✅	`add/medium_f16_threads=1-internal/262144`	126.16 µs	112.71 µs	-10.7%
✅	`gather/large_bf16_threads=1-internal/131072`	17.51 µs	15.20 µs	-13.1%
🟢	`gather/large_f16_threads=1-internal/131072`	20.57 µs	15.66 µs	-23.9%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 5.79 5.02 6.19 }

What this cannot catch

Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
Sub-threshold regressions that compound over multiple PRs
Performance changes that only manifest under GPU execution
Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)

codecov · 2026-08-20T02:34:30Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.12%. Comparing base (4a9f4ec) to head (b12824a).
⚠️ Report is 68 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #1486       +/-   ##
===========================================
- Coverage   82.10%   80.12%    -1.99%     
===========================================
  Files          12      376      +364     
  Lines        5471   164127   +158656     
  Branches     5471   164127   +158656     
===========================================
+ Hits         4492   131506   +127014     
- Misses        780    27794    +27014     
- Partials      199     4827     +4628

Flag	Coverage Δ
cli-ort-linux	`82.60% <ø> (?)`
cli-ort-windows	`82.19% <ø> (+0.09%)`	⬆️
offline	`80.03% <ø> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.
see 367 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

justinchuby and others added 2 commits August 19, 2026 16:53

justinchuby mentioned this pull request Aug 19, 2026

now.md: bank fp16-fairness reversal + PR #1486 native fusions + #1474 verdict #1489

Merged

justinchuby merged commit 1050ebe into main Aug 19, 2026
4 checks passed

justinchuby deleted the squad/ssm-reduce-fuse branch August 19, 2026 17:42

justinchuby mentioned this pull request Aug 19, 2026

cuda_ep: fuse gated-delta linear-attn decode — route f32 L2-norm ReduceSum off cuDNN to NVRTC block reduction #1495

Merged

justinchuby mentioned this pull request Aug 19, 2026

cuda_ep: fuse gated-delta LinearAttention gate chains (beta Sigmoid + decay Softplus) into the decode kernel (#3) #1496

Merged

justinchuby mentioned this pull request Aug 20, 2026

chore(squad): consolidate GEMV 6th-win + honest-ceiling campaign state #1519

Merged

Navigation Menu

Repository navigation

Fuse SSM/linear-attention reduction: drop fp32 cuDNN ReduceSum round-trip on native decode - #1486

Fuse SSM/linear-attention reduction: drop fp32 cuDNN ReduceSum round-trip on native decode#1486
justinchuby merged 2 commits into
mainfrom
squad/ssm-reduce-fuse

justinchuby commented Aug 19, 2026

Uh oh!

justinchuby commented Aug 19, 2026

Uh oh!

Uh oh!

github-actions Bot commented Aug 20, 2026

Uh oh!

codecov Bot commented Aug 20, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

justinchuby commented Aug 19, 2026

Summary

The problem (profiling finding)

The fix

Before / after — native decode (H200, --test-threads=1, matched A/B on the same build)

Parity

Capture preserved

Generality

Uh oh!

justinchuby commented Aug 19, 2026

Follow-up (2nd commit): fuse SSM Reciprocal(Sqrt(x)) → single rsqrt kernel

Evidence (H200, matched, incremental on top of the reduce fusion)

Combined result of this PR (both fusions)

Uh oh!

Uh oh!

github-actions Bot commented Aug 20, 2026

🔴 Benchmark Regression Detected

Uh oh!

codecov Bot commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Before / after — native decode (H200, `--test-threads=1`, matched A/B on the same build)

Follow-up (2nd commit): fuse SSM `Reciprocal(Sqrt(x))` → single `rsqrt` kernel

codecov Bot commented Aug 20, 2026 •

edited

Loading