perf(cuda): gate int4 f16 GEMV pipe vs plain by launch occupancy

[justinchuby]

Lever: occupancy-gated pipe-vs-plain int4 MatMulNBits GEMV

The block-32 scales-fp16 int4 MatMulNBits decode GEMV has two occupancy regimes. The prefetch-pipelined (pipe) entry uses more registers to hide the Long-Scoreboard weight-load latency — a win on grid-starved (warp-capped) projections, but a loss on launches that already fill the SMs many waves over, where its lower occupancy dominates. This routes well-occupied launches to the lower-register plain entry and keeps the pipe entry for grid-starved ones.

Both entries are byte-identical (same lane→nibble mapping, same fp16 accumulation order) — a pure occupancy/register trade, not a numeric change.

Before/after ncu (H200, 132 SMs, isolated, --graph-profiling node)

shape	grid (CTAs/SM)	pipe	plain	landed
LM head N=248320	31040 (~235/SM)	98.8µs	85.0µs (−14%)	→ plain
projection N=4096	512 (~3.9/SM)	4.66µs	4.79µs	→ pipe (kept)

Gate: route to plain when ceil(N/columns_per_block) >= mp_count*32 CTAs.

End-to-end tok/s (qwen3.5-0.8b-fp16io, H200)

Paired A/B (interleaved, env-toggled on the same binary to cancel shared-box noise): gate-ON beats the old all-pipe path by +1.3 to +1.6 tok/s every round (~242 → ~243.5 tok/s). The within-round paired delta is stable even as absolute tok/s drifts 239–244.

Byte-identity / golden lock

• Greedy tokens byte-identical gate-on vs gate-off (string-equal check: YES).
• qwen35_0_8b_text_decode_lock golden lock PASSES (ok).
• Unit tests pass: scales_f16_pipeline_is_bit_identical_to_scalar, fp16_gemv_variant_selection_is_structural, fp16_gemv_matches_dequant_reference, plus new scales_f16_pipe_well_occupied_routes_lm_head_to_plain.

Properties

• General — keys only on N, launch width, live SM count; no hardcoded shapes/head-dim/layer-count; works for arbitrary MatMulNBits dims and both fp16/fp32 accumulation paths. Not special-cased to qwen3.5.
• Capture-safe — launch-time constant, stable across CUDA-graph replays; no host syncs or per-step alloc.
• Opt-out — ONNX_GENAI_GEMV_PIPE_WELLOCC=0 restores the pre-gate all-pipe behavior.

Honest floor

Deep prefetch (PF>2) was evaluated and reverted: the grid-starved int4 projections here are all K=1024 → only 4 weight loads/warp, too little per-warp work to hide latency, and occupancy is warp-capped (one warp per output column). Raising it would require split-K, which reorders the fp32 partial reduction → not byte-identical (barred). Those projections are at their byte-identical latency floor (already ~4.3–4.7µs/call, beating ORT's cited 4733ns/call). The one clean, free, byte-identical win is de-pipelining the well-occupied LM head.

Working as Gaff (CUDA-kernel specialist).

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.25%. Comparing base (4a9f4ec) to head (40b588a).
⚠️ Report is 79 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #1501       +/-   ##
===========================================
- Coverage   82.10%   80.25%    -1.85%     
===========================================
  Files          12      377      +365     
  Lines        5471   166289   +160818     
  Branches     5471   166289   +160818     
===========================================
+ Hits         4492   133462   +128970     
- Misses        780    27994    +27214     
- Partials      199     4833     +4634     

Flag	Coverage Δ
cli-ort-linux	82.60% <ø> (?)
cli-ort-windows	82.19% <ø> (+0.09%)	⬆️
offline	80.17% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.
see 365 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.