[CUDA] Extend speculative decode GEMVs to 64 rows

[Tianlei Wu (tianleiwu)]

Description

• Extend the CUDA small-N GEMV path from short decode steps to speculative verification batches with up to 64 rows.
• Keep eligible FP16/BF16 MatMul shapes on the fused GEMV path instead of falling back to the general GEMM path.
• Extend block-scaled FP8 and NVFP4 weight-only GEMV launchers across row-tile boundaries by splitting larger speculative steps into supported sub-launches.
• Add coverage for row counts across the dispatch boundaries, including 9, 17, 33, and 64 rows.

Motivation and Context

Multi-token speculative verification produces matrix row counts beyond the original decode-only range. Falling back at those boundaries adds dequantization, workspace, and general GEMM overhead to a latency-sensitive path. This change keeps those shapes on the specialized kernels while preserving the existing fallback for ineligible dimensions.

Validation

• CUDA EP internal tests: 78 passed, 6 skipped across 20 suites. MatMulSmallNGemvOpTest.DispatchesEligibleShapesWhenEnabled passed and covers 8, 9, 33, and 64 rows.
• MatMulBlockQuantizedFp4WeightOpTest.GemvTensorCoreTilesFp16
• MatMulBlockQuantizedFp4WeightOpTest.GemvTensorCoreTilesBf16
• MatMulBlockQuantizedFp8WeightOpTest.GemvTensorCoreTilesFp16
• The three focused block-scaled tests passed in 19.0 seconds.
• File-scoped lintrunner and git diff --check passed for all 12 changed files.

[Copilot]

Pull request overview

Extends CUDA speculative-decoding GEMV paths to support up to 64 rows while retaining specialized fused kernels.

Changes:

• Processes standard small-N GEMV in 8-row chunks.
• Adds multi-tile and split-launch FP8/FP4 GEMV support.
• Expands dispatch-boundary tests through 64 rows.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
onnxruntime/test/providers/cuda/test_cases/matmul_small_n_gemv_op_test.cc	Tests larger eligible row counts.
onnxruntime/test/contrib_ops/matmul_block_scaled_fp8_test.cc	Expands FP8 tile and fallback coverage.
onnxruntime/test/contrib_ops/matmul_block_scaled_fp4_test.cc	Expands FP4 FP16/BF16 coverage.
onnxruntime/core/providers/cuda/math/matmul.cc	Delegates counter initialization to the launcher.
onnxruntime/core/providers/cuda/math/matmul_small_n_gemv.h	Documents chunked row processing.
onnxruntime/core/providers/cuda/math/matmul_small_n_gemv.cu	Implements 8-row chunked launches through M=64.
onnxruntime/contrib_ops/cuda/math/matmul_block_scaled_fp8.h	Exposes FP8 GEMV row-limit selection.
onnxruntime/contrib_ops/cuda/math/matmul_block_scaled_fp8.cu	Adds multi-tile and split-launch FP8 kernels.
onnxruntime/contrib_ops/cuda/math/matmul_block_scaled_fp8.cc	Routes eligible larger shapes to FP8 GEMV.
onnxruntime/contrib_ops/cuda/math/matmul_block_scaled_fp4.h	Exposes FP4 GEMV row-limit selection.
onnxruntime/contrib_ops/cuda/math/matmul_block_scaled_fp4.cu	Adds multi-tile and split-launch FP4 kernels.
onnxruntime/contrib_ops/cuda/math/matmul_block_scaled_fp4.cc	Routes eligible larger shapes to FP4 GEMV.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

You can commit the suggested changes from lintrunner.