[WebGPU] Support block-quantized FP8 dense MatMul by sayanshaw24 · Pull Request #33074 · microsoft/onnxruntime

Sayan Shaw (sayanshaw24) · 2026-10-02T22:08:18Z

Summary

Add WebGPU support for com.microsoft::MatMulBlockQuantizedFp8Weight, the long-named dense projection operator used by the FP8 model export. This remains necessary independently of MoE/QMoE support and targets main directly.

Read row-major E4M3FN weights with FP32 block scales [N, ceil(K/block_size)].
Support optional activation FP8 quantize/dequantize, bias, rank-one/batched inputs, partial blocks, non-word-aligned rows, and empty outputs/reductions.
Support FP16 activations. BF16 storage is not supported by this WebGPU kernel.
Use FP16 subgroup-matrix operands with FP32 accumulation for supported prefill configurations, with a scalar FP32-reduction fallback. This is not native FP8 matrix hardware compute.
Preserve NaNs during activation quantization and use explicit E4M3 ties-to-even rounding.
Preserve the FP16-only operator during cast insertion and prefer a registered WebGPU plugin over a built-in factory when applicable.

October 16 refresh and review fixes

Merged main at e88538cc6f and pushed the review fixes in 4905cd01f2.

Classify activation NaNs through their bits before saturation. A floating-point self-comparison was optimized away on the real Vulkan backend; the bitwise check passes the added NaN regression.
Use unequal positive/negative subnormal weights, so flushing both values to zero no longer leaves the reference unchanged.
Make the shared subgroup_id builtin opt-in per program and check the separate WGSL language feature. Ordinary shaders and scalar fallbacks no longer acquire that requirement.
Gate both dense and routed block-FP8 matrix paths, which share the builtin, on the language capability. Include the opt-in in the program cache key.
Add capability/cache-key regression coverage and document the operator and fallback behavior.

Current-head validation

Linux Release static WebGPU provider-test and general unit-test builds passed.
All 16 dense FP8 tests passed on H200/Vulkan with CPU EP fallback disabled. Matrix-specific cases ran rather than skipped, including distinct subgroup tiles and a second workgroup.
All 38 related WebGPU QMoE tests passed, covering the shared shader-generation change.
All four focused cast-transformer tests passed.
Incremental repository-standard lintrunner and git diff --check passed.
Fresh remote CI is running. Browser/Metal portability and macOS arm64 results must be confirmed by the new CI runs; local Vulkan results alone do not establish those.

Earlier evidence and limits

An earlier shared-plugin build ran builder-exported W8A16 and W8A8 dense projection nodes with CPU fallback disabled and matched their CPU references. Earlier synthetic prefill trials showed the tiled path faster than its forced scalar control, but these are historical measurements, not a new performance claim for this refreshed head.

This PR's focused validation is not a full-model quality or throughput qualification. Routed FP8 expert execution and native-FP8 WebGPU/Dawn proposal work remain separate scopes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

Review effort: Lite
Findings: 2 · 2

Open (4)

Creating a uint8_t Tensor view over B via const_cast<void*>(b->DataRaw()) weakens… · New
The shader indexes weights as 32-bit words (weights.getByOffset(offset / 4u)), so correctness… · New
The FP8 quantization logic relies on several format-specific magic constants (e.g., 448.0, -6.0,… · New
K != 0 is already guaranteed here because the K == 0 case returns earlier. Dropping the… · New

What changed in this PR

Adds a WebGPU implementation for the com.microsoft::MatMulBlockQuantizedFp8Weight operator (block-quantized FP8 weights with per-block FP32 scales), including an optional activation FP8 Q/DQ path and a focused WebGPU test suite.

Changes:

Register a new WebGPU contrib kernel for MatMulBlockQuantizedFp8Weight (guarded by float8 availability).
Implement WGSL shaders for FP8 weight decode + matmul (subgroup-matrix fast path + scalar fallback) and optional activation Q/DQ.
Add WebGPU provider tests covering odd-byte layouts, block tails, bias, activation Q/DQ corner cases, empty shapes, and invalid scale shape.

File	Description
onnxruntime/test/providers/webgpu/contrib_ops/matmul_block_quantized_fp8_weight_test.cc	Adds targeted WebGPU tests for FP8 block-quantized dense matmul and validation errors.
onnxruntime/contrib_ops/webgpu/webgpu_contrib_kernels.cc	Registers the new WebGPU kernel behind the float8 build flag.
onnxruntime/contrib_ops/webgpu/quantization/matmul_block_quantized_fp8_weight.wgsl.template	Implements FP8 decode + matmul shader with optional bias and matrix/scalar paths.
onnxruntime/contrib_ops/webgpu/quantization/matmul_block_quantized_fp8_weight.h	Declares the new WebGPU kernel class.
onnxruntime/contrib_ops/webgpu/quantization/matmul_block_quantized_fp8_weight.cc	Implements kernel plumbing, shape validation, optional activation Q/DQ, and dispatch configuration.
onnxruntime/contrib_ops/webgpu/quantization/matmul_block_quantized_fp8_activation.wgsl.template	Implements activation FP8 quantize/dequantize shader used for W8A8 numerics.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Avoid casting FP16-only operators to unsupported FP32 before EP assignment and prefer the registered WebGPU plugin when initializing a Python session. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Apply explicit E4M3 ties-to-even instead of backend-dependent WGSL round. Document the read-only FP8 byte alias and padded binding and extend tie coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Avoid backend-dependent reciprocal rounding at halfway cases. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Keep FP8 rounding decisions consistent with CUDA reciprocal quantization and test adjacent FP16 values at a non-power-of-two scale. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Sayan Shaw (sayanshaw24)

Publishing the previously drafted replies to the addressed review comments.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

🟡 Changes recommended

Activation quantization currently converts NaN inputs into finite saturated values.

1 open finding

Preserve NaN during FP8 activation saturation · New

4 resolved since last review

The shader indexes weights as 32-bit words (weights.getByOffset(offset / 4u)), so correctness…
Creating a uint8_t Tensor view over B via const_cast<void*>(b->DataRaw()) weakens…
K != 0 is already guaranteed here because the K == 0 case returns earlier. Dropping the…
The FP8 quantization logic relies on several format-specific magic constants (e.g., 448.0, -6.0,…

🧠 Review effort: Balanced

## Summary Follow up on #32794 with an optimized **routed expert** matmul for block-scaled FP8 QMoE. This is separate from the dense MatMul kernel in #33074. - Stage selected-expert E4M3 weights and FP16 activations into shared tiles and use supported 16x16x16 FP16-input subgroup-matrix instructions with FP32 accumulation. FP8 is decoded before the matrix instruction; this is **not native FP8 matrix hardware**. - Handle per-128x128 scales, signed FP8 codes, FC1/FC2 bias, odd-byte expert boundaries, and incomplete matrix/scale tiles. - Only dispatch the matrix path for direct expert batches with at least 128 rows and 256 input/output channels on adapters exposing the required subgroup configuration. Preserve the original scalar path for single-token decode, smaller routed groups, and other adapters. A 16-row synthetic case regressed with the matrix path, so it is deliberately excluded. ## Validation - Linux Release static WebGPU provider, provider tests, and Python WebGPU runtime built successfully. - All 17 focused WebGPU QMoE tests passed on NVIDIA H200/Vulkan, including a new 128-row, 257-channel case covering signed FP8 values, distinct 128x128 block scales, bias, tail tiles, and non-word-aligned expert offset. - Synthetic end-to-end QMoE with 128 rows, 2560 hidden channels, 640 intermediate channels, two experts, and 30 repeated calls: matrix path 4.4–5.1 ms/call versus original scalar 6.17 ms/call in this harness (about 1.2–1.4x). This includes routing and other kernel work; it is **not isolated matrix-kernel timing**. At 16 rows the original scalar path remains enabled (about 3.7 ms/call). - `clang-format --dry-run --Werror` and `git diff --check` passed. No official block-FP8 checkpoint or end-to-end model-quality claims: the local checkpoint is BF16. Stack base is #32794 for review; merge the PRs individually when ready. ## Stack update (October 5) Restacked onto #32794 after its separate-FC3 regression was added. The review diff remains limited to the three QMoE optimization files. All 18 focused stacked WebGPU QMoE tests passed on H200/Vulkan; CI is rerunning for the updated base. ## Current main update (October 5) Merged the latest #32794 base, which itself now includes current `main` and merged INT2/mixed-width QMoE support. The review diff remains the three tiled-FP8 files. All 15 focused H200 block-FP8 and merged INT2 compatibility tests pass; new-head CI is running. ## Review follow-up (October 6) - Apply FP32 block scales to FP32 subgroup-matrix partial sums, not FP16 staged weights; a +448/-448 cancellation test with scale 200 checks finite output. - Keep matrix workgroup dimensions within the adapter limit, falling back to scalar when its dispatch is legal; reject oversized scalar dispatch through the updated #32794 base. - Add mixed 128+4 row routing with distinct expert FC1/FC2/FC3 weights, scales, and nonzero biases, an independent SwiGLU reference, and an adapter-aware assertion that all three eligible projections actually dispatch the matrix path. - After merging #32794 head `40cebdeeef`, built the Linux Release provider tests and passed all 30 focused WebGPU QMoE tests on H200/Vulkan; formatting and `git diff --check` passed. This does not imply other platform CI has completed. ## October 6 base update Merged updated FP8 QMoE base 09737a3, inheriting orphan FC3-scale validation and synchronized operator docs that address the Windows GPU documentation failure (stacked head 23922ce). All 31 focused H200 WebGPU QMoE tests pass; fresh CI is pending. Previously drafted replies to three review threads have now been published. ## October 7 base update Merged updated #32794 base eb51a28 to inherit explicit rejection of unused FP8 activation-scale inputs. The new four-case regression and all 32 focused H200 WebGPU QMoE tests pass on stacked head 158c7d0. Fresh CI is pending. ## October 7 CI stack update Merged current main and updated #32794 (541e5c7), including the already-merged upstream CUDA GQA `is_int8` build fix #33179. The static WebGPU provider tests build and all 32 focused H200 routed-QMoE tests pass. New-head CI is pending. ## October 7 upstream CI fix merged Restacked on updated #32794 at 79873ae, including merged upstream #33168 ONNX node-test type conversion. Static WebGPU provider tests build and all 32 focused H200 routed-QMoE cases pass. Fresh CI is pending; previous-head failures are not evidence for this head. ## October 8 merged-base refresh and paired performance Merged current main after #32794 landed. The review diff remains the QMoE matrix-path implementation and its tests. In an integrated branch with #33058 and #33074, 52/52 H200 WebGPU QMoE and dense FP8 regressions passed. For a synthetic FP16-input, 2-expert block-FP8 QMoE (128 tokens, hidden 2560, intermediate 640), ONNX Runtime perf_test on H200/Vulkan measured 7.86 ms/run on the merged #32794 implementation versus 6.05 ms/run with the tiled path (median of three 100-run trial averages, ~1.30x). These are whole-session inference latencies, not isolated kernel timings or an official-FP8-checkpoint result. Current-head CI is pending. ## October 9 main refresh and subgroup ownership fix Merged main at 9a6089e, including merged #33058 and its gate tests/documentation. The PR now targets the merged FP8 QMoE implementation on main; #32794 is no longer an unmerged stack prerequisite. Addressed the new subgroup-ownership review in 7567ecd: the shared shader entry point exposes the actual `@builtin(subgroup_id)` as `subgroup_index`, and the QMoE matrix path uses it for accumulator initialization, left-operand loading, and result storage. Cooperative tile transfers still use local invocation indices. This matches the dense FP8 implementation in #33074. Validation on the updated head: - Linux Release WebGPU provider-test build succeeded. - 38/38 focused H200/Vulkan WebGPU QMoE tests passed, including the matrix-capability-aware mixed-routing/separate-FC3 regression and the merged gate cases. - 89/89 WGSL-template tests passed. - Captured GPU shaders use the subgroup-ID builtin for all three matrix tile offsets; no `lane / 32u` offset remains. - `clang-format --dry-run --Werror` and `git diff --check` passed. The subgroup-mapping fix was executed on H200, not an adapter with a noncontiguous subgroup mapping. Fresh GitHub CI is pending; the local results are not a claim that all platform CI has completed. --------- Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

🔵 Needs a closer look

Cross-backend validation of the subgroup-matrix shaders and shared shader-helper change remains necessary before final human approval.

2 open findings

🧠 Review effort: Balanced

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Strengthen non-canceling subnormal coverage and explicitly request subgroup_id only for supported dense and routed FP8 matrix programs. Cache the shader opt-in separately and document the capability fallback. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add WebGPU block-quantized FP8 dense matmul

cbcd5e9

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI balanced review requested due to automatic review settings October 2, 2026 22:08

Copilot AI reviewed Oct 2, 2026

View reviewed changes

Sayan Shaw (sayanshaw24) mentioned this pull request Oct 2, 2026

[WebGPU] Tile large block-FP8 QMoE expert matmuls #33075

Merged

Copilot started reviewing on behalf of Sayan Shaw (sayanshaw24) October 2, 2026 22:59

View session

Sayan Shaw (sayanshaw24) and others added 4 commits October 5, 2026 16:50

Declare FP8 activation shader type dependencies

c6ca2e2

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Merge remote-tracking branch 'origin/main' into webgpu-fp8-block-matmul

b7c0e85

Compare FP8 activation ties in input space

2f12bcd

Avoid backend-dependent reciprocal rounding at halfway cases. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Akshay Sonawane (apsonawane) reviewed Oct 6, 2026

View reviewed changes

Comment thread onnxruntime/contrib_ops/webgpu/quantization/matmul_block_quantized_fp8_weight.wgsl.template Outdated

kunal-vaishnavi reviewed Oct 6, 2026

View reviewed changes

Comment thread onnxruntime/test/providers/webgpu/contrib_ops/matmul_block_quantized_fp8_weight_test.cc

kunal-vaishnavi reviewed Oct 6, 2026

View reviewed changes

Comment thread onnxruntime/contrib_ops/webgpu/quantization/matmul_block_quantized_fp8_weight.wgsl.template

kunal-vaishnavi reviewed Oct 6, 2026

View reviewed changes

Comment thread onnxruntime/contrib_ops/webgpu/quantization/matmul_block_quantized_fp8_activation.wgsl.template Outdated

Sayan Shaw (sayanshaw24) and others added 2 commits October 6, 2026 16:48

Merge remote-tracking branch 'origin/main' into webgpu-fp8-block-matmul

ba64210

Fix FP8 WebGPU dispatch, subgroup ownership and quantization ties

38a1262

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Akshay Sonawane (apsonawane) reviewed Oct 6, 2026

View reviewed changes

Comment thread onnxruntime/contrib_ops/webgpu/quantization/matmul_block_quantized_fp8_activation.wgsl.template Outdated

Compare FP8 activation midpoints in reciprocal domain

dc9a8e1

Keep FP8 rounding decisions consistent with CUDA reciprocal quantization and test adjacent FP16 values at a non-power-of-two scale. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Sayan Shaw (sayanshaw24) commented Oct 6, 2026

View reviewed changes

Sayan Shaw (sayanshaw24) requested a review from Akshay Sonawane (apsonawane) October 6, 2026 21:18

Sayan Shaw (sayanshaw24) and others added 2 commits October 7, 2026 21:57

Merge main with ONNX materialized-node test fix

f86ef3c

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Merge remote-tracking branch 'origin/main' into webgpu-fp8-block-matmul

9268738

Akshay Sonawane (apsonawane) requested a balanced review from Copilot October 8, 2026 20:14

Copilot started reviewing on behalf of Akshay Sonawane (apsonawane) October 8, 2026 20:15

View session

Copilot AI reviewed Oct 8, 2026

View reviewed changes

Comment thread onnxruntime/contrib_ops/webgpu/quantization/matmul_block_quantized_fp8_activation.wgsl.template

kunal-vaishnavi requested a balanced review from Copilot October 16, 2026 20:33

Copilot started reviewing on behalf of kunal-vaishnavi October 16, 2026 20:33

View session

Copilot AI reviewed Oct 16, 2026

View reviewed changes

Comment thread onnxruntime/test/providers/webgpu/contrib_ops/matmul_block_quantized_fp8_weight_test.cc Outdated

kunal-vaishnavi reviewed Oct 16, 2026

View reviewed changes

Comment thread onnxruntime/core/providers/webgpu/shader_helper.cc Outdated

Sayan Shaw (sayanshaw24) and others added 2 commits October 16, 2026 22:24

Merge current main into WebGPU block-FP8 dense MatMul

d9718cf

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

kunal-vaishnavi approved these changes Oct 16, 2026

View reviewed changes

Navigation Menu

Repository navigation

[WebGPU] Support block-quantized FP8 dense MatMul - #33074

[WebGPU] Support block-quantized FP8 dense MatMul#33074
Sayan Shaw (sayanshaw24) wants to merge 13 commits into
mainfrom
sayanshaw/webgpu-fp8-block-matmul

Sayan Shaw (sayanshaw24) commented Oct 2, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Sayan Shaw (sayanshaw24) left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Sayan Shaw (sayanshaw24) commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

October 16 refresh and review fixes

Current-head validation

Earlier evidence and limits

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Sayan Shaw (sayanshaw24) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Sayan Shaw (sayanshaw24) commented Oct 2, 2026 •

edited

Loading