[WebGPU] Add block-scaled FP8 QMoE

[Sayan Shaw (sayanshaw24)]

Summary

Adds direct WebGPU W8A16 support for QMoE expert weights stored as FP8 E4M3 with 128x128 FP32 block scales, matching the Qwen 3.8 Flash Next MTP expert checkpoint format without requantization or dense weight materialization.

Changes

• Extends the existing quant_type="fp8" QMoE mode: block_size=128 consumes fc*_scales shaped [E, ceil(N/128), ceil(K/128)]; omitted block size retains the existing CUDA global-scale FP8 contract.
• Adds a selected-expert WebGPU shader that decodes raw E4M3 bytes and applies block scales during FP16/FP32 accumulation.
• Supports direct expert dispatch for prefill and indirect top-k expert dispatch for single-token decode.
• Keeps integer, FP4, NVFP4, and WFP4AFP8 behavior unchanged.
• Adds exact two-expert WebGPU coverage with distinct per-expert scale products.

Historical validation (initial Windows run)

• Windows Release onnxruntime_providers_webgpu builds successfully.
• Windows Release onnxruntime_provider_test builds successfully.
• MoETest.QMoETest_WebGPU_BlockFp8_128x128 passes locally on D3D12.
• clang-format --dry-run --Werror and git diff --check pass.
• Editor diagnostics report no errors in changed files.

Target / follow-up

• Targets main directly; the WebGPU MoE/QMoE prerequisite from Add WebGPU MoE and expand QMoE feature support #32447 is merged. This PR's review diff is limited to six FP8-related files, independent of the Qwen integration branch.
• GenAI export wiring for MTP block-FP8 QMoE and full-model chat/performance measurements remain follow-up work. The INT4 RTN model measurements are not validation of the block-FP8 checkpoint.

Historical validation (earlier main restack)

• Linux Release static WebGPU provider and provider-test builds passed on current main.
• All 14 MoETest.QMoETest_WebGPU_* tests passed on NVIDIA H200/Vulkan, including single-token decode, a 3x3 non-word-aligned expert-boundary case, a 129x129 four-scale-tile case, and negative integer-weight validation.
• Focused clang-format --dry-run --Werror and git diff --check passed.

Historical review follow-up (October 5)

Added a separate-FC3 block-FP8 single-token test covering k=2 indirect expert routing, broadcast input, SwiGLU, and distinct per-expert FC1/FC2/FC3 scales. All 17 focused QMoE WebGPU tests passed on H200/Vulkan. All review threads have replies and are resolved; CI is rerunning for the new commit.

Historical main update (October 5)

Merged current main after #32878 landed and reconciled block-FP8 routing with its INT2/mixed-width QMoE logic. The review diff remains limited to the FP8 QMoE files. An isolated Linux static WebGPU provider/test build and all 14 focused block-FP8 plus INT2 compatibility tests passed on H200; the stacked tiled follow-up also passed 15 focused H200 tests. New-head CI is running.

Latest validation (October 6, 40cebdeeef)

• Merged upstream main at aac791ef2408, resolved the QMoE schema/documentation conflicts, and confirmed GitHub reports this head as mergeable.
• The Linux static WebGPU onnxruntime_provider_test build passed; all 26 focused MoETest.QMoETest_WebGPU_* tests passed on NVIDIA H200/Vulkan. The numerical block-FP8 reference now covers both FP16 and FP32 inputs/outputs, including FP8 decoding, accumulation, and biases.
• Before FC1, FC2, or optional FC3 dispatch, the WebGPU path rejects workgroup dimensions that would be normalized and projections exceeding 32-bit shader addressing. Direct expert byte offsets and view sizes use 64-bit arithmetic; the shared limit also covers indirect expert indexing. The oversized-dispatch rejection has an H200 regression test.
• The six review threads concerning FP32 coverage, dispatch normalization, and expert-offset overflow have replies and are resolved. New-head CI is still pending; earlier test counts and CI statements above describe their respective historical heads, not the current result.

October 6 follow-up

Reject orphan block-FP8 FC3 scales instead of silently ignoring them (09737a3); 27 focused H200 WebGPU QMoE tests pass. Refreshed generated ContribOperators.md to match the schema and address Windows GPU documentation CI. Fresh CI is pending.

October 7 review follow-up

Reject all four optional activation-scale inputs (17–20) in WebGPU block-FP8 mode instead of silently ignoring them. The GPU-only negative regression supplies each input separately. Updated the schema and generated operator docs to state that activation scales are not accepted. All 28 focused H200 WebGPU QMoE tests pass on eb51a28; new-head CI is pending.

October 7 CI follow-up

Merged current main (6030cc6), including upstream CUDA GQA fix #33179: the previous Linux/Windows CUDA and TensorRT builds failed because is_int8 was used before its declaration in upstream CUDA code, not in this PR's WebGPU changes. The updated Linux static WebGPU provider test builds and all 28 focused H200 QMoE tests pass. Previous-head Python 3.14 and Windows QNN jobs failed ONNX materialized-node tests; Android minimal binary size exceeded its threshold by 1,494 bytes. New-head CI is running; these failures are not yet claimed fixed until the fresh checks report.

October 7 upstream CI fix merged

#33168 merged to main and corrects the ONNX wire-type to C-API-type conversion behind the old-head Windows QNN/Python 3.14 materialized-node failures. Merged main at fdf7f9d; static WebGPU tests build and all 28 focused H200 QMoE cases pass. The old-head Android binary-size check passed. New-head CI is running; no claim of CI green until the fresh checks finish.

[Copilot]

Copilot review overview

🟡 Changes recommended

Direct expert indexing and large-buffer bindings are incorrect, with additional validation, documentation, and decode-test gaps.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 2 · 1 · 3

Open (6)

• Double-offset expert-sliced BufferViews · New
• Missing uint8 validation for integer quantization · New
• Backing entries duplicate segmented buffer bindings · New
• Generated operator documentation is outdated · New
• Block-FP8 dimension documentation contradicts partial-tile support · New
• Missing coverage for indirect single-token expert dispatch · New

What changed in this PR

Adds WebGPU QMoE support for 128×128 block-scaled FP8 expert weights.

Changes:

• Adds FP8 validation, dispatch, and dequantization shader logic.
• Extends the QMoE schema and adds WebGPU coverage.
• Cleans up Engram gate declarations.

File	Description
onnxruntime/​contrib_ops/​webgpu/​moe/​qmoe.cc	Implements block-FP8 execution paths.
onnxruntime/​contrib_ops/​webgpu/​moe/​qmoe.h	Adds FP8 mode configuration.
onnxruntime/​contrib_ops/​webgpu/​moe/​block_fp8_expert_matmul.wgsl.template	Adds the FP8 expert matmul shader.
onnxruntime/​core/​graph/​contrib_ops/​contrib_defs.cc	Documents the new QMoE contract.
onnxruntime/​test/​providers/​webgpu/​contrib_ops/​moe_test.cc	Adds block-FP8 prefill coverage.
onnxruntime/​contrib_ops/​webgpu/​bert/​engram_gate.h	Removes obsolete and duplicate declarations.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

Review effort: Lite
Findings: 1 · 2

Open (3)

• When DISABLE_FLOAT8_TYPES is defined, this function still returns WGSL declaring `array<u32,… · New
• The shader loads scale from storage on every inner iteration even though it only changes once per… · New
• Building the FP8 dequant LUT as a large WGSL string via ostringstream and a 256-iteration loop… · New

Resolved since last review (6)

• Missing uint8 validation for integer quantization
• Double-offset expert-sliced BufferViews
• Backing entries duplicate segmented buffer bindings
• Missing coverage for indirect single-token expert dispatch
• Block-FP8 dimension documentation contradicts partial-tile support
• Generated operator documentation is outdated

[Copilot]

Copilot review overview

🟡 Changes recommended

The new separate-FC3 block-FP8 execution path lacks regression coverage.

Review effort: Balanced
Findings: 1

Open (1)

• Add block-FP8 test covering FC3 weights, scales, and expert routing · New

Resolved since last review (3)

• When DISABLE_FLOAT8_TYPES is defined, this function still returns WGSL declaring `array<u32,…
• Building the FP8 dequant LUT as a large WGSL string via ostringstream and a 256-iteration loop…
• The shader loads scale from storage on every inner iteration even though it only changes once per…

[kunal-vaishnavi]

From Copilot:

Beyond the already-raised FP32 coverage gap, could we add focused block-FP8 coverage for these execution combinations?

• Fused SwiGLU with swiglu_fusion=1 and 2, distinct gate/up values and scales, and non-default alpha/beta/limit.
• Multiple tokens routed to the same expert, so the new direct matmul actually executes with rows > 1; include nonzero expert biases.
• A rectangular scale grid with unequal N-block and K-block counts, such as 129x257.
• Top-k selection from more than k experts with unequal mixing weights, plus a single-token case with explicit router_weights to exercise the direct fallback.

The current successful multi-token FP8 fixtures assign one token to each expert, the scale-boundary fixture is square, and the k=2 fixtures select both experts with equal mixing weights. Also, the separate-FC3 test uses activation_type="silu", so it does not cover the fused SwiGLU layouts or the parameterized/clipped SwiGLU branch.

[Sayan Shaw (sayanshaw24)]

CI follow-up for head 6030cc6af8: the earlier CUDA is_int8 compile failures were resolved by merging upstream #33179, and the Android minimal binary-size check now passes. The two remaining failed checks, Windows QNN shared-lib and Linux Python 3.14, both fail onnx_test_node_materialized on INT2/UINT2/FLOAT8E8M0 tensor types. This is the unrelated ONNX-node-test loader conversion issue reproduced on main; upstream fixes are already under review in #33150 and #33168. Rerunning these deterministic jobs on the same head will not address it. Once one upstream fix merges, I can merge current main here (and into stacked #33075) to trigger fresh CI without folding unrelated test-loader code into this WebGPU PR. Other checks are still running.