Repository navigation
Fix isolated provider tests in Windows GPU CI - #33203
Merged
Merged
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Co-locating tests that mutate identical provider DLL paths introduces races under parallel CTest execution.
1 open finding
What changed in this PR
Moves four provider test executables into the standard output directory so Windows GPU artifact staging includes them.
Changes:
- Removes custom runtime output directories for provider bridge tests.
- Removes custom runtime output directories for optional provider probe tests.
| File | Description |
|---|---|
cmake/onnxruntime_unittests.cmake |
Uses the standard runtime output directory for four isolated provider tests. |
🧠 Review effort: Balanced
Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The revised paths preserve test isolation while matching the recursively packaged Windows GPU artifact layout.
0 open findings
1 resolved since last review
🧠 Review effort: Balanced
Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.
Tianlei Wu (tianleiwu)
approved these changes
Oct 8, 2026
Tianlei Wu (tianleiwu)
enabled auto-merge (squash)
October 8, 2026 16:29
Akshay Sonawane (apsonawane)
approved these changes
Oct 8, 2026
This was referenced Oct 8, 2026
kunal-vaishnavi
pushed a commit
that referenced
this pull request
Oct 9, 2026
## Summary - Replace the single-token QMoE gate full expert bubble sort with a stable parallel TopK reduction for normalized routing. Preserve descending-logit/index-ascending ties, softmax weight order, and the existing full sort when routing weights are not normalized (the denominator then needs all experts). - Cover boundary ties in 3-, 5-, 6-, and 512-expert pools with nonzero expert outputs. Skip the 512-expert case when either adapter workgroup limit is below 512. - Assert the selected gate indices directly for FP16/FP32 negative-infinity logits (`[0, -inf, -inf]` selects `[0, 1]`) and for all-equal logits with `k == num_experts` (`[0, 1, 2, 3]`). The negative-infinity assertion fails with the original comparator (it selects the invalid sentinel index 3). - Exercise FP16/FP32 nonnormalized routing with an excluded expert contributing appreciably to the all-expert softmax denominator. ## Validation - On head `45037061757c`, merged `origin/main` at `9f6baac7319a` (including #33168) without rebasing or gate conflicts. Rebuilt both the static test-all and split provider-test targets; all 28 focused WebGPU MoE tests passed on H200, including FP16/FP32 gate-index readback. Checked free GPU memory before the run; fresh CI for this head is in progress. - Main-based static WebGPU build and all 22 focused WebGPU QMoE tests passed on H200 after merging `main` at `aac791ef2408a266d8ee59dd5fa0ce11d1a5a0bb`, including the new tie, gate-index, and nonnormalized-routing regressions. - Qwen3.8-Flash-Next INT4 RTN/INT8-head export, 512 experts, k=10, FP16 I/O, one-token chunks, H200: 16-token prefill 11.64 s -> 0.90 s and steady decode ~0.674 s -> ~0.045 s/token. Baseline versus optimized logits were bit-identical (248,320 logits), and the first five generated token IDs matched. This full-model comparison used a QSA-enabled experimental build carrying the same single-token gate change because the export also requires the not-yet-merged SparseAttentionIndexer kernel; the isolated main-based plugin cannot load that export today. The PR itself does not depend on the QSA changes. - This PR changes only the single-token gate. Separate GPU-only multi-token prefill experiments are intentionally excluded pending model-quality validation. ## Main updates - October 5: merged `main` and added the follow-up requested on merged #32878: two experts use distinct INT2/INT4 packed weights as well as scales, with matching FP16-aware numerical references. The fixture explicitly rejects unsupported 8-bit encodings. - October 6: merged `main` at `aac791ef2408a266d8ee59dd5fa0ce11d1a5a0bb`; added direct gate-index assertions, 5-/6-expert pool tests, adapter-limit skip, and the full-denominator regression. That head was superseded by the fixture alignment fix below. - October 6 follow-up: pad the direct gate-index test fixture buffers to 4-byte boundaries for odd FP16 expert counts. On the prior head, macOS Dawn rejected a six-byte storage buffer and an eight-byte staging copy. The prior-head Windows job reported a lost device at the same gate-index test (0 ms), followed by cascading failures; its log does not establish whether the fixture caused the device loss. - October 7 follow-up: current-head arm64 Debug reproduced a real test-fixture readback error: `BufferManager::Download` rounds the copy to 16 bytes, but the `k=2` indices buffer was only 8 bytes. Align that buffer to 16 bytes. Windows novcpkg lost the WebGPU device at the same gate-index test and subsequently failed 1,538 unrelated tests; the static vcpkg aggregate reported the dependent test failure despite successful compilation. All 28 focused WebGPU MoE tests pass after the fix on H200. - October 7 main refresh: merged `main` at `9f6baac7319a` (including #33168), rebuilt the split static WebGPU provider-test target, and passed all 28 focused H200 WebGPU MoE tests on head `45037061757c`. Prior-head CI failures do not describe this new head; current-head checks are running. ## October 8 main refresh Merged `origin/main` at `864d87126864c64651f58a9b51d9006ad71b2c78` (including #33203) into the gate branch, resolving the test include conflict by retaining both sets of includes. Updated the direct gate-buffer test to call upstream `WebGpuDevice(0)` after the context-ID API change. On head `813f7999e55de5a8d29a7a02dfc869d514f327b4`, the static WebGPU provider-test build succeeded and all 34 focused H200/Vulkan `MoETest.QMoETest_WebGPU*` cases passed. Fresh CI is queued; no new-head CI result is claimed yet. ## Matched synthetic whole-inference benchmark (October 8) H200/Vulkan ONNX Runtime `perf_test`, synthetic single-token block-FP8 QMoE with 256 experts and top-4 routing, three 100-run trial averages: merged #32794 baseline **7.09 ms/run** median-trial mean versus the integrated gate branch **5.56 ms/run** (~**1.28x**). Individual trial averages ranged **5.71–7.12 ms** for baseline and **5.46–5.89 ms** for the integrated branch; these overlapping ranges warrant a variance caveat. This measures whole inference, **not** the isolated gate kernel or a full checkpoint. The separate combined branch passed **52/52** focused tests; that result is not claimed as an isolated #33058-head test run. ## October 8 documentation follow-up Added [WebGPU QMoE single-token gate documentation](https://github.com/sayanshaw24/onnxruntime/blob/8d38e0e31be0216e7d3b937578efa113fccee4ca/docs/contrib_ops/webgpu/qmoe_gate.md) on head `8d38e0e31b`, covering normalized top-k, tie and -inf handling, nonnormalized fallback, adapter limits, and benchmark scope. The previous-head Windows documentation-build job stopped while compiling unrelated CUDA sources before document generation, so its dependent validation failure does not establish a generated-doc mismatch. The previous-head Android job failed during NDK setup with a corrupt downloaded archive. Fresh-head CI is running. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Keep the provider bridge and optional provider probe test executables isolated while nesting their per-target runtime directories under the standard CMake configuration output directory. Their post-build commands continue to place the required fixture libraries beside each executable.
Motivation and Context
The split Windows GPU build/test pipeline recursively packages
RelWithDebInfo/RelWithDebInfo, while these four tests previously placed their isolated runtime directories beside that packaged directory. CTest metadata reached the GPU test machine, but the referenced working directories and executables did not, so all four tests were reported asNot Run.Changing the directory layout from
<target>/<config>to<config>/<target>preserves runtime isolation for parallel CTest execution and makes the existing artifact staging include each executable and its fixtures without pipeline-specific copies.Validation:
lintrunner cmake/onnxruntime_unittests.cmake— passedgit diff --check— passed