Forward user provider options to allocator init session by qjia7 · Pull Request #2177 · microsoft/onnxruntime-genai

qjia7 · 2026-05-22T07:33:21Z

Summary

EnsureDeviceOrtInit creates a dummy ORT session to obtain a device allocator before the real user session is created. For WebGPU EP, this first session triggers WebGpuContextFactory::CreateContext, which creates a singleton WebGpuContext whose Dawn device toggles (including skip_validation) are immutable after creation via std::call_once.
Previously the dummy session was created with empty provider options, so the WebGpuContext was always initialized with the default validation_mode=Basic, ignoring the user's validationMode: "disabled" from genai_config.json. When the real session arrived with the user's setting, the singleton already existed and the setting was silently dropped.
This caused Dawn to run the expensive TransformIndirectDispatchBuffer validation path for every indirect dispatch (~4,199 per inference run), because the skip_validation toggle was never enabled on the Dawn device.
Forward the user's provider options from config.model.decoder.session_options.provider_options to the dummy session, excluding enableGraphCapture (which would fail on the trivial model). This ensures singleton resources like WebGpuContext are initialized with the user's intended settings on the very first creation.

Test plan

Verified via runtime debug logging that CreateContext is now called with validation_mode=0 (Disabled) on the first call, and skip_validation toggle is included in Dawn device toggles
Verified model correctness with qwen3-1.7b-graph-prune (graph capture enabled)
Run e2e benchmark to measure performance improvement from skipping indirect dispatch validation

Benchmark: phi4-graph-prune (Vulkan, graph capture ON, RTX 5080)

Same ORT Vulkan DLL used for both runs. Only GenAI differs (main vs this PR).

Metric	`main`	This PR	Delta
Prefill (tps)	11,374	11,654	+2.5%
Token Gen (tps)	101	150	+49%

Params: -w 2 -r 2 --reuse_generator -l 1024 -g 300 -e follow_config

EnsureDeviceOrtInit creates a dummy ORT session to obtain a device allocator before the real user session. For WebGPU, this first session creates the WebGpuContext singleton whose Dawn device toggles are immutable after creation (via std::call_once). Previously the dummy session used empty provider options, so the context was always created with the default validation_mode=Basic, ignoring the user's validationMode=disabled from genai_config.json. Forward the user's provider options to the dummy session so that singleton resources are initialized with the correct settings. enableGraphCapture is excluded since the trivial model cannot use it.

Copilot

Pull request overview

This PR fixes WebGPU EP initialization so that the allocator init “dummy session” created in EnsureDeviceOrtInit receives the same user provider options as the real model session, ensuring singleton WebGPU resources (e.g., WebGpuContext / Dawn device toggles like skip_validation) are configured correctly on first creation.

Changes:

Forward matching user provider options into the dummy session used for allocator initialization.
Filter out enableGraphCapture for the dummy session to avoid failures with the trivial model.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

Also copy device_filtering_options from user's provider options so the dummy session selects the same EP device as the real session.

Limit the provider option forwarding in EnsureDeviceOrtInit to WebGPU only, since the singleton context issue is WebGPU-specific. Also add missing is_nemotron_speech_model_ declaration from main merge.

Change the dummy session's provider option forwarding from a blocklist (skip enableGraphCapture) to a whitelist of 7 global/singleton options (deviceId, webgpuInstance, webgpuDevice, dawnBackendType, powerPreference, validationMode, dawnProcTable). Per-session options like enableGraphCapture, bufferCacheMode, enablePIXCapture are excluded because they are meaningless for the trivial initialization model.

## Summary This PR updates WebGPU release behavior to improve default runtime performance while preserving explicit user control over validation. In non-debug builds, the provider enables Dawn's `skip_validation` toggle by default **only when `validationMode` is not explicitly configured or when explicitly configured to a mode weaker than strict validation**. If `validationMode` is explicitly set to a stricter validation level (> disabled), that setting is honored and `skip_validation` is not forced. Debug build behavior is unchanged. ## Motivation Release users generally prioritize performance, while advanced users need predictable diagnostic control when they explicitly request a stronger validation mode. This change provides a release-friendly default without removing configurability or forcing performance penalties when users explicitly enable validation for debugging. ## Behavior Matrix | Build Type | validationMode Explicitly Set | Behavior | |---|---|---| | Debug | No | No behavior change | | Debug | Yes | Respect explicit validationMode | | Release | No | Enable skip_validation by default | | Release | Yes (validationMode ≤ disabled) | Enable skip_validation | | Release | Yes (validationMode > disabled) | Respect explicit validationMode; do not force skip_validation | ## Implementation - Track whether `validationMode` was explicitly provided during provider option parsing. - Plumb explicitness into WebGpuContext initialization. - In non-debug builds: apply `skip_validation` default only when `validationMode` is absent or weaker than strict validation; otherwise honor explicit choice. ## Why this approach - Improves release-path defaults for performance-sensitive scenarios. - Preserves explicit override semantics for diagnostics/troubleshooting. - Respects user intent: if they explicitly enable strong validation, that choice is honored. - Keeps the change narrowly scoped and low risk. ## Related PRs - [PR](#28581) to use direct dispatch instead of indirect dispatch in the FA kernels to work-around the Dawn inefficiency with basic valiation mode - This is no longer needed - [PR](microsoft/onnxruntime-genai#2177) fix to create the dummy ORT session with user provided device knobs is still needed for other knobs (validation mode is just one such knob) ## Motivation and Context Perf improvement on **Vulkan backend** for **Qwen3-1.7B** model on a **Windows** machine with **RTX 5060Ti** card <img width="387" height="200" alt="image" src="https://github.com/user-attachments/assets/f2d79200-f7cf-48cd-973c-3ba6bdf3c5da" /> The default state for validationMode to disabled greatly helps deocde TPS on Vulkan as it helps reduce the indirect dispatch inefficiency on Vulkan. There is a very small marginal improvement on D3D12 as well although not as marked as the inefficiency with indirect dispatch on D3D12 backend is greatly helped by D3D12's aggressive sub-allocator pooling (something Vulkan backend lacks). NOTE: The "Branch" perf improvement numbers also includes the fusion in #28484 but the fusion's decode TPS gains is under 10%. The rest are contributed by the validationMode default. --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Replace the function-local static std::unordered_set<std::string> with a constexpr std::array<std::string_view, 7> and look up keys via std::find. Removes the hash-table allocation and uses zero-copy string_view comparison against the literal keys. Rename dummy_provider_options to init_session_provider_options so the variable reflects its role (provider options for the allocator init session) and matches the vocabulary used in the PR description.

The block comment above kWebGpuGlobalOptions still referred to the dummy session and cited bufferCacheMode, which is not a real per-session option key. Update both to match the new init_session_provider_options name and use enableInt64 as a real per-session example.

kunal-vaishnavi

The PR looks good to me. In the future, we should consider routing these initial session settings through the EP interface to avoid adding more EP-specific code in the base model.cpp file.

qjia7 · 2026-06-12T05:34:40Z

The PR looks good to me. In the future, we should consider routing these initial session settings through the EP interface to avoid adding more EP-specific code in the base model.cpp file.

Merged it. Will put it in my todo list. Thanks.

qjia7 requested a review from a team as a code owner May 22, 2026 07:33

Copilot AI review requested due to automatic review settings May 22, 2026 07:33

Copilot started reviewing on behalf of qjia7 May 22, 2026 07:33 View session

Copilot AI reviewed May 22, 2026

View reviewed changes

Comment thread src/models/model.cpp Outdated

qjia7 requested a review from Copilot May 22, 2026 07:41

Copilot started reviewing on behalf of qjia7 May 22, 2026 07:41 View session

Copilot AI reviewed May 22, 2026

View reviewed changes

Comment thread src/models/model.cpp Outdated

Comment thread src/models/model.cpp Outdated

qjia7 added 5 commits May 22, 2026 15:50

Forward device filtering options to allocator init session

2ee1529

Also copy device_filtering_options from user's provider options so the dummy session selects the same EP device as the real session.

Merge branch 'main' into fix-webgpu-validation-mode-ordering

042c02d

Scope provider option forwarding to WebGPU and add missing declaration

615b98b

Limit the provider option forwarding in EnsureDeviceOrtInit to WebGPU only, since the singleton context issue is WebGPU-specific. Also add missing is_nemotron_speech_model_ declaration from main merge.

Merge branch 'main' into fix-webgpu-validation-mode-ordering

d774af3

Remove unused is_nemotron_speech_model_ field after merging main

94b92e1

This was referenced May 26, 2026

[WebGPU] Avoid indirect dispatch in FlashAttention decode to fix perf issues with Vulkan backend + GraphCapture/GraphReplay microsoft/onnxruntime#28581

Draft

[WebGPU] Default non-debug validation mode set to disabled microsoft/onnxruntime#28679

Merged

guschmue previously approved these changes May 28, 2026

View reviewed changes

baijumeswani reviewed May 28, 2026

View reviewed changes

Comment thread src/models/model.cpp Outdated

qjia7 dismissed guschmue’s stale review via c54404f June 1, 2026 07:52

Fix clang-format: one initializer entry per line

47f4917

qjia7 requested a review from baijumeswani June 2, 2026 02:43

baijumeswani reviewed Jun 11, 2026

View reviewed changes

Comment thread src/models/model.cpp Outdated

baijumeswani reviewed Jun 11, 2026

View reviewed changes

Comment thread src/models/model.cpp Outdated

qjia7 added 3 commits June 11, 2026 09:53

Merge branch 'main' into fix-webgpu-validation-mode-ordering

c819ff7

baijumeswani approved these changes Jun 11, 2026

View reviewed changes

kunal-vaishnavi approved these changes Jun 11, 2026

View reviewed changes

qjia7 merged commit c1c43d6 into main Jun 12, 2026
63 of 70 checks passed

qjia7 deleted the fix-webgpu-validation-mode-ordering branch June 12, 2026 05:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Forward user provider options to allocator init session - #2177

Forward user provider options to allocator init session#2177
qjia7 merged 11 commits into
mainfrom
fix-webgpu-validation-mode-ordering

qjia7 commented May 22, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kunal-vaishnavi left a comment

Uh oh!

Uh oh!

qjia7 commented Jun 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

qjia7 commented May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Benchmark: phi4-graph-prune (Vulkan, graph capture ON, RTX 5080)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kunal-vaishnavi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

qjia7 commented Jun 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

qjia7 commented May 22, 2026 •

edited

Loading