[WebGPU] Restore ORT 1.24.4 compatibility with per-Session recording and serialized execution

[Xiaofei Han (xiaofeihan1)]

Description

Restore WebGPU plugin compatibility starting with ORT 1.24.4, while retaining Session-owned command recording in both execution modes. The additional real-host matrix below found an unresolved ORT 1.26.0 host-side rejection; this is not a claim that every intervening release currently works.

• Each Session exclusively owns its recording through unique_ptr; there is no context-owned shared legacy recording.
• Select the older-host compatibility policy with UseSerializedExecutionMode() and ShouldUseSerializedExecutionMode(), now isolated in ep/runtime_compatibility.h/.cc rather than the stream implementation.
• In serialized mode, Session Alloc and matching AllocOnStream use !IsRunActive() to submit cached-buffer clears outside Run and defer them during Run. Kernel scratch uses ordinary Tensor allocation through the same allocator, without an AllocForKernel API or the unavailable old-host KernelContext_GetSyncStream.
• Env allocations and concurrent-mode ordinary Session allocations retain independent, immediately submitted clears. Cached-buffer clearing and kernel dispatch batching remain enabled.
• For streamless framework copies, submit pending active-Session work before recording the copy independently. The context borrows a non-owning pointer to that recording; it does not own or share the command state. Session teardown discards only that Session's pending entries.
• Remove the context-wide Run gate and rely on the required caller-serialization contract. SerialRunsAfterRejectedStreamOverride covers a rejected stream override followed by successful serial Runs on both the affected Session and another Session.

Host ORT version	Default execution mode
1.26.0	Currently blocked by the host's plugin-version check; see validation below
1.28.x starting at 1.28.3	Concurrent independent Sessions
1.30.x starting at 1.30.1	Concurrent independent Sessions
1.31 and later	Concurrent independent Sessions
Other supported versions, including all 1.29.x	Serialized compatibility

Set ORT_WEBGPU_EP_FORCE_LEGACY=1 before plugin registration to force serialized compatibility. The existing override and onnxruntime_webgpu_legacy_test CTest name are retained for compatibility; neither implies shared recording. The override cannot enable concurrency on an unsupported host.

Serialized mode requires callers to serialize all WebGPU operations on the same device, including Session creation/destruction, Run, allocation, I/O binding, and Env transfers. Use sequential graph execution. This is not a fixed-CPU-thread requirement, and the plugin neither detects nor serializes overlapping operations. Callback reentry remains unsupported.

Input/output copy ordering

ORT 1.24.4 forwards supplied streams through batched CopyTensors; its separate single-tensor wrapper drops the stream. BindInput uses a single-input copy, and mixed source/destination device pairs can make a batch fall back to individual copies.

Path	Ordering mechanism
BindInput outside Run, CPU to GPU	Allocation submits any cached-buffer clear before returning, including through AllocOnStream. A subsequent streamless upload is ordered after it on the device queue; no active Run pointer is needed.
Input copy during Run, CPU to GPU	With a stream, the clear and upload use the same Session recording. Without a stream, flush the active Session first, then upload using a local recording.
Output copy during Run, GPU to CPU	Submit the producer before readback, either through its explicit stream or the active-Session pre-copy flush. This does not depend on OnRunEnd, which occurs after output copies.
Mixed-direction inputs/outputs	Each streamless copy follows the same submission boundary if the old host falls back to individual copies.
Streamless GPU-to-GPU copy within one context	Submit preceding active Session work, then submit the local copy before returning. Cross-context copies remain unsupported.

These are submission boundaries, not per-kernel CPU waits. Readback retains its existing completion wait. Compared with a shared recording, a streamless copy with pending Session work can require an additional queue submission; performance equivalence has not been established.

Motivation and Context

Allow independently distributed WebGPU plugins to support older applications without requiring the host-side single-copy stream-forwarding fix in #32666. Builds on #32874.

Following review, the implementation now keeps Session-owned recording rather than introducing a second ownership model for old hosts. A real ORT 1.24.4 mixed-output repro demonstrated the missing submission boundary: an independent GPU Neg branch fetched to CPU returned 0 instead of -1 when the Session's pending computation was not submitted before a streamless readback. The other independent CPU branch was fetched to GPU, triggering mixed-direction single-copy fallback. Both targets and the GPU input were preallocated. Adding the boundary makes both output orderings pass; this shows the need for ordering coordination, not shared recording ownership.

The alternative developed in #33228 has been incorporated here, and that duplicate PR is closed. This revision includes upstream main at e19a9ae831.

Scope: Removing the context-wide gate avoids permanently blocking other Sessions when a host error skips OnRunEnd. It does not guarantee recovery of all unfinished Run/capture state. Clarifying the host's OnRunStart/OnRunEnd error-path contract and complete failed-replay recovery remain separate follow-up work.

Validation

Validated commit 20ff4e2f63 on Windows x64 Release with the shared WebGPU plugin and EP API adapters. Current-host tests use ORT 1.32.0. W: was a temporary mapping to this worktree and was removed afterward.

Build: passed.

cmake --build W:\build\WebGpuCompat\Release --config Release --target onnxruntime_providers_webgpu onnxruntime_provider_test --parallel 8 -- /nologo /verbosity:minimal

Serialized compatibility: 41 executed tests passed, no failures or skips. CTest sets ORT_WEBGPU_EP_FORCE_LEGACY=1 before loading the plugin and selects serial-safe cases.

Set-Location W:\build\WebGpuCompat\Release
ctest -C Release -R '^onnxruntime_webgpu_legacy_test$' --output-on-failure

Concurrent-mode selection: 57 executed tests passed, no failures or skips. Two pre-existing disabled diagnostic tests were not enabled in either selection.

Set-Location W:\build\WebGpuCompat\Release\Release
$env:ORT_UNIT_TEST_MAIN_DYNAMIC_PLUGIN_EP_CONFIG_JSON='{"ep_library_registration_name":"WebGPU_PluginEP","ep_library_path":"onnxruntime_providers_webgpu.dll","selected_ep_name":"WebGpuExecutionProvider"}'
$env:ORT_WEBGPU_EP_FORCE_LEGACY='0'
.\onnxruntime_provider_test.exe '--gtest_filter=WebGpuPluginRecordingModeTest.*:WebGpuPluginSharedAllocatorRegistrationTest.*:WebGpuPluginSharedAllocatorTest.*:WebGpuSessionAllocatorDeathTest.*:PluginEpWebGpuConcurrency.*'

Coverage includes version boundaries, allocation-clear submission, scratch batching, serial and concurrent capture/replay, mixed feeds/fetches, and serial Runs after a rejected stream override.

Actual ORT 1.24.4: all 9 local-harness scenarios passed, 20 iterations each. The standalone API-24 C++ probe loaded the official 1.24.4 DLL and the plugin rebuilt from this commit; profiles confirmed CPU/WebGPU node placement. Cases cover CPU I/O, dirty-buffer BindInput, mixed-direction inputs and outputs in both orders, GPU-to-CPU-to-GPU graph execution, scratch reuse, and capture-enabled execution.

The following invocation used the existing local recording-ownership-probe directory ($probe); this harness is not part of the repository or CI:

$variant='serialized-policy-20ff4e2f63'
$dest=Join-Path $probe $variant
$env:ORT_WEBGPU_EP_FORCE_LEGACY='1'
python "$probe\run_probe.py" $variant "$dest\onnxruntime_providers_webgpu.dll"

Capture limitation: the actual 1.24.4 run exercised initial capture and its internal replay. Its subsequent Runs did not exercise host-driven replay, because that host lacks the plugin capture/replay callbacks. Repeated host-driven replay was covered by the current-host tests above.

Other operating systems and performance benchmarks were not tested in the above 20ff4e2f63 validation.

Additional local validation at 62e3504c2e

Removed only the redundant pre-capture OnRunStart flush. CaptureBegin still flushes before entering capture; streamless-copy, Run-end, and standalone replay submission boundaries are unchanged. The plugin rebuilt successfully, and the three focused serialized capture tests passed:

$env:ORT_UNIT_TEST_MAIN_DYNAMIC_PLUGIN_EP_CONFIG_JSON='{"ep_library_registration_name":"WebGPU_PluginEP","ep_library_path":"onnxruntime_providers_webgpu.dll","selected_ep_name":"WebGpuExecutionProvider"}'
$env:ORT_WEBGPU_EP_FORCE_LEGACY='1'
.\onnxruntime_provider_test.exe '--gtest_filter=PluginEpWebGpuConcurrency.SerialGraphCaptureWithCpuFeedsAndFetches:PluginEpWebGpuConcurrency.SerialSingleSessionMultipleGraphCaptureIds:PluginEpWebGpuConcurrency.GraphCaptureReplayInterleavedWithIdleSessionCpuBindInput'

A separate local matrix used one unchanged plugin DLL with official Windows x64 cores 1.24.4, 1.25.1, 1.26.0, 1.27.1, 1.28.3, 1.29.1, and 1.30.0, plus the local 1.32.0 Release core. Loaded module paths, actual core versions, and plugin hashes were checked.

• Native copy/allocation/capture probe: 63 scenarios passed, 9 failed, with 20 iterations per scenario. All nine scenarios passed on every tested core except 1.26.0. That core rejects OrtEp::ort_version_supported=32 because its kernel-creation check requires a value no greater than 26. This host restriction remains unresolved; forcing serialized mode does not address it.
• Capture support differs from execution-mode eligibility. In the simple native capture probe, 1.24.4 and 1.25.1 performed one initial internal replay; the other passing cores performed repeated host-driven replay.
• Local Phi-4 workload, capture OFF: all seven usable cores passed fresh-generator repeatability, serially interleaved-generator checks, multi-turn checks, and a repeated exact 1024-input/300-output-token workload. An isolated GenAI build compiled against API 24 was held fixed across cores; the normal installed package requires API 29. Performance measurements used two warmups and five measured runs, generator reuse and shared KV buffers, with profiling disabled.
• Capture ON is not valid for this model graph. It contains an If node. Tested cores from 1.26 onward reject capture at Session initialization. On 1.24.4 and 1.25.1, initialization succeeds but generated tokens differ after rewinding and repeating the same prompt. This also reproduces without profiling and with the pre-cleanup 20ff4e2f63 plugin, so it was not introduced by removing the redundant flush. No capture-ON throughput is reported for this model.
• Performance interpretation: default core graph optimizations were retained; newer cores use additional MLP fusion. Cross-version throughput is therefore not an isolated measurement of serialized-mode overhead. Generated tokens were repeatable within each core but not identical between the older and newer optimizer groups.

The matrix scripts and model are session-local, not checked into the repository or run by CI. $matrix below refers to the isolated phi4-core-matrix artifact directory:

python "$matrix\run_native.py"                   # failed overall: 63 passed, 9 failed on 1.26.0
python "$matrix\run_phi4.py" --phase smoke        # failed overall: 7 passed, 9 unsupported/failing combinations
python "$matrix\run_phi4.py" --phase correctness # passed: 21 checks; failing smoke combinations excluded
python "$matrix\run_phi4.py" --phase benchmark   # completed: 7 capture-OFF configurations; failing smoke combinations excluded

[Copilot]

Copilot review overview

🟡 Changes recommended

Failed replay cleanup can contaminate the shared legacy recording, and the compatibility path lacks automated CI execution.

Review effort: Balanced
Findings: 1 · 1

Open (2)

• Reset shared recording state after failed replay · New
• Add automated coverage for the legacy compatibility branch · New

What changed in this PR

Restores WebGPU plugin compatibility with ORT 1.24.4 through a shared serial command-recording path while retaining modern concurrency.

Changes:

• Adds runtime-version-based legacy recording and serialization.
• Updates allocation, transfer, capture, and replay behavior.
• Expands compatibility, ordering, and batching tests and documentation.

File	Description
plugin-ep-webgpu/​README.md	Documents compatibility and concurrency contracts.
plugin-ep-webgpu/​MIN_ONNXRUNTIME_VERSION	Lowers minimum ORT version to 1.24.4.
onnxruntime/​test/​providers/​webgpu/​plugin/​test_webgpu_concurrency.cc	Adds ordering and legacy serialization tests.
onnxruntime/​test/​providers/​webgpu/​plugin/​test_webgpu_allocators.cc	Adds clear-submission and batching tests.
onnxruntime/​core/​providers/​webgpu/​webgpu_provider_factory.cc	Routes legacy transfers through shared recording.
onnxruntime/​core/​providers/​webgpu/​webgpu_execution_provider.h	Makes command recording shareable.
onnxruntime/​core/​providers/​webgpu/​webgpu_execution_provider.cc	Implements legacy Run and replay coordination.
onnxruntime/​core/​providers/​webgpu/​webgpu_context.h	Stores shared legacy state and run gate.
onnxruntime/​core/​providers/​webgpu/​ep/​sync_stream.h	Declares legacy-mode detection.
onnxruntime/​core/​providers/​webgpu/​ep/​sync_stream.cc	Selects mode from runtime version or override.
onnxruntime/​core/​providers/​webgpu/​ep/​README.md	Documents execution and allocator contracts.
onnxruntime/​core/​providers/​webgpu/​ep/​factory.cc	Configures mode-specific allocators.
onnxruntime/​core/​providers/​webgpu/​ep/​api.cc	Logs compatibility-mode activation.
onnxruntime/​core/​providers/​webgpu/​ep/​allocator.h	Updates allocator contract documentation.
onnxruntime/​core/​providers/​webgpu/​compute_context.cc	Uses legacy scratch allocation behavior.
onnxruntime/​core/​providers/​webgpu/​buffer_manager.cc	Clarifies modern allocation behavior.
onnxruntime/​core/​providers/​webgpu/​allocator.h	Documents submission policies.
onnxruntime/​core/​providers/​webgpu/​allocator.cc	Implements legacy clear-submission policy.
cmake/​onnxruntime_providers_webgpu.cmake	Tracks minimum-version file changes.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

You can commit the suggested changes from lintrunner.

[Jiajia Qin (qjia7)]

Could we investigate whether legacy hosts can retain Session-owned recording, rather than introducing a separate context-owned recording path?

ORT 1.24.4 already forwards supplied streams through the batched CopyTensors path used for ordinary framework feed/fetch copies; #32666 fixes the separate single-tensor wrapper. Plain externally usable allocations can submit cached-buffer clears before returning, while Session-local scratch can retain deferred clears without calling the unavailable KernelContext_GetSyncStream API.

This suggests that Session-owned recording, with appropriate legacy allocation behavior and submission boundaries, may support serialized legacy execution and substantially simplify recording ownership, transfer handling, and teardown.

Could you trace the remaining single-copy/null-stream paths and identify a concrete case that cannot be ordered correctly with this approach? If shared recording is required, a focused failing regression would clarify why. Otherwise, could we retain Session-owned recording and keep only the necessary old-host compatibility branches?

Please validate the candidate on an actual 1.24.4 host, covering dirty-buffer BindInput, CPU feeds/fetches, mixed CPU/GPU execution, and capture/replay. This does not assume concurrency support on older cores; their serialization guarantees remain a separate requirement.

[Xiaofei Han (xiaofeihan1)]

Jiajia Qin (@qjia7) Thanks for the detailed suggestion in #33082 (comment). Your distinction between batched CopyTensors and the separate single-tensor wrapper was correct. I traced and exercised the remaining paths, and agree that context-owned recording is not required for serialized compatibility.

The implementation now in this PR (20ff4e2) keeps each recording exclusively owned by its Session. Serialized-mode Session allocations submit cached-buffer clears outside Run and defer them during Run; scratch uses the same ordinary allocator without KernelContext_GetSyncStream. Env allocations remain independent. The policy is now named UseSerializedExecutionMode() and lives in ep/runtime_compatibility.h/.cc, separate from the stream bridge.

One submission boundary is still needed: on 1.24.4, mixed-direction fetches can fall back to the single-tensor wrapper and lose their streams. In a concrete local repro, two independent branches produce a GPU Neg output fetched to a preallocated CPU target and a CPU Neg output fetched to a preallocated GPU target. The GPU input is also preallocated/uploaded before Run. The Session-owned prototype without the pre-copy flush returned 0 instead of -1 on the first iteration. The current implementation passes both output orderings by submitting pending active-Session work before the independent streamless copy. The context holds only a non-owning pointer for that boundary. This demonstrates a need for ordering coordination, not shared recording ownership.

I reran the final merged plugin on an actual ORT 1.24.4 host: all nine local-harness scenarios passed, 20 iterations each, including dirty-buffer BindInput, CPU feeds/fetches, mixed-direction copies, mixed CPU/GPU execution, and scratch reuse. Profiles confirm the intended provider placement. Capture coverage on 1.24.4 is specifically initial capture plus the internal replay; later Runs are ordinary execution because that host lacks the plugin capture/replay callbacks. Repeated host-driven replay is covered separately by the current-host tests, where all 41 serialized and 57 concurrent-mode tests passed. The standalone old-host harness is local, not a checked-in CI test.

The strict caller-serialization requirement remains. The context-wide overlap gate has been removed to avoid stale-gate failures when the host skips OnRunEnd; this does not add concurrency support or claim full failure recovery. I also updated the PR title/description to match the revised design, and closed the duplicate #33228 after incorporating its changes here.

[Jiajia Qin (qjia7)]

LGTM after all CI bots are green.

BTW, should we also cherry-pick #32902 to the 1.28 release branch so concurrent Runs using captured graphs are properly serialized? Since this PR enables concurrent mode on 1.28.3, should we keep that version in serialized mode until a patched release includes the fix?