[WebGPU] Support concurrent use with explicit Session streams

[Xiaofei Han (xiaofeihan1)]

Summary

Support concurrent use of independent WebGPU Sessions by separating Session command recording from context-shared device resources. The plugin uses explicit Session-owned streams rather than TLS or creation-thread-based transfer routing. This PR develops the explicit-stream alternative to #29851; it is no longer a frozen copy of the original historical implementation.

• Keep command recording per Session while sharing the default and initializer BufferManagers at the WebGpuContext level. Graph capture routes allocations to Session-owned per-graph BufferManagers during the relevant Run. Protect mutable recording and shared cache state separately.
• Implement OrtEp::CreateSyncStreamForDevice. Each plugin stream references its owning EP's recording; it does not create a separate WebGPU queue. Stream-bearing transfers preserve that ordering, including BindInput copies, while graph Memcpy kernels access their owning EP directly. No new public EP API is introduced.
• Keep streamless Env transfers on independent recording state. Stream flush and notification activation conservatively wait for completion; streamless GPU-to-GPU copies submit and wait before returning.
• Keep the plugin-only stream/notification bridge and Session allocator ABI wrapper in webgpu/ep. Native/built-in WebGPU uses EP recording directly and excludes these plugin adapters.
• Route plugin kernel scratch allocation in CreateGPUTensor through the kernel's explicit sync stream, preserving clear/copy/dispatch ordering without submitting each scratch allocation. Move plugin regression coverage into onnxruntime_provider_test and serialize shared shader-dump writes.

Allocator ownership and ordering

Env and Session device allocators reuse GpuBufferAllocator, held by AllocatorPtr; the ABI wrappers differ because only the Session wrapper exposes stream allocation.

Aspect	Plugin Env shared allocator	Plugin Session device allocator
Outer class (OrtAllocator)	onnxruntime::ep::adapter::Allocator	WebGpuSessionAllocator
Recording / lifetime	Getter owns independent recording state and retains the context; no Session required	Getter borrows the owning EP's recording; the Session must outlive allocator use and tensor frees
BufferManager	Context-shared default manager	Context-shared default manager, or Session-owned per-graph manager during a capture-enabled Run
Plain Alloc / null stream	Submit cached-buffer clears before returning; no AllocOnStream callback exposed	Submit cached-buffer clears before returning, including during Run
Matching AllocOnStream	Not exposed	Defer cached-buffer clears on the same Session recording; reject another Session's stream

The distinction is stream-ordered versus streamless allocation, not internal versus external API use. A shared buffer cache does not imply shared recording. Callers must operate on independent tensors or synchronize writes before another Session consumes shared data.

Related: #32561 #31627.

Validation

Local Windows validation of the allocator refactor:

• Reconfigured and built onnxruntime_provider_test for both plugin and native/built-in WebGPU configurations.
• Plugin: 34 enabled tests passed in two consecutive CTest runs, with no skips; two pre-existing disabled tests remained disabled.
• Native/built-in: 9 targeted allocator, transfer, and deferred-dispatch ordering tests passed, with no skips.
• The two new Env allocator regressions also passed in a separate run with fixture-owned plugin registration: tensors/allocator outliving a Session, and reused-buffer clears submitted without a Session. Coverage also checks that the Env allocator does not expose AllocOnStream.

Plugin coverage includes CPU I/O, graph-internal CPU/GPU copies, mixed feed copies, CPU outputs bound to GPU, concurrent Sessions, and same-Session/dedicated-Session allocator concurrency. This is targeted local validation, not a full-suite or current CI-success claim, and does not establish parallel Run execution on one Session.

These build/test runs preceded the final minimum-version change to 1.30.1. That change was checked at source/configuration level; the plugin was not rebuilt afterward or tested against an actual ORT 1.30.1 runtime.

Limitations and deferred work

• Runtime requirement: The WebGPU plugin minimum is now ONNX Runtime 1.30.1 (plugin-ep-webgpu/MIN_ONNXRUNTIME_VERSION), with stream support required. This replaces the previous 1.31.0 minimum; actual 1.30.1 runtime compatibility remains unvalidated locally.
• Native device requirement: Dawn's ImplicitDeviceSynchronization feature must be enabled. ORT requests it for internally created devices; applications providing an external device must request it when creating that device. Initialization rejects native devices without it, even for serial use. This requirement does not apply to WASM.
• Session-owned streams only: WebGPU's factory-level stream creation remains unsupported. Ordinary application Alloc/CreateTensor APIs do not accept a stream; the explicit-stream path is the existing framework/plugin API, not a new application Session-stream accessor.

Known correctness issues not addressed in this PR

• Built-in EP: external Session-allocator use overlapping a Run on the same Session (C3). Plain allocations through OrtApi::CreateAllocator followed by streamless OrtEnv::CopyTensors uploads are not supported while the allocator-owning Session has an active Run. A reused buffer's clear can remain deferred on the Session recording while the upload is submitted through the Env's separate recording; the later clear can overwrite the uploaded data. The ordering fix and a public-API regression remain deferred. This specific limitation is in the built-in EP path: plugin Session allocators submit plain Alloc clears before returning and defer only stream-ordered AllocOnStream calls.

• Concurrent Session execution with profiling enabled (C4). Profiling query resources, pending events, timestamp offsets, and lifecycle state remain shared at the WebGpuContext level. Per-Session recording locks do not isolate them, so overlapping profiled Sessions can race and mix or lose GPU events. State/lifecycle isolation or an explicit serialized/rejected-overlap policy, with regression coverage, remains deferred.

These are known correctness limitations, not runtime-enforced exclusions. Applications must avoid these overlaps until follow-up fixes are in place.

Other out-of-scope or unvalidated scenarios

• Concurrent graph capture, cross-device transfer, and arbitrary foreign stream overrides are not established by this validation.
• Parallel Run execution on a single Session is not claimed.
• Conservative waits and plain Session allocations can flush or wait for pending work. Correctness isolation does not guarantee execution overlap or freedom from contention; throughput and latency require separate performance validation.

[Copilot]

🟡 Changes recommended

Critical and moderate review findings remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds per-Session WebGPU command recording and stream-aware allocation/transfer handling while synchronizing shared context resources.

Changes:

• Isolates Session recording and graph-capture state.
• Adds explicit sync streams and stream-aware allocator behavior.
• Adds native and AutoEP concurrency coverage.

Unresolved findings affect allocator exception safety, dispatch ordering, synchronization, stream registration, error-scope ownership, and graph-copy routing.

File summaries

File	Summary
onnxruntime/test/providers/webgpu/webgpu_context_test.cc	Updates context tests for explicit recordings.
onnxruntime/test/providers/webgpu/concurrent_context_test.cc	Adds shared-context concurrency coverage.
onnxruntime/test/autoep/test_webgpu_concurrency.cc	Adds AutoEP concurrency coverage.
onnxruntime/core/providers/webgpu/webgpu_provider_factory.cc	Adds plugin transfer and allocator integration.
onnxruntime/core/providers/webgpu/webgpu_kernel.cc	Integrates kernel recording and graph-copy routing.
onnxruntime/core/providers/webgpu/webgpu_execution_provider.h	Defines Session recording ownership.
onnxruntime/core/providers/webgpu/webgpu_execution_provider.cc	Handles EP lifecycle, recording, flushing, and Memcpy routing.
onnxruntime/core/providers/webgpu/webgpu_context.h	Defines recording-aware context state.
onnxruntime/core/providers/webgpu/webgpu_context.cc	Implements recording, flushing, and capture handling.
onnxruntime/core/providers/webgpu/session_buffer_pool.cc	Handles Session buffer-pool management.
onnxruntime/core/providers/webgpu/program_manager.h	Updates pipeline-cache interfaces.
onnxruntime/core/providers/webgpu/program_manager.cc	Synchronizes shared pipeline-cache access.
onnxruntime/core/providers/webgpu/ep/README.md	Documents Session streams and limitations.
onnxruntime/core/providers/webgpu/ep/factory.cc	Registers plugin stream integration.
onnxruntime/core/providers/webgpu/ep/ep.h	Declares EP sync-stream interfaces.
onnxruntime/core/providers/webgpu/ep/ep.cc	Implements sync-stream creation.
onnxruntime/core/providers/webgpu/data_transfer.h	Defines stream-bound transfer interfaces.
onnxruntime/core/providers/webgpu/data_transfer.cc	Implements stream and notification handling.
onnxruntime/core/providers/webgpu/compute_context.h	Updates recording-aware compute helpers.
onnxruntime/core/providers/webgpu/compute_context.cc	Supports per-recording compute encoders.
onnxruntime/core/providers/webgpu/buffer_manager.h	Defines recording-scoped buffer APIs and synchronization.
onnxruntime/core/providers/webgpu/buffer_manager.cc	Synchronizes shared buffer-cache operations.
onnxruntime/core/providers/webgpu/allocator.h	Declares stream-aware allocator interfaces.
onnxruntime/core/providers/webgpu/allocator.cc	Implements recording-aware and stream-aware allocation.

Review details

Suppressed comments (4)

onnxruntime/core/providers/webgpu/buffer_manager.cc:545

• The GPU-to-GPU copy is recorded before recording.deferred_dispatches is encoded. For a graph path that copies a value after a deferred compute dispatch, Flush will submit this copy before that compute, so the destination can contain stale data. Encode the deferred dispatches before recording CopyBufferToBuffer, matching Download and cached-buffer clearing.

  auto& command_encoder = context_.GetCommandEncoder(recording);
  context_.EndComputePass(recording);
  command_encoder.CopyBufferToBuffer(src, 0, dst, 0, copy_size);

onnxruntime/core/providers/webgpu/ep/factory.cc:296

• IsStreamAwareImpl() now advertises factory-level stream support, but the callback used by the public CreateSyncStreamForEpDevice API still unconditionally returns ORT_NOT_IMPLEMENTED. Internal session setup calls Ep::CreateSyncStreamForDeviceImpl and therefore hides this mismatch, while applications creating a stream from an OrtEpDevice will fail despite the advertised capability. Please implement the factory callback (or update the awareness/registration contract so this public capability is not advertised while EP-level session streams remain registered).

  return true;
}

OrtStatus* ORT_API_CALL Factory::CreateSyncStreamForDeviceImpl(
    OrtEpFactory* /*this_ptr*/,
    const OrtMemoryDevice* /*memory_device*/,
    const OrtKeyValuePairs* /*stream_options*/,
    OrtSyncStreamImpl** stream) noexcept {
  EXCEPTION_TO_RETURNED_STATUS_BEGIN
  *stream = nullptr;
  return Api().ort.CreateStatus(ORT_NOT_IMPLEMENTED,
                                "CreateSyncStreamForDevice is not implemented for this EP factory.");

onnxruntime/core/providers/webgpu/webgpu_context.h:114

• This mutex only serializes one Session's recording, while WebGpuContext::PushErrorScope/PopErrorScope operate on the shared WGPUDevice error-scope stack. OnRunStart and OnRunEnd call them for every Session, so interleaved runs can pop another Session's scope, misattribute validation errors, or fail to pop. Protect the device-wide scope lifetime (or disable per-Session scopes when a device is shared); locking this per-recording mutex alone cannot establish the required LIFO pairing.

  std::recursive_mutex mutex;

onnxruntime/core/providers/webgpu/webgpu_execution_provider.cc:880

• OnRunStart pushes a validation error scope on context_.Device() and this path later pops it, but that scope stack is shared by every Session using context 0. Two runs can interleave A.Push, B.Push, A.Pop, B.Pop, causing each run to consume the other's validation errors and report a failure to the wrong caller; implicit device synchronization does not make the nesting per-session. The per-recording lock here cannot fix that, so serialize the complete error-scope lifetime or disable per-run scopes for shared concurrent contexts.

  Status flush_status = context_.Flush(BufferManager(), *recording_);

• Files reviewed: 24/24 changed files
• Comments generated: 4
• Review effort level: Lite (auto)

Note

Copilot is running an experiment and ran this review at Lite.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Xiaofei Han (xiaofeihan1)]

/run bots

[Jiajia Qin (qjia7)]

The per-Session recording and explicit-stream direction is appropriate, and the latest commits resolve the plugin build/test-coverage issue. Two correctness issues still block merge: native streamless Session allocations can submit their cached-buffer clear after caller data is uploaded, and profiling state remains shared across otherwise concurrent Sessions. There is also a conditional shader-dump race. Details are inline.

[Xiaofei Han (xiaofeihan1)]

I will add a version check to support 1.28.3 in the following PR.

[Xiaofei Han (xiaofeihan1)]

Reopen this PR to re-trigger the pipepline.

[Jiajia Qin (qjia7)]

Let's merge this first and handle the follow-up work separately. Thanks.

[Enes (altunenes)]

I confirmed fixed with ONNX Runtime 1.31. On macOS 26.5 (Apple Silicon),my repro no longer crashes and all nodes run on the WebGPU EP. Thank you so much for this!
closing:
#31627
#27592