[WebGPU] Accelerated model loading using Win32 and D3D12

[Sushanth Rajasankar (sushraja-msft)]

Description

This change adds accelerated external-weight loading for the native Windows WebGPU execution provider using documented Win32 and D3D12 APIs.

External initializers are loaded through documented Win32 and D3D12 APIs:

ReadFile(file, persistently mapped D3D12 upload memory)
  -> D3D12 copy queue / CopyBufferRegion
  -> default-heap tensor resource
  -> Dawn shared-buffer import

This avoids materializing weight payloads in application CPU memory and performs no application CPU payload copy.

Implementation

The reusable implementation is in the shared Windows platform layer as onnxruntime::windows::d3d12::D3D12FileBufferLoader. It has no dependency on Dawn, WebGPU, ORT tensors, or execution-provider interfaces. A separate WebGPU adapter prepares tensor ranges, invokes the shared loader, imports the D3D12 resources into Dawn, and owns their lifetime.

The loader includes:

• Overlapped, unbuffered Win32 reads.
• A persistent ring of twelve 16-MiB mapped upload buffers.
• Event-driven read completion.
• D3D12 copy-queue uploads and final resource-state transitions.
• Placed destination resources packed into heaps targeted at no more than 512 MiB.
• Committed-resource fallback for larger individual tensors.
• Overlap between destination allocation and the initial file-read wave.
• Support for multiple external-data files and non-contiguous ranges.
• GPU copies for aligned tensor padding, preserving existing bytes without a CPU payload copy.
• Cancellation, bounds validation, error propagation, and explicit heap/resource lifetime ownership.

WebGPU selects the requested physical adapter through DXGI and pins Dawn to the same adapter by LUID so the loading device and WebGPU device always refer to the same GPU. Initialization is synchronous and uses one accelerated loading path.

Provider option

"weightLoadAcceleration": "preferred"

Supported values:

• off: use the existing initializer path.
• preferred: attempt D3D12 accelerated loading and fall back to ordinary loading if it is unavailable.
• required: require D3D12 accelerated loading and report the underlying error.

Cancellation is always propagated and is never converted into a successful fallback.

Motivation

Large models commonly store most parameters as external ONNX initializer data. The ordinary native WebGPU path materializes those bytes in CPU-accessible memory and then uploads each initializer to a Dawn buffer. For multi-gigabyte models this adds CPU memory pressure, address-space activity, and extra copying to session creation.

The new shared loader keeps payload bytes in a bounded D3D12 upload ring and transfers them into GPU-local tensor resources without requiring an additional storage runtime deployment. The shared placement also allows other Windows D3D12-based execution providers to reuse the loader.

Measurements

Measurements used an approximately 2-billion-parameter-class decoder-only model containing:

• 454 external tensors.
• 1,291,602,752 payload bytes.
• 452 of 453 adjacent tensor boundaries contiguous.
• 3,072 non-tensor bytes inside the covered file range.

Test system:

• Intel Core Ultra 9 285K.
• Intel Graphics (0x8086:0x7D67).
• NVIDIA GeForce RTX 4070 (0x10DE:0x2709).
• SK hynix PC801 NVMe SSD.

ONNX Runtime session creation

Nine fresh onnxruntime_perf_test processes were run for each mode and GPU preference:

GPU preference	GPU	off median	preferred median	Improvement
High performance	NVIDIA GeForce RTX 4070	1.562 s	0.715 s	846 ms / 54.2%
Low power	Intel Graphics	1.420 s	0.669 s	751 ms / 52.9%

Observed ranges, including the first process:

GPU	off range	preferred range
NVIDIA	1.534-2.826 s	0.665-0.840 s
Intel	1.400-2.430 s	0.605-0.783 s

The first off process on each GPU was a cold-start outlier. required uses the same data path as preferred and differs only in fallback policy.

Standalone loader versus DirectStorage

A standalone 15-repetition comparison used alternating path order, excluded each path's first run from the steady-state median, and verified every destination byte:

GPU	DirectStorage	Win32 + D3D12 loader	Difference
NVIDIA	225.097 ms	187.070 ms	38.028 ms / 16.9% faster
Intel	262.044 ms	241.538 ms	20.506 ms / 7.8% faster

Isolated diagnostics show that the loader is storage-bound:

GPU	Raw unbuffered read	GPU copy only	Approximate rates
NVIDIA	182.050 ms	52.087 ms	7.10 GB/s read, 24.80 GB/s copy
Intel	210.477 ms	68.945 ms	6.14 GB/s read, 18.73 GB/s copy

Full byte-for-byte verification passed for all 1,291,602,752 payload bytes on both paths and both GPUs.

Validation

• Static Dawn provider and test targets build successfully.
• External Dawn provider and test targets build successfully.
• Shared Dawn provider and test targets build successfully.
• External-data-loader lifecycle and manager tests pass.
• WebGPU context/provider tests pass in static and shared Dawn configurations; device-precondition tests skip where expected.
• git diff --check passes.

[Copilot]

🟡 Changes recommended

Build-definition propagation, required-pipelined semantics, cleanup, empty tensors, and coverage have unresolved issues.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds experimental DirectStorage-based external-weight loading for native Windows WebGPU.

Changes:

• Adds batched disk-to-GPU loading and Dawn resource import.
• Extends external-data loader lifecycle and asynchronous WebGPU initialization.
• Adds configuration, packaging, benchmark support, and tests.

File summaries

File	Description
tools/nuget/generate_nuspec_for_native_nuget.py	Packages DirectStorage runtimes.
onnxruntime/test/providers/webgpu/webgpu_context_test.cc	Tests acceleration modes.
onnxruntime/test/perftest/ort_test_session.cc	Passes WebGPU benchmark options.
onnxruntime/test/framework/external_data_loader_test.cc	Tests loader batch lifecycle.
onnxruntime/core/session/inference_session.h	Tracks preload state.
onnxruntime/core/session/inference_session.cc	Starts external-data preloading.
onnxruntime/core/providers/webgpu/webgpu_provider_options.h	Defines acceleration policies.
onnxruntime/core/providers/webgpu/webgpu_provider_factory.cc	Parses the new option.
onnxruntime/core/providers/webgpu/webgpu_execution_provider.h	Adds DirectStorage state.
onnxruntime/core/providers/webgpu/webgpu_execution_provider.cc	Integrates loader and allocator.
onnxruntime/core/providers/webgpu/webgpu_context.h	Adds asynchronous initialization state.
onnxruntime/core/providers/webgpu/webgpu_context.cc	Implements pipelined device initialization.
onnxruntime/core/providers/webgpu/external_data_loader.h	Updates loader signature.
onnxruntime/core/providers/webgpu/external_data_loader.cc	Adapts WebAssembly loader.
onnxruntime/core/providers/webgpu/direct_storage_external_data_loader.h	Declares DirectStorage loading APIs.
onnxruntime/core/providers/webgpu/direct_storage_external_data_loader.cc	Implements batched loading and import.
onnxruntime/core/providers/webgpu/data_transfer.h	Defers buffer-manager lookup.
onnxruntime/core/providers/webgpu/data_transfer.cc	Uses deferred lookup.
onnxruntime/core/providers/webgpu/allocator.h	Defers UMA detection.
onnxruntime/core/providers/webgpu/allocator.cc	Initializes allocation mode lazily.
onnxruntime/core/providers/js/external_data_loader.h	Updates JS loader signature.
onnxruntime/core/providers/js/external_data_loader.cc	Adapts JS loader implementation.
onnxruntime/core/framework/tensorprotoutils.h	Adds external-data preparation API.
onnxruntime/core/framework/tensorprotoutils.cc	Validates and prepares external tensors.
onnxruntime/core/framework/session_state_utils.cc	Batches initializer preparation/loading.
onnxruntime/core/framework/external_data_loader.h	Extends loader lifecycle interface.
onnxruntime/core/framework/external_data_loader.cc	Provides default lifecycle hooks.
onnxruntime/core/framework/external_data_loader_manager.h	Adds preload and batch management.
onnxruntime/core/framework/external_data_loader_manager.cc	Coordinates loader lifecycles.
cmake/onnxruntime_providers_webgpu.cmake	Configures DirectStorage dependencies.
cmake/CMakeLists.txt	Adds DirectStorage build options.

Review details

• Files reviewed: 31/31 changed files
• Comments generated: 6
• Review effort level: Balanced

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

Cancellation propagation, queue sizing, preload filtering, and initialization portability contain blocking defects.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 31/31 changed files
• Comments generated: 5
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

Cancellation can be lost, a lazy callback can outlive its provider, and the feature lacks an enabled CI build.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 31/31 changed files
• Comments generated: 3
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

Preloading breaks caller-supplied external initializer overrides, and packaged feature availability is inconsistent across Windows Python wheels.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 32/32 changed files
• Comments generated: 2
• Review effort level: Balanced

[Copilot]

🟡 Changes recommended

The Windows wheel omits required runtime DLLs, and unresolved loader compatibility, boolean normalization, and cancellation defects remain.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (4)

Previously missed (2) — in code that hasn't changed since the last review.

cmake/onnxruntime_providers_webgpu.cmake:16

• This gate still permits onnxruntime_USE_EP_API_ADAPTERS, but the OrtEp adapter has no external-data-loader callback/bridge: the wrapped WebGPU provider's GetExternalDataLoader() is never registered with InferenceSession. In that build, every acceleration mode is ineffective and even required silently uses ordinary loading. Either implement an EP-API bridge or reject adapter/plugin builds here.
  onnxruntime/core/providers/webgpu/direct_storage_external_data_loader.cc:728
• The zero-request preload fast path never evaluates is_cancelled, so an all-empty external-initializer model can report successful preload even when cancellation is already requested. This contradicts the stated cancellation guarantee; continue initialization, but return MODEL_LOAD_CANCELED when the callback is true.

This issue also appears in the following locations of the same file:

• line 807
• line 860

onnxruntime/core/providers/webgpu/direct_storage_external_data_loader.cc:819

• The zero-request final batch likewise ignores the current cancellation callback. If there was no non-empty preload (for example, all external tensors are empty), cancellation requested after preparation is converted into success. Check is_cancelled in this branch and return MODEL_LOAD_CANCELED through fail_or_fallback.

  if (batch.request_count == 0) {
    common::Status preload_status = common::Status::OK();
    if (impl_->preload_future.valid()) {
      preload_status = impl_->preload_future.get();
    }
    impl_->preload_batch.reset();
    impl_->context.ContinueInitialize();
    if (!preload_status.IsOK()) {
      return fail_or_fallback(preload_status);
    }
    impl_->context.WaitForInitializeComplete();
    batch.finalized = true;
    return common::Status::OK();

onnxruntime/core/providers/webgpu/direct_storage_external_data_loader.cc:864

• After the DirectStorage transfer completes, cancellation is no longer polled while this potentially long Dawn initialization wait and the subsequent per-tensor import run. In pipelined mode, a cancellation arriving during that interval is therefore returned as a successful load. Recheck the callback after the wait (and during a large import loop) and propagate MODEL_LOAD_CANCELED via fail_or_fallback.

  impl_->context.WaitForInitializeComplete();

  const auto import_start = Clock::now();
  const auto import_status = [&]() -> common::Status {
    ORT_RETURN_IF_NOT(

• Files reviewed: 32/32 changed files
• Comments generated: 2
• Review effort level: Balanced

[Xavier Dupré (xadupre)]

I'm also trying to improve the loading time on GPU, see #32437. I'm using pinned buffers. I was planning to use DirectStorage APIs next.

[Sushanth Rajasankar (sushraja-msft)]

I'm also trying to improve the loading time on GPU, see #32437. I'm using pinned buffers. I was planning to use DirectStorage APIs next.

nice, let me know if the config options can be used in CUDA as well or if you would want it named differently.
"weightLoadAcceleration": "preferred-pipelined"

the difference between preffered vs preffered-pipelined is just that device feature querying can be parallelized with the weight loads. So pipelining does just that. required more throws an error if anything fails with direct storage based weight loading instead of falling back which is what the prefered mode does.

My change here also contains an extension point to
"An extended external-data-loader lifecycle that allows ORT to prepare and finalize an initializer batch without affecting other execution providers." does that work for CUDA as well or if we need another primitive that works across EPs we can collaborate on that.

[Copilot]

🟡 Changes recommended

Asynchronous initialization has a data race, and aborting a failed load can still wait for the entire preload.

Get a fresh assessment by requesting another Copilot review.

Review details

• Files reviewed: 34/34 changed files
• Comments generated: 2
• Review effort level: Balanced (auto)

Note

Copilot is running an experiment and ran this review at Balanced.

[github-actions]

You can commit the suggested changes from lintrunner.

[Sushanth Rajasankar (sushraja-msft)]

Superseded by #33055, which presents the final Win32 and D3D12 implementation as a clean change based on current main.