You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[WebGPU] Accelerated model loading using Win32 and D3D12 - #33055
Large ONNX models often store weights in external files. The existing native WebGPU loading path materializes each weight in tensor-sized, CPU-accessible memory before uploading it to the GPU. For large tensors and multi-gigabyte models, these allocations increase CPU memory pressure, address-space activity, and session creation time.
This change adds an accelerated Windows loading path that streams external weights through a bounded pool of twelve reusable 16 MiB Direct3D 12 (D3D12) upload buffers. File reads and GPU copies operate in chunks, so staging memory remains bounded instead of scaling with individual tensor sizes. The loader copies each chunk into its final GPU-local tensor resource, which WebGPU then imports into Dawn.
File reads, GPU copies, and upload-buffer reuse overlap through Windows events and D3D12 fences. Weight payloads are not materialized in application CPU memory, and the application does not perform an additional CPU payload copy.
Design
The reusable loader, onnxruntime::windows::d3d12::D3D12FileBufferLoader, is part of the shared Windows platform layer. It does not depend on Dawn, WebGPU, ONNX Runtime tensors, or execution-provider interfaces.
The WebGPU integration:
Selects the requested physical adapter through DirectX Graphics Infrastructure (DXGI).
Pins Dawn and the loading device to the same adapter by locally unique identifier (LUID).
Prepares external tensor ranges and invokes the shared loader.
Imports completed D3D12 resources into Dawn.
Retains the D3D12 resources and heaps for the required lifetime.
The shared loader provides:
Overlapped, unbuffered Win32 file reads.
Twelve persistent 16 MiB mapped upload buffers.
Event-driven read completion and D3D12 copy-queue uploads.
Overlap between destination allocation and the initial read wave.
Placed resources packed into heaps targeted at no more than 512 MiB.
Committed resources for tensors that are too large for those heaps.
Support for multiple external-data files and non-contiguous ranges.
GPU-side handling of aligned tensor padding.
Bounds validation, cancellation, error propagation, and explicit resource ownership.
Provider option
Enable the feature with:
{
"weightLoadAcceleration": "preferred"
}
Supported values:
off: Use the existing initializer loading path.
preferred: Use accelerated loading when available; otherwise, fall back to the existing path.
required: Require accelerated loading and return the underlying error if initialization fails.
Cancellation is always propagated and never triggers fallback.
Performance
Measurements used a decoder-only model with:
454 external tensors.
1,291,602,752 bytes of weight data.
452 of 453 adjacent tensor boundaries contiguous.
3,072 non-tensor bytes within the covered file range.
Test system:
Intel Core Ultra 9 285K.
Intel Graphics (0x8086:0x7D67).
NVIDIA GeForce RTX 4070 (0x10DE:0x2709).
SK hynix PC801 NVMe solid-state drive.
ONNX Runtime session creation
Nine fresh onnxruntime_perf_test processes were run for each mode and GPU preference.
GPU preference
GPU
off median
preferred median
Improvement
High performance
NVIDIA GeForce RTX 4070
1.562 s
0.715 s
846 ms / 54.2%
Low power
Intel Graphics
1.420 s
0.669 s
751 ms / 52.9%
Observed ranges, including the first process:
GPU
off range
preferred range
NVIDIA
1.534-2.826 s
0.665-0.840 s
Intel
1.400-2.430 s
0.605-0.783 s
The first off process on each GPU was a cold-start outlier. The required mode uses the same loading path as preferred; only its fallback policy differs.
Standalone comparison with DirectStorage
A 15-repetition comparison alternated the path order, excluded the first run of each path from the steady-state median, and verified every destination byte.
GPU
DirectStorage
This implementation
Difference
NVIDIA
225.097 ms
187.070 ms
38.028 ms / 16.9% faster
Intel
262.044 ms
241.538 ms
20.506 ms / 7.8% faster
Diagnostics indicate that the new path is storage-bound:
GPU
Raw unbuffered read
GPU copy only
Approximate rates
NVIDIA
182.050 ms
52.087 ms
7.10 GB/s read, 24.80 GB/s copy
Intel
210.477 ms
68.945 ms
6.14 GB/s read, 18.73 GB/s copy
Byte-for-byte verification passed for all 1,291,602,752 payload bytes on both loading paths and both GPUs.
Validation
Static, shared, and external Dawn provider and test targets build successfully.
External-data-loader lifecycle and manager tests pass.
WebGPU context and provider tests pass with static and shared Dawn configurations.
The reason will be displayed to describe this comment to others. Learn more.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
The reason will be displayed to describe this comment to others. Learn more.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
The reason will be displayed to describe this comment to others. Learn more.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Setting compile_only here does not guarantee a device-free context because default context 0 is process-global and initialized via call_once; earlier tests may already have created its device. Consequently these three tests do not reliably exercise the advertised no-device paths, and the required test can observe a non-null loader. Follow the established CompileOnlyContextDoesNotCreateDevice pattern and run each case through RunWithFreshDefaultContext(..., /*compile_only_parent=*/true).
In an external-Dawn build, a live default context that was first created with acceleration off can never enable it in a later session. The one-time initialization skips D3D12-device preselection when the first mode is off, and this recovery path is excluded under USE_EXTERNAL_DAWN, so WeightLoadingD3D12Device() remains null and a later required session fails (while the new “enabled after initial off” test merely skips on that failure). Please provide a matching D3D12 device for reused external-Dawn contexts, or reject/document that configuration and make the test assert the intended behavior rather than skipping it.
The reason will be displayed to describe this comment to others. Learn more.
Copilot review overview
🔵 Needs a closer look
The Node.js portability assertion mishandles expected no-GPU errors, and key D3D12 allocation branches lack focused coverage.
Review effort: Balanced Findings: None
Previously missed (2)
In code that hasn't changed since last review
Allow acceleration-specific errors in portability test
js/node/test/api/simple-api-tests.ts:115
On Windows builds with D3D12 file loading but no usable GPU, off fails with Dawn's adapter error, while required can fail earlier with the expected acceleration-specific DXGI/D3D12 error. Because unavailableProviderError is then set, this branch incorrectly requires those distinct valid errors to be identical, so the portability test fails on the exact device-unavailable configuration it intends to tolerate. Allow required to report either the baseline provider error or an acceleration-specific error.
Test heap rollover and committed-resource loading paths
The only direct loader test uses the default 512 MiB heap limit, so it exercises a placed resource in one heap but never this committed-resource branch or the rollover to a second heap. These are advertised loading paths with distinct ownership/allocation behavior. Add a focused test using a small max_destination_heap_size (for example 64 KiB) and small ranges that force both heap rollover and a committed destination, then verify every output byte.
The reason will be displayed to describe this comment to others. Learn more.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
The reason will be displayed to describe this comment to others. Learn more.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
The reason will be displayed to describe this comment to others. Learn more.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Mark the allocator parameter unused when D3D12 file loading is disabled so warning-as-error builds compile cleanly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use explicit source, read-plan, staging, and destination names. Store destination resources directly in BufferCollection. Detach destination resources when GPU completion cannot be tracked, and reject further loads.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Make the allocator-aware LoadTensor overload available to WebAssembly loaders while restricting batch preparation hooks to D3D12 file loading.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Explain why copy encoding shares the recording lock with public allocator operations.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove the test that shares one unsupported low-level command recording state across threads. Retain the independent data transfer coverage that matches the supported ownership model.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Propagate the WebGPU context ID through the accelerated allocator and external loader so their memory identity matches the current main branch interfaces.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Preserve uncaptured error and device-loss state for checked completion, and apply the required C++ formatting.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Route memory-backed external data through ordinary loaders while excluding tensor-creating file loaders such as D3D12 acceleration.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…gistration
Both WebGPU compile variants reject ModelPath at the D3D12 candidate-registration call added by upstream microsoft#33055. Integrate the exact already-tested upstream base 0066c38, align the helper declaration and implementation with ModelPath, and explicitly project Path() only for existing pathname parsing. Retain the ModelPath object for file validation rather than introducing an implicit conversion that loses its captured-directory context. Add a feature-guarded candidate-registration regression. Full production caller/helper and regression translation units compile with ENABLE_D3D12_FILE_LOADING and MSVC /W4 /WX.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Description
Large ONNX models often store weights in external files. The existing native WebGPU loading path materializes each weight in tensor-sized, CPU-accessible memory before uploading it to the GPU. For large tensors and multi-gigabyte models, these allocations increase CPU memory pressure, address-space activity, and session creation time.
This change adds an accelerated Windows loading path that streams external weights through a bounded pool of twelve reusable 16 MiB Direct3D 12 (D3D12) upload buffers. File reads and GPU copies operate in chunks, so staging memory remains bounded instead of scaling with individual tensor sizes. The loader copies each chunk into its final GPU-local tensor resource, which WebGPU then imports into Dawn.
File reads, GPU copies, and upload-buffer reuse overlap through Windows events and D3D12 fences. Weight payloads are not materialized in application CPU memory, and the application does not perform an additional CPU payload copy.
Design
The reusable loader,
onnxruntime::windows::d3d12::D3D12FileBufferLoader, is part of the shared Windows platform layer. It does not depend on Dawn, WebGPU, ONNX Runtime tensors, or execution-provider interfaces.The WebGPU integration:
The shared loader provides:
Provider option
Enable the feature with:
{ "weightLoadAcceleration": "preferred" }Supported values:
off: Use the existing initializer loading path.preferred: Use accelerated loading when available; otherwise, fall back to the existing path.required: Require accelerated loading and return the underlying error if initialization fails.Cancellation is always propagated and never triggers fallback.
Performance
Measurements used a decoder-only model with:
Test system:
0x8086:0x7D67).0x10DE:0x2709).ONNX Runtime session creation
Nine fresh
onnxruntime_perf_testprocesses were run for each mode and GPU preference.offmedianpreferredmedianObserved ranges, including the first process:
offrangepreferredrangeThe first
offprocess on each GPU was a cold-start outlier. Therequiredmode uses the same loading path aspreferred; only its fallback policy differs.Standalone comparison with DirectStorage
A 15-repetition comparison alternated the path order, excluded the first run of each path from the steady-state median, and verified every destination byte.
Diagnostics indicate that the new path is storage-bound:
Byte-for-byte verification passed for all 1,291,602,752 payload bytes on both loading paths and both GPUs.
Validation